Hanyu Yu

AttnRoute-MoE: Attention-Prior Routing for Mixture-of-Experts Vision Transformers

2026 · Zenodo (CERN European Organization for Nuclear Research)

Mixture-of-Experts (MoE) Vision Transformers improve Transformer efficiency by routing each image patch token to a sparse subset of specialized feed-forward network experts. Standard learned routers produce expert collapse early in training—one or two experts absorb …