Xiyang Dai

Residual Mixture of Experts

2022 · arXiv (Cornell University)

Mixture of Experts (MoE) is able to scale up vision transformers effectively. However, it requires prohibiting computation resources to train a large MoE transformer. In this paper, we propose Residual Mixture of Experts (RMoE), an …