Kha Pham

On the Role of Discrete Representation in Sparse Mixture of Experts

2024 · arXiv (Cornell University)

Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to relevant experts; …