Pierre Tholoniat

Differentially Private Training of Mixture of Experts Models

2024 · arXiv (Cornell University)

This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of …