Mahmoud Assran

A Closer Look at Codistillation for Distributed Training

2020 · arXiv (Cornell University)

Codistillation has been proposed as a mechanism to share knowledge among concurrently trained models by encouraging them to represent the same function through an auxiliary loss. This contrasts with the more commonly used fully-synchronous data-parallel …