Pavel Kisilev

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training

2024 · arXiv (Cornell University)

Distributed machine learning has recently become a critical paradigm for training large models on vast datasets. We examine the stochastic optimization problem for deep learning within synchronous parallel computing environments under communication constraints. While averaging …