Johannes Hagemann

Efficient Parallelization Layouts for Large-Scale Distributed Model Training

2023 · arXiv (Cornell University)

Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work …