Weiyang Wang

Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters

2024

This paper presents a low-cost network architecture for training large language models (LLMs) at hyperscale. We study the optimal parallelization strategy of LLMs and propose a novel datacenter network design tailored to LLM's unique communication …