ScaleAcross Explorer Optimizes Cross-Data Center AI Training

Meta has released a research paper detailing ScaleAcross Explorer, an optimizer aimed at enhancing communication efficiency during the training of large language models across various data centers. This paper, available on arXiv, tackles the increasing intricacies of distributing GPU resources across multiple locations, a process referred to as "scale-across" training. It identifies three critical design aspects: parallelism placement, scheduling, and network layer technologies. By taking a comprehensive approach to these factors, the optimizer seeks to expedite the exploration of the design landscape and facilitate effective training for cutting-edge models. The findings are based on Meta's extensive experience managing hundreds of thousands of GPUs in numerous data centers.