Mixture of Training: Modular Pretraining for Language Models

A recent study published on arXiv (2608.13277) presents Mixture of Training (MoT), a technique designed to break down the pre-training of language models into smaller, independently trainable tasks that can later be integrated into a unified model. This method divides a target Transformer into sequential layer blocks, training each block within a static pretrained aligner framework, and subsequently recombines the trained blocks with an optional brief end-to-end adaptation phase. The authors provide a proof of concept using a 1.3B-parameter Gemma-style model trained on C4, demonstrating that independently trained depth slices can form a functional language model. A quality-parity schedule achieves the same perplexity as the monolithic baseline while processing a greater number of tokens and exhibiting a shorter idealized layer-equivalent critical path post-aligner preparation. The effective computational benefit relies on reusing the aligner across various training sessions. This paper serves as a cross-type announcement and can be accessed via the given URL.