Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

A recent paper on arXiv (2608.02829) explores the feasibility of transforming a pretrained large transformer model into a smaller version, specifically examining the transition from 1.4B to 410M parameters within the Pythia series. The findings reveal a strong alignment in representations across different sizes (ridge R^2=0.84), although parameter alignment is weak. The study indicates that dense weight projection disrupts functionality, not merely an assembly issue, as it interferes with rotary, per-head, GELU, and LayerNorm structures. After utilizing the optimal linear operator, weight residuals appeared statistically similar to noise under shuffle controls, suggesting that the conversion's effectiveness is rooted in initialization. In matched-budget continued pre-training, the conversion is divided into two distinct components: least-squares compensation (function: optimal zero-shot) and variance-preserving rescale (dynamics: optimal endpoints). Compensation proves to be a token-efficient, low-cost advantage rather than a universal solution, demonstrating benefits at 30M tokens. This research, dated August 2026, enhances the understanding of model conversion and transfer learning in AI, with significant implications for efficient model deployment and training methodologies.