LoopUS: Post-Training Framework Converts LLMs into Looped Architectures
A team of researchers has introduced an innovative method known as Looped Depth Up-Scaling (LoopUS) that enhances large language models by restructuring them into looped formats. This method, which significantly reduces the need for extensive retraining, is built around three components: an encoder, a looped reasoning unit, and a decoder. LoopUS aims to boost reasoning performance while maintaining efficiency. The technique includes breaking down information into stages, utilizing a selective gate for hidden-state management, and incorporating random deep supervision to optimize memory during complex tasks. The full details are available in the study on arXiv, ID 2605.11011.
Key facts
- LoopUS stands for Looped Depth Up-Scaling.
- It is a post-training framework for converting pretrained LLMs into looped architectures.
- The framework recasts the LLM into an encoder, a looped reasoning block, and a decoder.
- Four core components are used: block decomposition, selective gate, random deep supervision, and additional mechanisms.
- Block decomposition is guided by staged representation dynamics.
- The selective gate is input-dependent and mitigates hidden-state drift.
- Random deep supervision enables memory-efficient learning over long recursive horizons.
- The work is published on arXiv with ID 2605.11011.
Entities
Institutions
- arXiv