ProxyCoT: Boosting Long-Context Reasoning in LLMs via Short Proxy Contexts

A novel training approach known as ProxyCoT enhances long-context reasoning in large language models (LLMs) by enabling the transfer of reasoning skills from brief proxy contexts to comprehensive long contexts. While existing LLMs can manage up to 10 million tokens, they struggle with intricate reasoning tasks involving lengthy sequences. ProxyCoT initially creates high-quality chain-of-thought reasoning traces within short proxy contexts, utilizing reinforcement learning or distillation from a larger teacher model. These traces are subsequently integrated into full long contexts via supervised fine-tuning. Experiments indicate that ProxyCoT consistently surpasses robust baselines while minimizing computational expenses. This method effectively bridges the performance divide between proxy and full contexts, which rely on the same foundational reasoning process. The research can be found on arXiv under ID 2605.20201.