Structured Reasoning Signals, Not Code, Improve Mathematical Reasoning in LMs

A recent study published on arXiv questions the belief that incorporating code enhances general reasoning in language models. Researchers conducted controlled pretraining using a 10-trillion-token dataset with distinct domain separation and discovered that executable code does not improve reasoning abilities; rather, it competes with knowledge-heavy tasks, including advanced mathematics. The reasoning improvements typically linked to code arise from structured reasoning across different domains, like combinations of code and text or math and text. By increasing the proportion of structured math-domain examples within a set math budget, significant improvements were observed in challenging mathematical reasoning tasks.