Test-Time Scaling Fails to Improve Accuracy in Knowledge-Intensive Tasks
A recent paper from arXiv (paper 2509.06861) questions the utility of test-time scaling in reasoning models for tasks requiring extensive knowledge. This study assesses 14 reasoning models using different test-time scaling techniques and reveals that increasing computation during inference does not reliably enhance accuracy; in fact, it often results in more hallucinations. The researchers note that longer reasoning chains lead to more attempts, many of which are erroneous, and that prolonged reasoning can solidify initial false beliefs with invented details, aligning with confirmation bias. The research offers an information-theoretic viewpoint on the limitations of test-time scaling for closed-book knowledge-intensive tasks, despite its effectiveness in other areas. The paper was announced as a replacement on arXiv, emphasizing the need to reevaluate the advantages of extra computation in these scenarios.
Key facts
- The study evaluates 14 reasoning models under different test-time scaling strategies.
- Increasing test-time computation does not consistently improve accuracy.
- Longer reasoning often leads to more hallucinations.
- Changes in hallucination rates are driven by the model's willingness to answer.
- Extended reasoning can reinforce early incorrect beliefs with fabricated details.
- The paper provides an information-theoretic perspective.
- The paper is available on arXiv with ID 2509.06861.
- The study focuses on closed-book knowledge-intensive tasks.
Entities
Institutions
- arXiv