TRACE: A New Benchmark for Temporal Reasoning in Large Reasoning Models
Researchers have unveiled TRACE, a novel framework aimed at assessing temporal reasoning in Large Reasoning Models (LRMs). This initiative, outlined in a paper available on arXiv (2607.04784), conceptualizes temporal reasoning as constraint satisfaction problems through Allen's Interval Algebra, facilitating meticulous management of logical complexity. TRACE incorporates a Trace-Based Verification Oracle to ensure the reliability of reasoning, overcoming the drawbacks of static benchmarks that may suffer from data contamination and lack of nuanced difficulty levels. The team developed TRACEBench, which features 1,200 synthesized test instances with varying difficulty. Evaluations of eight prominent LRMs on TRACEBench revealed a significant negative correlation between reasoning faithfulness and task complexity, indicating challenges with intricate temporal reasoning tasks. The paper serves as a cross-type announcement and is published on arXiv.
Key facts
- TRACE is a testing framework for temporal reasoning in Large Reasoning Models (LRMs).
- It models temporal reasoning as constraint satisfaction problems via Allen's Interval Algebra.
- TRACE includes a Trace-Based Verification Oracle to validate reasoning faithfulness.
- TRACEBench is a benchmark with 1,200 synthesized test instances across graded difficulty levels.
- Eight widely used LRMs were evaluated on TRACEBench.
- Results show a strong negative correlation between reasoning faithfulness and task complexity.
- The paper is available on arXiv with identifier 2607.04784.
- The announcement type is cross.
Entities
Institutions
- arXiv