TREAT: Benchmark for Theorem Recognition in LLMs
A new assessment framework called TREAT has been introduced to gauge how well large language models (LLMs) can recognize established theorem identities that come from transformations keeping theorem conditions equivalent. This benchmark tackles the challenge of figuring out when an unknown expression corresponds to a known formal object, which is crucial for AI systems handling different input styles and the formal constructs used by later tools. Rather than just rephrasing theorems, TREAT modifies the mathematical framework of theorems by presenting known results in various ways, including residual equations and optimization identities. The benchmark was created by collecting theorem pages and choosing entries with significant mathematical content. You can find the related paper on arXiv under the identifier 2608.07540.
Key facts
- TREAT is a benchmark for evaluating LLMs' ability to recover known theorem identities from equivalence-preserving formula-level transformations.
- The benchmark changes the mathematical form of theorem conditions, not just paraphrasing text.
- Transformations include residual equations, witness statements, optimization identities, set relations, operator forms, and proof-intermediate characterizations.
- The benchmark was built from scraped theorem pages, filtered for usable mathematical content.
- The paper is available on arXiv with identifier 2608.07540.
- The work addresses the challenge of recognizing unfamiliar formulations of known formal objects.
- AI systems increasingly operate between flexible input representations and formal objects used by downstream tools.
Entities
Institutions
- arXiv