ALPS Benchmark Measures Valid Creativity in LLMs via Mathematical Construction
A new benchmark called ALPS (Austin-Law Proof-Synthesis) has been developed by researchers to assess the genuine creativity of large language models (LLMs). This benchmark tackles the difficulty of determining if outputs that seem creative are genuinely original and effective. Each entry in ALPS consists of a singular equational law, which is validated to either necessitate the creation of an infinite mathematical structure that meets the law or to prove that no such structure can exist. Automated proof checking is used for submission verification, eliminating human involvement, while a public generator continuously creates new instances, ensuring LLMs are not tested on previously encountered problems. The benchmark employs a set of eight configurations. The related paper can be found on arXiv with the identifier 2608.15979.
Key facts
- ALPS is a benchmark for measuring valid creativity in LLMs.
- Each instance is an equational law requiring construction of an infinite structure or proof of nonexistence.
- Submissions are verified by automated proof checking with no human involvement.
- A public generator produces new instances without limit.
- LLMs are never evaluated on problems they may have seen.
- The benchmark uses a portfolio of eight configurations.
- The paper is on arXiv with identifier 2608.15979.
Entities
Institutions
- arXiv