New Benchmark for Evaluating Scientific Novelty Metrics
Researchers have introduced a benchmark to evaluate the reliability of scientific novelty metrics without requiring explicit novelty labels. The benchmark tests whether scores respond correctly to controlled manipulations of the pool of prior work, based on three axioms: scores should decrease as the pool covers more of a paper's content, increase as the pool loses relevance, and decrease as the pool moves later in time. The study demonstrates that even the most direct human signal, ICLR reviewer novelty scores, fails to satisfy these axioms, highlighting the challenges in automating novelty assessment. The work is motivated by the rise of AI scientists, where reliable novelty evaluation is crucial to avoid wasting resources on already-explored ideas. The benchmark provides a framework for comparing novelty metrics and improving their validity.
Key facts
- A benchmark for evaluating scientific novelty metrics is introduced.
- The benchmark does not require explicit novelty labels.
- It tests three axioms: scores fall as the pool covers more content, rise as the pool loses relevance, and fall as the pool moves later in time.
- ICLR reviewer novelty scores are used as a human signal and shown to fail the axioms.
- The work is motivated by the need for reliable novelty evaluation in AI scientists.
- The benchmark aims to prevent waste of attention and compute on already-explored ideas.
- Existing novelty metrics often validate against noisy signals like citation counts or peer review scores.
- The paper is available on arXiv with ID 2604.15145.
Entities
Institutions
- arXiv
- ICLR