OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting
Researchers propose OracleProto, a framework for evaluating LLM native forecasting capability by reconstructing resolved events with knowledge cutoff and temporal masking. This addresses the challenge of distinguishing genuine forecasting from memorized facts in retrospective benchmarks. The framework aims to provide reproducible evaluation for forecasting, a composite capability linking information gathering, evidence integration, judgment, and decision-making, which is in demand across finance, policy, industry, and scientific research.
Key facts
- OracleProto is a reproducible framework for benchmarking LLM native forecasting.
- It reconstructs resolved events into time-specific benchmarks.
- It uses knowledge cutoff and temporal masking to prevent data leakage.
- Live benchmarks expire once events resolve, limiting reproducibility.
- Retrospective benchmarks cannot distinguish forecasting from memorized facts.
- Prompting models to 'pretend not to know' is insufficient.
- Forecasting is a composite capability linking information gathering, evidence integration, judgment, and decision-making.
- Demand for forecasting exists across finance, policy, industry, and scientific research.
Entities
—