Beta-Bernoulli Calibrator Enhances LLM Probabilistic Forecasting
Researchers propose the Beta-Bernoulli Calibrator (BBC), a method to improve probabilistic forecasting by large language models (LLMs). BBC converts initial point forecasts into distributions over event likelihood, leveraging both binary outcomes and aggregated human forecasts. It models event likelihood as Beta(α, β) and outcomes as Bernoulli(p), outputting a calibrated point forecast and epistemic uncertainty. Tests show BBC yields better-calibrated and more accurate forecasts than traditional methods. The approach addresses the underexplored use of human agreement signals in LLM forecasting.
Key facts
- BBC converts initial point forecasts into distributions over event likelihood.
- BBC models event likelihood as Beta(α, β) and outcomes as Bernoulli(p).
- BBC uses supervision from both binary outcomes and human forecasts.
- BBC provides calibrated point forecasts and epistemic uncertainty.
- BBC yields better-calibrated and more accurate forecasts than traditional methods.
- The approach addresses underexplored use of human agreement signals.
- Probabilistic forecasting estimates likelihood of uncertain future events.
- Existing methods typically learn from binary outcomes to output verbalized forecasts.
Entities
—