Rehearse: Training-Free Method for LLM Confidence Calibration

A new approach called Rehearse (Experiential Rehearsal) has been developed by researchers to enhance the calibration of verbal confidence in large language models (LLMs) without requiring training. This technique, outlined in arXiv paper 2508.14390, tackles the prevalent issue of LLMs exhibiting confidence levels that do not correspond with their actual accuracy, a crucial factor for applications where safety is paramount. Unlike traditional prompt-based calibration methods that view the process as a one-time inference challenge, Rehearse allows models to learn from their own confidence assessments. During a credence-calibration game governed by a strictly proper scoring rule, models receive feedback on their previous confidence choices. This feedback is encapsulated in a post-game trajectory prefix, which highlights consistent over- or under-confidence. When faced with new questions, the model utilizes this calibration signal in its reasoning process. The method was evaluated using four LLMs, three benchmarks, and five random seeds, demonstrating enhanced calibration. The research paper can be accessed on arXiv.