LLM Self-Assessment: Effort and Ability Outperform Confidence
A recent study published on arXiv (2605.07806) introduces a multidimensional framework for self-assessment in Large Language Models, inspired by cognitive appraisal theory from psychology. The researchers identified six appraisal dimensions, such as effort and ability, in addition to conventional confidence metrics, and tested their effectiveness in forecasting model failures across 12 LLMs and 38 tasks across eight different domains. The results indicate that dimensions related to competence, especially effort and ability, either match or exceed confidence in predicting accuracy, while also providing more realistic estimates. These findings question the dependence on expressed confidence as a measure of reliability.
Key facts
- Study proposes multidimensional self-assessment for LLMs based on cognitive appraisal theory.
- Six appraisal dimensions (effort, ability, etc.) were evaluated alongside confidence.
- Tested on 12 LLMs and 38 tasks across eight domains.
- Effort and ability dimensions match or outperform confidence in predicting failure.
- Effort yields less overoptimistic estimates than confidence.
- Research challenges the use of confidence as a primary reliability metric.
- Published on arXiv with ID 2605.07806.
- Cognitive appraisal theory originates from human psychology.
Entities
Institutions
- arXiv