LLMs Show Overconfidence on Hard Tasks, Underconfidence on Easy Ones

A preregistered study on confidence calibration in large language models (LLMs) reveals that, like humans, LLMs tend to be overconfident on average, with confidence exceeding accuracy. However, this overconfidence is moderated by a strong hard-easy effect: models are most overconfident on difficult tests and substantially underconfident on easy ones. The researchers developed LifeEval, a benchmark designed to evaluate model calibration across varying difficulty levels. The study was conducted by authors affiliated with arXiv, a repository for preprint papers in computer science and artificial intelligence. The findings highlight a systematic bias in LLM confidence that mirrors human cognitive tendencies, with implications for trust and reliability in AI systems.