Uncertainty Estimation in Code LLMs: Multi-Sample P(True) Shows Promise, Self-Correction Fails

A recent study published on arXiv (2608.14659) explores the applicability of uncertainty estimation techniques from natural language processing to code generation, assessing if these methods can enhance code generation via selective self-correction. Researchers tested five uncertainty techniques—mean token entropy, verbalized confidence, P(True), entropy ensembles, and semantic entropy probes—using three small code LLMs on the HumanEval and BigCodeBench benchmarks. The findings indicated that multi-sample P(True) had the highest correlation with correctness, while the other methods, including semantic entropy probes, showed only weak correlations. The authors implemented three self-correction strategies based on these signals: adaptive decoding, uncertainty-based regeneration, and verification-based regeneration. Surprisingly, the results demonstrated that uncertainty-driven self-correction did not enhance code generation and, in some instances, worsened it, underscoring the limitations of current uncertainty estimation methods in this area and the need for more effective solutions.