First-Token Confidence Matches Semantic Self-Consistency for Hallucination Detection

The phi_first technique identifies hallucinations in large language models by assessing the normalized entropy of the top-K logits at the initial content-bearing token during a single greedy decode. This method either matches or surpasses the performance of semantic self-consistency, which necessitates multiple sampling and external inference. In tests involving three instruction-tuned models of 7-8B and two benchmarks, phi_first recorded an average AUROC of 0.820, while semantic agreement and standard surface-form self-consistency scored 0.793 and 0.791, respectively. Additionally, this approach is computationally efficient and shows a strong correlation with semantic agreement.