New Method Measures Black-Box Confidence via Reasoning Trajectories

Researchers have introduced a novel technique for assessing confidence in chain-of-thought reasoning using text-based APIs, known as the trajectory-confidence score. This system visualizes reasoning paths and evaluates consistency with external benchmarks through a unique softmax approach, avoiding the need for hidden states or calibration methods. Experiments across six diverse scenarios, such as MedQA-USMLE and GPQA Diamond, utilizing Gemini 3.1 Pro and Claude Sonnet 4.6, indicated that integrating this score with coverage and verbalized confidence at K=4 achieved superior outcomes compared to self-consistency at K=8, achieving a median AUC of 0.78 versus 0.71, with no vendor-related discrepancies observed.