PCFJudge Reduces Order Sensitivity in LLM Factuality Evaluation

A new method called PCFJudge addresses candidate-order sensitivity in listwise factuality evaluation by large language models (LLMs). The technique reruns the same factuality-first prompt over multiple orderings of candidate sets and aggregates scores, ranks, and uncertainty signals into a consensus decision. On RewardBench 2 Factuality, PCFJudge improves accuracy by up to 7 absolute points over direct judging. Development ablations reveal that permutation consensus itself, rather than heavier arbitration layers, drives the improvement. The research suggests that a significant portion of factuality-judging error stems from order instability.