Study Shows LLM Judge Performance Depends on Decision Rule, Not Just Accuracy

A new arXiv paper (2608.07813) investigates the effectiveness of LLM judges in reasoning pipelines, arguing that the cost of a judge's decision depends more on the decision rule it operates within than on its raw accuracy. The study, which uses frozen candidate pools from four GRPO policies, finds that an unconstrained scalar DeepSeek-R1-7B judge provides minimal improvement over answer-level majority voting: +1.0 percentage points on 500 GSM8K questions and +0.34 exact match on 300 HotpotQA questions. On a frozen-rule 30-question confirmation split, the judge is 10 points worse than majority, demonstrating that a confident judge can actually destroy accuracy. The authors then introduce Evidence-Locked Derive-Gate-Repair (EL-DGR), a task-adaptive non-compensatory decision rule. Under EL-DGR, a judge preference can override evidence-supported consensus only when accompanied by an extractive evidence certificate, and a repair is permitted only when neither alternative is certified and the repair itself is certified. The paper reports that with no change to the judge model, EL-DGR improves performance, suggesting that the decision rule is a critical factor in LLM judge deployment. The findings have implications for AI systems that use LLM judges to select answers in reasoning tasks, highlighting the importance of designing decision rules that prevent judges from making harmful decisions.