Cheap Open-Weight LLMs Match Frontier Judges for Grading Math Proofs

A new study from arXiv (2608.00004) investigates whether cost-effective open-weight language models can reliably grade natural-language mathematical proofs, a task traditionally requiring expensive frontier models. The researchers evaluated three cheap models—GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B—on a 200-instance validation sample of IMO-GradingBench, comparing their pass/fail agreement with human graders against that of Claude Opus 4.7 and Gemini 3.1 Pro. Results showed that the cheap judges achieved statistically indistinguishable agreement rates from the frontier models, at up to 100 times lower cost. Surprisingly, a majority vote of the three cheap models did not improve over its strongest individual member, but requiring unanimous agreement (all-three-pass) yielded the highest pass-agreement and precision, with minimal run-to-run variance across four replicates. The findings suggest that budget-friendly open-weight models can serve as reliable judges for math proof evaluation, potentially reducing costs in AI evaluation pipelines. The study extends to the full 1000-instance benchmark, exploring consensus rules to optimize grading accuracy.