Graph-Structured Rubrics Enhance LLM Evaluation Accuracy

A recent study published on arXiv (2608.12097) presents Graph-Structured Rubrics (GSR), a novel technique for organizing evaluation rubrics into structured graphs for LLM assessments. This method overcomes a shortcoming in existing rubric-based evaluators, which often view rubrics as simplistic criteria without clear composition. GSR creates a graph independent of responses prior to evaluation, where criterion nodes facilitate judgments and utilize transformation, reduction, and gating operators through designated ports. A specific output mapping, termed Readout, transforms the unique sink into a score or preference. The compilation phase discards improperly formed or incompatible graphs. For pointwise evaluations, rubric dimensions are assessed individually before graph integration, while pairwise evaluations apply the graph with one judgment per criterion for each candidate. Under GPT-OSS-120B, GSR enhances exact score agreement by 0.62 to 6.75 percentage points compared to Prometheus-style scoring across four pointwise datasets. This research was conducted by a team of researchers and announced as a new submission on arXiv.