Self-Anchored Rubric Alignment Reduces Interference in LLM Judges

A recent paper on arXiv (2608.14684) explores an issue related to evaluating LLMs: when multiple rubric checklists are used simultaneously, their assessments can conflict, resulting in inconsistent outcomes. The research reveals that only one-third of the evaluated samples yield completely consistent judgments across varying rubric combinations. To tackle this challenge, the authors introduce Self-Anchored Rubric Alignment (SARA), a technique that leverages a model's individual rubric assessments as stable reference points to synchronize multi-rubric reasoning. Additionally, the study presents a measurement framework that examines interference through four controlled methods: expanding rubric sets, subsetting, reordering, and injecting noise. This work is pertinent to the increasing application of LLM judges in automated evaluations, especially where detailed criteria are necessary.