LLM-as-a-Judge Bias Study Finds Style Bias Dominates, Debiasing Helps

A recent study thoroughly investigates strategies for mitigating bias in LLM-as-a-Judge systems. Researchers assessed nine debiasing approaches across five judge models from Google, Anthropic, OpenAI, and Meta, utilizing three benchmarks (MT-Bench, LLMBar, custom) and examining four types of bias. The results reveal that style bias is the most prevalent (0.76-0.92 across models), overshadowing position bias (≤0.04). While all models exhibit a preference for conciseness in expansion pairs, truncation controls demonstrate a precise quality-length distinction (0.92-1.00 accuracy). Although debiasing proves advantageous, its effectiveness varies by model; the combined budget strategy notably enhances Claude Sonnet 4 by +11%. The study underscores the lack of research on style bias, despite its dominance.