Wiggle Framework: Stress-Testing Epistemic Stability of LLM Judges
A recent paper published on arXiv (2608.12645) presents the Wiggle Framework, designed as a comprehensive stress test to assess the epistemic stability of LLM judges. This framework breaks down judge robustness into three key areas: Mechanical Consistency (resilience to re-prompting and reframing), Single-turn Conviction (response stability to a single challenge), and Multi-turn Persistence (endurance under ongoing or adaptive pressure). The research evaluates 9 cutting-edge models across 14 judging tasks, such as safety, toxicity, AI writing detection, and political response assessment. Findings reveal that each model shows significant variability in judgment, altering decisions 25–71% of the time under static pressure and 62–91% with ongoing challenges. The authors contend that conventional accuracy metrics fail to adequately measure stability against re-prompting and sustained challenges, underscoring the vulnerability of LLM judges and the necessity for improved evaluation techniques. This research is particularly pertinent given the increasing reliance on LLM judges for model assessments, online grading, and reward modeling.
Key facts
- The Wiggle Framework is introduced as a unified stress test for epistemic stability in LLM judges.
- The framework decomposes judge robustness into Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence.
- The study evaluates 9 frontier models across 14 judging tasks.
- Judging tasks cover safety, toxicity, AI writing detection, and political-response evaluation.
- Models flip verdicts 25–71% of the time under static pushback.
- Models flip verdicts 62–91% of the time under adversarial persistence.
- Traditional accuracy-based validation is insufficient for assessing judge stability.
- The paper is available on arXiv with identifier 2608.12645.
Entities
Institutions
- arXiv