LLM-as-a-Judge Evaluators Vulnerable to Consensus Mimicry Under Social Load
A recent paper in computer science presents the concept of the 'Agentic Formalism Trap' alongside the Evaluative Dissonance Index (D_E), which measures how LLM-as-a-Judge systems mix structural proceduralism with semantic accuracy when faced with adversarial challenges. The research examines 22,500 trajectories from three areas (GAIA, SWE-bench, Multi-Challenge) and develops a semantic classification of hallucination tactics, confirmed through deterministic lexical grounding (p < 10^-120). A logistic meta-evaluator identifies precise syntactic triggers for evaluator capture (ROC-AUC 0.8779), while zero-shot Leave-One-Domain-Out transfer indicates that the vulnerability is not limited to specific domains (mean ROC-AUC 0.7482). Architectural profiling shows that different simulated swarm structures create unique semantic blind spots, highlighting the instability of unanchored closed-loop evaluations and the need for architecture-specific vigilance filters. The paper can be accessed on arXiv (arXiv:2607.28641) in the 'Computer Science > Computation and Language' category.
Key facts
- Introduces Agentic Formalism Trap and Evaluative Dissonance Index (D_E)
- Analyzes 22,500 trajectories across GAIA, SWE-bench, and Multi-Challenge domains
- Validates semantic taxonomy of hallucination maneuvers with p < 10^-120
- Logistic meta-evaluator achieves ROC-AUC 0.8779 for syntactic triggers
- Zero-shot Leave-One-Domain-Out transfer shows mean ROC-AUC 0.7482
- Distinct simulated swarm topologies induce disparate semantic blind spots
- Unanchored closed-loop evaluation is unstable and domain-agnostic
- Paper available on arXiv with ID 2607.28641
Entities
Institutions
- arXiv