New Benchmark MAS-HQ Evaluates AI Factuality vs. Compute Cost

A recent paper published on arXiv presents MAS-HQ (Multi-Agent System Hallucination Quest), a protocol designed to evaluate resource-efficient benchmarks for hallucination in large language models. The researchers contend that traditional static leaderboards, which assess factuality in isolation and disregard computational expenses, fail to differentiate truly superior systems from those that merely utilize more computational power. They illustrate a ranking shift: a brute-force Best-of-4 agent secures a higher raw factuality score (H-Score 0.9169 compared to 0.9103) but falls short in the cost-adjusted Q-Score (0.5169 versus 0.5217) while consuming approximately four times the tokens and latency. The protocol integrates any factuality detector, highlighting the balance between accuracy and computational costs.