New LLM-as-a-Judge Standard for Multi-Hop RAG Evaluation

A team of researchers has introduced a baseline measurement standard aimed at evaluating retrieval-augmented generation (RAG) systems through LLM-as-a-judge, tackling challenges in multi-hop RAG assessment. This standard establishes parameters such as a fixed top-100 candidate pool, an evidence budget, an answer limit, a generator, and a prompt. It mandates pre-registered hypotheses, cluster-aware inference, precise cluster sign-flip verification, and replication by a second judge. They employed the Genetic Algorithm Decoder for Multi-hop Evidence Composition (GADMEC) to rigorously test 40 tasks. Their findings indicate that benchmarks based on clustering may exaggerate advancements.