Rule-Gated 7B Analytics Agent Outperforms Direct-Prompted 32B Baseline
A recent study has unveiled WarehouseReliabilityBench, a benchmark comprising 400 frozen tasks across two synthetic warehouses, aimed at assessing LLM analytics agents based on production failures rather than the accuracy of SQL syntax. This benchmark features inquiries with two acceptable business definitions, questions that the warehouse is unable to answer, deprecated columns following schema updates, and queries that may run successfully but yield incorrect business figures. Additionally, the research introduces QueryProof, a 7B agent that employs rules from a semantic layer and physical catalog to guide its actions, validating each response through deterministic post-execution checks. In an 80-task synthetic test split, QueryProof surpassed a directly prompted 32B baseline by +0.237, with a pre-registered paired bootstrap ensuring each claim verb was fixed prior to the existence of the numbers. The paper can be found on arXiv with the identifier 2608.09254.
Key facts
- WarehouseReliabilityBench consists of 400 frozen tasks over two synthetic warehouses.
- Roughly half of the correct responses in the benchmark are clarifications, abstentions, or refusals.
- QueryProof is a 7B agent that uses rules derived from a semantic layer and physical catalog.
- QueryProof gates every answer on deterministic post-execution checks.
- On an 80-task synthetic test split, QueryProof outperformed a direct-prompted 32B baseline by +0.237.
- The benchmark includes questions with two valid business definitions, unanswerable questions, deprecated columns, and queries returning wrong business numbers.
- The paper pre-registered a paired bootstrap to fix each claim verb before the numbers existed.
- The paper is available on arXiv under the identifier 2608.09254.
Entities
Institutions
- arXiv