AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games
A new method called AV-AIVAT (Anytime-Valid Action-Informed Value Assessment Tool) reduces the cost of evaluating AI agents in imperfect-information games by up to 74 times while maintaining statistical guarantees. The approach combines AIVAT, a variance reduction technique, with Confidence Sequences (CSs) to enable anytime-valid stopping. In tests across 15 LLM agent configurations and 71,439 paired Heads-Up No-Limit Hold'em (HUNL) hands, AIVAT reduced variance by a median of 54 times. AV-AIVAT allows evaluations to stop as soon as sufficient evidence is gathered, avoiding wasted resources. The method is particularly relevant for comparing agents in games like poker, where skill and luck are intertwined. The paper is available on arXiv under the identifier 2608.06362.
Key facts
- AV-AIVAT combines AIVAT with Confidence Sequences for anytime-valid stopping.
- AIVAT reduces variance by a median of 54x across 15 LLM agent configurations.
- The study used 71,439 paired Heads-Up No-Limit Hold'em (HUNL) hands.
- AV-AIVAT can reduce evaluation cost by up to 74 times.
- The method maintains statistical guarantees while allowing early stopping.
- The paper is available on arXiv with identifier 2608.06362.
- AIVAT uses conditional mean-zero corrections for variance reduction.
- The online value model in AV-AIVAT learns only from past data.
Entities
Institutions
- arXiv