Adaptive Auditing Framework for Generative AI Systems
A new paper on arXiv introduces a hypothesis testing framework for adaptive auditing of generative AI systems, addressing the challenge of drawing statistically rigorous conclusions from highly flexible testing paradigms. The framework pits two competing null hypotheses against each other: one from the model asserting no failure mode below a target threshold, and another from the auditor. This approach aims to provide anytime-valid guarantees despite limited sample sizes (often 10 to 50 cases) and data-dependent sampling and stopping decisions.
Key facts
- Paper title: Adaptive auditing of AI systems with anytime-valid guarantees
- Published on arXiv with ID 2605.07002
- Addresses adaptive testing paradigms for generative AI
- Sample sizes are often limited to 10 to 50 cases
- Framework uses two 'dueling' null hypotheses
- Model's null: no failure mode below target threshold
- Auditor's null: opposite claim
- Aims to provide statistically rigorous conclusions despite adaptive sampling
Entities
Institutions
- arXiv