AIR-BENCH Live: Self-Evolving Safety Benchmark for Foundation Models
AIR-BENCH Live is a self-evolving safety benchmark for foundation models, introduced as a successor to AIR-BENCH 2024. It features an automated pipeline that monitors government regulation and updates its risk taxonomy accordingly. The system classifies new policies against a four-tier risk taxonomy, proposing new granular categories when needed. A multi-agent, persona-driven prompt generation algorithm creates realistic, multilingual prompts with minimal human review. The current version expanded the benchmark from 314 to 335 granular risks, adding 21 new categories. This addresses the problem that static benchmarks become outdated as models improve and AI-safety legislation evolves.
Key facts
- AIR-BENCH Live is a self-evolving successor to AIR-BENCH 2024.
- An automated update pipeline monitors government regulation.
- The pipeline classifies new policies against a four-tier risk taxonomy.
- It can propose new granular categories for unmatched policies.
- A multi-agent, persona-driven prompt generation algorithm creates prompts.
- Prompts are realistic, multilingual, and require minimal human review.
- The benchmark expanded from 314 to 335 granular risks.
- 21 new categories were added in the current version.
Entities
—