DRBENCHER: Synthetic Benchmark for Web-Browsing AI Agents
Researchers have created a new tool called DRBENCHER, which acts as a synthetic benchmark generator for evaluating deep research agents that integrate web browsing with complex calculations. Unlike the usual benchmarks that test these abilities separately, DRBENCHER crafts questions that require both skills across five fields: biochemistry, finance, geophysics, security, and history. It meets four criteria: verifiability, complexity, difficulty, and diversity. Human evaluations show a 76% validity rate (84% without outdated data), with 35% of questions deemed difficult. This work is published on arXiv, under ID 2604.09251.
Key facts
- DRBENCHER is a synthetic benchmark generator for questions requiring both browsing and computation.
- It covers five domains: biochemistry, financial, geophysical, security, and history.
- Enforces four criteria: verifiability, complexity, difficulty, and diversity.
- Human evaluation shows 76% validity (84% excluding stale data).
- 35% of questions are considered challenging.
- Published on arXiv with ID 2604.09251.
Entities
Institutions
- arXiv