AgentSLABench: Benchmarking AI Agents Under Resource Constraints
AgentSLABench, a novel evaluation framework focused on resource awareness for autonomous AI agents, has been detailed in an arXiv publication (2608.00805). Unlike conventional benchmarks that solely assess accuracy, this framework evaluates correctness in conjunction with latency, cost, computing power, memory, and network usage within specified resource limits. It features 16 task environments divided into 6 categories, which include 5 core tasks: multi-hop QA, retail substitution, code generation, web shopping, and travel planning, along with 11 extended tasks. Each task operates in separate Docker containers with defined CPU, memory, time, and network constraints, utilizing sealed test sets with SHA256 hashes and a uniform profiling protocol. The study examines 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) and 4 specialized agents, aiming to deliver a detailed profile for each agent per task, incorporating task correctness as a key metric for evaluation in real-world resource-limited conditions.
Key facts
- AgentSLABench is a resource-aware evaluation framework for autonomous AI agents.
- It measures correctness alongside latency, cost, compute, memory, and network usage.
- It provides 16 task environments across 6 categories.
- The 5 core tasks are multi-hop QA, retail substitution, code generation, web shopping, and travel planning.
- It uses isolated Docker containers with declared CPU, memory, time, and network budgets.
- Test sets are sealed with SHA256 hashes.
- It profiles 5 general-purpose baseline agents: ReAct, PlanAndSolve, Reflexion, CoT, and Random.
- It also profiles 4 task-specialized agents.
- The framework is similar to systems profilers like perf, pprof, and cProfile, but adds task correctness.
- The paper is available on arXiv with identifier 2608.00805.
Entities
Institutions
- arXiv