AeroCopilotBench: New Benchmark Tests LLM Agents as Aviation Copilots
AeroCopilotBench, a novel two-tier benchmark, has been introduced for assessing large language model (LLM) agents in the role of aviation copilots. Accompanying this benchmark is the AeroCopilot Operational Environment (ACOE), which offers a reproducible interactive virtual cockpit for testing. The first tier evaluates aviation knowledge through 1,200 multiple-choice questions, while the second tier includes 73 emergency and abnormal scenarios based on manufacturers' Pilot's Operating Handbooks (POHs) and implemented within ACOE. ACOE translates natural-language procedures into actionable state transitions, final-state goals, and stringent safety constraints, allowing models to analyze cockpit status, identify issues, and manage aircraft systems via standardized tool interfaces. This research paper can be found on arXiv with the identifier 2608.16349.
Key facts
- AeroCopilotBench is a two-tier benchmark for evaluating LLM agents as aviation copilots.
- The AeroCopilot Operational Environment (ACOE) is a reproducible interactive virtual-cockpit test environment.
- Tier-1 evaluates aviation knowledge using 1,200 multiple-choice questions.
- Tier-2 comprises 73 emergency and abnormal tasks derived from manufacturers' Pilot's Operating Handbooks (POHs).
- ACOE converts natural-language procedures into executable state transitions, final-state goal conditions, and hard safety constraints.
- The benchmark enables models to interpret cockpit state, diagnose faults, and operate aircraft systems through standardized tool interfaces.
- The paper is published on arXiv with identifier 2608.16349.
- The work addresses the lack of systematic testing of procedural execution and safety compliance in interactive environments.
Entities
Institutions
- arXiv