SemPlan Benchmark Evaluates LLM-Based Structured Semantic Planning for Enterprise Data Queries
A new benchmark called SemPlan has been introduced to evaluate how natural-language interfaces handle enterprise data design. It focuses on turning vague user requests into actionable steps while dealing with issues like invalid queries and policy failures. SemPlan includes a synthetic bilingual dataset with 1,800 instances in English and Brazilian Portuguese, with 1,200 set aside for scientific evaluation. Four models were tested: direct SQL generation (A1), a baseline tool-agent (A2), structured semantic requests (A3), and a clarification version (A4). The overall accuracy was low, with A3 achieving the best results at 25.67%. A3 notably outperformed the others, highlighting the potential of structured semantic planning. The benchmark is available on arXiv under the identifier 2608.13612.
Key facts
- SemPlan Benchmark is a deterministic synthetic bilingual benchmark with 1,800 cases in English and Brazilian Portuguese.
- 1,200 cases form the frozen scientific evaluation subset.
- Four architectures are compared: direct SQL generation (A1), bounded tool-agent baseline (A2), structured semantic-request generation with deterministic planning (A3), and clarification/stateful semantic-plan variant (A4).
- Across 4,800 primary records, correctness rates: A1 22.25%, A2 22.58%, A3 25.67%, A4 24.25%.
- A3 had the highest observed correctness and significantly outperformed A1, A2, and A4.
- The benchmark addresses control of invalid queries, policy failures, cost, and nondeterminism.
- The paper is available on arXiv with identifier 2608.13612.
Entities
Institutions
- arXiv