OurBench: New Benchmark for Debugging Enterprise SQL with LLMs
A new benchmark called OurBench has been introduced to evaluate LLMs' ability to debug enterprise-level SQL code. Unlike existing text-to-SQL benchmarks that focus on generating correct SQL in one attempt, OurBench targets the iterative debugging process, which is more realistic for complex enterprise data engineering tasks. The benchmark comprises 469 queries with syntax errors (OurBenchSyn) and 516 queries with semantic errors (OurBenchSem), where code fails to meet user intent. Queries average over 14 components in complexity. OurBench uses an automated construction workflow that injects realistic bugs via reverse engineering, and an execution-free evaluation framework for fast, accurate assessment. This addresses the challenge that even advanced LLMs and experienced developers struggle to produce fully correct SQL in a single attempt.
Key facts
- OurBench is the first benchmark for enterprise-level SQL reasoning and debugging
- Contains 469 syntax error queries (OurBenchSyn) and 516 semantic error queries (OurBenchSem)
- Queries average over 14 components in complexity
- Uses reverse engineering to inject realistic bugs
- Employs an execution-free evaluation framework
- Aims to improve LLM debugging capabilities for enterprise SQL
- Addresses limitations of single-attempt text-to-SQL benchmarks
- Published on arXiv with ID 2601.18119
Entities
Institutions
- arXiv