Agentic-SQL Revisited: Autonomy-Based Taxonomy and Benchmark Analysis
A recent preprint on arXiv (2608.15389) introduces a classification system for Text-to-SQL models based on LLMs, categorizing them along an axis of inference autonomy that includes constrained, in-context, iterative, agentic, and reasoning-internalized generation. The researchers redefine the domain as an aggregation of leaderboards, compiling metrics from various authors and ensuring traceable origins for each entry. To substantiate their findings, they perform a case study utilizing the Spider benchmark, assessing 8B open-source models with and without chain-of-thought (CoT) supervision against few-shot DeepSeek V3 and GLM-4 benchmarks. Their analysis reveals four key trends: uneven transfer of Spider to BIRD and Spider 2.0; autonomy's trade-off for robustness; reasoning internalization's position between answer-only decoding and externally coordinated agents; and CoT improvements focusing on Hard and Extra-Hard queries. The authors also provide a Python harness for additional exploration.
Key facts
- arXiv preprint 2608.15389 proposes a taxonomy for LLM-based Text-to-SQL systems.
- Taxonomy is based on an inference-autonomy axis: constrained, in-context, iterative, agentic, reasoning-internalized.
- The study aggregates metrics reported by authors, with traceable provenance for each cell.
- A case study on Spider compares 8B open-source backbones with/without CoT supervision.
- Baselines include few-shot DeepSeek V3 and GLM-4.
- Four patterns: uneven transfer to BIRD and Spider 2.0; autonomy costs; reasoning internalization; CoT gains on Hard/Extra-Hard.
- A Python harness is released.
- The paper is available at arxiv.org/abs/2608.15389.
Entities
Institutions
- arXiv