AIM-Text-to-SQL Benchmark: SQL Accuracy Across 70+ LLMs
We evaluated SQL answers from 70+ LLMs on a 759-question BIRD-SQL subset. Each model selected a database from 11 candidates, then produced a query.
SQL accuracy is the percentage of scored queries that return the reference result. Incorrect routes and references flagged as broken are excluded. A reference query is the SQL supplied as the expected answer.
Each model reaches a different set of SQL questions because scoring depends on its database choices. Differences in these subsets and execution setup limit comparisons.
Text-to-SQL findings
Opus 5.5 recorded 87.8% SQL accuracy
claude-opus-5.5 matched the reference result on 444 of 506 scored questions. gemini-4-argon followed at 78.6% (407 of 518), then gemini-3.8-flash at 77.2% (342 of 443). These were the three highest SQL accuracy scores in the 81-model panel. Muse Wonder recorded 56.4% (285 of 505). GPT-6.1 Terra recorded 56.6% (280 of 495). MiniMax M3.1 Flash recorded 36.5% (177 of 485). Kimi K3.1 Pro recorded 52.3% (257 of 491). Muse Spark Open recorded 44.9% (219 of 488), and DeepSeek V4.1 Flash 1017 recorded 22.8% (108 of 474).
The score excludes incorrect database choices and references flagged as broken. Opus 5.5 used Claude Code. Gemini 4 Argon and Gemini 3.8 Flash used OpenRouter. The ranking reflects both execution conditions and differences in the questions each model routed correctly.
Three Pro variants scored below their base models
gpt-6.1-sol-pro recorded 63.6% SQL accuracy, compared with 65.1% for gpt-6.1-sol. gpt-6-sol-pro scored 53.6% against 53.8% for gpt-6-sol. gpt-6-luna-pro scored 44.2% against 45.4% for gpt-6-luna.
All six models used OpenRouter. These scores come from one run per model and different subsets of correctly routed questions. Repeated runs on the same SQL questions would be needed to assess whether the small gaps persist.
Cost vs. SQL accuracy
Run cost covers the retained routing and SQL responses from each model’s benchmark run. The target was 759 questions. Seven runs have fewer retained answers. Separate smoke tests, discarded retry attempts and jury calls are excluded. SQL accuracy uses each model’s correctly routed questions after excluding references flagged as broken.
gemini-3-flash-preview recorded 67.7% SQL accuracy at $7.67 per run. gemini-3.8-flash reached 77.2% at $26.48. Opus 5.5’s 87.8% score is paired with an estimated API-equivalent cost of $188.71. Gemini 4 Argon reached 78.6% at $41.41 in retained run costs. Muse Wonder reached 56.4% at $77.44 in retained run costs. GPT-6.1 Terra reached 56.6% at $7.19 in retained run costs. MiniMax M3.1 Flash reached 36.5% at $2.30 in retained run costs. Kimi K3.1 Pro reached 52.3% at $27.14 in retained run costs. Muse Spark Open cost $22.03 and DeepSeek V4.1 Flash 1017 cost $1.75 for the retained routing-plus-SQL runs.
The 78 API models use metered costs. Opus 5.5, Fable 5.1 and GPT-6 Astra have separately labeled estimates based on their recorded CLI tokens and OpenRouter prices checked on 1 October 2026, with recorded cache usage preserved.
Costs include database exploration, and SQL scores cover different question subsets. This comparison therefore describes the cost and accuracy of the combined benchmark workflow. Isolated SQL generation costs were not measured. The agentic RAG benchmark reports database selection separately.
SQL accuracy results
Before reference review measures exact result matching on all correctly routed questions. After answer review adds credit for answers accepted by the jury. Both comparison columns use the original denominator, while SQL accuracy excludes flagged references.
How models answered the same SQL question
The same question produced exact matches, correct answers with extra columns, and an unrelated institution. All eight models chose the correct database, college_completion. The table institution_details stores institution names in chronname, cities in city and states in state. These responses come from saved benchmark runs. Seven models used OpenRouter, while Opus 5.5 used Claude Code.
GPT-6.1 Terra, Kimi K3.1 Pro, Gemini 4 Argon and Opus 5.5 returned Georgia and passed execution match. Muse Spark Open and DeepSeek V4.1 Flash 1017 also identified Georgia, but included name and city columns. They failed the strict result comparison. All three answer-review jurors accepted each answer.
MiniMax M3.1 Flash and GLM 5.3 used LIKE '%Mercer%', which also returned Mercer County Community College in New Jersey. All three jurors classified each response as a model error. The reference passed the separate validity audit.
We selected this case to show how filtering and output columns affect evaluation. It illustrates individual responses. The full benchmark results above provide the overall model comparison.
Conditions affecting SQL accuracy
BIRD’s hints increased Gemini Flash’s SQL accuracy by 8.7 points
We repeated gemini-3-flash-preview with BIRD’s supplementary evidence. On questions routed correctly in both conditions, SQL accuracy rose from 67.5% to 76.2%. Strict accuracy, before reference review, rose from 55.0% to 61.6%.
This was a single-model experiment. It does not establish a common uplift for the 81-model panel. The main panel withholds the hints because they can also reveal which database to select.
Models reached different sets of SQL questions
Each model’s database choices determine which questions reach SQL scoring. For claude-opus-5, the strict denominator contains 717 questions, including 156 from the hardest routing group. For nova-lite-v1, it contains 285 questions, including 22 from that group. The shares are 21.8% and 7.7%.
Their SQL percentages therefore combine query-writing performance with the difficulty of the cases that reached scoring. A controlled SQL comparison would supply every model with the correct database for the same questions. That experiment has not been run.
Question selection also matters when comparing AI benchmarks that provide different schemas, hints or tool budgets.
Findings from reference and answer review
The reference audit flagged 236 of 759 queries
Five models assessed each reference query against its question and available schema information. They saw no candidate model answer. A majority of at least three marked 236 references as broken, or 31.1% of the set.
After the audit, the benchmark owner reviewed the flags. Independent human annotators have not repeated that review. These percentages describe the audited subset. They should not be generalized to all of BIRD.
Equivalent answers added 22.7 points to claude-sonnet-5’s score
claude-sonnet-5 passed 211 of 679 strictly scored questions, or 31.1%. A jury accepted another 154 answers as equivalent to the reference. These contributed 22.7 percentage points.
A further 117 answers received credit because the jury accepted the model’s query and rejected the reference. Those contributed 17.2 points. Together, the two categories raised the score to 71.0%, a 39.9-point increase.
Equivalent answers can return the requested information with an extra column or a different projection. Strict comparison can reject that result even when a reviewer accepts it as an answer to the question. Of the increase, 22.7 points come from the jury’s decisions about these cases. Reference defects account for the remaining 17.2 points.
Add as preferred sourceText-to-SQL benchmark methodology
Questions and execution
BIRD-SQL supplies natural-language questions, SQL references and database contents.1 We used a frozen selection of 759 questions from 11 databases, covering 560 train and 199 dev examples.
Models received short database descriptions and tools for listing tables, reading column definitions and executing SQL. Each question allowed up to nine calls with tools and one final response containing the chosen database and SQL. The schema preview limit was 4,000 characters, and query previews were capped at 50 rows.
The final candidate query and reference were executed against the same database file. Strict scoring compared their results, including an order-sensitive mode for questions that request an ordered answer. Missing, malformed and execution-failing candidate queries received no strict credit on otherwise eligible, correctly routed questions.
For the 78 OpenRouter runs, we requested temperature 0. Of these, 61 used the v3 tool-call recovery layer, 15 predated it, and two combined records from both versions. GPT-6 Astra ran through Codex CLI. Opus 5.5 and Fable 5.1 used Claude Code. All three CLI runs used medium reasoning effort. The GPT-6 Luna, Luna Pro, Sol and Sol Pro runs and GPT-5.2 used OpenRouter. These execution differences prevent attribution of a score gap to the model alone.
Seven runs completed fewer than 759 questions. Missing records are absent from the scoring denominators. The table reports the actual sample sizes for the main and supplementary scores. GPT-6.1 Terra, MiniMax M3.1 Flash and Kimi K3.1 Pro each completed all 759 questions through OpenRouter on 27 October. Earlier successful answers were preserved across measurement stages. DeepSeek V4.1 Flash 1017 and Muse Spark Open each completed all 759 questions through OpenRouter on 30 October.
SQL scoring
The main score excludes flagged questions from both its numerator and denominator. Three references marked uncertain remain in the calculation. The result files call this metric ex_gold_clean.
Two supplementary scores show how review changes the result:
Before reference review, the strict score measures exact execution match on all correctly routed questions. It includes questions with flagged references.
After answer review, the score adds credit when a separate jury accepts an eligible missed answer. It uses the same denominator as the strict score. The result files call this adjudicated accuracy.
Sample size (n) is the number of scored questions. The main score uses a smaller subset than the two supplementary scores.
For claude-sonnet-5, excluding flagged references reduced the accepted answers from 211 to 180 and the scored questions from 679 to 468. Its SQL accuracy was 38.5%.
Exclusion changes both counts. Answer review instead retains the original denominator and adds credit for accepted misses.
Reference and answer review
Auditing references and reviewing candidate answers require different inputs.
Reference auditors were claude-opus-4.8, gpt-5.6-sol, gemini-3.1-pro-preview, grok-4.5 and deepseek-v4-pro. Candidate answers were withheld to keep the flags independent of the model being scored.
Answer adjudication used three jurors from model families different from the candidate’s. The reference and candidate queries appeared in randomized A/B positions without labels revealing their roles. Ambiguous cases and cases with insufficient valid votes received no added credit. GPT-6.1 Terra, MiniMax M3.1 Flash and Kimi K3.1 Pro had at least two valid juror votes for every eligible case. Individual failed votes remain in the audit logs. One truncated GPT-6.1 Terra jury response was completed with a 4,000-token output limit, increased from 1,200. The jury model, prompt and scoring rules were unchanged. DeepSeek V4.1 Flash 1017 and Muse Spark Open had at least two valid juror votes for every eligible case.
Review coverage is asymmetric. Strict passes remain accepted, while eligible misses receive another assessment. Consequently, adjudication can retain false positives from strict scoring and add false positives of its own. It is a supplementary estimate, with neither guaranteed correctness nor a guaranteed upper bound on true accuracy.
A miss qualified for adjudication when the candidate query executed and the F1 overlap between its result rows and the reference was below 0.5. F1 combines the proportion of candidate rows found in the reference with the proportion of reference rows recovered by the candidate.
For example, gemini-3.5-flash-lite had 346 strict misses. Of these, 314 qualified for review. The jury accepted the candidate in 144 cases: 92 with a rejected reference and 52 with both answers accepted. It classified 105 as model errors and left 65 ambiguous. The other 32 misses comprised 27 with higher result overlap and five execution failures.
The jury accepted 45.9% of the 314 reviewed misses.
Limitations
Reference errors remain uncertain. The five-model audit inspected columns from tables used by each reference query, so it may miss an answer located in a different table. The audit can also reject a valid reference. A repeat review by independent human annotators is needed to estimate both kinds of error.
Jury judgments depend on LLM interpretation. No independent human review verified all adjudication verdicts. In the nova-lite-v1 control, accuracy rose from 18.9% under strict scoring to 31.6% after adjudication, showing that review did not automatically produce a high score. That control alone cannot establish jury accuracy.
Strict comparison can fail in both directions. Extra columns or different column order can reject a useful answer. Case normalization and rounding floating-point values to six significant figures can accept differences that matter.
Question-sampling intervals are available for strict scores and SQL accuracy in the result export. They omit reference-label uncertainty, jury error and variation between repeated SQL runs. No SQL repeat analysis is reported here.
These are results for a selected, hint-free workload with a database-routing requirement. They are not directly comparable with the BIRD leaderboard. The public train data also leaves a risk of training-data overlap, as discussed in the routing article.
Conclusion
claude-opus-5.5 led SQL accuracy at 87.8%, followed by gemini-4-argon at 78.6% and gemini-3.8-flash at 77.2%.
Each SQL score reflects the questions that model routed correctly and the references retained after review. A controlled SQL comparison would give every model the same questions and database information, with independent checks of reference labels and jury decisions.
Further reading
- Agentic RAG Benchmark: Multi-Database Routing
- 800+ Leading AI Benchmarks
- HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{AIM-Text-to-SQL Benchmark: SQL Accuracy Across 70+ LLMs}},
year = {2026},
month = oct,
howpublished = {\url{https://aimultiple.com/text-to-sql}},
note = {AIMultiple. Retrieved October 30, 2026}
}Results and timestamps of 1.5 thousand data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 5 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
9 updatesReplaced text-to-SQL benchmark results with a 36-model panel scored by strict, gold-clean, and adjudicated execution match.
Added a changelog to the Agentic RAG benchmark: multi-database routing and query generation section.
Replaced the methodology section with a summary and a reference to another article.
Reference Links
Comments 1
Share Your Thoughts
Your email address will not be published. All fields are required. Comments are left in their original language.
Curious, how much of the context engineering and specific prompting did you apply in your benchmarks. Or, was it to review the models only? I have found much higher return of correct and consistent responses. A higher fidelity. To do that, I needed to provide a most sophisticated prompt that fed the context window as the question was being asked. Not perfect, but better than those scores represented in this article when using the Grok 4.x .
Great point. This benchmark intentionally uses zero-shot, minimal prompting with temperature=0. No few-shot examples, no domain-specific instructions, no iterative refinement. The goal was to measure each model's baseline text-to-SQL capability. So your experience with Grok 4 getting higher fidelity through sophisticated context engineering is completely expected. A well-crafted prompt with detailed schema descriptions, few-shot examples, and domain-specific rules will improve any model's performance significantly. What this benchmark isolates is how well the model performs out-of-the-box when given only the raw question and retrieved schema, which helps compare the models' inherent SQL reasoning abilities on a level playing field. We'll make this clearer in the methodology section. Thanks for raising it.