Text-to-SQL Benchmark: SQL Accuracy Across 40+ LLMs
We evaluated SQL answers from 37 LLMs on a 759-question BIRD-SQL subset. Each model selected a database from 11 candidates, then produced a query.
SQL accuracy is the percentage of scored queries that return the reference result. Incorrect routes and references flagged as broken are excluded. A reference query is the SQL supplied as the expected answer.
Each model reaches a different set of SQL questions because scoring depends on its database choices. Differences in these subsets and execution setup limit comparisons.
Text-to-SQL findings
Opus 5.5 recorded 87.8% SQL accuracy
claude-opus-5.5 matched the reference result on 444 of 506 scored questions. gemini-3.8-flash followed at 77.2% (342 of 443), then gemini-3.7-flash at 72.5% (363 of 501). These were the three highest SQL accuracy scores in the 49-model panel.
The score excludes incorrect database choices and references flagged as broken. Opus ran through Claude Code and both Gemini models used OpenRouter, so the ranking also reflects different execution conditions and question subsets.
Gemini 3.7 and 3.8 Flash reversed order between routing and SQL
gemini-3.7-flash scored 85.3% on hard routing questions and 72.5% on SQL accuracy. gemini-3.8-flash recorded 69.0% and 77.2%, respectively.
Routing accuracy here covers the hardest questions. SQL accuracy covers each model’s correctly routed questions across all difficulty groups, after excluding flagged references. The two measures use different question subsets.
The agentic RAG benchmark reports database selection separately.
SQL accuracy results
Before reference review measures exact result matching on all correctly routed questions. After answer review adds credit for answers accepted by the jury. Both comparison columns use the original denominator, while SQL accuracy excludes flagged references.
Conditions affecting SQL accuracy
BIRD’s hints increased Gemini Flash’s SQL accuracy by 8.7 points
We repeated gemini-3-flash-preview with BIRD’s supplementary evidence. On questions routed correctly in both conditions, SQL accuracy rose from 67.5% to 76.2%. Strict accuracy, before reference review, rose from 55.0% to 61.6%.
This was a single-model experiment. It does not establish a common uplift for the 49-model panel. The main panel withholds the hints because they can also reveal which database to select.
Models reached different sets of SQL questions
Each model’s database choices determine which questions reach SQL scoring. For claude-opus-5, the strict denominator contains 717 questions, including 156 from the hardest routing group. For nova-lite-v1, it contains 285 questions, including 22 from that group. The shares are 21.8% and 7.7%.
Their SQL percentages therefore combine query-writing performance with the difficulty of the cases that reached scoring. A controlled SQL comparison would supply every model with the correct database for the same questions. That experiment has not been run.
Question selection also matters when comparing AI benchmarks that provide different schemas, hints or tool budgets.
Findings from reference and answer review
The reference audit flagged 236 of 759 queries
Five models assessed each reference query against its question and available schema information. They saw no candidate model answer. A majority of at least three marked 236 references as broken, or 31.1% of the set.
After the audit, the benchmark owner reviewed the flags. Independent human annotators have not repeated that review. These percentages describe the audited subset. They should not be generalized to all of BIRD.
Equivalent answers added 22.7 points to claude-sonnet-5’s score
claude-sonnet-5 passed 211 of 679 strictly scored questions, or 31.1%. A jury accepted another 154 answers as equivalent to the reference. These contributed 22.7 percentage points.
A further 117 answers received credit because the jury accepted the model’s query and rejected the reference. Those contributed 17.2 points. Together, the two categories raised the score to 71.0%, a 39.9-point increase.
Equivalent answers can return the requested information with an extra column or a different projection. Strict comparison can reject that result even when a reviewer accepts it as an answer to the question. Of the increase, 22.7 points come from the jury’s decisions about these cases. Reference defects account for the remaining 17.2 points.
Text-to-SQL benchmark methodology
Questions and execution
BIRD-SQL supplies natural-language questions, SQL references and database contents.1 We used a frozen selection of 759 questions from 11 databases, covering 560 train and 199 dev examples.
Models received short database descriptions and tools for listing tables, reading column definitions and executing SQL. Each question allowed up to nine calls with tools and one final response containing the chosen database and SQL. The schema preview limit was 4,000 characters, and query previews were capped at 50 rows.
The final candidate query and reference were executed against the same database file. Strict scoring compared their results, including an order-sensitive mode for questions that request an ordered answer. Missing, malformed and execution-failing candidate queries received no strict credit on otherwise eligible, correctly routed questions.
For the 46 OpenRouter runs, we requested temperature 0. Twenty-nine used the v3 tool-call recovery layer, 15 predated it, and two combined records from both versions. GPT-6 Astra ran through Codex CLI. Opus 5.5 and Fable 5.1 used Claude Code. All three CLI runs used medium reasoning effort. The GPT-6 Luna, Luna Pro, Sol and Sol Pro runs and GPT-5.2 used OpenRouter. These execution differences prevent attribution of a score gap to the model alone.
Seven runs completed fewer than 759 questions. Missing records are absent from the scoring denominators. The table reports the actual sample sizes for the main and supplementary scores. The ten API additions each completed all 759 questions after retries of failed requests, preserving earlier successful answers.
SQL scoring
The main score excludes flagged questions from both its numerator and denominator. Three references marked uncertain remain in the calculation. The result files call this metric ex_gold_clean.
Two supplementary scores show how review changes the result:
Before reference review, the strict score measures exact execution match on all correctly routed questions. It includes questions with flagged references.
After answer review, the score adds credit when a separate jury accepts an eligible missed answer. It uses the same denominator as the strict score. The result files call this adjudicated accuracy.
Sample size (n) is the number of scored questions. The main score uses a smaller subset than the two supplementary scores.
For claude-sonnet-5, excluding flagged references reduced the accepted answers from 211 to 180 and the scored questions from 679 to 468. Its SQL accuracy was 38.5%.
Exclusion changes both counts. Answer review instead retains the original denominator and adds credit for accepted misses.
Reference and answer review
Auditing references and reviewing candidate answers require different inputs.
Reference auditors were claude-opus-4.8, gpt-5.6-sol, gemini-3.1-pro-preview, grok-4.5 and deepseek-v4-pro. Candidate answers were withheld to keep the flags independent of the model being scored.
Answer adjudication used three jurors from model families different from the candidate’s. The reference and candidate queries appeared in randomized A/B positions without labels revealing their roles. Ambiguous cases and cases with insufficient valid votes received no added credit.
Review coverage is asymmetric. Strict passes remain accepted, while eligible misses receive another assessment. Consequently, adjudication can retain false positives from strict scoring and add false positives of its own. It is a supplementary estimate, with neither guaranteed correctness nor a guaranteed upper bound on true accuracy.
A miss qualified for adjudication when the candidate query executed and the F1 overlap between its result rows and the reference was below 0.5. F1 combines the proportion of candidate rows found in the reference with the proportion of reference rows recovered by the candidate.
For example, gemini-3.5-flash-lite had 346 strict misses. Of these, 314 qualified for review. The jury accepted the candidate in 144 cases: 92 with a rejected reference and 52 with both answers accepted. It classified 105 as model errors and left 65 ambiguous. The other 32 misses comprised 27 with higher result overlap and five execution failures.
The 144 accepted cases are 45.9% of reviewed misses, not of every failed answer.
Add as preferred sourceLimitations
Reference errors remain uncertain. The five-model audit inspected columns from tables used by each reference query, so it may miss an answer located in a different table. The audit can also reject a valid reference. A repeat review by independent human annotators is needed to estimate both kinds of error.
Jury judgments depend on LLM interpretation. No independent human review verified all adjudication verdicts. In the nova-lite-v1 control, accuracy rose from 18.9% under strict scoring to 31.6% after adjudication, showing that review did not automatically produce a high score. That control alone cannot establish jury accuracy.
Strict comparison can fail in both directions. Extra columns or different column order can reject a useful answer. Case normalization and rounding floating-point values to six significant figures can accept differences that matter.
Question-sampling intervals are available for strict scores and SQL accuracy in the result export. They omit reference-label uncertainty, jury error and variation between repeated SQL runs. No SQL repeat analysis is reported here.
These are results for a selected, hint-free workload with a database-routing requirement. They are not directly comparable with the BIRD leaderboard. The public train data also leaves a risk of training-data overlap, as discussed in the routing article.
Conclusion
Opus 5.5 led SQL accuracy at 87.8%, followed by Gemini 3.8 Flash at 77.2% and Gemini 3.7 Flash at 72.5%. The two Gemini models reversed order on hard-question routing.
Each SQL score reflects the questions that model routed correctly and the references retained after review. A controlled SQL comparison would give every model the same questions and database information, with independent checks of reference labels and jury decisions.
Further reading
- Agentic RAG Benchmark: Multi-Database Routing
- 800+ Leading AI Benchmarks
- HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Text-to-SQL Benchmark: SQL Accuracy Across 40+ LLMs}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/text-to-sql}},
note = {AIMultiple. Retrieved September 25, 2026}
}Results and timestamps of 348 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 4 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
9 updatesReplaced text-to-SQL benchmark results with a 36-model panel scored by strict, gold-clean, and adjudicated execution match.
Added a changelog to the Agentic RAG benchmark: multi-database routing and query generation section.
Replaced the methodology section with a summary and a reference to another article.
Reference Links
Comments 1
Share Your Thoughts
Your email address will not be published. All fields are required. Comments are left in their original language.
Curious, how much of the context engineering and specific prompting did you apply in your benchmarks. Or, was it to review the models only? I have found much higher return of correct and consistent responses. A higher fidelity. To do that, I needed to provide a most sophisticated prompt that fed the context window as the question was being asked. Not perfect, but better than those scores represented in this article when using the Grok 4.x .
Great point. This benchmark intentionally uses zero-shot, minimal prompting with temperature=0. No few-shot examples, no domain-specific instructions, no iterative refinement. The goal was to measure each model's baseline text-to-SQL capability. So your experience with Grok 4 getting higher fidelity through sophisticated context engineering is completely expected. A well-crafted prompt with detailed schema descriptions, few-shot examples, and domain-specific rules will improve any model's performance significantly. What this benchmark isolates is how well the model performs out-of-the-box when given only the raw question and retrieved schema, which helps compare the models' inherent SQL reasoning abilities on a level playing field. We'll make this clearer in the methodology section. Thanks for raising it.