Text-to-SQL: Comparison of LLM Accuracy
We ran 36 large language models over 759 questions from BIRD-SQL, each model writing SQL against a database it had to identify for itself out of 11 candidates. Every parseable query was executed against the real database and its result set compared with the result set of BIRD’s gold query. Missing, malformed and execution-failing queries scored as misses.
Every run was at temperature 0, zero-shot, with BIRD’s domain hint withheld. Nineteen of the 36 runs used the tool-call recovery layer described on the routing page, which parses tool calls the standard parser rejects. Fifteen runs predate it and two straddle the change.
Metrics explained
Strict execution match. Execution match over the questions the model routed correctly. Conditioning on the correct route removes direct wrong-database failures, but it does not isolate SQL ability fully, because each model reaches a different subset of questions.
Gold-clean. The same measurement with the questions whose gold SQL we judged broken dropped from the denominator. Those questions are not scored as passes, they are removed.
Adjudicated. The upper end. A blind three-model jury, drawn from families other than the model under test, reviews every answer the strict comparison rejected, where the model routed correctly, the SQL runs, and its rows overlap the gold’s by less than half. It decides whether the query is a different-but-equivalent formulation, whether the gold is wrong, or whether the model is wrong. Equivalent answers and broken golds are credited.
All three start from the full 759-question set rather than the hard subset, and none of them uses 759 as its denominator. Each is measured over the questions the model routed correctly, and the gold-clean column then drops the flagged golds. Chance does not apply to this axis, because a query either returns the gold result set or it does not.
Text-to-SQL benchmark findings
Right answers in the wrong shape cost one model 22.7 points
claude-sonnet-5 records 0.311 on strict execution match, 33rd of the 36 models and the lowest of any Anthropic row. Under blind adjudication the same run scores 0.710, 0.007 below qwen3.6-27b’s 0.717.
The 39.9-point gain splits in two. 154 of its 679 scored questions, 22.7 points, are answers the jury read as equivalent to the gold in a different shape. A further 117, 17.2 points, are questions where the jury judged the gold itself broken. Answer style explains the first of those two, not the sum. Harness failure accounts for none of it, because the run recorded no missing queries, one crash and no unrecovered answers.
Those same 154 answers are 34% of the 453 sonnet-5 misses that reached adjudication. Format-equivalent means the query returns the right information in a different projection, an extra column, a different column order, or a count where the gold returned the rows. Strict execution match scores that as a failure.
The tendency runs by model family rather than by capability tier. Rich-projection families lose a quarter to a third of their adjudicated misses this way, at Claude 25 to 34%, Kimi 24 to 34%, OpenAI’s 5.x reasoning line 26 to 32%, DeepSeek 31%, MiniMax 26 to 30% and GLM-5.x 28%. Gold-shaped writers lose far less, at Google 9 to 17%, Llama 9%, Mistral 18%, Grok 19% and OpenAI’s own small models 19 to 20%. Grok broke our first explanation of this pattern. It sits in the reasoning tier and loses 19%.
The adjudicated measurement places sonnet-5 17th, a gap of 16 positions.
The jury finds 46% of one model’s rejected answers are not model errors
We took 314 of the 346 answers the strict comparison rejected for gemini-3.5-flash-lite on questions it routed correctly, the ones whose SQL ran and whose rows overlapped the gold’s by less than half, and sent them to the blind, position-shuffled three-model jury. Those 346 span every difficulty bucket rather than the hard ones alone, so they match the population the SQL columns cover.
The jury split them four ways. Gold-defect, where the model is right and BIRD’s gold query is wrong, took 29.3%. Format-equivalent took 16.6%, genuine model error 33.4%, and ambiguous 20.7%. Adding the first two, 144 of the 314 adjudicated misses (46%) are not model errors. That share is over the 314 that reached the jury, not over all 346 rejected answers. The 32 held back either failed to execute or overlapped the gold too closely to be a clean disagreement.
Comparator strictness does not explain the gap. Loosening column-order matching buys 0.5 points, and 92% of the misses overlap the gold result by less than 0.5. Adjudication, rather than a looser comparator, puts that model in a range of 0.606 to 0.687, against a strict 0.464.
The misses the jury called gold-defect concentrate where no published correction reaches, at 33% of this model’s train-split misses against 20% of its dev-split misses. Every deterministic correction available to us was already spent. Re-scoring against BIRD’s own corrected dev release flipped 0 of 92 dev misses, an external correction set covered 39 of them and flipped 1, and a MySQL-against-SQLite engine audit found 4 divergent golds across the whole set, together about 1% of the 314.
Our five-model audit flags 31.1% of BIRD’s gold queries as broken
We audited the gold itself. Five frontier models, one per family, rated all 759 gold queries against their questions and schemas. No juror was ever shown any model’s output. The panel flagged 236 of 759 golds as broken, 31.1%, at a cost of $20.55 and with no failed jury calls.
The rate splits by BIRD partition, at 204 of 560 train golds (36%) against 32 of 199 dev golds (16%). An independent three-model panel run earlier reached 29.1%, and 207 of its 221 broken flags, 94%, are broken in this one. Over all 759 questions the two panels agree on 92.9%.
One published estimate covers BIRD’s train split, MotherDuck’s audit of 151 examples, and it puts the rate at 32.5%. The peer-reviewed audits of BIRD cover the dev split by explicit design, so MotherDuck’s 151 examples stand as the single external check on the 560 train questions in our frozen set, 73.8% of it.1
Dropping the broken golds from the denominator moves the best strict writer from 0.551 to 0.677, and per-model gains span +4.5 to +13.3 points. The top tier holds its order and mid-table neighbours move by up to three positions.
A 27B model ranks fifth on strict SQL accuracy
qwen3.6-27b records 0.471 strict and 0.590 gold-clean, fifth on the panel behind gemini-3-flash-preview (0.551 / 0.677), gemini-3.1-pro-preview (0.536 / 0.669), claude-fable-5 (0.531 / 0.655) and claude-opus-5 (0.523 / 0.643). Every OpenAI, Kimi and DeepSeek row records a lower strict score, and the run cost $15.28.
The column is measured over each model’s own correct routes, so a weaker router is graded on an easier selection. qwen3.6-27b routes 0.679, and the limitations section measures that effect at a 0.96 correlation between routing accuracy and denominator difficulty.
The ordering changes once the bracket’s upper end is used. On adjudicated scores qwen3.6-27b is fourteenth at 0.717 while claude-opus-5 leads at 0.824.
Routing skill and SQL writing skill are separate axes
kimi-k3 routes 0.832, behind claude-opus-5 at 0.848 and level with qwen3.8-max, and writes 0.482 gold-clean SQL against a panel best of 0.677. gemini-3-flash-preview reverses it, routing 0.753 with that panel-best gold-clean SQL. gpt-5.6-terra routes at 0.772 and writes 0.447.
The two axes are not measured on one denominator. Each model writes SQL for the questions it routed correctly and no others, and the better routers earn a harder set, which pulls any measured association between the axes downward.
The gold audit and the adjudication jury
Two juries did two different jobs.
The gold-validity jury rates gold queries. Its five jurors (claude-opus-4.8, gpt-5.6-sol, gemini-3.1-pro-preview, grok-4.5, deepseek-v4-pro) each see a question, the gold SQL, and the column lists of the tables that query touches, and answer whether the gold answers the question. No model output is ever shown. Its verdicts are frozen in a hash-pinned file and feed the gold-clean column.
The adjudication jury rates a specific model’s miss. Three models drawn from families other than the model under test see the question, the gold query and result, and the candidate query and result, with position shuffled, and vote on which of four categories the miss belongs to. It runs per model at about $6, and its verdicts feed the adjudicated column. Adjudication across the full 36-model panel cost $208.81.
The adjudicated column is an adjudication-adjusted, optimistic endpoint rather than independent ground truth. It can err in either direction. An ambiguous answer that was in fact correct gets no credit, and a jury false positive credits one that was not. Three measured properties bound how far it can be pushed.
The review is asymmetric. Misses get a second look and passes never do, so an error in the passing direction cannot be caught. Ambiguous verdicts, 15 to 23% depending on the model, are never credited, and near-misses and non-executing queries stay misses.
It does not erase the weak-model floor. As a negative control we adjudicated nova-lite-v1, the weakest row on the panel. Its score rises from 0.190 to 0.316 and stays 0.150 below the next row. The floor holds at llama 0.466, mistral 0.509 and gpt-5.4-nano 0.512.
The gold-defect share tracks model strength at a rank correlation of 0.906, measured across 33 models. gpt-5.6-sol’s misses are 40% gold-defect and 19% genuine error, where llama-4-maverick’s are 13% and 50%.
Twenty-four of the 36 models land at or above 0.68 adjudicated.
How SQL generation works in this benchmark
The model never receives a schema up front. It picks one of 11 database tools, reads the table list that comes back, asks for a table’s columns when it needs them, and may run exploratory queries against the database it has settled on before committing. Schema output is capped at 4,000 characters and query results at 50 rows.
The final query arrives on a mandatory finalization call sent with no tools attached, alongside the database the model declares. Scoring executes that query and BIRD’s gold query against the same database file and compares the two result sets, with a NULL sentinel and an order-sensitive mode for questions that specify an order.
Comparison is the strict step, and it is where a right answer can score as a miss. A query returning the same rows with an extra column, in a different column order, or as a count where the gold returned the rows, fails the comparison. That is the gap the adjudication jury measures rather than the one a looser comparator would close, which buys 0.5 points.
Benchmark methodology for text-to-SQL
This benchmark shares its harness with the agentic RAG benchmark, which describes the database selection, the difficulty taxonomy, the anonymisation, the agentic loop and the turn budget in full. Both pages report the same 759-question frozen BIRD-SQL subset, run over 11 databases the model has to choose between, at temperature 0 with the domain hint withheld. The routing page carries the routing axis, and this page carries the SQL axis.
Scoring: execution match. The model’s final query and BIRD’s gold query are both executed against the real database and their result sets compared, with a NULL sentinel and an order-sensitive mode for queries whose question specifies an order Denominator: questions the model routed correctly Reported form: a three-value bracket, strict execution match, then gold-clean, then adjudicated Gold audit: 5 frontier families, one model each, 759 golds rated, 236 flagged broken, hash-pinned Adjudication: 3 models per candidate, family-disjoint from the model under test, blind and position-shuffled Panel: 36 models, single run each, $874.53 for the runs and $208.81 for the adjudication
Why no LLM judges correctness at the floor. We measured the alternative before choosing. Across 2,203 records from the predecessor benchmark an LLM judge never failed a query that execution passed, at any threshold. That judge was shown both result sets, so its verdict is not independent of the execution outcome and the zero does not establish that a judge cannot be stricter. Execution stays as the floor and the jury appears higher up the bracket where its leniency is the point.
Why the hint is withheld. BIRD ships a domain hint with each question. Supplying it is worth 6 to 9 execution-match points, and it also moves routing accuracy by 5.7 points, which makes it a leak on the routing axis. Both pages therefore report the hint-free condition, and the SQL numbers here sit below what the same models would score under BIRD-standard conditions.
SQL accuracy across the panel, three ways
Ranked by the adjudicated column. Six rows ran short of 759 records after two retry passes, and the missing questions are absent from every denominator rather than counted as failures.
Limitations of the SQL axis
The gold is borrowed and disputed. Our 31.1% broken-gold figure is our own audited subset, not multi-annotator ground truth. There is no blinded duplicate pass, no inter-annotator agreement figure, and the human verification was done by the benchmark owner rather than an independent annotator. Jurors saw the column lists of the tables the gold query touches, so a gold that queries the wrong table is undetectable from that view and such defects are missed. The panel can also flag a working gold, an error in the other direction. No independent annotator has repeated the pass, so the residual error is unmeasured either way and the figure is not a floor.
The adjudicated column is an LLM instrument, not a second ground truth, and not a strict upper bound. Its three jurors are models, so the column inherits whatever a model panel gets wrong about SQL equivalence, and no human re-read the verdicts.
The comparator errs in both directions. Column order and column count cause false negatives, while case folding and six-significant-figure float rounding cause false positives. Comparator error and gold-query error are separate sources of uncertainty and can move a score in different directions.
Strict execution match is measured over each model’s own correctly-routed questions, and the better routers earn a harder set. Routing accuracy tracks the difficulty of the questions that reach a model’s SQL denominator. Across the 36 models that correlation is 0.96 (Pearson, over the mean cross-database neighbour count), and 0.97 against the share of hardest questions in that denominator. In concrete terms claude-opus-5 writes SQL for 717 questions of which 21.8% are among the 184 hardest, while nova-lite-v1 writes SQL for 285 questions of which 7.7% are. A weak router is graded on an easier selection. Treat this column as a per-model diagnostic rather than a cross-model ranking, and read the three columns as tiers. Closing the gap needs an oracle-route run, where every model writes SQL for the same questions. That run has not been done.
The confidence intervals cover sampling noise alone. The table above prints point estimates, and Wilson intervals for the strict and gold-clean columns ride in the published CSV beside every row. The intervals carry question-sampling noise and neither the juries’ uncertainty nor the gold flags’. Two panel runs repeated under identical conditions moved routing accuracy by 1.6 to 2.7 points, and this page reports no repeat measurement for the SQL columns.
These numbers are hint-free by construction, so they are not comparable to BIRD leaderboard scores for the same models.
Conclusion
Strict execution match against BIRD’s gold queries spans 0.190 to 0.551 across the 36 models, and that same panel spans 0.316 to 0.824 once a blind jury has re-read every miss. The distance between those two readings is the finding. For claude-sonnet-5 it is 39.9 points, of which 22.7 come from answers that return the right information in a shape the comparator rejects.
For a workload scored by strict execution match, gemini-3-flash-preview recorded 0.551 raw and 0.677 gold-clean at $7.67 per run. For a workload where an equivalent answer in a different projection is acceptable, claude-opus-5 recorded 0.824 adjudicated against gpt-5.6-sol’s 0.785, inside one confidence interval. For a per-model comparison of any kind, the denominator differs by model, so the three columns are diagnostics rather than a controlled leaderboard.
The ceiling on this axis is the gold, not the models. A third of BIRD’s gold queries fail our audit, the defects concentrate in the train split where no published correction reaches, and the two instruments that see past them are both LLM juries rather than human annotators. Moving the axis forward needs an oracle-route run so every model writes SQL for the same questions, and independent double-human annotation of a stratified gold sample. Neither has been done.
Add as preferred sourceFurther reading
- Agentic RAG Benchmark: Multi-Database Routing Across 36 LLMs
- 200+ Leading AI Benchmarks
- HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark
- +100 Datasets for ML & AI Models
- Best RAG Tools, Frameworks, and Libraries
FAQs
Because the gold queries are BIRD’s and a third of them fail our audit. Strict execution match is the floor, the gold-clean column removes the questions whose gold we judged broken, and the adjudicated column credits answers a blind jury read as equivalent. claude-sonnet-5 moves from 0.311 to 0.710 across that bracket, so a single number would misstate the panel by up to 39.9 points.
No. BIRD ships a domain hint with each question and supplies the database. This benchmark withholds the hint and makes the model choose the database out of 11. The hint alone is worth 6 to 9 execution-match points.
Not on this panel. Each model writes SQL for the questions it routed correctly, so a better router earns a harder denominator, at a 0.96 correlation between routing accuracy and denominator difficulty. kimi-k3 routes 0.832 and writes 0.482 gold-clean, while gemini-3-flash-preview routes 0.753 and writes the panel-best 0.677.
A gold query that does not answer its own question, as rated by five frontier models from five different families, none of which was shown any candidate answer. The panel flagged 236 of 759, and an earlier three-model panel independently flagged 221, of which 207 overlap.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Text-to-SQL: Comparison of LLM Accuracy}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/text-to-sql}},
note = {AIMultiple. Retrieved August 7, 2026}
}Reference Links
Comments 1
Share Your Thoughts
Your email address will not be published. All fields are required. Comments are left in their original language.
Curious, how much of the context engineering and specific prompting did you apply in your benchmarks. Or, was it to review the models only? I have found much higher return of correct and consistent responses. A higher fidelity. To do that, I needed to provide a most sophisticated prompt that fed the context window as the question was being asked. Not perfect, but better than those scores represented in this article when using the Grok 4.x .
Great point. This benchmark intentionally uses zero-shot, minimal prompting with temperature=0. No few-shot examples, no domain-specific instructions, no iterative refinement. The goal was to measure each model's baseline text-to-SQL capability. So your experience with Grok 4 getting higher fidelity through sophisticated context engineering is completely expected. A well-crafted prompt with detailed schema descriptions, few-shot examples, and domain-specific rules will improve any model's performance significantly. What this benchmark isolates is how well the model performs out-of-the-box when given only the raw question and retrieved schema, which helps compare the models' inherent SQL reasoning abilities on a level playing field. We'll make this clearer in the methodology section. Thanks for raising it.