AIM-Agentic RAG Benchmark: Routing Across 11 SQL Databases
We benchmarked 70+ LLMs on 759 questions that require choosing among 11 SQL databases. The benchmark measures whether each model identifies the right database, explores alternatives and states a final choice.
Routing accuracy is the percentage of scored questions for which the model explicitly names the correct database in its final answer. The headline uses the 184 questions flagged as difficult by both our similarity test and a jury of three LLMs. Missing explicit declarations receive no credit.
Database routing findings
Gemini 4 Argon recorded the highest routing accuracy at 94.0%
gemini-4-argon selected the correct database for 173 of 184 hard questions. claude-opus-5.5 followed at 90.2% (166 of 184). Muse Wonder recorded 86.4% (159 of 184).
Gemini 4 Argon led across all questions at 98.4% (747 of 759). It ran through OpenRouter. The observed scores reflect both model and execution conditions.
DeepSeek V4 Flash 0731 recorded 82.1% routing accuracy at $1.37
DeepSeek V4 Flash 0731 answered 151 of 184 hard questions correctly. Its retained responses across all 759 questions cost $1.37. Qwen3.8 Flash scored 79.3% at $2.08, while MiMo-V2.6-Pro reached 82.6% at $4.17.
These costs sum the successful responses retained in each result file. Failed and replaced retry attempts are excluded. These totals depend on the providers and caching conditions used during measurement. The cost chart includes 73 measured API totals and three separately labeled OpenRouter-equivalent estimates for Opus 5.5, Fable 5.1 and GPT-6 Astra. These estimates apply 1 October 2026 API prices to recorded CLI tokens and cache usage. They do not measure subscription spending.
Grok 4.6 gained 41.8 points after its first database choice
Grok 4.6’s hard-question routing accuracy rose from 38.0% on its first probe to 79.9% on its final declaration. GPT-6 Sol Pro gained 34.2 points within its run. Nova Lite lost 14.0 points.
These changes use questions with a single database on the first turn. They describe recorded trajectories. No separate intervention isolated the benefit of extra calls.
Across all 76 models, 7,570 hard-question records included a database switch:
This table takes the first database in recorded call order, including multi-database first turns. Final credit requires an explicit correct declaration. The percentage-point changes above use the narrower single-first-choice subset.
Database routing results
Models are sorted by routing accuracy, highest first. The methodology identifies differences in the software used to run them. These conditions matter when interpreting small score gaps.
The intervals estimate question-sampling uncertainty. Recorded costs cover the successful records in each run. They reflect the providers and caching conditions used at the time of measurement.
Conditions affecting database selection
Similar databases reduced routing accuracy by about 20 points
We tested the database selection method in a separate experiment on 128 questions. Each question kept its correct database, while the other 10 candidates came either from our selected group or from a random draw.
On the hard subset, claude-opus-4.8 scored 71.3% with similar candidates and 92.0% with random candidates. gemini-3.5-flash scored 77.0% and 96.6%. Both differences had p-values below 0.001 in paired tests.
This experiment used descriptions and one model response per question, without the agentic tool loop. It supports the narrower finding that selecting similar candidates makes database choice harder. The measured gap should not be presented as the effect of agentic exploration.
Names alone produced 68.8% and 71.1% accuracy in a pilot
In another 128-question pilot, claude-opus-4.8 selected the right database 68.8% of the time from names alone. Its score with names and descriptions was 67.2%. gemini-3.5-flash scored 71.1% from names and 75.0% from the full catalog.
Names such as california_schools and toxicology reveal the subject directly. For the main benchmark, we replace database names with aliases from db_01 to db_11. Descriptions still explain each database’s domain, and tool responses expose its table and column names.
Reliability of routing and cost comparisons
Repeat runs changed routing accuracy by 1.6 to 2.7 points
claude-opus-5 selected the correct database for 156 of 184 hard questions on its first run and 151 on its repeat. Its accuracy fell from 84.8% to 82.1%. qwen3.8-max moved from 153 correct answers to 150, a decline from 83.2% to 81.5%.
Both repeats used the same questions, anonymization, temperature and provider settings as their original runs. qwen3.8-max changed its answer on 13 hard questions even though its total moved by three. Improvements on some questions canceled errors on others.
These repeats show that small score gaps can disappear on another run. One repeat per model cannot establish a distribution of outcomes, and the other 35 models have no repeat measurement.
Recorded costs depend on provider and caching settings
Prompt caching reuses previously processed input to reduce billed input costs. The claude-opus-5 run cost $62.78, compared with an estimated $105.54 at uncached list rates for the same recorded usage. That estimate implies a 40.5% saving. It is an accounting comparison, without a separate uncached accuracy run.
Agentic RAG and database selection
Agentic RAG gives a model control over retrieval decisions. It can choose a source, inspect the response and make another request. A simple RAG pipeline retrieves context through a predetermined sequence before generating an answer.
This benchmark tests source selection through SQL database tools. It contains no document index, retrieved passages or answer-grounding score. Results therefore describe one part of an agentic retrieval system. Document search is covered separately in our agentic search benchmark.
A model starts with short database descriptions. It can request table lists, inspect columns, run SQL and change databases. Its final response contains a database choice and a query. Our text-to-SQL benchmark reports SQL accuracy and two supplementary review scores.
Agentic RAG benchmark methodology
The question set is a frozen subset of BIRD-SQL, which pairs natural-language questions with SQL reference queries.1
Two signals define difficulty. The similarity test counts how many of a question’s 20 nearest neighbors belong to other databases. Three LLMs assess whether the question is confounding across databases. Membership in the hardest group requires at least five cross-database neighbors and a confounding jury label.
Routing uses the database named under the explicit selected_database field. A route inferred from prose is excluded from the headline. SQL correctness is scored separately and does not determine routing credit.
The harness is the software that presents tools and records model responses. The panel contains 64 OpenRouter runs, two Claude Code runs and one Codex CLI run. Among the API runs, 47 used v3, 15 used an earlier version, and two combined records from both versions. In v3, a recovery layer parses supported alternative tool-call formats. For llama-4-maverick, the recorded non-native format share was 97.9%, so that result depends heavily on the adapter. minimax-m2.7 was excluded because it did not issue the database tool calls required by the task.
GPT-6 Astra ran inside Codex CLI. Opus 5.5 and Fable 5.1 ran through Claude Code. All three used medium reasoning effort and completed 759 questions. Claude Code runs used subscriptions. Their reported list-price equivalents are excluded from the API cost comparison. Astra reported no dollar cost. The GPT-6 Luna, Luna Pro, Sol and Sol Pro runs and GPT-5.2 used OpenRouter.
Add as preferred sourceLimitations
The hard subset concentrates on commerce. Four databases contribute 171 of its 184 questions, or 92.9%. Five of the 11 databases contribute none. This supports conclusions about choosing among similar databases within a domain, with limited evidence for other domain combinations.
Public benchmark data may have appeared in model training. Controls using 138 validated paraphrases and 65 newly authored questions found no accuracy decline in the tested lower-cost models. Their coverage was limited to six and three databases respectively. They do not resolve contamination for the full panel, and no control used databases released after the models’ training cutoffs.
Schema truncation can hide useful tables. In works_cycles, the 4,000-character limit leaves 26 of 66 tables visible in the initial schema listing. The model can request further details, but the initial view is incomplete.
Conclusion
gemini-4-argon recorded the highest hard-question routing score at 94.0%. DeepSeek V4 Flash 0731 reached 82.1% with $1.37 in retained-response costs across the full run.
Execution setup, retries and provider pricing limit comparisons. These results measure database selection among similar SQL sources. Document retrieval and grounded answer generation require separate evaluations.
Further reading
- Text-to-SQL: Comparison of LLM Accuracy
- RAG Frameworks: LangChain vs LangGraph vs LlamaIndex
- Best RAG Tools, Frameworks, and Libraries
- Agentic Search in 2026: Benchmark 8 Search APIs for Agents
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{AIM-Agentic RAG Benchmark: Routing Across 11 SQL Databases}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/agentic-rag}},
note = {AIMultiple. Retrieved September 29, 2026}
}Results and timestamps of 1.1 thousand data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 7 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
16 updatesUpdated the 'What makes a question hard in this benchmark?' section with new content.
Updated the methodology to clarify how the hardest questions are distributed across databases.
Replaced the Agentic RAG benchmark methodology section with a new one that covers 36 LLMs and 11 databases.
Added new models to the benchmark: Claude Fable 5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3, Claude Opus 4.7.
Updated the methodology section with new details on database environment, agent architecture, and evaluation process.
Added a section on long-context models versus agentic RAG.
Removed additional agentic metrics from the methodology.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.