Agentic RAG Benchmark: Multi-Database Routing Across 36 LLMs

We benchmarked 36 large language models on cross-database routing. Each model receives a natural language question and 11 SQL databases described at paragraph level, then has to decide which database holds the answer before it writes any SQL. The 11 databases were drawn from 80 BIRD-SQL candidates by clustering their description embeddings, so the candidates are semantically close to each other, and their real names are hidden behind the labels db_01 through db_11.

Every model saw the same frozen set of 759 questions at temperature 0, with BIRD’s domain hint withheld.¹ Each database is exposed as one tool with three actions, schema listing, table detail and query execution, under a ceiling of 10 API calls per question. The tool list is permuted for every question, so candidate position cannot be read as routing skill.

Routing accuracy on the hardest questions. The share of the 184 hardest questions where the model’s declared database matches the gold database. A question counts as one of the 184 when two independent signals agree. At least 5 of its 20 nearest neighbours in embedding space must belong to other databases, and a three-model jury must label it confounding. This is the headline routing metric.

The 184 questions come from 6 of the 11 databases, in shares of 51 (regional_sales), 50 (retail_world), 43 (superstore), 27 (works_cycles), 8 (debit_card_specializing) and 5 (college_completion). The four commerce databases carry 171 of the 184, or 93%. The other five databases contribute none, because they are semantically isolated enough that no question of theirs clears both signals. The number therefore measures routing among databases built to be hard to tell apart, and it is dominated by one commerce cluster rather than spread across eleven domains.

Declared route. The database the model states in its final answer under an explicit selected_database key. A run that names no database scores zero rather than inheriting its last tool call. Routes recovered from prose are excluded and reported separately.

Agentic RAG benchmark findings

Re-running the panel leader drops it to sixth

claude-opus-5 routed 0.848 of the hardest questions correctly, 156 of 184. We ran it a second time under identical conditions, same frozen set, same anonymisation, temperature 0, and it recorded 0.8207, 151 of 184. That second score is the published score of claude-fable-5, which sits sixth. Ranks 1 through 6 span 156 of 184 correct answers down to 151 of 184, a distance of five questions, which is the distance the leader itself moved between two runs.

A second model reproduced the pattern. qwen3.8-max moved from 0.8315 (153/184) to 0.8152 (150/184). The easy and medium buckets moved by 0.0000 in both models, and the rest of the movement sits in the three hard buckets, with qwen3.8-max also moving from 153 to 152 on the questions the neighbour count flags.

qwen3.8-max has the smaller aggregate delta, 1.6 points against 2.7, and the larger disagreement with itself, routing differently on 13 of 184 hard questions where opus-5 differs on 8. A model can reproduce its own score while disagreeing with itself on one hard question in fourteen.

Two limits travel with this. One repeat gives an absolute difference, not a standard deviation, so 1.6 to 2.7 points is a floor on the noise rather than a description of it. And 34 of the 36 panel rows have no repeat run at all. Read these scores in tiers, not ranks.

The 11 database names alone route within 4 points of the full catalogue

Shown the real database names and no descriptions at all, claude-opus-4.8 routes at 0.688 and gemini-3.5-flash at 0.711. Shown the full catalogue, real names plus all 11 paragraph descriptions, the same two models score 0.672 and 0.750, so for claude-opus-4.8 the names alone score higher than the whole catalogue.

Replacing the names with db_01 through db_11 costs opus-4.8 7.0 points and gemini 4.7. A run under real names measures name recognition alongside comprehension, which is why every number published here comes from the anonymised condition.

A third condition separates the two channels. Keeping the real names and permuting the descriptions between databases collapses routing to 0.148 and 0.055, measured on the same two models over a 128-question pilot.

Databases chosen to be confusable are about 20 points harder than random ones

We swapped the 10 distractor databases for 10 drawn at random from BIRD’s other 69, over the same 128 questions, same gold database, both arms anonymised. Hard_strict routing moves from 0.713 to 0.920 for claude-opus-4.8 and from 0.770 to 0.966 for gemini-3.5-flash. Both arms answer the same questions, so the test is McNemar with a continuity correction over the discordant pairs, giving p = 8.6e-04 and p = 2.4e-04. Eleven randomly chosen candidates leave both models near ceiling.

On the easy bucket the same McNemar contrast is not significant, p = 0.48 and p = 1.00. Cluster-selected distractors were meant to interfere with the questions the neighbour count flags as confusable and to leave the rest routable regardless of panel, and that is the pattern the two arms show.

This ablation is description-only and single-shot rather than agentic, with one random panel draw per question. It licenses the claim that cluster-discovery selection produces measurably harder routing than random selection. It does not license the claim that the neighbour count itself causes difficulty, which we tested separately and found non-significant on its own.

The top routing tier is not a price tier

qwen3.8-max and kimi-k3 both route 153 of 184 hard questions, 0.8315 each, at $25.14 and $40.34 per 759-question run. claude-fable-5 costs $121.55 and routes 151. Six models sit within 0.03 of the leader across a 5x range in run cost.

The cost column records what the run was billed, not model economics. OpenRouter routes a model slug to whichever upstream provider is serving it, and the identical 594 questions on the same model have cost $2.09 in one run and $8.33 in another. Read the column as an order of magnitude. One direction survives that drift. The two most expensive runs on the panel, claude-fable-5 at $121.55 and claude-opus-4.8 at $117.82, route no higher than qwen3.8-max at $25.14.

A model can run the agentic loop without benefiting from it

Restricted to questions whose first turn probed a single database, the accuracy gap between the first database a model touches and the one it finally declares is +25.0 points for claude-opus-5 and +38.0 for grok-4.5. For gpt-5.4-mini that gap is 0.0. For gpt-oss-120b it is -3.8 and for nova-lite-v1 it is -14.0. Both of those models abandon a correct first probe more often than they recover from a wrong one.

The two groups differ in exploration breadth. Pooled over the panel, models probe 1.12 distinct databases on easy questions and 1.93 on the hardest. kimi-k3 goes from 1.07 to 2.27 and grok-4.5 from 1.06 to 2.21, while gpt-oss-120b goes from 1.01 to 1.10 and gpt-5.4-nano from 1.11 to 1.20.

gpt-5.4-nano spends 6.5 turns per question and emits tool calls throughout, so the loop does run. Running the loop gains it 2.2 points.

Some providers ignored our request for sequential tool calls. Where a first turn touched several databases at once we drop it from this comparison, which then runs over fewer than 184 questions for some models (qwen3.8-max 86, glm-5.2 101, kimi-k3 105).

Probing, switching, and the gap between first probe and declared route

Exploration scales with the difficulty label. Pooled across all 36 models and 759 questions:

Across the 3,122 hardest-question records where a model changed route at least once, 1,717 moved from a wrong database to the right one and 90 moved the other way, a net gain of 1,627 records at a ratio near 19 to 1. A further 983 started on a wrong database and never reached the right one, and 332 left the correct database and came back to it. All four counts use the explicit-declaration route, the same definition as every other routing number here.

This asymmetry is why the benchmark scores the declared route instead of the first probe. The earlier version recorded the agent’s first database function call as its routing decision. On this panel that convention understates 30 of the 36 models by between 14.7 and 38.0 points.

Cost per run across the 36-model panel

The 36 runs cost $874.53 in total, from $0.47 to $121.55. Six of them scored fewer than 759 questions after residual provider errors (gemini-3.1-pro-preview 744, gemini-3-flash-preview 755, claude-haiku-4.5 750, nova-lite-v1 750, gpt-5.6-luna 758, gpt-5.6-terra 758).

Turn count ranges from 3.79 to 7.64 across the panel and feeds the bill alongside the per-token price. claude-opus-5 averages 3.92 turns per question at $62.78, gemini-3.1-pro-preview 7.02 at $80.01, and gpt-oss-120b 3.79 at $0.47.

Three of the 36 runs used prompt caching and the other 33 did not, so the rows are not billed on the same basis and the three cached rows sit lower than an uncached run of the same model would.

Prompt caching, and what the position-bias control costs

In this benchmark 44% of every tool-loop call is a byte-identical prefix, the system prompt at 362 tokens plus the 11 tool definitions at 2,259, so 2,621 tokens of a call that averages around 6,000. Three runs carried an explicit cache breakpoint on that prefix. claude-opus-5 read 62.2% of its input from cache and was billed $62.78 against $105.54 at list price, 40.5% off. qwen3.8-max read 68.2%, the highest we measured, on Alibaba rather than Anthropic.

The saving is bounded by a design decision. The tool list is permuted per question, so the prefix differs from byte 0 between questions and each question’s cache is written and read inside its own five or so calls, never across the run. Freezing the tool order would widen the cache and save about $8.95 per 759-question run. We kept the permutation and priced it.

Caching changed the bill and not the model’s input. A negative-test suite captures the real request bodies with the flag on and off and requires them byte-identical once the cache markers are stripped, and its load-bearing test mutates a single character under a marker and requires the check to catch it. One cost defect survived that suite anyway. The finalization turn deliberately sends no tool definitions, so its prefix misses every cached entry, and leaving a breakpoint on it wrote about 3.2M tokens of cache that nothing could read, roughly $4 of opus-5’s bill. An adversarial review after the change had shipped found it, with every guard green.

Agentic RAG and standard RAG

Retrieval-augmented generation puts a retrieval step in front of a language model. The question is embedded, the nearest chunks come back from one index, and the model answers from them. The path is fixed, and nothing in the pipeline chooses anything.

Agentic RAG hands the retrieval decisions to the model. It picks which source to query, reads what comes back, and can query again, switch source, or refine the query before it answers. Retrieval stops being a step that runs once and becomes a loop the model drives.

That first choice, which source holds the answer, is what this benchmark measures. Each model sees 11 SQL databases described at paragraph level, with their names hidden, and has to pick one before it writes any SQL. It may then probe a second and a third and change its mind. On the hardest questions the panel probes 1.93 databases on average, and 47.3% of runs touch more than one.

How agentic database routing works

The harness gives the model a question and a catalogue of 11 paragraph descriptions, one per database, with no table or column names in them. Alongside the catalogue it attaches 11 tools, one per database, each exposing three actions: get_schema returns the table list, get_table_schema returns one table’s columns, and execute_query runs SQL against that database. Schema output is capped at 4,000 characters and query results at 50 rows.

From there the model drives. It picks a tool, reads the response inside the same conversation, and may probe a different database, ask for one table’s detail, or run a query against the database it has settled on. It may switch database on any turn, and 47.3% of the hardest questions do get probed on more than one.

The loop ends in one of two ways. Either the model stops asking for tools, or it uses its ninth tool-enabled call. Either way the harness then sends one finalization call with no tools attached, in which the model states its chosen database and its SQL. That call runs unconditionally, for every model and every question, so the call budget is nine tool turns plus one, at most 10 API calls.

The equal budget is a correction to an earlier design. That design added the finalization turn for models that had not yet emitted SQL, which handed an extra call to one output habit rather than to ability. The ceiling is also not a quota. The panel averages between 3.79 and 7.64 tool turns per question, because a model may stop probing early.

Agentic RAG benchmark methodology

The benchmark measures two competencies separately. Routing decides which database holds the answer, and SQL generation decides whether the query returns the right rows. Routing is the contribution and carries the headline. SQL correctness is measured against BIRD’s gold queries and is reported with its own uncertainty on the text-to-SQL benchmark page.

• Dataset: BIRD-SQL train and dev splits, 759 questions frozen and hash-pinned.
• Databases: 11, selected from 80 BIRD candidates by agglomerative clustering of their description embeddings at cosine 0.65.
• Difficulty signal: each question’s 20 nearest neighbours counted by how many belong to a different database.
• Difficulty buckets: easy 222, medium 118, qq_only_hard 165, jury_only_hard 70, hard_strict 184.
• Primary condition: anonymised. Database names replaced by db_01 through db_11 in tool names and in schema output.
• Evidence condition: none. BIRD’s domain hint is withheld from every model.
• Tools: 1 per database, each exposing 3 actions (get_schema, get_table_schema, execute_query), 11 in total, order permuted per question.
• Vocabulary: an API call is one request to the model. A tool turn is an API call that carries the tool list and may come back with tool calls. A tool call is one database action inside a turn, at most 11 per turn. The finalization call is the last API call, sent with no tools.
• Turn budget: at most 10 API calls per question, up to 9 tool turns plus exactly one finalization call that always runs.
• Tool turns used: 3.79 to 7.64 per question on average across the panel, because a model may stop probing early. The recorded field counts tool turns and excludes the finalization call, so its ceiling is 9 rather than 10.
• Temperature: 0. Sequential tool calls requested. Questions shuffled once with a fixed seed before any slice.
• Routing metric: final routing accuracy over the explicitly declared database.
• SQL metric: execution match against BIRD’s gold query, reported as a three-value bracket.
• Panel: 36 models, single run each except two repeats, $874.53 on the panel runs plus $85.37 on the repeats.

Which databases carry the hard questions. Five of the eleven contribute none of the 184 hardest questions at all. They are california_schools plus the four semantically isolated ones (financial, synthea, superhero, toxicology), which do still supply 27 of the 165 neighbour-flagged questions between them.

The difficulty taxonomy. Two independent signals label each question. The first embeds all 1,922 questions belonging to the 11 databases and counts, among each question’s 20 nearest neighbours, how many sit in a different database. The second is a three-model jury asked whether the question is confounding across databases. The hardest bucket requires both rather than blending them.

Neither signal is significant on its own. The conjunction is. Measured on one model, routing accuracy on the questions both signals flag was 0.803 times the accuracy on that same database’s easy questions, 95% CI [0.685, 0.943], p = 0.007. We compare inside each database, then pool across databases with an inverse-variance log risk ratio and a 0.5 continuity correction. It pools over 5 databases rather than 6, because regional_sales carries hard questions but no easy ones and so has nothing to pair against. Three things keep the contrast exploratory. It rests on one model and five strata, and it is observational rather than randomised.

What the model sees. Each model gets a paragraph description of all 11 databases, with no table or column names, plus the 11 database tools. Anonymisation removes the name and not the domain. The descriptions still say what each database is about, and once the model calls get_schema it sees real table and column names. The condition measures description comprehension.

Scoring the route. The declared database is the one the model names under an explicit selected_database key on the finalization turn. A model that produces no explicit declaration scores zero for that question. Routes inferred from prose stay out of the headline and are published separately. An earlier version of this work carried three different definitions of the metric, and the prose-scraped variant inflated six panel rows by between 0.6 and 4.9 points.

Scoring the SQL. Correctness is decided by executing both queries and comparing result sets. Our five-model audit flagged 31.1% of BIRD’s gold queries as broken, so that axis is reported as a three-value bracket rather than a single score. The scoring rules, the gold audit and the full bracket live on the text-to-SQL page linked above.

Tool-call recovery. Five model families serialise tool calls in formats the standard parser does not accept. A recovery layer parses those forms rather than scoring them as silence, and every recovery is recorded per record and per run. Truncated calls are never reconstructed. Six of the 36 rows carry a non-zero recovery share. Five of them are footnote-sized, 0.4% to 4.4%. The sixth is larger. llama-4-maverick sits at 97.9%, so nearly every tool call in that row was adapter-recovered and that row measures the model plus the recovery layer rather than the model alone. One model, minimax-m2.7, narrates without emitting any tool call at all, and we exclude it from the panel instead of scoring it zero.

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

Models tested

Four rows carry a smaller hard-question denominator after two retry passes left residual provider errors, at 182 for gemini-3.1-pro-preview, gemini-3-flash-preview and nova-lite-v1, and 181 for claude-haiku-4.5. Those questions are dropped from the denominator rather than scored as failures, so each percentage is over the records that ran.

Routing accuracy (all 759): The same measurement over the whole frozen set, including the easy and medium control arms. The easy arm is saturated by design.

The 95% intervals: Every interval in the table above is a Wilson interval over the hard questions scored for that model, which is 184 for most rows and 182 or 181 for the four rows that lost questions to residual provider errors. It covers question-sampling uncertainty and nothing else. It does not cover the clustering of questions inside databases, and it does not cover run-to-run variation, which we measured separately at 1.6 to 2.7 points and which is the larger term for any two models near each other.

Chance and baselines. Chance is 0.091 across 11 databases. The majority class is 0.196 over the 759 questions and 0.277 on the hardest. Three non-agentic retrievers score 0.471 (TF-IDF), 0.495 (BM25) and 0.522 (embedding nearest database) overall, and 0.207 to 0.230 on the hardest. Those three were measured on the earlier 594-question set and have not been re-run on the 759, so they are historical reference points rather than baselines for this panel. Chance and the majority class are computed on the frozen 759 and are directly comparable.

Limitations

Both repeat-run models sit in the top tier, where the questions are hardest and the scores are most compressed. No mid-table model has been repeated, so the noise term outside the top tier is unmeasured.

Contamination controls bound the effect, they do not remove it. We rewrote the hard questions to preserve meaning and gold while changing the wording. On the 138 of 184 paraphrases that passed all three validity gates routing did not drop, and the panel effect ran significant in the opposite direction, 0.543 to 0.583, McNemar p = 0.032. Each question is scored in both arms, so the pairing is real: 87 pairs favoured the paraphrase against 60. Sixty-five questions authored from scratch and matched on difficulty, phrasing style and database mix scored within +0.012 of the published set, p = 0.74. That last comparison is between two independent question sets rather than paired, so it is a two-proportion comparison and the weaker of the two designs. The authored control covers 3 of the 11 databases, the paraphrase control 6, and both ran on cheap models alone. A control on databases published after the model cutoffs has not been attempted, and 560 of the 759 questions (73.8%) come from BIRD’s train split, the share most likely to have been memorised.

Dropping the unscored records moves the headline by at most 0.90 points. Six rows lost questions to residual provider errors and those questions leave the denominator rather than counting as failures, which inflates a score if the losses fall on hard questions. Scoring every lost question as wrong instead moves gemini-3.1-pro-preview 0.8242 to 0.8152 on the hardest questions (-0.90 pt) and 0.9140 to 0.8959 over the 759 (-1.81 pt), gemini-3-flash-preview -0.81 pt, claude-haiku-4.5 -0.79 pt, nova-lite-v1 -0.13 pt. Seven rank positions swap on the hardest questions, every one of them adjacent and every one inside the 1.6 to 2.7 point run-to-run noise. The convention is therefore not load-bearing at this panel’s resolution.

The hardest questions are a commerce-cluster measurement. Four commerce databases carry 171 of the 184 hard questions and five databases carry none. The headline generalises to routing among mutually confusable databases, which is what the benchmark was built to measure, and not to cross-domain routing in general. Widening it means adding hard questions from a second semantic cluster, which the current corpus cannot supply.

The panel ran on a mixed harness. Nineteen models were scored after the tool-call recovery layer was added, fifteen before it, and two straddle the change. A model in the earlier group that emitted a non-standard tool call had it scored as silence. Sweeping the recovery layer over all 27,306 stored records found no recoverable call in any of the earlier rows, so the measured cost of the mix is zero, but a serialisation no parser handles would be invisible to that sweep as well.

Runs are not bit-reproducible. OpenRouter served kimi-k2.6 from 19 distinct upstream providers inside a single run and deepseek-v4-pro from 12, and serialisation behaviour differs between them. Three of the 36 runs pinned a provider.

Anonymisation is name-only. The descriptions still name their domains, and the first tool call returns real table and column names, so this condition measures comprehension of a description rather than recognition of a name.

Schema output is truncated at 4,000 characters. works_cycles is the largest database at 66 tables, and 26 of them survive the cut. That database carries 149 of the 759 questions. Query results are capped at 50 rows.

The routing axis is running out of headroom. The easy control arm is exhausted at 222 of 222 for the top models, and the hardest bucket compresses six models into five questions. This benchmark can no longer separate frontier models from each other on routing.

Further reading

• RAG Frameworks: LangChain vs LangGraph vs LlamaIndex
• Best RAG Tools, Frameworks, and Libraries
• Agentic Search in 2026: Benchmark 8 Search APIs for Agents
• Top 5 Open-Source Agentic AI Frameworks in 2026
• Text-to-SQL: Comparison of LLM Accuracy

FAQs

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{sari2026,
  author = {Sarı, Ekrem},
  title  = {{Agentic RAG Benchmark: Multi-Database Routing Across 36 LLMs}},
  year   = {2026},
  month  = aug,
  howpublished    = {\url{https://aimultiple.com/agentic-rag}},
  note   = {AIMultiple. Retrieved August 11, 2026}
}

Reference Links

1.

BIRD-bench

Ekrem Sarı

AI Researcher

Follow On

Ekrem is an AI Researcher and Data Analyst at AIMultiple. He designs and runs hands-on benchmarks for AI and LLM systems.

View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment