Agentic RAG Benchmark: Routing Across 11 SQL Databases

Routing accuracy is the percentage of scored questions for which the model explicitly names the correct database in its final answer. The headline uses the 184 questions flagged as difficult by both our similarity test and a jury of three LLMs. Missing explicit declarations receive no credit.

Database routing findings

Opus 5.5 recorded the highest routing accuracy at 90.2%

claude-opus-5.5 selected the correct database for 166 of 184 hard questions. gemini-3.7-flash followed with 157 correct answers, or 85.3%, the highest score among the API runs. claude-opus-5 recorded 84.8%, while gpt-6-sol-pro and GPT-6 Astra each scored 84.2%.

Opus 5.5 also led across all 759 questions at 96.6%. It ran through Claude Code, while Gemini 3.7 Flash used OpenRouter. The observed scores reflect both model and execution conditions.

HY3 recorded 77.7% routing accuracy at a recorded cost of $1.88

HY3 answered 143 of 184 hard questions correctly. Its retained responses across all 759 questions cost $1.88. GPT-6 Luna scored 73.4% at $1.05, while Gemini 3.7 Flash reached 85.3% at $20.82.

These costs sum the successful responses retained in each result file. Failed and replaced retry attempts are excluded. These totals depend on the providers and caching conditions used during measurement. Subscription-based CLI runs are excluded from the cost chart.

GPT-6 Sol Pro gained 34.2 points after its first database choice

GPT-6 Sol Pro’s hard-question routing accuracy rose from 50.0% on its first probe to 84.2% on its final declaration. Grok 4.7 gained 34.8 points within its run. Nova Lite lost 14.0 points.

These changes use questions with a single database on the first turn. They describe recorded trajectories. No separate intervention isolated the benefit of extra calls.

Across all 49 models, 4,195 hard-question records included a database switch:

This table takes the first database in recorded call order, including multi-database first turns. Final credit requires an explicit correct declaration. The percentage-point changes above use the narrower single-first-choice subset.

Database routing results

Models are sorted by routing accuracy, highest first. The methodology identifies differences in the software used to run them. These conditions matter when interpreting small score gaps.

The intervals estimate question-sampling uncertainty. Recorded costs cover the successful records in each run. They reflect the providers and caching conditions used at the time of measurement.

Conditions affecting database selection

Similar databases reduced routing accuracy by about 20 points

We tested the database selection method in a separate experiment on 128 questions. Each question kept its correct database, while the other 10 candidates came either from our selected group or from a random draw.

On the hard subset, claude-opus-4.8 scored 71.3% with similar candidates and 92.0% with random candidates. gemini-3.5-flash scored 77.0% and 96.6%. Both differences had p-values below 0.001 in paired tests.

This experiment used descriptions and one model response per question, without the agentic tool loop. It supports the narrower finding that selecting similar candidates makes database choice harder. The measured gap should not be presented as the effect of agentic exploration.

Names alone produced 68.8% and 71.1% accuracy in a pilot

In another 128-question pilot, claude-opus-4.8 selected the right database 68.8% of the time from names alone. Its score with names and descriptions was 67.2%. gemini-3.5-flash scored 71.1% from names and 75.0% from the full catalog.

Names such as california_schools and toxicology reveal the subject directly. For the main benchmark, we replace database names with aliases from db_01 to db_11. Descriptions still explain each database’s domain, and tool responses expose its table and column names.

Reliability of routing and cost comparisons

Repeat runs changed routing accuracy by 1.6 to 2.7 points

claude-opus-5 selected the correct database for 156 of 184 hard questions on its first run and 151 on its repeat. Its accuracy fell from 84.8% to 82.1%. qwen3.8-max moved from 153 correct answers to 150, a decline from 83.2% to 81.5%.

Both repeats used the same questions, anonymization, temperature and provider settings as their original runs. qwen3.8-max changed its answer on 13 hard questions even though its total moved by three. Improvements on some questions canceled errors on others.

These repeats show that small score gaps can disappear on another run. One repeat per model cannot establish a distribution of outcomes, and the other 35 models have no repeat measurement.

Recorded costs depend on provider and caching settings

Prompt caching reuses previously processed input to reduce billed input costs. The claude-opus-5 run cost $62.78, compared with an estimated $105.54 at uncached list rates for the same recorded usage. That estimate implies a 40.5% saving. It is an accounting comparison, without a separate uncached accuracy run.

Agentic RAG and database selection

Agentic RAG gives a model control over retrieval decisions. It can choose a source, inspect the response and make another request. A simple RAG pipeline retrieves context through a predetermined sequence before generating an answer.

This benchmark tests source selection through SQL database tools. It contains no document index, retrieved passages or answer-grounding score. Results therefore describe one part of an agentic retrieval system. Document search is covered separately in our agentic search benchmark.

A model starts with short database descriptions. It can request table lists, inspect columns, run SQL and change databases. Its final response contains a database choice and a query. Our text-to-SQL benchmark reports SQL accuracy and two supplementary review scores.

Decision models for database routing

We compared Jev, Kev-9B, Laya English and six LLMs on 243 reviewed questions from the same 11 databases. Each model selected a database from supplied descriptions.

Laya ran locally on an Apple M4. Kev ran on a dedicated A100 service accessed through an SSH tunnel. The other models used hosted APIs. Latency includes the complete client request among valid responses. Accuracy includes every attempted question.

Jev selected the correct database for 240 of 243 questions, compared with 123 for Laya English: 117 more correct selections, a 48.1-percentage-point difference. Jev was the fastest hosted model by median latency in this run. Laya’s local deployment had the lowest median latency overall, alongside substantially lower accuracy.

Kev-9B selected 236 databases correctly, four fewer than Jev. Five of its seven errors involved the fuel debit-card database. Its median response took 952 ms including the network and SSH tunnel. The separately recorded server median was 470 ms. This A100 deployment used FP32 and reference kernels. Its speed depends on that serving configuration.

Gemini 3.8 Flash and Claude Opus 5 both selected the correct database for every question. Opus took 1.72 times as long at the median and its reported API cost was 7.58 times Gemini’s for these questions. Accuracy was equal for these two models on this set.

DeepSeek’s result includes 19 service or output failures: 16 HTTP 429 responses, two responses that broke the required single-alias format, and one exhausted output budget. It selected the correct database in 211 of its 224 valid responses, or 94.2%. Counting all attempts gives the 86.8% shown in the chart and table.

Routing costs: API pricing and hardware estimates

Jev selected the correct database for 98.8% of questions at an API cost of $0.0337 per 1,000 routing attempts. Qwen3.8 Flash recorded 97.9% accuracy at $0.0791, while Gemini 3.8 Flash reached 100.0% at $0.5337. We scaled costs from the same 243-question workload, including incorrect answers and excluding warmups. Jev’s API cost was about 1/120 of Claude Opus 5’s for the same 243 routing attempts.

Laya’s hardware estimate was $0.0141 per 1,000 attempts at 50.6% accuracy. Kev reached 97.1% at an estimated $0.1250. Both estimates assume that the rented machine continuously processes requests. At 10% utilization, each request carries ten times the rental cost, raising Laya to $0.1405 and Kev to $1.2503 per 1,000 attempts.

For the five LLMs with complete charge records, we divided reported API charges by 243 and multiplied by 1,000. Jev charges $0.042 per million input tokens, with free output tokens. Its 194,882 recorded input tokens cost $0.008185 across 243 attempts, or $0.0337 per 1,000.¹

Hardware cost per 1,000 attempts equals hourly rental price × mean processing seconds × 1,000 ÷ (3,600 × utilization). Laya averaged 0.2006 seconds per request on an Apple M4. Kev averaged 0.4734 seconds on the A100 server, excluding network and SSH delay. We applied those times to matching hardware at the quoted providers. Performance and billing totals at those providers remain unmeasured.

For Kev, we used $0.9508/hour for one A100 SXM 80 GB at Vast.ai, the lowest matching on-demand listing in our GPU rental price index data.

For Laya, we used Scaleway’s M4-S with 16 GB memory at €0.22/hour. The conversion uses the ECB’s 22 September rate of $1.1463 per euro. Scaleway requires a 24-hour minimum lease, making the minimum rental €5.28, or about $6.05, even for a short job.²³⁴

Laya’s model page listed no Hugging Face Inference Provider. Our estimate covers renting a machine and running the model on it. Setup time, storage, network extras, taxes and operations labor are excluded from both hardware estimates.⁵

How decision models choose a database

For Jev and Laya, the application supplies the context, a question and named options with descriptions. Their Choice interfaces return a selected option, probabilities over the options and a confidence score. Application code decides what to do with that result. For routing, the options are database aliases such as db_03 and db_07.⁶⁵

This format makes the output easy to use in code. A model can still choose the wrong database. A high confidence score also needs validation against observed correctness before an application uses it to skip review or trigger a fallback.

Laya’s English checkpoint uses a bidirectional ModernBERT encoder with a decision head that scores the supplied options. The option descriptions arrive with the request, so the application can change its available choices. Jev exposes a hosted Choice API. Its documentation places workflow control and actions in application code. The shared interface does not establish that the two models have the same internal architecture.⁵⁷

Kev-9B uses a LoRA adapter and a pointer head on Qwen3.5-9B-Base. It scores the supplied choices and returns their probabilities without generating answer text. We used its compatible Choice endpoint with the same question and database descriptions.⁸

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

Agentic RAG benchmark methodology

The question set is a frozen subset of BIRD-SQL, which pairs natural-language questions with SQL reference queries.⁹

Two signals define difficulty. The similarity test counts how many of a question’s 20 nearest neighbors belong to other databases. Three LLMs assess whether the question is confounding across databases. Membership in the hardest group requires at least five cross-database neighbors and a confounding jury label.

Routing uses the database named under the explicit selected_database field. A route inferred from prose is excluded from the headline. SQL correctness is scored separately and does not determine routing credit.

The harness is the software that presents tools and records model responses. The main panel contains 36 OpenRouter runs and one Codex CLI run. Among the API runs, 19 used v3, 15 used an earlier version, and two combined records from both versions. In v3, a recovery layer parses supported alternative tool-call formats. For llama-4-maverick, the recorded non-native format share was 97.9%, so that result depends heavily on the adapter. minimax-m2.7 was excluded because it did not issue the database tool calls required by the task.

GPT-6 Astra ran inside Codex CLI, whose recorded harness identifier is codex_cli_transcript_v1. Because the setup differs, score differences cannot be attributed to the model alone. GPT-6 Astra completed all 759 questions. It selected the correct database on 155 of 184 hard questions and recorded 95.3% routing accuracy across the full set. Its run did not report dollar costs.

Decision-model routing benchmark

The separate routing experiment used 243 unchanged questions selected from 394 candidates after reserving 17 development questions. The query set contained 204 originally easy and 39 originally hard questions. Those difficulty labels were not revalidated for this task, and selection was not a blind held-out evaluation.

Four database descriptions were corrected using source schemas and question text before collecting the measured predictions. The other seven retained their compact descriptions. Every model received the same question, 11 aliases and descriptions, and option order for that question. Reference SQL, source labels and supplementary evidence were withheld. Jev and Laya used their native Choice interface. The LLMs were instructed to return one alias. Models had no database tools.

We ran Jev 1.13.0 through its hosted API and the pinned Laya English checkpoint locally with Apple M4 MPS, 16 GiB memory and four Torch threads. Laya’s 404-token context-head limit and 48-token option limit produced no input truncation. Six LLMs ran through OpenRouter with fixed providers and reasoning settings. Requested reasoning was low for Gemini 3.8 Flash, minimal for Gemini 3.5 Flash Lite and high for Opus. It was disabled for DeepSeek, Qwen and Haiku. Each model had one excluded warmup, one request in flight and no retries. Model submission order rotated between questions.

Kev-9B was added on 22 September in a separate run, preserving the original eight results. Its 243 request payloads matched the original Jev inputs exactly, including option order. We pinned checkpoint revision 2629c06a and used an A100 SXM 80 GB with FP32, SDPA attention and unmerged LoRA. Date preprocessing and prefix caching were off. After checking the API on the 17 reserved development questions, we ran one excluded warmup and 243 serial requests with no retries. The service was reserved for our use. Every response confirmed that the full input was retained. Server timings include tokenization and synchronized inference. The chart uses client timings, including SSH and network overhead.

Accuracy divides correct, valid choices by all 243 planned questions, including service and output failures. Median latency measures the complete nonstreaming client request among valid responses, excluding warmup. Reported LLM API costs cover the measured requests. Laya hardware and Kev rental costs were not measured, and Jev’s list-price estimate is a different accounting basis. The comparison describes this curated set and these deployment conditions. Public source data, description revisions, reviewer judgment, different measurement dates and the absence of repeats limit generalization.

Limitations

The hard subset concentrates on commerce. Four databases contribute 171 of its 184 questions, or 92.9%. Five of the 11 databases contribute none. This supports conclusions about choosing among similar databases within a domain, with limited evidence for other domain combinations.

Seven runs contain fewer than 759 scored records after provider errors. Five also have fewer than 184 hard-question records. Their table denominators show the questions that completed. Missing records are excluded, which can favor a model if the omitted questions were harder.

Wilson intervals cover sampling uncertainty under a question-level model. They omit variation across repeat runs, dependence among questions from the same database and uncertainty introduced by the harness. Provider changes and tool-call recovery also limit comparisons between runs.

Public benchmark data may have appeared in model training. Controls using 138 validated paraphrases and 65 newly authored questions found no accuracy decline in the tested lower-cost models. Their coverage was limited to six and three databases respectively. They do not resolve contamination for the full panel, and no control used databases released after the models’ training cutoffs.

Schema truncation can hide useful tables. In works_cycles, the 4,000-character limit leaves 26 of 66 tables visible in the initial schema listing. The model can request further details, but the initial view is incomplete.

Further reading

• Text-to-SQL: Comparison of LLM Accuracy
• RAG Frameworks: LangChain vs LangGraph vs LlamaIndex
• Best RAG Tools, Frameworks, and Libraries
• Agentic Search in 2026: Benchmark 8 Search APIs for Agents

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{sari2026,
  author = {Sarı, Ekrem},
  title  = {{Agentic RAG Benchmark: Routing Across 11 SQL Databases}},
  year   = {2026},
  month  = sep,
  howpublished    = {\url{https://aimultiple.com/agentic-rag}},
  note   = {AIMultiple. Retrieved September 25, 2026}
}

Changelog

16 updates

All16General13Methodology3

202620252024

1. Aug 11

   Updated the 'What makes a question hard in this benchmark?' section with new content.

2. Aug 11

   Updated the methodology to clarify how the hardest questions are distributed across databases.

3. Aug 7

   Replaced the Agentic RAG benchmark methodology section with a new one that covers 36 LLMs and 11 databases.

4. Jun 29

   Added new models to the benchmark: Claude Fable 5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3, Claude Opus 4.7.

5. Feb 20

   Updated the methodology section with new details on database environment, agent architecture, and evaluation process.

6. Jan 23

   Added a section on long-context models versus agentic RAG.

7. Jan 23

   Removed additional agentic metrics from the methodology.

Reference Links

1.

Models - TypeSafe AI

Mintlify

2.

Apple Silicon Pricing | Scaleway

3.

Euro exchange rates charts

4.

Apple silicon M4 as-a-Service. Cloud Mac | Scaleway

5.

convaiinnovations/laya · Hugging Face

6.

Choice - TypeSafe AI

Mintlify

7.

How to build with TypeSafe - TypeSafe AI

Mintlify

8.

jaredpalmer/kev-9b · Hugging Face

9.

BIRD-bench

Ekrem Sarı

AI Researcher

Follow On

Ekrem is an AI Researcher and Data Scientist at AIMultiple. He designs and runs hands-on benchmarks for AI and LLM systems.

View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment