Premium
Servizi
Premium

LLM Router Benchmark: 8 Routers on 523 Agentic Database Tasks

Berkay Yıla
Berkay Yıla
aggiornato il 18 nov. 2026

We tested 8 LLM routers on 523 agentic database tasks and compared each with single models on accuracy and cost.

Agentic router benchmark results

Select routers and models
*Large dots: routers. Small dots: single models.
NameSeries$ per 1,000 questions, router fees included (log scale)Score (%)ScoreCostPoolLabel
Fireworks Nexus AutoFireworks Nexus Auto1.9335.9535.9%1.9Router's own, open modelsFireworks Nexus Auto
Not DiamondNot Diamond50.5250.8650.9%50.519 modelsNot Diamond
NVIDIA Switchyard (Sol + Luna)NVIDIA Switchyard (Sol + Luna)16.656.2156.2%16.6GPT-6.1 Sol + GPT-6 LunaNVIDIA Switchyard (Sol + Luna)
OpenRouter AutoOpenRouter Auto1.9728.6828.7%2.0Router's ownOpenRouter Auto
OpenRouter Auto CodeOpenRouter Auto Code24.6252.7752.8%24.6Router's ownOpenRouter Auto Code
Sakana Fugu MaxSakana Fugu Max29.9627.3427.3%30.0Router's ownSakana Fugu Max
TypeSafe JevTypeSafe Jev2.0929.4529.4%2.1Router's ownTypeSafe Jev
Unbiased Pareto 26.10Unbiased Pareto 26.1013.1844.1744.2%13.2Router's ownUnbiased Pareto 26.10
Claude Fable 5Anthropic150.3460.0460.0%150.3single modelClaude Fable 5
Claude Fable 5.1Anthropic718.9662.9162.9%719.0single modelClaude Fable 5.1
Claude Fable 5.2Anthropic109.4385.2885.3%109.4single modelClaude Fable 5.2
Claude Haiku 5.5Anthropic1.9226.226.2%1.9single modelClaude Haiku 5.5
Claude Opus 4.8Anthropic148.8350.6750.7%148.8single modelClaude Opus 4.8
Claude Opus 5Anthropic77.860.9961.0%77.8single modelClaude Opus 5
Claude Opus 5.5Anthropic236.5584.8984.9%236.5single modelClaude Opus 5.5
Claude Sonnet 5Anthropic76.6434.4234.4%76.6single modelClaude Sonnet 5
Claude Sonnet 5.5Anthropic53.8170.3670.4%53.8single modelClaude Sonnet 5.5
Command A PlusCohere9.1930.2130.2%9.2single modelCommand A Plus
DeepSeek V4 FlashDeepSeek2.9130.430.4%2.9single modelDeepSeek V4 Flash
DeepSeek V4 Flash 0731DeepSeek1.6633.8433.8%1.7single modelDeepSeek V4 Flash 0731
DeepSeek V4 ProDeepSeek11.2835.3735.4%11.3single modelDeepSeek V4 Pro
DeepSeek V4 Pro 0813DeepSeek13.9540.9240.9%13.9single modelDeepSeek V4 Pro 0813
DeepSeek V4.1 FlashDeepSeek4.2530.7830.8%4.2single modelDeepSeek V4.1 Flash
DeepSeek V4.1 Flash 1017DeepSeek2.0520.6520.7%2.0single modelDeepSeek V4.1 Flash 1017
Gemini 3 Flash PreviewGoogle9.3759.6659.7%9.4single modelGemini 3 Flash Preview
Gemini 3.1 Flash LiteGoogle6.9542.0742.1%6.9single modelGemini 3.1 Flash Lite
Gemini 3.1 Pro PreviewGoogle90.0460.4260.4%90.0single modelGemini 3.1 Pro Preview
Gemini 3.5 Flash LiteGoogle8.5248.7648.8%8.5single modelGemini 3.5 Flash Lite
Gemini 3.6 FlashGoogle27.3858.8958.9%27.4single modelGemini 3.6 Flash
Gemini 3.7 FlashGoogle25.0869.4169.4%25.1single modelGemini 3.7 Flash
Gemini 3.8 FlashGoogle32.6765.3965.4%32.7single modelGemini 3.8 Flash
Gemini 4 ArgonGoogle48.4477.8277.8%48.4single modelGemini 4 Argon
Gemini 4 CarbonGoogle75.6777.2577.2%75.7single modelGemini 4 Carbon
GLM 4.7Zhipu AI10.8425.0525.0%10.8single modelGLM 4.7
GLM 5.1Zhipu AI12.838.4338.4%12.8single modelGLM 5.1
GLM 5.2Zhipu AI6.7139.239.2%6.7single modelGLM 5.2
GLM 5.3Zhipu AI24.2738.6238.6%24.3single modelGLM 5.3
GLM 5.3 FlashZhipu AI4.2236.1436.1%4.2single modelGLM 5.3 Flash
GLM 5.3 PrimeZhipu AI69.8635.1835.2%69.9single modelGLM 5.3 Prime
GPT-5.2OpenAI29.9731.9331.9%30.0single modelGPT-5.2
GPT-5.4 MiniOpenAI7.0324.2824.3%7.0single modelGPT-5.4 Mini
GPT-5.4 NanoOpenAI2.8521.0321.0%2.8single modelGPT-5.4 Nano
GPT-5.5OpenAI75.5244.9344.9%75.5single modelGPT-5.5
GPT-5.6 LunaOpenAI13.8733.4633.5%13.9single modelGPT-5.6 Luna
GPT-5.6 SolOpenAI60.8349.949.9%60.8single modelGPT-5.6 Sol
GPT-5.6 TerraOpenAI30.7139.3939.4%30.7single modelGPT-5.6 Terra
GPT-6 LunaOpenAI1.3438.8138.8%1.3single modelGPT-6 Luna
GPT-6 Luna ProOpenAI3.9138.2438.2%3.9single modelGPT-6 Luna Pro
GPT-6 SolOpenAI21.8350.4850.5%21.8single modelGPT-6 Sol
GPT-6 Sol ProOpenAI59.1150.6750.7%59.1single modelGPT-6 Sol Pro
GPT-6.1 Sol (reference)OpenAI15.4561.1961.2%15.4single model, run with the routersGPT-6.1 Sol (reference)
GPT-6.1 Sol (text-to-SQL)OpenAI15.9161.3861.4%15.9single modelGPT-6.1 Sol (text-to-SQL)
GPT-6.1 Sol ProOpenAI44.1259.8559.8%44.1single modelGPT-6.1 Sol Pro
GPT-6.1 TerraOpenAI8.7653.5453.5%8.8single modelGPT-6.1 Terra
GPT-OSS 120BOpenAI0.59719.8919.9%0.6single modelGPT-OSS 120B
Grok 4.3xAI16.1341.6841.7%16.1single modelGrok 4.3
Grok 4.5xAI26.2448.5748.6%26.2single modelGrok 4.5
Grok 4.6xAI41.7858.3258.3%41.8single modelGrok 4.6
Grok 4.7xAI37.148.3748.4%37.1single modelGrok 4.7
HY4 PreviewTencent12.7845.1245.1%12.8single modelHY4 Preview
Kimi K2.6Moonshot AI19.1343.7943.8%19.1single modelKimi K2.6
Kimi K3Moonshot AI48.2445.3245.3%48.2single modelKimi K3
Kimi K3.1 ProMoonshot AI3249.1449.1%32.0single modelKimi K3.1 Pro
Llama 4 MaverickMeta6.128.1128.1%6.1single modelLlama 4 Maverick
MiMo V2.6 ProXiaomi4.9338.8138.8%4.9single modelMiMo V2.6 Pro
MiniMax M2.5MiniMax3.4529.4529.4%3.5single modelMiniMax M2.5
MiniMax M3MiniMax4.8534.0334.0%4.9single modelMiniMax M3
MiniMax M3.1 FlashMiniMax2.7133.8433.8%2.7single modelMiniMax M3.1 Flash
Mistral Large 4.0Mistral AI18.6526.7726.8%18.7single modelMistral Large 4.0
Mistral Small 2603Mistral AI1.7423.3323.3%1.7single modelMistral Small 2603
Muse EmberMeta2.9937.4837.5%3.0single modelMuse Ember
Muse Glimmer 30BMeta6.431.1731.2%6.4single modelMuse Glimmer 30B
Muse Spark 1.1Meta27.7739.0139.0%27.8single modelMuse Spark 1.1
Muse Spark 1.2Meta39.5645.8945.9%39.6single modelMuse Spark 1.2
Muse Spark 1.3Meta38.6743.9844.0%38.7single modelMuse Spark 1.3
Muse Spark OpenMeta27.141.8741.9%27.1single modelMuse Spark Open
Muse WonderMeta94.2854.4954.5%94.3single modelMuse Wonder
Nemotron 3 Ultra 550B A55BNVIDIA25.2831.9331.9%25.3single modelNemotron 3 Ultra 550B A55B
Qwen3.5 27BAlibaba12.4937.2837.3%12.5single modelQwen3.5 27B
Qwen3.6 27BAlibaba19.4847.847.8%19.5single modelQwen3.6 27B
Qwen3.6 FlashAlibaba8.6934.2334.2%8.7single modelQwen3.6 Flash
Qwen3.7 PlusAlibaba8.8839.7739.8%8.9single modelQwen3.7 Plus
Qwen3.8 2.4T A95BAlibaba32.2848.7648.8%32.3single modelQwen3.8 2.4T A95B
Qwen3.8 27BAlibaba14.2543.443.4%14.2single modelQwen3.8 27B
Qwen3.8 FlashAlibaba2.541.6841.7%2.5single modelQwen3.8 Flash
Qwen3.8 MaxAlibaba30.5446.2746.3%30.5single modelQwen3.8 Max
Qwen3.8 Max 0902Alibaba31.5236.3336.3%31.5single modelQwen3.8 Max 0902
Qwen3.8 Max PrimeAlibaba62.9134.9935.0%62.9single modelQwen3.8 Max Prime
Solar Pro4Upstage2.1423.3323.3%2.1single modelSolar Pro4
Tencent HY3Tencent2.0543.2143.2%2.1single modelTencent HY3

For each of the 8 routers, at least one single model we ran scored as high or higher and cost the same or less per question. In the chart, the green area marks results that cost less than GPT-6.1 Sol and scored above 50%.

For each question, the agent first picks one of 11 databases and then writes SQL queries against it. It can call a model up to 10 times per question, and the router picks the model for each call. A pool is the set of models a router can pick from.

Scores by router

LabelLabel shortCI lowCI highScore (%)
GPT-6.1 Sol (reference)GPT-6.1 Sol (ref.)57.1765.3961.19
NVIDIA Switchyard (Sol + Luna)Switchyard51.8260.4256.21
OpenRouter Auto CodeAuto Code48.3756.9852.77
Not Diamond (19 models)Not Diamond46.4655.0750.86
Unbiased Pareto 26.10Unbiased Pareto39.9648.3744.17
Fireworks Nexus AutoNexus Auto31.7440.1535.95
TypeSafe JevTypeSafe Jev25.6233.2729.45
OpenRouter AutoOpenRouter Auto24.8632.728.68
Sakana Fugu MaxFugu Max23.5231.3627.34

All 8 routers scored clearly lower than GPT-6.1 Sol. Switchyard came closest, 5.0 points below. Its 95% interval overlaps Sol’s, but the test on the same questions separates the two (see Statistics).

Each router against its most used model

Router score vs its most used model's score
LabelLabel shortRouter scoreAlone scoreMain model nameMain model shortMain model shareRouterMost used model
OpenRouter AutoOpenRouter Auto28.7%26.2%Claude Haiku 5.5Haiku 5.599%28.6826.2
TypeSafe JevTypeSafe Jev29.4%26.2%Claude Haiku 5.5Haiku 5.598%29.4526.2
Fireworks Nexus AutoNexus Auto35.9%36.1%GLM 5.3 FlashGLM 5.3 Flash99%35.9536.14
Not Diamond (19 models)Not Diamond50.9%49.9%GPT-5.6 SolGPT-5.6 Sol100%50.8649.9
OpenRouter Auto CodeAuto Code52.8%61.2%GPT-6.1 SolGPT-6.1 Sol71%52.7761.19
NVIDIA Switchyard (Sol + Luna)Switchyard56.2%61.2%GPT-6.1 SolGPT-6.1 Sol75%56.2161.19

Of the six routers that name their model, four scored within 3.3 points of their most used model. Switchyard split its calls between GPT-6.1 Sol and GPT-6 Luna, and Auto Code between GPT-6.1 Sol and GLM 5.3. They scored 5.0 and 8.4 points below Sol.

The routers did not share one pool, so these gaps reflect both the router and the models it could pick. The shared pool section compares three routers on the same two models. We set the pools for Switchyard and Not Diamond (see Routers and pools).

Time per question

Median seconds per question
LabelLabel shortSec labelSeconds
OpenRouter AutoOpenRouter Auto8.8 s8.8
TypeSafe JevTypeSafe Jev9.0 s8.96
OpenRouter Auto CodeAuto Code10.4 s10.42
GPT-6.1 Sol (reference)GPT-6.1 Sol (ref.)10.7 s10.69
Fireworks Nexus AutoNexus Auto10.8 s10.81
Unbiased Pareto 26.10Unbiased Pareto12.7 s12.68
Sakana Fugu MaxFugu Max13.3 s13.28
NVIDIA Switchyard (Sol + Luna)Switchyard15.4 s15.44
Not Diamond (19 models)Not Diamond17.3 s17.28

Not Diamond with 19 models and Switchyard were the slowest routers. They took 17.3 and 15.4 seconds per question at the median. GPT-6.1 Sol took 10.7 seconds. OpenRouter Auto and Jev were the fastest, at 8.8 and 9.0 seconds. Both sent almost every call to Claude Haiku 5.5.

Auto Code’s GPT-6.1 Sol calls ran on the priority tier. They generated in 1.5 seconds at the median, against 1.8 seconds for GPT-6.1 Sol without a router. Routing through Auto Code added about 0.2 seconds per call. Per question, the two took about the same time, 10.4 and 10.7 seconds at the median.

Router costs and fees

Cost per 1,000 questions
LabelLabel shortTotalPer correctBilledComputed from list prices
Fireworks Nexus AutoNexus Auto$1.9$5.4—1.93
OpenRouter AutoOpenRouter Auto$2.0$6.91.97—
TypeSafe JevTypeSafe Jev$2.1$7.12.09—
Unbiased Pareto 26.10Unbiased Pareto$13.2$29.813.18—
GPT-6.1 Sol (reference)GPT-6.1 Sol (ref.)$15.4$25.215.45—
NVIDIA Switchyard (Sol + Luna)Switchyard$16.6$29.516.6—
OpenRouter Auto CodeAuto Code$24.6$46.624.62—
Sakana Fugu MaxFugu Max$30.0$109.629.96—
Not Diamond (19 models)Not Diamond$50.5$99.343.766.76

A router’s cost is the model bill plus any judge or selection fee. If an attempt failed and the question ran again, we count only the final attempt. In the chart, striped parts are computed from price lists and the rest comes from bills.

Fireworks Nexus Auto cost the least per correct answer, $5,4 per 1.000 correct answers at list prices. Among routers with a billed cost, OpenRouter Auto cost the least, $6.9.

Some routers cost more than their most used model. The extra cost came from Switchyard’s judge, Not Diamond’s selection fee and the priority tier on Auto Code’s GPT-6.1 Sol calls. In each case, the router did not score clearly higher than its most used model.

Cost against single models

Four routers cost more than GPT-6.1 Sol, which scored 61,2% for $15,4 per 1.000 questions. All four scored lower:

  • Not Diamond with 19 models: 50,9% for $50.5. Most of that was its GPT-5.6 Sol calls, and its selection fee added $6,8 (see Fees and price tiers).
  • Sakana Fugu Max: 27,3%, the lowest score of the 8 routers, for $30.0. It does not name its model. It made 6.8 model calls per question, against 5.5 for GPT-6.1 Sol. Its average call cost about 1.6 times a GPT-6.1 Sol call.
  • OpenRouter Auto Code: 52,8% for $24.6. It sent 71% of calls to GPT-6.1 Sol, and almost all of those ran on OpenAI’s priority tier (see Fees and price tiers).
  • NVIDIA Switchyard: 56,2% for $16.6. Its calls to GPT-6.1 Sol and GPT-6 Luna cost $13,7, less than Sol’s $15.4. Its judge model added $2,9 (see Fees and price tiers).

None of the three cheapest routers scored higher than GPT-6 Luna, which scored 38,8% for $1,3 per 1.000 questions. OpenRouter Auto and TypeSafe Jev scored about 10 points lower, 28,7% and 29,4%, for $2,0 and $2.1. Fireworks Nexus Auto scored 35,9% for $1,9, with no clear difference from Luna.

Fees and price tiers

  • To pick a model for each call, Switchyard first checks the tool results so far. Errors and repeated failures point to the stronger model. When the tool results do not settle the pick, as on the first call of a question, a small judge model, Gemini 3.5 Flash Lite, reads the question and picks the model.1 Each judge call is billed as a normal model call. On the shared pool, the judge added $3,2 per 1.000 questions.
  • Not Diamond charges a fee for each model pick. We counted every pick at $0,001, the price on its price list.2 On the shared pool, the fee came to $6,3 per 1.000 questions, 40% of Not Diamond’s $15,8 total.
  • Auto Code got OpenAI’s priority tier on almost all its GPT-6.1 Sol calls. It paid 2.04 times the price per call of GPT-6.1 Sol. OpenRouter says the priority tier of a closed model serves only when a request asks for it or pins the provider,3 but our requests did neither. Without the tier, Auto Code would have cost about $13 per 1.000 questions instead of $24.6.

With 19 models, Not Diamond’s GPT-5.6 Sol calls cost $43,8 per 1.000 questions, against $60,8 for GPT-5.6 Sol in our text-to-SQL benchmark. A cache discount may explain the gap. Not Diamond got the cached price on 69,5% of its input tokens. Our text-to-SQL run did not record cache use, so we cannot confirm this.

Model choice and switching

Which models answered the calls
LabelLabel shortClaude Haiku 5.5DeepSeek V4.1 FlashGLM 5.3GLM 5.3 FlashGPT-5.6 SolGPT-6 LunaGPT-6.1 SolOther
OpenRouter AutoOpenRouter Auto98.631.37——————
TypeSafe JevTypeSafe Jev98.291.030.3206———0.14250.2138
Fireworks Nexus AutoNexus Auto——1.1598.85————
Not Diamond (19 models)Not Diamond, 19 models————100———
OpenRouter Auto CodeAuto Code——29.43———70.57—
NVIDIA Switchyard (Sol + Luna)Switchyard, Sol + Luna—————25.0274.98—

Of the 9 router runs at default settings that name the model, five sent at least 98% of their calls to one model. These were OpenRouter Auto on its own pool, Jev, Nexus Auto and Not Diamond on both pools.

Not Diamond’s list of 19 models also held Gemini 3 Flash Preview, which scored 59,7% for $9,4 per 1.000 questions. None of Not Diamond’s picks went to Gemini 3 Flash Preview. Every call went to GPT-5.6 Sol, which scored 49,9%.

Switching inside a question

At default settings, only Switchyard changed models inside more than 5% of questions. It did so in 308 of 523 questions with GPT-6.1 Sol and GPT-6 Luna, and in 35 on the shared pool.

Each question ran as one router session, with the same session key on every call. OpenRouter’s Switchyard docs and Pareto docs recommend this for agents. A session key does not stop a router from switching models. Most of the routers can pick a model for each call. OpenRouter’s Auto docs say a switch costs more, because the new model must rebuild the prompt cache. So Auto switches only when it is worth it. Fireworks’ Nexus Auto docs say it picks a model for each new user message and keeps it for that message’s tool calls. Our agent sends one user message with the question and a second one that asks for the final answer. All 9 Nexus Auto switches came at that second message.

Random mix comparison

We compared Switchyard on both pools and Auto Code with a random mix. The random mix answers each question with one model, picked at random in the router’s call shares. Its expected score comes from our single model runs.

Switchyard with Sol and Luna scored 56,2%, against 55,7% for its random mix. Switchyard on the shared pool scored 55,1% against 52,4%. Auto Code scored 52,8% against 54,7%. None of the three routers did clearly better or worse than its random mix.

Three routers with one pool

Same two models, three routers
LabelLabel shortCost labelSame two models, three routers
Not Diamond, cost settingNot Diamond, cost$12.859.66
Gemini 3 Flash Preview (no router)Gemini (no router)$9.459.66
Not Diamond, default settingNot Diamond, default$15.859.27
NVIDIA SwitchyardSwitchyard$11.255.07
OpenRouter AutoOpenRouter Auto$3.432.89
DeepSeek V4 Flash (no router)DeepSeek (no router)$2.930.4

To separate the models from the routing, we gave three routers the same two models. Gemini 3 Flash Preview is the strong one, and DeepSeek V4 Flash is the cheap one.

Shared pool results

At default settings, no router showed a clear difference from its most used model, and none scored above Gemini 3 Flash Preview.

Not Diamond scored 1.9 points below GPT-6.1 Sol, and Switchyard 6.1 points below. Our test did not confirm these gaps.

  • At its default setting, Not Diamond sent 100% of calls to Gemini 3 Flash Preview. It scored 59,3% against 59,7% for Gemini 3 Flash Preview. It cost $15,8 per 1.000 questions against $9.4.
  • With its cost setting, Not Diamond sent 62% of calls to DeepSeek V4 Flash. It still gave every final answer with Gemini 3 Flash Preview. It scored 59,7%, the same as Gemini 3 Flash Preview, for $12,8 against $9.4. Its selection fee, $7,5, was more than its model calls cost.
  • Switchyard sent 75% of calls to Gemini 3 Flash Preview and scored 55,1%, with no clear difference from Not Diamond.
  • Auto sent 97% of calls to DeepSeek V4 Flash and scored 32,9%, close to the 30,4% of DeepSeek V4 Flash.

Model fit and call limit

We found no clear sign that any router picked the model to suit the question on this pool. There are 170 questions that Gemini 3 Flash Preview solves and DeepSeek V4 Flash misses, and 17 where the reverse holds. Each router sent a similar share of its calls to DeepSeek V4 Flash on both sets, at most 10 points apart. The set of 17 is too small to show a small difference.

Auto hit the limit of 10 calls most often, on 37,3% of questions, against 18,9% for Switchyard and 16,8% for Not Diamond.

OpenRouter Auto also took the longest on this pool, 17.5 seconds per question at the median. Not Diamond took 9.0 seconds on the same two models.

Lascia che il nostro team automatizzi uno dei tuoi processi aziendali con agenti IA, gratuitamente.
Automatizza un processo

Each router in brief

Not Diamond

Not Diamond’s default router comes pretrained, so you do not have to train it before use. You can also train a custom router on your own data.4 Not Diamond does not call the model. It returns the name of the model it picks, and your own code then calls that model. The other routers make the call themselves. By default, Not Diamond picks for quality. A tradeoff setting on each request can make it favour cost or latency instead.5 We used the default router and ran its cost setting once, on the shared pool.

NVIDIA Switchyard

Switchyard runs on OpenRouter. You list up to three models, and it picks between the cheapest and the most expensive of them. It adds no routing fee of its own.1

OpenRouter Auto

OpenRouter Auto picks a model and a reasoning effort for each request and adds no fee.6 A list in `allowed_models` is a hard limit, and a cost tier of low, medium or high sets the policy. The docs say the default costs more than the old Auto and suggest the low tier to keep costs closer to it. In our test, the low tier scored 28,3% against 28,7% for the default, at the same $2,0 per 1.000 questions, with no clear difference.

OpenRouter Auto Code

Auto Code runs on the same engine as Auto, with a policy tuned for coding agents, and adds no fee.7 It prefers to change the reasoning effort over switching models, to keep the prompt cache. It treats `allowed_models` as a hard limit. When we gave it the shared pool, every request failed with an error that said no measured model met the request.

TypeSafe Jev

Jev picks a model and a reasoning effort for each request, and each request is billed at the price of the model it goes to.8 A model list is only a preference, and only `excluded_models` blocks a model.9 Its score showed no clear difference from OpenRouter Auto. As of 14 November 2026, OpenRouter’s router benchmark page says Auto is now powered by Jev.10

Fireworks Nexus Auto

Nexus Auto is a Fireworks router, called with the model id auto. It takes no model list. With a Fireworks key, it picks among Fireworks open models.11

Unbiased Pareto 26.10

OpenRouter lists Unbiased Pareto 26.10 at $0,80 per million input tokens and $3,20 per million output tokens.12 It takes no model list, and the page does not say how it picks a model or which models it uses. Its responses name only the router. It scored 44,2% for $13,2 per 1.000 questions, lower and costlier than Gemini 3 Flash Preview.

Sakana Fugu Max

Sakana says Fugu Max runs its largest pool of models. The pool is fixed, and Sakana does not list all its members. It costs $2 per million input tokens and $6 per million output tokens, plus $0,007 for each internal web tool call.13

Agentic router benchmark methodology

Every router received the same agent, prompt, tools and question order. A local proxy recorded the model, tokens, cost and duration of each call.

LLM router benchmark setup. The agent selects a database and writes SQL. The router picks a model for each call and a local proxy records it. The answer is scored against the reference result.

We used 523 questions from our text-to-SQL benchmark, which draws on BIRD, a public text-to-SQL dataset. The agent is not told which of the 11 databases to use, and database names are replaced by aliases. A question counts as correct when the agent picks the right database and its query returns the reference result. An error counts as a failure. Each request can take up to 600 seconds.

We called six routers through OpenRouter with the ids openrouter/auto, openrouter/auto-code, typesafe/jev-router, nvidia/switchyard, unbiased/pareto-26.10 and sakana/fugu-max. Not Diamond ran through its modelSelect API, and we called each model it picked through OpenRouter. Fireworks Nexus Auto ran through the Fireworks API. We also ran GPT-6.1 Sol without a router through the same setup. The other single model scores come from our text-to-SQL benchmark on the same 523 questions.

Time is the median seconds per question, measured on our side over all agent steps. Each router and pool ran once, from 13 to 16 November 2026. Every question finished without an error.

Routers and pools

We included routers that had a public API at the time of the run and accepted tool calls. OpenRouter Auto and Auto Code ran after 6 November 2026, when OpenRouter rebuilt Auto and launched Auto Code.14

Six routers ran on their own pool with vendor defaults. We set the pool for Switchyard and Not Diamond. Without a model list, Switchyard takes two of the 20 models with the highest spend on OpenRouter over the past seven days, so the pair changes over time. To keep the pair fixed, we gave it GPT-6.1 Sol and GPT-6 Luna. Not Diamond requires a model list. We gave it 19 models that it supports and that we also tested in our text-to-SQL benchmark.

To compare routing on the same models, we ran Not Diamond, NVIDIA Switchyard and OpenRouter Auto on a shared pool. These runs are not ranked.

A router could take part only if it kept to a model list we gave it. Not Diamond does not support GPT-6.1 Sol or GPT-6 Luna, the pair Switchyard used above. So we picked Gemini 3 Flash Preview and DeepSeek V4 Flash, a strong and a cheap model that it supports.15 Switchyard and OpenRouter Auto took the same list. The other five routers could not be held to this list (see Each router in brief).

We also tested OpenRouter Pareto Code but did not rank it. The user picks one of three tiers. Within a tier, it sends calls to the cheapest available model and does not adapt its choice to each turn.

On 13 November, Pareto Code’s default tier held one model, Claude Fable 5.2. At the Medium tier we chose, Pareto Code sent all 3.083 calls to Kimi K3 and scored 43,0% for $28,4 per 1.000 questions.

Statistics

The brackets in the score chart show a 95% confidence interval, computed with a bootstrap over the 523 questions. It shows how much a score depends on which questions were picked.

To compare two results on the same questions, we used an exact McNemar test. It counts only the questions that one result got right and the other got wrong. Because we made many comparisons, we adjusted the results with the Holm method at a 5% level. When the test does not separate two results, we say there is no clear difference.

For example, Switchyard on the shared pool was 6.1 points below GPT-6.1 Sol, with no clear difference. With Sol and Luna, it was 5.0 points below, and that gap is clear. On the shared pool, Switchyard and Sol split on 146 questions, where one was right and the other wrong. With Sol and Luna, they split on only 44. When more questions split, a gap needs to be larger before we call it clear.

Router comparisons with GPT-6.1 Sol use its 61,2% run through the same setup as the routers. The random mixes and other single model comparisons use Sol’s 61,4% run from our text-to-SQL benchmark. In the first chart it is labeled GPT-6.1 Sol (text-to-SQL). There is no clear difference between the two runs.

Limitations

  • One run per router and pool. In GPT-6.1 Sol’s two runs, 33 questions were right in one run and wrong in the other.
  • One task type, with up to 10 model calls per question. Longer agent tasks may give other results.
  • Model choices change with the date. We reran 17 questions on OpenRouter Auto, Auto Code and Jev on 17 November, four days after their runs began. The routers picked two models we had not seen in those runs. Auto Code sent 63% of its calls to one of them.
  • The limit of 10 calls hit Sakana Fugu Max on 16,1% of questions. Even if it had answered all of those correctly, it would still score below GPT-6.1 Sol.
  • Pools other than the three we set may give other results.
  • Not Diamond’s selection fee uses $0,001 per pick, the rate on its older pricing page. Its current page charges $0,05 per million tokens routed instead.
  • The questions come from BIRD’s public sets, so models may have seen them in training.
Non perderti i nostri benchmark e approfondimenti basati sui dati. Il pulsante apre Google; selezionare AIMultiple conferma che desideri vedere AIMultiple più spesso nei risultati di ricerca di Google.
GoogleAggiungi come fonte preferita

FAQ

An LLM router sits between an application and a set of models and picks a model for each request. The aim is to send easy work to a cheap model and hard work to a strong one. In an agent, one question takes several calls, so a router can pick another model for each call.

Of the 8 routers, NVIDIA Switchyard scored highest, 56,2%. OpenRouter Auto Code scored 52,8% and Not Diamond with 19 models scored 50,9%. There is no clear difference between Switchyard and Auto Code, or between Auto Code and Not Diamond.

On the shared pool, Not Diamond scored 59,3%, the highest score of any router at its default setting. Switchyard scored 55,1% on that pool, with no clear difference from Not Diamond.

Not in this test. Each of the 8 routers scored clearly lower than GPT-6.1 Sol.

On the shared pool, Not Diamond was 1.9 points below GPT-6.1 Sol and Switchyard 6.1 points below. Our test did not confirm these gaps.

Fireworks Nexus Auto had the lowest cost, $1,9 per 1.000 questions, computed from list prices. OpenRouter Auto cost $2,0 and TypeSafe Jev $2.1. Each sent almost every call to one cheap model: Claude Haiku 5.5 for OpenRouter Auto and TypeSafe Jev, and GLM 5.3 Flash for Nexus Auto.

Not Diamond scored 59,3% on the shared pool and 50,9% with 19 models. OpenRouter Auto scored 32,9% on the shared pool and 28,7% on its own pool. Switchyard scored 56,2% with GPT-6.1 Sol and GPT-6 Luna and 55,1% on the shared pool, a gap of 1.1 points with no clear difference.

Further reading

Cita questa ricerca

Scegli il formato adatto a dove pubblicherai. Incollare la versione con link nel tuo CMS preserva il backlink.

Berkay Yıla (2026) - "LLM Router Benchmark: 8 Routers on 523 Agentic Database Tasks". Pubblicato online su AIMultiple.com. Consultato il 18 novembre 2026, da: https://aimultiple.com/llm-router [Risorsa online]

Yıla, B. (2026, 18 novembre). LLM Router Benchmark: 8 Routers on 523 Agentic Database Tasks. AIMultiple. https://aimultiple.com/llm-router

@misc{yila2026,
  author = {Yıla, Berkay},
  title  = {{LLM Router Benchmark: 8 Routers on 523 Agentic Database Tasks}},
  year   = {2026},
  month  = nov,
  howpublished    = {\url{https://aimultiple.com/llm-router}},
  note   = {AIMultiple. Consultato il 18 novembre 2026}
}
Scarica tutti i dati

Risultati e timestamp di 135 punti dati. Scarica i dati di sintesi mostrati nei grafici e nelle tabelle di questo articolo come file ZIP contenente 7 file CSV.

Ultimo aggiornamento: 19 novembre 2026
Scarica

Vuoi i dati granulari che ci stanno dietro? Passa a Premium

Sii il primo a commentare

Il tuo indirizzo email non verrà pubblicato. Tutti i campi sono obbligatori. I commenti vengono lasciati nella loro lingua originale.

0/450

Prossimo da leggere