We tested 8 LLM routers on 523 agentic database tasks and compared each with single models on accuracy and cost.
Agentic router benchmark results
| Name | Series | $ per 1,000 questions, router fees included (log scale) | Score (%) | Score | Cost | Pool | Label |
|---|---|---|---|---|---|---|---|
| Fireworks Nexus Auto | Fireworks Nexus Auto | 1.93 | 35.95 | 35.9% | 1.9 | Router's own, open models | Fireworks Nexus Auto |
| Not Diamond | Not Diamond | 50.52 | 50.86 | 50.9% | 50.5 | 19 models | Not Diamond |
| NVIDIA Switchyard (Sol + Luna) | NVIDIA Switchyard (Sol + Luna) | 16.6 | 56.21 | 56.2% | 16.6 | GPT-6.1 Sol + GPT-6 Luna | NVIDIA Switchyard (Sol + Luna) |
| OpenRouter Auto | OpenRouter Auto | 1.97 | 28.68 | 28.7% | 2.0 | Router's own | OpenRouter Auto |
| OpenRouter Auto Code | OpenRouter Auto Code | 24.62 | 52.77 | 52.8% | 24.6 | Router's own | OpenRouter Auto Code |
| Sakana Fugu Max | Sakana Fugu Max | 29.96 | 27.34 | 27.3% | 30.0 | Router's own | Sakana Fugu Max |
| TypeSafe Jev | TypeSafe Jev | 2.09 | 29.45 | 29.4% | 2.1 | Router's own | TypeSafe Jev |
| Unbiased Pareto 26.10 | Unbiased Pareto 26.10 | 13.18 | 44.17 | 44.2% | 13.2 | Router's own | Unbiased Pareto 26.10 |
| Claude Fable 5 | Anthropic | 150.34 | 60.04 | 60.0% | 150.3 | single model | Claude Fable 5 |
| Claude Fable 5.1 | Anthropic | 718.96 | 62.91 | 62.9% | 719.0 | single model | Claude Fable 5.1 |
| Claude Fable 5.2 | Anthropic | 109.43 | 85.28 | 85.3% | 109.4 | single model | Claude Fable 5.2 |
| Claude Haiku 5.5 | Anthropic | 1.92 | 26.2 | 26.2% | 1.9 | single model | Claude Haiku 5.5 |
| Claude Opus 4.8 | Anthropic | 148.83 | 50.67 | 50.7% | 148.8 | single model | Claude Opus 4.8 |
| Claude Opus 5 | Anthropic | 77.8 | 60.99 | 61.0% | 77.8 | single model | Claude Opus 5 |
| Claude Opus 5.5 | Anthropic | 236.55 | 84.89 | 84.9% | 236.5 | single model | Claude Opus 5.5 |
| Claude Sonnet 5 | Anthropic | 76.64 | 34.42 | 34.4% | 76.6 | single model | Claude Sonnet 5 |
| Claude Sonnet 5.5 | Anthropic | 53.81 | 70.36 | 70.4% | 53.8 | single model | Claude Sonnet 5.5 |
| Command A Plus | Cohere | 9.19 | 30.21 | 30.2% | 9.2 | single model | Command A Plus |
| DeepSeek V4 Flash | DeepSeek | 2.91 | 30.4 | 30.4% | 2.9 | single model | DeepSeek V4 Flash |
| DeepSeek V4 Flash 0731 | DeepSeek | 1.66 | 33.84 | 33.8% | 1.7 | single model | DeepSeek V4 Flash 0731 |
| DeepSeek V4 Pro | DeepSeek | 11.28 | 35.37 | 35.4% | 11.3 | single model | DeepSeek V4 Pro |
| DeepSeek V4 Pro 0813 | DeepSeek | 13.95 | 40.92 | 40.9% | 13.9 | single model | DeepSeek V4 Pro 0813 |
| DeepSeek V4.1 Flash | DeepSeek | 4.25 | 30.78 | 30.8% | 4.2 | single model | DeepSeek V4.1 Flash |
| DeepSeek V4.1 Flash 1017 | DeepSeek | 2.05 | 20.65 | 20.7% | 2.0 | single model | DeepSeek V4.1 Flash 1017 |
| Gemini 3 Flash Preview | 9.37 | 59.66 | 59.7% | 9.4 | single model | Gemini 3 Flash Preview | |
| Gemini 3.1 Flash Lite | 6.95 | 42.07 | 42.1% | 6.9 | single model | Gemini 3.1 Flash Lite | |
| Gemini 3.1 Pro Preview | 90.04 | 60.42 | 60.4% | 90.0 | single model | Gemini 3.1 Pro Preview | |
| Gemini 3.5 Flash Lite | 8.52 | 48.76 | 48.8% | 8.5 | single model | Gemini 3.5 Flash Lite | |
| Gemini 3.6 Flash | 27.38 | 58.89 | 58.9% | 27.4 | single model | Gemini 3.6 Flash | |
| Gemini 3.7 Flash | 25.08 | 69.41 | 69.4% | 25.1 | single model | Gemini 3.7 Flash | |
| Gemini 3.8 Flash | 32.67 | 65.39 | 65.4% | 32.7 | single model | Gemini 3.8 Flash | |
| Gemini 4 Argon | 48.44 | 77.82 | 77.8% | 48.4 | single model | Gemini 4 Argon | |
| Gemini 4 Carbon | 75.67 | 77.25 | 77.2% | 75.7 | single model | Gemini 4 Carbon | |
| GLM 4.7 | Zhipu AI | 10.84 | 25.05 | 25.0% | 10.8 | single model | GLM 4.7 |
| GLM 5.1 | Zhipu AI | 12.8 | 38.43 | 38.4% | 12.8 | single model | GLM 5.1 |
| GLM 5.2 | Zhipu AI | 6.71 | 39.2 | 39.2% | 6.7 | single model | GLM 5.2 |
| GLM 5.3 | Zhipu AI | 24.27 | 38.62 | 38.6% | 24.3 | single model | GLM 5.3 |
| GLM 5.3 Flash | Zhipu AI | 4.22 | 36.14 | 36.1% | 4.2 | single model | GLM 5.3 Flash |
| GLM 5.3 Prime | Zhipu AI | 69.86 | 35.18 | 35.2% | 69.9 | single model | GLM 5.3 Prime |
| GPT-5.2 | OpenAI | 29.97 | 31.93 | 31.9% | 30.0 | single model | GPT-5.2 |
| GPT-5.4 Mini | OpenAI | 7.03 | 24.28 | 24.3% | 7.0 | single model | GPT-5.4 Mini |
| GPT-5.4 Nano | OpenAI | 2.85 | 21.03 | 21.0% | 2.8 | single model | GPT-5.4 Nano |
| GPT-5.5 | OpenAI | 75.52 | 44.93 | 44.9% | 75.5 | single model | GPT-5.5 |
| GPT-5.6 Luna | OpenAI | 13.87 | 33.46 | 33.5% | 13.9 | single model | GPT-5.6 Luna |
| GPT-5.6 Sol | OpenAI | 60.83 | 49.9 | 49.9% | 60.8 | single model | GPT-5.6 Sol |
| GPT-5.6 Terra | OpenAI | 30.71 | 39.39 | 39.4% | 30.7 | single model | GPT-5.6 Terra |
| GPT-6 Luna | OpenAI | 1.34 | 38.81 | 38.8% | 1.3 | single model | GPT-6 Luna |
| GPT-6 Luna Pro | OpenAI | 3.91 | 38.24 | 38.2% | 3.9 | single model | GPT-6 Luna Pro |
| GPT-6 Sol | OpenAI | 21.83 | 50.48 | 50.5% | 21.8 | single model | GPT-6 Sol |
| GPT-6 Sol Pro | OpenAI | 59.11 | 50.67 | 50.7% | 59.1 | single model | GPT-6 Sol Pro |
| GPT-6.1 Sol (reference) | OpenAI | 15.45 | 61.19 | 61.2% | 15.4 | single model, run with the routers | GPT-6.1 Sol (reference) |
| GPT-6.1 Sol (text-to-SQL) | OpenAI | 15.91 | 61.38 | 61.4% | 15.9 | single model | GPT-6.1 Sol (text-to-SQL) |
| GPT-6.1 Sol Pro | OpenAI | 44.12 | 59.85 | 59.8% | 44.1 | single model | GPT-6.1 Sol Pro |
| GPT-6.1 Terra | OpenAI | 8.76 | 53.54 | 53.5% | 8.8 | single model | GPT-6.1 Terra |
| GPT-OSS 120B | OpenAI | 0.597 | 19.89 | 19.9% | 0.6 | single model | GPT-OSS 120B |
| Grok 4.3 | xAI | 16.13 | 41.68 | 41.7% | 16.1 | single model | Grok 4.3 |
| Grok 4.5 | xAI | 26.24 | 48.57 | 48.6% | 26.2 | single model | Grok 4.5 |
| Grok 4.6 | xAI | 41.78 | 58.32 | 58.3% | 41.8 | single model | Grok 4.6 |
| Grok 4.7 | xAI | 37.1 | 48.37 | 48.4% | 37.1 | single model | Grok 4.7 |
| HY4 Preview | Tencent | 12.78 | 45.12 | 45.1% | 12.8 | single model | HY4 Preview |
| Kimi K2.6 | Moonshot AI | 19.13 | 43.79 | 43.8% | 19.1 | single model | Kimi K2.6 |
| Kimi K3 | Moonshot AI | 48.24 | 45.32 | 45.3% | 48.2 | single model | Kimi K3 |
| Kimi K3.1 Pro | Moonshot AI | 32 | 49.14 | 49.1% | 32.0 | single model | Kimi K3.1 Pro |
| Llama 4 Maverick | Meta | 6.1 | 28.11 | 28.1% | 6.1 | single model | Llama 4 Maverick |
| MiMo V2.6 Pro | Xiaomi | 4.93 | 38.81 | 38.8% | 4.9 | single model | MiMo V2.6 Pro |
| MiniMax M2.5 | MiniMax | 3.45 | 29.45 | 29.4% | 3.5 | single model | MiniMax M2.5 |
| MiniMax M3 | MiniMax | 4.85 | 34.03 | 34.0% | 4.9 | single model | MiniMax M3 |
| MiniMax M3.1 Flash | MiniMax | 2.71 | 33.84 | 33.8% | 2.7 | single model | MiniMax M3.1 Flash |
| Mistral Large 4.0 | Mistral AI | 18.65 | 26.77 | 26.8% | 18.7 | single model | Mistral Large 4.0 |
| Mistral Small 2603 | Mistral AI | 1.74 | 23.33 | 23.3% | 1.7 | single model | Mistral Small 2603 |
| Muse Ember | Meta | 2.99 | 37.48 | 37.5% | 3.0 | single model | Muse Ember |
| Muse Glimmer 30B | Meta | 6.4 | 31.17 | 31.2% | 6.4 | single model | Muse Glimmer 30B |
| Muse Spark 1.1 | Meta | 27.77 | 39.01 | 39.0% | 27.8 | single model | Muse Spark 1.1 |
| Muse Spark 1.2 | Meta | 39.56 | 45.89 | 45.9% | 39.6 | single model | Muse Spark 1.2 |
| Muse Spark 1.3 | Meta | 38.67 | 43.98 | 44.0% | 38.7 | single model | Muse Spark 1.3 |
| Muse Spark Open | Meta | 27.1 | 41.87 | 41.9% | 27.1 | single model | Muse Spark Open |
| Muse Wonder | Meta | 94.28 | 54.49 | 54.5% | 94.3 | single model | Muse Wonder |
| Nemotron 3 Ultra 550B A55B | NVIDIA | 25.28 | 31.93 | 31.9% | 25.3 | single model | Nemotron 3 Ultra 550B A55B |
| Qwen3.5 27B | Alibaba | 12.49 | 37.28 | 37.3% | 12.5 | single model | Qwen3.5 27B |
| Qwen3.6 27B | Alibaba | 19.48 | 47.8 | 47.8% | 19.5 | single model | Qwen3.6 27B |
| Qwen3.6 Flash | Alibaba | 8.69 | 34.23 | 34.2% | 8.7 | single model | Qwen3.6 Flash |
| Qwen3.7 Plus | Alibaba | 8.88 | 39.77 | 39.8% | 8.9 | single model | Qwen3.7 Plus |
| Qwen3.8 2.4T A95B | Alibaba | 32.28 | 48.76 | 48.8% | 32.3 | single model | Qwen3.8 2.4T A95B |
| Qwen3.8 27B | Alibaba | 14.25 | 43.4 | 43.4% | 14.2 | single model | Qwen3.8 27B |
| Qwen3.8 Flash | Alibaba | 2.5 | 41.68 | 41.7% | 2.5 | single model | Qwen3.8 Flash |
| Qwen3.8 Max | Alibaba | 30.54 | 46.27 | 46.3% | 30.5 | single model | Qwen3.8 Max |
| Qwen3.8 Max 0902 | Alibaba | 31.52 | 36.33 | 36.3% | 31.5 | single model | Qwen3.8 Max 0902 |
| Qwen3.8 Max Prime | Alibaba | 62.91 | 34.99 | 35.0% | 62.9 | single model | Qwen3.8 Max Prime |
| Solar Pro4 | Upstage | 2.14 | 23.33 | 23.3% | 2.1 | single model | Solar Pro4 |
| Tencent HY3 | Tencent | 2.05 | 43.21 | 43.2% | 2.1 | single model | Tencent HY3 |
For each of the 8 routers, at least one single model we ran scored as high or higher and cost the same or less per question. In the chart, the green area marks results that cost less than GPT-6.1 Sol and scored above 50%.
For each question, the agent first picks one of 11 databases and then writes SQL queries against it. It can call a model up to 10 times per question, and the router picks the model for each call. A pool is the set of models a router can pick from.
Scores by router
| Label | Label short | CI low | CI high | Score (%) |
|---|---|---|---|---|
| GPT-6.1 Sol (reference) | GPT-6.1 Sol (ref.) | 57.17 | 65.39 | 61.19 |
| NVIDIA Switchyard (Sol + Luna) | Switchyard | 51.82 | 60.42 | 56.21 |
| OpenRouter Auto Code | Auto Code | 48.37 | 56.98 | 52.77 |
| Not Diamond (19 models) | Not Diamond | 46.46 | 55.07 | 50.86 |
| Unbiased Pareto 26.10 | Unbiased Pareto | 39.96 | 48.37 | 44.17 |
| Fireworks Nexus Auto | Nexus Auto | 31.74 | 40.15 | 35.95 |
| TypeSafe Jev | TypeSafe Jev | 25.62 | 33.27 | 29.45 |
| OpenRouter Auto | OpenRouter Auto | 24.86 | 32.7 | 28.68 |
| Sakana Fugu Max | Fugu Max | 23.52 | 31.36 | 27.34 |
All 8 routers scored clearly lower than GPT-6.1 Sol. Switchyard came closest, 5.0 points below. Its 95% interval overlaps Sol’s, but the test on the same questions separates the two (see Statistics).
Each router against its most used model
| Label | Label short | Router score | Alone score | Main model name | Main model short | Main model share | Router | Most used model |
|---|---|---|---|---|---|---|---|---|
| OpenRouter Auto | OpenRouter Auto | 28.7% | 26.2% | Claude Haiku 5.5 | Haiku 5.5 | 99% | 28.68 | 26.2 |
| TypeSafe Jev | TypeSafe Jev | 29.4% | 26.2% | Claude Haiku 5.5 | Haiku 5.5 | 98% | 29.45 | 26.2 |
| Fireworks Nexus Auto | Nexus Auto | 35.9% | 36.1% | GLM 5.3 Flash | GLM 5.3 Flash | 99% | 35.95 | 36.14 |
| Not Diamond (19 models) | Not Diamond | 50.9% | 49.9% | GPT-5.6 Sol | GPT-5.6 Sol | 100% | 50.86 | 49.9 |
| OpenRouter Auto Code | Auto Code | 52.8% | 61.2% | GPT-6.1 Sol | GPT-6.1 Sol | 71% | 52.77 | 61.19 |
| NVIDIA Switchyard (Sol + Luna) | Switchyard | 56.2% | 61.2% | GPT-6.1 Sol | GPT-6.1 Sol | 75% | 56.21 | 61.19 |
Of the six routers that name their model, four scored within 3.3 points of their most used model. Switchyard split its calls between GPT-6.1 Sol and GPT-6 Luna, and Auto Code between GPT-6.1 Sol and GLM 5.3. They scored 5.0 and 8.4 points below Sol.
The routers did not share one pool, so these gaps reflect both the router and the models it could pick. The shared pool section compares three routers on the same two models. We set the pools for Switchyard and Not Diamond (see Routers and pools).
Time per question
| Label | Label short | Sec label | Seconds |
|---|---|---|---|
| OpenRouter Auto | OpenRouter Auto | 8.8 s | 8.8 |
| TypeSafe Jev | TypeSafe Jev | 9.0 s | 8.96 |
| OpenRouter Auto Code | Auto Code | 10.4 s | 10.42 |
| GPT-6.1 Sol (reference) | GPT-6.1 Sol (ref.) | 10.7 s | 10.69 |
| Fireworks Nexus Auto | Nexus Auto | 10.8 s | 10.81 |
| Unbiased Pareto 26.10 | Unbiased Pareto | 12.7 s | 12.68 |
| Sakana Fugu Max | Fugu Max | 13.3 s | 13.28 |
| NVIDIA Switchyard (Sol + Luna) | Switchyard | 15.4 s | 15.44 |
| Not Diamond (19 models) | Not Diamond | 17.3 s | 17.28 |
Not Diamond with 19 models and Switchyard were the slowest routers. They took 17.3 and 15.4 seconds per question at the median. GPT-6.1 Sol took 10.7 seconds. OpenRouter Auto and Jev were the fastest, at 8.8 and 9.0 seconds. Both sent almost every call to Claude Haiku 5.5.
Auto Code’s GPT-6.1 Sol calls ran on the priority tier. They generated in 1.5 seconds at the median, against 1.8 seconds for GPT-6.1 Sol without a router. Routing through Auto Code added about 0.2 seconds per call. Per question, the two took about the same time, 10.4 and 10.7 seconds at the median.
Router costs and fees
| Label | Label short | Total | Per correct | Billed | Computed from list prices |
|---|---|---|---|---|---|
| Fireworks Nexus Auto | Nexus Auto | $1.9 | $5.4 | — | 1.93 |
| OpenRouter Auto | OpenRouter Auto | $2.0 | $6.9 | 1.97 | — |
| TypeSafe Jev | TypeSafe Jev | $2.1 | $7.1 | 2.09 | — |
| Unbiased Pareto 26.10 | Unbiased Pareto | $13.2 | $29.8 | 13.18 | — |
| GPT-6.1 Sol (reference) | GPT-6.1 Sol (ref.) | $15.4 | $25.2 | 15.45 | — |
| NVIDIA Switchyard (Sol + Luna) | Switchyard | $16.6 | $29.5 | 16.6 | — |
| OpenRouter Auto Code | Auto Code | $24.6 | $46.6 | 24.62 | — |
| Sakana Fugu Max | Fugu Max | $30.0 | $109.6 | 29.96 | — |
| Not Diamond (19 models) | Not Diamond | $50.5 | $99.3 | 43.76 | 6.76 |
A router’s cost is the model bill plus any judge or selection fee. If an attempt failed and the question ran again, we count only the final attempt. In the chart, striped parts are computed from price lists and the rest comes from bills.
Fireworks Nexus Auto cost the least per correct answer, $5,4 per 1.000 correct answers at list prices. Among routers with a billed cost, OpenRouter Auto cost the least, $6.9.
Some routers cost more than their most used model. The extra cost came from Switchyard’s judge, Not Diamond’s selection fee and the priority tier on Auto Code’s GPT-6.1 Sol calls. In each case, the router did not score clearly higher than its most used model.
Cost against single models
Four routers cost more than GPT-6.1 Sol, which scored 61,2% for $15,4 per 1.000 questions. All four scored lower:
- Not Diamond with 19 models: 50,9% for $50.5. Most of that was its GPT-5.6 Sol calls, and its selection fee added $6,8 (see Fees and price tiers).
- Sakana Fugu Max: 27,3%, the lowest score of the 8 routers, for $30.0. It does not name its model. It made 6.8 model calls per question, against 5.5 for GPT-6.1 Sol. Its average call cost about 1.6 times a GPT-6.1 Sol call.
- OpenRouter Auto Code: 52,8% for $24.6. It sent 71% of calls to GPT-6.1 Sol, and almost all of those ran on OpenAI’s priority tier (see Fees and price tiers).
- NVIDIA Switchyard: 56,2% for $16.6. Its calls to GPT-6.1 Sol and GPT-6 Luna cost $13,7, less than Sol’s $15.4. Its judge model added $2,9 (see Fees and price tiers).
None of the three cheapest routers scored higher than GPT-6 Luna, which scored 38,8% for $1,3 per 1.000 questions. OpenRouter Auto and TypeSafe Jev scored about 10 points lower, 28,7% and 29,4%, for $2,0 and $2.1. Fireworks Nexus Auto scored 35,9% for $1,9, with no clear difference from Luna.
Fees and price tiers
- To pick a model for each call, Switchyard first checks the tool results so far. Errors and repeated failures point to the stronger model. When the tool results do not settle the pick, as on the first call of a question, a small judge model, Gemini 3.5 Flash Lite, reads the question and picks the model.1 Each judge call is billed as a normal model call. On the shared pool, the judge added $3,2 per 1.000 questions.
- Not Diamond charges a fee for each model pick. We counted every pick at $0,001, the price on its price list.2 On the shared pool, the fee came to $6,3 per 1.000 questions, 40% of Not Diamond’s $15,8 total.
- Auto Code got OpenAI’s priority tier on almost all its GPT-6.1 Sol calls. It paid 2.04 times the price per call of GPT-6.1 Sol. OpenRouter says the priority tier of a closed model serves only when a request asks for it or pins the provider,3 but our requests did neither. Without the tier, Auto Code would have cost about $13 per 1.000 questions instead of $24.6.
With 19 models, Not Diamond’s GPT-5.6 Sol calls cost $43,8 per 1.000 questions, against $60,8 for GPT-5.6 Sol in our text-to-SQL benchmark. A cache discount may explain the gap. Not Diamond got the cached price on 69,5% of its input tokens. Our text-to-SQL run did not record cache use, so we cannot confirm this.
Model choice and switching
| Label | Label short | Claude Haiku 5.5 | DeepSeek V4.1 Flash | GLM 5.3 | GLM 5.3 Flash | GPT-5.6 Sol | GPT-6 Luna | GPT-6.1 Sol | Other |
|---|---|---|---|---|---|---|---|---|---|
| OpenRouter Auto | OpenRouter Auto | 98.63 | 1.37 | — | — | — | — | — | — |
| TypeSafe Jev | TypeSafe Jev | 98.29 | 1.03 | 0.3206 | — | — | — | 0.1425 | 0.2138 |
| Fireworks Nexus Auto | Nexus Auto | — | — | 1.15 | 98.85 | — | — | — | — |
| Not Diamond (19 models) | Not Diamond, 19 models | — | — | — | — | 100 | — | — | — |
| OpenRouter Auto Code | Auto Code | — | — | 29.43 | — | — | — | 70.57 | — |
| NVIDIA Switchyard (Sol + Luna) | Switchyard, Sol + Luna | — | — | — | — | — | 25.02 | 74.98 | — |
Of the 9 router runs at default settings that name the model, five sent at least 98% of their calls to one model. These were OpenRouter Auto on its own pool, Jev, Nexus Auto and Not Diamond on both pools.
Not Diamond’s list of 19 models also held Gemini 3 Flash Preview, which scored 59,7% for $9,4 per 1.000 questions. None of Not Diamond’s picks went to Gemini 3 Flash Preview. Every call went to GPT-5.6 Sol, which scored 49,9%.
Switching inside a question
At default settings, only Switchyard changed models inside more than 5% of questions. It did so in 308 of 523 questions with GPT-6.1 Sol and GPT-6 Luna, and in 35 on the shared pool.
Each question ran as one router session, with the same session key on every call. OpenRouter’s Switchyard docs and Pareto docs recommend this for agents. A session key does not stop a router from switching models. Most of the routers can pick a model for each call. OpenRouter’s Auto docs say a switch costs more, because the new model must rebuild the prompt cache. So Auto switches only when it is worth it. Fireworks’ Nexus Auto docs say it picks a model for each new user message and keeps it for that message’s tool calls. Our agent sends one user message with the question and a second one that asks for the final answer. All 9 Nexus Auto switches came at that second message.
Random mix comparison
We compared Switchyard on both pools and Auto Code with a random mix. The random mix answers each question with one model, picked at random in the router’s call shares. Its expected score comes from our single model runs.
Switchyard with Sol and Luna scored 56,2%, against 55,7% for its random mix. Switchyard on the shared pool scored 55,1% against 52,4%. Auto Code scored 52,8% against 54,7%. None of the three routers did clearly better or worse than its random mix.
Three routers with one pool
| Label | Label short | Cost label | Same two models, three routers |
|---|---|---|---|
| Not Diamond, cost setting | Not Diamond, cost | $12.8 | 59.66 |
| Gemini 3 Flash Preview (no router) | Gemini (no router) | $9.4 | 59.66 |
| Not Diamond, default setting | Not Diamond, default | $15.8 | 59.27 |
| NVIDIA Switchyard | Switchyard | $11.2 | 55.07 |
| OpenRouter Auto | OpenRouter Auto | $3.4 | 32.89 |
| DeepSeek V4 Flash (no router) | DeepSeek (no router) | $2.9 | 30.4 |
To separate the models from the routing, we gave three routers the same two models. Gemini 3 Flash Preview is the strong one, and DeepSeek V4 Flash is the cheap one.
Shared pool results
At default settings, no router showed a clear difference from its most used model, and none scored above Gemini 3 Flash Preview.
Not Diamond scored 1.9 points below GPT-6.1 Sol, and Switchyard 6.1 points below. Our test did not confirm these gaps.
- At its default setting, Not Diamond sent 100% of calls to Gemini 3 Flash Preview. It scored 59,3% against 59,7% for Gemini 3 Flash Preview. It cost $15,8 per 1.000 questions against $9.4.
- With its cost setting, Not Diamond sent 62% of calls to DeepSeek V4 Flash. It still gave every final answer with Gemini 3 Flash Preview. It scored 59,7%, the same as Gemini 3 Flash Preview, for $12,8 against $9.4. Its selection fee, $7,5, was more than its model calls cost.
- Switchyard sent 75% of calls to Gemini 3 Flash Preview and scored 55,1%, with no clear difference from Not Diamond.
- Auto sent 97% of calls to DeepSeek V4 Flash and scored 32,9%, close to the 30,4% of DeepSeek V4 Flash.
Model fit and call limit
We found no clear sign that any router picked the model to suit the question on this pool. There are 170 questions that Gemini 3 Flash Preview solves and DeepSeek V4 Flash misses, and 17 where the reverse holds. Each router sent a similar share of its calls to DeepSeek V4 Flash on both sets, at most 10 points apart. The set of 17 is too small to show a small difference.
Auto hit the limit of 10 calls most often, on 37,3% of questions, against 18,9% for Switchyard and 16,8% for Not Diamond.
OpenRouter Auto also took the longest on this pool, 17.5 seconds per question at the median. Not Diamond took 9.0 seconds on the same two models.
Each router in brief
Not Diamond
Not Diamond’s default router comes pretrained, so you do not have to train it before use. You can also train a custom router on your own data.4 Not Diamond does not call the model. It returns the name of the model it picks, and your own code then calls that model. The other routers make the call themselves. By default, Not Diamond picks for quality. A tradeoff setting on each request can make it favour cost or latency instead.5 We used the default router and ran its cost setting once, on the shared pool.
NVIDIA Switchyard
Switchyard runs on OpenRouter. You list up to three models, and it picks between the cheapest and the most expensive of them. It adds no routing fee of its own.1
OpenRouter Auto
OpenRouter Auto picks a model and a reasoning effort for each request and adds no fee.6 A list in `allowed_models` is a hard limit, and a cost tier of low, medium or high sets the policy. The docs say the default costs more than the old Auto and suggest the low tier to keep costs closer to it. In our test, the low tier scored 28,3% against 28,7% for the default, at the same $2,0 per 1.000 questions, with no clear difference.
OpenRouter Auto Code
Auto Code runs on the same engine as Auto, with a policy tuned for coding agents, and adds no fee.7 It prefers to change the reasoning effort over switching models, to keep the prompt cache. It treats `allowed_models` as a hard limit. When we gave it the shared pool, every request failed with an error that said no measured model met the request.
TypeSafe Jev
Jev picks a model and a reasoning effort for each request, and each request is billed at the price of the model it goes to.8 A model list is only a preference, and only `excluded_models` blocks a model.9 Its score showed no clear difference from OpenRouter Auto. As of 14 November 2026, OpenRouter’s router benchmark page says Auto is now powered by Jev.10
Fireworks Nexus Auto
Nexus Auto is a Fireworks router, called with the model id auto. It takes no model list. With a Fireworks key, it picks among Fireworks open models.11
Unbiased Pareto 26.10
OpenRouter lists Unbiased Pareto 26.10 at $0,80 per million input tokens and $3,20 per million output tokens.12 It takes no model list, and the page does not say how it picks a model or which models it uses. Its responses name only the router. It scored 44,2% for $13,2 per 1.000 questions, lower and costlier than Gemini 3 Flash Preview.
Sakana Fugu Max
Sakana says Fugu Max runs its largest pool of models. The pool is fixed, and Sakana does not list all its members. It costs $2 per million input tokens and $6 per million output tokens, plus $0,007 for each internal web tool call.13
Agentic router benchmark methodology
Every router received the same agent, prompt, tools and question order. A local proxy recorded the model, tokens, cost and duration of each call.

We used 523 questions from our text-to-SQL benchmark, which draws on BIRD, a public text-to-SQL dataset. The agent is not told which of the 11 databases to use, and database names are replaced by aliases. A question counts as correct when the agent picks the right database and its query returns the reference result. An error counts as a failure. Each request can take up to 600 seconds.
We called six routers through OpenRouter with the ids openrouter/auto, openrouter/auto-code, typesafe/jev-router, nvidia/switchyard, unbiased/pareto-26.10 and sakana/fugu-max. Not Diamond ran through its modelSelect API, and we called each model it picked through OpenRouter. Fireworks Nexus Auto ran through the Fireworks API. We also ran GPT-6.1 Sol without a router through the same setup. The other single model scores come from our text-to-SQL benchmark on the same 523 questions.
Time is the median seconds per question, measured on our side over all agent steps. Each router and pool ran once, from 13 to 16 November 2026. Every question finished without an error.
Routers and pools
We included routers that had a public API at the time of the run and accepted tool calls. OpenRouter Auto and Auto Code ran after 6 November 2026, when OpenRouter rebuilt Auto and launched Auto Code.14
Six routers ran on their own pool with vendor defaults. We set the pool for Switchyard and Not Diamond. Without a model list, Switchyard takes two of the 20 models with the highest spend on OpenRouter over the past seven days, so the pair changes over time. To keep the pair fixed, we gave it GPT-6.1 Sol and GPT-6 Luna. Not Diamond requires a model list. We gave it 19 models that it supports and that we also tested in our text-to-SQL benchmark.
To compare routing on the same models, we ran Not Diamond, NVIDIA Switchyard and OpenRouter Auto on a shared pool. These runs are not ranked.
A router could take part only if it kept to a model list we gave it. Not Diamond does not support GPT-6.1 Sol or GPT-6 Luna, the pair Switchyard used above. So we picked Gemini 3 Flash Preview and DeepSeek V4 Flash, a strong and a cheap model that it supports.15 Switchyard and OpenRouter Auto took the same list. The other five routers could not be held to this list (see Each router in brief).
We also tested OpenRouter Pareto Code but did not rank it. The user picks one of three tiers. Within a tier, it sends calls to the cheapest available model and does not adapt its choice to each turn.
On 13 November, Pareto Code’s default tier held one model, Claude Fable 5.2. At the Medium tier we chose, Pareto Code sent all 3.083 calls to Kimi K3 and scored 43,0% for $28,4 per 1.000 questions.
Statistics
The brackets in the score chart show a 95% confidence interval, computed with a bootstrap over the 523 questions. It shows how much a score depends on which questions were picked.
To compare two results on the same questions, we used an exact McNemar test. It counts only the questions that one result got right and the other got wrong. Because we made many comparisons, we adjusted the results with the Holm method at a 5% level. When the test does not separate two results, we say there is no clear difference.
For example, Switchyard on the shared pool was 6.1 points below GPT-6.1 Sol, with no clear difference. With Sol and Luna, it was 5.0 points below, and that gap is clear. On the shared pool, Switchyard and Sol split on 146 questions, where one was right and the other wrong. With Sol and Luna, they split on only 44. When more questions split, a gap needs to be larger before we call it clear.
Router comparisons with GPT-6.1 Sol use its 61,2% run through the same setup as the routers. The random mixes and other single model comparisons use Sol’s 61,4% run from our text-to-SQL benchmark. In the first chart it is labeled GPT-6.1 Sol (text-to-SQL). There is no clear difference between the two runs.
Limitations
- One run per router and pool. In GPT-6.1 Sol’s two runs, 33 questions were right in one run and wrong in the other.
- One task type, with up to 10 model calls per question. Longer agent tasks may give other results.
- Model choices change with the date. We reran 17 questions on OpenRouter Auto, Auto Code and Jev on 17 November, four days after their runs began. The routers picked two models we had not seen in those runs. Auto Code sent 63% of its calls to one of them.
- The limit of 10 calls hit Sakana Fugu Max on 16,1% of questions. Even if it had answered all of those correctly, it would still score below GPT-6.1 Sol.
- Pools other than the three we set may give other results.
- Not Diamond’s selection fee uses $0,001 per pick, the rate on its older pricing page. Its current page charges $0,05 per million tokens routed instead.
- The questions come from BIRD’s public sets, so models may have seen them in training.
FAQ
An LLM router sits between an application and a set of models and picks a model for each request. The aim is to send easy work to a cheap model and hard work to a strong one. In an agent, one question takes several calls, so a router can pick another model for each call.
Of the 8 routers, NVIDIA Switchyard scored highest, 56,2%. OpenRouter Auto Code scored 52,8% and Not Diamond with 19 models scored 50,9%. There is no clear difference between Switchyard and Auto Code, or between Auto Code and Not Diamond.
On the shared pool, Not Diamond scored 59,3%, the highest score of any router at its default setting. Switchyard scored 55,1% on that pool, with no clear difference from Not Diamond.
Not in this test. Each of the 8 routers scored clearly lower than GPT-6.1 Sol.
On the shared pool, Not Diamond was 1.9 points below GPT-6.1 Sol and Switchyard 6.1 points below. Our test did not confirm these gaps.
Fireworks Nexus Auto had the lowest cost, $1,9 per 1.000 questions, computed from list prices. OpenRouter Auto cost $2,0 and TypeSafe Jev $2.1. Each sent almost every call to one cheap model: Claude Haiku 5.5 for OpenRouter Auto and TypeSafe Jev, and GLM 5.3 Flash for Nexus Auto.
Not Diamond scored 59,3% on the shared pool and 50,9% with 19 models. OpenRouter Auto scored 32,9% on the shared pool and 28,7% on its own pool. Switchyard scored 56,2% with GPT-6.1 Sol and GPT-6 Luna and 55,1% on the shared pool, a gap of 1.1 points with no clear difference.
Further reading
Cita questa ricerca
Scegli il formato adatto a dove pubblicherai. Incollare la versione con link nel tuo CMS preserva il backlink.
@misc{yila2026,
author = {Yıla, Berkay},
title = {{LLM Router Benchmark: 8 Routers on 523 Agentic Database Tasks}},
year = {2026},
month = nov,
howpublished = {\url{https://aimultiple.com/llm-router}},
note = {AIMultiple. Consultato il 18 novembre 2026}
}Risultati e timestamp di 135 punti dati. Scarica i dati di sintesi mostrati nei grafici e nelle tabelle di questo articolo come file ZIP contenente 7 file CSV.
Vuoi i dati granulari che ci stanno dietro? Passa a Premium
Sii il primo a commentare
Il tuo indirizzo email non verrà pubblicato. Tutti i campi sono obbligatori. I commenti vengono lasciati nella loro lingua originale.