LLM Latency Benchmark by Use Cases in 2026
We benchmarked 11 top large language models with a total of 1,320 requests, splitting reasoning and non-reasoning models, and measured first-token latency, per-token latency, and overall response time.
LLM latency benchmark
You can find details on how we measured latency here.
End-to-end response time by model
LLM latency benchmark results
We report reasoning and non-reasoning models separately. Reasoning models spend several seconds thinking before the first visible answer, so comparing them directly with non-reasoning models on latency would be misleading. Some models also change behavior by task: GPT-5.2 answers Q&A without reasoning but reasons on the coding prompts.
Short-answer case (Q&A)
In the short case what matters is how quickly the first answer token appears, since the whole response is only a sentence or two. Non-reasoning models cluster in the low-latency region:
- claude-opus-4-8: 0.75 seconds
- gpt-5-2: 0.8 seconds
- claude-haiku-4-5: 0.96 seconds
- claude-sonnet-5: 1.5 seconds
- claude-opus-4-7: 2 seconds
Reasoning models were slower to reach the first answer word even here, because they think first:
- mimo-v2-5: 1.2 seconds
- minimax-m3: 2.4 seconds
- deepseek-v4-flash: 3.6 seconds
- gemini-3-1-pro-preview: 9 seconds
- hy3-preview: 14.6 seconds
On short questions the thinking step is mostly overhead, since the answer itself is short.
Long-answer case (code generation)
In the long case the first-token delay matters less, and streaming spIn the long case the first-token delay matters less, and streaming speed and total response time take over. claude-haiku-4-5 led both, streaming at about 180 tokens per second and finishing in about 5.5 seconds. The other non-reasoning models finished in roughly 10 to 13 seconds: claude-sonnet-5 at 10 seconds, claude-opus-4-8 at 10.8, and claude-opus-4-7 at 12.7.
Reasoning models produced longer total times, because thinking and the long output add together:
- gemini-3-1-pro-preview: 18.5 seconds
- hy3-preview: 23.1 seconds
- deepseek-v4-flash: 24.3 seconds
- minimax-m3: 28.1 seconds
- mimo-v2-5: 35 seconds
Some stream quickly once they start (hy3-preview runs at about 200 tokens per second), but the upfront thinking keeps the total high.
Output length was capped rather than left open, so models are not compared on verbosity: about 128 tokens for short answers and 1,024 for long answers. Reasoning models were given extra budget beyond this limit to hold their thinking tokens, so their end-to-end times reflect thinking plus the answer rather than a response cut off mid-thought.
Short answers versus long answers
The contrast between the two cases is the main takeaway:
- In the short case the fastest models answered in under a second and the spread was narrow.
- In the long case non-reasoning models stayed near the front (Haiku fastest at about 5.5 seconds), while reasoning models fell well behind at 18 to 35 seconds.
- The order within the non-reasoning group also shifts: Haiku was one of several sub-second starters on short prompts, but pulls clearly ahead on long outputs because of its streaming speed.
Latency at the 90th percentile
The main pattern is that the tail multiplies for reasoning models rather than just shifting. On the coding task, MiniMax went from about 13 seconds at the median to 42 seconds at p90, and Hunyuan from about 16 to 46 seconds. The extra time is thinking, and thinking length varies from request to request, so these models are not only slower but less predictable.
A few points stand out:
- MiMo is the exception among reasoning models. It reasons on every request but thinks briefly, so its p90 stayed near 3.7 seconds. Reasoning by itself does not make a model unpredictable; the amount of thinking does.
- GPT-5.2 shows a tail only where it reasons. On Q&A its p90 was under 2 seconds, but on coding, where it turns reasoning on, its p90 rose to about 11 seconds.
- Non-reasoning models stayed tight. Haiku 4.5 and Opus 4.8 held p90 within about 2 to 2.5 seconds. Even Sonnet 5, which widened on long outputs (about 4 seconds at the median to 8 at p90), stayed far below the reasoning group.
The practical point is that the median can hide this. MiniMax’s median on coding, about 13 seconds, is not the worst in the group, but its p90 of about 42 seconds matches Hunyuan, so on a bad request it is as slow as anything measured.
LLM reasoning and its effect on speed
Reasoning models take longer to produce their first answer token because they generate internal chain-of-thought tokens before the visible answer. That thinking step is what raises their time to first answer, from about a second for non-reasoning models to several seconds or more.
Whether a model reasons is not always fixed. We detected reasoning per request from the number of reasoning tokens each response returned, then grouped the results by task. The share of requests on which each model reasoned:
GPT-5.2 shows this directly: it answered short Q&A prompts with no reasoning, but reasoned on most coding prompts. For a hybrid model like this, “reasoning” is not a fixed label; the model decides based on the task, and its latency shifts with that decision. This is why GPT-5.2 sits in the fast group on Q&A but drops back on coding.
We measured every model in its default state. Several are hybrids that can reason on demand but ship with thinking off: Claude Opus 4.8, for instance, supports extended thinking, but it is disabled by default, so it never produced reasoning tokens in our runs and stayed fast, as did the other Claude models (Opus 4.7, Sonnet 5, Haiku 4.5).
The rest reasoned on nearly every request, which is why they appear slower to first answer. So “never reasons” here means not by default, not that the capability is missing. GPT-5.2 stands out because it made this choice on its own, turning reasoning on for the harder coding prompts while leaving it off for Q&A.
Endpoint effect on latency for LLMs
A hosted model’s latency is not only the model. It also includes the provider’s serving stack, queueing, the network path, and geography. Our roster shows this inside a single model line. Claude Opus 4.8, served through Google Vertex in Europe, returned its first token in about 0.75 seconds, while Claude Opus 4.7, served through Anthropic’s own API, took about 2 seconds.
These are different versions, so the gap is not purely the endpoint, but the pattern matches a well-known effect: the same weights can be several times faster or slower depending on where and how they are served. Our measurement client ran in Europe, so the European Vertex endpoint had a shorter network path, which is part of why it came out ahead.
To see how we chose the provider for each model, read our benchmark methodology.
Max token limit and reasoning models
To measure latency fairly, we set a fixed maximum output length so that no model is rewarded or penalized for writing more. That cap is the token budget for the whole response, and it is where most failures came from:
- Reasoning models spend part of the budget on thinking before the answer begins.
- On hard coding prompts some think a lot, and unpredictably (MiniMax used about 2,500 to 3,700 thinking tokens on a single prompt).
- When thinking fills the whole budget, no tokens are left for the answer and the response comes back empty.
- This hit MiniMax hardest on coding, where the first pass completed fewer than half of its requests.
The takeaway is that the max token limit is a parameter worth watching: set it too low and a reasoning model can spend all of it thinking and never reach an answer. We raised the budget for reasoning models, gave thinking room on top of the answer length, and retried the failed requests. Most gaps filled, though a few still failed when a model’s thinking ran past even the larger budget.
Add as preferred sourceLLM latency benchmark methodology
We benchmarked 11 language models with 1,320 requests, measuring how quickly each model responds.
Models
We took the five most-used models on OpenRouter and added six more current models on top:
- Non-reasoning by default: Claude Opus 4.8, Claude Opus 4.7, Claude Sonnet 5, Claude Haiku 4.5, GPT-5.2
- Reasoning by default: DeepSeek V4 Flash, Xiaomi MiMo v2.5, MiniMax M3, Gemini 3.1 Pro, Tencent Hunyuan HY3, Claude Fable 5
Use cases and example prompts
We used two use cases with fixed input and output lengths, so results reflect speed and not how long a prompt or answer happened to be:
- Short answer (Q&A): short input (about 256 tokens), short output (capped at 128 tokens). Example: “What is the purpose of a primary key in a database?”
- Long answer (code generation): short input (about 512 tokens), long output (capped at 1,024 tokens). Example: “Build a multi-threaded URL downloader in Python that takes a list of image URLs and downloads them concurrently.”
Each use case used 20 prompts, and each prompt was sent 3 times per model, giving 60 measurements per model per use case. We report the median (p50) and the p90.
What we measured
We streamed every response and recorded when the request was sent, when the first token arrived, when the first answer token arrived (after any thinking), and when the last token arrived. From these we derived time to first token, time to first answer token, per-token latency and output speed, and end-to-end time.
Reasoning vs non-reasoning
Each response returns a usage summary that includes a reasoning-token count. We read this for every request and marked the request as reasoning when the count was above zero and non-reasoning when it was zero, instead of relying on the model’s label. This is how we found that GPT-5.2 reasons on coding but not on Q&A.
How we chose providers
The same model is served by different providers at different speeds, so we pinned each model to one provider instead of letting requests route freely. OpenRouter lists each provider’s uptime, latency, and throughput, and we used those figures to choose. We dropped any provider below about 98 percent uptime, then picked the fastest of the rest by latency and throughput. We pinned that provider for every request and logged the provider that actually served each response to confirm the pin held. A few of the fastest providers were not reachable with our account and returned rate-limit errors, so we used the fastest provider that served reliably.
The pinned providers were Claude Opus 4.8 on Google Vertex (Europe), Claude Opus 4.7 on Anthropic, Claude Sonnet 5 and GPT-5.2 on Azure, Claude Fable 5 and Claude Haiku 4.5 on Amazon Bedrock, DeepSeek V4 Flash on Novita, MiMo v2.5 on DeepInfra, MiniMax M3 on Together, Gemini 3.1 Pro on Google Vertex, and Hunyuan HY3 on GMICloud.
Controls
- All requests ran from a single client region.
- Output length was capped (128 for short, 1,024 for long) so verbosity does not distort latency. Reasoning models were given extra budget for thinking on top of this limit.
- Each prompt began with a unique block of text to defeat prompt caching, and we verified that no input tokens came from cache.
- All token counts use a single tokenizer so speed is comparable across models.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem and Şipi, Nazlı},
title = {{LLM Latency Benchmark by Use Cases in 2026}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/llm-latency-benchmark}},
note = {AIMultiple. Retrieved July 8, 2026}
}Results and timestamps of 1.3 thousand data points. Download the data used in this article as a ZIP file containing one CSV file and a README.
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.