VELC-Bench: Verification on Long Context Benchmark
The model’s ability to locate a specific metric in context, compare its value to a claim, and confirm or reject it. This tests fine-grained value matching under long-context conditions. The model must both retrieve the value and perform a precise comparison.
Results
The models are tested in the following context windows:
- anthropic/claude-fable-5: 850,000 tokens tested
- openai/gpt-5.5: 1,000,000 tokens
- google/gemini-3.1-pro-preview: 1,000,000 tokens
- google/gemini-3.5-flash: 1,000,000 tokens
- anthropic/claude-sonnet-4.6: 1,000,000 tokens
- qwen/qwen3.6-plus: 1,000,000 tokens
- moonshotai/kimi-k2.6: 200,000 tokens
- z-ai/glm-5.1: 200,000 tokens
- minimax/minimax-m2.7: 150,000 tokens
- openai/gpt-5.4-mini: 250,000 tokens
claude-fable-5 scores 90.0% on verify YES and 94.0% on verify NO. The gap matches the asymmetry described below: confirming a value requires finding it, while rejecting one only requires spotting a mismatch.
Question formats
Verify YES (the claim’s value is correct):
Claim: The Revenue for Q1 2026 Adobe (ADBE) is $6.40 billion.
Expected: YES
Verify NO (the claim’s value is wrong):
Claim: The Revenue for Q1 2026 Adobe (ADBE) is $7.92 billion.
Expected: NO
Data source
Same TAKEAWAYS-extracted metrics as direct recall. For each chosen metric:
- Verify YES items use the actual value from the transcript
- Verify NO items use a programmatically perturbed value (8–25% off, in either direction, with matching precision and units)
Scoring rule
Three-state detection on the model’s response:
- If the response contains a NOT MENTIONED phrase (e.g., “not mentioned,” “not discussed”) → predicted =
not_mentioned - Else if it contains “yes” → predicted =
yes - Else if it contains “no” → predicted =
no
Score = 1.0 if predicted == expected, else 0.0.
Detection priority is NOT MENTIONED > NO > YES to prevent “not mentioned” from accidentally matching “no” via the substring “not.”
claude-fable-5 is tested through Claude Code: it receives the 850,000-token haystack as a file and searches it with retrieval tools instead of reading it from its context window, so its scores measure the model together with the Claude Code harness.
Claude Sonnet 5 launched June 30, 2026 with a native 1M-token window at standard pricing, and Claude Opus 4.8 carries the same 1M window both are now part of this test set.1
Two open-weight models trained entirely on non-Nvidia hardware also entered the long-context field the same week: Meituan’s LongCat-2.0, a 1.6T-parameter MoE model with a native 1M-token window built on Chinese AI accelerators, and Huawei’s openPangu-2.0-Flash, a 92B-parameter MoE model with a 512K window trained on Ascend chips.2
Phase-by-phase interpretation
The asymmetry between YES and NO is informative: YES requires positive identification of a value (harder when the target is deeper), while NO requires only spotting a mismatch (easier when recently read).
Phases are 0.1, 0.5, and 0.9 of the context window, to see the difference in accuracy in different haystack positions.
What is a good performance?
Phase 2 YES ≥ 80% and NO ≥ 80% indicates the model can both confirm and reject across a haystack.
A model that scores very high on NO but low on YES is biased toward rejection. A model that scores very high on YES but low on NO is over-trusting claims.
Item count
50 verify_yes + 50 verify_no = 100 verify items.
What is a context window?
Anthropic’s technical documentation offers the canonical definition: the context window is “all the text a language model can reference when generating a response, including the response itself,” distinguishing it from the larger training corpus and instead representing a “working memory” for the model.3 IBM defines it in near-identical terms as the amount of text, in tokens, that the model can consider or “remember” at any one time.4 McKinsey frames the concept as comparable to human short-term memory, noting that the model can only “look at” a fixed amount of information at once before combining it with pretrained parameters to produce a response.5
The unit of measurement is the token. OpenAI’s developer documentation states that one token corresponds to roughly 4 characters or 0.75 words for English text, though the ratio varies by language and content.6 IBM describes a token as the smallest unit of language AI models use, a machine-readable representation of a word, part of a word, or punctuation.4
The window covers the system prompt, every message including tool results and documents, and the generated output combined. Anthropic’s documentation explicitly states that “the output Claude generates for the turn, including its extended thinking, counts too.”3 OpenAI confirms that for text generation models, the prompt and generated output combined must not exceed the maximum context length.6
Context window size is a fixed architectural property set at training time, not a user-adjustable setting. Hugging Face’s LLM course documentation notes that base Transformer architectures without extended-context modifications are limited to a fixed maximum sequence length and will crash or truncate input when that length is exceeded, though specialized techniques now let some deployed models handle far longer sequences.7 A model cannot be given a larger context window after deployment without retraining or specialized extension techniques.
Context window vs related terms
“Context length” functions as a direct synonym for “context window,” not a distinct concept. IBM presents both terms together, stating that the “context window (or ‘context length’)” is the amount of text the model can consider at any one time.4 OpenAI uses “maximum context length” to describe the same combined prompt-plus-output ceiling.6
The output token limit is a separate, narrower term. OpenAI exposes this as a distinct parameter (such as `max_tokens`) that caps only the generated portion of the response, whereas the context window constrains the total of input plus output.6
Context window is distinct from persistent or agentic memory across sessions. Anthropic’s documentation for its memory tool clarifies that the context window is ephemeral and session-bound: “Your context window might be reset at any moment, so you risk losing any progress that is not recorded in your memory directory.”8 Persistent memory, by contrast, stores information in external files that survive between sessions.8
Finally, the context window is not the corpus of data the model was trained on. Anthropic’s documentation draws this line explicitly: the context window “is different from the large corpus of data the language model was trained on, and instead represents a ‘working memory’ for the model.”3
How a context window works?
Before processing, raw text is converted into tokens through tokenization. The dominant method is byte pair encoding (BPE), originally a 1994 data compression algorithm adapted for neural machine translation in 2016, which builds a vocabulary by iteratively merging the most frequently co-occurring character sequences.9 BPE powers tokenizers in GPT-2, GPT-3, GPT-4, and LLaMA, among others.9 OpenAI notes that tokenization is context-sensitive; the same word can map to different tokens depending on capitalization or surrounding whitespace.10
Once tokenized, each token is converted into a vector and processed through the transformer architecture introduced in the 2017 paper “Attention Is All You Need.”11 The core mechanism, self-attention, computes Query, Key, and Value vectors for every token, then uses scaled dot-product attention so that each token attends to every other token in the sequence, including itself.11 This parallel comparison of all token pairs is what lets transformers replace the sequential processing of earlier recurrent architectures.
During autoregressive generation, the model produces one token at a time. Each new token’s self-attention step requires the Key and Value vectors of every prior token. Rather than recomputing these from scratch at each step, production LLMs store them after the first computation and reuse them; this stored set is called the KV cache.12 By reusing cached Key and Value matrices, per-step inference complexity drops, though the cache’s memory footprint grows linearly with context length and must reside in GPU memory for the duration of the request.12
Tokens outside the window are architecturally invisible. Because self-attention only computes relationships among tokens present in the current input tensor, and the KV cache only stores vectors for tokens that were fed into the model, a token never included in the request has no representation for the model to reference.11 This is why context window overflow triggers a hard API failure; Anthropic’s API returns a 400 `invalid_request_error` if input alone exceeds the limit, and generation halts with a `model_context_window_exceeded` stop reason if output would push the total past the ceiling.3
Context window sizes across leading AI models
As of mid-2026, the frontier API default has settled at roughly 1 million tokens, though the effective limit varies by access surface.
OpenAI’s GPT-5.6 family (Sol, Terra, and Luna) offers a 1.05 million token context window and 128,000 token max output via API.13 However, the ChatGPT consumer interface offers less: when a user manually selects “Thinking” mode, the combined window is 256,000 tokens (128,000 input plus 128,000 max output).14
Anthropic’s Claude Opus 5, and Sonnet 5 offer a 1 million token window via the Claude API and on Amazon Bedrock, Google Cloud, and Microsoft Foundry.3
Google’s Gemini 3.5 Flash supports a 1 million token input context window with up to 65,536 output tokens.15 Claims that newer Gemini 3.5 Pro reaches 2 million tokens could not be verified against primary Google documentation at the time of writing.
xAI’s Grok 4.3, launched April 30, 2026, carries a 1 million token context window.16 Meta’s Llama 4 Scout advertises a 10 million token context window reached via an iRoPE architecture, though the model was pretrained only up to 256,000 tokens, meaning the 10 million figure represents extrapolated capability.17
Mistral Large 3 remains capped at 256,000 tokens, an outlier among frontier models.18 DeepSeek-V4 ships in two variants supporting 1 million tokens, achieved through an efficient attention mechanism.19
Add as preferred sourceLong context windows vs retrieval-augmented generation
The choice between stuffing tokens into a long context window versus using retrieval-augmented generation (RAG) depends on corpus size and update frequency. Anthropic’s engineering guidance advises that if a knowledge base is smaller than 200,000 tokens (about 500 pages), the entire corpus can be included in the prompt, which with prompt caching becomes “significantly faster and more cost-effective.” Once the corpus exceeds that threshold, there is a need for a more scalable solution.20
Agentic AI and long-running tasks
Coding and research agents rely on context window size to hold accumulated tool outputs and conversation history. Anthropic’s engineering team documents context management as the central constraint for long-horizon tasks, recommending “just-in-time” context where agents keep lightweight references and dynamically load data at runtime rather than loading everything upfront.21
Claude Code implements this via sub-agents. In one documented example, a sub-agent tasked with researching a codebase read 6,100 tokens of file content and returned a 420-token summary to the parent session, keeping roughly 5,700 tokens of file content out of the primary agent’s context window entirely.22 Claude Code’s default context window for Sonnet 5 is 1 million tokens on the Anthropic API, with sessions auto-compacting by default at approximately 967,000 tokens, or about 96.7% of that limit, unless a configuration caps the window at 200,000 tokens instead.23
Document and codebase analysis
Google’s Gemini API documentation states that its 1 million token context window is equivalent to roughly 50,000 lines of code, illustrating the scale of a single context window rather than describing a dedicated codebase review use case.24
Cite this research
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem and Alper, Şevval},
title = {{VELC-Bench: Verification on Long Context Benchmark}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/ai-context-window}},
note = {AIMultiple. Retrieved August 4, 2026}
}Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.