RELC-Bench: Retrieval on Long Context Benchmark
RELC-Bench (RELC-Bench: Retrieval on Long Context Benchmark) aims to measure a model’s ability to find and extract a specific numeric value from one or more documents within its context. It tests whether the model can remember and retrieve a specific fact it just saw in the input.
Results
claude-fable-5 scores 97.0% on the 100 direct-recall items, flat across haystack positions (97.0% beginning, 97.1% middle, 97.0% end).
Methodology
Question format
A natural-language question asking for one numeric metric. Example:
Q: What was the Revenue for Q1 2026 Adobe (ADBE)?
Expected: $6.40 billion
Data source
The script parses the Takeaways section of each Motley Fool earnings transcript and extracts all numeric metrics. For each metric, the script verifies the number appears verbatim in the post-Takeaways transcript body (the actual conference call text), so the model has to read the real conversation, not the summary bullet. Summary bullets are removed from the texts.
Scoring rule
- Each item has a list of target values; the first is the primary target (the headline answer to the question)
- Score = 1.0 if the primary target matches any number in the prediction
- Score = 0.0 otherwise
- Refusals (“I don’t know”) score 0.0
- claude-fable-5 is tested through Claude Code: the haystack is provided as a file and the model retrieves from it with search tools instead of reading it from its context window. Its scores measure the model together with the Claude Code harness, and position invariance is expected in this setup because target depth does not apply to file search.
What good performance looks like
Phase 1 ≥ 85% (model reliably finds metrics in a single doc).
Phase 2 ≥ 90% (model navigates to the target in a haystack without distraction).
Position-invariant scores indicate true long-context capability; declining scores by depth indicate “lost in the middle.”
Item count
100 direct-recall items spread across 14 transcripts.
Further reading
- Cognitive Agents: Creating a Mind with LangChain
- 5 Open-Source Agentic AI Frameworks
- AI Apps with MCP Memory Benchmark & Tutorial
- Code Execution with MCP: A New Approach to AI Agent Efficiency
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem},
title = {{RELC-Bench: Retrieval on Long Context Benchmark}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/ai-memory}},
note = {AIMultiple. Retrieved July 7, 2026}
}Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Add as preferred source
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.