RELC-Bench: Retrieval on Long Context Benchmark
RELC-Bench (RELC-Bench: Retrieval on Long Context Benchmark) aims to measure a model’s ability to find and extract a specific numeric value from one or more documents within its context. It tests whether the model can remember and retrieve a specific fact it just saw in the input.
Results
claude-fable-5 scores 97.0% on the 100 direct-recall items, flat across haystack positions (97.0% beginning, 97.1% middle, 97.0% end).
Methodology
Question format
A natural-language question asking for one numeric metric. Example:
Q: What was the Revenue for Q1 2026 Adobe (ADBE)?
Expected: $6.40 billion
Data source
The script parses the Takeaways section of each Motley Fool earnings transcript and extracts all numeric metrics. For each metric, the script verifies the number appears verbatim in the post-Takeaways transcript body (the actual conference call text), so the model has to read the real conversation, not the summary bullet. Summary bullets are removed from the texts.
Scoring rule
- Each item has a list of target values; the first is the primary target (the headline answer to the question)
- Score = 1.0 if the primary target matches any number in the prediction
- Score = 0.0 otherwise
- Refusals (“I don’t know”) score 0.0
- claude-fable-5 is tested through Claude Code: the haystack is provided as a file and the model retrieves from it with search tools instead of reading it from its context window. Its scores measure the model together with the Claude Code harness, and position invariance is expected in this setup because target depth does not apply to file search.
What good performance looks like
Phase 1 ≥ 85% (model reliably finds metrics in a single doc).
Phase 2 ≥ 90% (model navigates to the target in a haystack without distraction).
Position-invariant scores indicate true long-context capability; declining scores by depth indicate “lost in the middle.”
Item count
100 direct-recall items spread across 14 transcripts.
What is AI memory?
AI memory is an artificial intelligence system’s ability to store and recall past experiences to improve decision-making, perception, and overall performance.1 Unlike human memory, which is subjective and selective, AI memory is a technical architecture, a structured store of information within neural networks or external databases.
The concept is distinct from a model’s context window, which is the finite, session-bound token buffer that serves as working memory during inference.2 While the context window resets between sessions and loses information when tokens are truncated, AI memory persists across sessions through external storage.3 It is also distinct from retrieval-augmented generation (RAG), which is read-only and treats relevance as a property of content rather than the user; AI memory requires a write path that extracts facts from conversations and updates storage when information changes.4
The field recognizes two primary tiers. Short-term memory, or working memory, tracks ongoing conversation within a session, while long-term memory stores user-specific or application-level data across sessions.5 Some frameworks add a third category, episodic memory, which allows agents to recall specific past experiences similar to how humans remember individual events.
In consumer products, “chat memory” refers specifically to user preferences and conversation gist surfaced automatically, which is architecturally narrower than “agent memory” as used in developer frameworks, the latter encompassing task state, tool-call history, and learned procedures.6
How AI memory works?
AI memory systems converge on a common write/retrieve/consolidate pipeline that differs from static retrieval. New information enters through an extraction phase that processes message pairs to identify candidate facts, passes through an update phase where an LLM decides to add, update, delete, or ignore each fact, and exits through a retrieval phase that performs semantic search at query time.7
Real-world applications of AI memory
Personal AI assistants
OpenAI’s ChatGPT memory feature builds persistent profiles from user conversations, files, and connected apps. A 2026 “Dreaming” update introduced background synthesis across many conversations, enabling memories to self-update over time; for example, a saved memory stating “the user is going to Singapore in July” automatically rewrites to “the user went to Singapore in July 2026” once the trip passes.8 Users manage memory through Settings > Personalization > Memory, though OpenAI notes the visible summary does not capture everything stored.
Anthropic’s Claude memory launched for Team plan users on September 11, 2025, extended to Enterprise on September 18, 2025, and reached all users including the free tier on March 2, 2026.9 On July 10, 2026, Anthropic redesigned the feature into categorized entries that Claude reads and updates during conversations, replacing the earlier single daily summary.
Google’s Gemini stores information users ask it to remember plus details inferred from past conversations when Personal context is enabled.10 The separate “Personal Intelligence” upgrade, when opted in, connects to Gmail and Photos without training models on that content.11
Enterprise agents and customer support
Fastweb and Vodafone Italy deployed a multi-agent system built with LangGraph and LangSmith serving 9.5 million customers, automatically injecting CRM data at the start of each interaction to identify customers and tailor responses. The system reports a 90% correctness rate, 82% resolution rate, and Customer Effort Score of 5.2 out of 7.12
Decagon’s deployment with Rippling connected the AI agent to internal APIs so the agent could pull individual employee account data rather than answering generically, increasing chat deflection from 38% to over 50%, a 32% relative improvement.13 Decagon’s work with Chime automated high-friction tasks including card replacement and deposit status checks, cutting customer support costs by 60% while doubling member satisfaction scores.14
Robotics, healthcare, and embodied AI
Diligent Robotics’ hospital logistics robot Moxi operates in more than 25 U.S. facilities and has completed over 1.25 million autonomous deliveries as of 2026, mapping each hospital environment using onboard sensors and navigating dynamic obstacles including badge-access doors.15 Moxi 2.0 runs on NVIDIA IGX Thor compute with 10 times the processing capacity of the original, trained on three years of real-world hospital delivery data.
NVIDIA’s ReMEmbR framework gives mobile robots long-horizon spatio-temporal memory to build and reason over ongoing records of what they have seen for navigation tasks.16 ASPIRE, released June 29, 2026, stores every validated debugging fix as a named, reusable code pattern, raising success rates from 20% to 92% on Robosuite’s bimanual object-handover benchmark through iterative debugging alone.17
In healthcare, Abridge’s ambient AI documentation platform is deployed by Northwell Health across 28 hospitals to reduce clinical documentation burden.18 Riverside Health piloted ambient documentation tools for two months before selecting Abridge for expanded deployment.19
Add as preferred sourceBenefits of AI memory
Memory systems reduce token costs and latency in long-horizon benchmarks compared to full-context replay. On the LoCoMo benchmark, which tests conversations averaging 300 turns across up to 35 sessions, Mem0 reports saving more than 90% in token cost versus replaying full conversation history, cutting p95 latency by 91% (1.44s vs. 17.12s), and achieving a 26% relative improvement in LLM-as-a-Judge score over OpenAI’s memory baseline.7
Zep reports 94.8% accuracy on the Deep Memory Retrieval benchmark versus 93.4% for MemGPT, up to 18.5% aggregate accuracy gains on LongMemEval paired with GPT-4o, and a 90% reduction in response latency compared to full-context baselines, with median response time dropping from 28.9 seconds to 2.58 seconds while context tokens dropped from approximately 115,000 to 1,600.20 21
The BEAM benchmark, testing up to 10 million tokens, found that its LIGHT memory framework improved performance by 3.5 to 12.69 percentage points on average over the strongest existing baselines, and that even LLMs with 1-million-token context windows degrade substantially as conversation length grows.22
OpenAI’s Dreaming architecture for ChatGPT improved time-sensitive memory accuracy from 9.4% to 75.1%, with factual recall of 82.8% and preference adherence of 71.3%, while cutting compute cost by roughly 5x.8
Challenges and risks of AI memory
Memory poisoning and security
Memory poisoning is classified as ASI06, “Memory and Context Poisoning,” in the OWASP Top 10 for Agentic Applications, defined as corruption of persistent agent memory leading to misalignment and malicious behavior across sessions.23 24 In February 2025, researcher Johann Rehberger demonstrated a “delayed tool invocation” attack against Google Gemini that caused the system to store false facts as long-term memories, including incorrect age information.25
MINJA (Memory INJection Attack), accepted at NeurIPS 2025, shows that attackers with only ordinary query-and-response interaction can poison long-term memory, reaching 98.2% injection success rate and 76.8% attack success rate averaged across tested agents.26 AgentPoison, published at NeurIPS 2024, achieves over 80% attack success rate while poisoning fewer than 0.1% of memory records, degrading benign task performance by less than 1%.27
Privacy and data governance
MEXTRA, a black-box Memory EXTRaction Attack, crafts queries to extract stored private data, achieving 83% complete-extraction rates against EHRAgent and 87% against web-shopping agents using 30 attack prompts.28 Increasing retrieval depth or total memory size both increase leakage rates.28
Italy’s data protection authority fined OpenAI €15 million in December 2024 for processing personal data to train ChatGPT without adequate legal basis, though the Court of Rome annulled the fine in March 2026 ruling that Ireland’s Data Protection Commission had jurisdiction as lead EU supervisory authority.29 30
OpenAI’s documentation states that turning memory off does not automatically delete previously stored memories; a full reset requires both disabling memory and separately clearing existing entries.
Technical limitations
Across ten evaluated memory approaches on the LoCoMo benchmark, most methods score lower on temporal-reasoning questions than on single-hop factual retrieval, though at least one approach reverses this pattern.7
Memory staleness presents another unresolved problem: when stored facts are contradicted by new information, systems struggle to distinguish current from outdated data. The MemConflict benchmark evaluates scenarios where stored facts become outdated or directly conflict, finding that existing systems fail to reliably resolve these conflicts.31
Additionally, performance on standardized benchmarks does not reliably translate to production reliability because synthetic environments oversimplify real usage.32
Further reading
- Cognitive Agents: Creating a Mind with LangChain
- 5 Open-Source Agentic AI Frameworks
- AI Apps with MCP Memory Benchmark & Tutorial
- Code Execution with MCP: A New Approach to AI Agent Efficiency
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem},
title = {{RELC-Bench: Retrieval on Long Context Benchmark}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/ai-memory}},
note = {AIMultiple. Retrieved August 4, 2026}
}Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.