RAG Evaluation Tools: W&B Weave vs Ragas vs TruLens
We benchmarked five retrieval-augmented generation (RAG) evaluation tools on 1,000+ adversarial questions, each paired with five passages. The comparison measures context relevance using GPT-4o and the tools’ default evaluation prompts.
RAG evaluation tools benchmark results
W&B Weave had the highest Top-1 accuracy at 94.5%. TruLens led the ranking metrics, including NDCG@5 at 0.932. Their positions depend on how each metric handles equal scores, especially when an evaluator returns binary outputs.
The benchmark methodology fixes the correct passage at the first position in each set. Top-1 accuracy gives that passage the win when it ties for the highest score.
Metrics explained
Top-1 accuracy measures how often the designated correct passage, called the golden context, receives the highest score. We also report strict Top-1, which requires it to be the unique maximum.
NDCG@5 (normalized discounted cumulative gain) evaluates the ordering of all five passages using relevance labels 4, 3, 2, 1 and 0. Spearman ρ measures the correlation between predicted scores and that relevance order. A perfect correlation is ρ = 1.0.
MRR (mean reciprocal rank) averages 1/rank for the golden context. Ranks one, two and three contribute 1.0, 0.5 and 0.33, respectively. MRR puts the golden context behind equally scored passages in the stored calculation. Its tie handling therefore differs from Top-1 accuracy.
Key findings
W&B Weave leads Top-1 accuracy and TruLens leads ranking
W&B Weave’s highest Top-1 accuracy accompanies the lowest NDCG@5, at 0.910, and a Spearman ρ of 0.669. TruLens leads on Spearman ρ at 0.750 and MRR at 0.594. Ties help explain the difference between W&B Weave’s identification and ranking results.
TruLens has the highest ratio of correct to reversed comparisons
For golden versus hard-negative passages, TruLens records 35.5% wins and 8.4% losses, a 4.2:1 ratio. Its four-point scale distinguishes more pairs than W&B Weave’s binary scale, with more reversals as well.
Every tool gives hard negatives higher mean scores than partial contexts
Hard negatives preserve the topic and most of the wording while changing an answer fact. All five tools give these passages higher mean scores than partial contexts, which lack the answer. This aggregate result shows why relevance scoring needs a separate check of factual accuracy.
DeepEval assigns lower scores to correct passages
DeepEval gives golden contexts a mean score of 0.46, compared with 0.82–0.91 for the other tools. Its NDCG@5 is 0.923, while its Top-1 accuracy is 78.1%. With statement-level scoring, additional details can lower a correct passage’s score even when it contains the answer.
UpTrain has the lowest full-ranking accuracy
UpTrain’s three possible scores cannot express five distinct relevance levels. It has the lowest five-way accuracy at 27.6% and the lowest golden-versus-hard-negative win-to-loss ratio at 1.4:1.
Discrimination: golden vs. hard negative
Golden and hard-negative passages differ in an answer fact. A win means the golden context scores higher, a loss means the hard negative scores higher, and a tie means the scores are equal.
W&B Weave has both the fewest wins and the fewest losses. About 80% of its comparisons are ties, limiting how often its binary score separates the two passages. Across all five contexts, its strict Top-1 accuracy is 8.3%, compared with 25.3% for TruLens.
Ranking quality
Pairwise accuracy counts correctly ordered pairs among the 10 pairs in each sample. Top-2 accuracy counts a sample as correct when either the golden or partial context shares the maximum score. Five-way accuracy requires all five scores to follow the expected non-increasing order.
Equal scores count as correct in pairwise and five-way accuracy. Pairwise accuracy uses s[i] >= s[j], and five-way accuracy accepts equal adjacent scores. W&B Weave leads all three measures, helped by ties being accepted as correct orderings. NDCG@5 and Spearman ρ evaluate the ordering with different treatment of ties.
Average scores by relevance level
In these group means, hard negatives outrank partial contexts for all five tools. Individual examples vary within each group.
How each tool evaluates context relevance
In the tested configurations, all five evaluators receive the query and one passage without a reference answer. Each uses GPT-4o, with differences in prompts, output scales and aggregation.
W&B Weave: Binary LLM prompt
W&B Weave is part of CoreWeave Forge, formerly Weights & Biases.12
W&B Weave makes one large language model (LLM) call per passage, asking for relevance “on a scale from 0 to 1.” Its integer response schema permits the binary values 0 and 1, which can give a partial context and a hard negative identical scores.
TruLens: 4-point Likert scale
TruLens asks the LLM to act as a “RELEVANCE grader” on a 0-3 scale. Its rubric covers four degrees of query relevance.
- 0: Irrelevant to the query
- 1: Relevant to some of the query
- 2: Relevant to most of the query
- 3: Relevant to the entirety of the query
Dividing the raw score by 3 gives four output values between 0.0 and 1.0, displayed as 0.0, 0.33, 0.67 and 1.0. This permits distinctions between partially and fully relevant passages, subject to the judge’s classification.
Ragas: Dual-judge averaging
Ragas uses two judge prompts with different wordings of the same criteria. Each prompt assigns 0 for irrelevant, 1 for partially relevant or 2 for fully relevant. Averaging and normalizing the judgments gives five values on a 0.0–1.0 scale.
Scores can be 0.0, 0.25, 0.5, 0.75 and 1.0. The average combines judgments from two prompt wordings. We did not separately test sensitivity to prompt changes.
UpTrain: Ternary classification (A/B/C)
UpTrain maps its A/B/C classifications to relevance scores.
- A (1.0): Context contains a full answer to the query
- B (0.5): Context contains information for a partial answer
- C (0.0): Context doesn’t contain any information to answer the query
Scores are 0.0, 0.5 and 1.0. A deceptive passage and a tangentially related one can still receive the same score.
DeepEval: Statement decomposition (G-Eval)
DeepEval splits a passage into statements and asks the judge to mark each as “yes” or “no” for relevance. It divides the relevant-statement count by the total, producing a fractional score between 0.0 and 1.0. For example, 7 relevant statements out of 10 produce 0.70.
Additional off-topic statements lower this fraction, even when the passage contains the correct answer. A shorter hard negative can consequently outscore a longer golden context.
The adversarial traps
Context can fail to answer a question through relation confusion, a misleading entity match or an omitted fact. These three illustrations come from earlier explanatory material and are separate from the scored v3 dataset.
The multi-hop trap (Relation confusion)
Multi-hop questions require following a chain of relationships. Answering a simpler question about the first named entity breaks that chain.
Question: “Who publishes the game series that Retro City Rampage is a parody of?” The target answer is Rockstar Games.
The entity distractor trap
A passage can identify the correct venue and location while describing a different event. Shared entity names can make that passage appear relevant.
Question: “…The Bridge Inn is the venue for which annual competition for telling lies, held in Cumbria, England?” The target answer is World’s Biggest Liar.
The partial relevance trap
A partial context supplies the topic and related entities while leaving out the requested answer.
Question: “Who wrote the lyrics of Portofino with a collaborator on ‘Fiddler on the Roof’?” The target answer is Richard Ney.
Add as preferred sourceRAG evaluation tools benchmark methodology
Adversarial dataset design
Each question is paired with five passages designed to occupy distinct relevance levels. Golden passages are paraphrased, and hard negatives replace an answer fact while preserving the surrounding structure.
Dataset
HaluEval contributes 480 general-knowledge questions, and HotPotQA contributes 530 questions requiring multi-hop reasoning. Together, they provide 1,010 questions and 5,050 question-context pairs per complete tool evaluation.
W&B Weave, TruLens, Ragas and UpTrain each have 1,010 complete result rows. DeepEval has 1,005 complete rows after five failed samples were excluded. A sample was excluded if any of its five context scores failed, so these failures were not assigned zero scores.
Project notes report 489 samples removed during answer-leak filtering before the final dataset was saved.
Cross-model protocol
Claude Sonnet 4.5 generated adversarial contexts, and GPT-4o evaluated relevance through OpenRouter. Using different models separates context generation from judging. Each evaluator received the same query-context inputs and used default evaluation prompts without custom instructions.
The project configuration sets temperature=0. Top-1 uses the first maximum returned by argmax, with the golden context fixed at index 0. Strict Top-1 requires a unique maximum. W&B Weave’s golden context ties for the maximum in about 86% of samples.
NDCG@5 uses relevance labels 4, 3, 2, 1 and 0. Spearman correlations are averaged across samples with nonconstant score vectors. The MRR implementation reverses an ascending score sort, placing the golden context behind tied passages in this result set.
Limitations
All results come from a single GPT-4o run with default evaluation prompts. Other judge models, prompt changes and naturally retrieved passages may produce different rankings. All hard negatives use entity swapping.
Tie conventions affect the ranking of tools, and the fixed position of the golden context favors Top-1 accuracy. Strict Top-1 provides a separate measure of unique identification.
DeepEval’s aggregate results exclude five samples, and Spearman excludes constant score vectors for each tool. Tool averages therefore use different eligible samples. Cost, latency, prompt robustness and combined-tool workflows were not benchmarked.
Context relevance is the evaluated metric. Answer faithfulness, which checks whether a generated answer is supported by retrieved material, and other RAG metrics are outside its measured scope.
Tool selection by use case
Binary relevance screening and passage ranking favor different tools in this test. The combined workflow is a proposal that requires separate validation.
Conclusion
W&B Weave’s binary output produces high Top-1 accuracy under this benchmark’s tie rule. TruLens and Ragas provide more score levels and lead the broader ranking measures.
All five tools achieve pairwise accuracy of 91% or higher, yet assign hard negatives higher mean scores than partial contexts. Relevance scoring can filter passages by topical fit. Factual verification and answer-faithfulness evaluation remain separate checks.
Further reading
- Embedding Models: 25 Models Benchmarked for Retrieval
- Open Source Embedding Models: 11 Models Benchmarked for Retrieval
- Vector Database Sizing and Selection Calculator
- Reranker Benchmark: Top 8 Models Compared
- Multimodal Embedding Models: Apple vs Meta vs OpenAI
- Top 10 Multilingual Embedding Models for RAG
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem and Sarı, Ekrem},
title = {{RAG Evaluation Tools: W&B Weave vs Ragas vs TruLens}},
year = {2026},
month = mar,
howpublished = {\url{https://aimultiple.com/rag-evaluation-tools}},
note = {AIMultiple. Retrieved March 17, 2026}
}Results and timestamps of 30 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 8 CSV files.
Want the granular data behind it? Join Premium
Changelog
2 updatesUpdated the benchmark results in the RAG evaluation tools benchmark results section.
Updated the "RAG evaluation tools benchmark results" section with new benchmark results and metrics.
Reference Links
Cem's work at AIMultiple has been cited by leading global publications including Business Insider, Forbes, Morning Brew, and Washington Post, global firms like Deloitte and HPE, NGOs like World Economic Forum, and supranational organizations like European Commission. [1], [2], [3], [4], [5]
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.