RAG Benchmarks: Embedding Models, Vector DBs and Rerankers
RAG improves LLM reliability by grounding responses in external data. We benchmark the RAG stack across embedding models, rerankers, vector databases, frameworks and evaluation tools on retrieval accuracy, latency and quality.
Explore RAG Benchmarks: Embedding Models, Vector DBs and Rerankers
AIM-Agentic RAG Benchmark: Routing Across 11 SQL Databases
RAGBenchmarkOct 14Routing accuracy is the percentage of scored questions for which the model explicitly names the correct database in its final answer. The headline uses the 184 questions flagged as difficult by both our similarity test and a jury of three LLMs. Missing explicit declarations receive no credit. gemini-4-argon selected the correct database for 173 of…
- RAGBenchmarkOct 14
RAG Evaluation Tools: W&B Weave vs Ragas vs TruLens
We benchmarked five retrieval-augmented generation (RAG) evaluation tools on 1,000+ adversarial questions, each paired with five passages. The comparison measures context relevance using GPT-4o and the tools’ default evaluation prompts. W&B Weave had the highest Top-1 accuracy at 94.5%. TruLens led the ranking metrics, including NDCG@5 at 0.932. Their positions depend on how each metric…
- EmbeddingsBenchmarkOct 1
Embedding Models: 25 Models Benchmarked for Retrieval
We benchmarked 25 embedding models on 200 retrieval questions across finance, legal contracts, technical support and medical literature. The four corpora contained 341,040 text passages in total. Each query searched its own domain. nDCG@10 measures how highly the recorded relevant passages appear in the first ten results. Scores range from 0 to 1, with higher…
- EmbeddingsBenchmarkSep 27
Open Source Embedding Models: 11 Models Benchmarked for Retrieval
nDCG@10 scores the placement of recorded relevant passages within the first ten search results. A score of 1 represents the ideal ranking for the recorded positives. Passages nearer the top receive more weight. PPLX Embed v1 4B led the comparison at 0.741 nDCG@10, followed by PPLX Embed v1 0.6B at 0.698. Multilingual E5 Large ranked…
- RAGBenchmarkSep 22
Reranker Benchmark: Top 8 Models Compared
We retrieved top-100 candidates with multilingual-e5-base, reranked them with each model, and evaluated the top-10 results against 300 queries, each referencing concrete details from its source review. The best reranker lifted Hit@1 from 62.67% to 83.00% (+20.33pp). Metrics explained: ΔHit@1 / ΔHit@10 shows the improvement over the baseline (no reranker) in percentage points (pp). For…
- Vector DBBenchmarkSep 4
Vector Database Benchmark: 7 Open-Source Engines for RAG
We benchmarked seven open-source, self-hosted vector databases as the retrieval layer of a RAG pipeline, each run one at a time on identical bge-m3 embeddings and real medical and technical queries, so the database index was the sole variable. The workload spanned MedRAG-50k, TechQA-28k, and a 2.25M-vector corpus across eight dimensions, from accuracy and retrieval…
- RAGBenchmarkAug 31
RAG Frameworks: LangChain vs LangGraph vs LlamaIndex
We benchmarked 5 RAG frameworks: LangChain, LangGraph, LlamaIndex, Haystack, and DSPy, by building the same agentic RAG workflow with standardized components: identical models (GPT-4.1-mini), embeddings (BGE-small), retriever (Qdrant), and tools (Tavily web search). This isolates each framework’s true overhead and token efficiency. The benchmark consisted of 100 queries, with each framework running the full set…
- RAGBenchmarkAug 31
Best RAG Tools, Frameworks, and Libraries
RAG improves LLM responses by grounding them in external data instead of just what the model memorized in training. We benchmarked the components a RAG system is built from and gathered the results in one place, with a practical guide to choosing each part of the stack. See our benchmark results for each RAG component,…