Embedding Models: 22 Models Benchmarked for Retrieval
We benchmarked 22 embedding models on 200 retrieval questions across finance, legal contracts, technical support and medical literature. The four corpora contained 341,040 text passages in total. Each query searched its own domain.
Embedding models benchmark results
nDCG@10 measures how highly the recorded relevant passages appear in the first ten results. Scores range from 0 to 1, with higher values rewarding better placement.
Recall@10 measures the share of each question’s recorded relevant passages retrieved within the first ten results, averaged across questions. Generated-answer accuracy was outside this evaluation.
The 95% confidence intervals (CIs) estimate uncertainty in average nDCG@10 from question sampling. Wider intervals indicate less precise estimates.
Questions were selected because BM25 missed their answering passages in its first ten results. Its zero score reflects that selection and limits comparisons with keyword retrieval on ordinary traffic.
Retrieval findings across 22 models
Voyage 4 Large recorded the highest nDCG@10 at 0.851
Its 95% CI was 0.817–0.884. Paired tests found no significant difference from Gemini Embedding 2, Voyage 4 or Gemini Embedding 001 after correcting for multiple comparisons. These results leave uncertainty about the ranking and do not establish equal performance.
Gemini Embedding 2 recorded the highest Recall@10 at 96.50%
Voyage 4 followed at 95.83% and Voyage 4 Large at 95.46%. No pairwise recall difference among the eight highest-recall models was significant after correction. Moving a relevant passage from fifth to first improves nDCG without changing recall.
PPLX Embed v1 4B led the open-weight models at 0.741 nDCG@10
PPLX Embed v1 0.6B followed at 0.698. The 4B model scored significantly above nine of the other ten open-weight models. Its gap to 0.6B was not significant. The open-source embedding model comparison covers this subset.
Domain results for finance, contracts, support and medicine
The six highest nDCG@10 scores within each domain. Each filter re-ranks the completed models on that domain’s 50 questions.
Each domain contributes equally to the overall average. MedRAG produced every model’s highest nDCG@10. With 50 questions per domain, small differences in these scores remain uncertain.
How text embeddings support retrieval
An embedding model converts text into a vector, a list of numerical values that can be compared with other vectors from a compatible model. In semantic retrieval, document passages are encoded in advance. At search time, the system encodes the query and ranks stored passages by vector similarity.1
For example, a question about cancelling a plan can be close to a passage about terminating a subscription despite different wording. The model’s training determines which relationships the vectors preserve.2 The diagram compresses this idea into two dimensions. Deployed embeddings generally contain many more coordinates.
Query and document formatting also matters. A model may expect separate retrieval roles or prefixes even when both outputs occupy a compatible vector space. We applied each model’s required query and document formatting.
A vector database stores and searches vectors, while the embedding model determines the representations being searched. Retrieved passages can then become context for an answer generator. This benchmark stops at passage retrieval and does not score generated answers.
Embedding models benchmark methodology
All models searched the same passages using the same 200 questions and 225 recorded relevant question-passage pairs. Questions and labels were frozen before comparing embedding scores. Query inputs contained the original question alone.
Finance passages came from FinQA annual reports and legal passages from CUAD v1 contracts.34 Technical support used TechQA technotes, and medical literature used a sample of MedRAG PubMed abstracts.56
Passages were whole paragraphs, tables with context, contract clauses or abstracts with titles. We excluded units longer than 256 tokens under the all-MiniLM-L6-v2 tokenizer. Accepted passages were kept intact, and the 512-token models received at most 459 tokens per passage. This benchmark therefore evaluates short passages.
A frontier LLMs drafted the questions from source passages. A team member verified 20% of the questions (40 of 200). The agent also judged BM25’s top 20 passages per question, adding 25 relevant passages to the original 200 source passages. These relevance labels received no human review.
Retrieval and statistical analysis
Retrieval compared each query with every passage in its domain. We used cosine similarity for 21 models and dot product for MultiQA MPNet Base Dot v1. BGE-M3 used its dense vectors.7 Query and document inputs used each model’s configured retrieval roles or prefixes.
Both metrics are averages across questions. We calculated 95% CIs with 10,000 bootstrap samples, resampling 50 questions within each domain. Pairwise comparisons used paired sign-flip tests with Holm correction across all 231 model pairs for each metric and scope. We called differences significant at a corrected p-value below 0.05.
Tested model IDs, dimensions and input limits
The runs took place through OpenRouter, with one fixed backend provider per model and fallbacks disabled. The table records the API identifiers, output dimensions and input limits used in those runs. These limits describe the benchmark configuration, including conservative bounds where applicable.
API identifiers do not guarantee an immutable model snapshot. Tokenizer revisions were pinned where available. Hosted model weights were not independently verified. Gemini input lengths were checked using conservative estimates. The short passages did not test long-context retrieval.
Add as preferred sourceScope and limitations
The questions form a BM25-selected development set. Results on ordinary production queries may differ. Relevance judgments were limited to the source passages and BM25’s top 20. Across all 22 models, 87% of distinct question-passage pairs retrieved in the top ten were unjudged and received no relevance credit.
Each model was run once. The CIs reflect question sampling and exclude variation between API runs or uncertainty from incomplete labels. Training-data contamination was not tested.
This evaluation covers text retrieval. Generated-answer accuracy, hybrid retrieval, reranking, latency and serving cost were outside its scope. The reranker benchmark, multilingual embedding comparison and multimodal embedding comparison cover separate workloads.
Conclusion
Voyage 4 Large recorded the highest nDCG@10, and Gemini Embedding 2 recorded the highest recall. Neither result establishes a clear winner over the closest alternatives. PPLX Embed v1 4B led the open-weight group.
Use the domain results to choose candidates, then compare them on representative queries with reviewed relevance labels.
Further reading
- Open-source embedding model benchmark
- Multilingual embedding models for RAG
- Multimodal embedding models
- Open-source vector database benchmark
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Embedding Models: 22 Models Benchmarked for Retrieval}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/embedding-models}},
note = {AIMultiple. Retrieved September 26, 2026}
}Results and timestamps of 95 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 5 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
5 updatesAdded Top-1 hit metric definition to the embedding benchmark methodology.
Replaced the benchmark with 15 models and a BM25 baseline tested on legal, customer support and healthcare corpora.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.