Embedding Models: 25 Models Benchmarked for Retrieval
We benchmarked 25 embedding models on 200 retrieval questions across finance, legal contracts, technical support and medical literature. The four corpora contained 341,040 text passages in total. Each query searched its own domain.
Embedding models benchmark results
nDCG@10 measures how highly the recorded relevant passages appear in the first ten results. Scores range from 0 to 1, with higher values rewarding better placement.
Recall@10 measures the share of each question’s recorded relevant passages retrieved within the first ten results, averaged across questions. Generated-answer accuracy was outside this evaluation.
The 95% confidence intervals (CIs) estimate uncertainty in average nDCG@10 from question sampling. Wider intervals indicate less precise estimates.
Questions were selected because BM25 missed their answering passages in its first ten results. Its zero score reflects that selection and limits comparisons with keyword retrieval on ordinary traffic.
Retrieval findings across 25 models
Voyage 4 Large recorded the highest nDCG@10 at 0.851
Its 95% CI was 0.817–0.884. Paired tests found no significant difference from Cohere Embed 5 Pro, Cohere Embed 5 Fast, Gemini Embedding 2, Voyage 4 or Gemini Embedding 001 after correcting for multiple comparisons. The ranking remains uncertain. A nonsignificant result does not establish equal performance.
Cohere Embed 5 Pro and Fast scored significantly above Embed v4.0
Pro scored 0.850 and Fast scored 0.839, compared with 0.772 for Cohere Embed v4.0. Their gains of 0.078 and 0.067 nDCG@10 were significant after correction across all 300 model pairs. The 0.010 difference between Pro and Fast was not significant.
Gemini Embedding 2 recorded the highest Recall@10 at 96.50%
Cohere Embed 5 Pro followed at 95.88% and Voyage 4 at 95.83%. Neither difference from Gemini Embedding 2 was significant after correction. Moving a relevant passage from fifth to first improves nDCG without changing recall.
PPLX Embed v1 4B led the open-weight models at 0.741 nDCG@10
PPLX Embed v1 0.6B followed at 0.698. The 4B model scored significantly above nine of the other ten open-weight models. Its gap to 0.6B was not significant. The open-source embedding model comparison covers this subset.
Domain results for finance, contracts, support and medicine
The six highest nDCG@10 scores within each domain. Each filter re-ranks the completed models on that domain’s 50 questions.
Each domain contributes equally to the overall average. Cohere Embed 5 Fast recorded the highest CUAD score at 0.762, and Pro recorded the highest TechQA score at 0.894. MedRAG produced every model’s highest nDCG@10. With 50 questions per domain, small differences in these scores remain uncertain.
Embedding cost and retrieval quality
We calculated the API cost of embedding the same 341,040 passages and 200 queries for each model.
PPLX Embed v1 0.6B had the lowest embedding cost at $0.13 and scored 0.698 nDCG@10. Voyage 4 Lite cost $0.62 and scored 0.793. At $1.87, Voyage 4 reached 0.829.
Cohere Embed 5 Fast cost $2.52 and scored 0.839, while Pro cost $3.40 and scored 0.850. Fast’s embedding cost was 26% lower on this workload. The difference in their retrieval scores was not statistically significant after correction for multiple comparisons.
Voyage 4 Large cost $3.74 and recorded the highest score at 0.851. Gemini Embedding 2 had the highest cost at $6.68 and scored 0.830. Spending more on embeddings did not consistently produce a higher retrieval score in this test.
Costs cover successful embedding calls used in the evaluation. Cohere totals use reported input tokens multiplied by the recorded tariff. Embed v4.0’s $3.72 cost is an estimate because its direct API tariff was unverified. Retries, diagnostics, account fees, storage and search compute are excluded. These amounts describe the tested workload and API routes. Production costs will depend on text volume, tokenization and provider pricing.
How text embeddings support retrieval
An embedding model converts text into a vector, a list of numerical values that can be compared with other vectors from a compatible model. In semantic retrieval, document passages are encoded in advance. At search time, the system encodes the query and ranks stored passages by vector similarity.1
For example, a question about cancelling a plan can be close to a passage about terminating a subscription despite different wording. The model’s training determines which relationships the vectors preserve.2 The diagram compresses this idea into two dimensions. Deployed embeddings generally contain many more coordinates.
Query and document formatting also matters. A model may expect separate retrieval roles or prefixes even when both outputs occupy a compatible vector space. We applied each model’s required query and document formatting.
A vector database stores and searches vectors, while the embedding model determines the representations being searched. Retrieved passages can then become context for an answer generator. This benchmark stops at passage retrieval and does not score generated answers.
Embedding models benchmark methodology
All models searched the same passages using the same 200 questions and 225 recorded relevant question-passage pairs. Questions and labels were frozen before comparing embedding scores. Query inputs contained the original question alone.
Finance passages came from FinQA annual reports and legal passages from CUAD v1 contracts.34 Technical support used TechQA technotes, and medical literature used a sample of MedRAG PubMed abstracts.56
Passages were whole paragraphs, tables with context, contract clauses or abstracts with titles. We excluded units longer than 256 tokens under the all-MiniLM-L6-v2 tokenizer. Accepted passages were kept intact, and the 512-token models received at most 459 tokens per passage. This benchmark therefore evaluates short passages.
A frontier LLMs drafted the questions from source passages. A team member verified 20% of the questions (40 of 200). LLMs also judged BM25’s top 20 passages per question, adding 25 relevant passages to the original 200 source passages.
Retrieval and statistical analysis
Retrieval compared each query with every passage in its domain. We used cosine similarity for 21 models and dot product for MultiQA MPNet Base Dot v1. BGE-M3 used its dense vectors.7 Query and document inputs used each model’s configured retrieval roles or prefixes.
Both metrics are averages across questions. We calculated 95% CIs with 10,000 bootstrap samples, resampling 50 questions within each domain. Pairwise comparisons used paired sign-flip tests with Holm correction across all 231 model pairs for each metric and scope. We called differences significant at a corrected p-value below 0.05.
Tested model IDs, dimensions and input limits
The original 22 models ran through OpenRouter, with one fixed backend provider per model and fallbacks disabled. The three Cohere models ran through the Cohere API using native document and query roles (search_document and search_query). The table records the API identifiers, output dimensions and input limits used, including conservative bounds.
API identifiers do not guarantee an immutable model snapshot. Tokenizer revisions were pinned where available. Hosted model weights were not independently verified. Gemini and Cohere input lengths were checked using conservative estimates. The short passages did not test long-context retrieval.
Add as preferred sourceScope and limitations
The questions form a BM25-selected development set. Results on ordinary production queries may differ. Relevance judgments were limited to the source passages and BM25’s top 20. Across all 25 models, 87% of distinct question-passage pairs retrieved in the top ten were unjudged and received no relevance credit.
Each model was run once. The CIs reflect question sampling and exclude variation between API runs or uncertainty from incomplete labels. Training-data contamination was not tested.
This evaluation covers text retrieval. Generated-answer accuracy, hybrid retrieval, reranking, latency and serving cost were outside its scope. The reranker benchmark, multilingual embedding comparison and multimodal embedding comparison cover separate workloads.
Conclusion
Voyage 4 Large recorded the highest nDCG@10, with no significant gap to Cohere Embed 5 Pro or Fast. Both Cohere models scored significantly above Embed v4.0. Gemini Embedding 2 led recall, and PPLX Embed v1 4B led the open-weight group.
Use the domain results to choose candidates, then compare them on representative queries with reviewed relevance labels. Measure latency and end-to-end serving cost separately.
Further reading
- Open-source embedding model benchmark
- Multilingual embedding models for RAG
- Multimodal embedding models
- Open-source vector database benchmark
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Embedding Models: 25 Models Benchmarked for Retrieval}},
year = {2026},
month = oct,
howpublished = {\url{https://aimultiple.com/embedding-models}},
note = {AIMultiple. Retrieved October 1, 2026}
}Results and timestamps of 180 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 8 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
5 updatesAdded Top-1 hit metric definition to the embedding benchmark methodology.
Replaced the benchmark with 15 models and a BM25 baseline tested on legal, customer support and healthcare corpora.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.