Open Source Embedding Models: 11 Models Benchmarked for Retrieval
We benchmarked 11 open-weight embedding models, 200 questions and 341,040 text passages across four corpora. Finance, legal contract, technical support and medical questions each searched their own domain's corpus.
nDCG@10 scores the placement of recorded relevant passages within the first ten search results. A score of 1 represents the ideal ranking for the recorded positives. Passages nearer the top receive more weight.
PPLX Embed v1 4B led the comparison at 0.741 nDCG@10, followed by PPLX Embed v1 0.6B at 0.698. Multilingual E5 Large ranked third at 0.628. The small gap between BGE Large and E5 Large is hidden by three-decimal rounding. Their unrounded scores determine the order.
Model | nDCG@10 | 95% CI | Recall@10 |
|---|---|---|---|
PPLX Embed v1 4B | 0.741 | 0.700–0.782 | 90.46% |
PPLX Embed v1 0.6B | 0.698 | 0.651–0.744 | 86.71% |
Multilingual E5 Large | 0.628 | 0.578–0.680 | 79.38% |
BGE Large EN v1.5 | 0.624 | 0.572–0.676 | 78.21% |
E5 Large v2 | 0.624 | 0.572–0.675 | 78.88% |
BGE-M3 (dense) | 0.614 | 0.562–0.666 | 79.50% |
GTE Large | 0.612 | 0.563–0.660 | 77.71% |
All-MPNet Base v2 | 0.608 | 0.556–0.661 | 77.33% |
GTE Base | 0.596 | 0.547–0.645 | 76.21% |
E5 Base v2 | 0.586 | 0.534–0.640 | 75.21% |
Recall@10 is the average fraction of each question’s recorded relevant passages found in the first ten results. It measures passage coverage and does not score whether a generated answer is correct.
The 95% confidence intervals (CIs) estimate uncertainty in average nDCG@10 from question sampling. Wider intervals indicate less precise estimates.
Questions were selected to challenge BM25, which scored zero on this set. We report that baseline alongside the selection procedure. Performance on ordinary search traffic remains untested. Inference was performed through hosted endpoints.
Retrieval findings within the open-weight models
PPLX Embed v1 4B recorded the highest nDCG@10 at 0.741
PPLX Embed v1 0.6B followed at 0.698. The 4B model scored significantly above nine of the other ten open-weight models after correcting for multiple comparisons. Its gap to 0.6B was not significant. Their Recall@10 scores were 90.46% and 86.71%, also without a significant difference. These tests do not establish equal performance.
BGE Large recorded the highest finance score at 0.604
PPLX 4B scored 0.596 on the same 50 finance questions. That difference was not significant after correction. BGE Large ranked fourth overall, illustrating how the observed order changes across domains.
GTE Large retrieved every recorded MedRAG positive within ten results
GTE Large reached 100% Recall@10 across the 50 medical questions. Its nDCG@10 was 0.894, compared with PPLX 4B’s 0.954. Recovering the recorded positives and ranking them first are separate outcomes. These scores concern passage retrieval and do not measure medical-answer accuracy.
Domain comparisons for the eleven models
Domain-level nDCG@10. The Finance, CUAD, TechQA and MedRAG views retain all eleven models and sort them by the selected domain’s score.
BGE-M3 reached 0.616 on CUAD and 0.743 on MedRAG. Those figures describe the dense-vector configuration used here. BGE-M3 also supports sparse and multi-vector retrieval. Those modes and hybrid combinations were outside this experiment.1
All four domains carry equal weight in the overall result. A model’s strong MedRAG performance can offset weaker finance retrieval in that average, even though the financial and medical use cases require different evidence.
Embedding representations and deployment choices
Embedding retrieval encodes passages and queries into compatible vectors, then compares those vectors to select candidate passages. Documents can be encoded before search. Each incoming query needs its own representation. Model-specific query and document prompts or roles should follow the model’s retrieval configuration.2
Storage and search behavior also depend on the vector database and its index configuration. Model choice, vector precision, index settings and retrieval depth should remain separate decisions when planning a deployment. The broader embedding model benchmark places these eleven entries alongside the commercial API models on the same questions.
Open source embedding models benchmark methodology
Every model searched the same passages with 200 frozen questions and 225 recorded relevant question-passage pairs. Questions and relevance labels were fixed before comparing embedding scores. Query inputs contained the original question alone.
Finance passages came from FinQA annual reports and legal passages from CUAD v1 contracts.34 Technical support used TechQA technotes, and medical literature used a sample of MedRAG PubMed abstracts.56
Passages were whole paragraphs, tables with context, contract clauses or abstracts with titles. We excluded units longer than 256 tokens under the all-MiniLM-L6-v2 tokenizer and kept accepted passages intact. The 512-token models received at most 459 tokens per passage, so this evaluation covers short passages.
Frontier LLMs drafted the questions from source passages. A team member verified 20% of the questions (40 of 200). The LLMs also judged BM25’s top 20 passages per question, adding 25 relevant passages to the original 200 source passages.
Retrieval and statistical analysis
Each query was compared with every passage in its domain. MultiQA used dot product, consistent with its model card.7 The other ten models used cosine similarity. BGE-M3 used its dense vectors.1 Inputs used each model’s configured retrieval roles or prefixes.
Both metrics are averages across questions. We calculated 95% CIs with 10,000 bootstrap samples, resampling 50 questions within each domain. Significance uses the full benchmark’s paired sign-flip tests, with Holm correction across all 231 pairs among its 22 models for each metric and scope. Differences are significant at a corrected p-value below 0.05.
Tested model IDs, dimensions and input limits
The runs took place through OpenRouter, with one fixed backend provider per model and fallbacks disabled. These are the API identifiers, output dimensions and input limits used in those runs.
API identifiers do not guarantee an immutable model snapshot. Tokenizer revisions were pinned, and hosted model weights were not independently verified. The input limits describe the benchmark configuration. The short passages did not test long-context retrieval.
Limits of the comparison
The questions form a BM25-selected development set. Results on ordinary production queries may differ. Relevance judgments were limited to the source passages and BM25’s top 20. Across these eleven models, approximately 85% of distinct question-passage pairs retrieved in the top ten were unjudged and received no relevance credit.
Each model was run once. The CIs reflect question sampling and exclude variation between API runs or uncertainty from incomplete labels. Training-data contamination was not tested.
The evaluation covers English text retrieval. Multilingual embedding workloads and multimodal embedding models need separate evidence. Generated-answer accuracy, hybrid retrieval, reranking, GPU memory, local throughput and serving cost were outside this comparison.
Conclusion
PPLX Embed v1 4B and 0.6B had the highest overall scores among the eleven open-weight models. Their difference was not statistically significant. BGE Large recorded the highest finance score, and GTE Large retrieved all recorded MedRAG positives within ten results.
Before deployment, test the intended query set and measure local serving speed, memory use and cost.
Add as preferred sourceFurther reading
- Embedding model benchmark
- Multilingual embedding models for RAG
- Open-source vector database benchmark
- Vector database sizing and selection
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Open Source Embedding Models: 11 Models Benchmarked for Retrieval}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/open-source-embedding-models}},
note = {AIMultiple. Retrieved September 26, 2026}
}Results and timestamps of 66 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 5 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
4 updatesReplaced the 16-model Amazon reviews Top-K benchmark with a 14-model CUAD, TechQA and MedRAG nDCG@3 and cost benchmark.
Added License & commercial use to the Open source embedding models benchmark overview.
Expanded the benchmark to include five additional open-source models.
Updated hardware, batch size, and precision in the Evaluation setup.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.