Open Source Embedding Models: 11 Models Benchmarked for Retrieval
We benchmarked 11 open-weight embedding models, 200 questions and 341,040 text passages across four corpora. Finance, legal contract, technical support and medical questions each searched their own domain's corpus.
nDCG@10 scores the placement of recorded relevant passages within the first ten search results. A score of 1 represents the ideal ranking for the recorded positives. Passages nearer the top receive more weight.
PPLX Embed v1 4B led the comparison at 0.741 nDCG@10, followed by PPLX Embed v1 0.6B at 0.698. Multilingual E5 Large ranked third at 0.628. The small gap between BGE Large and E5 Large is hidden by three-decimal rounding. Their unrounded scores determine the order.
Questions were selected to challenge BM25, which scored zero on this set. We report that baseline alongside the selection procedure. Performance on ordinary search traffic remains untested. Inference was performed through hosted endpoints.
Model | nDCG@10 | Recall@10 |
|---|---|---|
PPLX Embed v1 4B | 0.741 | 90.46% |
PPLX Embed v1 0.6B | 0.698 | 86.71% |
Multilingual E5 Large | 0.628 | 79.38% |
BGE Large EN v1.5 | 0.624 | 78.21% |
E5 Large v2 | 0.624 | 78.88% |
BGE-M3 (dense) | 0.614 | 79.50% |
GTE Large | 0.612 | 77.71% |
All-MPNet Base v2 | 0.608 | 77.33% |
GTE Base | 0.596 | 76.21% |
E5 Base v2 | 0.586 | 75.21% |
Recall@10 is the average fraction of each question’s recorded relevant passages found in the first ten results. It measures passage coverage and does not score whether a generated answer is correct.
Retrieval findings within the open-weight models
PPLX 4B led overall and in three domains
PPLX Embed v1 4B had the highest nDCG@10 among these eleven models on CUAD, TechQA and MedRAG. Its corresponding scores were 0.653, 0.762 and 0.954. The 0.6B variant ranked second overall, with 86.71% Recall@10 against the 4B variant’s 90.46%.
Moving from the 0.6B model to the 4B model increased overall nDCG by 0.043 points and recall by 3.75 percentage points. Local hardware requirements and inference speed were outside the test.
BGE Large exceeded PPLX 4B on finance
BGE Large EN v1.5 scored 0.604 nDCG@10 on finance, compared with PPLX 4B at 0.596 and PPLX 0.6B at 0.563. BGE Large therefore led the finance subset despite placing fourth overall.
That advantage did not extend to technical support. BGE Large scored 0.544 on TechQA, compared with PPLX 4B’s 0.762. A single overall ranking conceals this difference between workloads. The finance lead is also small enough that the absence of significance testing matters.
E5 Base exceeded E5 Large on finance
E5 Large v2 reached 0.624 overall, compared with E5 Base v2 at 0.586. On finance, however, E5 Base scored 0.553 and E5 Large scored 0.471. E5 Large’s overall improvement came from stronger results in the other three domains.
Multilingual E5 Large scored 0.628 overall and 0.675 on TechQA, placing behind the two PPLX variants on that technical-support subset. We tested English retrieval. Performance across languages requires separate evidence. A separate multilingual embedding comparison addresses that scope.
GTE Large recovered every recorded MedRAG positive within ten results
GTE Large reached 100% Recall@10 on MedRAG, while PPLX 4B reached 99%. Their nDCG@10 scores were 0.894 and 0.954, respectively. GTE Large recovered all the recorded positives, but PPLX 4B placed them higher on average according to the ranking metric.
The labels cover a limited set of positive passages. Other sufficient passages may be unjudged, and generated medical answers were outside the evaluation.
MultiQA trailed All-MPNet on the combined workload
MultiQA MPNet Base Dot v1 scored 0.574 nDCG@10 and 75.21% Recall@10. All-MPNet Base v2 scored 0.608 and 77.33%, respectively. MultiQA is included in both the overall and domain tables with the same completed-run requirements as every other entry.
We used dot-product similarity for MultiQA, consistent with its model card.1
Domain comparisons for the eleven models
Domain-level nDCG@10. The Finance, CUAD, TechQA and MedRAG views retain all eleven models and sort them by the selected domain’s score.
BGE-M3 reached 0.616 on CUAD and 0.743 on MedRAG. Those figures describe the dense-vector configuration used here. BGE-M3 also supports sparse and multi-vector retrieval. Those modes and hybrid combinations were outside this experiment.2
All four domains carry equal weight in the overall result. A model’s strong MedRAG performance can offset weaker finance retrieval in that average, even though the financial and medical use cases require different evidence.
Embedding representations and deployment choices
Embedding retrieval encodes passages and queries into compatible vectors, then compares those vectors to select candidate passages. Documents can be encoded before search. Each incoming query needs its own representation. Model-specific query and document prompts or roles should follow the model’s retrieval configuration.3
Storage and search behavior also depend on the vector database and its index configuration. Model choice, vector precision, index settings and retrieval depth should remain separate decisions when planning a deployment. The broader embedding model benchmark places these eleven entries alongside the commercial API models on the same questions.
Open source embedding models benchmark methodology
Each of the 200 frozen questions searched its own domain’s complete corpus. The corpora contained 50,657 finance passages, 71,656 CUAD passages, 190,658 TechQA passages and 28,069 MedRAG passages, totaling 341,040. There were 50 questions per domain.
We embedded each text passage as a separate retrieval unit. These chunks could contain several sentences, and a single source document could supply multiple passages. Finance and TechQA used complete paragraphs or tables with context. CUAD used contract passages. MedRAG retained each complete abstract together with its original title.
Relevance labels covered 225 unique positive question and passage pairs. These were derived from the selected answer-bearing passages and additional sufficient passages identified during top-twenty review. Query encoders received the original question, without the answer or supporting evidence. We used no query expansion, reranker or BM25 candidate prefilter.
We applied each model’s required formatting, query/document roles and similarity function. Saved vector and ranking replay checks, full-corpus completion and separate metric acceptance were required before adding results. MultiQA used dot product. The other ten open-weight entries used cosine similarity in their tested contracts.
Both metrics were calculated for each question and then averaged. Because the domains contain equal question counts, the overall score is also the equally weighted mean of the four domain scores. Corpus size does not change that weighting.
Limits of the comparison
We used a development workload selected for difficult BM25 cases. Performance on a blind production sample remains untested. The BM25 zero should not be used to calculate a relative uplift or to generalize about the value of lexical retrieval. The question-authoring workflow had no independent final approval panel.
English text retrieval results also do not establish multilingual, image or audio retrieval performance. Multimodal embedding models require a different evaluation.
Every included run passed completion and acceptance checks. Local throughput, GPU memory, quantization effects, energy use and serving cost require separate measurements with controlled hardware and execution settings.
Conclusion
PPLX Embed v1 4B and 0.6B provide the two highest overall quality reference points in this group. BGE Large’s finance result and GTE Large’s MedRAG recall show why domain-specific scores remain useful alongside the aggregate.
Applying these results to a deployment requires testing its query distribution with the same model-specific formatting and similarity functions. Retrieval quality and hardware measurements are both needed to compare deployment options.
Add as preferred sourceFurther reading
- Embedding model benchmark
- Multilingual embedding models for RAG
- Open-source vector database benchmark
- Vector database sizing and selection
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{Open Source Embedding Models: 11 Models Benchmarked for Retrieval}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/open-source-embedding-models}},
note = {AIMultiple. Retrieved September 25, 2026}
}Results and timestamps of 50 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 3 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
4 updatesReplaced the 16-model Amazon reviews Top-K benchmark with a 14-model CUAD, TechQA and MedRAG nDCG@3 and cost benchmark.
Added License & commercial use to the Open source embedding models benchmark overview.
Expanded the benchmark to include five additional open-source models.
Updated hardware, batch size, and precision in the Evaluation setup.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.