Vector Database Benchmark: 7 Open-Source Engines for RAG

We benchmarked seven open-source, self-hosted vector databases as the retrieval layer of a RAG pipeline, each run one at a time on identical bge-m3 embeddings and real medical and technical queries, so the database index was the sole variable. The workload spanned MedRAG-50k, TechQA-28k, and a 2.25M-vector corpus across eight dimensions, from accuracy and retrieval quality through speed, memory, filtered and hybrid search, build cost, and live churn.

Single-thread speed

Each engine is evaluated at the same recall level, the shared operating point where its search parameter (either ef or nprobe) is adjusted until it reaches a Recall@10 of 0.95, allowing for a fair comparison of speed numbers.

On a single client thread, Redis serves 764 QPS on MedRAG-50k at 559 MB peak RAM, the highest throughput and the lowest memory in the set. The single-thread order is Redis 764, Qdrant 377, Milvus 342, Weaviate 341, pgvector 257, Chroma 197, LanceDB 70, a 10x spread from top to bottom. LanceDB is the on-disk outlier, trading RAM for latency at a p95 near 22 ms. Redis stays fastest and LanceDB slowest on TechQA-28k (651 to 81) and at 2.25M (495 to 28), though the middle reshuffles at scale, where Milvus falls from third to sixth.

For RAG, tail latency drives the user experience more than raw throughput. The per-query latency, pooled over 100 runs of 154 queries at matched recall, tracks the throughput order.

Redis ran with persistence off (no RDB snapshot or append-only file), so its speed and memory figures are for a volatile, no-durability configuration.

Throughput under concurrency

Single-thread QPS answers how fast one query is. The production question is throughput under many concurrent clients, and the answer depends on how the client is built, not on the database alone.

A closed-loop client can be driven two ways, and both are common in Python RAG apps. One async or threaded process deserializes every concurrent result (gRPC/protobuf or RESP) on a single GIL, so the client, not the server, caps throughput. Many worker processes (the gunicorn -w N pattern) each get their own GIL, exposing far more of the server’s capacity. We measured both on one 32-vCPU box at a fixed ef=128, and recall-verified every point.

As request concurrency rises from 1 to 512 across up to 32 worker processes, the throughput lines cross. Redis starts highest at one request and ends lowest, while Weaviate climbs from the bottom of the pack to a plateau of 8330 QPS at 512 requests, passing 7,114 on the way at 32. On a single async process the ranking instead tracks Python-client parse efficiency, since Redis’s RESP is the lightest to deserialize and loses the least to the GIL.

Each engine’s single-process peak and 32-process peak are the ceilings across the whole 1-to-512 sweep, not the throughput at any one request count.

Redis holds the lowest single-query latency in the set (about 1.6 ms), and its single-threaded search core saturates around 1642 QPS at 32 concurrent requests, then anti-scales beyond that (RediSearch’s WORKERS query-thread pool is off by default). Weaviate and Milvus (multi-threaded servers) and pgvector (an independent Postgres backend per connection) scale across the 32 cores. pgvector has the lowest single-process peak (536) and the second-highest 32-process peak (4832). Under a p99-below-100ms budget the multi-process order is Weaviate 8290, pgvector 4828, Milvus 4725, Qdrant 1737, Redis 1642.

Qdrant’s heavier gRPC parse makes its co-located number client-bound rather than a server limit, since the process count alone moves it from 455 QPS on one process to 1859 on 32. Chroma and LanceDB were held out of the multi-process run. Chroma bakes ef at collection-create and anti-scales, with a p99 of 13 seconds at 512 requests, and LanceDB is embedded, so the network-client story does not apply.

Memory footprint

At 50k vectors, peak RAM runs from 559 MB (Redis, in-RAM, persistence off) to 3,300 MB (Milvus), with Weaviate at 1,201, LanceDB at 1,574, Chroma at 1,800, and pgvector at 2,024. The ranking changes at 2.25M, where the in-memory-versus-disk trade opens up.

Peak RAM at 2.25M, across the five in-memory engines, runs from 17.0 GB (Milvus) to 62.4 GB (Chroma), a 3.7x range, or 7.5 GB to 27.7 GB per million vectors. Milvus stays leanest because it offloads the index to disk (DiskANN-style) while the all-RAM HNSW engines grow fastest, so Milvus, the heaviest at 50k (3,300 MB), is the leanest at 2.25M (17.0 GB), with Chroma the heaviest at 62.4 GB.

The two on-disk engines sit outside this RAM comparison because their footprint is disk rather than RAM. pgvector holds an 18.4 GB on-disk index and LanceDB a 12.0 GB one at 2.25M, and their serving RAM is a page cache over that index rather than the full working set, which was not measured. One caveat sits on the in-memory figures too. They are a build-and-serve high-water mark, not a serving-only footprint, so a serving-only re-measure is an open refinement.

Freshness, writes, and deletes under live churn

The other seven dimensions measure a static bulk-load then read corpus. Real RAG knowledge bases add, update, and delete documents continuously while the index merges in the background. This churn workload ran on MedRAG-50k on a separate 8-vCPU box, so its throughput is internal to this section and not comparable to the 32-core speed numbers. The correctness signals (recall and tombstone checks) are hardware-independent.

Freshness (read-after-write). At the consistency setting each engine was configured with, every one is read-after-write. A newly written document is searchable immediately, with zero of about 150 writes not visible. The write-ack to search-visible latency is 2 ms p50 for Redis, 4 ms for Weaviate, 9 ms for Qdrant, 12 ms for Milvus, 14 ms for pgvector, 31 ms for LanceDB, and 41 ms for Chroma. Milvus finds the new row through a strong-consistency scan of the not-yet-sealed growing segment (the vector enters the persistent HNSW graph later), so its 12 ms is growing-segment searchability, a valid user-facing read-after-write.

Writes and reads under mixed load. With one writer targeting 150 writes/s and six readers querying for 60 seconds, read recall held between 0.96 and 1.00 for every engine, and no index degraded under concurrent writes. Single-row write throughput separates the engines by 57x, and the order is close to the reverse of the static build ranking.

LanceDB pays one copy-on-write commit per row and Chroma one HTTP add per row, which is why the fastest static builders are the slowest under churn. The write throughput here is the achieved rate against a 150/s target under concurrent reads, so the fast engines are capped near 150 rather than showing their peak. Read this as single-row, high-frequency online write behavior, not batch ingestion, which is these engines’ design point and is untested here.

Delete and compaction. Deleting a random 20% of the live set and reinserting 20% new documents, tombstone-leak was zero for every engine. A deleted id never resurfaced in search, and Recall@10 stayed 0.97 to 1.00 after the full cycle. Correctness holds under churn everywhere. The cost differs by engine. pgvector pays a 53-second VACUUM and Milvus an 11-second compaction, while the slow-write engines’ reinsert dominates (Chroma 276 s, LanceDB 1139 s for about 6,000 documents).

Weaviate ran this section at 3,000 initial documents rather than 30,000 (its synchronous bulk ingest is slow on the small box), so its read-QPS is at a smaller corpus. At that scale its corrected freshness is about 5 ms and its write rate about 102/s, which places both mid-pack.

Metrics explained

ANN Recall@10 is the fraction of the 10 true nearest vectors (per an exact brute-force kNN oracle over the same embeddings the DB indexed) that the approximate index returned. It isolates the database (index type, ef/nprobe, implementation), not the embedding.

nDCG@10 is a position-weighted 0-to-1 score of whether the human-labelled correct document lands near the top of the result list. It measures end-to-end retrieval relevance, which is largely a property of the embedding model, held constant here across every engine.

Δ (delta bridge) is the oracle’s nDCG@10 minus the database’s nDCG@10 at a given search setting. It converts each engine’s approximation error into the answer quality that its speed costs, reportable because we hold both an oracle and human labels.

Findings from the vector database benchmark

The seven engines tie on retrieval accuracy

At Recall@10 = 0.95 on MedRAG-50k, nDCG@10 lands between 0.803 (pgvector) and 0.817 (LanceDB), a spread of 0.014. The delta-to-oracle bridge runs 0.009 (LanceDB) to 0.023 (pgvector), so the database’s approximation costs at most 0.023 nDCG points of answer quality.

Under this embedding, corpus, k=10, and 0.95 operating point, database choice moves nDCG@10 by at most 0.014, against the 10x spread in single-thread throughput and the 3.7x spread in peak memory shown above. Index choice starts to move quality at Recall@10 = 0.99 or higher, at larger k, at far larger corpora, or with quantized indexes.

Retrieval reliability varies more by what a query asks than by which database answers it. Over the 154 MedRAG queries at the exact-kNN oracle, factoid lookups score 0.888 nDCG@10, conditional questions 0.836, comparative questions 0.779, and clause-existence questions 0.763. The 0.125 gap between factoid and clause-existence is present identically across all seven engines.

Metadata filters cost throughput, not recall

With a metadata predicate applied, worst-case Recall@10 stays between 0.968 (Redis) and 1.00 (Weaviate, pgvector, LanceDB) across a grid of selectivity (1/5/20/50%) and predicate correlation (scattered and clustered). The separation is in filtered throughput.

The separation is throughput. Chroma (11-19 QPS) and pgvector (10-56 QPS) hold recall at 20 to 40x lower filtered throughput than the rest. Milvus keeps the highest worst-case recall (0.984) with 268 to 732 QPS across selectivity, while Redis runs fastest at low selectivity (1,374 QPS at 1%) at a lower recall floor (0.968). At 1-5% selectivity, a recall of 1.00 is largely the query planner’s exact full-scan branch below each engine’s full-scan threshold, not filterable-HNSW, disclosed per engine.

*LanceDB’s filtered recall was first measured at 0.24 on clustered filters, then retracted. The sweep varied ef, but LanceDB’s recall knob is nprobes, so it ran at a default of about 9% of partitions. Re-measured at proper nprobes, clustered Recall@10 recovers from 0.24 to 0.76 (nprobes=128) to about 1.00 (nprobes=256), at 3 to 5x lower QPS. LanceDB holds filtered recall, slowly. A recall-matched, same-host filtered-throughput re-measure is still pending, so LanceDB’s filtered QPS is not listed.

Hybrid search adds up to 0.067 nDCG

Adding a standard BM25 keyword arm and fusing it with the dense arm through reciprocal rank fusion (RRF, k=60) lifts nDCG@10 by 0.030 to 0.067 for every engine with a keyword arm. On MedRAG the dense and BM25 arms are close to equal strength individually (about 0.80 each), so the lift recovers each arm’s mistakes rather than one arm dominating.

The engines with a strong BM25 arm gain the most, and pgvector and Weaviate gain less, tracking their weaker keyword arms (0.740 and 0.782). No engine’s hybrid falls below its dense arm once measured correctly.

What separates the engines is native fusion versus client-side round-trips, from Milvus at 340 QPS native to pgvector at 12 QPS client-side. Four engines fuse natively (Qdrant, Milvus, Weaviate, LanceDB). Redis runs native BM25 and KNN but no server-side fusion at this version, so the client issues two round trips and fuses in Python. pgvector has no fusion API, fuses client-side, and its keyword arm is Postgres ts_rank (term frequency with length normalization, no IDF), which with an OR full-text scan runs at 12 QPS. Chroma self-hosted has no ranked keyword search at all (BM25 is a Chroma Cloud feature), so it has no hybrid row.

Index build time spans 13x

At 50k vectors, building the index runs from 7 seconds (LanceDB) and 11 to 13 (Weaviate, Milvus) up to 88 seconds (pgvector). At 2.25M the spread reaches 13x. LanceDB and Milvus finish in 7 to 8 minutes, against 44 minutes for Redis and 92 for pgvector. Normalized to the box rate, build cost runs from €0.04 per 1M vectors (LanceDB) and €0.05 (Milvus) to €0.58 (pgvector). Build is a one-time cost, paid again only when the corpus is re-indexed.

Retrieval quality at 2.25M vectors

To test quality at scale while keeping real human labels, we built a 2.25M-vector corpus. It combines the 50k labelled MedRAG documents with 2.2M pubmed distractors, all in the same bge-m3 space, integrity-checked so no distractor is a mislabelled target. Standard billion-scale ANN datasets carry no human relevance labels, so they cannot show what happens next.

Geometric ANN recall holds at scale. Every engine still reaches Recall@10 above 0.973 at 2.25M (Qdrant and Milvus at 0.999), so the index finds the true nearest vectors. Semantic answer quality does not hold. nDCG@10 falls from about 0.81 at 50k to about 0.56 at 2.25M for every engine, because the correct document is buried among 2.2M distractors.

The collapse is not a database effect. It hits the exact-kNN oracle identically (oracle nDCG@10 at 2.25M is 0.572, and every engine sits at that ceiling around 0.55 to 0.57). Whether the cause is the embedding’s ceiling, corpus ambiguity, or single-positive labels that miss now-relevant near-duplicates is not adjudicated here. Separating those causes needs human re-labelling of the newly top-ranked documents. The reportable fact is that scaling the corpus 45x erased about a third of the answer quality (nDCG@10 0.81 to 0.56) while leaving every ANN index reporting near-perfect recall, an effect a geometric-only benchmark would miss.

Durability, HA, and security by engine

Speed, memory, and recall are the measured axes. For a production RAG deployment the non-measured axes (durability, high availability, access control) also separate these engines. This capability inventory is doc-grounded from each engine’s official documentation at the benchmarked open-source edition. It is a capability inventory, not a chaos test. Actual failover time, crash-recovery data-loss window, and tenant isolation under load are not measured here.

Among the seven, pgvector alone offers point-in-time recovery (through Postgres WAL archiving) and row-level security. Qdrant, Milvus, and Weaviate ship replication and RBAC in their open-source builds, with the caveat that Milvus’s high availability lives in the heavier distributed mode, not the standalone build measured here. None of the seven encrypts on-disk data natively. All rely on disk or volume encryption at the infrastructure layer.

For multi-tenant RAG, where a filter doubles as an access-control boundary, Qdrant, Milvus, Weaviate, and pgvector can enforce tenant isolation inside the database. Chroma and embedded LanceDB push that to the application. A correct pre-filter under-returns, a completeness risk, rather than leaking another tenant’s documents. A cross-tenant leak would need a buggy post-filter, which none of these engines used.

How approximate nearest-neighbor search works

A vector database stores one high-dimensional vector per document and, given a query vector, returns the k nearest by a distance metric (here cosine over L2-normalized vectors). Exact nearest-neighbor search compares the query against every stored vector, which is correct but scales linearly with corpus size. At 50k vectors that is fast. At millions it is too slow to serve.

Approximate nearest-neighbor (ANN) indexes give up some accuracy to search a fraction of the stored vectors. The dominant structure across these engines is HNSW, a layered proximity graph that walks from a coarse entry point down to a dense neighborhood, visiting a bounded number of candidates set by a search knob (ef for HNSW, nprobe for IVF). A larger ef visits more candidates, raising recall and lowering QPS. A smaller ef does the reverse.

Because that knob trades recall against speed continuously, comparing two engines at one fixed setting is misleading, since one may be spending more accuracy for its speed. The fair comparison sweeps the knob, draws each engine’s recall-versus-QPS curve, and reads them all at the same recall. That matched-recall point (Recall@10 = 0.95 here) is where the QPS, latency, and memory numbers in this benchmark are taken.

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

Benchmark methodology

Engines. Qdrant v1.18.1, Milvus v2.6.0, Weaviate 1.38.0, pgvector 0.8.x (Postgres 17), Chroma 1.5.0, Redis/RediSearch 8.2, and LanceDB 0.34.0, at pinned Docker images. Redis ran with persistence off (no RDB snapshot or append-only file).

Hardware. One Hetzner CCX53 (32 dedicated vCPU, 128 GB RAM, NVMe, nbg1) for accuracy, speed, memory, filtered, and build, one container up at a time, a fresh container per index build, and a single-thread client co-located. The live-churn dimension ran on a CCX33 (8 vCPU, 32 GB).

Embedding. bge-m3, 1024-dim, cosine on L2-normalized float32, k=10, seed 42, byte-identical across engines. Queries carry single-positive human labels (target_doc_id).

Corpora. MedRAG-50k (154 queries) and TechQA-28k (151 queries) for retrieval quality, and a 2.25M-vector corpus (50k labelled plus 2.2M distractors) for the scale tier. The hybrid dimension was re-run on local Docker, so its QPS is comparable among hybrid rows, not to the box numbers. Its nDCG and Δ are hardware-independent.

Statistics. 100+ runs per measurement, IQR outlier trim at 3.0x, perf_counter_ns timing. Query pools of 154 to 246 cap tail reporting at p95, occasionally p99. p99.9 is undefined at these counts.

Dual ground truth. A vector database has two independent ground truths, and this benchmark scores both. The geometric ground truth (ANN Recall@10 against the exact brute-force oracle over the same vectors) asks whether the index returned the true nearest vectors, isolating the database. The semantic ground truth (nDCG@10, MRR, Hit@k against human labels) asks whether the returned documents are relevant, which is largely the embedding, held constant. Because the embedding is frozen, ranking engines on raw semantic scores would rank the embedding. The database’s power shows as speed and memory at matched ANN recall, and the Δ bridge translates its approximation loss into answer quality.

Correctness anchor. Each per-engine recipe (filter and hybrid) was taken from official docs at the exact version and adversarially verified before coding. The hybrid dimension carries an independent BM25 answer key (two references agree at nDCG about 0.81), so any engine scoring far from it signals a harness bug, not a finding. That guard flagged three first-pass measurement errors, all in our harness (an async-index timing artifact on Weaviate, a wrong Postgres ranking function, and a SQL fusion bug that made HNSW ignore its ef setting), which we fixed by re-measuring all seven engines in one consistent environment, where the five already-correct engines reproduced their numbers exactly. The concurrency and churn dimensions carry the same anchor (recall and tombstone re-checks), so no reported number is fast-but-wrong.

Statistical significance

Each reported number is a bootstrap point estimate over 100+ runs, with a 95% confidence interval from 1,000 resamples, and a difference counts as real when the two intervals do not overlap.

The hybrid lift is the close call, so its intervals are the deciding evidence. Four engines clear zero and two do not:

Retrieval accuracy is the same rule read the other way. The seven engines’ nDCG@10 fall within overlapping intervals across a 0.014 spread, so none wins, which is what the tie means. The speed and memory orderings sit far outside that uncertainty, with gaps (10x single-thread throughput, 3.7x peak memory) that dwarf the bootstrap noise.

Engines tested

Limitations

Redis ran with persistence off, so an AOF fsync configuration would change its write latency and footprint from the volatile figures reported here.

Human labels are single-positive (one correct document per query), with no graded relevance or inter-annotator agreement, so nDCG@10, MRR@10, and Hit@10 carry near-identical signal on this data. The embedding is held constant by design. Embedding families, dimensions, and multilingual behavior are a separate benchmark. This run isolates the retrieval layer. The reranker (cross-encoder) and LLM answer-generation stages of a RAG pipeline are out of scope. Managed cloud engines (Pinecone, Zilliz, and others) are a later phase. The durability and security comparison is doc-grounded, not chaos-tested.

Further reading

• Embedding models benchmark
• Reranker benchmark
• Vector database calculator
• Open-source embedding models
• Multilingual embedding models
• Multimodal embeddings
• Agentic RAG frameworks

Cite this research

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{sari2026,
  author = {Sarı, Ekrem},
  title  = {{Vector Database Benchmark: 7 Open-Source Engines for RAG}},
  year   = {2026},
  month  = jul,
  howpublished    = {\url{https://aimultiple.com/open-source-vector-databases}},
  note   = {AIMultiple. Retrieved July 17, 2026}
}

Ekrem Sarı

AI Researcher

Follow On

Ekrem is an AI Researcher and Data Analyst at AIMultiple. He designs and runs hands-on benchmarks for AI and LLM systems.

View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment