Skip to content
ashvardanianPublic

About

Benchmark suite that compares vector search engines against each other on billion-scale datasets, from in-memory HNSW libraries like USearch, FAISS, & cuVS to managed databases like Qdrant, Weaviate, LanceDB, & Redis – built for reproducibility

Topics

Resources

Stars

37 stars

Watchers

2 watching

Forks

Repository files navigation

RetriEval

RetriEval benchmarks thumbnail

RetriEval is a benchmarking suite designed for Billion-scale Vector Search workloads. It's primarily used to benchmark in-process Search Engines on CPUs and GPUs, like USearch, FAISS, and cuVS, but it also reuses similar profiling logic for standalone databases like Qdrant, Weaviate, and Redis. It works with the same plain input format standardized by the BigANN benchmark, aiming for reproducible measurements – with shuffled parallel construction, incremental recall curves, normalized metrics, and machine-readable reports, capturing everything from machine topology to indexing hyper-parameters.

Quickstart

Install the default retri-eval-usearch binary:

cargo install --path .

Fetch the Unum Wiki 1M dataset — ~1.1 GB of vectors, queries, and ground truth — into data/wiki-1m/:

scripts/download.sh wiki-1m

Run a sweep over three quantizations and write JSON reports under results/:

retri-eval-usearch \
    --base-vectors data/wiki-1m/base.1M.fbin \
    --query-vectors data/wiki-1m/query.public.100K.fbin \
    --query-neighbors data/wiki-1m/groundtruth.public.100K.ibin \
    --data-type bf16,f16,i8 \
    --metric inner \
    --output results/

Generate plots from the results:

uv run scripts/plot.py results/ --output plots/

Backends

Search Engines

Backend Parallelism Quantization Metrics
USearch ForkUnion bf16, f16, e5m2, e4m3, e3m2, e2m3, i8, u8, b1 inner, euclidean, angular, hamming, jaccard
FAISS OpenMP f32, f16, bf16, u8, i8, b1 inner, euclidean
cuVS CUDA f32, f16, i8, u8 euclidean, inner, angular
  • USearch: The input is stored in the --data-type quantization, converted on the fly. e3m2 and e2m3 occupy byte slots; b1 is packed.
  • FAISS: Input is always f32. --data-type selects the internal scalar quantizer (SQfp16, SQbf16, SQ8_direct, etc.).
  • cuVS: Input is converted on the host to the --data-type, one of f32, f16, i8 or u8, before CAGRA builds on it. It reuses the shared --connectivity, --expansion-add and --expansion-search names and JSON keys, so its runs plot on the same axes as the HNSW backends. The mapping is an analogy, not an identity: --expansion-add is CAGRA's pre-prune candidate width rather than ef_construction, and --expansion-search its internal top-i list rather than ef_search.

A query walking a proximity graph from an entry point along a thick path, beside a table naming the connectivity, construction and search expansion flags in USearch, FAISS, Qdrant, Weaviate, Redis, SurrealDB and cuVS

retri-eval-usearch --data-type bf16 --metric euclidean ...
retri-eval-faiss --data-type f16 --metric euclidean ...
retri-eval-cuvs --metric euclidean ...

Bit layouts of every --data-type from f32 down to b1, split into sign, exponent, mantissa and integer bits, with each type's largest value, overflow and NaN behavior, and the share of its codes inside the range from minus one to one

Vector Databases

Server-side quantization is managed by the database engine, not the benchmark. Binary quantization is deterministic sign(x) per dim, and scalar quantization is per-dim min/max — neither trains a codebook, so both stay inside the "no learned logic" constraint the rest of the benchmark holds for the native backends. Product quantization is deliberately excluded everywhere.

Backend Client Docker Image Metrics Wire dtype sweep Server-side quantization
Qdrant qdrant-client, gRPC qdrant/qdrant:v1.19.1 ip, l2, cos, manhattan f32, f16, u8 none, binary, scalar
Redis redis, RESP redis:8.10 ip, l2, cos f32, f64, f16, bf16, u8, i8 —
Weaviate reqwest, REST semitechnologies/weaviate:1.39.7 ip, l2, cos f32 only none, binary
LanceDB lancedb, in-process, Arrow — ip, l2, cos f32 only — (exact scan) ¹
Turso turso = 0.8.1, native Rust — l2, cos f32 only — (exact scan)
SurrealDB reqwest, HTTP SQL surrealdb/surrealdb:v3.3.0 l2, cos f32 only — (HNSW)

¹ LanceDB's Rust client — lancedb 0.37 — exposes graph-based search only via IvfHnswFlat / IvfHnswSq / IvfHnswPq, all behind a k-means-trained IVF layer that cannot be disabled. No pure-HNSW variant is offered, so this benchmark leaves LanceDB on plain f32 + L2/IP/Cos until upstream adds one. Hamming is only available on IvfFlat, outside our graph path.

Turso runs in-process with exact f32 scans over an in-memory database, or a file in --data-dir whose benchmark table alone is recreated; it measures exact search, not a libSQL ANN index. The server backends run in Docker containers the binaries manage themselves; docker/README.md covers their lifecycle and per-server caveats.


Each backend is behind its own feature flag. Build only what you need:

cargo build --release --features usearch-backend    # USearch
cargo build --release --features usearch-cuda       # USearch with its CUDA kernels, for --residency vram
cargo build --release --features faiss-backend      # FAISS
cargo build --release --features qdrant-backend     # Qdrant
cargo build --release --features redis-backend      # Redis
cargo build --release --features lancedb-backend    # LanceDB
cargo build --release --features weaviate-backend   # Weaviate
cargo build --release --features cuvs-backend       # cuVS
cargo build --release --no-default-features --features turso-backend # Turso
cargo build --release --no-default-features --features surrealdb-backend # SurrealDB
cargo build --release --no-default-features --features ground-truth # retri-ground-truth

Or combine multiple:

cargo build --release --features usearch-backend,faiss-backend,qdrant-backend

Datasets

The BigANN collections are a good starting point, as most ship pre-computed ground-truth neighbors for recall evaluation. The tables below group the datasets by scale, and each name links to its download and run recipe in data/README.md, which also covers the file format and sharded inputs.

~1M Scale — Development & Testing

Dataset Scalar Type Dimensions Metric Base Size Ground Truth
Unum UForm Wiki f32 256 IP 1 GB 100K queries, yes
Arxiv with E5 f32 768 IP 7 GB 2.3M queries, yes

~10M Scale

Dataset Scalar Type Dimensions Metric Base Size Ground Truth
Meta BIGANN — SIFT u8 128 L2 1.2 GB 10K queries, yes
Microsoft Turing-ANNS f32 100 L2 3.7 GB 100K queries, yes
Microsoft SpaceV i8 100 L2 0.9 GB 30K queries, yes
Yandex Deep f32 96 L2 3.6 GB 10K queries, yes
Cohere Wiki EN b1 1024 Hamming 5.3 GB self-sampled ¹

¹ These sources ship vectors without ground truth, so their retri-download-* binaries sample queries with a fixed seed and compute exact Hamming top-K.

~100M Scale

Dataset Scalar Type Dimensions Metric Base Size Ground Truth
Meta BIGANN — SIFT u8 128 L2 12 GB 10K queries, yes
Microsoft Turing-ANNS f32 100 L2 37 GB 100K queries, yes
Microsoft SpaceV i8 100 L2 9.3 GB 30K queries, yes
Yandex Deep f32 96 L2 36 GB 10K queries, yes
Unum WikiVerse ² f16 128–4096³ Cos/IP 95-505GB pipeline pending
USearchMolecules PubChem MACCS b1 168 Hamming 2.4 GB self-sampled ¹
USearchMolecules PubChem ECFP4 b1 2048 Hamming 29 GB self-sampled ¹

² WikiVerse ships .f16bin, which RetriEval does not read yet. ³ nomic-embed at 768, arctic-embed and Qwen3 at 1024, e5-mistral at 4096; the ColBERT model's 128d per token needs multi-vector support.

~1B Scale

Dataset Scalar Type Dimensions Metric Base Size Ground Truth
Meta BIGANN — SIFT u8 128 L2 119 GB 10K queries, yes
Microsoft Turing-ANNS f32 100 L2 373 GB 100K queries, yes
Microsoft SpaceV i8 100 L2 93 GB 30K queries, yes
Yandex Text-to-Image f32 200 Cos 750 GB 100K queries, yes
Yandex Deep f32 96 L2 358 GB 10K queries, yes
USearchMolecules GDB-13 MACCS b1 168 Hamming 21 GB self-sampled ¹
USearchMolecules GDB-13 ECFP4 b1 2048 Hamming 250 GB self-sampled ¹
USearchMolecules Enamine REAL MACCS b1 168 Hamming 127 GB self-sampled ¹
USearchMolecules Enamine REAL ECFP4 b1 2048 Hamming 1.55 TB self-sampled ¹

Results

These numbers were measured for reference on DGX-B300 and DGX-H100 class nodes, the main drivers of AI neoclouds, and on single-socket AWS Graviton4 and Graviton5 hosts, and mostly guide the design of USearch. They shift with the data, the software versions and the hardware, so treat them as reference points for your own runs, not as fixed rankings.

Each table holds one row per backend, device and data type, so a new run is a new row and nothing else moves:

  • Device: the cores a run used and what they are, like 148× NVIDIA SM103 for the streaming multiprocessors of one B300 or 128× Intel Xeon6 for the physical cores of a dual-socket host. CPU rows count physical cores, not SMT threads: with two threads per core, the 96× and 128× hosts ran 192 and 256 threads.
  • Type: the stored type; every backend took its native input type, with no rescaling.
  • Connectivity: the graph degree, M.
  • Layer 0: the edges each vector keeps in the densest layer: 2×M in the HNSW graphs of USearch 2.25 and FAISS, M in the single-layer graphs of USearch 3.0 and cuVS.
  • Expansions: the construction and search expansion, split by /.
  • 1-r@10: 1-recall@10, the share of queries whose true nearest neighbor lands in the top 10.
  • 10-r@10: K-recall@10, the share of the true top 10 found.
  • Add/s and Search/s: vectors indexed and queries answered per second.
  • Memory: the backend's own peak figure, which src/README.md defines per backend.
  • Build: the wall time to index the whole base.

Every row follows the same methodology:

  • Tune each backend's settings until it reaches 99% 1-recall@10 or more on the dataset.
  • Never let a row build faster than it searches: when Add/s exceeds Search/s, raise the construction expansion and lower the search expansion until search is at least as fast at that recall.
  • Run every command on a 1M prefix first, then on a 10M prefix, before the full file.
  • Match the links each vector keeps: 64 means connectivity 64 in a single-layer graph and M=32 in HNSW.
  • Start every engine from the same expansions, and widen the construction only where a dataset misses 0.99.
  • Hold the 0.99 target on every dataset.
  • Give each engine its best pages and kernels, and name them under the table.
  • Build in 20 steps, and validate the ground truth against the base before the first insert.

Compare the fastest measured configuration meeting the same recall target, with the same recall definition, queries, K, and ground truth. Report build time and memory alongside query throughput, and keep query concurrency, batch size, and cache state comparable.

Two schematic throughput-recall curves, where engine A looks faster at its default setting but engine B answers more queries once both are swept to the same 0.99 recall

Each search reports both recall conventions, because they answer different questions and one of them saturates:

  • recall_at_k is 1-recall@K, FAISS's OneRecallAtRCriterion and the measure USearch reports: any of the top K within the true nearest distance.
  • intersection_at_k is K-recall@K, the share of the top K within the K-th true distance, order-independent, and what ann-benchmarks and cuVS's own harness publish as recall.
  • ndcg_at_k weights true hits by 1 / log2(rank + 1), normalized by ideal gain.
  • Ties are counted by exact distance: the harness recomputes the distance of every true neighbor and every answer from the base vectors, whatever the engine stores, so a neighbor tied with a true one counts whichever key the ground truth kept, within a relative 10⁻⁶ for rounding.
  • Neither 1-recall@K nor K-recall@K generally bounds the other, so the two are never interchangeable.
  • K is --top-k, which defaults to the ground-truth file's width, 10 on the Wiki sets and 100 on the BigANN ones.
  • 1-recall@K stops discriminating as K grows: at K=100 nearly any sane configuration finds the single true neighbor somewhere in the list. intersection_at_k stays informative there, and ndcg_at_k likewise normalizes over the full truth prefix.
  • Pass --top-k 10 for figures comparable across datasets of differing ground-truth width.

One list of ten returned answers scored four ways against the true neighbors: 0 for 1-recall@1, 1 for 1-recall@10, 0.7 for 10-recall@10 and 0.74 for NDCG@10

PubChem MACCS

168-bit binary fingerprints of PubChem molecules under Hamming, with settings calibrated at 10M and reused at 100M; recipes in data/README.md.

PubChem MACCS 10M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 b1 32 64 128 / 64 0.9696 … 36,347 35,767 4.7 GB 5.0m
FAISS 1.12 96× Intel Xeon6 b1 64 128 40 / 16 0.9661 … 95,230 293,795 7.6 GB 1.5m

PubChem MACCS 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 b1 32 64 128 / 64 0.8438 … 35,080 38,432 40.8 GB 54.6m
FAISS 1.12 96× Intel Xeon6 b1 64 128 40 / 16 … … … … ≥ 63 GB killed at 9h
USearch 3.0 148× NVIDIA SM103 b1 32 32 256 / 64 0.9907 0.9906 559,140 1,460,000 18.3 GB 3.4m

PubChem ECFP4

2048-bit ECFP4 fingerprints of PubChem molecules under Hamming, all 115,627,256 of them; recipes in data/README.md.

PubChem ECFP4 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 3.0 148× NVIDIA SM103 b1 32 32 512 / 512 0.9906 … 146,810 285,230 … 13.1m
USearch 3.0 192× Arm Graviton5 b1 32 32 512 / 512 0.9821 0.9038 104,412 57,807 53.6 GB 18.5m
FAISS 1.15.1 192× Arm Graviton5 b1 32 64 40 / 16 0.5623 0.4704 341,635 839,862 93.3 GB 5.6m
USearch 3.0 84× Arm Graviton4 b1 64 64 512 / 512 0.9931 0.9953 41,008 49,048 68.6 GB 47.0m
FAISS 1.15.1 84× Arm Graviton4 b1 32 64 512 / 1024 0.9870 0.9925 27,175 11,750 62.9 GB 1h 11m

Neither Graviton5 row reaches 99%, so both break the methodology above:

  • USearch on Graviton5 levels off at 0.9843 even at a search expansion of 1,024, short of the 0.9906 the B300 reaches at the same 512 / 512, and at that recall it builds faster than it searches.
  • FAISS's binary HNSW ran there at its compiled-in 40 / 16, since FAISS's parameter tuning has no dispatcher for IndexBinaryHNSW.
  • On Graviton4 the harness sets FAISS's expansions directly, and FAISS still levels off at 0.9870, where it builds faster than it searches.
  • 10-recall@10 near 0.90 is tie scoring: Hamming distances tie often, and those rows credited only the exact keys the ground truth kept among tied neighbors.

SIFT

Meta's 128-dimensional u8 descriptors under L2, compared at the setting that first reaches 99% recall@10; recipes in data/README.md.

SIFT 10M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 u8 16 32 128 / 256 0.9938 … 35,405 80,729 4.4 GB 4.8m
FAISS 1.12 96× Intel Xeon6 u8 16 32 128 / 256 0.9952 … 26,374 38,278 5.9 GB 5.6m
cuVS 26.10 148× NVIDIA SM103 u8 64 64 128 / 128 … 0.9966 204,838 1,204,000 … 0.8m
USearch 3.0 148× NVIDIA SM103 u8 32 32 256 / 128 0.9911 0.9780 558,510 1,040,000 2.6 GB 0.3m
FAISS 1.15.1 128× Intel Xeon6 u8 32 64 128 / 96 0.9900 0.9765 115,643 133,919 8.6 GB 1.4m
FAISS 1.15.1 128× Intel Xeon6 f32 32 64 128 / 96 0.9900 0.9765 59,688 61,027 14.4 GB 2.8m

SIFT 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 u8 16 32 128 / 256 0.9833 … 39,831 75,808 53.7 GB 48.7m
FAISS 1.12 96× Intel Xeon6 u8 16 32 128 / 256 … … … … ≥ 46 GB killed at 9h
cuVS 26.10 148× NVIDIA SM103 u8 64 64 128 / 256 … 0.9963 154,799 582,000 … 10.8m
USearch 3.0 148× NVIDIA SM103 u8 32 32 256 / 256 0.9913 0.9810 431,520 604,490 24.8 GB 3.9m
USearch 3.0 192× Arm Graviton5 u8 32 32 512 / 160 0.9925 0.9824 161,952 173,823 33.8 GB 10.3m
FAISS 1.15.1 192× Arm Graviton5 u8 32 64 256 / 128 0.9917 0.9792 84,536 178,600 56.9 GB 19.7m
USearch 3.0 84× Arm Graviton4 u8 64 64 128 / 128 0.9921 0.9775 185,046 208,835 46.7 GB 9.0m
FAISS 1.15.1 84× Arm Graviton4 u8 32 64 256 / 128 0.9912 0.9793 41,917 93,549 42.3 GB 39.8m

SpaceV

Microsoft's 100-dimensional i8 web-search embeddings under L2, compared at the setting that first reaches 99% 1-recall@10; recipes in data/README.md.

SpaceV 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 3.0 84× Arm Graviton4 i8 64 64 512 / 384 0.9934 0.9929 38,713 57,971 45.1 GB 43.1m
FAISS 1.15.1 84× Arm Graviton4 i8 32 64 512 / 640 0.9912 0.9908 14,975 15,773 39.5 GB 1h 51m

Deep

Yandex's 96-dimensional f32 image embeddings under L2, stored as bf16 by both backends and compared at the setting that first reaches 99% 1-recall@10; recipes in data/README.md.

Deep 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 3.0 84× Arm Graviton4 bf16 64 64 256 / 192 0.9934 0.9804 87,964 118,976 53.1 GB 18.9m
FAISS 1.15.1 84× Arm Graviton4 bf16 32 64 256 / 256 0.9929 0.9827 48,503 56,920 48.6 GB 34.4m

Turing-ANNS

Microsoft's 100-dimensional f32 web-search embeddings under L2, compared at the setting that first reaches 99% recall@10 across storage types; recipes in data/README.md.

Turing-ANNS 10M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 f32 48 96 768 / 384 0.9929 … 8,532 12,331 13.0 GB 18.3m
USearch 2.25 96× Intel Xeon6 bf16 48 96 768 / 384 0.9929 … 10,496 16,940 10.9 GB 14.1m
USearch 2.25 96× Intel Xeon6 f16 48 96 768 / 384 0.9929 … 10,969 20,246 10.9 GB 13.5m
USearch 2.25 96× Intel Xeon6 e5m2 48 96 768 / 384 0.9919 … 10,526 20,534 9.8 GB 13.5m
USearch 2.25 96× Intel Xeon6 e4m3 48 96 768 / 384 0.9930 … 7,353 12,106 9.8 GB 19.4m
USearch 2.25 96× Intel Xeon6 e3m2 48 96 768 / 384 0.9728 … 10,398 18,022 9.8 GB 13.3m
USearch 2.25 96× Intel Xeon6 e2m3 48 96 768 / 384 0.7941 … 10,935 21,313 9.8 GB 13.2m
FAISS 1.12 96× Intel Xeon6 f32 48 96 768 / 384 0.9944 … 7,491 16,486 14.1 GB 20.6m
FAISS 1.12 96× Intel Xeon6 bf16 48 96 768 / 384 0.9944 … 3,800 10,391 12.1 GB 39.4m
FAISS 1.12 96× Intel Xeon6 f16 48 96 768 / 384 0.9944 … 2,545 10,032 12.1 GB 1h 1m
USearch 3.0 148× NVIDIA SM103 bf16 32 32 1024 / 1024 0.9908 0.9831 95,870 155,810 3.4 GB 1.7m

Turing-ANNS 100M

Backend Device Type Connectivity Layer 0 Expansions 1-r@10 10-r@10 Add/s Search/s Memory Build
USearch 2.25 96× Intel Xeon6 f32 48 96 768 / 384 0.9929 … 6,646 10,398 139.6 GB 4h 1m
USearch 2.25 96× Intel Xeon6 bf16 48 96 768 / 384 0.9931 … 8,564 14,772 105.2 GB 3h 1m
USearch 2.25 96× Intel Xeon6 f16 48 96 768 / 384 0.9930 … 8,807 15,412 105.2 GB 2h 54m
USearch 2.25 96× Intel Xeon6 e5m2 48 96 768 / 384 0.9924 … 7,368 13,227 88.0 GB 3h 15m
USearch 3.0 148× NVIDIA SM103 bf16 32 32 1024 / 1024 0.9933 0.9868 88,660 156,790 32.1 GB 18.8m

CLI Reference

Each backend is a separate binary, and the manuals below list every flag of every binary with its default. A value spelled <…,...> takes comma-separated values and runs one configuration per combination. Counts take K, M, G or T in powers of 1000, like 10M, and sizes take KB to TB in powers of 1024, like 16GB. A zero connectivity or expansion lets USearch derive it, and the report records the connectivity the index header holds. A bad value prints --flag="value" does not parse, expected … and exits with status 1. A sweep also exits with status 1 when any configuration fails or none runs, while configurations a backend refuses to open are reported as skipped. Every retri-eval-* flag also reads RETRIEVAL_<FLAG> as its default, upper-cased with dashes as underscores, so --top-k reads RETRIEVAL_TOP_K. The flag on the command line overrides the variable, and switches like --self-search read 1, 0, true or false.

Shared by every retri-eval-* binary
retri-eval-*
  --base-vectors <PATH|GLOB>            required
      Base vectors to index: .fbin, .u8bin, .i8bin or .b1bin.
  --base-keys <PATH|GLOB>               generated
      Keys for the base vectors, one per vector: .i32bin, .u64bin or .uuidbin.
  --key-type <TYPE>                     u32
      Keys generated without --base-keys: u32, u64, uuid-v4 or uuid-v7.
  --query-vectors <PATH|GLOB>           required
      Query vectors to search with.
  --query-neighbors <PATH|GLOB>         required
      Ground-truth neighbor rows for those queries: .ibin, or .u64bin past 2^32 base rows.
  --top-k <K>                           ground-truth width
      Neighbors per query, the K every metric is taken at; at most the ground-truth width.
  --insertion-order <ORDER>             shuffled
      shuffled by --seed, or the file's original order.
  --seed <SEED>                         42
      32-bit seed of the shuffled order, or random; rejected with --insertion-order original.
  --steps <N>                           10
      Measurement steps; the base is inserted in this many equal parts.
  --inserts-per-core <N>                2048
      Vectors per add call for each backend thread: --threads on usearch and faiss, one elsewhere.
  --queries-per-core <N>                2048
      Queries per search call for each backend thread: --threads on usearch and faiss, one elsewhere.
  --output <DIR>                        none
      Directory for JSON result files; progress only when unset.
  --memory-limit <SIZE>                 cgroup memory.max, else all RAM
      Memory the engine may use, page cache included, like 16GB; servers get it as their
      container's memory and swap limit, beside the process's CPU affinity as their cpuset.
  --perf-counters                       off
      Record hardware counters per phase; needs the perf-counters feature on Linux.
  --max-base-vectors <N>                all
      Cap on the base vectors used, for calibration on a slice.
  --dims <N,...>                        native
      Matryoshka truncations, each at most the native width, and a multiple of 8 for .b1bin.
  --self-search                         off
      After the last insertion, replay indexed vectors as their own queries.
  --self-search-top-k <K>               10
      Neighbors per self-search query.
  --self-search-sample <N|FRACTION>     all
      Leading base rows to replay, a count like 50000 or a fraction like 0.1; needs --self-search.
Search engines: retri-eval-usearch, retri-eval-faiss & retri-eval-cuvs
retri-eval-usearch, retri-eval-faiss, retri-eval-cuvs
  --index <PATH>                        none
      Persisted index: searched if the path exists, else built and saved; needs a single config.

retri-eval-usearch
  --data-type <TYPE,...>                bf16
      Stored type: bf16, f16, e5m2, e4m3, e3m2, e2m3, i8, u8 or b1.
  --metric <METRIC,...>                 euclidean
      inner, euclidean, angular, hamming or jaccard; ip, l2, cos and tanimoto also parse.
  --connectivity <N,...>                0
      Graph connectivity, 0 to let USearch derive it.
  --expansion-add <N,...>               0
      Construction expansion, 0 to let USearch derive it.
  --expansion-search <N,...>            0
      Search expansion, 0 to let USearch derive it.
  --memory-domains <NODES>              every node the process may use
      Comma-separated NUMA nodes, one shard on each.
  --threads <N,...>                     0
      Threads, 0 for every core in the affinity mask.
  --residency <WHERE>                   ram
      ram, vram, or nvme: built in RAM, saved to --index and searched as a view on the last step.
  --max-refines <N>                     0
      Refinement rounds after the last insert step, stopping at the first that does not improve the graph.
  --nvme-paths <PATHS>                  where --index points
      Devices or directories an NVMe view serves its parts from.
  --devices <ORDINALS>                  one free device
      GPU ordinals, one shard on each.

retri-eval-faiss
  --data-type <TYPE,...>                bf16
      Scalar quantizer over f32 input: f32, f16, bf16, u8, i8 or b1.
  --metric <METRIC,...>                 euclidean
      inner or euclidean, also spelled ip and l2; b1 data always uses Hamming.
  --connectivity <N,...>                32
      HNSW M.
  --expansion-add <N,...>               128
      HNSW construction width.
  --expansion-search <N,...>            64
      HNSW search width.
  --threads <N>                         0
      OpenMP threads, 0 for every core in the affinity mask.

retri-eval-cuvs
  --data-type <TYPE,...>                f32
      f32, f16, i8 or u8, converted on the host before CAGRA builds on it.
  --metric <METRIC,...>                 euclidean
      euclidean, inner or angular, also spelled l2, ip and cos.
  --connectivity <N,...>                32
      CAGRA's output graph degree.
  --expansion-add <N,...>               64
      CAGRA's intermediate graph degree before pruning.
  --expansion-search <N,...>            64
      CAGRA's internal top-i list.
  --search-width <N|auto,...>           auto
      Graph nodes that start each search iteration, or auto for cuVS's choice.
  --min-iterations <N>                  cuVS's choice
      Minimum search iterations.
  --max-iterations <N>                  cuVS's choice
      Maximum search iterations.
  --num-random-samplings <N>            cuVS's choice
      Random sampling rounds for the initial search points.
  --build-algo <ALGORITHM>              auto
      Graph build algorithm, auto or nn_descent.
  --devices <ORDINALS>                  0
      GPU ordinals; CAGRA builds and searches on the first.
Vector databases: retri-eval-qdrant, -redis, -weaviate, -surrealdb, -lancedb & -turso
retri-eval-qdrant, retri-eval-redis, retri-eval-weaviate, retri-eval-surrealdb
  --startup-time-limit <DURATION>       120s
      Time limit for container start and readiness.
  --connectivity <N,...>                16
      HNSW M.
  --expansion-add <N,...>               128
      HNSW construction width.
  --expansion-search <N,...>            64
      HNSW search width.

retri-eval-qdrant
  --data-type <TYPE,...>                f32
      Storage type: f32, f16 or u8.
  --metric <METRIC,...>                 l2
      ip, cos, l2 or manhattan.
  --quantization <MODE,...>             none
      Server-side none, binary or scalar.
  --grpc-port <PORT>                    6334
      Host gRPC port.
  --http-port <PORT>                    6333
      Host HTTP port.
  --vectors-per-upsert <N>              10000
      Vectors per upsert request.

retri-eval-redis
  --data-type <TYPE,...>                f32
      f32, f64, f16, bf16, u8 or i8; u8 and i8 need Redis 8.
  --metric <METRIC,...>                 l2
      l2, ip or cos.
  --port <PORT>                         6379
      Host port.
  --vectors-per-upsert <N>              1000
      Vectors per pipeline flush.

retri-eval-weaviate
  --metric <METRIC,...>                 l2
      l2, ip or cos.
  --quantization <MODE,...>             none
      Server-side none or binary.
  --port <PORT>                         8080
      Host HTTP port.

retri-eval-surrealdb
  --metric <METRIC,...>                 l2
      l2 or cos.
  --port <PORT>                         8000
      Host HTTP port.

retri-eval-lancedb
  --metric <METRIC>                     l2
      One of ip, cos or l2.
  --data-dir <DIR>                      data
      LanceDB storage directory.

retri-eval-turso
  --metric <METRIC>                     l2
      One of l2 or cos.
  --data-dir <DIR>                      in memory
      Directory of the Turso database file; only its bench table is recreated.
Dataset tools: retri-generate, retri-ground-truth, retri-download-molecules & retri-download-cohere
retri-generate, retri-download-molecules, retri-download-cohere
  --threads <N>                         0
      Ground-truth threads, 0 for all cores.
  --seed <SEED>                         42
      32-bit seed of generation or query sampling, or random.
  --output <DIR>                        required
      Output directory.

retri-generate
  --format <FORMAT>                     required
      b1bin for clustered binary with Hamming ground truth, or fbin for Gaussian f32 with L2.
  --base-count <N>                      required
      Base vectors to generate.
  --query-count <N>                     required
      Query vectors to generate.
  --dims <N>                            required
      Bits for b1bin, a multiple of 8, or scalars for fbin.
  --clusters <N>                        256
      Cluster centers.
  --noise <PROBABILITY>                 0.1
      Bit-flip probability from 0 to 1.
  --top-k <K>                           10
      Neighbors per query in the ground-truth file.

retri-ground-truth
  --base-vectors <PATH|GLOB>            required
      Base vectors: .fbin, .u8bin, .i8bin or .b1bin.
  --query-vectors <PATH>                required
      Query vectors of the base's scalar type and width.
  --metric <METRIC>                     required
      euclidean over .fbin, .i8bin and .u8bin, or hamming over .b1bin.
  --top-k <K>                           100
      Neighbors per query in the ground-truth file.
  --threads <N>                         0
      Ground-truth threads, 0 for all cores.
  --output <PATH>                       required
      Ground-truth file to write: neighbor rows, then their distances.

retri-download-molecules, retri-download-cohere
  --query-count <N>                     10000
      Queries sampled from the base, along with the ground truth.
  --limit <N>                           all
      Rows to extract.
  --download-concurrency <N>            4
      Concurrent shard downloads.
  --keep-parquet                        off
      Keep the Parquet shards under <output>/parquet/.

retri-download-molecules
  --source <SOURCE>                     required
      pubchem (115M), gdb13 (977M) or enamine (6.04B).
  --fingerprint <COLUMN>                required unless --conformers
      maccs (166 bits), pubchem (881), ecfp4 (2048) or fcfp4 (2048).
  --conformers                          off
      Each embedded molecule's lowest-energy conformer, as base.part_N.f16bin xyz points beside
      counts.part_N.ibin points per cloud; excludes --fingerprint and --top-k. With
      --query-count, also query.N.f16bin and query.N.counts.ibin of N sampled clouds.
  --top-k <K>                           none
      Neighbors per query in the ground-truth file, which only this flag writes.
  --hold-out                            off
      Move the sampled queries out of the base parts first, so the truth covers the rest.
  --part-rows <N>                       512000000
      Rows per base part, a multiple of the bucket's 1M-row shards.
  --url-prefix <URL>                    the public bucket
      S3 or HTTPS prefix of the shards.

retri-download-cohere
  --top-k <K>                           10
      Neighbors per query in the ground-truth file.
  --language <CODE>                     en
      Language config of the Hugging Face dataset.
  --no-text                             off
      Skip the aligned titles.txt, texts.txt and urls.txt.

Self-Recall

--self-search answers a different question from --query-neighbors, and closes two gaps the ground-truth path leaves open.

Shipped query files are small — SIFT has 10K queries, a single batch finishing in under a tenth of a second on a modern GPU. A QPS figure measured over that burst is real, but says nothing about sustained throughput. Self-recall replays indexed vectors as queries, so an unsampled 100M-vector index issues 100M queries, making it the only phase in a report measured under load. Use --self-search-sample to cap the sweep (e.g. 0.1 for a tenth, or 50000 for an absolute count) when a full pass is too slow, as it is on the server backends.

It is also the metric that survives --max-base-vectors most cleanly. Ground-truth files name neighbors drawn from the whole published base, so capping a run to a slice drags recall down by roughly the cap ratio no matter how good the index is — a quarter-size slice of the SIFT base reports about 0.25, not 0.99. The harness does not divide that back out: the correction would assume the indexed slice is a uniform sample of the base, and a leading-prefix cap is not one. It reports the raw figure and, in dataset.vectors_count, the divisor — coverage is steps[].vectors_indexed / dataset.vectors_count, plotted as coverage.png. Self-recall needs no such correction at all: identity truth is computed against exactly the vectors that were inserted.

Three caveats are worth knowing before quoting the numbers. Exact-duplicate vectors in the base can cost a hit, since either twin may take rank 1, so a self-recall below 1.0 bounds the index's error rate rather than measuring it exactly. A self-query is a graph node rather than an out-of-sample point, so traversal converges faster than it would on real queries — treat the throughput as an optimistic bound, not as a substitute for a query-set measurement. And --self-search-sample replays the leading N rows rather than a random draw, because contiguous slices are what keep the queries zero-copy and the throughput figure clean. On a base whose row order carries structure, only the default full sweep is unbiased.

Note also that a self-search's recall_at_k is very nearly redundant with its recall_at_1: a vector's own distance to itself is zero, which is minimal, so if the index retrieves it at all it lands at rank 1. Raising --self-search-top-k therefore changes how much search work each query does — which is the point when you are measuring throughput — but not the recall value.

Keys & Order

Keys stay as wide as their input: a keys file is memory-mapped at its own width, and generated keys are computed from the row on demand, so neither costs memory per row. uuid-v7 keys hold the row in their timestamp and counter, so they sort as rows do, while uuid-v4 keys scatter over the whole key space. Each engine stores keys natively where it can:

  • USearch takes every key as its 16-byte Key.
  • Qdrant keeps keys that fit 64 bits as numeric point ids and the rest as UUID point ids.
  • Weaviate and SurrealDB use each key as the object or record UUID, integers zero-padded.
  • Redis spells keys as UUIDs in the hash names.
  • LanceDB sizes its id column to UInt32, UInt64 or FixedSizeBinary(16).
  • FAISS, cuVS and Turso store base rows as their ids and name hits by the rows' keys, so keys of any width cost them nothing; cuVS saves those rows to a .rows file beside its index.

Duplicate keys suit USearch and the engines that store rows, FAISS, cuVS and Turso, since the others overwrite a repeated id.

--insertion-order shuffled is a seeded Feistel permutation computed per position, so a shuffle of any size stores nothing per row.

Observability

Building with --features perf-counters and passing --perf-counters adds per-phase hardware counters on Linux, and an external perf stat sidecar works on any build. src/README.md covers both, along with how each backend measures memory_bytes, which differs from one backend to the next.

Output Format

One JSON file per backend configuration, written to --output <dir>. Files are auto-named <backend>-<hash>.json.

{
  "machine": { "cpu_model": "Intel Xeon 6776P", "physical_cores": 96,
               "budget": { "cores": 192, "cpus": "0-191", "memory_limit": 549755813888 }, ... },
  "dataset": { "base_vectors_path": "...", "vectors_count": 10000000, "dimensions": 100, ... },
  "config": { "backend": "usearch", "data_type": "f32", "metric": "l2", "connectivity": 16,
              "dimensions": 100, "vectors_count": 10000000, "top_k": 10, ... },
  "steps": [
    {
      "vectors_indexed": 1000000,
      "memory_bytes": 412000000, "process_anon_bytes": 436000000,
      "add": { "elapsed": 12.3, "throughput": 81300 },
      "refine": { "passes": 2, "elapsed": 6.8, "changed": [412000, 61000] },
      "ground_truth_search": {
        "queries": 100000, "passes": 23, "top_k": 10,
        "elapsed": 5.1, "throughput": 451000,
        "recall_at_1": 0.0942, "recall_at_k": 0.2815,
        "intersection_at_k": 0.2611, "ndcg_at_k": 0.1847
      },
      "self_search": {
        "queries": 1000000, "passes": 1, "top_k": 10,
        "elapsed": 1.6, "throughput": 627541,
        "recall_at_1": 0.9958, "recall_at_k": 0.9961
      }
    }
  ]
}

A step pairs the index state (vectors_indexed, memory_bytes, process_anon_bytes) with the phases that acted on it. add.elapsed includes the wait until a server finishes indexing what was added. A ground-truth search runs one untimed pass to warm the index, then repeats every query until at least 5 seconds pass, and passes counts the timed ones. refine appears on the last insert step when --max-refines ran refinement rounds, which stop at the first that does not improve the graph; passes counts the rounds that ran and changed the rows each one re-wired. add is null on the --index load path, where nothing was inserted; self_search appears on at most one step — the last — because it runs once against the finished index, and that step is the index state it ran against. Its metrics drop the self_ prefix because the container already says it, and it carries no intersection_at_k or ndcg_at_k: identity truth is a single key per query, so neither a set overlap nor a ranked gain says anything beyond recall_at_1.

dataset describes the input files and config describes the run, so dataset.vectors_count is the base file's row count while config.vectors_count is what --max-base-vectors actually indexed; divide steps[].vectors_indexed by the former for the share of the ground truth a step could possibly have found. dataset.dimensions is likewise the file's width, and config.dimensions the width a --dims sweep truncated to.

Development

The repository pins Rust 1.100. CI runs cargo fmt --check, Ruff on the plotting script, then cargo clippy and cargo test per feature set; pass the same --features locally for optional native engines. src/ holds the shared library in bench.rs, one binary per backend and retri-generate, scripts/ the dataset downloaders and plotting, .githooks/ the Git hooks, and docker/ the compose files for the server backends. Arm the hooks once per clone with git config core.hooksPath .githooks.

About

Benchmark suite that compares vector search engines against each other on billion-scale datasets, from in-memory HNSW libraries like USearch, FAISS, & cuVS to managed databases like Qdrant, Weaviate, LanceDB, & Redis – built for reproducibility

Topics

Resources

Stars

37 stars

Watchers

2 watching

Forks

Contributors

Languages