RetriEval is a benchmarking suite designed for Billion-scale Vector Search workloads. It's primarily used to benchmark in-process Search Engines on CPUs and GPUs, like USearch, FAISS, and cuVS, but it also reuses similar profiling logic for standalone databases like Qdrant, Weaviate, and Redis. It works with the same plain input format standardized by the BigANN benchmark, aiming for reproducible measurements – with shuffled parallel construction, incremental recall curves, normalized metrics, and machine-readable reports, capturing everything from machine topology to indexing hyper-parameters.
Install the default retri-eval-usearch binary:
cargo install --path .Fetch the Unum Wiki 1M dataset — ~1.1 GB of vectors, queries, and ground truth — into data/wiki-1m/:
scripts/download.sh wiki-1mRun a sweep over three quantizations and write JSON reports under results/:
retri-eval-usearch \
--base-vectors data/wiki-1m/base.1M.fbin \
--query-vectors data/wiki-1m/query.public.100K.fbin \
--query-neighbors data/wiki-1m/groundtruth.public.100K.ibin \
--data-type bf16,f16,i8 \
--metric inner \
--output results/Generate plots from the results:
uv run scripts/plot.py results/ --output plots/| Backend | Parallelism | Quantization | Metrics |
|---|---|---|---|
| USearch | ForkUnion | bf16, f16, e5m2, e4m3, e3m2, e2m3, i8, u8, b1 | inner, euclidean, angular, hamming, jaccard |
| FAISS | OpenMP | f32, f16, bf16, u8, i8, b1 | inner, euclidean |
| cuVS | CUDA | f32, f16, i8, u8 | euclidean, inner, angular |
- USearch: The input is stored in the
--data-typequantization, converted on the fly.e3m2ande2m3occupy byte slots;b1is packed. - FAISS: Input is always f32.
--data-typeselects the internal scalar quantizer (SQfp16, SQbf16, SQ8_direct, etc.). - cuVS: Input is converted on the host to the
--data-type, one of f32, f16, i8 or u8, before CAGRA builds on it. It reuses the shared--connectivity,--expansion-addand--expansion-searchnames and JSON keys, so its runs plot on the same axes as the HNSW backends. The mapping is an analogy, not an identity:--expansion-addis CAGRA's pre-prune candidate width rather thanef_construction, and--expansion-searchits internal top-i list rather thanef_search.
retri-eval-usearch --data-type bf16 --metric euclidean ...
retri-eval-faiss --data-type f16 --metric euclidean ...
retri-eval-cuvs --metric euclidean ...Server-side quantization is managed by the database engine, not the benchmark.
Binary quantization is deterministic sign(x) per dim, and scalar quantization is per-dim min/max — neither trains a codebook, so both stay inside the "no learned logic" constraint the rest of the benchmark holds for the native backends.
Product quantization is deliberately excluded everywhere.
| Backend | Client | Docker Image | Metrics | Wire dtype sweep | Server-side quantization |
|---|---|---|---|---|---|
| Qdrant | qdrant-client, gRPC |
qdrant/qdrant:v1.19.1 |
ip, l2, cos, manhattan | f32, f16, u8 |
none, binary, scalar |
| Redis | redis, RESP |
redis:8.10 |
ip, l2, cos | f32, f64, f16, bf16, u8, i8 |
— |
| Weaviate | reqwest, REST |
semitechnologies/weaviate:1.39.7 |
ip, l2, cos | f32 only |
none, binary |
| LanceDB | lancedb, in-process, Arrow |
— | ip, l2, cos | f32 only |
— (exact scan) ¹ |
| Turso | turso = 0.8.1, native Rust |
— | l2, cos | f32 only |
— (exact scan) |
| SurrealDB | reqwest, HTTP SQL |
surrealdb/surrealdb:v3.3.0 |
l2, cos | f32 only |
— (HNSW) |
¹ LanceDB's Rust client — lancedb 0.37 — exposes graph-based search only via IvfHnswFlat / IvfHnswSq / IvfHnswPq, all behind a k-means-trained IVF layer that cannot be disabled.
No pure-HNSW variant is offered, so this benchmark leaves LanceDB on plain f32 + L2/IP/Cos until upstream adds one.
Hamming is only available on IvfFlat, outside our graph path.
Turso runs in-process with exact f32 scans over an in-memory database, or a file in --data-dir whose benchmark table alone is recreated; it measures exact search, not a libSQL ANN index.
The server backends run in Docker containers the binaries manage themselves; docker/README.md covers their lifecycle and per-server caveats.
Each backend is behind its own feature flag. Build only what you need:
cargo build --release --features usearch-backend # USearch
cargo build --release --features usearch-cuda # USearch with its CUDA kernels, for --residency vram
cargo build --release --features faiss-backend # FAISS
cargo build --release --features qdrant-backend # Qdrant
cargo build --release --features redis-backend # Redis
cargo build --release --features lancedb-backend # LanceDB
cargo build --release --features weaviate-backend # Weaviate
cargo build --release --features cuvs-backend # cuVS
cargo build --release --no-default-features --features turso-backend # Turso
cargo build --release --no-default-features --features surrealdb-backend # SurrealDB
cargo build --release --no-default-features --features ground-truth # retri-ground-truthOr combine multiple:
cargo build --release --features usearch-backend,faiss-backend,qdrant-backendThe BigANN collections are a good starting point, as most ship pre-computed ground-truth neighbors for recall evaluation.
The tables below group the datasets by scale, and each name links to its download and run recipe in data/README.md, which also covers the file format and sharded inputs.
| Dataset | Scalar Type | Dimensions | Metric | Base Size | Ground Truth |
|---|---|---|---|---|---|
| Unum UForm Wiki | f32 |
256 | IP | 1 GB | 100K queries, yes |
| Arxiv with E5 | f32 |
768 | IP | 7 GB | 2.3M queries, yes |
| Dataset | Scalar Type | Dimensions | Metric | Base Size | Ground Truth |
|---|---|---|---|---|---|
| Meta BIGANN — SIFT | u8 |
128 | L2 | 1.2 GB | 10K queries, yes |
| Microsoft Turing-ANNS | f32 |
100 | L2 | 3.7 GB | 100K queries, yes |
| Microsoft SpaceV | i8 |
100 | L2 | 0.9 GB | 30K queries, yes |
| Yandex Deep | f32 |
96 | L2 | 3.6 GB | 10K queries, yes |
| Cohere Wiki EN | b1 |
1024 | Hamming | 5.3 GB | self-sampled ¹ |
¹ These sources ship vectors without ground truth, so their
retri-download-*binaries sample queries with a fixed seed and compute exact Hamming top-K.
| Dataset | Scalar Type | Dimensions | Metric | Base Size | Ground Truth |
|---|---|---|---|---|---|
| Meta BIGANN — SIFT | u8 |
128 | L2 | 12 GB | 10K queries, yes |
| Microsoft Turing-ANNS | f32 |
100 | L2 | 37 GB | 100K queries, yes |
| Microsoft SpaceV | i8 |
100 | L2 | 9.3 GB | 30K queries, yes |
| Yandex Deep | f32 |
96 | L2 | 36 GB | 10K queries, yes |
| Unum WikiVerse ² | f16 |
128–4096³ | Cos/IP | 95-505GB | pipeline pending |
| USearchMolecules PubChem MACCS | b1 |
168 | Hamming | 2.4 GB | self-sampled ¹ |
| USearchMolecules PubChem ECFP4 | b1 |
2048 | Hamming | 29 GB | self-sampled ¹ |
² WikiVerse ships
.f16bin, which RetriEval does not read yet. ³ nomic-embed at 768, arctic-embed and Qwen3 at 1024, e5-mistral at 4096; the ColBERT model's 128d per token needs multi-vector support.
| Dataset | Scalar Type | Dimensions | Metric | Base Size | Ground Truth |
|---|---|---|---|---|---|
| Meta BIGANN — SIFT | u8 |
128 | L2 | 119 GB | 10K queries, yes |
| Microsoft Turing-ANNS | f32 |
100 | L2 | 373 GB | 100K queries, yes |
| Microsoft SpaceV | i8 |
100 | L2 | 93 GB | 30K queries, yes |
| Yandex Text-to-Image | f32 |
200 | Cos | 750 GB | 100K queries, yes |
| Yandex Deep | f32 |
96 | L2 | 358 GB | 10K queries, yes |
| USearchMolecules GDB-13 MACCS | b1 |
168 | Hamming | 21 GB | self-sampled ¹ |
| USearchMolecules GDB-13 ECFP4 | b1 |
2048 | Hamming | 250 GB | self-sampled ¹ |
| USearchMolecules Enamine REAL MACCS | b1 |
168 | Hamming | 127 GB | self-sampled ¹ |
| USearchMolecules Enamine REAL ECFP4 | b1 |
2048 | Hamming | 1.55 TB | self-sampled ¹ |
These numbers were measured for reference on DGX-B300 and DGX-H100 class nodes, the main drivers of AI neoclouds, and on single-socket AWS Graviton4 and Graviton5 hosts, and mostly guide the design of USearch. They shift with the data, the software versions and the hardware, so treat them as reference points for your own runs, not as fixed rankings.
Each table holds one row per backend, device and data type, so a new run is a new row and nothing else moves:
Device: the cores a run used and what they are, like148× NVIDIA SM103for the streaming multiprocessors of one B300 or128× Intel Xeon6for the physical cores of a dual-socket host. CPU rows count physical cores, not SMT threads: with two threads per core, the96×and128×hosts ran 192 and 256 threads.Type: the stored type; every backend took its native input type, with no rescaling.Connectivity: the graph degree, M.Layer 0: the edges each vector keeps in the densest layer: 2×M in the HNSW graphs of USearch 2.25 and FAISS, M in the single-layer graphs of USearch 3.0 and cuVS.Expansions: the construction and search expansion, split by/.1-r@10: 1-recall@10, the share of queries whose true nearest neighbor lands in the top 10.10-r@10: K-recall@10, the share of the true top 10 found.Add/sandSearch/s: vectors indexed and queries answered per second.Memory: the backend's own peak figure, whichsrc/README.mddefines per backend.Build: the wall time to index the whole base.
Every row follows the same methodology:
- Tune each backend's settings until it reaches 99% 1-recall@10 or more on the dataset.
- Never let a row build faster than it searches: when
Add/sexceedsSearch/s, raise the construction expansion and lower the search expansion until search is at least as fast at that recall. - Run every command on a 1M prefix first, then on a 10M prefix, before the full file.
- Match the links each vector keeps: 64 means connectivity 64 in a single-layer graph and M=32 in HNSW.
- Start every engine from the same expansions, and widen the construction only where a dataset misses 0.99.
- Hold the 0.99 target on every dataset.
- Give each engine its best pages and kernels, and name them under the table.
- Build in 20 steps, and validate the ground truth against the base before the first insert.
Compare the fastest measured configuration meeting the same recall target, with the same recall definition, queries, K, and ground truth. Report build time and memory alongside query throughput, and keep query concurrency, batch size, and cache state comparable.
Each search reports both recall conventions, because they answer different questions and one of them saturates:
recall_at_kis 1-recall@K, FAISS'sOneRecallAtRCriterionand the measure USearch reports: any of the top K within the true nearest distance.intersection_at_kis K-recall@K, the share of the top K within the K-th true distance, order-independent, and what ann-benchmarks and cuVS's own harness publish as recall.ndcg_at_kweights true hits by1 / log2(rank + 1), normalized by ideal gain.- Ties are counted by exact distance: the harness recomputes the distance of every true neighbor and every answer from the base vectors, whatever the engine stores, so a neighbor tied with a true one counts whichever key the ground truth kept, within a relative 10⁻⁶ for rounding.
- Neither 1-recall@K nor K-recall@K generally bounds the other, so the two are never interchangeable.
- K is
--top-k, which defaults to the ground-truth file's width, 10 on the Wiki sets and 100 on the BigANN ones. - 1-recall@K stops discriminating as K grows: at K=100 nearly any sane configuration finds the single true neighbor somewhere in the list.
intersection_at_kstays informative there, andndcg_at_klikewise normalizes over the full truth prefix. - Pass
--top-k 10for figures comparable across datasets of differing ground-truth width.
168-bit binary fingerprints of PubChem molecules under Hamming, with settings calibrated at 10M and reused at 100M; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | b1 |
32 | 64 | 128 / 64 | 0.9696 | … | 36,347 | 35,767 | 4.7 GB | 5.0m |
| FAISS 1.12 | 96× Intel Xeon6 | b1 |
64 | 128 | 40 / 16 | 0.9661 | … | 95,230 | 293,795 | 7.6 GB | 1.5m |
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | b1 |
32 | 64 | 128 / 64 | 0.8438 | … | 35,080 | 38,432 | 40.8 GB | 54.6m |
| FAISS 1.12 | 96× Intel Xeon6 | b1 |
64 | 128 | 40 / 16 | … | … | … | … | ≥ 63 GB | killed at 9h |
| USearch 3.0 | 148× NVIDIA SM103 | b1 |
32 | 32 | 256 / 64 | 0.9907 | 0.9906 | 559,140 | 1,460,000 | 18.3 GB | 3.4m |
2048-bit ECFP4 fingerprints of PubChem molecules under Hamming, all 115,627,256 of them; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 3.0 | 148× NVIDIA SM103 | b1 |
32 | 32 | 512 / 512 | 0.9906 | … | 146,810 | 285,230 | … | 13.1m |
| USearch 3.0 | 192× Arm Graviton5 | b1 |
32 | 32 | 512 / 512 | 0.9821 | 0.9038 | 104,412 | 57,807 | 53.6 GB | 18.5m |
| FAISS 1.15.1 | 192× Arm Graviton5 | b1 |
32 | 64 | 40 / 16 | 0.5623 | 0.4704 | 341,635 | 839,862 | 93.3 GB | 5.6m |
| USearch 3.0 | 84× Arm Graviton4 | b1 |
64 | 64 | 512 / 512 | 0.9931 | 0.9953 | 41,008 | 49,048 | 68.6 GB | 47.0m |
| FAISS 1.15.1 | 84× Arm Graviton4 | b1 |
32 | 64 | 512 / 1024 | 0.9870 | 0.9925 | 27,175 | 11,750 | 62.9 GB | 1h 11m |
Neither Graviton5 row reaches 99%, so both break the methodology above:
- USearch on Graviton5 levels off at 0.9843 even at a search expansion of 1,024, short of the 0.9906 the B300 reaches at the same 512 / 512, and at that recall it builds faster than it searches.
- FAISS's binary HNSW ran there at its compiled-in 40 / 16, since FAISS's parameter tuning has no dispatcher for
IndexBinaryHNSW. - On Graviton4 the harness sets FAISS's expansions directly, and FAISS still levels off at 0.9870, where it builds faster than it searches.
- 10-recall@10 near 0.90 is tie scoring: Hamming distances tie often, and those rows credited only the exact keys the ground truth kept among tied neighbors.
Meta's 128-dimensional u8 descriptors under L2, compared at the setting that first reaches 99% recall@10; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | u8 |
16 | 32 | 128 / 256 | 0.9938 | … | 35,405 | 80,729 | 4.4 GB | 4.8m |
| FAISS 1.12 | 96× Intel Xeon6 | u8 |
16 | 32 | 128 / 256 | 0.9952 | … | 26,374 | 38,278 | 5.9 GB | 5.6m |
| cuVS 26.10 | 148× NVIDIA SM103 | u8 |
64 | 64 | 128 / 128 | … | 0.9966 | 204,838 | 1,204,000 | … | 0.8m |
| USearch 3.0 | 148× NVIDIA SM103 | u8 |
32 | 32 | 256 / 128 | 0.9911 | 0.9780 | 558,510 | 1,040,000 | 2.6 GB | 0.3m |
| FAISS 1.15.1 | 128× Intel Xeon6 | u8 |
32 | 64 | 128 / 96 | 0.9900 | 0.9765 | 115,643 | 133,919 | 8.6 GB | 1.4m |
| FAISS 1.15.1 | 128× Intel Xeon6 | f32 |
32 | 64 | 128 / 96 | 0.9900 | 0.9765 | 59,688 | 61,027 | 14.4 GB | 2.8m |
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | u8 |
16 | 32 | 128 / 256 | 0.9833 | … | 39,831 | 75,808 | 53.7 GB | 48.7m |
| FAISS 1.12 | 96× Intel Xeon6 | u8 |
16 | 32 | 128 / 256 | … | … | … | … | ≥ 46 GB | killed at 9h |
| cuVS 26.10 | 148× NVIDIA SM103 | u8 |
64 | 64 | 128 / 256 | … | 0.9963 | 154,799 | 582,000 | … | 10.8m |
| USearch 3.0 | 148× NVIDIA SM103 | u8 |
32 | 32 | 256 / 256 | 0.9913 | 0.9810 | 431,520 | 604,490 | 24.8 GB | 3.9m |
| USearch 3.0 | 192× Arm Graviton5 | u8 |
32 | 32 | 512 / 160 | 0.9925 | 0.9824 | 161,952 | 173,823 | 33.8 GB | 10.3m |
| FAISS 1.15.1 | 192× Arm Graviton5 | u8 |
32 | 64 | 256 / 128 | 0.9917 | 0.9792 | 84,536 | 178,600 | 56.9 GB | 19.7m |
| USearch 3.0 | 84× Arm Graviton4 | u8 |
64 | 64 | 128 / 128 | 0.9921 | 0.9775 | 185,046 | 208,835 | 46.7 GB | 9.0m |
| FAISS 1.15.1 | 84× Arm Graviton4 | u8 |
32 | 64 | 256 / 128 | 0.9912 | 0.9793 | 41,917 | 93,549 | 42.3 GB | 39.8m |
Microsoft's 100-dimensional i8 web-search embeddings under L2, compared at the setting that first reaches 99% 1-recall@10; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 3.0 | 84× Arm Graviton4 | i8 |
64 | 64 | 512 / 384 | 0.9934 | 0.9929 | 38,713 | 57,971 | 45.1 GB | 43.1m |
| FAISS 1.15.1 | 84× Arm Graviton4 | i8 |
32 | 64 | 512 / 640 | 0.9912 | 0.9908 | 14,975 | 15,773 | 39.5 GB | 1h 51m |
Yandex's 96-dimensional f32 image embeddings under L2, stored as bf16 by both backends and compared at the setting that first reaches 99% 1-recall@10; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 3.0 | 84× Arm Graviton4 | bf16 |
64 | 64 | 256 / 192 | 0.9934 | 0.9804 | 87,964 | 118,976 | 53.1 GB | 18.9m |
| FAISS 1.15.1 | 84× Arm Graviton4 | bf16 |
32 | 64 | 256 / 256 | 0.9929 | 0.9827 | 48,503 | 56,920 | 48.6 GB | 34.4m |
Microsoft's 100-dimensional f32 web-search embeddings under L2, compared at the setting that first reaches 99% recall@10 across storage types; recipes in data/README.md.
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | f32 |
48 | 96 | 768 / 384 | 0.9929 | … | 8,532 | 12,331 | 13.0 GB | 18.3m |
| USearch 2.25 | 96× Intel Xeon6 | bf16 |
48 | 96 | 768 / 384 | 0.9929 | … | 10,496 | 16,940 | 10.9 GB | 14.1m |
| USearch 2.25 | 96× Intel Xeon6 | f16 |
48 | 96 | 768 / 384 | 0.9929 | … | 10,969 | 20,246 | 10.9 GB | 13.5m |
| USearch 2.25 | 96× Intel Xeon6 | e5m2 |
48 | 96 | 768 / 384 | 0.9919 | … | 10,526 | 20,534 | 9.8 GB | 13.5m |
| USearch 2.25 | 96× Intel Xeon6 | e4m3 |
48 | 96 | 768 / 384 | 0.9930 | … | 7,353 | 12,106 | 9.8 GB | 19.4m |
| USearch 2.25 | 96× Intel Xeon6 | e3m2 |
48 | 96 | 768 / 384 | 0.9728 | … | 10,398 | 18,022 | 9.8 GB | 13.3m |
| USearch 2.25 | 96× Intel Xeon6 | e2m3 |
48 | 96 | 768 / 384 | 0.7941 | … | 10,935 | 21,313 | 9.8 GB | 13.2m |
| FAISS 1.12 | 96× Intel Xeon6 | f32 |
48 | 96 | 768 / 384 | 0.9944 | … | 7,491 | 16,486 | 14.1 GB | 20.6m |
| FAISS 1.12 | 96× Intel Xeon6 | bf16 |
48 | 96 | 768 / 384 | 0.9944 | … | 3,800 | 10,391 | 12.1 GB | 39.4m |
| FAISS 1.12 | 96× Intel Xeon6 | f16 |
48 | 96 | 768 / 384 | 0.9944 | … | 2,545 | 10,032 | 12.1 GB | 1h 1m |
| USearch 3.0 | 148× NVIDIA SM103 | bf16 |
32 | 32 | 1024 / 1024 | 0.9908 | 0.9831 | 95,870 | 155,810 | 3.4 GB | 1.7m |
| Backend | Device | Type | Connectivity | Layer 0 | Expansions | 1-r@10 | 10-r@10 | Add/s | Search/s | Memory | Build |
|---|---|---|---|---|---|---|---|---|---|---|---|
| USearch 2.25 | 96× Intel Xeon6 | f32 |
48 | 96 | 768 / 384 | 0.9929 | … | 6,646 | 10,398 | 139.6 GB | 4h 1m |
| USearch 2.25 | 96× Intel Xeon6 | bf16 |
48 | 96 | 768 / 384 | 0.9931 | … | 8,564 | 14,772 | 105.2 GB | 3h 1m |
| USearch 2.25 | 96× Intel Xeon6 | f16 |
48 | 96 | 768 / 384 | 0.9930 | … | 8,807 | 15,412 | 105.2 GB | 2h 54m |
| USearch 2.25 | 96× Intel Xeon6 | e5m2 |
48 | 96 | 768 / 384 | 0.9924 | … | 7,368 | 13,227 | 88.0 GB | 3h 15m |
| USearch 3.0 | 148× NVIDIA SM103 | bf16 |
32 | 32 | 1024 / 1024 | 0.9933 | 0.9868 | 88,660 | 156,790 | 32.1 GB | 18.8m |
Each backend is a separate binary, and the manuals below list every flag of every binary with its default.
A value spelled <…,...> takes comma-separated values and runs one configuration per combination.
Counts take K, M, G or T in powers of 1000, like 10M, and sizes take KB to TB in powers of 1024, like 16GB.
A zero connectivity or expansion lets USearch derive it, and the report records the connectivity the index header holds.
A bad value prints --flag="value" does not parse, expected … and exits with status 1.
A sweep also exits with status 1 when any configuration fails or none runs, while configurations a backend refuses to open are reported as skipped.
Every retri-eval-* flag also reads RETRIEVAL_<FLAG> as its default, upper-cased with dashes as underscores, so --top-k reads RETRIEVAL_TOP_K.
The flag on the command line overrides the variable, and switches like --self-search read 1, 0, true or false.
Shared by every retri-eval-* binary
retri-eval-*
--base-vectors <PATH|GLOB> required
Base vectors to index: .fbin, .u8bin, .i8bin or .b1bin.
--base-keys <PATH|GLOB> generated
Keys for the base vectors, one per vector: .i32bin, .u64bin or .uuidbin.
--key-type <TYPE> u32
Keys generated without --base-keys: u32, u64, uuid-v4 or uuid-v7.
--query-vectors <PATH|GLOB> required
Query vectors to search with.
--query-neighbors <PATH|GLOB> required
Ground-truth neighbor rows for those queries: .ibin, or .u64bin past 2^32 base rows.
--top-k <K> ground-truth width
Neighbors per query, the K every metric is taken at; at most the ground-truth width.
--insertion-order <ORDER> shuffled
shuffled by --seed, or the file's original order.
--seed <SEED> 42
32-bit seed of the shuffled order, or random; rejected with --insertion-order original.
--steps <N> 10
Measurement steps; the base is inserted in this many equal parts.
--inserts-per-core <N> 2048
Vectors per add call for each backend thread: --threads on usearch and faiss, one elsewhere.
--queries-per-core <N> 2048
Queries per search call for each backend thread: --threads on usearch and faiss, one elsewhere.
--output <DIR> none
Directory for JSON result files; progress only when unset.
--memory-limit <SIZE> cgroup memory.max, else all RAM
Memory the engine may use, page cache included, like 16GB; servers get it as their
container's memory and swap limit, beside the process's CPU affinity as their cpuset.
--perf-counters off
Record hardware counters per phase; needs the perf-counters feature on Linux.
--max-base-vectors <N> all
Cap on the base vectors used, for calibration on a slice.
--dims <N,...> native
Matryoshka truncations, each at most the native width, and a multiple of 8 for .b1bin.
--self-search off
After the last insertion, replay indexed vectors as their own queries.
--self-search-top-k <K> 10
Neighbors per self-search query.
--self-search-sample <N|FRACTION> all
Leading base rows to replay, a count like 50000 or a fraction like 0.1; needs --self-search.
Search engines: retri-eval-usearch, retri-eval-faiss & retri-eval-cuvs
retri-eval-usearch, retri-eval-faiss, retri-eval-cuvs
--index <PATH> none
Persisted index: searched if the path exists, else built and saved; needs a single config.
retri-eval-usearch
--data-type <TYPE,...> bf16
Stored type: bf16, f16, e5m2, e4m3, e3m2, e2m3, i8, u8 or b1.
--metric <METRIC,...> euclidean
inner, euclidean, angular, hamming or jaccard; ip, l2, cos and tanimoto also parse.
--connectivity <N,...> 0
Graph connectivity, 0 to let USearch derive it.
--expansion-add <N,...> 0
Construction expansion, 0 to let USearch derive it.
--expansion-search <N,...> 0
Search expansion, 0 to let USearch derive it.
--memory-domains <NODES> every node the process may use
Comma-separated NUMA nodes, one shard on each.
--threads <N,...> 0
Threads, 0 for every core in the affinity mask.
--residency <WHERE> ram
ram, vram, or nvme: built in RAM, saved to --index and searched as a view on the last step.
--max-refines <N> 0
Refinement rounds after the last insert step, stopping at the first that does not improve the graph.
--nvme-paths <PATHS> where --index points
Devices or directories an NVMe view serves its parts from.
--devices <ORDINALS> one free device
GPU ordinals, one shard on each.
retri-eval-faiss
--data-type <TYPE,...> bf16
Scalar quantizer over f32 input: f32, f16, bf16, u8, i8 or b1.
--metric <METRIC,...> euclidean
inner or euclidean, also spelled ip and l2; b1 data always uses Hamming.
--connectivity <N,...> 32
HNSW M.
--expansion-add <N,...> 128
HNSW construction width.
--expansion-search <N,...> 64
HNSW search width.
--threads <N> 0
OpenMP threads, 0 for every core in the affinity mask.
retri-eval-cuvs
--data-type <TYPE,...> f32
f32, f16, i8 or u8, converted on the host before CAGRA builds on it.
--metric <METRIC,...> euclidean
euclidean, inner or angular, also spelled l2, ip and cos.
--connectivity <N,...> 32
CAGRA's output graph degree.
--expansion-add <N,...> 64
CAGRA's intermediate graph degree before pruning.
--expansion-search <N,...> 64
CAGRA's internal top-i list.
--search-width <N|auto,...> auto
Graph nodes that start each search iteration, or auto for cuVS's choice.
--min-iterations <N> cuVS's choice
Minimum search iterations.
--max-iterations <N> cuVS's choice
Maximum search iterations.
--num-random-samplings <N> cuVS's choice
Random sampling rounds for the initial search points.
--build-algo <ALGORITHM> auto
Graph build algorithm, auto or nn_descent.
--devices <ORDINALS> 0
GPU ordinals; CAGRA builds and searches on the first.
Vector databases: retri-eval-qdrant, -redis, -weaviate, -surrealdb, -lancedb & -turso
retri-eval-qdrant, retri-eval-redis, retri-eval-weaviate, retri-eval-surrealdb
--startup-time-limit <DURATION> 120s
Time limit for container start and readiness.
--connectivity <N,...> 16
HNSW M.
--expansion-add <N,...> 128
HNSW construction width.
--expansion-search <N,...> 64
HNSW search width.
retri-eval-qdrant
--data-type <TYPE,...> f32
Storage type: f32, f16 or u8.
--metric <METRIC,...> l2
ip, cos, l2 or manhattan.
--quantization <MODE,...> none
Server-side none, binary or scalar.
--grpc-port <PORT> 6334
Host gRPC port.
--http-port <PORT> 6333
Host HTTP port.
--vectors-per-upsert <N> 10000
Vectors per upsert request.
retri-eval-redis
--data-type <TYPE,...> f32
f32, f64, f16, bf16, u8 or i8; u8 and i8 need Redis 8.
--metric <METRIC,...> l2
l2, ip or cos.
--port <PORT> 6379
Host port.
--vectors-per-upsert <N> 1000
Vectors per pipeline flush.
retri-eval-weaviate
--metric <METRIC,...> l2
l2, ip or cos.
--quantization <MODE,...> none
Server-side none or binary.
--port <PORT> 8080
Host HTTP port.
retri-eval-surrealdb
--metric <METRIC,...> l2
l2 or cos.
--port <PORT> 8000
Host HTTP port.
retri-eval-lancedb
--metric <METRIC> l2
One of ip, cos or l2.
--data-dir <DIR> data
LanceDB storage directory.
retri-eval-turso
--metric <METRIC> l2
One of l2 or cos.
--data-dir <DIR> in memory
Directory of the Turso database file; only its bench table is recreated.
Dataset tools: retri-generate, retri-ground-truth, retri-download-molecules & retri-download-cohere
retri-generate, retri-download-molecules, retri-download-cohere
--threads <N> 0
Ground-truth threads, 0 for all cores.
--seed <SEED> 42
32-bit seed of generation or query sampling, or random.
--output <DIR> required
Output directory.
retri-generate
--format <FORMAT> required
b1bin for clustered binary with Hamming ground truth, or fbin for Gaussian f32 with L2.
--base-count <N> required
Base vectors to generate.
--query-count <N> required
Query vectors to generate.
--dims <N> required
Bits for b1bin, a multiple of 8, or scalars for fbin.
--clusters <N> 256
Cluster centers.
--noise <PROBABILITY> 0.1
Bit-flip probability from 0 to 1.
--top-k <K> 10
Neighbors per query in the ground-truth file.
retri-ground-truth
--base-vectors <PATH|GLOB> required
Base vectors: .fbin, .u8bin, .i8bin or .b1bin.
--query-vectors <PATH> required
Query vectors of the base's scalar type and width.
--metric <METRIC> required
euclidean over .fbin, .i8bin and .u8bin, or hamming over .b1bin.
--top-k <K> 100
Neighbors per query in the ground-truth file.
--threads <N> 0
Ground-truth threads, 0 for all cores.
--output <PATH> required
Ground-truth file to write: neighbor rows, then their distances.
retri-download-molecules, retri-download-cohere
--query-count <N> 10000
Queries sampled from the base, along with the ground truth.
--limit <N> all
Rows to extract.
--download-concurrency <N> 4
Concurrent shard downloads.
--keep-parquet off
Keep the Parquet shards under <output>/parquet/.
retri-download-molecules
--source <SOURCE> required
pubchem (115M), gdb13 (977M) or enamine (6.04B).
--fingerprint <COLUMN> required unless --conformers
maccs (166 bits), pubchem (881), ecfp4 (2048) or fcfp4 (2048).
--conformers off
Each embedded molecule's lowest-energy conformer, as base.part_N.f16bin xyz points beside
counts.part_N.ibin points per cloud; excludes --fingerprint and --top-k. With
--query-count, also query.N.f16bin and query.N.counts.ibin of N sampled clouds.
--top-k <K> none
Neighbors per query in the ground-truth file, which only this flag writes.
--hold-out off
Move the sampled queries out of the base parts first, so the truth covers the rest.
--part-rows <N> 512000000
Rows per base part, a multiple of the bucket's 1M-row shards.
--url-prefix <URL> the public bucket
S3 or HTTPS prefix of the shards.
retri-download-cohere
--top-k <K> 10
Neighbors per query in the ground-truth file.
--language <CODE> en
Language config of the Hugging Face dataset.
--no-text off
Skip the aligned titles.txt, texts.txt and urls.txt.
--self-search answers a different question from --query-neighbors, and closes two gaps the ground-truth path leaves open.
Shipped query files are small — SIFT has 10K queries, a single batch finishing in under a tenth of a second on a modern GPU.
A QPS figure measured over that burst is real, but says nothing about sustained throughput.
Self-recall replays indexed vectors as queries, so an unsampled 100M-vector index issues 100M queries, making it the only phase in a report measured under load.
Use --self-search-sample to cap the sweep (e.g. 0.1 for a tenth, or 50000 for an absolute count) when a full pass is too slow, as it is on the server backends.
It is also the metric that survives --max-base-vectors most cleanly.
Ground-truth files name neighbors drawn from the whole published base, so capping a run to a slice drags recall down by roughly the cap ratio no matter how good the index is — a quarter-size slice of the SIFT base reports about 0.25, not 0.99.
The harness does not divide that back out: the correction would assume the indexed slice is a uniform sample of the base, and a leading-prefix cap is not one.
It reports the raw figure and, in dataset.vectors_count, the divisor — coverage is steps[].vectors_indexed / dataset.vectors_count, plotted as coverage.png.
Self-recall needs no such correction at all: identity truth is computed against exactly the vectors that were inserted.
Three caveats are worth knowing before quoting the numbers.
Exact-duplicate vectors in the base can cost a hit, since either twin may take rank 1, so a self-recall below 1.0 bounds the index's error rate rather than measuring it exactly.
A self-query is a graph node rather than an out-of-sample point, so traversal converges faster than it would on real queries — treat the throughput as an optimistic bound, not as a substitute for a query-set measurement.
And --self-search-sample replays the leading N rows rather than a random draw, because contiguous slices are what keep the queries zero-copy and the throughput figure clean.
On a base whose row order carries structure, only the default full sweep is unbiased.
Note also that a self-search's recall_at_k is very nearly redundant with its recall_at_1: a vector's own distance to itself is zero, which is minimal, so if the index retrieves it at all it lands at rank 1.
Raising --self-search-top-k therefore changes how much search work each query does — which is the point when you are measuring throughput — but not the recall value.
Keys stay as wide as their input: a keys file is memory-mapped at its own width, and generated keys are computed from the row on demand, so neither costs memory per row.
uuid-v7 keys hold the row in their timestamp and counter, so they sort as rows do, while uuid-v4 keys scatter over the whole key space.
Each engine stores keys natively where it can:
- USearch takes every key as its 16-byte
Key. - Qdrant keeps keys that fit 64 bits as numeric point ids and the rest as UUID point ids.
- Weaviate and SurrealDB use each key as the object or record UUID, integers zero-padded.
- Redis spells keys as UUIDs in the hash names.
- LanceDB sizes its
idcolumn toUInt32,UInt64orFixedSizeBinary(16). - FAISS, cuVS and Turso store base rows as their ids and name hits by the rows' keys, so keys of any width cost them nothing; cuVS saves those rows to a
.rowsfile beside its index.
Duplicate keys suit USearch and the engines that store rows, FAISS, cuVS and Turso, since the others overwrite a repeated id.
--insertion-order shuffled is a seeded Feistel permutation computed per position, so a shuffle of any size stores nothing per row.
Building with --features perf-counters and passing --perf-counters adds per-phase hardware counters on Linux, and an external perf stat sidecar works on any build.
src/README.md covers both, along with how each backend measures memory_bytes, which differs from one backend to the next.
One JSON file per backend configuration, written to --output <dir>.
Files are auto-named <backend>-<hash>.json.
{
"machine": { "cpu_model": "Intel Xeon 6776P", "physical_cores": 96,
"budget": { "cores": 192, "cpus": "0-191", "memory_limit": 549755813888 }, ... },
"dataset": { "base_vectors_path": "...", "vectors_count": 10000000, "dimensions": 100, ... },
"config": { "backend": "usearch", "data_type": "f32", "metric": "l2", "connectivity": 16,
"dimensions": 100, "vectors_count": 10000000, "top_k": 10, ... },
"steps": [
{
"vectors_indexed": 1000000,
"memory_bytes": 412000000, "process_anon_bytes": 436000000,
"add": { "elapsed": 12.3, "throughput": 81300 },
"refine": { "passes": 2, "elapsed": 6.8, "changed": [412000, 61000] },
"ground_truth_search": {
"queries": 100000, "passes": 23, "top_k": 10,
"elapsed": 5.1, "throughput": 451000,
"recall_at_1": 0.0942, "recall_at_k": 0.2815,
"intersection_at_k": 0.2611, "ndcg_at_k": 0.1847
},
"self_search": {
"queries": 1000000, "passes": 1, "top_k": 10,
"elapsed": 1.6, "throughput": 627541,
"recall_at_1": 0.9958, "recall_at_k": 0.9961
}
}
]
}A step pairs the index state (vectors_indexed, memory_bytes, process_anon_bytes) with the phases that acted on it.
add.elapsed includes the wait until a server finishes indexing what was added.
A ground-truth search runs one untimed pass to warm the index, then repeats every query until at least 5 seconds pass, and passes counts the timed ones.
refine appears on the last insert step when --max-refines ran refinement rounds, which stop at the first that does not improve the graph; passes counts the rounds that ran and changed the rows each one re-wired.
add is null on the --index load path, where nothing was inserted; self_search appears on at most one step — the last — because it runs once against the finished index, and that step is the index state it ran against.
Its metrics drop the self_ prefix because the container already says it, and it carries no intersection_at_k or ndcg_at_k: identity truth is a single key per query, so neither a set overlap nor a ranked gain says anything beyond recall_at_1.
dataset describes the input files and config describes the run, so dataset.vectors_count is the base file's row count while config.vectors_count is what --max-base-vectors actually indexed; divide steps[].vectors_indexed by the former for the share of the ground truth a step could possibly have found.
dataset.dimensions is likewise the file's width, and config.dimensions the width a --dims sweep truncated to.
The repository pins Rust 1.100.
CI runs cargo fmt --check, Ruff on the plotting script, then cargo clippy and cargo test per feature set; pass the same --features locally for optional native engines.
src/ holds the shared library in bench.rs, one binary per backend and retri-generate, scripts/ the dataset downloaders and plotting, .githooks/ the Git hooks, and docker/ the compose files for the server backends.
Arm the hooks once per clone with git config core.hooksPath .githooks.
