tidaldb/docs/profiling/usearch-tuning.md

USearch Parameter Tuning

Summary

The production default (VectorIndexConfig::default()) is M=16, ef_construction=400, ef_search=200, F16. m12p3 re-ran the grid search at the production shape (1536-D) — not the historical 128-D — with a real exact brute-force oracle, and validated the F16/Int8 recall and memory frontier there.

What changed in m12p3. The earlier version of this doc reported an extrapolated 128-D table (values from published ANN-Benchmarks, not measured here). That is replaced below by measured 1536-D numbers from cargo run --release --example ann_grid_search. The example builds a UsearchIndex and a BruteForceIndex oracle over the same deterministic, id-keyed corpus and reports recall@10 vs the exact oracle, single-thread mean/p99 search latency, build time, and the true memory_usage() footprint.

Grid Search Setup (m12p3, measured)

Parameter	Values
M (connectivity)	16, 24, 32
ef_construction	400
ef_search (per-query)	128, 200, 400, 600
Quantization	F32, F16, Int8
Dataset size	100,000 vectors (and 1M on the k3s node)
Dimensionality	1536D (production embedding width)
Corpus shape	clustered (Gaussian mixture) — see below for why not uniform-random
Distance metric	L2 (L2-normalized → equivalent to cosine)
Recall metric	recall@10 vs exact brute-force cosine (200-query average)
Hardware	Apple Silicon laptop (release build)

Method

1. Build a deterministic, id-keyed clustered corpus (SplitMix64 Gaussian mixture — see below) so the run is reproducible, the recall numbers are comparable across machines, and recall@k is a meaningful metric at scale.
2. Build a BruteForceIndex (F32) over the same vectors and compute the exact top-10 for every query once — the ground truth the HNSW approximates.
3. For each (M, ef_construction) build the graph once and sweep ef_search (a per-query knob — no rebuild). For quantization, build F32/F16/Int8 at the recall-frontier graph.
4. Record recall@10, mean + p99 search latency (µs), build time (s), and the true in-memory footprint, extrapolated to 1M vectors.

The corpus must be clustered, not uniform-random

A subtle but decisive measurement point: uniform-random high-dimensional vectors are a pathological ANN benchmark. By concentration of measure, every pair of random unit vectors in 1536-D sits at cosine ≈ 0, so beyond a tiny perturbation a query's true top-10 is an arbitrary draw from a thick equidistant shell — recall@10 then measures impossible tie-breaking, not index quality, and it gets worse as the corpus grows (the shell thickens). Measured on uniform-random data, recall@10 fell from ~0.97 at 10k to ~0.54 at 100k — a corpus-size artifact, not an index regression.

Real text/image embeddings instead live on a low-dimensional manifold with clusters: a point's neighbours are its cluster-mates, distinctly closer than the bulk. The harness therefore builds a Gaussian mixture (--clusters, --spread-milli; default ~100 points/cluster, spread 0.5 ⇒ intra-cluster cosine ≈ 0.89, inter ≈ 0) — the shape related/for_you queries actually run against, where recall@10 is a meaningful, scale-stable metric. All numbers below use it.

Results — HNSW parameter sweep (1536D, F16, 100k clustered corpus, measured)

M / ef_c / ef_s	recall@10	mean (µs)	p99 (µs)	build (s)	mem/1M (GB)
16 / 400 / 128	0.9970	670	1018	19.7	5.21
16 / 400 / 200	0.9970	1129	1394	19.7	5.21
16 / 400 / 400	0.9970	2292	2719	19.7	5.21
16 / 400 / 600	0.9970	3477	4255	19.7	5.21
24 / 400 / 128	0.9985	1074	1299	39.0	5.53
24 / 400 / 200	0.9985	1812	2239	39.0	5.53
24 / 400 / 400	0.9985	3536	4088	39.0	5.53
32 / 400 / 200	0.9975	2331	2857	42.5	5.53

Every point clears recall@10 ≥ 0.95 and p99 ≤ 10ms (raw single-thread ANN cost). Two readings:

• ef_search is the latency lever, and recall saturates early. At M=16, recall is already 0.997 at ef_s=128 (p99 1.0ms) and does not improve with a wider beam — only latency grows (ef_s=600 → p99 4.3ms). So the cheap operating point is ef_s≈128; the per-query override lets a latency-sensitive caller pick it without rebuilding.
• M=24 buys ~0.15% recall for ~1.6× latency and 2× build — worth it only when the last fraction of recall matters. M=32 is not better than M=24 here.

Results — quantization sweep (1536D, M=24, ef_c=400, ef_s=400, 100k, measured)

quantization	recall@10	mean (µs)	p99 (µs)	mem/1M (GB)
F32	0.9985	3642	5126	10.53
F16	0.9960	3569	4680	5.53
Int8	0.7150	1389	1678	3.03

• F16 is the right default: it costs only 0.25% recall vs F32 (0.9960 vs 0.9985) for half the memory — validating the "<1% recall loss" claim at 1536-D, not just the 128-D it was previously checked at.
• Int8 is NOT viable at 1536-D as a drop-in: recall collapses to 0.715 (a 28% loss). The memory saving (3.0 vs 5.5 GB/1M) is real but does not justify losing a quarter of the neighbours; Int8 would need quantization-aware scaling before it is usable. The roadmap's "Int8 ≈ 1.8 GB/1M as a RAM fallback" option is therefore rejected at this dim on accuracy grounds.

Memory note. The per-1M figures are the true USearch footprint (memory_usage(), graph links + quantized vectors), not a vectors-only estimate. F16 ≈ 5.2 GB/1M, not the ~3.4 GB the roadmap projected from 1536 × 2 bytes alone — the ~1.8 GB difference is the HNSW proximity graph (M=16). Size pods at ≈ 5.5 GB/1M for F16, ≈ 10.5 GB/1M for F32.

Live cross-check (real engine, HTTP)

The standalone grid above measures the index in isolation. The authoritative end-to-end number is tidal-stress --verify-recall against a real server: at 20k/1536-D (clustered), the engine's /vector_search returned recall@10 = 1.0000 vs the brute-force cosine oracle at the default beam and at --recall-ef-search 400 — confirming both the index quality and that the per-query ef_search is honored over the wire. (The harness's end-to-end p99 of ~19 ms there is HTTP round-trip + 1536-float (de)serialization on one laptop, not ANN — the raw search is ~1 ms per the grid; it is a single-node wire cost, not an index limit.)

Decision

Default stays M=16, ef_construction=400, ef_search=200, F16 — the grid validates it at the production shape: recall@10 0.997, p99 ≈ 1.4 ms raw ANN at 100k/1536-D clustered, 5.2 GB/1M. No change is warranted.

• Latency-sensitive reads can drop to ef_search=128 per request (recall still 0.997, p99 ≈ 1.0 ms) via the per-query override.
• Maximum recall (e.g. an offline eval) can use M=24, recall 0.9985, at ~1.6× latency — but M=24 is not the default because M=16 already clears G2 (≥0.95) and G1 (≤10 ms) with wide margin.
• F32 only if an application cannot tolerate the 0.25% F16 gap; doubles RAM.
• Int8 rejected at 1536-D (recall 0.715).

Per-query ef_search (m12p3: now honored)

M and ef_construction are graph properties fixed at build time. ef_search is a per-query knob: a low-latency query can pass ef_search=64 and a high-recall query ef_search=400 against the same shared index, concurrently.

Before m12p3, UsearchIndex accepted ef_search for trait compliance but ignored it (logging a warning) because USearch 2.24 has no per-call beam argument — the only knob is the index-global change_expansion_search. m12p3 makes the override real and race-free via an RwLock epoch guard (UsearchIndex::with_expansion): searches that agree on ef_search run in parallel under a shared guard; only a query that changes the live beam width takes the exclusive guard for its (set, search) window. The override reaches the wire through vector_search_items(.., ef_search) and the /vector_search ef_search field, and tidal-stress --verify-recall --recall-ef-search sweeps it open-loop.

The ef_search axis in the table above is the recall/latency trade this knob now controls per request — higher ef_search recovers more true neighbours at a latency cost, exactly as the HNSW theory predicts (spec 07-vector-retrieval.md).

Brute-force → HNSW crossover (m12p3: dimension-aware)

The brute-force backend (exact, used for small slots) scans every vector under a read lock, at count × dim cost. A single fixed crossover is wrong: 10,000 vectors is a 1.3M-FMA scan at 128-D (sub-ms) but a 15.4M-FMA scan at 1536-D (tens of ms, under a lock that also blocks writers). usearch_min_vectors(dim) now scales the crossover so a brute-force scan stays within ~4M FMAs: ≈10,000 at/under 128-D (unchanged), ≈2,600 at 1536-D. Above the crossover a slot is the production HNSW; below it stays exact. See storage/vector/registry.rs.

Regression Guard

tidal/tests/vector_usearch.rs:

• recall_at_10_above_threshold — default config (M=16, ef_c=400) recall@10 > 0.95.
• usearch_per_query_ef_search_is_honored — a wide per-query ef_search recovers strictly more true neighbours than a starved beam (proves the override is live).

Reproduce

# Production shape on one laptop (oracle ≈ 0.6 GB at 100k/1536D):
cargo run --release --example ann_grid_search -- \
    --corpus 100000 --dim 1536 --queries 200 --k 10

# 1M shape (oracle ≈ 6 GB; run on the k3s node):
cargo run --release --example ann_grid_search -- \
    --corpus 1000000 --dim 1536 --queries 200 --k 10