tidaldb/tidal
jordan e5fd19eb73 fix(vector): load a persisted slot graph by its own backend, not an assumed one
Found by running the standalone server end to end: after a clean shutdown that
logged "persisted HNSW graphs to disk (next boot loads instead of rebuilding)",
the next boot logged

  WARN persisted HNSW graph failed to load; falling back to rebuild
       slot="content_vector" error=USearch load failed: Failed to read vectors

and rebuilt the index. Every boot. Correct results, no data loss, a WARN nobody
reads - and the optimization was dead.

Cause: `checkpoint_graphs` saves whichever index the slot holds, and
`build_slot_index` holds a BRUTE-FORCE index below the dimension-aware
crossover. Both write to the same `<kind>__<slot>.usearch` path, so a small slot
persisted a `BFVI` file that `load_persisted_slot` then handed to the USearch
reader. Observed on a 6-item, 128-D store; at production scale the slot is
USearch-backed, which is why the cluster never surfaced it - but every dev,
staging, and small-tenant instance pays a full index rebuild on every start, and
that rebuild is what the 20-minute startup probe budget exists for.

Fix: sniff the file's magic (brute-force `MAGIC` is now `pub(crate)` so the
reader and writer cannot drift) and dispatch to the matching loader. A
brute-force graph is still rejected when `expected_count` has grown past the
crossover, so the rebuild upgrades the backend to HNSW rather than pinning a
linear scan forever. Log lines now say "vector graph" and name the backend
instead of claiming HNSW for both.

Tests: `brute_force_slot_graph_round_trips_through_registry_persistence` covers
the case that was broken (the existing round-trip test forces USearch, which is
why it passed throughout), and
`grown_corpus_rejects_a_brute_force_graph_so_the_rebuild_upgrades_it` pins the
upgrade path. Verified live: the same data directory that produced the WARN now
logs `loaded persisted vector graph from disk (skipped full rebuild)
backend="brute-force" count=6`, with feed, text search, and vector search all
returning the pre-restart results.
2026-08-17 17:47:55 -06:00
..
benches Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10 2026-08-03 02:16:04 -06:00
examples feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
proptest-regressions feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
src fix(vector): load a persisted slot graph by its own backend, not an assumed one 2026-08-17 17:47:55 -06:00
tests fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
build.rs feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger 2026-02-20 20:32:00 -07:00
Cargo.lock feat: M0p1 runtime skeleton, M0p2 tooling & diagnostics, m1p4 signal ledger 2026-02-20 20:32:00 -07:00
Cargo.toml Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10 2026-08-03 02:16:04 -06:00
CLAUDE.md feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00