tidaldb/tidal-server/tests
jx12n a0399550d6 feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.

Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.

Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.

Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
2026-06-15 13:09:20 -06:00
..
support feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
cluster_chaos.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_cross_shard_reads.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_e2e.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_election.rs feat(m11): Raft leader election over WAL stream (m11p4) 2026-06-11 23:30:24 -06:00
cluster_faults.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_graph_persistence.rs feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
cluster_grpc.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_lifecycle.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
cluster_membership.rs fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider 2026-06-14 20:29:37 -06:00
cluster_multiproc.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_quorum.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_region.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_reseed.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
cluster_routes.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_runbook.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_security.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_sharding.rs feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
middleware.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
reseed_install.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
standalone_offload.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
standalone.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
vector_search.rs feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00