tidaldb/k8s/cluster
jx12n a0399550d6 feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.

Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.

Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.

Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
2026-06-15 13:09:20 -06:00
..
certs.yaml fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 2026-06-14 22:41:59 -06:00
kustomization.yaml feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
namespace.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
poddisruptionbudget.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
schema-configmap.yaml bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling 2026-06-13 21:40:58 -06:00
secret.example.yaml feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
service-client.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
service-peers.yaml feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
statefulset.yaml feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
topology-configmap.yaml feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00