tidaldb/tidal-stress/k8s
jx12n a0399550d6 feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.

Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.

Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.

Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
2026-06-15 13:09:20 -06:00
..
recall-verify-job.yaml feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
recall-verify-skipseed-job.yaml feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
rollout-trickle-job.yaml feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
stress-job-1536-peach.yaml bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling 2026-06-13 21:40:58 -06:00
stress-job-m11p6-baseline.yaml feat(deploy+stress): m11p6/mTLS deploy fixes — HTTPS probes + TLS-aware generator 2026-06-13 18:39:30 -06:00
stress-job-t2a.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
stress-job-t2b.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
stress-job-t3.yaml feat(k8s): m11p5 cluster manifest — local-path PVCs, initContainer, T3 tooling 2026-06-12 22:01:37 -06:00
stress-job-t5.yaml feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
stress-job.yaml feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
t4-load-job.yaml fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 2026-06-14 22:41:59 -06:00
t4-seed-job.yaml fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 2026-06-14 22:41:59 -06:00