tidaldb/docs/profiling/m12-cluster-deploy-findings.md

M12 cluster deploy + recall findings (k3s, 1536-dim)

Date: 2026-06-15. Cluster: orchard9-k3sf, namespace tidaldb-cluster, 3 nodes (4 vCPU / 16 GiB each), local-path PVCs. Image built from HEAD 8e39ee1 + the working-tree T4-TLS fixes. Goal: deploy M12 to the real cluster and get the real recall@10 + read p99 at the 1536-dim production shape (the m11p6 image 404'd /vector_search).

Headline result (real, live cluster)

100k items / 1536-dim, recall@10 vs a brute-force cosine oracle (1000-query pool), /vector_search pure k-NN against one node:

metric	value	gate	verdict
recall@10	0.9869	≥ 0.95	MET
read p99 @ knee	8.71 ms @ 200 rps	≤ 10 ms	MET (at 100k)
read knee (single node)	~200 rps	—	saturates fast

Beyond the knee a single node sheds/errors hard: 62% error at 500 rps (achieved 267), 100% at 1000+. 1536-dim k-NN reads saturate one node at ~250 rps.

Honest scope:

• This is 100k, not 1M. The G1 gate is p99 ≤ 10 ms @ 1M/1536-D — still unproven; at 1M the candidate set grows and p99 rises. 100k clears it at 8.71 ms.
• recall 0.9869 is below the standalone-laptop 0.9997 because the measured node's HNSW carries tombstones from the upsert-reseed (remove+add, below) — a freshly-built index has no tombstones and recalls higher. Re-measure on a fresh index for the canonical number.
• Measured against a single healthy replica with the corpus already resident (--skip-seed), because the cluster's write path is unstable at 1536-dim (below) — a full clean seed+query could not complete.

Bug found + fixed: USearch insert was not an upsert (reseed wedge)

VectorIndex::insert's contract is "Replaces any existing vector with the same ID." The USearch backend (tidal/src/storage/vector/usearch_index.rs) violated it: it computed is_new but then unconditionally add, which a multi:false index rejects with Duplicate keys not allowed in high-level wrappers whenever the key exists.

Impact: a reseeding follower's post-snapshot WAL replay re-applies embeddings the snapshot already loaded. Every overlapping embedding failed add → WAL blob replay failed → applied_events never advanced past the snapshot frontier → the catch-up receiver deadlocked on a backlog it could not apply (inbound channel full forever). With all three replicas behind the compaction horizon simultaneously, none would self-restart (quorum protection) → the cluster was unrecoverable.

Fix: make insert a true upsert — remove the existing key, then add. Added a regression test (usearch_insert_replaces_existing_key). Verified in production: operator-forced rolling reseed (followers first from the ahead leader, leader last) completed with dupkey=0 and converged 3/3 to lag=0, no data loss.

Tradeoff: remove+add tombstones the old slot, so a reseed leaves the index tombstoned (the recall dip above). A periodic compaction/rebuild reclaims it.

Open upstream issues (NOT bandaided — flagged for the devs)

1. 1536-dim write path destabilizes the cluster. The recall seed's quorum write-burst (100k embeddings) trips the leader's partition detector: it marks the followers partitioned:true, reachable:false (the followers actually applied the data and self-report healthy) → leader believes it lost quorum → writes stall AND reads on the leader time out (HTTP 408, the cross-region gather waits on followers it thinks are gone). The view does not self-heal; only a leader restart + re-election clears it. Root cause is almost certainly the replication ship/ack path not keeping pace with 1536-dim apply (each apply is an HNSW insert at ef_construction=400), tripping heartbeat/ack timeouts. This blocks ingesting the production-shape corpus and is the top blocker to "M12 production-ready at 1536-D."

2. HNSW index is rebuilt on every boot, not persisted. Open logs using USearch (HNSW) backend ... count=100000 then spends ~5.5 min single-core rebuilding the graph before binding HTTP. 1M would imply a ~50–70 min boot. The long outage is what let the WAL compact past restarting nodes and triggered the reseed cascade in the first place. Persist/mmap the graph so boot is a load, not a rebuild. (Mitigation applied: startup probe budget raised to 20 min; pod memory limit 2Gi→4Gi — 100k/1536 peaks ~1.9 GiB.)

3. The 3-group shards: topology runs as a single shard at runtime. /cluster/status reports one shard (shard 0, RF=3) on every node despite the topology's shards: block. Relevant to the T5 sharded-throughput gate — the sharding is not actually engaging.

Deploy mechanics (for repeat)

• Cross-compile on macOS-arm64 → x86_64-linux (GCC 15.2 = glibc 2.41 → trixie runtime; binary NEEDs libmvec.so.1, absent on bookworm). Runtime image debian:trixie-slim + libstdc++6/libgcc-s1, built via the amd64builder docker-container buildx builder (QEMU). Pushed to registry.threesix.ai.
• Two-tier PKI (k8s/cluster/certs.yaml, working-tree) applied cleanly: ca.crt is now a real CA (CA:TRUE); leaf carries the wildcard SAN + ready-only client Service SAN (needed for T4 scale-to-5).
• Deployed image: registry.threesix.ai/tidal/server:m12-rc2 (sha256:1cfa30e5938002a881545b60f21cdfa50af6f87be8190d1ed4b1ca5a83d5bc3b) = m12 + the upsert fix.

rc3 update (2026-06-15): three fixes + verification

m12-rc3 (sha256:bceadce5419d5fcd78d3491f9d53cbd4055ee131f5ec7e9efa75b0f36474d1f7) = rc2 + three changes (built/deployed; combined cargo check green; per-area unit/chaos tests pass):

1. G1 index tuning — DONE & verified at 100k. Grid search (in-process, clean index) at 1536-D: ef_search 200→64 holds recall@10 0.998 at ~half the p99 (2434µs→1221µs); Int8 disqualified (recall collapses to ~0.71). Default set: ANN_DEFAULT_EF_SEARCH=64 (candidate_gen.rs). End-to-end on the cluster (100k/1536, follower target): read-knee moved 200→500 rps, p99 7.06ms, recall@10 0.967 (tombstone-depressed; clean index ≈0.998) — G1+G2 MET at the 100k knee. 1M still unproven (blocked by the ingest issue below).

2. False-partition status view — FIXED & verified. Under the 100k/1536 write burst, partitioned_peers stayed 0 throughout (previously the leader falsely flagged followers partitioned). The detector now uses a gRPC last-contact liveness signal; genuine-partition detection intact (chaos tests pass).

3. HNSW persist-on-boot — IMPLEMENTED, NOT yet effective on this cluster. Two real gaps found in verification: (a) the save is wired into Db::shutdown_inner but the k8s SIGTERM path doesn't reach it — no {data_dir}/vector/ file appeared after a kubectl delete pod (fix in progress: wire the save into the cluster serve SIGTERM shutdown); (b) this cluster reseeds on almost every restart (term-mismatch / accumulated PVC churn), and the reseed path rebuilds from the snapshot's raw vectors, bypassing graph-load. Both must be closed for the persist fix to pay off here.

NEW blockers surfaced by rc3 verification (the real 1536-D ingest wall)

The false-partition fix removed the view symptom, but the underlying defect — the leader's HTTP/gather plane degrades under 1536-D load — has two more faces that still block production-shape ingest:

• Write-forward timeout under burst. Seeding 100k to a follower target: the follower→leader HTTP forward (/items,/embeddings) succeeds ~1950 writes then every forward times out (leader unreachable ... 503, latency 15000ms) and the seed stalls. On an IDLE cluster the same forward takes ~47ms — it is purely load-induced HTTP-plane saturation on the leader. (Fix in progress.)
• Leader read-gather hang. Even idle, /vector_search on the leader times out (HTTP 000 @ 8s) while followers serve it in ~50ms — the leader's cross-region/shard read gather hangs. Reads must currently be sent to followers.

Net: at 1536-D the leader cannot ingest a sustained quorum write load nor serve vector reads; followers serve reads fine. This is the top blocker to T4/T5/1M and to a clean (tombstone-free) recall number. Root cause is the apply/HTTP-plane contention; the status-view fix was necessary but not sufficient.

rc4 (2026-06-15): ingest admission-control + SIGTERM bounded-drain

m12-rc4 (sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193) adds the two gap fixes:

• Ingest admission control + forward retry (fixes the write-forward timeout). Leader /items+/embeddings apply now goes through the bounded ClusterWritePool (was unbounded spawn_blocking) → sheds as fast 429+retry_after instead of timing out; the follower→leader forwarder retries 3× honoring 429/backoff within a 28s budget (< the 30s route timeout). Idempotent upserts make retry safe. 147 tidal-server tests pass.
• SIGTERM bounded-drain (fixes the persist-save not running). The cluster with_graceful_shutdown blocked forever on sibling keep-alive connections → 60s grace → SIGKILL → no Drop → no graph save. rc4 adds a 15s drain deadline
  • an &self-reachable deterministic close (ShardReplica.db → ArcSwapOption) so checkpoint_embedding_graphs always runs. Real multi-process SIGTERM e2e test asserts the shutdown checkpoint is written.

STILL-OPEN catch-up/term bugs (surface under restart churn; NOT yet fixed)

The rc4 rollout exposed two more pre-existing replication bugs that make every rolling restart a slog on these (heavily-churned) PVCs — distinct from anything the four fixes above address:

• Term-gate rejects same-term catch-up chunks. A reseeding/catching-up node pulls a catch-up stream and the chunks are refused: catch-up chunk refused by the term gate ... chunk_term=10 rejection=Stale { current_term: 10 } — chunk_term == current_term should NOT be stale, but it is, so the pull retry-loops forever (the node never converges via the stream). Workaround: restart the node so it takes the snapshot-install path instead of the stream.
• Reseed-on-restart churn. A node restart that coincides with a term bump (e.g. the leader restarting during a rolling update) term-mismatches on rejoin and does a full snapshot reseed (rebuild-from-snapshot, ~5-7min) rather than a cheap WAL catch-up — even when only a few events behind. This both slows every rollout AND bypasses the new HNSW graph-load (reseed rebuilds from raw vectors).

These two are the next upstream targets: a correct term-gate (accept == term) plus a streaming catch-up that actually converges would remove the reseed churn, let restarts be fast clean reloads, and make the HNSW-persist fix pay off.

rc4 cluster verification (2026-06-15)

• Ingest admission-control + forward-retry — VERIFIED. The 100k/1536 seed directed at a FOLLOWER (forwards to leader) now commits steadily (leader applied climbed monotonically ~52k events over 10 min) with zero forward to leader failed 503s, zero false partition (partitioned stayed 0), and only brief graceful 429 backpressure — exactly the design. The pre-fix build hard-stalled at ~1500 writes in 90s. (Throughput is slow only because it upserts the existing corpus; correctness/liveness is the point.) This unblocks the 1536-D ingest path for T4/T5/1M.
• HNSW persist-on-boot — NOT yet effective on the cluster. Despite the rc4 SIGTERM bounded-drain fix (which makes the shutdown path RUN — the e2e test asserts checkpoint.meta is written), a clean follower restart on the real 1536-D cluster still showed using USearch backend (rebuild) and no {data_dir}/vector/ file — the graph save does not materialize via the SIGTERM→bounded-drain→ArcSwap close→Drop→shutdown_inner path, even though Agent's direct-shutdown() unit test (dim=4000) writes the file. This is a reconciliation gap between the bounded-drain/ArcSwap-close and the graph-save-in-shutdown_inner (Drop may not run the full shutdown body), on top of the reseed-churn that would bypass graph-LOAD anyway. Implemented + unit-tested, but not production-verified.

Net state after the fan-out (G1 + 2 cluster fixes)

item	status
G1 ef_search=64 tuning	DONE — verified 100k (knee 500 rps, p99 7.06ms, recall 0.967≥0.95); 1M unproven
Fix #1a false-partition view	DONE — verified (partitioned_peers=0 under the write burst)
Fix #1b ingest saturation (forward-503)	DONE — verified (seed commits steadily, 0 forward-503)
Fix #2 HNSW persist	implemented + unit-tested; cluster save doesn't materialize (open)
term-gate same-term Stale reject	open (workaround: restart→snapshot-install)
reseed-on-restart churn	open (term-mismatch → full snapshot reseed)

Deployed: m12-rc4 (sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193).