# M12 cluster deploy + recall findings (k3s, 1536-dim)

Date: 2026-06-15. Cluster: orchard9-k3sf, namespace `tidaldb-cluster`, 3 nodes
(4 vCPU / 16 GiB each), local-path PVCs. Image built from HEAD `8e39ee1` + the
working-tree T4-TLS fixes. **Goal: deploy M12 to the real cluster and get the
real recall@10 + read p99 at the 1536-dim production shape** (the m11p6 image
404'd `/vector_search`).

## Headline result (real, live cluster)

100k items / 1536-dim, recall@10 vs a brute-force cosine oracle (1000-query pool),
`/vector_search` pure k-NN against one node:

| metric | value | gate | verdict |
|--------|-------|------|---------|
| recall@10 | **0.9869** | ≥ 0.95 | **MET** |
| read p99 @ knee | **8.71 ms** @ 200 rps | ≤ 10 ms | **MET (at 100k)** |
| read knee (single node) | **~200 rps** | — | saturates fast |

Beyond the knee a single node sheds/errors hard: 62% error at 500 rps (achieved
267), 100% at 1000+. 1536-dim k-NN reads saturate one node at ~250 rps.

Honest scope:
- This is **100k, not 1M**. The G1 gate is p99 ≤ 10 ms @ **1M**/1536-D — still
  unproven; at 1M the candidate set grows and p99 rises. 100k clears it at 8.71 ms.
- recall 0.9869 is **below the standalone-laptop 0.9997** because the measured
  node's HNSW carries **tombstones from the upsert-reseed** (remove+add, below) —
  a freshly-built index has no tombstones and recalls higher. Re-measure on a
  fresh index for the canonical number.
- Measured against a **single healthy replica** with the corpus already resident
  (`--skip-seed`), because the cluster's **write path is unstable at 1536-dim**
  (below) — a full clean seed+query could not complete.

## Bug found + fixed: USearch insert was not an upsert (reseed wedge)

`VectorIndex::insert`'s contract is "Replaces any existing vector with the same
ID." The USearch backend (`tidal/src/storage/vector/usearch_index.rs`) violated
it: it computed `is_new` but then **unconditionally `add`**, which a `multi:false`
index rejects with `Duplicate keys not allowed in high-level wrappers` whenever
the key exists.

Impact: a reseeding follower's post-snapshot WAL replay re-applies embeddings the
snapshot already loaded. Every overlapping embedding failed `add` →
`WAL blob replay failed` → applied_events never advanced past the snapshot
frontier → the catch-up receiver deadlocked on a backlog it could not apply
(`inbound channel full` forever). With all three replicas behind the compaction
horizon simultaneously, none would self-restart (quorum protection) → the cluster
was unrecoverable.

Fix: make `insert` a true upsert — `remove` the existing key, then `add`. Added a
regression test (`usearch_insert_replaces_existing_key`). Verified in production:
operator-forced rolling reseed (followers first from the ahead leader, leader
last) completed with `dupkey=0` and converged 3/3 to lag=0, no data loss.

Tradeoff: `remove`+`add` tombstones the old slot, so a reseed leaves the index
tombstoned (the recall dip above). A periodic compaction/rebuild reclaims it.

## Open upstream issues (NOT bandaided — flagged for the devs)

1. **1536-dim write path destabilizes the cluster.** The recall seed's quorum
   write-burst (100k embeddings) trips the leader's partition detector: it marks
   the followers `partitioned:true, reachable:false` (the followers actually
   applied the data and self-report healthy) → leader believes it lost quorum →
   writes stall AND reads on the leader time out (HTTP 408, the cross-region
   gather waits on followers it thinks are gone). The view does not self-heal;
   only a leader restart + re-election clears it. Root cause is almost certainly
   the replication ship/ack path not keeping pace with 1536-dim apply (each apply
   is an HNSW insert at ef_construction=400), tripping heartbeat/ack timeouts.
   **This blocks ingesting the production-shape corpus and is the top blocker to
   "M12 production-ready at 1536-D."**

2. **HNSW index is rebuilt on every boot, not persisted.** Open logs
   `using USearch (HNSW) backend ... count=100000` then spends ~5.5 min
   single-core rebuilding the graph before binding HTTP. 1M would imply a ~50–70
   min boot. The long outage is what let the WAL compact past restarting nodes
   and triggered the reseed cascade in the first place. Persist/mmap the graph so
   boot is a load, not a rebuild. (Mitigation applied: startup probe budget
   raised to 20 min; pod memory limit 2Gi→4Gi — 100k/1536 peaks ~1.9 GiB.)

3. **The 3-group `shards:` topology runs as a single shard at runtime.**
   `/cluster/status` reports one shard (shard 0, RF=3) on every node despite the
   topology's `shards:` block. Relevant to the T5 sharded-throughput gate — the
   sharding is not actually engaging.

## Deploy mechanics (for repeat)

- Cross-compile on macOS-arm64 → x86_64-linux (GCC 15.2 = glibc 2.41 → trixie
  runtime; binary NEEDs `libmvec.so.1`, absent on bookworm). Runtime image
  `debian:trixie-slim` + `libstdc++6`/`libgcc-s1`, built via the `amd64builder`
  docker-container buildx builder (QEMU). Pushed to `registry.threesix.ai`.
- Two-tier PKI (`k8s/cluster/certs.yaml`, working-tree) applied cleanly: `ca.crt`
  is now a real CA (CA:TRUE); leaf carries the wildcard SAN + ready-only client
  Service SAN (needed for T4 scale-to-5).
- Deployed image: `registry.threesix.ai/tidal/server:m12-rc2`
  (`sha256:1cfa30e5938002a881545b60f21cdfa50af6f87be8190d1ed4b1ca5a83d5bc3b`) =
  m12 + the upsert fix.

## rc3 update (2026-06-15): three fixes + verification

`m12-rc3` (`sha256:bceadce5419d5fcd78d3491f9d53cbd4055ee131f5ec7e9efa75b0f36474d1f7`)
= rc2 + three changes (built/deployed; combined `cargo check` green; per-area
unit/chaos tests pass):

1. **G1 index tuning — DONE & verified at 100k.** Grid search (in-process, clean
   index) at 1536-D: `ef_search` 200→64 holds recall@10 0.998 at ~half the p99
   (2434µs→1221µs); Int8 disqualified (recall collapses to ~0.71). Default set:
   `ANN_DEFAULT_EF_SEARCH=64` (candidate_gen.rs). End-to-end on the cluster
   (100k/1536, follower target): read-knee moved **200→500 rps**, p99 **7.06ms**,
   recall@10 **0.967** (tombstone-depressed; clean index ≈0.998) — **G1+G2 MET at
   the 100k knee**. 1M still unproven (blocked by the ingest issue below).

2. **False-partition status view — FIXED & verified.** Under the 100k/1536 write
   burst, `partitioned_peers` stayed **0** throughout (previously the leader
   falsely flagged followers partitioned). The detector now uses a gRPC
   last-contact liveness signal; genuine-partition detection intact (chaos tests
   pass).

3. **HNSW persist-on-boot — IMPLEMENTED, NOT yet effective on this cluster.** Two
   real gaps found in verification: (a) the save is wired into `Db::shutdown_inner`
   but the k8s SIGTERM path doesn't reach it — no `{data_dir}/vector/` file
   appeared after a `kubectl delete pod` (fix in progress: wire the save into the
   cluster serve SIGTERM shutdown); (b) this cluster reseeds on almost every
   restart (term-mismatch / accumulated PVC churn), and the reseed path rebuilds
   from the snapshot's raw vectors, bypassing graph-load. Both must be closed for
   the persist fix to pay off here.

### NEW blockers surfaced by rc3 verification (the real 1536-D ingest wall)

The false-partition fix removed the *view* symptom, but the underlying defect —
**the leader's HTTP/gather plane degrades under 1536-D load** — has two more faces
that still block production-shape ingest:

- **Write-forward timeout under burst.** Seeding 100k to a *follower* target: the
  follower→leader HTTP forward (`/items`,`/embeddings`) succeeds ~1950 writes then
  every forward times out (`leader unreachable ... 503, latency 15000ms`) and the
  seed stalls. On an IDLE cluster the same forward takes ~47ms — it is purely
  load-induced HTTP-plane saturation on the leader. (Fix in progress.)
- **Leader read-gather hang.** Even idle, `/vector_search` on the **leader**
  times out (HTTP 000 @ 8s) while **followers** serve it in ~50ms — the leader's
  cross-region/shard read gather hangs. Reads must currently be sent to followers.

Net: at 1536-D the leader cannot ingest a sustained quorum write load nor serve
vector reads; followers serve reads fine. This is the top blocker to T4/T5/1M and
to a clean (tombstone-free) recall number. Root cause is the apply/HTTP-plane
contention; the status-view fix was necessary but not sufficient.

## rc4 (2026-06-15): ingest admission-control + SIGTERM bounded-drain

`m12-rc4` (`sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193`)
adds the two gap fixes:
- **Ingest admission control + forward retry (fixes the write-forward timeout).**
  Leader `/items`+`/embeddings` apply now goes through the bounded `ClusterWritePool`
  (was unbounded `spawn_blocking`) → sheds as fast 429+retry_after instead of
  timing out; the follower→leader forwarder retries 3× honoring 429/backoff within
  a 28s budget (< the 30s route timeout). Idempotent upserts make retry safe. 147
  tidal-server tests pass.
- **SIGTERM bounded-drain (fixes the persist-save not running).** The cluster
  `with_graceful_shutdown` blocked forever on sibling keep-alive connections →
  60s grace → SIGKILL → no `Drop` → no graph save. rc4 adds a 15s drain deadline
  + an `&self`-reachable deterministic close (`ShardReplica.db` → `ArcSwapOption`)
  so `checkpoint_embedding_graphs` always runs. Real multi-process SIGTERM e2e test
  asserts the shutdown checkpoint is written.

### STILL-OPEN catch-up/term bugs (surface under restart churn; NOT yet fixed)

The rc4 rollout exposed two more pre-existing replication bugs that make every
rolling restart a slog on these (heavily-churned) PVCs — distinct from anything
the four fixes above address:
- **Term-gate rejects same-term catch-up chunks.** A reseeding/catching-up node
  pulls a catch-up stream and the chunks are refused:
  `catch-up chunk refused by the term gate ... chunk_term=10 rejection=Stale { current_term: 10 }`
  — `chunk_term == current_term` should NOT be stale, but it is, so the pull
  retry-loops forever (the node never converges via the stream). Workaround:
  restart the node so it takes the snapshot-install path instead of the stream.
- **Reseed-on-restart churn.** A node restart that coincides with a term bump
  (e.g. the leader restarting during a rolling update) term-mismatches on rejoin
  and does a full snapshot reseed (rebuild-from-snapshot, ~5-7min) rather than a
  cheap WAL catch-up — even when only a few events behind. This both slows every
  rollout AND bypasses the new HNSW graph-load (reseed rebuilds from raw vectors).

These two are the next upstream targets: a correct term-gate (accept `==` term)
plus a streaming catch-up that actually converges would remove the reseed churn,
let restarts be fast clean reloads, and make the HNSW-persist fix pay off.

### rc4 cluster verification (2026-06-15)

- **Ingest admission-control + forward-retry — VERIFIED.** The 100k/1536 seed
  directed at a FOLLOWER (forwards to leader) now commits **steadily** (leader
  applied climbed monotonically ~52k events over 10 min) with **zero
  `forward to leader failed` 503s**, **zero false partition** (`partitioned`
  stayed 0), and only brief graceful 429 backpressure — exactly the design. The
  pre-fix build hard-stalled at ~1500 writes in 90s. (Throughput is slow only
  because it upserts the existing corpus; correctness/liveness is the point.)
  This unblocks the 1536-D ingest path for T4/T5/1M.
- **HNSW persist-on-boot — NOT yet effective on the cluster.** Despite the rc4
  SIGTERM bounded-drain fix (which makes the shutdown path RUN — the e2e test
  asserts `checkpoint.meta` is written), a clean follower restart on the real
  1536-D cluster still showed `using USearch backend` (rebuild) and **no
  `{data_dir}/vector/` file** — the graph save does not materialize via the
  SIGTERM→bounded-drain→`ArcSwap` close→`Drop`→`shutdown_inner` path, even though
  Agent's direct-`shutdown()` unit test (dim=4000) writes the file. This is a
  reconciliation gap between the bounded-drain/ArcSwap-close and the
  graph-save-in-`shutdown_inner` (Drop may not run the full shutdown body), on
  top of the reseed-churn that would bypass graph-LOAD anyway. Implemented +
  unit-tested, but not production-verified.

### Net state after the fan-out (G1 + 2 cluster fixes)

| item | status |
|------|--------|
| G1 ef_search=64 tuning | **DONE** — verified 100k (knee 500 rps, p99 7.06ms, recall 0.967≥0.95); 1M unproven |
| Fix #1a false-partition view | **DONE** — verified (partitioned_peers=0 under the write burst) |
| Fix #1b ingest saturation (forward-503) | **DONE** — verified (seed commits steadily, 0 forward-503) |
| Fix #2 HNSW persist | implemented + unit-tested; **cluster save doesn't materialize** (open) |
| term-gate same-term `Stale` reject | **open** (workaround: restart→snapshot-install) |
| reseed-on-restart churn | **open** (term-mismatch → full snapshot reseed) |

Deployed: `m12-rc4`
(`sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193`).