tidaldb/docs/profiling/m12-cluster-deploy-findings.md

M12 cluster deploy + recall findings (k3s, 1536-dim)

⇒ STATUS (2026-06-17): the m12p4 3-shard layer is now FIXED in rc8

Operator deployed M12 to the real 3-shard k3s cluster (rc5→rc7), clean-slate re-seeded a COMPLETE 100k/1536 corpus, then root-caused and FIXED the five m12p4 3-shard catch-up/read/reseed bugs (each adversarially verified for failover-safety before implementation). m12-rc8 (sha256:5e3e5128562d5de98e73809f468ed6654e3445fecaf486f7879d9e9c6c553cb5). Workspace lib tests green (tidal-server 147/147, tidaldb 1910/1910; compaction 17/17; S=1 reseed 3/3). The 5 fixes:

1. Bug 3 (keystone) — reseed boot-install was never shard-aware in S>1. run_boot_install ran ONCE on the PARENT /data/db, but the per-group marker

   • divergent WAL live in /data/db/shard-NNNNN/ → it found no marker, no-op'd, and the divergent shard re-quarantined + self-restarted forever (186 loops). Fix: new run_boot_install_for_region runs the install per hosted shard subdir, and run_boot_install now takes a ShardId — candidates_for resolves THAT group's per-shard grpc_addr, discover_leader appends ?shard=N — so a divergent shard heals from ITS OWN leader (the original directory-only idea would have installed shard-0's WAL/term into shard-N: cross-shard contamination, caught in verification). node::shard_subdir → pub(crate) (single formatter). main.rs calls the per-region entry. (reseed.rs, main.rs, node.rs)

2. Bug 4 — an elected LEADER never recorded its own joined_term (only the follower join_term_check set it), so cluster_promote read the elected shard as topology-era and routed /cluster/shards/{id}/transfer down the legacy fenced leg → 500. Fix: become_leader_for_term calls a new monotonic ElectionRuntime::note_self_won_term(term). (node.rs, election_driver.rs)

3. Bug 2 — boot self-heal self-pulled my_shard (a node can't serve its own stream to itself) → PeerUnreachable(self) forever. Fix: guard leader_shard != my_shard. The "never reconverges" half was Bug 3 (a quarantined node clears only via the now-working reseed). (node.rs)

4. Bug 1 — cross-shard read collapse (p99 36s, 500-storm). scatter_merge fail-fasted (?) on one shard's error and offload_read had no admission bound. Fix: scatter_merge serves the surviving groups as a degraded partial on a per-shard error (logs it; the recall harness still measures true recall); a bounded read-admission semaphore sheds 429 instead of piling onto the blocking pool into a 36s p99. (Recall recovery itself comes from Bugs 2+3 restoring the empty shards.) (node.rs, offload.rs)

5. Bug 6 (rc9) — divergent-rejoin reseed loop. Surfaced when a deposed leader (frontier AT/ABOVE the new term's baseline, with old-term data) rejoins: the decide_join→ReseedRequired/Quarantine latch used from_seqno = frontier+1, which lands ABOVE the leader's WAL tail, so wal_covers answers needed=false → the reseed install no-ops → the divergent suffix is re-detected → self-restart loop. Fix: latch from_seqno = stream_baseline (threaded through join_term_check from the heartbeat), which is <= baseline so wal_covers returns needed=true → the snapshot installs and re-baselines onto the leader's committed history (discarding the divergent suffix — uncommitted writes the cluster elected past, so no acked-write loss). m12-rc9 (sha256:818c923368ecca6e433c42a13672132e9b467c449fa23c035201fe7189f1f52b). (election_driver.rs, node.rs note_quarantined)

6. Bug 5 — WAL over-compaction forced reseed on a brief restart. Compaction reclaimed every sealed segment below the checkpoint, so a follower down across one checkpoint hit the compacted-below refusal. Fix: retain the WAL_RETENTION_SEGMENTS=4 most-recent sealed segments past the checkpoint (online + a new compact_wal_retained for shutdown); a too-far-behind follower still correctly reseeds. (tidal/src/wal/compaction.rs, tidal/src/db/lifecycle.rs)

Original handoff detail (pre-fix) retained below for context.

⇒ DEV HANDOFF (2026-06-16, pre-rc8)

Operator deployed M12 to the real 3-shard k3s cluster (rc5→rc7) and clean-slate re-seeded a COMPLETE 100k/1536 corpus. Five fixes landed in-tree (working-tree diff; rc7 image is built from them) and are verified live; the m12p4 sharded catch-up/read layer is NOT production-ready on k3s and is handed back to you.

FIXED + verified (keep these):

fix	file	what it cures
TidalDb::close_shared(&self) called explicitly from ShardReplica::shutdown	tidal/src/db/lifecycle.rs, tidal-server/src/cluster/node.rs	HNSW graph save never ran on cluster SIGTERM (relied on drop(db) hitting refcount 0; lingering request clones prevented it). Now runs.
concurrent per-shard shutdown (thread::scope) + grace 60→600s + TIDAL_SHUTDOWN_DRAIN_MS=3000	node.rs (ClusterNode::shutdown), k8s/cluster/statefulset.yaml	3 shards saved SERIALLY overran the grace → SIGKILL mid-save. Now all 3 save concurrently in-window (verified: 3× "persisted HNSW graphs to disk").
count_alive_other_voters uses the CA-trusting self.blocking_client	node.rs ~2820	reseed self-restart quorum poll built a BARE reqwest client → every TLS peer poll failed → alive_voters=0 → reseed NEVER auto-healed on a TLS cluster. Verified: now alive_voters=2, self-restart fires.
seed retry: 8→40 attempts, backoff→500ms (~12s budget)	tidal-stress/src/client.rs retry_write	a fast bulk seed silently DROPPED ~0.4–4% of the corpus when a leader shed 429. Now seeds 100000/100000.
single-shard "blocker #3" was a STALE ConfigMap (shards: commented out live)	k8s/cluster/topology-configmap.yaml (re-applied)	NOT a code bug.

OPEN — m12p4 sharded path, needs your repair (gates T5/T4/1M-on-shards BLOCKED):

1. Cross-shard /vector_search collapses even on a healthy follower with all 3 shards LOCAL + lag=0: p99 36 s, recall ~0.42 (≈ consulting one shard), 70–100% error at ≥100 rps. The local scatter-merge / read-offload path (vector_search → scatter_merge over hosted_dbs, node.rs ~7204) hangs and returns partial. This is the gate-blocker — reads are the SLA.
2. Catch-up "peer shard sN not in configuration" (tidal-net PeerUnreachable, error.rs:13): a rejoining node's per-shard gRPC pool never registers the source peer → its catch-up stream can't open → it never reconverges (seen on shards 0/2).
3. term-0 topology-era divergence trap: a leader that took term-0 bootstrap writes, hit by a term transition under the seed, gets a divergent suffix → self-restarts (correct now), but then STALLS rejoining on (2) + a term-behind refusal. Net: self-restart fires but the node can't actually rejoin the 3-shard cluster (quorum survives 2/3; it does not self-repair to 3/3).
4. 3-shard leader-transfer verb 500s for an already-elected shard (cluster_promote/promote_local routes the fenced transfer through the legacy leg, fenced at joined_term≥1).
5. WAL compacts past a briefly-restarted follower → forced reseed instead of cheap stream catch-up (periodic-checkpoint compaction is too aggressive for the rollout window).

Recommendation: the single-shard topology has a working read path (prior session: recall 0.967, p99 7 ms @ 100k) — G1/G2/T4/1M reads are provable there today. The sharded gates need (1)+(2)+(3) fixed first. Also note the rc6 grace bump to 600s makes rolling restarts slow on a loaded cluster (the graph save is the long pole); a periodic graph checkpoint (the checkpoint_graphs helper is reusable from the periodic thread) would let the grace drop back and survive SIGKILL too.

---

Date: 2026-06-15. Cluster: orchard9-k3sf, namespace tidaldb-cluster, 3 nodes (4 vCPU / 16 GiB each), local-path PVCs. Image built from HEAD 8e39ee1 + the working-tree T4-TLS fixes. Goal: deploy M12 to the real cluster and get the real recall@10 + read p99 at the 1536-dim production shape (the m11p6 image 404'd /vector_search).

Headline result (real, live cluster)

100k items / 1536-dim, recall@10 vs a brute-force cosine oracle (1000-query pool), /vector_search pure k-NN against one node:

metric	value	gate	verdict
recall@10	0.9869	≥ 0.95	MET
read p99 @ knee	8.71 ms @ 200 rps	≤ 10 ms	MET (at 100k)
read knee (single node)	~200 rps	—	saturates fast

Beyond the knee a single node sheds/errors hard: 62% error at 500 rps (achieved 267), 100% at 1000+. 1536-dim k-NN reads saturate one node at ~250 rps.

Honest scope:

• This is 100k, not 1M. The G1 gate is p99 ≤ 10 ms @ 1M/1536-D — still unproven; at 1M the candidate set grows and p99 rises. 100k clears it at 8.71 ms.
• recall 0.9869 is below the standalone-laptop 0.9997 because the measured node's HNSW carries tombstones from the upsert-reseed (remove+add, below) — a freshly-built index has no tombstones and recalls higher. Re-measure on a fresh index for the canonical number.
• Measured against a single healthy replica with the corpus already resident (--skip-seed), because the cluster's write path is unstable at 1536-dim (below) — a full clean seed+query could not complete.

Bug found + fixed: USearch insert was not an upsert (reseed wedge)

VectorIndex::insert's contract is "Replaces any existing vector with the same ID." The USearch backend (tidal/src/storage/vector/usearch_index.rs) violated it: it computed is_new but then unconditionally add, which a multi:false index rejects with Duplicate keys not allowed in high-level wrappers whenever the key exists.

Impact: a reseeding follower's post-snapshot WAL replay re-applies embeddings the snapshot already loaded. Every overlapping embedding failed add → WAL blob replay failed → applied_events never advanced past the snapshot frontier → the catch-up receiver deadlocked on a backlog it could not apply (inbound channel full forever). With all three replicas behind the compaction horizon simultaneously, none would self-restart (quorum protection) → the cluster was unrecoverable.

Fix: make insert a true upsert — remove the existing key, then add. Added a regression test (usearch_insert_replaces_existing_key). Verified in production: operator-forced rolling reseed (followers first from the ahead leader, leader last) completed with dupkey=0 and converged 3/3 to lag=0, no data loss.

Tradeoff: remove+add tombstones the old slot, so a reseed leaves the index tombstoned (the recall dip above). A periodic compaction/rebuild reclaims it.

Open upstream issues (NOT bandaided — flagged for the devs)

1. 1536-dim write path destabilizes the cluster. The recall seed's quorum write-burst (100k embeddings) trips the leader's partition detector: it marks the followers partitioned:true, reachable:false (the followers actually applied the data and self-report healthy) → leader believes it lost quorum → writes stall AND reads on the leader time out (HTTP 408, the cross-region gather waits on followers it thinks are gone). The view does not self-heal; only a leader restart + re-election clears it. Root cause is almost certainly the replication ship/ack path not keeping pace with 1536-dim apply (each apply is an HNSW insert at ef_construction=400), tripping heartbeat/ack timeouts. This blocks ingesting the production-shape corpus and is the top blocker to "M12 production-ready at 1536-D."

2. HNSW index is rebuilt on every boot, not persisted. Open logs using USearch (HNSW) backend ... count=100000 then spends ~5.5 min single-core rebuilding the graph before binding HTTP. 1M would imply a ~50–70 min boot. The long outage is what let the WAL compact past restarting nodes and triggered the reseed cascade in the first place. Persist/mmap the graph so boot is a load, not a rebuild. (Mitigation applied: startup probe budget raised to 20 min; pod memory limit 2Gi→4Gi — 100k/1536 peaks ~1.9 GiB.)

3. The 3-group shards: topology runs as a single shard at runtime. /cluster/status reports one shard (shard 0, RF=3) on every node despite the topology's shards: block. Relevant to the T5 sharded-throughput gate — the sharding is not actually engaging.

Deploy mechanics (for repeat)

• Cross-compile on macOS-arm64 → x86_64-linux (GCC 15.2 = glibc 2.41 → trixie runtime; binary NEEDs libmvec.so.1, absent on bookworm). Runtime image debian:trixie-slim + libstdc++6/libgcc-s1, built via the amd64builder docker-container buildx builder (QEMU). Pushed to registry.threesix.ai.
• Two-tier PKI (k8s/cluster/certs.yaml, working-tree) applied cleanly: ca.crt is now a real CA (CA:TRUE); leaf carries the wildcard SAN + ready-only client Service SAN (needed for T4 scale-to-5).
• Deployed image: registry.threesix.ai/tidal/server:m12-rc2 (sha256:1cfa30e5938002a881545b60f21cdfa50af6f87be8190d1ed4b1ca5a83d5bc3b) = m12 + the upsert fix.

rc3 update (2026-06-15): three fixes + verification

m12-rc3 (sha256:bceadce5419d5fcd78d3491f9d53cbd4055ee131f5ec7e9efa75b0f36474d1f7) = rc2 + three changes (built/deployed; combined cargo check green; per-area unit/chaos tests pass):

1. G1 index tuning — DONE & verified at 100k. Grid search (in-process, clean index) at 1536-D: ef_search 200→64 holds recall@10 0.998 at ~half the p99 (2434µs→1221µs); Int8 disqualified (recall collapses to ~0.71). Default set: ANN_DEFAULT_EF_SEARCH=64 (candidate_gen.rs). End-to-end on the cluster (100k/1536, follower target): read-knee moved 200→500 rps, p99 7.06ms, recall@10 0.967 (tombstone-depressed; clean index ≈0.998) — G1+G2 MET at the 100k knee. 1M still unproven (blocked by the ingest issue below).

2. False-partition status view — FIXED & verified. Under the 100k/1536 write burst, partitioned_peers stayed 0 throughout (previously the leader falsely flagged followers partitioned). The detector now uses a gRPC last-contact liveness signal; genuine-partition detection intact (chaos tests pass).

3. HNSW persist-on-boot — IMPLEMENTED, NOT yet effective on this cluster. Two real gaps found in verification: (a) the save is wired into Db::shutdown_inner but the k8s SIGTERM path doesn't reach it — no {data_dir}/vector/ file appeared after a kubectl delete pod (fix in progress: wire the save into the cluster serve SIGTERM shutdown); (b) this cluster reseeds on almost every restart (term-mismatch / accumulated PVC churn), and the reseed path rebuilds from the snapshot's raw vectors, bypassing graph-load. Both must be closed for the persist fix to pay off here.

NEW blockers surfaced by rc3 verification (the real 1536-D ingest wall)

The false-partition fix removed the view symptom, but the underlying defect — the leader's HTTP/gather plane degrades under 1536-D load — has two more faces that still block production-shape ingest:

• Write-forward timeout under burst. Seeding 100k to a follower target: the follower→leader HTTP forward (/items,/embeddings) succeeds ~1950 writes then every forward times out (leader unreachable ... 503, latency 15000ms) and the seed stalls. On an IDLE cluster the same forward takes ~47ms — it is purely load-induced HTTP-plane saturation on the leader. (Fix in progress.)
• Leader read-gather hang. Even idle, /vector_search on the leader times out (HTTP 000 @ 8s) while followers serve it in ~50ms — the leader's cross-region/shard read gather hangs. Reads must currently be sent to followers.

Net: at 1536-D the leader cannot ingest a sustained quorum write load nor serve vector reads; followers serve reads fine. This is the top blocker to T4/T5/1M and to a clean (tombstone-free) recall number. Root cause is the apply/HTTP-plane contention; the status-view fix was necessary but not sufficient.

rc4 (2026-06-15): ingest admission-control + SIGTERM bounded-drain

m12-rc4 (sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193) adds the two gap fixes:

• Ingest admission control + forward retry (fixes the write-forward timeout). Leader /items+/embeddings apply now goes through the bounded ClusterWritePool (was unbounded spawn_blocking) → sheds as fast 429+retry_after instead of timing out; the follower→leader forwarder retries 3× honoring 429/backoff within a 28s budget (< the 30s route timeout). Idempotent upserts make retry safe. 147 tidal-server tests pass.
• SIGTERM bounded-drain (fixes the persist-save not running). The cluster with_graceful_shutdown blocked forever on sibling keep-alive connections → 60s grace → SIGKILL → no Drop → no graph save. rc4 adds a 15s drain deadline
  • an &self-reachable deterministic close (ShardReplica.db → ArcSwapOption) so checkpoint_embedding_graphs always runs. Real multi-process SIGTERM e2e test asserts the shutdown checkpoint is written.

STILL-OPEN catch-up/term bugs (surface under restart churn; NOT yet fixed)

The rc4 rollout exposed two more pre-existing replication bugs that make every rolling restart a slog on these (heavily-churned) PVCs — distinct from anything the four fixes above address:

• Term-gate rejects same-term catch-up chunks. A reseeding/catching-up node pulls a catch-up stream and the chunks are refused: catch-up chunk refused by the term gate ... chunk_term=10 rejection=Stale { current_term: 10 } — chunk_term == current_term should NOT be stale, but it is, so the pull retry-loops forever (the node never converges via the stream). Workaround: restart the node so it takes the snapshot-install path instead of the stream.
• Reseed-on-restart churn. A node restart that coincides with a term bump (e.g. the leader restarting during a rolling update) term-mismatches on rejoin and does a full snapshot reseed (rebuild-from-snapshot, ~5-7min) rather than a cheap WAL catch-up — even when only a few events behind. This both slows every rollout AND bypasses the new HNSW graph-load (reseed rebuilds from raw vectors).

These two are the next upstream targets: a correct term-gate (accept == term) plus a streaming catch-up that actually converges would remove the reseed churn, let restarts be fast clean reloads, and make the HNSW-persist fix pay off.

rc4 cluster verification (2026-06-15)

• Ingest admission-control + forward-retry — VERIFIED. The 100k/1536 seed directed at a FOLLOWER (forwards to leader) now commits steadily (leader applied climbed monotonically ~52k events over 10 min) with zero forward to leader failed 503s, zero false partition (partitioned stayed 0), and only brief graceful 429 backpressure — exactly the design. The pre-fix build hard-stalled at ~1500 writes in 90s. (Throughput is slow only because it upserts the existing corpus; correctness/liveness is the point.) This unblocks the 1536-D ingest path for T4/T5/1M.
• HNSW persist-on-boot — NOT yet effective on the cluster. Despite the rc4 SIGTERM bounded-drain fix (which makes the shutdown path RUN — the e2e test asserts checkpoint.meta is written), a clean follower restart on the real 1536-D cluster still showed using USearch backend (rebuild) and no {data_dir}/vector/ file — the graph save does not materialize via the SIGTERM→bounded-drain→ArcSwap close→Drop→shutdown_inner path, even though Agent's direct-shutdown() unit test (dim=4000) writes the file. This is a reconciliation gap between the bounded-drain/ArcSwap-close and the graph-save-in-shutdown_inner (Drop may not run the full shutdown body), on top of the reseed-churn that would bypass graph-LOAD anyway. Implemented + unit-tested, but not production-verified.

Net state after the fan-out (G1 + 2 cluster fixes)

item	status
G1 ef_search=64 tuning	DONE — verified 100k (knee 500 rps, p99 7.06ms, recall 0.967≥0.95); 1M unproven
Fix #1a false-partition view	DONE — verified (partitioned_peers=0 under the write burst)
Fix #1b ingest saturation (forward-503)	DONE — verified (seed commits steadily, 0 forward-503)
Fix #2 HNSW persist	implemented + unit-tested; cluster save doesn't materialize (open)
term-gate same-term Stale reject	open (workaround: restart→snapshot-install)
reseed-on-restart churn	open (term-mismatch → full snapshot reseed)

Deployed: m12-rc4 (sha256:28d0c45e59c487cf0d5e6a574c4c9997b82c6a358d67e74f326ff2c0c0925193).

rc5 + CLEAN-SLATE 3-shard rebuild (2026-06-16)

Operator-authorized clean-slate wipe (StatefulSet + 3 PVCs deleted, kustomization re-applied) → fresh 3-shard cluster on m12-rc5 (sha256:8792b283d6c4d41c0b303afa777fa1a7583a94cca932d73580132c3a0e60e01b). This removed the accumulated-PVC-churn confound and enabled the 3-shard topology that the single-shard layout could not host in place.

rc5 = rc4 + the persist-save fix. Root cause of "Fix #2 doesn't materialize": ShardReplica::shutdown relied on drop(db) reaching TidalDb::Drop (refcount 0) to run shutdown_inner→checkpoint_embedding_graphs. Under load a request-scoped clone keeps the Arc alive, so Drop never fires and the graph is never saved. Fix: a new pub fn TidalDb::close_shared(&self) (idempotent via the closed CAS) that ShardReplica::shutdown calls EXPLICITLY on the shared handle — the checkpoint now runs regardless of refcount. Verified: m12p6_graph_persistence (save/load roundtrip) + cluster_graph_persistence (multiproc SIGTERM reaches the close) both green; native cargo check clean.

Blocker #3 (single-shard at runtime) — RESOLVED, was a stale ConfigMap. The LIVE tidaldb-cluster-topology ConfigMap had the shards: block COMMENTED OUT (the on-disk manifest had it enabled but was never re-applied). The fresh deploy re-applied it: /cluster/status now reports 3 shards, every pod carries shard-00000/01/02 subdirs, and a 100k seed hash-balances ~even across them (≈42k events/shard). NOT a code bug.

Ingest is FAST on a fresh 3-shard cluster. The 100k seed (200k writes: items + embeddings) committed at ~680 events/s aggregate, lag 0 throughout — vs the prior cluster's ~80–200/s. The old slowness was upsert-tombstoning over an existing corpus + single-shard, not a fundamental ceiling. ⇒ a 1M seed is ~25 min, not hours.

Still-open (noted, not yet fixed)

• 3-shard leader-transfer verb 500s on a fresh mixed-era cluster. On the parallel fresh boot tidaldb-0 won leadership of ALL 3 shards. POST /cluster/shards/{id}/transfer to rebalance returns 503 ("promotion target returned 500"): for a shard already in an elected term (≥1) the fenced transfer routes through the internal legacy-promote leg, which promote_local fences at joined_term ≥ 1 → 500. Operator-convenience verb (not a correctness issue); the cluster serves fine with one pod leading all shards. Deferred — the fix is in the delicate election machinery the code repeatedly warns against duplicating.
• term-gate same-term Stale + reseed-on-restart churn: to be re-evaluated on the clean cluster (hypothesis: largely PVC-churn cruft).

Fix #2 persist — root-caused on the cluster, rc6 (2026-06-16)

Restarting a follower (tidaldb-2, ~32k vectors/slot/shard) on rc5 showed the persist SAVE now runs (rc4 wrote nothing): /data/db/shard-00000/vector/ held a complete 141 MB item__content_vector.usearch.tmp — close_shared executed checkpoint_graphs. But the .tmp was never renamed to the final .usearch, and shards 1/2 had no vector/ dir at all → the boot correctly REBUILT (load_persisted_slot rejects a .tmp). Root cause: ClusterNode::shutdown saved the 3 shards serially (node.rs:4161), and a single 1536-dim slot's USearch serialize+fsync is the long pole, so 3 shards overran the 60s SIGTERM grace and k8s SIGKILLed mid-save.

rc6 = rc5 + (a) ClusterNode::shutdown now saves the hosted shards concurrently via thread::scope (each ShardReplica::shutdown is &self and touches only its own db/shard); (b) terminationGracePeriodSeconds 60→600 and TIDAL_SHUTDOWN_DRAIN_MS=3000 so the save starts promptly and has ample budget. m12-rc6 (sha256:8fc56cb8e9014812df3de81fbdb4cfed6fdecd3a19ec7c2eb836a3defebdf068). Open: a SIGKILL/crash still skips the save (boot rebuilds) — the production-robust follow-up is a periodic graph checkpoint (the checkpoint_graphs helper is already reusable from the periodic thread; not yet wired because the save is heavy and needs an interval + change-detection design).

Seed completeness — single-leader 429 saturation

The fresh parallel boot left tidaldb-0 leading ALL 3 shards (the election race; the per-shard transfer verb 500s, above). A 100k recall seed then dropped 4396/100000 items: every write funnels through tidaldb-0's ONE bounded write pool, which sheds 429 under sustained load; the stress seed retries 429 only 8× (~900ms total backoff) and then drops. Real product behavior (backpressure), but it means a single-leader cluster can't absorb a fast bulk seed. Mitigations: balanced leaders (3 pools) and/or lower --seed-concurrency. The rc6 rollout terminates tidaldb-0 LAST, so leadership fails over OFF it — a free rebalance.

rc7 (2026-06-16): self-restart auto-heal + the wedge it exposed → 2nd wipe

The rc6 rollout (the FIRST restart of a loaded 3-shard cluster) exposed three real catch-up/reseed bugs (NOT cruft — clean cluster):

1. WAL compacts past a briefly-restarted follower (shard 1: "WAL compacted below seqno 1, earliest 63873") → forced reseed instead of cheap stream catch-up. Aggravated by (3).
2. Reseed self-restart ALWAYS refused on a TLS cluster — count_alive_other_voters built a BARE reqwest::blocking::Client (no cluster CA), so every https://peer:9500/cluster/status/local poll failed TLS verify → alive_voters_excluding_self=0 → refuse (would "break quorum") → a node that needed a reseed wedged forever. FIXED (rc7): use the CA-trusting self.blocking_client. VERIFIED live: the same path now logs alive_voters_excluding_self=2 and the self-restart FIRES.
3. Shard-2 catch-up "peer shard s2 not in configuration" — the per-shard gRPC pool lacks the source peer while reseed-pending; likely downstream of (2) (incomplete group membership), to re-confirm now that (2) auto-heals.

Persist save VERIFIED on the cluster (rc6): a self-restart logged persisted HNSW graphs to disk for ALL THREE shards (shard-00000/01/02) concurrently within the grace window — the rc6 parallel-save + 600s grace works.

But: the rc7 self-restart then LOOPED on the churned cluster — a divergent term-0 suffix (tail_term=0 vs elected term=1, frontier 65108) that the boot snapshot-install did not clear → quarantine→self-restart→re-detect→loop. This is mixed-era churn (the term-0 topology-bootstrap leadership orphaning uncommitted data across a leadership change). Resolved by a 2nd operator-authorized clean-slate wipe → pristine rc7 baseline (sha256:d42fe27c4d6a63e5c5103886d4bfa25998d992c4dfd3b536f01469a1edab94d0). Lesson: commit fully (lag=0) before any leadership-changing restart; the term-0 topology era is the divergence trap. The boot-install-not-clearing-a-divergent- suffix loop is a remaining upstream item for the devs.

The 3-shard topology is NOT production-ready on real k3s (dev handoff)

With the seed-retry fix the corpus seeds COMPLETE (100000/100000 in 488.5s), but exercising the seeded 3-shard cluster surfaced that the 3-shard catch-up/read paths are broken — distinct from (and on top of) the persist / self-restart fixes, which DO work. These are the m12p4 sharding code (kind-tested only); they need upstream work before T5/T4/1M can pass on 3 shards:

1. Cross-shard /vector_search collapses under load. A recall ramp on a busy LEADER (tidaldb-0, owns 2 groups) returned p99 33,000 ms, 81–100% error, recall 0.3082 — the ≈⅓ recall is the tell that the read consulted ONE shard. Root: the read target (a leader) self-restarted mid-ramp (see 3) and, while degraded, its sibling-group dbs were unavailable so the gather fell to a remote fan-out that timed out. Reads MUST be sent to a non-leader; even then the gather is fragile.
2. Catch-up "peer shard sN not in configuration" recurs (now shard 0 AND 2): a rejoining node's per-shard gRPC pool never registers the source peer, so its catch-up stream can't open → it can't converge. This blocks clean rejoin.
3. A leader hits the term-0 divergence trap under the seed and self-restarts (exitCode 0 — the Bug-B fix firing), then STALLS rejoining on (2) plus a term-behind refusal (puller term 0 behind source's 1; rejoin). Net: a node that self-restarts to heal cannot actually rejoin the 3-shard cluster. Quorum survives (2/3), the corpus is intact on the healthy two, but the cluster does not self-repair to 3/3.

Net: the rc5–rc7 fixes (persist, parallel-save, self-restart TLS-trust, seed-retry) are real and verified. But the m12p4 3-shard catch-up + cross-shard read layer needs upstream repair before the sharded-topology gates (T5, and T4/1M on shards) can run. The single-shard topology had a working read path (prior session: recall 0.967, p99 7 ms @ 100k) — the read gates G1/G2 (and T4/1M reads) are provable there today; T5 inherently needs the (currently broken) sharded path.