tidaldb/docs/profiling
jx12n 727fbfcb6b fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9)
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:

1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
   visits each hosted group's own shard subdir; per-group leader discovery
   appends `?shard=N` so a divergent shard heals from its own leader (not
   shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
   `note_self_won_term` so the elected shard's `joined_term` is set and
   `cluster_promote` routes rebalances correctly (was: topology-era mis-read
   → legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
   prevents a node pulling its own stream (its stream isn't a registered peer)
   → eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
   instead of `?`-failing the whole read; bounded read-admission semaphore
   (`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
   most-recent sealed segments; online path gets the same retention clamp.
   Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
   `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
   returns `needed=true` and the snapshot installs instead of looping.

Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
2026-06-16 22:34:21 -06:00
..
hotspot-analysis.md feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9 2026-02-23 22:41:16 -07:00
m12-cluster-deploy-findings.md fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) 2026-06-16 22:34:21 -06:00
m12p4-t5-sharded-throughput.md feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
m12p5-idle-readiness-elasticity.md fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 2026-06-14 22:41:59 -06:00
scale-baselines.md feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00
signal-memory-analysis.md feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9 2026-02-23 22:41:16 -07:00
signal-rollup-eval.md feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9 2026-02-23 22:41:16 -07:00
social-scale.md feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00
tantivy-merge-tuning.md feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9 2026-02-23 22:41:16 -07:00
usearch-tuning.md feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00