tidaldb/tidal-server/src/cluster
jordan 5b3cfe59d9 fix(cluster): never boot-pull against the topology leader post-election
This is the defect that kept tidaldb-0 looping, and the per-key instrument named it
exactly. Live group 1 held:

  keys: [[0, 13540659], [1, 13540652], [2, 13540661]]

Current leader tidaldb-2 is key 2 and the group was fully converged there at
13540661. Key 1 is a STALE position left from when tidaldb-1 led the group.

`node.rs`'s follower boot self-heal pulled `shard_of_region(leader)` where `leader`
is the BOOT TOPOLOGY leader — dead config after any election, as the topology
comment itself says. For group 1 that is tidaldb-1, i.e. key 1, so the pull went
out at 13540652 + 1 = 13540653, which tidaldb-1's WAL had compacted below (earliest
13540657). Permanent `snapshot-required` → marker latch → `reseed_self_restart` →
repeat. The old comment claimed "term fencing + later election traffic rescue it";
they do not, because the refusal re-latches faster than the rescue converges.

The boot pull is now confined to the genuine topology era (durable term 0), where
the topology leader IS authoritative. Post-election, convergence is driven by the
heartbeat path (which carries the CURRENT leader's frontier and works on an idle
cluster since m12p5) and by the receiver's gap detection on real ship traffic —
both keyed to the leader actually shipping, never a historical one.

Gates: mp_follower_reseeds_via_snapshot_after_compaction and
mp_multi_group_node_converges_after_reseeding_several_groups both pass;
mp_quarantined_node_reseeds_without_wipe still passes, which is the term-0 path
this change deliberately leaves intact.
2026-08-21 11:35:31 -06:00
..
audit.rs fix(m11): review remediation + tidal-stress perf sweep + perf wave 2 2026-06-13 12:28:04 -06:00
election_driver.rs fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail 2026-08-20 22:23:35 -06:00
forward.rs feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
http_tls.rs fix(m11): review remediation + tidal-stress perf sweep + perf wave 2 2026-06-13 12:28:04 -06:00
join_boot.rs feat(m12): election-divergence-fix + soak-eval streak + release tooling 2026-06-18 13:08:53 -06:00
membership.rs feat(m11): data-plane sharding × replication (m11p6 L0-L2) 2026-06-12 23:06:41 -06:00
mod.rs docs: withdraw the pre-release "not ready for production" disclaimer 2026-07-30 19:03:34 -06:00
node.rs fix(cluster): never boot-pull against the topology leader post-election 2026-08-21 11:35:31 -06:00
reseed_restart.rs fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator) 2026-06-18 21:06:18 -06:00
reseed.rs fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator) 2026-06-18 21:06:18 -06:00
routes.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
security.rs fix(m11): review remediation + tidal-stress perf sweep + perf wave 2 2026-06-13 12:28:04 -06:00
snapshot.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
state.rs docs: withdraw the pre-release "not ready for production" disclaimer 2026-07-30 19:03:34 -06:00
topology.rs feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 2026-06-13 09:17:49 -06:00
transport.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00