fix(cluster): never boot-pull against the topology leader post-election

This is the defect that kept tidaldb-0 looping, and the per-key instrument named it
exactly. Live group 1 held:

  keys: [[0, 13540659], [1, 13540652], [2, 13540661]]

Current leader tidaldb-2 is key 2 and the group was fully converged there at
13540661. Key 1 is a STALE position left from when tidaldb-1 led the group.

`node.rs`'s follower boot self-heal pulled `shard_of_region(leader)` where `leader`
is the BOOT TOPOLOGY leader — dead config after any election, as the topology
comment itself says. For group 1 that is tidaldb-1, i.e. key 1, so the pull went
out at 13540652 + 1 = 13540653, which tidaldb-1's WAL had compacted below (earliest
13540657). Permanent `snapshot-required` → marker latch → `reseed_self_restart` →
repeat. The old comment claimed "term fencing + later election traffic rescue it";
they do not, because the refusal re-latches faster than the rescue converges.

The boot pull is now confined to the genuine topology era (durable term 0), where
the topology leader IS authoritative. Post-election, convergence is driven by the
heartbeat path (which carries the CURRENT leader's frontier and works on an idle
cluster since m12p5) and by the receiver's gap detection on real ship traffic —
both keyed to the leader actually shipping, never a historical one.

Gates: mp_follower_reseeds_via_snapshot_after_compaction and
mp_multi_group_node_converges_after_reseeding_several_groups both pass;
mp_quarantined_node_reseeds_without_wipe still passes, which is the term-0 path
this change deliberately leaves intact.
This commit is contained in:
jordan 2026-08-21 11:35:31 -06:00
parent 925a616cda
commit 5b3cfe59d9

View File

@ -996,12 +996,30 @@ impl ShardReplica {
// install, the pull must target the DISCOVERED leader's shard from the
// seeded artifact tail + 1 — NOT the boot topology leader, which for a
// reseeded ex-leader is the node itself (a self-pull never converges).
// The plain self-heal still targets the boot topology leader for a
// non-install follower boot (leadership moved? term fencing + later
// election traffic rescue it, the existing m11p2 semantics).
// The plain self-heal targets the boot topology leader — but ONLY in the
// genuine topology era (durable term 0). Post-election that field is dead
// config, and pulling against it is actively harmful:
//
// `applied_seqno` is keyed per LEADER REGION, so a group retains a position
// for every leadership it has followed. Live tidaldb-0's group 1 held
// keys [[0, 13540659], [1, 13540652], [2, 13540661]] — current leader
// tidaldb-2 (key 2) fully converged at 13540661, alongside a STALE key 1
// from when tidaldb-1 led the group. The old code pulled the topology
// leader's key regardless, i.e. key 1 from 13540652 + 1 = 13540653, which
// tidaldb-1's WAL had compacted below (earliest 13540657) → a permanent
// `snapshot-required` refusal → marker latch → `reseed_self_restart` loop.
// The comment here used to claim "term fencing + later election traffic
// rescue it"; they do not, because the refusal re-latches faster than the
// rescue converges.
//
// Post-election, convergence is driven by the heartbeat path
// (`note_leader_frontier_for_readiness`, which carries the CURRENT leader's
// frontier and works on an idle cluster since m12p5) and by the receiver's
// gap detection on real ship traffic — both keyed to the leader that is
// actually shipping, never a historical one.
if let Some((discovered_shard, recovered_tail)) = install_target {
transport.request_catchup(discovered_shard, recovered_tail + 1);
} else if !is_leader_at_boot {
} else if !is_leader_at_boot && hard_state.current_term == 0 {
let leader_shard = shard_of_region(leader);
// A node can NEVER pull its own stream from itself: only the node whose
// `source_shard == S` serves shard S's stream (server.rs returns