From 0919b0a4bf946587a660a88d2c975d2147baaa5a Mon Sep 17 00:00:00 2001 From: jx12n Date: Fri, 19 Jun 2026 01:46:46 -0600 Subject: [PATCH] fix(m12): seed-join learner auto-promotes after a snapshot install (report the caught-up frontier on the heartbeat) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes a regression the reseed-loop fix (c8ea05b "Fix 1") introduced: a node that seed-joins and converges via a SNAPSHOT INSTALL never auto-promotes Learner -> Voter. It catches up fully (applied == leader frontier, lag 0) but sits a Learner forever; `mp_seed_join_snapshot_catchup` caught it (base 580142d passes, c8ea05b on fails — bisected). Root cause: the leader's durable per-peer `learner_mark` (which the auto-promotion gate reads: `flushed - learner_mark <= learner_promote_lag`) advances ONLY from a follower frontier-report, which the receiver emits AFTER applying a streamed event. Before Fix 1 a joiner seeded its frontier from `last_wal_seq()` (0 on the empty WAL a checkpoint restore leaves), so it re-pulled the whole log from seqno 1 and THOSE stream applies emitted the reports that advanced `learner_mark`. Fix 1 seeds the frontier to `snapshot_seq` to stop the prod reseed loop, so the leader has nothing to ship, no stream applies, and the joiner never tells the leader it is caught up. Fix: in the m12p5 heartbeat idle-readiness drive (`note_leader_frontier_for_readiness`) report this node's caught-up frontier back to the leader via the existing `ReportApplied` channel. The heartbeat flows on an idle cluster and carries the current term, so the report is both recurring (survives the join/registration race) and term-correct (the term-checked `update_peer_for_term` fold accepts it — a boot-time report stamped term 0 does not). SCOPED to a Learner: a Voter's frontier already reaches the leader via ship-acks and DOES feed `compute_commit`, so folding one off the heartbeat could perturb the same-term commit gate (Raft fig-8); a learner mark never feeds `compute_commit`, so this is provably commit-safe. `notify_applied` dedups, so a steady follower never spams. Verified: mp_seed_join_snapshot_catchup PASS (joiner promotes, 4-voter quorum); safety preserved — mp_quarantined (divergent quarantine+reseed, no wipe), mp_graceful_rolling_restart_under_load_no_reseed (0-reseed), the failover oracle (no false quarantine), tidal-server lib 154/154, engine durability 4/4, clippy -D clean. The 5600-item voter-reseed e2e are inert to this Learner-scoped change (host RAM cannot run them locally; verified live). --- tidal-server/src/cluster/node.rs | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tidal-server/src/cluster/node.rs b/tidal-server/src/cluster/node.rs index e10f68e..ac992c5 100644 --- a/tidal-server/src/cluster/node.rs +++ b/tidal-server/src/cluster/node.rs @@ -3379,6 +3379,34 @@ impl ShardReplica { .lag_gauge() .update_leader_seqno_for(leader_shard, leader_last_seq); let applied = self.applied_for_leader_shard(leader_shard); + // m12 seed-join promotion fix: report this follower's caught-up frontier + // BACK to the leader on the heartbeat path. The leader's durable per-peer + // `learner_mark` advances ONLY from a follower frontier-report, which the + // receiver emits after APPLYING a streamed event. A node that converged via + // a SNAPSHOT INSTALL (frontier seeded at boot, the leader's WAL already + // covering it) has nothing to stream-apply, so it never tells the leader it + // is caught up — and the auto-promotion gate (`flushed - learner_mark <= + // learner_promote_lag`) strands a seed-joined Learner forever (regression: + // `mp_seed_join_snapshot_catchup`; before Fix 1's frontier-seed the joiner + // re-pulled from seqno 1 and THOSE applies emitted the reports that promoted + // it). The heartbeat flows on an IDLE cluster and carries the CURRENT term, + // so this report is both recurring (survives the join/registration race) and + // term-correct (the term-checked `update_peer_for_term` fold accepts it, + // unlike a boot-time report stamped term 0). `notify_applied` dedups per + // shard (only an ADVANCED frontier pushes), so a steady follower never spams. + // Safe: it reports this node's TRUE applied frontier (never above what it + // durably holds, so it cannot over-credit a not-caught-up node), and learner + // marks never feed the quorum commit index — they gate promotion only. + // SCOPED to a LEARNER: a Voter's frontier already reaches the leader via + // the ship-ack path, and a voter mark DOES feed `compute_commit`, so folding + // one off the heartbeat could perturb the same-term commit gate (Raft fig-8); + // a learner mark never feeds `compute_commit`, so this is provably + // commit-safe and is exactly the signal the auto-promotion gate consumes. + if applied > 0 + && self.membership.self_role() == Some(tidaldb::wal::format::MemberRole::Learner) + { + self.transport.notify_applied(leader_shard, applied); + } if (self.install_boot || self.seed_joiner) && !self.converged.load(Ordering::Acquire) { self.note_lag_for_readiness(leader_last_seq.saturating_sub(applied)); }