jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(m12): seed-join learner auto-promotes after a snapshot install (report the caught-up frontier on the heartbeat)

Browse Source 
Fixes a regression the reseed-loop fix (c8ea05b "Fix 1") introduced: a node
that seed-joins and converges via a SNAPSHOT INSTALL never auto-promotes
Learner -> Voter. It catches up fully (applied == leader frontier, lag 0) but
sits a Learner forever; `mp_seed_join_snapshot_catchup` caught it (base
580142d passes, c8ea05b on fails — bisected).

Root cause: the leader's durable per-peer `learner_mark` (which the
auto-promotion gate reads: `flushed - learner_mark <= learner_promote_lag`)
advances ONLY from a follower frontier-report, which the receiver emits AFTER
applying a streamed event. Before Fix 1 a joiner seeded its frontier from
`last_wal_seq()` (0 on the empty WAL a checkpoint restore leaves), so it
re-pulled the whole log from seqno 1 and THOSE stream applies emitted the
reports that advanced `learner_mark`. Fix 1 seeds the frontier to
`snapshot_seq` to stop the prod reseed loop, so the leader has nothing to
ship, no stream applies, and the joiner never tells the leader it is caught
up.

Fix: in the m12p5 heartbeat idle-readiness drive
(`note_leader_frontier_for_readiness`) report this node's caught-up frontier
back to the leader via the existing `ReportApplied` channel. The heartbeat
flows on an idle cluster and carries the current term, so the report is both
recurring (survives the join/registration race) and term-correct (the
term-checked `update_peer_for_term` fold accepts it — a boot-time report
stamped term 0 does not). SCOPED to a Learner: a Voter's frontier already
reaches the leader via ship-acks and DOES feed `compute_commit`, so folding
one off the heartbeat could perturb the same-term commit gate (Raft fig-8); a
learner mark never feeds `compute_commit`, so this is provably commit-safe.
`notify_applied` dedups, so a steady follower never spams.

Verified: mp_seed_join_snapshot_catchup PASS (joiner promotes, 4-voter
quorum); safety preserved — mp_quarantined (divergent quarantine+reseed, no
wipe), mp_graceful_rolling_restart_under_load_no_reseed (0-reseed), the
failover oracle (no false quarantine), tidal-server lib 154/154, engine
durability 4/4, clippy -D clean. The 5600-item voter-reseed e2e are inert to
this Learner-scoped change (host RAM cannot run them locally; verified live).
This commit is contained in:
jx12n 2026-06-19 01:46:46 -06:00
parent 973f073558
commit 0919b0a4bf
1 changed files with 28 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
tidal-server/src/cluster/node.rs
View File

@ -3379,6 +3379,34 @@ impl ShardReplica {
.lag_gauge()
.update_leader_seqno_for(leader_shard, leader_last_seq);
let applied = self.applied_for_leader_shard(leader_shard);
// m12 seed-join promotion fix: report this follower's caught-up frontier
// BACK to the leader on the heartbeat path. The leader's durable per-peer
// `learner_mark` advances ONLY from a follower frontier-report, which the
// receiver emits after APPLYING a streamed event. A node that converged via
// a SNAPSHOT INSTALL (frontier seeded at boot, the leader's WAL already
// covering it) has nothing to stream-apply, so it never tells the leader it
// is caught up — and the auto-promotion gate (`flushed - learner_mark <=
// learner_promote_lag`) strands a seed-joined Learner forever (regression:
// `mp_seed_join_snapshot_catchup`; before Fix 1's frontier-seed the joiner
// re-pulled from seqno 1 and THOSE applies emitted the reports that promoted
// it). The heartbeat flows on an IDLE cluster and carries the CURRENT term,
// so this report is both recurring (survives the join/registration race) and
// term-correct (the term-checked `update_peer_for_term` fold accepts it,
// unlike a boot-time report stamped term 0). `notify_applied` dedups per
// shard (only an ADVANCED frontier pushes), so a steady follower never spams.
// Safe: it reports this node's TRUE applied frontier (never above what it
// durably holds, so it cannot over-credit a not-caught-up node), and learner
// marks never feed the quorum commit index — they gate promotion only.
// SCOPED to a LEARNER: a Voter's frontier already reaches the leader via
// the ship-ack path, and a voter mark DOES feed `compute_commit`, so folding
// one off the heartbeat could perturb the same-term commit gate (Raft fig-8);
// a learner mark never feeds `compute_commit`, so this is provably
// commit-safe and is exactly the signal the auto-promotion gate consumes.
if applied > 0
&& self.membership.self_role() == Some(tidaldb::wal::format::MemberRole::Learner)
{
self.transport.notify_applied(leader_shard, applied);
}
if (self.install_boot || self.seed_joiner) && !self.converged.load(Ordering::Acquire) {
self.note_lag_for_readiness(leader_last_seq.saturating_sub(applied));
}