The live rc3 deploy on tidaldb-2 (which hosts all 3 shards) revealed a second
loop the install-only term-marker synthesis (prior commit) does not reach: a
shard that was reseeded in an earlier loop iteration has an EMPTY WAL
(tail_term=0) but a frontier that already COVERS the leader's baseline. On the
next boot the leader answers needed=false (its WAL covers the frontier), so the
shard never installs, never gets a synthesized term marker, and decide_join —
comparing (tail_term, frontier) with tail_term first — classifies it
ReseedRequired forever. tidaldb-2 went 35 CrashLoops -> shard 2 converged via Fix
2 but shards 0/1 kept looping (restarts still climbing, ready=false).
Two complementary fixes:
- decide_join: in the `own < prev_log` arm, a frontier at/above the leader's
baseline is CAUGHT UP (it holds every committed entry, only the term-marker
record is missing) -> Clean, not a futile reseed. A frontier short of the
baseline is genuinely behind -> ReseedRequired. Divergence is unaffected: a
future-term marker and an un-replicated leader-acked suffix both quarantine
BEFORE this arm, so the frontier-covers-baseline Clean never reaches a
divergent node (verified: mp_quarantined still quarantines).
- note_term_joined: when the WAL-tail term is stale on a clean join, durably
append the kind-3 term marker (guarded, so once per stale term — never a WAL
write per heartbeat), generalizing the install-boot synthesis to caught-up
shards that never install. Falls back to the in-memory fold on append failure
(decide_join's frontier arm keeps the node Clean regardless).
Tests: decide_join now asserts caught-up->Clean, behind->ReseedRequired,
divergent->Quarantine. Verified: cluster_reseed 4/4 (rolling restart 0-reseed x2,
quarantine reseed, failover oracle — no false quarantine), tidal-server lib
154/154, engine durability 4/4.
fe56d1bf3e