da736b8 replaced `applied >= leader_last_seq` with `applied >= marker.from_seqno`
and was still wrong, for the same underlying reason: the applied frontier is a
HIGH-WATER-MARK, not a contiguity proof. A term join re-bases it onto the new
leader's stream (`replication_state().advance(.., baseline + 1)`), so it leaps
across history the node never received. Any predicate built on it discharges
markers for nodes that still have a hole.
Measured, not argued. `mp_follower_reseeds_via_snapshot_after_compaction` stops a
follower at frontier 9, compacts the leader so it retains only from 15722, and
the follower's frontier is re-based to 16810. Both predicates discharge the
marker there; the node skips its reseed and then reports `lag_events: 0` while
missing 10..15721 and serving reads from a log with a hole. Production showed the
identical shape: `applied 13540660` against a marker resuming at 13540653 that no
live WAL could serve.
The marker is now discharged only on POSITIVE EVIDENCE that the stream served the
latching range: a `StreamSegments` pull that began at or below the marker's
`from_seqno` and ran to completion. New `CatchupServedSink` in tidal-net fires on
`PullOutcome::Complete`; `NodeCatchupServedSink` routes it to
`discharge_reseed_marker_if_served`. `ReseedMarker::discharged_by_served_range`
replaces `discharged_by`. The other sound discharge is unchanged: a snapshot
install replaces the data dir and takes the marker with it.
Both frontier-based call sites are gone, with the reasoning recorded where they
were. The election-won site is deliberately NOT replaced: winning proves the log
beats a quorum's under the vote restriction, which is not contiguity, so
discharging there could promote a leader with a hole.
The owner-test for this mechanism was RED ON BASELINE and is now green. It also
gained the premise assertion it never had: it used to assert only the consequence
(`reseed_required == true`), so when its fixture stopped forcing compaction it
failed 40s later looking like a follower bug. `assert_history_compacted_past` now
checks the leader actually dropped the follower's resume seq, and prints the
retained segment floors. Its content probe ("every probed offline item is
searchable on the reseeded follower") is what proves the hole is really gone.
Suite state: cluster_reseed's other tests pass individually.
mp_graceful_rolling_restart_under_load_no_reseed remains red on baseline
(pre-existing, verified by stash). mp_quarantined_node_reseeds_without_wipe and
mp_election_position_consistent_across_roles_after_failover pass alone but can
fail in-suite: this fix makes the compaction test run its full 565s reseed
instead of failing fast at 40s, which shifts timing for later tests on shared
fixed ports. Order sensitivity is pre-existing, not introduced here.
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
