tidaldb/tidal-server
jordan da736b8eb2 fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail
A follower that latched `reseed_required` from a genuine `snapshot-required`
refusal could clear its own marker ~200ms later and so never run the boot
reseed that was the only way to close the gap. With
`replication.reseed_self_restart: true` it exit-looped: latch -> clear ->
exit(0) -> boot with no marker -> re-latch. Production tidaldb-0 did this 196
times in 21h on 2026-08-20 while the cluster ran on 2 of 3 voters.

`clear_stale_reseed_marker_if_caught_up(applied >= leader_last_seq)` compared
the applied frontier against the LEADER'S TAIL and documented the invariant "a
node genuinely behind a COMPACTED gap never reaches caught_up". That is false:
on a quiet shard any node meets the leader's tail, including one missing
committed history it can never refetch. The clear also reset
`tidaldb_cluster_reseed_required`, so the gauge flapped 1->0 every 30s and
`TidalDBClusterReseedPending` (`== 1 for 10m`) could never fire - the code path
that broke the reseed also erased the signal that would have reported it.

The discharge decision now belongs to the marker. `ReseedMarker::discharged_by`
requires a stream-dischargeable reason AND an applied frontier that reached the
marker's own `from_seqno` - the very entry whose absence latched it. A compacted
gap can never satisfy that, so the reseed runs; a node merely behind a shippable
tail satisfies it as soon as the stream serves that entry, so the m12
false-alarm self-heal still works (and now clears sooner, since it no longer
waits to meet a moving leader tail).

The two conditions previously shared `ReseedReason::SnapshotRequired`, so reason
alone could not discriminate. The term-join arm's `frontier > baseline` case
deliberately sets `from_seqno = baseline`, BELOW the node's own frontier, so a
bare `applied >= from_seqno` would discharge it instantly - it holds divergent
post-baseline data only a snapshot can discard. It gets its own never-lag-
dischargeable reason, `DivergentPostBaseline = 3`. Adding a discriminant is the
sanctioned forward-only extension; a downgrade that meets one refuses to decode
it, per the existing kind-3/kind-4 precedent.

The election-won call site passed a hardcoded `true`; it now passes the leader's
durable flushed frontier (`applied_seqno` never advances on a leader), and a
`DivergentPostBaseline` node is not campaign-suppressed so it can reach there.

Tests: three deterministic predicate tests pinning the incident's exact seqnos
(13540653 vs earliest-available 13540657), the false-alarm discharge, and the
never-discharge of every structural reason.

Pre-existing and NOT introduced here: cluster_reseed's
`mp_follower_reseeds_via_snapshot_after_compaction` and
`mp_graceful_rolling_restart_under_load_no_reseed` fail on baseline main
(verified by stashing this change). The first is the owner-test for this exact
mechanism - its leader compaction no longer forces a `snapshot-required`, so it
never reached the clear path and never guarded it. Tracked separately.
2026-08-20 22:23:35 -06:00
..
benches feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
config feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
src fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail 2026-08-20 22:23:35 -06:00
tests fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
Cargo.toml feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00