tidaldb/tidal-server
jordan 7450cc7ef1 fix(cluster): readiness must prove convergence, not merely lack a marker
Closes the multi-group reseed defect. `is_ready` gated convergence behind
`install_boot || seed_joiner`, so a plain restarted voter fell straight through to
ready — admitted to the client VIP before it had learned the leader's frontier,
let alone caught up. The doc comment called that intentional ("keeps today's
behavior"). It is the same anti-pattern as the marker-discharge bug: asserting
health from ABSENCE of bad news.

`lag_events` could not contradict it. Lag is `leader_seqno - applied`, an unsigned
subtraction against a gauge that reads 0 until the frontier is known, so a node
that has learned nothing computes 0 - 0 = 0 and looks perfectly caught up. Both
halves together are how a PVC-wiped tidaldb-0 entered the VIP with an EMPTY corpus
and how the repro node reported all groups clean while missing items:

  shard 0: applied_events 24, lag_events 0
  shard 1: applied_events 14, lag_events 0
  shard 2: applied_events  0, lag_events 0, leader null

after 5600 items were written.

Now: convergence is required for EVERY boot, `note_lag_for_readiness` takes the
leader frontier and refuses to latch on a zero (no information is not
convergence), and it is driven on every boot rather than only joiner boots — the
heartbeat carries the frontier, so this works on an idle cluster (m12p5).
`reseeding` becomes `!converged` for all boots, which also makes the status field
mean what it says.

Only ESTABLISHED leadership self-certifies. The first cut tested
`current_leader()`, which is seeded from the TOPOLOGY FILE — and in a sharded
topology group `s` names node `s` as its term-0 leader, so a booting node
self-certified convergence for a group it merely believed it led while holding none
of its data. The election-runtime role is the honest source; the durable §1.4-1
rule is that a restart always boots a follower. The leader arm stays load-bearing
for bootstrap: a fresh cluster's leader has `last_seq == 0` and would otherwise be
permanently 503.

Gate: mp_multi_group_node_converges_after_reseeding_several_groups now PASSES and
is un-ignored. All three groups converge against real frontiers (applied 3797/3726/
3747 == leader_seqno, terms 1/5/3) and every probed item is readable, in 2 restarts
of a ceiling of 5. mp_follower_reseeds_via_snapshot_after_compaction and
mp_quarantined_node_reseeds_without_wipe still pass, so bootstrap and the
quarantine reseed are unaffected.
2026-08-21 10:41:38 -06:00
..
benches feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
config feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 2026-06-09 17:06:34 -06:00
src fix(cluster): readiness must prove convergence, not merely lack a marker 2026-08-21 10:41:38 -06:00
tests fix(cluster): readiness must prove convergence, not merely lack a marker 2026-08-21 10:41:38 -06:00
BUILD.bazel feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening 2026-06-07 18:29:38 -06:00
Cargo.toml feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00