tidaldb/tidal-server/tests
jordan 7450cc7ef1 fix(cluster): readiness must prove convergence, not merely lack a marker
Closes the multi-group reseed defect. `is_ready` gated convergence behind
`install_boot || seed_joiner`, so a plain restarted voter fell straight through to
ready — admitted to the client VIP before it had learned the leader's frontier,
let alone caught up. The doc comment called that intentional ("keeps today's
behavior"). It is the same anti-pattern as the marker-discharge bug: asserting
health from ABSENCE of bad news.

`lag_events` could not contradict it. Lag is `leader_seqno - applied`, an unsigned
subtraction against a gauge that reads 0 until the frontier is known, so a node
that has learned nothing computes 0 - 0 = 0 and looks perfectly caught up. Both
halves together are how a PVC-wiped tidaldb-0 entered the VIP with an EMPTY corpus
and how the repro node reported all groups clean while missing items:

  shard 0: applied_events 24, lag_events 0
  shard 1: applied_events 14, lag_events 0
  shard 2: applied_events  0, lag_events 0, leader null

after 5600 items were written.

Now: convergence is required for EVERY boot, `note_lag_for_readiness` takes the
leader frontier and refuses to latch on a zero (no information is not
convergence), and it is driven on every boot rather than only joiner boots — the
heartbeat carries the frontier, so this works on an idle cluster (m12p5).
`reseeding` becomes `!converged` for all boots, which also makes the status field
mean what it says.

Only ESTABLISHED leadership self-certifies. The first cut tested
`current_leader()`, which is seeded from the TOPOLOGY FILE — and in a sharded
topology group `s` names node `s` as its term-0 leader, so a booting node
self-certified convergence for a group it merely believed it led while holding none
of its data. The election-runtime role is the honest source; the durable §1.4-1
rule is that a restart always boots a follower. The leader arm stays load-bearing
for bootstrap: a fresh cluster's leader has `last_seq == 0` and would otherwise be
permanently 503.

Gate: mp_multi_group_node_converges_after_reseeding_several_groups now PASSES and
is un-ignored. All three groups converge against real frontiers (applied 3797/3726/
3747 == leader_seqno, terms 1/5/3) and every probed item is readable, in 2 restarts
of a ceiling of 5. mp_follower_reseeds_via_snapshot_after_compaction and
mp_quarantined_node_reseeds_without_wipe still pass, so bootstrap and the
quarantine reseed are unaffected.
2026-08-21 10:41:38 -06:00
..
support feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
cluster_chaos.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_cross_shard_reads.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_e2e.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_election.rs feat(m11): Raft leader election over WAL stream (m11p4) 2026-06-11 23:30:24 -06:00
cluster_faults.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_graph_persistence.rs fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
cluster_grpc.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_lifecycle.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
cluster_membership.rs feat(m12): election-divergence-fix + soak-eval streak + release tooling 2026-06-18 13:08:53 -06:00
cluster_multiproc.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_quorum.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_region.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_reseed.rs fix(cluster): readiness must prove convergence, not merely lack a marker 2026-08-21 10:41:38 -06:00
cluster_routes.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_runbook.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_security.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_sharding.rs feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
middleware.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
reseed_install.rs fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator) 2026-06-18 21:06:18 -06:00
standalone_offload.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
standalone.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
vector_search.rs feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00