tidaldb/tidal-server/tests
jordan afdda7cc0f fix(cluster): discharge a reseed marker on served evidence, never on a frontier
da736b8 replaced `applied >= leader_last_seq` with `applied >= marker.from_seqno`
and was still wrong, for the same underlying reason: the applied frontier is a
HIGH-WATER-MARK, not a contiguity proof. A term join re-bases it onto the new
leader's stream (`replication_state().advance(.., baseline + 1)`), so it leaps
across history the node never received. Any predicate built on it discharges
markers for nodes that still have a hole.

Measured, not argued. `mp_follower_reseeds_via_snapshot_after_compaction` stops a
follower at frontier 9, compacts the leader so it retains only from 15722, and
the follower's frontier is re-based to 16810. Both predicates discharge the
marker there; the node skips its reseed and then reports `lag_events: 0` while
missing 10..15721 and serving reads from a log with a hole. Production showed the
identical shape: `applied 13540660` against a marker resuming at 13540653 that no
live WAL could serve.

The marker is now discharged only on POSITIVE EVIDENCE that the stream served the
latching range: a `StreamSegments` pull that began at or below the marker's
`from_seqno` and ran to completion. New `CatchupServedSink` in tidal-net fires on
`PullOutcome::Complete`; `NodeCatchupServedSink` routes it to
`discharge_reseed_marker_if_served`. `ReseedMarker::discharged_by_served_range`
replaces `discharged_by`. The other sound discharge is unchanged: a snapshot
install replaces the data dir and takes the marker with it.

Both frontier-based call sites are gone, with the reasoning recorded where they
were. The election-won site is deliberately NOT replaced: winning proves the log
beats a quorum's under the vote restriction, which is not contiguity, so
discharging there could promote a leader with a hole.

The owner-test for this mechanism was RED ON BASELINE and is now green. It also
gained the premise assertion it never had: it used to assert only the consequence
(`reseed_required == true`), so when its fixture stopped forcing compaction it
failed 40s later looking like a follower bug. `assert_history_compacted_past` now
checks the leader actually dropped the follower's resume seq, and prints the
retained segment floors. Its content probe ("every probed offline item is
searchable on the reseeded follower") is what proves the hole is really gone.

Suite state: cluster_reseed's other tests pass individually.
mp_graceful_rolling_restart_under_load_no_reseed remains red on baseline
(pre-existing, verified by stash). mp_quarantined_node_reseeds_without_wipe and
mp_election_position_consistent_across_roles_after_failover pass alone but can
fail in-suite: this fix makes the compaction test run its full 565s reseed
instead of failing fast at 40s, which shifts timing for later tests on shared
fixed ports. Order sensitivity is pre-existing, not introduced here.
2026-08-21 00:40:06 -06:00
..
support feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild 2026-06-15 13:09:20 -06:00
cluster_chaos.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_cross_shard_reads.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_e2e.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_election.rs feat(m11): Raft leader election over WAL stream (m11p4) 2026-06-11 23:30:24 -06:00
cluster_faults.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_graph_persistence.rs fleet remediation: make the workspace gate runnable, then fix what it caught 2026-08-16 12:38:14 -06:00
cluster_grpc.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_lifecycle.rs feat(m11): membership, snapshot install, and reseed (m11p5) 2026-06-12 19:55:54 -06:00
cluster_membership.rs feat(m12): election-divergence-fix + soak-eval streak + release tooling 2026-06-18 13:08:53 -06:00
cluster_multiproc.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_quorum.rs feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline 2026-06-13 15:23:59 -06:00
cluster_region.rs feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) 2026-06-14 15:17:35 -06:00
cluster_reseed.rs fix(cluster): discharge a reseed marker on served evidence, never on a frontier 2026-08-21 00:40:06 -06:00
cluster_routes.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_runbook.rs feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 2026-06-10 14:07:33 -06:00
cluster_security.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
cluster_sharding.rs feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) 2026-06-13 18:23:43 -06:00
middleware.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
reseed_install.rs fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator) 2026-06-18 21:06:18 -06:00
standalone_offload.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
standalone.rs feat(m11): cluster security (m11p7) + perf instrumentation floor 2026-06-13 01:25:35 -06:00
vector_search.rs feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning 2026-06-14 11:07:09 -06:00