From aa94fd9b1fc65b4149ad31f7f0cc702677338ada Mon Sep 17 00:00:00 2001 From: jx12n Date: Sun, 14 Jun 2026 16:21:00 -0600 Subject: [PATCH] feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup --- .../m12p5-idle-readiness-elasticity.md | 131 ++++++++++++++++++ k8s/cluster/certs.yaml | 20 ++- scripts/gen-cluster-certs.sh | 11 +- tidal-net/proto/wal_shipping.proto | 11 ++ tidal-net/src/server.rs | 1 + tidal-net/src/sources.rs | 9 +- tidal-net/tests/election_rpc.rs | 14 +- tidal-net/tests/snapshot_fetch.rs | 1 + tidal-server/src/cluster/election_driver.rs | 9 ++ tidal-server/src/cluster/node.rs | 61 ++++++++ tidal-server/tests/cluster_membership.rs | 122 ++++++++++++++++ 11 files changed, 373 insertions(+), 17 deletions(-) create mode 100644 docs/profiling/m12p5-idle-readiness-elasticity.md diff --git a/docs/profiling/m12p5-idle-readiness-elasticity.md b/docs/profiling/m12p5-idle-readiness-elasticity.md new file mode 100644 index 0000000..81ed6de --- /dev/null +++ b/docs/profiling/m12p5-idle-readiness-elasticity.md @@ -0,0 +1,131 @@ +# m12p5 — Idle-readiness fix + elasticity under load (T4) + +Status as of 2026-06-14. Closes the **idle-readiness stall** and the **cert SAN +scale gap**; the 1M/1536-dim T4 run on k3s remains the project's standing +Ref-A/k3s dependency (the machinery and the fix that unblocks it are proven +locally over real OS processes). + +## 1. The idle-readiness stall (WORKLOG 2026-06-13: an 11.5h hang) + +A snapshot-installed joiner (`install_boot` / `seed_joiner`) serves `503` on its +readiness probe (`/health` → `region_health` → `is_ready`) until its sticky +`converged` latch flips. Pre-m12p5 the ONLY thing that flips that latch is +`note_lag_for_readiness`, and it is called from exactly one place: +`local_status` — i.e. when something hits the node's `/cluster/status/local` +(an operator/monitoring poll, or the leader's `/cluster/status` aggregator +querying it). + +On a cluster with write traffic this is masked: ship traffic keeps the lag gauge +fresh and monitoring keeps poking status. On a **fully idle** cluster (no writes, +no status polls) neither happens, so a freshly caught-up joiner sits `503` +forever and never joins the Service VIP. The k8s `readinessProbe` is `/health` +(`statefulset.yaml`), so the pod never receives traffic — an 11.5h observed stall. + +Root-cause specifics (read on `44b768b`): + +- `node.rs::note_lag_for_readiness` sets `converged` only when a lag value ≤ + `learner_promote_lag` is *recomputed* — there is no periodic self-driven check. +- `region_health` (the readiness handler) reads `is_ready()` but never recomputes + lag, so polling readiness does not advance convergence. +- The lag gauge (`leader_seqno_for(shard)`) is seeded only by **received ship + segments**; on idle nothing ships, so the gauge would even read a stale/0 value + — converging on it directly is unsafe (a behind-but-unshipped joiner would read + lag 0 from an uninitialized gauge and false-converge). + +## 2. The fix — converge from the heartbeat + +The leader **heartbeat** flows every heartbeat interval regardless of write +traffic and already proves "the leader is alive at term T". m12p5 makes it also +carry the leader's **live flushed frontier**: + +- **Proto** (`wal_shipping.proto`): new `HeartbeatRequest.leader_last_seq = 14` — + the leader's `ship_feed.flushed_seq()` at heartbeat time (same stream numbering + as a follower's per-shard `applied_seqno`). proto3 zero-default `0` = a pre-m12p5 + leader → the follower falls back to the status-poll path (no behavior change). +- **Leader** (`election_heartbeat`): stamps `leader_last_seq` on every heartbeat. +- **Follower** (`election_driver::on_heartbeat` → `node::note_leader_frontier_for_readiness`): + on every ACCEPTED heartbeat, folds `leader_last_seq` into the lag gauge + (monotonic — keeps the `lag_segments` metric/`local_status` truthful on idle for + every follower) and, for a joiner that is not yet converged, computes + `lag = leader_last_seq − applied_seqno(leader_shard)` and drives the existing + `note_lag_for_readiness`. The convergence uses a **real** leader frontier (never + the uninitialized-0 gauge), so a still-behind joiner stays `503` until it + actually catches up. +- **Leader-elect edge** (`become_leader_for_term`): a joiner that WINS leadership + receives no heartbeats, so it converges its latch on activation — by + construction it is caught up to its own log. Without this a promoted-then-elected + joiner would stay `503` forever on an idle cluster. + +Net: a caught-up joiner converges within a heartbeat interval (~100ms here) of +catching up, with no write traffic and no status poll. + +## 3. The cert SAN scale gap + +`k8s/cluster/certs.yaml` (and `scripts/gen-cluster-certs.sh`) enumerated SANs for +`tidaldb-0/1/2` only. Scaling the StatefulSet to 5 (T4) gives `tidaldb-3/4` no +matching SAN, so the inter-node mTLS handshake to the new pods fails. + +Fix: a **wildcard pod SAN** `*.tidaldb-peers.tidaldb-cluster.svc.cluster.local` +covers every ordinal, so scale-up/down needs no cert re-issue. The explicit +`tidaldb-0/1/2` names and the headless/client Service names are retained +(belt-and-suspenders for any strict verifier). tidalDB dials peers over +tonic → rustls → webpki, which matches a wildcard against the single leftmost DNS +label per RFC 6125 — exactly the pod-ordinal label. + +Verified for real (openssl leaf generated by the updated script): + +``` +$ openssl verify -CAfile ca.crt \ + -verify_hostname tidaldb-7.tidaldb-peers.tidaldb-cluster.svc.cluster.local tls.crt +tls.crt: OK # tidaldb-7 is NOT explicitly listed — matched by the wildcard +``` + +## 4. Verification + +### Idle-readiness regression test (real OS processes, tier-3) + +`tidal-server/tests/cluster_membership.rs::mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic`: + +1. 3-node elected cluster; heavy seed → graceful leader restart → WAL compaction + past seq 1, so the later joiner takes the **snapshot-install** path + (`install_boot`/`seed_joiner` true → readiness IS gated on `converged`). A + small `needed=false` join boots a voter and never engages the gate, so it + cannot reproduce the stall — this setup is load-bearing. +2. Go **fully idle**, then `add_node` (which returns on `/health/startup`, an + unconditional 200 — NOT on cluster-readiness). +3. **Gate**: poll ONLY the joiner's `/health` (never `/cluster/status/local`, + which would drive the old latch and mask the bug). With zero writes the joiner + flips ready in **<101ms** (one measured run: 257µs; another: 101ms). +4. **Honesty**: head/tail items are then searchable on the joiner — `converged ⟹ + caught up`. + +**Negative control (proves it is a real gate, not wiring).** With the +heartbeat-convergence call (`note_leader_frontier_for_readiness`) commented out, +the identical test 503s for the full 30s convergence budget and fails — exactly +the pre-m12p5 stall. Re-enabling the call makes it pass in <101ms. + +### Suites green (2026-06-14, local) + +- `cluster_membership` (tier-3, real processes): **6/6** — includes the new idle + test, `mp_seed_join_snapshot_catchup`, and the T4 `mp_scale_3_5_3_under_load_zero_loss`. +- `tidal-net` unit/integration (proto roundtrip + election RPC incl. the new + `leader_last_seq` on the wire): green. +- `tidaldb` lib: 1903 passed. `clippy -D warnings` clean across `tidaldb`, + `tidal-net`, `tidal-server` (incl. `--features cluster-e2e --all-targets`). + +## 5. What remains — T4 at 1M/1536-dim on k3s + +The exit gate's full form ("joiner reaches lag=0 ≤ 5 min on a 1M/1536D corpus; +p99 impact < 2× baseline for < 60s; zero loss") requires the production-shape +corpus, which needs multi-node k3s/Ref-A. This is the SAME standing dependency +called out for m12p1-p4: the local kubeconfig cannot reach Ref-A. What is proven +locally: + +- the 3→5→3 scale-up machinery, zero acked loss, quorum on the grown set + (`mp_scale_3_5_3_under_load_zero_loss`); +- the idle-readiness fix that lets an idle scale-up actually join the VIP — the + specific blocker that would have stalled a real k3s scale-up. + +When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D, +`kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load, +record joiner-lag-to-0 and the p99 envelope, then scale back to 3. diff --git a/k8s/cluster/certs.yaml b/k8s/cluster/certs.yaml index 2895cff..7391c30 100644 --- a/k8s/cluster/certs.yaml +++ b/k8s/cluster/certs.yaml @@ -12,9 +12,14 @@ # `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`, # `ca.crt`. # -# ONE shared node cert with EVERY pod's stable DNS as a SAN (the standard -# StatefulSet pattern): any pod may present it for its own DNS name, and a peer -# dialing `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. +# ONE shared node cert with a WILDCARD pod SAN (the standard StatefulSet pattern): +# any pod may present it for its own DNS name, and a peer dialing +# `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. The +# wildcard `*.tidaldb-peers...` covers EVERY pod ordinal (tidaldb-0, -1, … -N), +# so scaling the StatefulSet up or down with `--seed` needs NO cert re-issue +# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke +# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single +# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label. --- apiVersion: cert-manager.io/v1 kind: Issuer @@ -48,10 +53,13 @@ spec: usages: - server auth # the gRPC + HTTP server identity - client auth # the gRPC mTLS client identity (peer dials) - # SANs: every initial pod's stable headless-Service DNS, plus the headless and - # client Services. Scaling past 3 with `--seed` requires adding the new pod's - # DNS here (or switching to a per-pod Certificate template). + # SANs: a WILDCARD over every pod's stable headless-Service DNS (covers any + # ordinal, so scale-up/down needs no cert re-issue), the explicit initial-pod + # names (belt-and-suspenders for any strict verifier that distrusts a wildcard- + # only leaf), plus the headless and client Services. Keep + # `scripts/gen-cluster-certs.sh` in sync. dnsNames: + - "*.tidaldb-peers.tidaldb-cluster.svc.cluster.local" - tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local - tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local - tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local diff --git a/scripts/gen-cluster-certs.sh b/scripts/gen-cluster-certs.sh index 4d4d3a4..b0c1519 100755 --- a/scripts/gen-cluster-certs.sh +++ b/scripts/gen-cluster-certs.sh @@ -39,9 +39,14 @@ CA_DIR="ca-private" mkdir -p "$CA_DIR" chmod 700 "$CA_DIR" -# Build the SAN list: every pod's stable headless-Service DNS + the headless and -# client Services + loopback (so a local test cluster on 127.0.0.1 also verifies). -SANS="DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1" +# Build the SAN list: a WILDCARD over every pod's stable headless-Service DNS +# (covers any ordinal, so scaling the StatefulSet needs no cert re-issue — m12p5; +# rustls/webpki matches it against the single leftmost label `tidaldb-N` per +# RFC 6125), plus the headless and client Services + loopback (a local 127.0.0.1 +# cluster also verifies). Mirrors k8s/cluster/certs.yaml's wildcard dnsNames. The +# explicit per-pod entries (0..N-1) are belt-and-suspenders for any strict +# verifier that distrusts wildcard-only leaves. +SANS="DNS:*.${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1" for i in $(seq 0 $((N - 1))); do SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local" done diff --git a/tidal-net/proto/wal_shipping.proto b/tidal-net/proto/wal_shipping.proto index 2e2229c..58660e4 100644 --- a/tidal-net/proto/wal_shipping.proto +++ b/tidal-net/proto/wal_shipping.proto @@ -94,6 +94,17 @@ message HeartbeatRequest { // rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") = // a pre-m11p8 peer, treated as version-unknown (no warning). string build_version = 13; + // The leader's CURRENT flushed WAL frontier at heartbeat time (m12p5): the + // live high-water-mark every follower should converge to (its + // `ship_feed.flushed_seq()`, same numbering as the segment `leader_last_seq` + // and a follower's per-shard `applied_seqno`). Unlike `stream_baseline` (the + // term's immutable activation point), this advances as the leader writes — so + // a follower can compute lag and converge readiness from the HEARTBEAT, which + // flows even on an IDLE cluster, instead of waiting for observed ship traffic + // (the idle-readiness stall, m12p5). proto3 zero-default (0) = a pre-m12p5 + // leader, treated as "unknown" — the follower falls back to the status-poll + // readiness path. + uint64 leader_last_seq = 14; } // Heartbeat acknowledgement. diff --git a/tidal-net/src/server.rs b/tidal-net/src/server.rs index 7dbf811..62da728 100644 --- a/tidal-net/src/server.rs +++ b/tidal-net/src/server.rs @@ -725,6 +725,7 @@ impl WalShipping for WalShippingService { tail_term: req.prev_log_term, frontier: req.prev_log_seq, }, + req.leader_last_seq, ); return Ok(Response::new(HeartbeatResponse { acknowledged: true, diff --git a/tidal-net/src/sources.rs b/tidal-net/src/sources.rs index 381e675..7e16bae 100644 --- a/tidal-net/src/sources.rs +++ b/tidal-net/src/sources.rs @@ -294,14 +294,19 @@ pub trait ElectionHooks: Send + Sync + 'static { /// A leader heartbeat: term + leadership + the term's activation /// baseline + the leader's election-time log position - /// `(prev_log_term, prev_log_seq)`. Drives the failure detector and the - /// term-join divergence check. + /// `(prev_log_term, prev_log_seq)` + the leader's LIVE flushed frontier + /// (`leader_last_seq`). Drives the failure detector, the term-join + /// divergence check, and (m12p5) the sticky readiness latch — a caught-up + /// joiner converges from the heartbeat (which flows on an idle cluster), + /// not only from observed ship traffic. `leader_last_seq == 0` means a + /// pre-m12p5 leader conveyed no frontier (the readiness drive is skipped). fn on_heartbeat( &self, term: u64, leader_region: u16, stream_baseline: u64, prev_log: tidaldb::replication::LogPosition, + leader_last_seq: u64, ) -> HeartbeatExchange; /// A pre-vote or vote request. The grant (and any term adoption) is diff --git a/tidal-net/tests/election_rpc.rs b/tidal-net/tests/election_rpc.rs index 8295eda..5f00492 100644 --- a/tidal-net/tests/election_rpc.rs +++ b/tidal-net/tests/election_rpc.rs @@ -117,11 +117,11 @@ impl ElectionHooks for ScriptedHooks { leader_region: u16, baseline: u64, _prev_log: tidaldb::replication::LogPosition, + leader_last_seq: u64, ) -> HeartbeatExchange { - self.seen - .lock() - .unwrap() - .push(format!("hb:{term}:{leader_region}:{baseline}")); + self.seen.lock().unwrap().push(format!( + "hb:{term}:{leader_region}:{baseline}:{leader_last_seq}" + )); HeartbeatExchange { term: self.term, accepted: term >= self.term, @@ -327,6 +327,7 @@ fn heartbeat_exchange_carries_term_and_acceptance() { term: 7, leader_region: 0, stream_baseline: 12, + leader_last_seq: 99, ..Default::default() }, &tx, @@ -366,8 +367,9 @@ fn heartbeat_exchange_carries_term_and_acceptance() { let seen = hooks.seen.lock().unwrap(); assert!( - seen.contains(&"hb:7:0:12".to_string()), - "the term's activation baseline rode the heartbeat: {seen:?}" + seen.contains(&"hb:7:0:12:99".to_string()), + "the term's activation baseline AND the leader's live frontier (m12p5 \ + leader_last_seq) rode the heartbeat: {seen:?}" ); } diff --git a/tidal-net/tests/snapshot_fetch.rs b/tidal-net/tests/snapshot_fetch.rs index 12d7c7f..bc3e940 100644 --- a/tidal-net/tests/snapshot_fetch.rs +++ b/tidal-net/tests/snapshot_fetch.rs @@ -357,6 +357,7 @@ fn fetch_snapshot_term_fence_refuses_stale_puller() { _leader_region: u16, _stream_baseline: u64, _prev_log: tidaldb::replication::LogPosition, + _leader_last_seq: u64, ) -> tidal_net::HeartbeatExchange { tidal_net::HeartbeatExchange { term: self.0, diff --git a/tidal-server/src/cluster/election_driver.rs b/tidal-server/src/cluster/election_driver.rs index 919e0f4..5b05ab7 100644 --- a/tidal-server/src/cluster/election_driver.rs +++ b/tidal-server/src/cluster/election_driver.rs @@ -551,6 +551,7 @@ impl tidal_net::ElectionHooks for NodeElectionHooks { leader_region: u16, stream_baseline: u64, prev_log: LogPosition, + leader_last_seq: u64, ) -> HeartbeatExchange { let now = Instant::now(); let current = self.runtime.current_term(); @@ -580,6 +581,14 @@ impl tidal_net::ElectionHooks for NodeElectionHooks { .lock_machine() .on_leader_contact(term, RegionId(leader_region), now); self.runtime.execute(actions, now); + // m12p5 idle-readiness: the ACCEPTED heartbeat carries the leader's live + // flushed frontier. Seed the lag gauge and drive the sticky readiness + // latch from it — a caught-up joiner converges on the heartbeat (which + // flows even when no writes happen), not only after observed ship traffic + // recomputes lag, or an external status poll happens to call `local_status`. + if let Some(node) = self.runtime.node.upgrade() { + node.note_leader_frontier_for_readiness(RegionId(leader_region), leader_last_seq); + } HeartbeatExchange { term: self.runtime.current_term(), accepted: true, diff --git a/tidal-server/src/cluster/node.rs b/tidal-server/src/cluster/node.rs index 4daf4ff..0ba379a 100644 --- a/tidal-server/src/cluster/node.rs +++ b/tidal-server/src/cluster/node.rs @@ -2446,6 +2446,11 @@ impl ShardReplica { stream_baseline: self.stream_baseline.load(Ordering::Acquire), prev_log_term: prev.tail_term, prev_log_seq: prev.frontier, + // m12p5: the leader's LIVE flushed frontier (not the immutable term + // baseline) so a follower converges readiness from the heartbeat — + // which flows on an idle cluster — instead of waiting for ship + // traffic to seed its lag gauge (the idle-readiness stall). + leader_last_seq: self.ship_feed.flushed_seq(), ..Default::default() } } @@ -2486,6 +2491,19 @@ impl ShardReplica { "election won: term marker journaled, ship queue active, writes open" ); + // m12p5 idle-readiness: a joiner that WINS leadership is, by construction, + // caught up to its own log — but it will never receive a heartbeat to + // drive `note_leader_frontier_for_readiness`, so converge the sticky latch + // here. Without this a promoted-then-elected joiner would stay 503 forever + // on an idle cluster (`is_ready` gates on `converged` for a joiner boot). + if (self.install_boot || self.seed_joiner) && !self.converged.swap(true, Ordering::AcqRel) { + tracing::info!( + term, + region = %self.region_name, + "joiner won leadership → trivially converged; readiness sticky-ready (m12p5)" + ); + } + // §3.2 — the activation membership record (the linchpin): once the // kind-3 marker is durable AND the membership era has begun (a kind-4 // record is in the log), re-append the CURRENT roster as a fresh kind-4 @@ -2800,6 +2818,49 @@ impl ShardReplica { } } + /// m12p5 idle-readiness drive: fold a leader heartbeat's live flushed + /// frontier into the lag gauge and the sticky readiness latch. + /// + /// The pre-m12p5 readiness latch (`note_lag_for_readiness`) only fired when + /// something recomputed lag — observed ship traffic seeding the gauge, or an + /// external `/cluster/status/local` poll calling `local_status`. On an IDLE + /// cluster neither happens, so a freshly caught-up joiner could sit 503 for + /// hours (WORKLOG 2026-06-13: an 11.5h stall) and never join the VIP. The + /// heartbeat, by contrast, flows every heartbeat interval regardless of write + /// traffic and now carries `leader_last_seq` — the leader's live frontier in + /// the SAME stream numbering as a follower's per-shard `applied_seqno`. + /// + /// Seeding the gauge (monotonic) keeps the lag metric/status truthful on an + /// idle cluster for every follower; the readiness note then converges a + /// caught-up joiner using a REAL leader frontier (never the uninitialized-0 + /// gauge reading, which would false-converge a still-behind joiner). + /// `leader_last_seq == 0` = a pre-m12p5 leader conveyed nothing → no-op + /// (the status-poll path still applies). + pub(crate) fn note_leader_frontier_for_readiness( + &self, + leader_region: RegionId, + leader_last_seq: u64, + ) { + if leader_last_seq == 0 { + return; + } + let Ok(db) = self.db() else { return }; + let leader_shard = shard_of_region(leader_region); + // Keep the lag gauge (hence the `lag_segments` metric and `local_status`) + // fresh on an idle cluster, not only after ship traffic. Monotonic per + // shard, so a stale/lower heartbeat can never walk it back. + db.control_plane() + .lag_gauge() + .update_leader_seqno_for(leader_shard, leader_last_seq); + if (self.install_boot || self.seed_joiner) && !self.converged.load(Ordering::Acquire) { + let applied = db + .replication_state() + .applied_seqno(leader_shard) + .unwrap_or(0); + self.note_lag_for_readiness(leader_last_seq.saturating_sub(applied)); + } + } + /// Whether this node is READY to serve (m11p5 §4 readiness predicate). /// /// 503 while shutting down, quarantined, REMOVED, or an install/seed-join diff --git a/tidal-server/tests/cluster_membership.rs b/tidal-server/tests/cluster_membership.rs index b585107..fe24dcd 100644 --- a/tidal-server/tests/cluster_membership.rs +++ b/tidal-server/tests/cluster_membership.rs @@ -483,6 +483,128 @@ fn mp_seed_join_snapshot_catchup() { println!("[seed-join] every probed seeded item is searchable on the joiner — exit gate 1 met"); } +/// m12p5 EXIT GATE — idle-cluster readiness convergence. +/// +/// THE BUG (WORKLOG 2026-06-13, an 11.5h stall). A snapshot-installed joiner's +/// sticky readiness latch (`converged`, the `/health` readinessProbe gate) was +/// driven ONLY by `note_lag_for_readiness`, which fires when something recomputes +/// lag — observed ship traffic seeding the lag gauge, or an external +/// `/cluster/status/local` poll calling `local_status`. On an IDLE cluster (no +/// writes, no operator/monitoring status polls) neither happens, so a freshly +/// caught-up joiner stayed `503` indefinitely and never joined the Service VIP. +/// +/// THE FIX (m12p5). The leader heartbeat — which flows every heartbeat interval +/// regardless of write traffic — now carries its LIVE flushed frontier +/// (`leader_last_seq`). The follower folds it into the lag gauge and the readiness +/// latch on every accepted heartbeat, so a caught-up joiner converges from the +/// heartbeat, not from write traffic or a status poll. +/// +/// THE GATE. Reuse exit-gate-1's install-boot setup (heavy seed → compact past +/// seq 1 → the joiner takes the SNAPSHOT path, so `install_boot`/`seed_joiner` +/// are true and readiness IS gated on `converged` — a small `needed=false` join +/// boots a voter and never engages the gate). Then go FULLY IDLE and seed-join. +/// The joiner's `/health` must flip `200` within the convergence budget **while +/// this test issues zero writes and never polls the joiner's +/// `/cluster/status/local`** (which would drive the old latch and mask the bug). +/// Pre-m12p5 this `503`'d until the budget elapsed. +#[test] +fn mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic() { + let extra = format!("{FAST_ELECTION_YAML}\n{PROMOTE_LAG_YAML}"); + let opts = ClusterOptions::new(3).with_topology_extra(&extra).with_env( + 0, + "TIDAL_RESEED_HANDSHAKE_MS", + RESEED_HANDSHAKE_MS, + ); + let mut cluster = MultiProcCluster::start_with(opts); + + // ── Seed heavy content (WAL > one 16 MiB segment), then force compaction past + // seq 1 via a graceful leader restart — the install-boot setup of + // `mp_seed_join_snapshot_catchup`. This is what makes the later joiner take the + // SNAPSHOT path, so its readiness is gated on the `converged` latch (the bug). + let seeded = seeded_items(); + let blob = blob_value(); + for entity in 1..=3u64 { + write_heavy_item(&cluster, LEADER, entity, "", false); + } + for entity in 4..=seeded { + write_heavy_item(&cluster, LEADER, entity, &blob, true); + } + // Generous, item-scaled convergence budgets: 320 heavy items over loopback can + // trail the bare 30s budget on a contended/cold runner (the convergence here is + // pure SETUP, not the gate under test), and the post-restart re-ship repeats it. + let setup_budget = convergence_budget() + Duration::from_secs(seeded / 50); + cluster.wait_converged_all(setup_budget); + cluster.restart_graceful(LEADER, &[]); + let _leader_idx = current_leader_idx(&cluster).unwrap_or_else(|| { + await_elected_leader(&cluster, &[0, 1, 2], 0, Duration::from_secs(15)).0 + }); + cluster.wait_converged_all(setup_budget); + let leader_idx = current_leader_idx(&cluster).expect("a leader after re-election"); + println!("[idle-ready] heavy seed + compaction done; leader is node {leader_idx} — going IDLE"); + + // ── Seed-join onto the now-IDLE cluster. `add_node` returns once the joiner's + // PROCESS is up (`/health/startup`, an unconditional 200), NOT when it is + // cluster-ready (`/health`). From here the test issues ZERO writes and never + // polls the joiner's `/cluster/status/local`. + let joiner = cluster.add_node(leader_idx); + let joiner_name = cluster.region_name(joiner).to_string(); + println!("[idle-ready] seed-joined node {joiner} ('{joiner_name}') — process up; cluster idle"); + + // ── THE GATE: the snapshot-installed joiner's readiness probe (`/health` → + // region_health → is_ready → converged) flips 200 within budget, driven ONLY + // by the leader heartbeat's live-frontier compare. NO writes, NO status poll on + // the joiner. Pre-m12p5 this 503'd until the budget elapsed (the 11.5h stall). + let ready_start = Instant::now(); + let deadline = ready_start + convergence_budget(); + loop { + if cluster.get(joiner, "/health").status().is_success() { + break; + } + assert!( + Instant::now() < deadline, + "IDLE-READINESS REGRESSION (m12p5): snapshot joiner did not flip /health ready \ + within {:?} on an idle cluster with no write traffic and no status poll — the \ + heartbeat must converge it. Pre-m12p5 this 503'd until the budget elapsed.", + convergence_budget() + ); + thread::sleep(Duration::from_millis(100)); + } + println!( + "[idle-ready] snapshot joiner flipped /health READY in {:?} on an idle cluster", + ready_start.elapsed() + ); + + // ── HONESTY: readiness must mean actually-caught-up, not a premature latch. + // A handful of head/tail probes are searchable on the joiner (snapshot carried + // the compacted history, the stream the suffix). Uses `/search`, NOT + // `/cluster/status/local`, so it never retroactively drives the latch (which + // already flipped above). Poll past the ~2s text-index auto-commit. + let probes: Vec = { + let mut p = vec![1u64, seeded / 2, seeded]; + p.retain(|&e| (1..=seeded).contains(&e)); + p.sort_unstable(); + p.dedup(); + p + }; + for entity in probes { + let probe_deadline = Instant::now() + Duration::from_secs(20); + loop { + if item_searchable(&cluster, joiner, entity) { + break; + } + assert!( + Instant::now() < probe_deadline, + "joiner reported READY but is missing seeded item {entity} — converged must \ + imply caught up (the heartbeat compare uses a real leader frontier, not a 0 gauge)" + ); + thread::sleep(Duration::from_millis(200)); + } + } + println!( + "[idle-ready] joiner has content parity — idle convergence was honest (m12p5 gate met)" + ); +} + /// A voter removed via the verb decommissions cleanly: the roster shrinks /// everywhere, the removed node reports removed/not-ready, quorum follows the /// smaller set, and the removed node has NO reseed marker.