feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN

Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).

- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
  back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
  lag gauge (monotonic per shard) and drives the readiness latch using a REAL
  leader frontier (never the uninitialized-0 gauge, which would false-converge a
  still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
  — snapshot joiner flips /health ready on an idle cluster with zero writes and
  no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
  scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
  cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
  names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
This commit is contained in:
jx12n 2026-06-14 16:21:00 -06:00
parent 31ee612f27
commit aa94fd9b1f
11 changed files with 373 additions and 17 deletions

View File

@ -0,0 +1,131 @@
# m12p5 — Idle-readiness fix + elasticity under load (T4)
Status as of 2026-06-14. Closes the **idle-readiness stall** and the **cert SAN
scale gap**; the 1M/1536-dim T4 run on k3s remains the project's standing
Ref-A/k3s dependency (the machinery and the fix that unblocks it are proven
locally over real OS processes).
## 1. The idle-readiness stall (WORKLOG 2026-06-13: an 11.5h hang)
A snapshot-installed joiner (`install_boot` / `seed_joiner`) serves `503` on its
readiness probe (`/health` → `region_health``is_ready`) until its sticky
`converged` latch flips. Pre-m12p5 the ONLY thing that flips that latch is
`note_lag_for_readiness`, and it is called from exactly one place:
`local_status` — i.e. when something hits the node's `/cluster/status/local`
(an operator/monitoring poll, or the leader's `/cluster/status` aggregator
querying it).
On a cluster with write traffic this is masked: ship traffic keeps the lag gauge
fresh and monitoring keeps poking status. On a **fully idle** cluster (no writes,
no status polls) neither happens, so a freshly caught-up joiner sits `503`
forever and never joins the Service VIP. The k8s `readinessProbe` is `/health`
(`statefulset.yaml`), so the pod never receives traffic — an 11.5h observed stall.
Root-cause specifics (read on `44b768b`):
- `node.rs::note_lag_for_readiness` sets `converged` only when a lag value ≤
`learner_promote_lag` is *recomputed* — there is no periodic self-driven check.
- `region_health` (the readiness handler) reads `is_ready()` but never recomputes
lag, so polling readiness does not advance convergence.
- The lag gauge (`leader_seqno_for(shard)`) is seeded only by **received ship
segments**; on idle nothing ships, so the gauge would even read a stale/0 value
— converging on it directly is unsafe (a behind-but-unshipped joiner would read
lag 0 from an uninitialized gauge and false-converge).
## 2. The fix — converge from the heartbeat
The leader **heartbeat** flows every heartbeat interval regardless of write
traffic and already proves "the leader is alive at term T". m12p5 makes it also
carry the leader's **live flushed frontier**:
- **Proto** (`wal_shipping.proto`): new `HeartbeatRequest.leader_last_seq = 14`
the leader's `ship_feed.flushed_seq()` at heartbeat time (same stream numbering
as a follower's per-shard `applied_seqno`). proto3 zero-default `0` = a pre-m12p5
leader → the follower falls back to the status-poll path (no behavior change).
- **Leader** (`election_heartbeat`): stamps `leader_last_seq` on every heartbeat.
- **Follower** (`election_driver::on_heartbeat` → `node::note_leader_frontier_for_readiness`):
on every ACCEPTED heartbeat, folds `leader_last_seq` into the lag gauge
(monotonic — keeps the `lag_segments` metric/`local_status` truthful on idle for
every follower) and, for a joiner that is not yet converged, computes
`lag = leader_last_seq applied_seqno(leader_shard)` and drives the existing
`note_lag_for_readiness`. The convergence uses a **real** leader frontier (never
the uninitialized-0 gauge), so a still-behind joiner stays `503` until it
actually catches up.
- **Leader-elect edge** (`become_leader_for_term`): a joiner that WINS leadership
receives no heartbeats, so it converges its latch on activation — by
construction it is caught up to its own log. Without this a promoted-then-elected
joiner would stay `503` forever on an idle cluster.
Net: a caught-up joiner converges within a heartbeat interval (~100ms here) of
catching up, with no write traffic and no status poll.
## 3. The cert SAN scale gap
`k8s/cluster/certs.yaml` (and `scripts/gen-cluster-certs.sh`) enumerated SANs for
`tidaldb-0/1/2` only. Scaling the StatefulSet to 5 (T4) gives `tidaldb-3/4` no
matching SAN, so the inter-node mTLS handshake to the new pods fails.
Fix: a **wildcard pod SAN** `*.tidaldb-peers.tidaldb-cluster.svc.cluster.local`
covers every ordinal, so scale-up/down needs no cert re-issue. The explicit
`tidaldb-0/1/2` names and the headless/client Service names are retained
(belt-and-suspenders for any strict verifier). tidalDB dials peers over
tonic → rustls → webpki, which matches a wildcard against the single leftmost DNS
label per RFC 6125 — exactly the pod-ordinal label.
Verified for real (openssl leaf generated by the updated script):
```
$ openssl verify -CAfile ca.crt \
-verify_hostname tidaldb-7.tidaldb-peers.tidaldb-cluster.svc.cluster.local tls.crt
tls.crt: OK # tidaldb-7 is NOT explicitly listed — matched by the wildcard
```
## 4. Verification
### Idle-readiness regression test (real OS processes, tier-3)
`tidal-server/tests/cluster_membership.rs::mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic`:
1. 3-node elected cluster; heavy seed → graceful leader restart → WAL compaction
past seq 1, so the later joiner takes the **snapshot-install** path
(`install_boot`/`seed_joiner` true → readiness IS gated on `converged`). A
small `needed=false` join boots a voter and never engages the gate, so it
cannot reproduce the stall — this setup is load-bearing.
2. Go **fully idle**, then `add_node` (which returns on `/health/startup`, an
unconditional 200 — NOT on cluster-readiness).
3. **Gate**: poll ONLY the joiner's `/health` (never `/cluster/status/local`,
which would drive the old latch and mask the bug). With zero writes the joiner
flips ready in **<101ms** (one measured run: 257µs; another: 101ms).
4. **Honesty**: head/tail items are then searchable on the joiner — `converged ⟹
caught up`.
**Negative control (proves it is a real gate, not wiring).** With the
heartbeat-convergence call (`note_leader_frontier_for_readiness`) commented out,
the identical test 503s for the full 30s convergence budget and fails — exactly
the pre-m12p5 stall. Re-enabling the call makes it pass in <101ms.
### Suites green (2026-06-14, local)
- `cluster_membership` (tier-3, real processes): **6/6** — includes the new idle
test, `mp_seed_join_snapshot_catchup`, and the T4 `mp_scale_3_5_3_under_load_zero_loss`.
- `tidal-net` unit/integration (proto roundtrip + election RPC incl. the new
`leader_last_seq` on the wire): green.
- `tidaldb` lib: 1903 passed. `clippy -D warnings` clean across `tidaldb`,
`tidal-net`, `tidal-server` (incl. `--features cluster-e2e --all-targets`).
## 5. What remains — T4 at 1M/1536-dim on k3s
The exit gate's full form ("joiner reaches lag=0 ≤ 5 min on a 1M/1536D corpus;
p99 impact < 2× baseline for < 60s; zero loss") requires the production-shape
corpus, which needs multi-node k3s/Ref-A. This is the SAME standing dependency
called out for m12p1-p4: the local kubeconfig cannot reach Ref-A. What is proven
locally:
- the 3→5→3 scale-up machinery, zero acked loss, quorum on the grown set
(`mp_scale_3_5_3_under_load_zero_loss`);
- the idle-readiness fix that lets an idle scale-up actually join the VIP — the
specific blocker that would have stalled a real k3s scale-up.
When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D,
`kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load,
record joiner-lag-to-0 and the p99 envelope, then scale back to 3.

View File

@ -12,9 +12,14 @@
# `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`, # `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`,
# `ca.crt`. # `ca.crt`.
# #
# ONE shared node cert with EVERY pod's stable DNS as a SAN (the standard # ONE shared node cert with a WILDCARD pod SAN (the standard StatefulSet pattern):
# StatefulSet pattern): any pod may present it for its own DNS name, and a peer # any pod may present it for its own DNS name, and a peer dialing
# dialing `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. # `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. The
# wildcard `*.tidaldb-peers...` covers EVERY pod ordinal (tidaldb-0, -1, … -N),
# so scaling the StatefulSet up or down with `--seed` needs NO cert re-issue
# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
--- ---
apiVersion: cert-manager.io/v1 apiVersion: cert-manager.io/v1
kind: Issuer kind: Issuer
@ -48,10 +53,13 @@ spec:
usages: usages:
- server auth # the gRPC + HTTP server identity - server auth # the gRPC + HTTP server identity
- client auth # the gRPC mTLS client identity (peer dials) - client auth # the gRPC mTLS client identity (peer dials)
# SANs: every initial pod's stable headless-Service DNS, plus the headless and # SANs: a WILDCARD over every pod's stable headless-Service DNS (covers any
# client Services. Scaling past 3 with `--seed` requires adding the new pod's # ordinal, so scale-up/down needs no cert re-issue), the explicit initial-pod
# DNS here (or switching to a per-pod Certificate template). # names (belt-and-suspenders for any strict verifier that distrusts a wildcard-
# only leaf), plus the headless and client Services. Keep
# `scripts/gen-cluster-certs.sh` in sync.
dnsNames: dnsNames:
- "*.tidaldb-peers.tidaldb-cluster.svc.cluster.local"
- tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local - tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local - tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local - tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local

View File

@ -39,9 +39,14 @@ CA_DIR="ca-private"
mkdir -p "$CA_DIR" mkdir -p "$CA_DIR"
chmod 700 "$CA_DIR" chmod 700 "$CA_DIR"
# Build the SAN list: every pod's stable headless-Service DNS + the headless and # Build the SAN list: a WILDCARD over every pod's stable headless-Service DNS
# client Services + loopback (so a local test cluster on 127.0.0.1 also verifies). # (covers any ordinal, so scaling the StatefulSet needs no cert re-issue — m12p5;
SANS="DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1" # rustls/webpki matches it against the single leftmost label `tidaldb-N` per
# RFC 6125), plus the headless and client Services + loopback (a local 127.0.0.1
# cluster also verifies). Mirrors k8s/cluster/certs.yaml's wildcard dnsNames. The
# explicit per-pod entries (0..N-1) are belt-and-suspenders for any strict
# verifier that distrusts wildcard-only leaves.
SANS="DNS:*.${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1"
for i in $(seq 0 $((N - 1))); do for i in $(seq 0 $((N - 1))); do
SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local" SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local"
done done

View File

@ -94,6 +94,17 @@ message HeartbeatRequest {
// rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") = // rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") =
// a pre-m11p8 peer, treated as version-unknown (no warning). // a pre-m11p8 peer, treated as version-unknown (no warning).
string build_version = 13; string build_version = 13;
// The leader's CURRENT flushed WAL frontier at heartbeat time (m12p5): the
// live high-water-mark every follower should converge to (its
// `ship_feed.flushed_seq()`, same numbering as the segment `leader_last_seq`
// and a follower's per-shard `applied_seqno`). Unlike `stream_baseline` (the
// term's immutable activation point), this advances as the leader writes so
// a follower can compute lag and converge readiness from the HEARTBEAT, which
// flows even on an IDLE cluster, instead of waiting for observed ship traffic
// (the idle-readiness stall, m12p5). proto3 zero-default (0) = a pre-m12p5
// leader, treated as "unknown" the follower falls back to the status-poll
// readiness path.
uint64 leader_last_seq = 14;
} }
// Heartbeat acknowledgement. // Heartbeat acknowledgement.

View File

@ -725,6 +725,7 @@ impl WalShipping for WalShippingService {
tail_term: req.prev_log_term, tail_term: req.prev_log_term,
frontier: req.prev_log_seq, frontier: req.prev_log_seq,
}, },
req.leader_last_seq,
); );
return Ok(Response::new(HeartbeatResponse { return Ok(Response::new(HeartbeatResponse {
acknowledged: true, acknowledged: true,

View File

@ -294,14 +294,19 @@ pub trait ElectionHooks: Send + Sync + 'static {
/// A leader heartbeat: term + leadership + the term's activation /// A leader heartbeat: term + leadership + the term's activation
/// baseline + the leader's election-time log position /// baseline + the leader's election-time log position
/// `(prev_log_term, prev_log_seq)`. Drives the failure detector and the /// `(prev_log_term, prev_log_seq)` + the leader's LIVE flushed frontier
/// term-join divergence check. /// (`leader_last_seq`). Drives the failure detector, the term-join
/// divergence check, and (m12p5) the sticky readiness latch — a caught-up
/// joiner converges from the heartbeat (which flows on an idle cluster),
/// not only from observed ship traffic. `leader_last_seq == 0` means a
/// pre-m12p5 leader conveyed no frontier (the readiness drive is skipped).
fn on_heartbeat( fn on_heartbeat(
&self, &self,
term: u64, term: u64,
leader_region: u16, leader_region: u16,
stream_baseline: u64, stream_baseline: u64,
prev_log: tidaldb::replication::LogPosition, prev_log: tidaldb::replication::LogPosition,
leader_last_seq: u64,
) -> HeartbeatExchange; ) -> HeartbeatExchange;
/// A pre-vote or vote request. The grant (and any term adoption) is /// A pre-vote or vote request. The grant (and any term adoption) is

View File

@ -117,11 +117,11 @@ impl ElectionHooks for ScriptedHooks {
leader_region: u16, leader_region: u16,
baseline: u64, baseline: u64,
_prev_log: tidaldb::replication::LogPosition, _prev_log: tidaldb::replication::LogPosition,
leader_last_seq: u64,
) -> HeartbeatExchange { ) -> HeartbeatExchange {
self.seen self.seen.lock().unwrap().push(format!(
.lock() "hb:{term}:{leader_region}:{baseline}:{leader_last_seq}"
.unwrap() ));
.push(format!("hb:{term}:{leader_region}:{baseline}"));
HeartbeatExchange { HeartbeatExchange {
term: self.term, term: self.term,
accepted: term >= self.term, accepted: term >= self.term,
@ -327,6 +327,7 @@ fn heartbeat_exchange_carries_term_and_acceptance() {
term: 7, term: 7,
leader_region: 0, leader_region: 0,
stream_baseline: 12, stream_baseline: 12,
leader_last_seq: 99,
..Default::default() ..Default::default()
}, },
&tx, &tx,
@ -366,8 +367,9 @@ fn heartbeat_exchange_carries_term_and_acceptance() {
let seen = hooks.seen.lock().unwrap(); let seen = hooks.seen.lock().unwrap();
assert!( assert!(
seen.contains(&"hb:7:0:12".to_string()), seen.contains(&"hb:7:0:12:99".to_string()),
"the term's activation baseline rode the heartbeat: {seen:?}" "the term's activation baseline AND the leader's live frontier (m12p5 \
leader_last_seq) rode the heartbeat: {seen:?}"
); );
} }

View File

@ -357,6 +357,7 @@ fn fetch_snapshot_term_fence_refuses_stale_puller() {
_leader_region: u16, _leader_region: u16,
_stream_baseline: u64, _stream_baseline: u64,
_prev_log: tidaldb::replication::LogPosition, _prev_log: tidaldb::replication::LogPosition,
_leader_last_seq: u64,
) -> tidal_net::HeartbeatExchange { ) -> tidal_net::HeartbeatExchange {
tidal_net::HeartbeatExchange { tidal_net::HeartbeatExchange {
term: self.0, term: self.0,

View File

@ -551,6 +551,7 @@ impl tidal_net::ElectionHooks for NodeElectionHooks {
leader_region: u16, leader_region: u16,
stream_baseline: u64, stream_baseline: u64,
prev_log: LogPosition, prev_log: LogPosition,
leader_last_seq: u64,
) -> HeartbeatExchange { ) -> HeartbeatExchange {
let now = Instant::now(); let now = Instant::now();
let current = self.runtime.current_term(); let current = self.runtime.current_term();
@ -580,6 +581,14 @@ impl tidal_net::ElectionHooks for NodeElectionHooks {
.lock_machine() .lock_machine()
.on_leader_contact(term, RegionId(leader_region), now); .on_leader_contact(term, RegionId(leader_region), now);
self.runtime.execute(actions, now); self.runtime.execute(actions, now);
// m12p5 idle-readiness: the ACCEPTED heartbeat carries the leader's live
// flushed frontier. Seed the lag gauge and drive the sticky readiness
// latch from it — a caught-up joiner converges on the heartbeat (which
// flows even when no writes happen), not only after observed ship traffic
// recomputes lag, or an external status poll happens to call `local_status`.
if let Some(node) = self.runtime.node.upgrade() {
node.note_leader_frontier_for_readiness(RegionId(leader_region), leader_last_seq);
}
HeartbeatExchange { HeartbeatExchange {
term: self.runtime.current_term(), term: self.runtime.current_term(),
accepted: true, accepted: true,

View File

@ -2446,6 +2446,11 @@ impl ShardReplica {
stream_baseline: self.stream_baseline.load(Ordering::Acquire), stream_baseline: self.stream_baseline.load(Ordering::Acquire),
prev_log_term: prev.tail_term, prev_log_term: prev.tail_term,
prev_log_seq: prev.frontier, prev_log_seq: prev.frontier,
// m12p5: the leader's LIVE flushed frontier (not the immutable term
// baseline) so a follower converges readiness from the heartbeat —
// which flows on an idle cluster — instead of waiting for ship
// traffic to seed its lag gauge (the idle-readiness stall).
leader_last_seq: self.ship_feed.flushed_seq(),
..Default::default() ..Default::default()
} }
} }
@ -2486,6 +2491,19 @@ impl ShardReplica {
"election won: term marker journaled, ship queue active, writes open" "election won: term marker journaled, ship queue active, writes open"
); );
// m12p5 idle-readiness: a joiner that WINS leadership is, by construction,
// caught up to its own log — but it will never receive a heartbeat to
// drive `note_leader_frontier_for_readiness`, so converge the sticky latch
// here. Without this a promoted-then-elected joiner would stay 503 forever
// on an idle cluster (`is_ready` gates on `converged` for a joiner boot).
if (self.install_boot || self.seed_joiner) && !self.converged.swap(true, Ordering::AcqRel) {
tracing::info!(
term,
region = %self.region_name,
"joiner won leadership → trivially converged; readiness sticky-ready (m12p5)"
);
}
// §3.2 — the activation membership record (the linchpin): once the // §3.2 — the activation membership record (the linchpin): once the
// kind-3 marker is durable AND the membership era has begun (a kind-4 // kind-3 marker is durable AND the membership era has begun (a kind-4
// record is in the log), re-append the CURRENT roster as a fresh kind-4 // record is in the log), re-append the CURRENT roster as a fresh kind-4
@ -2800,6 +2818,49 @@ impl ShardReplica {
} }
} }
/// m12p5 idle-readiness drive: fold a leader heartbeat's live flushed
/// frontier into the lag gauge and the sticky readiness latch.
///
/// The pre-m12p5 readiness latch (`note_lag_for_readiness`) only fired when
/// something recomputed lag — observed ship traffic seeding the gauge, or an
/// external `/cluster/status/local` poll calling `local_status`. On an IDLE
/// cluster neither happens, so a freshly caught-up joiner could sit 503 for
/// hours (WORKLOG 2026-06-13: an 11.5h stall) and never join the VIP. The
/// heartbeat, by contrast, flows every heartbeat interval regardless of write
/// traffic and now carries `leader_last_seq` — the leader's live frontier in
/// the SAME stream numbering as a follower's per-shard `applied_seqno`.
///
/// Seeding the gauge (monotonic) keeps the lag metric/status truthful on an
/// idle cluster for every follower; the readiness note then converges a
/// caught-up joiner using a REAL leader frontier (never the uninitialized-0
/// gauge reading, which would false-converge a still-behind joiner).
/// `leader_last_seq == 0` = a pre-m12p5 leader conveyed nothing → no-op
/// (the status-poll path still applies).
pub(crate) fn note_leader_frontier_for_readiness(
&self,
leader_region: RegionId,
leader_last_seq: u64,
) {
if leader_last_seq == 0 {
return;
}
let Ok(db) = self.db() else { return };
let leader_shard = shard_of_region(leader_region);
// Keep the lag gauge (hence the `lag_segments` metric and `local_status`)
// fresh on an idle cluster, not only after ship traffic. Monotonic per
// shard, so a stale/lower heartbeat can never walk it back.
db.control_plane()
.lag_gauge()
.update_leader_seqno_for(leader_shard, leader_last_seq);
if (self.install_boot || self.seed_joiner) && !self.converged.load(Ordering::Acquire) {
let applied = db
.replication_state()
.applied_seqno(leader_shard)
.unwrap_or(0);
self.note_lag_for_readiness(leader_last_seq.saturating_sub(applied));
}
}
/// Whether this node is READY to serve (m11p5 §4 readiness predicate). /// Whether this node is READY to serve (m11p5 §4 readiness predicate).
/// ///
/// 503 while shutting down, quarantined, REMOVED, or an install/seed-join /// 503 while shutting down, quarantined, REMOVED, or an install/seed-join

View File

@ -483,6 +483,128 @@ fn mp_seed_join_snapshot_catchup() {
println!("[seed-join] every probed seeded item is searchable on the joiner — exit gate 1 met"); println!("[seed-join] every probed seeded item is searchable on the joiner — exit gate 1 met");
} }
/// m12p5 EXIT GATE — idle-cluster readiness convergence.
///
/// THE BUG (WORKLOG 2026-06-13, an 11.5h stall). A snapshot-installed joiner's
/// sticky readiness latch (`converged`, the `/health` readinessProbe gate) was
/// driven ONLY by `note_lag_for_readiness`, which fires when something recomputes
/// lag — observed ship traffic seeding the lag gauge, or an external
/// `/cluster/status/local` poll calling `local_status`. On an IDLE cluster (no
/// writes, no operator/monitoring status polls) neither happens, so a freshly
/// caught-up joiner stayed `503` indefinitely and never joined the Service VIP.
///
/// THE FIX (m12p5). The leader heartbeat — which flows every heartbeat interval
/// regardless of write traffic — now carries its LIVE flushed frontier
/// (`leader_last_seq`). The follower folds it into the lag gauge and the readiness
/// latch on every accepted heartbeat, so a caught-up joiner converges from the
/// heartbeat, not from write traffic or a status poll.
///
/// THE GATE. Reuse exit-gate-1's install-boot setup (heavy seed → compact past
/// seq 1 → the joiner takes the SNAPSHOT path, so `install_boot`/`seed_joiner`
/// are true and readiness IS gated on `converged` — a small `needed=false` join
/// boots a voter and never engages the gate). Then go FULLY IDLE and seed-join.
/// The joiner's `/health` must flip `200` within the convergence budget **while
/// this test issues zero writes and never polls the joiner's
/// `/cluster/status/local`** (which would drive the old latch and mask the bug).
/// Pre-m12p5 this `503`'d until the budget elapsed.
#[test]
fn mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic() {
let extra = format!("{FAST_ELECTION_YAML}\n{PROMOTE_LAG_YAML}");
let opts = ClusterOptions::new(3).with_topology_extra(&extra).with_env(
0,
"TIDAL_RESEED_HANDSHAKE_MS",
RESEED_HANDSHAKE_MS,
);
let mut cluster = MultiProcCluster::start_with(opts);
// ── Seed heavy content (WAL > one 16 MiB segment), then force compaction past
// seq 1 via a graceful leader restart — the install-boot setup of
// `mp_seed_join_snapshot_catchup`. This is what makes the later joiner take the
// SNAPSHOT path, so its readiness is gated on the `converged` latch (the bug).
let seeded = seeded_items();
let blob = blob_value();
for entity in 1..=3u64 {
write_heavy_item(&cluster, LEADER, entity, "", false);
}
for entity in 4..=seeded {
write_heavy_item(&cluster, LEADER, entity, &blob, true);
}
// Generous, item-scaled convergence budgets: 320 heavy items over loopback can
// trail the bare 30s budget on a contended/cold runner (the convergence here is
// pure SETUP, not the gate under test), and the post-restart re-ship repeats it.
let setup_budget = convergence_budget() + Duration::from_secs(seeded / 50);
cluster.wait_converged_all(setup_budget);
cluster.restart_graceful(LEADER, &[]);
let _leader_idx = current_leader_idx(&cluster).unwrap_or_else(|| {
await_elected_leader(&cluster, &[0, 1, 2], 0, Duration::from_secs(15)).0
});
cluster.wait_converged_all(setup_budget);
let leader_idx = current_leader_idx(&cluster).expect("a leader after re-election");
println!("[idle-ready] heavy seed + compaction done; leader is node {leader_idx} — going IDLE");
// ── Seed-join onto the now-IDLE cluster. `add_node` returns once the joiner's
// PROCESS is up (`/health/startup`, an unconditional 200), NOT when it is
// cluster-ready (`/health`). From here the test issues ZERO writes and never
// polls the joiner's `/cluster/status/local`.
let joiner = cluster.add_node(leader_idx);
let joiner_name = cluster.region_name(joiner).to_string();
println!("[idle-ready] seed-joined node {joiner} ('{joiner_name}') — process up; cluster idle");
// ── THE GATE: the snapshot-installed joiner's readiness probe (`/health` →
// region_health → is_ready → converged) flips 200 within budget, driven ONLY
// by the leader heartbeat's live-frontier compare. NO writes, NO status poll on
// the joiner. Pre-m12p5 this 503'd until the budget elapsed (the 11.5h stall).
let ready_start = Instant::now();
let deadline = ready_start + convergence_budget();
loop {
if cluster.get(joiner, "/health").status().is_success() {
break;
}
assert!(
Instant::now() < deadline,
"IDLE-READINESS REGRESSION (m12p5): snapshot joiner did not flip /health ready \
within {:?} on an idle cluster with no write traffic and no status poll the \
heartbeat must converge it. Pre-m12p5 this 503'd until the budget elapsed.",
convergence_budget()
);
thread::sleep(Duration::from_millis(100));
}
println!(
"[idle-ready] snapshot joiner flipped /health READY in {:?} on an idle cluster",
ready_start.elapsed()
);
// ── HONESTY: readiness must mean actually-caught-up, not a premature latch.
// A handful of head/tail probes are searchable on the joiner (snapshot carried
// the compacted history, the stream the suffix). Uses `/search`, NOT
// `/cluster/status/local`, so it never retroactively drives the latch (which
// already flipped above). Poll past the ~2s text-index auto-commit.
let probes: Vec<u64> = {
let mut p = vec![1u64, seeded / 2, seeded];
p.retain(|&e| (1..=seeded).contains(&e));
p.sort_unstable();
p.dedup();
p
};
for entity in probes {
let probe_deadline = Instant::now() + Duration::from_secs(20);
loop {
if item_searchable(&cluster, joiner, entity) {
break;
}
assert!(
Instant::now() < probe_deadline,
"joiner reported READY but is missing seeded item {entity} — converged must \
imply caught up (the heartbeat compare uses a real leader frontier, not a 0 gauge)"
);
thread::sleep(Duration::from_millis(200));
}
}
println!(
"[idle-ready] joiner has content parity — idle convergence was honest (m12p5 gate met)"
);
}
/// A voter removed via the verb decommissions cleanly: the roster shrinks /// A voter removed via the verb decommissions cleanly: the roster shrinks
/// everywhere, the removed node reports removed/not-ready, quorum follows the /// everywhere, the removed node reports removed/not-ready, quorum follows the
/// smaller set, and the removed node has NO reseed marker. /// smaller set, and the removed node has NO reseed marker.