feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
This commit is contained in:
parent
31ee612f27
commit
aa94fd9b1f
131
docs/profiling/m12p5-idle-readiness-elasticity.md
Normal file
131
docs/profiling/m12p5-idle-readiness-elasticity.md
Normal file
@ -0,0 +1,131 @@
|
|||||||
|
# m12p5 — Idle-readiness fix + elasticity under load (T4)
|
||||||
|
|
||||||
|
Status as of 2026-06-14. Closes the **idle-readiness stall** and the **cert SAN
|
||||||
|
scale gap**; the 1M/1536-dim T4 run on k3s remains the project's standing
|
||||||
|
Ref-A/k3s dependency (the machinery and the fix that unblocks it are proven
|
||||||
|
locally over real OS processes).
|
||||||
|
|
||||||
|
## 1. The idle-readiness stall (WORKLOG 2026-06-13: an 11.5h hang)
|
||||||
|
|
||||||
|
A snapshot-installed joiner (`install_boot` / `seed_joiner`) serves `503` on its
|
||||||
|
readiness probe (`/health` → `region_health` → `is_ready`) until its sticky
|
||||||
|
`converged` latch flips. Pre-m12p5 the ONLY thing that flips that latch is
|
||||||
|
`note_lag_for_readiness`, and it is called from exactly one place:
|
||||||
|
`local_status` — i.e. when something hits the node's `/cluster/status/local`
|
||||||
|
(an operator/monitoring poll, or the leader's `/cluster/status` aggregator
|
||||||
|
querying it).
|
||||||
|
|
||||||
|
On a cluster with write traffic this is masked: ship traffic keeps the lag gauge
|
||||||
|
fresh and monitoring keeps poking status. On a **fully idle** cluster (no writes,
|
||||||
|
no status polls) neither happens, so a freshly caught-up joiner sits `503`
|
||||||
|
forever and never joins the Service VIP. The k8s `readinessProbe` is `/health`
|
||||||
|
(`statefulset.yaml`), so the pod never receives traffic — an 11.5h observed stall.
|
||||||
|
|
||||||
|
Root-cause specifics (read on `44b768b`):
|
||||||
|
|
||||||
|
- `node.rs::note_lag_for_readiness` sets `converged` only when a lag value ≤
|
||||||
|
`learner_promote_lag` is *recomputed* — there is no periodic self-driven check.
|
||||||
|
- `region_health` (the readiness handler) reads `is_ready()` but never recomputes
|
||||||
|
lag, so polling readiness does not advance convergence.
|
||||||
|
- The lag gauge (`leader_seqno_for(shard)`) is seeded only by **received ship
|
||||||
|
segments**; on idle nothing ships, so the gauge would even read a stale/0 value
|
||||||
|
— converging on it directly is unsafe (a behind-but-unshipped joiner would read
|
||||||
|
lag 0 from an uninitialized gauge and false-converge).
|
||||||
|
|
||||||
|
## 2. The fix — converge from the heartbeat
|
||||||
|
|
||||||
|
The leader **heartbeat** flows every heartbeat interval regardless of write
|
||||||
|
traffic and already proves "the leader is alive at term T". m12p5 makes it also
|
||||||
|
carry the leader's **live flushed frontier**:
|
||||||
|
|
||||||
|
- **Proto** (`wal_shipping.proto`): new `HeartbeatRequest.leader_last_seq = 14` —
|
||||||
|
the leader's `ship_feed.flushed_seq()` at heartbeat time (same stream numbering
|
||||||
|
as a follower's per-shard `applied_seqno`). proto3 zero-default `0` = a pre-m12p5
|
||||||
|
leader → the follower falls back to the status-poll path (no behavior change).
|
||||||
|
- **Leader** (`election_heartbeat`): stamps `leader_last_seq` on every heartbeat.
|
||||||
|
- **Follower** (`election_driver::on_heartbeat` → `node::note_leader_frontier_for_readiness`):
|
||||||
|
on every ACCEPTED heartbeat, folds `leader_last_seq` into the lag gauge
|
||||||
|
(monotonic — keeps the `lag_segments` metric/`local_status` truthful on idle for
|
||||||
|
every follower) and, for a joiner that is not yet converged, computes
|
||||||
|
`lag = leader_last_seq − applied_seqno(leader_shard)` and drives the existing
|
||||||
|
`note_lag_for_readiness`. The convergence uses a **real** leader frontier (never
|
||||||
|
the uninitialized-0 gauge), so a still-behind joiner stays `503` until it
|
||||||
|
actually catches up.
|
||||||
|
- **Leader-elect edge** (`become_leader_for_term`): a joiner that WINS leadership
|
||||||
|
receives no heartbeats, so it converges its latch on activation — by
|
||||||
|
construction it is caught up to its own log. Without this a promoted-then-elected
|
||||||
|
joiner would stay `503` forever on an idle cluster.
|
||||||
|
|
||||||
|
Net: a caught-up joiner converges within a heartbeat interval (~100ms here) of
|
||||||
|
catching up, with no write traffic and no status poll.
|
||||||
|
|
||||||
|
## 3. The cert SAN scale gap
|
||||||
|
|
||||||
|
`k8s/cluster/certs.yaml` (and `scripts/gen-cluster-certs.sh`) enumerated SANs for
|
||||||
|
`tidaldb-0/1/2` only. Scaling the StatefulSet to 5 (T4) gives `tidaldb-3/4` no
|
||||||
|
matching SAN, so the inter-node mTLS handshake to the new pods fails.
|
||||||
|
|
||||||
|
Fix: a **wildcard pod SAN** `*.tidaldb-peers.tidaldb-cluster.svc.cluster.local`
|
||||||
|
covers every ordinal, so scale-up/down needs no cert re-issue. The explicit
|
||||||
|
`tidaldb-0/1/2` names and the headless/client Service names are retained
|
||||||
|
(belt-and-suspenders for any strict verifier). tidalDB dials peers over
|
||||||
|
tonic → rustls → webpki, which matches a wildcard against the single leftmost DNS
|
||||||
|
label per RFC 6125 — exactly the pod-ordinal label.
|
||||||
|
|
||||||
|
Verified for real (openssl leaf generated by the updated script):
|
||||||
|
|
||||||
|
```
|
||||||
|
$ openssl verify -CAfile ca.crt \
|
||||||
|
-verify_hostname tidaldb-7.tidaldb-peers.tidaldb-cluster.svc.cluster.local tls.crt
|
||||||
|
tls.crt: OK # tidaldb-7 is NOT explicitly listed — matched by the wildcard
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Verification
|
||||||
|
|
||||||
|
### Idle-readiness regression test (real OS processes, tier-3)
|
||||||
|
|
||||||
|
`tidal-server/tests/cluster_membership.rs::mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic`:
|
||||||
|
|
||||||
|
1. 3-node elected cluster; heavy seed → graceful leader restart → WAL compaction
|
||||||
|
past seq 1, so the later joiner takes the **snapshot-install** path
|
||||||
|
(`install_boot`/`seed_joiner` true → readiness IS gated on `converged`). A
|
||||||
|
small `needed=false` join boots a voter and never engages the gate, so it
|
||||||
|
cannot reproduce the stall — this setup is load-bearing.
|
||||||
|
2. Go **fully idle**, then `add_node` (which returns on `/health/startup`, an
|
||||||
|
unconditional 200 — NOT on cluster-readiness).
|
||||||
|
3. **Gate**: poll ONLY the joiner's `/health` (never `/cluster/status/local`,
|
||||||
|
which would drive the old latch and mask the bug). With zero writes the joiner
|
||||||
|
flips ready in **<101ms** (one measured run: 257µs; another: 101ms).
|
||||||
|
4. **Honesty**: head/tail items are then searchable on the joiner — `converged ⟹
|
||||||
|
caught up`.
|
||||||
|
|
||||||
|
**Negative control (proves it is a real gate, not wiring).** With the
|
||||||
|
heartbeat-convergence call (`note_leader_frontier_for_readiness`) commented out,
|
||||||
|
the identical test 503s for the full 30s convergence budget and fails — exactly
|
||||||
|
the pre-m12p5 stall. Re-enabling the call makes it pass in <101ms.
|
||||||
|
|
||||||
|
### Suites green (2026-06-14, local)
|
||||||
|
|
||||||
|
- `cluster_membership` (tier-3, real processes): **6/6** — includes the new idle
|
||||||
|
test, `mp_seed_join_snapshot_catchup`, and the T4 `mp_scale_3_5_3_under_load_zero_loss`.
|
||||||
|
- `tidal-net` unit/integration (proto roundtrip + election RPC incl. the new
|
||||||
|
`leader_last_seq` on the wire): green.
|
||||||
|
- `tidaldb` lib: 1903 passed. `clippy -D warnings` clean across `tidaldb`,
|
||||||
|
`tidal-net`, `tidal-server` (incl. `--features cluster-e2e --all-targets`).
|
||||||
|
|
||||||
|
## 5. What remains — T4 at 1M/1536-dim on k3s
|
||||||
|
|
||||||
|
The exit gate's full form ("joiner reaches lag=0 ≤ 5 min on a 1M/1536D corpus;
|
||||||
|
p99 impact < 2× baseline for < 60s; zero loss") requires the production-shape
|
||||||
|
corpus, which needs multi-node k3s/Ref-A. This is the SAME standing dependency
|
||||||
|
called out for m12p1-p4: the local kubeconfig cannot reach Ref-A. What is proven
|
||||||
|
locally:
|
||||||
|
|
||||||
|
- the 3→5→3 scale-up machinery, zero acked loss, quorum on the grown set
|
||||||
|
(`mp_scale_3_5_3_under_load_zero_loss`);
|
||||||
|
- the idle-readiness fix that lets an idle scale-up actually join the VIP — the
|
||||||
|
specific blocker that would have stalled a real k3s scale-up.
|
||||||
|
|
||||||
|
When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D,
|
||||||
|
`kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load,
|
||||||
|
record joiner-lag-to-0 and the p99 envelope, then scale back to 3.
|
||||||
@ -12,9 +12,14 @@
|
|||||||
# `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`,
|
# `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`,
|
||||||
# `ca.crt`.
|
# `ca.crt`.
|
||||||
#
|
#
|
||||||
# ONE shared node cert with EVERY pod's stable DNS as a SAN (the standard
|
# ONE shared node cert with a WILDCARD pod SAN (the standard StatefulSet pattern):
|
||||||
# StatefulSet pattern): any pod may present it for its own DNS name, and a peer
|
# any pod may present it for its own DNS name, and a peer dialing
|
||||||
# dialing `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list.
|
# `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. The
|
||||||
|
# wildcard `*.tidaldb-peers...` covers EVERY pod ordinal (tidaldb-0, -1, … -N),
|
||||||
|
# so scaling the StatefulSet up or down with `--seed` needs NO cert re-issue
|
||||||
|
# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
|
||||||
|
# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
|
||||||
|
# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
|
||||||
---
|
---
|
||||||
apiVersion: cert-manager.io/v1
|
apiVersion: cert-manager.io/v1
|
||||||
kind: Issuer
|
kind: Issuer
|
||||||
@ -48,10 +53,13 @@ spec:
|
|||||||
usages:
|
usages:
|
||||||
- server auth # the gRPC + HTTP server identity
|
- server auth # the gRPC + HTTP server identity
|
||||||
- client auth # the gRPC mTLS client identity (peer dials)
|
- client auth # the gRPC mTLS client identity (peer dials)
|
||||||
# SANs: every initial pod's stable headless-Service DNS, plus the headless and
|
# SANs: a WILDCARD over every pod's stable headless-Service DNS (covers any
|
||||||
# client Services. Scaling past 3 with `--seed` requires adding the new pod's
|
# ordinal, so scale-up/down needs no cert re-issue), the explicit initial-pod
|
||||||
# DNS here (or switching to a per-pod Certificate template).
|
# names (belt-and-suspenders for any strict verifier that distrusts a wildcard-
|
||||||
|
# only leaf), plus the headless and client Services. Keep
|
||||||
|
# `scripts/gen-cluster-certs.sh` in sync.
|
||||||
dnsNames:
|
dnsNames:
|
||||||
|
- "*.tidaldb-peers.tidaldb-cluster.svc.cluster.local"
|
||||||
- tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
- tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
||||||
- tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
- tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
||||||
- tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
- tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
||||||
|
|||||||
@ -39,9 +39,14 @@ CA_DIR="ca-private"
|
|||||||
mkdir -p "$CA_DIR"
|
mkdir -p "$CA_DIR"
|
||||||
chmod 700 "$CA_DIR"
|
chmod 700 "$CA_DIR"
|
||||||
|
|
||||||
# Build the SAN list: every pod's stable headless-Service DNS + the headless and
|
# Build the SAN list: a WILDCARD over every pod's stable headless-Service DNS
|
||||||
# client Services + loopback (so a local test cluster on 127.0.0.1 also verifies).
|
# (covers any ordinal, so scaling the StatefulSet needs no cert re-issue — m12p5;
|
||||||
SANS="DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1"
|
# rustls/webpki matches it against the single leftmost label `tidaldb-N` per
|
||||||
|
# RFC 6125), plus the headless and client Services + loopback (a local 127.0.0.1
|
||||||
|
# cluster also verifies). Mirrors k8s/cluster/certs.yaml's wildcard dnsNames. The
|
||||||
|
# explicit per-pod entries (0..N-1) are belt-and-suspenders for any strict
|
||||||
|
# verifier that distrusts wildcard-only leaves.
|
||||||
|
SANS="DNS:*.${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1"
|
||||||
for i in $(seq 0 $((N - 1))); do
|
for i in $(seq 0 $((N - 1))); do
|
||||||
SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local"
|
SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local"
|
||||||
done
|
done
|
||||||
|
|||||||
@ -94,6 +94,17 @@ message HeartbeatRequest {
|
|||||||
// rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") =
|
// rolling upgrade REQUIRES N/N+1 to interoperate. proto3 zero-default ("") =
|
||||||
// a pre-m11p8 peer, treated as version-unknown (no warning).
|
// a pre-m11p8 peer, treated as version-unknown (no warning).
|
||||||
string build_version = 13;
|
string build_version = 13;
|
||||||
|
// The leader's CURRENT flushed WAL frontier at heartbeat time (m12p5): the
|
||||||
|
// live high-water-mark every follower should converge to (its
|
||||||
|
// `ship_feed.flushed_seq()`, same numbering as the segment `leader_last_seq`
|
||||||
|
// and a follower's per-shard `applied_seqno`). Unlike `stream_baseline` (the
|
||||||
|
// term's immutable activation point), this advances as the leader writes — so
|
||||||
|
// a follower can compute lag and converge readiness from the HEARTBEAT, which
|
||||||
|
// flows even on an IDLE cluster, instead of waiting for observed ship traffic
|
||||||
|
// (the idle-readiness stall, m12p5). proto3 zero-default (0) = a pre-m12p5
|
||||||
|
// leader, treated as "unknown" — the follower falls back to the status-poll
|
||||||
|
// readiness path.
|
||||||
|
uint64 leader_last_seq = 14;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Heartbeat acknowledgement.
|
// Heartbeat acknowledgement.
|
||||||
|
|||||||
@ -725,6 +725,7 @@ impl WalShipping for WalShippingService {
|
|||||||
tail_term: req.prev_log_term,
|
tail_term: req.prev_log_term,
|
||||||
frontier: req.prev_log_seq,
|
frontier: req.prev_log_seq,
|
||||||
},
|
},
|
||||||
|
req.leader_last_seq,
|
||||||
);
|
);
|
||||||
return Ok(Response::new(HeartbeatResponse {
|
return Ok(Response::new(HeartbeatResponse {
|
||||||
acknowledged: true,
|
acknowledged: true,
|
||||||
|
|||||||
@ -294,14 +294,19 @@ pub trait ElectionHooks: Send + Sync + 'static {
|
|||||||
|
|
||||||
/// A leader heartbeat: term + leadership + the term's activation
|
/// A leader heartbeat: term + leadership + the term's activation
|
||||||
/// baseline + the leader's election-time log position
|
/// baseline + the leader's election-time log position
|
||||||
/// `(prev_log_term, prev_log_seq)`. Drives the failure detector and the
|
/// `(prev_log_term, prev_log_seq)` + the leader's LIVE flushed frontier
|
||||||
/// term-join divergence check.
|
/// (`leader_last_seq`). Drives the failure detector, the term-join
|
||||||
|
/// divergence check, and (m12p5) the sticky readiness latch — a caught-up
|
||||||
|
/// joiner converges from the heartbeat (which flows on an idle cluster),
|
||||||
|
/// not only from observed ship traffic. `leader_last_seq == 0` means a
|
||||||
|
/// pre-m12p5 leader conveyed no frontier (the readiness drive is skipped).
|
||||||
fn on_heartbeat(
|
fn on_heartbeat(
|
||||||
&self,
|
&self,
|
||||||
term: u64,
|
term: u64,
|
||||||
leader_region: u16,
|
leader_region: u16,
|
||||||
stream_baseline: u64,
|
stream_baseline: u64,
|
||||||
prev_log: tidaldb::replication::LogPosition,
|
prev_log: tidaldb::replication::LogPosition,
|
||||||
|
leader_last_seq: u64,
|
||||||
) -> HeartbeatExchange;
|
) -> HeartbeatExchange;
|
||||||
|
|
||||||
/// A pre-vote or vote request. The grant (and any term adoption) is
|
/// A pre-vote or vote request. The grant (and any term adoption) is
|
||||||
|
|||||||
@ -117,11 +117,11 @@ impl ElectionHooks for ScriptedHooks {
|
|||||||
leader_region: u16,
|
leader_region: u16,
|
||||||
baseline: u64,
|
baseline: u64,
|
||||||
_prev_log: tidaldb::replication::LogPosition,
|
_prev_log: tidaldb::replication::LogPosition,
|
||||||
|
leader_last_seq: u64,
|
||||||
) -> HeartbeatExchange {
|
) -> HeartbeatExchange {
|
||||||
self.seen
|
self.seen.lock().unwrap().push(format!(
|
||||||
.lock()
|
"hb:{term}:{leader_region}:{baseline}:{leader_last_seq}"
|
||||||
.unwrap()
|
));
|
||||||
.push(format!("hb:{term}:{leader_region}:{baseline}"));
|
|
||||||
HeartbeatExchange {
|
HeartbeatExchange {
|
||||||
term: self.term,
|
term: self.term,
|
||||||
accepted: term >= self.term,
|
accepted: term >= self.term,
|
||||||
@ -327,6 +327,7 @@ fn heartbeat_exchange_carries_term_and_acceptance() {
|
|||||||
term: 7,
|
term: 7,
|
||||||
leader_region: 0,
|
leader_region: 0,
|
||||||
stream_baseline: 12,
|
stream_baseline: 12,
|
||||||
|
leader_last_seq: 99,
|
||||||
..Default::default()
|
..Default::default()
|
||||||
},
|
},
|
||||||
&tx,
|
&tx,
|
||||||
@ -366,8 +367,9 @@ fn heartbeat_exchange_carries_term_and_acceptance() {
|
|||||||
|
|
||||||
let seen = hooks.seen.lock().unwrap();
|
let seen = hooks.seen.lock().unwrap();
|
||||||
assert!(
|
assert!(
|
||||||
seen.contains(&"hb:7:0:12".to_string()),
|
seen.contains(&"hb:7:0:12:99".to_string()),
|
||||||
"the term's activation baseline rode the heartbeat: {seen:?}"
|
"the term's activation baseline AND the leader's live frontier (m12p5 \
|
||||||
|
leader_last_seq) rode the heartbeat: {seen:?}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -357,6 +357,7 @@ fn fetch_snapshot_term_fence_refuses_stale_puller() {
|
|||||||
_leader_region: u16,
|
_leader_region: u16,
|
||||||
_stream_baseline: u64,
|
_stream_baseline: u64,
|
||||||
_prev_log: tidaldb::replication::LogPosition,
|
_prev_log: tidaldb::replication::LogPosition,
|
||||||
|
_leader_last_seq: u64,
|
||||||
) -> tidal_net::HeartbeatExchange {
|
) -> tidal_net::HeartbeatExchange {
|
||||||
tidal_net::HeartbeatExchange {
|
tidal_net::HeartbeatExchange {
|
||||||
term: self.0,
|
term: self.0,
|
||||||
|
|||||||
@ -551,6 +551,7 @@ impl tidal_net::ElectionHooks for NodeElectionHooks {
|
|||||||
leader_region: u16,
|
leader_region: u16,
|
||||||
stream_baseline: u64,
|
stream_baseline: u64,
|
||||||
prev_log: LogPosition,
|
prev_log: LogPosition,
|
||||||
|
leader_last_seq: u64,
|
||||||
) -> HeartbeatExchange {
|
) -> HeartbeatExchange {
|
||||||
let now = Instant::now();
|
let now = Instant::now();
|
||||||
let current = self.runtime.current_term();
|
let current = self.runtime.current_term();
|
||||||
@ -580,6 +581,14 @@ impl tidal_net::ElectionHooks for NodeElectionHooks {
|
|||||||
.lock_machine()
|
.lock_machine()
|
||||||
.on_leader_contact(term, RegionId(leader_region), now);
|
.on_leader_contact(term, RegionId(leader_region), now);
|
||||||
self.runtime.execute(actions, now);
|
self.runtime.execute(actions, now);
|
||||||
|
// m12p5 idle-readiness: the ACCEPTED heartbeat carries the leader's live
|
||||||
|
// flushed frontier. Seed the lag gauge and drive the sticky readiness
|
||||||
|
// latch from it — a caught-up joiner converges on the heartbeat (which
|
||||||
|
// flows even when no writes happen), not only after observed ship traffic
|
||||||
|
// recomputes lag, or an external status poll happens to call `local_status`.
|
||||||
|
if let Some(node) = self.runtime.node.upgrade() {
|
||||||
|
node.note_leader_frontier_for_readiness(RegionId(leader_region), leader_last_seq);
|
||||||
|
}
|
||||||
HeartbeatExchange {
|
HeartbeatExchange {
|
||||||
term: self.runtime.current_term(),
|
term: self.runtime.current_term(),
|
||||||
accepted: true,
|
accepted: true,
|
||||||
|
|||||||
@ -2446,6 +2446,11 @@ impl ShardReplica {
|
|||||||
stream_baseline: self.stream_baseline.load(Ordering::Acquire),
|
stream_baseline: self.stream_baseline.load(Ordering::Acquire),
|
||||||
prev_log_term: prev.tail_term,
|
prev_log_term: prev.tail_term,
|
||||||
prev_log_seq: prev.frontier,
|
prev_log_seq: prev.frontier,
|
||||||
|
// m12p5: the leader's LIVE flushed frontier (not the immutable term
|
||||||
|
// baseline) so a follower converges readiness from the heartbeat —
|
||||||
|
// which flows on an idle cluster — instead of waiting for ship
|
||||||
|
// traffic to seed its lag gauge (the idle-readiness stall).
|
||||||
|
leader_last_seq: self.ship_feed.flushed_seq(),
|
||||||
..Default::default()
|
..Default::default()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -2486,6 +2491,19 @@ impl ShardReplica {
|
|||||||
"election won: term marker journaled, ship queue active, writes open"
|
"election won: term marker journaled, ship queue active, writes open"
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// m12p5 idle-readiness: a joiner that WINS leadership is, by construction,
|
||||||
|
// caught up to its own log — but it will never receive a heartbeat to
|
||||||
|
// drive `note_leader_frontier_for_readiness`, so converge the sticky latch
|
||||||
|
// here. Without this a promoted-then-elected joiner would stay 503 forever
|
||||||
|
// on an idle cluster (`is_ready` gates on `converged` for a joiner boot).
|
||||||
|
if (self.install_boot || self.seed_joiner) && !self.converged.swap(true, Ordering::AcqRel) {
|
||||||
|
tracing::info!(
|
||||||
|
term,
|
||||||
|
region = %self.region_name,
|
||||||
|
"joiner won leadership → trivially converged; readiness sticky-ready (m12p5)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// §3.2 — the activation membership record (the linchpin): once the
|
// §3.2 — the activation membership record (the linchpin): once the
|
||||||
// kind-3 marker is durable AND the membership era has begun (a kind-4
|
// kind-3 marker is durable AND the membership era has begun (a kind-4
|
||||||
// record is in the log), re-append the CURRENT roster as a fresh kind-4
|
// record is in the log), re-append the CURRENT roster as a fresh kind-4
|
||||||
@ -2800,6 +2818,49 @@ impl ShardReplica {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// m12p5 idle-readiness drive: fold a leader heartbeat's live flushed
|
||||||
|
/// frontier into the lag gauge and the sticky readiness latch.
|
||||||
|
///
|
||||||
|
/// The pre-m12p5 readiness latch (`note_lag_for_readiness`) only fired when
|
||||||
|
/// something recomputed lag — observed ship traffic seeding the gauge, or an
|
||||||
|
/// external `/cluster/status/local` poll calling `local_status`. On an IDLE
|
||||||
|
/// cluster neither happens, so a freshly caught-up joiner could sit 503 for
|
||||||
|
/// hours (WORKLOG 2026-06-13: an 11.5h stall) and never join the VIP. The
|
||||||
|
/// heartbeat, by contrast, flows every heartbeat interval regardless of write
|
||||||
|
/// traffic and now carries `leader_last_seq` — the leader's live frontier in
|
||||||
|
/// the SAME stream numbering as a follower's per-shard `applied_seqno`.
|
||||||
|
///
|
||||||
|
/// Seeding the gauge (monotonic) keeps the lag metric/status truthful on an
|
||||||
|
/// idle cluster for every follower; the readiness note then converges a
|
||||||
|
/// caught-up joiner using a REAL leader frontier (never the uninitialized-0
|
||||||
|
/// gauge reading, which would false-converge a still-behind joiner).
|
||||||
|
/// `leader_last_seq == 0` = a pre-m12p5 leader conveyed nothing → no-op
|
||||||
|
/// (the status-poll path still applies).
|
||||||
|
pub(crate) fn note_leader_frontier_for_readiness(
|
||||||
|
&self,
|
||||||
|
leader_region: RegionId,
|
||||||
|
leader_last_seq: u64,
|
||||||
|
) {
|
||||||
|
if leader_last_seq == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let Ok(db) = self.db() else { return };
|
||||||
|
let leader_shard = shard_of_region(leader_region);
|
||||||
|
// Keep the lag gauge (hence the `lag_segments` metric and `local_status`)
|
||||||
|
// fresh on an idle cluster, not only after ship traffic. Monotonic per
|
||||||
|
// shard, so a stale/lower heartbeat can never walk it back.
|
||||||
|
db.control_plane()
|
||||||
|
.lag_gauge()
|
||||||
|
.update_leader_seqno_for(leader_shard, leader_last_seq);
|
||||||
|
if (self.install_boot || self.seed_joiner) && !self.converged.load(Ordering::Acquire) {
|
||||||
|
let applied = db
|
||||||
|
.replication_state()
|
||||||
|
.applied_seqno(leader_shard)
|
||||||
|
.unwrap_or(0);
|
||||||
|
self.note_lag_for_readiness(leader_last_seq.saturating_sub(applied));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether this node is READY to serve (m11p5 §4 readiness predicate).
|
/// Whether this node is READY to serve (m11p5 §4 readiness predicate).
|
||||||
///
|
///
|
||||||
/// 503 while shutting down, quarantined, REMOVED, or an install/seed-join
|
/// 503 while shutting down, quarantined, REMOVED, or an install/seed-join
|
||||||
|
|||||||
@ -483,6 +483,128 @@ fn mp_seed_join_snapshot_catchup() {
|
|||||||
println!("[seed-join] every probed seeded item is searchable on the joiner — exit gate 1 met");
|
println!("[seed-join] every probed seeded item is searchable on the joiner — exit gate 1 met");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// m12p5 EXIT GATE — idle-cluster readiness convergence.
|
||||||
|
///
|
||||||
|
/// THE BUG (WORKLOG 2026-06-13, an 11.5h stall). A snapshot-installed joiner's
|
||||||
|
/// sticky readiness latch (`converged`, the `/health` readinessProbe gate) was
|
||||||
|
/// driven ONLY by `note_lag_for_readiness`, which fires when something recomputes
|
||||||
|
/// lag — observed ship traffic seeding the lag gauge, or an external
|
||||||
|
/// `/cluster/status/local` poll calling `local_status`. On an IDLE cluster (no
|
||||||
|
/// writes, no operator/monitoring status polls) neither happens, so a freshly
|
||||||
|
/// caught-up joiner stayed `503` indefinitely and never joined the Service VIP.
|
||||||
|
///
|
||||||
|
/// THE FIX (m12p5). The leader heartbeat — which flows every heartbeat interval
|
||||||
|
/// regardless of write traffic — now carries its LIVE flushed frontier
|
||||||
|
/// (`leader_last_seq`). The follower folds it into the lag gauge and the readiness
|
||||||
|
/// latch on every accepted heartbeat, so a caught-up joiner converges from the
|
||||||
|
/// heartbeat, not from write traffic or a status poll.
|
||||||
|
///
|
||||||
|
/// THE GATE. Reuse exit-gate-1's install-boot setup (heavy seed → compact past
|
||||||
|
/// seq 1 → the joiner takes the SNAPSHOT path, so `install_boot`/`seed_joiner`
|
||||||
|
/// are true and readiness IS gated on `converged` — a small `needed=false` join
|
||||||
|
/// boots a voter and never engages the gate). Then go FULLY IDLE and seed-join.
|
||||||
|
/// The joiner's `/health` must flip `200` within the convergence budget **while
|
||||||
|
/// this test issues zero writes and never polls the joiner's
|
||||||
|
/// `/cluster/status/local`** (which would drive the old latch and mask the bug).
|
||||||
|
/// Pre-m12p5 this `503`'d until the budget elapsed.
|
||||||
|
#[test]
|
||||||
|
fn mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic() {
|
||||||
|
let extra = format!("{FAST_ELECTION_YAML}\n{PROMOTE_LAG_YAML}");
|
||||||
|
let opts = ClusterOptions::new(3).with_topology_extra(&extra).with_env(
|
||||||
|
0,
|
||||||
|
"TIDAL_RESEED_HANDSHAKE_MS",
|
||||||
|
RESEED_HANDSHAKE_MS,
|
||||||
|
);
|
||||||
|
let mut cluster = MultiProcCluster::start_with(opts);
|
||||||
|
|
||||||
|
// ── Seed heavy content (WAL > one 16 MiB segment), then force compaction past
|
||||||
|
// seq 1 via a graceful leader restart — the install-boot setup of
|
||||||
|
// `mp_seed_join_snapshot_catchup`. This is what makes the later joiner take the
|
||||||
|
// SNAPSHOT path, so its readiness is gated on the `converged` latch (the bug).
|
||||||
|
let seeded = seeded_items();
|
||||||
|
let blob = blob_value();
|
||||||
|
for entity in 1..=3u64 {
|
||||||
|
write_heavy_item(&cluster, LEADER, entity, "", false);
|
||||||
|
}
|
||||||
|
for entity in 4..=seeded {
|
||||||
|
write_heavy_item(&cluster, LEADER, entity, &blob, true);
|
||||||
|
}
|
||||||
|
// Generous, item-scaled convergence budgets: 320 heavy items over loopback can
|
||||||
|
// trail the bare 30s budget on a contended/cold runner (the convergence here is
|
||||||
|
// pure SETUP, not the gate under test), and the post-restart re-ship repeats it.
|
||||||
|
let setup_budget = convergence_budget() + Duration::from_secs(seeded / 50);
|
||||||
|
cluster.wait_converged_all(setup_budget);
|
||||||
|
cluster.restart_graceful(LEADER, &[]);
|
||||||
|
let _leader_idx = current_leader_idx(&cluster).unwrap_or_else(|| {
|
||||||
|
await_elected_leader(&cluster, &[0, 1, 2], 0, Duration::from_secs(15)).0
|
||||||
|
});
|
||||||
|
cluster.wait_converged_all(setup_budget);
|
||||||
|
let leader_idx = current_leader_idx(&cluster).expect("a leader after re-election");
|
||||||
|
println!("[idle-ready] heavy seed + compaction done; leader is node {leader_idx} — going IDLE");
|
||||||
|
|
||||||
|
// ── Seed-join onto the now-IDLE cluster. `add_node` returns once the joiner's
|
||||||
|
// PROCESS is up (`/health/startup`, an unconditional 200), NOT when it is
|
||||||
|
// cluster-ready (`/health`). From here the test issues ZERO writes and never
|
||||||
|
// polls the joiner's `/cluster/status/local`.
|
||||||
|
let joiner = cluster.add_node(leader_idx);
|
||||||
|
let joiner_name = cluster.region_name(joiner).to_string();
|
||||||
|
println!("[idle-ready] seed-joined node {joiner} ('{joiner_name}') — process up; cluster idle");
|
||||||
|
|
||||||
|
// ── THE GATE: the snapshot-installed joiner's readiness probe (`/health` →
|
||||||
|
// region_health → is_ready → converged) flips 200 within budget, driven ONLY
|
||||||
|
// by the leader heartbeat's live-frontier compare. NO writes, NO status poll on
|
||||||
|
// the joiner. Pre-m12p5 this 503'd until the budget elapsed (the 11.5h stall).
|
||||||
|
let ready_start = Instant::now();
|
||||||
|
let deadline = ready_start + convergence_budget();
|
||||||
|
loop {
|
||||||
|
if cluster.get(joiner, "/health").status().is_success() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
Instant::now() < deadline,
|
||||||
|
"IDLE-READINESS REGRESSION (m12p5): snapshot joiner did not flip /health ready \
|
||||||
|
within {:?} on an idle cluster with no write traffic and no status poll — the \
|
||||||
|
heartbeat must converge it. Pre-m12p5 this 503'd until the budget elapsed.",
|
||||||
|
convergence_budget()
|
||||||
|
);
|
||||||
|
thread::sleep(Duration::from_millis(100));
|
||||||
|
}
|
||||||
|
println!(
|
||||||
|
"[idle-ready] snapshot joiner flipped /health READY in {:?} on an idle cluster",
|
||||||
|
ready_start.elapsed()
|
||||||
|
);
|
||||||
|
|
||||||
|
// ── HONESTY: readiness must mean actually-caught-up, not a premature latch.
|
||||||
|
// A handful of head/tail probes are searchable on the joiner (snapshot carried
|
||||||
|
// the compacted history, the stream the suffix). Uses `/search`, NOT
|
||||||
|
// `/cluster/status/local`, so it never retroactively drives the latch (which
|
||||||
|
// already flipped above). Poll past the ~2s text-index auto-commit.
|
||||||
|
let probes: Vec<u64> = {
|
||||||
|
let mut p = vec![1u64, seeded / 2, seeded];
|
||||||
|
p.retain(|&e| (1..=seeded).contains(&e));
|
||||||
|
p.sort_unstable();
|
||||||
|
p.dedup();
|
||||||
|
p
|
||||||
|
};
|
||||||
|
for entity in probes {
|
||||||
|
let probe_deadline = Instant::now() + Duration::from_secs(20);
|
||||||
|
loop {
|
||||||
|
if item_searchable(&cluster, joiner, entity) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
Instant::now() < probe_deadline,
|
||||||
|
"joiner reported READY but is missing seeded item {entity} — converged must \
|
||||||
|
imply caught up (the heartbeat compare uses a real leader frontier, not a 0 gauge)"
|
||||||
|
);
|
||||||
|
thread::sleep(Duration::from_millis(200));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
println!(
|
||||||
|
"[idle-ready] joiner has content parity — idle convergence was honest (m12p5 gate met)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// A voter removed via the verb decommissions cleanly: the roster shrinks
|
/// A voter removed via the verb decommissions cleanly: the roster shrinks
|
||||||
/// everywhere, the removed node reports removed/not-ready, quorum follows the
|
/// everywhere, the removed node reports removed/not-ready, quorum follows the
|
||||||
/// smaller set, and the removed node has NO reseed marker.
|
/// smaller set, and the removed node has NO reseed marker.
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user