tidaldb/docs/profiling/m12p5-idle-readiness-elasticity.md

# m12p5 — Idle-readiness fix + elasticity under load (T4)

Status as of 2026-06-14. Closes the **idle-readiness stall** and the **cert SAN
scale gap**; the 1M/1536-dim T4 run on k3s remains the project's standing
Ref-A/k3s dependency (the machinery and the fix that unblocks it are proven
locally over real OS processes).

## 1. The idle-readiness stall (WORKLOG 2026-06-13: an 11.5h hang)

A snapshot-installed joiner (`install_boot` / `seed_joiner`) serves `503` on its
readiness probe (`/health` → `region_health` → `is_ready`) until its sticky
`converged` latch flips. Pre-m12p5 the ONLY thing that flips that latch is
`note_lag_for_readiness`, and it is called from exactly one place:
`local_status` — i.e. when something hits the node's `/cluster/status/local`
(an operator/monitoring poll, or the leader's `/cluster/status` aggregator
querying it).

On a cluster with write traffic this is masked: ship traffic keeps the lag gauge
fresh and monitoring keeps poking status. On a **fully idle** cluster (no writes,
no status polls) neither happens, so a freshly caught-up joiner sits `503`
forever and never joins the Service VIP. The k8s `readinessProbe` is `/health`
(`statefulset.yaml`), so the pod never receives traffic — an 11.5h observed stall.

Root-cause specifics (read on `44b768b`):

- `node.rs::note_lag_for_readiness` sets `converged` only when a lag value ≤
  `learner_promote_lag` is *recomputed* — there is no periodic self-driven check.
- `region_health` (the readiness handler) reads `is_ready()` but never recomputes
  lag, so polling readiness does not advance convergence.
- The lag gauge (`leader_seqno_for(shard)`) is seeded only by **received ship
  segments**; on idle nothing ships, so the gauge would even read a stale/0 value
  — converging on it directly is unsafe (a behind-but-unshipped joiner would read
  lag 0 from an uninitialized gauge and false-converge).

## 2. The fix — converge from the heartbeat

The leader **heartbeat** flows every heartbeat interval regardless of write
traffic and already proves "the leader is alive at term T". m12p5 makes it also
carry the leader's **live flushed frontier**:

- **Proto** (`wal_shipping.proto`): new `HeartbeatRequest.leader_last_seq = 14` —
  the leader's `ship_feed.flushed_seq()` at heartbeat time (same stream numbering
  as a follower's per-shard `applied_seqno`). proto3 zero-default `0` = a pre-m12p5
  leader → the follower falls back to the status-poll path (no behavior change).
- **Leader** (`election_heartbeat`): stamps `leader_last_seq` on every heartbeat.
- **Follower** (`election_driver::on_heartbeat` → `node::note_leader_frontier_for_readiness`):
  on every ACCEPTED heartbeat, folds `leader_last_seq` into the lag gauge
  (monotonic — keeps the `lag_segments` metric/`local_status` truthful on idle for
  every follower) and, for a joiner that is not yet converged, computes
  `lag = leader_last_seq − applied_seqno(leader_shard)` and drives the existing
  `note_lag_for_readiness`. The convergence uses a **real** leader frontier (never
  the uninitialized-0 gauge), so a still-behind joiner stays `503` until it
  actually catches up.
- **Leader-elect edge** (`become_leader_for_term`): a joiner that WINS leadership
  receives no heartbeats, so it converges its latch on activation — by
  construction it is caught up to its own log. Without this a promoted-then-elected
  joiner would stay `503` forever on an idle cluster.

Net: a caught-up joiner converges within a heartbeat interval (~100ms here) of
catching up, with no write traffic and no status poll.

## 3. The cert SAN scale gap

`k8s/cluster/certs.yaml` (and `scripts/gen-cluster-certs.sh`) enumerated SANs for
`tidaldb-0/1/2` only. Scaling the StatefulSet to 5 (T4) gives `tidaldb-3/4` no
matching SAN, so the inter-node mTLS handshake to the new pods fails.

Fix: a **wildcard pod SAN** `*.tidaldb-peers.tidaldb-cluster.svc.cluster.local`
covers every ordinal, so scale-up/down needs no cert re-issue. The explicit
`tidaldb-0/1/2` names and the headless/client Service names are retained
(belt-and-suspenders for any strict verifier). tidalDB dials peers over
tonic → rustls → webpki, which matches a wildcard against the single leftmost DNS
label per RFC 6125 — exactly the pod-ordinal label.

Verified for real (openssl leaf generated by the updated script):

```
$ openssl verify -CAfile ca.crt \
    -verify_hostname tidaldb-7.tidaldb-peers.tidaldb-cluster.svc.cluster.local tls.crt
tls.crt: OK     # tidaldb-7 is NOT explicitly listed — matched by the wildcard
```

## 4. Verification

### Idle-readiness regression test (real OS processes, tier-3)

`tidal-server/tests/cluster_membership.rs::mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic`:

1. 3-node elected cluster; heavy seed → graceful leader restart → WAL compaction
   past seq 1, so the later joiner takes the **snapshot-install** path
   (`install_boot`/`seed_joiner` true → readiness IS gated on `converged`). A
   small `needed=false` join boots a voter and never engages the gate, so it
   cannot reproduce the stall — this setup is load-bearing.
2. Go **fully idle**, then `add_node` (which returns on `/health/startup`, an
   unconditional 200 — NOT on cluster-readiness).
3. **Gate**: poll ONLY the joiner's `/health` (never `/cluster/status/local`,
   which would drive the old latch and mask the bug). With zero writes the joiner
   flips ready in **<101ms** (one measured run: 257µs; another: 101ms).
4. **Honesty**: head/tail items are then searchable on the joiner — `converged ⟹
   caught up`.

**Negative control (proves it is a real gate, not wiring).** With the
heartbeat-convergence call (`note_leader_frontier_for_readiness`) commented out,
the identical test 503s for the full 30s convergence budget and fails — exactly
the pre-m12p5 stall. Re-enabling the call makes it pass in <101ms.

### Suites green (2026-06-14, local)

- `cluster_membership` (tier-3, real processes): **6/6** — includes the new idle
  test, `mp_seed_join_snapshot_catchup`, and the T4 `mp_scale_3_5_3_under_load_zero_loss`.
- `tidal-net` unit/integration (proto roundtrip + election RPC incl. the new
  `leader_last_seq` on the wire): green.
- `tidaldb` lib: 1903 passed. `clippy -D warnings` clean across `tidaldb`,
  `tidal-net`, `tidal-server` (incl. `--features cluster-e2e --all-targets`).

## 5. What remains — T4 at 1M/1536-dim on k3s

The exit gate's full form ("joiner reaches lag=0 ≤ 5 min on a 1M/1536D corpus;
p99 impact < 2× baseline for < 60s; zero loss") requires the production-shape
corpus, which needs multi-node k3s/Ref-A. This is the SAME standing dependency
called out for m12p1-p4: the local kubeconfig cannot reach Ref-A. What is proven
locally:

- the 3→5→3 scale-up machinery, zero acked loss, quorum on the grown set
  (`mp_scale_3_5_3_under_load_zero_loss`);
- the idle-readiness fix that lets an idle scale-up actually join the VIP — the
  specific blocker that would have stalled a real k3s scale-up.

When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D,
`kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load,
record joiner-lag-to-0 and the p99 envelope, then scale back to 3.