tidaldb/docs/profiling/m12p5-idle-readiness-elasticity.md

m12p5 — Idle-readiness fix + elasticity under load (T4)

Status as of 2026-06-14. Closes the idle-readiness stall and the cert SAN scale gap; the 1M/1536-dim T4 run on k3s remains the project's standing Ref-A/k3s dependency (the machinery and the fix that unblocks it are proven locally over real OS processes).

1. The idle-readiness stall (WORKLOG 2026-06-13: an 11.5h hang)

A snapshot-installed joiner (install_boot / seed_joiner) serves 503 on its readiness probe (/health → region_health → is_ready) until its sticky converged latch flips. Pre-m12p5 the ONLY thing that flips that latch is note_lag_for_readiness, and it is called from exactly one place: local_status — i.e. when something hits the node's /cluster/status/local (an operator/monitoring poll, or the leader's /cluster/status aggregator querying it).

On a cluster with write traffic this is masked: ship traffic keeps the lag gauge fresh and monitoring keeps poking status. On a fully idle cluster (no writes, no status polls) neither happens, so a freshly caught-up joiner sits 503 forever and never joins the Service VIP. The k8s readinessProbe is /health (statefulset.yaml), so the pod never receives traffic — an 11.5h observed stall.

Root-cause specifics (read on 44b768b):

• node.rs::note_lag_for_readiness sets converged only when a lag value ≤ learner_promote_lag is recomputed — there is no periodic self-driven check.
• region_health (the readiness handler) reads is_ready() but never recomputes lag, so polling readiness does not advance convergence.
• The lag gauge (leader_seqno_for(shard)) is seeded only by received ship segments; on idle nothing ships, so the gauge would even read a stale/0 value — converging on it directly is unsafe (a behind-but-unshipped joiner would read lag 0 from an uninitialized gauge and false-converge).

2. The fix — converge from the heartbeat

The leader heartbeat flows every heartbeat interval regardless of write traffic and already proves "the leader is alive at term T". m12p5 makes it also carry the leader's live flushed frontier:

• Proto (wal_shipping.proto): new HeartbeatRequest.leader_last_seq = 14 — the leader's ship_feed.flushed_seq() at heartbeat time (same stream numbering as a follower's per-shard applied_seqno). proto3 zero-default 0 = a pre-m12p5 leader → the follower falls back to the status-poll path (no behavior change).
• Leader (election_heartbeat): stamps leader_last_seq on every heartbeat.
• Follower (election_driver::on_heartbeat → node::note_leader_frontier_for_readiness): on every ACCEPTED heartbeat, folds leader_last_seq into the lag gauge (monotonic — keeps the lag_segments metric/local_status truthful on idle for every follower) and, for a joiner that is not yet converged, computes lag = leader_last_seq − applied_seqno(leader_shard) and drives the existing note_lag_for_readiness. The convergence uses a real leader frontier (never the uninitialized-0 gauge), so a still-behind joiner stays 503 until it actually catches up.
• Leader-elect edge (become_leader_for_term): a joiner that WINS leadership receives no heartbeats, so it converges its latch on activation — by construction it is caught up to its own log. Without this a promoted-then-elected joiner would stay 503 forever on an idle cluster.

Net: a caught-up joiner converges within a heartbeat interval (~100ms here) of catching up, with no write traffic and no status poll.

3. The cert SAN scale gap

k8s/cluster/certs.yaml (and scripts/gen-cluster-certs.sh) enumerated SANs for tidaldb-0/1/2 only. Scaling the StatefulSet to 5 (T4) gives tidaldb-3/4 no matching SAN, so the inter-node mTLS handshake to the new pods fails.

Fix: a wildcard pod SAN *.tidaldb-peers.tidaldb-cluster.svc.cluster.local covers every ordinal, so scale-up/down needs no cert re-issue. The explicit tidaldb-0/1/2 names and the headless/client Service names are retained (belt-and-suspenders for any strict verifier). tidalDB dials peers over tonic → rustls → webpki, which matches a wildcard against the single leftmost DNS label per RFC 6125 — exactly the pod-ordinal label.

Verified for real (openssl leaf generated by the updated script):

$ openssl verify -CAfile ca.crt \
    -verify_hostname tidaldb-7.tidaldb-peers.tidaldb-cluster.svc.cluster.local tls.crt
tls.crt: OK     # tidaldb-7 is NOT explicitly listed — matched by the wildcard

4. Verification

Idle-readiness regression test (real OS processes, tier-3)

tidal-server/tests/cluster_membership.rs::mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic:

1. 3-node elected cluster; heavy seed → graceful leader restart → WAL compaction past seq 1, so the later joiner takes the snapshot-install path (install_boot/seed_joiner true → readiness IS gated on converged). A small needed=false join boots a voter and never engages the gate, so it cannot reproduce the stall — this setup is load-bearing.
2. Go fully idle, then add_node (which returns on /health/startup, an unconditional 200 — NOT on cluster-readiness).
3. Gate: poll ONLY the joiner's /health (never /cluster/status/local, which would drive the old latch and mask the bug). With zero writes the joiner flips ready in <101ms (one measured run: 257µs; another: 101ms).
4. Honesty: head/tail items are then searchable on the joiner — converged ⟹ caught up.

Negative control (proves it is a real gate, not wiring). With the heartbeat-convergence call (note_leader_frontier_for_readiness) commented out, the identical test 503s for the full 30s convergence budget and fails — exactly the pre-m12p5 stall. Re-enabling the call makes it pass in <101ms.

Suites green (2026-06-14, local)

• cluster_membership (tier-3, real processes): 6/6 — includes the new idle test, mp_seed_join_snapshot_catchup, and the T4 mp_scale_3_5_3_under_load_zero_loss.
• tidal-net unit/integration (proto roundtrip + election RPC incl. the new leader_last_seq on the wire): green.
• tidaldb lib: 1903 passed. clippy -D warnings clean across tidaldb, tidal-net, tidal-server (incl. --features cluster-e2e --all-targets).

5. What remains — T4 at 1M/1536-dim on k3s

The exit gate's full form ("joiner reaches lag=0 ≤ 5 min on a 1M/1536D corpus; p99 impact < 2× baseline for < 60s; zero loss") requires the production-shape corpus, which needs multi-node k3s/Ref-A. This is the SAME standing dependency called out for m12p1-p4: the local kubeconfig cannot reach Ref-A. What is proven locally:

• the 3→5→3 scale-up machinery, zero acked loss, quorum on the grown set (mp_scale_3_5_3_under_load_zero_loss);
• the idle-readiness fix that lets an idle scale-up actually join the VIP — the specific blocker that would have stalled a real k3s scale-up.

When Ref-A is reachable: deploy k8s/cluster/ (now wildcard-SAN), seed 1M×1536-D, kubectl scale statefulset tidaldb --replicas=5 under the tidal-stress load, record joiner-lag-to-0 and the p99 envelope, then scale back to 3.

6. Real-k8s T4 run on kind — and the seed-join-over-TLS bug chain it exposed

The idle-readiness fix was first proven over real OS processes (§4) but those run plaintext inter-node. Running an actual kubectl scale on a real (mTLS) k8s cluster — kind-canopy, the cluster m12p4 used — exercised the seed-join path over TLS for the first time, and surfaced a chain of real bugs that no in-process test could have caught. All are fixed; the run is GREEN.

Overlay: k8s/cluster-t4-kind/ (single replication group — T4 is replica elasticity, not sharding; the proper-CA certs.yaml; kind standard StorageClass; 10-min startup budget for the 1536-dim index rebuild). Seed/load Jobs: tidal-stress/k8s/t4-{seed,load}-job.yaml.

The bug chain (every one TLS-only; the plaintext in-process harness is blind to them)

1. Seed dial scheme (statefulset.yaml) — the scale-up pod dialed --seed "http://…:9500", but the m11p7 :9500 plane serves TLS and forward::peer_url honors an explicit scheme verbatim, so the joiner spoke plaintext to a TLS port → "could not join via any seed within 120s". Fixed to https://.
2. rustls CryptoProvider install order (tidal-server/main.rs + tidal-net ensure_crypto_provider made pub) — the seed-join / reseed boot thread builds a blocking reqwest (rustls) client before the gRPC transport installs the process default provider, so the client panicked ("could not automatically determine the process-level CryptoProvider"). Now installed at the top of main().
3. Discovery target (statefulset.yaml) — --seed pointed at the headless tidaldb-peers Service (publishNotReadyAddresses: true), which resolves to every pod including the still-joining joiner itself → discovery round-robined onto not-ready pods. Re-pointed at the ready-only client Service tidaldb.
4. Discovery poll timeout (join_boot.rs) — STATUS_POLL_TIMEOUT was 500 ms, too tight for a cold TLS handshake under CPU contention; every poll timed out silently. Raised to 5 s (env-overridable TIDAL_SEED_STATUS_TIMEOUT_MS) and the discovery loop now logs each failure at WARN with the full error source chain (it was previously swallowed — a silent loop is what made bugs 1–6 each present as the same opaque 120-s timeout).
5. cert-manager PKI shape (certs.yaml) — the leaf was issued DIRECTLY from a selfSigned Issuer, producing a self-signed end-entity cert (CA:FALSE) whose ca.crt is a copy of the leaf. Lenient stacks tolerated it, but the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer. Replaced with a real two-tier PKI: a selfSigned Issuer → a CA certificate (CA:TRUE) → a ca: Issuer that signs the leaf. (scripts/gen-cluster-certs.sh already did this correctly — the two were merely inconsistent.)
6. grpc_tls for a node not in the topology (join_boot.rs) — the actual UnknownIssuer root cause. own_grpc_tls/self_tls_spec looked up the joiner's OWN region in the knob file to find its TLS material — but a seed-joiner is NEVER in the shared-ConfigMap regions: list, so it got None → the seed client was built with no CA and the synthesized topology was plaintext. Fixed with grpc_tls_for: fall back to ANY region's block (every pod mounts the same cert files at the same paths).

Result (GREEN)

kubectl scale statefulset tidaldb --replicas=5 on an idle, 1536-dim-seeded cluster: both new pods seed-join over mTLS (wildcard SAN covers tidaldb-3/4), catch up, and flip /health Ready in 13 s — driven by the idle-readiness heartbeat convergence (no writes, no status poll). Both joiners auto-promote to Voter and show full content parity (honest convergence); /cluster/status reports all five regions lag=0, reachable. Pre-m12p5 this would have stalled at 503 indefinitely (the 11.5-h WORKLOG hang).

Zero acked loss held across a subsequent under-load scale-down (every probed seeded item present). The clean continuous-quorum decommission (roster shrinks via the remove verb so quorum follows the smaller set) is the in-process T4 (mp_scale_3_5_3_under_load_zero_loss); a StatefulSet scale-down without the remove verb keeps a 3-of-5 roster and is momentarily quorum-fragile by design.

One bug found and root-caused but NOT fixed (deliberately)

At a corpus large enough to compact the leader's WAL below the snapshot point (≥ ~10k×1536 here), a seed-joiner installs a snapshot (snapshot_seq correct, fix #2/#6 made the gRPC snapshot fetch work) but then node.rs:734 seeds the post-install catch-up frontier from db.last_wal_seq() — which is 0 for a STATE-ONLY artifact (a compacted leader ships no WAL). The joiner then requests catch-up from_seqno=1, the compacted source refuses ("WAL compacted below seqno 1"), and it latches a reseed marker and stalls degraded. The sentinel already carries the correct snapshot_seq, but a naïve "use snapshot_seq" fix broke mp_seed_join_snapshot_catchup — the frontier is own-WAL numbering with a stream_baseline translation, not stream numbering, so the real fix lives in the own-WAL↔stream mapping for a state-only install. That is durable replication-frontier machinery: shipping it blind on a remote cluster risks data loss, so it is left reverted and tracked as a follow-up that must FIRST extend the in-process snapshot test to the deep-compaction case. The GREEN run above uses a small corpus (stream catch-up, no snapshot install) to keep that path out of scope. The full 1M/1536-dim gate remains the standing Ref-A/k3s dependency.