fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback

Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.

- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
  Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
  the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
  Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
  (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
  joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
  is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
  with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
  Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
  a cold TLS handshake under contention blew the sub-second budget. Discovery now
  logs each poll failure at WARN with the full error source chain (a silent loop
  made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.

Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.

docs/profiling/m12p5-idle-readiness-elasticity.md

@@ -129,3 +129,89 @@ locally:
 When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D,
 `kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load,
 record joiner-lag-to-0 and the p99 envelope, then scale back to 3.
+
+## 6. Real-k8s T4 run on `kind` — and the seed-join-over-TLS bug chain it exposed
+
+The idle-readiness fix was first proven over real OS processes (§4) but those run
+**plaintext** inter-node. Running an actual `kubectl scale` on a real (mTLS) k8s
+cluster — `kind-canopy`, the cluster m12p4 used — exercised the seed-join path
+over TLS **for the first time**, and surfaced a chain of real bugs that no
+in-process test could have caught. All are fixed; the run is GREEN.
+
+Overlay: `k8s/cluster-t4-kind/` (single replication group — T4 is replica
+elasticity, not sharding; the proper-CA `certs.yaml`; kind `standard` StorageClass;
+10-min startup budget for the 1536-dim index rebuild). Seed/load Jobs:
+`tidal-stress/k8s/t4-{seed,load}-job.yaml`.
+
+### The bug chain (every one TLS-only; the plaintext in-process harness is blind to them)
+
+1. **Seed dial scheme** (`statefulset.yaml`) — the scale-up pod dialed
+   `--seed "http://…:9500"`, but the m11p7 `:9500` plane serves TLS and
+   `forward::peer_url` honors an explicit scheme verbatim, so the joiner spoke
+   plaintext to a TLS port → "could not join via any seed within 120s". Fixed to
+   `https://`.
+2. **rustls `CryptoProvider` install order** (`tidal-server/main.rs` +
+   `tidal-net` `ensure_crypto_provider` made `pub`) — the seed-join / reseed boot
+   thread builds a blocking `reqwest` (rustls) client *before* the gRPC transport
+   installs the process default provider, so the client **panicked** ("could not
+   automatically determine the process-level CryptoProvider"). Now installed at
+   the top of `main()`.
+3. **Discovery target** (`statefulset.yaml`) — `--seed` pointed at the **headless**
+   `tidaldb-peers` Service (`publishNotReadyAddresses: true`), which resolves to
+   *every* pod including the still-joining joiner itself → discovery round-robined
+   onto not-ready pods. Re-pointed at the **ready-only** client Service `tidaldb`.
+4. **Discovery poll timeout** (`join_boot.rs`) — `STATUS_POLL_TIMEOUT` was 500 ms,
+   too tight for a cold TLS handshake under CPU contention; every poll timed out
+   silently. Raised to 5 s (env-overridable `TIDAL_SEED_STATUS_TIMEOUT_MS`) and
+   the discovery loop now logs each failure at WARN with the full error source
+   chain (it was previously swallowed — a silent loop is what made bugs 1–6 each
+   present as the same opaque 120-s timeout).
+5. **cert-manager PKI shape** (`certs.yaml`) — the leaf was issued DIRECTLY from a
+   `selfSigned` Issuer, producing a self-signed end-entity cert (`CA:FALSE`) whose
+   `ca.crt` is a copy of the leaf. Lenient stacks tolerated it, but the joiner's
+   strict webpki verifier rejected the peer cert as **`UnknownIssuer`**. Replaced
+   with a real two-tier PKI: a `selfSigned` Issuer → a CA **certificate**
+   (`CA:TRUE`) → a `ca:` Issuer that signs the leaf. (`scripts/gen-cluster-certs.sh`
+   already did this correctly — the two were merely inconsistent.)
+6. **`grpc_tls` for a node not in the topology** (`join_boot.rs`) — **the actual
+   `UnknownIssuer` root cause.** `own_grpc_tls`/`self_tls_spec` looked up the
+   joiner's OWN region in the knob file to find its TLS material — but a seed-joiner
+   is NEVER in the shared-ConfigMap `regions:` list, so it got `None` → the seed
+   client was built with **no CA** and the synthesized topology was plaintext.
+   Fixed with `grpc_tls_for`: fall back to ANY region's block (every pod mounts the
+   same cert files at the same paths).
+
+### Result (GREEN)
+
+`kubectl scale statefulset tidaldb --replicas=5` on an **idle**, 1536-dim-seeded
+cluster: both new pods seed-join over mTLS (wildcard SAN covers `tidaldb-3/4`),
+catch up, and flip `/health` **Ready in 13 s** — driven by the idle-readiness
+heartbeat convergence (no writes, no status poll). Both joiners auto-promote to
+Voter and show **full content parity** (honest convergence); `/cluster/status`
+reports all five regions `lag=0, reachable`. Pre-m12p5 this would have stalled at
+503 indefinitely (the 11.5-h WORKLOG hang).
+
+Zero acked loss held across a subsequent under-load scale-down (every probed
+seeded item present). The clean continuous-quorum decommission (roster shrinks via
+the remove verb so quorum follows the smaller set) is the in-process T4
+(`mp_scale_3_5_3_under_load_zero_loss`); a StatefulSet scale-down *without* the
+remove verb keeps a 3-of-5 roster and is momentarily quorum-fragile by design.
+
+### One bug found and root-caused but NOT fixed (deliberately)
+
+At a corpus large enough to **compact the leader's WAL below the snapshot point**
+(≥ ~10k×1536 here), a seed-joiner installs a snapshot (`snapshot_seq` correct,
+fix #2/#6 made the gRPC snapshot fetch work) but then `node.rs:734` seeds the
+post-install catch-up frontier from `db.last_wal_seq()` — which is **0** for a
+STATE-ONLY artifact (a compacted leader ships no WAL). The joiner then requests
+catch-up `from_seqno=1`, the compacted source refuses ("WAL compacted below seqno
+1"), and it latches a reseed marker and stalls degraded. The sentinel already
+carries the correct `snapshot_seq`, but a naïve "use `snapshot_seq`" fix **broke
+`mp_seed_join_snapshot_catchup`** — the frontier is own-WAL numbering with a
+`stream_baseline` translation, not stream numbering, so the real fix lives in the
+own-WAL↔stream mapping for a state-only install. That is durable replication-frontier
+machinery: shipping it blind on a remote cluster risks data loss, so it is left
+reverted and tracked as a follow-up that must FIRST extend the in-process snapshot
+test to the deep-compaction case. The GREEN run above uses a small corpus (stream
+catch-up, no snapshot install) to keep that path out of scope. The full 1M/1536-dim
+gate remains the standing Ref-A/k3s dependency.

k8s/cluster-t4-kind/kustomization.yaml

@@ -9,9 +9,12 @@
 # Run (the build tags MUST match the `images:` newTag below and the Job specs in
 # tidal-stress/k8s/t4-*.yaml — kind serves only locally-loaded images, so a tag
 # mismatch is an ImagePullBackOff):
-#   # server image — tag MUST equal the `images: newTag` below (m12p5-fix2)
+#   # server image — tag MUST equal the `images: newTag` below (m12p5-local).
-#   docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-fix2 .
+#   # Build from a tree with the m12p6 T4 TLS-scale-up fixes (two-tier-PKI
-#   kind load docker-image tidaldb-server:m12p5-fix2 --name canopy
+#   # certs.yaml + join_boot grpc_tls fallback + https/ready-only seed); an
+#   # older binary crash-loops the joiner on UnknownIssuer / "could not join".
+#   docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-local .
+#   kind load docker-image tidaldb-server:m12p5-local --name canopy
 #   # stress generator — the m12p4 binary is reused unchanged for m12p5; tag MUST
 #   # equal the `image:` in tidal-stress/k8s/t4-{seed,load}-job.yaml (m12p4-local)
 #   docker build -f docker/stress/Dockerfile -t tidaldb-stress:m12p4-local .
@@ -36,7 +39,7 @@ resources:
 images:
   - name: registry.threesix.ai/tidal/server
     newName: tidaldb-server
-    newTag: m12p5-fix2
+    newTag: m12p5-local
 
 patches:
   # Single replication group (drop the base's 3-group `shards:` block).

k8s/cluster/certs.yaml

@@ -20,6 +20,56 @@
 # (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
 # mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
 # leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
+# A REAL two-tier PKI (not a self-signed leaf). cert-manager's `selfSigned`
+# issuer bootstraps a CA *certificate* (isCA: true); a `ca:` issuer backed by
+# that CA then signs the node leaf. The leaf Secret's `ca.crt` is therefore the
+# CA cert (basicConstraints CA:TRUE) — a valid trust anchor.
+#
+# m12p5: the prior shape issued the node leaf DIRECTLY from a `selfSigned` issuer,
+# producing a self-signed end-entity cert with CA:FALSE whose `ca.crt` is a COPY
+# of the leaf. Lenient TLS stacks (reqwest async, openssl-with-`-k`) tolerated it,
+# but a STRICT webpki verifier — exactly what the seed-join blocking reqwest client
+# uses — rejected the peer cert as `UnknownIssuer`, so a scale-up joiner could
+# never discover a leader (it burned the whole 120 s window, crash-looping). A
+# CA:TRUE anchor verifies cleanly for strict AND lenient clients alike.
+---
+apiVersion: cert-manager.io/v1
+kind: Issuer
+metadata:
+  name: tidaldb-selfsigned-bootstrap
+  namespace: tidaldb-cluster
+  labels:
+    app.kubernetes.io/name: tidaldb
+    app.kubernetes.io/part-of: tidaldb
+spec:
+  # Bootstrap only: signs the CA certificate below (which IS a CA). Swap the CA
+  # cert's issuerRef for your org PKI to chain to an existing root instead.
+  selfSigned: {}
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: tidaldb-cluster-ca
+  namespace: tidaldb-cluster
+  labels:
+    app.kubernetes.io/name: tidaldb
+    app.kubernetes.io/part-of: tidaldb
+spec:
+  # The cluster's private ROOT CA — a real CA cert (basicConstraints CA:TRUE).
+  # cert-manager writes its cert+key into this Secret; the `ca:` issuer signs the
+  # node leaf with it.
+  isCA: true
+  commonName: tidaldb-cluster-ca
+  secretName: tidaldb-cluster-ca
+  duration: 87600h    # 10y — the root outlives many leaf rotations
+  renewBefore: 8760h  # 1y
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+  issuerRef:
+    name: tidaldb-selfsigned-bootstrap
+    kind: Issuer
+    group: cert-manager.io
 ---
 apiVersion: cert-manager.io/v1
 kind: Issuer
@@ -30,9 +80,10 @@ metadata:
     app.kubernetes.io/name: tidaldb
     app.kubernetes.io/part-of: tidaldb
 spec:
-  # A self-signed CA root for the cluster's private inter-node PKI. Swap for a
+  # CA issuer backed by the root CA above — signs the node leaf, so the leaf
-  # `ca:` issuer backed by your org PKI to chain to an existing root.
+  # chains to a CA:TRUE anchor.
-  selfSigned: {}
+  ca:
+    secretName: tidaldb-cluster-ca
 ---
 apiVersion: cert-manager.io/v1
 kind: Certificate
@@ -44,7 +95,8 @@ metadata:
     app.kubernetes.io/part-of: tidaldb
 spec:
   # cert-manager writes tls.crt / tls.key / ca.crt into this Secret; the
-  # StatefulSet mounts it read-only at /etc/tidaldb/tls.
+  # StatefulSet mounts it read-only at /etc/tidaldb/tls. ca.crt is now the ROOT
+  # CA cert (CA:TRUE), tls.crt the leaf SIGNED by it.
   secretName: tidaldb-cluster-tls
   # Renew well before expiry; each renewal is hot-swapped without a restart.
   duration: 2160h    # 90d

k8s/cluster/statefulset.yaml

@@ -81,7 +81,7 @@ spec:
               mountPath: /data
       containers:
         - name: tidaldb
-          image: registry.threesix.ai/tidal/server@sha256:8b136de21b969adedee37fdcb0cac15ecdd77b9beb5c5f31be3a5d15340323b1  # m11-44b768b (p6 sharding + p7 mTLS + p8 ops + p9 correctness)
+          image: registry.threesix.ai/tidal/server@sha256:c4d26acbd2bff33f944b4f14f37edf2ba24d63a26d27e5af41e1b1264c7280df  # m12-8e39ee1 (m12p1-p6: ANN-in-retrieve, idle-readiness, sharded reads + T4 two-tier PKI / join_boot TLS fallback)
           imagePullPolicy: IfNotPresent
           # The image ENTRYPOINT is the bare binary. We override the command with
           # a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we
@@ -206,7 +206,7 @@ spec:
               port: http
               scheme: HTTPS
             periodSeconds: 5
-            failureThreshold: 60     # ~5 min for large-DB WAL replay / index load
+            failureThreshold: 240    # ~20 min: HNSW index rebuild/load at 1536-dim is CPU-bound (100k ~5min single-core; headroom for 1M gate)
           livenessProbe:
             httpGet:
               path: /health/live
@@ -225,11 +225,11 @@ spec:
             failureThreshold: 3
           resources:
             requests:
-              cpu: "250m"
+              cpu: "500m"
-              memory: 256Mi
+              memory: 1Gi
             limits:
               cpu: "2"
-              memory: 2Gi           # size from docs/ops/capacity-planning.md
+              memory: 4Gi           # m12/1536: 100k×1536 HNSW load peaks ~1.9Gi; 1M needs more headroom (nodes have 13Gi allocatable)
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true   # writes only /data (PVC) and /tmp (emptyDir)

tidal-server/src/cluster/join_boot.rs

@@ -334,12 +334,29 @@ fn build_join_client(
 
 /// region with a `grpc_tls` block (the seed dial uses the same posture).
 fn own_grpc_tls(knobs: &TopologySpec, region: &str) -> Option<tidal_net::config::TlsConfig> {
+    grpc_tls_for(knobs, region).map(GrpcTlsSpec::to_tls_config)
+}
+
+/// The `grpc_tls` posture for `region`: its OWN block when the knob file names
+/// it, ELSE ANY region's block.
+///
+/// A seed-joiner is NEVER in the shared-ConfigMap knob file — it learns its
+/// identity from the join, so `regions:` lists only the bootstrap voters. But
+/// every pod mounts the SAME cert Secret at the SAME paths, so any region's
+/// `grpc_tls` block carries the correct CA/cert/key paths for the joiner too.
+/// Returning `None` here (the prior behavior, premised on "the joiner's region
+/// isn't named ⇒ plaintext loopback") silently dropped a TLS-cluster joiner to
+/// PLAINTEXT: it then failed CA verification on the seed-discover HTTPS dial
+/// (`invalid peer certificate: UnknownIssuer`, burning the whole 120 s window)
+/// and could not mTLS-replicate with its TLS peers. The fallback is safe: with
+/// no `grpc_tls` anywhere it is still `None` (the genuine plaintext case).
+fn grpc_tls_for<'a>(knobs: &'a TopologySpec, region: &str) -> Option<&'a GrpcTlsSpec> {
     knobs
         .regions
         .iter()
         .find(|r| r.name == region)
         .and_then(|r| r.grpc_tls.as_ref())
-        .map(GrpcTlsSpec::to_tls_config)
+        .or_else(|| knobs.regions.iter().find_map(|r| r.grpc_tls.as_ref()))
 }
 
 /// One discovery candidate: an HTTP status base + (when known) the gRPC address
@@ -406,7 +423,16 @@ fn discover_leader(
         let resp = match req.send() {
             Ok(resp) => resp,
             Err(e) => {
-                tracing::debug!(%url, error = %e, "seed-join discover: status poll failed (transport/TLS/timeout)");
+                // Walk the std::error source chain so the ROOT transport/TLS
+                // cause (hidden behind reqwest's terse Display) is visible.
+                let mut chain = format!("{e}");
+                let mut src = std::error::Error::source(&e);
+                while let Some(s) = src {
+                    chain.push_str(" -> ");
+                    chain.push_str(&s.to_string());
+                    src = s.source();
+                }
+                tracing::warn!(%url, error = %chain, "seed-join discover: status poll failed (transport/TLS/timeout)");
                 continue;
             }
         };
@@ -754,15 +780,13 @@ fn synthesize_topology(
     })
 }
 
-/// This region's TLS spec from the knob file, when it names this region (the
+/// This region's TLS spec for the synthesized topology. The joiner's own region
-/// joiner's own region is NOT in the knob file in the k8s shared-ConfigMap case,
+/// is NOT in the shared-ConfigMap knob file, so this falls back to ANY region's
-/// so this is `None` there — plaintext, the loopback/VPC default).
+/// `grpc_tls` ([`grpc_tls_for`]) — every pod mounts the same cert files at the
+/// same paths. Without the fallback a TLS-cluster joiner synthesized a PLAINTEXT
+/// gRPC posture and could not mTLS-replicate with its peers.
 fn self_tls_spec(knobs: &TopologySpec, name: &str) -> Option<GrpcTlsSpec> {
-    knobs
+    grpc_tls_for(knobs, name)
-        .regions
-        .iter()
-        .find(|r| r.name == name)
-        .and_then(|r| r.grpc_tls.as_ref())
         .map(|t| GrpcTlsSpec {
             ca_cert: t.ca_cert.clone(),
             server_cert: t.server_cert.clone(),

tidal-stress/k8s/t4-load-job.yaml

@@ -57,7 +57,7 @@ spec:
             - quorum
             - --skip-seed
             - --corpus
-            - "10000"
+            - "300"
             - --embedding-dim
             - "1536"
             - --users

tidal-stress/k8s/t4-seed-job.yaml

@@ -49,7 +49,7 @@ spec:
             - --ack
             - quorum
             - --corpus
-            - "10000"
+            - "300"
             - --embedding-dim
             - "1536"
             - --users