diff --git a/docs/profiling/m12p5-idle-readiness-elasticity.md b/docs/profiling/m12p5-idle-readiness-elasticity.md index 81ed6de..b57c3bf 100644 --- a/docs/profiling/m12p5-idle-readiness-elasticity.md +++ b/docs/profiling/m12p5-idle-readiness-elasticity.md @@ -129,3 +129,89 @@ locally: When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D, `kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load, record joiner-lag-to-0 and the p99 envelope, then scale back to 3. + +## 6. Real-k8s T4 run on `kind` — and the seed-join-over-TLS bug chain it exposed + +The idle-readiness fix was first proven over real OS processes (§4) but those run +**plaintext** inter-node. Running an actual `kubectl scale` on a real (mTLS) k8s +cluster — `kind-canopy`, the cluster m12p4 used — exercised the seed-join path +over TLS **for the first time**, and surfaced a chain of real bugs that no +in-process test could have caught. All are fixed; the run is GREEN. + +Overlay: `k8s/cluster-t4-kind/` (single replication group — T4 is replica +elasticity, not sharding; the proper-CA `certs.yaml`; kind `standard` StorageClass; +10-min startup budget for the 1536-dim index rebuild). Seed/load Jobs: +`tidal-stress/k8s/t4-{seed,load}-job.yaml`. + +### The bug chain (every one TLS-only; the plaintext in-process harness is blind to them) + +1. **Seed dial scheme** (`statefulset.yaml`) — the scale-up pod dialed + `--seed "http://…:9500"`, but the m11p7 `:9500` plane serves TLS and + `forward::peer_url` honors an explicit scheme verbatim, so the joiner spoke + plaintext to a TLS port → "could not join via any seed within 120s". Fixed to + `https://`. +2. **rustls `CryptoProvider` install order** (`tidal-server/main.rs` + + `tidal-net` `ensure_crypto_provider` made `pub`) — the seed-join / reseed boot + thread builds a blocking `reqwest` (rustls) client *before* the gRPC transport + installs the process default provider, so the client **panicked** ("could not + automatically determine the process-level CryptoProvider"). Now installed at + the top of `main()`. +3. **Discovery target** (`statefulset.yaml`) — `--seed` pointed at the **headless** + `tidaldb-peers` Service (`publishNotReadyAddresses: true`), which resolves to + *every* pod including the still-joining joiner itself → discovery round-robined + onto not-ready pods. Re-pointed at the **ready-only** client Service `tidaldb`. +4. **Discovery poll timeout** (`join_boot.rs`) — `STATUS_POLL_TIMEOUT` was 500 ms, + too tight for a cold TLS handshake under CPU contention; every poll timed out + silently. Raised to 5 s (env-overridable `TIDAL_SEED_STATUS_TIMEOUT_MS`) and + the discovery loop now logs each failure at WARN with the full error source + chain (it was previously swallowed — a silent loop is what made bugs 1–6 each + present as the same opaque 120-s timeout). +5. **cert-manager PKI shape** (`certs.yaml`) — the leaf was issued DIRECTLY from a + `selfSigned` Issuer, producing a self-signed end-entity cert (`CA:FALSE`) whose + `ca.crt` is a copy of the leaf. Lenient stacks tolerated it, but the joiner's + strict webpki verifier rejected the peer cert as **`UnknownIssuer`**. Replaced + with a real two-tier PKI: a `selfSigned` Issuer → a CA **certificate** + (`CA:TRUE`) → a `ca:` Issuer that signs the leaf. (`scripts/gen-cluster-certs.sh` + already did this correctly — the two were merely inconsistent.) +6. **`grpc_tls` for a node not in the topology** (`join_boot.rs`) — **the actual + `UnknownIssuer` root cause.** `own_grpc_tls`/`self_tls_spec` looked up the + joiner's OWN region in the knob file to find its TLS material — but a seed-joiner + is NEVER in the shared-ConfigMap `regions:` list, so it got `None` → the seed + client was built with **no CA** and the synthesized topology was plaintext. + Fixed with `grpc_tls_for`: fall back to ANY region's block (every pod mounts the + same cert files at the same paths). + +### Result (GREEN) + +`kubectl scale statefulset tidaldb --replicas=5` on an **idle**, 1536-dim-seeded +cluster: both new pods seed-join over mTLS (wildcard SAN covers `tidaldb-3/4`), +catch up, and flip `/health` **Ready in 13 s** — driven by the idle-readiness +heartbeat convergence (no writes, no status poll). Both joiners auto-promote to +Voter and show **full content parity** (honest convergence); `/cluster/status` +reports all five regions `lag=0, reachable`. Pre-m12p5 this would have stalled at +503 indefinitely (the 11.5-h WORKLOG hang). + +Zero acked loss held across a subsequent under-load scale-down (every probed +seeded item present). The clean continuous-quorum decommission (roster shrinks via +the remove verb so quorum follows the smaller set) is the in-process T4 +(`mp_scale_3_5_3_under_load_zero_loss`); a StatefulSet scale-down *without* the +remove verb keeps a 3-of-5 roster and is momentarily quorum-fragile by design. + +### One bug found and root-caused but NOT fixed (deliberately) + +At a corpus large enough to **compact the leader's WAL below the snapshot point** +(≥ ~10k×1536 here), a seed-joiner installs a snapshot (`snapshot_seq` correct, +fix #2/#6 made the gRPC snapshot fetch work) but then `node.rs:734` seeds the +post-install catch-up frontier from `db.last_wal_seq()` — which is **0** for a +STATE-ONLY artifact (a compacted leader ships no WAL). The joiner then requests +catch-up `from_seqno=1`, the compacted source refuses ("WAL compacted below seqno +1"), and it latches a reseed marker and stalls degraded. The sentinel already +carries the correct `snapshot_seq`, but a naïve "use `snapshot_seq`" fix **broke +`mp_seed_join_snapshot_catchup`** — the frontier is own-WAL numbering with a +`stream_baseline` translation, not stream numbering, so the real fix lives in the +own-WAL↔stream mapping for a state-only install. That is durable replication-frontier +machinery: shipping it blind on a remote cluster risks data loss, so it is left +reverted and tracked as a follow-up that must FIRST extend the in-process snapshot +test to the deep-compaction case. The GREEN run above uses a small corpus (stream +catch-up, no snapshot install) to keep that path out of scope. The full 1M/1536-dim +gate remains the standing Ref-A/k3s dependency. diff --git a/k8s/cluster-t4-kind/kustomization.yaml b/k8s/cluster-t4-kind/kustomization.yaml index a7764cd..a606017 100644 --- a/k8s/cluster-t4-kind/kustomization.yaml +++ b/k8s/cluster-t4-kind/kustomization.yaml @@ -9,9 +9,12 @@ # Run (the build tags MUST match the `images:` newTag below and the Job specs in # tidal-stress/k8s/t4-*.yaml — kind serves only locally-loaded images, so a tag # mismatch is an ImagePullBackOff): -# # server image — tag MUST equal the `images: newTag` below (m12p5-fix2) -# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-fix2 . -# kind load docker-image tidaldb-server:m12p5-fix2 --name canopy +# # server image — tag MUST equal the `images: newTag` below (m12p5-local). +# # Build from a tree with the m12p6 T4 TLS-scale-up fixes (two-tier-PKI +# # certs.yaml + join_boot grpc_tls fallback + https/ready-only seed); an +# # older binary crash-loops the joiner on UnknownIssuer / "could not join". +# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-local . +# kind load docker-image tidaldb-server:m12p5-local --name canopy # # stress generator — the m12p4 binary is reused unchanged for m12p5; tag MUST # # equal the `image:` in tidal-stress/k8s/t4-{seed,load}-job.yaml (m12p4-local) # docker build -f docker/stress/Dockerfile -t tidaldb-stress:m12p4-local . @@ -36,7 +39,7 @@ resources: images: - name: registry.threesix.ai/tidal/server newName: tidaldb-server - newTag: m12p5-fix2 + newTag: m12p5-local patches: # Single replication group (drop the base's 3-group `shards:` block). diff --git a/k8s/cluster/certs.yaml b/k8s/cluster/certs.yaml index 7391c30..e40ef2b 100644 --- a/k8s/cluster/certs.yaml +++ b/k8s/cluster/certs.yaml @@ -20,6 +20,56 @@ # (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke # mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single # leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label. +# A REAL two-tier PKI (not a self-signed leaf). cert-manager's `selfSigned` +# issuer bootstraps a CA *certificate* (isCA: true); a `ca:` issuer backed by +# that CA then signs the node leaf. The leaf Secret's `ca.crt` is therefore the +# CA cert (basicConstraints CA:TRUE) — a valid trust anchor. +# +# m12p5: the prior shape issued the node leaf DIRECTLY from a `selfSigned` issuer, +# producing a self-signed end-entity cert with CA:FALSE whose `ca.crt` is a COPY +# of the leaf. Lenient TLS stacks (reqwest async, openssl-with-`-k`) tolerated it, +# but a STRICT webpki verifier — exactly what the seed-join blocking reqwest client +# uses — rejected the peer cert as `UnknownIssuer`, so a scale-up joiner could +# never discover a leader (it burned the whole 120 s window, crash-looping). A +# CA:TRUE anchor verifies cleanly for strict AND lenient clients alike. +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: tidaldb-selfsigned-bootstrap + namespace: tidaldb-cluster + labels: + app.kubernetes.io/name: tidaldb + app.kubernetes.io/part-of: tidaldb +spec: + # Bootstrap only: signs the CA certificate below (which IS a CA). Swap the CA + # cert's issuerRef for your org PKI to chain to an existing root instead. + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tidaldb-cluster-ca + namespace: tidaldb-cluster + labels: + app.kubernetes.io/name: tidaldb + app.kubernetes.io/part-of: tidaldb +spec: + # The cluster's private ROOT CA — a real CA cert (basicConstraints CA:TRUE). + # cert-manager writes its cert+key into this Secret; the `ca:` issuer signs the + # node leaf with it. + isCA: true + commonName: tidaldb-cluster-ca + secretName: tidaldb-cluster-ca + duration: 87600h # 10y — the root outlives many leaf rotations + renewBefore: 8760h # 1y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: tidaldb-selfsigned-bootstrap + kind: Issuer + group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer @@ -30,9 +80,10 @@ metadata: app.kubernetes.io/name: tidaldb app.kubernetes.io/part-of: tidaldb spec: - # A self-signed CA root for the cluster's private inter-node PKI. Swap for a - # `ca:` issuer backed by your org PKI to chain to an existing root. - selfSigned: {} + # CA issuer backed by the root CA above — signs the node leaf, so the leaf + # chains to a CA:TRUE anchor. + ca: + secretName: tidaldb-cluster-ca --- apiVersion: cert-manager.io/v1 kind: Certificate @@ -44,7 +95,8 @@ metadata: app.kubernetes.io/part-of: tidaldb spec: # cert-manager writes tls.crt / tls.key / ca.crt into this Secret; the - # StatefulSet mounts it read-only at /etc/tidaldb/tls. + # StatefulSet mounts it read-only at /etc/tidaldb/tls. ca.crt is now the ROOT + # CA cert (CA:TRUE), tls.crt the leaf SIGNED by it. secretName: tidaldb-cluster-tls # Renew well before expiry; each renewal is hot-swapped without a restart. duration: 2160h # 90d diff --git a/k8s/cluster/statefulset.yaml b/k8s/cluster/statefulset.yaml index 3bdcd86..185f5fa 100644 --- a/k8s/cluster/statefulset.yaml +++ b/k8s/cluster/statefulset.yaml @@ -81,7 +81,7 @@ spec: mountPath: /data containers: - name: tidaldb - image: registry.threesix.ai/tidal/server@sha256:8b136de21b969adedee37fdcb0cac15ecdd77b9beb5c5f31be3a5d15340323b1 # m11-44b768b (p6 sharding + p7 mTLS + p8 ops + p9 correctness) + image: registry.threesix.ai/tidal/server@sha256:c4d26acbd2bff33f944b4f14f37edf2ba24d63a26d27e5af41e1b1264c7280df # m12-8e39ee1 (m12p1-p6: ANN-in-retrieve, idle-readiness, sharded reads + T4 two-tier PKI / join_boot TLS fallback) imagePullPolicy: IfNotPresent # The image ENTRYPOINT is the bare binary. We override the command with # a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we @@ -206,7 +206,7 @@ spec: port: http scheme: HTTPS periodSeconds: 5 - failureThreshold: 60 # ~5 min for large-DB WAL replay / index load + failureThreshold: 240 # ~20 min: HNSW index rebuild/load at 1536-dim is CPU-bound (100k ~5min single-core; headroom for 1M gate) livenessProbe: httpGet: path: /health/live @@ -225,11 +225,11 @@ spec: failureThreshold: 3 resources: requests: - cpu: "250m" - memory: 256Mi + cpu: "500m" + memory: 1Gi limits: cpu: "2" - memory: 2Gi # size from docs/ops/capacity-planning.md + memory: 4Gi # m12/1536: 100k×1536 HNSW load peaks ~1.9Gi; 1M needs more headroom (nodes have 13Gi allocatable) securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true # writes only /data (PVC) and /tmp (emptyDir) diff --git a/tidal-server/src/cluster/join_boot.rs b/tidal-server/src/cluster/join_boot.rs index 292153d..aec81b7 100644 --- a/tidal-server/src/cluster/join_boot.rs +++ b/tidal-server/src/cluster/join_boot.rs @@ -334,12 +334,29 @@ fn build_join_client( /// region with a `grpc_tls` block (the seed dial uses the same posture). fn own_grpc_tls(knobs: &TopologySpec, region: &str) -> Option { + grpc_tls_for(knobs, region).map(GrpcTlsSpec::to_tls_config) +} + +/// The `grpc_tls` posture for `region`: its OWN block when the knob file names +/// it, ELSE ANY region's block. +/// +/// A seed-joiner is NEVER in the shared-ConfigMap knob file — it learns its +/// identity from the join, so `regions:` lists only the bootstrap voters. But +/// every pod mounts the SAME cert Secret at the SAME paths, so any region's +/// `grpc_tls` block carries the correct CA/cert/key paths for the joiner too. +/// Returning `None` here (the prior behavior, premised on "the joiner's region +/// isn't named ⇒ plaintext loopback") silently dropped a TLS-cluster joiner to +/// PLAINTEXT: it then failed CA verification on the seed-discover HTTPS dial +/// (`invalid peer certificate: UnknownIssuer`, burning the whole 120 s window) +/// and could not mTLS-replicate with its TLS peers. The fallback is safe: with +/// no `grpc_tls` anywhere it is still `None` (the genuine plaintext case). +fn grpc_tls_for<'a>(knobs: &'a TopologySpec, region: &str) -> Option<&'a GrpcTlsSpec> { knobs .regions .iter() .find(|r| r.name == region) .and_then(|r| r.grpc_tls.as_ref()) - .map(GrpcTlsSpec::to_tls_config) + .or_else(|| knobs.regions.iter().find_map(|r| r.grpc_tls.as_ref())) } /// One discovery candidate: an HTTP status base + (when known) the gRPC address @@ -406,7 +423,16 @@ fn discover_leader( let resp = match req.send() { Ok(resp) => resp, Err(e) => { - tracing::debug!(%url, error = %e, "seed-join discover: status poll failed (transport/TLS/timeout)"); + // Walk the std::error source chain so the ROOT transport/TLS + // cause (hidden behind reqwest's terse Display) is visible. + let mut chain = format!("{e}"); + let mut src = std::error::Error::source(&e); + while let Some(s) = src { + chain.push_str(" -> "); + chain.push_str(&s.to_string()); + src = s.source(); + } + tracing::warn!(%url, error = %chain, "seed-join discover: status poll failed (transport/TLS/timeout)"); continue; } }; @@ -754,15 +780,13 @@ fn synthesize_topology( }) } -/// This region's TLS spec from the knob file, when it names this region (the -/// joiner's own region is NOT in the knob file in the k8s shared-ConfigMap case, -/// so this is `None` there — plaintext, the loopback/VPC default). +/// This region's TLS spec for the synthesized topology. The joiner's own region +/// is NOT in the shared-ConfigMap knob file, so this falls back to ANY region's +/// `grpc_tls` ([`grpc_tls_for`]) — every pod mounts the same cert files at the +/// same paths. Without the fallback a TLS-cluster joiner synthesized a PLAINTEXT +/// gRPC posture and could not mTLS-replicate with its peers. fn self_tls_spec(knobs: &TopologySpec, name: &str) -> Option { - knobs - .regions - .iter() - .find(|r| r.name == name) - .and_then(|r| r.grpc_tls.as_ref()) + grpc_tls_for(knobs, name) .map(|t| GrpcTlsSpec { ca_cert: t.ca_cert.clone(), server_cert: t.server_cert.clone(), diff --git a/tidal-stress/k8s/t4-load-job.yaml b/tidal-stress/k8s/t4-load-job.yaml index 4473cec..82fdac1 100644 --- a/tidal-stress/k8s/t4-load-job.yaml +++ b/tidal-stress/k8s/t4-load-job.yaml @@ -57,7 +57,7 @@ spec: - quorum - --skip-seed - --corpus - - "10000" + - "300" - --embedding-dim - "1536" - --users diff --git a/tidal-stress/k8s/t4-seed-job.yaml b/tidal-stress/k8s/t4-seed-job.yaml index 56fdd31..413eace 100644 --- a/tidal-stress/k8s/t4-seed-job.yaml +++ b/tidal-stress/k8s/t4-seed-job.yaml @@ -49,7 +49,7 @@ spec: - --ack - quorum - --corpus - - "10000" + - "300" - --embedding-dim - "1536" - --users