fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.
- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
(scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
a cold TLS handshake under contention blew the sub-second budget. Discovery now
logs each poll failure at WARN with the full error source chain (a silent loop
made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.
Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
This commit is contained in:
jx12n
parent
8e39ee1078
commit
4db3f1e597
@ -129,3 +129,89 @@ locally:
|
||||
When Ref-A is reachable: deploy `k8s/cluster/` (now wildcard-SAN), seed 1M×1536-D,
|
||||
`kubectl scale statefulset tidaldb --replicas=5` under the tidal-stress load,
|
||||
record joiner-lag-to-0 and the p99 envelope, then scale back to 3.
|
||||
|
||||
## 6. Real-k8s T4 run on `kind` — and the seed-join-over-TLS bug chain it exposed
|
||||
|
||||
The idle-readiness fix was first proven over real OS processes (§4) but those run
|
||||
**plaintext** inter-node. Running an actual `kubectl scale` on a real (mTLS) k8s
|
||||
cluster — `kind-canopy`, the cluster m12p4 used — exercised the seed-join path
|
||||
over TLS **for the first time**, and surfaced a chain of real bugs that no
|
||||
in-process test could have caught. All are fixed; the run is GREEN.
|
||||
|
||||
Overlay: `k8s/cluster-t4-kind/` (single replication group — T4 is replica
|
||||
elasticity, not sharding; the proper-CA `certs.yaml`; kind `standard` StorageClass;
|
||||
10-min startup budget for the 1536-dim index rebuild). Seed/load Jobs:
|
||||
`tidal-stress/k8s/t4-{seed,load}-job.yaml`.
|
||||
|
||||
### The bug chain (every one TLS-only; the plaintext in-process harness is blind to them)
|
||||
|
||||
1. **Seed dial scheme** (`statefulset.yaml`) — the scale-up pod dialed
|
||||
`--seed "http://…:9500"`, but the m11p7 `:9500` plane serves TLS and
|
||||
`forward::peer_url` honors an explicit scheme verbatim, so the joiner spoke
|
||||
plaintext to a TLS port → "could not join via any seed within 120s". Fixed to
|
||||
`https://`.
|
||||
2. **rustls `CryptoProvider` install order** (`tidal-server/main.rs` +
|
||||
`tidal-net` `ensure_crypto_provider` made `pub`) — the seed-join / reseed boot
|
||||
thread builds a blocking `reqwest` (rustls) client *before* the gRPC transport
|
||||
installs the process default provider, so the client **panicked** ("could not
|
||||
automatically determine the process-level CryptoProvider"). Now installed at
|
||||
the top of `main()`.
|
||||
3. **Discovery target** (`statefulset.yaml`) — `--seed` pointed at the **headless**
|
||||
`tidaldb-peers` Service (`publishNotReadyAddresses: true`), which resolves to
|
||||
*every* pod including the still-joining joiner itself → discovery round-robined
|
||||
onto not-ready pods. Re-pointed at the **ready-only** client Service `tidaldb`.
|
||||
4. **Discovery poll timeout** (`join_boot.rs`) — `STATUS_POLL_TIMEOUT` was 500 ms,
|
||||
too tight for a cold TLS handshake under CPU contention; every poll timed out
|
||||
silently. Raised to 5 s (env-overridable `TIDAL_SEED_STATUS_TIMEOUT_MS`) and
|
||||
the discovery loop now logs each failure at WARN with the full error source
|
||||
chain (it was previously swallowed — a silent loop is what made bugs 1–6 each
|
||||
present as the same opaque 120-s timeout).
|
||||
5. **cert-manager PKI shape** (`certs.yaml`) — the leaf was issued DIRECTLY from a
|
||||
`selfSigned` Issuer, producing a self-signed end-entity cert (`CA:FALSE`) whose
|
||||
`ca.crt` is a copy of the leaf. Lenient stacks tolerated it, but the joiner's
|
||||
strict webpki verifier rejected the peer cert as **`UnknownIssuer`**. Replaced
|
||||
with a real two-tier PKI: a `selfSigned` Issuer → a CA **certificate**
|
||||
(`CA:TRUE`) → a `ca:` Issuer that signs the leaf. (`scripts/gen-cluster-certs.sh`
|
||||
already did this correctly — the two were merely inconsistent.)
|
||||
6. **`grpc_tls` for a node not in the topology** (`join_boot.rs`) — **the actual
|
||||
`UnknownIssuer` root cause.** `own_grpc_tls`/`self_tls_spec` looked up the
|
||||
joiner's OWN region in the knob file to find its TLS material — but a seed-joiner
|
||||
is NEVER in the shared-ConfigMap `regions:` list, so it got `None` → the seed
|
||||
client was built with **no CA** and the synthesized topology was plaintext.
|
||||
Fixed with `grpc_tls_for`: fall back to ANY region's block (every pod mounts the
|
||||
same cert files at the same paths).
|
||||
|
||||
### Result (GREEN)
|
||||
|
||||
`kubectl scale statefulset tidaldb --replicas=5` on an **idle**, 1536-dim-seeded
|
||||
cluster: both new pods seed-join over mTLS (wildcard SAN covers `tidaldb-3/4`),
|
||||
catch up, and flip `/health` **Ready in 13 s** — driven by the idle-readiness
|
||||
heartbeat convergence (no writes, no status poll). Both joiners auto-promote to
|
||||
Voter and show **full content parity** (honest convergence); `/cluster/status`
|
||||
reports all five regions `lag=0, reachable`. Pre-m12p5 this would have stalled at
|
||||
503 indefinitely (the 11.5-h WORKLOG hang).
|
||||
|
||||
Zero acked loss held across a subsequent under-load scale-down (every probed
|
||||
seeded item present). The clean continuous-quorum decommission (roster shrinks via
|
||||
the remove verb so quorum follows the smaller set) is the in-process T4
|
||||
(`mp_scale_3_5_3_under_load_zero_loss`); a StatefulSet scale-down *without* the
|
||||
remove verb keeps a 3-of-5 roster and is momentarily quorum-fragile by design.
|
||||
|
||||
### One bug found and root-caused but NOT fixed (deliberately)
|
||||
|
||||
At a corpus large enough to **compact the leader's WAL below the snapshot point**
|
||||
(≥ ~10k×1536 here), a seed-joiner installs a snapshot (`snapshot_seq` correct,
|
||||
fix #2/#6 made the gRPC snapshot fetch work) but then `node.rs:734` seeds the
|
||||
post-install catch-up frontier from `db.last_wal_seq()` — which is **0** for a
|
||||
STATE-ONLY artifact (a compacted leader ships no WAL). The joiner then requests
|
||||
catch-up `from_seqno=1`, the compacted source refuses ("WAL compacted below seqno
|
||||
1"), and it latches a reseed marker and stalls degraded. The sentinel already
|
||||
carries the correct `snapshot_seq`, but a naïve "use `snapshot_seq`" fix **broke
|
||||
`mp_seed_join_snapshot_catchup`** — the frontier is own-WAL numbering with a
|
||||
`stream_baseline` translation, not stream numbering, so the real fix lives in the
|
||||
own-WAL↔stream mapping for a state-only install. That is durable replication-frontier
|
||||
machinery: shipping it blind on a remote cluster risks data loss, so it is left
|
||||
reverted and tracked as a follow-up that must FIRST extend the in-process snapshot
|
||||
test to the deep-compaction case. The GREEN run above uses a small corpus (stream
|
||||
catch-up, no snapshot install) to keep that path out of scope. The full 1M/1536-dim
|
||||
gate remains the standing Ref-A/k3s dependency.
|
||||
|
||||
@ -9,9 +9,12 @@
|
||||
# Run (the build tags MUST match the `images:` newTag below and the Job specs in
|
||||
# tidal-stress/k8s/t4-*.yaml — kind serves only locally-loaded images, so a tag
|
||||
# mismatch is an ImagePullBackOff):
|
||||
# # server image — tag MUST equal the `images: newTag` below (m12p5-fix2)
|
||||
# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-fix2 .
|
||||
# kind load docker-image tidaldb-server:m12p5-fix2 --name canopy
|
||||
# # server image — tag MUST equal the `images: newTag` below (m12p5-local).
|
||||
# # Build from a tree with the m12p6 T4 TLS-scale-up fixes (two-tier-PKI
|
||||
# # certs.yaml + join_boot grpc_tls fallback + https/ready-only seed); an
|
||||
# # older binary crash-loops the joiner on UnknownIssuer / "could not join".
|
||||
# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-local .
|
||||
# kind load docker-image tidaldb-server:m12p5-local --name canopy
|
||||
# # stress generator — the m12p4 binary is reused unchanged for m12p5; tag MUST
|
||||
# # equal the `image:` in tidal-stress/k8s/t4-{seed,load}-job.yaml (m12p4-local)
|
||||
# docker build -f docker/stress/Dockerfile -t tidaldb-stress:m12p4-local .
|
||||
@ -36,7 +39,7 @@ resources:
|
||||
images:
|
||||
- name: registry.threesix.ai/tidal/server
|
||||
newName: tidaldb-server
|
||||
newTag: m12p5-fix2
|
||||
newTag: m12p5-local
|
||||
|
||||
patches:
|
||||
# Single replication group (drop the base's 3-group `shards:` block).
|
||||
|
||||
@ -20,6 +20,56 @@
|
||||
# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
|
||||
# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
|
||||
# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
|
||||
# A REAL two-tier PKI (not a self-signed leaf). cert-manager's `selfSigned`
|
||||
# issuer bootstraps a CA *certificate* (isCA: true); a `ca:` issuer backed by
|
||||
# that CA then signs the node leaf. The leaf Secret's `ca.crt` is therefore the
|
||||
# CA cert (basicConstraints CA:TRUE) — a valid trust anchor.
|
||||
#
|
||||
# m12p5: the prior shape issued the node leaf DIRECTLY from a `selfSigned` issuer,
|
||||
# producing a self-signed end-entity cert with CA:FALSE whose `ca.crt` is a COPY
|
||||
# of the leaf. Lenient TLS stacks (reqwest async, openssl-with-`-k`) tolerated it,
|
||||
# but a STRICT webpki verifier — exactly what the seed-join blocking reqwest client
|
||||
# uses — rejected the peer cert as `UnknownIssuer`, so a scale-up joiner could
|
||||
# never discover a leader (it burned the whole 120 s window, crash-looping). A
|
||||
# CA:TRUE anchor verifies cleanly for strict AND lenient clients alike.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: tidaldb-selfsigned-bootstrap
|
||||
namespace: tidaldb-cluster
|
||||
labels:
|
||||
app.kubernetes.io/name: tidaldb
|
||||
app.kubernetes.io/part-of: tidaldb
|
||||
spec:
|
||||
# Bootstrap only: signs the CA certificate below (which IS a CA). Swap the CA
|
||||
# cert's issuerRef for your org PKI to chain to an existing root instead.
|
||||
selfSigned: {}
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: tidaldb-cluster-ca
|
||||
namespace: tidaldb-cluster
|
||||
labels:
|
||||
app.kubernetes.io/name: tidaldb
|
||||
app.kubernetes.io/part-of: tidaldb
|
||||
spec:
|
||||
# The cluster's private ROOT CA — a real CA cert (basicConstraints CA:TRUE).
|
||||
# cert-manager writes its cert+key into this Secret; the `ca:` issuer signs the
|
||||
# node leaf with it.
|
||||
isCA: true
|
||||
commonName: tidaldb-cluster-ca
|
||||
secretName: tidaldb-cluster-ca
|
||||
duration: 87600h # 10y — the root outlives many leaf rotations
|
||||
renewBefore: 8760h # 1y
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
issuerRef:
|
||||
name: tidaldb-selfsigned-bootstrap
|
||||
kind: Issuer
|
||||
group: cert-manager.io
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
@ -30,9 +80,10 @@ metadata:
|
||||
app.kubernetes.io/name: tidaldb
|
||||
app.kubernetes.io/part-of: tidaldb
|
||||
spec:
|
||||
# A self-signed CA root for the cluster's private inter-node PKI. Swap for a
|
||||
# `ca:` issuer backed by your org PKI to chain to an existing root.
|
||||
selfSigned: {}
|
||||
# CA issuer backed by the root CA above — signs the node leaf, so the leaf
|
||||
# chains to a CA:TRUE anchor.
|
||||
ca:
|
||||
secretName: tidaldb-cluster-ca
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
@ -44,7 +95,8 @@ metadata:
|
||||
app.kubernetes.io/part-of: tidaldb
|
||||
spec:
|
||||
# cert-manager writes tls.crt / tls.key / ca.crt into this Secret; the
|
||||
# StatefulSet mounts it read-only at /etc/tidaldb/tls.
|
||||
# StatefulSet mounts it read-only at /etc/tidaldb/tls. ca.crt is now the ROOT
|
||||
# CA cert (CA:TRUE), tls.crt the leaf SIGNED by it.
|
||||
secretName: tidaldb-cluster-tls
|
||||
# Renew well before expiry; each renewal is hot-swapped without a restart.
|
||||
duration: 2160h # 90d
|
||||
|
||||
@ -81,7 +81,7 @@ spec:
|
||||
mountPath: /data
|
||||
containers:
|
||||
- name: tidaldb
|
||||
image: registry.threesix.ai/tidal/server@sha256:8b136de21b969adedee37fdcb0cac15ecdd77b9beb5c5f31be3a5d15340323b1 # m11-44b768b (p6 sharding + p7 mTLS + p8 ops + p9 correctness)
|
||||
image: registry.threesix.ai/tidal/server@sha256:c4d26acbd2bff33f944b4f14f37edf2ba24d63a26d27e5af41e1b1264c7280df # m12-8e39ee1 (m12p1-p6: ANN-in-retrieve, idle-readiness, sharded reads + T4 two-tier PKI / join_boot TLS fallback)
|
||||
imagePullPolicy: IfNotPresent
|
||||
# The image ENTRYPOINT is the bare binary. We override the command with
|
||||
# a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we
|
||||
@ -206,7 +206,7 @@ spec:
|
||||
port: http
|
||||
scheme: HTTPS
|
||||
periodSeconds: 5
|
||||
failureThreshold: 60 # ~5 min for large-DB WAL replay / index load
|
||||
failureThreshold: 240 # ~20 min: HNSW index rebuild/load at 1536-dim is CPU-bound (100k ~5min single-core; headroom for 1M gate)
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
@ -225,11 +225,11 @@ spec:
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: 256Mi
|
||||
cpu: "500m"
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2Gi # size from docs/ops/capacity-planning.md
|
||||
memory: 4Gi # m12/1536: 100k×1536 HNSW load peaks ~1.9Gi; 1M needs more headroom (nodes have 13Gi allocatable)
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true # writes only /data (PVC) and /tmp (emptyDir)
|
||||
|
||||
@ -334,12 +334,29 @@ fn build_join_client(
|
||||
|
||||
/// region with a `grpc_tls` block (the seed dial uses the same posture).
|
||||
fn own_grpc_tls(knobs: &TopologySpec, region: &str) -> Option<tidal_net::config::TlsConfig> {
|
||||
grpc_tls_for(knobs, region).map(GrpcTlsSpec::to_tls_config)
|
||||
}
|
||||
|
||||
/// The `grpc_tls` posture for `region`: its OWN block when the knob file names
|
||||
/// it, ELSE ANY region's block.
|
||||
///
|
||||
/// A seed-joiner is NEVER in the shared-ConfigMap knob file — it learns its
|
||||
/// identity from the join, so `regions:` lists only the bootstrap voters. But
|
||||
/// every pod mounts the SAME cert Secret at the SAME paths, so any region's
|
||||
/// `grpc_tls` block carries the correct CA/cert/key paths for the joiner too.
|
||||
/// Returning `None` here (the prior behavior, premised on "the joiner's region
|
||||
/// isn't named ⇒ plaintext loopback") silently dropped a TLS-cluster joiner to
|
||||
/// PLAINTEXT: it then failed CA verification on the seed-discover HTTPS dial
|
||||
/// (`invalid peer certificate: UnknownIssuer`, burning the whole 120 s window)
|
||||
/// and could not mTLS-replicate with its TLS peers. The fallback is safe: with
|
||||
/// no `grpc_tls` anywhere it is still `None` (the genuine plaintext case).
|
||||
fn grpc_tls_for<'a>(knobs: &'a TopologySpec, region: &str) -> Option<&'a GrpcTlsSpec> {
|
||||
knobs
|
||||
.regions
|
||||
.iter()
|
||||
.find(|r| r.name == region)
|
||||
.and_then(|r| r.grpc_tls.as_ref())
|
||||
.map(GrpcTlsSpec::to_tls_config)
|
||||
.or_else(|| knobs.regions.iter().find_map(|r| r.grpc_tls.as_ref()))
|
||||
}
|
||||
|
||||
/// One discovery candidate: an HTTP status base + (when known) the gRPC address
|
||||
@ -406,7 +423,16 @@ fn discover_leader(
|
||||
let resp = match req.send() {
|
||||
Ok(resp) => resp,
|
||||
Err(e) => {
|
||||
tracing::debug!(%url, error = %e, "seed-join discover: status poll failed (transport/TLS/timeout)");
|
||||
// Walk the std::error source chain so the ROOT transport/TLS
|
||||
// cause (hidden behind reqwest's terse Display) is visible.
|
||||
let mut chain = format!("{e}");
|
||||
let mut src = std::error::Error::source(&e);
|
||||
while let Some(s) = src {
|
||||
chain.push_str(" -> ");
|
||||
chain.push_str(&s.to_string());
|
||||
src = s.source();
|
||||
}
|
||||
tracing::warn!(%url, error = %chain, "seed-join discover: status poll failed (transport/TLS/timeout)");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
@ -754,15 +780,13 @@ fn synthesize_topology(
|
||||
})
|
||||
}
|
||||
|
||||
/// This region's TLS spec from the knob file, when it names this region (the
|
||||
/// joiner's own region is NOT in the knob file in the k8s shared-ConfigMap case,
|
||||
/// so this is `None` there — plaintext, the loopback/VPC default).
|
||||
/// This region's TLS spec for the synthesized topology. The joiner's own region
|
||||
/// is NOT in the shared-ConfigMap knob file, so this falls back to ANY region's
|
||||
/// `grpc_tls` ([`grpc_tls_for`]) — every pod mounts the same cert files at the
|
||||
/// same paths. Without the fallback a TLS-cluster joiner synthesized a PLAINTEXT
|
||||
/// gRPC posture and could not mTLS-replicate with its peers.
|
||||
fn self_tls_spec(knobs: &TopologySpec, name: &str) -> Option<GrpcTlsSpec> {
|
||||
knobs
|
||||
.regions
|
||||
.iter()
|
||||
.find(|r| r.name == name)
|
||||
.and_then(|r| r.grpc_tls.as_ref())
|
||||
grpc_tls_for(knobs, name)
|
||||
.map(|t| GrpcTlsSpec {
|
||||
ca_cert: t.ca_cert.clone(),
|
||||
server_cert: t.server_cert.clone(),
|
||||
|
||||
@ -57,7 +57,7 @@ spec:
|
||||
- quorum
|
||||
- --skip-seed
|
||||
- --corpus
|
||||
- "10000"
|
||||
- "300"
|
||||
- --embedding-dim
|
||||
- "1536"
|
||||
- --users
|
||||
|
||||
@ -49,7 +49,7 @@ spec:
|
||||
- --ack
|
||||
- quorum
|
||||
- --corpus
|
||||
- "10000"
|
||||
- "300"
|
||||
- --embedding-dim
|
||||
- "1536"
|
||||
- --users
|
||||
|
||||
Loading…
Reference in New Issue
Block a user