jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
tidaldb/k8s/cluster/certs.yaml
jx12n 4db3f1e597 fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback 
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.

- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
  Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
  the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
  Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
  (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
  joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
  is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
  with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
  Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
  a cold TLS handshake under contention blew the sub-second budget. Discovery now
  logs each poll failure at WARN with the full error source chain (a silent loop
  made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.

Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
2026-06-14 22:41:59 -06:00

124 lines
5.2 KiB
YAML
Raw Permalink Blame History

# m11p7 inter-node TLS — cluster CA + the shared node cert (cert-manager).
#
# The whole cluster runs over mutual TLS: gRPC replication (mTLS — client certs
# required) and the inter-node HTTP plane (server TLS + signed node tokens). This
# manifest provisions the material with cert-manager so renewal is automatic and
# hot — cert-manager rewrites the mounted Secret, the kubelet swaps the `..data`
# symlink, and tidalDB's content-hash cert poller hot-swaps the in-memory cert
# with ZERO connection drop (no pod restart). See docs/runbooks/cluster.md §10.
#
# Prereq: cert-manager installed in the cluster (https://cert-manager.io). If you
# do not run cert-manager, provision the same Secret out-of-band — e.g. with
# `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`,
# `ca.crt`.
#
# ONE shared node cert with a WILDCARD pod SAN (the standard StatefulSet pattern):
# any pod may present it for its own DNS name, and a peer dialing
# `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. The
# wildcard `*.tidaldb-peers...` covers EVERY pod ordinal (tidaldb-0, -1, … -N),
# so scaling the StatefulSet up or down with `--seed` needs NO cert re-issue
# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
# A REAL two-tier PKI (not a self-signed leaf). cert-manager's `selfSigned`
# issuer bootstraps a CA *certificate* (isCA: true); a `ca:` issuer backed by
# that CA then signs the node leaf. The leaf Secret's `ca.crt` is therefore the
# CA cert (basicConstraints CA:TRUE) — a valid trust anchor.
#
# m12p5: the prior shape issued the node leaf DIRECTLY from a `selfSigned` issuer,
# producing a self-signed end-entity cert with CA:FALSE whose `ca.crt` is a COPY
# of the leaf. Lenient TLS stacks (reqwest async, openssl-with-`-k`) tolerated it,
# but a STRICT webpki verifier — exactly what the seed-join blocking reqwest client
# uses — rejected the peer cert as `UnknownIssuer`, so a scale-up joiner could
# never discover a leader (it burned the whole 120 s window, crash-looping). A
# CA:TRUE anchor verifies cleanly for strict AND lenient clients alike.
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: tidaldb-selfsigned-bootstrap
namespace: tidaldb-cluster
labels:
app.kubernetes.io/name: tidaldb
app.kubernetes.io/part-of: tidaldb
spec:
# Bootstrap only: signs the CA certificate below (which IS a CA). Swap the CA
# cert's issuerRef for your org PKI to chain to an existing root instead.
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tidaldb-cluster-ca
namespace: tidaldb-cluster
labels:
app.kubernetes.io/name: tidaldb
app.kubernetes.io/part-of: tidaldb
spec:
# The cluster's private ROOT CA — a real CA cert (basicConstraints CA:TRUE).
# cert-manager writes its cert+key into this Secret; the `ca:` issuer signs the
# node leaf with it.
isCA: true
commonName: tidaldb-cluster-ca
secretName: tidaldb-cluster-ca
duration: 87600h # 10y — the root outlives many leaf rotations
renewBefore: 8760h # 1y
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: tidaldb-selfsigned-bootstrap
kind: Issuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: tidaldb-cluster-ca-issuer
namespace: tidaldb-cluster
labels:
app.kubernetes.io/name: tidaldb
app.kubernetes.io/part-of: tidaldb
spec:
# CA issuer backed by the root CA above — signs the node leaf, so the leaf
# chains to a CA:TRUE anchor.
ca:
secretName: tidaldb-cluster-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tidaldb-cluster-tls
namespace: tidaldb-cluster
labels:
app.kubernetes.io/name: tidaldb
app.kubernetes.io/part-of: tidaldb
spec:
# cert-manager writes tls.crt / tls.key / ca.crt into this Secret; the
# StatefulSet mounts it read-only at /etc/tidaldb/tls. ca.crt is now the ROOT
# CA cert (CA:TRUE), tls.crt the leaf SIGNED by it.
secretName: tidaldb-cluster-tls
# Renew well before expiry; each renewal is hot-swapped without a restart.
duration: 2160h # 90d
renewBefore: 720h # 30d
isCA: false
usages:
- server auth # the gRPC + HTTP server identity
- client auth # the gRPC mTLS client identity (peer dials)
# SANs: a WILDCARD over every pod's stable headless-Service DNS (covers any
# ordinal, so scale-up/down needs no cert re-issue), the explicit initial-pod
# names (belt-and-suspenders for any strict verifier that distrusts a wildcard-
# only leaf), plus the headless and client Services. Keep
# `scripts/gen-cluster-certs.sh` in sync.
dnsNames:
- "*.tidaldb-peers.tidaldb-cluster.svc.cluster.local"
- tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb-peers.tidaldb-cluster.svc.cluster.local
- tidaldb.tidaldb-cluster.svc.cluster.local
issuerRef:
name: tidaldb-cluster-ca-issuer
kind: Issuer
group: cert-manager.io
Reference in New Issue View Git Blame Copy Permalink