aa94fd9b1f
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
74 lines
3.6 KiB
Bash
Executable File
74 lines
3.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Generate the tidalDB inter-node TLS material (m11p7) for clusters NOT using
|
|
# cert-manager: a self-signed cluster CA + ONE shared node cert whose SANs cover
|
|
# every pod's stable DNS (the StatefulSet pattern). Output matches the cert-manager
|
|
# Secret shape — tls.crt / tls.key / ca.crt — so the topology grpc_tls paths and
|
|
# the k8s mount are identical either way.
|
|
#
|
|
# Usage:
|
|
# scripts/gen-cluster-certs.sh [OUT_DIR] [NAMESPACE] [STATEFULSET] [HEADLESS_SVC] [N]
|
|
# Defaults reproduce k8s/cluster/: out=./cluster-certs ns=tidaldb-cluster
|
|
# sts=tidaldb headless=tidaldb-peers N=3 (pods tidaldb-0..2).
|
|
#
|
|
# Then either mount the files at /etc/tidaldb/tls, or load them as the Secret the
|
|
# StatefulSet expects:
|
|
# kubectl -n tidaldb-cluster create secret generic tidaldb-cluster-tls \
|
|
# --from-file=tls.crt=cluster-certs/tls.crt \
|
|
# --from-file=tls.key=cluster-certs/tls.key \
|
|
# --from-file=ca.crt=cluster-certs/ca.crt
|
|
#
|
|
# Local (non-k8s) clusters point each region's grpc_tls at these files directly
|
|
# (ca_cert=ca.crt, server_cert=client_cert=tls.crt, server_key=client_key=tls.key).
|
|
set -euo pipefail
|
|
|
|
OUT_DIR="${1:-./cluster-certs}"
|
|
NAMESPACE="${2:-tidaldb-cluster}"
|
|
STS="${3:-tidaldb}"
|
|
HEADLESS="${4:-tidaldb-peers}"
|
|
N="${5:-3}"
|
|
|
|
mkdir -p "$OUT_DIR"
|
|
cd "$OUT_DIR"
|
|
|
|
# The CA ROOT private key is the cluster's crown jewel — anyone holding it can
|
|
# mint any node identity and forge the whole mTLS + node-token trust. Keep it in a
|
|
# SEPARATE, more-protected directory so it never lands in the secret-mount glob
|
|
# (the deployable Secret is only ca.crt / tls.crt / tls.key). Move ca-private/ to
|
|
# offline storage immediately after issuance.
|
|
CA_DIR="ca-private"
|
|
mkdir -p "$CA_DIR"
|
|
chmod 700 "$CA_DIR"
|
|
|
|
# Build the SAN list: a WILDCARD over every pod's stable headless-Service DNS
|
|
# (covers any ordinal, so scaling the StatefulSet needs no cert re-issue — m12p5;
|
|
# rustls/webpki matches it against the single leftmost label `tidaldb-N` per
|
|
# RFC 6125), plus the headless and client Services + loopback (a local 127.0.0.1
|
|
# cluster also verifies). Mirrors k8s/cluster/certs.yaml's wildcard dnsNames. The
|
|
# explicit per-pod entries (0..N-1) are belt-and-suspenders for any strict
|
|
# verifier that distrusts wildcard-only leaves.
|
|
SANS="DNS:*.${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${HEADLESS}.${NAMESPACE}.svc.cluster.local,DNS:${STS}.${NAMESPACE}.svc.cluster.local,DNS:localhost,IP:127.0.0.1"
|
|
for i in $(seq 0 $((N - 1))); do
|
|
SANS="${SANS},DNS:${STS}-${i}.${HEADLESS}.${NAMESPACE}.svc.cluster.local"
|
|
done
|
|
|
|
echo "==> cluster CA (private key -> ${CA_DIR}/ca.key, NOT a deployable file)"
|
|
openssl genrsa -out "${CA_DIR}/ca.key" 4096 2>/dev/null
|
|
openssl req -x509 -new -nodes -key "${CA_DIR}/ca.key" -sha256 -days 3650 \
|
|
-subj "/CN=tidaldb-cluster-ca" -out ca.crt
|
|
|
|
echo "==> node leaf (SANs: ${SANS})"
|
|
openssl genrsa -out tls.key 4096 2>/dev/null
|
|
openssl req -new -key tls.key -subj "/CN=tidaldb-cluster" -out node.csr
|
|
# serverAuth + clientAuth so the same leaf is the gRPC mTLS client identity AND
|
|
# the gRPC/HTTP server identity.
|
|
openssl x509 -req -in node.csr -CA ca.crt -CAkey "${CA_DIR}/ca.key" -CAcreateserial \
|
|
-days 825 -sha256 -out tls.crt \
|
|
-extfile <(printf 'subjectAltName=%s\nextendedKeyUsage=serverAuth,clientAuth\n' "$SANS")
|
|
rm -f node.csr "${CA_DIR}/ca.srl"
|
|
|
|
chmod 600 "${CA_DIR}/ca.key" tls.key
|
|
echo "==> wrote DEPLOYABLE ${OUT_DIR}/{ca.crt,tls.crt,tls.key}"
|
|
echo " CA ROOT KEY at ${OUT_DIR}/${CA_DIR}/ca.key — move it to OFFLINE storage now;"
|
|
echo " it must NEVER be copied to a pod or included in the cluster Secret."
|
|
echo " grpc_tls: ca_cert=ca.crt server_cert=client_cert=tls.crt server_key=client_key=tls.key"
|