aa94fd9b1f
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
72 lines
3.2 KiB
YAML
72 lines
3.2 KiB
YAML
# m11p7 inter-node TLS — cluster CA + the shared node cert (cert-manager).
|
|
#
|
|
# The whole cluster runs over mutual TLS: gRPC replication (mTLS — client certs
|
|
# required) and the inter-node HTTP plane (server TLS + signed node tokens). This
|
|
# manifest provisions the material with cert-manager so renewal is automatic and
|
|
# hot — cert-manager rewrites the mounted Secret, the kubelet swaps the `..data`
|
|
# symlink, and tidalDB's content-hash cert poller hot-swaps the in-memory cert
|
|
# with ZERO connection drop (no pod restart). See docs/runbooks/cluster.md §10.
|
|
#
|
|
# Prereq: cert-manager installed in the cluster (https://cert-manager.io). If you
|
|
# do not run cert-manager, provision the same Secret out-of-band — e.g. with
|
|
# `scripts/gen-cluster-certs.sh` (openssl) — keeping the keys `tls.crt`, `tls.key`,
|
|
# `ca.crt`.
|
|
#
|
|
# ONE shared node cert with a WILDCARD pod SAN (the standard StatefulSet pattern):
|
|
# any pod may present it for its own DNS name, and a peer dialing
|
|
# `tidaldb-N.tidaldb-peers...` verifies the name against the SAN list. The
|
|
# wildcard `*.tidaldb-peers...` covers EVERY pod ordinal (tidaldb-0, -1, … -N),
|
|
# so scaling the StatefulSet up or down with `--seed` needs NO cert re-issue
|
|
# (m12p5: the prior cert enumerated tidaldb-0/1/2 only, so T4 scale-to-5 broke
|
|
# mTLS on tidaldb-3/4). rustls/webpki matches a wildcard against the single
|
|
# leftmost DNS label per RFC 6125, which is exactly the pod-ordinal label.
|
|
---
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Issuer
|
|
metadata:
|
|
name: tidaldb-cluster-ca-issuer
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
# A self-signed CA root for the cluster's private inter-node PKI. Swap for a
|
|
# `ca:` issuer backed by your org PKI to chain to an existing root.
|
|
selfSigned: {}
|
|
---
|
|
apiVersion: cert-manager.io/v1
|
|
kind: Certificate
|
|
metadata:
|
|
name: tidaldb-cluster-tls
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
# cert-manager writes tls.crt / tls.key / ca.crt into this Secret; the
|
|
# StatefulSet mounts it read-only at /etc/tidaldb/tls.
|
|
secretName: tidaldb-cluster-tls
|
|
# Renew well before expiry; each renewal is hot-swapped without a restart.
|
|
duration: 2160h # 90d
|
|
renewBefore: 720h # 30d
|
|
isCA: false
|
|
usages:
|
|
- server auth # the gRPC + HTTP server identity
|
|
- client auth # the gRPC mTLS client identity (peer dials)
|
|
# SANs: a WILDCARD over every pod's stable headless-Service DNS (covers any
|
|
# ordinal, so scale-up/down needs no cert re-issue), the explicit initial-pod
|
|
# names (belt-and-suspenders for any strict verifier that distrusts a wildcard-
|
|
# only leaf), plus the headless and client Services. Keep
|
|
# `scripts/gen-cluster-certs.sh` in sync.
|
|
dnsNames:
|
|
- "*.tidaldb-peers.tidaldb-cluster.svc.cluster.local"
|
|
- tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
|
- tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
|
- tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
|
- tidaldb-peers.tidaldb-cluster.svc.cluster.local
|
|
- tidaldb.tidaldb-cluster.svc.cluster.local
|
|
issuerRef:
|
|
name: tidaldb-cluster-ca-issuer
|
|
kind: Issuer
|
|
group: cert-manager.io
|