4db3f1e597
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.
- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
(scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
a cold TLS handshake under contention blew the sub-second budget. Discovery now
logs each poll failure at WARN with the full error source chain (a silent loop
made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.
Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
89 lines
4.1 KiB
YAML
89 lines
4.1 KiB
YAML
# m12p5 T4 (idle-readiness + scale 3→5→3 elasticity) overlay for local `kind`.
|
|
#
|
|
# Reuses the canonical k8s/cluster/ base but (1) collapses to a SINGLE
|
|
# replication group (topology-singlegroup-patch.yaml — T4 is replica elasticity,
|
|
# not sharding), (2) swaps the registry image for the locally-built m12p5 tag,
|
|
# (3) repoints the PVC at kind's `standard` StorageClass, and (4) gives each pod
|
|
# room for a 1536-dim corpus + a catch-up joiner.
|
|
#
|
|
# Run (the build tags MUST match the `images:` newTag below and the Job specs in
|
|
# tidal-stress/k8s/t4-*.yaml — kind serves only locally-loaded images, so a tag
|
|
# mismatch is an ImagePullBackOff):
|
|
# # server image — tag MUST equal the `images: newTag` below (m12p5-local).
|
|
# # Build from a tree with the m12p6 T4 TLS-scale-up fixes (two-tier-PKI
|
|
# # certs.yaml + join_boot grpc_tls fallback + https/ready-only seed); an
|
|
# # older binary crash-loops the joiner on UnknownIssuer / "could not join".
|
|
# docker build -f docker/deploy/Dockerfile -t tidaldb-server:m12p5-local .
|
|
# kind load docker-image tidaldb-server:m12p5-local --name canopy
|
|
# # stress generator — the m12p4 binary is reused unchanged for m12p5; tag MUST
|
|
# # equal the `image:` in tidal-stress/k8s/t4-{seed,load}-job.yaml (m12p4-local)
|
|
# docker build -f docker/stress/Dockerfile -t tidaldb-stress:m12p4-local .
|
|
# kind load docker-image tidaldb-stress:m12p4-local --name canopy
|
|
# kubectl create namespace tidaldb-cluster
|
|
# kubectl -n tidaldb-cluster create secret generic tidaldb-credentials \
|
|
# --from-literal=TIDAL_API_KEY="$(openssl rand -hex 32)" \
|
|
# --from-literal=TIDAL_CLUSTER_KEY="$(openssl rand -hex 32)"
|
|
# kubectl apply -k k8s/cluster-t4-kind/
|
|
# kubectl apply -f tidal-stress/k8s/t4-seed-job.yaml # seed the 1536-dim corpus
|
|
# kubectl apply -f tidal-stress/k8s/t4-load-job.yaml # steady load across the scale
|
|
# # scale 3→5 (idle-readiness fix + wildcard-SAN mTLS), then 5→3.
|
|
# kubectl -n tidaldb-cluster scale statefulset tidaldb --replicas=5
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: tidaldb-cluster
|
|
|
|
resources:
|
|
- ../cluster
|
|
|
|
images:
|
|
- name: registry.threesix.ai/tidal/server
|
|
newName: tidaldb-server
|
|
newTag: m12p5-local
|
|
|
|
patches:
|
|
# Single replication group (drop the base's 3-group `shards:` block).
|
|
- path: topology-singlegroup-patch.yaml
|
|
target:
|
|
kind: ConfigMap
|
|
name: tidaldb-cluster-topology
|
|
|
|
# kind ships `standard` (rancher.io/local-path), not the `local-path` the base
|
|
# names; repoint the PVC so claims bind. Bump the per-pod ceiling so a 1536-dim
|
|
# corpus + a joining replica's catch-up index fit (the 2Gi base is sized for a
|
|
# smoke corpus, not a real read-recall shape).
|
|
- target:
|
|
kind: StatefulSet
|
|
name: tidaldb
|
|
patch: |-
|
|
- op: replace
|
|
path: /spec/volumeClaimTemplates/0/spec/storageClassName
|
|
value: standard
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/limits/memory
|
|
value: 6Gi
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/limits/cpu
|
|
value: "3"
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/resources/requests/memory
|
|
value: 512Mi
|
|
# The 1536-dim HNSW index is rebuilt from durable storage on every open;
|
|
# under CPU contention on a single shared kind node that can exceed the
|
|
# base 5-min startup budget (a restarted/joining pod 503s its startup probe
|
|
# → SIGKILL → re-rebuild → cascade). Widen the budget to 10 min so a real
|
|
# catch-up + index rebuild finishes before the probe gives up.
|
|
- op: replace
|
|
path: /spec/template/spec/containers/0/startupProbe/failureThreshold
|
|
value: 120
|
|
# Local-run convenience: only roll pods with ordinal >= 3 (the joiners) on a
|
|
# template change, so iterating on the scale-up command never restarts the
|
|
# seeded 0/1/2 (and never triggers a simultaneous index-rebuild cascade on
|
|
# the shared kind node). NOT for the canonical base — production rolls all.
|
|
- op: add
|
|
path: /spec/updateStrategy
|
|
value:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
partition: 3
|