4db3f1e597
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.
- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
(scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
a cold TLS handshake under contention blew the sub-second budget. Discovery now
logs each poll failure at WARN with the full error source chain (a silent loop
made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.
Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
112 lines
3.7 KiB
YAML
112 lines
3.7 KiB
YAML
# T4 (m12p5) — sustained load during the 3→5→3 scale (Phase B).
|
|
#
|
|
# A single generator holds a steady ack=quorum read+write mix for ~5 min, long
|
|
# enough to span a scale-down (5→3) and scale-up (3→5) under load. `--skip-seed`
|
|
# (the corpus is already registered by t4-seed-job) keeps the id space; the mix
|
|
# carries item/embed WRITES so the transition is exercised by live quorum writes,
|
|
# not just reads. `--max-error-pct 0` is the zero-acked-loss gate: a quorum write
|
|
# that cannot reach its majority during a membership change fails to ack → error
|
|
# → the Job exits non-zero. `--json-summary` captures the per-stage p99 envelope
|
|
# (before / during / after the scale) for the evidence table.
|
|
#
|
|
# NO --poll-status: this phase is UNDER LOAD (the idle-readiness isolation is the
|
|
# separate idle Phase A driven from the host), so status polling is unneeded and
|
|
# omitted to keep the offered load pure.
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/t4-load-job.yaml
|
|
# Watch: kubectl logs -f job/tidal-stress-t4-load -n tidaldb-cluster
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-stress-t4-load
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: tidaldb-stress:m12p4-local
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --ack
|
|
- quorum
|
|
- --skip-seed
|
|
- --corpus
|
|
- "300"
|
|
- --embedding-dim
|
|
- "1536"
|
|
- --users
|
|
- "100000"
|
|
# ~5 min steady hold: three 100s stages so the per-stage JSON gives a
|
|
# before / during / after p99 read across the scale window.
|
|
- --ramp
|
|
- "400:100,400:100,400:100"
|
|
- --mix
|
|
- "feed=4,search=1,view=2,like=1,item=1,embed=1"
|
|
- --max-inflight
|
|
- "2000"
|
|
- --json-summary
|
|
- /tmp/t4-load-summary.json
|
|
- --max-error-pct
|
|
- "0"
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests:
|
|
cpu: 500m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "3"
|
|
memory: 1Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: cluster-tls
|
|
mountPath: /etc/tidaldb/tls
|
|
readOnly: true
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|
|
- name: tmp
|
|
emptyDir: {}
|