jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

k8s(cluster): three voters is the desired state, on the read-path fix image

Browse Source 
Two outages in one session came from this file claiming `replicas: 0` while the
cluster served traffic: a plain `kubectl apply` scaled it to zero, twice. Source
now states production intent. Parking stays an explicit `kubectl scale`
divergence recorded in k3s-fleet/cluster-state.yaml, and
scripts/restore-fleet.sh remains the guarded path back - its storage, image, and
per-node capacity preflights are exactly what a bare apply does not do.

Pins server:m12-consumer-readpath-20260818@sha256:9191233d..., the build with the
transport-aware read budget and the scatter-degraded counters. Verified after the
rolling update: 3/3 Ready, all regions lag 0, every replica answering 3/3 shards
with degraded=false on the default budget.
This commit is contained in:
jordan 2026-08-17 20:52:32 -06:00
parent 3370394666
commit d923b036af
1 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
k8s/cluster/statefulset.yaml
View File

@ -21,7 +21,15 @@ metadata:
app.kubernetes.io/component: cluster-node app.kubernetes.io/component: cluster-node
spec: spec:
serviceName: tidaldb-peers # the headless peer Service — stable per-pod DNS serviceName: tidaldb-peers # the headless peer Service — stable per-pod DNS
replicas: 0 # parked; scripts/restore-fleet.sh restores three voters explicitly # PRODUCTION as of 2026-08-18: three voters is the desired state, not a value
# only the restore script may set. `replicas: 0` lived here while the cluster
# was parked and it cost two outages in one session: a plain `kubectl apply` of
# this file scaled a LIVE cluster to zero, twice, because the manifest still
# claimed parked. Source states intent; parking is an explicit `kubectl scale`
# divergence recorded in k3s-fleet/cluster-state.yaml, and
# scripts/restore-fleet.sh remains the guarded path back (its storage, image,
# and per-node capacity preflights are what a bare apply does not do).
replicas: 3
# Parallel: bring all pods up at once. There is no ordered-bootstrap # Parallel: bring all pods up at once. There is no ordered-bootstrap
# dependency — siblings boot in any order (an unreachable-at-startup peer is # dependency — siblings boot in any order (an unreachable-at-startup peer is
# normal; the election + catch-up timer converge them). Ordered start would # normal; the election + catch-up timer converge them). Ordered start would
@ -83,7 +91,7 @@ spec:
mountPath: /data mountPath: /data
containers: containers:
- name: tidaldb - name: tidaldb
image: registry.threesix.ai/tidal/server:m12-fleet-remediation-20260813@sha256:2e4baaf974ad2cf650609b1843689122bf4fd2751bcb8749906df8089223f4a3 image: registry.threesix.ai/tidal/server:m12-consumer-readpath-20260818@sha256:9191233d012e41fd92bb6d0efea83b60010995a6fc6e3ebc894f3d7c5f72872a
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
# The image ENTRYPOINT is the bare binary. We override the command with # The image ENTRYPOINT is the bare binary. We override the command with
# a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we # a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we