jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

k8s(cluster): declare the limits production actually runs

Browse Source 
The manifest said cpu 2 / memory 6Gi while the live StatefulSet ran cpu 3 /
memory 7Gi, so `kubectl apply -k k8s/cluster/` DOWNGRADED production every time
it was run - a silent capacity regression on the write path, delivered by the
very command used to deploy a fix. Verified before and after with kubectl diff:
the cpu/memory hunks are now absent, so an apply is a no-op on resources.

The live values are the intended ones; this file is now the source of truth for
them. The comment keeps the reasoning that matters - the limit exists to keep the
measured query/apply burst reachable without reserving it, and when neighbours
burst too the answer is more REQUEST, not more limit, because CFS throttling is
how CockroachDB was pushed into multi-second Raft stalls on this fleet with nodes
70% idle.
This commit is contained in:
jordan 2026-08-22 19:49:37 -06:00
parent 388e445a38
commit c9adec040d
1 changed files with 22 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
k8s/cluster/statefulset.yaml
View File

@ -287,17 +287,28 @@ spec:
# footprint, not a gate artifact: it stays at 4 GiB. # footprint, not a gate artifact: it stays at 4 GiB.
memory: 4Gi memory: 4Gi
limits: limits:
# Two cores keeps the measured query/apply burst reachable on the # 3 cores / 7Gi, matching what production actually runs.
# tightest node without reserving it. Note the ratio: a burstable #
# pod whose neighbours are also bursting gets CFS-throttled, which # These were 2 cores / 6Gi here while the live StatefulSet ran 3/7,
# is precisely how CockroachDB was pushed into multi-second Raft # so `kubectl apply -k k8s/cluster/` DOWNGRADED production every
# stalls on this fleet with nodes 70% idle. The quorum alerts above # time it was run - a silent capacity regression on the write path,
# are the detector for that; more request is the fix. # applied by the very command used to deploy a fix. The live values
cpu: "2" # are the intended ones and this file is now the source of truth for
# Four independent OOMKills occurred at 3.97-4.00 GiB. Six GiB is # them; raise BOTH together or not at all.
# measured peak plus 50% recovery/profiling headroom; the exact #
# internal growth source still requires heap/allocation profiling. # Why a limit above the request at all: it keeps the measured
memory: 6Gi # query/apply burst reachable on the tightest node without reserving
# it. Note the ratio - a burstable pod whose neighbours are also
# bursting gets CFS-throttled, which is precisely how CockroachDB was
# pushed into multi-second Raft stalls on this fleet with nodes 70%
# idle. The quorum alerts above are the detector for that; more
# REQUEST is the fix, not more limit.
cpu: "3"
# Four independent OOMKills occurred at 3.97-4.00 GiB. Six GiB was
# measured peak plus 50% headroom; production carries 7Gi and that is
# what is declared here. The exact internal growth source still
# requires heap/allocation profiling.
memory: 7Gi
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
readOnlyRootFilesystem: true # writes only /data (PVC) and /tmp (emptyDir) readOnlyRootFilesystem: true # writes only /data (PVC) and /tmp (emptyDir)