jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(k8s): reconcile tidaldb StatefulSet resources with live cluster (300m/4Gi requests, 3/7Gi limits)

Browse Source
This commit is contained in:
jx12n 2026-09-03 05:15:35 -06:00
parent 2dc00538e8
commit 34cc5f08ca
1 changed files with 18 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
k8s/cluster/statefulset.yaml
View File

@ -309,8 +309,9 @@ spec:
# #
# 300m is what the tightest pinned node (k3s-agent-1, 355m free) # 300m is what the tightest pinned node (k3s-agent-1, 355m free)
# can actually reserve for a voter, and it is honest for a FIRST # can actually reserve for a voter, and it is honest for a FIRST
# CONSUMER's load - not for the gate. The limit below keeps the # CONSUMER's load - not for the gate. It also matches measured p95
# measured burst reachable without reserving it. # (302/235/233m per pod). The limit below keeps the measured burst
# reachable without reserving it.
# #
# This is provisioned optimism with named detectors: if real write # This is provisioned optimism with named detectors: if real write
# volume approaches the knee, TidalDBClusterQuorumLag, # volume approaches the knee, TidalDBClusterQuorumLag,
@ -321,7 +322,10 @@ spec:
cpu: "300m" cpu: "300m"
# Baseline working set was ~3.6 GiB before load, and full placement # Baseline working set was ~3.6 GiB before load, and full placement
# means every pod holds the WHOLE 1536-D corpus. This is a resident # means every pod holds the WHOLE 1536-D corpus. This is a resident
# footprint, not a gate artifact: it stays at 4 GiB. # footprint, not a gate artifact: it stays at 4 GiB. An 11d
# observation put pods at 4.4-5.7Gi, so the 1Gi this file once
# declared would make every replica a Burstable-over-request
# eviction candidate.
memory: 4Gi memory: 4Gi
limits: limits:
# 3 cores / 7Gi, matching what production actually runs. # 3 cores / 7Gi, matching what production actually runs.
@ -333,6 +337,14 @@ spec:
# are the intended ones and this file is now the source of truth for # are the intended ones and this file is now the source of truth for
# them; raise BOTH together or not at all. # them; raise BOTH together or not at all.
# #
# Why 3 and not 2 (m12 read-SLA fix): the cgroup cpu quota is what
# the engine's available_parallelism() reads (SEARCH_GATE /
# worker_threads sizing). At "2" a cross-shard search burst starved
# the async reactor plus the election/heartbeat/apply loops - reads
# hung to the 30s route timeout, and the starved control plane
# churned elections into reseed self-exit. "3" leaves ~1 core for
# kubelet/system on the 4-core nodes.
#
# Why a limit above the request at all: it keeps the measured # Why a limit above the request at all: it keeps the measured
# query/apply burst reachable on the tightest node without reserving # query/apply burst reachable on the tightest node without reserving
# it. Note the ratio - a burstable pod whose neighbours are also # it. Note the ratio - a burstable pod whose neighbours are also
@ -343,8 +355,9 @@ spec:
cpu: "3" cpu: "3"
# Four independent OOMKills occurred at 3.97-4.00 GiB. Six GiB was # Four independent OOMKills occurred at 3.97-4.00 GiB. Six GiB was
# measured peak plus 50% headroom; production carries 7Gi and that is # measured peak plus 50% headroom; production carries 7Gi and that is
# what is declared here. The exact internal growth source still # what is declared here - tidaldb-0 held 5751Mi, 94% of the old 6Gi
# requires heap/allocation profiling. # ceiling. The exact internal growth source still requires
# heap/allocation profiling.
memory: 7Gi memory: 7Gi
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false