jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(soak): re-scope nightly soak 500->200 rps (measured sustainable) + fix image drift

Browse Source 
500 rps failed the capacity gate every night (peach mix is write-heavy; the
write path is structurally capped on 3-node RF3 full placement). A settled-
cluster capacity sweep measured the mix clean (0.00% error, 0 follower
restarts) at 100/150/200/250 rps; set the nightly target to 200 rps with margin
for a 1h x 30-night endurance run. SLO gates unchanged.

Also fix the image drift: the manifest pinned stress:m12-soak-eval (never
pushed -> ImagePull NotFound), while the live cronjob already ran the pullable
stress:m12-rc7-seedretry. The manifest now matches the live, pullable image.
This commit is contained in:
jx12n 2026-06-19 02:23:16 -06:00
parent 0919b0a4bf
commit 7cb9724911
1 changed files with 15 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
tidal-stress/k8s/soak-nightly-cronjob.yaml
View File

@ -22,12 +22,17 @@
# The streak passes when 30 CONSECUTIVE nightly Jobs have exited 0 with zero # The streak passes when 30 CONSECUTIVE nightly Jobs have exited 0 with zero
# under-load pod restarts (the monitor records restarts; see soak-monitor.yaml). # under-load pod restarts (the monitor records restarts; see soak-monitor.yaml).
# #
# SUSTAINED RATE: 500 rps mixed (peach: feed/search/view/like/skip/item/embed) — # SUSTAINED RATE: 200 rps mixed (peach: feed/search/view/like/skip/item/embed).
# deliberately BELOW the SLA knee. The rc12 read-SLA gate sustained 500 rps at # RE-SCOPED 2026-06-19 from 500 rps. The original 500 rps cited the rc12 READ-SLA
# p99 9.28ms / recall 0.9989 / 0% err / 0 under-load restarts, so 500 rps is a # gate, but the peach mix is WRITE-heavy (view/like/skip ≈ 90% of ops) and the
# safe endurance rate that exercises BOTH read and write paths without parking # write path is structurally capped on this 3-node RF3 full-placement cluster
# the cluster at its ceiling (a soak proves stability over time, not peak — the # (every follower applies every 1536-D write; T5). At 500 rps the soak FAILED
# 1-hour 100k-DAU peak run is the separate `--ramp 3900:3600` throughput gate). # every night (2026-06-17/18/19: p99 198-305ms, error 12-33%). A 2026-06-19
# capacity sweep on a settled cluster measured the peach mix CLEAN (0.00% error,
# p99 schedule-lag <5ms, 0 follower restarts) at 100/150/200/250 rps; the knee is
# between 250 and 500. 200 rps is the measured sustainable rate with margin for a
# 1-hour × 30-consecutive-night endurance run. Raise only with more write capacity
# (≥5 nodes / partitioned placement) — see orchard9-k3sf cluster-state.yaml.
# #
# WHY A CRONJOB, NOT ONE LONG JOB: the GA bar is literally "30 CONSECUTIVE DAYS" # WHY A CRONJOB, NOT ONE LONG JOB: the GA bar is literally "30 CONSECUTIVE DAYS"
# of an independent nightly verdict — a CronJob produces exactly that audit trail # of an independent nightly verdict — a CronJob produces exactly that audit trail
@ -82,7 +87,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: soak - name: soak
image: registry.threesix.ai/tidal/stress:m12-soak-eval image: registry.threesix.ai/tidal/stress:m12-rc7-seedretry
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
# entrypoint is tidal-stress; wrap it so we can stamp the result file # entrypoint is tidal-stress; wrap it so we can stamp the result file
# name with the date and append a ledger line regardless of verdict. # name with the date and append a ledger line regardless of verdict.
@ -92,7 +97,7 @@ spec:
set -u set -u
DATE="$(date -u +%Y-%m-%d)" DATE="$(date -u +%Y-%m-%d)"
OUT="/results/soak-$DATE.json" OUT="/results/soak-$DATE.json"
echo "soak $DATE start $(date -u +%H:%M:%SZ) target=cluster image=m12-soak-eval" echo "soak $DATE start $(date -u +%H:%M:%SZ) target=cluster image=m12-rc7-seedretry-200rps"
tidal-stress \ tidal-stress \
--target https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \ --target https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \
--target https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \ --target https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \
@ -103,7 +108,7 @@ spec:
--embedding-dim 1536 \ --embedding-dim 1536 \
--mix peach \ --mix peach \
--users 50000 \ --users 50000 \
--ramp "500:3600" \ --ramp "200:3600" \
--json-summary "$OUT" \ --json-summary "$OUT" \
--max-p99-ms 150 \ --max-p99-ms 150 \
--max-error-pct 1 \ --max-error-pct 1 \
@ -120,7 +125,7 @@ spec:
PASSED="$(grep -o '"passed"[: ]*[a-z]*' "$OUT" 2>/dev/null | head -1 | grep -o '[a-z]*$')" PASSED="$(grep -o '"passed"[: ]*[a-z]*' "$OUT" 2>/dev/null | head -1 | grep -o '[a-z]*$')"
P99="$(grep -o '"overall_p99_ms"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" P99="$(grep -o '"overall_p99_ms"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')"
ERRF="$(grep -o '"error_rate"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" ERRF="$(grep -o '"error_rate"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')"
printf '%s\t%s\trc=%s\tpassed=%s\tp99_ms=%s\terr_frac=%s\timage=m12-soak-eval\n' \ printf '%s\t%s\trc=%s\tpassed=%s\tp99_ms=%s\terr_frac=%s\timage=m12-rc7-seedretry-200rps\n' \
"$DATE" "$VERD" "$RC" "${PASSED:-?}" "${P99:-?}" "${ERRF:-?}" >> /results/ledger.tsv "$DATE" "$VERD" "$RC" "${PASSED:-?}" "${P99:-?}" "${ERRF:-?}" >> /results/ledger.tsv
echo "soak $DATE end $(date -u +%H:%M:%SZ) verdict=$VERD rc=$RC p99=${P99:-?} err=${ERRF:-?}" echo "soak $DATE end $(date -u +%H:%M:%SZ) verdict=$VERD rc=$RC p99=${P99:-?} err=${ERRF:-?}"
# Refresh the streak immediately, CONSULTING restarts.tsv: a night # Refresh the streak immediately, CONSULTING restarts.tsv: a night