tidaldb/docs/runbooks/nightly-soak.md

Soak: pre-release load and regression gate

Run this before tagging a release. It was a nightly-soak step in .woodpecker.yaml until 2026-08-30 and moved here unchanged — the commands below are the step's commands verbatim, so the coverage survives in a runnable form rather than only in git history.

Why this is not a CI step

The step never actually ran: the nightly cron it was gated on was never created, so for 216 days it produced zero signal while reading like standing coverage.

When the cron was finally configured, this step was deliberately excluded, and the reason is a measurement rather than a preference. The soak drives 1000 rps for 600s and fails the build if p99 > 250ms or errors > 1%. Measured free CPU on the k3s cluster, 2026-08-30:

node	allocatable CPU	free CPU (requests)	free memory
k3s-agent-1	4000m	1700m	4193Mi
k3s-server-1	3000m	435m	2309Mi
k3s-server-2	3000m	550m	2257Mi

A p99 gate of 250ms cannot be met from 1700m of contended CPU shared with production tidalDB. The step would fail nightly on starvation, not regression — a false alarm every morning, which is the fake-coverage defect inverted rather than fixed. A gate that cannot distinguish its own failure mode from the thing it is watching for is not a gate.

So it runs here, by hand, on hardware where its numbers mean something.

Where to run it

Anywhere with ≥ 2 dedicated cores and no co-tenant under load. A developer workstation qualifies; the shared k3s nodes do not. If you only have the k3s cluster, the honest options are to raise its thresholds to match the hardware (and say so in the output) or to skip it and record that you skipped it — not to run it and read the result as meaningful.

Run

export TIDAL_SOAK_RPS=1000
export TIDAL_SOAK_SECS=600
export TIDAL_SOAK_MAX_P99_MS=250
export TIDAL_SOAK_MAX_ERROR_PCT=1
# Soak target port — an UNCLAIMED slot in the project's reserved dev band
# (59520-59529): 59520=site, 59521=iknowyou, so the soak uses 59526.
export TIDAL_SOAK_PORT=59526

cargo build -p tidal-server -p tidal-stress

PORT="${TIDAL_SOAK_PORT}"
TARGET="${TIDAL_SOAK_TARGET:-http://127.0.0.1:$PORT}"
if [ -z "$TIDAL_SOAK_TARGET" ]; then
  mkdir -p /tmp/soak-data
  ./target/debug/tidal-server standalone --listen "127.0.0.1:$PORT" \
    --schema tidal-server/config/default-schema.yaml --data-dir /tmp/soak-data &
  SRV=$!
  # Reap the background server + scratch dir on ANY exit (success, gate failure,
  # or the boot-failure exit below) so a re-run is idempotent.
  trap 'kill "$SRV" 2>/dev/null; rm -rf /tmp/soak-data' EXIT
  up=0
  for i in $(seq 1 100); do
    curl -sf "$TARGET/health/startup" >/dev/null 2>&1 && { up=1; break; } || sleep 0.3
  done
  # Fail FAST and unambiguously on a boot failure (bad schema, port already
  # bound) rather than soaking a dead target and mislabeling it as an error-rate
  # regression on the trend line.
  if [ "$up" != "1" ]; then echo "soak target failed to start at $TARGET"; exit 1; fi
fi

./target/debug/tidal-stress --target "$TARGET" \
  --ramp "${TIDAL_SOAK_RPS}:${TIDAL_SOAK_SECS}" --corpus 5000 --mix peach \
  --json-summary soak-summary.json \
  --max-error-pct "${TIDAL_SOAK_MAX_ERROR_PCT}" --max-p99-ms "${TIDAL_SOAK_MAX_P99_MS}" --fail-on-knee

cat soak-summary.json

The $${VAR} escaping in the original CI step is not needed here. Woodpecker preprocesses a bare ${VAR} before the shell sees it, so the step had to write $${VAR} to pass a literal through. In a plain shell the single ${VAR} form above is correct.

What to expect

soak-summary.json is the artifact — keep it with the release notes so there is a trend line rather than a single opinion. A --fail-on-knee failure means throughput stopped scaling before the target rps, which is a different finding from a p99 breach and worth reporting separately.

Targeting the live cluster

Setting TIDAL_SOAK_TARGET to the live Ref-A cluster and TIDAL_SOAK_SECS=3600 gives the GA-bar 1-hour 100k-DAU soak. This soaks production at 1000 rps. Do not set it casually, and never as part of an unattended run.