# Soak: pre-release load and regression gate

Run this before tagging a release. It was a `nightly-soak` step in
`.woodpecker.yaml` until 2026-08-30 and moved here **unchanged** — the commands
below are the step's commands verbatim, so the coverage survives in a runnable
form rather than only in git history.

## Why this is not a CI step

The step never actually ran: the `nightly` cron it was gated on was never
created, so for 216 days it produced zero signal while reading like standing
coverage.

When the cron was finally configured, this step was **deliberately excluded**,
and the reason is a measurement rather than a preference. The soak drives
**1000 rps for 600s** and fails the build if **p99 > 250ms** or **errors > 1%**.
Measured free CPU on the k3s cluster, 2026-08-30:

| node | allocatable CPU | free CPU (requests) | free memory |
| --- | --- | --- | --- |
| `k3s-agent-1` | 4000m | 1700m | 4193Mi |
| `k3s-server-1` | 3000m | 435m | 2309Mi |
| `k3s-server-2` | 3000m | 550m | 2257Mi |

A p99 gate of 250ms cannot be met from 1700m of contended CPU shared with
production tidalDB. The step would fail nightly on **starvation, not
regression** — a false alarm every morning, which is the fake-coverage defect
inverted rather than fixed. A gate that cannot distinguish its own failure mode
from the thing it is watching for is not a gate.

So it runs **here**, by hand, on hardware where its numbers mean something.

## Where to run it

Anywhere with **≥ 2 dedicated cores** and no co-tenant under load. A developer
workstation qualifies; the shared k3s nodes do not. If you only have the k3s
cluster, the honest options are to raise its thresholds to match the hardware
(and say so in the output) or to skip it and record that you skipped it — not to
run it and read the result as meaningful.

## Run

```bash
export TIDAL_SOAK_RPS=1000
export TIDAL_SOAK_SECS=600
export TIDAL_SOAK_MAX_P99_MS=250
export TIDAL_SOAK_MAX_ERROR_PCT=1
# Soak target port — an UNCLAIMED slot in the project's reserved dev band
# (59520-59529): 59520=site, 59521=iknowyou, so the soak uses 59526.
export TIDAL_SOAK_PORT=59526

cargo build -p tidal-server -p tidal-stress

PORT="${TIDAL_SOAK_PORT}"
TARGET="${TIDAL_SOAK_TARGET:-http://127.0.0.1:$PORT}"
if [ -z "$TIDAL_SOAK_TARGET" ]; then
  mkdir -p /tmp/soak-data
  ./target/debug/tidal-server standalone --listen "127.0.0.1:$PORT" \
    --schema tidal-server/config/default-schema.yaml --data-dir /tmp/soak-data &
  SRV=$!
  # Reap the background server + scratch dir on ANY exit (success, gate failure,
  # or the boot-failure exit below) so a re-run is idempotent.
  trap 'kill "$SRV" 2>/dev/null; rm -rf /tmp/soak-data' EXIT
  up=0
  for i in $(seq 1 100); do
    curl -sf "$TARGET/health/startup" >/dev/null 2>&1 && { up=1; break; } || sleep 0.3
  done
  # Fail FAST and unambiguously on a boot failure (bad schema, port already
  # bound) rather than soaking a dead target and mislabeling it as an error-rate
  # regression on the trend line.
  if [ "$up" != "1" ]; then echo "soak target failed to start at $TARGET"; exit 1; fi
fi

./target/debug/tidal-stress --target "$TARGET" \
  --ramp "${TIDAL_SOAK_RPS}:${TIDAL_SOAK_SECS}" --corpus 5000 --mix peach \
  --json-summary soak-summary.json \
  --max-error-pct "${TIDAL_SOAK_MAX_ERROR_PCT}" --max-p99-ms "${TIDAL_SOAK_MAX_P99_MS}" --fail-on-knee

cat soak-summary.json
```

> The `$${VAR}` escaping in the original CI step is **not** needed here.
> Woodpecker preprocesses a bare `${VAR}` before the shell sees it, so the step
> had to write `$${VAR}` to pass a literal through. In a plain shell the single
> `${VAR}` form above is correct.

## What to expect

`soak-summary.json` is the artifact — keep it with the release notes so there is
a trend line rather than a single opinion. A `--fail-on-knee` failure means
throughput stopped scaling before the target rps, which is a different finding
from a p99 breach and worth reporting separately.

## Targeting the live cluster

Setting `TIDAL_SOAK_TARGET` to the live Ref-A cluster and `TIDAL_SOAK_SECS=3600`
gives the GA-bar 1-hour 100k-DAU soak. **This soaks production at 1000 rps.**
Do not set it casually, and never as part of an unattended run.