# tidal-stress process — running the next checkpoint

How to take the cluster from "deployed" to "gated" without re-learning the
operational traps. Read [README.md](README.md) for the tool, [WORKLOG.md](WORKLOG.md)
for what's been run. This doc is the **repeatable flow**.

The legacy plan in `../tmp/stress-test-plan.md` is correct on *gates and intent*
but **stale on addressing** — it names the old 3-StatefulSet ClusterIPs
(`10.43.99.11-13`) and namespace `tidaldb`. Use the m11p5 DNS targets from the
README instead. Migrating that plan in place is step 0 below.

---

## The standard checkpoint loop

Every checkpoint, old or new, follows the same five beats:

1. **Pre-flight.** Context + cluster healthy before anything else.
   ```bash
   export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
   kubectl config current-context                       # must be the k3s one
   kubectl get pods -n tidaldb-cluster                  # all 3 → 1/1 Running, 0 restarts
   kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 &
   curl -sf localhost:19500/cluster/status | jq .       # leader set, every region lag=0
   ```
   Do **not** start a run with any pod restarting or mid-reseed (`0/1 Running`).
   A reseed cycle from a prior run takes ~120–150s to settle — wait it out.

2. **Arm the Job.** Copy the nearest `k8s/stress-job-*.yaml`, set the m11p5 DNS
   targets (README), pin `--ack quorum`, `--corpus 20000`, `--poll-status`.
   For a fresh PVC the first run seeds; add `--skip-seed` on subsequent runs in
   the same session.

3. **Run + chaos.** Apply the Job; if the checkpoint has a chaos arm (kill /
   scale), drive it from a script that talks to the **VIP over a port-forward**,
   never `kubectl exec` into a tidaldb pod (see traps below).

4. **Capture.** Paste the `CAPACITY VERDICT` block + `[pre/post-stage]` lag lines
   into `docs/ops/stress-test-<phase>.md`. Record the one number the next
   checkpoint must not regress.

5. **Record + re-arm.** Update `WORKLOG.md` (newest first) and the checkpoint
   table in `README.md`. `kubectl delete job <name> -n tidaldb-cluster` to re-arm.

---

## Operational traps (each cost a run — do not relearn)

| Trap | Symptom | Rule |
|------|---------|------|
| **Longhorn storage** | 74–85% quorum-write error rate | PVCs MUST be `local-path` (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput. |
| **`kubectl exec` for leader detection** | `container not found`, bogus 17–19s "elections" | Read leader via `curl <VIP>/cluster/status`, never exec into a tidaldb pod (it may be the one dying). |
| **`"none (election in progress)"`** | loop tries to delete a pod by that literal string | Filter leader name with `^tidaldb-[0-9]+$`; retry until a real name. |
| **`kubectl wait` across force-delete** | hangs past its own `--timeout` | Don't gate the loop on `kubectl wait`. Poll `kubectl get pod -o jsonpath` for `Ready=True`, or bound the wait and move on. |
| **Port-forward dies on VIP kill** | curl to `:19500` refuses mid-run | The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp. |

---

## Next: T4 — elasticity under load (unblocked on m11p5)

m11p5 added the membership + snapshot-install path T4 needs. **Gate:** scale
3→5→3 online under steady load; each joiner reaches `lag=0` within 5 min on a
100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss.

m11p5 makes scale-up a plain `kubectl scale` — pod ordinal ≥3 boots with
`--seed` (the headless Service resolves to a live peer), joins as a **learner**,
and **auto-promotes to a voter** once caught up (statefulset.yaml §ORD≥3 branch).

```bash
# 1. Seed a large corpus first (100k items so catch-up is non-trivial):
#    one run with --corpus 100000 --ramp smoke, let it seed, then stop.

# 2. Start steady-state load (leader path, quorum), let it reach a stable stage.
kubectl apply -f k8s/stress-job.yaml          # peach-100k or a pinned rps:secs

# 3. Scale up under load and watch the joiners catch up:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5
watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"'

# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min,
#    and the stress verdict shows p99 < 2× baseline for < 60s during the join.

# 5. Scale back down, then run a full ramp to confirm no residual impact:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3
```

Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k
corpus — that's the path under test. Confirm it promotes learner→voter (the
region appears in `/cluster/status` as reachable, lag=0) before declaring pass.

**T4 watch-item — keep load running through the whole join window.** A reseeded
node's K8s readiness only flips when it *observes* replication traffic
(WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node `0/1`
for 11.5h). So T4 must hold write load continuously from before the scale-up
until after both joiners are Ready — never let the cluster go idle mid-join, or
the readiness gate stalls on a node that is actually caught up. If you see a node
with `/cluster/status` lag=0 but pod `0/1`, that's this bug, not a catch-up
failure — and it's an **upstream fix**, not a manifest workaround.

---

## Then: T-read — the vector-search gate (NOT YET BUILT)

This is the gap that blocks any honest "competes with vector DBs" claim. Every
number to date is write-path. The tool can already drive reads (`--mix reads`,
ops `feed`/`search`) and report **latency/throughput/error**, but it has **no
recall oracle** — it cannot tell you whether the results are *correct*.

**What exists today** (latency/throughput only):
```bash
# Read-latency under load — runnable now, but only answers "how fast", not "how right":
#   --mix reads --ramp peach-100k --skip-seed --poll-status
kubectl apply -f k8s/stress-job.yaml   # with mix=reads
```

**What must be built** for a real gate (new work in `src/`):
1. **Ground-truth oracle.** For a sample of query vectors, compute exact
   top-k by brute-force cosine over the seeded corpus (offline or a `--verify`
   subcommand). Store as `(query_id → [item_id…])`.
2. **Recall@k metric.** During the read ramp, issue the same queries against the
   cluster, compare returned ids to the oracle, emit **recall@10** alongside the
   existing latency histogram.
3. **Query-QPS verdict.** Extend the capacity verdict with a read knee: QPS at
   which query p99 breaches SLO *while recall@10 stays ≥ target*.

**Proposed gate:** recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a
128-dim / 20k corpus — then re-run at **1536-dim** (thepeach's real width;
~2.3× RAM/item per capacity-planning.md) since that's the production shape.

Until T-read exists, scope every performance claim to the **write path** and say
so explicitly.

---

## Checkpoint order from here

```
T3 (done) ─→ T4 elasticity ─→ T-read recall  ─┐
                                              ├─→ T5 sharded throughput (needs p6)
            build recall oracle first  ───────┘
```

T4 is pure ops (no tool change) — do it next. T-read needs `src/` work (the
oracle + recall metric) — schedule that build before claiming any read number.
T5 stays blocked until p6 lands the shard groups.