tidaldb/tidal-stress/PROCESS.md

tidal-stress process — running the next checkpoint

How to take the cluster from "deployed" to "gated" without re-learning the operational traps. Read README.md for the tool, WORKLOG.md for what's been run. This doc is the repeatable flow.

The legacy plan in ../tmp/stress-test-plan.md is correct on gates and intent but stale on addressing — it names the old 3-StatefulSet ClusterIPs (10.43.99.11-13) and namespace tidaldb. Use the m11p5 DNS targets from the README instead. Migrating that plan in place is step 0 below.

---

The standard checkpoint loop

Every checkpoint, old or new, follows the same five beats:

1. Pre-flight. Context + cluster healthy before anything else.

   export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
   kubectl config current-context                       # must be the k3s one
   kubectl get pods -n tidaldb-cluster                  # all 3 → 1/1 Running, 0 restarts
   kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 &
   curl -sf localhost:19500/cluster/status | jq .       # leader set, every region lag=0
   

   Do not start a run with any pod restarting or mid-reseed (0/1 Running). A reseed cycle from a prior run takes ~120–150s to settle — wait it out.

2. Arm the Job. Copy the nearest k8s/stress-job-*.yaml, set the m11p5 DNS targets (README), pin --ack quorum, --corpus 20000, --poll-status. For a fresh PVC the first run seeds; add --skip-seed on subsequent runs in the same session.

3. Run + chaos. Apply the Job; if the checkpoint has a chaos arm (kill / scale), drive it from a script that talks to the VIP over a port-forward, never kubectl exec into a tidaldb pod (see traps below).

4. Capture. Paste the CAPACITY VERDICT block + [pre/post-stage] lag lines into docs/ops/stress-test-<phase>.md. Record the one number the next checkpoint must not regress.

5. Record + re-arm. Update WORKLOG.md (newest first) and the checkpoint table in README.md. kubectl delete job <name> -n tidaldb-cluster to re-arm.

---

Operational traps (each cost a run — do not relearn)

Trap	Symptom	Rule
Longhorn storage	74–85% quorum-write error rate	PVCs MUST be local-path (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput.
kubectl exec for leader detection	container not found, bogus 17–19s "elections"	Read leader via curl <VIP>/cluster/status, never exec into a tidaldb pod (it may be the one dying).
"none (election in progress)"	loop tries to delete a pod by that literal string	Filter leader name with ^tidaldb-[0-9]+$; retry until a real name.
kubectl wait across force-delete	hangs past its own --timeout	Don't gate the loop on kubectl wait. Poll kubectl get pod -o jsonpath for Ready=True, or bound the wait and move on.
Port-forward dies on VIP kill	curl to :19500 refuses mid-run	The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp.

---

Next: T4 — elasticity under load (unblocked on m11p5)

m11p5 added the membership + snapshot-install path T4 needs. Gate: scale 3→5→3 online under steady load; each joiner reaches lag=0 within 5 min on a 100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss.

m11p5 makes scale-up a plain kubectl scale — pod ordinal ≥3 boots with --seed (the headless Service resolves to a live peer), joins as a learner, and auto-promotes to a voter once caught up (statefulset.yaml §ORD≥3 branch).

# 1. Seed a large corpus first (100k items so catch-up is non-trivial):
#    one run with --corpus 100000 --ramp smoke, let it seed, then stop.

# 2. Start steady-state load (leader path, quorum), let it reach a stable stage.
kubectl apply -f k8s/stress-job.yaml          # peach-100k or a pinned rps:secs

# 3. Scale up under load and watch the joiners catch up:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5
watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"'

# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min,
#    and the stress verdict shows p99 < 2× baseline for < 60s during the join.

# 5. Scale back down, then run a full ramp to confirm no residual impact:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3

Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k corpus — that's the path under test. Confirm it promotes learner→voter (the region appears in /cluster/status as reachable, lag=0) before declaring pass.

T4 watch-item — keep load running through the whole join window. A reseeded node's K8s readiness only flips when it observes replication traffic (WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node 0/1 for 11.5h). So T4 must hold write load continuously from before the scale-up until after both joiners are Ready — never let the cluster go idle mid-join, or the readiness gate stalls on a node that is actually caught up. If you see a node with /cluster/status lag=0 but pod 0/1, that's this bug, not a catch-up failure — and it's an upstream fix, not a manifest workaround.

---

Then: T-read — the vector-search gate (NOT YET BUILT)

This is the gap that blocks any honest "competes with vector DBs" claim. Every number to date is write-path. The tool can already drive reads (--mix reads, ops feed/search) and report latency/throughput/error, but it has no recall oracle — it cannot tell you whether the results are correct.

What exists today (latency/throughput only):

# Read-latency under load — runnable now, but only answers "how fast", not "how right":
#   --mix reads --ramp peach-100k --skip-seed --poll-status
kubectl apply -f k8s/stress-job.yaml   # with mix=reads

What must be built for a real gate (new work in src/):

1. Ground-truth oracle. For a sample of query vectors, compute exact top-k by brute-force cosine over the seeded corpus (offline or a --verify subcommand). Store as (query_id → [item_id…]).
2. Recall@k metric. During the read ramp, issue the same queries against the cluster, compare returned ids to the oracle, emit recall@10 alongside the existing latency histogram.
3. Query-QPS verdict. Extend the capacity verdict with a read knee: QPS at which query p99 breaches SLO while recall@10 stays ≥ target.

Proposed gate: recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a 128-dim / 20k corpus — then re-run at 1536-dim (thepeach's real width; ~2.3× RAM/item per capacity-planning.md) since that's the production shape.

Until T-read exists, scope every performance claim to the write path and say so explicitly.

---

Checkpoint order from here

T3 (done) ─→ T4 elasticity ─→ T-read recall  ─┐
                                              ├─→ T5 sharded throughput (needs p6)
            build recall oracle first  ───────┘

T4 is pure ops (no tool change) — do it next. T-read needs src/ work (the oracle + recall metric) — schedule that build before claiming any read number. T5 stays blocked until p6 lands the shard groups.