46741a3a8c
README.md — tool reference pinned to the m11p5 cluster (tidaldb-cluster ns, pod-DNS targets, local-path storage); retires the stale 10.43.99.11-13 ClusterIPs. WORKLOG.md — m11p5 migration + T3 gate (10/10 kills, max 6157ms), the three run-costing traps (Longhorn storage, exec leader-detection, kubectl wait hang), reseed behavior, and the vector-DB positioning note. Records a m11p5 finding: a reseeded node stays NotReady forever on an idle cluster because the first-converged readiness check needs observed replication traffic — flagged as an upstream fix, not a manifest bandaid. PROCESS.md — the repeatable five-beat checkpoint loop, the operational-traps table, the T4 elasticity flow (now unblocked on m11p5, with the idle-readiness watch-item), and the T-read vector-search gate that still needs a recall oracle built before any read-path/vector-DB claim.
146 lines
7.3 KiB
Markdown
146 lines
7.3 KiB
Markdown
# tidal-stress process — running the next checkpoint
|
||
|
||
How to take the cluster from "deployed" to "gated" without re-learning the
|
||
operational traps. Read [README.md](README.md) for the tool, [WORKLOG.md](WORKLOG.md)
|
||
for what's been run. This doc is the **repeatable flow**.
|
||
|
||
The legacy plan in `../tmp/stress-test-plan.md` is correct on *gates and intent*
|
||
but **stale on addressing** — it names the old 3-StatefulSet ClusterIPs
|
||
(`10.43.99.11-13`) and namespace `tidaldb`. Use the m11p5 DNS targets from the
|
||
README instead. Migrating that plan in place is step 0 below.
|
||
|
||
---
|
||
|
||
## The standard checkpoint loop
|
||
|
||
Every checkpoint, old or new, follows the same five beats:
|
||
|
||
1. **Pre-flight.** Context + cluster healthy before anything else.
|
||
```bash
|
||
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
|
||
kubectl config current-context # must be the k3s one
|
||
kubectl get pods -n tidaldb-cluster # all 3 → 1/1 Running, 0 restarts
|
||
kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 &
|
||
curl -sf localhost:19500/cluster/status | jq . # leader set, every region lag=0
|
||
```
|
||
Do **not** start a run with any pod restarting or mid-reseed (`0/1 Running`).
|
||
A reseed cycle from a prior run takes ~120–150s to settle — wait it out.
|
||
|
||
2. **Arm the Job.** Copy the nearest `k8s/stress-job-*.yaml`, set the m11p5 DNS
|
||
targets (README), pin `--ack quorum`, `--corpus 20000`, `--poll-status`.
|
||
For a fresh PVC the first run seeds; add `--skip-seed` on subsequent runs in
|
||
the same session.
|
||
|
||
3. **Run + chaos.** Apply the Job; if the checkpoint has a chaos arm (kill /
|
||
scale), drive it from a script that talks to the **VIP over a port-forward**,
|
||
never `kubectl exec` into a tidaldb pod (see traps below).
|
||
|
||
4. **Capture.** Paste the `CAPACITY VERDICT` block + `[pre/post-stage]` lag lines
|
||
into `docs/ops/stress-test-<phase>.md`. Record the one number the next
|
||
checkpoint must not regress.
|
||
|
||
5. **Record + re-arm.** Update `WORKLOG.md` (newest first) and the checkpoint
|
||
table in `README.md`. `kubectl delete job <name> -n tidaldb-cluster` to re-arm.
|
||
|
||
---
|
||
|
||
## Operational traps (each cost a run — do not relearn)
|
||
|
||
| Trap | Symptom | Rule |
|
||
|------|---------|------|
|
||
| **Longhorn storage** | 74–85% quorum-write error rate | PVCs MUST be `local-path` (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput. |
|
||
| **`kubectl exec` for leader detection** | `container not found`, bogus 17–19s "elections" | Read leader via `curl <VIP>/cluster/status`, never exec into a tidaldb pod (it may be the one dying). |
|
||
| **`"none (election in progress)"`** | loop tries to delete a pod by that literal string | Filter leader name with `^tidaldb-[0-9]+$`; retry until a real name. |
|
||
| **`kubectl wait` across force-delete** | hangs past its own `--timeout` | Don't gate the loop on `kubectl wait`. Poll `kubectl get pod -o jsonpath` for `Ready=True`, or bound the wait and move on. |
|
||
| **Port-forward dies on VIP kill** | curl to `:19500` refuses mid-run | The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp. |
|
||
|
||
---
|
||
|
||
## Next: T4 — elasticity under load (unblocked on m11p5)
|
||
|
||
m11p5 added the membership + snapshot-install path T4 needs. **Gate:** scale
|
||
3→5→3 online under steady load; each joiner reaches `lag=0` within 5 min on a
|
||
100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss.
|
||
|
||
m11p5 makes scale-up a plain `kubectl scale` — pod ordinal ≥3 boots with
|
||
`--seed` (the headless Service resolves to a live peer), joins as a **learner**,
|
||
and **auto-promotes to a voter** once caught up (statefulset.yaml §ORD≥3 branch).
|
||
|
||
```bash
|
||
# 1. Seed a large corpus first (100k items so catch-up is non-trivial):
|
||
# one run with --corpus 100000 --ramp smoke, let it seed, then stop.
|
||
|
||
# 2. Start steady-state load (leader path, quorum), let it reach a stable stage.
|
||
kubectl apply -f k8s/stress-job.yaml # peach-100k or a pinned rps:secs
|
||
|
||
# 3. Scale up under load and watch the joiners catch up:
|
||
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5
|
||
watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"'
|
||
|
||
# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min,
|
||
# and the stress verdict shows p99 < 2× baseline for < 60s during the join.
|
||
|
||
# 5. Scale back down, then run a full ramp to confirm no residual impact:
|
||
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3
|
||
```
|
||
|
||
Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k
|
||
corpus — that's the path under test. Confirm it promotes learner→voter (the
|
||
region appears in `/cluster/status` as reachable, lag=0) before declaring pass.
|
||
|
||
**T4 watch-item — keep load running through the whole join window.** A reseeded
|
||
node's K8s readiness only flips when it *observes* replication traffic
|
||
(WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node `0/1`
|
||
for 11.5h). So T4 must hold write load continuously from before the scale-up
|
||
until after both joiners are Ready — never let the cluster go idle mid-join, or
|
||
the readiness gate stalls on a node that is actually caught up. If you see a node
|
||
with `/cluster/status` lag=0 but pod `0/1`, that's this bug, not a catch-up
|
||
failure — and it's an **upstream fix**, not a manifest workaround.
|
||
|
||
---
|
||
|
||
## Then: T-read — the vector-search gate (NOT YET BUILT)
|
||
|
||
This is the gap that blocks any honest "competes with vector DBs" claim. Every
|
||
number to date is write-path. The tool can already drive reads (`--mix reads`,
|
||
ops `feed`/`search`) and report **latency/throughput/error**, but it has **no
|
||
recall oracle** — it cannot tell you whether the results are *correct*.
|
||
|
||
**What exists today** (latency/throughput only):
|
||
```bash
|
||
# Read-latency under load — runnable now, but only answers "how fast", not "how right":
|
||
# --mix reads --ramp peach-100k --skip-seed --poll-status
|
||
kubectl apply -f k8s/stress-job.yaml # with mix=reads
|
||
```
|
||
|
||
**What must be built** for a real gate (new work in `src/`):
|
||
1. **Ground-truth oracle.** For a sample of query vectors, compute exact
|
||
top-k by brute-force cosine over the seeded corpus (offline or a `--verify`
|
||
subcommand). Store as `(query_id → [item_id…])`.
|
||
2. **Recall@k metric.** During the read ramp, issue the same queries against the
|
||
cluster, compare returned ids to the oracle, emit **recall@10** alongside the
|
||
existing latency histogram.
|
||
3. **Query-QPS verdict.** Extend the capacity verdict with a read knee: QPS at
|
||
which query p99 breaches SLO *while recall@10 stays ≥ target*.
|
||
|
||
**Proposed gate:** recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a
|
||
128-dim / 20k corpus — then re-run at **1536-dim** (thepeach's real width;
|
||
~2.3× RAM/item per capacity-planning.md) since that's the production shape.
|
||
|
||
Until T-read exists, scope every performance claim to the **write path** and say
|
||
so explicitly.
|
||
|
||
---
|
||
|
||
## Checkpoint order from here
|
||
|
||
```
|
||
T3 (done) ─→ T4 elasticity ─→ T-read recall ─┐
|
||
├─→ T5 sharded throughput (needs p6)
|
||
build recall oracle first ───────┘
|
||
```
|
||
|
||
T4 is pure ops (no tool change) — do it next. T-read needs `src/` work (the
|
||
oracle + recall metric) — schedule that build before claiming any read number.
|
||
T5 stays blocked until p6 lands the shard groups.
|