From 46741a3a8c7b2819a62b4adbe4d8d761a49f292e Mon Sep 17 00:00:00 2001 From: jx12n Date: Sat, 13 Jun 2026 09:23:29 -0600 Subject: [PATCH] docs(tidal-stress): README + worklog + continue-the-process flow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md — tool reference pinned to the m11p5 cluster (tidaldb-cluster ns, pod-DNS targets, local-path storage); retires the stale 10.43.99.11-13 ClusterIPs. WORKLOG.md — m11p5 migration + T3 gate (10/10 kills, max 6157ms), the three run-costing traps (Longhorn storage, exec leader-detection, kubectl wait hang), reseed behavior, and the vector-DB positioning note. Records a m11p5 finding: a reseeded node stays NotReady forever on an idle cluster because the first-converged readiness check needs observed replication traffic — flagged as an upstream fix, not a manifest bandaid. PROCESS.md — the repeatable five-beat checkpoint loop, the operational-traps table, the T4 elasticity flow (now unblocked on m11p5, with the idle-readiness watch-item), and the T-read vector-search gate that still needs a recall oracle built before any read-path/vector-DB claim. --- tidal-stress/PROCESS.md | 145 ++++++++++++++++++++++++++++++++++++++++ tidal-stress/README.md | 106 +++++++++++++++++++++++++++++ tidal-stress/WORKLOG.md | 136 +++++++++++++++++++++++++++++++++++++ 3 files changed, 387 insertions(+) create mode 100644 tidal-stress/PROCESS.md create mode 100644 tidal-stress/README.md create mode 100644 tidal-stress/WORKLOG.md diff --git a/tidal-stress/PROCESS.md b/tidal-stress/PROCESS.md new file mode 100644 index 0000000..d4e7a79 --- /dev/null +++ b/tidal-stress/PROCESS.md @@ -0,0 +1,145 @@ +# tidal-stress process — running the next checkpoint + +How to take the cluster from "deployed" to "gated" without re-learning the +operational traps. Read [README.md](README.md) for the tool, [WORKLOG.md](WORKLOG.md) +for what's been run. This doc is the **repeatable flow**. + +The legacy plan in `../tmp/stress-test-plan.md` is correct on *gates and intent* +but **stale on addressing** — it names the old 3-StatefulSet ClusterIPs +(`10.43.99.11-13`) and namespace `tidaldb`. Use the m11p5 DNS targets from the +README instead. Migrating that plan in place is step 0 below. + +--- + +## The standard checkpoint loop + +Every checkpoint, old or new, follows the same five beats: + +1. **Pre-flight.** Context + cluster healthy before anything else. + ```bash + export KUBECONFIG=~/.kube/orchard9-k3sf.yaml + kubectl config current-context # must be the k3s one + kubectl get pods -n tidaldb-cluster # all 3 → 1/1 Running, 0 restarts + kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 & + curl -sf localhost:19500/cluster/status | jq . # leader set, every region lag=0 + ``` + Do **not** start a run with any pod restarting or mid-reseed (`0/1 Running`). + A reseed cycle from a prior run takes ~120–150s to settle — wait it out. + +2. **Arm the Job.** Copy the nearest `k8s/stress-job-*.yaml`, set the m11p5 DNS + targets (README), pin `--ack quorum`, `--corpus 20000`, `--poll-status`. + For a fresh PVC the first run seeds; add `--skip-seed` on subsequent runs in + the same session. + +3. **Run + chaos.** Apply the Job; if the checkpoint has a chaos arm (kill / + scale), drive it from a script that talks to the **VIP over a port-forward**, + never `kubectl exec` into a tidaldb pod (see traps below). + +4. **Capture.** Paste the `CAPACITY VERDICT` block + `[pre/post-stage]` lag lines + into `docs/ops/stress-test-.md`. Record the one number the next + checkpoint must not regress. + +5. **Record + re-arm.** Update `WORKLOG.md` (newest first) and the checkpoint + table in `README.md`. `kubectl delete job -n tidaldb-cluster` to re-arm. + +--- + +## Operational traps (each cost a run — do not relearn) + +| Trap | Symptom | Rule | +|------|---------|------| +| **Longhorn storage** | 74–85% quorum-write error rate | PVCs MUST be `local-path` (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput. | +| **`kubectl exec` for leader detection** | `container not found`, bogus 17–19s "elections" | Read leader via `curl /cluster/status`, never exec into a tidaldb pod (it may be the one dying). | +| **`"none (election in progress)"`** | loop tries to delete a pod by that literal string | Filter leader name with `^tidaldb-[0-9]+$`; retry until a real name. | +| **`kubectl wait` across force-delete** | hangs past its own `--timeout` | Don't gate the loop on `kubectl wait`. Poll `kubectl get pod -o jsonpath` for `Ready=True`, or bound the wait and move on. | +| **Port-forward dies on VIP kill** | curl to `:19500` refuses mid-run | The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp. | + +--- + +## Next: T4 — elasticity under load (unblocked on m11p5) + +m11p5 added the membership + snapshot-install path T4 needs. **Gate:** scale +3→5→3 online under steady load; each joiner reaches `lag=0` within 5 min on a +100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss. + +m11p5 makes scale-up a plain `kubectl scale` — pod ordinal ≥3 boots with +`--seed` (the headless Service resolves to a live peer), joins as a **learner**, +and **auto-promotes to a voter** once caught up (statefulset.yaml §ORD≥3 branch). + +```bash +# 1. Seed a large corpus first (100k items so catch-up is non-trivial): +# one run with --corpus 100000 --ramp smoke, let it seed, then stop. + +# 2. Start steady-state load (leader path, quorum), let it reach a stable stage. +kubectl apply -f k8s/stress-job.yaml # peach-100k or a pinned rps:secs + +# 3. Scale up under load and watch the joiners catch up: +kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5 +watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"' + +# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min, +# and the stress verdict shows p99 < 2× baseline for < 60s during the join. + +# 5. Scale back down, then run a full ramp to confirm no residual impact: +kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3 +``` + +Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k +corpus — that's the path under test. Confirm it promotes learner→voter (the +region appears in `/cluster/status` as reachable, lag=0) before declaring pass. + +**T4 watch-item — keep load running through the whole join window.** A reseeded +node's K8s readiness only flips when it *observes* replication traffic +(WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node `0/1` +for 11.5h). So T4 must hold write load continuously from before the scale-up +until after both joiners are Ready — never let the cluster go idle mid-join, or +the readiness gate stalls on a node that is actually caught up. If you see a node +with `/cluster/status` lag=0 but pod `0/1`, that's this bug, not a catch-up +failure — and it's an **upstream fix**, not a manifest workaround. + +--- + +## Then: T-read — the vector-search gate (NOT YET BUILT) + +This is the gap that blocks any honest "competes with vector DBs" claim. Every +number to date is write-path. The tool can already drive reads (`--mix reads`, +ops `feed`/`search`) and report **latency/throughput/error**, but it has **no +recall oracle** — it cannot tell you whether the results are *correct*. + +**What exists today** (latency/throughput only): +```bash +# Read-latency under load — runnable now, but only answers "how fast", not "how right": +# --mix reads --ramp peach-100k --skip-seed --poll-status +kubectl apply -f k8s/stress-job.yaml # with mix=reads +``` + +**What must be built** for a real gate (new work in `src/`): +1. **Ground-truth oracle.** For a sample of query vectors, compute exact + top-k by brute-force cosine over the seeded corpus (offline or a `--verify` + subcommand). Store as `(query_id → [item_id…])`. +2. **Recall@k metric.** During the read ramp, issue the same queries against the + cluster, compare returned ids to the oracle, emit **recall@10** alongside the + existing latency histogram. +3. **Query-QPS verdict.** Extend the capacity verdict with a read knee: QPS at + which query p99 breaches SLO *while recall@10 stays ≥ target*. + +**Proposed gate:** recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a +128-dim / 20k corpus — then re-run at **1536-dim** (thepeach's real width; +~2.3× RAM/item per capacity-planning.md) since that's the production shape. + +Until T-read exists, scope every performance claim to the **write path** and say +so explicitly. + +--- + +## Checkpoint order from here + +``` +T3 (done) ─→ T4 elasticity ─→ T-read recall ─┐ + ├─→ T5 sharded throughput (needs p6) + build recall oracle first ───────┘ +``` + +T4 is pure ops (no tool change) — do it next. T-read needs `src/` work (the +oracle + recall metric) — schedule that build before claiming any read number. +T5 stays blocked until p6 lands the shard groups. diff --git a/tidal-stress/README.md b/tidal-stress/README.md new file mode 100644 index 0000000..fc97f3b --- /dev/null +++ b/tidal-stress/README.md @@ -0,0 +1,106 @@ +# tidal-stress + +Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the +`thepeach` feed workload (signals + vector embeddings) against a live cluster and +reports a per-stage capacity verdict. + +- **Worklog** (what's been run, what we learned): [WORKLOG.md](WORKLOG.md) +- **Process** (how to run the next checkpoint): [PROCESS.md](PROCESS.md) + +--- + +## Current target cluster (as of 2026-06-13) + +The cluster moved to the **m11p5 single-StatefulSet architecture**. The old +3-StatefulSet / static-ClusterIP model (namespace `tidaldb`, IPs `10.43.99.11-13`) +is **retired** — any manifest or doc still naming those IPs is stale. + +| Fact | Value | +|------|-------| +| Namespace | `tidaldb-cluster` | +| Pods | `tidaldb-{0,1,2}` (one StatefulSet, 3 replicas) | +| Peer DNS | `tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500` (HTTP), `:9601` (gRPC) | +| Client VIP | `tidaldb.tidaldb-cluster.svc.cluster.local:9500` (readiness-gated) | +| Server image | `registry.threesix.ai/tidal/server@sha256:173e803…` (`:m11p5`) | +| Stress image | `registry.threesix.ai/tidal/stress@sha256:3a75c311…` (`:m11p3`) | +| Storage | local-path 5Gi/pod (on-node NVMe) — **NOT** Longhorn (see WORKLOG) | +| CPU/pod | limit `2` (the write pool is ~2 workers on the leader) | + +Targets for any new Job manifest — use pod DNS for `--target` (so status-polling +reaches survivors during a kill window) and the VIP for `--leader-url`: + +```yaml +args: + - --target + - http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --target + - http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --target + - http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --leader-url + - http://tidaldb.tidaldb-cluster.svc.cluster.local:9500 +``` + +--- + +## Run pattern + +Every run deploys the generator as an in-cluster Job (port-forward adds +API-server serialization latency — never use it for capacity numbers; only the +kill loop port-forwards, and only to read `/cluster/status`). + +```bash +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml +kubectl apply -f k8s/.yaml +kubectl logs -f job/ -n tidaldb-cluster +kubectl delete job -n tidaldb-cluster # re-arm before re-running +``` + +## CLI flags (authoritative — from `src/main.rs`) + +| Flag | Default | Notes | +|------|---------|-------| +| `--target ` | (required, repeatable) | Region gateway; reads round-robin across all | +| `--leader-url ` | none | Pin leader-path writes here to skip the forward hop | +| `--api-key` | `$TIDAL_API_KEY` | Bearer; the cluster requires it | +| `--ack ` | topology default | Sent as `x-tidal-ack` per write | +| `--ramp ` | `peach-100k` | Presets: `smoke`, `quick`, `peach-100k`, `max` | +| `--stage-secs ` | 45 | Hold per preset stage; 300–600 for soak | +| `--mix ` | `peach` | Presets: `peach`, `reads`, `writes`. Ops: `feed,search,view,like,skip,item,embed` | +| `--write-path ` | `leader` | `sharded` removes the single-leader funnel (not replicated) | +| `--corpus ` | 10000 | Items+embeddings to seed; 20k for gate runs | +| `--users ` | 50000 | Virtual user id space | +| `--skip-seed` | false | Set after the first run of a session (corpus persists on PVC) | +| `--embedding-dim ` | 128 | Deployed schema = 128; thepeach real = 1536 | +| `--hot-skew ` | 1.3 | Power-law concentration onto hot items | +| `--poll-status` | false | Poll `/cluster/status` between stages for lag — **always set when measuring lag** | +| `--stop-on-knee` | false | Stop at first SLO-breaching stage | +| `--dau ` | 100000 | DAU the verdict translates the ceiling against | + +SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA); +error rate ≥ 1% (429/408/503/5xx/transport) = the knee. + +## Layout + +``` +src/ generator (scheduler, workload model, client, metrics) +k8s/ Job manifests — one per checkpoint + stress-job.yaml generic ramp + stress-job-t2a.yaml T2-A quorum throughput + stress-job-t2b.yaml T2-B acked-loss under kills + stress-job-t3.yaml T3 automatic-failover gate +scripts/ + t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods) +``` + +## Checkpoint status + +| ID | Gate | Status | +|----|------|--------| +| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done | +| T2-A | ≥1000 quorum writes/s | ✓ 2980/s | +| T2-B | 0 acked loss across kills | ✓ | +| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) | +| **T4** | scale 3→5→3 under load, joiner ≤5min | **next** | +| T-read | vector-search recall@k + query QPS/p99 | **not built** (see PROCESS) | +| T5 | sharded ≥5000 quorum writes/s | blocked on p6 | diff --git a/tidal-stress/WORKLOG.md b/tidal-stress/WORKLOG.md new file mode 100644 index 0000000..237e48c --- /dev/null +++ b/tidal-stress/WORKLOG.md @@ -0,0 +1,136 @@ +# tidal-stress worklog + +Newest first. Chronicle of stress-test runs against the live k3s cluster, the +gates they cleared, and the operational lessons each one cost. + +--- + +## 2026-06-12/13 — m11p5 migration + T3 automatic-failover gate + +### What shipped + +Rolled the cluster from m11p3/p4 (3 StatefulSets, static ClusterIPs) to **m11p5** +(1 StatefulSet × 3 replicas, headless-Service DNS, elastic membership). New +namespace `tidaldb-cluster`. Then gated **T3** (automatic leader-failover). + +**Result: T3 PASS — 10/10 leader kills, max election 6157ms (gate <10 000ms), +zero acked loss, post-ramp lag=0.** + +| Kill | Victim → leader | ms | Kill | Victim → leader | ms | +|------|-----------------|----|----|-----------------|----| +| 1 | tidaldb-0 → tidaldb-2 | 2752 | 6 | tidaldb-2 → tidaldb-0 | 2656 | +| 2 | tidaldb-2 → tidaldb-0 | 4312 | 7 | tidaldb-0 → tidaldb-1 | 3517 | +| 3 | tidaldb-0 → tidaldb-2 | 2180 | 8 | tidaldb-1 → tidaldb-0 | 2758¹ | +| 4 | tidaldb-2 → tidaldb-0 | 3675 | 9 | tidaldb-0 → tidaldb-1 | 2235 | +| 5 | tidaldb-0 → tidaldb-2 | 2168 | 10 | tidaldb-1 → tidaldb-2 | 6157 | + +¹ Kill 8 measured from Raft logs (election won 03:55:42.758, kill 03:55:40) — +the port-forward died when the VIP routed to the killed pod, so the loop's own +timer recorded a false TIMEOUT. The election itself was clean. + +Stress envelope during the gate: 1500 rps quorum writes, 900s, 1.35M ops, +1.51% error (all 503s = quorum rejections inside the ~2–4s kill windows, not +acked losses), post-ramp lag=0 on all three nodes. + +### Three things that cost us a run each + +1. **Longhorn storage → 74–85% error rate.** The new cluster first provisioned + Longhorn 10Gi PVCs (networked storage). Every quorum write needs follower + fsync; over Longhorn that's a network hop on the commit path, which collapsed + throughput. Two full T3 runs were invalid before we spotted it. **Fix:** + `storageClassName: local-path`, 5Gi — matches the on-node NVMe that T2-A was + measured on. The first valid run came straight after the switch. + +2. **`kubectl exec` leader-detection died on the killed pod.** The v1/v2 kill + loops exec'd into a pod to read cluster status; when that pod was the one + being killed (or mid-initContainer on reboot), exec returned + `container not found` and the loop logged bogus 17–19s "election" times. The + status API also returns `"none (election in progress)"` as the leader field + mid-election, which the loop tried to `kubectl delete`. **Fix:** v3 uses a + port-forward to the VIP + `curl /cluster/status` with a strict + `^tidaldb-[0-9]+$` leader regex — never execs into a tidaldb pod. + +3. **`kubectl wait --for=condition=Ready` hangs across a force-delete+recreate.** + When a pod is force-deleted and the StatefulSet recreates it with the same + name, `kubectl wait` latches the old object and blocks past its own + `--timeout`. Had to `kill` the hung wait twice (kills 7 and 9) to let the loop + advance. Kills 9–10 were driven by hand with a fresh port-forward. + +### The reseed observation (expected, not a bug) + +At 1500 rps a ~50s pod outage opens a WAL gap large enough that the returning +pod can't catch up by streaming segments — it latches `reseed_required` +(m11p5 §2.4), self-exits, and installs a full snapshot on its next boot. +Snapshot install takes ~120–150s, during which the pod is `0/1 Running`. Quorum +holds throughout (2/3 voters always up), and **a reseeding pod still +participates in elections** — kill 4 elected tidaldb-0 in 3675ms while it was +mid-reseed. So the slow K8s readiness is orthogonal to election health; the gate +measures election time from the Raft log, not pod readiness. + +### FINDING — reseeded node stays NotReady forever on an idle cluster (m11p5 bug) + +Surfaced 11h after the gate, verifying cluster state. tidaldb-1 (kill 10's +victim) installed its snapshot cleanly at **03:59:42** (`snapshot_seq=1441537`) +and was a healthy follower from that moment — `/cluster/status` showed it +reachable, `lag_events=0`; `/health` returned `ok:true`. But K8s kept it +**`0/1 Running` for 11.5 hours**. The readiness flag did not flip to ready until +**15:21:25**, when it finally logged `joiner first-converged … readiness is now +sticky-ready`. + +Root cause: the install-boot readiness predicate (§4 hysteresis) flips ready only +when it *observes* `lag <= learner_promote_lag` through the catch-up +receiver/ship path. The stress job ended right after kill 10, so the cluster went +idle — **no new events were shipped, so the convergence-observation path was +never exercised**, and readiness never flipped. A fully-caught-up node sat out of +the client VIP for 11h on a technicality. + +Impact: in production this bites a **scale-up during a quiet period** — the new +node installs, catches up, but never joins the client Service until write traffic +resumes. It also means T4's joiner-Ready timing is only valid *under sustained +load*; an idle-window join would hang the readiness gate. + +This is an **upstream tidaldb fix**, not a stress-harness workaround: the +first-converged check should fire on a caught-up node even with zero inbound +replication (e.g. evaluate against a leader heartbeat / committed-index compare, +not only against observed ship traffic). Do not bandaid it in the manifest. +Filed as a T4 watch-item in PROCESS.md. + +### Strategic note (vector-DB positioning) + +Discussed where tidaldb sits vs other databases. CockroachDB is a different +animal (distributed SQL) — not the comp. The real comp is **vector DBs** +(Qdrant/Milvus/Weaviate/pgvector), and the honest read is: + +- tidaldb's differentiator is **synchronous quorum durability + <6s auto-failover + on the write path** — which almost no vector DB has (they treat vectors as a + rebuildable derived index and skip write durability). That's a real niche + *when the signals are source-of-truth* (thepeach view/like/skip are). +- We should **not** race them on ANN QPS/recall/ingest — a dedicated engine wins + that and should. +- **Gap:** every number we have is write-path only (`--mix writes`). We have + zero read-path data — no recall@k, no query QPS/p99. That's exactly the axis + vector DBs are judged on, and the tool can measure read latency/throughput but + **cannot measure recall** (no ground-truth oracle). Building that is the + T-read checkpoint in PROCESS.md. + +--- + +## 2026-06-11 — m11p4 regression + m11p3 T2 gate + +m11p3 (quorum acks, one replicated log) cleared **T2-A** (2980 quorum writes/s, +p99=49ms, knee ~5k rps) and **T2-B** (3 leader kills, lag=0 after each, zero +acked loss). m11p4 (catch-up timer retry + WAL TSEG segment versioning) held the +gate with no regression (2975 writes/s, −0.2%). Full results: +`docs/ops/stress-test-p3-t2.md`, `docs/ops/stress-test-p4-regression.md`. + +This was still on the **old 3-StatefulSet / static-ClusterIP** architecture — +the `10.43.99.11-13` IPs in `tmp/stress-test-plan.md` date from here and are now +stale (see PROCESS.md for the migration). + +--- + +## 2026-06-10 — T0 baseline + +Replicated `/signals` ~90/s (WAL fsync + synchronous gRPC ship per write, ~2 +workers on a cpu-2 pod); sharded `/sharded/signals` 3669/s at 0 errors, knee not +reached. Full results: `docs/ops/stress-test-thepeach.md`.