jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(tidal-stress): README + worklog + continue-the-process flow

Browse Source 
README.md — tool reference pinned to the m11p5 cluster (tidaldb-cluster ns,
pod-DNS targets, local-path storage); retires the stale 10.43.99.11-13 ClusterIPs.

WORKLOG.md — m11p5 migration + T3 gate (10/10 kills, max 6157ms), the three
run-costing traps (Longhorn storage, exec leader-detection, kubectl wait hang),
reseed behavior, and the vector-DB positioning note. Records a m11p5 finding:
a reseeded node stays NotReady forever on an idle cluster because the
first-converged readiness check needs observed replication traffic — flagged as
an upstream fix, not a manifest bandaid.

PROCESS.md — the repeatable five-beat checkpoint loop, the operational-traps
table, the T4 elasticity flow (now unblocked on m11p5, with the idle-readiness
watch-item), and the T-read vector-search gate that still needs a recall oracle
built before any read-path/vector-DB claim.
This commit is contained in:
jx12n 2026-06-13 09:23:29 -06:00
parent d5d1e7d81a
commit 46741a3a8c
3 changed files with 387 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

145
tidal-stress/PROCESS.md Normal file
View File

@ -0,0 +1,145 @@
# tidal-stress process — running the next checkpoint
How to take the cluster from "deployed" to "gated" without re-learning the
operational traps. Read [README.md](README.md) for the tool, [WORKLOG.md](WORKLOG.md)
for what's been run. This doc is the **repeatable flow**.
The legacy plan in `../tmp/stress-test-plan.md` is correct on *gates and intent*
but **stale on addressing** — it names the old 3-StatefulSet ClusterIPs
(`10.43.99.11-13`) and namespace `tidaldb`. Use the m11p5 DNS targets from the
README instead. Migrating that plan in place is step 0 below.
---
## The standard checkpoint loop
Every checkpoint, old or new, follows the same five beats:
1. **Pre-flight.** Context + cluster healthy before anything else.
```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl config current-context # must be the k3s one
kubectl get pods -n tidaldb-cluster # all 3 → 1/1 Running, 0 restarts
kubectl port-forward -n tidaldb-cluster svc/tidaldb 19500:9500 &
curl -sf localhost:19500/cluster/status | jq . # leader set, every region lag=0
```
Do **not** start a run with any pod restarting or mid-reseed (`0/1 Running`).
A reseed cycle from a prior run takes ~120–150s to settle — wait it out.
2. **Arm the Job.** Copy the nearest `k8s/stress-job-*.yaml`, set the m11p5 DNS
targets (README), pin `--ack quorum`, `--corpus 20000`, `--poll-status`.
For a fresh PVC the first run seeds; add `--skip-seed` on subsequent runs in
the same session.
3. **Run + chaos.** Apply the Job; if the checkpoint has a chaos arm (kill /
scale), drive it from a script that talks to the **VIP over a port-forward**,
never `kubectl exec` into a tidaldb pod (see traps below).
4. **Capture.** Paste the `CAPACITY VERDICT` block + `[pre/post-stage]` lag lines
into `docs/ops/stress-test-<phase>.md`. Record the one number the next
checkpoint must not regress.
5. **Record + re-arm.** Update `WORKLOG.md` (newest first) and the checkpoint
table in `README.md`. `kubectl delete job <name> -n tidaldb-cluster` to re-arm.
---
## Operational traps (each cost a run — do not relearn)
| Trap | Symptom | Rule |
|------|---------|------|
| **Longhorn storage** | 74–85% quorum-write error rate | PVCs MUST be `local-path` (5Gi). Networked storage on the fsync commit path is fatal to quorum throughput. |
| **`kubectl exec` for leader detection** | `container not found`, bogus 17–19s "elections" | Read leader via `curl <VIP>/cluster/status`, never exec into a tidaldb pod (it may be the one dying). |
| **`"none (election in progress)"`** | loop tries to delete a pod by that literal string | Filter leader name with `^tidaldb-[0-9]+$`; retry until a real name. |
| **`kubectl wait` across force-delete** | hangs past its own `--timeout` | Don't gate the loop on `kubectl wait`. Poll `kubectl get pod -o jsonpath` for `Ready=True`, or bound the wait and move on. |
| **Port-forward dies on VIP kill** | curl to `:19500` refuses mid-run | The VIP routed to the killed pod. Re-establish the port-forward before resuming; for failover timing, fall back to the Raft log timestamp. |
---
## Next: T4 — elasticity under load (unblocked on m11p5)
m11p5 added the membership + snapshot-install path T4 needs. **Gate:** scale
3→5→3 online under steady load; each joiner reaches `lag=0` within 5 min on a
100k-item corpus; p99 impact < 2× baseline for < 60s; zero acked loss.
m11p5 makes scale-up a plain `kubectl scale` — pod ordinal ≥3 boots with
`--seed` (the headless Service resolves to a live peer), joins as a **learner**,
and **auto-promotes to a voter** once caught up (statefulset.yaml §ORD≥3 branch).
```bash
# 1. Seed a large corpus first (100k items so catch-up is non-trivial):
# one run with --corpus 100000 --ramp smoke, let it seed, then stop.
# 2. Start steady-state load (leader path, quorum), let it reach a stable stage.
kubectl apply -f k8s/stress-job.yaml # peach-100k or a pinned rps:secs
# 3. Scale up under load and watch the joiners catch up:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=5
watch 'curl -s localhost:19500/cluster/status | jq ".regions[] | {name, lag_events, partitioned}"'
# 4. Pass when tidaldb-3 and tidaldb-4 show lag_events=0 within 5 min,
# and the stress verdict shows p99 < 2× baseline for < 60s during the join.
# 5. Scale back down, then run a full ramp to confirm no residual impact:
kubectl scale statefulset tidaldb -n tidaldb-cluster --replicas=3
```
Watch for: a joiner on a fresh PVC will snapshot-install (not stream) for a 100k
corpus — that's the path under test. Confirm it promotes learner→voter (the
region appears in `/cluster/status` as reachable, lag=0) before declaring pass.
**T4 watch-item — keep load running through the whole join window.** A reseeded
node's K8s readiness only flips when it *observes* replication traffic
(WORKLOG 2026-06-13 finding: an idle cluster left a fully-caught-up node `0/1`
for 11.5h). So T4 must hold write load continuously from before the scale-up
until after both joiners are Ready — never let the cluster go idle mid-join, or
the readiness gate stalls on a node that is actually caught up. If you see a node
with `/cluster/status` lag=0 but pod `0/1`, that's this bug, not a catch-up
failure — and it's an **upstream fix**, not a manifest workaround.
---
## Then: T-read — the vector-search gate (NOT YET BUILT)
This is the gap that blocks any honest "competes with vector DBs" claim. Every
number to date is write-path. The tool can already drive reads (`--mix reads`,
ops `feed`/`search`) and report **latency/throughput/error**, but it has **no
recall oracle** — it cannot tell you whether the results are *correct*.
**What exists today** (latency/throughput only):
```bash
# Read-latency under load — runnable now, but only answers "how fast", not "how right":
# --mix reads --ramp peach-100k --skip-seed --poll-status
kubectl apply -f k8s/stress-job.yaml # with mix=reads
```
**What must be built** for a real gate (new work in `src/`):
1. **Ground-truth oracle.** For a sample of query vectors, compute exact
top-k by brute-force cosine over the seeded corpus (offline or a `--verify`
subcommand). Store as `(query_id → [item_id…])`.
2. **Recall@k metric.** During the read ramp, issue the same queries against the
cluster, compare returned ids to the oracle, emit **recall@10** alongside the
existing latency histogram.
3. **Query-QPS verdict.** Extend the capacity verdict with a read knee: QPS at
which query p99 breaches SLO *while recall@10 stays ≥ target*.
**Proposed gate:** recall@10 ≥ 0.95 at ≥ X query QPS with query p99 ≤ 50ms, on a
128-dim / 20k corpus — then re-run at **1536-dim** (thepeach's real width;
~2.3× RAM/item per capacity-planning.md) since that's the production shape.
Until T-read exists, scope every performance claim to the **write path** and say
so explicitly.
---
## Checkpoint order from here
```
T3 (done) ─→ T4 elasticity ─→ T-read recall ─┐
├─→ T5 sharded throughput (needs p6)
build recall oracle first ───────┘
```
T4 is pure ops (no tool change) — do it next. T-read needs `src/` work (the
oracle + recall metric) — schedule that build before claiming any read number.
T5 stays blocked until p6 lands the shard groups.

106
tidal-stress/README.md Normal file
View File

@ -0,0 +1,106 @@
# tidal-stress
Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the
`thepeach` feed workload (signals + vector embeddings) against a live cluster and
reports a per-stage capacity verdict.
- **Worklog** (what's been run, what we learned): [WORKLOG.md](WORKLOG.md)
- **Process** (how to run the next checkpoint): [PROCESS.md](PROCESS.md)
---
## Current target cluster (as of 2026-06-13)
The cluster moved to the **m11p5 single-StatefulSet architecture**. The old
3-StatefulSet / static-ClusterIP model (namespace `tidaldb`, IPs `10.43.99.11-13`)
is **retired** — any manifest or doc still naming those IPs is stale.
| Fact | Value |
|------|-------|
| Namespace | `tidaldb-cluster` |
| Pods | `tidaldb-{0,1,2}` (one StatefulSet, 3 replicas) |
| Peer DNS | `tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500` (HTTP), `:9601` (gRPC) |
| Client VIP | `tidaldb.tidaldb-cluster.svc.cluster.local:9500` (readiness-gated) |
| Server image | `registry.threesix.ai/tidal/server@sha256:173e803…` (`:m11p5`) |
| Stress image | `registry.threesix.ai/tidal/stress@sha256:3a75c311…` (`:m11p3`) |
| Storage | local-path 5Gi/pod (on-node NVMe) — **NOT** Longhorn (see WORKLOG) |
| CPU/pod | limit `2` (the write pool is ~2 workers on the leader) |
Targets for any new Job manifest — use pod DNS for `--target` (so status-polling
reaches survivors during a kill window) and the VIP for `--leader-url`:
```yaml
args:
- --target
- http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --leader-url
- http://tidaldb.tidaldb-cluster.svc.cluster.local:9500
```
---
## Run pattern
Every run deploys the generator as an in-cluster Job (port-forward adds
API-server serialization latency — never use it for capacity numbers; only the
kill loop port-forwards, and only to read `/cluster/status`).
```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f k8s/<job>.yaml
kubectl logs -f job/<job-name> -n tidaldb-cluster
kubectl delete job <job-name> -n tidaldb-cluster # re-arm before re-running
```
## CLI flags (authoritative — from `src/main.rs`)
| Flag | Default | Notes |
|------|---------|-------|
| `--target <url>` | (required, repeatable) | Region gateway; reads round-robin across all |
| `--leader-url <url>` | none | Pin leader-path writes here to skip the forward hop |
| `--api-key` | `$TIDAL_API_KEY` | Bearer; the cluster requires it |
| `--ack <leader\|quorum>` | topology default | Sent as `x-tidal-ack` per write |
| `--ramp <preset\|rps:secs,…>` | `peach-100k` | Presets: `smoke`, `quick`, `peach-100k`, `max` |
| `--stage-secs <n>` | 45 | Hold per preset stage; 300–600 for soak |
| `--mix <preset\|op=w,…>` | `peach` | Presets: `peach`, `reads`, `writes`. Ops: `feed,search,view,like,skip,item,embed` |
| `--write-path <leader\|sharded>` | `leader` | `sharded` removes the single-leader funnel (not replicated) |
| `--corpus <n>` | 10000 | Items+embeddings to seed; 20k for gate runs |
| `--users <n>` | 50000 | Virtual user id space |
| `--skip-seed` | false | Set after the first run of a session (corpus persists on PVC) |
| `--embedding-dim <n>` | 128 | Deployed schema = 128; thepeach real = 1536 |
| `--hot-skew <f>` | 1.3 | Power-law concentration onto hot items |
| `--poll-status` | false | Poll `/cluster/status` between stages for lag — **always set when measuring lag** |
| `--stop-on-knee` | false | Stop at first SLO-breaching stage |
| `--dau <n>` | 100000 | DAU the verdict translates the ceiling against |
SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA);
error rate ≥ 1% (429/408/503/5xx/transport) = the knee.
## Layout
```
src/ generator (scheduler, workload model, client, metrics)
k8s/ Job manifests — one per checkpoint
stress-job.yaml generic ramp
stress-job-t2a.yaml T2-A quorum throughput
stress-job-t2b.yaml T2-B acked-loss under kills
stress-job-t3.yaml T3 automatic-failover gate
scripts/
t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods)
```
## Checkpoint status
| ID | Gate | Status |
|----|------|--------|
| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done |
| T2-A | ≥1000 quorum writes/s | ✓ 2980/s |
| T2-B | 0 acked loss across kills | ✓ |
| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) |
| **T4** | scale 3→5→3 under load, joiner ≤5min | **next** |
| T-read | vector-search recall@k + query QPS/p99 | **not built** (see PROCESS) |
| T5 | sharded ≥5000 quorum writes/s | blocked on p6 |

136
tidal-stress/WORKLOG.md Normal file
View File

@ -0,0 +1,136 @@
# tidal-stress worklog
Newest first. Chronicle of stress-test runs against the live k3s cluster, the
gates they cleared, and the operational lessons each one cost.
---
## 2026-06-12/13 — m11p5 migration + T3 automatic-failover gate
### What shipped
Rolled the cluster from m11p3/p4 (3 StatefulSets, static ClusterIPs) to **m11p5**
(1 StatefulSet × 3 replicas, headless-Service DNS, elastic membership). New
namespace `tidaldb-cluster`. Then gated **T3** (automatic leader-failover).
**Result: T3 PASS — 10/10 leader kills, max election 6157ms (gate <10 000ms),
zero acked loss, post-ramp lag=0.**
| Kill | Victim → leader | ms | Kill | Victim → leader | ms |
|------|-----------------|----|----|-----------------|----|
| 1 | tidaldb-0 → tidaldb-2 | 2752 | 6 | tidaldb-2 → tidaldb-0 | 2656 |
| 2 | tidaldb-2 → tidaldb-0 | 4312 | 7 | tidaldb-0 → tidaldb-1 | 3517 |
| 3 | tidaldb-0 → tidaldb-2 | 2180 | 8 | tidaldb-1 → tidaldb-0 | 2758¹ |
| 4 | tidaldb-2 → tidaldb-0 | 3675 | 9 | tidaldb-0 → tidaldb-1 | 2235 |
| 5 | tidaldb-0 → tidaldb-2 | 2168 | 10 | tidaldb-1 → tidaldb-2 | 6157 |
¹ Kill 8 measured from Raft logs (election won 03:55:42.758, kill 03:55:40) —
the port-forward died when the VIP routed to the killed pod, so the loop's own
timer recorded a false TIMEOUT. The election itself was clean.
Stress envelope during the gate: 1500 rps quorum writes, 900s, 1.35M ops,
1.51% error (all 503s = quorum rejections inside the ~2–4s kill windows, not
acked losses), post-ramp lag=0 on all three nodes.
### Three things that cost us a run each
1. **Longhorn storage → 74–85% error rate.** The new cluster first provisioned
Longhorn 10Gi PVCs (networked storage). Every quorum write needs follower
fsync; over Longhorn that's a network hop on the commit path, which collapsed
throughput. Two full T3 runs were invalid before we spotted it. **Fix:**
`storageClassName: local-path`, 5Gi — matches the on-node NVMe that T2-A was
measured on. The first valid run came straight after the switch.
2. **`kubectl exec` leader-detection died on the killed pod.** The v1/v2 kill
loops exec'd into a pod to read cluster status; when that pod was the one
being killed (or mid-initContainer on reboot), exec returned
`container not found` and the loop logged bogus 17–19s "election" times. The
status API also returns `"none (election in progress)"` as the leader field
mid-election, which the loop tried to `kubectl delete`. **Fix:** v3 uses a
port-forward to the VIP + `curl /cluster/status` with a strict
`^tidaldb-[0-9]+$` leader regex — never execs into a tidaldb pod.
3. **`kubectl wait --for=condition=Ready` hangs across a force-delete+recreate.**
When a pod is force-deleted and the StatefulSet recreates it with the same
name, `kubectl wait` latches the old object and blocks past its own
`--timeout`. Had to `kill` the hung wait twice (kills 7 and 9) to let the loop
advance. Kills 9–10 were driven by hand with a fresh port-forward.
### The reseed observation (expected, not a bug)
At 1500 rps a ~50s pod outage opens a WAL gap large enough that the returning
pod can't catch up by streaming segments — it latches `reseed_required`
(m11p5 §2.4), self-exits, and installs a full snapshot on its next boot.
Snapshot install takes ~120–150s, during which the pod is `0/1 Running`. Quorum
holds throughout (2/3 voters always up), and **a reseeding pod still
participates in elections** — kill 4 elected tidaldb-0 in 3675ms while it was
mid-reseed. So the slow K8s readiness is orthogonal to election health; the gate
measures election time from the Raft log, not pod readiness.
### FINDING — reseeded node stays NotReady forever on an idle cluster (m11p5 bug)
Surfaced 11h after the gate, verifying cluster state. tidaldb-1 (kill 10's
victim) installed its snapshot cleanly at **03:59:42** (`snapshot_seq=1441537`)
and was a healthy follower from that moment — `/cluster/status` showed it
reachable, `lag_events=0`; `/health` returned `ok:true`. But K8s kept it
**`0/1 Running` for 11.5 hours**. The readiness flag did not flip to ready until
**15:21:25**, when it finally logged `joiner first-converged … readiness is now
sticky-ready`.
Root cause: the install-boot readiness predicate (§4 hysteresis) flips ready only
when it *observes* `lag <= learner_promote_lag` through the catch-up
receiver/ship path. The stress job ended right after kill 10, so the cluster went
idle — **no new events were shipped, so the convergence-observation path was
never exercised**, and readiness never flipped. A fully-caught-up node sat out of
the client VIP for 11h on a technicality.
Impact: in production this bites a **scale-up during a quiet period** — the new
node installs, catches up, but never joins the client Service until write traffic
resumes. It also means T4's joiner-Ready timing is only valid *under sustained
load*; an idle-window join would hang the readiness gate.
This is an **upstream tidaldb fix**, not a stress-harness workaround: the
first-converged check should fire on a caught-up node even with zero inbound
replication (e.g. evaluate against a leader heartbeat / committed-index compare,
not only against observed ship traffic). Do not bandaid it in the manifest.
Filed as a T4 watch-item in PROCESS.md.
### Strategic note (vector-DB positioning)
Discussed where tidaldb sits vs other databases. CockroachDB is a different
animal (distributed SQL) — not the comp. The real comp is **vector DBs**
(Qdrant/Milvus/Weaviate/pgvector), and the honest read is:
- tidaldb's differentiator is **synchronous quorum durability + <6s auto-failover
on the write path** — which almost no vector DB has (they treat vectors as a
rebuildable derived index and skip write durability). That's a real niche
*when the signals are source-of-truth* (thepeach view/like/skip are).
- We should **not** race them on ANN QPS/recall/ingest — a dedicated engine wins
that and should.
- **Gap:** every number we have is write-path only (`--mix writes`). We have
zero read-path data — no recall@k, no query QPS/p99. That's exactly the axis
vector DBs are judged on, and the tool can measure read latency/throughput but
**cannot measure recall** (no ground-truth oracle). Building that is the
T-read checkpoint in PROCESS.md.
---
## 2026-06-11 — m11p4 regression + m11p3 T2 gate
m11p3 (quorum acks, one replicated log) cleared **T2-A** (2980 quorum writes/s,
p99=49ms, knee ~5k rps) and **T2-B** (3 leader kills, lag=0 after each, zero
acked loss). m11p4 (catch-up timer retry + WAL TSEG segment versioning) held the
gate with no regression (2975 writes/s, −0.2%). Full results:
`docs/ops/stress-test-p3-t2.md`, `docs/ops/stress-test-p4-regression.md`.
This was still on the **old 3-StatefulSet / static-ClusterIP** architecture —
the `10.43.99.11-13` IPs in `tmp/stress-test-plan.md` date from here and are now
stale (see PROCESS.md for the migration).
---
## 2026-06-10 — T0 baseline
Replicated `/signals` ~90/s (WAL fsync + synchronous gRPC ship per write, ~2
workers on a cpu-2 pod); sharded `/sharded/signals` 3669/s at 0 errors, knee not
reached. Full results: `docs/ops/stress-test-thepeach.md`.