tidaldb/tidal-stress/WORKLOG.md

tidal-stress worklog

Newest first. Chronicle of stress-test runs against the live k3s cluster, the gates they cleared, and the operational lessons each one cost.

---

2026-06-12/13 — m11p5 migration + T3 automatic-failover gate

What shipped

Rolled the cluster from m11p3/p4 (3 StatefulSets, static ClusterIPs) to m11p5 (1 StatefulSet × 3 replicas, headless-Service DNS, elastic membership). New namespace tidaldb-cluster. Then gated T3 (automatic leader-failover).

Result: T3 PASS — 10/10 leader kills, max election 6157ms (gate <10 000ms), zero acked loss, post-ramp lag=0.

Kill	Victim → leader	ms	Kill	Victim → leader	ms
1	tidaldb-0 → tidaldb-2	2752	6	tidaldb-2 → tidaldb-0	2656
2	tidaldb-2 → tidaldb-0	4312	7	tidaldb-0 → tidaldb-1	3517
3	tidaldb-0 → tidaldb-2	2180	8	tidaldb-1 → tidaldb-0	2758¹
4	tidaldb-2 → tidaldb-0	3675	9	tidaldb-0 → tidaldb-1	2235
5	tidaldb-0 → tidaldb-2	2168	10	tidaldb-1 → tidaldb-2	6157

¹ Kill 8 measured from Raft logs (election won 03:55:42.758, kill 03:55:40) — the port-forward died when the VIP routed to the killed pod, so the loop's own timer recorded a false TIMEOUT. The election itself was clean.

Stress envelope during the gate: 1500 rps quorum writes, 900s, 1.35M ops, 1.51% error (all 503s = quorum rejections inside the ~2–4s kill windows, not acked losses), post-ramp lag=0 on all three nodes.

Three things that cost us a run each

1. Longhorn storage → 74–85% error rate. The new cluster first provisioned Longhorn 10Gi PVCs (networked storage). Every quorum write needs follower fsync; over Longhorn that's a network hop on the commit path, which collapsed throughput. Two full T3 runs were invalid before we spotted it. Fix: storageClassName: local-path, 5Gi — matches the on-node NVMe that T2-A was measured on. The first valid run came straight after the switch.

2. kubectl exec leader-detection died on the killed pod. The v1/v2 kill loops exec'd into a pod to read cluster status; when that pod was the one being killed (or mid-initContainer on reboot), exec returned container not found and the loop logged bogus 17–19s "election" times. The status API also returns "none (election in progress)" as the leader field mid-election, which the loop tried to kubectl delete. Fix: v3 uses a port-forward to the VIP + curl /cluster/status with a strict ^tidaldb-[0-9]+$ leader regex — never execs into a tidaldb pod.

3. kubectl wait --for=condition=Ready hangs across a force-delete+recreate. When a pod is force-deleted and the StatefulSet recreates it with the same name, kubectl wait latches the old object and blocks past its own --timeout. Had to kill the hung wait twice (kills 7 and 9) to let the loop advance. Kills 9–10 were driven by hand with a fresh port-forward.

The reseed observation (expected, not a bug)

At 1500 rps a ~50s pod outage opens a WAL gap large enough that the returning pod can't catch up by streaming segments — it latches reseed_required (m11p5 §2.4), self-exits, and installs a full snapshot on its next boot. Snapshot install takes ~120–150s, during which the pod is 0/1 Running. Quorum holds throughout (2/3 voters always up), and a reseeding pod still participates in elections — kill 4 elected tidaldb-0 in 3675ms while it was mid-reseed. So the slow K8s readiness is orthogonal to election health; the gate measures election time from the Raft log, not pod readiness.

FINDING — reseeded node stays NotReady forever on an idle cluster (m11p5 bug)

Surfaced 11h after the gate, verifying cluster state. tidaldb-1 (kill 10's victim) installed its snapshot cleanly at 03:59:42 (snapshot_seq=1441537) and was a healthy follower from that moment — /cluster/status showed it reachable, lag_events=0; /health returned ok:true. But K8s kept it 0/1 Running for 11.5 hours. The readiness flag did not flip to ready until 15:21:25, when it finally logged joiner first-converged … readiness is now sticky-ready.

Root cause: the install-boot readiness predicate (§4 hysteresis) flips ready only when it observes lag <= learner_promote_lag through the catch-up receiver/ship path. The stress job ended right after kill 10, so the cluster went idle — no new events were shipped, so the convergence-observation path was never exercised, and readiness never flipped. A fully-caught-up node sat out of the client VIP for 11h on a technicality.

Impact: in production this bites a scale-up during a quiet period — the new node installs, catches up, but never joins the client Service until write traffic resumes. It also means T4's joiner-Ready timing is only valid under sustained load; an idle-window join would hang the readiness gate.

This is an upstream tidaldb fix, not a stress-harness workaround: the first-converged check should fire on a caught-up node even with zero inbound replication (e.g. evaluate against a leader heartbeat / committed-index compare, not only against observed ship traffic). Do not bandaid it in the manifest. Filed as a T4 watch-item in PROCESS.md.

Strategic note (vector-DB positioning)

Discussed where tidaldb sits vs other databases. CockroachDB is a different animal (distributed SQL) — not the comp. The real comp is vector DBs (Qdrant/Milvus/Weaviate/pgvector), and the honest read is:

• tidaldb's differentiator is synchronous quorum durability + <6s auto-failover on the write path — which almost no vector DB has (they treat vectors as a rebuildable derived index and skip write durability). That's a real niche when the signals are source-of-truth (thepeach view/like/skip are).
• We should not race them on ANN QPS/recall/ingest — a dedicated engine wins that and should.
• Gap: every number we have is write-path only (--mix writes). We have zero read-path data — no recall@k, no query QPS/p99. That's exactly the axis vector DBs are judged on, and the tool can measure read latency/throughput but cannot measure recall (no ground-truth oracle). Building that is the T-read checkpoint in PROCESS.md.

---

2026-06-11 — m11p4 regression + m11p3 T2 gate

m11p3 (quorum acks, one replicated log) cleared T2-A (2980 quorum writes/s, p99=49ms, knee ~5k rps) and T2-B (3 leader kills, lag=0 after each, zero acked loss). m11p4 (catch-up timer retry + WAL TSEG segment versioning) held the gate with no regression (2975 writes/s, −0.2%). Full results: docs/ops/stress-test-p3-t2.md, docs/ops/stress-test-p4-regression.md.

This was still on the old 3-StatefulSet / static-ClusterIP architecture — the 10.43.99.11-13 IPs in tmp/stress-test-plan.md date from here and are now stale (see PROCESS.md for the migration).

---

2026-06-10 — T0 baseline

Replicated /signals ~90/s (WAL fsync + synchronous gRPC ship per write, ~2 workers on a cpu-2 pod); sharded /sharded/signals 3669/s at 0 errors, knee not reached. Full results: docs/ops/stress-test-thepeach.md.