tidaldb/tidal-stress

tidal-stress

Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the thepeach feed workload (signals + vector embeddings) against a live cluster and reports a per-stage capacity verdict.

• Worklog (what's been run, what we learned): WORKLOG.md
• Process (how to run the next checkpoint): PROCESS.md

---

Current target cluster (as of 2026-06-13)

The cluster moved to the m11p5 single-StatefulSet architecture. The old 3-StatefulSet / static-ClusterIP model (namespace tidaldb, IPs 10.43.99.11-13) is retired — any manifest or doc still naming those IPs is stale.

Fact	Value
Namespace	tidaldb-cluster
Pods	tidaldb-{0,1,2} (one StatefulSet, 3 replicas)
Peer DNS	tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 (HTTP), :9601 (gRPC)
Client VIP	tidaldb.tidaldb-cluster.svc.cluster.local:9500 (readiness-gated)
Server image	registry.threesix.ai/tidal/server@sha256:173e803… (:m11p5)
Stress image	registry.threesix.ai/tidal/stress@sha256:3a75c311… (:m11p3)
Storage	local-path 5Gi/pod (on-node NVMe) — NOT Longhorn (see WORKLOG)
CPU/pod	limit 2 (the write pool is ~2 workers on the leader)

Targets for any new Job manifest — use pod DNS for --target (so status-polling reaches survivors during a kill window) and the VIP for --leader-url:

args:
  - --target
  - http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
  - --target
  - http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
  - --target
  - http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
  - --leader-url
  - http://tidaldb.tidaldb-cluster.svc.cluster.local:9500

---

Run pattern

Every run deploys the generator as an in-cluster Job (port-forward adds API-server serialization latency — never use it for capacity numbers; only the kill loop port-forwards, and only to read /cluster/status).

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f k8s/<job>.yaml
kubectl logs -f job/<job-name> -n tidaldb-cluster
kubectl delete job <job-name> -n tidaldb-cluster    # re-arm before re-running

CLI flags (authoritative — from src/main.rs)

Flag	Default	Notes
--target <url>	(required, repeatable)	Region gateway; reads round-robin across all
--leader-url <url>	none	Pin leader-path writes here to skip the forward hop
--api-key	$TIDAL_API_KEY	Bearer; the cluster requires it
--ack <leader|quorum>	topology default	Sent as x-tidal-ack per write
--ramp <preset|rps:secs,…>	peach-100k	Presets: smoke, quick, peach-100k, max
--stage-secs <n>	45	Hold per preset stage; 300–600 for soak
--mix <preset|op=w,…>	peach	Presets: peach, reads, writes. Ops: feed,search,view,like,skip,item,embed
--write-path <leader|sharded>	leader	sharded removes the single-leader funnel (not replicated)
--corpus <n>	10000	Items+embeddings to seed; 20k for gate runs
--users <n>	50000	Virtual user id space
--skip-seed	false	Set after the first run of a session (corpus persists on PVC)
--embedding-dim <n>	128	Deployed schema = 128; thepeach real = 1536
--hot-skew <f>	1.3	Power-law concentration onto hot items
--poll-status	false	Poll /cluster/status between stages for lag — always set when measuring lag
--stop-on-knee	false	Stop at first SLO-breaching stage
--dau <n>	100000	DAU the verdict translates the ceiling against

SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA); error rate ≥ 1% (429/408/503/5xx/transport) = the knee.

Layout

src/            generator (scheduler, workload model, client, metrics)
k8s/            Job manifests — one per checkpoint
  stress-job.yaml       generic ramp
  stress-job-t2a.yaml   T2-A quorum throughput
  stress-job-t2b.yaml   T2-B acked-loss under kills
  stress-job-t3.yaml    T3 automatic-failover gate
scripts/
  t3-kill-loop-v3.sh    HTTP-polling leader-kill loop (no exec into pods)

Checkpoint status

ID	Gate	Status
T0	baseline (~90/s replicated, 3669/s sharded)	✓ done
T2-A	≥1000 quorum writes/s	✓ 2980/s
T2-B	0 acked loss across kills	✓
T3	leader-kill failover <10s p99 ×10	✓ max 6157ms (m11p5)
T4	scale 3→5→3 under load, joiner ≤5min	next
T-read	vector-search recall@k + query QPS/p99	not built (see PROCESS)
T5	sharded ≥5000 quorum writes/s	blocked on p6