tidaldb/docs/ops/stress-test-thepeach.md

Capacity stress test — thepeach feed workload (100k DAU)

Run with the tidal-stress crate (an open-loop, coordinated-omission-corrected load generator) against the live 3-region multi-process cluster on k3s (tidaldb namespace; 3 pods, one region each, cpu limit "2" per pod). Date: 2026-06-10.

TL;DR. Reads scale trivially (feed p99 <15ms at thousands of rps). The replicated write path (/signals → single leader) saturates at ~90 signals/s — far short of a 100k-DAU peak — because every write funnels to one leader, gated by a ~2-worker write pool, each write blocking on WAL fsync + a synchronous cross-region gRPC ship. The sharded write path (/sharded/signals, hash-distributed across all 3 regions) cleanly sustains 3,669 signals/s at 0 errors and only ~27% cluster CPU — that is essentially the 100k-DAU evening peak, with headroom. Overload degrades gracefully (HTTP 429 backpressure; zero pod restarts, replication stays caught up). For thepeach's actual plan — embedding the engine in-process — 100k DAU is not even a question (in-process signal write ≈ 82ns ⇒ ~12M signals/s/core).

The workload

thepeach is a character.ai-style AI-companion feed app; its tidaldb integration is planned (experiment E2 / roadmap R8: feed ranking via named tidaldb profiles). The test models that future hot path — a feed session — mapped onto the deployed schema (signals view/like/skip, 128-dim content_vector):

• A session = home feed (24 tiles/page, infinite scroll) → view/like/skip on tiles → occasional search → repeat. Per thepeach's user-graph spec, signals outnumber feed reads by 1–2 orders of magnitude ("10–1000+ events/session").
• Default mix (peach): ~78% view, ~10% skip, ~4% like, ~6.5% feed reads, ~1.6% search, a trickle of item/embedding registration. Hot-content power-law entity selection (a few viral items get most signals).
• 100k-DAU model: ~5 sessions/user/day × ~5 feed pages × ~15 signal-equivalents ⇒ ~780 signals/s average, ~3,900 signals/s at a 5× evening peak; feed reads ~1/20th of that.

Methodology

tidal-stress is open-loop (constant arrival rate, not N-workers-in-a-loop), so a server stall inflates the latency percentiles a closed-loop test would hide (coordinated-omission correction: latency measured from each request's intended send time). It ramps a throttleable rate low→high, reports per-op p50/p90/p99/p999

• max latency and a backpressure-aware status breakdown (429 = write backpressure, 408 = concurrency-queue timeout, 503 = leader/region down), and prints a capacity verdict translated to supported DAU. Run in-cluster as a k8s Job (a kubectl port-forward serializes everything through the API server and adds tens of ms — useless for capacity numbers). See tidal-stress/k8s/stress-job.yaml.

Results

Reads — never the bottleneck

/feed p50 ~3ms, p99 <15ms; /search p50 ~2ms — sustained even at 4,000 rps total. tidalDB's own RETRIEVE SLA (p99 <50ms) holds comfortably over the cluster network hop.

Replicated write path (/signals → leader) — ~90 signals/s

view ok/s was pinned at 76, 76, 74, 71 across offered rates of 100→4,000 rps — that flat line is the ceiling (~90 signals/s incl. like/skip). Everything above it shed as HTTP 429 (honor-retry backpressure). No 5xx, no timeouts, reads unaffected, replication lag stayed 0–3.

offered (total rps)	signal ok/s	errors	feed p99
100	~90	1.6% (429)	8ms
500	~90	73% (429)	5ms
1,500	~90	86% (429)	8ms
4,000	~90	89% (429)	9ms

Bottleneck: a single leader (us-east) + a write pool of available_parallelism clamped to [2,8] (≈2 on a cpu-"2" pod), each write blocking on WAL fsync and a synchronous gRPC ship to both siblings (~178ms/write). ⇒ ~2k DAU at peak, ~11k DAU at average. Not enough for 100k DAU on this path.

Sharded write path (/sharded/signals) — 3,669 signals/s, 0 errors

Hash-distributes writes across all 3 regions (applied to the owning region's local store; not replicated — reads use /sharded/* scatter-gather to merge across shards). One generator pod:

offered (total rps)	signal ok/s	errors	view p99	cluster cpu/region
100	92	0%	21ms	idle
500	458	0%	20ms	~140m
1,500	1,378	0%	22ms	~300m
4,000	3,669	0%	31ms	~540m (27%)

The knee was not reached — the cluster was only ~27% utilized at 3,669 signals/s ⇒ ~94k DAU at peak, ~470k DAU at average. ≈ handles the 100k-DAU peak with headroom.

Pushing past one generator — a connection wall, not the engine

A single generator pod tops out ~4–5k rps (its own CPU/connection limit). A 3-pod parallel fleet (aggregate >5k rps) collapsed with 40s connection timeouts + 503s while the cluster CPU stayed idle — successful writes were still fast (p50 ~12ms) but new connections hung. This is a connection- establishment wall (k8s ClusterIP/conntrack/accept-queue under connection churn from many pods), hit before the engine's compute ceiling. The database engine was the bottleneck for neither the clean run nor the collapse.

Verdict — can we handle 100k DAU? more?

Deployment mode	100k DAU?	Ceiling found	Headroom
Embedded engine (thepeach's plan: Arc<TidalDb>, db.signal())	Trivially yes	in-process write ≈82ns ⇒ ~12M signals/s/core	RAM-bound, not CPU (capacity-planning.md)
HTTP cluster, sharded writes	Yes	≥3,669 signals/s @ 27% cpu (knee not reached)	~2–3× compute; connection wall ~5k aggregate rps
HTTP cluster, replicated /signals	No	~90 signals/s (single leader)	needs sharding or more leader cpu

Can we handle more? On the sharded/embedded paths, yes — substantially. The replicated single-leader path does not scale writes by design.

Recommendations

1. thepeach's planned in-process embedding sidesteps all of this — no HTTP, no leader funnel, no connection layer; the engine does millions of signals/s per core. At 100k–10M DAU the constraint is RAM for the item/embedding index (capacity-planning.md), not signal write throughput. thepeach's real 1536-dim embeddings cost ~2.3× the RAM/item vs the deployed 128-dim (~470MB vs ~200MB per 100k items).
2. If using the HTTP cluster for the signal firehose, use /sharded/* (or raise the leader pod's CPU to lift the write pool toward its 8-worker clamp — ~4× the replicated ceiling to ~360/s, still far short of peak; the single- leader replicated model fundamentally doesn't scale writes).
3. Graceful under overload: the cluster sheds excess writes as 429 and never fell over (0 pod restarts across every run, replication stayed caught up) — a client that honors the ~50ms retry hint degrades cleanly.
4. To push past ~5k aggregate HTTP rps: connection discipline (keep-alive reuse, bounded pools), more region replicas, or front the regions with an ingress/mesh that tolerates connection churn — the next wall is networking, not the database.

Reproduce

# Build + push the image (cross-compile path), then:
kubectl apply -f tidal-stress/k8s/stress-job.yaml          # leader path, peach-100k ramp
kubectl logs -f job/tidal-stress -n tidaldb
# Sharded comparison: set --write-path sharded in the args.

See tidal-stress/ for the generator (open-loop scheduler in scheduler.rs, the thepeach workload model in workload.rs).