# Capacity stress test — thepeach feed workload (100k DAU)

Run with the `tidal-stress` crate (an open-loop, coordinated-omission-corrected
load generator) against the live 3-region multi-process cluster on k3s
(`tidaldb` namespace; 3 pods, one region each, cpu limit `"2"` per pod). Date:
2026-06-10.

> **TL;DR.** Reads scale trivially (feed p99 <15ms at thousands of rps). The
> *replicated* write path (`/signals` → single leader) saturates at **~90
> signals/s** — far short of a 100k-DAU peak — because every write funnels to one
> leader, gated by a ~2-worker write pool, each write blocking on WAL fsync + a
> synchronous cross-region gRPC ship. The *sharded* write path
> (`/sharded/signals`, hash-distributed across all 3 regions) cleanly sustains
> **3,669 signals/s at 0 errors and only ~27% cluster CPU** — that is essentially
> the 100k-DAU evening peak, with headroom. Overload degrades **gracefully**
> (HTTP 429 backpressure; zero pod restarts, replication stays caught up).
> **For thepeach's actual plan — embedding the engine in-process — 100k DAU is
> not even a question** (in-process signal write ≈ 82ns ⇒ ~12M signals/s/core).

## The workload

thepeach is a character.ai-style AI-companion feed app; its tidaldb integration
is *planned* (experiment E2 / roadmap R8: feed ranking via named tidaldb
profiles). The test models that future hot path — a feed session — mapped onto
the deployed schema (signals `view`/`like`/`skip`, 128-dim `content_vector`):

- A session = home feed (24 tiles/page, infinite scroll) → `view`/`like`/`skip`
  on tiles → occasional search → repeat. Per thepeach's user-graph spec, signals
  outnumber feed reads by 1–2 orders of magnitude ("10–1000+ events/session").
- Default mix (`peach`): ~78% `view`, ~10% `skip`, ~4% `like`, ~6.5% feed reads,
  ~1.6% search, a trickle of item/embedding registration. Hot-content power-law
  entity selection (a few viral items get most signals).
- 100k-DAU model: ~5 sessions/user/day × ~5 feed pages × ~15 signal-equivalents
  ⇒ **~780 signals/s average, ~3,900 signals/s at a 5× evening peak**; feed reads
  ~1/20th of that.

## Methodology

`tidal-stress` is **open-loop** (constant arrival rate, not N-workers-in-a-loop),
so a server stall inflates the latency percentiles a closed-loop test would hide
(coordinated-omission correction: latency measured from each request's *intended*
send time). It ramps a throttleable rate low→high, reports per-op p50/p90/p99/p999
+ max latency and a **backpressure-aware** status breakdown (429 = write
backpressure, 408 = concurrency-queue timeout, 503 = leader/region down), and
prints a capacity verdict translated to supported DAU. Run in-cluster as a k8s
Job (a `kubectl port-forward` serializes everything through the API server and
adds tens of ms — useless for capacity numbers). See `tidal-stress/k8s/stress-job.yaml`.

## Results

### Reads — never the bottleneck
`/feed` p50 ~3ms, p99 <15ms; `/search` p50 ~2ms — sustained even at 4,000 rps
total. tidalDB's own RETRIEVE SLA (p99 <50ms) holds comfortably over the cluster
network hop.

### Replicated write path (`/signals` → leader) — ~90 signals/s
`view` ok/s was pinned at **76, 76, 74, 71** across offered rates of 100→4,000
rps — that flat line *is* the ceiling (~90 signals/s incl. like/skip). Everything
above it shed as **HTTP 429** (honor-retry backpressure). No 5xx, no timeouts,
reads unaffected, replication lag stayed 0–3.

| offered (total rps) | signal ok/s | errors | feed p99 |
|---|---|---|---|
| 100 | ~90 | 1.6% (429) | 8ms |
| 500 | ~90 | 73% (429) | 5ms |
| 1,500 | ~90 | 86% (429) | 8ms |
| 4,000 | ~90 | 89% (429) | 9ms |

Bottleneck: a single leader (us-east) + a write pool of `available_parallelism`
clamped to [2,8] (≈2 on a cpu-`"2"` pod), each write blocking on WAL fsync **and
a synchronous gRPC ship to both siblings** (~178ms/write). ⇒ ~2k DAU at peak,
~11k DAU at average. **Not enough for 100k DAU on this path.**

### Sharded write path (`/sharded/signals`) — 3,669 signals/s, 0 errors
Hash-distributes writes across all 3 regions (applied to the owning region's
local store; **not replicated** — reads use `/sharded/*` scatter-gather to merge
across shards). One generator pod:

| offered (total rps) | signal ok/s | errors | view p99 | cluster cpu/region |
|---|---|---|---|---|
| 100 | 92 | 0% | 21ms | idle |
| 500 | 458 | 0% | 20ms | ~140m |
| 1,500 | 1,378 | 0% | 22ms | ~300m |
| 4,000 | **3,669** | **0%** | 31ms | ~540m (27%) |

The knee was **not reached** — the cluster was only ~27% utilized at 3,669
signals/s ⇒ **~94k DAU at peak, ~470k DAU at average**. ≈ handles the 100k-DAU
peak with headroom.

### Pushing past one generator — a connection wall, not the engine
A single generator pod tops out ~4–5k rps (its own CPU/connection limit). A
3-pod parallel fleet (aggregate >5k rps) **collapsed with 40s connection
timeouts + 503s while the cluster CPU stayed idle** — successful writes were
still fast (p50 ~12ms) but new connections hung. This is a **connection-
establishment wall** (k8s ClusterIP/conntrack/accept-queue under connection
churn from many pods), hit *before* the engine's compute ceiling. The database
engine was the bottleneck for **neither** the clean run nor the collapse.

## Verdict — can we handle 100k DAU? more?

| Deployment mode | 100k DAU? | Ceiling found | Headroom |
|---|---|---|---|
| **Embedded engine** (thepeach's plan: `Arc<TidalDb>`, `db.signal()`) | **Trivially yes** | in-process write ≈82ns ⇒ ~12M signals/s/core | RAM-bound, not CPU (capacity-planning.md) |
| **HTTP cluster, sharded writes** | **Yes** | ≥3,669 signals/s @ 27% cpu (knee not reached) | ~2–3× compute; connection wall ~5k aggregate rps |
| **HTTP cluster, replicated `/signals`** | **No** | ~90 signals/s (single leader) | needs sharding or more leader cpu |

**Can we handle more?** On the sharded/embedded paths, yes — substantially. The
replicated single-leader path does not scale writes by design.

## Recommendations

1. **thepeach's planned in-process embedding sidesteps all of this** — no HTTP,
   no leader funnel, no connection layer; the engine does millions of signals/s
   per core. At 100k–10M DAU the constraint is **RAM for the item/embedding
   index** (capacity-planning.md), not signal write throughput. thepeach's real
   1536-dim embeddings cost ~2.3× the RAM/item vs the deployed 128-dim
   (`~470MB` vs `~200MB` per 100k items).
2. **If using the HTTP cluster for the signal firehose, use `/sharded/*`** (or
   raise the leader pod's CPU to lift the write pool toward its 8-worker clamp —
   ~4× the replicated ceiling to ~360/s, still far short of peak; the single-
   leader replicated model fundamentally doesn't scale writes).
3. **Graceful under overload**: the cluster sheds excess writes as 429 and never
   fell over (0 pod restarts across every run, replication stayed caught up) — a
   client that honors the ~50ms retry hint degrades cleanly.
4. **To push past ~5k aggregate HTTP rps**: connection discipline (keep-alive
   reuse, bounded pools), more region replicas, or front the regions with an
   ingress/mesh that tolerates connection churn — the next wall is networking,
   not the database.

## Reproduce

```bash
# Build + push the image (cross-compile path), then:
kubectl apply -f tidal-stress/k8s/stress-job.yaml          # leader path, peach-100k ramp
kubectl logs -f job/tidal-stress -n tidaldb
# Sharded comparison: set --write-path sharded in the args.
```

See `tidal-stress/` for the generator (open-loop scheduler in `scheduler.rs`,
the thepeach workload model in `workload.rs`).