f640764d89
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load generator + capacity ramp for the standalone and multi-process cluster surfaces, modeling a thepeach feed session (feed reads + view/like/skip signals + search, signal-dominated per their user-graph spec). Throttleable target rate, ramp presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/). Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures latency from each request's intended send time, so a server stall inflates the percentiles a closed-loop test hides; it shed-and-counts rather than blocking when the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps. Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md): reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at ~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship); the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the 100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns).
141 lines
7.5 KiB
Markdown
141 lines
7.5 KiB
Markdown
# Capacity stress test — thepeach feed workload (100k DAU)
|
||
|
||
Run with the `tidal-stress` crate (an open-loop, coordinated-omission-corrected
|
||
load generator) against the live 3-region multi-process cluster on k3s
|
||
(`tidaldb` namespace; 3 pods, one region each, cpu limit `"2"` per pod). Date:
|
||
2026-06-10.
|
||
|
||
> **TL;DR.** Reads scale trivially (feed p99 <15ms at thousands of rps). The
|
||
> *replicated* write path (`/signals` → single leader) saturates at **~90
|
||
> signals/s** — far short of a 100k-DAU peak — because every write funnels to one
|
||
> leader, gated by a ~2-worker write pool, each write blocking on WAL fsync + a
|
||
> synchronous cross-region gRPC ship. The *sharded* write path
|
||
> (`/sharded/signals`, hash-distributed across all 3 regions) cleanly sustains
|
||
> **3,669 signals/s at 0 errors and only ~27% cluster CPU** — that is essentially
|
||
> the 100k-DAU evening peak, with headroom. Overload degrades **gracefully**
|
||
> (HTTP 429 backpressure; zero pod restarts, replication stays caught up).
|
||
> **For thepeach's actual plan — embedding the engine in-process — 100k DAU is
|
||
> not even a question** (in-process signal write ≈ 82ns ⇒ ~12M signals/s/core).
|
||
|
||
## The workload
|
||
|
||
thepeach is a character.ai-style AI-companion feed app; its tidaldb integration
|
||
is *planned* (experiment E2 / roadmap R8: feed ranking via named tidaldb
|
||
profiles). The test models that future hot path — a feed session — mapped onto
|
||
the deployed schema (signals `view`/`like`/`skip`, 128-dim `content_vector`):
|
||
|
||
- A session = home feed (24 tiles/page, infinite scroll) → `view`/`like`/`skip`
|
||
on tiles → occasional search → repeat. Per thepeach's user-graph spec, signals
|
||
outnumber feed reads by 1–2 orders of magnitude ("10–1000+ events/session").
|
||
- Default mix (`peach`): ~78% `view`, ~10% `skip`, ~4% `like`, ~6.5% feed reads,
|
||
~1.6% search, a trickle of item/embedding registration. Hot-content power-law
|
||
entity selection (a few viral items get most signals).
|
||
- 100k-DAU model: ~5 sessions/user/day × ~5 feed pages × ~15 signal-equivalents
|
||
⇒ **~780 signals/s average, ~3,900 signals/s at a 5× evening peak**; feed reads
|
||
~1/20th of that.
|
||
|
||
## Methodology
|
||
|
||
`tidal-stress` is **open-loop** (constant arrival rate, not N-workers-in-a-loop),
|
||
so a server stall inflates the latency percentiles a closed-loop test would hide
|
||
(coordinated-omission correction: latency measured from each request's *intended*
|
||
send time). It ramps a throttleable rate low→high, reports per-op p50/p90/p99/p999
|
||
+ max latency and a **backpressure-aware** status breakdown (429 = write
|
||
backpressure, 408 = concurrency-queue timeout, 503 = leader/region down), and
|
||
prints a capacity verdict translated to supported DAU. Run in-cluster as a k8s
|
||
Job (a `kubectl port-forward` serializes everything through the API server and
|
||
adds tens of ms — useless for capacity numbers). See `tidal-stress/k8s/stress-job.yaml`.
|
||
|
||
## Results
|
||
|
||
### Reads — never the bottleneck
|
||
`/feed` p50 ~3ms, p99 <15ms; `/search` p50 ~2ms — sustained even at 4,000 rps
|
||
total. tidalDB's own RETRIEVE SLA (p99 <50ms) holds comfortably over the cluster
|
||
network hop.
|
||
|
||
### Replicated write path (`/signals` → leader) — ~90 signals/s
|
||
`view` ok/s was pinned at **76, 76, 74, 71** across offered rates of 100→4,000
|
||
rps — that flat line *is* the ceiling (~90 signals/s incl. like/skip). Everything
|
||
above it shed as **HTTP 429** (honor-retry backpressure). No 5xx, no timeouts,
|
||
reads unaffected, replication lag stayed 0–3.
|
||
|
||
| offered (total rps) | signal ok/s | errors | feed p99 |
|
||
|---|---|---|---|
|
||
| 100 | ~90 | 1.6% (429) | 8ms |
|
||
| 500 | ~90 | 73% (429) | 5ms |
|
||
| 1,500 | ~90 | 86% (429) | 8ms |
|
||
| 4,000 | ~90 | 89% (429) | 9ms |
|
||
|
||
Bottleneck: a single leader (us-east) + a write pool of `available_parallelism`
|
||
clamped to [2,8] (≈2 on a cpu-`"2"` pod), each write blocking on WAL fsync **and
|
||
a synchronous gRPC ship to both siblings** (~178ms/write). ⇒ ~2k DAU at peak,
|
||
~11k DAU at average. **Not enough for 100k DAU on this path.**
|
||
|
||
### Sharded write path (`/sharded/signals`) — 3,669 signals/s, 0 errors
|
||
Hash-distributes writes across all 3 regions (applied to the owning region's
|
||
local store; **not replicated** — reads use `/sharded/*` scatter-gather to merge
|
||
across shards). One generator pod:
|
||
|
||
| offered (total rps) | signal ok/s | errors | view p99 | cluster cpu/region |
|
||
|---|---|---|---|---|
|
||
| 100 | 92 | 0% | 21ms | idle |
|
||
| 500 | 458 | 0% | 20ms | ~140m |
|
||
| 1,500 | 1,378 | 0% | 22ms | ~300m |
|
||
| 4,000 | **3,669** | **0%** | 31ms | ~540m (27%) |
|
||
|
||
The knee was **not reached** — the cluster was only ~27% utilized at 3,669
|
||
signals/s ⇒ **~94k DAU at peak, ~470k DAU at average**. ≈ handles the 100k-DAU
|
||
peak with headroom.
|
||
|
||
### Pushing past one generator — a connection wall, not the engine
|
||
A single generator pod tops out ~4–5k rps (its own CPU/connection limit). A
|
||
3-pod parallel fleet (aggregate >5k rps) **collapsed with 40s connection
|
||
timeouts + 503s while the cluster CPU stayed idle** — successful writes were
|
||
still fast (p50 ~12ms) but new connections hung. This is a **connection-
|
||
establishment wall** (k8s ClusterIP/conntrack/accept-queue under connection
|
||
churn from many pods), hit *before* the engine's compute ceiling. The database
|
||
engine was the bottleneck for **neither** the clean run nor the collapse.
|
||
|
||
## Verdict — can we handle 100k DAU? more?
|
||
|
||
| Deployment mode | 100k DAU? | Ceiling found | Headroom |
|
||
|---|---|---|---|
|
||
| **Embedded engine** (thepeach's plan: `Arc<TidalDb>`, `db.signal()`) | **Trivially yes** | in-process write ≈82ns ⇒ ~12M signals/s/core | RAM-bound, not CPU (capacity-planning.md) |
|
||
| **HTTP cluster, sharded writes** | **Yes** | ≥3,669 signals/s @ 27% cpu (knee not reached) | ~2–3× compute; connection wall ~5k aggregate rps |
|
||
| **HTTP cluster, replicated `/signals`** | **No** | ~90 signals/s (single leader) | needs sharding or more leader cpu |
|
||
|
||
**Can we handle more?** On the sharded/embedded paths, yes — substantially. The
|
||
replicated single-leader path does not scale writes by design.
|
||
|
||
## Recommendations
|
||
|
||
1. **thepeach's planned in-process embedding sidesteps all of this** — no HTTP,
|
||
no leader funnel, no connection layer; the engine does millions of signals/s
|
||
per core. At 100k–10M DAU the constraint is **RAM for the item/embedding
|
||
index** (capacity-planning.md), not signal write throughput. thepeach's real
|
||
1536-dim embeddings cost ~2.3× the RAM/item vs the deployed 128-dim
|
||
(`~470MB` vs `~200MB` per 100k items).
|
||
2. **If using the HTTP cluster for the signal firehose, use `/sharded/*`** (or
|
||
raise the leader pod's CPU to lift the write pool toward its 8-worker clamp —
|
||
~4× the replicated ceiling to ~360/s, still far short of peak; the single-
|
||
leader replicated model fundamentally doesn't scale writes).
|
||
3. **Graceful under overload**: the cluster sheds excess writes as 429 and never
|
||
fell over (0 pod restarts across every run, replication stayed caught up) — a
|
||
client that honors the ~50ms retry hint degrades cleanly.
|
||
4. **To push past ~5k aggregate HTTP rps**: connection discipline (keep-alive
|
||
reuse, bounded pools), more region replicas, or front the regions with an
|
||
ingress/mesh that tolerates connection churn — the next wall is networking,
|
||
not the database.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
# Build + push the image (cross-compile path), then:
|
||
kubectl apply -f tidal-stress/k8s/stress-job.yaml # leader path, peach-100k ramp
|
||
kubectl logs -f job/tidal-stress -n tidaldb
|
||
# Sharded comparison: set --write-path sharded in the args.
|
||
```
|
||
|
||
See `tidal-stress/` for the generator (open-loop scheduler in `scheduler.rs`,
|
||
the thepeach workload model in `workload.rs`).
|