81093a6779
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
64 lines
3.4 KiB
Markdown
64 lines
3.4 KiB
Markdown
# Benchmark — 1536-dim production shape, peach mix (2026-06-14)
|
||
|
||
Live 3-node m11p6 cluster (`m11-44b768b`), mTLS, single replication group,
|
||
ack=quorum, local-path PVCs, 2 vCPU / 2Gi per pod. Schema `content_vector`
|
||
**1536-dim** (text-embedding-3-small — thepeach production width). Mix `peach`:
|
||
feed-profile reads + search + signal writes in production ratio (writes dominate;
|
||
the feed read hits `/feed?profile=…` — the `db.retrieve(profile)` path thepeach
|
||
E2/R8 will call). 20k-item corpus, 8-stage ramp, 120s/stage.
|
||
|
||
## Per-stage (total rps / write p99 / feed-read p99 / error)
|
||
|
||
| Stage | Target | Achieved | write p99 | feed p99 | search p99 | error | verdict |
|
||
|------:|-------:|---------:|----------:|---------:|-----------:|------:|---------|
|
||
| 1 | 50 | 50 | 43ms | 9ms | 45ms* | 0.00% | clean |
|
||
| 2 | 150 | 150 | 40ms | 6ms | 5ms | 0.00% | clean |
|
||
| 3 | 400 | 400 | 40ms | 8ms | 6ms | 0.00% | clean |
|
||
| 4 | 800 | 800 | 43ms | 9ms | 7ms | 0.00% | clean, comfortable |
|
||
| **5** | **1500** | **1498** | **57ms** | **11ms** | **9ms** | **0.14%** | **highest within SLO** |
|
||
| 6 | 3000 | 2976 | 4.56s | 170ms | 173ms | 6.67% | **knee — SLO breach** |
|
||
| 7 | 5000 | 3700 | 4.61s | 151ms | 153ms | 8.78% | saturated |
|
||
| 8 | 8000 | 3718 | 7.32s | 156ms | 165ms | 10.72% | saturated |
|
||
|
||
\* stage-1 search p99 is a cold-start single-sample artifact (97 requests).
|
||
|
||
## Findings
|
||
|
||
1. **1536-dim costs ~nothing on throughput vs 128-dim.** Knee is stage 6
|
||
(~2,976 rps) — identical to the 128-dim run (2,981). The write bottleneck is
|
||
quorum-commit + the 2-worker leader pool, **not** vector size. Larger
|
||
embeddings did not move the throughput ceiling.
|
||
|
||
2. **The vector read path is cheap and is NOT the bottleneck.** Feed-profile
|
||
retrieve stays p99 3–11ms through stage 5 and only ~170ms even past the knee,
|
||
while writes blow up to 4.5s. The thing we were worried about — vector search
|
||
at production width — is a non-issue for latency. Search p99 ≤ 9ms through 1500 rps.
|
||
|
||
3. **Quorum writes are the sole ceiling**, and it's CPU-bound on the leader's
|
||
2-core / ~2-worker pool. Zero 503s at every stage (quorum never timed out);
|
||
the knee is 429/in-flight-cap backpressure, not server quorum failure.
|
||
Post-ramp lag=0, no pod restarts.
|
||
|
||
4. **Memory is the dim-sensitive resource.** 20k×1536 items → 567–751 MiB/pod
|
||
(vs 229–408 MiB at 128-dim, ~12× per-vector). Modest at 20k corpus; the
|
||
binding constraint at real corpus scale. Capacity-plan RAM = corpus × 1536 ×
|
||
4B × index-overhead, not throughput.
|
||
|
||
## Recommended operating target
|
||
|
||
The knee is ~3,000 rps. For a **high-quality sustained target with real margin**
|
||
(absorbs spikes, survives a single-node failover that transiently ~halves write
|
||
capacity, keeps write p99 within the 50ms SLA):
|
||
|
||
**Target: ≤ 1,000 signal-ingest rps sustained (~1,200 rps full peach mix).**
|
||
|
||
- ~40% of the knee → 2.5× headroom; survives one node loss without breaching.
|
||
- Write p99 ~45ms (within the 50ms in-process SLA), feed p99 < 10ms, error ~0%.
|
||
- ≈ 23k DAU at a realistic 5× evening peak, or ≈ 117k DAU against average load.
|
||
- 1,500 rps is the *highest within SLO* but write p99 (57ms) and p999 (≈150ms)
|
||
are at the edge — operate below it, not at it.
|
||
|
||
**Scale levers when traffic grows past this** (both available, both unproven —
|
||
gated behind T5): (a) more CPU per leader (2→8 workers, ~linear on the write
|
||
pool); (b) m11p6 sharding — hash-route writes across S groups for ~S× the knee.
|