81093a6779
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
3.4 KiB
Benchmark — 1536-dim production shape, peach mix (2026-06-14)
Live 3-node m11p6 cluster (m11-44b768b), mTLS, single replication group,
ack=quorum, local-path PVCs, 2 vCPU / 2Gi per pod. Schema content_vector
1536-dim (text-embedding-3-small — thepeach production width). Mix peach:
feed-profile reads + search + signal writes in production ratio (writes dominate;
the feed read hits /feed?profile=… — the db.retrieve(profile) path thepeach
E2/R8 will call). 20k-item corpus, 8-stage ramp, 120s/stage.
Per-stage (total rps / write p99 / feed-read p99 / error)
| Stage | Target | Achieved | write p99 | feed p99 | search p99 | error | verdict |
|---|---|---|---|---|---|---|---|
| 1 | 50 | 50 | 43ms | 9ms | 45ms* | 0.00% | clean |
| 2 | 150 | 150 | 40ms | 6ms | 5ms | 0.00% | clean |
| 3 | 400 | 400 | 40ms | 8ms | 6ms | 0.00% | clean |
| 4 | 800 | 800 | 43ms | 9ms | 7ms | 0.00% | clean, comfortable |
| 5 | 1500 | 1498 | 57ms | 11ms | 9ms | 0.14% | highest within SLO |
| 6 | 3000 | 2976 | 4.56s | 170ms | 173ms | 6.67% | knee — SLO breach |
| 7 | 5000 | 3700 | 4.61s | 151ms | 153ms | 8.78% | saturated |
| 8 | 8000 | 3718 | 7.32s | 156ms | 165ms | 10.72% | saturated |
* stage-1 search p99 is a cold-start single-sample artifact (97 requests).
Findings
-
1536-dim costs ~nothing on throughput vs 128-dim. Knee is stage 6 (~2,976 rps) — identical to the 128-dim run (2,981). The write bottleneck is quorum-commit + the 2-worker leader pool, not vector size. Larger embeddings did not move the throughput ceiling.
-
The vector read path is cheap and is NOT the bottleneck. Feed-profile retrieve stays p99 3–11ms through stage 5 and only ~170ms even past the knee, while writes blow up to 4.5s. The thing we were worried about — vector search at production width — is a non-issue for latency. Search p99 ≤ 9ms through 1500 rps.
-
Quorum writes are the sole ceiling, and it's CPU-bound on the leader's 2-core / ~2-worker pool. Zero 503s at every stage (quorum never timed out); the knee is 429/in-flight-cap backpressure, not server quorum failure. Post-ramp lag=0, no pod restarts.
-
Memory is the dim-sensitive resource. 20k×1536 items → 567–751 MiB/pod (vs 229–408 MiB at 128-dim, ~12× per-vector). Modest at 20k corpus; the binding constraint at real corpus scale. Capacity-plan RAM = corpus × 1536 × 4B × index-overhead, not throughput.
Recommended operating target
The knee is ~3,000 rps. For a high-quality sustained target with real margin (absorbs spikes, survives a single-node failover that transiently ~halves write capacity, keeps write p99 within the 50ms SLA):
Target: ≤ 1,000 signal-ingest rps sustained (~1,200 rps full peach mix).
- ~40% of the knee → 2.5× headroom; survives one node loss without breaching.
- Write p99 ~45ms (within the 50ms in-process SLA), feed p99 < 10ms, error ~0%.
- ≈ 23k DAU at a realistic 5× evening peak, or ≈ 117k DAU against average load.
- 1,500 rps is the highest within SLO but write p99 (57ms) and p999 (≈150ms) are at the edge — operate below it, not at it.
Scale levers when traffic grows past this (both available, both unproven — gated behind T5): (a) more CPU per leader (2→8 workers, ~linear on the write pool); (b) m11p6 sharding — hash-route writes across S groups for ~S× the knee.