bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling

Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
2026-06-13 21:40:58 -06:00 · 2026-06-13 21:40:58 -06:00 · 81093a6779
commit 81093a6779
parent df0e1b98de
4 changed files with 174 additions and 6 deletions
--- a/docs/ops/benchmark-1536-peach.md
+++ b/docs/ops/benchmark-1536-peach.md
@ -0,0 +1,63 @@
+# Benchmark — 1536-dim production shape, peach mix (2026-06-14)
+
+Live 3-node m11p6 cluster (`m11-44b768b`), mTLS, single replication group,
+ack=quorum, local-path PVCs, 2 vCPU / 2Gi per pod. Schema `content_vector`
+**1536-dim** (text-embedding-3-small — thepeach production width). Mix `peach`:
+feed-profile reads + search + signal writes in production ratio (writes dominate;
+the feed read hits `/feed?profile=…` — the `db.retrieve(profile)` path thepeach
+E2/R8 will call). 20k-item corpus, 8-stage ramp, 120s/stage.
+
+## Per-stage (total rps / write p99 / feed-read p99 / error)
+
+| Stage | Target | Achieved | write p99 | feed p99 | search p99 | error | verdict |
+|------:|-------:|---------:|----------:|---------:|-----------:|------:|---------|
+| 1 | 50   | 50    | 43ms | 9ms  | 45ms* | 0.00% | clean |
+| 2 | 150  | 150   | 40ms | 6ms  | 5ms   | 0.00% | clean |
+| 3 | 400  | 400   | 40ms | 8ms  | 6ms   | 0.00% | clean |
+| 4 | 800  | 800   | 43ms | 9ms  | 7ms   | 0.00% | clean, comfortable |
+| **5** | **1500** | **1498** | **57ms** | **11ms** | **9ms** | **0.14%** | **highest within SLO** |
+| 6 | 3000 | 2976  | 4.56s | 170ms | 173ms | 6.67% | **knee — SLO breach** |
+| 7 | 5000 | 3700  | 4.61s | 151ms | 153ms | 8.78% | saturated |
+| 8 | 8000 | 3718  | 7.32s | 156ms | 165ms | 10.72% | saturated |
+
+\* stage-1 search p99 is a cold-start single-sample artifact (97 requests).
+
+## Findings
+
+1. **1536-dim costs ~nothing on throughput vs 128-dim.** Knee is stage 6
+   (~2,976 rps) — identical to the 128-dim run (2,981). The write bottleneck is
+   quorum-commit + the 2-worker leader pool, **not** vector size. Larger
+   embeddings did not move the throughput ceiling.
+
+2. **The vector read path is cheap and is NOT the bottleneck.** Feed-profile
+   retrieve stays p99 3–11ms through stage 5 and only ~170ms even past the knee,
+   while writes blow up to 4.5s. The thing we were worried about — vector search
+   at production width — is a non-issue for latency. Search p99 ≤ 9ms through 1500 rps.
+
+3. **Quorum writes are the sole ceiling**, and it's CPU-bound on the leader's
+   2-core / ~2-worker pool. Zero 503s at every stage (quorum never timed out);
+   the knee is 429/in-flight-cap backpressure, not server quorum failure.
+   Post-ramp lag=0, no pod restarts.
+
+4. **Memory is the dim-sensitive resource.** 20k×1536 items → 567–751 MiB/pod
+   (vs 229–408 MiB at 128-dim, ~12× per-vector). Modest at 20k corpus; the
+   binding constraint at real corpus scale. Capacity-plan RAM = corpus × 1536 ×
+   4B × index-overhead, not throughput.
+
+## Recommended operating target
+
+The knee is ~3,000 rps. For a **high-quality sustained target with real margin**
+(absorbs spikes, survives a single-node failover that transiently ~halves write
+capacity, keeps write p99 within the 50ms SLA):
+
+**Target: ≤ 1,000 signal-ingest rps sustained (~1,200 rps full peach mix).**
+
+- ~40% of the knee → 2.5× headroom; survives one node loss without breaching.
+- Write p99 ~45ms (within the 50ms in-process SLA), feed p99 < 10ms, error ~0%.
+- ≈ 23k DAU at a realistic 5× evening peak, or ≈ 117k DAU against average load.
+- 1,500 rps is the *highest within SLO* but write p99 (57ms) and p999 (≈150ms)
+  are at the edge — operate below it, not at it.
+
+**Scale levers when traffic grows past this** (both available, both unproven —
+gated behind T5): (a) more CPU per leader (2→8 workers, ~linear on the write
+pool); (b) m11p6 sharding — hash-route writes across S groups for ~S× the knee.
--- a/k8s/cluster/schema-configmap.yaml
+++ b/k8s/cluster/schema-configmap.yaml
@ -48,4 +48,4 @@ data:
    embedding_slots:
      - name: content_vector
        entity: item
-        dimensions: 128
+        dimensions: 1536   # thepeach production width (text-embedding-3-small) — see docs/specs/experiments
--- a/tidal-stress/k8s/stress-job-1536-peach.yaml
+++ b/tidal-stress/k8s/stress-job-1536-peach.yaml
@ -0,0 +1,104 @@
+# 1536-dim production-representative benchmark — the REAL thepeach shape.
+#
+# Mix: `peach` (feed-profile reads + signal writes in production ratio), NOT
+# write-only. Embeddings: 1536-dim (text-embedding-3-small), the production width.
+# The feed read exercises tidaldb's named-profile retrieve (db.retrieve(profile)),
+# which is what thepeach E2/R8 (services/api/feed.rs) will call.
+# First run on the m11-44b768b image: confirms the HTTPS :9500 client plane,
+# cluster-key auth, and m11p6 single-group replication all carry a real write
+# load end-to-end, and re-establishes the T2-A capacity number on the new stack.
+#
+# TLS: the :9500 plane serves a private-CA cert (tidaldb-cluster-tls). The
+# generator trusts it via the mounted ca.crt (--ca-cert) — verified TLS, not
+# --insecure. Targets MUST be https:// and use the cert's DNS SANs (pod-DNS / VIP),
+# never pod IPs (not in the SAN list).
+#
+# Apply:  kubectl apply -f tidal-stress/k8s/stress-job-m11p6-baseline.yaml
+# Watch:  kubectl logs -f job/tidal-stress-1536 -n tidaldb-cluster
+# Rearm:  kubectl delete job tidal-stress-1536 -n tidaldb-cluster
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: tidal-stress-1536
+  namespace: tidaldb-cluster
+  labels:
+    app.kubernetes.io/name: tidal-stress
+    app.kubernetes.io/part-of: tidaldb
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 7200
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: tidal-stress
+        app.kubernetes.io/part-of: tidaldb
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: stress
+          image: registry.threesix.ai/tidal/stress@sha256:e130aa871f5df17a14a9e13e7df606c602b95a03d8eba49490ff7481e6e2b2b3  # m11-44b768b (TLS-aware)
+          imagePullPolicy: IfNotPresent
+          args:
+            - --target
+            - https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
+            - --target
+            - https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
+            - --target
+            - https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
+            - --leader-url
+            - https://tidaldb.tidaldb-cluster.svc.cluster.local:9500   # VIP — routes to any ready pod; m11p6 hash-routes writes to the group leader
+            - --ca-cert
+            - /etc/tidaldb/tls/ca.crt
+            - --ack
+            - quorum
+            - --ramp
+            - peach-100k
+            - --stage-secs
+            - "120"
+            - --mix
+            - peach
+            - --embedding-dim
+            - "1536"            # thepeach production width (text-embedding-3-small)
+            - --corpus
+            - "20000"
+            - --users
+            - "100000"
+            - --poll-status
+          env:
+            - name: TIDAL_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: tidaldb-credentials
+                  key: TIDAL_API_KEY
+            - name: TIDAL_STRESS_LOG
+              value: warn
+          resources:
+            requests:
+              cpu: 250m
+              memory: 256Mi
+            limits:
+              cpu: "3"
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: cluster-tls
+              mountPath: /etc/tidaldb/tls
+              readOnly: true
+      volumes:
+        - name: cluster-tls
+          secret:
+            secretName: tidaldb-cluster-tls
+            items:
+              - key: ca.crt
+                path: ca.crt
--- a/tidal-stress/src/main.rs
+++ b/tidal-stress/src/main.rs
@ -109,8 +109,9 @@ struct Cli {
    #[arg(long, default_value_t = 24)]
    feed_limit: u32,

-    /// Embedding width. The deployed schema's content_vector is 128 (thepeach's
-    /// real text-embedding-3-small is 1536 — a schema change, noted in the verdict).
+    /// Embedding width. MUST match the deployed schema's content_vector
+    /// dimensions. thepeach production is 1536 (text-embedding-3-small); the
+    /// 128 default is the legacy smoke width. A mismatch fails the seed.
    #[arg(long, default_value_t = 128)]
    embedding_dim: usize,

@ -249,10 +250,10 @@ async fn run() -> Result<()> {
    println!(
        "  embedding dim : {}{}",
        cli.embedding_dim,
-        if cli.embedding_dim == 128 {
-            ""
+        if cli.embedding_dim == 1536 {
+            "  (thepeach production width — text-embedding-3-small)"
        } else {
-            "  (NB: deployed schema is 128)"
+            "  (NB: must match the deployed schema's content_vector dimensions)"
        }
    );
    println!(