diff --git a/docs/ops/benchmark-1536-peach.md b/docs/ops/benchmark-1536-peach.md new file mode 100644 index 0000000..0adaa03 --- /dev/null +++ b/docs/ops/benchmark-1536-peach.md @@ -0,0 +1,63 @@ +# Benchmark — 1536-dim production shape, peach mix (2026-06-14) + +Live 3-node m11p6 cluster (`m11-44b768b`), mTLS, single replication group, +ack=quorum, local-path PVCs, 2 vCPU / 2Gi per pod. Schema `content_vector` +**1536-dim** (text-embedding-3-small — thepeach production width). Mix `peach`: +feed-profile reads + search + signal writes in production ratio (writes dominate; +the feed read hits `/feed?profile=…` — the `db.retrieve(profile)` path thepeach +E2/R8 will call). 20k-item corpus, 8-stage ramp, 120s/stage. + +## Per-stage (total rps / write p99 / feed-read p99 / error) + +| Stage | Target | Achieved | write p99 | feed p99 | search p99 | error | verdict | +|------:|-------:|---------:|----------:|---------:|-----------:|------:|---------| +| 1 | 50 | 50 | 43ms | 9ms | 45ms* | 0.00% | clean | +| 2 | 150 | 150 | 40ms | 6ms | 5ms | 0.00% | clean | +| 3 | 400 | 400 | 40ms | 8ms | 6ms | 0.00% | clean | +| 4 | 800 | 800 | 43ms | 9ms | 7ms | 0.00% | clean, comfortable | +| **5** | **1500** | **1498** | **57ms** | **11ms** | **9ms** | **0.14%** | **highest within SLO** | +| 6 | 3000 | 2976 | 4.56s | 170ms | 173ms | 6.67% | **knee — SLO breach** | +| 7 | 5000 | 3700 | 4.61s | 151ms | 153ms | 8.78% | saturated | +| 8 | 8000 | 3718 | 7.32s | 156ms | 165ms | 10.72% | saturated | + +\* stage-1 search p99 is a cold-start single-sample artifact (97 requests). + +## Findings + +1. **1536-dim costs ~nothing on throughput vs 128-dim.** Knee is stage 6 + (~2,976 rps) — identical to the 128-dim run (2,981). The write bottleneck is + quorum-commit + the 2-worker leader pool, **not** vector size. Larger + embeddings did not move the throughput ceiling. + +2. **The vector read path is cheap and is NOT the bottleneck.** Feed-profile + retrieve stays p99 3–11ms through stage 5 and only ~170ms even past the knee, + while writes blow up to 4.5s. The thing we were worried about — vector search + at production width — is a non-issue for latency. Search p99 ≤ 9ms through 1500 rps. + +3. **Quorum writes are the sole ceiling**, and it's CPU-bound on the leader's + 2-core / ~2-worker pool. Zero 503s at every stage (quorum never timed out); + the knee is 429/in-flight-cap backpressure, not server quorum failure. + Post-ramp lag=0, no pod restarts. + +4. **Memory is the dim-sensitive resource.** 20k×1536 items → 567–751 MiB/pod + (vs 229–408 MiB at 128-dim, ~12× per-vector). Modest at 20k corpus; the + binding constraint at real corpus scale. Capacity-plan RAM = corpus × 1536 × + 4B × index-overhead, not throughput. + +## Recommended operating target + +The knee is ~3,000 rps. For a **high-quality sustained target with real margin** +(absorbs spikes, survives a single-node failover that transiently ~halves write +capacity, keeps write p99 within the 50ms SLA): + +**Target: ≤ 1,000 signal-ingest rps sustained (~1,200 rps full peach mix).** + +- ~40% of the knee → 2.5× headroom; survives one node loss without breaching. +- Write p99 ~45ms (within the 50ms in-process SLA), feed p99 < 10ms, error ~0%. +- ≈ 23k DAU at a realistic 5× evening peak, or ≈ 117k DAU against average load. +- 1,500 rps is the *highest within SLO* but write p99 (57ms) and p999 (≈150ms) + are at the edge — operate below it, not at it. + +**Scale levers when traffic grows past this** (both available, both unproven — +gated behind T5): (a) more CPU per leader (2→8 workers, ~linear on the write +pool); (b) m11p6 sharding — hash-route writes across S groups for ~S× the knee. diff --git a/k8s/cluster/schema-configmap.yaml b/k8s/cluster/schema-configmap.yaml index a6c6b5c..0dc0b02 100644 --- a/k8s/cluster/schema-configmap.yaml +++ b/k8s/cluster/schema-configmap.yaml @@ -48,4 +48,4 @@ data: embedding_slots: - name: content_vector entity: item - dimensions: 128 + dimensions: 1536 # thepeach production width (text-embedding-3-small) — see docs/specs/experiments diff --git a/tidal-stress/k8s/stress-job-1536-peach.yaml b/tidal-stress/k8s/stress-job-1536-peach.yaml new file mode 100644 index 0000000..74ec164 --- /dev/null +++ b/tidal-stress/k8s/stress-job-1536-peach.yaml @@ -0,0 +1,104 @@ +# 1536-dim production-representative benchmark — the REAL thepeach shape. +# +# Mix: `peach` (feed-profile reads + signal writes in production ratio), NOT +# write-only. Embeddings: 1536-dim (text-embedding-3-small), the production width. +# The feed read exercises tidaldb's named-profile retrieve (db.retrieve(profile)), +# which is what thepeach E2/R8 (services/api/feed.rs) will call. +# First run on the m11-44b768b image: confirms the HTTPS :9500 client plane, +# cluster-key auth, and m11p6 single-group replication all carry a real write +# load end-to-end, and re-establishes the T2-A capacity number on the new stack. +# +# TLS: the :9500 plane serves a private-CA cert (tidaldb-cluster-tls). The +# generator trusts it via the mounted ca.crt (--ca-cert) — verified TLS, not +# --insecure. Targets MUST be https:// and use the cert's DNS SANs (pod-DNS / VIP), +# never pod IPs (not in the SAN list). +# +# Apply: kubectl apply -f tidal-stress/k8s/stress-job-m11p6-baseline.yaml +# Watch: kubectl logs -f job/tidal-stress-1536 -n tidaldb-cluster +# Rearm: kubectl delete job tidal-stress-1536 -n tidaldb-cluster +apiVersion: batch/v1 +kind: Job +metadata: + name: tidal-stress-1536 + namespace: tidaldb-cluster + labels: + app.kubernetes.io/name: tidal-stress + app.kubernetes.io/part-of: tidaldb +spec: + backoffLimit: 0 + ttlSecondsAfterFinished: 7200 + template: + metadata: + labels: + app.kubernetes.io/name: tidal-stress + app.kubernetes.io/part-of: tidaldb + spec: + restartPolicy: Never + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: stress + image: registry.threesix.ai/tidal/stress@sha256:e130aa871f5df17a14a9e13e7df606c602b95a03d8eba49490ff7481e6e2b2b3 # m11-44b768b (TLS-aware) + imagePullPolicy: IfNotPresent + args: + - --target + - https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --target + - https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --target + - https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 + - --leader-url + - https://tidaldb.tidaldb-cluster.svc.cluster.local:9500 # VIP — routes to any ready pod; m11p6 hash-routes writes to the group leader + - --ca-cert + - /etc/tidaldb/tls/ca.crt + - --ack + - quorum + - --ramp + - peach-100k + - --stage-secs + - "120" + - --mix + - peach + - --embedding-dim + - "1536" # thepeach production width (text-embedding-3-small) + - --corpus + - "20000" + - --users + - "100000" + - --poll-status + env: + - name: TIDAL_API_KEY + valueFrom: + secretKeyRef: + name: tidaldb-credentials + key: TIDAL_API_KEY + - name: TIDAL_STRESS_LOG + value: warn + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: "3" + memory: 1Gi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: cluster-tls + mountPath: /etc/tidaldb/tls + readOnly: true + volumes: + - name: cluster-tls + secret: + secretName: tidaldb-cluster-tls + items: + - key: ca.crt + path: ca.crt diff --git a/tidal-stress/src/main.rs b/tidal-stress/src/main.rs index 9c7b2cc..79a00de 100644 --- a/tidal-stress/src/main.rs +++ b/tidal-stress/src/main.rs @@ -109,8 +109,9 @@ struct Cli { #[arg(long, default_value_t = 24)] feed_limit: u32, - /// Embedding width. The deployed schema's content_vector is 128 (thepeach's - /// real text-embedding-3-small is 1536 — a schema change, noted in the verdict). + /// Embedding width. MUST match the deployed schema's content_vector + /// dimensions. thepeach production is 1536 (text-embedding-3-small); the + /// 128 default is the legacy smoke width. A mismatch fails the seed. #[arg(long, default_value_t = 128)] embedding_dim: usize, @@ -249,10 +250,10 @@ async fn run() -> Result<()> { println!( " embedding dim : {}{}", cli.embedding_dim, - if cli.embedding_dim == 128 { - "" + if cli.embedding_dim == 1536 { + " (thepeach production width — text-embedding-3-small)" } else { - " (NB: deployed schema is 128)" + " (NB: must match the deployed schema's content_vector dimensions)" } ); println!(