81093a6779
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
105 lines
3.7 KiB
YAML
105 lines
3.7 KiB
YAML
# 1536-dim production-representative benchmark — the REAL thepeach shape.
|
|
#
|
|
# Mix: `peach` (feed-profile reads + signal writes in production ratio), NOT
|
|
# write-only. Embeddings: 1536-dim (text-embedding-3-small), the production width.
|
|
# The feed read exercises tidaldb's named-profile retrieve (db.retrieve(profile)),
|
|
# which is what thepeach E2/R8 (services/api/feed.rs) will call.
|
|
# First run on the m11-44b768b image: confirms the HTTPS :9500 client plane,
|
|
# cluster-key auth, and m11p6 single-group replication all carry a real write
|
|
# load end-to-end, and re-establishes the T2-A capacity number on the new stack.
|
|
#
|
|
# TLS: the :9500 plane serves a private-CA cert (tidaldb-cluster-tls). The
|
|
# generator trusts it via the mounted ca.crt (--ca-cert) — verified TLS, not
|
|
# --insecure. Targets MUST be https:// and use the cert's DNS SANs (pod-DNS / VIP),
|
|
# never pod IPs (not in the SAN list).
|
|
#
|
|
# Apply: kubectl apply -f tidal-stress/k8s/stress-job-m11p6-baseline.yaml
|
|
# Watch: kubectl logs -f job/tidal-stress-1536 -n tidaldb-cluster
|
|
# Rearm: kubectl delete job tidal-stress-1536 -n tidaldb-cluster
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tidal-stress-1536
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 7200
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidal-stress
|
|
app.kubernetes.io/part-of: tidaldb
|
|
spec:
|
|
restartPolicy: Never
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: stress
|
|
image: registry.threesix.ai/tidal/stress@sha256:e130aa871f5df17a14a9e13e7df606c602b95a03d8eba49490ff7481e6e2b2b3 # m11-44b768b (TLS-aware)
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --target
|
|
- https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --target
|
|
- https://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
|
|
- --leader-url
|
|
- https://tidaldb.tidaldb-cluster.svc.cluster.local:9500 # VIP — routes to any ready pod; m11p6 hash-routes writes to the group leader
|
|
- --ca-cert
|
|
- /etc/tidaldb/tls/ca.crt
|
|
- --ack
|
|
- quorum
|
|
- --ramp
|
|
- peach-100k
|
|
- --stage-secs
|
|
- "120"
|
|
- --mix
|
|
- peach
|
|
- --embedding-dim
|
|
- "1536" # thepeach production width (text-embedding-3-small)
|
|
- --corpus
|
|
- "20000"
|
|
- --users
|
|
- "100000"
|
|
- --poll-status
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-credentials
|
|
key: TIDAL_API_KEY
|
|
- name: TIDAL_STRESS_LOG
|
|
value: warn
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "3"
|
|
memory: 1Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: cluster-tls
|
|
mountPath: /etc/tidaldb/tls
|
|
readOnly: true
|
|
volumes:
|
|
- name: cluster-tls
|
|
secret:
|
|
secretName: tidaldb-cluster-tls
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|