81093a6779
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
52 lines
1.7 KiB
YAML
52 lines
1.7 KiB
YAML
# The tidalDB schema the cluster pods load at startup with `--schema`.
|
|
#
|
|
# Identical SHAPE to the standalone set's k8s/schema-configmap.yaml (signal types
|
|
# + decay, text fields, embedding slots) — every region in the cluster MUST load
|
|
# the same schema. Mounting it as a ConfigMap lets you evolve the schema without
|
|
# rebuilding the image; the schema is read once at boot, so roll the StatefulSet
|
|
# (kubectl rollout restart) to apply changes, and only make backward-compatible
|
|
# changes on a populated data dir (see docs/ops/recovery.md § schema mismatch).
|
|
#
|
|
# Schema format reference: docs/guides/server-deployment.md and API.md.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: tidaldb-schema
|
|
namespace: tidaldb-cluster
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/part-of: tidaldb
|
|
data:
|
|
schema.yaml: |
|
|
signals:
|
|
- name: view
|
|
entity: item
|
|
decay:
|
|
exponential:
|
|
half_life_seconds: 604800 # 7 days
|
|
windows: [one_hour, twenty_four_hours, seven_days]
|
|
velocity: true
|
|
positive_engagement: true # folds into the user preference vector
|
|
- name: like
|
|
entity: item
|
|
decay:
|
|
exponential:
|
|
half_life_seconds: 1209600 # 14 days
|
|
windows: [twenty_four_hours, seven_days, thirty_days, all_time]
|
|
velocity: false
|
|
positive_engagement: true
|
|
- name: skip
|
|
entity: item
|
|
decay:
|
|
permanent: true
|
|
velocity: false
|
|
text_fields:
|
|
- name: title
|
|
kind: text
|
|
- name: category
|
|
kind: keyword
|
|
embedding_slots:
|
|
- name: content_vector
|
|
entity: item
|
|
dimensions: 1536 # thepeach production width (text-embedding-3-small) — see docs/specs/experiments
|