Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production
width) and ran the realistic peach mix (feed-profile reads + signal writes) on the
m11p6 mTLS cluster.
Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps
(128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader
pool, not vector size. The vector READ path (feed-profile retrieve — the
db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps,
never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod
at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput.
Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of
knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA.
Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects
the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
Deploying the m11-44b768b image (p6 sharding + p7 mTLS + p8 ops + p9 correctness)
surfaced two blockers; both fixed here.
1. statefulset.yaml: the m11p7 change made the :9500 HTTP plane serve TLS, but the
startup/liveness/readiness probes still used scheme HTTP — kubelet got a TLS
handshake back ("malformed HTTP response \x15\x03\x03") and pods never went
Ready. Set scheme: HTTPS on all three probes (kubelet skips cert verification
for httpGet probes, so the cert's DNS-only SANs are fine). Image pinned to the
m11-44b768b amd64 digest.
2. tidal-stress: the generator's reqwest client did default cert verification and
had no way to trust the cluster's private CA, so https:// targets failed. Added
--ca-cert <pem> (verified TLS against the mounted tidaldb-cluster-tls ca.crt)
and --insecure (skip verification, escape hatch). New StressError::CaCert for
the PEM read fault.
stress-job-m11p6-baseline.yaml: T2-A-equivalent quorum-write throughput run on the
new stack — https:// targets, ca.crt mounted from the tidaldb-cluster-tls Secret,
--ca-cert verified TLS. Drops the removed --write-path flag (m11p6 unified the
write path to hash-routing).
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load
generator + capacity ramp for the standalone and multi-process cluster surfaces,
modeling a thepeach feed session (feed reads + view/like/skip signals + search,
signal-dominated per their user-graph spec). Throttleable target rate, ramp
presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom
mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a
backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict
translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/).
Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures
latency from each request's intended send time, so a server stall inflates the
percentiles a closed-loop test hides; it shed-and-counts rather than blocking when
the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps.
Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md):
reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at
~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship);
the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the
100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod
restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns).
