1b5bcbacd7
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit
|
||
|---|---|---|
| .. | ||
| benches | ||
| k8s | ||
| scripts | ||
| src | ||
| Cargo.toml | ||
| PROCESS.md | ||
| README.md | ||
| WORKLOG.md | ||
tidal-stress
Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the
thepeach feed workload (signals + vector embeddings) against a live cluster and
reports a per-stage capacity verdict.
- Worklog (what's been run, what we learned): WORKLOG.md
- Process (how to run the next checkpoint): PROCESS.md
Current target cluster (as of 2026-06-13)
The cluster moved to the m11p5 single-StatefulSet architecture. The old
3-StatefulSet / static-ClusterIP model (namespace tidaldb, IPs 10.43.99.11-13)
is retired — any manifest or doc still naming those IPs is stale.
| Fact | Value |
|---|---|
| Namespace | tidaldb-cluster |
| Pods | tidaldb-{0,1,2} (one StatefulSet, 3 replicas) |
| Peer DNS | tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 (HTTP), :9601 (gRPC) |
| Client VIP | tidaldb.tidaldb-cluster.svc.cluster.local:9500 (readiness-gated) |
| Server image | registry.threesix.ai/tidal/server@sha256:173e803… (:m11p5) |
| Stress image | registry.threesix.ai/tidal/stress@sha256:3a75c311… (:m11p3) |
| Storage | local-path 5Gi/pod (on-node NVMe) — NOT Longhorn (see WORKLOG) |
| CPU/pod | limit 2 (the write pool is ~2 workers on the leader) |
Targets for any new Job manifest — use pod DNS for --target (so status-polling
reaches survivors during a kill window) and the VIP for --leader-url:
args:
- --target
- http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --leader-url
- http://tidaldb.tidaldb-cluster.svc.cluster.local:9500
Run pattern
Every run deploys the generator as an in-cluster Job (port-forward adds
API-server serialization latency — never use it for capacity numbers; only the
kill loop port-forwards, and only to read /cluster/status).
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f k8s/<job>.yaml
kubectl logs -f job/<job-name> -n tidaldb-cluster
kubectl delete job <job-name> -n tidaldb-cluster # re-arm before re-running
CLI flags (authoritative — from src/main.rs)
| Flag | Default | Notes |
|---|---|---|
--target <url> |
(required, repeatable) | Region gateway; reads round-robin across all |
--leader-url <url> |
none | Pin leader-path writes here to skip the forward hop |
--api-key |
$TIDAL_API_KEY |
Bearer; the cluster requires it |
--ack <leader|quorum> |
topology default | Sent as x-tidal-ack per write |
--ramp <preset|rps:secs,…> |
peach-100k |
Presets: smoke, quick, peach-100k, max |
--stage-secs <n> |
45 | Hold per preset stage; 300–600 for soak |
--mix <preset|op=w,…> |
peach |
Presets: peach, reads, writes. Ops: feed,search,view,like,skip,item,embed |
--write-path <leader|sharded> |
leader |
sharded removes the single-leader funnel (not replicated) |
--corpus <n> |
10000 | Items+embeddings to seed; 20k for gate runs |
--users <n> |
50000 | Virtual user id space |
--skip-seed |
false | Set after the first run of a session (corpus persists on PVC) |
--embedding-dim <n> |
128 | Deployed schema = 128; thepeach real = 1536 |
--hot-skew <f> |
1.3 | Power-law concentration onto hot items |
--poll-status |
false | Poll /cluster/status between stages for lag — always set when measuring lag |
--stop-on-knee |
false | Stop at first SLO-breaching stage |
--dau <n> |
100000 | DAU the verdict translates the ceiling against |
SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA); error rate ≥ 1% (429/408/503/5xx/transport) = the knee.
Layout
src/ generator (scheduler, workload model, client, metrics)
k8s/ Job manifests — one per checkpoint
stress-job.yaml generic ramp
stress-job-t2a.yaml T2-A quorum throughput
stress-job-t2b.yaml T2-B acked-loss under kills
stress-job-t3.yaml T3 automatic-failover gate
scripts/
t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods)
Checkpoint status
| ID | Gate | Status |
|---|---|---|
| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done |
| T2-A | ≥1000 quorum writes/s | ✓ 2980/s |
| T2-B | 0 acked loss across kills | ✓ |
| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) |
| T4 | scale 3→5→3 under load, joiner ≤5min | next |
| T-read | vector-search recall@k + query QPS/p99 | not built (see PROCESS) |
| T5 | sharded ≥5000 quorum writes/s | blocked on p6 |