31ee612f27
Scale write throughput across data-shard groups while keeping a single unified read surface: - scatter_gather.rs: pooled fan-out across shard groups (replaces per-request client construction); cross-shard query results merged on one node - cluster/node.rs: cross-shard read routing — a read on any node gathers from every shard group's leader and unions results - cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM on the real mTLS plane) - dto.rs: cross-shard query/result DTOs - k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers, topology-configmap) - k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A - tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job - tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind - docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings |
||
|---|---|---|
| .. | ||
| benches | ||
| k8s | ||
| scripts | ||
| src | ||
| Cargo.toml | ||
| PROCESS.md | ||
| README.md | ||
| WORKLOG.md | ||
tidal-stress
Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the
thepeach feed workload (signals + vector embeddings) against a live cluster and
reports a per-stage capacity verdict.
- Worklog (what's been run, what we learned): WORKLOG.md
- Process (how to run the next checkpoint): PROCESS.md
Current target cluster (as of 2026-06-13)
The cluster moved to the m11p5 single-StatefulSet architecture. The old
3-StatefulSet / static-ClusterIP model (namespace tidaldb, IPs 10.43.99.11-13)
is retired — any manifest or doc still naming those IPs is stale.
| Fact | Value |
|---|---|
| Namespace | tidaldb-cluster |
| Pods | tidaldb-{0,1,2} (one StatefulSet, 3 replicas) |
| Peer DNS | tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 (HTTP), :9601 (gRPC) |
| Client VIP | tidaldb.tidaldb-cluster.svc.cluster.local:9500 (readiness-gated) |
| Server image | registry.threesix.ai/tidal/server@sha256:173e803… (:m11p5) |
| Stress image | registry.threesix.ai/tidal/stress@sha256:3a75c311… (:m11p3) |
| Storage | local-path 5Gi/pod (on-node NVMe) — NOT Longhorn (see WORKLOG) |
| CPU/pod | limit 2 (the write pool is ~2 workers on the leader) |
Targets for any new Job manifest — use pod DNS for --target (so status-polling
reaches survivors during a kill window) and the VIP for --leader-url:
args:
- --target
- http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --leader-url
- http://tidaldb.tidaldb-cluster.svc.cluster.local:9500
Run pattern
Every run deploys the generator as an in-cluster Job (port-forward adds
API-server serialization latency — never use it for capacity numbers; only the
kill loop port-forwards, and only to read /cluster/status).
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f k8s/<job>.yaml
kubectl logs -f job/<job-name> -n tidaldb-cluster
kubectl delete job <job-name> -n tidaldb-cluster # re-arm before re-running
CLI flags (authoritative — from src/main.rs)
| Flag | Default | Notes |
|---|---|---|
--target <url> |
(required, repeatable) | Region gateway; reads round-robin across all |
--leader-url <url> |
none | Pin leader-path writes here to skip the forward hop |
--api-key |
$TIDAL_API_KEY |
Bearer; the cluster requires it |
--ack <leader|quorum> |
topology default | Sent as x-tidal-ack per write |
--ramp <preset|rps:secs,…> |
peach-100k |
Presets: smoke, quick, peach-100k, max |
--stage-secs <n> |
45 | Hold per preset stage; 300–600 for soak |
--mix <preset|op=w,…> |
peach |
Presets: peach, reads, writes. Ops: feed,search,view,like,skip,item,embed |
--write-path <leader|sharded> |
leader |
sharded removes the single-leader funnel (not replicated) |
--corpus <n> |
10000 | Items+embeddings to seed; 20k for gate runs |
--users <n> |
50000 | Virtual user id space |
--skip-seed |
false | Set after the first run of a session (corpus persists on PVC) |
--embedding-dim <n> |
128 | Deployed schema = 128; thepeach real = 1536 |
--hot-skew <f> |
1.3 | Power-law concentration onto hot items |
--poll-status |
false | Poll /cluster/status between stages for lag — always set when measuring lag |
--stop-on-knee |
false | Stop at first SLO-breaching stage |
--dau <n> |
100000 | DAU the verdict translates the ceiling against |
SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA); error rate ≥ 1% (429/408/503/5xx/transport) = the knee.
Layout
src/ generator (scheduler, workload model, client, metrics)
k8s/ Job manifests — one per checkpoint
stress-job.yaml generic ramp
stress-job-t2a.yaml T2-A quorum throughput
stress-job-t2b.yaml T2-B acked-loss under kills
stress-job-t3.yaml T3 automatic-failover gate
scripts/
t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods)
Checkpoint status
| ID | Gate | Status |
|---|---|---|
| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done |
| T2-A | ≥1000 quorum writes/s | ✓ 2980/s |
| T2-B | 0 acked loss across kills | ✓ |
| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) |
| T4 | scale 3→5→3 under load, joiner ≤5min | next |
| T-read | vector-search recall@k + query QPS/p99 | not built (see PROCESS) |
| T5 | sharded ≥5000 quorum writes/s | blocked on p6 |