a0399550d6
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
|
||
|---|---|---|
| .. | ||
| benches | ||
| k8s | ||
| scripts | ||
| src | ||
| Cargo.toml | ||
| PROCESS.md | ||
| README.md | ||
| WORKLOG.md | ||
tidal-stress
Open-loop capacity ramp + chaos harness for the tidalDB cluster. Drives the
thepeach feed workload (signals + vector embeddings) against a live cluster and
reports a per-stage capacity verdict.
- Worklog (what's been run, what we learned): WORKLOG.md
- Process (how to run the next checkpoint): PROCESS.md
Current target cluster (as of 2026-06-13)
The cluster moved to the m11p5 single-StatefulSet architecture. The old
3-StatefulSet / static-ClusterIP model (namespace tidaldb, IPs 10.43.99.11-13)
is retired — any manifest or doc still naming those IPs is stale.
| Fact | Value |
|---|---|
| Namespace | tidaldb-cluster |
| Pods | tidaldb-{0,1,2} (one StatefulSet, 3 replicas) |
| Peer DNS | tidaldb-N.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 (HTTP), :9601 (gRPC) |
| Client VIP | tidaldb.tidaldb-cluster.svc.cluster.local:9500 (readiness-gated) |
| Server image | registry.threesix.ai/tidal/server@sha256:173e803… (:m11p5) |
| Stress image | registry.threesix.ai/tidal/stress@sha256:3a75c311… (:m11p3) |
| Storage | local-path 5Gi/pod (on-node NVMe) — NOT Longhorn (see WORKLOG) |
| CPU/pod | limit 2 (the write pool is ~2 workers on the leader) |
Targets for any new Job manifest — use pod DNS for --target (so status-polling
reaches survivors during a kill window) and the VIP for --leader-url:
args:
- --target
- http://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --target
- http://tidaldb-2.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500
- --leader-url
- http://tidaldb.tidaldb-cluster.svc.cluster.local:9500
Run pattern
Every run deploys the generator as an in-cluster Job (port-forward adds
API-server serialization latency — never use it for capacity numbers; only the
kill loop port-forwards, and only to read /cluster/status).
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f k8s/<job>.yaml
kubectl logs -f job/<job-name> -n tidaldb-cluster
kubectl delete job <job-name> -n tidaldb-cluster # re-arm before re-running
CLI flags (authoritative — from src/main.rs)
| Flag | Default | Notes |
|---|---|---|
--target <url> |
(required, repeatable) | Region gateway; reads round-robin across all |
--leader-url <url> |
none | Pin leader-path writes here to skip the forward hop |
--api-key |
$TIDAL_API_KEY |
Bearer; the cluster requires it |
--ack <leader|quorum> |
topology default | Sent as x-tidal-ack per write |
--ramp <preset|rps:secs,…> |
peach-100k |
Presets: smoke, quick, peach-100k, max |
--stage-secs <n> |
45 | Hold per preset stage; 300–600 for soak |
--mix <preset|op=w,…> |
peach |
Presets: peach, reads, writes. Ops: feed,search,view,like,skip,item,embed |
--write-path <leader|sharded> |
leader |
sharded removes the single-leader funnel (not replicated) |
--corpus <n> |
10000 | Items+embeddings to seed; 20k for gate runs |
--users <n> |
50000 | Virtual user id space |
--skip-seed |
false | Set after the first run of a session (corpus persists on PVC) |
--embedding-dim <n> |
128 | Deployed schema = 128; thepeach real = 1536 |
--hot-skew <f> |
1.3 | Power-law concentration onto hot items |
--poll-status |
false | Poll /cluster/status between stages for lag — always set when measuring lag |
--stop-on-knee |
false | Stop at first SLO-breaching stage |
--dau <n> |
100000 | DAU the verdict translates the ceiling against |
SLO: feed p99 ≤ 150ms (network-hop allowance over the in-process 50ms SLA); error rate ≥ 1% (429/408/503/5xx/transport) = the knee.
Layout
src/ generator (scheduler, workload model, client, metrics)
k8s/ Job manifests — one per checkpoint
stress-job.yaml generic ramp
stress-job-t2a.yaml T2-A quorum throughput
stress-job-t2b.yaml T2-B acked-loss under kills
stress-job-t3.yaml T3 automatic-failover gate
scripts/
t3-kill-loop-v3.sh HTTP-polling leader-kill loop (no exec into pods)
Checkpoint status
| ID | Gate | Status |
|---|---|---|
| T0 | baseline (~90/s replicated, 3669/s sharded) | ✓ done |
| T2-A | ≥1000 quorum writes/s | ✓ 2980/s |
| T2-B | 0 acked loss across kills | ✓ |
| T3 | leader-kill failover <10s p99 ×10 | ✓ max 6157ms (m11p5) |
| T4 | scale 3→5→3 under load, joiner ≤5min | next |
| T-read | vector-search recall@k + query QPS/p99 | not built (see PROCESS) |
| T5 | sharded ≥5000 quorum writes/s | blocked on p6 |