44820d4f81
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Four data points settle this. mp_rolling_upgrade_no_loss_no_stall passed in pipelines #5 and #6 and failed in #9 and #10, all on the same 3 CPU / 6Gi step and all four ending identically: cluster_lifecycle.rs:362 timed out: WAL relay alone must reconverge all three nodes to 1e-6 after the rolling upgrade The same test on the same commit passes locally in 19.61s. It spawns three real tidal-server processes — each with its own WAL, HNSW index, gRPC transport and tokio runtime — and asks them to reconverge to 1e-6 inside 180s, on a node with ~1700m free CPU shared with the production cluster. Two passes and two failures is a coin flip, and a coin flip that blocks image builds teaches everyone to re-run until it goes green, which is how a gate stops being one. I did not raise the budget again. That would be loosening a measured threshold to hide the hardware, and it is the third time this session that the honest answer was "the number is right, the environment is the finding". This is the escalation task 01 prescribed verbatim: a tier-3 three-process test does not belong on a 4-CPU shared node, so it becomes a documented pre-release step run where it demonstrably passes. It is in docs/runbooks/deploy-verification.md with the command and the expected 20s, and it still runs nightly inside cluster_lifecycle where a flake costs a re-read of the morning report instead of a blocked release. The image is now gated by `fast-suites`: 51 tests across nine deterministic in-process suites, no spawned processes, no convergence budget to starve — so its verdict means the same thing on a loaded shared node as on a workstation. It catches the class of regression that actually reached main today: cluster_routes asserting a wire fabrication deleted hours earlier. Also removes the last resource anchor that made step order load-bearing. Moving a step broke an anchor defined on it three times in one session (resources-light, cargo_env, resources-heavy); with the gate gone the heavy shape has exactly one consumer, so it is written inline with the reason recorded. |
||
|---|---|---|
| .. | ||
| cluster.md | ||
| deploy-verification.md | ||
| disaster-recovery.md | ||
| kubernetes.md | ||
| nightly-soak.md | ||
| on-call.md | ||
| README.md | ||
tidalDB Runbooks — index
Operational documentation for the cluster deployment (ns tidaldb-cluster,
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
and history live in the orchard9-k3sf repo (cluster-state.yaml,
deployments/history/tidaldb.md).
| When you need to… | Open |
|---|---|
| Verify a deploy end to end — every layer, with the command and its expected output | deploy-verification.md |
| Respond to an incident / page — symptom → signal → fix | on-call.md |
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | cluster.md |
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | kubernetes.md |
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | disaster-recovery.md |
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | ../ops/recovery.md |
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | ../ops/capacity-planning.md |
| Soak before a release — 1000 rps load with p99 / error-rate gates (moved out of CI on 2026-08-30; the shared runner cannot meet a 250ms p99) | nightly-soak.md |
| Read the metrics / wire dashboards & alerts | ../ops/observability.md, ../ops/grafana-tidaldb.json, ../ops/prometheus-alerts.yaml |
| Understand the live perf/topology findings (dev handoff) | ../profiling/m12-cluster-deploy-findings.md |
Current production facts (2026-06-19)
- Image:
registry.threesix.ai/tidal/server@sha256:171505745b…(m12-writeburst-rc7). - Shape: 1 StatefulSet
tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on:9500, metrics:9091. - Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
- Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
- Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
- Must-watch signal:
tidaldb_cluster_peer_breaker_state(0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
Alert rules in
../ops/prometheus-alerts.yamlare accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to aPrometheusRuleis the one open observability step (seeon-call.md§3).