d21a202a56
Every command in it was run against the live orchard9-k3sf deployment and its
output recorded before commit. Nothing is aspirational, and the three defects
found while dogfooding it are fixed rather than left for the reader:
* the backup check sorted ALL backups by timestamp and selected a
restore-canary run (20 items, 1 volume) — it would have "passed" while
telling you nothing about the fleet. Now filters on the schedule label.
* the certificate check dialled the hostname, which fails on a workstation
behind a split-DNS resolver. Now connects by IP with SNI.
* a prose line was sitting inside a bash fence.
Sections 1-8 verify what is deployed today. Section 9 is deliberately separate:
HTTP metrics, the operator/data credential split, and structured logs are
committed and tested but INERT until an image roll, so their absence is not
mistaken for a regression. Five dashboard panels are legitimately empty for the
same reason and the doc says which.
Carries the two measurement traps this deploy actually produced, because both
generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind
and reads like a dead node), and pod-to-pod reachability must not be probed with
/dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that
briefly looked like a cluster partition).
Also records the known-red reseed tests as environmental rather than regressions:
bisect against the preceding commit shows all three fail identically there, on
the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship
channels exist, against the harness's own 3s client timeout.
32 lines
2.8 KiB
Markdown
32 lines
2.8 KiB
Markdown
# tidalDB Runbooks — index
|
||
|
||
Operational documentation for the **cluster** deployment (ns `tidaldb-cluster`,
|
||
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
|
||
and history live in the `orchard9-k3sf` repo (`cluster-state.yaml`,
|
||
`deployments/history/tidaldb.md`).
|
||
|
||
| When you need to… | Open |
|
||
|---|---|
|
||
| **Verify a deploy end to end** — every layer, with the command and its expected output | [`deploy-verification.md`](deploy-verification.md) |
|
||
| **Respond to an incident / page** — symptom → signal → fix | [`on-call.md`](on-call.md) |
|
||
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | [`cluster.md`](cluster.md) |
|
||
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | [`kubernetes.md`](kubernetes.md) |
|
||
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | [`disaster-recovery.md`](disaster-recovery.md) |
|
||
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | [`../ops/recovery.md`](../ops/recovery.md) |
|
||
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | [`../ops/capacity-planning.md`](../ops/capacity-planning.md) |
|
||
| Read the metrics / wire dashboards & alerts | [`../ops/observability.md`](../ops/observability.md), [`../ops/grafana-tidaldb.json`](../ops/grafana-tidaldb.json), [`../ops/prometheus-alerts.yaml`](../ops/prometheus-alerts.yaml) |
|
||
| Understand the live perf/topology findings (dev handoff) | [`../profiling/m12-cluster-deploy-findings.md`](../profiling/m12-cluster-deploy-findings.md) |
|
||
|
||
## Current production facts (2026-06-19)
|
||
|
||
- **Image:** `registry.threesix.ai/tidal/server@sha256:171505745b…` (`m12-writeburst-rc7`).
|
||
- **Shape:** 1 StatefulSet `tidaldb`, 3 pods = 3 regions = 3 voters, **full-placement RF3** (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on `:9500`, metrics `:9091`.
|
||
- **Guarantees live:** quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
|
||
- **Shipped fixes in this image:** reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
|
||
- **Known limits:** read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
|
||
- **Must-watch signal:** `tidaldb_cluster_peer_breaker_state` (0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
|
||
|
||
> Alert rules in `../ops/prometheus-alerts.yaml` are accurate but **design-reference
|
||
> — not yet loaded by the live Prometheus.** Promoting them to a `PrometheusRule` is
|
||
> the one open observability step (see [`on-call.md`](on-call.md) §3).
|