71e80ef655
The suite had not been run since 2026-08-23 and deps were not installed. Running it
against the freshly rolled m12-vsc-20260830 found four failures. Every one was the
harness doing its job; three were stale pins it explicitly told me to invert.
REAL FINDING, caught by the suite and nothing else: tidaldb-2 was NotReady mid-run.
It had exited(0) with {"reason":"reseed_self_restart","shard":1}, reinstalled a
snapshot and converged. Designed behavior - but the suite sampled readiness ONCE and
reported a self-healing cluster as broken. Readiness is now polled via
waitForPodsReady with a bounded budget and the whole timeline attached as evidence.
Deliberately not Playwright retries: retries:0 is correct here, because a live check
that only passes on attempt two has told you something true.
STALE PINS INVERTED (each verified live first, not taken on the message's word):
- 06-logs: ANSI escapes are gone (0 in a 5-line sample), BUG-006 resolved on this
image. Now pinned so a regression to coloured output fails.
- 09-operator-authority + CAP-015 capture: tidaldb_http_* exists (185 series
against a 552 baseline). Runbook 9.1 moved from inert to LIVE. CAP-015 keeps its
purpose - state the gaps - and now names the one that is still real: no JSON_LOGS.
- The transient /search 500 and public 502 were tidaldb-2's restart window, not
defects; both surfaces returned 200 on eight retries afterwards.
THRESHOLD CALIBRATED AGAINST A MEASUREMENT, TWICE. My first fix capped
consecutive ship failures at 500, guessing a restart burst was ~100. Measurement
killed it: a reseed restart is a ~2 minute absence, which at the shipper's 100ms
cadence is ~1200-2000 failures - observed exactly 1950, then "peer recovered", with
peer_acked_seqno back at the frontier. A COUNT cannot separate "a peer restarted"
from "shipping is stuck"; it only encodes how long the peer was away. The test now
compares the newest distress line against the newest recovery line and fails only
when distress is newer. Same correction applied to the alert in k3s-fleet.
STALE EVIDENCE WAS THE WORST GAP. demo/public/captures and capture-manifest.json
still described m12-admin-gate-20260823 - two image rolls stale - while
demo:preflight reported "audited perfect" about week-old frames, and the rendered
title card read "image m12-admin-gate-20260823 - 32 checks green". The capture suite
writes to test-results/demo-captures/ and the copy-and-merge step into the published
set simply did not exist; it was done by hand once. Added demo/promote.ts: copies
frames, verifies each PNG against its fragment hash, and stamps buildRevision and
verifiedImage from the live StatefulSet. Verdicts land `pending`, so preflight fails
until the frames are audited - that failure is the gate. scenes.ts now derives the
image tag and check count from the manifest, and preflight fails if a literal is
pasted back in (proven by pasting one back in).
All 10 captures were opened individually at full resolution; the audit note is stored
in the manifest beside each verdict rather than only in prose.
Green: 34 e2e + 5 hermetic semantics + 10 captures + preflight + 2107 lib.
Video: demo/out/deploy-verification.mp4, 90.05s 1920x1080 h264, title card now
reading "image m12-vsc-20260830 - 34 checks green".
CLAUDE.md gains a Deploy Verification section and AGENTS.md a short mandatory
pointer: every deploy is verified through this harness, and maintaining it is part
of the change, not follow-up. The suite pins current reality including defects, so a
correct improvement WILL turn it red - and that is the harness working.
|
||
|---|---|---|
| .. | ||
| cluster.md | ||
| deploy-verification.md | ||
| disaster-recovery.md | ||
| kubernetes.md | ||
| on-call.md | ||
| README.md | ||
tidalDB Runbooks — index
Operational documentation for the cluster deployment (ns tidaldb-cluster,
single StatefulSet, full-placement RF3, quorum-ack + automatic election). Live state
and history live in the orchard9-k3sf repo (cluster-state.yaml,
deployments/history/tidaldb.md).
| When you need to… | Open |
|---|---|
| Verify a deploy end to end — every layer, with the command and its expected output | deploy-verification.md |
| Respond to an incident / page — symptom → signal → fix | on-call.md |
| Operate the cluster — API, failover, partition, membership, scale, rolling upgrade | cluster.md |
| Run it on Kubernetes — deploy, digest-pin, probes, scale up/down, troubleshoot pods | kubernetes.md |
| Back up / restore / DR — object-store export, restore, byte-verify, query-proof, R2, rebuild | disaster-recovery.md |
| Recover a single node / standalone engine — corrupt keyspace, WAL, lock, schema | ../ops/recovery.md |
| Size a deployment — single-node tables + the measured Ref-A cluster envelope | ../ops/capacity-planning.md |
| Read the metrics / wire dashboards & alerts | ../ops/observability.md, ../ops/grafana-tidaldb.json, ../ops/prometheus-alerts.yaml |
| Understand the live perf/topology findings (dev handoff) | ../profiling/m12-cluster-deploy-findings.md |
Current production facts (2026-06-19)
- Image:
registry.threesix.ai/tidal/server@sha256:171505745b…(m12-writeburst-rc7). - Shape: 1 StatefulSet
tidaldb, 3 pods = 3 regions = 3 voters, full-placement RF3 (every pod hosts all 3 shard groups; gRPC 9601/9602/9603), HTTPS+mTLS on:9500, metrics:9091. - Guarantees live: quorum-ack writes, automatic election/failover, elastic seed-join (learner→voter), inter-node mTLS, per-node Prometheus.
- Shipped fixes in this image: reseed-loop resolved, seed-join promotion, election-divergence/quarantine, read-SLA (CPU oversubscription), and the write-burst false-partition fix.
- Known limits: read p99 ≤10 ms to ~1000 rps (ceiling ~1–1.5k/s, CPU-bound); write knee ~250 rps; soak at 200 rps; 1M corpus needs >16 GiB nodes; 2.5× write-scaling needs ≥5 nodes + partitioned placement.
- Must-watch signal:
tidaldb_cluster_peer_breaker_state(0/1/2) — a live peer stuck at 1 is the write-burst/partition pattern.
Alert rules in
../ops/prometheus-alerts.yamlare accurate but design-reference — not yet loaded by the live Prometheus. Promoting them to aPrometheusRuleis the one open observability step (seeon-call.md§3).