tidaldb/docs/planning/milestone-8/phase-10/task-07-runbook-verification-docs.md

# Task 07: Runbook-verification suite + docs/ROADMAP/CHANGELOG updates

## Delivers

1. `tidal-server/tests/cluster_runbook.rs` (`cluster-e2e` feature): executes EVERY
   operation documented in `docs/runbooks/cluster.md` (§5 core API, §6 management, §7
   sharded, §9 failover drill, §10 partition drill, §11 shutdown) against a real
   3-process cluster and asserts each response matches the documented shape.
2. The documentation truth-up: the runbook rewritten for multi-process mode, stale
   cross-references fixed, ROADMAP/CHANGELOG updated to record m8p10 completion.

## Complexity: M

## Dependencies

Tasks 04, 05, 06 (the suite drives the final shipped surface; docs describe what shipped).

## Technical Design

### Runbook-verification suite

One test per runbook section against a shared `MultiProcCluster` (plus proxies where the
drill needs them):

- §5: health probes (`/health`, `/health/startup`, `/health/live`, `/openapi.json` —
  assert the OpenAPI document includes every `/cluster/*`, `/sharded/*`, and `/hardnegs`
  route), items/embeddings (201/204 + strict-dimension 400 + zero-norm 400), signals
  (204 + undeclared-signal 400), region reads (`?region=` valid + unknown→400, limit
  clamp).
- §6: status (documented keys: `leader`, `relay_log_len`,
  `regions[].{name,applied_events,lag_events,partitioned,reachable}`), promote
  (`{ok, leader}` + unknown→400), partition/heal (`{ok, partitioned|healed}`),
  reconcile (`{ok, region, local_elapsed_ms, remote_elapsed_ms}`).
- §7: sharded writes + `scatter_gather` response block keys
  (`degraded, shards_queried, elapsed_ms, shard_deadline_ms`), `deadline_ms` honored.
- §9 failover drill and §10 partition drill: scripted exactly as the runbook numbers the
  steps — the test IS the drill (use the proxy for §10's real isolation; assert each
  step's documented observation).
- §11: SIGTERM a node → readiness flips 503 during drain, process exits 0, restart
  recovers from WAL (data-dir intact).
- Auth: with `TIDAL_API_KEY` set, protected routes 401 without bearer; probes and
  `/openapi.json` stay open.

Shape assertions check key presence/types (serde into typed structs mirroring the
documented samples), not exact values.

### Documentation updates

- **`docs/runbooks/cluster.md`** — rewrite:
  - Status box: multi-process mode EXISTS (`--region`, one process per region, real
    process isolation, real gRPC between hosts); single-process mode remains the
    dev/demo default; both still behind the experimental gate; honest remaining limits
    (writes are leader-durable not quorum-acked; leadership propagation is operator
    promote + fan-out, no automatic failure detection/election).
  - §3 topology: `http_addr`/`grpc_addr` required per region in multi-process mode,
    `--region`/`TIDAL_REGION`, `--data-dir`, `TIDAL_HLC_SKEW_MS` (test/ops escape
    hatch), example 3-process launch script.
  - §5: default read region is LOCAL in multi-process mode; `?region=` forwards;
    writes forward to the leader from any node; items/embeddings broadcast contract
    (per-peer failure reporting).
  - §6: `reachable` field, `/cluster/reconcile`, `/hardnegs`.
  - §8: 204 = leader durability (unchanged); quorum-ack re-pointed at a post-M8
    follow-up (no longer "tracked as m8p10").
  - §9/§10 drills updated for multi-process (kill a real process; real network
    partition guidance referencing the chaos suite's proxy approach; iptables/pfctl
    notes for operators who want host-level injection).
  - Cross-references: drop the stale "kubernetes.md planned / server-deployment.md
    planned, not yet written" parentheticals (both exist) and link them properly.
- **`docs/runbooks/kubernetes.md`**: update the status box paragraph — multi-process
  cluster mode now exists (m8p10 shipped); standalone-single-replica remains the
  recommended production deployment until quorum-ack/auto-failover land; add a short
  "experimental: cluster mode on k8s" note (StatefulSet-per-region sketch, explicitly
  experimental, no manifests promised).
- **`tidal-server/tests/cluster_e2e.rs` module docs**: the "KNOWN GAP — m8p10" header is
  now resolved; reframe as the single-process smoke suite complementing
  `cluster_multiproc.rs`/`cluster_chaos.rs`/`cluster_runbook.rs`.
- **`docs/planning/ROADMAP.md`**: m8p10 PARTIAL→COMPLETE (tier-2 + tier-3 detail line in
  the phase table mirroring sibling entries); M8 summary updated (no longer
  NEAR-COMPLETE; "Done When (M8 Full)" checklist satisfied — note each item); G2 marked
  RESOLVED with the same strikethrough convention as G1; line 37 M8 row updated; the §8
  quorum-ack follow-up recorded as a new tracked gap (post-M8, severity Low/Medium).
- **`CHANGELOG.md`**: m8p10 entry following the existing milestone-entry format.
- **`docs/ops/monitoring.md`**: only if tasks 02/03 surfaced new metrics; otherwise add
  the one-line pointer that per-region lag is observable via `/cluster/status`
  (`lag_events`) in cluster mode.

Doc-guard: `scripts/check-docs.sh` must stay green (canonical homes only; no mirrors).

## Acceptance Criteria

- [ ] Every runbook operation (§5–§11) executed against the real 3-process cluster with
      shape assertions; failover and partition drills scripted step-for-step
- [ ] `/openapi.json` documents every cluster-mode route the runbook names
- [ ] Runbook, kubernetes runbook, cluster_e2e module docs, ROADMAP, CHANGELOG all
      updated as specified; no stale "m8p10 pending" claims remain anywhere in docs/
      (grep `m8p10` and re-justify every remaining mention)
- [ ] `scripts/check-docs.sh` passes
- [ ] Suite green via `cargo test -p tidal-server --features cluster-e2e --test cluster_runbook`