jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
a6f663f002
tidaldb/docs/planning/milestone-8/phase-10/task-07-runbook-verification-docs.md
jx12n 8a0950260f feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
2026-06-10 14:07:33 -06:00

5.6 KiB
Raw Blame History

Task 07: Runbook-verification suite + docs/ROADMAP/CHANGELOG updates

Delivers

  1. tidal-server/tests/cluster_runbook.rs (cluster-e2e feature): executes EVERY operation documented in docs/runbooks/cluster.md (§5 core API, §6 management, §7 sharded, §9 failover drill, §10 partition drill, §11 shutdown) against a real 3-process cluster and asserts each response matches the documented shape.
  2. The documentation truth-up: the runbook rewritten for multi-process mode, stale cross-references fixed, ROADMAP/CHANGELOG updated to record m8p10 completion.

Complexity: M

Dependencies

Tasks 04, 05, 06 (the suite drives the final shipped surface; docs describe what shipped).

Technical Design

Runbook-verification suite

One test per runbook section against a shared MultiProcCluster (plus proxies where the drill needs them):

  • §5: health probes (/health, /health/startup, /health/live, /openapi.json — assert the OpenAPI document includes every /cluster/*, /sharded/*, and /hardnegs route), items/embeddings (201/204 + strict-dimension 400 + zero-norm 400), signals (204 + undeclared-signal 400), region reads (?region= valid + unknown→400, limit clamp).
  • §6: status (documented keys: leader, relay_log_len, regions[].{name,applied_events,lag_events,partitioned,reachable}), promote ({ok, leader} + unknown→400), partition/heal ({ok, partitioned|healed}), reconcile ({ok, region, local_elapsed_ms, remote_elapsed_ms}).
  • §7: sharded writes + scatter_gather response block keys (degraded, shards_queried, elapsed_ms, shard_deadline_ms), deadline_ms honored.
  • §9 failover drill and §10 partition drill: scripted exactly as the runbook numbers the steps — the test IS the drill (use the proxy for §10's real isolation; assert each step's documented observation).
  • §11: SIGTERM a node → readiness flips 503 during drain, process exits 0, restart recovers from WAL (data-dir intact).
  • Auth: with TIDAL_API_KEY set, protected routes 401 without bearer; probes and /openapi.json stay open.

Shape assertions check key presence/types (serde into typed structs mirroring the documented samples), not exact values.

Documentation updates

  • docs/runbooks/cluster.md — rewrite:
    • Status box: multi-process mode EXISTS (--region, one process per region, real process isolation, real gRPC between hosts); single-process mode remains the dev/demo default; both still behind the experimental gate; honest remaining limits (writes are leader-durable not quorum-acked; leadership propagation is operator promote + fan-out, no automatic failure detection/election).
    • §3 topology: http_addr/grpc_addr required per region in multi-process mode, --region/TIDAL_REGION, --data-dir, TIDAL_HLC_SKEW_MS (test/ops escape hatch), example 3-process launch script.
    • §5: default read region is LOCAL in multi-process mode; ?region= forwards; writes forward to the leader from any node; items/embeddings broadcast contract (per-peer failure reporting).
    • §6: reachable field, /cluster/reconcile, /hardnegs.
    • §8: 204 = leader durability (unchanged); quorum-ack re-pointed at a post-M8 follow-up (no longer "tracked as m8p10").
    • §9/§10 drills updated for multi-process (kill a real process; real network partition guidance referencing the chaos suite's proxy approach; iptables/pfctl notes for operators who want host-level injection).
    • Cross-references: drop the stale "kubernetes.md planned / server-deployment.md planned, not yet written" parentheticals (both exist) and link them properly.
  • docs/runbooks/kubernetes.md: update the status box paragraph — multi-process cluster mode now exists (m8p10 shipped); standalone-single-replica remains the recommended production deployment until quorum-ack/auto-failover land; add a short "experimental: cluster mode on k8s" note (StatefulSet-per-region sketch, explicitly experimental, no manifests promised).
  • tidal-server/tests/cluster_e2e.rs module docs: the "KNOWN GAP — m8p10" header is now resolved; reframe as the single-process smoke suite complementing cluster_multiproc.rs/cluster_chaos.rs/cluster_runbook.rs.
  • docs/planning/ROADMAP.md: m8p10 PARTIAL→COMPLETE (tier-2 + tier-3 detail line in the phase table mirroring sibling entries); M8 summary updated (no longer NEAR-COMPLETE; "Done When (M8 Full)" checklist satisfied — note each item); G2 marked RESOLVED with the same strikethrough convention as G1; line 37 M8 row updated; the §8 quorum-ack follow-up recorded as a new tracked gap (post-M8, severity Low/Medium).
  • CHANGELOG.md: m8p10 entry following the existing milestone-entry format.
  • docs/ops/monitoring.md: only if tasks 02/03 surfaced new metrics; otherwise add the one-line pointer that per-region lag is observable via /cluster/status (lag_events) in cluster mode.

Doc-guard: scripts/check-docs.sh must stay green (canonical homes only; no mirrors).

Acceptance Criteria

  • Every runbook operation (§5–§11) executed against the real 3-process cluster with shape assertions; failover and partition drills scripted step-for-step
  • /openapi.json documents every cluster-mode route the runbook names
  • Runbook, kubernetes runbook, cluster_e2e module docs, ROADMAP, CHANGELOG all updated as specified; no stale "m8p10 pending" claims remain anywhere in docs/ (grep m8p10 and re-justify every remaining mention)
  • scripts/check-docs.sh passes
  • Suite green via cargo test -p tidal-server --features cluster-e2e --test cluster_runbook