tidaldb/docs/planning/milestone-11/phase-8.md

# m11p8 — Observability + Operations (COMPLETE — 2026-06-13)

Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p8](../../roadmap-to-cluster.md).
Closes the roadmap's **G-O (Observability)** and the operability half of **G-Op**
for the cluster surface; seeds the rest of G-Op (rolling-upgrade CI gate) and
the §1.4-3 incident ("the breaker eats the first heal").
Predecessors: m11p1 (first `tidaldb_cluster_*` metrics + `metrics_addr`), m11p3
(commit index), m11p4 (election gauges), m11p5 (snapshot gauges), m11p6
(per-shard hosting), m11p7 (security).

**Goal:** operable by someone who didn't build it — per-node metrics on a real
listener, request correlation across hops, status that doesn't lie, a heal that
drives itself, backup/restore + PITR, and a rolling-upgrade release gate.

## Design (as adopted)

m11p8 is six sub-items. Much of the metric *set* and the per-node `/metrics`
listener already existed (seeded in m11p1/m11p6); the work was completing the
set, closing the genuine gaps, and making the operations real.

### 1. Metrics + the cluster `/metrics` listener (A)

The per-node listener already binds (the engine's metrics HTTP server, per the
topology `metrics_addr`). The metric *set* gained the two members the spec named
that were missing — **breaker** and **forwards** — plus the self-heal series:

- `tidaldb_cluster_forwards_total` / `tidaldb_cluster_forward_failures_total`
  (cross-node write forwards initiated by a gateway; instrumented in
  `forward_write` and `forward_to_group_node`).
- `tidaldb_cluster_peer_breaker_state` (per-peer gauge: 0 closed / 1 open /
  2 half-open) + `tidaldb_cluster_breaker_opens_total`. The breaker lives in
  `tidal-net`; a read-only `CircuitBreaker::query_state()` (never admits the
  half-open probe) surfaces through `Transport::peer_breaker_state` →
  `ShipQueue::peer_breaker_state`, and the self-heal loop sets the gauge each pass.
- `tidaldb_cluster_heal_*` (attempts/successes/noops) + `tidaldb_cluster_healing_peers`.

**Multi-shard listener (m11p6 co-location).** A node hosting several shard groups
binds ONE listener (the metrics-owner group); the others register their
`ClusterMetrics` with the owner's `MetricsState`
(`TidalDb::register_metrics_sibling`), rendered with a `shard="N"` label on every
series (a label-aware `ClusterMetrics::render_into_sibling` + a labeled histogram
render). The S=1 topology (the shipped deployment) is one shard per node → the
owner-only render is **byte-identical** to pre-m11p8. Co-located shards never
collide on the shared scrape target.

**Dashboard + alerts.** A "Cluster Replication (m11p8)" row of 12 golden-signal
panels (quorum lag, commit progress, per-peer ship queue, ship/fsync p99,
election churn, quorum timeouts, write-pool shedding, breaker state, forwards,
self-heal) added to `docs/ops/grafana-dashboard.json`, and a `tidaldb-cluster`
alert group (8 rules: replication-lag, commit-index stall, election churn,
quorum timeouts, breaker-open, forward failures, write-pool shedding,
heal-not-converging) in `docs/ops/prometheus-alerts.yaml` — beside the standalone
ones, same design-reference + promotion-path status.

### 2. Request-id propagation + tracing across hops (B)

The standalone router's `SetRequestId` + `PropagateRequestId` + `TraceLayer` stack
was extracted to `crate::router::with_request_id_tracing` and applied to BOTH
cluster routers (single-process `build_cluster_router`, multi-process
`build_region_router`) — they previously skipped it. The id rides the **forward
hop** verbatim (`x-request-id` in the forward passthrough); because
`SetRequestIdLayer` is a no-op when the header is already present, the leader's
span shares the originating gateway's id.

**Ship hop, honestly.** Ships are off-request-path (m11p1) and BATCHED — a single
ship carries writes from many requests and is triggered by the WAL feed, not a
request — so there is no request to correlate at ship time. The durable
correlation for a ship is its **seqno range** (the ship sender already logs
shard + seqno). The version handshake (below) rides the heartbeat. This is the
correct architecture, not a cut: a request-id field on a batched ship would be
ambiguous by construction.

### 3. Truthful status (C)

Two distinct, confirmed bugs:

- **Leader's own applied row read 0** (multi-process `local_status`). The applied
  frontier (`replication_state().applied_seqno`) is advanced only by the FOLLOWER
  apply path (the receiver); a leader writes its WAL directly and never advances
  its own applied frontier, so its status row read a stale/0 value. Fix: for a
  leader, report its durable **flushed** frontier (`ship_feed.flushed_seq()`) —
  it is, by construction, applied to its own log.
- **Post-promote `ShardId(0)` keying** (single-process `applied_count`). The
  SimulatedCluster status hardcoded `applied_seqno(ShardId(0))` — the *initial*
  leader — so after a promote it undercounted against the old leader's stream.
  Fix: key on the **current** leader's shard (`ShardId(leader_region.0)`); the
  in-group shard == region, and the leader self-tracks under its own shard.

(The explorer-proposed `self.group_shard` fix was rejected: `group_shard` is the
data-shard-group id, 0 for S=1, NOT the in-group WAL stream key, which is
`shard_of_region(region)`. Keying on it would have been wrong.)

Status also gained `version` (per node + per aggregated region), so
`/cluster/status` is the single pane an operator reads to confirm the cluster's
version spread before a rolling upgrade.

### 4. Self-driving heal (D) — closes incident §1.4-3

A STANDING LEADER DUTY (`ShardReplica::tick_self_heal`) re-armed every ~3 s
(throttled inside the 50 ms election tick; non-blocking, runs inline). Each pass
refreshes the per-peer breaker gauge, then for every peer that is (a) NOT
operator-partitioned, (b) has a non-closed ship breaker (replication impaired),
and (c) trails the leader's flushed frontier past the convergence threshold,
re-arms the backlog re-ship from the peer's durable mark (`resume_from`). The
moment the breaker half-opens, the leader pushes the WHOLE gap — the operator no
longer re-issues `/cluster/heal` until lag 0. Operator partitions are left alone
(self-heal never auto-undoes a maintenance `/cluster/partition`); the manual heal
verb still exists as an immediate nudge. Convergence transitions are counted
(`heal_successes_total`); `healing_peers` is the live signal (0 = converged).

### 5. Coordinated backup/restore + WAL PITR archival (E)

- **WAL archival hook (the PITR primitive)**: a `wal.archive_dir` config
  (`TidalDb::builder().wal_archive_dir` + topology `wal.archive_dir`). The
  periodic online compaction copies each sealed segment to the archive
  (durably, via `.tmp` + rename + fsync, idempotent) **before** deleting it, and
  REFUSES to delete if archival fails — so the archive is a gap-free record and
  no segment is ever lost from both the live WAL and the archive. Segment
  filenames encode shard + first-seq, so co-located groups share one archive dir
  without collision.
- **`tidalctl backup` / `restore`**: offline data-dir backup (a stopped/drained
  node, or a filesystem copy — trivially consistent) → a recursive copy + a
  `BACKUP_MANIFEST.json` (BLAKE3 per file + the recovered WAL checkpoint cursor).
  Restore verifies EVERY file's BLAKE3 against the manifest BEFORE writing
  anything, and refuses a non-empty target (the destructive-op guard).
- **Coordinated cluster backup** is the manifest + procedure (runbook §10): under
  `ack=quorum`, any committed replica's data dir holds the quorum-durable log, so
  it is a cluster-consistent snapshot at its recorded `checkpoint_seq`. Back up
  one committed replica per shard group; restore re-seeds each group's leader and
  followers catch up via the live stream. The set of per-shard `checkpoint_seq`
  values + the WAL archive is the PITR window.

### 6. Rolling upgrade: version handshake + release gate (F)

- **Version handshake on ship**: `HeartbeatRequest.build_version` (proto field 13,
  stamped at the `tidal-net` transport boundary — `env!("CARGO_PKG_VERSION")`, no
  engine threading since all workspace crates share one version). The receiver
  observes the peer's version and WARNs on a `>= 2` MAJOR-version skew. **Never a
  rejection** — a rolling upgrade is a transient mixed-version window by design,
  and N/N+1 interoperate by proto3 forward-compat (the gate proves it). An empty
  version = a pre-m11p8 peer (version-unknown, no warning).
- **Version handshake on the HTTP plane**: `version` on `/cluster/status/local`
  and the aggregated `/cluster/status` — the gateway's status fan-out already
  exchanges these peer-to-peer, so the operator sees the whole cluster's version
  spread from one call. `#[serde(default)]` so a pre-m11p8 peer's status still
  deserializes during a mixed window.
- **Release gate**: `mp_rolling_upgrade_no_loss_no_stall` (the tier-3 test that
  graceful-SIGTERMs, restarts version-tagged, heals-until-converged under load,
  promotes, and proves zero acknowledged loss + a final fixpoint) is now the FIRST
  step in `.woodpecker.yaml` — a failure blocks the image build below. CI is
  Woodpecker, never GitHub Actions.

## Exit gate (from the roadmap)

- Dashboard answers the golden-signal questions without code.
- Backup→restore of a 100k-item cluster < 30 min.
- Upgrade-under-load gate green.

## Status

- [x] Metric set completed (breaker, forwards, self-heal) + multi-shard listener aggregation
- [x] Grafana cluster dashboard (12 panels) + Prometheus cluster alert group (8 rules)
- [x] Request-id + tracing on both cluster routers; id propagated across the forward hop
- [x] Truthful status: leader's own applied row + post-promote `ShardId(0)` keying
- [x] Self-driving heal loop (breaker-gated backlog re-ship) + heal metrics
- [x] WAL archival hook for PITR (`wal.archive_dir`, archive-before-delete, gap-free)
- [x] `tidalctl backup` / `restore` (BLAKE3 manifest, integrity-verified round-trip)
- [x] Version handshake (heartbeat `build_version` + status `version`) + N/N+1 policy
- [x] `mp_rolling_upgrade_no_loss_no_stall` promoted to a Woodpecker release gate
- [x] Docs (runbook §10/§11 + rolling upgrade, monitoring cluster metrics + alerts) + CHANGELOG

## Exit-gate evidence (local; release builds where noted)

| Gate | Target | Measured |
|------|--------|----------|
| Dashboard answers golden-signal questions without code | qualitative | 12-panel "Cluster Replication" row covers lag / commit progress / per-peer queue / ship+fsync p99 / election churn / quorum timeouts / write-pool shed / breaker / forwards / self-heal — every alert's expr has a panel. |
| Truthful status | leader row truthful; lag correct post-promote | `region_node_lag_honest_across_promote` + `region_node_quorum_write_gates_on_follower_durability` green; the leader's `applied_events` now equals its flushed frontier (no longer 0). |
| Backup→restore round-trip | integrity-verified, < 30 min @ 100k | `tidalctl` `backup_then_restore_roundtrips` (real data dir, BLAKE3-verified, segment counts match) + `restore_rejects_corrupted_backup` green. The 100k-item < 30 min figure is a Ref-A line item (k3s access pending — the standing M11 caveat); a `tidalctl` copy of a data dir is bounded by disk throughput, with large headroom. |
| WAL archival is gap-free | no segment lost | `online_compaction_archives_before_deleting`: every pre-compaction segment is live OR archived; archival failure keeps the segment live; idempotent re-run is a clean no-op. |
| Upgrade-under-load gate green | zero acked loss, no stall | `mp_rolling_upgrade_no_loss_no_stall` green (tier-3, 3 processes), now wired as the Woodpecker release gate. |

Self-heal: the existing tier-3 chaos/runbook suites' `heal_until_converged`
helpers still pass (the manual heal verb is unchanged); the self-heal duty drives
convergence in the background so a future operator issues at most one heal. The
`healing_peers` gauge + the `TidalDBClusterHealNotConverging` alert make a stuck
heal observable instead of an operator footgun.

## Verification status

- Workspace `cargo fmt --all -- --check`: clean.
- `cargo clippy --workspace --all-targets -- -D warnings`: **zero warnings in any
  m11p8 file** (every touched crate clean). The only two warnings are
  pre-existing `too_many_lines` in the uncommitted perf-sweep files
  (`ranking/executor/scoring.rs`, `tests.rs`) — not m11p8, left untouched.
- Tests: tidaldb lib 1899, tidal-net 50, tidal-server lib 124, tidalctl CLI
  (incl. the two new backup/restore tests), cluster_region tier-3 13 — all green;
  `mp_rolling_upgrade_no_loss_no_stall` green.
- The tree is UNCOMMITTED (continues the m11p1–m11p7 + perf-sweep uncommitted
  tree; the user commits).