# Phase 10: Multi-Node UAT (m8p10)

## Delivers

True multi-process cluster mode — each `tidal-server cluster --region <name>` process owns
exactly ONE region and peers with sibling processes over real gRPC — plus the tier-3 chaos
suite that the ROADMAP defines for this phase: multi-process UAT of all 5 original M8 UAT
scenario steps, real network-partition injection, clock-skew simulation, rolling upgrade,
cluster-runbook verification, and the performance assertions (replication < 2s p99,
failover < 10s, reconciliation < 100ms over localhost).

The single-process cluster mode (m8p8) is retained unchanged as the dev/demo default; the
multi-process mode is additive, selected by the new `--region` flag.

## Dependencies

- m8p7 (gRPC transport, `tidal-net`) — COMPLETE
- m8p8 (multi-node `tidal-server`, cluster routes) — COMPLETE
- m8p9 (cross-node scatter-gather queries) — COMPLETE
- Engine primitives: `Transport` trait, `spawn_receiver`/`start_replication`,
  `ReplicationState`, `ReplicationLagGauge`, `ReconciliationEngine`/`StateSnapshot`,
  `Hlc`, `encode_batch`, `redeliver_missed` invariants — all shipped (m8p1–p6)

## Research References

- `docs/specs/14-scale-architecture.md` §7 (query routing, deadlines), §9 (replication SLA)
- `docs/planning/ROADMAP.md` "Phase 10: Multi-Node UAT (m8p10)" — the authoritative spec
- `docs/planning/ROADMAP.md` Milestone 8 "UAT Scenario" (the original 5 steps)
- `docs/runbooks/cluster.md` — the operations that must verify against the real cluster
- `tidal/src/testing/cluster.rs` + `cluster_transport.rs` — the load-bearing relay
  invariants (seqno atomicity/rollback, `single_event_payload`, `redeliver_missed`
  destination contract) that the production relay must inherit, not duplicate

## Acceptance Criteria (Phase Level)

From ROADMAP Phase 10 (verbatim):

- [ ] 3-process cluster starts, seeds data, and passes all 5 original M8 UAT scenario steps over gRPC
- [ ] Network partition between two followers: writes continue on leader; heal restores convergence with no data loss
- [ ] Rolling upgrade: mixed-version window produces no errors or data corruption
- [ ] Every cluster runbook operation works against the real cluster
- [ ] Performance: replication < 2s, failover < 10s, reconciliation < 100ms (all over localhost)

## Task Execution Order

| Task | Title | Complexity | Depends on |
|------|-------|------------|------------|
| 01 | Engine relay module, HLC clock offset, StateSnapshot wire serde | L | — |
| 02 | Single-region cluster mode (`--region`, `RegionClusterState`) | XL | 01 |
| 03 | Cross-process routes: forwarding, promote fan-out, status aggregation, reconcile, sharded | XL | 02 |
| 04 | Multi-process harness + UAT steps 1/2/5 + perf assertions | L | 03 |
| 05 | TCP-proxy partition injection + UAT steps 3/4 + reconcile timing | L | 04 |
| 06 | Clock-skew and rolling-upgrade scenarios | M | 04 |
| 07 | Runbook-verification suite + docs/ROADMAP/CHANGELOG updates | M | 04, 05, 06 |

## Module Location

- Engine: `tidal/src/replication/relay.rs` (new), `tidal/src/replication/crdt/hlc.rs`,
  `tidal/src/replication/reconcile.rs`, `tidal/src/db/builder.rs`,
  `tidal/src/db/replication_ops.rs`
- Server: `tidal-server/src/cluster/` (split from `cluster.rs` per the tracked
  Maintainability-S follow-up: `topology.rs`, `transport.rs`, `state.rs`, `node.rs`,
  `routes.rs`, `forward.rs`)
- Tests: `tidal-server/tests/cluster_multiproc.rs`, `cluster_chaos.rs`,
  `cluster_runbook.rs` (all `#![cfg(feature = "cluster-e2e")]`)
- Docs: `docs/runbooks/cluster.md`, `docs/runbooks/kubernetes.md`,
  `docs/planning/ROADMAP.md`, `CHANGELOG.md`

## Notes

- **Partition injection mechanism.** The ROADMAP spec allows iptables/pfctl *or*
  proxy-based injection ("toxiproxy or similar"). We implement an in-harness TCP relay
  proxy: peers connect through real TCP relays the test controls; severing a relay is a
  real network partition (connections die, gRPC ships fail, circuit breaker opens) with no
  root privileges and identical behavior on macOS/Linux. iptables/pfctl require root and
  are not portable to dev machines; the proxy approach is the spec-sanctioned alternative.
- **Write-durability contract unchanged.** 204 = leader durability (storage + WAL fsync).
  A quorum-ack write contract is NOT in this phase's ROADMAP acceptance criteria; the
  runbook §8 text that loosely tied it to m8p10 is updated in task 07 to track it as a
  post-M8 follow-up. Scope decision recorded here so it is explicit, not silent.
- **Default read region in multi-process mode is the LOCAL region** (each process serves
  its own replica); `?region=` forwards to the owning process. Single-process mode keeps
  leader-default. Documented in the runbook rewrite.
- **G3 (Knuth vs Jump Consistent Hash unification) is not a prerequisite**: both routing
  tiers remain internally consistent (engine `ShardRouter` is the single router for
  `/sharded/*` writes and reads). G3 stays open as an m8p9 gap.
- **Experimental gate retained** for both cluster modes (multi-process is new; quorum-ack
  and automatic failure detection remain future work). The WARN text differentiates the
  modes honestly.
- Tier-3 suites stay behind the existing `cluster-e2e` feature so the default test run
  stays fast.