8a0950260f
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.
92 lines
5.2 KiB
Markdown
92 lines
5.2 KiB
Markdown
# Phase 10: Multi-Node UAT (m8p10)
|
||
|
||
## Delivers
|
||
|
||
True multi-process cluster mode — each `tidal-server cluster --region <name>` process owns
|
||
exactly ONE region and peers with sibling processes over real gRPC — plus the tier-3 chaos
|
||
suite that the ROADMAP defines for this phase: multi-process UAT of all 5 original M8 UAT
|
||
scenario steps, real network-partition injection, clock-skew simulation, rolling upgrade,
|
||
cluster-runbook verification, and the performance assertions (replication < 2s p99,
|
||
failover < 10s, reconciliation < 100ms over localhost).
|
||
|
||
The single-process cluster mode (m8p8) is retained unchanged as the dev/demo default; the
|
||
multi-process mode is additive, selected by the new `--region` flag.
|
||
|
||
## Dependencies
|
||
|
||
- m8p7 (gRPC transport, `tidal-net`) — COMPLETE
|
||
- m8p8 (multi-node `tidal-server`, cluster routes) — COMPLETE
|
||
- m8p9 (cross-node scatter-gather queries) — COMPLETE
|
||
- Engine primitives: `Transport` trait, `spawn_receiver`/`start_replication`,
|
||
`ReplicationState`, `ReplicationLagGauge`, `ReconciliationEngine`/`StateSnapshot`,
|
||
`Hlc`, `encode_batch`, `redeliver_missed` invariants — all shipped (m8p1–p6)
|
||
|
||
## Research References
|
||
|
||
- `docs/specs/14-scale-architecture.md` §7 (query routing, deadlines), §9 (replication SLA)
|
||
- `docs/planning/ROADMAP.md` "Phase 10: Multi-Node UAT (m8p10)" — the authoritative spec
|
||
- `docs/planning/ROADMAP.md` Milestone 8 "UAT Scenario" (the original 5 steps)
|
||
- `docs/runbooks/cluster.md` — the operations that must verify against the real cluster
|
||
- `tidal/src/testing/cluster.rs` + `cluster_transport.rs` — the load-bearing relay
|
||
invariants (seqno atomicity/rollback, `single_event_payload`, `redeliver_missed`
|
||
destination contract) that the production relay must inherit, not duplicate
|
||
|
||
## Acceptance Criteria (Phase Level)
|
||
|
||
From ROADMAP Phase 10 (verbatim):
|
||
|
||
- [ ] 3-process cluster starts, seeds data, and passes all 5 original M8 UAT scenario steps over gRPC
|
||
- [ ] Network partition between two followers: writes continue on leader; heal restores convergence with no data loss
|
||
- [ ] Rolling upgrade: mixed-version window produces no errors or data corruption
|
||
- [ ] Every cluster runbook operation works against the real cluster
|
||
- [ ] Performance: replication < 2s, failover < 10s, reconciliation < 100ms (all over localhost)
|
||
|
||
## Task Execution Order
|
||
|
||
| Task | Title | Complexity | Depends on |
|
||
|------|-------|------------|------------|
|
||
| 01 | Engine relay module, HLC clock offset, StateSnapshot wire serde | L | — |
|
||
| 02 | Single-region cluster mode (`--region`, `RegionClusterState`) | XL | 01 |
|
||
| 03 | Cross-process routes: forwarding, promote fan-out, status aggregation, reconcile, sharded | XL | 02 |
|
||
| 04 | Multi-process harness + UAT steps 1/2/5 + perf assertions | L | 03 |
|
||
| 05 | TCP-proxy partition injection + UAT steps 3/4 + reconcile timing | L | 04 |
|
||
| 06 | Clock-skew and rolling-upgrade scenarios | M | 04 |
|
||
| 07 | Runbook-verification suite + docs/ROADMAP/CHANGELOG updates | M | 04, 05, 06 |
|
||
|
||
## Module Location
|
||
|
||
- Engine: `tidal/src/replication/relay.rs` (new), `tidal/src/replication/crdt/hlc.rs`,
|
||
`tidal/src/replication/reconcile.rs`, `tidal/src/db/builder.rs`,
|
||
`tidal/src/db/replication_ops.rs`
|
||
- Server: `tidal-server/src/cluster/` (split from `cluster.rs` per the tracked
|
||
Maintainability-S follow-up: `topology.rs`, `transport.rs`, `state.rs`, `node.rs`,
|
||
`routes.rs`, `forward.rs`)
|
||
- Tests: `tidal-server/tests/cluster_multiproc.rs`, `cluster_chaos.rs`,
|
||
`cluster_runbook.rs` (all `#![cfg(feature = "cluster-e2e")]`)
|
||
- Docs: `docs/runbooks/cluster.md`, `docs/runbooks/kubernetes.md`,
|
||
`docs/planning/ROADMAP.md`, `CHANGELOG.md`
|
||
|
||
## Notes
|
||
|
||
- **Partition injection mechanism.** The ROADMAP spec allows iptables/pfctl *or*
|
||
proxy-based injection ("toxiproxy or similar"). We implement an in-harness TCP relay
|
||
proxy: peers connect through real TCP relays the test controls; severing a relay is a
|
||
real network partition (connections die, gRPC ships fail, circuit breaker opens) with no
|
||
root privileges and identical behavior on macOS/Linux. iptables/pfctl require root and
|
||
are not portable to dev machines; the proxy approach is the spec-sanctioned alternative.
|
||
- **Write-durability contract unchanged.** 204 = leader durability (storage + WAL fsync).
|
||
A quorum-ack write contract is NOT in this phase's ROADMAP acceptance criteria; the
|
||
runbook §8 text that loosely tied it to m8p10 is updated in task 07 to track it as a
|
||
post-M8 follow-up. Scope decision recorded here so it is explicit, not silent.
|
||
- **Default read region in multi-process mode is the LOCAL region** (each process serves
|
||
its own replica); `?region=` forwards to the owning process. Single-process mode keeps
|
||
leader-default. Documented in the runbook rewrite.
|
||
- **G3 (Knuth vs Jump Consistent Hash unification) is not a prerequisite**: both routing
|
||
tiers remain internally consistent (engine `ShardRouter` is the single router for
|
||
`/sharded/*` writes and reads). G3 stays open as an m8p9 gap.
|
||
- **Experimental gate retained** for both cluster modes (multi-process is new; quorum-ack
|
||
and automatic failure detection remain future work). The WARN text differentiates the
|
||
modes honestly.
|
||
- Tier-3 suites stay behind the existing `cluster-e2e` feature so the default test run
|
||
stays fast.
|