jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
tidaldb/docs/planning/milestone-8/phase-10/OVERVIEW.md
jx12n 8a0950260f feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites 
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
2026-06-10 14:07:33 -06:00

5.2 KiB
Raw Permalink Blame History

Phase 10: Multi-Node UAT (m8p10)

Delivers

True multi-process cluster mode — each tidal-server cluster --region <name> process owns exactly ONE region and peers with sibling processes over real gRPC — plus the tier-3 chaos suite that the ROADMAP defines for this phase: multi-process UAT of all 5 original M8 UAT scenario steps, real network-partition injection, clock-skew simulation, rolling upgrade, cluster-runbook verification, and the performance assertions (replication < 2s p99, failover < 10s, reconciliation < 100ms over localhost).

The single-process cluster mode (m8p8) is retained unchanged as the dev/demo default; the multi-process mode is additive, selected by the new --region flag.

Dependencies

  • m8p7 (gRPC transport, tidal-net) — COMPLETE
  • m8p8 (multi-node tidal-server, cluster routes) — COMPLETE
  • m8p9 (cross-node scatter-gather queries) — COMPLETE
  • Engine primitives: Transport trait, spawn_receiver/start_replication, ReplicationState, ReplicationLagGauge, ReconciliationEngine/StateSnapshot, Hlc, encode_batch, redeliver_missed invariants — all shipped (m8p1–p6)

Research References

  • docs/specs/14-scale-architecture.md §7 (query routing, deadlines), §9 (replication SLA)
  • docs/planning/ROADMAP.md "Phase 10: Multi-Node UAT (m8p10)" — the authoritative spec
  • docs/planning/ROADMAP.md Milestone 8 "UAT Scenario" (the original 5 steps)
  • docs/runbooks/cluster.md — the operations that must verify against the real cluster
  • tidal/src/testing/cluster.rs + cluster_transport.rs — the load-bearing relay invariants (seqno atomicity/rollback, single_event_payload, redeliver_missed destination contract) that the production relay must inherit, not duplicate

Acceptance Criteria (Phase Level)

From ROADMAP Phase 10 (verbatim):

  • 3-process cluster starts, seeds data, and passes all 5 original M8 UAT scenario steps over gRPC
  • Network partition between two followers: writes continue on leader; heal restores convergence with no data loss
  • Rolling upgrade: mixed-version window produces no errors or data corruption
  • Every cluster runbook operation works against the real cluster
  • Performance: replication < 2s, failover < 10s, reconciliation < 100ms (all over localhost)

Task Execution Order

Task Title Complexity Depends on
01 Engine relay module, HLC clock offset, StateSnapshot wire serde L —
02 Single-region cluster mode (--region, RegionClusterState) XL 01
03 Cross-process routes: forwarding, promote fan-out, status aggregation, reconcile, sharded XL 02
04 Multi-process harness + UAT steps 1/2/5 + perf assertions L 03
05 TCP-proxy partition injection + UAT steps 3/4 + reconcile timing L 04
06 Clock-skew and rolling-upgrade scenarios M 04
07 Runbook-verification suite + docs/ROADMAP/CHANGELOG updates M 04, 05, 06

Module Location

  • Engine: tidal/src/replication/relay.rs (new), tidal/src/replication/crdt/hlc.rs, tidal/src/replication/reconcile.rs, tidal/src/db/builder.rs, tidal/src/db/replication_ops.rs
  • Server: tidal-server/src/cluster/ (split from cluster.rs per the tracked Maintainability-S follow-up: topology.rs, transport.rs, state.rs, node.rs, routes.rs, forward.rs)
  • Tests: tidal-server/tests/cluster_multiproc.rs, cluster_chaos.rs, cluster_runbook.rs (all #![cfg(feature = "cluster-e2e")])
  • Docs: docs/runbooks/cluster.md, docs/runbooks/kubernetes.md, docs/planning/ROADMAP.md, CHANGELOG.md

Notes

  • Partition injection mechanism. The ROADMAP spec allows iptables/pfctl or proxy-based injection ("toxiproxy or similar"). We implement an in-harness TCP relay proxy: peers connect through real TCP relays the test controls; severing a relay is a real network partition (connections die, gRPC ships fail, circuit breaker opens) with no root privileges and identical behavior on macOS/Linux. iptables/pfctl require root and are not portable to dev machines; the proxy approach is the spec-sanctioned alternative.
  • Write-durability contract unchanged. 204 = leader durability (storage + WAL fsync). A quorum-ack write contract is NOT in this phase's ROADMAP acceptance criteria; the runbook §8 text that loosely tied it to m8p10 is updated in task 07 to track it as a post-M8 follow-up. Scope decision recorded here so it is explicit, not silent.
  • Default read region in multi-process mode is the LOCAL region (each process serves its own replica); ?region= forwards to the owning process. Single-process mode keeps leader-default. Documented in the runbook rewrite.
  • G3 (Knuth vs Jump Consistent Hash unification) is not a prerequisite: both routing tiers remain internally consistent (engine ShardRouter is the single router for /sharded/* writes and reads). G3 stays open as an m8p9 gap.
  • Experimental gate retained for both cluster modes (multi-process is new; quorum-ack and automatic failure detection remain future work). The WARN text differentiates the modes honestly.
  • Tier-3 suites stay behind the existing cluster-e2e feature so the default test run stays fast.