jx12n 8a0950260f feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites

Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.

2026-06-10 14:07:33 -06:00

3.7 KiB

Raw Permalink Blame History

Task 04: Multi-process harness + UAT steps 1/2/5 + perf assertions

Delivers

tidal-server/tests/cluster_multiproc.rs (#![cfg(feature = "cluster-e2e")]): a harness that spawns one OS process PER REGION (the thing cluster_e2e.rs explicitly cannot do) and the first tranche of tier-3 UAT: replication (step 1), leader-crash failover (step 2), routing-config tenant move (step 5), write forwarding, status aggregation, and the replication/failover performance assertions.

Complexity: L

Dependencies

Task 03 (full cross-process surface).

Technical Design

Harness (`MultiProcCluster`, shared via `tests/support/multiproc.rs` mod for tasks 05–07)

Reuse cluster_e2e.rs patterns: free_addr(), binary resolution + one cargo build, tempdir config, SIGTERM-then-kill Drop. Extend:
- Topology with per-region explicit grpc_addr AND http_addr (free ports), shared by all processes; optional address-rewrite hook so task 05 can interpose proxies.
- Spawn each node: tidal-server cluster --region <name> --listen <http_addr> --topology … --schema … [--data-dir <tmp>/region-N] + TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1; per-node extra env (skew, version tag) via a builder hook.
- kill_hard(idx) (SIGKILL — real crash), restart(idx, env_overrides) (same data-dir), wait_healthy(idx), wait_converged_all(timeout) — polls EVERY follower process's own /cluster/status/local until lag_events == 0 (not just node 0 — the exact gap the old harness documented).
- Generous tier-3 budgets: 60s boot, 30s convergence default.

Tests

mp_uat_step1_replication_under_2s: seed items+embeddings on leader (broadcast), write ≥100 signals to the leader, record per-signal write→follower-applied latency by polling both followers' local status; assert convergence, p99 < 2s, and feed parity: each follower's own /feed (local region) returns the seeded items with scores matching the leader's feed to 1e-6.
mp_uat_step2_leader_crash_failover_under_10s: seed + converge; kill_hard(leader); POST /cluster/promote {eu-west} on a survivor; poll until both survivors' status agree on the new leader; POST /signals (to the OTHER survivor — proves forwarding to the new leader); assert pre-crash data still served on both; assert promote→first-successful-write elapsed < 10s; no data loss (item count + feed parity across survivors).
mp_uat_step5_tenant_routing_flip: tenant reads pinned ?region=us-east while a background read loop runs; flip the pin to ?region=eu-west (routing config only — no data movement; replication already placed the data); assert ZERO failed reads during the flip window and identical results from the new region.
mp_write_forwarding: POST /signals to a follower → 204 → converges everywhere.
mp_status_aggregation: /cluster/status on each of the 3 nodes reports all 3 regions, same leader, reachable: true everywhere.

Perf measurements use Instant wall-clock in-test; report actuals with the assertion so failures are diagnosable (assert!(p99 < 2s, "p99 was {p99:?}")).

Acceptance Criteria

Harness spawns one process per region; convergence verified against EVERY process
UAT step 1 passes: cross-process replication, feed parity 1e-6, p99 lag < 2s
UAT step 2 passes: SIGKILL'd leader, promoted follower, zero data loss, failover < 10s
UAT step 5 passes: routing-config-only region move with zero read downtime
Forwarded writes and aggregated status proven over real processes
Suite runs green via cargo test -p tidal-server --features cluster-e2e --test cluster_multiproc
Default (non-feature) workspace test run unaffected and fast

3.7 KiB Raw Permalink Blame History Unescape Escape