# Task 04: Multi-process harness + UAT steps 1/2/5 + perf assertions

## Delivers

`tidal-server/tests/cluster_multiproc.rs` (`#![cfg(feature = "cluster-e2e")]`): a
harness that spawns one OS process PER REGION (the thing `cluster_e2e.rs` explicitly
cannot do) and the first tranche of tier-3 UAT: replication (step 1), leader-crash
failover (step 2), routing-config tenant move (step 5), write forwarding, status
aggregation, and the replication/failover performance assertions.

## Complexity: L

## Dependencies

Task 03 (full cross-process surface).

## Technical Design

### Harness (`MultiProcCluster`, shared via `tests/support/multiproc.rs` mod for tasks 05–07)

- Reuse `cluster_e2e.rs` patterns: `free_addr()`, binary resolution + one `cargo build`,
  tempdir config, SIGTERM-then-kill Drop. Extend:
  - Topology with per-region explicit `grpc_addr` AND `http_addr` (free ports), shared by
    all processes; optional address-rewrite hook so task 05 can interpose proxies.
  - Spawn each node: `tidal-server cluster --region <name> --listen <http_addr>
    --topology … --schema … [--data-dir <tmp>/region-N]` +
    `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`; per-node extra env (skew, version tag) via a
    builder hook.
  - `kill_hard(idx)` (SIGKILL — real crash), `restart(idx, env_overrides)` (same
    data-dir), `wait_healthy(idx)`, `wait_converged_all(timeout)` — polls EVERY follower
    process's own `/cluster/status/local` until `lag_events == 0` (not just node 0 — the
    exact gap the old harness documented).
  - Generous tier-3 budgets: 60s boot, 30s convergence default.

### Tests

- `mp_uat_step1_replication_under_2s`: seed items+embeddings on leader (broadcast), write
  ≥100 signals to the leader, record per-signal write→follower-applied latency by polling
  both followers' local status; assert convergence, p99 < 2s, and feed parity: each
  follower's own `/feed` (local region) returns the seeded items with scores matching the
  leader's feed to 1e-6.
- `mp_uat_step2_leader_crash_failover_under_10s`: seed + converge; `kill_hard(leader)`;
  `POST /cluster/promote {eu-west}` on a survivor; poll until both survivors' status
  agree on the new leader; `POST /signals` (to the OTHER survivor — proves forwarding to
  the new leader); assert pre-crash data still served on both; assert
  promote→first-successful-write elapsed < 10s; no data loss (item count + feed parity
  across survivors).
- `mp_uat_step5_tenant_routing_flip`: tenant reads pinned `?region=us-east` while a
  background read loop runs; flip the pin to `?region=eu-west` (routing config only —
  no data movement; replication already placed the data); assert ZERO failed reads
  during the flip window and identical results from the new region.
- `mp_write_forwarding`: `POST /signals` to a follower → 204 → converges everywhere.
- `mp_status_aggregation`: `/cluster/status` on each of the 3 nodes reports all 3 regions,
  same leader, `reachable: true` everywhere.

Perf measurements use `Instant` wall-clock in-test; report actuals with the assertion so
failures are diagnosable (`assert!(p99 < 2s, "p99 was {p99:?}")`).

## Acceptance Criteria

- [ ] Harness spawns one process per region; convergence verified against EVERY process
- [ ] UAT step 1 passes: cross-process replication, feed parity 1e-6, p99 lag < 2s
- [ ] UAT step 2 passes: SIGKILL'd leader, promoted follower, zero data loss,
      failover < 10s
- [ ] UAT step 5 passes: routing-config-only region move with zero read downtime
- [ ] Forwarded writes and aggregated status proven over real processes
- [ ] Suite runs green via
      `cargo test -p tidal-server --features cluster-e2e --test cluster_multiproc`
- [ ] Default (non-feature) workspace test run unaffected and fast