# Task 05: TCP-proxy partition injection + UAT steps 3/4 + reconcile timing

## Delivers

Real network-partition injection for the multi-process cluster via an in-harness TCP
relay proxy (the ROADMAP-sanctioned "toxiproxy or similar" alternative — root-free,
macOS/Linux portable), and the partition-centric tier-3 tests: degraded global query
(UAT step 3), heal + deterministic CRDT reconciliation with no loss/duplication
(UAT step 4), and the reconciliation < 100ms performance assertion. Lives in
`tidal-server/tests/cluster_chaos.rs` (`cluster-e2e` feature), reusing the task-04
harness support module.

## Complexity: L

## Dependencies

Task 04 (harness + support module).

## Technical Design

### `PartitionProxy` (in `tests/support/`)

Plain `std::net::TcpListener` + per-connection pump threads:

```rust
struct PartitionProxy {
    listen: SocketAddr,            // free port the PEERS dial
    target: SocketAddr,            // the region's real grpc/http listener
    severed: Arc<AtomicBool>,
    conns: Arc<Mutex<Vec<TcpStream>>>, // live streams, shutdown() on sever
}
impl PartitionProxy {
    fn start(target: SocketAddr) -> Self;
    fn sever(&self);   // refuse new conns (accept→drop) AND shutdown existing streams
    fn heal_link(&self);
}
```

Severing kills established TCP streams (both directions) and refuses new connects —
the leader's in-flight `ShipSegment` fails for real, the circuit breaker opens, HTTP
fan-outs time out. This is a genuine network-layer partition between OS processes.

### Topology wiring

Harness option `with_proxies(region)`: the isolated region's `grpc_addr` and `http_addr`
as PUBLISHED IN THE TOPOLOGY are proxy listen ports; the process itself binds the real
targets. The test client talks to the region's REAL http addr directly (operator's
console survives the partition; only peer traffic is severed) — matching the runbook
partition drill's "read the stale follower" step.

### Tests

- `mp_uat_step3_degraded_query_during_partition`: 3 nodes, ap-south proxied. Seed +
  converge. `sever(ap-south)`. Writes to leader keep returning 204 (best-effort ship —
  the write-durability contract); ap-south's local `applied` stalls while leader
  `last_seq` grows (lag climbs, observed via direct addrs); aggregated `/cluster/status`
  on the leader shows ap-south `reachable: false`/worst-case lag. `GET /sharded/feed`
  (deadline 1000ms) on the leader → 200, `degraded: true`, `unavailable_shards ==
  ["ap-south"]`, results still contain live-shard items. Direct read of ap-south's own
  `/feed` serves the PRE-partition data (eventual consistency demonstrated).
- `mp_uat_step4_heal_reconcile_no_loss_no_dup`: continue from a partition with N writes
  during the window plus a `POST /hardnegs` hide recorded on the leader for a seeded
  user/item. `heal_link()` then `POST /cluster/heal {ap-south}` on the leader →
  `wait_converged_all`; assert ap-south applied == leader last_seq (no loss) and decay
  parity 1e-6 (no duplication); call `/cluster/heal` AGAIN → scores unchanged
  (idempotent redelivery). Then `POST /cluster/reconcile {region: ap-south}` on the
  leader → hide present on ap-south (hard negative converged, "hides remain hidden":
  the hidden item no longer appears in ap-south's `/feed?user_id=…` for that user);
  assert `local_elapsed_ms < 100 && remote_elapsed_ms < 100`; repeat reconcile → no-op,
  state identical (determinism/idempotence).
- `mp_partition_between_followers`: ROADMAP criterion verbatim — proxy BOTH followers,
  sever the eu-west↔ap-south direction only (each follower's proxy severed for nothing
  but… follower↔follower links carry no replication traffic in the leader-ships
  topology, so the meaningful assertion is): writes continue on the leader and BOTH
  followers converge (their leader links are intact) — i.e. a follower↔follower
  partition is invisible to correctness. Then additionally sever leader→ap-south to show
  the contrast (lag climbs only then). This makes the "partition between two followers:
  writes continue on leader; heal restores convergence with no data loss" criterion an
  explicit, honest test.

Tier-3 timing: every wait bounded with diagnosable timeout messages; suite total budget
documented in the file header.

## Acceptance Criteria

- [ ] Partition is a real TCP-level severance between OS processes (no engine flags
      involved in the chaos path)
- [ ] UAT step 3 passes: degraded scatter-gather (`degraded: true`, named shard), leader
      writes continue, stale follower serves
- [ ] UAT step 4 passes: heal → zero loss, zero duplication (parity 1e-6, idempotent
      re-heal), reconcile converges hard negatives ("hides remain hidden" via /feed),
      reconcile merge+apply < 100ms both sides, idempotent on repeat
- [ ] Follower↔follower partition criterion covered explicitly
- [ ] Suite green via `cargo test -p tidal-server --features cluster-e2e --test cluster_chaos`