tidaldb/docs/planning/milestone-8/phase-10/task-05-partition-injection.md

Task 05: TCP-proxy partition injection + UAT steps 3/4 + reconcile timing

Delivers

Real network-partition injection for the multi-process cluster via an in-harness TCP relay proxy (the ROADMAP-sanctioned "toxiproxy or similar" alternative — root-free, macOS/Linux portable), and the partition-centric tier-3 tests: degraded global query (UAT step 3), heal + deterministic CRDT reconciliation with no loss/duplication (UAT step 4), and the reconciliation < 100ms performance assertion. Lives in tidal-server/tests/cluster_chaos.rs (cluster-e2e feature), reusing the task-04 harness support module.

Complexity: L

Dependencies

Task 04 (harness + support module).

Technical Design

PartitionProxy (in tests/support/)

Plain std::net::TcpListener + per-connection pump threads:

struct PartitionProxy {
    listen: SocketAddr,            // free port the PEERS dial
    target: SocketAddr,            // the region's real grpc/http listener
    severed: Arc<AtomicBool>,
    conns: Arc<Mutex<Vec<TcpStream>>>, // live streams, shutdown() on sever
}
impl PartitionProxy {
    fn start(target: SocketAddr) -> Self;
    fn sever(&self);   // refuse new conns (accept→drop) AND shutdown existing streams
    fn heal_link(&self);
}

Severing kills established TCP streams (both directions) and refuses new connects — the leader's in-flight ShipSegment fails for real, the circuit breaker opens, HTTP fan-outs time out. This is a genuine network-layer partition between OS processes.

Topology wiring

Harness option with_proxies(region): the isolated region's grpc_addr and http_addr as PUBLISHED IN THE TOPOLOGY are proxy listen ports; the process itself binds the real targets. The test client talks to the region's REAL http addr directly (operator's console survives the partition; only peer traffic is severed) — matching the runbook partition drill's "read the stale follower" step.

Tests

• mp_uat_step3_degraded_query_during_partition: 3 nodes, ap-south proxied. Seed + converge. sever(ap-south). Writes to leader keep returning 204 (best-effort ship — the write-durability contract); ap-south's local applied stalls while leader last_seq grows (lag climbs, observed via direct addrs); aggregated /cluster/status on the leader shows ap-south reachable: false/worst-case lag. GET /sharded/feed (deadline 1000ms) on the leader → 200, degraded: true, unavailable_shards == ["ap-south"], results still contain live-shard items. Direct read of ap-south's own /feed serves the PRE-partition data (eventual consistency demonstrated).
• mp_uat_step4_heal_reconcile_no_loss_no_dup: continue from a partition with N writes during the window plus a POST /hardnegs hide recorded on the leader for a seeded user/item. heal_link() then POST /cluster/heal {ap-south} on the leader → wait_converged_all; assert ap-south applied == leader last_seq (no loss) and decay parity 1e-6 (no duplication); call /cluster/heal AGAIN → scores unchanged (idempotent redelivery). Then POST /cluster/reconcile {region: ap-south} on the leader → hide present on ap-south (hard negative converged, "hides remain hidden": the hidden item no longer appears in ap-south's /feed?user_id=… for that user); assert local_elapsed_ms < 100 && remote_elapsed_ms < 100; repeat reconcile → no-op, state identical (determinism/idempotence).
• mp_partition_between_followers: ROADMAP criterion verbatim — proxy BOTH followers, sever the eu-west↔ap-south direction only (each follower's proxy severed for nothing but… follower↔follower links carry no replication traffic in the leader-ships topology, so the meaningful assertion is): writes continue on the leader and BOTH followers converge (their leader links are intact) — i.e. a follower↔follower partition is invisible to correctness. Then additionally sever leader→ap-south to show the contrast (lag climbs only then). This makes the "partition between two followers: writes continue on leader; heal restores convergence with no data loss" criterion an explicit, honest test.

Tier-3 timing: every wait bounded with diagnosable timeout messages; suite total budget documented in the file header.

Acceptance Criteria

• Partition is a real TCP-level severance between OS processes (no engine flags involved in the chaos path)
• UAT step 3 passes: degraded scatter-gather (degraded: true, named shard), leader writes continue, stale follower serves
• UAT step 4 passes: heal → zero loss, zero duplication (parity 1e-6, idempotent re-heal), reconcile converges hard negatives ("hides remain hidden" via /feed), reconcile merge+apply < 100ms both sides, idempotent on repeat
• Follower↔follower partition criterion covered explicitly
• Suite green via cargo test -p tidal-server --features cluster-e2e --test cluster_chaos