8a0950260f
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.
95 lines
4.9 KiB
Markdown
95 lines
4.9 KiB
Markdown
# Task 05: TCP-proxy partition injection + UAT steps 3/4 + reconcile timing
|
|
|
|
## Delivers
|
|
|
|
Real network-partition injection for the multi-process cluster via an in-harness TCP
|
|
relay proxy (the ROADMAP-sanctioned "toxiproxy or similar" alternative — root-free,
|
|
macOS/Linux portable), and the partition-centric tier-3 tests: degraded global query
|
|
(UAT step 3), heal + deterministic CRDT reconciliation with no loss/duplication
|
|
(UAT step 4), and the reconciliation < 100ms performance assertion. Lives in
|
|
`tidal-server/tests/cluster_chaos.rs` (`cluster-e2e` feature), reusing the task-04
|
|
harness support module.
|
|
|
|
## Complexity: L
|
|
|
|
## Dependencies
|
|
|
|
Task 04 (harness + support module).
|
|
|
|
## Technical Design
|
|
|
|
### `PartitionProxy` (in `tests/support/`)
|
|
|
|
Plain `std::net::TcpListener` + per-connection pump threads:
|
|
|
|
```rust
|
|
struct PartitionProxy {
|
|
listen: SocketAddr, // free port the PEERS dial
|
|
target: SocketAddr, // the region's real grpc/http listener
|
|
severed: Arc<AtomicBool>,
|
|
conns: Arc<Mutex<Vec<TcpStream>>>, // live streams, shutdown() on sever
|
|
}
|
|
impl PartitionProxy {
|
|
fn start(target: SocketAddr) -> Self;
|
|
fn sever(&self); // refuse new conns (accept→drop) AND shutdown existing streams
|
|
fn heal_link(&self);
|
|
}
|
|
```
|
|
|
|
Severing kills established TCP streams (both directions) and refuses new connects —
|
|
the leader's in-flight `ShipSegment` fails for real, the circuit breaker opens, HTTP
|
|
fan-outs time out. This is a genuine network-layer partition between OS processes.
|
|
|
|
### Topology wiring
|
|
|
|
Harness option `with_proxies(region)`: the isolated region's `grpc_addr` and `http_addr`
|
|
as PUBLISHED IN THE TOPOLOGY are proxy listen ports; the process itself binds the real
|
|
targets. The test client talks to the region's REAL http addr directly (operator's
|
|
console survives the partition; only peer traffic is severed) — matching the runbook
|
|
partition drill's "read the stale follower" step.
|
|
|
|
### Tests
|
|
|
|
- `mp_uat_step3_degraded_query_during_partition`: 3 nodes, ap-south proxied. Seed +
|
|
converge. `sever(ap-south)`. Writes to leader keep returning 204 (best-effort ship —
|
|
the write-durability contract); ap-south's local `applied` stalls while leader
|
|
`last_seq` grows (lag climbs, observed via direct addrs); aggregated `/cluster/status`
|
|
on the leader shows ap-south `reachable: false`/worst-case lag. `GET /sharded/feed`
|
|
(deadline 1000ms) on the leader → 200, `degraded: true`, `unavailable_shards ==
|
|
["ap-south"]`, results still contain live-shard items. Direct read of ap-south's own
|
|
`/feed` serves the PRE-partition data (eventual consistency demonstrated).
|
|
- `mp_uat_step4_heal_reconcile_no_loss_no_dup`: continue from a partition with N writes
|
|
during the window plus a `POST /hardnegs` hide recorded on the leader for a seeded
|
|
user/item. `heal_link()` then `POST /cluster/heal {ap-south}` on the leader →
|
|
`wait_converged_all`; assert ap-south applied == leader last_seq (no loss) and decay
|
|
parity 1e-6 (no duplication); call `/cluster/heal` AGAIN → scores unchanged
|
|
(idempotent redelivery). Then `POST /cluster/reconcile {region: ap-south}` on the
|
|
leader → hide present on ap-south (hard negative converged, "hides remain hidden":
|
|
the hidden item no longer appears in ap-south's `/feed?user_id=…` for that user);
|
|
assert `local_elapsed_ms < 100 && remote_elapsed_ms < 100`; repeat reconcile → no-op,
|
|
state identical (determinism/idempotence).
|
|
- `mp_partition_between_followers`: ROADMAP criterion verbatim — proxy BOTH followers,
|
|
sever the eu-west↔ap-south direction only (each follower's proxy severed for nothing
|
|
but… follower↔follower links carry no replication traffic in the leader-ships
|
|
topology, so the meaningful assertion is): writes continue on the leader and BOTH
|
|
followers converge (their leader links are intact) — i.e. a follower↔follower
|
|
partition is invisible to correctness. Then additionally sever leader→ap-south to show
|
|
the contrast (lag climbs only then). This makes the "partition between two followers:
|
|
writes continue on leader; heal restores convergence with no data loss" criterion an
|
|
explicit, honest test.
|
|
|
|
Tier-3 timing: every wait bounded with diagnosable timeout messages; suite total budget
|
|
documented in the file header.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Partition is a real TCP-level severance between OS processes (no engine flags
|
|
involved in the chaos path)
|
|
- [ ] UAT step 3 passes: degraded scatter-gather (`degraded: true`, named shard), leader
|
|
writes continue, stale follower serves
|
|
- [ ] UAT step 4 passes: heal → zero loss, zero duplication (parity 1e-6, idempotent
|
|
re-heal), reconcile converges hard negatives ("hides remain hidden" via /feed),
|
|
reconcile merge+apply < 100ms both sides, idempotent on repeat
|
|
- [ ] Follower↔follower partition criterion covered explicitly
|
|
- [ ] Suite green via `cargo test -p tidal-server --features cluster-e2e --test cluster_chaos`
|