# m8p6: End-to-End UAT

## Delivers

A comprehensive end-to-end test suite that exercises the complete UAT scenario:
3 regions, 5 shards per region, 25K signals/sec, network partition, failover,
partition heal, deterministic reconciliation, and tenant migration. This is the
gate for M8 completion.

Deliverables:
- `m8_uat` integration test suite matching all 5 UAT scenario steps
- `SimulatedCluster`: test harness that creates a multi-region tidalDB cluster using `InProcessTransport`
- `NetworkPartition`: injectable fault that blocks `Transport::send_segment` between specified regions
- `ShardCrash`: injectable fault that drops a shard primary and triggers follower promotion
- Performance assertions: cross-region replication < 2s p99, failover < 10s

## Dependencies

- **Requires:** All phases 8.1-8.5
- **Files created:**
  - `tidal/tests/m8_uat.rs` -- integration test suite
  - `tidal/src/testing/cluster.rs` -- `SimulatedCluster` harness
  - `tidal/src/testing/faults.rs` -- `NetworkPartition`, `ShardCrash` fault injection

## Research References

- `docs/research/tidaldb_wal.md` -- invariant checklist for replication correctness

## Acceptance Criteria (Phase Level)

- [ ] **UAT Step 1:** Write signals for a user in us-east, read in eu-west after < 2 seconds; verified by `ReplicationLagGauge` assertion and `read_decay_score` equivalence
- [ ] **UAT Step 2:** Crash an entire shard primary (simulated); follower is promoted within 10 seconds; all acknowledged signals are present on the promoted follower; no data loss
- [ ] **UAT Step 3:** Execute `RETRIEVE items COHORT locale:EU` while ap-south is partitioned; query succeeds using available shards; results include items from non-partitioned regions only; degradation flag set in `QueryStats`
- [ ] **UAT Step 4:** Heal the partition; `ReconciliationEngine` runs; after reconciliation: no duplicate signal counts (verified by sum of all events across all regions); hard negatives never leaked; decay scores on all shards match analytical formula to 6 decimal places
- [ ] **UAT Step 5:** Move a tenant to a new region by changing routing config; during migration: zero downtime, all queries succeed; after migration: tenant's data is on new region only; old region's copy is GC'd
- [ ] Invariant: no signal event is lost or double-counted across the entire test run (verified by WAL event count == materialized signal count on all shards)
- [ ] Invariant: hard negatives (hide/mute/block) are monotonically enforced -- once hidden, never visible during convergence

## Task Execution Order

```
Task 01: SimulatedCluster Harness ──────┐
                                         ├──> Task 03: UAT Scenario Tests (Steps 1-5)
Task 02: Fault Injection ────────────────┘         │
                                                    v
                                          Task 04: Performance Assertions + CI
```

Tasks 01 and 02 are parallelizable. Task 03 depends on both. Task 04 depends on 03.

## Module Location

| File | Status | Contains |
|------|--------|----------|
| `tidal/tests/m8_uat.rs` | NEW | All UAT scenario tests |
| `tidal/src/testing/cluster.rs` | NEW | `SimulatedCluster` harness |
| `tidal/src/testing/faults.rs` | NEW | `NetworkPartition`, `ShardCrash` fault injection |

## Notes

### All tests use InProcessTransport

No actual network. The `NetworkPartition` fault works by intercepting `send_segment` calls and dropping them for the specified region pair.

### Deterministic reconciliation verification

After partition heal, we replay all WAL segments from both sides of the partition through a single-node `TidalDb` (the ground truth). We then compare every signal count and decay score on every shard against this ground truth. Any divergence fails the test.

### Performance assertions are soft

The 2s p99 target is for in-process transport. Real network latency is additive. The test verifies that replication logic itself adds < 100ms overhead; the remaining budget is for network RTT.

## Done When

`cargo test --test m8_uat` passes all 5 UAT scenario steps with 25K signals/sec sustained throughput across 3 simulated regions, verifying no signal loss, no duplicate counts, no leaked hard negatives, and correct decay scores after partition heal and reconciliation.