# Task 06: Clock-skew and rolling-upgrade scenarios

## Delivers

The remaining two ROADMAP chaos scenarios, in the same `cluster_chaos.rs` suite (or a
sibling `cluster_lifecycle.rs` if size warrants — implementer's call, same feature gate):
HLC clock-skew with ±500ms across processes, and a 3-node rolling upgrade with a
continuously-written mixed-version window.

## Complexity: M

## Dependencies

Task 04 (harness restart/env hooks); task 01 (HLC offset, `TIDAL_HLC_SKEW_MS`).

## Technical Design

### Clock skew (`mp_clock_skew_reconciliation_stays_causal`)

Spawn 3 nodes with `TIDAL_HLC_SKEW_MS` = `+500`, `0`, `-500` (each process's HLC
genuinely runs skewed — the engine offset from task 01, no mocks). Scenario:

1. Seed + converge; signals replicate normally (decay timestamps are `Timestamp::now()`,
   unaffected by HLC skew — assert decay parity 1e-6 still holds; this pins the scope of
   the skew mechanism).
2. Divergence: hide a (user,item) on the skewed-BEHIND node, and (after partitioning it
   via the task-05 proxy) hide a different pair on the skewed-AHEAD leader; heal.
3. `/cluster/reconcile` in BOTH directions; assert: both nodes converge to the identical
   hard-negative state (LWW is deterministic despite 1s of relative skew — HLC `update()`
   advances past remote timestamps, `max(wall, last_seen+1)`); repeated reconciles are
   no-ops (stability ⇒ no timestamp oscillation); skewed-behind node's hides are not
   lost (its register still wins where it is the only writer — hide semantics intact).
4. Assert reconcile responses stay < 100ms (skew must not break the perf envelope).

### Rolling upgrade (`mp_rolling_upgrade_no_loss_no_stall`)

ROADMAP mechanism: "simulate by restarting with different config." All nodes run
persistent `--data-dir`s.

1. Boot 3 nodes ("version N": env `TIDAL_VERSION_TAG=N` — informational; config has
   `write_workers: 2`).
2. Start a background writer: continuous `POST /signals` round-robin against ALL nodes
   (forwarding finds the leader), tallying successes/failures.
3. Upgrade followers first, one at a time: SIGTERM (graceful), restart same data-dir with
   "version N+1" (env tag + `write_workers: 4`), `wait_healthy`, `POST /cluster/heal` for
   the restarted region (covers segments shipped while it was down — its WAL recovery
   plus heal close the gap), `wait_converged_all`.
4. Leader last: `POST /cluster/promote` to an already-upgraded follower, then restart the
   old leader the same way — the real rolling-upgrade choreography.
5. Stop the writer. Assert: zero write errors during the whole window (a request window
   overlapping a node's restart may retry once — forwarding to the CURRENT leader must
   make every logical write eventually succeed; tally semantics defined in-test);
   final convergence on all 3 (counts equal, decay parity 1e-6 — restarted nodes
   recovered their pre-restart state from WAL, no loss); lag returns to 0 (no stall);
   leader is the promoted node.

Engine note: `replication::upgrade::RollingUpgradeCoordinator` exists from earlier M8
work — the implementer reads it first and reuses/asserts through it where it fits the
server surface; if it is engine-internal version-gating only, note that in the test
header rather than forcing it in.

## Acceptance Criteria

- [ ] Skew test: processes genuinely run with ±500ms HLC offsets; reconciliation
      deterministic and stable both directions; hides intact; decay unaffected;
      reconcile < 100ms under skew
- [ ] Rolling upgrade: continuous writes across the full 3-node restart choreography
      with zero lost logical writes, zero corruption (WAL recovery proven by post-restart
      parity), zero replication stall, promote-before-leader-restart exercised
- [ ] Both suites green under `--features cluster-e2e`