tidaldb/docs/planning/milestone-8/phase-10/task-06-skew-rolling-upgrade.md

Task 06: Clock-skew and rolling-upgrade scenarios

Delivers

The remaining two ROADMAP chaos scenarios, in the same cluster_chaos.rs suite (or a sibling cluster_lifecycle.rs if size warrants — implementer's call, same feature gate): HLC clock-skew with ±500ms across processes, and a 3-node rolling upgrade with a continuously-written mixed-version window.

Complexity: M

Dependencies

Task 04 (harness restart/env hooks); task 01 (HLC offset, TIDAL_HLC_SKEW_MS).

Technical Design

Clock skew (mp_clock_skew_reconciliation_stays_causal)

Spawn 3 nodes with TIDAL_HLC_SKEW_MS = +500, 0, -500 (each process's HLC genuinely runs skewed — the engine offset from task 01, no mocks). Scenario:

1. Seed + converge; signals replicate normally (decay timestamps are Timestamp::now(), unaffected by HLC skew — assert decay parity 1e-6 still holds; this pins the scope of the skew mechanism).
2. Divergence: hide a (user,item) on the skewed-BEHIND node, and (after partitioning it via the task-05 proxy) hide a different pair on the skewed-AHEAD leader; heal.
3. /cluster/reconcile in BOTH directions; assert: both nodes converge to the identical hard-negative state (LWW is deterministic despite 1s of relative skew — HLC update() advances past remote timestamps, max(wall, last_seen+1)); repeated reconciles are no-ops (stability ⇒ no timestamp oscillation); skewed-behind node's hides are not lost (its register still wins where it is the only writer — hide semantics intact).
4. Assert reconcile responses stay < 100ms (skew must not break the perf envelope).

Rolling upgrade (mp_rolling_upgrade_no_loss_no_stall)

ROADMAP mechanism: "simulate by restarting with different config." All nodes run persistent --data-dirs.

1. Boot 3 nodes ("version N": env TIDAL_VERSION_TAG=N — informational; config has write_workers: 2).
2. Start a background writer: continuous POST /signals round-robin against ALL nodes (forwarding finds the leader), tallying successes/failures.
3. Upgrade followers first, one at a time: SIGTERM (graceful), restart same data-dir with "version N+1" (env tag + write_workers: 4), wait_healthy, POST /cluster/heal for the restarted region (covers segments shipped while it was down — its WAL recovery plus heal close the gap), wait_converged_all.
4. Leader last: POST /cluster/promote to an already-upgraded follower, then restart the old leader the same way — the real rolling-upgrade choreography.
5. Stop the writer. Assert: zero write errors during the whole window (a request window overlapping a node's restart may retry once — forwarding to the CURRENT leader must make every logical write eventually succeed; tally semantics defined in-test); final convergence on all 3 (counts equal, decay parity 1e-6 — restarted nodes recovered their pre-restart state from WAL, no loss); lag returns to 0 (no stall); leader is the promoted node.

Engine note: replication::upgrade::RollingUpgradeCoordinator exists from earlier M8 work — the implementer reads it first and reuses/asserts through it where it fits the server surface; if it is engine-internal version-gating only, note that in the test header rather than forcing it in.

Acceptance Criteria

• Skew test: processes genuinely run with ±500ms HLC offsets; reconciliation deterministic and stable both directions; hides intact; decay unaffected; reconcile < 100ms under skew
• Rolling upgrade: continuous writes across the full 3-node restart choreography with zero lost logical writes, zero corruption (WAL recovery proven by post-restart parity), zero replication stall, promote-before-leader-restart exercised
• Both suites green under --features cluster-e2e