8a0950260f
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.
71 lines
3.7 KiB
Markdown
71 lines
3.7 KiB
Markdown
# Task 06: Clock-skew and rolling-upgrade scenarios
|
|
|
|
## Delivers
|
|
|
|
The remaining two ROADMAP chaos scenarios, in the same `cluster_chaos.rs` suite (or a
|
|
sibling `cluster_lifecycle.rs` if size warrants — implementer's call, same feature gate):
|
|
HLC clock-skew with ±500ms across processes, and a 3-node rolling upgrade with a
|
|
continuously-written mixed-version window.
|
|
|
|
## Complexity: M
|
|
|
|
## Dependencies
|
|
|
|
Task 04 (harness restart/env hooks); task 01 (HLC offset, `TIDAL_HLC_SKEW_MS`).
|
|
|
|
## Technical Design
|
|
|
|
### Clock skew (`mp_clock_skew_reconciliation_stays_causal`)
|
|
|
|
Spawn 3 nodes with `TIDAL_HLC_SKEW_MS` = `+500`, `0`, `-500` (each process's HLC
|
|
genuinely runs skewed — the engine offset from task 01, no mocks). Scenario:
|
|
|
|
1. Seed + converge; signals replicate normally (decay timestamps are `Timestamp::now()`,
|
|
unaffected by HLC skew — assert decay parity 1e-6 still holds; this pins the scope of
|
|
the skew mechanism).
|
|
2. Divergence: hide a (user,item) on the skewed-BEHIND node, and (after partitioning it
|
|
via the task-05 proxy) hide a different pair on the skewed-AHEAD leader; heal.
|
|
3. `/cluster/reconcile` in BOTH directions; assert: both nodes converge to the identical
|
|
hard-negative state (LWW is deterministic despite 1s of relative skew — HLC `update()`
|
|
advances past remote timestamps, `max(wall, last_seen+1)`); repeated reconciles are
|
|
no-ops (stability ⇒ no timestamp oscillation); skewed-behind node's hides are not
|
|
lost (its register still wins where it is the only writer — hide semantics intact).
|
|
4. Assert reconcile responses stay < 100ms (skew must not break the perf envelope).
|
|
|
|
### Rolling upgrade (`mp_rolling_upgrade_no_loss_no_stall`)
|
|
|
|
ROADMAP mechanism: "simulate by restarting with different config." All nodes run
|
|
persistent `--data-dir`s.
|
|
|
|
1. Boot 3 nodes ("version N": env `TIDAL_VERSION_TAG=N` — informational; config has
|
|
`write_workers: 2`).
|
|
2. Start a background writer: continuous `POST /signals` round-robin against ALL nodes
|
|
(forwarding finds the leader), tallying successes/failures.
|
|
3. Upgrade followers first, one at a time: SIGTERM (graceful), restart same data-dir with
|
|
"version N+1" (env tag + `write_workers: 4`), `wait_healthy`, `POST /cluster/heal` for
|
|
the restarted region (covers segments shipped while it was down — its WAL recovery
|
|
plus heal close the gap), `wait_converged_all`.
|
|
4. Leader last: `POST /cluster/promote` to an already-upgraded follower, then restart the
|
|
old leader the same way — the real rolling-upgrade choreography.
|
|
5. Stop the writer. Assert: zero write errors during the whole window (a request window
|
|
overlapping a node's restart may retry once — forwarding to the CURRENT leader must
|
|
make every logical write eventually succeed; tally semantics defined in-test);
|
|
final convergence on all 3 (counts equal, decay parity 1e-6 — restarted nodes
|
|
recovered their pre-restart state from WAL, no loss); lag returns to 0 (no stall);
|
|
leader is the promoted node.
|
|
|
|
Engine note: `replication::upgrade::RollingUpgradeCoordinator` exists from earlier M8
|
|
work — the implementer reads it first and reuses/asserts through it where it fits the
|
|
server surface; if it is engine-internal version-gating only, note that in the test
|
|
header rather than forcing it in.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Skew test: processes genuinely run with ±500ms HLC offsets; reconciliation
|
|
deterministic and stable both directions; hides intact; decay unaffected;
|
|
reconcile < 100ms under skew
|
|
- [ ] Rolling upgrade: continuous writes across the full 3-node restart choreography
|
|
with zero lost logical writes, zero corruption (WAL recovery proven by post-restart
|
|
parity), zero replication stall, promote-before-leader-restart exercised
|
|
- [ ] Both suites green under `--features cluster-e2e`
|