8a0950260f
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.
1043 lines
48 KiB
Rust
1043 lines
48 KiB
Rust
//! Tier-3 MULTI-PROCESS lifecycle chaos suite (m8p10 task 06): clock skew and
|
|
//! rolling upgrade.
|
|
//!
|
|
//! These two scenarios are kept in their OWN file (not `cluster_chaos.rs`, which is
|
|
//! already at a healthy size and owns the partition-injection suite) so each suite
|
|
//! stays independently runnable; both share the same `mod support;` tier-3 harness
|
|
//! and the same `cluster-e2e` feature gate.
|
|
//!
|
|
//! Run:
|
|
//! ```bash
|
|
//! cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle -- --nocapture
|
|
//! ```
|
|
//!
|
|
//! # What each test proves
|
|
//!
|
|
//! ## `mp_clock_skew_reconciliation_stays_causal`
|
|
//!
|
|
//! Three OS processes run with GENUINELY skewed HLCs (`TIDAL_HLC_SKEW_MS` = `+500`,
|
|
//! `0`, `-500`). The env var flows through `main.rs` → `TidalDbBuilder::
|
|
//! with_hlc_offset_ms` → `Hlc::for_shard_with_offset`, which the engine reads ONLY
|
|
//! when stamping hard-negative LWW registers in `take_crdt_snapshot` — signal-decay
|
|
//! timestamps use `Timestamp::now()` and are deliberately UNAFFECTED. The test pins
|
|
//! that scope and the convergence guarantee:
|
|
//!
|
|
//! * **Phase A — normal replication under skew.** Seed + converge; the leader's
|
|
//! `view` signals replicate over the WAL relay and every follower ranks an
|
|
//! IDENTICAL feed to 1e-6. Because decay reads the real wall clock, ±500ms of HLC
|
|
//! skew leaves decayed scores byte-for-byte equal across the three skewed
|
|
//! processes — proof the skew mechanism touches HLC (reconcile LWW) ONLY.
|
|
//! * **Phase B — divergence under a real partition.** We sever the skewed-BEHIND
|
|
//! follower (`-500ms`, ap-south) from every peer with the TCP proxy, then create
|
|
//! NODE-LOCAL divergence honestly (see "Divergence creation" below): the
|
|
//! skewed-AHEAD real leader (`+500ms`, us-east) hides one pair, and the severed
|
|
//! skewed-behind node — promoted to leader in its OWN partitioned view, a genuine
|
|
//! split-brain — hides a DIFFERENT pair. Each pair therefore has exactly ONE
|
|
//! writer.
|
|
//! * **Heal + reconcile BOTH directions.** After healing the link and re-promoting
|
|
//! the real leader (resolving the split brain), `/cluster/reconcile` is driven
|
|
//! from both sides. LWW is deterministic despite 1s of relative skew (HLC
|
|
//! `update()` advances past remote timestamps, `max(wall, last_seen+1)`), so both
|
|
//! nodes converge to the IDENTICAL hard-negative state: each hide is effective on
|
|
//! BOTH nodes (verified by the `/feed?user_id=` control-vs-filtered pattern). A
|
|
//! repeated reconcile leaves the converged STATE unchanged (no timestamp
|
|
//! oscillation; we assert on state, never on `ops_applied`, which is > 0 by
|
|
//! design). Reconcile merge+apply stays < 100ms both sides under skew.
|
|
//!
|
|
//! ### Divergence creation (honest choice)
|
|
//!
|
|
//! The task doc's divergence is "hide on the skewed-BEHIND node and a DIFFERENT pair
|
|
//! on the skewed-AHEAD leader during the window", which needs a NODE-LOCAL hide on
|
|
//! the partitioned follower. `/hardnegs` on a non-leader FORWARDS to the leader; while
|
|
//! severed that forward fails — which is the forwarding contract working, not a bug.
|
|
//! Rather than reach for the `x-tidal-internal` marker (a stand-in), we take the
|
|
//! CLEANER real path the runbook itself sanctions: promote the severed follower to
|
|
//! leader IN ITS OWN VIEW (`/cluster/promote` applies `promote_local` unconditionally;
|
|
//! the fan-out to severed peers fails, leaving a genuine SPLIT BRAIN — two
|
|
//! leader-views during the partition). A plain external `/hardnegs` to that
|
|
//! split-brain leader then applies LOCALLY because `state.is_leader()` is true on its
|
|
//! own view. This is exactly the production hazard reconciliation exists to repair:
|
|
//! two nodes each accepted a divergent write while partitioned. On heal we re-promote
|
|
//! the real leader to collapse the split brain, then reconcile — the durable
|
|
//! `Tag::HardNeg` rows survive the leadership change (they are store rows, not
|
|
//! leader state), so both hides converge by LWW.
|
|
//!
|
|
//! ## `mp_rolling_upgrade_no_loss_no_stall`
|
|
//!
|
|
//! Three processes on PERSISTENT data dirs (the harness always passes `--data-dir`
|
|
//! and reuses it on restart, so a restart recovers pre-restart state from the WAL).
|
|
//! A background writer thread continuously `POST /signals` round-robin across ALL
|
|
//! three live nodes (followers forward to the current leader) while we perform the
|
|
//! full rolling-upgrade choreography:
|
|
//!
|
|
//! * Boot "version N" (env `TIDAL_VERSION_TAG=N`).
|
|
//! * Upgrade followers one at a time UNDER LOAD: graceful SIGTERM (clean drain +
|
|
//! checkpoint + WAL fsync — NOT a crash), restart same data dir with
|
|
//! `TIDAL_VERSION_TAG=N+1`, `wait_healthy`, `heal_until_converged_busy` (the leader's
|
|
//! gRPC circuit breaker opened while the node was down — threshold 5, reset 30s — so
|
|
//! one heal can ship into an open breaker; we re-issue exactly like a runbook
|
|
//! operator, tolerating transient 408/429 from the pool-backed heal route under the
|
|
//! concurrent writer).
|
|
//! * Drain the writer (and read the zero-loss tally).
|
|
//! * Leader LAST: `/cluster/promote` an already-upgraded follower, `wait_leader_agreed`
|
|
//! on the live nodes, then graceful-restart the OLD leader the same way. Because the
|
|
//! old leader's topology file still names region 0 as leader, it boots with a STALE
|
|
//! leader view (a transient split brain) — so we re-promote the chosen leader after
|
|
//! it rejoins and `wait_leader_agreed` on ALL three, the real operator step.
|
|
//! * Heal each restarted node (the runbook recovery) and assert the WAL relay ALONE
|
|
//! reconverges it to the leader's ranking — NO `/cluster/reconcile` in the
|
|
//! convergence path. A final `/cluster/reconcile` round is then asserted to be a
|
|
//! true no-op on scores (the anti-entropy fixpoint, now reached immediately).
|
|
//!
|
|
//! Final asserts: the cluster converges to identical state via the WAL relay alone
|
|
//! (feed/decay parity to 1e-6 across all three — no loss, no duplication), the final
|
|
//! reconcile is an exact no-op (no creep), zero ACKNOWLEDGED logical writes lost
|
|
//! (the spec's hard guarantee, criterion 7: `lost == 0`), the leader is the promoted
|
|
//! node, and each restarted node served its pre-restart items after restart (WAL
|
|
//! recovery presence).
|
|
//!
|
|
//! ### Convergence is driven by the WAL RELAY (three reported bugs, now FIXED)
|
|
//!
|
|
//! The spec (docs/specs/14-scale-architecture.md §"consistency guarantees") makes signal
|
|
//! aggregates EVENTUAL (bounded-staleness), recovered by anti-entropy; the hard
|
|
//! guarantee is "no ACKNOWLEDGED signal events lost" (criterion 7). The m8p10 tier-3 UAT
|
|
//! surfaced three real product bugs that meant the WAL relay alone did NOT deliver the
|
|
//! eventual convergence the model promises after a restart. All three are now FIXED, so
|
|
//! this test asserts the REAL contract: heal + the WAL relay reconverge every restarted
|
|
//! node WITHOUT `/cluster/reconcile`, and the final reconcile is a true no-op:
|
|
//!
|
|
//! 1. **Lag gauge meaningless across a leadership change — FIXED.** The lag gauge now
|
|
//! tracks the leader high-water-mark PER SOURCE SHARD
|
|
//! (`ReplicationLagGauge::leader_seqno_for`), and `local_status` computes lag against
|
|
//! the CURRENT leader's shard. A fully-caught-up rejoined node reports lag 0, so
|
|
//! `converged()` (lag 0 + applied >= leader hwm) is an honest convergence signal.
|
|
//! 2. **`reconcile_with` NOT idempotent for signal state — FIXED.** `take_crdt_snapshot`
|
|
//! now attributes every node's signal contribution to ONE canonical replication shard
|
|
//! (`ShardId::SINGLE`), not the local shard, so a fully-replicated event is ONE LWW
|
|
//! register on every node — `merge` is idempotent. Reconcile of converged nodes is an
|
|
//! exact fixpoint immediately (no 0.5 → 0.375 → … creep). The final reconcile round
|
|
//! below asserts this directly (feeds byte-unchanged).
|
|
//! 3. **Graceful-restart signal loss the WAL relay wouldn't backfill — FIXED.** Items
|
|
//! are HTTP-broadcast (not WAL-relayed), so a node down during a broadcast missed the
|
|
//! items forever; `/cluster/heal` now ALSO re-broadcasts every item's metadata +
|
|
//! embeddings to the healed node (an idempotent upsert), making heal the single
|
|
//! recovery verb. Combined with the durable follower WAL replay (signal state already
|
|
//! survives a restart), heal leaves the node with EXACTLY the leader's data — proven
|
|
//! by the per-restart WAL-relay parity assertion (1e-6, no reconcile).
|
|
//!
|
|
//! ### Write-tally semantics (the zero-loss gate)
|
|
//!
|
|
//! A "logical write" is one intended signal POST. During a restart window a request
|
|
//! may transiently fail (connection refused while the process is down; 503 when a
|
|
//! follower forwards to a leader that is restarting or holds a stale view; 429 under
|
|
//! write-pool backpressure; 408 request timeout). The writer RETRIES the SAME logical
|
|
//! write (cycling target nodes) until it 204s, counting retries; `stop` is checked only
|
|
//! at the top of each logical write so a committed write is always driven to completion.
|
|
//! The gate is `lost == 0`: every logical write eventually returns 204 (the spec's
|
|
//! "no acknowledged events lost"). `total` is the number of logical writes, `retried`
|
|
//! the number that needed >= 1 retry, `lost` the number that never succeeded within
|
|
//! [`MAX_WRITE_ATTEMPTS`] (must be 0).
|
|
//!
|
|
//! ## `RollingUpgradeCoordinator` decision
|
|
//!
|
|
//! `tidaldb::replication::upgrade::RollingUpgradeCoordinator` is an ENGINE-INTERNAL
|
|
//! drain/rejoin gate over `ShardId`s on a single in-process `ControlPlane`: `drain`
|
|
//! marks a shard not-routable (refusing to leave zero serving shards) and `rejoin`
|
|
//! clears it. It is NOT surfaced on the multi-process server's HTTP API — there is no
|
|
//! `/cluster/drain` route, and each process owns exactly one region (its own
|
|
//! `ControlPlane`), so a coordinator in process A cannot drain process B. The
|
|
//! server-level rolling upgrade is instead choreographed by the OPERATOR surface that
|
|
//! DOES exist over the network — `/cluster/promote` (move leadership off the node
|
|
//! about to restart) + graceful SIGTERM (drain in-flight, checkpoint, WAL fsync) +
|
|
//! `/cluster/heal` (redeliver segments shipped while the node was down). Asserting
|
|
//! through the in-process coordinator here would test the wrong process's state, so
|
|
//! this suite exercises the real network choreography and documents the coordinator's
|
|
//! scope in this header rather than forcing it in. (Its own unit tests in
|
|
//! `upgrade.rs` cover the drain/rejoin invariant.)
|
|
//!
|
|
//! # Budget
|
|
//!
|
|
//! Tier-3 over real OS processes. Two tests: the skew test is a boot + converge +
|
|
//! one partition/heal/reconcile window; the upgrade test is a boot + four graceful
|
|
//! restarts, each followed by a breaker-bounded heal. Worst case the upgrade test's
|
|
//! heals each wait up to the 30s breaker reset, but in practice an eager-ship probe
|
|
//! (the continuous writer, while it runs) closes the breaker far sooner. The writer is
|
|
//! paced (~5 writes/sec) so it never starves the leader's small write pool and the
|
|
//! pool-backed heal can land. Every wait is poll-with-deadline; whole-suite wall budget
|
|
//! < 5 minutes on a developer laptop.
|
|
|
|
#![cfg(feature = "cluster-e2e")]
|
|
// Tier-3 harness allows, mirroring `cluster_chaos.rs` / `cluster_multiproc.rs`:
|
|
// `unwrap` on known-good fixtures is idiomatic test noise; the lossy numeric casts
|
|
// are the same pervasive-and-intentional scoring math the crate config documents.
|
|
#![allow(
|
|
clippy::unwrap_used,
|
|
clippy::missing_panics_doc,
|
|
clippy::too_many_lines,
|
|
clippy::cast_precision_loss,
|
|
clippy::cast_possible_truncation,
|
|
clippy::cast_sign_loss,
|
|
clippy::items_after_statements
|
|
)]
|
|
|
|
mod support;
|
|
|
|
use std::{
|
|
sync::{
|
|
Arc,
|
|
atomic::{AtomicBool, AtomicU64, Ordering},
|
|
},
|
|
thread,
|
|
time::{Duration, Instant},
|
|
};
|
|
|
|
use support::{
|
|
multiproc::{
|
|
BREAKER_RESET, ClusterOptions, MultiProcCluster, convergence_budget,
|
|
seed_items_and_embeddings, write_view,
|
|
},
|
|
partition::proxied_rewrite,
|
|
};
|
|
|
|
/// Region 0 = `us-east` = the initial leader in every topology.
|
|
const LEADER: usize = 0;
|
|
/// Region 1 = `eu-west` (a follower; promoted to leader in the upgrade test).
|
|
const EU_WEST: usize = 1;
|
|
/// Region 2 = `ap-south` (a follower; the skewed-BEHIND node in the skew test).
|
|
const AP_SOUTH: usize = 2;
|
|
|
|
// The leader-ships circuit breaker ([`BREAKER_RESET`], the `tidal-net` default)
|
|
// opens after 5 consecutive failed ships and stays open for 30s. After a node
|
|
// returns, the leader's transport cannot reach it until the breaker half-opens
|
|
// and one probe ship closes it — so a SINGLE `/cluster/heal` can ship into the
|
|
// open breaker and be a no-op. Tests bound post-return convergence at
|
|
// BREAKER_RESET + the convergence budget, re-issuing heal exactly like a
|
|
// runbook operator. (Same reasoning as `cluster_chaos.rs`.)
|
|
|
|
// ── Shared seeding / feed helpers (canonical bodies in support::multiproc) ──────
|
|
|
|
/// A node's local-region feed as a sorted `(entity_id, score)` vector.
|
|
fn feed_pairs(
|
|
cluster: &MultiProcCluster,
|
|
idx: usize,
|
|
profile: &str,
|
|
limit: u32,
|
|
) -> Vec<(u64, f64)> {
|
|
let body = cluster.get_json(idx, &format!("/feed?profile={profile}&limit={limit}"));
|
|
let mut pairs: Vec<(u64, f64)> = body["items"]
|
|
.as_array()
|
|
.unwrap_or(&Vec::new())
|
|
.iter()
|
|
.map(|it| {
|
|
(
|
|
it["entity_id"].as_u64().unwrap(),
|
|
it["score"].as_f64().unwrap(),
|
|
)
|
|
})
|
|
.collect();
|
|
pairs.sort_by_key(|(id, _)| *id);
|
|
pairs
|
|
}
|
|
|
|
/// A user-scoped feed's item-id set (sorted). Hard negatives for `user_id` are
|
|
/// filtered by the engine's user-context stage, so a hidden item is ABSENT here.
|
|
fn feed_item_ids_for_user(
|
|
cluster: &MultiProcCluster,
|
|
idx: usize,
|
|
user_id: u64,
|
|
profile: &str,
|
|
limit: u32,
|
|
) -> Vec<u64> {
|
|
let body = cluster.get_json(
|
|
idx,
|
|
&format!("/feed?profile={profile}&limit={limit}&user_id={user_id}"),
|
|
);
|
|
let mut ids: Vec<u64> = body["items"]
|
|
.as_array()
|
|
.unwrap_or(&Vec::new())
|
|
.iter()
|
|
.map(|it| it["entity_id"].as_u64().unwrap())
|
|
.collect();
|
|
ids.sort_unstable();
|
|
ids
|
|
}
|
|
|
|
/// Assert two feed views carry the SAME items with scores equal to 1e-6.
|
|
fn assert_feed_parity(label: &str, a: &[(u64, f64)], b: &[(u64, f64)]) {
|
|
assert_eq!(
|
|
a.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
|
|
b.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
|
|
"{label}: feed item sets differ"
|
|
);
|
|
for ((id_a, score_a), (_, score_b)) in a.iter().zip(b.iter()) {
|
|
assert!(
|
|
(score_a - score_b).abs() <= 1e-6,
|
|
"{label}: score for item {id_a} differs: {score_a} vs {score_b}"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// One node's `applied_events` from its OWN `/cluster/status/local` (direct addr).
|
|
fn applied(cluster: &MultiProcCluster, idx: usize) -> u64 {
|
|
cluster
|
|
.local_status(idx)
|
|
.and_then(|st| st["applied_events"].as_u64())
|
|
.unwrap_or(0)
|
|
}
|
|
|
|
/// True when every `followers` entry has applied up to (or past) the leader's
|
|
/// high-water-mark with zero lag. Non-panicking, so a retry loop can poll it.
|
|
fn converged(cluster: &MultiProcCluster, followers: &[usize]) -> bool {
|
|
let Some(target) = cluster.leader_last_seq() else {
|
|
return false;
|
|
};
|
|
followers.iter().all(|&idx| {
|
|
cluster.local_status(idx).is_some_and(|st| {
|
|
let applied = st["applied_events"].as_u64().unwrap_or(0);
|
|
let lag = st["lag_events"].as_u64().unwrap_or(u64::MAX);
|
|
lag == 0 && applied >= target
|
|
})
|
|
})
|
|
}
|
|
|
|
/// Drive convergence under CONCURRENT write load the way an operator does: re-issue
|
|
/// `POST /cluster/heal` for `region` until every `followers` entry converges, allowing
|
|
/// for the circuit-breaker reset window. The rolling-upgrade writer keeps hammering the
|
|
/// leader, so its write pool — and the axum request-timeout layer in front of the
|
|
/// pool-backed heal route — can make a `/cluster/heal` return 408 (request timeout) or
|
|
/// 429 (pool saturated) instead of 200. That is genuine production behavior, not a bug:
|
|
/// an operator re-issues the heal. So (unlike the chaos suite's strict heal, which runs
|
|
/// with no concurrent writer and asserts a 200) this TOLERATES a transient non-200 heal
|
|
/// and simply re-issues, bounded by the breaker reset + convergence budget; a heal that
|
|
/// DOES 200 is the one that redelivered. We always confirm convergence on STATE.
|
|
fn heal_until_converged_busy(cluster: &MultiProcCluster, region: &str, followers: &[usize]) {
|
|
let deadline = Instant::now() + BREAKER_RESET + convergence_budget();
|
|
loop {
|
|
// Re-issue heal; a 408/429/503 under write load is a no-op we retry, not a
|
|
// failure. Only a 4xx that is NOT timeout/backpressure (e.g. 400 unknown
|
|
// region) would be a real bug — but the region name is always valid here.
|
|
let status = cluster
|
|
.post(
|
|
LEADER,
|
|
"/cluster/heal",
|
|
&serde_json::json!({ "region": region }),
|
|
)
|
|
.status()
|
|
.as_u16();
|
|
assert!(
|
|
matches!(status, 200 | 408 | 429 | 503),
|
|
"/cluster/heal returned an unexpected status {status} (expected 200, or a \
|
|
transient 408/429/503 under write load)"
|
|
);
|
|
let check_deadline = Instant::now() + Duration::from_secs(3);
|
|
while Instant::now() <= check_deadline {
|
|
if converged(cluster, followers) {
|
|
return;
|
|
}
|
|
thread::sleep(Duration::from_millis(100));
|
|
}
|
|
assert!(
|
|
Instant::now() <= deadline,
|
|
"region '{region}' did not converge under write load within budget; \
|
|
leader hwm={:?}, follower applied={:?}",
|
|
cluster.leader_last_seq(),
|
|
followers
|
|
.iter()
|
|
.map(|&i| applied(cluster, i))
|
|
.collect::<Vec<_>>()
|
|
);
|
|
}
|
|
}
|
|
|
|
/// Poll `pred()` until true or `budget` elapses, asserting with `msg` on timeout.
|
|
fn poll_until(budget: Duration, msg: &str, mut pred: impl FnMut() -> bool) {
|
|
let deadline = Instant::now() + budget;
|
|
while !pred() {
|
|
assert!(Instant::now() <= deadline, "timed out: {msg}");
|
|
thread::sleep(Duration::from_millis(50));
|
|
}
|
|
}
|
|
|
|
// ── Clock-skew reconciliation ────────────────────────────────────────────────
|
|
|
|
#[test]
|
|
fn mp_clock_skew_reconciliation_stays_causal() {
|
|
// Skewed-AHEAD leader, on-time follower, skewed-BEHIND follower. The env var
|
|
// genuinely offsets each process's HLC (parsed i64 in main.rs → engine), so
|
|
// this is real ±500ms skew, not a mock.
|
|
let (rewrite, proxies) = proxied_rewrite(&["ap-south"]);
|
|
let cluster = MultiProcCluster::start_with(
|
|
ClusterOptions::new(3)
|
|
.with_env(LEADER, "TIDAL_HLC_SKEW_MS", "500")
|
|
.with_env(EU_WEST, "TIDAL_HLC_SKEW_MS", "0")
|
|
.with_env(AP_SOUTH, "TIDAL_HLC_SKEW_MS", "-500")
|
|
.with_rewrite(rewrite),
|
|
);
|
|
|
|
const ITEMS: u64 = 12;
|
|
/// The user whose hide the AHEAD leader records (on pair A).
|
|
const AHEAD_USER: u64 = 11;
|
|
/// The item the AHEAD leader (+500ms) hides.
|
|
const AHEAD_ITEM: u64 = 4;
|
|
/// The user whose hide the BEHIND split-brain node records (on pair B).
|
|
const BEHIND_USER: u64 = 22;
|
|
/// The item the BEHIND follower (-500ms) hides.
|
|
const BEHIND_ITEM: u64 = 9;
|
|
|
|
// ── PHASE A: normal replication under ±500ms skew ──────────────────────────
|
|
seed_items_and_embeddings(&cluster, LEADER, ITEMS);
|
|
for entity_id in 1..=ITEMS {
|
|
write_view(&cluster, LEADER, entity_id, entity_id as f64);
|
|
}
|
|
cluster.wait_converged_all(convergence_budget());
|
|
|
|
// Decay reads the real wall clock, so ±500ms of HLC skew leaves the ranked
|
|
// feed byte-identical across the three skewed processes. This pins the scope:
|
|
// the skew mechanism is HLC-only (reconcile LWW), never signal decay.
|
|
let leader_feed = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
|
|
let eu_feed = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32);
|
|
let ap_feed = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
|
|
assert!(!leader_feed.is_empty(), "leader must rank the seeded items");
|
|
assert_feed_parity(
|
|
"leader(+500) vs eu-west(0) under skew",
|
|
&leader_feed,
|
|
&eu_feed,
|
|
);
|
|
assert_feed_parity(
|
|
"leader(+500) vs ap-south(-500) under skew",
|
|
&leader_feed,
|
|
&ap_feed,
|
|
);
|
|
println!(
|
|
"[skew] PHASE A: {} items rank identically (1e-6) across +500ms / 0 / -500ms processes \
|
|
(decay is wall-clock, unaffected by HLC skew)",
|
|
leader_feed.len()
|
|
);
|
|
|
|
// ── PHASE B: divergence under a real partition (split-brain hide) ──────────
|
|
// Sever the skewed-BEHIND follower (ap-south, -500ms) from every peer.
|
|
proxies.region("ap-south").sever_all();
|
|
println!("[skew] PHASE B: severed ap-south (-500ms) from all peers");
|
|
|
|
// AHEAD leader (+500ms, us-east) hides pair A. It is the real leader, so a plain
|
|
// external /hardnegs applies locally.
|
|
let resp = cluster.post(
|
|
LEADER,
|
|
"/hardnegs",
|
|
&serde_json::json!({ "user_id": AHEAD_USER, "item_id": AHEAD_ITEM }),
|
|
);
|
|
assert_eq!(resp.status().as_u16(), 204, "ahead-leader hardneg must 204");
|
|
|
|
// BEHIND follower (-500ms, ap-south): promote it to leader IN ITS OWN VIEW. The
|
|
// fan-out to the severed peers fails (it cannot reach them), leaving a genuine
|
|
// SPLIT BRAIN — exactly the production hazard reconciliation repairs. promote_local
|
|
// applies unconditionally, so ap-south now answers is_leader:true on its own view.
|
|
let ap_region = cluster.region_name(AP_SOUTH).to_string();
|
|
let resp = cluster.post(
|
|
AP_SOUTH,
|
|
"/cluster/promote",
|
|
&serde_json::json!({ "region": ap_region }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
200,
|
|
"split-brain self-promote on the severed follower must 200: {}",
|
|
resp.status()
|
|
);
|
|
poll_until(
|
|
Duration::from_secs(5),
|
|
"severed ap-south must adopt its own leader view (split brain)",
|
|
|| {
|
|
cluster
|
|
.local_status(AP_SOUTH)
|
|
.and_then(|st| st["is_leader"].as_bool())
|
|
== Some(true)
|
|
},
|
|
);
|
|
|
|
// A plain external /hardnegs on the split-brain leader (ap-south) applies LOCALLY
|
|
// (state.is_leader() is true on its own view) — a node-local divergent hide on a
|
|
// DIFFERENT pair (B). No internal marker, no forward: a real partitioned write.
|
|
let resp = cluster.post(
|
|
AP_SOUTH,
|
|
"/hardnegs",
|
|
&serde_json::json!({ "user_id": BEHIND_USER, "item_id": BEHIND_ITEM }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
204,
|
|
"split-brain follower hardneg must apply locally (204): {}",
|
|
resp.status()
|
|
);
|
|
println!(
|
|
"[skew] divergence created: +500 leader hid ({AHEAD_USER},{AHEAD_ITEM}); \
|
|
-500 split-brain ap-south hid ({BEHIND_USER},{BEHIND_ITEM})"
|
|
);
|
|
|
|
// Before reconcile, each hide exists on exactly ONE node.
|
|
assert!(
|
|
!feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32)
|
|
.contains(&AHEAD_ITEM),
|
|
"ahead hide must be effective on the leader pre-reconcile"
|
|
);
|
|
assert!(
|
|
feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32)
|
|
.contains(&BEHIND_ITEM),
|
|
"leader must NOT yet know the behind node's hide pre-reconcile"
|
|
);
|
|
|
|
// ── HEAL + collapse the split brain, then reconcile BOTH directions ────────
|
|
proxies.region("ap-south").heal_all();
|
|
// Re-promote the real leader (us-east) to collapse the split brain — the runbook
|
|
// step after a partitioned node rejoins. The durable Tag::HardNeg rows survive the
|
|
// leadership change (store rows, not leader state), so the behind hide is intact.
|
|
let leader_region = cluster.region_name(LEADER).to_string();
|
|
let resp = cluster.post(
|
|
LEADER,
|
|
"/cluster/promote",
|
|
&serde_json::json!({ "region": leader_region }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
200,
|
|
"re-promote real leader must 200"
|
|
);
|
|
cluster.wait_leader_agreed(&leader_region, Duration::from_secs(15));
|
|
println!("[skew] healed + collapsed split brain (leader = {leader_region})");
|
|
|
|
// Reconcile leader -> ap-south (the call exchanges snapshots bidirectionally:
|
|
// ships ours into ap-south's merge AND applies ap-south's back). Assert < 100ms.
|
|
let (l_ms, r_ms) = reconcile(&cluster, LEADER, &ap_region);
|
|
assert!(
|
|
l_ms < 100 && r_ms < 100,
|
|
"reconcile under skew must be < 100ms both sides: leader-side local={l_ms}ms remote={r_ms}ms"
|
|
);
|
|
println!("[skew] reconcile leader->ap-south: local={l_ms}ms remote={r_ms}ms (both < 100ms)");
|
|
|
|
// Reconcile the OTHER direction (ap-south -> leader) to prove symmetry under skew.
|
|
let (l_ms2, r_ms2) = reconcile(&cluster, AP_SOUTH, &leader_region);
|
|
assert!(
|
|
l_ms2 < 100 && r_ms2 < 100,
|
|
"reverse reconcile under skew must be < 100ms both sides: local={l_ms2}ms remote={r_ms2}ms"
|
|
);
|
|
println!("[skew] reconcile ap-south->leader: local={l_ms2}ms remote={r_ms2}ms (both < 100ms)");
|
|
|
|
// ── CONVERGENCE: both hides effective on BOTH nodes, LWW deterministic ─────
|
|
assert_converged_hide(
|
|
&cluster,
|
|
LEADER,
|
|
AP_SOUTH,
|
|
AHEAD_USER,
|
|
AHEAD_ITEM,
|
|
"ahead hide",
|
|
);
|
|
assert_converged_hide(
|
|
&cluster,
|
|
LEADER,
|
|
AP_SOUTH,
|
|
BEHIND_USER,
|
|
BEHIND_ITEM,
|
|
"behind hide",
|
|
);
|
|
println!(
|
|
"[skew] converged: both hides effective on leader AND ap-south despite ±500ms skew \
|
|
(skewed-behind node's hide NOT lost; LWW deterministic)"
|
|
);
|
|
|
|
// ── STABILITY: a repeat reconcile leaves the STATE unchanged ───────────────
|
|
// (ops_applied may be > 0 by design — the LWW plan re-resolves the same hides —
|
|
// so we assert on STATE, never on ops_applied: no timestamp oscillation.)
|
|
let leader_before = (
|
|
feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32),
|
|
feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32),
|
|
);
|
|
let ap_before = (
|
|
feed_item_ids_for_user(&cluster, AP_SOUTH, AHEAD_USER, "trending", ITEMS as u32),
|
|
feed_item_ids_for_user(&cluster, AP_SOUTH, BEHIND_USER, "trending", ITEMS as u32),
|
|
);
|
|
let (l_ms3, r_ms3) = reconcile(&cluster, LEADER, &ap_region);
|
|
assert!(
|
|
l_ms3 < 100 && r_ms3 < 100,
|
|
"repeat reconcile must stay < 100ms both sides: local={l_ms3}ms remote={r_ms3}ms"
|
|
);
|
|
let leader_after = (
|
|
feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32),
|
|
feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32),
|
|
);
|
|
let ap_after = (
|
|
feed_item_ids_for_user(&cluster, AP_SOUTH, AHEAD_USER, "trending", ITEMS as u32),
|
|
feed_item_ids_for_user(&cluster, AP_SOUTH, BEHIND_USER, "trending", ITEMS as u32),
|
|
);
|
|
assert_eq!(
|
|
leader_before, leader_after,
|
|
"repeat reconcile changed leader hide state"
|
|
);
|
|
assert_eq!(
|
|
ap_before, ap_after,
|
|
"repeat reconcile changed ap-south hide state"
|
|
);
|
|
println!(
|
|
"[skew] STABLE: repeat reconcile left both nodes' hide state identical (no oscillation)"
|
|
);
|
|
}
|
|
|
|
/// Drive `POST /cluster/reconcile` from `from_idx` against `with_region`, asserting a
|
|
/// 200, and return `(local_elapsed_ms, remote_elapsed_ms)`.
|
|
fn reconcile(cluster: &MultiProcCluster, from_idx: usize, with_region: &str) -> (u64, u64) {
|
|
let resp = cluster.post(
|
|
from_idx,
|
|
"/cluster/reconcile",
|
|
&serde_json::json!({ "region": with_region }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
200,
|
|
"reconcile from node {from_idx} with '{with_region}' must 200: {}",
|
|
resp.status()
|
|
);
|
|
let body: serde_json::Value = resp.json().unwrap();
|
|
assert_eq!(
|
|
body["ok"].as_bool(),
|
|
Some(true),
|
|
"reconcile ok must be true: {body}"
|
|
);
|
|
(
|
|
body["local_elapsed_ms"].as_u64().unwrap(),
|
|
body["remote_elapsed_ms"].as_u64().unwrap(),
|
|
)
|
|
}
|
|
|
|
/// Assert the WAL relay ALONE (heal redelivery + item backfill — no
|
|
/// `/cluster/reconcile`) reconverges `follower` to the current leader's ranking to
|
|
/// 1e-6. The background writer is still running, so the leader's feed moves between
|
|
/// reads; we capture leader then follower back-to-back and retry until a snapshot
|
|
/// pair matches, bounded by the convergence budget. A persistent mismatch is a real
|
|
/// loss the WAL relay failed to close (bug 3), surfaced — not masked.
|
|
fn assert_wal_relay_parity_to_leader(
|
|
cluster: &MultiProcCluster,
|
|
follower: usize,
|
|
limit: u32,
|
|
label: &str,
|
|
) {
|
|
let leader_idx = current_leader_idx(cluster);
|
|
let deadline = Instant::now() + convergence_budget();
|
|
loop {
|
|
// Capture the follower first, then the leader: if they match, the follower is
|
|
// at least as current as a leader snapshot taken AFTER it, so the relay has
|
|
// delivered everything up to that point (a conservative, race-safe check).
|
|
let f = feed_pairs(cluster, follower, "trending", limit);
|
|
let l = feed_pairs(cluster, leader_idx, "trending", limit);
|
|
let matched = f.len() == l.len()
|
|
&& f.iter()
|
|
.zip(l.iter())
|
|
.all(|((fi, fs), (li, ls))| fi == li && (fs - ls).abs() <= 1e-6);
|
|
if matched {
|
|
println!(
|
|
"[upgrade] {label}: WAL relay alone reconverged to the leader's ranking \
|
|
({} items, 1e-6) — NO reconcile needed",
|
|
f.len()
|
|
);
|
|
return;
|
|
}
|
|
assert!(
|
|
Instant::now() <= deadline,
|
|
"{label}: WAL relay alone did NOT reconverge the restarted node to the leader \
|
|
within budget (bug 3 would manifest here as a permanent gap); \
|
|
follower={f:?} leader={l:?}"
|
|
);
|
|
thread::sleep(Duration::from_millis(100));
|
|
}
|
|
}
|
|
|
|
/// Index of the region every node currently agrees is the leader (from the leader's
|
|
/// own local status). Falls back to `LEADER` if unresolved.
|
|
fn current_leader_idx(cluster: &MultiProcCluster) -> usize {
|
|
for i in 0..cluster.len() {
|
|
if let Some(name) = cluster
|
|
.local_status(i)
|
|
.and_then(|st| st["leader"].as_str().map(str::to_string))
|
|
&& let Some(idx) = (0..cluster.len()).find(|&j| cluster.region_name(j) == name)
|
|
{
|
|
return idx;
|
|
}
|
|
}
|
|
LEADER
|
|
}
|
|
|
|
/// Whether all three nodes' `trending` feeds agree to 1e-6 (item set + scores).
|
|
fn three_way_parity(cluster: &MultiProcCluster, leader_idx: usize, limit: u32) -> bool {
|
|
let a = feed_pairs(cluster, leader_idx, "trending", limit);
|
|
let others: Vec<usize> = (0..cluster.len()).filter(|&i| i != leader_idx).collect();
|
|
others.iter().all(|&idx| {
|
|
let b = feed_pairs(cluster, idx, "trending", limit);
|
|
a.len() == b.len()
|
|
&& a.iter()
|
|
.zip(b.iter())
|
|
.all(|((ia, sa), (ib, sb))| ia == ib && (sa - sb).abs() <= 1e-6)
|
|
})
|
|
}
|
|
|
|
/// Assert a hide for `(user,item)` is effective on BOTH nodes: the control un-scoped
|
|
/// feed still ranks the item on each (proving it is a FILTER, not a disappearance),
|
|
/// while the `?user_id=` feed omits it on each.
|
|
fn assert_converged_hide(
|
|
cluster: &MultiProcCluster,
|
|
node_a: usize,
|
|
node_b: usize,
|
|
user: u64,
|
|
item: u64,
|
|
label: &str,
|
|
) {
|
|
const LIMIT: u32 = 12;
|
|
for &node in &[node_a, node_b] {
|
|
let control: Vec<u64> = feed_pairs(cluster, node, "trending", LIMIT)
|
|
.into_iter()
|
|
.map(|(id, _)| id)
|
|
.collect();
|
|
assert!(
|
|
control.contains(&item),
|
|
"{label}: control un-scoped feed on node {node} must still rank item {item}: {control:?}"
|
|
);
|
|
let scoped = feed_item_ids_for_user(cluster, node, user, "trending", LIMIT);
|
|
assert!(
|
|
!scoped.contains(&item),
|
|
"{label}: item {item} must be hidden from user {user} on node {node}: {scoped:?}"
|
|
);
|
|
}
|
|
}
|
|
|
|
// ── Rolling upgrade: no loss, no stall ───────────────────────────────────────
|
|
|
|
/// A background writer's tally. `total` logical writes, `retried` that needed >= 1
|
|
/// retry, `lost` that never landed (the zero-loss gate asserts `lost == 0`).
|
|
struct WriterTally {
|
|
total: AtomicU64,
|
|
retried: AtomicU64,
|
|
lost: AtomicU64,
|
|
}
|
|
|
|
impl WriterTally {
|
|
const fn new() -> Self {
|
|
Self {
|
|
total: AtomicU64::new(0),
|
|
retried: AtomicU64::new(0),
|
|
lost: AtomicU64::new(0),
|
|
}
|
|
}
|
|
|
|
fn snapshot(&self) -> (u64, u64, u64) {
|
|
(
|
|
self.total.load(Ordering::Relaxed),
|
|
self.retried.load(Ordering::Relaxed),
|
|
self.lost.load(Ordering::Relaxed),
|
|
)
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn mp_rolling_upgrade_no_loss_no_stall() {
|
|
// Persistent data dirs are guaranteed by the harness (it always passes
|
|
// --data-dir and reuses it on restart). Boot "version N". The cluster handle
|
|
// stays owned on THIS thread (restarts need &mut self); the writer thread gets
|
|
// only the stable node base URLs + its own client, so the two never alias.
|
|
let mut cluster = MultiProcCluster::start_with(
|
|
ClusterOptions::new(3)
|
|
.with_env(LEADER, "TIDAL_VERSION_TAG", "N")
|
|
.with_env(EU_WEST, "TIDAL_VERSION_TAG", "N")
|
|
.with_env(AP_SOUTH, "TIDAL_VERSION_TAG", "N"),
|
|
);
|
|
|
|
const ITEMS: u64 = 16;
|
|
seed_items_and_embeddings(&cluster, LEADER, ITEMS);
|
|
// Steady-state warmup so every node shares a converged baseline before the
|
|
// continuous writer + restarts begin.
|
|
for entity_id in 1..=ITEMS {
|
|
write_view(&cluster, LEADER, entity_id, 1.0);
|
|
}
|
|
cluster.wait_converged_all(convergence_budget());
|
|
|
|
// Spot-check fixture: an item written on the leader BEFORE any restart, used to
|
|
// prove WAL recovery (each restarted node still serves it AFTER its restart).
|
|
let pre_restart_feed = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
|
|
let pre_restart_ids: Vec<u64> = pre_restart_feed.iter().map(|(id, _)| *id).collect();
|
|
assert!(
|
|
!pre_restart_ids.is_empty(),
|
|
"leader must rank seeded items pre-restart"
|
|
);
|
|
|
|
// ── Background writer: continuous round-robin POST /signals across ALL nodes ─
|
|
// The writer holds only the node URLs (ports are stable across restarts) and its
|
|
// own blocking client — fully decoupled from the &mut harness on this thread.
|
|
let node_urls: Vec<String> = (0..cluster.len()).map(|i| cluster.node(i)).collect();
|
|
let tally = Arc::new(WriterTally::new());
|
|
let stop = Arc::new(AtomicBool::new(false));
|
|
let writer = {
|
|
let tally = Arc::clone(&tally);
|
|
let stop = Arc::clone(&stop);
|
|
thread::spawn(move || run_writer(&node_urls, &tally, &stop, ITEMS))
|
|
};
|
|
|
|
// Let the writer establish a cadence before the first restart.
|
|
thread::sleep(Duration::from_millis(300));
|
|
|
|
// ── Upgrade FOLLOWERS one at a time (graceful SIGTERM → restart N+1 → heal) ──
|
|
for &follower in &[EU_WEST, AP_SOUTH] {
|
|
let region = cluster.region_name(follower).to_string();
|
|
println!("[upgrade] gracefully stopping follower {region} for upgrade to N+1");
|
|
cluster.restart_graceful(follower, &[("TIDAL_VERSION_TAG", "N+1")]);
|
|
// The leader's breaker opened while this follower was down; drive heal through
|
|
// the reset exactly like an operator — tolerating transient 408/429 from the
|
|
// pool-backed heal route under the concurrent writer.
|
|
heal_until_converged_busy(&cluster, ®ion, &[follower]);
|
|
println!("[upgrade] follower {region} upgraded to N+1; WAL-relay caught up (lag 0)");
|
|
|
|
// WAL recovery: the just-restarted follower still SERVES the pre-restart items
|
|
// (presence).
|
|
let post_pairs = feed_pairs(&cluster, follower, "trending", ITEMS as u32);
|
|
let post_ids: Vec<u64> = post_pairs.iter().map(|(id, _)| *id).collect();
|
|
for id in &pre_restart_ids {
|
|
assert!(
|
|
post_ids.contains(id),
|
|
"{region}: pre-restart item {id} lost after restart (WAL recovery): {post_ids:?}"
|
|
);
|
|
}
|
|
|
|
// STRONGER CONTRACT (bugs 1+3 fixed): heal alone — `/cluster/heal` redelivers
|
|
// the missed signal segments AND backfills the missed item metadata, with an
|
|
// honest lag gauge — must reconverge the restarted follower to the LEADER's
|
|
// ranking WITHOUT any `/cluster/reconcile` in the path. We poll the follower's
|
|
// feed against the leader's to 1e-6 (the writer is still running, so the leader
|
|
// moves; poll until a snapshot pair matches). This is the eventual-consistency
|
|
// guarantee the WAL relay now delivers on its own.
|
|
assert_wal_relay_parity_to_leader(&cluster, follower, ITEMS as u32, ®ion);
|
|
}
|
|
|
|
// ── Drain the writer before the leadership handoff ──────────────────────────
|
|
// The two FOLLOWER upgrades above ran under continuous write load (the whole point:
|
|
// no lost LOGICAL writes during a restart — the lost==0 gate below). We drain before
|
|
// the leader handoff so the handoff happens over a quiescent cluster (faithful
|
|
// operator practice for a planned failover) and so the final anti-entropy reconcile
|
|
// snapshots a stable state.
|
|
stop.store(true, Ordering::Relaxed);
|
|
writer.join().expect("writer thread joined");
|
|
let (total, retried, lost) = tally.snapshot();
|
|
println!(
|
|
"[upgrade] writer drained before handoff; tally: total={total} retried={retried} lost={lost}"
|
|
);
|
|
|
|
// ── Leader LAST: promote an upgraded follower, then restart the old leader ───
|
|
let new_leader = cluster.region_name(EU_WEST).to_string();
|
|
println!("[upgrade] promoting upgraded follower {new_leader} before restarting the old leader");
|
|
let resp = cluster.post(
|
|
AP_SOUTH,
|
|
"/cluster/promote",
|
|
&serde_json::json!({ "region": new_leader }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
200,
|
|
"promote must 200: {}",
|
|
resp.status()
|
|
);
|
|
// The old leader is still up here, so it acks the fan-out and every LIVE node
|
|
// agrees eu-west leads.
|
|
cluster.wait_leader_agreed(&new_leader, Duration::from_secs(15));
|
|
|
|
let old_leader = cluster.region_name(LEADER).to_string();
|
|
println!("[upgrade] gracefully restarting the OLD leader {old_leader} to N+1");
|
|
cluster.restart_graceful(LEADER, &[("TIDAL_VERSION_TAG", "N+1")]);
|
|
// The restarted old leader read `leader: us-east` from its topology file, so it
|
|
// boots with a STALE leader view (transient split brain). Re-promote the chosen
|
|
// leader to collapse it — the real operator step after a node rejoins.
|
|
let resp = cluster.post(
|
|
LEADER,
|
|
"/cluster/promote",
|
|
&serde_json::json!({ "region": new_leader }),
|
|
);
|
|
assert_eq!(
|
|
resp.status().as_u16(),
|
|
200,
|
|
"re-promote after old-leader rejoin must 200: {}",
|
|
resp.status()
|
|
);
|
|
cluster.wait_leader_agreed(&new_leader, Duration::from_secs(15));
|
|
println!("[upgrade] old leader {old_leader} rejoined as a follower of {new_leader}");
|
|
|
|
// ── WAL RELAY ALONE reconverges the restarted old leader (bugs 1+3 fixed) ────
|
|
// The new leader heals the rejoined old leader: `/cluster/heal` redelivers the
|
|
// signal segments it missed while down AND backfills the item metadata, with an
|
|
// honest per-shard lag gauge. The writer is drained, so the cluster is quiescent
|
|
// (a planned-failover handoff over a quiescent cluster is faithful operator
|
|
// practice — we drain BEFORE the handoff for exactly this reason). After heal,
|
|
// ALL THREE feeds must agree to 1e-6 WITHOUT any `/cluster/reconcile` — the WAL
|
|
// relay is the convergence mechanism, and reaching three-way parity (no stall)
|
|
// with every item present (no loss) and scores equal (no duplication) is the
|
|
// exactly-once proof. Before the bug fixes this required a multi-round reconcile
|
|
// fixpoint loop; now the relay alone closes the gap.
|
|
heal_until_converged_busy(&cluster, &old_leader, &[LEADER]);
|
|
let new_leader_idx = EU_WEST;
|
|
poll_until(
|
|
convergence_budget(),
|
|
"WAL relay alone must reconverge all three nodes to 1e-6 after the rolling upgrade",
|
|
|| three_way_parity(&cluster, new_leader_idx, ITEMS as u32),
|
|
);
|
|
println!(
|
|
"[upgrade] WAL relay alone reconverged all three nodes (1e-6) — NO reconcile in the \
|
|
convergence path"
|
|
);
|
|
|
|
// ── Final reconcile is a TRUE no-op (bug 2 fixed: idempotent, fixpoint at once) ─
|
|
// Capture each node's feed, drive ONE pairwise reconcile round in BOTH directions,
|
|
// and assert every node's feed is BYTE-for-byte unchanged. Before bug 2 the signal
|
|
// merge summed already-replicated per-node contributions, so reconcile of converged
|
|
// nodes crept the scores (0.5 → 0.375 → …). Now `merge` is idempotent: reconcile of
|
|
// converged nodes is an exact fixpoint immediately, no creep.
|
|
let before_leader = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32);
|
|
let before_us = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
|
|
let before_ap = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
|
|
let us_region = cluster.region_name(LEADER).to_string();
|
|
let ap_region = cluster.region_name(AP_SOUTH).to_string();
|
|
let _ = reconcile(&cluster, EU_WEST, &us_region);
|
|
let _ = reconcile(&cluster, LEADER, &new_leader);
|
|
let _ = reconcile(&cluster, EU_WEST, &ap_region);
|
|
let _ = reconcile(&cluster, AP_SOUTH, &new_leader);
|
|
assert_feed_parity(
|
|
"final reconcile no-op: eu-west unchanged",
|
|
&before_leader,
|
|
&feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32),
|
|
);
|
|
assert_feed_parity(
|
|
"final reconcile no-op: us-east unchanged",
|
|
&before_us,
|
|
&feed_pairs(&cluster, LEADER, "trending", ITEMS as u32),
|
|
);
|
|
assert_feed_parity(
|
|
"final reconcile no-op: ap-south unchanged",
|
|
&before_ap,
|
|
&feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32),
|
|
);
|
|
println!("[upgrade] final reconcile is an exact no-op on scores (bug 2 fixed: true fixpoint)");
|
|
|
|
// ── Final state: exactly-once (no stall, no loss), leader is the promoted node ─
|
|
let leader_feed = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32); // current leader
|
|
let f1 = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
|
|
let f2 = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
|
|
assert_feed_parity("post-upgrade leader(eu-west) vs us-east", &leader_feed, &f1);
|
|
assert_feed_parity(
|
|
"post-upgrade leader(eu-west) vs ap-south",
|
|
&leader_feed,
|
|
&f2,
|
|
);
|
|
println!(
|
|
"[upgrade] exactly-once + no stall: {} items rank identically (1e-6) across all three upgraded nodes",
|
|
leader_feed.len()
|
|
);
|
|
|
|
// No acknowledged writes lost (the spec's hard guarantee, criterion 7): every
|
|
// logical write the writer issued returned 204 exactly once.
|
|
assert_eq!(
|
|
lost, 0,
|
|
"rolling upgrade LOST {lost} acknowledged logical writes (of {total})"
|
|
);
|
|
assert!(
|
|
total > 0,
|
|
"writer must have issued writes during the upgrade window"
|
|
);
|
|
assert!(
|
|
retried > 0,
|
|
"expected >= 1 retried write across the restart windows; got {retried} (window too quiet?)"
|
|
);
|
|
|
|
// WAL recovery proven for the restarted old leader: an item written BEFORE its
|
|
// restart is still served by it AFTER (and now ranks identically to the cluster).
|
|
let old_leader_ids: Vec<u64> = f1.iter().map(|(id, _)| *id).collect();
|
|
for id in &pre_restart_ids {
|
|
assert!(
|
|
old_leader_ids.contains(id),
|
|
"old leader {old_leader}: pre-restart item {id} lost after restart (WAL recovery): {old_leader_ids:?}"
|
|
);
|
|
}
|
|
|
|
// The new leader is the promoted node.
|
|
let leader_now = cluster
|
|
.local_status(EU_WEST)
|
|
.and_then(|st| st["leader"].as_str().map(str::to_string));
|
|
assert_eq!(
|
|
leader_now.as_deref(),
|
|
Some(new_leader.as_str()),
|
|
"final leader must be the promoted node {new_leader}"
|
|
);
|
|
}
|
|
|
|
/// Per logical write, the maximum number of (node-cycling) attempts before we give
|
|
/// up and count it LOST. Generous: four restart windows, each bounded by boot +
|
|
/// breaker reset, so a healthy cluster lands a write in a handful of attempts; this
|
|
/// cap only fires on a genuine, persistent failure (the exact bug the gate catches).
|
|
const MAX_WRITE_ATTEMPTS: u64 = 400;
|
|
|
|
/// Inter-write pacing for the background writer. A rolling upgrade rides STEADY
|
|
/// production traffic, not a saturation storm: the leader's cluster write pool has
|
|
/// only `MIN_WRITE_WORKERS = 2` slots, shared by signal writes AND the pool-backed
|
|
/// `/cluster/heal` redelivery. Pacing leaves headroom so a heal can land (otherwise
|
|
/// the writer starves both worker slots and the recovery path never runs — a
|
|
/// test-induced deadlock, not a product behavior). ~50 writes/sec is plenty of
|
|
/// continuous traffic to overlap every restart window with live writes.
|
|
const WRITE_PACING: Duration = Duration::from_millis(200);
|
|
|
|
/// The background writer: round-robins a `view` signal across the node base URLs,
|
|
/// retrying the SAME logical write (cycling targets) until it 204s. Tolerates the
|
|
/// failure modes a restart window produces — connection refused while a process is
|
|
/// down, 503 (`NotLeader` / leader-unreachable while forwarding), 429 (write-pool
|
|
/// backpressure), 408 (request timeout).
|
|
///
|
|
/// `stop` is checked only at the TOP of each logical write, so a write the writer has
|
|
/// COMMITTED to (incremented `total`) is always driven to completion or to `lost`,
|
|
/// never silently abandoned — the test settles a healthy steady-state window before
|
|
/// flipping `stop`, so the last committed write lands. A logical write that never
|
|
/// 204s within [`MAX_WRITE_ATTEMPTS`] is counted LOST (the zero-loss gate then fails,
|
|
/// as it should). Uses its OWN blocking client + the stable node URLs, so it never
|
|
/// aliases the &mut harness on the test thread.
|
|
fn run_writer(node_urls: &[String], tally: &WriterTally, stop: &AtomicBool, items: u64) {
|
|
let client = reqwest::blocking::Client::builder()
|
|
.build()
|
|
.expect("build writer client");
|
|
let mut node = 0usize;
|
|
let mut entity = 1u64;
|
|
while !stop.load(Ordering::Relaxed) {
|
|
tally.total.fetch_add(1, Ordering::Relaxed);
|
|
let mut attempts = 0u64;
|
|
let landed = loop {
|
|
let target = &node_urls[node % node_urls.len()];
|
|
let resp = client
|
|
.post(format!("{target}/signals"))
|
|
.json(&serde_json::json!({ "entity_id": entity, "signal": "view", "weight": 1.0 }))
|
|
.timeout(Duration::from_secs(3))
|
|
.send();
|
|
if let Ok(r) = resp
|
|
&& r.status().as_u16() == 204
|
|
{
|
|
break true;
|
|
}
|
|
attempts += 1;
|
|
if attempts >= MAX_WRITE_ATTEMPTS {
|
|
break false; // genuine persistent failure → LOST
|
|
}
|
|
// Cycle to another node (the target may be the one restarting; a live node
|
|
// forwards to the current leader) and retry the SAME logical write.
|
|
node += 1;
|
|
thread::sleep(Duration::from_millis(25));
|
|
};
|
|
if landed {
|
|
if attempts > 0 {
|
|
tally.retried.fetch_add(1, Ordering::Relaxed);
|
|
}
|
|
} else {
|
|
tally.lost.fetch_add(1, Ordering::Relaxed);
|
|
}
|
|
node += 1;
|
|
entity = (entity % items) + 1;
|
|
// Steady-traffic pacing: leave write-pool headroom for the pool-backed heal.
|
|
thread::sleep(WRITE_PACING);
|
|
}
|
|
}
|