//! Tier-3 MULTI-PROCESS lifecycle chaos suite (m8p10 task 06): clock skew and
//! rolling upgrade.
//!
//! These two scenarios are kept in their OWN file (not `cluster_chaos.rs`, which is
//! already at a healthy size and owns the partition-injection suite) so each suite
//! stays independently runnable; both share the same `mod support;` tier-3 harness
//! and the same `cluster-e2e` feature gate.
//!
//! Run:
//! ```bash
//! cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle -- --nocapture
//! ```
//!
//! # What each test proves
//!
//! ## `mp_clock_skew_reconciliation_stays_causal`
//!
//! Three OS processes run with GENUINELY skewed HLCs (`TIDAL_HLC_SKEW_MS` = `+500`,
//! `0`, `-500`). The env var flows through `main.rs` → `TidalDbBuilder::
//! with_hlc_offset_ms` → `Hlc::for_shard_with_offset`, which the engine reads ONLY
//! when stamping hard-negative LWW registers in `take_crdt_snapshot` — signal-decay
//! timestamps use `Timestamp::now()` and are deliberately UNAFFECTED. The test pins
//! that scope and the convergence guarantee:
//!
//! * **Phase A — normal replication under skew.** Seed + converge; the leader's
//!   `view` signals replicate over the WAL relay and every follower ranks an
//!   IDENTICAL feed to 1e-6. Because decay reads the real wall clock, ±500ms of HLC
//!   skew leaves decayed scores byte-for-byte equal across the three skewed
//!   processes — proof the skew mechanism touches HLC (reconcile LWW) ONLY.
//! * **Phase B — divergence under a real partition.** We sever the skewed-BEHIND
//!   follower (`-500ms`, ap-south) from every peer with the TCP proxy, then create
//!   NODE-LOCAL divergence honestly (see "Divergence creation" below): the
//!   skewed-AHEAD real leader (`+500ms`, us-east) hides one pair, and the severed
//!   skewed-behind node — promoted to leader in its OWN partitioned view, a genuine
//!   split-brain — hides a DIFFERENT pair. Each pair therefore has exactly ONE
//!   writer.
//! * **Heal + reconcile BOTH directions.** After healing the link and re-promoting
//!   the real leader (resolving the split brain), `/cluster/reconcile` is driven
//!   from both sides. LWW is deterministic despite 1s of relative skew (HLC
//!   `update()` advances past remote timestamps, `max(wall, last_seen+1)`), so both
//!   nodes converge to the IDENTICAL hard-negative state: each hide is effective on
//!   BOTH nodes (verified by the `/feed?user_id=` control-vs-filtered pattern). A
//!   repeated reconcile leaves the converged STATE unchanged (no timestamp
//!   oscillation; we assert on state, never on `ops_applied`, which is > 0 by
//!   design). Reconcile merge+apply stays < 100ms both sides under skew.
//!
//! ### Divergence creation (honest choice)
//!
//! The task doc's divergence is "hide on the skewed-BEHIND node and a DIFFERENT pair
//! on the skewed-AHEAD leader during the window", which needs a NODE-LOCAL hide on
//! the partitioned follower. `/hardnegs` on a non-leader FORWARDS to the leader; while
//! severed that forward fails — which is the forwarding contract working, not a bug.
//! Rather than reach for the `x-tidal-internal` marker (a stand-in), we take the
//! CLEANER real path the runbook itself sanctions: promote the severed follower to
//! leader IN ITS OWN VIEW (`/cluster/promote` applies `promote_local` unconditionally;
//! the fan-out to severed peers fails, leaving a genuine SPLIT BRAIN — two
//! leader-views during the partition). A plain external `/hardnegs` to that
//! split-brain leader then applies LOCALLY because `state.is_leader()` is true on its
//! own view. This is exactly the production hazard reconciliation exists to repair:
//! two nodes each accepted a divergent write while partitioned. On heal we re-promote
//! the real leader to collapse the split brain, then reconcile — the durable
//! `Tag::HardNeg` rows survive the leadership change (they are store rows, not
//! leader state), so both hides converge by LWW.
//!
//! ## `mp_rolling_upgrade_no_loss_no_stall`
//!
//! Three processes on PERSISTENT data dirs (the harness always passes `--data-dir`
//! and reuses it on restart, so a restart recovers pre-restart state from the WAL).
//! A background writer thread continuously `POST /signals` round-robin across ALL
//! three live nodes (followers forward to the current leader) while we perform the
//! full rolling-upgrade choreography:
//!
//! * Boot "version N" (env `TIDAL_VERSION_TAG=N`).
//! * Upgrade followers one at a time UNDER LOAD: graceful SIGTERM (clean drain +
//!   checkpoint + WAL fsync — NOT a crash), restart same data dir with
//!   `TIDAL_VERSION_TAG=N+1`, `wait_healthy`, `heal_until_converged_busy` (the leader's
//!   gRPC circuit breaker opened while the node was down — threshold 5, reset 30s — so
//!   one heal can ship into an open breaker; we re-issue exactly like a runbook
//!   operator, tolerating transient 408/429 from the pool-backed heal route under the
//!   concurrent writer).
//! * Drain the writer (and read the zero-loss tally).
//! * Leader LAST: `/cluster/promote` an already-upgraded follower, `wait_leader_agreed`
//!   on the live nodes, then graceful-restart the OLD leader the same way. Because the
//!   old leader's topology file still names region 0 as leader, it boots with a STALE
//!   leader view (a transient split brain) — so we re-promote the chosen leader after
//!   it rejoins and `wait_leader_agreed` on ALL three, the real operator step.
//! * Heal each restarted node (the runbook recovery) and assert the WAL relay ALONE
//!   reconverges it to the leader's ranking — NO `/cluster/reconcile` in the
//!   convergence path. A final `/cluster/reconcile` round is then asserted to be a
//!   true no-op on scores (the anti-entropy fixpoint, now reached immediately).
//!
//! Final asserts: the cluster converges to identical state via the WAL relay alone
//! (feed/decay parity to 1e-6 across all three — no loss, no duplication), the final
//! reconcile is an exact no-op (no creep), zero ACKNOWLEDGED logical writes lost
//! (the spec's hard guarantee, criterion 7: `lost == 0`), the leader is the promoted
//! node, and each restarted node served its pre-restart items after restart (WAL
//! recovery presence).
//!
//! ### Convergence is driven by the WAL RELAY (three reported bugs, now FIXED)
//!
//! The spec (docs/specs/14-scale-architecture.md §"consistency guarantees") makes signal
//! aggregates EVENTUAL (bounded-staleness), recovered by anti-entropy; the hard
//! guarantee is "no ACKNOWLEDGED signal events lost" (criterion 7). The m8p10 tier-3 UAT
//! surfaced three real product bugs that meant the WAL relay alone did NOT deliver the
//! eventual convergence the model promises after a restart. All three are now FIXED, so
//! this test asserts the REAL contract: heal + the WAL relay reconverge every restarted
//! node WITHOUT `/cluster/reconcile`, and the final reconcile is a true no-op:
//!
//! 1. **Lag gauge meaningless across a leadership change — FIXED.** The lag gauge now
//!    tracks the leader high-water-mark PER SOURCE SHARD
//!    (`ReplicationLagGauge::leader_seqno_for`), and `local_status` computes lag against
//!    the CURRENT leader's shard. A fully-caught-up rejoined node reports lag 0, so
//!    `converged()` (lag 0 + applied >= leader hwm) is an honest convergence signal.
//! 2. **`reconcile_with` NOT idempotent for signal state — FIXED.** `take_crdt_snapshot`
//!    now attributes every node's signal contribution to ONE canonical replication shard
//!    (`ShardId::SINGLE`), not the local shard, so a fully-replicated event is ONE LWW
//!    register on every node — `merge` is idempotent. Reconcile of converged nodes is an
//!    exact fixpoint immediately (no 0.5 → 0.375 → … creep). The final reconcile round
//!    below asserts this directly (feeds byte-unchanged).
//! 3. **Graceful-restart item loss — FIXED twice over.** m8p10 patched it by making
//!    `/cluster/heal` re-broadcast items over HTTP; m11p2 removed the root cause:
//!    items and embeddings ride the one replicated WAL log (kind-1/2 records), so a
//!    restarted node converges by pure log catch-up — live pushes plus its boot-time
//!    `StreamSegments` pull — leaving it with EXACTLY the leader's data, proven by
//!    the per-restart parity assertion (1e-6, no reconcile, no operator verb).
//!
//! ### Write-tally semantics (the zero-loss gate)
//!
//! A "logical write" is one intended signal POST. During a restart window a request
//! may transiently fail (connection refused while the process is down; 503 when a
//! follower forwards to a leader that is restarting or holds a stale view; 429 under
//! write-pool backpressure; 408 request timeout). The writer RETRIES the SAME logical
//! write (cycling target nodes) until it 204s, counting retries; `stop` is checked only
//! at the top of each logical write so a committed write is always driven to completion.
//! The gate is `lost == 0`: every logical write eventually returns 204 (the spec's
//! "no acknowledged events lost"). `total` is the number of logical writes, `retried`
//! the number that needed >= 1 retry, `lost` the number that never succeeded within
//! [`MAX_WRITE_ATTEMPTS`] (must be 0).
//!
//! ## `RollingUpgradeCoordinator` decision
//!
//! `tidaldb::replication::upgrade::RollingUpgradeCoordinator` is an ENGINE-INTERNAL
//! drain/rejoin gate over `ShardId`s on a single in-process `ControlPlane`: `drain`
//! marks a shard not-routable (refusing to leave zero serving shards) and `rejoin`
//! clears it. It is NOT surfaced on the multi-process server's HTTP API — there is no
//! `/cluster/drain` route, and each process owns exactly one region (its own
//! `ControlPlane`), so a coordinator in process A cannot drain process B. The
//! server-level rolling upgrade is instead choreographed by the OPERATOR surface that
//! DOES exist over the network — `/cluster/promote` (move leadership off the node
//! about to restart) + graceful SIGTERM (drain in-flight, checkpoint, WAL fsync) +
//! `/cluster/heal` (redeliver segments shipped while the node was down). Asserting
//! through the in-process coordinator here would test the wrong process's state, so
//! this suite exercises the real network choreography and documents the coordinator's
//! scope in this header rather than forcing it in. (Its own unit tests in
//! `upgrade.rs` cover the drain/rejoin invariant.)
//!
//! # Budget
//!
//! Tier-3 over real OS processes. Two tests: the skew test is a boot + converge +
//! one partition/heal/reconcile window; the upgrade test is a boot + four graceful
//! restarts, each followed by a breaker-bounded heal. Worst case the upgrade test's
//! heals each wait up to the 30s breaker reset, but in practice an eager-ship probe
//! (the continuous writer, while it runs) closes the breaker far sooner. The writer is
//! paced (~5 writes/sec) so it never starves the leader's small write pool and the
//! pool-backed heal can land. Every wait is poll-with-deadline; whole-suite wall budget
//! < 5 minutes on a developer laptop.

#![cfg(feature = "cluster-e2e")]
// Tier-3 harness allows, mirroring `cluster_chaos.rs` / `cluster_multiproc.rs`:
// `unwrap` on known-good fixtures is idiomatic test noise; the lossy numeric casts
// are the same pervasive-and-intentional scoring math the crate config documents.
#![allow(
    clippy::unwrap_used,
    clippy::missing_panics_doc,
    clippy::too_many_lines,
    clippy::cast_precision_loss,
    clippy::cast_possible_truncation,
    clippy::cast_sign_loss,
    clippy::items_after_statements
)]

mod support;

/// This suite choreographs the LEGACY operator drills (manual promote,
/// deliberate split-brain via a severed self-promote, rolling restarts).
/// Auto-election is pinned OFF so the m11p4 failure detector cannot race the
/// drills — the automatic path has its own exit-gate suite
/// (`cluster_election.rs`).
const LEGACY_ELECTION_YAML: &str = "election:\n  auto_election: false";

use std::{
    sync::{
        Arc,
        atomic::{AtomicBool, AtomicU64, Ordering},
    },
    thread,
    time::{Duration, Instant},
};

use support::{
    multiproc::{
        BREAKER_RESET, ClusterOptions, MultiProcCluster, convergence_budget,
        seed_items_and_embeddings, write_view,
    },
    partition::proxied_rewrite,
};

/// Region 0 = `us-east` = the initial leader in every topology.
const LEADER: usize = 0;
/// Region 1 = `eu-west` (a follower; promoted to leader in the upgrade test).
const EU_WEST: usize = 1;
/// Region 2 = `ap-south` (a follower; the skewed-BEHIND node in the skew test).
const AP_SOUTH: usize = 2;

// The leader-ships circuit breaker ([`BREAKER_RESET`], the `tidal-net` default)
// opens after 5 consecutive failed ships and stays open for 30s. After a node
// returns, the leader's transport cannot reach it until the breaker half-opens
// and one probe ship closes it — so a SINGLE `/cluster/heal` can ship into the
// open breaker and be a no-op. Tests bound post-return convergence at
// BREAKER_RESET + the convergence budget, re-issuing heal exactly like a
// runbook operator. (Same reasoning as `cluster_chaos.rs`.)

// ── Shared seeding / feed helpers (canonical bodies in support::multiproc) ──────

/// A node's local-region feed as a sorted `(entity_id, score)` vector.
fn feed_pairs(
    cluster: &MultiProcCluster,
    idx: usize,
    profile: &str,
    limit: u32,
) -> Vec<(u64, f64)> {
    let body = cluster.get_json(idx, &format!("/feed?profile={profile}&limit={limit}"));
    let mut pairs: Vec<(u64, f64)> = body["items"]
        .as_array()
        .unwrap_or(&Vec::new())
        .iter()
        .map(|it| {
            (
                it["entity_id"].as_u64().unwrap(),
                it["score"].as_f64().unwrap(),
            )
        })
        .collect();
    pairs.sort_by_key(|(id, _)| *id);
    pairs
}

/// A user-scoped feed's item-id set (sorted). Hard negatives for `user_id` are
/// filtered by the engine's user-context stage, so a hidden item is ABSENT here.
fn feed_item_ids_for_user(
    cluster: &MultiProcCluster,
    idx: usize,
    user_id: u64,
    profile: &str,
    limit: u32,
) -> Vec<u64> {
    let body = cluster.get_json(
        idx,
        &format!("/feed?profile={profile}&limit={limit}&user_id={user_id}"),
    );
    let mut ids: Vec<u64> = body["items"]
        .as_array()
        .unwrap_or(&Vec::new())
        .iter()
        .map(|it| it["entity_id"].as_u64().unwrap())
        .collect();
    ids.sort_unstable();
    ids
}

/// Assert two feed views carry the SAME items with scores equal to 1e-6.
fn assert_feed_parity(label: &str, a: &[(u64, f64)], b: &[(u64, f64)]) {
    assert_eq!(
        a.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
        b.iter().map(|(id, _)| *id).collect::<Vec<_>>(),
        "{label}: feed item sets differ"
    );
    for ((id_a, score_a), (_, score_b)) in a.iter().zip(b.iter()) {
        assert!(
            (score_a - score_b).abs() <= 1e-6,
            "{label}: score for item {id_a} differs: {score_a} vs {score_b}"
        );
    }
}

/// One node's `applied_events` from its OWN `/cluster/status/local` (direct addr).
fn applied(cluster: &MultiProcCluster, idx: usize) -> u64 {
    cluster
        .local_status(idx)
        .and_then(|st| st["applied_events"].as_u64())
        .unwrap_or(0)
}

/// True when every `followers` entry has applied up to (or past) the leader's
/// high-water-mark with zero lag. Non-panicking, so a retry loop can poll it.
fn converged(cluster: &MultiProcCluster, followers: &[usize]) -> bool {
    let Some(target) = cluster.leader_last_seq() else {
        return false;
    };
    followers.iter().all(|&idx| {
        cluster.local_status(idx).is_some_and(|st| {
            let applied = st["applied_events"].as_u64().unwrap_or(0);
            let lag = st["lag_events"].as_u64().unwrap_or(u64::MAX);
            lag == 0 && applied >= target
        })
    })
}

/// Drive convergence under CONCURRENT write load the way an operator does: re-issue
/// `POST /cluster/heal` for `region` until every `followers` entry converges, allowing
/// for the circuit-breaker reset window. The rolling-upgrade writer keeps hammering the
/// leader, so its write pool — and the axum request-timeout layer in front of the
/// pool-backed heal route — can make a `/cluster/heal` return 408 (request timeout) or
/// 429 (pool saturated) instead of 200. That is genuine production behavior, not a bug:
/// an operator re-issues the heal. So (unlike the chaos suite's strict heal, which runs
/// with no concurrent writer and asserts a 200) this TOLERATES a transient non-200 heal
/// and simply re-issues, bounded by the breaker reset + convergence budget; a heal that
/// DOES 200 is the one that redelivered. We always confirm convergence on STATE.
fn heal_until_converged_busy(cluster: &MultiProcCluster, region: &str, followers: &[usize]) {
    let deadline = Instant::now() + BREAKER_RESET + convergence_budget();
    loop {
        // Re-issue heal; a 408/429/503 under write load is a no-op we retry, not a
        // failure. Only a 4xx that is NOT timeout/backpressure (e.g. 400 unknown
        // region) would be a real bug — but the region name is always valid here.
        let status = cluster
            .post(
                LEADER,
                "/cluster/heal",
                &serde_json::json!({ "region": region }),
            )
            .status()
            .as_u16();
        assert!(
            matches!(status, 200 | 408 | 429 | 503),
            "/cluster/heal returned an unexpected status {status} (expected 200, or a \
             transient 408/429/503 under write load)"
        );
        let check_deadline = Instant::now() + Duration::from_secs(3);
        while Instant::now() <= check_deadline {
            if converged(cluster, followers) {
                return;
            }
            thread::sleep(Duration::from_millis(100));
        }
        assert!(
            Instant::now() <= deadline,
            "region '{region}' did not converge under write load within budget; \
             leader hwm={:?}, follower applied={:?}",
            cluster.leader_last_seq(),
            followers
                .iter()
                .map(|&i| applied(cluster, i))
                .collect::<Vec<_>>()
        );
    }
}

/// Poll `pred()` until true or `budget` elapses, asserting with `msg` on timeout.
fn poll_until(budget: Duration, msg: &str, mut pred: impl FnMut() -> bool) {
    let deadline = Instant::now() + budget;
    while !pred() {
        assert!(Instant::now() <= deadline, "timed out: {msg}");
        thread::sleep(Duration::from_millis(50));
    }
}

// ── Clock-skew reconciliation ────────────────────────────────────────────────

#[test]
fn mp_clock_skew_reconciliation_stays_causal() {
    // Skewed-AHEAD leader, on-time follower, skewed-BEHIND follower. The env var
    // genuinely offsets each process's HLC (parsed i64 in main.rs → engine), so
    // this is real ±500ms skew, not a mock.
    // All three regions proxied so ap-south can be isolated in BOTH
    // directions (m11p4): an inbound-only sever leaves its outbound votes,
    // ships and heartbeats flowing, which turns the intended split-brain
    // into a genuine election win.
    let (rewrite, proxies) = proxied_rewrite(&["us-east", "eu-west", "ap-south"]);
    let cluster = MultiProcCluster::start_with(
        ClusterOptions::new(3).with_topology_extra(LEGACY_ELECTION_YAML)
            .with_env(LEADER, "TIDAL_HLC_SKEW_MS", "500")
            .with_env(EU_WEST, "TIDAL_HLC_SKEW_MS", "0")
            .with_env(AP_SOUTH, "TIDAL_HLC_SKEW_MS", "-500")
            .with_rewrite(rewrite),
    );

    const ITEMS: u64 = 12;
    /// The user whose hide the AHEAD leader records (on pair A).
    const AHEAD_USER: u64 = 11;
    /// The item the AHEAD leader (+500ms) hides.
    const AHEAD_ITEM: u64 = 4;
    /// The user whose hide the BEHIND split-brain node records (on pair B).
    const BEHIND_USER: u64 = 22;
    /// The item the BEHIND follower (-500ms) hides.
    const BEHIND_ITEM: u64 = 9;

    // ── PHASE A: normal replication under ±500ms skew ──────────────────────────
    seed_items_and_embeddings(&cluster, LEADER, ITEMS);
    for entity_id in 1..=ITEMS {
        write_view(&cluster, LEADER, entity_id, entity_id as f64);
    }
    cluster.wait_converged_all(convergence_budget());

    // Decay reads the real wall clock, so ±500ms of HLC skew leaves the ranked
    // feed byte-identical across the three skewed processes. This pins the scope:
    // the skew mechanism is HLC-only (reconcile LWW), never signal decay.
    let leader_feed = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
    let eu_feed = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32);
    let ap_feed = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
    assert!(!leader_feed.is_empty(), "leader must rank the seeded items");
    assert_feed_parity(
        "leader(+500) vs eu-west(0) under skew",
        &leader_feed,
        &eu_feed,
    );
    assert_feed_parity(
        "leader(+500) vs ap-south(-500) under skew",
        &leader_feed,
        &ap_feed,
    );
    println!(
        "[skew] PHASE A: {} items rank identically (1e-6) across +500ms / 0 / -500ms processes \
         (decay is wall-clock, unaffected by HLC skew)",
        leader_feed.len()
    );

    // ── PHASE B: divergence under a real partition (split-brain hide) ──────────
    // Sever the skewed-BEHIND follower (ap-south, -500ms) from every peer.
    support::partition::isolate_region(&proxies, "ap-south", &["us-east", "eu-west"]);
    println!("[skew] PHASE B: severed ap-south (-500ms) from all peers");

    // AHEAD leader (+500ms, us-east) hides pair A. It is the real leader, so a plain
    // external /hardnegs applies locally.
    let resp = cluster.post(
        LEADER,
        "/hardnegs",
        &serde_json::json!({ "user_id": AHEAD_USER, "item_id": AHEAD_ITEM }),
    );
    assert_eq!(resp.status().as_u16(), 204, "ahead-leader hardneg must 204");

    // BEHIND follower (-500ms, ap-south): promote it to leader IN ITS OWN VIEW. The
    // fan-out to the severed peers fails (it cannot reach them), leaving a genuine
    // SPLIT BRAIN — exactly the production hazard reconciliation repairs. promote_local
    // applies unconditionally, so ap-south now answers is_leader:true on its own view.
    let ap_region = cluster.region_name(AP_SOUTH).to_string();
    let resp = cluster.post(
        AP_SOUTH,
        "/cluster/promote",
        &serde_json::json!({ "region": ap_region }),
    );
    assert_eq!(
        resp.status().as_u16(),
        200,
        "split-brain self-promote on the severed follower must 200: {}",
        resp.status()
    );
    poll_until(
        Duration::from_secs(5),
        "severed ap-south must adopt its own leader view (split brain)",
        || {
            cluster
                .local_status(AP_SOUTH)
                .and_then(|st| st["is_leader"].as_bool())
                == Some(true)
        },
    );

    // A plain external /hardnegs on the split-brain leader (ap-south) applies LOCALLY
    // (state.is_leader() is true on its own view) — a node-local divergent hide on a
    // DIFFERENT pair (B). No internal marker, no forward: a real partitioned write.
    let resp = cluster.post(
        AP_SOUTH,
        "/hardnegs",
        &serde_json::json!({ "user_id": BEHIND_USER, "item_id": BEHIND_ITEM }),
    );
    assert_eq!(
        resp.status().as_u16(),
        204,
        "split-brain follower hardneg must apply locally (204): {}",
        resp.status()
    );
    println!(
        "[skew] divergence created: +500 leader hid ({AHEAD_USER},{AHEAD_ITEM}); \
         -500 split-brain ap-south hid ({BEHIND_USER},{BEHIND_ITEM})"
    );

    // Before reconcile, each hide exists on exactly ONE node.
    assert!(
        !feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32)
            .contains(&AHEAD_ITEM),
        "ahead hide must be effective on the leader pre-reconcile"
    );
    assert!(
        feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32)
            .contains(&BEHIND_ITEM),
        "leader must NOT yet know the behind node's hide pre-reconcile"
    );

    // ── HEAL + collapse the split brain, then reconcile BOTH directions ────────
    support::partition::rejoin_region(&proxies, "ap-south", &["us-east", "eu-west"]);
    // Re-promote the real leader (us-east) to collapse the split brain — the runbook
    // step after a partitioned node rejoins. The durable Tag::HardNeg rows survive the
    // leadership change (store rows, not leader state), so the behind hide is intact.
    let leader_region = cluster.region_name(LEADER).to_string();
    let resp = cluster.post(
        LEADER,
        "/cluster/promote",
        &serde_json::json!({ "region": leader_region }),
    );
    assert_eq!(
        resp.status().as_u16(),
        200,
        "re-promote real leader must 200"
    );
    cluster.wait_leader_agreed(&leader_region, Duration::from_secs(15));
    println!("[skew] healed + collapsed split brain (leader = {leader_region})");

    // Reconcile leader -> ap-south (the call exchanges snapshots bidirectionally:
    // ships ours into ap-south's merge AND applies ap-south's back). Assert < 100ms.
    let (l_ms, r_ms) = reconcile(&cluster, LEADER, &ap_region);
    assert!(
        l_ms < 100 && r_ms < 100,
        "reconcile under skew must be < 100ms both sides: leader-side local={l_ms}ms remote={r_ms}ms"
    );
    println!("[skew] reconcile leader->ap-south: local={l_ms}ms remote={r_ms}ms (both < 100ms)");

    // Reconcile the OTHER direction (ap-south -> leader) to prove symmetry under skew.
    let (l_ms2, r_ms2) = reconcile(&cluster, AP_SOUTH, &leader_region);
    assert!(
        l_ms2 < 100 && r_ms2 < 100,
        "reverse reconcile under skew must be < 100ms both sides: local={l_ms2}ms remote={r_ms2}ms"
    );
    println!("[skew] reconcile ap-south->leader: local={l_ms2}ms remote={r_ms2}ms (both < 100ms)");

    // ── CONVERGENCE: both hides effective on BOTH nodes, LWW deterministic ─────
    assert_converged_hide(
        &cluster,
        LEADER,
        AP_SOUTH,
        AHEAD_USER,
        AHEAD_ITEM,
        "ahead hide",
    );
    assert_converged_hide(
        &cluster,
        LEADER,
        AP_SOUTH,
        BEHIND_USER,
        BEHIND_ITEM,
        "behind hide",
    );
    println!(
        "[skew] converged: both hides effective on leader AND ap-south despite ±500ms skew \
         (skewed-behind node's hide NOT lost; LWW deterministic)"
    );

    // ── STABILITY: a repeat reconcile leaves the STATE unchanged ───────────────
    // (ops_applied may be > 0 by design — the LWW plan re-resolves the same hides —
    // so we assert on STATE, never on ops_applied: no timestamp oscillation.)
    let leader_before = (
        feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32),
        feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32),
    );
    let ap_before = (
        feed_item_ids_for_user(&cluster, AP_SOUTH, AHEAD_USER, "trending", ITEMS as u32),
        feed_item_ids_for_user(&cluster, AP_SOUTH, BEHIND_USER, "trending", ITEMS as u32),
    );
    let (l_ms3, r_ms3) = reconcile(&cluster, LEADER, &ap_region);
    assert!(
        l_ms3 < 100 && r_ms3 < 100,
        "repeat reconcile must stay < 100ms both sides: local={l_ms3}ms remote={r_ms3}ms"
    );
    let leader_after = (
        feed_item_ids_for_user(&cluster, LEADER, AHEAD_USER, "trending", ITEMS as u32),
        feed_item_ids_for_user(&cluster, LEADER, BEHIND_USER, "trending", ITEMS as u32),
    );
    let ap_after = (
        feed_item_ids_for_user(&cluster, AP_SOUTH, AHEAD_USER, "trending", ITEMS as u32),
        feed_item_ids_for_user(&cluster, AP_SOUTH, BEHIND_USER, "trending", ITEMS as u32),
    );
    assert_eq!(
        leader_before, leader_after,
        "repeat reconcile changed leader hide state"
    );
    assert_eq!(
        ap_before, ap_after,
        "repeat reconcile changed ap-south hide state"
    );
    println!(
        "[skew] STABLE: repeat reconcile left both nodes' hide state identical (no oscillation)"
    );
}

/// Drive `POST /cluster/reconcile` from `from_idx` against `with_region`, asserting a
/// 200, and return `(local_elapsed_ms, remote_elapsed_ms)`.
fn reconcile(cluster: &MultiProcCluster, from_idx: usize, with_region: &str) -> (u64, u64) {
    let resp = cluster.post(
        from_idx,
        "/cluster/reconcile",
        &serde_json::json!({ "region": with_region }),
    );
    assert_eq!(
        resp.status().as_u16(),
        200,
        "reconcile from node {from_idx} with '{with_region}' must 200: {}",
        resp.status()
    );
    let body: serde_json::Value = resp.json().unwrap();
    assert_eq!(
        body["ok"].as_bool(),
        Some(true),
        "reconcile ok must be true: {body}"
    );
    (
        body["local_elapsed_ms"].as_u64().unwrap(),
        body["remote_elapsed_ms"].as_u64().unwrap(),
    )
}

/// Assert the WAL relay ALONE (heal redelivery + item backfill — no
/// `/cluster/reconcile`) reconverges `follower` to the current leader's ranking to
/// 1e-6. The background writer is still running, so the leader's feed moves between
/// reads; we capture leader then follower back-to-back and retry until a snapshot
/// pair matches, bounded by the convergence budget. A persistent mismatch is a real
/// loss the WAL relay failed to close (bug 3), surfaced — not masked.
fn assert_wal_relay_parity_to_leader(
    cluster: &MultiProcCluster,
    follower: usize,
    limit: u32,
    label: &str,
) {
    let leader_idx = current_leader_idx(cluster);
    let deadline = Instant::now() + convergence_budget();
    loop {
        // Capture the follower first, then the leader: if they match, the follower is
        // at least as current as a leader snapshot taken AFTER it, so the relay has
        // delivered everything up to that point (a conservative, race-safe check).
        let f = feed_pairs(cluster, follower, "trending", limit);
        let l = feed_pairs(cluster, leader_idx, "trending", limit);
        let matched = f.len() == l.len()
            && f.iter()
                .zip(l.iter())
                .all(|((fi, fs), (li, ls))| fi == li && (fs - ls).abs() <= 1e-6);
        if matched {
            println!(
                "[upgrade] {label}: WAL relay alone reconverged to the leader's ranking \
                 ({} items, 1e-6) — NO reconcile needed",
                f.len()
            );
            return;
        }
        assert!(
            Instant::now() <= deadline,
            "{label}: WAL relay alone did NOT reconverge the restarted node to the leader \
             within budget (bug 3 would manifest here as a permanent gap); \
             follower={f:?} leader={l:?}"
        );
        thread::sleep(Duration::from_millis(100));
    }
}

/// Index of the region every node currently agrees is the leader (from the leader's
/// own local status). Falls back to `LEADER` if unresolved.
fn current_leader_idx(cluster: &MultiProcCluster) -> usize {
    for i in 0..cluster.len() {
        if let Some(name) = cluster
            .local_status(i)
            .and_then(|st| st["leader"].as_str().map(str::to_string))
            && let Some(idx) = (0..cluster.len()).find(|&j| cluster.region_name(j) == name)
        {
            return idx;
        }
    }
    LEADER
}

/// Whether all three nodes' `trending` feeds agree to 1e-6 (item set + scores).
fn three_way_parity(cluster: &MultiProcCluster, leader_idx: usize, limit: u32) -> bool {
    let a = feed_pairs(cluster, leader_idx, "trending", limit);
    let others: Vec<usize> = (0..cluster.len()).filter(|&i| i != leader_idx).collect();
    others.iter().all(|&idx| {
        let b = feed_pairs(cluster, idx, "trending", limit);
        a.len() == b.len()
            && a.iter()
                .zip(b.iter())
                .all(|((ia, sa), (ib, sb))| ia == ib && (sa - sb).abs() <= 1e-6)
    })
}

/// Assert a hide for `(user,item)` is effective on BOTH nodes: the control un-scoped
/// feed still ranks the item on each (proving it is a FILTER, not a disappearance),
/// while the `?user_id=` feed omits it on each.
fn assert_converged_hide(
    cluster: &MultiProcCluster,
    node_a: usize,
    node_b: usize,
    user: u64,
    item: u64,
    label: &str,
) {
    const LIMIT: u32 = 12;
    for &node in &[node_a, node_b] {
        let control: Vec<u64> = feed_pairs(cluster, node, "trending", LIMIT)
            .into_iter()
            .map(|(id, _)| id)
            .collect();
        assert!(
            control.contains(&item),
            "{label}: control un-scoped feed on node {node} must still rank item {item}: {control:?}"
        );
        let scoped = feed_item_ids_for_user(cluster, node, user, "trending", LIMIT);
        assert!(
            !scoped.contains(&item),
            "{label}: item {item} must be hidden from user {user} on node {node}: {scoped:?}"
        );
    }
}

// ── Rolling upgrade: no loss, no stall ───────────────────────────────────────

/// A background writer's tally. `total` logical writes, `retried` that needed >= 1
/// retry, `lost` that never landed (the zero-loss gate asserts `lost == 0`).
struct WriterTally {
    total: AtomicU64,
    retried: AtomicU64,
    lost: AtomicU64,
}

impl WriterTally {
    const fn new() -> Self {
        Self {
            total: AtomicU64::new(0),
            retried: AtomicU64::new(0),
            lost: AtomicU64::new(0),
        }
    }

    fn snapshot(&self) -> (u64, u64, u64) {
        (
            self.total.load(Ordering::Relaxed),
            self.retried.load(Ordering::Relaxed),
            self.lost.load(Ordering::Relaxed),
        )
    }
}

#[test]
fn mp_rolling_upgrade_no_loss_no_stall() {
    // Persistent data dirs are guaranteed by the harness (it always passes
    // --data-dir and reuses it on restart). Boot "version N". The cluster handle
    // stays owned on THIS thread (restarts need &mut self); the writer thread gets
    // only the stable node base URLs + its own client, so the two never alias.
    let mut cluster = MultiProcCluster::start_with(
        ClusterOptions::new(3).with_topology_extra(LEGACY_ELECTION_YAML)
            .with_env(LEADER, "TIDAL_VERSION_TAG", "N")
            .with_env(EU_WEST, "TIDAL_VERSION_TAG", "N")
            .with_env(AP_SOUTH, "TIDAL_VERSION_TAG", "N"),
    );

    const ITEMS: u64 = 16;
    seed_items_and_embeddings(&cluster, LEADER, ITEMS);
    // Steady-state warmup so every node shares a converged baseline before the
    // continuous writer + restarts begin.
    for entity_id in 1..=ITEMS {
        write_view(&cluster, LEADER, entity_id, 1.0);
    }
    cluster.wait_converged_all(convergence_budget());

    // Spot-check fixture: an item written on the leader BEFORE any restart, used to
    // prove WAL recovery (each restarted node still serves it AFTER its restart).
    let pre_restart_feed = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
    let pre_restart_ids: Vec<u64> = pre_restart_feed.iter().map(|(id, _)| *id).collect();
    assert!(
        !pre_restart_ids.is_empty(),
        "leader must rank seeded items pre-restart"
    );

    // ── Background writer: continuous round-robin POST /signals across ALL nodes ─
    // The writer holds only the node URLs (ports are stable across restarts) and its
    // own blocking client — fully decoupled from the &mut harness on this thread.
    let node_urls: Vec<String> = (0..cluster.len()).map(|i| cluster.node(i)).collect();
    let tally = Arc::new(WriterTally::new());
    let stop = Arc::new(AtomicBool::new(false));
    let writer = {
        let tally = Arc::clone(&tally);
        let stop = Arc::clone(&stop);
        thread::spawn(move || run_writer(&node_urls, &tally, &stop, ITEMS))
    };

    // Let the writer establish a cadence before the first restart.
    thread::sleep(Duration::from_millis(300));

    // ── Upgrade FOLLOWERS one at a time (graceful SIGTERM → restart N+1 → heal) ──
    for &follower in &[EU_WEST, AP_SOUTH] {
        let region = cluster.region_name(follower).to_string();
        println!("[upgrade] gracefully stopping follower {region} for upgrade to N+1");
        cluster.restart_graceful(follower, &[("TIDAL_VERSION_TAG", "N+1")]);
        // The leader's breaker opened while this follower was down; drive heal through
        // the reset exactly like an operator — tolerating transient 408/429 from the
        // pool-backed heal route under the concurrent writer.
        heal_until_converged_busy(&cluster, &region, &[follower]);
        println!("[upgrade] follower {region} upgraded to N+1; WAL-relay caught up (lag 0)");

        // WAL recovery: the just-restarted follower still SERVES the pre-restart items
        // (presence).
        let post_pairs = feed_pairs(&cluster, follower, "trending", ITEMS as u32);
        let post_ids: Vec<u64> = post_pairs.iter().map(|(id, _)| *id).collect();
        for id in &pre_restart_ids {
            assert!(
                post_ids.contains(id),
                "{region}: pre-restart item {id} lost after restart (WAL recovery): {post_ids:?}"
            );
        }

        // STRONGER CONTRACT (bugs 1+3 fixed): heal alone — `/cluster/heal` redelivers
        // the missed signal segments AND backfills the missed item metadata, with an
        // honest lag gauge — must reconverge the restarted follower to the LEADER's
        // ranking WITHOUT any `/cluster/reconcile` in the path. We poll the follower's
        // feed against the leader's to 1e-6 (the writer is still running, so the leader
        // moves; poll until a snapshot pair matches). This is the eventual-consistency
        // guarantee the WAL relay now delivers on its own.
        assert_wal_relay_parity_to_leader(&cluster, follower, ITEMS as u32, &region);
    }

    // ── Drain the writer before the leadership handoff ──────────────────────────
    // The two FOLLOWER upgrades above ran under continuous write load (the whole point:
    // no lost LOGICAL writes during a restart — the lost==0 gate below). We drain before
    // the leader handoff so the handoff happens over a quiescent cluster (faithful
    // operator practice for a planned failover) and so the final anti-entropy reconcile
    // snapshots a stable state.
    stop.store(true, Ordering::Relaxed);
    writer.join().expect("writer thread joined");
    let (total, retried, lost) = tally.snapshot();
    println!(
        "[upgrade] writer drained before handoff; tally: total={total} retried={retried} lost={lost}"
    );

    // ── Leader LAST: promote an upgraded follower, then restart the old leader ───
    let new_leader = cluster.region_name(EU_WEST).to_string();
    println!("[upgrade] promoting upgraded follower {new_leader} before restarting the old leader");
    let resp = cluster.post(
        AP_SOUTH,
        "/cluster/promote",
        &serde_json::json!({ "region": new_leader }),
    );
    assert_eq!(
        resp.status().as_u16(),
        200,
        "promote must 200: {}",
        resp.status()
    );
    // The old leader is still up here, so it acks the fan-out and every LIVE node
    // agrees eu-west leads.
    cluster.wait_leader_agreed(&new_leader, Duration::from_secs(15));

    let old_leader = cluster.region_name(LEADER).to_string();
    println!("[upgrade] gracefully restarting the OLD leader {old_leader} to N+1");
    cluster.restart_graceful(LEADER, &[("TIDAL_VERSION_TAG", "N+1")]);
    // The restarted old leader read `leader: us-east` from its topology file, so it
    // boots with a STALE leader view (transient split brain). Re-promote the chosen
    // leader to collapse it — the real operator step after a node rejoins.
    let resp = cluster.post(
        LEADER,
        "/cluster/promote",
        &serde_json::json!({ "region": new_leader }),
    );
    assert_eq!(
        resp.status().as_u16(),
        200,
        "re-promote after old-leader rejoin must 200: {}",
        resp.status()
    );
    cluster.wait_leader_agreed(&new_leader, Duration::from_secs(15));
    println!("[upgrade] old leader {old_leader} rejoined as a follower of {new_leader}");

    // ── WAL RELAY ALONE reconverges the restarted old leader (bugs 1+3 fixed) ────
    // The new leader heals the rejoined old leader: `/cluster/heal` redelivers the
    // signal segments it missed while down AND backfills the item metadata, with an
    // honest per-shard lag gauge. The writer is drained, so the cluster is quiescent
    // (a planned-failover handoff over a quiescent cluster is faithful operator
    // practice — we drain BEFORE the handoff for exactly this reason). After heal,
    // ALL THREE feeds must agree to 1e-6 WITHOUT any `/cluster/reconcile` — the WAL
    // relay is the convergence mechanism, and reaching three-way parity (no stall)
    // with every item present (no loss) and scores equal (no duplication) is the
    // exactly-once proof. Before the bug fixes this required a multi-round reconcile
    // fixpoint loop; now the relay alone closes the gap.
    heal_until_converged_busy(&cluster, &old_leader, &[LEADER]);
    let new_leader_idx = EU_WEST;
    poll_until(
        convergence_budget(),
        "WAL relay alone must reconverge all three nodes to 1e-6 after the rolling upgrade",
        || three_way_parity(&cluster, new_leader_idx, ITEMS as u32),
    );
    println!(
        "[upgrade] WAL relay alone reconverged all three nodes (1e-6) — NO reconcile in the \
         convergence path"
    );

    // ── Final reconcile is a TRUE no-op (bug 2 fixed: idempotent, fixpoint at once) ─
    // Capture each node's feed, drive ONE pairwise reconcile round in BOTH directions,
    // and assert every node's feed is BYTE-for-byte unchanged. Before bug 2 the signal
    // merge summed already-replicated per-node contributions, so reconcile of converged
    // nodes crept the scores (0.5 → 0.375 → …). Now `merge` is idempotent: reconcile of
    // converged nodes is an exact fixpoint immediately, no creep.
    let before_leader = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32);
    let before_us = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
    let before_ap = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
    let us_region = cluster.region_name(LEADER).to_string();
    let ap_region = cluster.region_name(AP_SOUTH).to_string();
    let _ = reconcile(&cluster, EU_WEST, &us_region);
    let _ = reconcile(&cluster, LEADER, &new_leader);
    let _ = reconcile(&cluster, EU_WEST, &ap_region);
    let _ = reconcile(&cluster, AP_SOUTH, &new_leader);
    assert_feed_parity(
        "final reconcile no-op: eu-west unchanged",
        &before_leader,
        &feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32),
    );
    assert_feed_parity(
        "final reconcile no-op: us-east unchanged",
        &before_us,
        &feed_pairs(&cluster, LEADER, "trending", ITEMS as u32),
    );
    assert_feed_parity(
        "final reconcile no-op: ap-south unchanged",
        &before_ap,
        &feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32),
    );
    println!("[upgrade] final reconcile is an exact no-op on scores (bug 2 fixed: true fixpoint)");

    // ── Final state: exactly-once (no stall, no loss), leader is the promoted node ─
    let leader_feed = feed_pairs(&cluster, EU_WEST, "trending", ITEMS as u32); // current leader
    let f1 = feed_pairs(&cluster, LEADER, "trending", ITEMS as u32);
    let f2 = feed_pairs(&cluster, AP_SOUTH, "trending", ITEMS as u32);
    assert_feed_parity("post-upgrade leader(eu-west) vs us-east", &leader_feed, &f1);
    assert_feed_parity(
        "post-upgrade leader(eu-west) vs ap-south",
        &leader_feed,
        &f2,
    );
    println!(
        "[upgrade] exactly-once + no stall: {} items rank identically (1e-6) across all three upgraded nodes",
        leader_feed.len()
    );

    // No acknowledged writes lost (the spec's hard guarantee, criterion 7): every
    // logical write the writer issued returned 204 exactly once.
    assert_eq!(
        lost, 0,
        "rolling upgrade LOST {lost} acknowledged logical writes (of {total})"
    );
    assert!(
        total > 0,
        "writer must have issued writes during the upgrade window"
    );
    assert!(
        retried > 0,
        "expected >= 1 retried write across the restart windows; got {retried} (window too quiet?)"
    );

    // WAL recovery proven for the restarted old leader: an item written BEFORE its
    // restart is still served by it AFTER (and now ranks identically to the cluster).
    let old_leader_ids: Vec<u64> = f1.iter().map(|(id, _)| *id).collect();
    for id in &pre_restart_ids {
        assert!(
            old_leader_ids.contains(id),
            "old leader {old_leader}: pre-restart item {id} lost after restart (WAL recovery): {old_leader_ids:?}"
        );
    }

    // The new leader is the promoted node.
    let leader_now = cluster
        .local_status(EU_WEST)
        .and_then(|st| st["leader"].as_str().map(str::to_string));
    assert_eq!(
        leader_now.as_deref(),
        Some(new_leader.as_str()),
        "final leader must be the promoted node {new_leader}"
    );
}

/// Per logical write, the maximum number of (node-cycling) attempts before we give
/// up and count it LOST. Generous: four restart windows, each bounded by boot +
/// breaker reset, so a healthy cluster lands a write in a handful of attempts; this
/// cap only fires on a genuine, persistent failure (the exact bug the gate catches).
const MAX_WRITE_ATTEMPTS: u64 = 400;

/// Inter-write pacing for the background writer. A rolling upgrade rides STEADY
/// production traffic, not a saturation storm: the leader's cluster write pool has
/// only `MIN_WRITE_WORKERS = 2` slots, shared by signal writes AND the pool-backed
/// `/cluster/heal` redelivery. Pacing leaves headroom so a heal can land (otherwise
/// the writer starves both worker slots and the recovery path never runs — a
/// test-induced deadlock, not a product behavior). ~50 writes/sec is plenty of
/// continuous traffic to overlap every restart window with live writes.
const WRITE_PACING: Duration = Duration::from_millis(200);

/// The background writer: round-robins a `view` signal across the node base URLs,
/// retrying the SAME logical write (cycling targets) until it 204s. Tolerates the
/// failure modes a restart window produces — connection refused while a process is
/// down, 503 (`NotLeader` / leader-unreachable while forwarding), 429 (write-pool
/// backpressure), 408 (request timeout).
///
/// `stop` is checked only at the TOP of each logical write, so a write the writer has
/// COMMITTED to (incremented `total`) is always driven to completion or to `lost`,
/// never silently abandoned — the test settles a healthy steady-state window before
/// flipping `stop`, so the last committed write lands. A logical write that never
/// 204s within [`MAX_WRITE_ATTEMPTS`] is counted LOST (the zero-loss gate then fails,
/// as it should). Uses its OWN blocking client + the stable node URLs, so it never
/// aliases the &mut harness on the test thread.
fn run_writer(node_urls: &[String], tally: &WriterTally, stop: &AtomicBool, items: u64) {
    let client = reqwest::blocking::Client::builder()
        .build()
        .expect("build writer client");
    let mut node = 0usize;
    let mut entity = 1u64;
    while !stop.load(Ordering::Relaxed) {
        tally.total.fetch_add(1, Ordering::Relaxed);
        let mut attempts = 0u64;
        let landed = loop {
            let target = &node_urls[node % node_urls.len()];
            let resp = client
                .post(format!("{target}/signals"))
                .json(&serde_json::json!({ "entity_id": entity, "signal": "view", "weight": 1.0 }))
                .timeout(Duration::from_secs(3))
                .send();
            if let Ok(r) = resp
                && r.status().as_u16() == 204
            {
                break true;
            }
            attempts += 1;
            if attempts >= MAX_WRITE_ATTEMPTS {
                break false; // genuine persistent failure → LOST
            }
            // Cycle to another node (the target may be the one restarting; a live node
            // forwards to the current leader) and retry the SAME logical write.
            node += 1;
            thread::sleep(Duration::from_millis(25));
        };
        if landed {
            if attempts > 0 {
                tally.retried.fetch_add(1, Ordering::Relaxed);
            }
        } else {
            tally.lost.fetch_add(1, Ordering::Relaxed);
        }
        node += 1;
        entity = (entity % items) + 1;
        // Steady-traffic pacing: leave write-pool headroom for the pool-backed heal.
        thread::sleep(WRITE_PACING);
    }
}