# m11p4 — Failure Detection, Election, Fencing (COMPLETE — 2026-06-12) Phase spec and exit gate: [docs/roadmap-to-cluster.md §4/m11p4](../../roadmap-to-cluster.md). Closes ROADMAP gap **G5** ("operator-driven failover; restarted node believes the static topology — split-brain seed", incident §1.4-1). Predecessors: [phase-1.md](phase-1.md) (ship queue), [phase-2.md](phase-2.md) (one replicated log), [phase-3.md](phase-3.md) (quorum machinery — the prerequisite this phase's election rides on). **Already shipped in this phase** (d0a52e4, 2026-06-11): catch-up timer retry (re-arm-on-skip liveness) + TSEG segment version header + structured "snapshot required" refusal. This document covers the remaining XL work. **Goal:** "a machine died" is a non-event. Kill the leader under load → writes resume <10 s p99 with zero operator action and zero acknowledged loss; a restarted or partitioned ex-leader cannot accept a single write; election churn under flapping links is bounded. ## Design (as adopted) ### 0. Open question #2 resolved: purpose-built election, no raft crate The roadmap deferred "minimal purpose-built Raft over `tidal-net` vs. a vetted raft crate" to this phase. Decision: **purpose-built election-only Raft** (Raft §5.1–§5.4 election subset + pre-vote + check-quorum + leadership transfer), for one structural reason: **the replicated log already exists.** m11p2 made the WAL the stream; m11p3 built the quorum commit index over it. A raft crate (openraft, raft-rs) owns its own log and storage contracts — adopting one means either running a second log beside the WAL or contorting the crate to delegate its log to ours. The state that actually needs consensus is tiny — `(term, leader)` — and the safety-critical machinery (durable votes, the up-to-date restriction, term fencing) maps directly onto primitives we already have (`flushed_seq()`, the `stream_baseline` persistence pattern, the commit-index epoch). Membership stays static this phase (the topology file); m11p5 makes it dynamic through this phase's metadata machinery. The election state machine is **pure and deterministic** (no clocks, no I/O — inputs carry `Instant`s, randomness is an injected seed), so the entire protocol is unit-testable without processes or sleeps. ### 1. The durable election state (kills restart amnesia) A small per-node file, `data_dir/election_state`, written with the exact `stream_baseline` discipline (write `.tmp`, `sync_file_durable` [F_FULLFSYNC on macOS], atomic rename, `sync_dir_durable`), carrying a magic + format version + checksum so corruption is detectable: | Field | Type | Meaning | |---|---|---| | `current_term` | u64 | Highest term this node has seen. **Strictly monotonic** — a term is adopted only upward; no inbound message can ever lower it. | | `voted_for` | Option | Vote cast in `current_term` (at most one). | The WAL-tail term deliberately does NOT live in this file — see §1b. **Persistence is a fence, not bookkeeping**: the file is fsynced BEFORE any externally visible action that depends on it — before a vote response is sent, before a higher term is acknowledged, before leadership is assumed. Votes and term changes are rare (not per-write), so the fsync cost is irrelevant. All election-state mutations serialize under one lock, and the boot-time persist completes before the HTTP routes serve a single request (design-review C6). **Boot rule** (the incident §1.4-1 fix), four cases (design-review C1): 1. File exists and reads clean → the node **always boots as a follower**, regardless of what the file or the topology says — leadership is learned from heartbeats or won by election, never assumed from a restart. 2. File exists but is corrupt (bad magic/version/checksum) → **refuse to boot** with a FATAL error naming the file. A node that cannot prove its term knowledge must not guess at term 0 — guessing re-opens the §1.4-1 amnesia hole through the side door of a disk fault. 3. File absent but the data dir is NOT fresh (a WAL exists) → the file was deleted out from under a node that has run before. Boot **as a follower** at term 0 with a loud ERROR (never assume topology leadership), and persist the recreated state immediately. Terms recover upward from the first heartbeat/ship the node hears. 4. File absent and the data dir is fresh → a genuinely new node: boot with the topology's term-0 roles exactly as today, persisting `{term: 0}` before serving. Existing deployments boot byte-for-byte identically on first start. **Term-0 conflicting-leader guard** (design-review C5): a term-0 leader that receives another node's term-0 *leadership assertion* (heartbeat or ship stamped `term=0, leader=other`) — possible only through mismatched topology files, a deployment error — logs ERROR and starts an election. Term ≥ 1 leadership is unique by votes; the election converges the misdeployment instead of letting two config-believing leaders run open-loop. ### 1b. WAL term markers (the up-to-date restriction, crash-atomic) Raft's vote restriction compares `(lastLogTerm, lastLogIndex)`. The first design draft persisted a `last_applied_term` in the election file; the adversarial review (C7) found the killer: that file and the WAL frontier can never be made crash-atomic — a crash between the two persists yields a node whose up-to-date claim lies in the **unsafe** direction (it can win an election while missing committed entries). Fix: the term lives **in the log itself**. On becoming leader of term T (any T ≥ 1), the leader journals a `TermMarker(T)` record — a new small WAL record kind beside m11p2's item/embedding blobs — as its first term-T entry. It replicates to every follower through the normal stream like any other record; followers fold it by recording the term (no storage effect). The node's **WAL-tail term** is then the term of the last marker at or below its durable frontier — recovered from the WAL on boot, advanced on apply, and *atomic with the frontier by construction* (they are the same fsync stream). No marker (a pre-election log) reads as term 0. The vote comparison becomes `(wal_tail_term, durable_frontier)`, lexicographic, both read from the same WAL state. A crash at any point yields a node that understates — never overstates — its log, which is the safe direction (it may lose an election it deserved; it can never win one it shouldn't, because every committed entry's quorum contains an un-crashed holder whose refusal stands). Wire compatibility falls out of "markers only exist at term ≥ 1": a p4 binary on a term-0 cluster never writes one, so a rolling m11p3→m11p4 upgrade ships zero unknown record kinds; the first election (only possible once a vote quorum of upgraded binaries exists) is the moment the format becomes live. Runbook: complete the binary upgrade before relying on auto-failover. ### 2. Term model: term 0 is the topology, terms ≥ 1 are elected - **Term 0** = the static topology file's leader. A fresh cluster boots leading/following at term 0 with no election, preserving every existing boot path and test. - **Any election or fenced transfer moves the cluster to term ≥ 1.** From then on the durable state is authoritative and the topology's `leader` field is dead weight (kept for bootstrap only). - Proto compatibility falls out: proto3 defaults absent term fields to 0, so a pre-m11p4 peer's ships/reports read as term-0 traffic. A node still at term 0 interoperates with them; a node at term ≥ 1 rejects them (`FAILED_PRECONDITION`) — exactly the fencing semantics, applied to version skew. ### 3. Election protocol Roles: `Follower → PreCandidate → Candidate → Leader` (and anything → `Follower` on observing a higher term). - **Failure detector**: the leader sends `Heartbeat` to every peer each `heartbeat_interval` (default 500 ms), now carrying `(term, leader_region, stream_baseline)`. A follower that hears nothing valid for a randomized `election_timeout` ∈ [1500 ms, 3000 ms) starts a **pre-vote**. - **Pre-vote** (disruption guard): asks "would you vote for me in term+1?" without bumping any term. A voter refuses if it heard from a live leader within `election_timeout_min` (lease check) or if the candidate's log loses the up-to-date comparison. Only a pre-vote majority proceeds — a partitioned or flapping node cannot inflate terms or depose a healthy leader from the outside. - **Vote**: candidate persists `term+1, voted_for=self`, then requests votes. A voter grants iff `req.term ≥ current_term`, it hasn't voted for another candidate in that term (durable before reply), and the candidate is **up-to-date**: `(wal_tail_term, durable_frontier) ≥` the voter's, compared lexicographically — Raft's §5.4.1 restriction with both values derived from the voter's own WAL (§1b). Majority → leader. - **Becoming leader**: persist election state, then journal the `TermMarker(T)` (§1b), then the promote mechanics verbatim: baseline = own `flushed_seq()` (now ≥ the marker's seqno), `persist_stream_baseline`, `ship_queue.activate_from(baseline, term)` (which activates the commit index *scoped to T* and bumps its epoch), leader view updated, heartbeats start. - **Step-down ordering** (design-review C3): on any step-down — higher-term contact, check-quorum loss, fenced transfer — the **leadership view flips first** (`leader = None` or the new leader), THEN the ship queue deactivates. The write path's `is_leader()` reads the view, so once it flips no new write is accepted; the queue deactivation then fails in-flight `ack=quorum` waits with the existing `Demoted` path. The residual window (a request that read the old view and stages after the flip) produces a leader-local-durable, never-shipped write — exactly the documented `ack=leader` crash contract, surfaced at rejoin as a divergent suffix (§5). - **Check-quorum / lease step-down**: a leader that cannot reach a majority of peers (heartbeat acks) within `leader_lease_ms` **steps down** (stops accepting writes; quorum waits fail). The lease bound (design-review C2): the leader's countdown starts at its last successful quorum contact while a follower's election timer starts at the last heartbeat it *received* — the two anchors differ by up to one heartbeat interval plus RTT, NOT by wall-clock offset (all timers are monotonic `Instant` intervals; clock offsets cancel). The enforced constraint is therefore `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms` (validated at topology load), guaranteeing the deposed leader stops before any successor can be elected. This bounds the minority-partitioned-leader window for `ack=leader` writes; `ack=quorum` writes were never at risk (the commit index cannot reach majority from a minority partition). - **No pre-vote deadlock** (design-review C9): the lease refusal can only outlive a real leader if something keeps *sending* heartbeats. A stepped-down leader stops heartbeating by construction (only the Leader role sends), so after a leader dies or steps down every follower's lease window drains within one `election_timeout` and pre-votes start granting. Asymmetric partitions degrade exactly one node (it pre-votes, is refused by peers that still hear the leader, and never inflates a term — the pre-vote's whole point); the majority side keeps serving throughout. - **Timers** use `std::time::Instant` exclusively — monotonic, immune to the chaos suites' `TIDAL_HLC_SKEW_MS` wall-clock skew injection (HLC remains data-plane-only for hard-negative LWW). ### 4. Fencing everywhere Every replication RPC carries the sender's term; every receiver gates: | RPC | Stamp | Stale-term reaction | Higher-term reaction | |---|---|---|---| | `ShipSegment` | `term, leader_region` | reject `FAILED_PRECONDITION` before enqueue | persist & adopt term, treat as leader contact | | `StreamSegments` request | puller's `term` | source at a higher term refuses the pull | source at a lower term steps down (it is deposed and must not serve its possibly-divergent tail — design-review C14) | | `StreamSegments` chunks | same as ship (chunks are `ShipSegmentRequest`s) | puller drops the chunk + aborts the pull (a stale source can never inject history) | adopt | | `Heartbeat` | `term, leader_region, stream_baseline` | respond `accepted=false, term` (sender steps down) | adopt, reset election timer, run the §5 term-join check | | `ReportApplied` | reporter's `term` | leader folds **only if reporter term == the commit index's activation term** (design-review C4/C8/C12: the gate reads the term the index was activated with, so a stale, future, or cross-leadership report can never advance it; heal's `resume_from` folds through the same gate — C13) | leader steps down | | `RequestVote` / `TimeoutNow` | `term` | refuse, return own term | adopt | | HTTP forward (internal-marked writes) | already rejected on a non-leader (`NotLeader` 503); the error body now names `(term, leader-or-none, leader_http_addr)` so the forwarder follows one redirect to the named leader or returns a retryable 503 "election in progress" (design-review C11) | — | — | A ship/report/vote response carrying a higher term is a **step-down signal** to the sender — the deposed leader learns its term is stale from its own outbound traffic even if it never hears a heartbeat. The `CommitIndex` itself is term-scoped: `activate(baseline, term)` records the term, `update_peer` folds only same-term reports, so every quorum input — ship-ack hints, `ReportApplied`, heal resumes — passes one gate. ### 5. Rejoin, and the divergent-suffix boundary The check runs exactly once per term, at **term-join** (the first valid leader contact carrying term T — heartbeat or ship — when adopting T), and compares the node's durable WAL frontier against the term's activation baseline B (persisted by the leader at activation and announced in every heartbeat — design-review C10: B is immutable for the term, so the check never races a moving target; a frontier that grows past B *within* term T is just normal replication, never re-checked): - **frontier ≤ B**: clean rejoin. Catch-up via the existing `StreamSegments` pull (gap-triggered, timer-retried). The node's WAL-tail term advances to T when the term's marker record applies (§1b). - **frontier > B with WAL-tail term < T**: the node holds a **divergent suffix** — writes that were leader-durable on a dead leader but never quorum-committed, while the cluster elected past them. The node **quarantines**: it refuses to apply or report (receiver halt latch — the existing `died()` health surface), logs ERROR naming the suffix range and the reseed runbook, sets `tidaldb_cluster_divergence` and stays fenced from the data plane. It still votes (its stale WAL-tail term makes it lose every up-to-date comparison against healthy peers, so it can never win). - **frontier > B with WAL-tail term == T**: impossible at term-join (T-term entries only exist after the leader's activation, and joining is what admits them) — and after joining, it is the normal shape of a follower tracking the leader. Restarts *within* a term re-join idempotently: the tail term already equals T, so no quarantine. Why quarantine instead of Raft-style suffix truncation: tidalDB followers apply segments **on receipt** (before quorum commit) — by the time divergence is discovered, the suffix's effects are already folded into storage (decayed counters), and unapplying signal arithmetic is not a thing. Truncating the WAL alone would lie about state. The honest recovery is a reseed — which is m11p5's snapshot+stream work, and exactly what the already-shipped p4 increment's `FAILED_PRECONDITION ("snapshot required")` path points at. Probability note: with RF=3 a quorum-committed suffix holder is a member of *every* election majority (the vote restriction protects it), so wrongful quarantine is impossible at n=3; the quarantined case is strictly leader-ack-only data, which is the documented leader-ack contract. ### 6. Promote becomes a fenced transfer `POST /cluster/promote {region}`: - **Leader alive** (the maintenance drain): forwarded to the current leader, which waits until the target's durable mark reaches the leader's flushed frontier (bounded; 503 naming the lag on timeout), then sends `TimeoutNow` — the target starts an immediate election (skipping pre-vote and timeout; the leader sanctioned it), wins on term+1, and the old leader steps down on the first higher-term contact. No drain-the-world: the catch-up wait IS the drain. - **Leader dead/unreachable** (the legacy failover drill): the target runs a normal pre-vote + election among survivors. With auto-election on, the operator usually never gets here — the verb remains as a manual override and a way to *choose* the successor. - **Mixed-version fallback**: if a majority of peers answer `Unimplemented` to vote RPCs (pre-m11p4 binaries), promote falls back to the legacy fan-out promote at term 0 with a loud WARN. Auto-election quietly waits until the fleet is upgraded (vote quorum impossible until then). ### 7. What goes where | Crate | Work | |---|---| | `tidal` | `replication/election.rs`: pure `ElectionState` step machine (inputs → actions) + config; `replication/election_durable.rs`: the fsync'd `(current_term, voted_for)` file with magic/version/checksum + the four-case boot classification; the `TermMarker` WAL record kind + recovery/apply tracking of the WAL-tail term (§1b); `CommitIndex.activate(baseline, term)` + same-term fold gate; `WalSegmentPayload.term/leader_region`; ship queue stamps terms (shared `Arc`); receiver-side divergence latch. | | `tidal-net` | Proto: term fields on `ShipSegmentRequest/Response`, `StreamRequest`, `HeartbeatRequest/Response`, `AppliedReport`; new `RequestVote` (with `prevote`/`transfer` flags) + `TimeoutNow` RPCs. Server: late-bound `ElectionHooks` (vote/heartbeat/observed-term) mirroring the `AppliedSink` pattern; term gates on ship/stream/report handlers. Client: `PeerPool::{request_vote, timeout_now, heartbeat}` — vote RPCs bypass the circuit breaker (an election must be able to probe a peer the breaker quarantined; vote traffic is rare and self-limiting). Config: `election` knob block. | | `tidal-server` | The **election driver** (one tokio task per region state): ticks the state machine, executes actions (persist → send → transition), owns the heartbeat fan-out and failure detection. Boot path: durable-state detection → rejoin-as-follower. Leadership transitions route through the existing `promote_local` mechanics. `leader` becomes `RwLock>` (leaderless windows are real and reported honestly — forwards during an election return retryable 503 "election in progress"). `/cluster/status` grows `term`, `role`; election metrics (`tidaldb_cluster_term`, `_role`, `_elections_started_total`, `_leader_changes_total`, `_stepdowns_total`, `_votes_{granted,denied}_total`, `_divergence`). Promote rework per §6. | | tests | Pure state-machine property/unit tests (no clocks); `tidal-net` vote-RPC tests over real sockets; tier-3 `cluster_election.rs`: the three exit gates below. | ### 8. Configuration (topology `election` block, all serde-defaulted) | Knob | Default | Constraint | |---|---|---| | `heartbeat_interval_ms` | 300 | | | `election_timeout_min_ms` | 1500 | ≥ 3× heartbeat interval | | `election_timeout_max_ms` | 3000 | > min | | `leader_lease_ms` | 900 | `leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms` (the C2 bound — validated at topology load) | | `auto_election` | true | false = detector + fencing only; elections fire only through `/cluster/promote`. After a full-cluster restart in this mode the cluster is deliberately leaderless until an operator promotes (design-review C16 — that is the posture the knob selects). | Failover arithmetic: detect (≤3 s worst-case timeout) + pre-vote+vote (2 RTTs + 2 fsyncs) + activation (µs) ≈ **3–4 s typical, comfortably inside the 10 s p99 gate**, without making the detector so twitchy that a fsync stall triggers an election (the 10–50 ms F_FULLFSYNC tail from p1 informs the 1500 ms floor). Heartbeats are async fan-out tasks on the transport's runtime with a per-call timeout well under the interval — N peers at 300 ms is negligible load and can never stall the data plane (design-review C15). ## Exit gate (from the roadmap, restated as tests) 1. **Auto-failover, zero acked loss**: SIGKILL the leader under `ack=quorum` load with auto-election on → writes resume on the new leader <10 s p99, zero operator verbs, and the m11p3 ledger checker (frontier + content proofs) passes across repeated random kill-points. 2. **Fencing under partition + restart**: partition the old leader away mid-election, let it restart → it boots a follower (durable state), its ships/votes carry a stale term and are rejected, and it **cannot accept a single write** (every write attempt 503s) until it heals and rejoins. 3. **Bounded churn**: flapping links (proxy sever/heal cycles) produce bounded elections (pre-vote absorbs the flaps; no term explosion, no livelock), and the cluster converges to exactly one leader per term. ## Status - [x] Design adopted (this document; 16 confirmed adversarial-review holes folded in pre-implementation) - [x] Election core (state machine + durable state) — `tidal` - [x] Transport (proto, hooks, gates, client verbs) — `tidal-net` - [x] Driver + boot + fencing + promote rework — `tidal-server` - [x] Exit-gate suites green - [x] Docs (runbook §9, CHANGELOG, roadmap, status fields) ## As built (deltas vs the adopted design) 1. **The vote-restriction frontier is STREAM-relative, not own-WAL-relative.** The gate run exposed it: a reseeded node's local WAL numbering diverges from stream numbering after a baseline jump (observed live: stream-applied 38 vs own-WAL 22), so raw local frontiers are not comparable across nodes. `LogPosition.frontier` is the node's durable position **in the last joined term's stream numbering**: own `flushed_seq()` when it led that term, `applied_seqno(shard of that term's leader)` otherwise. The term marker therefore carries `leader_region`, recovered with the term. 2. **Divergence is judged against the leader's ELECTION-TIME position, only from heartbeats.** The adopted baseline-comparison was cross-numbering for the same reason. The heartbeat carries `(prev_log_term, prev_log_seq)` — the same pair the vote restriction compared — and a joining node is divergent iff its own position exceeds it lexicographically. A NEW-term ship arriving before the first heartbeat defers with a retryable `JoinPending` (UNAVAILABLE) instead of guessing; `stream_baseline` on the heartbeat is used only for the frontier jump. 3. **"Topology era" ends at the first JOINED election, not at one's own durable term.** A failed campaign (an isolated operator override, the chaos drills' deliberate split-brain) inflates a node's durable term with no elected leadership existing; gating term-0 traffic, the legacy promote and the check-quorum fallback on `joined_term` instead of `current_term` keeps that world functional while elected clusters stay absolutely fenced. 4. **Check-quorum step-down is gated on `auto_election`** — with elections off no successor can exist, so stepping down would only convert a follower outage into total write unavailability; the knob deliberately preserves the m11p3 availability posture. 5. The transfer's catch-up wait reads the **commit index's peer mark** (fed by `ReportApplied`), not the ship queue's acked frontier — the latter stalls ~30s behind an open circuit breaker after the target restarts. 6. The frontier-report gate at the handler references the **commit index's activation term** (the same value the sink's race-free in-lock gate uses), not the machine's possibly-candidacy-inflated term. 7. A rejoining term-0 ex-leader's boot view is `None`, never `Some(self-from-topology)` — caught by the gate before it shipped: the adopted text only forbade *assuming the role*, but a self-pointing VIEW still opened the §1.4-1 write-acceptance hole with the queue parked. ## Exit-gate evidence (local 3-process clusters, tier-3, 2026-06-12) | Gate | Result | |---|---| | Auto-failover, zero acked loss | 5 rounds × random kill points under `ack=quorum` load, ZERO operator verbs: elections in **0.59–0.98 s** (budget 10 s); frontier invariant (elected leader's election-time position ≥ every acked seq) + content invariant (every acked item searchable on the new leader) held every round, including rounds where the killed node rejoined via quarantine + reseed | | Fencing under partition + restart | leader isolated (bidirectional TCP severs) → survivors elected in ~0.7 s; the ex-leader restarted while partitioned and was **0-for-~30** on write attempts across the fence window, then healed and rejoined the new term as a converged follower | | Bounded churn | 4 sub-timeout flaps absorbed with no elections + 3 super-timeout flaps → converged to ONE leader on ONE term ≤ 15, serving quorum writes | Full verification: 2,312 default-feature tests + all six tier-3 suites green; workspace clippy clean (`--all-targets --all-features`, 0 warnings). `cluster_quorum`/`cluster_lifecycle` pin `auto_election: false` (they validate the manual drill, which the knob preserves); the quorum ledger drill gained the promote-retry step (the election can refuse a stale max-applied sample — the protocol now enforces the rule the runbook used to assign the operator). ## Post-implementation review (seven-dimension, 2026-06-12) Twenty-one findings, adversarially verified: **10 confirmed → all fixed**, 11 refuted. The fixes that mattered: - **`joined_term` boots at ZERO** (three reviewers converged on it): it tracks terms whose JOIN CHECK passed, and the durable term is not evidence of that — initializing from it both fenced legitimate topology-era traffic after a failed candidacy and skipped the divergence check on restart. The fix surfaced a latent hole in the join check itself: the design's `tail_term == joining term ⇒ clean` clause (§5, third bullet) was never implemented, so the corrected init would have false-quarantined every mid-term rejoiner. Both landed together; the failover gates re-proved them. - **Torn-pair races**: `WalTermMark`'s `(term, seq, leader_region)` and the node's activation `(prev_log_term, prev_log_seq)` were split atomics — a reader between stores could pick the new term with the previous marker's REGION and read a frontier in the wrong stream's numbering. Both are now single-lock values (written once per term, read a few times a second), and the activation pair publishes only AFTER the term marker is durable, so an aborted activation never exposes it. - Unit coverage added for `on_activation_failed` and `transfer_to` (the two uncovered machine transitions). Notable refutations (recorded for the next reviewer): the "stale `wal_term_mark` during join" claim — the mark and the frontier advance through the same apply path, so a node's position is always internally consistent; a behind position simply joins clean and catches up. And the quarantine-metric-never-cleared claim — clearing rides m11p5's reseed machinery by design. ## Carried hazards (tracked, not regressions) - A reseeded node converges on the live stream but does not hold pre-baseline history; quorum marks it contributes cover ranges it genuinely holds, but full-history re-replication is **m11p5's snapshot transfer** (same gap the d0a52e4 increment's "snapshot required" refusal points at). - `mp_rolling_upgrade_no_loss_no_stall` is load-sensitive on saturated hosts (the post-restart heal races the 30 s circuit breaker inside its 60 s budget; widen via the harness env on slow runners). m11p8's self-driving heal removes the pattern. - Data below a term's activation baseline is unreachable through that term's stream for followers that were behind at the transfer — pre-existing m11p2 semantics, unchanged by this phase, resolved by p5 snapshots.