tidaldb/docs/planning/milestone-11/phase-4.md

m11p4 — Failure Detection, Election, Fencing (COMPLETE — 2026-06-12)

Phase spec and exit gate: docs/roadmap-to-cluster.md §4/m11p4. Closes ROADMAP gap G5 ("operator-driven failover; restarted node believes the static topology — split-brain seed", incident §1.4-1). Predecessors: phase-1.md (ship queue), phase-2.md (one replicated log), phase-3.md (quorum machinery — the prerequisite this phase's election rides on).

Already shipped in this phase (d0a52e4, 2026-06-11): catch-up timer retry (re-arm-on-skip liveness) + TSEG segment version header + structured "snapshot required" refusal. This document covers the remaining XL work.

Goal: "a machine died" is a non-event. Kill the leader under load → writes resume <10 s p99 with zero operator action and zero acknowledged loss; a restarted or partitioned ex-leader cannot accept a single write; election churn under flapping links is bounded.

Design (as adopted)

0. Open question #2 resolved: purpose-built election, no raft crate

The roadmap deferred "minimal purpose-built Raft over tidal-net vs. a vetted raft crate" to this phase. Decision: purpose-built election-only Raft (Raft §5.1–§5.4 election subset + pre-vote + check-quorum + leadership transfer), for one structural reason: the replicated log already exists. m11p2 made the WAL the stream; m11p3 built the quorum commit index over it. A raft crate (openraft, raft-rs) owns its own log and storage contracts — adopting one means either running a second log beside the WAL or contorting the crate to delegate its log to ours. The state that actually needs consensus is tiny — (term, leader) — and the safety-critical machinery (durable votes, the up-to-date restriction, term fencing) maps directly onto primitives we already have (flushed_seq(), the stream_baseline persistence pattern, the commit-index epoch). Membership stays static this phase (the topology file); m11p5 makes it dynamic through this phase's metadata machinery.

The election state machine is pure and deterministic (no clocks, no I/O — inputs carry Instants, randomness is an injected seed), so the entire protocol is unit-testable without processes or sleeps.

1. The durable election state (kills restart amnesia)

A small per-node file, data_dir/election_state, written with the exact stream_baseline discipline (write .tmp, sync_file_durable [F_FULLFSYNC on macOS], atomic rename, sync_dir_durable), carrying a magic + format version + checksum so corruption is detectable:

Field	Type	Meaning
current_term	u64	Highest term this node has seen. Strictly monotonic — a term is adopted only upward; no inbound message can ever lower it.
voted_for	Option	Vote cast in current_term (at most one).

The WAL-tail term deliberately does NOT live in this file — see §1b.

Persistence is a fence, not bookkeeping: the file is fsynced BEFORE any externally visible action that depends on it — before a vote response is sent, before a higher term is acknowledged, before leadership is assumed. Votes and term changes are rare (not per-write), so the fsync cost is irrelevant. All election-state mutations serialize under one lock, and the boot-time persist completes before the HTTP routes serve a single request (design-review C6).

Boot rule (the incident §1.4-1 fix), four cases (design-review C1):

1. File exists and reads clean → the node always boots as a follower, regardless of what the file or the topology says — leadership is learned from heartbeats or won by election, never assumed from a restart.
2. File exists but is corrupt (bad magic/version/checksum) → refuse to boot with a FATAL error naming the file. A node that cannot prove its term knowledge must not guess at term 0 — guessing re-opens the §1.4-1 amnesia hole through the side door of a disk fault.
3. File absent but the data dir is NOT fresh (a WAL exists) → the file was deleted out from under a node that has run before. Boot as a follower at term 0 with a loud ERROR (never assume topology leadership), and persist the recreated state immediately. Terms recover upward from the first heartbeat/ship the node hears.
4. File absent and the data dir is fresh → a genuinely new node: boot with the topology's term-0 roles exactly as today, persisting {term: 0} before serving. Existing deployments boot byte-for-byte identically on first start.

Term-0 conflicting-leader guard (design-review C5): a term-0 leader that receives another node's term-0 leadership assertion (heartbeat or ship stamped term=0, leader=other) — possible only through mismatched topology files, a deployment error — logs ERROR and starts an election. Term ≥ 1 leadership is unique by votes; the election converges the misdeployment instead of letting two config-believing leaders run open-loop.

1b. WAL term markers (the up-to-date restriction, crash-atomic)

Raft's vote restriction compares (lastLogTerm, lastLogIndex). The first design draft persisted a last_applied_term in the election file; the adversarial review (C7) found the killer: that file and the WAL frontier can never be made crash-atomic — a crash between the two persists yields a node whose up-to-date claim lies in the unsafe direction (it can win an election while missing committed entries).

Fix: the term lives in the log itself. On becoming leader of term T (any T ≥ 1), the leader journals a TermMarker(T) record — a new small WAL record kind beside m11p2's item/embedding blobs — as its first term-T entry. It replicates to every follower through the normal stream like any other record; followers fold it by recording the term (no storage effect). The node's WAL-tail term is then the term of the last marker at or below its durable frontier — recovered from the WAL on boot, advanced on apply, and atomic with the frontier by construction (they are the same fsync stream). No marker (a pre-election log) reads as term 0.

The vote comparison becomes (wal_tail_term, durable_frontier), lexicographic, both read from the same WAL state. A crash at any point yields a node that understates — never overstates — its log, which is the safe direction (it may lose an election it deserved; it can never win one it shouldn't, because every committed entry's quorum contains an un-crashed holder whose refusal stands).

Wire compatibility falls out of "markers only exist at term ≥ 1": a p4 binary on a term-0 cluster never writes one, so a rolling m11p3→m11p4 upgrade ships zero unknown record kinds; the first election (only possible once a vote quorum of upgraded binaries exists) is the moment the format becomes live. Runbook: complete the binary upgrade before relying on auto-failover.

2. Term model: term 0 is the topology, terms ≥ 1 are elected

• Term 0 = the static topology file's leader. A fresh cluster boots leading/following at term 0 with no election, preserving every existing boot path and test.
• Any election or fenced transfer moves the cluster to term ≥ 1. From then on the durable state is authoritative and the topology's leader field is dead weight (kept for bootstrap only).
• Proto compatibility falls out: proto3 defaults absent term fields to 0, so a pre-m11p4 peer's ships/reports read as term-0 traffic. A node still at term 0 interoperates with them; a node at term ≥ 1 rejects them (FAILED_PRECONDITION) — exactly the fencing semantics, applied to version skew.

3. Election protocol

Roles: Follower → PreCandidate → Candidate → Leader (and anything → Follower on observing a higher term).

• Failure detector: the leader sends Heartbeat to every peer each heartbeat_interval (default 500 ms), now carrying (term, leader_region, stream_baseline). A follower that hears nothing valid for a randomized election_timeout ∈ [1500 ms, 3000 ms) starts a pre-vote.
• Pre-vote (disruption guard): asks "would you vote for me in term+1?" without bumping any term. A voter refuses if it heard from a live leader within election_timeout_min (lease check) or if the candidate's log loses the up-to-date comparison. Only a pre-vote majority proceeds — a partitioned or flapping node cannot inflate terms or depose a healthy leader from the outside.
• Vote: candidate persists term+1, voted_for=self, then requests votes. A voter grants iff req.term ≥ current_term, it hasn't voted for another candidate in that term (durable before reply), and the candidate is up-to-date: (wal_tail_term, durable_frontier) ≥ the voter's, compared lexicographically — Raft's §5.4.1 restriction with both values derived from the voter's own WAL (§1b). Majority → leader.
• Becoming leader: persist election state, then journal the TermMarker(T) (§1b), then the promote mechanics verbatim: baseline = own flushed_seq() (now ≥ the marker's seqno), persist_stream_baseline, ship_queue.activate_from(baseline, term) (which activates the commit index scoped to T and bumps its epoch), leader view updated, heartbeats start.
• Step-down ordering (design-review C3): on any step-down — higher-term contact, check-quorum loss, fenced transfer — the leadership view flips first (leader = None or the new leader), THEN the ship queue deactivates. The write path's is_leader() reads the view, so once it flips no new write is accepted; the queue deactivation then fails in-flight ack=quorum waits with the existing Demoted path. The residual window (a request that read the old view and stages after the flip) produces a leader-local-durable, never-shipped write — exactly the documented ack=leader crash contract, surfaced at rejoin as a divergent suffix (§5).
• Check-quorum / lease step-down: a leader that cannot reach a majority of peers (heartbeat acks) within leader_lease_ms steps down (stops accepting writes; quorum waits fail). The lease bound (design-review C2): the leader's countdown starts at its last successful quorum contact while a follower's election timer starts at the last heartbeat it received — the two anchors differ by up to one heartbeat interval plus RTT, NOT by wall-clock offset (all timers are monotonic Instant intervals; clock offsets cancel). The enforced constraint is therefore leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms (validated at topology load), guaranteeing the deposed leader stops before any successor can be elected. This bounds the minority-partitioned-leader window for ack=leader writes; ack=quorum writes were never at risk (the commit index cannot reach majority from a minority partition).
• No pre-vote deadlock (design-review C9): the lease refusal can only outlive a real leader if something keeps sending heartbeats. A stepped-down leader stops heartbeating by construction (only the Leader role sends), so after a leader dies or steps down every follower's lease window drains within one election_timeout and pre-votes start granting. Asymmetric partitions degrade exactly one node (it pre-votes, is refused by peers that still hear the leader, and never inflates a term — the pre-vote's whole point); the majority side keeps serving throughout.
• Timers use std::time::Instant exclusively — monotonic, immune to the chaos suites' TIDAL_HLC_SKEW_MS wall-clock skew injection (HLC remains data-plane-only for hard-negative LWW).

4. Fencing everywhere

Every replication RPC carries the sender's term; every receiver gates:

RPC	Stamp	Stale-term reaction	Higher-term reaction
ShipSegment	term, leader_region	reject FAILED_PRECONDITION before enqueue	persist & adopt term, treat as leader contact
StreamSegments request	puller's term	source at a higher term refuses the pull	source at a lower term steps down (it is deposed and must not serve its possibly-divergent tail — design-review C14)
StreamSegments chunks	same as ship (chunks are ShipSegmentRequests)	puller drops the chunk + aborts the pull (a stale source can never inject history)	adopt
Heartbeat	term, leader_region, stream_baseline	respond accepted=false, term (sender steps down)	adopt, reset election timer, run the §5 term-join check
ReportApplied	reporter's term	leader folds only if reporter term == the commit index's activation term (design-review C4/C8/C12: the gate reads the term the index was activated with, so a stale, future, or cross-leadership report can never advance it; heal's resume_from folds through the same gate — C13)	leader steps down
RequestVote / TimeoutNow	term	refuse, return own term	adopt
HTTP forward (internal-marked writes)	already rejected on a non-leader (NotLeader 503); the error body now names (term, leader-or-none, leader_http_addr) so the forwarder follows one redirect to the named leader or returns a retryable 503 "election in progress" (design-review C11)	—	—

A ship/report/vote response carrying a higher term is a step-down signal to the sender — the deposed leader learns its term is stale from its own outbound traffic even if it never hears a heartbeat. The CommitIndex itself is term-scoped: activate(baseline, term) records the term, update_peer folds only same-term reports, so every quorum input — ship-ack hints, ReportApplied, heal resumes — passes one gate.

5. Rejoin, and the divergent-suffix boundary

The check runs exactly once per term, at term-join (the first valid leader contact carrying term T — heartbeat or ship — when adopting T), and compares the node's durable WAL frontier against the term's activation baseline B (persisted by the leader at activation and announced in every heartbeat — design-review C10: B is immutable for the term, so the check never races a moving target; a frontier that grows past B within term T is just normal replication, never re-checked):

• frontier ≤ B: clean rejoin. Catch-up via the existing StreamSegments pull (gap-triggered, timer-retried). The node's WAL-tail term advances to T when the term's marker record applies (§1b).
• frontier > B with WAL-tail term < T: the node holds a divergent suffix — writes that were leader-durable on a dead leader but never quorum-committed, while the cluster elected past them. The node quarantines: it refuses to apply or report (receiver halt latch — the existing died() health surface), logs ERROR naming the suffix range and the reseed runbook, sets tidaldb_cluster_divergence and stays fenced from the data plane. It still votes (its stale WAL-tail term makes it lose every up-to-date comparison against healthy peers, so it can never win).
• frontier > B with WAL-tail term == T: impossible at term-join (T-term entries only exist after the leader's activation, and joining is what admits them) — and after joining, it is the normal shape of a follower tracking the leader. Restarts within a term re-join idempotently: the tail term already equals T, so no quarantine.

Why quarantine instead of Raft-style suffix truncation: tidalDB followers apply segments on receipt (before quorum commit) — by the time divergence is discovered, the suffix's effects are already folded into storage (decayed counters), and unapplying signal arithmetic is not a thing. Truncating the WAL alone would lie about state. The honest recovery is a reseed — which is m11p5's snapshot+stream work, and exactly what the already-shipped p4 increment's FAILED_PRECONDITION ("snapshot required") path points at. Probability note: with RF=3 a quorum-committed suffix holder is a member of every election majority (the vote restriction protects it), so wrongful quarantine is impossible at n=3; the quarantined case is strictly leader-ack-only data, which is the documented leader-ack contract.

6. Promote becomes a fenced transfer

POST /cluster/promote {region}:

• Leader alive (the maintenance drain): forwarded to the current leader, which waits until the target's durable mark reaches the leader's flushed frontier (bounded; 503 naming the lag on timeout), then sends TimeoutNow — the target starts an immediate election (skipping pre-vote and timeout; the leader sanctioned it), wins on term+1, and the old leader steps down on the first higher-term contact. No drain-the-world: the catch-up wait IS the drain.
• Leader dead/unreachable (the legacy failover drill): the target runs a normal pre-vote + election among survivors. With auto-election on, the operator usually never gets here — the verb remains as a manual override and a way to choose the successor.
• Mixed-version fallback: if a majority of peers answer Unimplemented to vote RPCs (pre-m11p4 binaries), promote falls back to the legacy fan-out promote at term 0 with a loud WARN. Auto-election quietly waits until the fleet is upgraded (vote quorum impossible until then).

7. What goes where

Crate	Work
tidal	replication/election.rs: pure ElectionState step machine (inputs → actions) + config; replication/election_durable.rs: the fsync'd (current_term, voted_for) file with magic/version/checksum + the four-case boot classification; the TermMarker WAL record kind + recovery/apply tracking of the WAL-tail term (§1b); CommitIndex.activate(baseline, term) + same-term fold gate; WalSegmentPayload.term/leader_region; ship queue stamps terms (shared Arc<AtomicU64>); receiver-side divergence latch.
tidal-net	Proto: term fields on ShipSegmentRequest/Response, StreamRequest, HeartbeatRequest/Response, AppliedReport; new RequestVote (with prevote/transfer flags) + TimeoutNow RPCs. Server: late-bound ElectionHooks (vote/heartbeat/observed-term) mirroring the AppliedSink pattern; term gates on ship/stream/report handlers. Client: PeerPool::{request_vote, timeout_now, heartbeat} — vote RPCs bypass the circuit breaker (an election must be able to probe a peer the breaker quarantined; vote traffic is rare and self-limiting). Config: election knob block.
tidal-server	The election driver (one tokio task per region state): ticks the state machine, executes actions (persist → send → transition), owns the heartbeat fan-out and failure detection. Boot path: durable-state detection → rejoin-as-follower. Leadership transitions route through the existing promote_local mechanics. leader becomes RwLock<Option<RegionId>> (leaderless windows are real and reported honestly — forwards during an election return retryable 503 "election in progress"). /cluster/status grows term, role; election metrics (tidaldb_cluster_term, _role, _elections_started_total, _leader_changes_total, _stepdowns_total, _votes_{granted,denied}_total, _divergence). Promote rework per §6.
tests	Pure state-machine property/unit tests (no clocks); tidal-net vote-RPC tests over real sockets; tier-3 cluster_election.rs: the three exit gates below.

8. Configuration (topology election block, all serde-defaulted)

Knob	Default	Constraint
heartbeat_interval_ms	300
election_timeout_min_ms	1500	≥ 3× heartbeat interval
election_timeout_max_ms	3000	> min
leader_lease_ms	900	leader_lease_ms + heartbeat_interval_ms < election_timeout_min_ms (the C2 bound — validated at topology load)
auto_election	true	false = detector + fencing only; elections fire only through /cluster/promote. After a full-cluster restart in this mode the cluster is deliberately leaderless until an operator promotes (design-review C16 — that is the posture the knob selects).

Failover arithmetic: detect (≤3 s worst-case timeout) + pre-vote+vote (2 RTTs + 2 fsyncs) + activation (µs) ≈ 3–4 s typical, comfortably inside the 10 s p99 gate, without making the detector so twitchy that a fsync stall triggers an election (the 10–50 ms F_FULLFSYNC tail from p1 informs the 1500 ms floor). Heartbeats are async fan-out tasks on the transport's runtime with a per-call timeout well under the interval — N peers at 300 ms is negligible load and can never stall the data plane (design-review C15).

Exit gate (from the roadmap, restated as tests)

1. Auto-failover, zero acked loss: SIGKILL the leader under ack=quorum load with auto-election on → writes resume on the new leader <10 s p99, zero operator verbs, and the m11p3 ledger checker (frontier + content proofs) passes across repeated random kill-points.
2. Fencing under partition + restart: partition the old leader away mid-election, let it restart → it boots a follower (durable state), its ships/votes carry a stale term and are rejected, and it cannot accept a single write (every write attempt 503s) until it heals and rejoins.
3. Bounded churn: flapping links (proxy sever/heal cycles) produce bounded elections (pre-vote absorbs the flaps; no term explosion, no livelock), and the cluster converges to exactly one leader per term.

Status

• Design adopted (this document; 16 confirmed adversarial-review holes folded in pre-implementation)
• Election core (state machine + durable state) — tidal
• Transport (proto, hooks, gates, client verbs) — tidal-net
• Driver + boot + fencing + promote rework — tidal-server
• Exit-gate suites green
• Docs (runbook §9, CHANGELOG, roadmap, status fields)

As built (deltas vs the adopted design)

1. The vote-restriction frontier is STREAM-relative, not own-WAL-relative. The gate run exposed it: a reseeded node's local WAL numbering diverges from stream numbering after a baseline jump (observed live: stream-applied 38 vs own-WAL 22), so raw local frontiers are not comparable across nodes. LogPosition.frontier is the node's durable position in the last joined term's stream numbering: own flushed_seq() when it led that term, applied_seqno(shard of that term's leader) otherwise. The term marker therefore carries leader_region, recovered with the term.
2. Divergence is judged against the leader's ELECTION-TIME position, only from heartbeats. The adopted baseline-comparison was cross-numbering for the same reason. The heartbeat carries (prev_log_term, prev_log_seq) — the same pair the vote restriction compared — and a joining node is divergent iff its own position exceeds it lexicographically. A NEW-term ship arriving before the first heartbeat defers with a retryable JoinPending (UNAVAILABLE) instead of guessing; stream_baseline on the heartbeat is used only for the frontier jump.
3. "Topology era" ends at the first JOINED election, not at one's own durable term. A failed campaign (an isolated operator override, the chaos drills' deliberate split-brain) inflates a node's durable term with no elected leadership existing; gating term-0 traffic, the legacy promote and the check-quorum fallback on joined_term instead of current_term keeps that world functional while elected clusters stay absolutely fenced.
4. Check-quorum step-down is gated on auto_election — with elections off no successor can exist, so stepping down would only convert a follower outage into total write unavailability; the knob deliberately preserves the m11p3 availability posture.
5. The transfer's catch-up wait reads the commit index's peer mark (fed by ReportApplied), not the ship queue's acked frontier — the latter stalls ~30s behind an open circuit breaker after the target restarts.
6. The frontier-report gate at the handler references the commit index's activation term (the same value the sink's race-free in-lock gate uses), not the machine's possibly-candidacy-inflated term.
7. A rejoining term-0 ex-leader's boot view is None, never Some(self-from-topology) — caught by the gate before it shipped: the adopted text only forbade assuming the role, but a self-pointing VIEW still opened the §1.4-1 write-acceptance hole with the queue parked.

Exit-gate evidence (local 3-process clusters, tier-3, 2026-06-12)

Gate	Result
Auto-failover, zero acked loss	5 rounds × random kill points under ack=quorum load, ZERO operator verbs: elections in 0.59–0.98 s (budget 10 s); frontier invariant (elected leader's election-time position ≥ every acked seq) + content invariant (every acked item searchable on the new leader) held every round, including rounds where the killed node rejoined via quarantine + reseed
Fencing under partition + restart	leader isolated (bidirectional TCP severs) → survivors elected in ~0.7 s; the ex-leader restarted while partitioned and was 0-for-~30 on write attempts across the fence window, then healed and rejoined the new term as a converged follower
Bounded churn	4 sub-timeout flaps absorbed with no elections + 3 super-timeout flaps → converged to ONE leader on ONE term ≤ 15, serving quorum writes

Full verification: 2,312 default-feature tests + all six tier-3 suites green; workspace clippy clean (--all-targets --all-features, 0 warnings). cluster_quorum/cluster_lifecycle pin auto_election: false (they validate the manual drill, which the knob preserves); the quorum ledger drill gained the promote-retry step (the election can refuse a stale max-applied sample — the protocol now enforces the rule the runbook used to assign the operator).

Post-implementation review (seven-dimension, 2026-06-12)

Twenty-one findings, adversarially verified: 10 confirmed → all fixed, 11 refuted. The fixes that mattered:

• joined_term boots at ZERO (three reviewers converged on it): it tracks terms whose JOIN CHECK passed, and the durable term is not evidence of that — initializing from it both fenced legitimate topology-era traffic after a failed candidacy and skipped the divergence check on restart. The fix surfaced a latent hole in the join check itself: the design's tail_term == joining term ⇒ clean clause (§5, third bullet) was never implemented, so the corrected init would have false-quarantined every mid-term rejoiner. Both landed together; the failover gates re-proved them.
• Torn-pair races: WalTermMark's (term, seq, leader_region) and the node's activation (prev_log_term, prev_log_seq) were split atomics — a reader between stores could pick the new term with the previous marker's REGION and read a frontier in the wrong stream's numbering. Both are now single-lock values (written once per term, read a few times a second), and the activation pair publishes only AFTER the term marker is durable, so an aborted activation never exposes it.
• Unit coverage added for on_activation_failed and transfer_to (the two uncovered machine transitions).

Notable refutations (recorded for the next reviewer): the "stale wal_term_mark during join" claim — the mark and the frontier advance through the same apply path, so a node's position is always internally consistent; a behind position simply joins clean and catches up. And the quarantine-metric-never-cleared claim — clearing rides m11p5's reseed machinery by design.

Carried hazards (tracked, not regressions)

• A reseeded node converges on the live stream but does not hold pre-baseline history; quorum marks it contributes cover ranges it genuinely holds, but full-history re-replication is m11p5's snapshot transfer (same gap the d0a52e4 increment's "snapshot required" refusal points at).
• mp_rolling_upgrade_no_loss_no_stall is load-sensitive on saturated hosts (the post-restart heal races the 30 s circuit breaker inside its 60 s budget; widen via the harness env on slow runners). m11p8's self-driving heal removes the pattern.
• Data below a term's activation baseline is unreachable through that term's stream for followers that were behind at the transfer — pre-existing m11p2 semantics, unchanged by this phase, resolved by p5 snapshots.