tidaldb/CHANGELOG.md

# Changelog

All notable changes to tidalDB will be documented in this file.

## [Unreleased]

### Added

**Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix**

- **Fault injection compiled out of production.** A non-default `fault-injection`
  cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks
  (`tidal/src/fault.rs`): **slow-fsync** (`TIDAL_FAULT_FSYNC_DELAY_MS` sleeps
  before each durable fsync) and **disk-full** (`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`
  returns a real `ENOSPC`/errno-28 once cumulative segment bytes cross the
  threshold). The production image build never passes the feature, so the hooks
  are compiled out entirely — a disk/fsync fault a stray env var could trip in
  prod is a 3am footgun we refuse to ship; the safety is structural. Inert until
  armed even when compiled in. The tier-3 harness builds the spawned binary with
  the feature.
- **First-class invariant checkers** (`tidal-server/tests/support/invariants.rs`):
  the no-acked-loss `AckLedger` (frontier + content, extracted from the m11p3
  ledger gate, which now consumes it), cross-replica `assert_feed_parity` /
  `feed_item_ids`, `assert_single_leader_per_term` (membership safety, reads the
  m11p6 `shards[]` rows), and `MonotonicCounters` (per-node applied frontier +
  leader commit index never regress, with legitimate epoch resets forgiven).
- **New fault-class suite** (`tidal-server/tests/cluster_faults.rs`, tier-3): a
  disk-full follower degrades gracefully (receiver halts, node alive, zero acked
  loss, restart recovers to parity); a slow-fsync follower lags then converges
  while the fast follower supplies quorum; both followers slow → `ack=quorum`
  returns a retryable 503 naming the laggards while `ack=leader` is unaffected;
  an asymmetric partition (inbound severed, outbound up) causes no split brain
  (pre-vote + check-quorum) and no loss. 4/4 green.
- **Soak + regression gates** in `tidal-stress`: `--json-summary <path>` (a
  machine-readable per-stage p99/throughput/error roll-up for trend lines) and
  `--max-p99-ms` / `--max-error-pct` / `--fail-on-knee` gates that exit non-zero
  on a regression (the tool always exited 0 before).
- **Nightly pipeline** (`.woodpecker.yaml`): a cron `nightly` flow (chaos suites
  with elevated kill-points + the `fault-injection` feature, then the gated
  `tidal-stress` soak) beside the existing push release gate, event-routed by
  per-step `when`. Woodpecker, never GitHub Actions.
- **Guarantee traceability** (`docs/planning/milestone-11/guarantee-traceability.md`):
  every roadmap §2 guarantee mapped to its named automated test(s). Closes the
  **G-C** apparatus; the 30-consecutive-days-green half of the GA bar is a
  calendar criterion the nightly pipeline accrues.

**Observability + operations (m11p8) — complete cluster metric set on a per-node `/metrics` listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate**

- **Metrics.** Completed the `tidaldb_cluster_*` set with the two members the
  roadmap named that were missing — **breaker** (`tidaldb_cluster_peer_breaker_state`
  0/1/2 + `tidaldb_cluster_breaker_opens_total`, surfaced read-only from the
  `tidal-net` circuit breaker via a new `Transport::peer_breaker_state`) and
  **forwards** (`tidaldb_cluster_forwards_total` / `_forward_failures_total`,
  instrumented at the gateway forward path) — plus the self-heal series
  (`heal_attempts/successes/noops_total`, `healing_peers`). When several shard
  groups co-locate on one node (m11p6) the metrics-owner serves the single
  `/metrics` listener and the siblings register their series under a `shard="N"`
  label (`TidalDb::register_metrics_sibling`); a single-shard node is
  byte-identical to before. A 12-panel "Cluster Replication" Grafana row +
  an 8-rule `tidaldb-cluster` Prometheus alert group ship beside the standalone
  ones.
- **Request-id + tracing.** Both cluster routers (single- and multi-process) now
  carry the standalone router's `SetRequestId` + `PropagateRequestId` +
  `TraceLayer` stack (extracted to `router::with_request_id_tracing`); the id
  rides the follower→leader forward hop verbatim, so the leader's span shares the
  gateway's `x-request-id`. (Ships are off-request-path and batched, so they
  correlate by seqno, not a request-id — by design.)
- **Truthful status.** Fixed the leader's own `applied_events` reading 0 (a leader
  writes its WAL directly and never advances its own applied frontier — now
  reports its flushed frontier) and the single-process post-promote `ShardId(0)`
  lag-keying undercount (now keys on the current leader's shard).
- **Self-driving heal.** A standing leader duty re-arms the backlog re-ship for a
  stuck (breaker-open, behind) peer every ~3s, so it converges through breaker
  resets with no operator `/cluster/heal` loop — closing the §1.4-3 footgun.
  Observable via `healing_peers` + the `TidalDBClusterHealNotConverging` alert.
- **WAL PITR archival.** `wal.archive_dir` (topology + builder): the online
  compaction copies each sealed segment to the archive — durably, before deletion,
  refusing to delete if archival fails — so the archive is a gap-free PITR record.
- **Backup/restore.** `tidalctl backup` / `restore`: offline data-dir backup with
  a BLAKE3 `BACKUP_MANIFEST.json` (+ the WAL checkpoint cursor); restore verifies
  every file's hash before writing and refuses a non-empty target. Coordinated
  cluster backup = back up one committed replica per shard group (the runbook
  drill).
- **Rolling upgrade.** A wire version handshake (`HeartbeatRequest.build_version`,
  stamped at the transport boundary; `>= 2`-major skew WARNs, never rejects) +
  `version` on `/cluster/status`. `mp_rolling_upgrade_no_loss_no_stall` promoted
  to the FIRST step of `.woodpecker.yaml` (the release gate; a failure blocks the
  image build). See [milestone-11/phase-8.md](docs/planning/milestone-11/phase-8.md).

**Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit**
- **The cluster stops trusting the network.** gRPC replication mTLS is now the
  intended posture: the inbound server is served over a custom `tokio-rustls`
  acceptor (not tonic's fixed `.tls_config()`) fed a `DynamicCertResolver`
  (`ArcSwap<CertifiedKey>`), preserving mutual TLS exactly (a `WebPkiClientVerifier`
  over the cluster CA — a foreign/absent client cert fails the handshake before
  any RPC). Plaintext is an explicit `insecure: true` with a loud startup WARN.
- **Cert + bearer rotation WITHOUT restart.** A content-hash poller re-reads the
  cert files (k8s `..data` symlink swaps that inotify misses) and atomically
  swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys,
  so a rotation **drops zero requests** (verified under concurrent load). Outbound
  peer channels rebuild from the refreshed files. The bearer and a new shared
  cluster key live behind `ArcSwap`, read per request and reloaded from
  `TIDAL_API_KEY_FILE` / `TIDAL_CLUSTER_KEY_FILE`.
- **Authenticated inter-node HTTP with per-node identity.** The axum listener
  serves TLS (reusing the same hot-swappable resolver — one rotation covers both
  planes); forwards/broadcasts/scatter/status/seed-join dial `https://` with the
  cluster CA. A forwarding node mints an `x-tidal-node-token` (keyed-BLAKE3 MAC
  over node-id + expiry under the cluster key — no new crypto dependency) so a
  foreign pod cannot forge a sibling identity. The `x-tidal-internal` marker is
  now honored ONLY from a verified sibling (marker without a valid node token →
  403): the marker stays a routing hint, never an authorization bypass. All
  opt-in via `grpc_tls` / the cluster key — absent ⇒ pre-m11p7 behavior, so every
  existing deployment and test is byte-for-byte unchanged.
- **Admin-verb audit log.** promote / partition / heal / join / member-remove /
  reseed each emit one structured record (principal, term, target, outcome) to a
  `tidal_audit` tracing target + an optional append-only JSONL file
  (`TIDAL_AUDIT_LOG`), on the operator-originated leg only (no double-audit on the
  forwarded re-apply).
- **Per-principal HTTP rate limit.** The engine's token-bucket `RateLimiter`
  (now re-exported from the crate root) gates all three routers keyed by
  principal; verified sibling nodes are exempt (replication is never throttled);
  a deny is 429 + `Retry-After`. Off by default (`TIDAL_RATE_LIMIT_RPS`).
- **Reference deployment + tooling.** `k8s/cluster/` gains cert-manager
  Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region
  `grpc_tls` topology block; `scripts/gen-cluster-certs.sh` (openssl) provisions
  the same Secret shape without cert-manager. Exit gate verified: foreign pod
  rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext
  inter-node links. See [docs/planning/milestone-11/phase-7.md](docs/planning/milestone-11/phase-7.md).

**Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference**
- **Nodes are cattle; topology is data, not files.** `grpc_addr` is now an
  **advertised** address — a hostname or an IP — split from a new optional
  per-region `grpc_bind` (the local SocketAddr to bind). A literal `grpc_addr`
  binds itself byte-for-byte as before (every existing topology keeps working);
  a hostname binds `0.0.0.0:<port>`. `tidal-net`'s peer map retypes from
  `SocketAddr` to `String`, so `Channel::from_shared` makes hyper **re-resolve
  DNS on every reconnect** — the pod-rescheduled-onto-a-new-IP case the
  `SocketAddr::parse`-only constraint made structurally impossible is now fixed
  by construction. One shared topology file can name every region by its per-pod
  DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names
  require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code
  change). Proven by `cluster_membership.rs::mp_dns_hostname_topology_replicates`
  (a hostname topology a pre-p5 binary would have refused to parse boots,
  replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
- **A new node joins via snapshot + stream and serves quorum in minutes.**
  `FetchSnapshot` is a new server-streaming RPC in the `WalShipping` service:
  the leader stages `TidalDb::create_backup` under `<data_dir>/snapshots/` and
  streams it term-stamped + term-fenced exactly like `StreamSegments`
  (manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to
  end). The handler answers a **no-snapshot-needed** header when the WAL can
  still serve the puller's range and streams the artifact only when it cannot —
  the `needed` decision is **baseline-aware** (`from_seqno ≤ stream_baseline OR
  < earliest WAL seq`), because a "the WAL still covers it" answer is a lie at
  or below a baseline the stream clamp will never serve, and looped marker boots
  forever. Install is a **boot-time** operation (never a live data-dir swap):
  fetch into a sibling staging dir, copy the node's own identity files
  (`election_state`, `membership`) INTO staging, write a `COMPLETE` sentinel,
  then rename — every crash window is an idempotent redo. A retention pin taken
  at backup **start** keeps the segment chain covering the artifact alive while
  any joiner streams, with a hard cap + `tidaldb_cluster_snapshot_pin_force_drops_total`
  so a dead joiner can't freeze compaction forever.
- **Reseed is self-healing — no operator verb, no `wipe_data_dir`.** A running
  follower that hits a **typed** snapshot-required refusal (the `StreamSegments`
  trailer is now structured — `x-tidal-catchup: snapshot-required | rejoin |
  stepping-down` — so an ordinary election term-mismatch never latches reseed
  markers fleet-wide; the marker latches **only** on `snapshot-required`), or the
  m11p4 divergence quarantine, durably latches `reseed_required` (ElectionStore
  file discipline), surfaces it in `/cluster/status/local` and the
  `tidaldb_cluster_reseed_required` gauge, and keeps serving degraded with
  **voting enabled** (a reseed needs a leader, and the leader may need this
  node's vote — reseed-blocks-voting would deadlock the cluster). The reseed
  runs on the next boot; `replication.reseed_self_restart: true` (default
  **false**; k8s sets it `true`) drains and clean-exits once the marker latches —
  **refused** when the remaining voters can't sustain quorum without this node.
  A successful reseed boot clears the quarantine latch and
  `tidaldb_cluster_divergence_quarantined` — closing the m11p4 carried hazard
  that quarantine clearing "rides m11p5's reseed machinery." `cluster_reseed.rs`
  proves the full quarantine → marker → restart → converged → gauges-cleared
  loop with no `wipe_data_dir` in the test.
- **The three-way term-join rule closes p4's last carried hazard** (pre-baseline
  history was unreachable through a term's stream for followers behind at the
  transfer). The first reseed drill exposed that such a follower jumps its
  frontier past pre-baseline history with `lag=0` and never asks for the gap —
  silently missing data with nothing to fire the refusal. The heartbeat now
  carries the leader's election-time position in the previous stream's numbering
  (`prev_log`), and the join check is three-way: `own > prev_log` → divergent
  suffix → quarantine (p4); `own < prev_log` → genuinely missing committed-era
  history → **latch `reseed_required`** (new); `own == prev_log` (or within-term
  rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL
  is the leader's copy, so `tail_term` equals the leader's term).
- **Membership is data on the one log: kind-4 records, learner → voter.** A new
  `MembershipRecord` WAL blob kind (kind-4, beside kind-0 signals, kind-1/2
  item/embedding blobs, kind-3 term markers) is journaled by the leader,
  replicated through the normal stream, and folded by followers into a
  `ClusterMembership` cell (the `WalTermMark` pattern). Records carry the FULL
  roster (latest record wins; no merge logic); membership **epoch 0 = the
  topology file**, so a cluster with no kind-4 record behaves byte-for-byte as
  today. `POST /cluster/join {name, grpc_addr, http_addr}` on any node forwards
  to the leader, which assigns `id = max(all ids ever) + 1`, appends a **Learner**
  record, and answers only after it is quorum-committed (idempotent by name);
  member ids are **permanent** — removal tombstones (a `Removed` record) keep ids
  burned, never renumbered or reused. **Auto-promotion is a standing leader duty**
  (re-armed on every activation and apply, evaluated on commit-index publishes,
  driven solely from the applied `ClusterMembership` — it survives the
  joining-era leader's death) that promotes a learner within
  `replication.learner_promote_lag` (default 1024) of the flushed frontier, or
  the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server
  (one at a time, each same-term-quorum-commit-gated); on every leadership
  activation the new leader **re-appends its full current membership** immediately
  after the kind-3 term marker (the Raft no-op-entry analogue), which closes both
  the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs:
  `GET /cluster/members`, `POST /cluster/members/remove`, `POST /cluster/reseed`.
- **The even-voter-count `majority()` arithmetic was a latent bug — fixed.**
  `ElectionConfig::majority()` was `peers.len()/2 + 1`: correct at n=3, **wrong
  for every even voter count** (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D}
  could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with
  zero RPCs). `CommitIndex` already used the correct `div_ceil` form — the two
  formulas in one subsystem disagreed. Since p5 makes even sizes mandatory
  transit states (3→4→5→4→3), this landed **first**, with even-n property tests:
  `majority() = (peers.len() + 1).div_ceil(2) + 1` (true `floor(n/2)+1` over the
  full voter set), spelled to read identically to the commit-index site so they
  can never drift again.
- **Learner marks never count toward quorum.** A role-blind `CommitIndex` that
  saw learner marks would treat two learners "committing" a write no voter holds
  as acked — acked-write loss. `CommitIndex` is now role-aware: voter marks feed
  the k-th-largest selection; learner marks live in a side map (promotion input,
  transfer-wait input) and are excluded from `needed`. In-flight `ack=quorum`
  waits are **re-evaluated** against a new config on every conf-change (a shrink
  may satisfy waiters instantly), never failed; a report from an unknown id logs
  at WARN with a counter instead of being silently dropped. `ElectionState`,
  `ShipQueue`, and `PeerPool` reconfigure behind one fenced apply path so the
  four peer-set copies can never disagree about the roster.
- **Seed-join boot.** `tidal-server cluster --region <name> --seed
  http://host:port (repeatable) --advertise-grpc host:port --advertise-http
  host:port --metrics addr`: the joiner skips the "every region declared"
  topology gate, learns its roster + assigned id + current term from any
  reachable seed, persists them to a durable membership cache + `election_state`
  (persist-before-act), and installs a snapshot when behind. A restart boots from
  the cache without the seed. A `--seed` boot **still requires the local
  topology/config file** for the behavioral knob blocks (`replication:`, `wal:`,
  `election:`, `timeouts:`, `grpc_tls`) — a bare `--seed` with neither
  `--topology` nor `TIDAL_CONFIG` refuses to boot naming the rule (so a joiner
  never silently inherits the wrong ack default or election timing).
- **Kubernetes reference: one StatefulSet.** `k8s/cluster/` (namespace
  `tidaldb-cluster`, `replicas: 3`, `podManagementPolicy: Parallel`,
  topology-spread, PDB `maxUnavailable: 1`) with ONE shared bootstrap topology
  ConfigMap (per-pod DNS advertised addresses, `0.0.0.0` binds). Scaling past 3
  does NOT edit it — pod N≥3 boots `--seed` + the same mounted file and joins as
  a learner; rolling node replace is `kubectl delete pod` (PVC retained → boot
  catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness:
  `converged` (boot catch-up completed at least once AND `lag ≤
  learner_promote_lag` — hysteresis, never `lag == 0` which an open-loop load
  keeps perpetually false); a restarted existing voter is Ready on today's terms.
- Exit-gate suites (tier-3, `cluster-e2e`): `cluster_membership.rs`
  (`mp_seed_join_snapshot_catchup`, `mp_scale_3_5_3_under_load_zero_loss` —
  `lost=0`, max acked seq 1467, p99 impact <2× across the 3→5→3 joins,
  `mp_dns_hostname_topology_replicates`) and `cluster_reseed.rs` (quarantine →
  marker → restart → converged, gauges cleared). Localhost-loopback catch-up:
  5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget);
  the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line
  items (k3s access still pending — the standing M11 caveat).
- **Mixed-version / downgrade caveats.** A kind-4 record shipped to a **pre-p5
  follower** is an unknown batch kind → `WalError::Corruption` → its receiver's
  torn-state halt latch, **permanent across restarts** (both followers halted =
  quorum-write outage). So `HeartbeatResponse`/`ReportApplied` carry a
  `capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
  leader **refuses `JoinCluster` and every conf-change until all current voters
  report kind-4 capability** — "complete the binary upgrade before the first
  conf-change" is structurally enforced, not operator discipline. Pre-p5 peers
  answer `Unimplemented` to `JoinCluster`/`FetchSnapshot` (the joiner reports it
  loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim):
  once any kind-4 record is in a node's WAL, **downgrade below p5 requires a
  reseed**.

**Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5**
- **"A machine died" is a non-event.** Every multi-process cluster node runs a
  purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced
  leadership transfer) over the existing `tidal-net` transport: leader
  heartbeats every 300ms (config `election.heartbeat_interval_ms`); a follower
  that hears no leader for a randomized 1500–3000ms starts a pre-vote; a
  majority elects. SIGKILL the leader under `ack=quorum` load → a survivor is
  elected and writes resume in well under a second locally, with **zero
  acknowledged-write loss across repeated random kill points** (tier-3 gate
  `cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss`).
  No raft crate: the WAL is already the replicated log (m11p2) and the quorum
  commit index (m11p3) already proves majority durability — the election adds
  only the tiny consensus state `(term, leader)` on top of them.
- **The WAL carries its own election history.** An elected leader's FIRST log
  entry is a kind-3 **term-marker record** that replicates like any record, so
  every replica's `(lastLogTerm, lastLogIndex)` — Raft's vote restriction — is
  derived from one fsync stream and can never disagree across a crash. The
  frontier half is compared in the **last joined term's stream numbering**
  (a reseeded node's own WAL numbering diverges from the stream's after a
  baseline jump; comparing raw local frontiers across nodes would let a
  behind node win). Hard state `(current_term, voted_for)` persists in
  `data_dir/election_state` (magic + version + checksum; corrupt → refuse to
  boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE
  any vote reply or leadership claim leaves the node.
- **Term fencing everywhere (kills incident §1.4-1).** Every replication RPC
  carries the sender's term: stale-term ships/chunks/heartbeats/frontier
  reports are rejected, the quorum commit index folds only reports stamped
  with its activation term (race-free, under the index's own lock), and a
  restarted ex-leader boots as a FOLLOWER from its durable state — never from
  the topology file. A partitioned ex-leader that restarts cannot accept a
  single write (`mp_fenced_ex_leader_restart_cannot_write`); a leader that
  loses majority contact steps down within `election.leader_lease_ms`
  (default 900; validated `lease + heartbeat < election_timeout_min` so a
  deposed leader stops before any successor can exist).
- **Divergent suffixes quarantine instead of lying.** A node whose log
  extends past what the elected leadership subsumed (leader-acked,
  never-quorum-acked writes on a dead leader) detects it at term-join — the
  heartbeat carries the leader's election-time log position — and fences
  itself from the data plane (status `quarantined: true`, metric
  `tidaldb_cluster_divergence_quarantined`, ERROR naming the reseed runbook)
  while still voting. Followers apply on receipt, so un-applying is not a
  thing: reseed is the honest recovery (m11p5 snapshots automate it).
- **`/cluster/promote` is now a fenced transfer**: with a live leader it
  drains (waits for the target to hold the flushed prefix, breaker-immune via
  the commit index's marks) then sanctions an immediate election; with a dead
  leader the target campaigns. The election refuses a target whose log lags —
  the m11p3 "promote the max-applied survivor" operator rule is now enforced
  by the protocol. The legacy term-0 fan-out survives only on clusters that
  have never elected (mixed-version rollouts; the chaos drills' deliberate
  isolated-node override) and is permanently retired per node at its first
  joined election. `election.auto_election: false` preserves the full
  pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
- **Bounded churn under flapping links**: the pre-vote (no term inflation
  without a majority probe) plus the leader-freshness lease absorb short
  flaps entirely and bound terms under long ones
  (`mp_flapping_links_bounded_churn`). Election observability:
  `tidaldb_cluster_election_term/_role/_elections_started_total/
  _leader_changes_total`, and `/cluster/status/local` grows `term`, `role`,
  `quarantined`, `prev_log_term/seq`.
- New tier-3 exit-gate suite `cluster_election.rs` (auto-failover ledger,
  fencing under partition+restart, bounded churn); `cluster_quorum.rs` and
  `cluster_lifecycle.rs` pin `auto_election: false` (they validate the manual
  drill, which remains supported); the partition harness gained
  bidirectional isolation (`isolate_region`) — severing only a node's inbound
  edges leaves its outbound heartbeats/votes flowing, which silently defeats
  partition scenarios.

**Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, `TSEG` segment header, structured "snapshot required"**
- **Failed catch-up pulls retry on a timer.** The pull trigger was event-only:
  a follower whose `StreamSegments` pull failed (e.g. the leader's gRPC server
  not yet ready during a rolling restart) waited for the next PUSHED segment
  to re-expose the gap — in an idle cluster that push never comes, and the
  follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers
  stuck at lag=136507). A failed pull now arms a one-shot timer
  (`replication.catchup_retry_ms`, default 30000; transport
  `catchup_retry_interval`) that re-pulls from the CURRENT applied frontier.
  Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when
  consumed by the rate limit or an in-flight pull, so the gap always keeps a
  standing wake-up until a pull completes. Verified over real sockets:
  `catchup_retry.rs` reproduces the incident (pull fails, leader appears,
  zero pushes) and proves timer-only self-heal — and that a clean completion
  arms nothing.
- **WAL segment files are format-versioned.** New segments open with an
  8-byte header (`TSEG` magic + version byte + reserved); pre-m11p4
  headerless segments stay readable as implicit version 0 — no migration. A
  segment this binary cannot identify (unknown header version, unrecognized
  leading bytes, unparseable `.seg` filename) surfaces as the new
  `WalError::SegmentFormatUnknown` at open — previously it scanned as
  empty (`segments=0`) and recovery's torn-tail repair could TRUNCATE the
  foreign file to zero. Foreign-format files are never repaired, truncated,
  or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
- **Unservable catch-up is a structured refusal.** `SegmentSource::collect_from`
  returns typed `SegmentReadError::{Unavailable,Failed}`;
  `TidalDb::read_wal_batches` returns the typed `WalError` (was stringified
  `TidalError`). The `StreamSegments` handler maps `Unavailable` to
  `FAILED_PRECONDITION` — `"segments not available from seq N; snapshot
  required"` — and the follower logs it distinctly (*catch-up unservable …
  needs a snapshot (m11p5) or an operator reseed*) instead of burying it as a
  transient. The on-disk segment format is now documented
  (`tidal/src/wal/segment.rs` module docs + spec 01 §2.2).

**Quorum-acked writes (m11p3) — `ack=leader|quorum`, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)**
- **`ack=quorum`** is an opt-in durability contract for every replicated write
  (`/signals`, `/items`, `/embeddings`): success means a **majority of the
  replica set durably holds the write** (leader + `floor(n/2)` followers, each
  storage-applied and own-WAL-fsynced), so an acked write survives the
  permanent loss of any single node — including the leader. Deployment default
  via topology `replication.ack`; per-request override via the **`x-tidal-ack`**
  header (forwarded verbatim by gateways). `ack=leader` (the default) is the
  m0–m11p2 contract unchanged.
- **Durable frontier reports.** Followers PUSH their durably-applied frontier
  to the leader once per apply round (new `ReportApplied` RPC, fired by the
  segment receiver through the new `Transport::notify_applied`) — batch-level
  and fully decoupled from ship acks, so the commit index stays fresh even
  when outbound ships stall (gap-parked follower, pull-based catch-up, quiet
  leader). Ship acks keep their m11p2 instant floor-hint semantics — both
  inputs are durable-true because a follower's frontier only ever advances
  after its storage apply + own-WAL fsync. (The first design held each ship
  ack until its segment's apply; measured under open-loop load, that couples
  ship cadence to apply latency and one gap-parked follower spirals into
  total quorum collapse — the report push is what shipped.)
- **Follower blob applies are group-committed.** m11p2 applied replicated
  items/embeddings one record at a time — one solo follower fsync per item,
  capping item apply at the fsync floor (~100/s on macOS) and stalling the
  quorum frontier behind any item burst. The receiver now hands each apply
  round's blob records to the engine as ONE batch (`apply_replicated_blobs`:
  validate all → stage all WAL appends → wait all → upsert storage), and the
  WAL writer flushes queued blobs under ONE group fsync. Measured: corpus
  seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
- **Commit index.** The leader folds durable acks (and heal resumes) into
  per-peer durable marks; the commit index is the k-th largest (k =
  `floor(n/2)`), leadership-scoped (promote resets it to the stream baseline;
  demotion fails in-flight waiters — a demoted leader never claims quorum).
  Handlers await it through a watch-channel bridge — **fully async, zero
  threads parked per waiter** (the thread-per-wait design measurably collapsed
  at 1k rps open-loop by exhausting the blocking pool and starving the very
  completions that advance the index). Quorum capacity measured on a real
  3-process localhost cluster (release, writes mix): **3,600 quorum
  signal-writes/s within SLO** (p50 ~45ms, zero errors, replication lag ≤3
  events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack
  figure, vs the ≥50% gate.
- **Honest timeout semantics.** Quorum not confirmed within
  `replication.quorum_timeout_ms` (default 2000) → a **retryable 503** naming
  the laggard regions, the commit index, and needed/confirmed counts. The
  write is in the leader's log and may still commit: retries are
  at-least-once (items/embeddings retries are idempotent upserts; signal
  retries can double-count — decided in-phase: no idempotency-key machinery,
  documented in runbook §8 with the session-write precedent for callers that
  need exact-once).
- **`x-tidal-seq`** on every cluster write response: the write's seqno in the
  replicated log (relayed through forwards) — an exact durability cursor
  against `commit_index` in `/cluster/status/local` (which also gains `ack`).
  `tidaldb_cluster_relay_durable_seq` now reports the commit index
  (`relay_last_seq − relay_durable_seq` = quorum lag); new counter
  `tidaldb_cluster_quorum_timeouts_total`.
- **The ledger gate (exit gate, run for real):** tier-3
  `mp_quorum_ledger_zero_acked_loss_across_killpoints` SIGKILLs the leader
  under concurrent quorum load and proves zero acknowledged loss on the
  promoted max-applied survivor — frontier invariant (max acked seq ≤
  survivor applied) plus per-item content probes. **167/167 kill points
  passed on the final design** (batches of 64 + 91 + 12; kill timings spread
  120–598ms across fresh 3-process clusters; an earlier 100/100 run had
  validated the superseded ack-holding design before it was replaced — see
  docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics
  (one follower down: quorum commits; both down: fast 503 naming laggards
  while `ack=leader` flows; heal: recovers) and in-process gRPC coverage
  (override headers, forwarded quorum writes, blob writes, 400 on bad mode).
- **Rolling-upgrade order (mixed-version caveat):** a pre-m11p3 leader
  neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack` — it
  silently applies LEADER-ack semantics to a `quorum` request (a durability
  downgrade the caller cannot see). Upgrade the leader first: `ack=quorum`
  is then honored immediately (commit-index freshness rides the m11p2
  ship-ack floor hints until the followers upgrade too). Replication and
  heal are unaffected by either order. Runbook §8 records the procedure.

**One replicated log (m11p2) — items/embeddings ride the WAL, `StreamSegments` catch-up, HTTP broadcast deleted**
- **The leader's WAL is now THE replicated log.** The group-commit writer hands
  every fsynced batch to a bounded in-memory **ship feed** (`wal::feed::WalShipFeed`);
  the ship queue pushes those already-encoded bytes verbatim — byte-identical on
  the leader's disk, the wire, and the follower's apply path — and stream seqnos
  are WAL seqnos, so they **survive restarts** (the m8p10 relay-reset hazard is
  gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning
  machinery left the server write path entirely (`/signals` stages straight
  through the engine's group commit; a WAL fsync failure now surfaces per-write
  exactly like single-node).
- **Item metadata and embeddings are replicated mutations.** `/items` and
  `/embeddings` journal kind-1/2 **blob records** (WAL header `flags` byte =
  batch kind; one record, one seqno) BEFORE storage, on the same stream as
  signals. Followers apply them kind-aware — WAL-first into their own log, then
  idempotent storage upserts — and recovery replays them. The m8p10 HTTP
  item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast,
  authed side-POSTs — the source of both 2026-06-10 live bugs) is **deleted**;
  those bug classes are now impossible by construction. Cluster `/items` returns
  a plain 201 and `/embeddings` a plain 204 (no broadcast-report body).
- **`StreamSegments` implemented — catch-up is follower-pulled.** The ship feed's
  tail is bounded; a peer that falls behind it is skipped ahead, and the
  follower pulls the hole itself via the (previously declared-unimplemented)
  server-streaming RPC over the leader's durable, BLAKE3-verified segments.
  Pulls trigger on detected gaps, on follower boot (self-driving restart
  catch-up), and on the leader's heal nudge (`POST /cluster/catchup`, internal,
  forwards the operator's own bearer credential). Pulled chunks flow through the
  same inbound apply path as live ships. Ship acks piggyback the follower's
  applied seqno (`ShipSegmentResponse.applied_seqno`), so retries of
  already-applied data prune and heal is `resume_from` + nudge — no redelivery
  scan, no O(items) traffic.
- **Promote carries a stream baseline.** A promoted leader's stream starts at its
  promote-time flushed frontier (persisted in `data_dir/stream_baseline`); the
  fan-out body and every catch-up chunk announce it, so peers jump their
  frontier past pre-stream history instead of parking on a phantom gap. Ship
  queues are leadership-gated (`activate_from`/`deactivate`): a follower's
  replicated applies never echo back at its peers.
- **Multi-process cluster mode now requires `--data-dir`** (validated at
  startup): the durable WAL is the replication stream. Standalone single-node
  deployments are untouched — blob journaling and the ship feed are gated on
  cluster peers, so single-node item writes keep fjall-only durability with
  zero extra fsyncs.
- New tier-3 suite `mp_items_ride_the_log_and_catchup_stream`: items written on
  the leader AND through a follower gateway converge everywhere via the log
  (feed parity 1e-6), and a follower stopped through item+embedding+signal
  writes restarts and converges via its boot-time `StreamSegments` pull with no
  heal verb and no HTTP item traffic.

**Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first `tidaldb_cluster_*` metrics**
- The replicated `/signals` write path no longer serializes every writer onto a
  solo group-commit fsync nor ships to followers on the request path. Writes are
  **staged** (seqno + WAL submission + relay log push, microseconds, atomic with
  rollback) and **completed** (shared group-commit fsync + in-memory fold) in two
  phases, so concurrent writers coalesce into one fsync; follower shipping moved
  to per-peer sender threads that coalesce contiguous runs into multi-event
  batches with a windowed in-flight budget (topology knobs
  `replication.{batch_max_events,window,retry_ms}`). The 204 contract is
  unchanged (leader durability only) — it now returns at leader fsync. Measured
  on a real 3-process localhost cluster (release build, thepeach mix):
  **4,534 replicated signal-writes/s sustained within SLO** vs ~90/s before
  (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
- **Durable-frontier shipping + relay poisoning.** Senders only ship the
  leader's contiguous fsynced prefix (an event a follower holds but the leader
  could lose is silent divergence); a staged write whose fsync fails **poisons**
  the relay — further cluster writes are rejected and the ship frontier freezes
  (the CockroachDB/Postgres fsync-failure posture).
- **Follower group-commit coalescing.** The segment receiver drains its inbound
  backlog (`Transport::try_recv_segment`) and applies it through ONE shared
  group commit (`SignalLedger::apply_replicated_events`), in range-disjoint
  groups so duplicate/subset re-ships cannot double-fold. Without this the
  follower apply ceiling was ~`events-per-segment / fsync-cost` (~1.8k events/s
  measured) and lag grew without bound under m11p1 leader rates.
- **First cluster metrics + cluster `/metrics` listener.** Cluster mode
  previously had no metrics endpoint at all. New per-region topology
  `metrics_addr` wires the engine's Prometheus listener; new `tidaldb_cluster_*`
  series: ship RTT + batch-size histograms, per-peer ship counters/gauges
  (`peer_shard` labels), WAL fsync latency + group-commit fill histograms,
  write-pool depth/rejections, relay committed + durable frontiers. WAL
  group-commit knobs are deployment config (`wal.{batch_size,batch_timeout_ms}`
  topology block; `wal_batch_size`/`wal_batch_timeout` builder methods).
- Engine API: `TidalDb::signal_staged`/`StagedSignal::wait`,
  `SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}`,
  `ShipQueue` (per-peer windowed batch senders with pause/resume),
  `WalWriter::append_signal_staged`, `WalSender::append_record_staged`,
  `range_payload`/`encode_run`. Relay log entries are now `RelayEvent` (raw
  event records, re-encoded deterministically at ship time) instead of
  pre-encoded single-event bytes.
- Ship-sender failure logging is transition-based (first + every 50th
  consecutive failure WARN with the running count, recovery INFO) — the
  per-retry WARN flood could fill an undrained log pipe and stall the process.
  The multiproc test harness now discards child logs via `/dev/null` (a piped
  fd nobody drains deadlocks the node once the kernel buffer fills), with
  `TIDAL_TEST_NODE_LOGS=inherit` to stream them while debugging.

**M9 — Community Sync & Revocation**
- Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata`. Community signal reconciliation via `CrdtSignalState` with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

**M10 — Governance & Agent Rights**
- Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

**Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE**
- Multi-process cluster mode: `tidal-server cluster --region <name> [--data-dir <p>]`
  (env `TIDAL_REGION`) runs **one process per region** (`RegionClusterState`), each
  owning one `TidalDb` and one `GrpcTransport` that binds this region's `grpc_addr`
  and dials every sibling's real `grpc_addr` — real process/host isolation. The
  topology requires per-region `grpc_addr` AND `http_addr` in this mode;
  single-process mode (no `--region`) is unchanged as the dev/demo default. Both
  modes stay behind the experimental gate (`--experimental-cluster` /
  `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) with mode-specific WARN text.
- New routes / behaviors on the multi-process surface: `GET /cluster/status/local`
  (per-node status), `GET /cluster/status` aggregates ALL regions with a `reachable`
  field (unreachable ⇒ `reachable:false` + worst-case lag), `POST /cluster/reconcile
  {region}` (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact
  no-op on scores), `POST /hardnegs {user_id,item_id}` (user-scoped hide; converges
  via reconcile, filtered from `/feed?user_id`). Writes on a non-leader **forward to
  the leader**; `?region=` reads forward to the owning region; default reads are
  **LOCAL** in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast
  to peers with a per-peer report: `POST /items` → 201 `{replicated_to, failed}`,
  `POST /embeddings` → **200** with the same report on the leader path (NOT 204 — a
  204 cannot carry the report; the forwarded/internal path stays 204). `/sharded/*`
  fans out across processes (degraded semantics preserved). `promote` fans out to all
  peers (`{ok, leader, acked, failed}`).
- `/cluster/heal` is the single recovery verb: redelivers missed signal segments
  (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region
  (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold
  5, reset 30s) is open, so re-issue `/cluster/heal` until `/cluster/status` shows
  lag 0.
- `TIDAL_HLC_SKEW_MS` (multi-process): signed ms offset applied to the process's HLC
  (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
- Tier-3 UAT over real OS processes (feature `cluster-e2e`): `cluster_multiproc` (5),
  `cluster_chaos` (3, REAL network-partition injection via a root-free in-harness TCP
  relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl
  remain an operator option), `cluster_lifecycle` (2, ±500ms clock skew + rolling
  upgrade with zero acknowledged-write loss), `cluster_runbook` (9, every
  `docs/runbooks/cluster.md` §5–§11 operation), plus `cluster_e2e` (2, single-process
  smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover
  ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

### Fixed

**Four production bugs surfaced and fixed by the m8p10 tier-3 UAT**
- **Silent data loss on out-of-order ships.** The replication applied high-water-mark
  swallowed sequence gaps when eager ships arrived out of order. The applied frontier
  is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
- **Non-idempotent reconcile (score creep).** `take_crdt_snapshot` attributed replicated
  signal streams per-node, so reconciling already-converged nodes crept the decayed
  scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical
  replication shard (`ShardId::SINGLE`), making `merge` idempotent — reconcile of
  converged nodes is an exact fixpoint.
- **Lag gauge conflated leader streams across a promotion.** A converged node reported a
  permanent phantom lag after a `/cluster/promote` moved leadership to a different shard.
  The lag gauge now tracks the leader high-water-mark per source shard
  (`ReplicationLagGauge::leader_seqno_for`), computing lag against the current leader.
- **Items missed during a broadcast were never backfilled.** A node down/partitioned
  during an item/embedding broadcast was permanently missing that data even at signal
  lag 0. `/cluster/heal` now re-broadcasts item metadata + embeddings to the healed
  region (idempotent upserts), making heal the single recovery verb.

### Changed

**Cluster mode (m8p8): real gRPC replication**
- `tidal-server`'s `ClusterState` now wires each follower region to a real
  `tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
  crossbeam channels — replication between regions traverses real gRPC/TCP
  (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
  per region) and self-heal a transient bind race by retrying on a fresh port.
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
  offloaded to a dedicated thread so they never block the axum reactor.
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
  blocks on its own runtime).
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
  single-process cluster mode is honest that it replicates over real gRPC but
  runs all regions in one process (no host/process isolation). (True
  multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry
  above.)

**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
  runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
  `rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
  runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
  bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
  image serves a functional 3-region cluster (write replicates to both followers
  over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
  HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
  + promote over real OS processes, feature-gated).

## [0.1.0] - 2026-02-23

### Added

**Core Database Engine**
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration

**Signal System**
- Typed signal recording with exponential decay scoring
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
- Signal velocity tracking
- WAL-backed signal durability with crash recovery
- Periodic signal checkpointing to fjall (every 30s)
- WAL compaction after each checkpoint

**Retrieval (RETRIEVE query)**
- 5-stage pipeline: universe, filter, score, diversify, return
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
- Custom ranking profiles via `SchemaBuilder`
- Diversity enforcement (max N per category/creator)
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more

**Search (SEARCH query)**
- BM25 full-text search via Tantivy
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
- Creator search with `entity_kind(EntityKind::Creator)`
- `similar_to(EntityId)` for content-based recommendations
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
- Autocomplete suggestions via `db.suggest()`

**Entity Model**
- Three built-in entity types: `Item`, `User`, `Creator`
- Metadata storage as `HashMap<String, String>`
- Embedding slots (up to 4 per entity type) via USearch
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`

**Sessions**
- Session lifecycle: `open_session`, `close_session`
- Cross-session preference vector updates (EMA blend)
- Session snapshots with signal state and preference vectors
- Session serialization format v0x03 with backward compatibility

**Social Graph**
- Creator follower/following indexes
- Cohort membership (user segments)
- CoEngagementIndex for co-viewing patterns with LRU eviction
- Social graph filter for "followed creator" content scoping

**Collections**
- Named collections with `Private`, `Shared`, `Public` visibility
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
- `FilterExpr::InCollection` for collection-scoped retrieval
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`

**Observability**
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
- `TidalDb::diagnostics()` -- structured health snapshot
- WAL diagnostics and recovery tools

**Safety**
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
- Export request limit: 500K signals max per request
- `FilterExpr` complexity limit: 256 nodes max
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
- Schema fingerprint persistence detects decay parameter changes on reopen
- Bounded `closed_sessions` cache (10K max, LRU eviction)
- Metrics server non-loopback bind warning

**CLI (`tidalctl`)**
- `tidalctl` binary for database inspection and diagnostics

**RLHF / ML Export**
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics

### Stability

tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.

---

*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*