tidaldb/CHANGELOG.md

# Changelog

All notable changes to tidalDB will be documented in this file.

## [Unreleased]

### Stability

tidalDB is **production-ready**. M0–M12 are shipped and the pre-release
"not yet recommended for production" disclaimer has been withdrawn from the
documentation set. The API surface and on-disk data format are stable for
shipped features: additive changes ship in minor releases, and any breaking
change to a public API or a persisted format ships with a documented migration
path in this file. This supersedes the `0.1.0` "no stability guarantees" note
below.

### Breaking

**`/sharded/*` writes now require `x-tidal-ack: local` (wire-visible)**

`POST /sharded/{items,embeddings,signals}` hash-partitions by `entity_id` and
applies the write to the **owning region's local store with no WAL append**. It
does not ride the leader relay, so the data has **redundancy 1 regardless of the
replication factor** — an RF3 cluster configured `ack: quorum` does not replicate
it. That is intentional (parallel write throughput across shard owners: 3,669
signals/s measured, vs ~90/s on the replicated path), but the endpoint answered
`201`/`204` with nothing at the call site, in the response, or in the OpenAPI
saying so. An operator probed with `/sharded/embeddings`, found each embedding on
exactly one of three nodes, and filed a durability incident that had to be
retracted.

The three write routes now return **400** unless the request carries
`x-tidal-ack: local`, with a body naming the header and pointing at the
replicating alternative. The existing `x-tidal-ack` header is reused — no second
durability knob — and `local` remains rejected on `/items` / `/embeddings` /
`/signals`, which always replicate. `GET /sharded/{feed,search}` are reads and are
**not** affected. Nothing was rerouted: a `/sharded/*` write that opts in is
byte-for-byte as before, and the surface never silently replicates.

*Migration:* add `-H 'x-tidal-ack: local'` to acknowledge single-copy durability,
or move the write to `POST /items` / `/embeddings` / `/signals` for a replicated
one. Live traffic showed **0 requests** to `/sharded/*` over ~2h of production
load, and the two in-repo callers were migrated with this change.

### Changed

**`/cluster/status` `regions[]` frontiers may now be `null` (wire-visible)**

`applied_events` and `lag_events` are now `Option<u64>` — JSON `null` when the
aggregating node has no frontier report for a region. Previously an unreachable
peer was reported as `applied_events: 0, lag_events: <leader high-water-mark>`;
because lag is `hwm − applied`, a 500 ms probe timeout was rendered as the
leader's entire history as a deficit. Measured live: every node reported both
peers `partitioned: true, reachable: false, lag_events: 13322258` while the **same
response's** `shards[]` array showed all three replicas converged at identical
frontiers with lag 0. An operator reading the obvious field concludes the cluster
is dead.

The root cause was a type that could not express "unknown", so the fix is at the
type: `null` where the value is not known, and no value is derived from one that
is not. That covers three former fabrications — the unreachable peer, a leader
with no ack mark for a peer, and a region whose lag was computed against a leader
high-water-mark the aggregator never learned. `reachable` and `partitioned` are
unchanged and still distinguish a genuinely partitioned peer from an unknown one;
`shards[]` is unchanged (it was the trustworthy half throughout).

*Migration:* clients typed against `applied_events` / `lag_events` must accept
`null`. `tidalctl` renders it as `NO REPORT` / `?` and no longer treats it as a
deficit, so `tidalctl cluster-status && deploy` is usable as a gate again — a
known deficit, an unreachable region, a `partitioned: true` region, a pending
reseed, or a local shard lag still exit 2.

### Fixed

**`/feed` and `/search` `rank` is now dense and ascending (wire-visible)**

Under full placement — the production shape, every shard group on every node —
`missing_groups()` is empty and both handlers return straight out of
`scatter_merge`, which sorted and truncated but never re-stamped `rank`. Each
hosted group ranks its own slice `1..k` locally, the slices were concatenated and
score-sorted, and the per-group counters reached the wire as-is: live
`/search?query=verification` returned ranks **`1, 1, 2`**. The sibling
`merge_cross_shard` already re-stamped for the partial-placement path and
documented exactly why; full placement returned before reaching it.

`scatter_merge` now takes the same `set_rank` closure and stamps after `truncate`,
so there is one rank-stamping mechanism rather than two. **Ordering is unchanged**
— the stamp is a renumbering, not a re-sort, and scores were always correctly
descending. `/vector_search` carries no rank field and passes a no-op. The `S=1`
fast path still returns the engine's own already-dense ranks; both shapes are now
asserted.

*Migration:* a client that deduplicated or keyed on `rank` while working around
the duplicates can drop the workaround. `rank` is `1..n` over the returned page.

**`vector_search` distances now honor the documented `[0.0, 4.0]` contract (wire-visible)**

The read path passed the caller's raw query vector straight to the index while the
write path L2-normalized every stored vector, so the two sides lived in different
spaces. With unit `v`, `d = |q|² − 2q·v + 1`: a non-unit query shifted and scaled
every distance by `|q|²`. Measured on a live RF3 cluster, `/vector_search` returned
distances of **591–1174** against a contract promising `≤ 4`, and an exact match
scored `|q|² − 1` instead of ~0.

`db::query_ops::vector_search_items` now normalizes the query with the canonical
`storage::vector::l2_normalize` before searching. A zero-norm query — which has no
direction, so "nearest by cosine" is undefined — is rejected with **400** instead of
returning an arbitrary ranking.

**Ranking is unchanged.** `|q|²` and `1` are constant across candidates, so the
ordering was already correct cosine order; that is why this went unnoticed. Absolute
uses of the number were the casualties: distance thresholding, dedup-by-distance,
cross-query comparison, and recall measurement.

*Migration:* any client comparing distances against a hard-coded threshold derived
from the old unnormalized values must re-derive it. An exact match now scores ~0
(`2.4e-7` measured; the f16 quantization floor is ~`1e-3`, so compare with a
tolerance, never equality) and all distances fall in `[0.0, 4.0]`.

**Blob replication is now observable (`tidaldb_cluster_blobs_*`)**

The blob path — item metadata, embeddings, term markers, cluster membership — had
**zero** instrumentation: `lag_events` counts WAL apply, so a blob that never shipped
or never applied moved no number anywhere. Added per-kind counters
`tidaldb_cluster_blobs_originated_total`, `_applied_total` and `_apply_failed_total`,
labelled by record kind with cardinality fixed at 4 by construction. Comparing
`originated` on the writing node against `applied` on each peer localises a
replication gap to enqueue, ship, or apply in one query.

**Every co-located shard group now reports its vector count**

`tidaldb_usearch_vector_count` was rendered only for the node's metrics-owner group,
so on a 3-group node two thirds of the corpus had no vector-count series and could
diverge unobserved. Co-located groups now render
`tidaldb_usearch_vector_count{shard="N"}`; the owner's series stays unlabeled for wire
compatibility, so an alert grouped `by (shard)` buckets each replica set separately
without double-counting.

### Added

**Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`**

- **Online preference clustering (`entities/multi_preference.rs`).** A warm user
  (≥ `COLD_START_N = 5` interactions) maintains up to `K_MAX` preference clusters
  built by online sequential k-means with a DP-means threshold split: a new
  engagement updates its nearest cluster (per-cluster adaptive EMA) or, past the
  split threshold and under the cap, opens a new cluster; at the cap the nearest
  cluster absorbs it. Per-cluster *importance* composes the canonical forward-decay
  kernel anchored to each engagement's timestamp, so stale interests fade.
- **Top-M ANN fan-out.** At query time `for_you` selects the top-`M` clusters by
  current importance, issues `M` ANN queries (`candidate_gen::ann_candidates_multi`),
  and merges by best (min) distance; the personalization boost is the max cosine
  over all clusters. Users below the cold-start threshold keep the single
  adaptive-LR vector (`entities/preference.rs`). Design: `docs/research/multi-vector-preference.md`.

**Idle-readiness + TLS scale-up (m12p5–m12p6) — followers converge readiness on an idle cluster, and elasticity is proven over REAL mTLS on k8s**

- **Idle-readiness convergence (m12p5).** The leader heartbeat now carries its live
  frontier (`leader_last_seq`), so a caught-up follower flips `/health` Ready on an
  idle cluster instead of stalling until the next status poll or ship. Cert SAN
  wildcard widened for scale-to-5.
- **TLS scale-up (m12p6).** A real `kubectl scale 3→5` exercised seed-join over
  mTLS on k8s (kind) for the first time: joiners flip Ready in ~13s via the
  idle-readiness heartbeat, auto-promote to Voter, reach full content parity at
  lag 0, with zero acked DATA loss across scale-down. Fixes span a six-bug chain —
  `https://` seed scheme, rustls `CryptoProvider` install order, headless seed
  Service, cold-handshake poll timeout, two-tier cert-manager PKI, and the
  `grpc_tls_for` CA fallback for a not-yet-in-topology joiner. Persists the HNSW
  graph and skips a suspect graph on reseed-pending close.

**Sharded ingestion (m12p4) — scatter-gather across shard groups with cross-shard unified reads**

- **Scatter-gather pool + cross-shard reads.** Writes hash-route across a 3-group
  `shards:` topology; reads unify across groups (L4). Ran REAL on kind with a
  2-generator load job. Fixed an HTTP/2 204 forward-relay bug (a synthesized JSON
  body on a 204 relay triggered an h2 `RST_STREAM`). Confirmed with data: at fixed
  per-pod CPU, full-placement sharding scales failover, not write throughput;
  the ≥2.5×-and-≥5,000/s scaling target remains Ref-A/k3s-pending.

**Index tuning + recall/memory at the production shape (m12p3) — the G2 work: per-query `ef_search` is now honored, the brute-force crossover scales with dimensionality, and the HNSW recall/latency/memory frontier is measured at 1536-D with a real exact oracle**

- **Per-query `ef_search` override — now real.** `UsearchIndex::search` /
  `filtered_search` honor a per-request `ef_search` instead of silently dropping
  it to the index default (pre-m12p3 behaviour: accepted for trait compliance,
  logged a warning, ignored — `USearch` 2.24 has no per-call beam argument). The
  override is race-free via an `RwLock` epoch guard (`with_expansion`): searches
  that agree on `ef_search` run in parallel under a shared guard; only a query
  that changes the live beam width takes the exclusive guard for its
  `(set, search)` window — not a per-search mutex. The knob was already plumbed
  end-to-end in m12p1 (`vector_search_items(.., ef_search)`, the `/vector_search`
  `ef_search` field, `tidal-stress --recall-ef-search`); m12p3 makes it move
  recall. `ef_search=0` selects the slot default.
- **Dimension-aware brute-force → HNSW crossover.** The exact `BruteForceIndex`
  scans every vector under a read lock at `count × dim` cost, so a fixed 10,000
  crossover meant a 15.4M-FMA scan at 1536-D (tens of ms, blocking writers).
  `usearch_min_vectors(dim)` now keeps a brute-force scan within ~4M FMAs:
  ≈10,000 at/under 128-D (byte-compatible with pre-m12p3), ≈2,600 at 1536-D —
  flipping high-dim mid-size slots to HNSW before the scan blows the SLA.
- **`memory_usage()`** exposed on `UsearchIndex` (the true graph + vector
  footprint from `USearch`, not the `index_stats` lower bound) for pod sizing.
- **Grid-search harness.** `cargo run --release --example ann_grid_search` builds
  a `UsearchIndex` + an exact `BruteForceIndex` oracle over the same
  deterministic id-keyed corpus and reports, per `(M, ef_construction, ef_search,
  quantization)` point, measured recall@10 vs the oracle, mean/p99 search
  latency, build time, and the true footprint — the tool that produces the
  documented `M`/`ef` and the F32/F16/Int8 recall+memory numbers.
- **Measured at 1536-D (100k clustered corpus, real exact oracle).** The
  production default (M=16, ef_c=400, F16) **clears G1 and G2**: recall@10 0.997
  at p99 ≈ 1.4 ms raw ANN; ef_search is the latency lever (recall saturates by
  ef_s=128 → p99 ≈ 1.0 ms). F16 costs only 0.25% recall vs F32 for half the RAM
  (≈ 5.2 GB/1M true footprint incl. graph); **Int8 rejected** at 1536-D (recall
  0.715, −28%). Live `tidal-stress --verify-recall` against a real server gave
  recall@10 = 1.0000 at 20k/1536-D, default beam and `--recall-ef-search 400`.
  Finding: the recall corpus is now **clustered** (Gaussian mixture) in both the
  grid harness and `tidal-stress` (`recall::embedding_for`) — uniform-random
  high-dim vectors are pathological for recall@k (≈0.97 at 10k → ≈0.54 at 100k, a
  measurement artifact, not an index regression).
  Full frontier + the 1M command in
  [docs/profiling/usearch-tuning.md](docs/profiling/usearch-tuning.md) and
  [docs/profiling/scale-baselines.md](docs/profiling/scale-baselines.md).
  `docs/specs/07-vector-retrieval.md` updated: per-query `ef_search` is
  IMPLEMENTED, not deferred.

**ANN candidate generation in RETRIEVE (m12p2) — the G1 unblock: `for_you`/`related` source candidates by nearest-neighbour, `trending` by a cached per-signal-type top-K, so the feed stays relevant AND bounded as the corpus grows past the scan cap**

- **ANN in retrieve.** `CandidateStrategy::Ann` is now wired into the RETRIEVE
  executor (it previously fell back to a scan with a warning). The db layer
  resolves the query vector — the user's **preference vector** for `for_you`, the
  **seed item's embedding** (`similar_to`) for `related` — and Stage 1 runs an
  `O(ef_search)` HNSW search over the item content slot instead of scanning an
  arbitrary low-id slice of the universe. `for_you` and `related` are now `Ann`
  profiles. Graceful: no registry / no preference vector / no seed ⇒ it degrades
  to a scan (anonymous reads and cold-start users still serve), so every
  embedding-less schema and pre-m12p2 caller is unchanged.
- **Cached `SignalRanked`.** The O(N) ledger scan behind `SignalRanked` is now a
  cached per-signal-type top-K (`signals/ledger/hot_top_k.rs`): O(K) on the
  served path, with a bounded O(N) rebuild only when stale. Decay preserves
  relative order (same λ), so a cache is valid until the next write; small
  ledgers rebuild on any write (always fresh), large ledgers throttle the rebuild
  off the read hot path (1s). `trending` now uses `SignalRanked(view)`, so it
  ranks the actually-viewed corpus at any id — not the low-id scan slice.
- **`related` over HTTP.** `GET /feed?profile=related&similar_to=<id>` resolves
  the seed's embedding and runs ANN — "more like this" on the read surface
  (`similar_to` added to `FeedQuery`, threaded through all three feed handlers).
- **Harness `/feed` measurement.** `tidal-stress` gains `--feed-profile <name>`
  (force every feed read to one profile, for per-profile retrieve p99) and
  `--seed-preferences` (build a preference vector per user so `for_you` exercises
  ANN, not the scan fallback).
- **Verified real** against a 1536-dim standalone server: **trending retrieve
  p99 = 3.5–7.7ms** (under the 10ms G1 target) under concurrent writes, via the
  cached top-K; `for_you` retrieve is ANN-backed (preference vectors built) at
  p99 ≈ 24ms, dominated by the Stage-3 preference-boost recompute (per-candidate
  embedding read) — flagged for m12p3's index/score tuning (the risk register's
  "materialized-score layer"). ANN candidate recall is the m12p1 `/vector_search`
  probe (0.9997). New engine tests prove ANN/`related`/`trending` reach the
  relevant items at high ids a scan can never reach, plus cache freshness.

**Read-recall harness — measurement truth for ranking-at-scale (m12p1): recall@k + true p99 at the production shape, so G1/G2 steer on real numbers instead of absent ones**

- **Pure k-NN probe.** `TidalDb::vector_search_items(query, k, ef_search)` returns
  the raw HNSW nearest neighbours over the item content embedding slot — NO
  profile scoring, fusion, or diversity — so the result is the ANN index quality
  in isolation (the G2 metric). Exposed as `POST /vector_search` on the standalone
  router and the multi-process region node (merge-by-distance across hosted shard
  groups); a dimension-mismatched query is a 400, not a 500.
- **The oracle.** `tidal-stress --verify-recall` seeds the corpus with
  **deterministic, id-keyed** embeddings (reproducible with `--skip-seed`), holds
  a **brute-force cosine ground truth** in RAM, and ramps `/vector_search` probes
  **open-loop** (coordinated-omission corrected). It reports per-stage **true p99**
  (a genuine tail, not a closed-loop mean) AND **mean recall@k**, plus the
  **read-knee** — the highest sustained QPS where `p99 ≤ target AND recall@k ≥
  target` both hold — with a machine-readable JSON summary and `--fail-on-knee`
  PASS/FAIL exit. Knobs: `--recall-k`, `--recall-queries`, `--read-p99-target-ms`,
  `--recall-target`, `--recall-ef-search`.
- **Verified end-to-end** against a real standalone server at 1536-dim:
  **recall@10 = 0.9997** at 20k items (HNSW M=16/ef=400/F16 vs brute-force cosine,
  far above the 0.95 G2 target); the harness also exercises the read-knee verdict
  (both branches) and gate exit codes. The 100k/1M exit-gate runs use the same
  harness on the k3s cluster (the brute-force oracle needs ~6 GB RAM at 1M).
- **No more mean-as-p99.** Repaired the fabricated p99 column in
  `docs/profiling/social-scale.md` (the `social` bench is Criterion = mean only)
  and the `scale.rs` / scale-baselines framing; every closed-loop number is now
  labelled an *isolated per-op mean (regression tripwire)*, with the p99/recall
  tail SLOs signed off only by the open-loop harness. Added a 1536-dim
  HNSW-vs-brute `recall@10` bench (`benches/vector.rs`) and a recall-harness
  section to `docs/profiling/scale-baselines.md`.

**Sharding × replication + rebalancing (m11p6) — the "replicated XOR sharded" split is over: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes**

- **One write surface.** `/items`//`/embeddings`//`/signals` now hash-route the
  entity to the owning shard group's leader (the engine's FNV-1a `ShardRouter`)
  AND replicate at RF — sharding and replication at once. The old
  `/sharded/*`-vs-leader split in `tidal-stress` (`WritePath::Leader|Sharded`) is
  gone; it drives the one unified path, spreading writes round-robin across
  gateways (or pinning one with `--leader-url`). `x-tidal-ack`/`x-tidal-seq`,
  quorum await, and `NotLeader`/`QuorumTimeout` are per-group; `NotLeader` names
  the group.
- **Rebalancing verbs (L3).** `POST /cluster/shards/{id}/transfer` moves one
  group's leadership (the m11p4 fenced transfer scoped to the group);
  `POST /cluster/shards/{id}/replicas` adds/removes a replica (the m11p5 join /
  fenced-removal per group). A `?shard=` selector threads through every per-shard
  admin verb (promote/heal/partition/catchup/reseed/members/join) and is
  PROPAGATED on every intra-group forward/broadcast (`ShardReplica::admin_path`)
  so the receiving sibling targets the same group. **S=1 is byte-for-byte** (no
  selector emitted, no shard in the `NotLeader` body).
- **Tier-3 exit gate** (`cluster_sharding.rs`): 3 nodes × 3 shards × RF=3 over
  real OS processes — SIGKILL a node under `ack=quorum` load → ONLY its
  shard-leaderships re-elect (survivor groups keep theirs), reads never stop, and
  every acked write is present on its shard's new leader (zero acked loss), across
  random kill points. Plus a rebalance-verb test (transfer + `?shard=` promote
  move exactly one group). Harness: `MultiProcCluster::start_sharded`
  (per-(node,shard) ports, `shards:` emission, `agreed_shard_leaders`).
- **Throughput.** Local 3×3 release cluster sustains 3,000 quorum signal-writes/s
  at 0% error with per-node CPU ≈30% and replication lag ~0 — generator-bound,
  not engine-bound. The ≥5,000/s + ≥2.5×-single-shard scaling is Ref-A (k3s
  Linux/`fdatasync`/multi-generator) — the standing access caveat since p1.
- Known follow-up (tracked): per-group-aware node readiness (today `is_ready` is
  node-global across co-hosted groups) and cross-node read fan-out under PARTIAL
  placement; the exit gate runs full placement, which these do not touch.

**Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix**

- **Fault injection compiled out of production.** A non-default `fault-injection`
  cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks
  (`tidal/src/fault.rs`): **slow-fsync** (`TIDAL_FAULT_FSYNC_DELAY_MS` sleeps
  before each durable fsync) and **disk-full** (`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`
  returns a real `ENOSPC`/errno-28 once cumulative segment bytes cross the
  threshold). The production image build never passes the feature, so the hooks
  are compiled out entirely — a disk/fsync fault a stray env var could trip in
  prod is a 3am footgun we refuse to ship; the safety is structural. Inert until
  armed even when compiled in. The tier-3 harness builds the spawned binary with
  the feature.
- **First-class invariant checkers** (`tidal-server/tests/support/invariants.rs`):
  the no-acked-loss `AckLedger` (frontier + content, extracted from the m11p3
  ledger gate, which now consumes it), cross-replica `assert_feed_parity` /
  `feed_item_ids`, `assert_single_leader_per_term` (membership safety, reads the
  m11p6 `shards[]` rows), and `MonotonicCounters` (per-node applied frontier +
  leader commit index never regress, with legitimate epoch resets forgiven).
- **New fault-class suite** (`tidal-server/tests/cluster_faults.rs`, tier-3): a
  disk-full follower degrades gracefully (receiver halts, node alive, zero acked
  loss, restart recovers to parity); a slow-fsync follower lags then converges
  while the fast follower supplies quorum; both followers slow → `ack=quorum`
  returns a retryable 503 naming the laggards while `ack=leader` is unaffected;
  an asymmetric partition (inbound severed, outbound up) causes no split brain
  (pre-vote + check-quorum) and no loss. 4/4 green.
- **Soak + regression gates** in `tidal-stress`: `--json-summary <path>` (a
  machine-readable per-stage p99/throughput/error roll-up for trend lines) and
  `--max-p99-ms` / `--max-error-pct` / `--fail-on-knee` gates that exit non-zero
  on a regression (the tool always exited 0 before).
- **Nightly pipeline** (`.woodpecker.yaml`): a cron `nightly` flow (chaos suites
  with elevated kill-points + the `fault-injection` feature, then the gated
  `tidal-stress` soak) beside the existing push release gate, event-routed by
  per-step `when`. Woodpecker, never GitHub Actions.
- **Guarantee traceability** (`docs/planning/milestone-11/guarantee-traceability.md`):
  every roadmap §2 guarantee mapped to its named automated test(s). Closes the
  **G-C** apparatus; the 30-consecutive-days-green half of the GA bar is a
  calendar criterion the nightly pipeline accrues.

**Observability + operations (m11p8) — complete cluster metric set on a per-node `/metrics` listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate**

- **Metrics.** Completed the `tidaldb_cluster_*` set with the two members the
  roadmap named that were missing — **breaker** (`tidaldb_cluster_peer_breaker_state`
  0/1/2 + `tidaldb_cluster_breaker_opens_total`, surfaced read-only from the
  `tidal-net` circuit breaker via a new `Transport::peer_breaker_state`) and
  **forwards** (`tidaldb_cluster_forwards_total` / `_forward_failures_total`,
  instrumented at the gateway forward path) — plus the self-heal series
  (`heal_attempts/successes/noops_total`, `healing_peers`). When several shard
  groups co-locate on one node (m11p6) the metrics-owner serves the single
  `/metrics` listener and the siblings register their series under a `shard="N"`
  label (`TidalDb::register_metrics_sibling`); a single-shard node is
  byte-identical to before. A 12-panel "Cluster Replication" Grafana row +
  an 8-rule `tidaldb-cluster` Prometheus alert group ship beside the standalone
  ones.
- **Request-id + tracing.** Both cluster routers (single- and multi-process) now
  carry the standalone router's `SetRequestId` + `PropagateRequestId` +
  `TraceLayer` stack (extracted to `router::with_request_id_tracing`); the id
  rides the follower→leader forward hop verbatim, so the leader's span shares the
  gateway's `x-request-id`. (Ships are off-request-path and batched, so they
  correlate by seqno, not a request-id — by design.)
- **Truthful status.** Fixed the leader's own `applied_events` reading 0 (a leader
  writes its WAL directly and never advances its own applied frontier — now
  reports its flushed frontier) and the single-process post-promote `ShardId(0)`
  lag-keying undercount (now keys on the current leader's shard).
- **Self-driving heal.** A standing leader duty re-arms the backlog re-ship for a
  stuck (breaker-open, behind) peer every ~3s, so it converges through breaker
  resets with no operator `/cluster/heal` loop — closing the §1.4-3 footgun.
  Observable via `healing_peers` + the `TidalDBClusterHealNotConverging` alert.
- **WAL PITR archival.** `wal.archive_dir` (topology + builder): the online
  compaction copies each sealed segment to the archive — durably, before deletion,
  refusing to delete if archival fails — so the archive is a gap-free PITR record.
- **Backup/restore.** `tidalctl backup` / `restore`: offline data-dir backup with
  a BLAKE3 `BACKUP_MANIFEST.json` (+ the WAL checkpoint cursor); restore verifies
  every file's hash before writing and refuses a non-empty target. Coordinated
  cluster backup = back up one committed replica per shard group (the runbook
  drill).
- **Rolling upgrade.** A wire version handshake (`HeartbeatRequest.build_version`,
  stamped at the transport boundary; `>= 2`-major skew WARNs, never rejects) +
  `version` on `/cluster/status`. `mp_rolling_upgrade_no_loss_no_stall` promoted
  to the FIRST step of `.woodpecker.yaml` (the release gate; a failure blocks the
  image build). See [milestone-11/phase-8.md](docs/planning/milestone-11/phase-8.md).

**Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit**
- **The cluster stops trusting the network.** gRPC replication mTLS is now the
  intended posture: the inbound server is served over a custom `tokio-rustls`
  acceptor (not tonic's fixed `.tls_config()`) fed a `DynamicCertResolver`
  (`ArcSwap<CertifiedKey>`), preserving mutual TLS exactly (a `WebPkiClientVerifier`
  over the cluster CA — a foreign/absent client cert fails the handshake before
  any RPC). Plaintext is an explicit `insecure: true` with a loud startup WARN.
- **Cert + bearer rotation WITHOUT restart.** A content-hash poller re-reads the
  cert files (k8s `..data` symlink swaps that inotify misses) and atomically
  swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys,
  so a rotation **drops zero requests** (verified under concurrent load). Outbound
  peer channels rebuild from the refreshed files. The bearer and a new shared
  cluster key live behind `ArcSwap`, read per request and reloaded from
  `TIDAL_API_KEY_FILE` / `TIDAL_CLUSTER_KEY_FILE`.
- **Authenticated inter-node HTTP with per-node identity.** The axum listener
  serves TLS (reusing the same hot-swappable resolver — one rotation covers both
  planes); forwards/broadcasts/scatter/status/seed-join dial `https://` with the
  cluster CA. A forwarding node mints an `x-tidal-node-token` (keyed-BLAKE3 MAC
  over node-id + expiry under the cluster key — no new crypto dependency) so a
  foreign pod cannot forge a sibling identity. The `x-tidal-internal` marker is
  now honored ONLY from a verified sibling (marker without a valid node token →
  403): the marker stays a routing hint, never an authorization bypass. All
  opt-in via `grpc_tls` / the cluster key — absent ⇒ pre-m11p7 behavior, so every
  existing deployment and test is byte-for-byte unchanged.
- **Admin-verb audit log.** promote / partition / heal / join / member-remove /
  reseed each emit one structured record (principal, term, target, outcome) to a
  `tidal_audit` tracing target + an optional append-only JSONL file
  (`TIDAL_AUDIT_LOG`), on the operator-originated leg only (no double-audit on the
  forwarded re-apply).
- **Per-principal HTTP rate limit.** The engine's token-bucket `RateLimiter`
  (now re-exported from the crate root) gates all three routers keyed by
  principal; verified sibling nodes are exempt (replication is never throttled);
  a deny is 429 + `Retry-After`. Off by default (`TIDAL_RATE_LIMIT_RPS`).
- **Reference deployment + tooling.** `k8s/cluster/` gains cert-manager
  Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region
  `grpc_tls` topology block; `scripts/gen-cluster-certs.sh` (openssl) provisions
  the same Secret shape without cert-manager. Exit gate verified: foreign pod
  rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext
  inter-node links. See [docs/planning/milestone-11/phase-7.md](docs/planning/milestone-11/phase-7.md).

**Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference**
- **Nodes are cattle; topology is data, not files.** `grpc_addr` is now an
  **advertised** address — a hostname or an IP — split from a new optional
  per-region `grpc_bind` (the local SocketAddr to bind). A literal `grpc_addr`
  binds itself byte-for-byte as before (every existing topology keeps working);
  a hostname binds `0.0.0.0:<port>`. `tidal-net`'s peer map retypes from
  `SocketAddr` to `String`, so `Channel::from_shared` makes hyper **re-resolve
  DNS on every reconnect** — the pod-rescheduled-onto-a-new-IP case the
  `SocketAddr::parse`-only constraint made structurally impossible is now fixed
  by construction. One shared topology file can name every region by its per-pod
  DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names
  require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code
  change). Proven by `cluster_membership.rs::mp_dns_hostname_topology_replicates`
  (a hostname topology a pre-p5 binary would have refused to parse boots,
  replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
- **A new node joins via snapshot + stream and serves quorum in minutes.**
  `FetchSnapshot` is a new server-streaming RPC in the `WalShipping` service:
  the leader stages `TidalDb::create_backup` under `<data_dir>/snapshots/` and
  streams it term-stamped + term-fenced exactly like `StreamSegments`
  (manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to
  end). The handler answers a **no-snapshot-needed** header when the WAL can
  still serve the puller's range and streams the artifact only when it cannot —
  the `needed` decision is **baseline-aware** (`from_seqno ≤ stream_baseline OR
  < earliest WAL seq`), because a "the WAL still covers it" answer is a lie at
  or below a baseline the stream clamp will never serve, and looped marker boots
  forever. Install is a **boot-time** operation (never a live data-dir swap):
  fetch into a sibling staging dir, copy the node's own identity files
  (`election_state`, `membership`) INTO staging, write a `COMPLETE` sentinel,
  then rename — every crash window is an idempotent redo. A retention pin taken
  at backup **start** keeps the segment chain covering the artifact alive while
  any joiner streams, with a hard cap + `tidaldb_cluster_snapshot_pin_force_drops_total`
  so a dead joiner can't freeze compaction forever.
- **Reseed is self-healing — no operator verb, no `wipe_data_dir`.** A running
  follower that hits a **typed** snapshot-required refusal (the `StreamSegments`
  trailer is now structured — `x-tidal-catchup: snapshot-required | rejoin |
  stepping-down` — so an ordinary election term-mismatch never latches reseed
  markers fleet-wide; the marker latches **only** on `snapshot-required`), or the
  m11p4 divergence quarantine, durably latches `reseed_required` (ElectionStore
  file discipline), surfaces it in `/cluster/status/local` and the
  `tidaldb_cluster_reseed_required` gauge, and keeps serving degraded with
  **voting enabled** (a reseed needs a leader, and the leader may need this
  node's vote — reseed-blocks-voting would deadlock the cluster). The reseed
  runs on the next boot; `replication.reseed_self_restart: true` (default
  **false**; k8s sets it `true`) drains and clean-exits once the marker latches —
  **refused** when the remaining voters can't sustain quorum without this node.
  A successful reseed boot clears the quarantine latch and
  `tidaldb_cluster_divergence_quarantined` — closing the m11p4 carried hazard
  that quarantine clearing "rides m11p5's reseed machinery." `cluster_reseed.rs`
  proves the full quarantine → marker → restart → converged → gauges-cleared
  loop with no `wipe_data_dir` in the test.
- **The three-way term-join rule closes p4's last carried hazard** (pre-baseline
  history was unreachable through a term's stream for followers behind at the
  transfer). The first reseed drill exposed that such a follower jumps its
  frontier past pre-baseline history with `lag=0` and never asks for the gap —
  silently missing data with nothing to fire the refusal. The heartbeat now
  carries the leader's election-time position in the previous stream's numbering
  (`prev_log`), and the join check is three-way: `own > prev_log` → divergent
  suffix → quarantine (p4); `own < prev_log` → genuinely missing committed-era
  history → **latch `reseed_required`** (new); `own == prev_log` (or within-term
  rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL
  is the leader's copy, so `tail_term` equals the leader's term).
- **Membership is data on the one log: kind-4 records, learner → voter.** A new
  `MembershipRecord` WAL blob kind (kind-4, beside kind-0 signals, kind-1/2
  item/embedding blobs, kind-3 term markers) is journaled by the leader,
  replicated through the normal stream, and folded by followers into a
  `ClusterMembership` cell (the `WalTermMark` pattern). Records carry the FULL
  roster (latest record wins; no merge logic); membership **epoch 0 = the
  topology file**, so a cluster with no kind-4 record behaves byte-for-byte as
  today. `POST /cluster/join {name, grpc_addr, http_addr}` on any node forwards
  to the leader, which assigns `id = max(all ids ever) + 1`, appends a **Learner**
  record, and answers only after it is quorum-committed (idempotent by name);
  member ids are **permanent** — removal tombstones (a `Removed` record) keep ids
  burned, never renumbered or reused. **Auto-promotion is a standing leader duty**
  (re-armed on every activation and apply, evaluated on commit-index publishes,
  driven solely from the applied `ClusterMembership` — it survives the
  joining-era leader's death) that promotes a learner within
  `replication.learner_promote_lag` (default 1024) of the flushed frontier, or
  the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server
  (one at a time, each same-term-quorum-commit-gated); on every leadership
  activation the new leader **re-appends its full current membership** immediately
  after the kind-3 term marker (the Raft no-op-entry analogue), which closes both
  the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs:
  `GET /cluster/members`, `POST /cluster/members/remove`, `POST /cluster/reseed`.
- **The even-voter-count `majority()` arithmetic was a latent bug — fixed.**
  `ElectionConfig::majority()` was `peers.len()/2 + 1`: correct at n=3, **wrong
  for every even voter count** (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D}
  could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with
  zero RPCs). `CommitIndex` already used the correct `div_ceil` form — the two
  formulas in one subsystem disagreed. Since p5 makes even sizes mandatory
  transit states (3→4→5→4→3), this landed **first**, with even-n property tests:
  `majority() = (peers.len() + 1).div_ceil(2) + 1` (true `floor(n/2)+1` over the
  full voter set), spelled to read identically to the commit-index site so they
  can never drift again.
- **Learner marks never count toward quorum.** A role-blind `CommitIndex` that
  saw learner marks would treat two learners "committing" a write no voter holds
  as acked — acked-write loss. `CommitIndex` is now role-aware: voter marks feed
  the k-th-largest selection; learner marks live in a side map (promotion input,
  transfer-wait input) and are excluded from `needed`. In-flight `ack=quorum`
  waits are **re-evaluated** against a new config on every conf-change (a shrink
  may satisfy waiters instantly), never failed; a report from an unknown id logs
  at WARN with a counter instead of being silently dropped. `ElectionState`,
  `ShipQueue`, and `PeerPool` reconfigure behind one fenced apply path so the
  four peer-set copies can never disagree about the roster.
- **Seed-join boot.** `tidal-server cluster --region <name> --seed
  http://host:port (repeatable) --advertise-grpc host:port --advertise-http
  host:port --metrics addr`: the joiner skips the "every region declared"
  topology gate, learns its roster + assigned id + current term from any
  reachable seed, persists them to a durable membership cache + `election_state`
  (persist-before-act), and installs a snapshot when behind. A restart boots from
  the cache without the seed. A `--seed` boot **still requires the local
  topology/config file** for the behavioral knob blocks (`replication:`, `wal:`,
  `election:`, `timeouts:`, `grpc_tls`) — a bare `--seed` with neither
  `--topology` nor `TIDAL_CONFIG` refuses to boot naming the rule (so a joiner
  never silently inherits the wrong ack default or election timing).
- **Kubernetes reference: one StatefulSet.** `k8s/cluster/` (namespace
  `tidaldb-cluster`, `replicas: 3`, `podManagementPolicy: Parallel`,
  topology-spread, PDB `maxUnavailable: 1`) with ONE shared bootstrap topology
  ConfigMap (per-pod DNS advertised addresses, `0.0.0.0` binds). Scaling past 3
  does NOT edit it — pod N≥3 boots `--seed` + the same mounted file and joins as
  a learner; rolling node replace is `kubectl delete pod` (PVC retained → boot
  catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness:
  `converged` (boot catch-up completed at least once AND `lag ≤
  learner_promote_lag` — hysteresis, never `lag == 0` which an open-loop load
  keeps perpetually false); a restarted existing voter is Ready on today's terms.
- Exit-gate suites (tier-3, `cluster-e2e`): `cluster_membership.rs`
  (`mp_seed_join_snapshot_catchup`, `mp_scale_3_5_3_under_load_zero_loss` —
  `lost=0`, max acked seq 1467, p99 impact <2× across the 3→5→3 joins,
  `mp_dns_hostname_topology_replicates`) and `cluster_reseed.rs` (quarantine →
  marker → restart → converged, gauges cleared). Localhost-loopback catch-up:
  5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget);
  the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line
  items (k3s access still pending — the standing M11 caveat).
- **Mixed-version / downgrade caveats.** A kind-4 record shipped to a **pre-p5
  follower** is an unknown batch kind → `WalError::Corruption` → its receiver's
  torn-state halt latch, **permanent across restarts** (both followers halted =
  quorum-write outage). So `HeartbeatResponse`/`ReportApplied` carry a
  `capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
  leader **refuses `JoinCluster` and every conf-change until all current voters
  report kind-4 capability** — "complete the binary upgrade before the first
  conf-change" is structurally enforced, not operator discipline. Pre-p5 peers
  answer `Unimplemented` to `JoinCluster`/`FetchSnapshot` (the joiner reports it
  loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim):
  once any kind-4 record is in a node's WAL, **downgrade below p5 requires a
  reseed**.

**Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5**
- **"A machine died" is a non-event.** Every multi-process cluster node runs a
  purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced
  leadership transfer) over the existing `tidal-net` transport: leader
  heartbeats every 300ms (config `election.heartbeat_interval_ms`); a follower
  that hears no leader for a randomized 1500–3000ms starts a pre-vote; a
  majority elects. SIGKILL the leader under `ack=quorum` load → a survivor is
  elected and writes resume in well under a second locally, with **zero
  acknowledged-write loss across repeated random kill points** (tier-3 gate
  `cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss`).
  No raft crate: the WAL is already the replicated log (m11p2) and the quorum
  commit index (m11p3) already proves majority durability — the election adds
  only the tiny consensus state `(term, leader)` on top of them.
- **The WAL carries its own election history.** An elected leader's FIRST log
  entry is a kind-3 **term-marker record** that replicates like any record, so
  every replica's `(lastLogTerm, lastLogIndex)` — Raft's vote restriction — is
  derived from one fsync stream and can never disagree across a crash. The
  frontier half is compared in the **last joined term's stream numbering**
  (a reseeded node's own WAL numbering diverges from the stream's after a
  baseline jump; comparing raw local frontiers across nodes would let a
  behind node win). Hard state `(current_term, voted_for)` persists in
  `data_dir/election_state` (magic + version + checksum; corrupt → refuse to
  boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE
  any vote reply or leadership claim leaves the node.
- **Term fencing everywhere (kills incident §1.4-1).** Every replication RPC
  carries the sender's term: stale-term ships/chunks/heartbeats/frontier
  reports are rejected, the quorum commit index folds only reports stamped
  with its activation term (race-free, under the index's own lock), and a
  restarted ex-leader boots as a FOLLOWER from its durable state — never from
  the topology file. A partitioned ex-leader that restarts cannot accept a
  single write (`mp_fenced_ex_leader_restart_cannot_write`); a leader that
  loses majority contact steps down within `election.leader_lease_ms`
  (default 900; validated `lease + heartbeat < election_timeout_min` so a
  deposed leader stops before any successor can exist).
- **Divergent suffixes quarantine instead of lying.** A node whose log
  extends past what the elected leadership subsumed (leader-acked,
  never-quorum-acked writes on a dead leader) detects it at term-join — the
  heartbeat carries the leader's election-time log position — and fences
  itself from the data plane (status `quarantined: true`, metric
  `tidaldb_cluster_divergence_quarantined`, ERROR naming the reseed runbook)
  while still voting. Followers apply on receipt, so un-applying is not a
  thing: reseed is the honest recovery (m11p5 snapshots automate it).
- **`/cluster/promote` is now a fenced transfer**: with a live leader it
  drains (waits for the target to hold the flushed prefix, breaker-immune via
  the commit index's marks) then sanctions an immediate election; with a dead
  leader the target campaigns. The election refuses a target whose log lags —
  the m11p3 "promote the max-applied survivor" operator rule is now enforced
  by the protocol. The legacy term-0 fan-out survives only on clusters that
  have never elected (mixed-version rollouts; the chaos drills' deliberate
  isolated-node override) and is permanently retired per node at its first
  joined election. `election.auto_election: false` preserves the full
  pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
- **Bounded churn under flapping links**: the pre-vote (no term inflation
  without a majority probe) plus the leader-freshness lease absorb short
  flaps entirely and bound terms under long ones
  (`mp_flapping_links_bounded_churn`). Election observability:
  `tidaldb_cluster_election_term/_role/_elections_started_total/
  _leader_changes_total`, and `/cluster/status/local` grows `term`, `role`,
  `quarantined`, `prev_log_term/seq`.
- New tier-3 exit-gate suite `cluster_election.rs` (auto-failover ledger,
  fencing under partition+restart, bounded churn); `cluster_quorum.rs` and
  `cluster_lifecycle.rs` pin `auto_election: false` (they validate the manual
  drill, which remains supported); the partition harness gained
  bidirectional isolation (`isolate_region`) — severing only a node's inbound
  edges leaves its outbound heartbeats/votes flowing, which silently defeats
  partition scenarios.

**Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, `TSEG` segment header, structured "snapshot required"**
- **Failed catch-up pulls retry on a timer.** The pull trigger was event-only:
  a follower whose `StreamSegments` pull failed (e.g. the leader's gRPC server
  not yet ready during a rolling restart) waited for the next PUSHED segment
  to re-expose the gap — in an idle cluster that push never comes, and the
  follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers
  stuck at lag=136507). A failed pull now arms a one-shot timer
  (`replication.catchup_retry_ms`, default 30000; transport
  `catchup_retry_interval`) that re-pulls from the CURRENT applied frontier.
  Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when
  consumed by the rate limit or an in-flight pull, so the gap always keeps a
  standing wake-up until a pull completes. Verified over real sockets:
  `catchup_retry.rs` reproduces the incident (pull fails, leader appears,
  zero pushes) and proves timer-only self-heal — and that a clean completion
  arms nothing.
- **WAL segment files are format-versioned.** New segments open with an
  8-byte header (`TSEG` magic + version byte + reserved); pre-m11p4
  headerless segments stay readable as implicit version 0 — no migration. A
  segment this binary cannot identify (unknown header version, unrecognized
  leading bytes, unparseable `.seg` filename) surfaces as the new
  `WalError::SegmentFormatUnknown` at open — previously it scanned as
  empty (`segments=0`) and recovery's torn-tail repair could TRUNCATE the
  foreign file to zero. Foreign-format files are never repaired, truncated,
  or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
- **Unservable catch-up is a structured refusal.** `SegmentSource::collect_from`
  returns typed `SegmentReadError::{Unavailable,Failed}`;
  `TidalDb::read_wal_batches` returns the typed `WalError` (was stringified
  `TidalError`). The `StreamSegments` handler maps `Unavailable` to
  `FAILED_PRECONDITION` — `"segments not available from seq N; snapshot
  required"` — and the follower logs it distinctly (*catch-up unservable …
  needs a snapshot (m11p5) or an operator reseed*) instead of burying it as a
  transient. The on-disk segment format is now documented
  (`tidal/src/wal/segment.rs` module docs + spec 01 §2.2).

**Quorum-acked writes (m11p3) — `ack=leader|quorum`, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)**
- **`ack=quorum`** is an opt-in durability contract for every replicated write
  (`/signals`, `/items`, `/embeddings`): success means a **majority of the
  replica set durably holds the write** (leader + `floor(n/2)` followers, each
  storage-applied and own-WAL-fsynced), so an acked write survives the
  permanent loss of any single node — including the leader. Deployment default
  via topology `replication.ack`; per-request override via the **`x-tidal-ack`**
  header (forwarded verbatim by gateways). `ack=leader` (the default) is the
  m0–m11p2 contract unchanged.
- **Durable frontier reports.** Followers PUSH their durably-applied frontier
  to the leader once per apply round (new `ReportApplied` RPC, fired by the
  segment receiver through the new `Transport::notify_applied`) — batch-level
  and fully decoupled from ship acks, so the commit index stays fresh even
  when outbound ships stall (gap-parked follower, pull-based catch-up, quiet
  leader). Ship acks keep their m11p2 instant floor-hint semantics — both
  inputs are durable-true because a follower's frontier only ever advances
  after its storage apply + own-WAL fsync. (The first design held each ship
  ack until its segment's apply; measured under open-loop load, that couples
  ship cadence to apply latency and one gap-parked follower spirals into
  total quorum collapse — the report push is what shipped.)
- **Follower blob applies are group-committed.** m11p2 applied replicated
  items/embeddings one record at a time — one solo follower fsync per item,
  capping item apply at the fsync floor (~100/s on macOS) and stalling the
  quorum frontier behind any item burst. The receiver now hands each apply
  round's blob records to the engine as ONE batch (`apply_replicated_blobs`:
  validate all → stage all WAL appends → wait all → upsert storage), and the
  WAL writer flushes queued blobs under ONE group fsync. Measured: corpus
  seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
- **Commit index.** The leader folds durable acks (and heal resumes) into
  per-peer durable marks; the commit index is the k-th largest (k =
  `floor(n/2)`), leadership-scoped (promote resets it to the stream baseline;
  demotion fails in-flight waiters — a demoted leader never claims quorum).
  Handlers await it through a watch-channel bridge — **fully async, zero
  threads parked per waiter** (the thread-per-wait design measurably collapsed
  at 1k rps open-loop by exhausting the blocking pool and starving the very
  completions that advance the index). Quorum capacity measured on a real
  3-process localhost cluster (release, writes mix): **3,600 quorum
  signal-writes/s within SLO** (p50 ~45ms, zero errors, replication lag ≤3
  events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack
  figure, vs the ≥50% gate.
- **Honest timeout semantics.** Quorum not confirmed within
  `replication.quorum_timeout_ms` (default 2000) → a **retryable 503** naming
  the laggard regions, the commit index, and needed/confirmed counts. The
  write is in the leader's log and may still commit: retries are
  at-least-once (items/embeddings retries are idempotent upserts; signal
  retries can double-count — decided in-phase: no idempotency-key machinery,
  documented in runbook §8 with the session-write precedent for callers that
  need exact-once).
- **`x-tidal-seq`** on every cluster write response: the write's seqno in the
  replicated log (relayed through forwards) — an exact durability cursor
  against `commit_index` in `/cluster/status/local` (which also gains `ack`).
  `tidaldb_cluster_relay_durable_seq` now reports the commit index
  (`relay_last_seq − relay_durable_seq` = quorum lag); new counter
  `tidaldb_cluster_quorum_timeouts_total`.
- **The ledger gate (exit gate, run for real):** tier-3
  `mp_quorum_ledger_zero_acked_loss_across_killpoints` SIGKILLs the leader
  under concurrent quorum load and proves zero acknowledged loss on the
  promoted max-applied survivor — frontier invariant (max acked seq ≤
  survivor applied) plus per-item content probes. **167/167 kill points
  passed on the final design** (batches of 64 + 91 + 12; kill timings spread
  120–598ms across fresh 3-process clusters; an earlier 100/100 run had
  validated the superseded ack-holding design before it was replaced — see
  docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics
  (one follower down: quorum commits; both down: fast 503 naming laggards
  while `ack=leader` flows; heal: recovers) and in-process gRPC coverage
  (override headers, forwarded quorum writes, blob writes, 400 on bad mode).
- **Rolling-upgrade order (mixed-version caveat):** a pre-m11p3 leader
  neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack` — it
  silently applies LEADER-ack semantics to a `quorum` request (a durability
  downgrade the caller cannot see). Upgrade the leader first: `ack=quorum`
  is then honored immediately (commit-index freshness rides the m11p2
  ship-ack floor hints until the followers upgrade too). Replication and
  heal are unaffected by either order. Runbook §8 records the procedure.

**One replicated log (m11p2) — items/embeddings ride the WAL, `StreamSegments` catch-up, HTTP broadcast deleted**
- **The leader's WAL is now THE replicated log.** The group-commit writer hands
  every fsynced batch to a bounded in-memory **ship feed** (`wal::feed::WalShipFeed`);
  the ship queue pushes those already-encoded bytes verbatim — byte-identical on
  the leader's disk, the wire, and the follower's apply path — and stream seqnos
  are WAL seqnos, so they **survive restarts** (the m8p10 relay-reset hazard is
  gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning
  machinery left the server write path entirely (`/signals` stages straight
  through the engine's group commit; a WAL fsync failure now surfaces per-write
  exactly like single-node).
- **Item metadata and embeddings are replicated mutations.** `/items` and
  `/embeddings` journal kind-1/2 **blob records** (WAL header `flags` byte =
  batch kind; one record, one seqno) BEFORE storage, on the same stream as
  signals. Followers apply them kind-aware — WAL-first into their own log, then
  idempotent storage upserts — and recovery replays them. The m8p10 HTTP
  item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast,
  authed side-POSTs — the source of both 2026-06-10 live bugs) is **deleted**;
  those bug classes are now impossible by construction. Cluster `/items` returns
  a plain 201 and `/embeddings` a plain 204 (no broadcast-report body).
- **`StreamSegments` implemented — catch-up is follower-pulled.** The ship feed's
  tail is bounded; a peer that falls behind it is skipped ahead, and the
  follower pulls the hole itself via the (previously declared-unimplemented)
  server-streaming RPC over the leader's durable, BLAKE3-verified segments.
  Pulls trigger on detected gaps, on follower boot (self-driving restart
  catch-up), and on the leader's heal nudge (`POST /cluster/catchup`, internal,
  forwards the operator's own bearer credential). Pulled chunks flow through the
  same inbound apply path as live ships. Ship acks piggyback the follower's
  applied seqno (`ShipSegmentResponse.applied_seqno`), so retries of
  already-applied data prune and heal is `resume_from` + nudge — no redelivery
  scan, no O(items) traffic.
- **Promote carries a stream baseline.** A promoted leader's stream starts at its
  promote-time flushed frontier (persisted in `data_dir/stream_baseline`); the
  fan-out body and every catch-up chunk announce it, so peers jump their
  frontier past pre-stream history instead of parking on a phantom gap. Ship
  queues are leadership-gated (`activate_from`/`deactivate`): a follower's
  replicated applies never echo back at its peers.
- **Multi-process cluster mode now requires `--data-dir`** (validated at
  startup): the durable WAL is the replication stream. Standalone single-node
  deployments are untouched — blob journaling and the ship feed are gated on
  cluster peers, so single-node item writes keep fjall-only durability with
  zero extra fsyncs.
- New tier-3 suite `mp_items_ride_the_log_and_catchup_stream`: items written on
  the leader AND through a follower gateway converge everywhere via the log
  (feed parity 1e-6), and a follower stopped through item+embedding+signal
  writes restarts and converges via its boot-time `StreamSegments` pull with no
  heal verb and no HTTP item traffic.

**Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first `tidaldb_cluster_*` metrics**
- The replicated `/signals` write path no longer serializes every writer onto a
  solo group-commit fsync nor ships to followers on the request path. Writes are
  **staged** (seqno + WAL submission + relay log push, microseconds, atomic with
  rollback) and **completed** (shared group-commit fsync + in-memory fold) in two
  phases, so concurrent writers coalesce into one fsync; follower shipping moved
  to per-peer sender threads that coalesce contiguous runs into multi-event
  batches with a windowed in-flight budget (topology knobs
  `replication.{batch_max_events,window,retry_ms}`). The 204 contract is
  unchanged (leader durability only) — it now returns at leader fsync. Measured
  on a real 3-process localhost cluster (release build, thepeach mix):
  **4,534 replicated signal-writes/s sustained within SLO** vs ~90/s before
  (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
- **Durable-frontier shipping + relay poisoning.** Senders only ship the
  leader's contiguous fsynced prefix (an event a follower holds but the leader
  could lose is silent divergence); a staged write whose fsync fails **poisons**
  the relay — further cluster writes are rejected and the ship frontier freezes
  (the CockroachDB/Postgres fsync-failure posture).
- **Follower group-commit coalescing.** The segment receiver drains its inbound
  backlog (`Transport::try_recv_segment`) and applies it through ONE shared
  group commit (`SignalLedger::apply_replicated_events`), in range-disjoint
  groups so duplicate/subset re-ships cannot double-fold. Without this the
  follower apply ceiling was ~`events-per-segment / fsync-cost` (~1.8k events/s
  measured) and lag grew without bound under m11p1 leader rates.
- **First cluster metrics + cluster `/metrics` listener.** Cluster mode
  previously had no metrics endpoint at all. New per-region topology
  `metrics_addr` wires the engine's Prometheus listener; new `tidaldb_cluster_*`
  series: ship RTT + batch-size histograms, per-peer ship counters/gauges
  (`peer_shard` labels), WAL fsync latency + group-commit fill histograms,
  write-pool depth/rejections, relay committed + durable frontiers. WAL
  group-commit knobs are deployment config (`wal.{batch_size,batch_timeout_ms}`
  topology block; `wal_batch_size`/`wal_batch_timeout` builder methods).
- Engine API: `TidalDb::signal_staged`/`StagedSignal::wait`,
  `SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}`,
  `ShipQueue` (per-peer windowed batch senders with pause/resume),
  `WalWriter::append_signal_staged`, `WalSender::append_record_staged`,
  `range_payload`/`encode_run`. Relay log entries are now `RelayEvent` (raw
  event records, re-encoded deterministically at ship time) instead of
  pre-encoded single-event bytes.
- Ship-sender failure logging is transition-based (first + every 50th
  consecutive failure WARN with the running count, recovery INFO) — the
  per-retry WARN flood could fill an undrained log pipe and stall the process.
  The multiproc test harness now discards child logs via `/dev/null` (a piped
  fd nobody drains deadlocks the node once the kernel buffer fills), with
  `TIDAL_TEST_NODE_LOGS=inherit` to stream them while debugging.

**M9 — Community Sync & Revocation**
- Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata`. Community signal reconciliation via `CrdtSignalState` with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

**M10 — Governance & Agent Rights**
- Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

**Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE**
- Multi-process cluster mode: `tidal-server cluster --region <name> [--data-dir <p>]`
  (env `TIDAL_REGION`) runs **one process per region** (`RegionClusterState`), each
  owning one `TidalDb` and one `GrpcTransport` that binds this region's `grpc_addr`
  and dials every sibling's real `grpc_addr` — real process/host isolation. The
  topology requires per-region `grpc_addr` AND `http_addr` in this mode;
  single-process mode (no `--region`) is unchanged as the dev/demo default. Both
  modes stay behind the experimental gate (`--experimental-cluster` /
  `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) with mode-specific WARN text.
- New routes / behaviors on the multi-process surface: `GET /cluster/status/local`
  (per-node status), `GET /cluster/status` aggregates ALL regions with a `reachable`
  field (unreachable ⇒ `reachable:false` + worst-case lag), `POST /cluster/reconcile
  {region}` (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact
  no-op on scores), `POST /hardnegs {user_id,item_id}` (user-scoped hide; converges
  via reconcile, filtered from `/feed?user_id`). Writes on a non-leader **forward to
  the leader**; `?region=` reads forward to the owning region; default reads are
  **LOCAL** in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast
  to peers with a per-peer report: `POST /items` → 201 `{replicated_to, failed}`,
  `POST /embeddings` → **200** with the same report on the leader path (NOT 204 — a
  204 cannot carry the report; the forwarded/internal path stays 204). `/sharded/*`
  fans out across processes (degraded semantics preserved). `promote` fans out to all
  peers (`{ok, leader, acked, failed}`).
- `/cluster/heal` is the single recovery verb: redelivers missed signal segments
  (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region
  (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold
  5, reset 30s) is open, so re-issue `/cluster/heal` until `/cluster/status` shows
  lag 0.
- `TIDAL_HLC_SKEW_MS` (multi-process): signed ms offset applied to the process's HLC
  (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
- Tier-3 UAT over real OS processes (feature `cluster-e2e`): `cluster_multiproc` (5),
  `cluster_chaos` (3, REAL network-partition injection via a root-free in-harness TCP
  relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl
  remain an operator option), `cluster_lifecycle` (2, ±500ms clock skew + rolling
  upgrade with zero acknowledged-write loss), `cluster_runbook` (9, every
  `docs/runbooks/cluster.md` §5–§11 operation), plus `cluster_e2e` (2, single-process
  smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover
  ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

### Fixed

**Four production bugs surfaced and fixed by the m8p10 tier-3 UAT**
- **Silent data loss on out-of-order ships.** The replication applied high-water-mark
  swallowed sequence gaps when eager ships arrived out of order. The applied frontier
  is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
- **Non-idempotent reconcile (score creep).** `take_crdt_snapshot` attributed replicated
  signal streams per-node, so reconciling already-converged nodes crept the decayed
  scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical
  replication shard (`ShardId::SINGLE`), making `merge` idempotent — reconcile of
  converged nodes is an exact fixpoint.
- **Lag gauge conflated leader streams across a promotion.** A converged node reported a
  permanent phantom lag after a `/cluster/promote` moved leadership to a different shard.
  The lag gauge now tracks the leader high-water-mark per source shard
  (`ReplicationLagGauge::leader_seqno_for`), computing lag against the current leader.
- **Items missed during a broadcast were never backfilled.** A node down/partitioned
  during an item/embedding broadcast was permanently missing that data even at signal
  lag 0. `/cluster/heal` now re-broadcasts item metadata + embeddings to the healed
  region (idempotent upserts), making heal the single recovery verb.

### Changed

**Cluster mode (m8p8): real gRPC replication**
- `tidal-server`'s `ClusterState` now wires each follower region to a real
  `tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
  crossbeam channels — replication between regions traverses real gRPC/TCP
  (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
  per region) and self-heal a transient bind race by retrying on a fresh port.
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
  offloaded to a dedicated thread so they never block the axum reactor.
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
  blocks on its own runtime).
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
  single-process cluster mode is honest that it replicates over real gRPC but
  runs all regions in one process (no host/process isolation). (True
  multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry
  above.)

**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
  runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
  `rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
  runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
  bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
  image serves a functional 3-region cluster (write replicates to both followers
  over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
  HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
  + promote over real OS processes, feature-gated).

## [0.1.0] - 2026-02-23

### Added

**Core Database Engine**
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration

**Signal System**
- Typed signal recording with exponential decay scoring
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
- Signal velocity tracking
- WAL-backed signal durability with crash recovery
- Periodic signal checkpointing to fjall (every 30s)
- WAL compaction after each checkpoint

**Retrieval (RETRIEVE query)**
- 5-stage pipeline: universe, filter, score, diversify, return
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
- Custom ranking profiles via `SchemaBuilder`
- Diversity enforcement (max N per category/creator)
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more

**Search (SEARCH query)**
- BM25 full-text search via Tantivy
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
- Creator search with `entity_kind(EntityKind::Creator)`
- `similar_to(EntityId)` for content-based recommendations
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
- Autocomplete suggestions via `db.suggest()`

**Entity Model**
- Three built-in entity types: `Item`, `User`, `Creator`
- Metadata storage as `HashMap<String, String>`
- Embedding slots (up to 4 per entity type) via USearch
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`

**Sessions**
- Session lifecycle: `open_session`, `close_session`
- Cross-session preference vector updates (EMA blend)
- Session snapshots with signal state and preference vectors
- Session serialization format v0x03 with backward compatibility

**Social Graph**
- Creator follower/following indexes
- Cohort membership (user segments)
- CoEngagementIndex for co-viewing patterns with LRU eviction
- Social graph filter for "followed creator" content scoping

**Collections**
- Named collections with `Private`, `Shared`, `Public` visibility
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
- `FilterExpr::InCollection` for collection-scoped retrieval
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`

**Observability**
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
- `TidalDb::diagnostics()` -- structured health snapshot
- WAL diagnostics and recovery tools

**Safety**
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
- Export request limit: 500K signals max per request
- `FilterExpr` complexity limit: 256 nodes max
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
- Schema fingerprint persistence detects decay parameter changes on reopen
- Bounded `closed_sessions` cache (10K max, LRU eviction)
- Metrics server non-loopback bind warning

**CLI (`tidalctl`)**
- `tidalctl` binary for database inspection and diagnostics

**RLHF / ML Export**
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics

### Stability

tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.

---

*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*