tidaldb/CHANGELOG.md

Changelog

All notable changes to tidalDB will be documented in this file.

[Unreleased]

Stability

tidalDB is production-ready. M0–M12 are shipped and the pre-release "not yet recommended for production" disclaimer has been withdrawn from the documentation set. The API surface and on-disk data format are stable for shipped features: additive changes ship in minor releases, and any breaking change to a public API or a persisted format ships with a documented migration path in this file. This supersedes the 0.1.0 "no stability guarantees" note below.

Breaking

Sort::New orders by real created_at, not by entity ID (wire-visible)

Sort::New used entity_id as f64 as its score. That was wrong twice over. It assumed IDs are assigned in creation order, which caller-supplied u64 IDs do not guarantee; and it used the ID's magnitude as the base score, so on a catalog of N items the sort term contributed ~N against a boost sum in the single digits. Recency did not merely participate in the ranking, it annihilated every boost, and normalize() runs on the final sum so it could not rescue the ratio. The score is now negated age in hours (newer = closer to 0 = higher), which preserves "newest first" on a scale comparable to a boost sum.

Ordering changes wherever entity IDs were not monotonic with creation time. Three built-in profiles set Sort::New — new, recent_uploads and the brief extra — and any custom profile using it is affected. Items with no created_at available score 0.0 (treated as brand new) and tie, rather than ordering by an ID that carries no recency meaning.

Under ReducedCandidates load the candidate cap now selects survivors via the created_at index read newest-first, not by descending entity ID. The old key was only correct while Sort::New itself ranked by ID; left alone it would have discarded the genuinely newest items before scoring, leaving the ranking wrong only when degraded. RangeIndex::top_n_descending is new and public for this.

Fixed

Sort::Hot is age-aware; it was ranking on view count alone

score_hot hardcoded age_hours = 24.0 for every candidate, so the divisor in log10(max(views,1)) / (age_hours + 2)^gravity was constant across the candidate set and Sort::Hot reduced exactly to log10(max(views, 1)) — a view-count ranking wearing a recency sort's name. Anyone tuning gravity was tuning a no-op. All four built-in Hot profiles were affected: hot, for_you, following, and the brief extra.

The in-code comment justified this by saying a per-entity created_at lookup would need an EntityId -> created_at_ns reverse map that "is not built". That was stale: created_at has been materialized into item metadata on every write since Items::metadata_with_created_at, the executor has held an EntityId -> metadata map since M6p3, and the WAL record ships the materialized map so replicas agree. No new index, storage change, schema change or migration was required — the scorer reads the map it already had, the same way read_duration does.

Measured on a real server, 10 items with equal view counts and ages spanning 2–20 days: before, every item scored 0.5 (an all-equal set folded to the normalizer's neutral midpoint) and the feed returned oldest-first forever; after, scores run 1.0 → 0.0 strictly descending, newest first.

LIMIT, deliberately not papered over: hot's numerator is log10(max(views, 1)), which is exactly 0.0 for 0 or 1 views, so the age divisor cannot differentiate a zero-signal corpus — every candidate still scores 0.0 and ties. Age-awareness takes effect from the second view onward. Making a cold-start corpus rank newest-first requires recency to be additive rather than a pure divisor, which changes ordering for every existing Hot consumer, so it is a separate decision and is not folded in here. sort_hot_zero_view_corpus_still_ ties_regardless_of_age pins the current behaviour so the limit cannot be rediscovered by accident.

Every metadata-based sort was dead on the SEARCH path

alphabetical_asc, alphabetical_desc, shortest and longest silently tied on GET /search for any query without a session context. Two independent causes, both in query::search::executor::pipeline: the metadata pre-load was gated on session_context.is_some() alone and never consulted profile.sort, and the ProfileExecutor it built never had with_item_metadata called at all — the map it did compute was passed only as the keyword-hint argument, which the sort scorers do not read. So shortest/longest scored NEG_INFINITY and the alphabetical sorts scored the missing-title sentinel, for every candidate.

Both query executors now derive the decision from Sort::needs_item_metadata, an exhaustive match on the enum itself. The knowledge lived in a matches! inside one executor while a second executor had its own, different copy — which is precisely how a metadata-reading sort came to be omitted from both. A new variant now has one place to declare this and the compiler forces the author to visit it.

Sort::Hot/Sort::New were missing from needs_metadata_for_sort

On the RETRIEVE path the metadata point-read is skipped unless the profile needs it. Both age-derived sorts were absent from that list, so a profile with no session and no diversity block never loaded the map the fix above depends on. Measured cost of including them, 2,000 candidates (the realistic ceiling — scan_candidates caps at max(limit * 10, 200) and limit > 500 is rejected): 7.25 ms, 3.6 µs per candidate, guarded by a test that fails above 250 ms.

POST /sharded/{items,embeddings,signals} hash-partitions by entity_id and applies the write to the owning region's local store with no WAL append. It does not ride the leader relay, so the data has redundancy 1 regardless of the replication factor — an RF3 cluster configured ack: quorum does not replicate it. That is intentional (parallel write throughput across shard owners: 3,669 signals/s measured, vs ~90/s on the replicated path), but the endpoint answered 201/204 with nothing at the call site, in the response, or in the OpenAPI saying so. An operator probed with /sharded/embeddings, found each embedding on exactly one of three nodes, and filed a durability incident that had to be retracted.

The three write routes now return 400 unless the request carries x-tidal-ack: local, with a body naming the header and pointing at the replicating alternative. The existing x-tidal-ack header is reused — no second durability knob — and local remains rejected on /items / /embeddings / /signals, which always replicate. GET /sharded/{feed,search} are reads and are not affected. Nothing was rerouted: a /sharded/* write that opts in is byte-for-byte as before, and the surface never silently replicates.

Migration: add -H 'x-tidal-ack: local' to acknowledge single-copy durability, or move the write to POST /items / /embeddings / /signals for a replicated one. Live traffic showed 0 requests to /sharded/* over ~2h of production load, and the two in-repo callers were migrated with this change.

Changed

/cluster/status regions[] frontiers may now be null (wire-visible)

applied_events and lag_events are now Option<u64> — JSON null when the aggregating node has no frontier report for a region. Previously an unreachable peer was reported as applied_events: 0, lag_events: <leader high-water-mark>; because lag is hwm − applied, a 500 ms probe timeout was rendered as the leader's entire history as a deficit. Measured live: every node reported both peers partitioned: true, reachable: false, lag_events: 13322258 while the same response's shards[] array showed all three replicas converged at identical frontiers with lag 0. An operator reading the obvious field concludes the cluster is dead.

The root cause was a type that could not express "unknown", so the fix is at the type: null where the value is not known, and no value is derived from one that is not. That covers three former fabrications — the unreachable peer, a leader with no ack mark for a peer, and a region whose lag was computed against a leader high-water-mark the aggregator never learned. reachable and partitioned are unchanged and still distinguish a genuinely partitioned peer from an unknown one; shards[] is unchanged (it was the trustworthy half throughout).

Migration: clients typed against applied_events / lag_events must accept null. tidalctl renders it as NO REPORT / ? and no longer treats it as a deficit, so tidalctl cluster-status && deploy is usable as a gate again — a known deficit, an unreachable region, a partitioned: true region, a pending reseed, or a local shard lag still exit 2.

Fixed

Embeddings are validated BEFORE the WAL append; malformed vectors now 400 (wire-visible) — fleet-hardening task 17

POST /embeddings appended the blob to the leader's WAL first and validated it second. A probe posted a 128-dimension vector to a slot the live schema declares at 1536: the client got a 500, but the record was already durable, so it shipped to both followers, neither could apply it, and both halted their receivers. Shard 1 froze (leader 13540698, followers pinned at 13540694/13540693, lag growing) and writes to the group returned 503 — a quorum-write outage from one malformed HTTP request. Neither a restart (boot self-heal re-pulls the same record) nor POST /cluster/reseed escaped it: the snapshot is captured at the leader's applied frontier, which is itself behind the poison, so the reseeded follower replayed straight back into it. Only forcing a leader election recovered the group.

Fixed on both sides of the durability boundary:

• Write path. The dimension check now runs before wal_blob_first, against the schema-declared slot width and this node's registered width. A mismatched, empty, non-finite, or zero-norm vector is rejected 400 (TidalError::invalid_input) and appends nothing — it cannot ship, so no follower can be poisoned by it. The comparison is storage::vector::validate_dimensions, which the apply path's normalize_and_store also calls: one rule, one definition, because it is the apply-path copy that halts replication when the two disagree.
• Receive path. A replicated record that is unapplicable on every replica — its verdict reads only the record's own bytes and the shared schema — is now skipped rather than halted on: dropped before it enters this node's own log, counted once per record on tidaldb_cluster_blobs_apply_failed_total, logged at ERROR with the seqno range, and the round's remaining records still apply so the frontier advances past it. Failures that might apply later still halt, deliberately: a node-local disk/lock fault, an unknown batch kind a newer binary understands, or a malformed term/membership record whose loss would diverge the roster. Skipping those would trade an availability bug for silent data loss.

ReplicatedBlobApplier::apply_blobs now returns BlobApplyOutcome { rejected } instead of () (in-crate trait; no HTTP or on-disk surface changes).

Migration: a client that treated a 500 from /embeddings as retryable must treat the 400 as terminal and fix the vector width. Runbook §5 documented the 500 as "a known wart, tracked post-M8" — it was not cosmetic, and §5 plus the new §16.6 (halted receiver: recovery by leadership change, and why reseed cannot escape a poison record) now say so.

/feed and /search rank is now dense and ascending (wire-visible)

Under full placement — the production shape, every shard group on every node — missing_groups() is empty and both handlers return straight out of scatter_merge, which sorted and truncated but never re-stamped rank. Each hosted group ranks its own slice 1..k locally, the slices were concatenated and score-sorted, and the per-group counters reached the wire as-is: live /search?query=verification returned ranks 1, 1, 2. The sibling merge_cross_shard already re-stamped for the partial-placement path and documented exactly why; full placement returned before reaching it.

scatter_merge now takes the same set_rank closure and stamps after truncate, so there is one rank-stamping mechanism rather than two. Ordering is unchanged — the stamp is a renumbering, not a re-sort, and scores were always correctly descending. /vector_search carries no rank field and passes a no-op. The S=1 fast path still returns the engine's own already-dense ranks; both shapes are now asserted.

Migration: a client that deduplicated or keyed on rank while working around the duplicates can drop the workaround. rank is 1..n over the returned page.

vector_search distances now honor the documented [0.0, 4.0] contract (wire-visible)

The read path passed the caller's raw query vector straight to the index while the write path L2-normalized every stored vector, so the two sides lived in different spaces. With unit v, d = |q|² − 2q·v + 1: a non-unit query shifted and scaled every distance by |q|². Measured on a live RF3 cluster, /vector_search returned distances of 591–1174 against a contract promising ≤ 4, and an exact match scored |q|² − 1 instead of ~0.

db::query_ops::vector_search_items now normalizes the query with the canonical storage::vector::l2_normalize before searching. A zero-norm query — which has no direction, so "nearest by cosine" is undefined — is rejected with 400 instead of returning an arbitrary ranking.

Ranking is unchanged. |q|² and 1 are constant across candidates, so the ordering was already correct cosine order; that is why this went unnoticed. Absolute uses of the number were the casualties: distance thresholding, dedup-by-distance, cross-query comparison, and recall measurement.

Migration: any client comparing distances against a hard-coded threshold derived from the old unnormalized values must re-derive it. An exact match now scores ~0 (2.4e-7 measured; the f16 quantization floor is ~1e-3, so compare with a tolerance, never equality) and all distances fall in [0.0, 4.0].

Blob replication is now observable (tidaldb_cluster_blobs_*)

The blob path — item metadata, embeddings, term markers, cluster membership — had zero instrumentation: lag_events counts WAL apply, so a blob that never shipped or never applied moved no number anywhere. Added per-kind counters tidaldb_cluster_blobs_originated_total, _applied_total and _apply_failed_total, labelled by record kind with cardinality fixed at 4 by construction. Comparing originated on the writing node against applied on each peer localises a replication gap to enqueue, ship, or apply in one query.

Every co-located shard group now reports its vector count

tidaldb_usearch_vector_count was rendered only for the node's metrics-owner group, so on a 3-group node two thirds of the corpus had no vector-count series and could diverge unobserved. Co-located groups now render tidaldb_usearch_vector_count{shard="N"}; the owner's series stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets each replica set separately without double-counting.

A converged, unquarantined, voting pod no longer sits permanently 503, and /health no longer names a cause it never checked — fleet-hardening task 18

During the 2026-08-31 staged roll, tidaldb-2 logged latched: 2 reseed markers (from_seqno 13322280 and 13540699) and discharged: 1, then answered 503 {"cause":"joiner boot not yet converged"} indefinitely while /cluster/status/local reported reseed_required=false, quarantined=false, membership_role=voter and lag=0 on all three shards — every group having already logged "first-converged against a KNOWN leader frontier ... readiness is now sticky-ready". It happened on all three pods and each needed a manual kubectl delete pod.

Three defects, all in tidal-server/src/cluster/node.rs:

• The latch and the discharge were not the same shape. A shard group held one readiness flag plus one durable marker slot that a re-latch overwrote, so a group that latched at two different frontiers (a term join at one, a later snapshot-required refusal at another) let one completed catch-up pull decide both — clearing readiness with the lower gap still a hole in one latch order, and clearing nothing in the other. Outstanding gaps are now a set keyed by (shard, from_seqno), each discharged only by evidence about itself (the unchanged ReseedMarker::discharged_by_served_range — never a frontier comparison), and the durable slot is re-derived from what remains as the conservative collapse: lowest resume point, least-dischargeable reason. The latch WARN now fires once per distinct gap and the discharge INFO once per gap cleared, so latched and discharged count the same population.
• A refused reseed_self_restart was never re-evaluated. The §2.4 quorum refusal is correct and unchanged — it fired during the roll with alive_voters_excluding_self=1 < majority=2 and is why the cluster stayed up — but its only re-drive was "a future latch re-evaluates", and a node whose gap is real gets no future latch once the catch-up retry stops refusing. It waited forever on a quorum event nobody polled; kubectl delete pod performed by hand exactly the restart the node had refused. A held refusal now re-evaluates every 15s until the gap heals, the refusal clears, or the exit fires.
• /health reported a cause it had not tested. The 503 ladder tested three gates and printed "joiner boot not yet converged" for everything else — including the latched reseed marker, which was the gate that actually held. Every gate now has its own arm; the reseed arm names the outstanding gaps (reseed_marker_latched: shard group 1 holds an unhealed reseed gap. Outstanding node-wide: [s1@13540699(snapshot_required)] ...) and says when a self-restart is refused; and the terminal arm reports unready: reason unavailable instead of inventing a state. A probe that lies about its reason is worse than one that admits it has none.

The readiness gate itself is unchanged in spirit: a node holding stale data it is about to discard still drains from the client VIP (the property that stopped a PVC-wiped tidaldb-0 serving an empty corpus). The bug was that it never stopped draining.

Wire-visible: /cluster/status/local and each shards[] row gain reseed_gaps (the outstanding frontiers, lowest first), and the flat/per-group reseed_required now reports the live gap set rather than the marker file — so it can no longer answer false while /health answers 503 for a latched marker.

Added

Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in for_you

• Online preference clustering (entities/multi_preference.rs). A warm user (≥ COLD_START_N = 5 interactions) maintains up to K_MAX preference clusters built by online sequential k-means with a DP-means threshold split: a new engagement updates its nearest cluster (per-cluster adaptive EMA) or, past the split threshold and under the cap, opens a new cluster; at the cap the nearest cluster absorbs it. Per-cluster importance composes the canonical forward-decay kernel anchored to each engagement's timestamp, so stale interests fade.
• Top-M ANN fan-out. At query time for_you selects the top-M clusters by current importance, issues M ANN queries (candidate_gen::ann_candidates_multi), and merges by best (min) distance; the personalization boost is the max cosine over all clusters. Users below the cold-start threshold keep the single adaptive-LR vector (entities/preference.rs). Design: docs/research/multi-vector-preference.md.

Idle-readiness + TLS scale-up (m12p5–m12p6) — followers converge readiness on an idle cluster, and elasticity is proven over REAL mTLS on k8s

• Idle-readiness convergence (m12p5). The leader heartbeat now carries its live frontier (leader_last_seq), so a caught-up follower flips /health Ready on an idle cluster instead of stalling until the next status poll or ship. Cert SAN wildcard widened for scale-to-5.
• TLS scale-up (m12p6). A real kubectl scale 3→5 exercised seed-join over mTLS on k8s (kind) for the first time: joiners flip Ready in ~13s via the idle-readiness heartbeat, auto-promote to Voter, reach full content parity at lag 0, with zero acked DATA loss across scale-down. Fixes span a six-bug chain — https:// seed scheme, rustls CryptoProvider install order, headless seed Service, cold-handshake poll timeout, two-tier cert-manager PKI, and the grpc_tls_for CA fallback for a not-yet-in-topology joiner. Persists the HNSW graph and skips a suspect graph on reseed-pending close.

Sharded ingestion (m12p4) — scatter-gather across shard groups with cross-shard unified reads

• Scatter-gather pool + cross-shard reads. Writes hash-route across a 3-group shards: topology; reads unify across groups (L4). Ran REAL on kind with a 2-generator load job. Fixed an HTTP/2 204 forward-relay bug (a synthesized JSON body on a 204 relay triggered an h2 RST_STREAM). Confirmed with data: at fixed per-pod CPU, full-placement sharding scales failover, not write throughput; the ≥2.5×-and-≥5,000/s scaling target remains Ref-A/k3s-pending.

Index tuning + recall/memory at the production shape (m12p3) — the G2 work: per-query ef_search is now honored, the brute-force crossover scales with dimensionality, and the HNSW recall/latency/memory frontier is measured at 1536-D with a real exact oracle

• Per-query ef_search override — now real. UsearchIndex::search / filtered_search honor a per-request ef_search instead of silently dropping it to the index default (pre-m12p3 behaviour: accepted for trait compliance, logged a warning, ignored — USearch 2.24 has no per-call beam argument). The override is race-free via an RwLock epoch guard (with_expansion): searches that agree on ef_search run in parallel under a shared guard; only a query that changes the live beam width takes the exclusive guard for its (set, search) window — not a per-search mutex. The knob was already plumbed end-to-end in m12p1 (vector_search_items(.., ef_search), the /vector_search ef_search field, tidal-stress --recall-ef-search); m12p3 makes it move recall. ef_search=0 selects the slot default.
• Dimension-aware brute-force → HNSW crossover. The exact BruteForceIndex scans every vector under a read lock at count × dim cost, so a fixed 10,000 crossover meant a 15.4M-FMA scan at 1536-D (tens of ms, blocking writers). usearch_min_vectors(dim) now keeps a brute-force scan within ~4M FMAs: ≈10,000 at/under 128-D (byte-compatible with pre-m12p3), ≈2,600 at 1536-D — flipping high-dim mid-size slots to HNSW before the scan blows the SLA.
• memory_usage() exposed on UsearchIndex (the true graph + vector footprint from USearch, not the index_stats lower bound) for pod sizing.
• Grid-search harness. cargo run --release --example ann_grid_search builds a UsearchIndex + an exact BruteForceIndex oracle over the same deterministic id-keyed corpus and reports, per (M, ef_construction, ef_search, quantization) point, measured recall@10 vs the oracle, mean/p99 search latency, build time, and the true footprint — the tool that produces the documented M/ef and the F32/F16/Int8 recall+memory numbers.
• Measured at 1536-D (100k clustered corpus, real exact oracle). The production default (M=16, ef_c=400, F16) clears G1 and G2: recall@10 0.997 at p99 ≈ 1.4 ms raw ANN; ef_search is the latency lever (recall saturates by ef_s=128 → p99 ≈ 1.0 ms). F16 costs only 0.25% recall vs F32 for half the RAM (≈ 5.2 GB/1M true footprint incl. graph); Int8 rejected at 1536-D (recall 0.715, −28%). Live tidal-stress --verify-recall against a real server gave recall@10 = 1.0000 at 20k/1536-D, default beam and --recall-ef-search 400. Finding: the recall corpus is now clustered (Gaussian mixture) in both the grid harness and tidal-stress (recall::embedding_for) — uniform-random high-dim vectors are pathological for recall@k (≈0.97 at 10k → ≈0.54 at 100k, a measurement artifact, not an index regression). Full frontier + the 1M command in docs/profiling/usearch-tuning.md and docs/profiling/scale-baselines.md. docs/specs/07-vector-retrieval.md updated: per-query ef_search is IMPLEMENTED, not deferred.

ANN candidate generation in RETRIEVE (m12p2) — the G1 unblock: for_you/related source candidates by nearest-neighbour, trending by a cached per-signal-type top-K, so the feed stays relevant AND bounded as the corpus grows past the scan cap

• ANN in retrieve. CandidateStrategy::Ann is now wired into the RETRIEVE executor (it previously fell back to a scan with a warning). The db layer resolves the query vector — the user's preference vector for for_you, the seed item's embedding (similar_to) for related — and Stage 1 runs an O(ef_search) HNSW search over the item content slot instead of scanning an arbitrary low-id slice of the universe. for_you and related are now Ann profiles. Graceful: no registry / no preference vector / no seed ⇒ it degrades to a scan (anonymous reads and cold-start users still serve), so every embedding-less schema and pre-m12p2 caller is unchanged.
• Cached SignalRanked. The O(N) ledger scan behind SignalRanked is now a cached per-signal-type top-K (signals/ledger/hot_top_k.rs): O(K) on the served path, with a bounded O(N) rebuild only when stale. Decay preserves relative order (same λ), so a cache is valid until the next write; small ledgers rebuild on any write (always fresh), large ledgers throttle the rebuild off the read hot path (1s). trending now uses SignalRanked(view), so it ranks the actually-viewed corpus at any id — not the low-id scan slice.
• related over HTTP. GET /feed?profile=related&similar_to=<id> resolves the seed's embedding and runs ANN — "more like this" on the read surface (similar_to added to FeedQuery, threaded through all three feed handlers).
• Harness /feed measurement. tidal-stress gains --feed-profile <name> (force every feed read to one profile, for per-profile retrieve p99) and --seed-preferences (build a preference vector per user so for_you exercises ANN, not the scan fallback).
• Verified real against a 1536-dim standalone server: trending retrieve p99 = 3.5–7.7ms (under the 10ms G1 target) under concurrent writes, via the cached top-K; for_you retrieve is ANN-backed (preference vectors built) at p99 ≈ 24ms, dominated by the Stage-3 preference-boost recompute (per-candidate embedding read) — flagged for m12p3's index/score tuning (the risk register's "materialized-score layer"). ANN candidate recall is the m12p1 /vector_search probe (0.9997). New engine tests prove ANN/related/trending reach the relevant items at high ids a scan can never reach, plus cache freshness.

Read-recall harness — measurement truth for ranking-at-scale (m12p1): recall@k + true p99 at the production shape, so G1/G2 steer on real numbers instead of absent ones

• Pure k-NN probe. TidalDb::vector_search_items(query, k, ef_search) returns the raw HNSW nearest neighbours over the item content embedding slot — NO profile scoring, fusion, or diversity — so the result is the ANN index quality in isolation (the G2 metric). Exposed as POST /vector_search on the standalone router and the multi-process region node (merge-by-distance across hosted shard groups); a dimension-mismatched query is a 400, not a 500.
• The oracle. tidal-stress --verify-recall seeds the corpus with deterministic, id-keyed embeddings (reproducible with --skip-seed), holds a brute-force cosine ground truth in RAM, and ramps /vector_search probes open-loop (coordinated-omission corrected). It reports per-stage true p99 (a genuine tail, not a closed-loop mean) AND mean recall@k, plus the read-knee — the highest sustained QPS where p99 ≤ target AND recall@k ≥ target both hold — with a machine-readable JSON summary and --fail-on-knee PASS/FAIL exit. Knobs: --recall-k, --recall-queries, --read-p99-target-ms, --recall-target, --recall-ef-search.
• Verified end-to-end against a real standalone server at 1536-dim: recall@10 = 0.9997 at 20k items (HNSW M=16/ef=400/F16 vs brute-force cosine, far above the 0.95 G2 target); the harness also exercises the read-knee verdict (both branches) and gate exit codes. The 100k/1M exit-gate runs use the same harness on the k3s cluster (the brute-force oracle needs ~6 GB RAM at 1M).
• No more mean-as-p99. Repaired the fabricated p99 column in docs/profiling/social-scale.md (the social bench is Criterion = mean only) and the scale.rs / scale-baselines framing; every closed-loop number is now labelled an isolated per-op mean (regression tripwire), with the p99/recall tail SLOs signed off only by the open-loop harness. Added a 1536-dim HNSW-vs-brute recall@10 bench (benches/vector.rs) and a recall-harness section to docs/profiling/scale-baselines.md.

Sharding × replication + rebalancing (m11p6) — the "replicated XOR sharded" split is over: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes

• One write surface. /items///embeddings///signals now hash-route the entity to the owning shard group's leader (the engine's FNV-1a ShardRouter) AND replicate at RF — sharding and replication at once. The old /sharded/*-vs-leader split in tidal-stress (WritePath::Leader|Sharded) is gone; it drives the one unified path, spreading writes round-robin across gateways (or pinning one with --leader-url). x-tidal-ack/x-tidal-seq, quorum await, and NotLeader/QuorumTimeout are per-group; NotLeader names the group.
• Rebalancing verbs (L3). POST /cluster/shards/{id}/transfer moves one group's leadership (the m11p4 fenced transfer scoped to the group); POST /cluster/shards/{id}/replicas adds/removes a replica (the m11p5 join / fenced-removal per group). A ?shard= selector threads through every per-shard admin verb (promote/heal/partition/catchup/reseed/members/join) and is PROPAGATED on every intra-group forward/broadcast (ShardReplica::admin_path) so the receiving sibling targets the same group. S=1 is byte-for-byte (no selector emitted, no shard in the NotLeader body).
• Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over real OS processes — SIGKILL a node under ack=quorum load → ONLY its shard-leaderships re-elect (survivor groups keep theirs), reads never stop, and every acked write is present on its shard's new leader (zero acked loss), across random kill points. Plus a rebalance-verb test (transfer + ?shard= promote move exactly one group). Harness: MultiProcCluster::start_sharded (per-(node,shard) ports, shards: emission, agreed_shard_leaders).
• Throughput. Local 3×3 release cluster sustains 3,000 quorum signal-writes/s at 0% error with per-node CPU ≈30% and replication lag ~0 — generator-bound, not engine-bound. The ≥5,000/s + ≥2.5×-single-shard scaling is Ref-A (k3s Linux/fdatasync/multi-generator) — the standing access caveat since p1.
• Known follow-up (tracked): per-group-aware node readiness (today is_ready is node-global across co-hosted groups) and cross-node read fan-out under PARTIAL placement; the exit gate runs full placement, which these do not touch.

Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix

• Fault injection compiled out of production. A non-default fault-injection cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks (tidal/src/fault.rs): slow-fsync (TIDAL_FAULT_FSYNC_DELAY_MS sleeps before each durable fsync) and disk-full (TIDAL_FAULT_DISK_FULL_AFTER_BYTES returns a real ENOSPC/errno-28 once cumulative segment bytes cross the threshold). The production image build never passes the feature, so the hooks are compiled out entirely — a disk/fsync fault a stray env var could trip in prod is a 3am footgun we refuse to ship; the safety is structural. Inert until armed even when compiled in. The tier-3 harness builds the spawned binary with the feature.
• First-class invariant checkers (tidal-server/tests/support/invariants.rs): the no-acked-loss AckLedger (frontier + content, extracted from the m11p3 ledger gate, which now consumes it), cross-replica assert_feed_parity / feed_item_ids, assert_single_leader_per_term (membership safety, reads the m11p6 shards[] rows), and MonotonicCounters (per-node applied frontier + leader commit index never regress, with legitimate epoch resets forgiven).
• New fault-class suite (tidal-server/tests/cluster_faults.rs, tier-3): a disk-full follower degrades gracefully (receiver halts, node alive, zero acked loss, restart recovers to parity); a slow-fsync follower lags then converges while the fast follower supplies quorum; both followers slow → ack=quorum returns a retryable 503 naming the laggards while ack=leader is unaffected; an asymmetric partition (inbound severed, outbound up) causes no split brain (pre-vote + check-quorum) and no loss. 4/4 green.
• Soak + regression gates in tidal-stress: --json-summary <path> (a machine-readable per-stage p99/throughput/error roll-up for trend lines) and --max-p99-ms / --max-error-pct / --fail-on-knee gates that exit non-zero on a regression (the tool always exited 0 before).
• Nightly pipeline (.woodpecker.yaml): a cron nightly flow (chaos suites with elevated kill-points + the fault-injection feature, then the gated tidal-stress soak) beside the existing push release gate, event-routed by per-step when. Woodpecker, never GitHub Actions.
• Guarantee traceability (docs/planning/milestone-11/guarantee-traceability.md): every roadmap §2 guarantee mapped to its named automated test(s). Closes the G-C apparatus; the 30-consecutive-days-green half of the GA bar is a calendar criterion the nightly pipeline accrues.

Observability + operations (m11p8) — complete cluster metric set on a per-node /metrics listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate

• Metrics. Completed the tidaldb_cluster_* set with the two members the roadmap named that were missing — breaker (tidaldb_cluster_peer_breaker_state 0/1/2 + tidaldb_cluster_breaker_opens_total, surfaced read-only from the tidal-net circuit breaker via a new Transport::peer_breaker_state) and forwards (tidaldb_cluster_forwards_total / _forward_failures_total, instrumented at the gateway forward path) — plus the self-heal series (heal_attempts/successes/noops_total, healing_peers). When several shard groups co-locate on one node (m11p6) the metrics-owner serves the single /metrics listener and the siblings register their series under a shard="N" label (TidalDb::register_metrics_sibling); a single-shard node is byte-identical to before. A 12-panel "Cluster Replication" Grafana row + an 8-rule tidaldb-cluster Prometheus alert group ship beside the standalone ones.
• Request-id + tracing. Both cluster routers (single- and multi-process) now carry the standalone router's SetRequestId + PropagateRequestId + TraceLayer stack (extracted to router::with_request_id_tracing); the id rides the follower→leader forward hop verbatim, so the leader's span shares the gateway's x-request-id. (Ships are off-request-path and batched, so they correlate by seqno, not a request-id — by design.)
• Truthful status. Fixed the leader's own applied_events reading 0 (a leader writes its WAL directly and never advances its own applied frontier — now reports its flushed frontier) and the single-process post-promote ShardId(0) lag-keying undercount (now keys on the current leader's shard).
• Self-driving heal. A standing leader duty re-arms the backlog re-ship for a stuck (breaker-open, behind) peer every ~3s, so it converges through breaker resets with no operator /cluster/heal loop — closing the §1.4-3 footgun. Observable via healing_peers + the TidalDBClusterHealNotConverging alert.
• WAL PITR archival. wal.archive_dir (topology + builder): the online compaction copies each sealed segment to the archive — durably, before deletion, refusing to delete if archival fails — so the archive is a gap-free PITR record.
• Backup/restore. tidalctl backup / restore: offline data-dir backup with a BLAKE3 BACKUP_MANIFEST.json (+ the WAL checkpoint cursor); restore verifies every file's hash before writing and refuses a non-empty target. Coordinated cluster backup = back up one committed replica per shard group (the runbook drill).
• Rolling upgrade. A wire version handshake (HeartbeatRequest.build_version, stamped at the transport boundary; >= 2-major skew WARNs, never rejects) + version on /cluster/status. mp_rolling_upgrade_no_loss_no_stall promoted to the FIRST step of .woodpecker.yaml (the release gate; a failure blocks the image build). See milestone-11/phase-8.md.

Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit

• The cluster stops trusting the network. gRPC replication mTLS is now the intended posture: the inbound server is served over a custom tokio-rustls acceptor (not tonic's fixed .tls_config()) fed a DynamicCertResolver (ArcSwap<CertifiedKey>), preserving mutual TLS exactly (a WebPkiClientVerifier over the cluster CA — a foreign/absent client cert fails the handshake before any RPC). Plaintext is an explicit insecure: true with a loud startup WARN.
• Cert + bearer rotation WITHOUT restart. A content-hash poller re-reads the cert files (k8s ..data symlink swaps that inotify misses) and atomically swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys, so a rotation drops zero requests (verified under concurrent load). Outbound peer channels rebuild from the refreshed files. The bearer and a new shared cluster key live behind ArcSwap, read per request and reloaded from TIDAL_API_KEY_FILE / TIDAL_CLUSTER_KEY_FILE.
• Authenticated inter-node HTTP with per-node identity. The axum listener serves TLS (reusing the same hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/scatter/status/seed-join dial https:// with the cluster CA. A forwarding node mints an x-tidal-node-token (keyed-BLAKE3 MAC over node-id + expiry under the cluster key — no new crypto dependency) so a foreign pod cannot forge a sibling identity. The x-tidal-internal marker is now honored ONLY from a verified sibling (marker without a valid node token → 403): the marker stays a routing hint, never an authorization bypass. All opt-in via grpc_tls / the cluster key — absent ⇒ pre-m11p7 behavior, so every existing deployment and test is byte-for-byte unchanged.
• Admin-verb audit log. promote / partition / heal / join / member-remove / reseed each emit one structured record (principal, term, target, outcome) to a tidal_audit tracing target + an optional append-only JSONL file (TIDAL_AUDIT_LOG), on the operator-originated leg only (no double-audit on the forwarded re-apply).
• Per-principal HTTP rate limit. The engine's token-bucket RateLimiter (now re-exported from the crate root) gates all three routers keyed by principal; verified sibling nodes are exempt (replication is never throttled); a deny is 429 + Retry-After. Off by default (TIDAL_RATE_LIMIT_RPS).
• Reference deployment + tooling. k8s/cluster/ gains cert-manager Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region grpc_tls topology block; scripts/gen-cluster-certs.sh (openssl) provisions the same Secret shape without cert-manager. Exit gate verified: foreign pod rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext inter-node links. See docs/planning/milestone-11/phase-7.md.

Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference

• Nodes are cattle; topology is data, not files. grpc_addr is now an advertised address — a hostname or an IP — split from a new optional per-region grpc_bind (the local SocketAddr to bind). A literal grpc_addr binds itself byte-for-byte as before (every existing topology keeps working); a hostname binds 0.0.0.0:<port>. tidal-net's peer map retypes from SocketAddr to String, so Channel::from_shared makes hyper re-resolve DNS on every reconnect — the pod-rescheduled-onto-a-new-IP case the SocketAddr::parse-only constraint made structurally impossible is now fixed by construction. One shared topology file can name every region by its per-pod DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code change). Proven by cluster_membership.rs::mp_dns_hostname_topology_replicates (a hostname topology a pre-p5 binary would have refused to parse boots, replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
• A new node joins via snapshot + stream and serves quorum in minutes. FetchSnapshot is a new server-streaming RPC in the WalShipping service: the leader stages TidalDb::create_backup under <data_dir>/snapshots/ and streams it term-stamped + term-fenced exactly like StreamSegments (manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to end). The handler answers a no-snapshot-needed header when the WAL can still serve the puller's range and streams the artifact only when it cannot — the needed decision is baseline-aware (from_seqno ≤ stream_baseline OR < earliest WAL seq), because a "the WAL still covers it" answer is a lie at or below a baseline the stream clamp will never serve, and looped marker boots forever. Install is a boot-time operation (never a live data-dir swap): fetch into a sibling staging dir, copy the node's own identity files (election_state, membership) INTO staging, write a COMPLETE sentinel, then rename — every crash window is an idempotent redo. A retention pin taken at backup start keeps the segment chain covering the artifact alive while any joiner streams, with a hard cap + tidaldb_cluster_snapshot_pin_force_drops_total so a dead joiner can't freeze compaction forever.
• Reseed is self-healing — no operator verb, no wipe_data_dir. A running follower that hits a typed snapshot-required refusal (the StreamSegments trailer is now structured — x-tidal-catchup: snapshot-required | rejoin | stepping-down — so an ordinary election term-mismatch never latches reseed markers fleet-wide; the marker latches only on snapshot-required), or the m11p4 divergence quarantine, durably latches reseed_required (ElectionStore file discipline), surfaces it in /cluster/status/local and the tidaldb_cluster_reseed_required gauge, and keeps serving degraded with voting enabled (a reseed needs a leader, and the leader may need this node's vote — reseed-blocks-voting would deadlock the cluster). The reseed runs on the next boot; replication.reseed_self_restart: true (default false; k8s sets it true) drains and clean-exits once the marker latches — refused when the remaining voters can't sustain quorum without this node. A successful reseed boot clears the quarantine latch and tidaldb_cluster_divergence_quarantined — closing the m11p4 carried hazard that quarantine clearing "rides m11p5's reseed machinery." cluster_reseed.rs proves the full quarantine → marker → restart → converged → gauges-cleared loop with no wipe_data_dir in the test.
• The three-way term-join rule closes p4's last carried hazard (pre-baseline history was unreachable through a term's stream for followers behind at the transfer). The first reseed drill exposed that such a follower jumps its frontier past pre-baseline history with lag=0 and never asks for the gap — silently missing data with nothing to fire the refusal. The heartbeat now carries the leader's election-time position in the previous stream's numbering (prev_log), and the join check is three-way: own > prev_log → divergent suffix → quarantine (p4); own < prev_log → genuinely missing committed-era history → latch reseed_required (new); own == prev_log (or within-term rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL is the leader's copy, so tail_term equals the leader's term).
• Membership is data on the one log: kind-4 records, learner → voter. A new MembershipRecord WAL blob kind (kind-4, beside kind-0 signals, kind-1/2 item/embedding blobs, kind-3 term markers) is journaled by the leader, replicated through the normal stream, and folded by followers into a ClusterMembership cell (the WalTermMark pattern). Records carry the FULL roster (latest record wins; no merge logic); membership epoch 0 = the topology file, so a cluster with no kind-4 record behaves byte-for-byte as today. POST /cluster/join {name, grpc_addr, http_addr} on any node forwards to the leader, which assigns id = max(all ids ever) + 1, appends a Learner record, and answers only after it is quorum-committed (idempotent by name); member ids are permanent — removal tombstones (a Removed record) keep ids burned, never renumbered or reused. Auto-promotion is a standing leader duty (re-armed on every activation and apply, evaluated on commit-index publishes, driven solely from the applied ClusterMembership — it survives the joining-era leader's death) that promotes a learner within replication.learner_promote_lag (default 1024) of the flushed frontier, or the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server (one at a time, each same-term-quorum-commit-gated); on every leadership activation the new leader re-appends its full current membership immediately after the kind-3 term marker (the Raft no-op-entry analogue), which closes both the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs: GET /cluster/members, POST /cluster/members/remove, POST /cluster/reseed.
• The even-voter-count majority() arithmetic was a latent bug — fixed. ElectionConfig::majority() was peers.len()/2 + 1: correct at n=3, wrong for every even voter count (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D} could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with zero RPCs). CommitIndex already used the correct div_ceil form — the two formulas in one subsystem disagreed. Since p5 makes even sizes mandatory transit states (3→4→5→4→3), this landed first, with even-n property tests: majority() = (peers.len() + 1).div_ceil(2) + 1 (true floor(n/2)+1 over the full voter set), spelled to read identically to the commit-index site so they can never drift again.
• Learner marks never count toward quorum. A role-blind CommitIndex that saw learner marks would treat two learners "committing" a write no voter holds as acked — acked-write loss. CommitIndex is now role-aware: voter marks feed the k-th-largest selection; learner marks live in a side map (promotion input, transfer-wait input) and are excluded from needed. In-flight ack=quorum waits are re-evaluated against a new config on every conf-change (a shrink may satisfy waiters instantly), never failed; a report from an unknown id logs at WARN with a counter instead of being silently dropped. ElectionState, ShipQueue, and PeerPool reconfigure behind one fenced apply path so the four peer-set copies can never disagree about the roster.
• Seed-join boot. tidal-server cluster --region <name> --seed http://host:port (repeatable) --advertise-grpc host:port --advertise-http host:port --metrics addr: the joiner skips the "every region declared" topology gate, learns its roster + assigned id + current term from any reachable seed, persists them to a durable membership cache + election_state (persist-before-act), and installs a snapshot when behind. A restart boots from the cache without the seed. A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule (so a joiner never silently inherits the wrong ack default or election timing).
• Kubernetes reference: one StatefulSet. k8s/cluster/ (namespace tidaldb-cluster, replicas: 3, podManagementPolicy: Parallel, topology-spread, PDB maxUnavailable: 1) with ONE shared bootstrap topology ConfigMap (per-pod DNS advertised addresses, 0.0.0.0 binds). Scaling past 3 does NOT edit it — pod N≥3 boots --seed + the same mounted file and joins as a learner; rolling node replace is kubectl delete pod (PVC retained → boot catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness: converged (boot catch-up completed at least once AND lag ≤ learner_promote_lag — hysteresis, never lag == 0 which an open-loop load keeps perpetually false); a restarted existing voter is Ready on today's terms.
• Exit-gate suites (tier-3, cluster-e2e): cluster_membership.rs (mp_seed_join_snapshot_catchup, mp_scale_3_5_3_under_load_zero_loss — lost=0, max acked seq 1467, p99 impact <2× across the 3→5→3 joins, mp_dns_hostname_topology_replicates) and cluster_reseed.rs (quarantine → marker → restart → converged, gauges cleared). Localhost-loopback catch-up: 5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget); the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line items (k3s access still pending — the standing M11 caveat).
• Mixed-version / downgrade caveats. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → its receiver's torn-state halt latch, permanent across restarts (both followers halted = quorum-write outage). So HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses JoinCluster and every conf-change until all current voters report kind-4 capability — "complete the binary upgrade before the first conf-change" is structurally enforced, not operator discipline. Pre-p5 peers answer Unimplemented to JoinCluster/FetchSnapshot (the joiner reports it loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrade below p5 requires a reseed.

Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5

• "A machine died" is a non-event. Every multi-process cluster node runs a purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced leadership transfer) over the existing tidal-net transport: leader heartbeats every 300ms (config election.heartbeat_interval_ms); a follower that hears no leader for a randomized 1500–3000ms starts a pre-vote; a majority elects. SIGKILL the leader under ack=quorum load → a survivor is elected and writes resume in well under a second locally, with zero acknowledged-write loss across repeated random kill points (tier-3 gate cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss). No raft crate: the WAL is already the replicated log (m11p2) and the quorum commit index (m11p3) already proves majority durability — the election adds only the tiny consensus state (term, leader) on top of them.
• The WAL carries its own election history. An elected leader's FIRST log entry is a kind-3 term-marker record that replicates like any record, so every replica's (lastLogTerm, lastLogIndex) — Raft's vote restriction — is derived from one fsync stream and can never disagree across a crash. The frontier half is compared in the last joined term's stream numbering (a reseeded node's own WAL numbering diverges from the stream's after a baseline jump; comparing raw local frontiers across nodes would let a behind node win). Hard state (current_term, voted_for) persists in data_dir/election_state (magic + version + checksum; corrupt → refuse to boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE any vote reply or leadership claim leaves the node.
• Term fencing everywhere (kills incident §1.4-1). Every replication RPC carries the sender's term: stale-term ships/chunks/heartbeats/frontier reports are rejected, the quorum commit index folds only reports stamped with its activation term (race-free, under the index's own lock), and a restarted ex-leader boots as a FOLLOWER from its durable state — never from the topology file. A partitioned ex-leader that restarts cannot accept a single write (mp_fenced_ex_leader_restart_cannot_write); a leader that loses majority contact steps down within election.leader_lease_ms (default 900; validated lease + heartbeat < election_timeout_min so a deposed leader stops before any successor can exist).
• Divergent suffixes quarantine instead of lying. A node whose log extends past what the elected leadership subsumed (leader-acked, never-quorum-acked writes on a dead leader) detects it at term-join — the heartbeat carries the leader's election-time log position — and fences itself from the data plane (status quarantined: true, metric tidaldb_cluster_divergence_quarantined, ERROR naming the reseed runbook) while still voting. Followers apply on receipt, so un-applying is not a thing: reseed is the honest recovery (m11p5 snapshots automate it).
• /cluster/promote is now a fenced transfer: with a live leader it drains (waits for the target to hold the flushed prefix, breaker-immune via the commit index's marks) then sanctions an immediate election; with a dead leader the target campaigns. The election refuses a target whose log lags — the m11p3 "promote the max-applied survivor" operator rule is now enforced by the protocol. The legacy term-0 fan-out survives only on clusters that have never elected (mixed-version rollouts; the chaos drills' deliberate isolated-node override) and is permanently retired per node at its first joined election. election.auto_election: false preserves the full pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
• Bounded churn under flapping links: the pre-vote (no term inflation without a majority probe) plus the leader-freshness lease absorb short flaps entirely and bound terms under long ones (mp_flapping_links_bounded_churn). Election observability: tidaldb_cluster_election_term/_role/_elections_started_total/ _leader_changes_total, and /cluster/status/local grows term, role, quarantined, prev_log_term/seq.
• New tier-3 exit-gate suite cluster_election.rs (auto-failover ledger, fencing under partition+restart, bounded churn); cluster_quorum.rs and cluster_lifecycle.rs pin auto_election: false (they validate the manual drill, which remains supported); the partition harness gained bidirectional isolation (isolate_region) — severing only a node's inbound edges leaves its outbound heartbeats/votes flowing, which silently defeats partition scenarios.

Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, TSEG segment header, structured "snapshot required"

• Failed catch-up pulls retry on a timer. The pull trigger was event-only: a follower whose StreamSegments pull failed (e.g. the leader's gRPC server not yet ready during a rolling restart) waited for the next PUSHED segment to re-expose the gap — in an idle cluster that push never comes, and the follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers stuck at lag=136507). A failed pull now arms a one-shot timer (replication.catchup_retry_ms, default 30000; transport catchup_retry_interval) that re-pulls from the CURRENT applied frontier. Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when consumed by the rate limit or an in-flight pull, so the gap always keeps a standing wake-up until a pull completes. Verified over real sockets: catchup_retry.rs reproduces the incident (pull fails, leader appears, zero pushes) and proves timer-only self-heal — and that a clean completion arms nothing.
• WAL segment files are format-versioned. New segments open with an 8-byte header (TSEG magic + version byte + reserved); pre-m11p4 headerless segments stay readable as implicit version 0 — no migration. A segment this binary cannot identify (unknown header version, unrecognized leading bytes, unparseable .seg filename) surfaces as the new WalError::SegmentFormatUnknown at open — previously it scanned as empty (segments=0) and recovery's torn-tail repair could TRUNCATE the foreign file to zero. Foreign-format files are never repaired, truncated, or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
• Unservable catch-up is a structured refusal. SegmentSource::collect_from returns typed SegmentReadError::{Unavailable,Failed}; TidalDb::read_wal_batches returns the typed WalError (was stringified TidalError). The StreamSegments handler maps Unavailable to FAILED_PRECONDITION — "segments not available from seq N; snapshot required" — and the follower logs it distinctly (catch-up unservable … needs a snapshot (m11p5) or an operator reseed) instead of burying it as a transient. The on-disk segment format is now documented (tidal/src/wal/segment.rs module docs + spec 01 §2.2).

Quorum-acked writes (m11p3) — ack=leader|quorum, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)

• ack=quorum is an opt-in durability contract for every replicated write (/signals, /items, /embeddings): success means a majority of the replica set durably holds the write (leader + floor(n/2) followers, each storage-applied and own-WAL-fsynced), so an acked write survives the permanent loss of any single node — including the leader. Deployment default via topology replication.ack; per-request override via the x-tidal-ack header (forwarded verbatim by gateways). ack=leader (the default) is the m0–m11p2 contract unchanged.
• Durable frontier reports. Followers PUSH their durably-applied frontier to the leader once per apply round (new ReportApplied RPC, fired by the segment receiver through the new Transport::notify_applied) — batch-level and fully decoupled from ship acks, so the commit index stays fresh even when outbound ships stall (gap-parked follower, pull-based catch-up, quiet leader). Ship acks keep their m11p2 instant floor-hint semantics — both inputs are durable-true because a follower's frontier only ever advances after its storage apply + own-WAL fsync. (The first design held each ship ack until its segment's apply; measured under open-loop load, that couples ship cadence to apply latency and one gap-parked follower spirals into total quorum collapse — the report push is what shipped.)
• Follower blob applies are group-committed. m11p2 applied replicated items/embeddings one record at a time — one solo follower fsync per item, capping item apply at the fsync floor (~100/s on macOS) and stalling the quorum frontier behind any item burst. The receiver now hands each apply round's blob records to the engine as ONE batch (apply_replicated_blobs: validate all → stage all WAL appends → wait all → upsert storage), and the WAL writer flushes queued blobs under ONE group fsync. Measured: corpus seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
• Commit index. The leader folds durable acks (and heal resumes) into per-peer durable marks; the commit index is the k-th largest (k = floor(n/2)), leadership-scoped (promote resets it to the stream baseline; demotion fails in-flight waiters — a demoted leader never claims quorum). Handlers await it through a watch-channel bridge — fully async, zero threads parked per waiter (the thread-per-wait design measurably collapsed at 1k rps open-loop by exhausting the blocking pool and starving the very completions that advance the index). Quorum capacity measured on a real 3-process localhost cluster (release, writes mix): 3,600 quorum signal-writes/s within SLO (p50 ~45ms, zero errors, replication lag ≤3 events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack figure, vs the ≥50% gate.
• Honest timeout semantics. Quorum not confirmed within replication.quorum_timeout_ms (default 2000) → a retryable 503 naming the laggard regions, the commit index, and needed/confirmed counts. The write is in the leader's log and may still commit: retries are at-least-once (items/embeddings retries are idempotent upserts; signal retries can double-count — decided in-phase: no idempotency-key machinery, documented in runbook §8 with the session-write precedent for callers that need exact-once).
• x-tidal-seq on every cluster write response: the write's seqno in the replicated log (relayed through forwards) — an exact durability cursor against commit_index in /cluster/status/local (which also gains ack). tidaldb_cluster_relay_durable_seq now reports the commit index (relay_last_seq − relay_durable_seq = quorum lag); new counter tidaldb_cluster_quorum_timeouts_total.
• The ledger gate (exit gate, run for real): tier-3 mp_quorum_ledger_zero_acked_loss_across_killpoints SIGKILLs the leader under concurrent quorum load and proves zero acknowledged loss on the promoted max-applied survivor — frontier invariant (max acked seq ≤ survivor applied) plus per-item content probes. 167/167 kill points passed on the final design (batches of 64 + 91 + 12; kill timings spread 120–598ms across fresh 3-process clusters; an earlier 100/100 run had validated the superseded ack-holding design before it was replaced — see docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics (one follower down: quorum commits; both down: fast 503 naming laggards while ack=leader flows; heal: recovers) and in-process gRPC coverage (override headers, forwarded quorum writes, blob writes, 400 on bad mode).
• Rolling-upgrade order (mixed-version caveat): a pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack — it silently applies LEADER-ack semantics to a quorum request (a durability downgrade the caller cannot see). Upgrade the leader first: ack=quorum is then honored immediately (commit-index freshness rides the m11p2 ship-ack floor hints until the followers upgrade too). Replication and heal are unaffected by either order. Runbook §8 records the procedure.

One replicated log (m11p2) — items/embeddings ride the WAL, StreamSegments catch-up, HTTP broadcast deleted

• The leader's WAL is now THE replicated log. The group-commit writer hands every fsynced batch to a bounded in-memory ship feed (wal::feed::WalShipFeed); the ship queue pushes those already-encoded bytes verbatim — byte-identical on the leader's disk, the wire, and the follower's apply path — and stream seqnos are WAL seqnos, so they survive restarts (the m8p10 relay-reset hazard is gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning machinery left the server write path entirely (/signals stages straight through the engine's group commit; a WAL fsync failure now surfaces per-write exactly like single-node).
• Item metadata and embeddings are replicated mutations. /items and /embeddings journal kind-1/2 blob records (WAL header flags byte = batch kind; one record, one seqno) BEFORE storage, on the same stream as signals. Followers apply them kind-aware — WAL-first into their own log, then idempotent storage upserts — and recovery replays them. The m8p10 HTTP item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast, authed side-POSTs — the source of both 2026-06-10 live bugs) is deleted; those bug classes are now impossible by construction. Cluster /items returns a plain 201 and /embeddings a plain 204 (no broadcast-report body).
• StreamSegments implemented — catch-up is follower-pulled. The ship feed's tail is bounded; a peer that falls behind it is skipped ahead, and the follower pulls the hole itself via the (previously declared-unimplemented) server-streaming RPC over the leader's durable, BLAKE3-verified segments. Pulls trigger on detected gaps, on follower boot (self-driving restart catch-up), and on the leader's heal nudge (POST /cluster/catchup, internal, forwards the operator's own bearer credential). Pulled chunks flow through the same inbound apply path as live ships. Ship acks piggyback the follower's applied seqno (ShipSegmentResponse.applied_seqno), so retries of already-applied data prune and heal is resume_from + nudge — no redelivery scan, no O(items) traffic.
• Promote carries a stream baseline. A promoted leader's stream starts at its promote-time flushed frontier (persisted in data_dir/stream_baseline); the fan-out body and every catch-up chunk announce it, so peers jump their frontier past pre-stream history instead of parking on a phantom gap. Ship queues are leadership-gated (activate_from/deactivate): a follower's replicated applies never echo back at its peers.
• Multi-process cluster mode now requires --data-dir (validated at startup): the durable WAL is the replication stream. Standalone single-node deployments are untouched — blob journaling and the ship feed are gated on cluster peers, so single-node item writes keep fjall-only durability with zero extra fsyncs.
• New tier-3 suite mp_items_ride_the_log_and_catchup_stream: items written on the leader AND through a follower gateway converge everywhere via the log (feed parity 1e-6), and a follower stopped through item+embedding+signal writes restarts and converges via its boot-time StreamSegments pull with no heal verb and no HTTP item traffic.

Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first tidaldb_cluster_* metrics

• The replicated /signals write path no longer serializes every writer onto a solo group-commit fsync nor ships to followers on the request path. Writes are staged (seqno + WAL submission + relay log push, microseconds, atomic with rollback) and completed (shared group-commit fsync + in-memory fold) in two phases, so concurrent writers coalesce into one fsync; follower shipping moved to per-peer sender threads that coalesce contiguous runs into multi-event batches with a windowed in-flight budget (topology knobs replication.{batch_max_events,window,retry_ms}). The 204 contract is unchanged (leader durability only) — it now returns at leader fsync. Measured on a real 3-process localhost cluster (release build, thepeach mix): 4,534 replicated signal-writes/s sustained within SLO vs ~90/s before (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
• Durable-frontier shipping + relay poisoning. Senders only ship the leader's contiguous fsynced prefix (an event a follower holds but the leader could lose is silent divergence); a staged write whose fsync fails poisons the relay — further cluster writes are rejected and the ship frontier freezes (the CockroachDB/Postgres fsync-failure posture).
• Follower group-commit coalescing. The segment receiver drains its inbound backlog (Transport::try_recv_segment) and applies it through ONE shared group commit (SignalLedger::apply_replicated_events), in range-disjoint groups so duplicate/subset re-ships cannot double-fold. Without this the follower apply ceiling was ~events-per-segment / fsync-cost (~1.8k events/s measured) and lag grew without bound under m11p1 leader rates.
• First cluster metrics + cluster /metrics listener. Cluster mode previously had no metrics endpoint at all. New per-region topology metrics_addr wires the engine's Prometheus listener; new tidaldb_cluster_* series: ship RTT + batch-size histograms, per-peer ship counters/gauges (peer_shard labels), WAL fsync latency + group-commit fill histograms, write-pool depth/rejections, relay committed + durable frontiers. WAL group-commit knobs are deployment config (wal.{batch_size,batch_timeout_ms} topology block; wal_batch_size/wal_batch_timeout builder methods).
• Engine API: TidalDb::signal_staged/StagedSignal::wait, SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}, ShipQueue (per-peer windowed batch senders with pause/resume), WalWriter::append_signal_staged, WalSender::append_record_staged, range_payload/encode_run. Relay log entries are now RelayEvent (raw event records, re-encoded deterministically at ship time) instead of pre-encoded single-event bytes.
• Ship-sender failure logging is transition-based (first + every 50th consecutive failure WARN with the running count, recovery INFO) — the per-retry WARN flood could fill an undrained log pipe and stall the process. The multiproc test harness now discards child logs via /dev/null (a piped fd nobody drains deadlocks the node once the kernel buffer fills), with TIDAL_TEST_NODE_LOGS=inherit to stream them while debugging.

M9 — Community Sync & Revocation

• Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: SignalScope, CommunityId, Membership, MembershipEpoch, PolicyMetadata. Community signal reconciliation via CrdtSignalState with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

M10 — Governance & Agent Rights

• Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE

• Multi-process cluster mode: tidal-server cluster --region <name> [--data-dir <p>] (env TIDAL_REGION) runs one process per region (RegionClusterState), each owning one TidalDb and one GrpcTransport that binds this region's grpc_addr and dials every sibling's real grpc_addr — real process/host isolation. The topology requires per-region grpc_addr AND http_addr in this mode; single-process mode (no --region) is unchanged as the dev/demo default. Both modes stay behind the experimental gate (--experimental-cluster / TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) with mode-specific WARN text.
• New routes / behaviors on the multi-process surface: GET /cluster/status/local (per-node status), GET /cluster/status aggregates ALL regions with a reachable field (unreachable ⇒ reachable:false + worst-case lag), POST /cluster/reconcile {region} (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact no-op on scores), POST /hardnegs {user_id,item_id} (user-scoped hide; converges via reconcile, filtered from /feed?user_id). Writes on a non-leader forward to the leader; ?region= reads forward to the owning region; default reads are LOCAL in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast to peers with a per-peer report: POST /items → 201 {replicated_to, failed}, POST /embeddings → 200 with the same report on the leader path (NOT 204 — a 204 cannot carry the report; the forwarded/internal path stays 204). /sharded/* fans out across processes (degraded semantics preserved). promote fans out to all peers ({ok, leader, acked, failed}).
• /cluster/heal is the single recovery verb: redelivers missed signal segments (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold 5, reset 30s) is open, so re-issue /cluster/heal until /cluster/status shows lag 0.
• TIDAL_HLC_SKEW_MS (multi-process): signed ms offset applied to the process's HLC (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
• Tier-3 UAT over real OS processes (feature cluster-e2e): cluster_multiproc (5), cluster_chaos (3, REAL network-partition injection via a root-free in-harness TCP relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl remain an operator option), cluster_lifecycle (2, ±500ms clock skew + rolling upgrade with zero acknowledged-write loss), cluster_runbook (9, every docs/runbooks/cluster.md §5–§11 operation), plus cluster_e2e (2, single-process smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

Fixed

Four production bugs surfaced and fixed by the m8p10 tier-3 UAT

• Silent data loss on out-of-order ships. The replication applied high-water-mark swallowed sequence gaps when eager ships arrived out of order. The applied frontier is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
• Non-idempotent reconcile (score creep). take_crdt_snapshot attributed replicated signal streams per-node, so reconciling already-converged nodes crept the decayed scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical replication shard (ShardId::SINGLE), making merge idempotent — reconcile of converged nodes is an exact fixpoint.
• Lag gauge conflated leader streams across a promotion. A converged node reported a permanent phantom lag after a /cluster/promote moved leadership to a different shard. The lag gauge now tracks the leader high-water-mark per source shard (ReplicationLagGauge::leader_seqno_for), computing lag against the current leader.
• Items missed during a broadcast were never backfilled. A node down/partitioned during an item/embedding broadcast was permanently missing that data even at signal lag 0. /cluster/heal now re-broadcasts item metadata + embeddings to the healed region (idempotent upserts), making heal the single recovery verb.

Changed

Cluster mode (m8p8): real gRPC replication

• tidal-server's ClusterState now wires each follower region to a real tidal-net GrpcTransport (self-loop over loopback gRPC) instead of in-process crossbeam channels — replication between regions traverses real gRPC/TCP (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
• Follower gRPC ports are auto-allocated from the topology (grpc_addr optional per region) and self-heal a transient bind race by retrying on a fresh port.
• Blocking gRPC ships triggered by POST /signals and POST /cluster/heal are offloaded to a dedicated thread so they never block the axum reactor.
• ClusterState is constructed off the async reactor (GrpcTransport::new blocks on its own runtime).
• The experimental opt-in gate and docker/cluster/Dockerfile are updated: single-process cluster mode is honest that it replicates over real gRPC but runs all regions in one process (no host/process isolation). (True multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry above.)

Docker build fixes (all three images now that tidal-server pulls tidal-net)

• Install protobuf-compiler in the builder stage — tidal-net's build script runs tonic-build, which needs protoc to compile the WAL-shipping .proto.
• Pin the builder base to bookworm (rust:1.91-bookworm / rust:1.91-slim-bookworm) so its glibc matches the debian:bookworm-slim runtime; the default trixie base emitted a libmvec.so.1 dependency absent on bookworm, aborting the binary at startup. Verified: docker run of the cluster image serves a functional 3-region cluster (write replicates to both followers over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
• New tests: tidal-server/tests/cluster_grpc.rs (in-process gRPC replication + HTTP offload path) and a hardened tier-3 cluster_e2e.rs (multi-process smoke
  • promote over real OS processes, feature-gated).

[0.1.0] - 2026-02-23

Added

Core Database Engine

• TidalDb embeddable database with ephemeral() and with_data_dir() open modes
• SchemaBuilder for defining signal types, decay parameters, and ranking profiles
• TidalDbBuilder fluent builder with schema, data directory, metrics, and rate limiter configuration

Signal System

• Typed signal recording with exponential decay scoring
• Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
• Windowed aggregation: OneHour, TwentyFourHours, SevenDays, AllTime
• Signal velocity tracking
• WAL-backed signal durability with crash recovery
• Periodic signal checkpointing to fjall (every 30s)
• WAL compaction after each checkpoint

Retrieval (RETRIEVE query)

• 5-stage pipeline: universe, filter, score, diversify, return
• Filter expressions: Eq, In, Gt, Lt, And, Or, Not, InCollection, InProgress, MinSignal, MaxSignal, NearLocation
• Built-in ranking profiles: trending, for_you, new, popular, recent, and 20+ more
• Custom ranking profiles via SchemaBuilder
• Diversity enforcement (max N per category/creator)
• Sort modes: Relevance, Trending, Newest, MostLiked, MostViewed, MostFollowed, AlphabeticalAsc/Desc, Shortest/Longest, LiveViewerCount, DateSaved, and more

Search (SEARCH query)

• BM25 full-text search via Tantivy
• Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
• Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
• Creator search with entity_kind(EntityKind::Creator)
• similar_to(EntityId) for content-based recommendations
• Scope pre-filters: Trending, CohortTrending, Following, Category, Collection
• Autocomplete suggestions via db.suggest()

Entity Model

• Three built-in entity types: Item, User, Creator
• Metadata storage as HashMap<String, String>
• Embedding slots (up to 4 per entity type) via USearch
• Relationships: Follows, Blocks, Hide, Mute, InteractionWeight

Sessions

• Session lifecycle: open_session, close_session
• Cross-session preference vector updates (EMA blend)
• Session snapshots with signal state and preference vectors
• Session serialization format v0x03 with backward compatibility

Social Graph

• Creator follower/following indexes
• Cohort membership (user segments)
• CoEngagementIndex for co-viewing patterns with LRU eviction
• Social graph filter for "followed creator" content scoping

Collections

• Named collections with Private, Shared, Public visibility
• create_collection, add_to_collection, remove_from_collection, list_collections
• FilterExpr::InCollection for collection-scoped retrieval
• Saved searches with save_search, list_saved_searches, retrieve_saved_search

Observability

• enable_metrics(addr) -- Prometheus-format /metrics endpoint + /healthz JSON
• 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
• tidaldb_checkpoint_failures_total counter for checkpoint monitoring
• TidalDb::diagnostics() -- structured health snapshot
• WAL diagnostics and recovery tools

Safety

• Signal weight NaN/Inf validation (returns TidalError::InvalidInput)
• Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
• Export request limit: 500K signals max per request
• FilterExpr complexity limit: 256 nodes max
• Data directory lock (tidaldb.lock) prevents dual-process corruption
• Schema fingerprint persistence detects decay parameter changes on reopen
• Bounded closed_sessions cache (10K max, LRU eviction)
• Metrics server non-loopback bind warning

CLI (tidalctl)

• tidalctl binary for database inspection and diagnostics

RLHF / ML Export

• db.export_signals(ExportRequest) -- WAL-based signal export for training data
• db.user_session_summary(user_id, since_ns) -- aggregated session statistics

Stability

tidalDB 0.1.0 is pre-1.0. No API or data format stability guarantees are made for 0.x releases. Upgrade guides will be provided for each minor version bump. Do not upgrade 0.x to 0.y on a live data directory without reading the release notes.

---

Format based on Keep a Changelog