tidaldb/CHANGELOG.md

Changelog

All notable changes to tidalDB will be documented in this file.

[Unreleased]

Stability

tidalDB is production-ready. M0–M12 are shipped and the pre-release "not yet recommended for production" disclaimer has been withdrawn from the documentation set. The API surface and on-disk data format are stable for shipped features: additive changes ship in minor releases, and any breaking change to a public API or a persisted format ships with a documented migration path in this file. This supersedes the 0.1.0 "no stability guarantees" note below.

Fixed

vector_search distances now honor the documented [0.0, 4.0] contract (wire-visible)

The read path passed the caller's raw query vector straight to the index while the write path L2-normalized every stored vector, so the two sides lived in different spaces. With unit v, d = |q|² − 2q·v + 1: a non-unit query shifted and scaled every distance by |q|². Measured on a live RF3 cluster, /vector_search returned distances of 591–1174 against a contract promising ≤ 4, and an exact match scored |q|² − 1 instead of ~0.

db::query_ops::vector_search_items now normalizes the query with the canonical storage::vector::l2_normalize before searching. A zero-norm query — which has no direction, so "nearest by cosine" is undefined — is rejected with 400 instead of returning an arbitrary ranking.

Ranking is unchanged. |q|² and 1 are constant across candidates, so the ordering was already correct cosine order; that is why this went unnoticed. Absolute uses of the number were the casualties: distance thresholding, dedup-by-distance, cross-query comparison, and recall measurement.

Migration: any client comparing distances against a hard-coded threshold derived from the old unnormalized values must re-derive it. An exact match now scores ~0 (2.4e-7 measured; the f16 quantization floor is ~1e-3, so compare with a tolerance, never equality) and all distances fall in [0.0, 4.0].

Blob replication is now observable (tidaldb_cluster_blobs_*)

The blob path — item metadata, embeddings, term markers, cluster membership — had zero instrumentation: lag_events counts WAL apply, so a blob that never shipped or never applied moved no number anywhere. Added per-kind counters tidaldb_cluster_blobs_originated_total, _applied_total and _apply_failed_total, labelled by record kind with cardinality fixed at 4 by construction. Comparing originated on the writing node against applied on each peer localises a replication gap to enqueue, ship, or apply in one query.

Every co-located shard group now reports its vector count

tidaldb_usearch_vector_count was rendered only for the node's metrics-owner group, so on a 3-group node two thirds of the corpus had no vector-count series and could diverge unobserved. Co-located groups now render tidaldb_usearch_vector_count{shard="N"}; the owner's series stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets each replica set separately without double-counting.

Added

Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in for_you

• Online preference clustering (entities/multi_preference.rs). A warm user (≥ COLD_START_N = 5 interactions) maintains up to K_MAX preference clusters built by online sequential k-means with a DP-means threshold split: a new engagement updates its nearest cluster (per-cluster adaptive EMA) or, past the split threshold and under the cap, opens a new cluster; at the cap the nearest cluster absorbs it. Per-cluster importance composes the canonical forward-decay kernel anchored to each engagement's timestamp, so stale interests fade.
• Top-M ANN fan-out. At query time for_you selects the top-M clusters by current importance, issues M ANN queries (candidate_gen::ann_candidates_multi), and merges by best (min) distance; the personalization boost is the max cosine over all clusters. Users below the cold-start threshold keep the single adaptive-LR vector (entities/preference.rs). Design: docs/research/multi-vector-preference.md.

Idle-readiness + TLS scale-up (m12p5–m12p6) — followers converge readiness on an idle cluster, and elasticity is proven over REAL mTLS on k8s

• Idle-readiness convergence (m12p5). The leader heartbeat now carries its live frontier (leader_last_seq), so a caught-up follower flips /health Ready on an idle cluster instead of stalling until the next status poll or ship. Cert SAN wildcard widened for scale-to-5.
• TLS scale-up (m12p6). A real kubectl scale 3→5 exercised seed-join over mTLS on k8s (kind) for the first time: joiners flip Ready in ~13s via the idle-readiness heartbeat, auto-promote to Voter, reach full content parity at lag 0, with zero acked DATA loss across scale-down. Fixes span a six-bug chain — https:// seed scheme, rustls CryptoProvider install order, headless seed Service, cold-handshake poll timeout, two-tier cert-manager PKI, and the grpc_tls_for CA fallback for a not-yet-in-topology joiner. Persists the HNSW graph and skips a suspect graph on reseed-pending close.

Sharded ingestion (m12p4) — scatter-gather across shard groups with cross-shard unified reads

• Scatter-gather pool + cross-shard reads. Writes hash-route across a 3-group shards: topology; reads unify across groups (L4). Ran REAL on kind with a 2-generator load job. Fixed an HTTP/2 204 forward-relay bug (a synthesized JSON body on a 204 relay triggered an h2 RST_STREAM). Confirmed with data: at fixed per-pod CPU, full-placement sharding scales failover, not write throughput; the ≥2.5×-and-≥5,000/s scaling target remains Ref-A/k3s-pending.

Index tuning + recall/memory at the production shape (m12p3) — the G2 work: per-query ef_search is now honored, the brute-force crossover scales with dimensionality, and the HNSW recall/latency/memory frontier is measured at 1536-D with a real exact oracle

• Per-query ef_search override — now real. UsearchIndex::search / filtered_search honor a per-request ef_search instead of silently dropping it to the index default (pre-m12p3 behaviour: accepted for trait compliance, logged a warning, ignored — USearch 2.24 has no per-call beam argument). The override is race-free via an RwLock epoch guard (with_expansion): searches that agree on ef_search run in parallel under a shared guard; only a query that changes the live beam width takes the exclusive guard for its (set, search) window — not a per-search mutex. The knob was already plumbed end-to-end in m12p1 (vector_search_items(.., ef_search), the /vector_search ef_search field, tidal-stress --recall-ef-search); m12p3 makes it move recall. ef_search=0 selects the slot default.
• Dimension-aware brute-force → HNSW crossover. The exact BruteForceIndex scans every vector under a read lock at count × dim cost, so a fixed 10,000 crossover meant a 15.4M-FMA scan at 1536-D (tens of ms, blocking writers). usearch_min_vectors(dim) now keeps a brute-force scan within ~4M FMAs: ≈10,000 at/under 128-D (byte-compatible with pre-m12p3), ≈2,600 at 1536-D — flipping high-dim mid-size slots to HNSW before the scan blows the SLA.
• memory_usage() exposed on UsearchIndex (the true graph + vector footprint from USearch, not the index_stats lower bound) for pod sizing.
• Grid-search harness. cargo run --release --example ann_grid_search builds a UsearchIndex + an exact BruteForceIndex oracle over the same deterministic id-keyed corpus and reports, per (M, ef_construction, ef_search, quantization) point, measured recall@10 vs the oracle, mean/p99 search latency, build time, and the true footprint — the tool that produces the documented M/ef and the F32/F16/Int8 recall+memory numbers.
• Measured at 1536-D (100k clustered corpus, real exact oracle). The production default (M=16, ef_c=400, F16) clears G1 and G2: recall@10 0.997 at p99 ≈ 1.4 ms raw ANN; ef_search is the latency lever (recall saturates by ef_s=128 → p99 ≈ 1.0 ms). F16 costs only 0.25% recall vs F32 for half the RAM (≈ 5.2 GB/1M true footprint incl. graph); Int8 rejected at 1536-D (recall 0.715, −28%). Live tidal-stress --verify-recall against a real server gave recall@10 = 1.0000 at 20k/1536-D, default beam and --recall-ef-search 400. Finding: the recall corpus is now clustered (Gaussian mixture) in both the grid harness and tidal-stress (recall::embedding_for) — uniform-random high-dim vectors are pathological for recall@k (≈0.97 at 10k → ≈0.54 at 100k, a measurement artifact, not an index regression). Full frontier + the 1M command in docs/profiling/usearch-tuning.md and docs/profiling/scale-baselines.md. docs/specs/07-vector-retrieval.md updated: per-query ef_search is IMPLEMENTED, not deferred.

ANN candidate generation in RETRIEVE (m12p2) — the G1 unblock: for_you/related source candidates by nearest-neighbour, trending by a cached per-signal-type top-K, so the feed stays relevant AND bounded as the corpus grows past the scan cap

• ANN in retrieve. CandidateStrategy::Ann is now wired into the RETRIEVE executor (it previously fell back to a scan with a warning). The db layer resolves the query vector — the user's preference vector for for_you, the seed item's embedding (similar_to) for related — and Stage 1 runs an O(ef_search) HNSW search over the item content slot instead of scanning an arbitrary low-id slice of the universe. for_you and related are now Ann profiles. Graceful: no registry / no preference vector / no seed ⇒ it degrades to a scan (anonymous reads and cold-start users still serve), so every embedding-less schema and pre-m12p2 caller is unchanged.
• Cached SignalRanked. The O(N) ledger scan behind SignalRanked is now a cached per-signal-type top-K (signals/ledger/hot_top_k.rs): O(K) on the served path, with a bounded O(N) rebuild only when stale. Decay preserves relative order (same λ), so a cache is valid until the next write; small ledgers rebuild on any write (always fresh), large ledgers throttle the rebuild off the read hot path (1s). trending now uses SignalRanked(view), so it ranks the actually-viewed corpus at any id — not the low-id scan slice.
• related over HTTP. GET /feed?profile=related&similar_to=<id> resolves the seed's embedding and runs ANN — "more like this" on the read surface (similar_to added to FeedQuery, threaded through all three feed handlers).
• Harness /feed measurement. tidal-stress gains --feed-profile <name> (force every feed read to one profile, for per-profile retrieve p99) and --seed-preferences (build a preference vector per user so for_you exercises ANN, not the scan fallback).
• Verified real against a 1536-dim standalone server: trending retrieve p99 = 3.5–7.7ms (under the 10ms G1 target) under concurrent writes, via the cached top-K; for_you retrieve is ANN-backed (preference vectors built) at p99 ≈ 24ms, dominated by the Stage-3 preference-boost recompute (per-candidate embedding read) — flagged for m12p3's index/score tuning (the risk register's "materialized-score layer"). ANN candidate recall is the m12p1 /vector_search probe (0.9997). New engine tests prove ANN/related/trending reach the relevant items at high ids a scan can never reach, plus cache freshness.

Read-recall harness — measurement truth for ranking-at-scale (m12p1): recall@k + true p99 at the production shape, so G1/G2 steer on real numbers instead of absent ones

• Pure k-NN probe. TidalDb::vector_search_items(query, k, ef_search) returns the raw HNSW nearest neighbours over the item content embedding slot — NO profile scoring, fusion, or diversity — so the result is the ANN index quality in isolation (the G2 metric). Exposed as POST /vector_search on the standalone router and the multi-process region node (merge-by-distance across hosted shard groups); a dimension-mismatched query is a 400, not a 500.
• The oracle. tidal-stress --verify-recall seeds the corpus with deterministic, id-keyed embeddings (reproducible with --skip-seed), holds a brute-force cosine ground truth in RAM, and ramps /vector_search probes open-loop (coordinated-omission corrected). It reports per-stage true p99 (a genuine tail, not a closed-loop mean) AND mean recall@k, plus the read-knee — the highest sustained QPS where p99 ≤ target AND recall@k ≥ target both hold — with a machine-readable JSON summary and --fail-on-knee PASS/FAIL exit. Knobs: --recall-k, --recall-queries, --read-p99-target-ms, --recall-target, --recall-ef-search.
• Verified end-to-end against a real standalone server at 1536-dim: recall@10 = 0.9997 at 20k items (HNSW M=16/ef=400/F16 vs brute-force cosine, far above the 0.95 G2 target); the harness also exercises the read-knee verdict (both branches) and gate exit codes. The 100k/1M exit-gate runs use the same harness on the k3s cluster (the brute-force oracle needs ~6 GB RAM at 1M).
• No more mean-as-p99. Repaired the fabricated p99 column in docs/profiling/social-scale.md (the social bench is Criterion = mean only) and the scale.rs / scale-baselines framing; every closed-loop number is now labelled an isolated per-op mean (regression tripwire), with the p99/recall tail SLOs signed off only by the open-loop harness. Added a 1536-dim HNSW-vs-brute recall@10 bench (benches/vector.rs) and a recall-harness section to docs/profiling/scale-baselines.md.

Sharding × replication + rebalancing (m11p6) — the "replicated XOR sharded" split is over: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes

• One write surface. /items///embeddings///signals now hash-route the entity to the owning shard group's leader (the engine's FNV-1a ShardRouter) AND replicate at RF — sharding and replication at once. The old /sharded/*-vs-leader split in tidal-stress (WritePath::Leader|Sharded) is gone; it drives the one unified path, spreading writes round-robin across gateways (or pinning one with --leader-url). x-tidal-ack/x-tidal-seq, quorum await, and NotLeader/QuorumTimeout are per-group; NotLeader names the group.
• Rebalancing verbs (L3). POST /cluster/shards/{id}/transfer moves one group's leadership (the m11p4 fenced transfer scoped to the group); POST /cluster/shards/{id}/replicas adds/removes a replica (the m11p5 join / fenced-removal per group). A ?shard= selector threads through every per-shard admin verb (promote/heal/partition/catchup/reseed/members/join) and is PROPAGATED on every intra-group forward/broadcast (ShardReplica::admin_path) so the receiving sibling targets the same group. S=1 is byte-for-byte (no selector emitted, no shard in the NotLeader body).
• Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over real OS processes — SIGKILL a node under ack=quorum load → ONLY its shard-leaderships re-elect (survivor groups keep theirs), reads never stop, and every acked write is present on its shard's new leader (zero acked loss), across random kill points. Plus a rebalance-verb test (transfer + ?shard= promote move exactly one group). Harness: MultiProcCluster::start_sharded (per-(node,shard) ports, shards: emission, agreed_shard_leaders).
• Throughput. Local 3×3 release cluster sustains 3,000 quorum signal-writes/s at 0% error with per-node CPU ≈30% and replication lag ~0 — generator-bound, not engine-bound. The ≥5,000/s + ≥2.5×-single-shard scaling is Ref-A (k3s Linux/fdatasync/multi-generator) — the standing access caveat since p1.
• Known follow-up (tracked): per-group-aware node readiness (today is_ready is node-global across co-hosted groups) and cross-node read fan-out under PARTIAL placement; the exit gate runs full placement, which these do not touch.

Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix

• Fault injection compiled out of production. A non-default fault-injection cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks (tidal/src/fault.rs): slow-fsync (TIDAL_FAULT_FSYNC_DELAY_MS sleeps before each durable fsync) and disk-full (TIDAL_FAULT_DISK_FULL_AFTER_BYTES returns a real ENOSPC/errno-28 once cumulative segment bytes cross the threshold). The production image build never passes the feature, so the hooks are compiled out entirely — a disk/fsync fault a stray env var could trip in prod is a 3am footgun we refuse to ship; the safety is structural. Inert until armed even when compiled in. The tier-3 harness builds the spawned binary with the feature.
• First-class invariant checkers (tidal-server/tests/support/invariants.rs): the no-acked-loss AckLedger (frontier + content, extracted from the m11p3 ledger gate, which now consumes it), cross-replica assert_feed_parity / feed_item_ids, assert_single_leader_per_term (membership safety, reads the m11p6 shards[] rows), and MonotonicCounters (per-node applied frontier + leader commit index never regress, with legitimate epoch resets forgiven).
• New fault-class suite (tidal-server/tests/cluster_faults.rs, tier-3): a disk-full follower degrades gracefully (receiver halts, node alive, zero acked loss, restart recovers to parity); a slow-fsync follower lags then converges while the fast follower supplies quorum; both followers slow → ack=quorum returns a retryable 503 naming the laggards while ack=leader is unaffected; an asymmetric partition (inbound severed, outbound up) causes no split brain (pre-vote + check-quorum) and no loss. 4/4 green.
• Soak + regression gates in tidal-stress: --json-summary <path> (a machine-readable per-stage p99/throughput/error roll-up for trend lines) and --max-p99-ms / --max-error-pct / --fail-on-knee gates that exit non-zero on a regression (the tool always exited 0 before).
• Nightly pipeline (.woodpecker.yaml): a cron nightly flow (chaos suites with elevated kill-points + the fault-injection feature, then the gated tidal-stress soak) beside the existing push release gate, event-routed by per-step when. Woodpecker, never GitHub Actions.
• Guarantee traceability (docs/planning/milestone-11/guarantee-traceability.md): every roadmap §2 guarantee mapped to its named automated test(s). Closes the G-C apparatus; the 30-consecutive-days-green half of the GA bar is a calendar criterion the nightly pipeline accrues.

Observability + operations (m11p8) — complete cluster metric set on a per-node /metrics listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate

• Metrics. Completed the tidaldb_cluster_* set with the two members the roadmap named that were missing — breaker (tidaldb_cluster_peer_breaker_state 0/1/2 + tidaldb_cluster_breaker_opens_total, surfaced read-only from the tidal-net circuit breaker via a new Transport::peer_breaker_state) and forwards (tidaldb_cluster_forwards_total / _forward_failures_total, instrumented at the gateway forward path) — plus the self-heal series (heal_attempts/successes/noops_total, healing_peers). When several shard groups co-locate on one node (m11p6) the metrics-owner serves the single /metrics listener and the siblings register their series under a shard="N" label (TidalDb::register_metrics_sibling); a single-shard node is byte-identical to before. A 12-panel "Cluster Replication" Grafana row + an 8-rule tidaldb-cluster Prometheus alert group ship beside the standalone ones.
• Request-id + tracing. Both cluster routers (single- and multi-process) now carry the standalone router's SetRequestId + PropagateRequestId + TraceLayer stack (extracted to router::with_request_id_tracing); the id rides the follower→leader forward hop verbatim, so the leader's span shares the gateway's x-request-id. (Ships are off-request-path and batched, so they correlate by seqno, not a request-id — by design.)
• Truthful status. Fixed the leader's own applied_events reading 0 (a leader writes its WAL directly and never advances its own applied frontier — now reports its flushed frontier) and the single-process post-promote ShardId(0) lag-keying undercount (now keys on the current leader's shard).
• Self-driving heal. A standing leader duty re-arms the backlog re-ship for a stuck (breaker-open, behind) peer every ~3s, so it converges through breaker resets with no operator /cluster/heal loop — closing the §1.4-3 footgun. Observable via healing_peers + the TidalDBClusterHealNotConverging alert.
• WAL PITR archival. wal.archive_dir (topology + builder): the online compaction copies each sealed segment to the archive — durably, before deletion, refusing to delete if archival fails — so the archive is a gap-free PITR record.
• Backup/restore. tidalctl backup / restore: offline data-dir backup with a BLAKE3 BACKUP_MANIFEST.json (+ the WAL checkpoint cursor); restore verifies every file's hash before writing and refuses a non-empty target. Coordinated cluster backup = back up one committed replica per shard group (the runbook drill).
• Rolling upgrade. A wire version handshake (HeartbeatRequest.build_version, stamped at the transport boundary; >= 2-major skew WARNs, never rejects) + version on /cluster/status. mp_rolling_upgrade_no_loss_no_stall promoted to the FIRST step of .woodpecker.yaml (the release gate; a failure blocks the image build). See milestone-11/phase-8.md.

Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit

• The cluster stops trusting the network. gRPC replication mTLS is now the intended posture: the inbound server is served over a custom tokio-rustls acceptor (not tonic's fixed .tls_config()) fed a DynamicCertResolver (ArcSwap<CertifiedKey>), preserving mutual TLS exactly (a WebPkiClientVerifier over the cluster CA — a foreign/absent client cert fails the handshake before any RPC). Plaintext is an explicit insecure: true with a loud startup WARN.
• Cert + bearer rotation WITHOUT restart. A content-hash poller re-reads the cert files (k8s ..data symlink swaps that inotify misses) and atomically swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys, so a rotation drops zero requests (verified under concurrent load). Outbound peer channels rebuild from the refreshed files. The bearer and a new shared cluster key live behind ArcSwap, read per request and reloaded from TIDAL_API_KEY_FILE / TIDAL_CLUSTER_KEY_FILE.
• Authenticated inter-node HTTP with per-node identity. The axum listener serves TLS (reusing the same hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/scatter/status/seed-join dial https:// with the cluster CA. A forwarding node mints an x-tidal-node-token (keyed-BLAKE3 MAC over node-id + expiry under the cluster key — no new crypto dependency) so a foreign pod cannot forge a sibling identity. The x-tidal-internal marker is now honored ONLY from a verified sibling (marker without a valid node token → 403): the marker stays a routing hint, never an authorization bypass. All opt-in via grpc_tls / the cluster key — absent ⇒ pre-m11p7 behavior, so every existing deployment and test is byte-for-byte unchanged.
• Admin-verb audit log. promote / partition / heal / join / member-remove / reseed each emit one structured record (principal, term, target, outcome) to a tidal_audit tracing target + an optional append-only JSONL file (TIDAL_AUDIT_LOG), on the operator-originated leg only (no double-audit on the forwarded re-apply).
• Per-principal HTTP rate limit. The engine's token-bucket RateLimiter (now re-exported from the crate root) gates all three routers keyed by principal; verified sibling nodes are exempt (replication is never throttled); a deny is 429 + Retry-After. Off by default (TIDAL_RATE_LIMIT_RPS).
• Reference deployment + tooling. k8s/cluster/ gains cert-manager Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region grpc_tls topology block; scripts/gen-cluster-certs.sh (openssl) provisions the same Secret shape without cert-manager. Exit gate verified: foreign pod rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext inter-node links. See docs/planning/milestone-11/phase-7.md.

Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference

• Nodes are cattle; topology is data, not files. grpc_addr is now an advertised address — a hostname or an IP — split from a new optional per-region grpc_bind (the local SocketAddr to bind). A literal grpc_addr binds itself byte-for-byte as before (every existing topology keeps working); a hostname binds 0.0.0.0:<port>. tidal-net's peer map retypes from SocketAddr to String, so Channel::from_shared makes hyper re-resolve DNS on every reconnect — the pod-rescheduled-onto-a-new-IP case the SocketAddr::parse-only constraint made structurally impossible is now fixed by construction. One shared topology file can name every region by its per-pod DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code change). Proven by cluster_membership.rs::mp_dns_hostname_topology_replicates (a hostname topology a pre-p5 binary would have refused to parse boots, replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
• A new node joins via snapshot + stream and serves quorum in minutes. FetchSnapshot is a new server-streaming RPC in the WalShipping service: the leader stages TidalDb::create_backup under <data_dir>/snapshots/ and streams it term-stamped + term-fenced exactly like StreamSegments (manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to end). The handler answers a no-snapshot-needed header when the WAL can still serve the puller's range and streams the artifact only when it cannot — the needed decision is baseline-aware (from_seqno ≤ stream_baseline OR < earliest WAL seq), because a "the WAL still covers it" answer is a lie at or below a baseline the stream clamp will never serve, and looped marker boots forever. Install is a boot-time operation (never a live data-dir swap): fetch into a sibling staging dir, copy the node's own identity files (election_state, membership) INTO staging, write a COMPLETE sentinel, then rename — every crash window is an idempotent redo. A retention pin taken at backup start keeps the segment chain covering the artifact alive while any joiner streams, with a hard cap + tidaldb_cluster_snapshot_pin_force_drops_total so a dead joiner can't freeze compaction forever.
• Reseed is self-healing — no operator verb, no wipe_data_dir. A running follower that hits a typed snapshot-required refusal (the StreamSegments trailer is now structured — x-tidal-catchup: snapshot-required | rejoin | stepping-down — so an ordinary election term-mismatch never latches reseed markers fleet-wide; the marker latches only on snapshot-required), or the m11p4 divergence quarantine, durably latches reseed_required (ElectionStore file discipline), surfaces it in /cluster/status/local and the tidaldb_cluster_reseed_required gauge, and keeps serving degraded with voting enabled (a reseed needs a leader, and the leader may need this node's vote — reseed-blocks-voting would deadlock the cluster). The reseed runs on the next boot; replication.reseed_self_restart: true (default false; k8s sets it true) drains and clean-exits once the marker latches — refused when the remaining voters can't sustain quorum without this node. A successful reseed boot clears the quarantine latch and tidaldb_cluster_divergence_quarantined — closing the m11p4 carried hazard that quarantine clearing "rides m11p5's reseed machinery." cluster_reseed.rs proves the full quarantine → marker → restart → converged → gauges-cleared loop with no wipe_data_dir in the test.
• The three-way term-join rule closes p4's last carried hazard (pre-baseline history was unreachable through a term's stream for followers behind at the transfer). The first reseed drill exposed that such a follower jumps its frontier past pre-baseline history with lag=0 and never asks for the gap — silently missing data with nothing to fire the refusal. The heartbeat now carries the leader's election-time position in the previous stream's numbering (prev_log), and the join check is three-way: own > prev_log → divergent suffix → quarantine (p4); own < prev_log → genuinely missing committed-era history → latch reseed_required (new); own == prev_log (or within-term rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL is the leader's copy, so tail_term equals the leader's term).
• Membership is data on the one log: kind-4 records, learner → voter. A new MembershipRecord WAL blob kind (kind-4, beside kind-0 signals, kind-1/2 item/embedding blobs, kind-3 term markers) is journaled by the leader, replicated through the normal stream, and folded by followers into a ClusterMembership cell (the WalTermMark pattern). Records carry the FULL roster (latest record wins; no merge logic); membership epoch 0 = the topology file, so a cluster with no kind-4 record behaves byte-for-byte as today. POST /cluster/join {name, grpc_addr, http_addr} on any node forwards to the leader, which assigns id = max(all ids ever) + 1, appends a Learner record, and answers only after it is quorum-committed (idempotent by name); member ids are permanent — removal tombstones (a Removed record) keep ids burned, never renumbered or reused. Auto-promotion is a standing leader duty (re-armed on every activation and apply, evaluated on commit-index publishes, driven solely from the applied ClusterMembership — it survives the joining-era leader's death) that promotes a learner within replication.learner_promote_lag (default 1024) of the flushed frontier, or the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server (one at a time, each same-term-quorum-commit-gated); on every leadership activation the new leader re-appends its full current membership immediately after the kind-3 term marker (the Raft no-op-entry analogue), which closes both the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs: GET /cluster/members, POST /cluster/members/remove, POST /cluster/reseed.
• The even-voter-count majority() arithmetic was a latent bug — fixed. ElectionConfig::majority() was peers.len()/2 + 1: correct at n=3, wrong for every even voter count (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D} could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with zero RPCs). CommitIndex already used the correct div_ceil form — the two formulas in one subsystem disagreed. Since p5 makes even sizes mandatory transit states (3→4→5→4→3), this landed first, with even-n property tests: majority() = (peers.len() + 1).div_ceil(2) + 1 (true floor(n/2)+1 over the full voter set), spelled to read identically to the commit-index site so they can never drift again.
• Learner marks never count toward quorum. A role-blind CommitIndex that saw learner marks would treat two learners "committing" a write no voter holds as acked — acked-write loss. CommitIndex is now role-aware: voter marks feed the k-th-largest selection; learner marks live in a side map (promotion input, transfer-wait input) and are excluded from needed. In-flight ack=quorum waits are re-evaluated against a new config on every conf-change (a shrink may satisfy waiters instantly), never failed; a report from an unknown id logs at WARN with a counter instead of being silently dropped. ElectionState, ShipQueue, and PeerPool reconfigure behind one fenced apply path so the four peer-set copies can never disagree about the roster.
• Seed-join boot. tidal-server cluster --region <name> --seed http://host:port (repeatable) --advertise-grpc host:port --advertise-http host:port --metrics addr: the joiner skips the "every region declared" topology gate, learns its roster + assigned id + current term from any reachable seed, persists them to a durable membership cache + election_state (persist-before-act), and installs a snapshot when behind. A restart boots from the cache without the seed. A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule (so a joiner never silently inherits the wrong ack default or election timing).
• Kubernetes reference: one StatefulSet. k8s/cluster/ (namespace tidaldb-cluster, replicas: 3, podManagementPolicy: Parallel, topology-spread, PDB maxUnavailable: 1) with ONE shared bootstrap topology ConfigMap (per-pod DNS advertised addresses, 0.0.0.0 binds). Scaling past 3 does NOT edit it — pod N≥3 boots --seed + the same mounted file and joins as a learner; rolling node replace is kubectl delete pod (PVC retained → boot catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness: converged (boot catch-up completed at least once AND lag ≤ learner_promote_lag — hysteresis, never lag == 0 which an open-loop load keeps perpetually false); a restarted existing voter is Ready on today's terms.
• Exit-gate suites (tier-3, cluster-e2e): cluster_membership.rs (mp_seed_join_snapshot_catchup, mp_scale_3_5_3_under_load_zero_loss — lost=0, max acked seq 1467, p99 impact <2× across the 3→5→3 joins, mp_dns_hostname_topology_replicates) and cluster_reseed.rs (quarantine → marker → restart → converged, gauges cleared). Localhost-loopback catch-up: 5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget); the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line items (k3s access still pending — the standing M11 caveat).
• Mixed-version / downgrade caveats. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → its receiver's torn-state halt latch, permanent across restarts (both followers halted = quorum-write outage). So HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses JoinCluster and every conf-change until all current voters report kind-4 capability — "complete the binary upgrade before the first conf-change" is structurally enforced, not operator discipline. Pre-p5 peers answer Unimplemented to JoinCluster/FetchSnapshot (the joiner reports it loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrade below p5 requires a reseed.

Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5

• "A machine died" is a non-event. Every multi-process cluster node runs a purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced leadership transfer) over the existing tidal-net transport: leader heartbeats every 300ms (config election.heartbeat_interval_ms); a follower that hears no leader for a randomized 1500–3000ms starts a pre-vote; a majority elects. SIGKILL the leader under ack=quorum load → a survivor is elected and writes resume in well under a second locally, with zero acknowledged-write loss across repeated random kill points (tier-3 gate cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss). No raft crate: the WAL is already the replicated log (m11p2) and the quorum commit index (m11p3) already proves majority durability — the election adds only the tiny consensus state (term, leader) on top of them.
• The WAL carries its own election history. An elected leader's FIRST log entry is a kind-3 term-marker record that replicates like any record, so every replica's (lastLogTerm, lastLogIndex) — Raft's vote restriction — is derived from one fsync stream and can never disagree across a crash. The frontier half is compared in the last joined term's stream numbering (a reseeded node's own WAL numbering diverges from the stream's after a baseline jump; comparing raw local frontiers across nodes would let a behind node win). Hard state (current_term, voted_for) persists in data_dir/election_state (magic + version + checksum; corrupt → refuse to boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE any vote reply or leadership claim leaves the node.
• Term fencing everywhere (kills incident §1.4-1). Every replication RPC carries the sender's term: stale-term ships/chunks/heartbeats/frontier reports are rejected, the quorum commit index folds only reports stamped with its activation term (race-free, under the index's own lock), and a restarted ex-leader boots as a FOLLOWER from its durable state — never from the topology file. A partitioned ex-leader that restarts cannot accept a single write (mp_fenced_ex_leader_restart_cannot_write); a leader that loses majority contact steps down within election.leader_lease_ms (default 900; validated lease + heartbeat < election_timeout_min so a deposed leader stops before any successor can exist).
• Divergent suffixes quarantine instead of lying. A node whose log extends past what the elected leadership subsumed (leader-acked, never-quorum-acked writes on a dead leader) detects it at term-join — the heartbeat carries the leader's election-time log position — and fences itself from the data plane (status quarantined: true, metric tidaldb_cluster_divergence_quarantined, ERROR naming the reseed runbook) while still voting. Followers apply on receipt, so un-applying is not a thing: reseed is the honest recovery (m11p5 snapshots automate it).
• /cluster/promote is now a fenced transfer: with a live leader it drains (waits for the target to hold the flushed prefix, breaker-immune via the commit index's marks) then sanctions an immediate election; with a dead leader the target campaigns. The election refuses a target whose log lags — the m11p3 "promote the max-applied survivor" operator rule is now enforced by the protocol. The legacy term-0 fan-out survives only on clusters that have never elected (mixed-version rollouts; the chaos drills' deliberate isolated-node override) and is permanently retired per node at its first joined election. election.auto_election: false preserves the full pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
• Bounded churn under flapping links: the pre-vote (no term inflation without a majority probe) plus the leader-freshness lease absorb short flaps entirely and bound terms under long ones (mp_flapping_links_bounded_churn). Election observability: tidaldb_cluster_election_term/_role/_elections_started_total/ _leader_changes_total, and /cluster/status/local grows term, role, quarantined, prev_log_term/seq.
• New tier-3 exit-gate suite cluster_election.rs (auto-failover ledger, fencing under partition+restart, bounded churn); cluster_quorum.rs and cluster_lifecycle.rs pin auto_election: false (they validate the manual drill, which remains supported); the partition harness gained bidirectional isolation (isolate_region) — severing only a node's inbound edges leaves its outbound heartbeats/votes flowing, which silently defeats partition scenarios.

Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, TSEG segment header, structured "snapshot required"

• Failed catch-up pulls retry on a timer. The pull trigger was event-only: a follower whose StreamSegments pull failed (e.g. the leader's gRPC server not yet ready during a rolling restart) waited for the next PUSHED segment to re-expose the gap — in an idle cluster that push never comes, and the follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers stuck at lag=136507). A failed pull now arms a one-shot timer (replication.catchup_retry_ms, default 30000; transport catchup_retry_interval) that re-pulls from the CURRENT applied frontier. Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when consumed by the rate limit or an in-flight pull, so the gap always keeps a standing wake-up until a pull completes. Verified over real sockets: catchup_retry.rs reproduces the incident (pull fails, leader appears, zero pushes) and proves timer-only self-heal — and that a clean completion arms nothing.
• WAL segment files are format-versioned. New segments open with an 8-byte header (TSEG magic + version byte + reserved); pre-m11p4 headerless segments stay readable as implicit version 0 — no migration. A segment this binary cannot identify (unknown header version, unrecognized leading bytes, unparseable .seg filename) surfaces as the new WalError::SegmentFormatUnknown at open — previously it scanned as empty (segments=0) and recovery's torn-tail repair could TRUNCATE the foreign file to zero. Foreign-format files are never repaired, truncated, or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
• Unservable catch-up is a structured refusal. SegmentSource::collect_from returns typed SegmentReadError::{Unavailable,Failed}; TidalDb::read_wal_batches returns the typed WalError (was stringified TidalError). The StreamSegments handler maps Unavailable to FAILED_PRECONDITION — "segments not available from seq N; snapshot required" — and the follower logs it distinctly (catch-up unservable … needs a snapshot (m11p5) or an operator reseed) instead of burying it as a transient. The on-disk segment format is now documented (tidal/src/wal/segment.rs module docs + spec 01 §2.2).

Quorum-acked writes (m11p3) — ack=leader|quorum, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)

• ack=quorum is an opt-in durability contract for every replicated write (/signals, /items, /embeddings): success means a majority of the replica set durably holds the write (leader + floor(n/2) followers, each storage-applied and own-WAL-fsynced), so an acked write survives the permanent loss of any single node — including the leader. Deployment default via topology replication.ack; per-request override via the x-tidal-ack header (forwarded verbatim by gateways). ack=leader (the default) is the m0–m11p2 contract unchanged.
• Durable frontier reports. Followers PUSH their durably-applied frontier to the leader once per apply round (new ReportApplied RPC, fired by the segment receiver through the new Transport::notify_applied) — batch-level and fully decoupled from ship acks, so the commit index stays fresh even when outbound ships stall (gap-parked follower, pull-based catch-up, quiet leader). Ship acks keep their m11p2 instant floor-hint semantics — both inputs are durable-true because a follower's frontier only ever advances after its storage apply + own-WAL fsync. (The first design held each ship ack until its segment's apply; measured under open-loop load, that couples ship cadence to apply latency and one gap-parked follower spirals into total quorum collapse — the report push is what shipped.)
• Follower blob applies are group-committed. m11p2 applied replicated items/embeddings one record at a time — one solo follower fsync per item, capping item apply at the fsync floor (~100/s on macOS) and stalling the quorum frontier behind any item burst. The receiver now hands each apply round's blob records to the engine as ONE batch (apply_replicated_blobs: validate all → stage all WAL appends → wait all → upsert storage), and the WAL writer flushes queued blobs under ONE group fsync. Measured: corpus seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
• Commit index. The leader folds durable acks (and heal resumes) into per-peer durable marks; the commit index is the k-th largest (k = floor(n/2)), leadership-scoped (promote resets it to the stream baseline; demotion fails in-flight waiters — a demoted leader never claims quorum). Handlers await it through a watch-channel bridge — fully async, zero threads parked per waiter (the thread-per-wait design measurably collapsed at 1k rps open-loop by exhausting the blocking pool and starving the very completions that advance the index). Quorum capacity measured on a real 3-process localhost cluster (release, writes mix): 3,600 quorum signal-writes/s within SLO (p50 ~45ms, zero errors, replication lag ≤3 events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack figure, vs the ≥50% gate.
• Honest timeout semantics. Quorum not confirmed within replication.quorum_timeout_ms (default 2000) → a retryable 503 naming the laggard regions, the commit index, and needed/confirmed counts. The write is in the leader's log and may still commit: retries are at-least-once (items/embeddings retries are idempotent upserts; signal retries can double-count — decided in-phase: no idempotency-key machinery, documented in runbook §8 with the session-write precedent for callers that need exact-once).
• x-tidal-seq on every cluster write response: the write's seqno in the replicated log (relayed through forwards) — an exact durability cursor against commit_index in /cluster/status/local (which also gains ack). tidaldb_cluster_relay_durable_seq now reports the commit index (relay_last_seq − relay_durable_seq = quorum lag); new counter tidaldb_cluster_quorum_timeouts_total.
• The ledger gate (exit gate, run for real): tier-3 mp_quorum_ledger_zero_acked_loss_across_killpoints SIGKILLs the leader under concurrent quorum load and proves zero acknowledged loss on the promoted max-applied survivor — frontier invariant (max acked seq ≤ survivor applied) plus per-item content probes. 167/167 kill points passed on the final design (batches of 64 + 91 + 12; kill timings spread 120–598ms across fresh 3-process clusters; an earlier 100/100 run had validated the superseded ack-holding design before it was replaced — see docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics (one follower down: quorum commits; both down: fast 503 naming laggards while ack=leader flows; heal: recovers) and in-process gRPC coverage (override headers, forwarded quorum writes, blob writes, 400 on bad mode).
• Rolling-upgrade order (mixed-version caveat): a pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack — it silently applies LEADER-ack semantics to a quorum request (a durability downgrade the caller cannot see). Upgrade the leader first: ack=quorum is then honored immediately (commit-index freshness rides the m11p2 ship-ack floor hints until the followers upgrade too). Replication and heal are unaffected by either order. Runbook §8 records the procedure.

One replicated log (m11p2) — items/embeddings ride the WAL, StreamSegments catch-up, HTTP broadcast deleted

• The leader's WAL is now THE replicated log. The group-commit writer hands every fsynced batch to a bounded in-memory ship feed (wal::feed::WalShipFeed); the ship queue pushes those already-encoded bytes verbatim — byte-identical on the leader's disk, the wire, and the follower's apply path — and stream seqnos are WAL seqnos, so they survive restarts (the m8p10 relay-reset hazard is gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning machinery left the server write path entirely (/signals stages straight through the engine's group commit; a WAL fsync failure now surfaces per-write exactly like single-node).
• Item metadata and embeddings are replicated mutations. /items and /embeddings journal kind-1/2 blob records (WAL header flags byte = batch kind; one record, one seqno) BEFORE storage, on the same stream as signals. Followers apply them kind-aware — WAL-first into their own log, then idempotent storage upserts — and recovery replays them. The m8p10 HTTP item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast, authed side-POSTs — the source of both 2026-06-10 live bugs) is deleted; those bug classes are now impossible by construction. Cluster /items returns a plain 201 and /embeddings a plain 204 (no broadcast-report body).
• StreamSegments implemented — catch-up is follower-pulled. The ship feed's tail is bounded; a peer that falls behind it is skipped ahead, and the follower pulls the hole itself via the (previously declared-unimplemented) server-streaming RPC over the leader's durable, BLAKE3-verified segments. Pulls trigger on detected gaps, on follower boot (self-driving restart catch-up), and on the leader's heal nudge (POST /cluster/catchup, internal, forwards the operator's own bearer credential). Pulled chunks flow through the same inbound apply path as live ships. Ship acks piggyback the follower's applied seqno (ShipSegmentResponse.applied_seqno), so retries of already-applied data prune and heal is resume_from + nudge — no redelivery scan, no O(items) traffic.
• Promote carries a stream baseline. A promoted leader's stream starts at its promote-time flushed frontier (persisted in data_dir/stream_baseline); the fan-out body and every catch-up chunk announce it, so peers jump their frontier past pre-stream history instead of parking on a phantom gap. Ship queues are leadership-gated (activate_from/deactivate): a follower's replicated applies never echo back at its peers.
• Multi-process cluster mode now requires --data-dir (validated at startup): the durable WAL is the replication stream. Standalone single-node deployments are untouched — blob journaling and the ship feed are gated on cluster peers, so single-node item writes keep fjall-only durability with zero extra fsyncs.
• New tier-3 suite mp_items_ride_the_log_and_catchup_stream: items written on the leader AND through a follower gateway converge everywhere via the log (feed parity 1e-6), and a follower stopped through item+embedding+signal writes restarts and converges via its boot-time StreamSegments pull with no heal verb and no HTTP item traffic.

Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first tidaldb_cluster_* metrics

• The replicated /signals write path no longer serializes every writer onto a solo group-commit fsync nor ships to followers on the request path. Writes are staged (seqno + WAL submission + relay log push, microseconds, atomic with rollback) and completed (shared group-commit fsync + in-memory fold) in two phases, so concurrent writers coalesce into one fsync; follower shipping moved to per-peer sender threads that coalesce contiguous runs into multi-event batches with a windowed in-flight budget (topology knobs replication.{batch_max_events,window,retry_ms}). The 204 contract is unchanged (leader durability only) — it now returns at leader fsync. Measured on a real 3-process localhost cluster (release build, thepeach mix): 4,534 replicated signal-writes/s sustained within SLO vs ~90/s before (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
• Durable-frontier shipping + relay poisoning. Senders only ship the leader's contiguous fsynced prefix (an event a follower holds but the leader could lose is silent divergence); a staged write whose fsync fails poisons the relay — further cluster writes are rejected and the ship frontier freezes (the CockroachDB/Postgres fsync-failure posture).
• Follower group-commit coalescing. The segment receiver drains its inbound backlog (Transport::try_recv_segment) and applies it through ONE shared group commit (SignalLedger::apply_replicated_events), in range-disjoint groups so duplicate/subset re-ships cannot double-fold. Without this the follower apply ceiling was ~events-per-segment / fsync-cost (~1.8k events/s measured) and lag grew without bound under m11p1 leader rates.
• First cluster metrics + cluster /metrics listener. Cluster mode previously had no metrics endpoint at all. New per-region topology metrics_addr wires the engine's Prometheus listener; new tidaldb_cluster_* series: ship RTT + batch-size histograms, per-peer ship counters/gauges (peer_shard labels), WAL fsync latency + group-commit fill histograms, write-pool depth/rejections, relay committed + durable frontiers. WAL group-commit knobs are deployment config (wal.{batch_size,batch_timeout_ms} topology block; wal_batch_size/wal_batch_timeout builder methods).
• Engine API: TidalDb::signal_staged/StagedSignal::wait, SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}, ShipQueue (per-peer windowed batch senders with pause/resume), WalWriter::append_signal_staged, WalSender::append_record_staged, range_payload/encode_run. Relay log entries are now RelayEvent (raw event records, re-encoded deterministically at ship time) instead of pre-encoded single-event bytes.
• Ship-sender failure logging is transition-based (first + every 50th consecutive failure WARN with the running count, recovery INFO) — the per-retry WARN flood could fill an undrained log pipe and stall the process. The multiproc test harness now discards child logs via /dev/null (a piped fd nobody drains deadlocks the node once the kernel buffer fills), with TIDAL_TEST_NODE_LOGS=inherit to stream them while debugging.

M9 — Community Sync & Revocation

• Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: SignalScope, CommunityId, Membership, MembershipEpoch, PolicyMetadata. Community signal reconciliation via CrdtSignalState with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

M10 — Governance & Agent Rights

• Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE

• Multi-process cluster mode: tidal-server cluster --region <name> [--data-dir <p>] (env TIDAL_REGION) runs one process per region (RegionClusterState), each owning one TidalDb and one GrpcTransport that binds this region's grpc_addr and dials every sibling's real grpc_addr — real process/host isolation. The topology requires per-region grpc_addr AND http_addr in this mode; single-process mode (no --region) is unchanged as the dev/demo default. Both modes stay behind the experimental gate (--experimental-cluster / TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) with mode-specific WARN text.
• New routes / behaviors on the multi-process surface: GET /cluster/status/local (per-node status), GET /cluster/status aggregates ALL regions with a reachable field (unreachable ⇒ reachable:false + worst-case lag), POST /cluster/reconcile {region} (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact no-op on scores), POST /hardnegs {user_id,item_id} (user-scoped hide; converges via reconcile, filtered from /feed?user_id). Writes on a non-leader forward to the leader; ?region= reads forward to the owning region; default reads are LOCAL in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast to peers with a per-peer report: POST /items → 201 {replicated_to, failed}, POST /embeddings → 200 with the same report on the leader path (NOT 204 — a 204 cannot carry the report; the forwarded/internal path stays 204). /sharded/* fans out across processes (degraded semantics preserved). promote fans out to all peers ({ok, leader, acked, failed}).
• /cluster/heal is the single recovery verb: redelivers missed signal segments (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold 5, reset 30s) is open, so re-issue /cluster/heal until /cluster/status shows lag 0.
• TIDAL_HLC_SKEW_MS (multi-process): signed ms offset applied to the process's HLC (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
• Tier-3 UAT over real OS processes (feature cluster-e2e): cluster_multiproc (5), cluster_chaos (3, REAL network-partition injection via a root-free in-harness TCP relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl remain an operator option), cluster_lifecycle (2, ±500ms clock skew + rolling upgrade with zero acknowledged-write loss), cluster_runbook (9, every docs/runbooks/cluster.md §5–§11 operation), plus cluster_e2e (2, single-process smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

Fixed

Four production bugs surfaced and fixed by the m8p10 tier-3 UAT

• Silent data loss on out-of-order ships. The replication applied high-water-mark swallowed sequence gaps when eager ships arrived out of order. The applied frontier is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
• Non-idempotent reconcile (score creep). take_crdt_snapshot attributed replicated signal streams per-node, so reconciling already-converged nodes crept the decayed scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical replication shard (ShardId::SINGLE), making merge idempotent — reconcile of converged nodes is an exact fixpoint.
• Lag gauge conflated leader streams across a promotion. A converged node reported a permanent phantom lag after a /cluster/promote moved leadership to a different shard. The lag gauge now tracks the leader high-water-mark per source shard (ReplicationLagGauge::leader_seqno_for), computing lag against the current leader.
• Items missed during a broadcast were never backfilled. A node down/partitioned during an item/embedding broadcast was permanently missing that data even at signal lag 0. /cluster/heal now re-broadcasts item metadata + embeddings to the healed region (idempotent upserts), making heal the single recovery verb.

Changed

Cluster mode (m8p8): real gRPC replication

• tidal-server's ClusterState now wires each follower region to a real tidal-net GrpcTransport (self-loop over loopback gRPC) instead of in-process crossbeam channels — replication between regions traverses real gRPC/TCP (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
• Follower gRPC ports are auto-allocated from the topology (grpc_addr optional per region) and self-heal a transient bind race by retrying on a fresh port.
• Blocking gRPC ships triggered by POST /signals and POST /cluster/heal are offloaded to a dedicated thread so they never block the axum reactor.
• ClusterState is constructed off the async reactor (GrpcTransport::new blocks on its own runtime).
• The experimental opt-in gate and docker/cluster/Dockerfile are updated: single-process cluster mode is honest that it replicates over real gRPC but runs all regions in one process (no host/process isolation). (True multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry above.)

Docker build fixes (all three images now that tidal-server pulls tidal-net)

• Install protobuf-compiler in the builder stage — tidal-net's build script runs tonic-build, which needs protoc to compile the WAL-shipping .proto.
• Pin the builder base to bookworm (rust:1.91-bookworm / rust:1.91-slim-bookworm) so its glibc matches the debian:bookworm-slim runtime; the default trixie base emitted a libmvec.so.1 dependency absent on bookworm, aborting the binary at startup. Verified: docker run of the cluster image serves a functional 3-region cluster (write replicates to both followers over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
• New tests: tidal-server/tests/cluster_grpc.rs (in-process gRPC replication + HTTP offload path) and a hardened tier-3 cluster_e2e.rs (multi-process smoke
  • promote over real OS processes, feature-gated).

[0.1.0] - 2026-02-23

Added

Core Database Engine

• TidalDb embeddable database with ephemeral() and with_data_dir() open modes
• SchemaBuilder for defining signal types, decay parameters, and ranking profiles
• TidalDbBuilder fluent builder with schema, data directory, metrics, and rate limiter configuration

Signal System

• Typed signal recording with exponential decay scoring
• Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
• Windowed aggregation: OneHour, TwentyFourHours, SevenDays, AllTime
• Signal velocity tracking
• WAL-backed signal durability with crash recovery
• Periodic signal checkpointing to fjall (every 30s)
• WAL compaction after each checkpoint

Retrieval (RETRIEVE query)

• 5-stage pipeline: universe, filter, score, diversify, return
• Filter expressions: Eq, In, Gt, Lt, And, Or, Not, InCollection, InProgress, MinSignal, MaxSignal, NearLocation
• Built-in ranking profiles: trending, for_you, new, popular, recent, and 20+ more
• Custom ranking profiles via SchemaBuilder
• Diversity enforcement (max N per category/creator)
• Sort modes: Relevance, Trending, Newest, MostLiked, MostViewed, MostFollowed, AlphabeticalAsc/Desc, Shortest/Longest, LiveViewerCount, DateSaved, and more

Search (SEARCH query)

• BM25 full-text search via Tantivy
• Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
• Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
• Creator search with entity_kind(EntityKind::Creator)
• similar_to(EntityId) for content-based recommendations
• Scope pre-filters: Trending, CohortTrending, Following, Category, Collection
• Autocomplete suggestions via db.suggest()

Entity Model

• Three built-in entity types: Item, User, Creator
• Metadata storage as HashMap<String, String>
• Embedding slots (up to 4 per entity type) via USearch
• Relationships: Follows, Blocks, Hide, Mute, InteractionWeight

Sessions

• Session lifecycle: open_session, close_session
• Cross-session preference vector updates (EMA blend)
• Session snapshots with signal state and preference vectors
• Session serialization format v0x03 with backward compatibility

Social Graph

• Creator follower/following indexes
• Cohort membership (user segments)
• CoEngagementIndex for co-viewing patterns with LRU eviction
• Social graph filter for "followed creator" content scoping

Collections

• Named collections with Private, Shared, Public visibility
• create_collection, add_to_collection, remove_from_collection, list_collections
• FilterExpr::InCollection for collection-scoped retrieval
• Saved searches with save_search, list_saved_searches, retrieve_saved_search

Observability

• enable_metrics(addr) -- Prometheus-format /metrics endpoint + /healthz JSON
• 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
• tidaldb_checkpoint_failures_total counter for checkpoint monitoring
• TidalDb::diagnostics() -- structured health snapshot
• WAL diagnostics and recovery tools

Safety

• Signal weight NaN/Inf validation (returns TidalError::InvalidInput)
• Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
• Export request limit: 500K signals max per request
• FilterExpr complexity limit: 256 nodes max
• Data directory lock (tidaldb.lock) prevents dual-process corruption
• Schema fingerprint persistence detects decay parameter changes on reopen
• Bounded closed_sessions cache (10K max, LRU eviction)
• Metrics server non-loopback bind warning

CLI (tidalctl)

• tidalctl binary for database inspection and diagnostics

RLHF / ML Export

• db.export_signals(ExportRequest) -- WAL-based signal export for training data
• db.user_session_summary(user_id, since_ns) -- aggregated session statistics

Stability

tidalDB 0.1.0 is pre-1.0. No API or data format stability guarantees are made for 0.x releases. Upgrade guides will be provided for each minor version bump. Do not upgrade 0.x to 0.y on a live data directory without reading the release notes.

---

Format based on Keep a Changelog