tidaldb/CHANGELOG.md

Changelog

All notable changes to tidalDB will be documented in this file.

[Unreleased]

Added

Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix

• Fault injection compiled out of production. A non-default fault-injection cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks (tidal/src/fault.rs): slow-fsync (TIDAL_FAULT_FSYNC_DELAY_MS sleeps before each durable fsync) and disk-full (TIDAL_FAULT_DISK_FULL_AFTER_BYTES returns a real ENOSPC/errno-28 once cumulative segment bytes cross the threshold). The production image build never passes the feature, so the hooks are compiled out entirely — a disk/fsync fault a stray env var could trip in prod is a 3am footgun we refuse to ship; the safety is structural. Inert until armed even when compiled in. The tier-3 harness builds the spawned binary with the feature.
• First-class invariant checkers (tidal-server/tests/support/invariants.rs): the no-acked-loss AckLedger (frontier + content, extracted from the m11p3 ledger gate, which now consumes it), cross-replica assert_feed_parity / feed_item_ids, assert_single_leader_per_term (membership safety, reads the m11p6 shards[] rows), and MonotonicCounters (per-node applied frontier + leader commit index never regress, with legitimate epoch resets forgiven).
• New fault-class suite (tidal-server/tests/cluster_faults.rs, tier-3): a disk-full follower degrades gracefully (receiver halts, node alive, zero acked loss, restart recovers to parity); a slow-fsync follower lags then converges while the fast follower supplies quorum; both followers slow → ack=quorum returns a retryable 503 naming the laggards while ack=leader is unaffected; an asymmetric partition (inbound severed, outbound up) causes no split brain (pre-vote + check-quorum) and no loss. 4/4 green.
• Soak + regression gates in tidal-stress: --json-summary <path> (a machine-readable per-stage p99/throughput/error roll-up for trend lines) and --max-p99-ms / --max-error-pct / --fail-on-knee gates that exit non-zero on a regression (the tool always exited 0 before).
• Nightly pipeline (.woodpecker.yaml): a cron nightly flow (chaos suites with elevated kill-points + the fault-injection feature, then the gated tidal-stress soak) beside the existing push release gate, event-routed by per-step when. Woodpecker, never GitHub Actions.
• Guarantee traceability (docs/planning/milestone-11/guarantee-traceability.md): every roadmap §2 guarantee mapped to its named automated test(s). Closes the G-C apparatus; the 30-consecutive-days-green half of the GA bar is a calendar criterion the nightly pipeline accrues.

Observability + operations (m11p8) — complete cluster metric set on a per-node /metrics listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate

• Metrics. Completed the tidaldb_cluster_* set with the two members the roadmap named that were missing — breaker (tidaldb_cluster_peer_breaker_state 0/1/2 + tidaldb_cluster_breaker_opens_total, surfaced read-only from the tidal-net circuit breaker via a new Transport::peer_breaker_state) and forwards (tidaldb_cluster_forwards_total / _forward_failures_total, instrumented at the gateway forward path) — plus the self-heal series (heal_attempts/successes/noops_total, healing_peers). When several shard groups co-locate on one node (m11p6) the metrics-owner serves the single /metrics listener and the siblings register their series under a shard="N" label (TidalDb::register_metrics_sibling); a single-shard node is byte-identical to before. A 12-panel "Cluster Replication" Grafana row + an 8-rule tidaldb-cluster Prometheus alert group ship beside the standalone ones.
• Request-id + tracing. Both cluster routers (single- and multi-process) now carry the standalone router's SetRequestId + PropagateRequestId + TraceLayer stack (extracted to router::with_request_id_tracing); the id rides the follower→leader forward hop verbatim, so the leader's span shares the gateway's x-request-id. (Ships are off-request-path and batched, so they correlate by seqno, not a request-id — by design.)
• Truthful status. Fixed the leader's own applied_events reading 0 (a leader writes its WAL directly and never advances its own applied frontier — now reports its flushed frontier) and the single-process post-promote ShardId(0) lag-keying undercount (now keys on the current leader's shard).
• Self-driving heal. A standing leader duty re-arms the backlog re-ship for a stuck (breaker-open, behind) peer every ~3s, so it converges through breaker resets with no operator /cluster/heal loop — closing the §1.4-3 footgun. Observable via healing_peers + the TidalDBClusterHealNotConverging alert.
• WAL PITR archival. wal.archive_dir (topology + builder): the online compaction copies each sealed segment to the archive — durably, before deletion, refusing to delete if archival fails — so the archive is a gap-free PITR record.
• Backup/restore. tidalctl backup / restore: offline data-dir backup with a BLAKE3 BACKUP_MANIFEST.json (+ the WAL checkpoint cursor); restore verifies every file's hash before writing and refuses a non-empty target. Coordinated cluster backup = back up one committed replica per shard group (the runbook drill).
• Rolling upgrade. A wire version handshake (HeartbeatRequest.build_version, stamped at the transport boundary; >= 2-major skew WARNs, never rejects) + version on /cluster/status. mp_rolling_upgrade_no_loss_no_stall promoted to the FIRST step of .woodpecker.yaml (the release gate; a failure blocks the image build). See milestone-11/phase-8.md.

Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit

• The cluster stops trusting the network. gRPC replication mTLS is now the intended posture: the inbound server is served over a custom tokio-rustls acceptor (not tonic's fixed .tls_config()) fed a DynamicCertResolver (ArcSwap<CertifiedKey>), preserving mutual TLS exactly (a WebPkiClientVerifier over the cluster CA — a foreign/absent client cert fails the handshake before any RPC). Plaintext is an explicit insecure: true with a loud startup WARN.
• Cert + bearer rotation WITHOUT restart. A content-hash poller re-reads the cert files (k8s ..data symlink swaps that inotify misses) and atomically swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys, so a rotation drops zero requests (verified under concurrent load). Outbound peer channels rebuild from the refreshed files. The bearer and a new shared cluster key live behind ArcSwap, read per request and reloaded from TIDAL_API_KEY_FILE / TIDAL_CLUSTER_KEY_FILE.
• Authenticated inter-node HTTP with per-node identity. The axum listener serves TLS (reusing the same hot-swappable resolver — one rotation covers both planes); forwards/broadcasts/scatter/status/seed-join dial https:// with the cluster CA. A forwarding node mints an x-tidal-node-token (keyed-BLAKE3 MAC over node-id + expiry under the cluster key — no new crypto dependency) so a foreign pod cannot forge a sibling identity. The x-tidal-internal marker is now honored ONLY from a verified sibling (marker without a valid node token → 403): the marker stays a routing hint, never an authorization bypass. All opt-in via grpc_tls / the cluster key — absent ⇒ pre-m11p7 behavior, so every existing deployment and test is byte-for-byte unchanged.
• Admin-verb audit log. promote / partition / heal / join / member-remove / reseed each emit one structured record (principal, term, target, outcome) to a tidal_audit tracing target + an optional append-only JSONL file (TIDAL_AUDIT_LOG), on the operator-originated leg only (no double-audit on the forwarded re-apply).
• Per-principal HTTP rate limit. The engine's token-bucket RateLimiter (now re-exported from the crate root) gates all three routers keyed by principal; verified sibling nodes are exempt (replication is never throttled); a deny is 429 + Retry-After. Off by default (TIDAL_RATE_LIMIT_RPS).
• Reference deployment + tooling. k8s/cluster/ gains cert-manager Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region grpc_tls topology block; scripts/gen-cluster-certs.sh (openssl) provisions the same Secret shape without cert-manager. Exit gate verified: foreign pod rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext inter-node links. See docs/planning/milestone-11/phase-7.md.

Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference

• Nodes are cattle; topology is data, not files. grpc_addr is now an advertised address — a hostname or an IP — split from a new optional per-region grpc_bind (the local SocketAddr to bind). A literal grpc_addr binds itself byte-for-byte as before (every existing topology keeps working); a hostname binds 0.0.0.0:<port>. tidal-net's peer map retypes from SocketAddr to String, so Channel::from_shared makes hyper re-resolve DNS on every reconnect — the pod-rescheduled-onto-a-new-IP case the SocketAddr::parse-only constraint made structurally impossible is now fixed by construction. One shared topology file can name every region by its per-pod DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code change). Proven by cluster_membership.rs::mp_dns_hostname_topology_replicates (a hostname topology a pre-p5 binary would have refused to parse boots, replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
• A new node joins via snapshot + stream and serves quorum in minutes. FetchSnapshot is a new server-streaming RPC in the WalShipping service: the leader stages TidalDb::create_backup under <data_dir>/snapshots/ and streams it term-stamped + term-fenced exactly like StreamSegments (manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to end). The handler answers a no-snapshot-needed header when the WAL can still serve the puller's range and streams the artifact only when it cannot — the needed decision is baseline-aware (from_seqno ≤ stream_baseline OR < earliest WAL seq), because a "the WAL still covers it" answer is a lie at or below a baseline the stream clamp will never serve, and looped marker boots forever. Install is a boot-time operation (never a live data-dir swap): fetch into a sibling staging dir, copy the node's own identity files (election_state, membership) INTO staging, write a COMPLETE sentinel, then rename — every crash window is an idempotent redo. A retention pin taken at backup start keeps the segment chain covering the artifact alive while any joiner streams, with a hard cap + tidaldb_cluster_snapshot_pin_force_drops_total so a dead joiner can't freeze compaction forever.
• Reseed is self-healing — no operator verb, no wipe_data_dir. A running follower that hits a typed snapshot-required refusal (the StreamSegments trailer is now structured — x-tidal-catchup: snapshot-required | rejoin | stepping-down — so an ordinary election term-mismatch never latches reseed markers fleet-wide; the marker latches only on snapshot-required), or the m11p4 divergence quarantine, durably latches reseed_required (ElectionStore file discipline), surfaces it in /cluster/status/local and the tidaldb_cluster_reseed_required gauge, and keeps serving degraded with voting enabled (a reseed needs a leader, and the leader may need this node's vote — reseed-blocks-voting would deadlock the cluster). The reseed runs on the next boot; replication.reseed_self_restart: true (default false; k8s sets it true) drains and clean-exits once the marker latches — refused when the remaining voters can't sustain quorum without this node. A successful reseed boot clears the quarantine latch and tidaldb_cluster_divergence_quarantined — closing the m11p4 carried hazard that quarantine clearing "rides m11p5's reseed machinery." cluster_reseed.rs proves the full quarantine → marker → restart → converged → gauges-cleared loop with no wipe_data_dir in the test.
• The three-way term-join rule closes p4's last carried hazard (pre-baseline history was unreachable through a term's stream for followers behind at the transfer). The first reseed drill exposed that such a follower jumps its frontier past pre-baseline history with lag=0 and never asks for the gap — silently missing data with nothing to fire the refusal. The heartbeat now carries the leader's election-time position in the previous stream's numbering (prev_log), and the join check is three-way: own > prev_log → divergent suffix → quarantine (p4); own < prev_log → genuinely missing committed-era history → latch reseed_required (new); own == prev_log (or within-term rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL is the leader's copy, so tail_term equals the leader's term).
• Membership is data on the one log: kind-4 records, learner → voter. A new MembershipRecord WAL blob kind (kind-4, beside kind-0 signals, kind-1/2 item/embedding blobs, kind-3 term markers) is journaled by the leader, replicated through the normal stream, and folded by followers into a ClusterMembership cell (the WalTermMark pattern). Records carry the FULL roster (latest record wins; no merge logic); membership epoch 0 = the topology file, so a cluster with no kind-4 record behaves byte-for-byte as today. POST /cluster/join {name, grpc_addr, http_addr} on any node forwards to the leader, which assigns id = max(all ids ever) + 1, appends a Learner record, and answers only after it is quorum-committed (idempotent by name); member ids are permanent — removal tombstones (a Removed record) keep ids burned, never renumbered or reused. Auto-promotion is a standing leader duty (re-armed on every activation and apply, evaluated on commit-index publishes, driven solely from the applied ClusterMembership — it survives the joining-era leader's death) that promotes a learner within replication.learner_promote_lag (default 1024) of the flushed frontier, or the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server (one at a time, each same-term-quorum-commit-gated); on every leadership activation the new leader re-appends its full current membership immediately after the kind-3 term marker (the Raft no-op-entry analogue), which closes both the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs: GET /cluster/members, POST /cluster/members/remove, POST /cluster/reseed.
• The even-voter-count majority() arithmetic was a latent bug — fixed. ElectionConfig::majority() was peers.len()/2 + 1: correct at n=3, wrong for every even voter count (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D} could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with zero RPCs). CommitIndex already used the correct div_ceil form — the two formulas in one subsystem disagreed. Since p5 makes even sizes mandatory transit states (3→4→5→4→3), this landed first, with even-n property tests: majority() = (peers.len() + 1).div_ceil(2) + 1 (true floor(n/2)+1 over the full voter set), spelled to read identically to the commit-index site so they can never drift again.
• Learner marks never count toward quorum. A role-blind CommitIndex that saw learner marks would treat two learners "committing" a write no voter holds as acked — acked-write loss. CommitIndex is now role-aware: voter marks feed the k-th-largest selection; learner marks live in a side map (promotion input, transfer-wait input) and are excluded from needed. In-flight ack=quorum waits are re-evaluated against a new config on every conf-change (a shrink may satisfy waiters instantly), never failed; a report from an unknown id logs at WARN with a counter instead of being silently dropped. ElectionState, ShipQueue, and PeerPool reconfigure behind one fenced apply path so the four peer-set copies can never disagree about the roster.
• Seed-join boot. tidal-server cluster --region <name> --seed http://host:port (repeatable) --advertise-grpc host:port --advertise-http host:port --metrics addr: the joiner skips the "every region declared" topology gate, learns its roster + assigned id + current term from any reachable seed, persists them to a durable membership cache + election_state (persist-before-act), and installs a snapshot when behind. A restart boots from the cache without the seed. A --seed boot still requires the local topology/config file for the behavioral knob blocks (replication:, wal:, election:, timeouts:, grpc_tls) — a bare --seed with neither --topology nor TIDAL_CONFIG refuses to boot naming the rule (so a joiner never silently inherits the wrong ack default or election timing).
• Kubernetes reference: one StatefulSet. k8s/cluster/ (namespace tidaldb-cluster, replicas: 3, podManagementPolicy: Parallel, topology-spread, PDB maxUnavailable: 1) with ONE shared bootstrap topology ConfigMap (per-pod DNS advertised addresses, 0.0.0.0 binds). Scaling past 3 does NOT edit it — pod N≥3 boots --seed + the same mounted file and joins as a learner; rolling node replace is kubectl delete pod (PVC retained → boot catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness: converged (boot catch-up completed at least once AND lag ≤ learner_promote_lag — hysteresis, never lag == 0 which an open-loop load keeps perpetually false); a restarted existing voter is Ready on today's terms.
• Exit-gate suites (tier-3, cluster-e2e): cluster_membership.rs (mp_seed_join_snapshot_catchup, mp_scale_3_5_3_under_load_zero_loss — lost=0, max acked seq 1467, p99 impact <2× across the 3→5→3 joins, mp_dns_hostname_topology_replicates) and cluster_reseed.rs (quarantine → marker → restart → converged, gauges cleared). Localhost-loopback catch-up: 5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget); the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line items (k3s access still pending — the standing M11 caveat).
• Mixed-version / downgrade caveats. A kind-4 record shipped to a pre-p5 follower is an unknown batch kind → WalError::Corruption → its receiver's torn-state halt latch, permanent across restarts (both followers halted = quorum-write outage). So HeartbeatResponse/ReportApplied carry a capabilities bit-field (proto3 zero-default = pre-p5 = incapable) and the leader refuses JoinCluster and every conf-change until all current voters report kind-4 capability — "complete the binary upgrade before the first conf-change" is structurally enforced, not operator discipline. Pre-p5 peers answer Unimplemented to JoinCluster/FetchSnapshot (the joiner reports it loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim): once any kind-4 record is in a node's WAL, downgrade below p5 requires a reseed.

Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5

• "A machine died" is a non-event. Every multi-process cluster node runs a purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced leadership transfer) over the existing tidal-net transport: leader heartbeats every 300ms (config election.heartbeat_interval_ms); a follower that hears no leader for a randomized 1500–3000ms starts a pre-vote; a majority elects. SIGKILL the leader under ack=quorum load → a survivor is elected and writes resume in well under a second locally, with zero acknowledged-write loss across repeated random kill points (tier-3 gate cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss). No raft crate: the WAL is already the replicated log (m11p2) and the quorum commit index (m11p3) already proves majority durability — the election adds only the tiny consensus state (term, leader) on top of them.
• The WAL carries its own election history. An elected leader's FIRST log entry is a kind-3 term-marker record that replicates like any record, so every replica's (lastLogTerm, lastLogIndex) — Raft's vote restriction — is derived from one fsync stream and can never disagree across a crash. The frontier half is compared in the last joined term's stream numbering (a reseeded node's own WAL numbering diverges from the stream's after a baseline jump; comparing raw local frontiers across nodes would let a behind node win). Hard state (current_term, voted_for) persists in data_dir/election_state (magic + version + checksum; corrupt → refuse to boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE any vote reply or leadership claim leaves the node.
• Term fencing everywhere (kills incident §1.4-1). Every replication RPC carries the sender's term: stale-term ships/chunks/heartbeats/frontier reports are rejected, the quorum commit index folds only reports stamped with its activation term (race-free, under the index's own lock), and a restarted ex-leader boots as a FOLLOWER from its durable state — never from the topology file. A partitioned ex-leader that restarts cannot accept a single write (mp_fenced_ex_leader_restart_cannot_write); a leader that loses majority contact steps down within election.leader_lease_ms (default 900; validated lease + heartbeat < election_timeout_min so a deposed leader stops before any successor can exist).
• Divergent suffixes quarantine instead of lying. A node whose log extends past what the elected leadership subsumed (leader-acked, never-quorum-acked writes on a dead leader) detects it at term-join — the heartbeat carries the leader's election-time log position — and fences itself from the data plane (status quarantined: true, metric tidaldb_cluster_divergence_quarantined, ERROR naming the reseed runbook) while still voting. Followers apply on receipt, so un-applying is not a thing: reseed is the honest recovery (m11p5 snapshots automate it).
• /cluster/promote is now a fenced transfer: with a live leader it drains (waits for the target to hold the flushed prefix, breaker-immune via the commit index's marks) then sanctions an immediate election; with a dead leader the target campaigns. The election refuses a target whose log lags — the m11p3 "promote the max-applied survivor" operator rule is now enforced by the protocol. The legacy term-0 fan-out survives only on clusters that have never elected (mixed-version rollouts; the chaos drills' deliberate isolated-node override) and is permanently retired per node at its first joined election. election.auto_election: false preserves the full pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
• Bounded churn under flapping links: the pre-vote (no term inflation without a majority probe) plus the leader-freshness lease absorb short flaps entirely and bound terms under long ones (mp_flapping_links_bounded_churn). Election observability: tidaldb_cluster_election_term/_role/_elections_started_total/ _leader_changes_total, and /cluster/status/local grows term, role, quarantined, prev_log_term/seq.
• New tier-3 exit-gate suite cluster_election.rs (auto-failover ledger, fencing under partition+restart, bounded churn); cluster_quorum.rs and cluster_lifecycle.rs pin auto_election: false (they validate the manual drill, which remains supported); the partition harness gained bidirectional isolation (isolate_region) — severing only a node's inbound edges leaves its outbound heartbeats/votes flowing, which silently defeats partition scenarios.

Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, TSEG segment header, structured "snapshot required"

• Failed catch-up pulls retry on a timer. The pull trigger was event-only: a follower whose StreamSegments pull failed (e.g. the leader's gRPC server not yet ready during a rolling restart) waited for the next PUSHED segment to re-expose the gap — in an idle cluster that push never comes, and the follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers stuck at lag=136507). A failed pull now arms a one-shot timer (replication.catchup_retry_ms, default 30000; transport catchup_retry_interval) that re-pulls from the CURRENT applied frontier. Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when consumed by the rate limit or an in-flight pull, so the gap always keeps a standing wake-up until a pull completes. Verified over real sockets: catchup_retry.rs reproduces the incident (pull fails, leader appears, zero pushes) and proves timer-only self-heal — and that a clean completion arms nothing.
• WAL segment files are format-versioned. New segments open with an 8-byte header (TSEG magic + version byte + reserved); pre-m11p4 headerless segments stay readable as implicit version 0 — no migration. A segment this binary cannot identify (unknown header version, unrecognized leading bytes, unparseable .seg filename) surfaces as the new WalError::SegmentFormatUnknown at open — previously it scanned as empty (segments=0) and recovery's torn-tail repair could TRUNCATE the foreign file to zero. Foreign-format files are never repaired, truncated, or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
• Unservable catch-up is a structured refusal. SegmentSource::collect_from returns typed SegmentReadError::{Unavailable,Failed}; TidalDb::read_wal_batches returns the typed WalError (was stringified TidalError). The StreamSegments handler maps Unavailable to FAILED_PRECONDITION — "segments not available from seq N; snapshot required" — and the follower logs it distinctly (catch-up unservable … needs a snapshot (m11p5) or an operator reseed) instead of burying it as a transient. The on-disk segment format is now documented (tidal/src/wal/segment.rs module docs + spec 01 §2.2).

Quorum-acked writes (m11p3) — ack=leader|quorum, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)

• ack=quorum is an opt-in durability contract for every replicated write (/signals, /items, /embeddings): success means a majority of the replica set durably holds the write (leader + floor(n/2) followers, each storage-applied and own-WAL-fsynced), so an acked write survives the permanent loss of any single node — including the leader. Deployment default via topology replication.ack; per-request override via the x-tidal-ack header (forwarded verbatim by gateways). ack=leader (the default) is the m0–m11p2 contract unchanged.
• Durable frontier reports. Followers PUSH their durably-applied frontier to the leader once per apply round (new ReportApplied RPC, fired by the segment receiver through the new Transport::notify_applied) — batch-level and fully decoupled from ship acks, so the commit index stays fresh even when outbound ships stall (gap-parked follower, pull-based catch-up, quiet leader). Ship acks keep their m11p2 instant floor-hint semantics — both inputs are durable-true because a follower's frontier only ever advances after its storage apply + own-WAL fsync. (The first design held each ship ack until its segment's apply; measured under open-loop load, that couples ship cadence to apply latency and one gap-parked follower spirals into total quorum collapse — the report push is what shipped.)
• Follower blob applies are group-committed. m11p2 applied replicated items/embeddings one record at a time — one solo follower fsync per item, capping item apply at the fsync floor (~100/s on macOS) and stalling the quorum frontier behind any item burst. The receiver now hands each apply round's blob records to the engine as ONE batch (apply_replicated_blobs: validate all → stage all WAL appends → wait all → upsert storage), and the WAL writer flushes queued blobs under ONE group fsync. Measured: corpus seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
• Commit index. The leader folds durable acks (and heal resumes) into per-peer durable marks; the commit index is the k-th largest (k = floor(n/2)), leadership-scoped (promote resets it to the stream baseline; demotion fails in-flight waiters — a demoted leader never claims quorum). Handlers await it through a watch-channel bridge — fully async, zero threads parked per waiter (the thread-per-wait design measurably collapsed at 1k rps open-loop by exhausting the blocking pool and starving the very completions that advance the index). Quorum capacity measured on a real 3-process localhost cluster (release, writes mix): 3,600 quorum signal-writes/s within SLO (p50 ~45ms, zero errors, replication lag ≤3 events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack figure, vs the ≥50% gate.
• Honest timeout semantics. Quorum not confirmed within replication.quorum_timeout_ms (default 2000) → a retryable 503 naming the laggard regions, the commit index, and needed/confirmed counts. The write is in the leader's log and may still commit: retries are at-least-once (items/embeddings retries are idempotent upserts; signal retries can double-count — decided in-phase: no idempotency-key machinery, documented in runbook §8 with the session-write precedent for callers that need exact-once).
• x-tidal-seq on every cluster write response: the write's seqno in the replicated log (relayed through forwards) — an exact durability cursor against commit_index in /cluster/status/local (which also gains ack). tidaldb_cluster_relay_durable_seq now reports the commit index (relay_last_seq − relay_durable_seq = quorum lag); new counter tidaldb_cluster_quorum_timeouts_total.
• The ledger gate (exit gate, run for real): tier-3 mp_quorum_ledger_zero_acked_loss_across_killpoints SIGKILLs the leader under concurrent quorum load and proves zero acknowledged loss on the promoted max-applied survivor — frontier invariant (max acked seq ≤ survivor applied) plus per-item content probes. 167/167 kill points passed on the final design (batches of 64 + 91 + 12; kill timings spread 120–598ms across fresh 3-process clusters; an earlier 100/100 run had validated the superseded ack-holding design before it was replaced — see docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics (one follower down: quorum commits; both down: fast 503 naming laggards while ack=leader flows; heal: recovers) and in-process gRPC coverage (override headers, forwarded quorum writes, blob writes, 400 on bad mode).
• Rolling-upgrade order (mixed-version caveat): a pre-m11p3 leader neither serves the ReportApplied RPC nor recognizes x-tidal-ack — it silently applies LEADER-ack semantics to a quorum request (a durability downgrade the caller cannot see). Upgrade the leader first: ack=quorum is then honored immediately (commit-index freshness rides the m11p2 ship-ack floor hints until the followers upgrade too). Replication and heal are unaffected by either order. Runbook §8 records the procedure.

One replicated log (m11p2) — items/embeddings ride the WAL, StreamSegments catch-up, HTTP broadcast deleted

• The leader's WAL is now THE replicated log. The group-commit writer hands every fsynced batch to a bounded in-memory ship feed (wal::feed::WalShipFeed); the ship queue pushes those already-encoded bytes verbatim — byte-identical on the leader's disk, the wire, and the follower's apply path — and stream seqnos are WAL seqnos, so they survive restarts (the m8p10 relay-reset hazard is gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning machinery left the server write path entirely (/signals stages straight through the engine's group commit; a WAL fsync failure now surfaces per-write exactly like single-node).
• Item metadata and embeddings are replicated mutations. /items and /embeddings journal kind-1/2 blob records (WAL header flags byte = batch kind; one record, one seqno) BEFORE storage, on the same stream as signals. Followers apply them kind-aware — WAL-first into their own log, then idempotent storage upserts — and recovery replays them. The m8p10 HTTP item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast, authed side-POSTs — the source of both 2026-06-10 live bugs) is deleted; those bug classes are now impossible by construction. Cluster /items returns a plain 201 and /embeddings a plain 204 (no broadcast-report body).
• StreamSegments implemented — catch-up is follower-pulled. The ship feed's tail is bounded; a peer that falls behind it is skipped ahead, and the follower pulls the hole itself via the (previously declared-unimplemented) server-streaming RPC over the leader's durable, BLAKE3-verified segments. Pulls trigger on detected gaps, on follower boot (self-driving restart catch-up), and on the leader's heal nudge (POST /cluster/catchup, internal, forwards the operator's own bearer credential). Pulled chunks flow through the same inbound apply path as live ships. Ship acks piggyback the follower's applied seqno (ShipSegmentResponse.applied_seqno), so retries of already-applied data prune and heal is resume_from + nudge — no redelivery scan, no O(items) traffic.
• Promote carries a stream baseline. A promoted leader's stream starts at its promote-time flushed frontier (persisted in data_dir/stream_baseline); the fan-out body and every catch-up chunk announce it, so peers jump their frontier past pre-stream history instead of parking on a phantom gap. Ship queues are leadership-gated (activate_from/deactivate): a follower's replicated applies never echo back at its peers.
• Multi-process cluster mode now requires --data-dir (validated at startup): the durable WAL is the replication stream. Standalone single-node deployments are untouched — blob journaling and the ship feed are gated on cluster peers, so single-node item writes keep fjall-only durability with zero extra fsyncs.
• New tier-3 suite mp_items_ride_the_log_and_catchup_stream: items written on the leader AND through a follower gateway converge everywhere via the log (feed parity 1e-6), and a follower stopped through item+embedding+signal writes restarts and converges via its boot-time StreamSegments pull with no heal verb and no HTTP item traffic.

Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first tidaldb_cluster_* metrics

• The replicated /signals write path no longer serializes every writer onto a solo group-commit fsync nor ships to followers on the request path. Writes are staged (seqno + WAL submission + relay log push, microseconds, atomic with rollback) and completed (shared group-commit fsync + in-memory fold) in two phases, so concurrent writers coalesce into one fsync; follower shipping moved to per-peer sender threads that coalesce contiguous runs into multi-event batches with a windowed in-flight budget (topology knobs replication.{batch_max_events,window,retry_ms}). The 204 contract is unchanged (leader durability only) — it now returns at leader fsync. Measured on a real 3-process localhost cluster (release build, thepeach mix): 4,534 replicated signal-writes/s sustained within SLO vs ~90/s before (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
• Durable-frontier shipping + relay poisoning. Senders only ship the leader's contiguous fsynced prefix (an event a follower holds but the leader could lose is silent divergence); a staged write whose fsync fails poisons the relay — further cluster writes are rejected and the ship frontier freezes (the CockroachDB/Postgres fsync-failure posture).
• Follower group-commit coalescing. The segment receiver drains its inbound backlog (Transport::try_recv_segment) and applies it through ONE shared group commit (SignalLedger::apply_replicated_events), in range-disjoint groups so duplicate/subset re-ships cannot double-fold. Without this the follower apply ceiling was ~events-per-segment / fsync-cost (~1.8k events/s measured) and lag grew without bound under m11p1 leader rates.
• First cluster metrics + cluster /metrics listener. Cluster mode previously had no metrics endpoint at all. New per-region topology metrics_addr wires the engine's Prometheus listener; new tidaldb_cluster_* series: ship RTT + batch-size histograms, per-peer ship counters/gauges (peer_shard labels), WAL fsync latency + group-commit fill histograms, write-pool depth/rejections, relay committed + durable frontiers. WAL group-commit knobs are deployment config (wal.{batch_size,batch_timeout_ms} topology block; wal_batch_size/wal_batch_timeout builder methods).
• Engine API: TidalDb::signal_staged/StagedSignal::wait, SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}, ShipQueue (per-peer windowed batch senders with pause/resume), WalWriter::append_signal_staged, WalSender::append_record_staged, range_payload/encode_run. Relay log entries are now RelayEvent (raw event records, re-encoded deterministically at ship time) instead of pre-encoded single-event bytes.
• Ship-sender failure logging is transition-based (first + every 50th consecutive failure WARN with the running count, recovery INFO) — the per-retry WARN flood could fill an undrained log pipe and stall the process. The multiproc test harness now discards child logs via /dev/null (a piped fd nobody drains deadlocks the node once the kernel buffer fills), with TIDAL_TEST_NODE_LOGS=inherit to stream them while debugging.

M9 — Community Sync & Revocation

• Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: SignalScope, CommunityId, Membership, MembershipEpoch, PolicyMetadata. Community signal reconciliation via CrdtSignalState with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

M10 — Governance & Agent Rights

• Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE

• Multi-process cluster mode: tidal-server cluster --region <name> [--data-dir <p>] (env TIDAL_REGION) runs one process per region (RegionClusterState), each owning one TidalDb and one GrpcTransport that binds this region's grpc_addr and dials every sibling's real grpc_addr — real process/host isolation. The topology requires per-region grpc_addr AND http_addr in this mode; single-process mode (no --region) is unchanged as the dev/demo default. Both modes stay behind the experimental gate (--experimental-cluster / TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) with mode-specific WARN text.
• New routes / behaviors on the multi-process surface: GET /cluster/status/local (per-node status), GET /cluster/status aggregates ALL regions with a reachable field (unreachable ⇒ reachable:false + worst-case lag), POST /cluster/reconcile {region} (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact no-op on scores), POST /hardnegs {user_id,item_id} (user-scoped hide; converges via reconcile, filtered from /feed?user_id). Writes on a non-leader forward to the leader; ?region= reads forward to the owning region; default reads are LOCAL in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast to peers with a per-peer report: POST /items → 201 {replicated_to, failed}, POST /embeddings → 200 with the same report on the leader path (NOT 204 — a 204 cannot carry the report; the forwarded/internal path stays 204). /sharded/* fans out across processes (degraded semantics preserved). promote fans out to all peers ({ok, leader, acked, failed}).
• /cluster/heal is the single recovery verb: redelivers missed signal segments (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold 5, reset 30s) is open, so re-issue /cluster/heal until /cluster/status shows lag 0.
• TIDAL_HLC_SKEW_MS (multi-process): signed ms offset applied to the process's HLC (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
• Tier-3 UAT over real OS processes (feature cluster-e2e): cluster_multiproc (5), cluster_chaos (3, REAL network-partition injection via a root-free in-harness TCP relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl remain an operator option), cluster_lifecycle (2, ±500ms clock skew + rolling upgrade with zero acknowledged-write loss), cluster_runbook (9, every docs/runbooks/cluster.md §5–§11 operation), plus cluster_e2e (2, single-process smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

Fixed

Four production bugs surfaced and fixed by the m8p10 tier-3 UAT

• Silent data loss on out-of-order ships. The replication applied high-water-mark swallowed sequence gaps when eager ships arrived out of order. The applied frontier is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
• Non-idempotent reconcile (score creep). take_crdt_snapshot attributed replicated signal streams per-node, so reconciling already-converged nodes crept the decayed scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical replication shard (ShardId::SINGLE), making merge idempotent — reconcile of converged nodes is an exact fixpoint.
• Lag gauge conflated leader streams across a promotion. A converged node reported a permanent phantom lag after a /cluster/promote moved leadership to a different shard. The lag gauge now tracks the leader high-water-mark per source shard (ReplicationLagGauge::leader_seqno_for), computing lag against the current leader.
• Items missed during a broadcast were never backfilled. A node down/partitioned during an item/embedding broadcast was permanently missing that data even at signal lag 0. /cluster/heal now re-broadcasts item metadata + embeddings to the healed region (idempotent upserts), making heal the single recovery verb.

Changed

Cluster mode (m8p8): real gRPC replication

• tidal-server's ClusterState now wires each follower region to a real tidal-net GrpcTransport (self-loop over loopback gRPC) instead of in-process crossbeam channels — replication between regions traverses real gRPC/TCP (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
• Follower gRPC ports are auto-allocated from the topology (grpc_addr optional per region) and self-heal a transient bind race by retrying on a fresh port.
• Blocking gRPC ships triggered by POST /signals and POST /cluster/heal are offloaded to a dedicated thread so they never block the axum reactor.
• ClusterState is constructed off the async reactor (GrpcTransport::new blocks on its own runtime).
• The experimental opt-in gate and docker/cluster/Dockerfile are updated: single-process cluster mode is honest that it replicates over real gRPC but runs all regions in one process (no host/process isolation). (True multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry above.)

Docker build fixes (all three images now that tidal-server pulls tidal-net)

• Install protobuf-compiler in the builder stage — tidal-net's build script runs tonic-build, which needs protoc to compile the WAL-shipping .proto.
• Pin the builder base to bookworm (rust:1.91-bookworm / rust:1.91-slim-bookworm) so its glibc matches the debian:bookworm-slim runtime; the default trixie base emitted a libmvec.so.1 dependency absent on bookworm, aborting the binary at startup. Verified: docker run of the cluster image serves a functional 3-region cluster (write replicates to both followers over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
• New tests: tidal-server/tests/cluster_grpc.rs (in-process gRPC replication + HTTP offload path) and a hardened tier-3 cluster_e2e.rs (multi-process smoke
  • promote over real OS processes, feature-gated).

[0.1.0] - 2026-02-23

Added

Core Database Engine

• TidalDb embeddable database with ephemeral() and with_data_dir() open modes
• SchemaBuilder for defining signal types, decay parameters, and ranking profiles
• TidalDbBuilder fluent builder with schema, data directory, metrics, and rate limiter configuration

Signal System

• Typed signal recording with exponential decay scoring
• Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
• Windowed aggregation: OneHour, TwentyFourHours, SevenDays, AllTime
• Signal velocity tracking
• WAL-backed signal durability with crash recovery
• Periodic signal checkpointing to fjall (every 30s)
• WAL compaction after each checkpoint

Retrieval (RETRIEVE query)

• 5-stage pipeline: universe, filter, score, diversify, return
• Filter expressions: Eq, In, Gt, Lt, And, Or, Not, InCollection, InProgress, MinSignal, MaxSignal, NearLocation
• Built-in ranking profiles: trending, for_you, new, popular, recent, and 20+ more
• Custom ranking profiles via SchemaBuilder
• Diversity enforcement (max N per category/creator)
• Sort modes: Relevance, Trending, Newest, MostLiked, MostViewed, MostFollowed, AlphabeticalAsc/Desc, Shortest/Longest, LiveViewerCount, DateSaved, and more

Search (SEARCH query)

• BM25 full-text search via Tantivy
• Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
• Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
• Creator search with entity_kind(EntityKind::Creator)
• similar_to(EntityId) for content-based recommendations
• Scope pre-filters: Trending, CohortTrending, Following, Category, Collection
• Autocomplete suggestions via db.suggest()

Entity Model

• Three built-in entity types: Item, User, Creator
• Metadata storage as HashMap<String, String>
• Embedding slots (up to 4 per entity type) via USearch
• Relationships: Follows, Blocks, Hide, Mute, InteractionWeight

Sessions

• Session lifecycle: open_session, close_session
• Cross-session preference vector updates (EMA blend)
• Session snapshots with signal state and preference vectors
• Session serialization format v0x03 with backward compatibility

Social Graph

• Creator follower/following indexes
• Cohort membership (user segments)
• CoEngagementIndex for co-viewing patterns with LRU eviction
• Social graph filter for "followed creator" content scoping

Collections

• Named collections with Private, Shared, Public visibility
• create_collection, add_to_collection, remove_from_collection, list_collections
• FilterExpr::InCollection for collection-scoped retrieval
• Saved searches with save_search, list_saved_searches, retrieve_saved_search

Observability

• enable_metrics(addr) -- Prometheus-format /metrics endpoint + /healthz JSON
• 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
• tidaldb_checkpoint_failures_total counter for checkpoint monitoring
• TidalDb::diagnostics() -- structured health snapshot
• WAL diagnostics and recovery tools

Safety

• Signal weight NaN/Inf validation (returns TidalError::InvalidInput)
• Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
• Export request limit: 500K signals max per request
• FilterExpr complexity limit: 256 nodes max
• Data directory lock (tidaldb.lock) prevents dual-process corruption
• Schema fingerprint persistence detects decay parameter changes on reopen
• Bounded closed_sessions cache (10K max, LRU eviction)
• Metrics server non-loopback bind warning

CLI (tidalctl)

• tidalctl binary for database inspection and diagnostics

RLHF / ML Export

• db.export_signals(ExportRequest) -- WAL-based signal export for training data
• db.user_session_summary(user_id, since_ns) -- aggregated session statistics

Stability

tidalDB 0.1.0 is pre-1.0. No API or data format stability guarantees are made for 0.x releases. Upgrade guides will be provided for each minor version bump. Do not upgrade 0.x to 0.y on a live data directory without reading the release notes.

---

Format based on Keep a Changelog