tidaldb/CHANGELOG.md

# Changelog

All notable changes to tidalDB will be documented in this file.

## [Unreleased]

### Added

**One replicated log (m11p2) — items/embeddings ride the WAL, `StreamSegments` catch-up, HTTP broadcast deleted**
- **The leader's WAL is now THE replicated log.** The group-commit writer hands
  every fsynced batch to a bounded in-memory **ship feed** (`wal::feed::WalShipFeed`);
  the ship queue pushes those already-encoded bytes verbatim — byte-identical on
  the leader's disk, the wire, and the follower's apply path — and stream seqnos
  are WAL seqnos, so they **survive restarts** (the m8p10 relay-reset hazard is
  gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning
  machinery left the server write path entirely (`/signals` stages straight
  through the engine's group commit; a WAL fsync failure now surfaces per-write
  exactly like single-node).
- **Item metadata and embeddings are replicated mutations.** `/items` and
  `/embeddings` journal kind-1/2 **blob records** (WAL header `flags` byte =
  batch kind; one record, one seqno) BEFORE storage, on the same stream as
  signals. Followers apply them kind-aware — WAL-first into their own log, then
  idempotent storage upserts — and recovery replays them. The m8p10 HTTP
  item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast,
  authed side-POSTs — the source of both 2026-06-10 live bugs) is **deleted**;
  those bug classes are now impossible by construction. Cluster `/items` returns
  a plain 201 and `/embeddings` a plain 204 (no broadcast-report body).
- **`StreamSegments` implemented — catch-up is follower-pulled.** The ship feed's
  tail is bounded; a peer that falls behind it is skipped ahead, and the
  follower pulls the hole itself via the (previously declared-unimplemented)
  server-streaming RPC over the leader's durable, BLAKE3-verified segments.
  Pulls trigger on detected gaps, on follower boot (self-driving restart
  catch-up), and on the leader's heal nudge (`POST /cluster/catchup`, internal,
  forwards the operator's own bearer credential). Pulled chunks flow through the
  same inbound apply path as live ships. Ship acks piggyback the follower's
  applied seqno (`ShipSegmentResponse.applied_seqno`), so retries of
  already-applied data prune and heal is `resume_from` + nudge — no redelivery
  scan, no O(items) traffic.
- **Promote carries a stream baseline.** A promoted leader's stream starts at its
  promote-time flushed frontier (persisted in `data_dir/stream_baseline`); the
  fan-out body and every catch-up chunk announce it, so peers jump their
  frontier past pre-stream history instead of parking on a phantom gap. Ship
  queues are leadership-gated (`activate_from`/`deactivate`): a follower's
  replicated applies never echo back at its peers.
- **Multi-process cluster mode now requires `--data-dir`** (validated at
  startup): the durable WAL is the replication stream. Standalone single-node
  deployments are untouched — blob journaling and the ship feed are gated on
  cluster peers, so single-node item writes keep fjall-only durability with
  zero extra fsyncs.
- New tier-3 suite `mp_items_ride_the_log_and_catchup_stream`: items written on
  the leader AND through a follower gateway converge everywhere via the log
  (feed parity 1e-6), and a follower stopped through item+embedding+signal
  writes restarts and converges via its boot-time `StreamSegments` pull with no
  heal verb and no HTTP item traffic.

**Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first `tidaldb_cluster_*` metrics**
- The replicated `/signals` write path no longer serializes every writer onto a
  solo group-commit fsync nor ships to followers on the request path. Writes are
  **staged** (seqno + WAL submission + relay log push, microseconds, atomic with
  rollback) and **completed** (shared group-commit fsync + in-memory fold) in two
  phases, so concurrent writers coalesce into one fsync; follower shipping moved
  to per-peer sender threads that coalesce contiguous runs into multi-event
  batches with a windowed in-flight budget (topology knobs
  `replication.{batch_max_events,window,retry_ms}`). The 204 contract is
  unchanged (leader durability only) — it now returns at leader fsync. Measured
  on a real 3-process localhost cluster (release build, thepeach mix):
  **4,534 replicated signal-writes/s sustained within SLO** vs ~90/s before
  (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
- **Durable-frontier shipping + relay poisoning.** Senders only ship the
  leader's contiguous fsynced prefix (an event a follower holds but the leader
  could lose is silent divergence); a staged write whose fsync fails **poisons**
  the relay — further cluster writes are rejected and the ship frontier freezes
  (the CockroachDB/Postgres fsync-failure posture).
- **Follower group-commit coalescing.** The segment receiver drains its inbound
  backlog (`Transport::try_recv_segment`) and applies it through ONE shared
  group commit (`SignalLedger::apply_replicated_events`), in range-disjoint
  groups so duplicate/subset re-ships cannot double-fold. Without this the
  follower apply ceiling was ~`events-per-segment / fsync-cost` (~1.8k events/s
  measured) and lag grew without bound under m11p1 leader rates.
- **First cluster metrics + cluster `/metrics` listener.** Cluster mode
  previously had no metrics endpoint at all. New per-region topology
  `metrics_addr` wires the engine's Prometheus listener; new `tidaldb_cluster_*`
  series: ship RTT + batch-size histograms, per-peer ship counters/gauges
  (`peer_shard` labels), WAL fsync latency + group-commit fill histograms,
  write-pool depth/rejections, relay committed + durable frontiers. WAL
  group-commit knobs are deployment config (`wal.{batch_size,batch_timeout_ms}`
  topology block; `wal_batch_size`/`wal_batch_timeout` builder methods).
- Engine API: `TidalDb::signal_staged`/`StagedSignal::wait`,
  `SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}`,
  `ShipQueue` (per-peer windowed batch senders with pause/resume),
  `WalWriter::append_signal_staged`, `WalSender::append_record_staged`,
  `range_payload`/`encode_run`. Relay log entries are now `RelayEvent` (raw
  event records, re-encoded deterministically at ship time) instead of
  pre-encoded single-event bytes.
- Ship-sender failure logging is transition-based (first + every 50th
  consecutive failure WARN with the running count, recovery INFO) — the
  per-retry WARN flood could fill an undrained log pipe and stall the process.
  The multiproc test harness now discards child logs via `/dev/null` (a piped
  fd nobody drains deadlocks the node once the kernel buffer fills), with
  `TIDAL_TEST_NODE_LOGS=inherit` to stream them while debugging.

**M9 — Community Sync & Revocation**
- Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata`. Community signal reconciliation via `CrdtSignalState` with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

**M10 — Governance & Agent Rights**
- Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

**Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE**
- Multi-process cluster mode: `tidal-server cluster --region <name> [--data-dir <p>]`
  (env `TIDAL_REGION`) runs **one process per region** (`RegionClusterState`), each
  owning one `TidalDb` and one `GrpcTransport` that binds this region's `grpc_addr`
  and dials every sibling's real `grpc_addr` — real process/host isolation. The
  topology requires per-region `grpc_addr` AND `http_addr` in this mode;
  single-process mode (no `--region`) is unchanged as the dev/demo default. Both
  modes stay behind the experimental gate (`--experimental-cluster` /
  `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) with mode-specific WARN text.
- New routes / behaviors on the multi-process surface: `GET /cluster/status/local`
  (per-node status), `GET /cluster/status` aggregates ALL regions with a `reachable`
  field (unreachable ⇒ `reachable:false` + worst-case lag), `POST /cluster/reconcile
  {region}` (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact
  no-op on scores), `POST /hardnegs {user_id,item_id}` (user-scoped hide; converges
  via reconcile, filtered from `/feed?user_id`). Writes on a non-leader **forward to
  the leader**; `?region=` reads forward to the owning region; default reads are
  **LOCAL** in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast
  to peers with a per-peer report: `POST /items` → 201 `{replicated_to, failed}`,
  `POST /embeddings` → **200** with the same report on the leader path (NOT 204 — a
  204 cannot carry the report; the forwarded/internal path stays 204). `/sharded/*`
  fans out across processes (degraded semantics preserved). `promote` fans out to all
  peers (`{ok, leader, acked, failed}`).
- `/cluster/heal` is the single recovery verb: redelivers missed signal segments
  (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region
  (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold
  5, reset 30s) is open, so re-issue `/cluster/heal` until `/cluster/status` shows
  lag 0.
- `TIDAL_HLC_SKEW_MS` (multi-process): signed ms offset applied to the process's HLC
  (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
- Tier-3 UAT over real OS processes (feature `cluster-e2e`): `cluster_multiproc` (5),
  `cluster_chaos` (3, REAL network-partition injection via a root-free in-harness TCP
  relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl
  remain an operator option), `cluster_lifecycle` (2, ±500ms clock skew + rolling
  upgrade with zero acknowledged-write loss), `cluster_runbook` (9, every
  `docs/runbooks/cluster.md` §5–§11 operation), plus `cluster_e2e` (2, single-process
  smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover
  ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

### Fixed

**Four production bugs surfaced and fixed by the m8p10 tier-3 UAT**
- **Silent data loss on out-of-order ships.** The replication applied high-water-mark
  swallowed sequence gaps when eager ships arrived out of order. The applied frontier
  is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
- **Non-idempotent reconcile (score creep).** `take_crdt_snapshot` attributed replicated
  signal streams per-node, so reconciling already-converged nodes crept the decayed
  scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical
  replication shard (`ShardId::SINGLE`), making `merge` idempotent — reconcile of
  converged nodes is an exact fixpoint.
- **Lag gauge conflated leader streams across a promotion.** A converged node reported a
  permanent phantom lag after a `/cluster/promote` moved leadership to a different shard.
  The lag gauge now tracks the leader high-water-mark per source shard
  (`ReplicationLagGauge::leader_seqno_for`), computing lag against the current leader.
- **Items missed during a broadcast were never backfilled.** A node down/partitioned
  during an item/embedding broadcast was permanently missing that data even at signal
  lag 0. `/cluster/heal` now re-broadcasts item metadata + embeddings to the healed
  region (idempotent upserts), making heal the single recovery verb.

### Changed

**Cluster mode (m8p8): real gRPC replication**
- `tidal-server`'s `ClusterState` now wires each follower region to a real
  `tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
  crossbeam channels — replication between regions traverses real gRPC/TCP
  (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
  per region) and self-heal a transient bind race by retrying on a fresh port.
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
  offloaded to a dedicated thread so they never block the axum reactor.
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
  blocks on its own runtime).
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
  single-process cluster mode is honest that it replicates over real gRPC but
  runs all regions in one process (no host/process isolation). (True
  multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry
  above.)

**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
  runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
  `rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
  runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
  bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
  image serves a functional 3-region cluster (write replicates to both followers
  over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
  HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
  + promote over real OS processes, feature-gated).

## [0.1.0] - 2026-02-23

### Added

**Core Database Engine**
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration

**Signal System**
- Typed signal recording with exponential decay scoring
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
- Signal velocity tracking
- WAL-backed signal durability with crash recovery
- Periodic signal checkpointing to fjall (every 30s)
- WAL compaction after each checkpoint

**Retrieval (RETRIEVE query)**
- 5-stage pipeline: universe, filter, score, diversify, return
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
- Custom ranking profiles via `SchemaBuilder`
- Diversity enforcement (max N per category/creator)
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more

**Search (SEARCH query)**
- BM25 full-text search via Tantivy
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
- Creator search with `entity_kind(EntityKind::Creator)`
- `similar_to(EntityId)` for content-based recommendations
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
- Autocomplete suggestions via `db.suggest()`

**Entity Model**
- Three built-in entity types: `Item`, `User`, `Creator`
- Metadata storage as `HashMap<String, String>`
- Embedding slots (up to 4 per entity type) via USearch
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`

**Sessions**
- Session lifecycle: `open_session`, `close_session`
- Cross-session preference vector updates (EMA blend)
- Session snapshots with signal state and preference vectors
- Session serialization format v0x03 with backward compatibility

**Social Graph**
- Creator follower/following indexes
- Cohort membership (user segments)
- CoEngagementIndex for co-viewing patterns with LRU eviction
- Social graph filter for "followed creator" content scoping

**Collections**
- Named collections with `Private`, `Shared`, `Public` visibility
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
- `FilterExpr::InCollection` for collection-scoped retrieval
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`

**Observability**
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
- `TidalDb::diagnostics()` -- structured health snapshot
- WAL diagnostics and recovery tools

**Safety**
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
- Export request limit: 500K signals max per request
- `FilterExpr` complexity limit: 256 nodes max
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
- Schema fingerprint persistence detects decay parameter changes on reopen
- Bounded `closed_sessions` cache (10K max, LRU eviction)
- Metrics server non-loopback bind warning

**CLI (`tidalctl`)**
- `tidalctl` binary for database inspection and diagnostics

**RLHF / ML Export**
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics

### Stability

tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.

---

*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*