tidaldb/CHANGELOG.md

Changelog

All notable changes to tidalDB will be documented in this file.

[Unreleased]

Added

One replicated log (m11p2) — items/embeddings ride the WAL, StreamSegments catch-up, HTTP broadcast deleted

• The leader's WAL is now THE replicated log. The group-commit writer hands every fsynced batch to a bounded in-memory ship feed (wal::feed::WalShipFeed); the ship queue pushes those already-encoded bytes verbatim — byte-identical on the leader's disk, the wire, and the follower's apply path — and stream seqnos are WAL seqnos, so they survive restarts (the m8p10 relay-reset hazard is gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning machinery left the server write path entirely (/signals stages straight through the engine's group commit; a WAL fsync failure now surfaces per-write exactly like single-node).
• Item metadata and embeddings are replicated mutations. /items and /embeddings journal kind-1/2 blob records (WAL header flags byte = batch kind; one record, one seqno) BEFORE storage, on the same stream as signals. Followers apply them kind-aware — WAL-first into their own log, then idempotent storage upserts — and recovery replays them. The m8p10 HTTP item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast, authed side-POSTs — the source of both 2026-06-10 live bugs) is deleted; those bug classes are now impossible by construction. Cluster /items returns a plain 201 and /embeddings a plain 204 (no broadcast-report body).
• StreamSegments implemented — catch-up is follower-pulled. The ship feed's tail is bounded; a peer that falls behind it is skipped ahead, and the follower pulls the hole itself via the (previously declared-unimplemented) server-streaming RPC over the leader's durable, BLAKE3-verified segments. Pulls trigger on detected gaps, on follower boot (self-driving restart catch-up), and on the leader's heal nudge (POST /cluster/catchup, internal, forwards the operator's own bearer credential). Pulled chunks flow through the same inbound apply path as live ships. Ship acks piggyback the follower's applied seqno (ShipSegmentResponse.applied_seqno), so retries of already-applied data prune and heal is resume_from + nudge — no redelivery scan, no O(items) traffic.
• Promote carries a stream baseline. A promoted leader's stream starts at its promote-time flushed frontier (persisted in data_dir/stream_baseline); the fan-out body and every catch-up chunk announce it, so peers jump their frontier past pre-stream history instead of parking on a phantom gap. Ship queues are leadership-gated (activate_from/deactivate): a follower's replicated applies never echo back at its peers.
• Multi-process cluster mode now requires --data-dir (validated at startup): the durable WAL is the replication stream. Standalone single-node deployments are untouched — blob journaling and the ship feed are gated on cluster peers, so single-node item writes keep fjall-only durability with zero extra fsyncs.
• New tier-3 suite mp_items_ride_the_log_and_catchup_stream: items written on the leader AND through a follower gateway converge everywhere via the log (feed parity 1e-6), and a follower stopped through item+embedding+signal writes restarts and converges via its boot-time StreamSegments pull with no heal verb and no HTTP item traffic.

Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first tidaldb_cluster_* metrics

• The replicated /signals write path no longer serializes every writer onto a solo group-commit fsync nor ships to followers on the request path. Writes are staged (seqno + WAL submission + relay log push, microseconds, atomic with rollback) and completed (shared group-commit fsync + in-memory fold) in two phases, so concurrent writers coalesce into one fsync; follower shipping moved to per-peer sender threads that coalesce contiguous runs into multi-event batches with a windowed in-flight budget (topology knobs replication.{batch_max_events,window,retry_ms}). The 204 contract is unchanged (leader durability only) — it now returns at leader fsync. Measured on a real 3-process localhost cluster (release build, thepeach mix): 4,534 replicated signal-writes/s sustained within SLO vs ~90/s before (~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
• Durable-frontier shipping + relay poisoning. Senders only ship the leader's contiguous fsynced prefix (an event a follower holds but the leader could lose is silent divergence); a staged write whose fsync fails poisons the relay — further cluster writes are rejected and the ship frontier freezes (the CockroachDB/Postgres fsync-failure posture).
• Follower group-commit coalescing. The segment receiver drains its inbound backlog (Transport::try_recv_segment) and applies it through ONE shared group commit (SignalLedger::apply_replicated_events), in range-disjoint groups so duplicate/subset re-ships cannot double-fold. Without this the follower apply ceiling was ~events-per-segment / fsync-cost (~1.8k events/s measured) and lag grew without bound under m11p1 leader rates.
• First cluster metrics + cluster /metrics listener. Cluster mode previously had no metrics endpoint at all. New per-region topology metrics_addr wires the engine's Prometheus listener; new tidaldb_cluster_* series: ship RTT + batch-size histograms, per-peer ship counters/gauges (peer_shard labels), WAL fsync latency + group-commit fill histograms, write-pool depth/rejections, relay committed + durable frontiers. WAL group-commit knobs are deployment config (wal.{batch_size,batch_timeout_ms} topology block; wal_batch_size/wal_batch_timeout builder methods).
• Engine API: TidalDb::signal_staged/StagedSignal::wait, SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}, ShipQueue (per-peer windowed batch senders with pause/resume), WalWriter::append_signal_staged, WalSender::append_record_staged, range_payload/encode_run. Relay log entries are now RelayEvent (raw event records, re-encoded deterministically at ship time) instead of pre-encoded single-event bytes.
• Ship-sender failure logging is transition-based (first + every 50th consecutive failure WARN with the running count, recovery INFO) — the per-retry WARN flood could fill an undrained log pipe and stall the process. The multiproc test harness now discards child logs via /dev/null (a piped fd nobody drains deadlocks the node once the kernel buffer fills), with TIDAL_TEST_NODE_LOGS=inherit to stream them while debugging.

M9 — Community Sync & Revocation

• Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: SignalScope, CommunityId, Membership, MembershipEpoch, PolicyMetadata. Community signal reconciliation via CrdtSignalState with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.

M10 — Governance & Agent Rights

• Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.

Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE

• Multi-process cluster mode: tidal-server cluster --region <name> [--data-dir <p>] (env TIDAL_REGION) runs one process per region (RegionClusterState), each owning one TidalDb and one GrpcTransport that binds this region's grpc_addr and dials every sibling's real grpc_addr — real process/host isolation. The topology requires per-region grpc_addr AND http_addr in this mode; single-process mode (no --region) is unchanged as the dev/demo default. Both modes stay behind the experimental gate (--experimental-cluster / TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1) with mode-specific WARN text.
• New routes / behaviors on the multi-process surface: GET /cluster/status/local (per-node status), GET /cluster/status aggregates ALL regions with a reachable field (unreachable ⇒ reachable:false + worst-case lag), POST /cluster/reconcile {region} (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact no-op on scores), POST /hardnegs {user_id,item_id} (user-scoped hide; converges via reconcile, filtered from /feed?user_id). Writes on a non-leader forward to the leader; ?region= reads forward to the owning region; default reads are LOCAL in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast to peers with a per-peer report: POST /items → 201 {replicated_to, failed}, POST /embeddings → 200 with the same report on the leader path (NOT 204 — a 204 cannot carry the report; the forwarded/internal path stays 204). /sharded/* fans out across processes (degraded semantics preserved). promote fans out to all peers ({ok, leader, acked, failed}).
• /cluster/heal is the single recovery verb: redelivers missed signal segments (gap-aware) AND re-broadcasts item metadata + embeddings to the healed region (idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold 5, reset 30s) is open, so re-issue /cluster/heal until /cluster/status shows lag 0.
• TIDAL_HLC_SKEW_MS (multi-process): signed ms offset applied to the process's HLC (reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
• Tier-3 UAT over real OS processes (feature cluster-e2e): cluster_multiproc (5), cluster_chaos (3, REAL network-partition injection via a root-free in-harness TCP relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl remain an operator option), cluster_lifecycle (2, ±500ms clock skew + rolling upgrade with zero acknowledged-write loss), cluster_runbook (9, every docs/runbooks/cluster.md §5–§11 operation), plus cluster_e2e (2, single-process smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover ~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).

Fixed

Four production bugs surfaced and fixed by the m8p10 tier-3 UAT

• Silent data loss on out-of-order ships. The replication applied high-water-mark swallowed sequence gaps when eager ships arrived out of order. The applied frontier is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
• Non-idempotent reconcile (score creep). take_crdt_snapshot attributed replicated signal streams per-node, so reconciling already-converged nodes crept the decayed scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical replication shard (ShardId::SINGLE), making merge idempotent — reconcile of converged nodes is an exact fixpoint.
• Lag gauge conflated leader streams across a promotion. A converged node reported a permanent phantom lag after a /cluster/promote moved leadership to a different shard. The lag gauge now tracks the leader high-water-mark per source shard (ReplicationLagGauge::leader_seqno_for), computing lag against the current leader.
• Items missed during a broadcast were never backfilled. A node down/partitioned during an item/embedding broadcast was permanently missing that data even at signal lag 0. /cluster/heal now re-broadcasts item metadata + embeddings to the healed region (idempotent upserts), making heal the single recovery verb.

Changed

Cluster mode (m8p8): real gRPC replication

• tidal-server's ClusterState now wires each follower region to a real tidal-net GrpcTransport (self-loop over loopback gRPC) instead of in-process crossbeam channels — replication between regions traverses real gRPC/TCP (serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
• Follower gRPC ports are auto-allocated from the topology (grpc_addr optional per region) and self-heal a transient bind race by retrying on a fresh port.
• Blocking gRPC ships triggered by POST /signals and POST /cluster/heal are offloaded to a dedicated thread so they never block the axum reactor.
• ClusterState is constructed off the async reactor (GrpcTransport::new blocks on its own runtime).
• The experimental opt-in gate and docker/cluster/Dockerfile are updated: single-process cluster mode is honest that it replicates over real gRPC but runs all regions in one process (no host/process isolation). (True multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry above.)

Docker build fixes (all three images now that tidal-server pulls tidal-net)

• Install protobuf-compiler in the builder stage — tidal-net's build script runs tonic-build, which needs protoc to compile the WAL-shipping .proto.
• Pin the builder base to bookworm (rust:1.91-bookworm / rust:1.91-slim-bookworm) so its glibc matches the debian:bookworm-slim runtime; the default trixie base emitted a libmvec.so.1 dependency absent on bookworm, aborting the binary at startup. Verified: docker run of the cluster image serves a functional 3-region cluster (write replicates to both followers over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
• New tests: tidal-server/tests/cluster_grpc.rs (in-process gRPC replication + HTTP offload path) and a hardened tier-3 cluster_e2e.rs (multi-process smoke
  • promote over real OS processes, feature-gated).

[0.1.0] - 2026-02-23

Added

Core Database Engine

• TidalDb embeddable database with ephemeral() and with_data_dir() open modes
• SchemaBuilder for defining signal types, decay parameters, and ranking profiles
• TidalDbBuilder fluent builder with schema, data directory, metrics, and rate limiter configuration

Signal System

• Typed signal recording with exponential decay scoring
• Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
• Windowed aggregation: OneHour, TwentyFourHours, SevenDays, AllTime
• Signal velocity tracking
• WAL-backed signal durability with crash recovery
• Periodic signal checkpointing to fjall (every 30s)
• WAL compaction after each checkpoint

Retrieval (RETRIEVE query)

• 5-stage pipeline: universe, filter, score, diversify, return
• Filter expressions: Eq, In, Gt, Lt, And, Or, Not, InCollection, InProgress, MinSignal, MaxSignal, NearLocation
• Built-in ranking profiles: trending, for_you, new, popular, recent, and 20+ more
• Custom ranking profiles via SchemaBuilder
• Diversity enforcement (max N per category/creator)
• Sort modes: Relevance, Trending, Newest, MostLiked, MostViewed, MostFollowed, AlphabeticalAsc/Desc, Shortest/Longest, LiveViewerCount, DateSaved, and more

Search (SEARCH query)

• BM25 full-text search via Tantivy
• Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
• Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
• Creator search with entity_kind(EntityKind::Creator)
• similar_to(EntityId) for content-based recommendations
• Scope pre-filters: Trending, CohortTrending, Following, Category, Collection
• Autocomplete suggestions via db.suggest()

Entity Model

• Three built-in entity types: Item, User, Creator
• Metadata storage as HashMap<String, String>
• Embedding slots (up to 4 per entity type) via USearch
• Relationships: Follows, Blocks, Hide, Mute, InteractionWeight

Sessions

• Session lifecycle: open_session, close_session
• Cross-session preference vector updates (EMA blend)
• Session snapshots with signal state and preference vectors
• Session serialization format v0x03 with backward compatibility

Social Graph

• Creator follower/following indexes
• Cohort membership (user segments)
• CoEngagementIndex for co-viewing patterns with LRU eviction
• Social graph filter for "followed creator" content scoping

Collections

• Named collections with Private, Shared, Public visibility
• create_collection, add_to_collection, remove_from_collection, list_collections
• FilterExpr::InCollection for collection-scoped retrieval
• Saved searches with save_search, list_saved_searches, retrieve_saved_search

Observability

• enable_metrics(addr) -- Prometheus-format /metrics endpoint + /healthz JSON
• 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
• tidaldb_checkpoint_failures_total counter for checkpoint monitoring
• TidalDb::diagnostics() -- structured health snapshot
• WAL diagnostics and recovery tools

Safety

• Signal weight NaN/Inf validation (returns TidalError::InvalidInput)
• Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
• Export request limit: 500K signals max per request
• FilterExpr complexity limit: 256 nodes max
• Data directory lock (tidaldb.lock) prevents dual-process corruption
• Schema fingerprint persistence detects decay parameter changes on reopen
• Bounded closed_sessions cache (10K max, LRU eviction)
• Metrics server non-loopback bind warning

CLI (tidalctl)

• tidalctl binary for database inspection and diagnostics

RLHF / ML Export

• db.export_signals(ExportRequest) -- WAL-based signal export for training data
• db.user_session_summary(user_id, since_ns) -- aggregated session statistics

Stability

tidalDB 0.1.0 is pre-1.0. No API or data format stability guarantees are made for 0.x releases. Upgrade guides will be provided for each minor version bump. Do not upgrade 0.x to 0.y on a live data directory without reading the release notes.

---

Format based on Keep a Changelog