3bcfb3c576
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
128 lines
6.2 KiB
Markdown
128 lines
6.2 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to tidalDB will be documented in this file.
|
|
|
|
## [Unreleased]
|
|
|
|
### Changed
|
|
|
|
**Cluster mode (m8p8): real gRPC replication**
|
|
- `tidal-server`'s `ClusterState` now wires each follower region to a real
|
|
`tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
|
|
crossbeam channels — replication between regions traverses real gRPC/TCP
|
|
(serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
|
|
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
|
|
per region) and self-heal a transient bind race by retrying on a fresh port.
|
|
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
|
|
offloaded to a dedicated thread so they never block the axum reactor.
|
|
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
|
|
blocks on its own runtime).
|
|
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
|
|
cluster mode is honest that it replicates over real gRPC but still runs all
|
|
regions in one process (no host/process isolation — true multi-process
|
|
deployment remains m8p10).
|
|
|
|
**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
|
|
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
|
|
runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
|
|
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
|
|
`rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
|
|
runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
|
|
bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
|
|
image serves a functional 3-region cluster (write replicates to both followers
|
|
over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
|
|
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
|
|
HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
|
|
+ promote over real OS processes, feature-gated).
|
|
|
|
## [0.1.0] - 2026-02-23
|
|
|
|
### Added
|
|
|
|
**Core Database Engine**
|
|
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
|
|
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
|
|
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration
|
|
|
|
**Signal System**
|
|
- Typed signal recording with exponential decay scoring
|
|
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
|
|
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
|
|
- Signal velocity tracking
|
|
- WAL-backed signal durability with crash recovery
|
|
- Periodic signal checkpointing to fjall (every 30s)
|
|
- WAL compaction after each checkpoint
|
|
|
|
**Retrieval (RETRIEVE query)**
|
|
- 5-stage pipeline: universe, filter, score, diversify, return
|
|
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
|
|
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
|
|
- Custom ranking profiles via `SchemaBuilder`
|
|
- Diversity enforcement (max N per category/creator)
|
|
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more
|
|
|
|
**Search (SEARCH query)**
|
|
- BM25 full-text search via Tantivy
|
|
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
|
|
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
|
|
- Creator search with `entity_kind(EntityKind::Creator)`
|
|
- `similar_to(EntityId)` for content-based recommendations
|
|
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
|
|
- Autocomplete suggestions via `db.suggest()`
|
|
|
|
**Entity Model**
|
|
- Three built-in entity types: `Item`, `User`, `Creator`
|
|
- Metadata storage as `HashMap<String, String>`
|
|
- Embedding slots (up to 4 per entity type) via USearch
|
|
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`
|
|
|
|
**Sessions**
|
|
- Session lifecycle: `open_session`, `close_session`
|
|
- Cross-session preference vector updates (EMA blend)
|
|
- Session snapshots with signal state and preference vectors
|
|
- Session serialization format v0x03 with backward compatibility
|
|
|
|
**Social Graph**
|
|
- Creator follower/following indexes
|
|
- Cohort membership (user segments)
|
|
- CoEngagementIndex for co-viewing patterns with LRU eviction
|
|
- Social graph filter for "followed creator" content scoping
|
|
|
|
**Collections**
|
|
- Named collections with `Private`, `Shared`, `Public` visibility
|
|
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
|
|
- `FilterExpr::InCollection` for collection-scoped retrieval
|
|
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`
|
|
|
|
**Observability**
|
|
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
|
|
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
|
|
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
|
|
- `TidalDb::diagnostics()` -- structured health snapshot
|
|
- WAL diagnostics and recovery tools
|
|
|
|
**Safety**
|
|
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
|
|
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
|
|
- Export request limit: 500K signals max per request
|
|
- `FilterExpr` complexity limit: 256 nodes max
|
|
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
|
|
- Schema fingerprint persistence detects decay parameter changes on reopen
|
|
- Bounded `closed_sessions` cache (10K max, LRU eviction)
|
|
- Metrics server non-loopback bind warning
|
|
|
|
**CLI (`tidalctl`)**
|
|
- `tidalctl` binary for database inspection and diagnostics
|
|
|
|
**RLHF / ML Export**
|
|
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
|
|
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics
|
|
|
|
### Stability
|
|
|
|
tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.
|
|
|
|
---
|
|
|
|
*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*
|