jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening

Browse Source 
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build
- Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference
- Add docker standalone/cluster/deploy images, compose, and prometheus config
- Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry
- Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites
- Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
This commit is contained in:
jx12n 2026-06-07 18:29:38 -06:00
parent b16025b8b2
commit 3bcfb3c576
617 changed files with 132626 additions and 8839 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
.dockerignore Normal file
View File

@ -0,0 +1,25 @@
# Allowlist .dockerignore — ignore everything, then re-include only the
# workspace sources cargo needs to build `-p tidal-server` from the repo root.
# The `tidal/docker/*` Dockerfiles `COPY . .` against this repository root.
*
# Workspace manifests + lockfile (cargo needs every member manifest present to
# resolve the workspace graph, even when building a single member).
!Cargo.toml
!Cargo.lock
# Member crates
!tidal
!tidal-net
!tidal-server
!tidalctl
!applications
# Re-exclude heavy / generated / secret paths nested inside the allowed dirs.
**/target
**/node_modules
**/.next
**/*.tsbuildinfo
**/.env
**/.env.*
.git

77
Cargo.lock generated
View File

@ -358,6 +358,28 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.7.9"
@ -707,6 +729,15 @@ version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a822ea5bc7590f9d40f1ba12c0dc3c2760f3482c6984db1573ad11031420831"
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "codespan-reporting"
version = "0.13.1"
@ -1075,6 +1106,12 @@ version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75b325c5dbd37f80359721ad39aca5a29fb04c89279657cffdda8736d0c0b9d2"
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "either"
version = "1.15.0"
@ -1271,6 +1308,12 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures-channel"
version = "0.3.32"
@ -1941,6 +1984,16 @@ dependencies = [
"libc",
]
[[package]]
name = "libyml"
version = "0.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3302702afa434ffa30847a83305f0a69d6abd74293b6554c18ec85c7ef30c980"
dependencies = [
"anyhow",
"version_check",
]
[[package]]
name = "link-cplusplus"
version = "1.0.12"
@ -2901,6 +2954,7 @@ version = "0.23.36"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c665f33d38cea657d9614f766881e4d510e0eda4239891eea56b4cadcf01801b"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"ring",
@ -2947,6 +3001,7 @@ version = "0.103.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
@ -3108,16 +3163,18 @@ dependencies = [
]
[[package]]
name = "serde_yaml"
version = "0.9.34+deprecated"
name = "serde_yml"
version = "0.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
checksum = "59e2dd588bf1597a252c3b920e0143eb99b0f76e4e082f4c92ce34fbc9e71ddd"
dependencies = [
"indexmap 2.13.0",
"itoa",
"libyml",
"memchr",
"ryu",
"serde",
"unsafe-libyaml",
"version_check",
]
[[package]]
@ -3512,6 +3569,7 @@ dependencies = [
"criterion",
"prost",
"rcgen",
"rustls",
"rustls-pemfile",
"tempfile",
"thiserror 2.0.18",
@ -3529,12 +3587,14 @@ version = "0.1.0"
dependencies = [
"axum 0.8.8",
"clap",
"reqwest",
"serde",
"serde_json",
"serde_yaml",
"serde_yml",
"subtle",
"tempfile",
"thiserror 2.0.18",
"tidal-net",
"tidaldb",
"tokio",
"tower 0.5.3",
@ -3547,6 +3607,7 @@ dependencies = [
name = "tidalctl"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
"tidaldb",
]
@ -3976,12 +4037,6 @@ version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
[[package]]
name = "unsafe-libyaml"
version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
[[package]]
name = "untrusted"
version = "0.9.0"

126
applications/forage/engine/src/lib.rs
View File

@ -1640,6 +1640,69 @@ fn canonicalize_url(url: &str) -> String {
}
}
/// Round-robin interleave items by category to ensure the cold-start exploit
/// pool spans ≥3 categories. Preserves score ordering within each category.
///
/// Algorithm: group items by category (preserving input order), then pull
/// one item at a time from each group until `limit` items are collected.
fn interleave_categories(
items: Vec<tidaldb::query::retrieve::RetrieveResult>,
meta_map: &HashMap<u64, SeedItem>,
limit: usize,
) -> Vec<tidaldb::query::retrieve::RetrieveResult> {
use std::collections::BTreeMap;
// Group by category, preserving within-group order (best score first).
let mut by_cat: BTreeMap<String, Vec<tidaldb::query::retrieve::RetrieveResult>> =
BTreeMap::new();
let mut no_meta: Vec<tidaldb::query::retrieve::RetrieveResult> = Vec::new();
for item in items {
if let Some(seed) = meta_map.get(&item.entity_id.as_u64()) {
by_cat.entry(seed.category.clone()).or_default().push(item);
} else {
no_meta.push(item);
}
}
// Round-robin across categories (BTreeMap gives deterministic order).
let mut result = Vec::with_capacity(limit);
let mut iterators: Vec<std::vec::IntoIter<tidaldb::query::retrieve::RetrieveResult>> =
by_cat.into_values().map(|v| v.into_iter()).collect();
if iterators.is_empty() {
// All items lacked metadata; fall back to uncategorized list.
result.extend(no_meta.into_iter().take(limit));
return result;
}
let mut idx = 0;
while result.len() < limit {
let start = idx;
let mut advanced = false;
loop {
if let Some(item) = iterators[idx].next() {
result.push(item);
advanced = true;
}
idx = (idx + 1) % iterators.len();
if idx == start {
break;
}
}
if !advanced {
break; // all iterators exhausted
}
}
// Append any uncategorized items to fill if needed.
for item in no_meta {
if result.len() >= limit {
break;
}
result.push(item);
}
result
}
#[cfg(test)]
mod canon_tests {
use super::canonicalize_url;
@ -1706,66 +1769,3 @@ mod canon_tests {
);
}
}
/// Round-robin interleave items by category to ensure the cold-start exploit
/// pool spans ≥3 categories. Preserves score ordering within each category.
///
/// Algorithm: group items by category (preserving input order), then pull
/// one item at a time from each group until `limit` items are collected.
fn interleave_categories(
items: Vec<tidaldb::query::retrieve::RetrieveResult>,
meta_map: &HashMap<u64, SeedItem>,
limit: usize,
) -> Vec<tidaldb::query::retrieve::RetrieveResult> {
use std::collections::BTreeMap;
// Group by category, preserving within-group order (best score first).
let mut by_cat: BTreeMap<String, Vec<tidaldb::query::retrieve::RetrieveResult>> =
BTreeMap::new();
let mut no_meta: Vec<tidaldb::query::retrieve::RetrieveResult> = Vec::new();
for item in items {
if let Some(seed) = meta_map.get(&item.entity_id.as_u64()) {
by_cat.entry(seed.category.clone()).or_default().push(item);
} else {
no_meta.push(item);
}
}
// Round-robin across categories (BTreeMap gives deterministic order).
let mut result = Vec::with_capacity(limit);
let mut iterators: Vec<std::vec::IntoIter<tidaldb::query::retrieve::RetrieveResult>> =
by_cat.into_values().map(|v| v.into_iter()).collect();
if iterators.is_empty() {
// All items lacked metadata; fall back to uncategorized list.
result.extend(no_meta.into_iter().take(limit));
return result;
}
let mut idx = 0;
while result.len() < limit {
let start = idx;
let mut advanced = false;
loop {
if let Some(item) = iterators[idx].next() {
result.push(item);
advanced = true;
}
idx = (idx + 1) % iterators.len();
if idx == start {
break;
}
}
if !advanced {
break; // all iterators exhausted
}
}
// Append any uncategorized items to fill if needed.
for item in no_meta {
if result.len() >= limit {
break;
}
result.push(item);
}
result
}

131
docs/planning/ROADMAP.md
View File

@ -35,8 +35,8 @@ A single embeddable database can replace the 6-system content ranking stack by t
| M6 | Full Surface Coverage | Every use case, every sort mode, every filter, every feedback loop | UC-01 through UC-14 complete |
| M7 | Production Hardening | Crash safety, graceful degradation, operational readiness | All UCs at production quality |
| M8 | Distributed Fabric | Multi-region, multi-tenant replication keeps agent-memory semantics intact | Hosted tidalDB, cloud/edge deployments, shared agent substrate (in-process primitives COMPLETE; multi-node PARTIAL — transport + HTTP surface done, GrpcTransport wiring + tier-3 tests pending) |
| M9 | Community Sync & Revocation | Local embeddable profiles can opt into community personalization and safely leave/purge contributions | Community personalization, federated taste graphs, shared feeds |
| M10 | Governance & Agent Rights | Community rules and agent-scoped permissions control what signals influence ranking | User-owned AI personalization at scale, policy-compliant agents |
| M9 | Community Sync & Revocation | Local embeddable profiles can opt into community personalization and safely leave/purge contributions | Community personalization, federated taste graphs, shared feeds — ✅ COMPLETE (2026-06-06) |
| M10 | Governance & Agent Rights | Community rules and agent-scoped permissions control what signals influence ranking | User-owned AI personalization at scale, policy-compliant agents — ✅ COMPLETE (2026-06-06) |
### Embeddable → Distributed Path
@ -120,7 +120,7 @@ The roadmap now has two tracks:
| **m8p5: Control Plane + Multi-Tenancy + Routing** | COMPLETE | 1194 lib + 5 m8p5_multitenancy integration tests; TenantId/TenantConfig, TenantRateLimiter (AtomicU64 CAS token-bucket), TenantRouter (Jump Consistent Hash), ControlPlane (shard heartbeat + health), TenantMigration state machine, RollingUpgradeCoordinator |
| **m8p6: End-to-End UAT (In-Process)** | COMPLETE | 1,206 lib + 8 m8_uat tests (0.11s); SimulatedCluster (signal-replay harness, 3 regions), NetworkPartition/ShardCrash RAII fault injection, 5 UAT steps + 3 perf assertions; p99 replication < 2s, failover < 10s, CRDT reconciliation < 100ms |
| **m8p7: Network Transport (gRPC)** | COMPLETE | `tidal-net` crate: `GrpcTransport` implementing `Transport` trait via tonic 0.12; `GrpcTransportFactory`; per-peer circuit breaker (Closed/Open/HalfOpen); mutual TLS via rustls; proto `WalShipping` service (ShipSegment, StreamSegments, Heartbeat); 16 tests (contract, mTLS, reconnection); benchmark harness |
| **m8p8: Multi-Node tidal-server** | PARTIAL | `cluster` subcommand with topology YAML; `ClusterState` wrapping `SimulatedCluster` (in-process crossbeam channels — GrpcTransport not yet wired); cluster HTTP routes; region-aware reads; leader-routed writes; `docker/cluster/Dockerfile` rebuilt (ENTRYPOINT+CMD) |
| **m8p8: Multi-Node tidal-server** | COMPLETE | `cluster` subcommand with topology YAML; `ClusterState` wraps `SimulatedCluster` wired to **real `GrpcTransport`** per follower (self-loop over loopback gRPC; auto-allocated ports w/ bind-race retry); cluster HTTP routes; region-aware reads; leader-routed writes; blocking gRPC ships offloaded off the axum reactor; `cluster_grpc.rs` (2 in-process gRPC tests) + `cluster_e2e.rs` (tier-3 multi-process: smoke + promote); `docker/cluster/Dockerfile` opts in + runs. Single-process only (host/process isolation is m8p10) |
| **m8p9: Cross-Node Query Routing** | COMPLETE | `scatter_gather` module: entity-sharded writes via `hash(entity_id) % num_shards`; scatter-gather RETRIEVE/SEARCH across all shards; K-way merge by score; deadline propagation (50ms budget - 5ms overhead); partial failure with `degraded: true` + `unavailable_shards`; sharded HTTP routes (`/sharded/feed`, `/sharded/search`, `/sharded/items`, `/sharded/signals`) |
| **m8p10: Multi-Node UAT** | PARTIAL | Tier-2 (same-process, real gRPC): 8 tests in tidal-net/tests/multi_node_uat.rs — replication, idempotency, mixed signals, 3-node convergence, partition/heal, perf (100 signals in ~230ms). Tier-3 (multi-process harness with OS process isolation, iptables partition injection): NOT STARTED |
@ -140,13 +140,15 @@ The roadmap now has two tracks:
**M7 Production Hardening: COMPLETE** — m7p1–m7p4 + Enterprise Readiness all done. Crash recovery (BLAKE3 integrity, WAL compaction), 4-stage graceful degradation, per-agent rate limiting, session TTL sweeper, scale to 1M items, Prometheus metrics, tidalctl diagnostics, RLHF export.
**M8 Distributed Fabric: NEAR-COMPLETE** — In-process primitives (m8p1–p6) COMPLETE: shard routing, WAL shipping, CRDT reconciliation, session continuity, multi-tenancy, control plane. Multi-node (m8p7–p10) PARTIAL: `tidal-net` crate with `GrpcTransport` (tonic, mTLS, circuit breaker) COMPLETE; `tidal-server cluster` subcommand with HTTP routes, region-aware reads COMPLETE but uses in-process SimulatedCluster transport (GrpcTransport not yet wired — m8p8 gap); scatter-gather query routing COMPLETE; tier-2 gRPC UAT tests (8 tests, same-process) COMPLETE; tier-3 multi-process test harness NOT STARTED (m8p10 gap). 1,209 lib + 22 tidal-net + 23 tidal-server tests passing.
**M8 Distributed Fabric: NEAR-COMPLETE** — In-process primitives (m8p1–p6) COMPLETE: shard routing, WAL shipping, CRDT reconciliation, session continuity, multi-tenancy, control plane. Multi-node (m8p7–p9) COMPLETE: `tidal-net` crate with `GrpcTransport` (tonic, mTLS, circuit breaker); `tidal-server cluster` subcommand with HTTP routes + region-aware reads, now wired to **real `GrpcTransport`** for replication (m8p8 — `ClusterState` builds a self-loop gRPC transport per follower; the in-process crossbeam path is gone); scatter-gather query routing. m8p10 PARTIAL: tier-2 gRPC UAT (8 same-process tests) + an in-process `cluster_grpc.rs` HTTP-path test + a tier-3 multi-process `cluster_e2e.rs` (smoke + promote over OS processes, feature-gated) COMPLETE; remaining tier-3 work (iptables partition injection, clock-skew, rolling-upgrade) NOT STARTED. 1,209 lib + 22 tidal-net + 25 tidal-server tests passing.
**Forage: COMPLETE** — All 5 phases done (P0: demo loop, P1: real signal surface, P2: semantic embeddings, P3: adaptive MAB, P4: bridge/surprise moment). Chrome extension + forage-server + forage-engine + forage-embedder sidecar all operational.
**iknowyou / Aeries: IN PROGRESS (as of 2026-02-24)** — M1–M4 complete. M5 (Communication Brief) is in progress with core implementation live; acceptance validation pending.
**Next (engine):** M9 — Community Sync & Revocation (now unblocked with M8 complete). M10 — Governance & Agent Rights.
**Engine status:** M0–M10 **COMPLETE**. M9 (Community Sync & Revocation) and M10 (Governance & Agent Rights) shipped 2026-06-06 — see *Implementation Status (M9–M10 as-built)* below.
**Next (engine):** the cross-cutting community-overlay UAT surface — a single RETRIEVE that blends local + community layers and folds membership-epoch / policy-version / purge-watermark inline into `Results.policy_metadata` (the per-phase mechanisms and direct read APIs all exist; this is the unifying query surface). Then a multi-node M9/M10 UAT harness, and the deferred `writer_agent` u16 interning on the WAL v3 envelope.
**Next (product):** iknowyou M5 acceptance pass, then M6 Closed Loop (session lifecycle + preference drift validation).
---
@ -2761,12 +2763,19 @@ These phases take the proven in-process primitives and deliver actual multi-node
**Acceptance Criteria:**
- [ ] `tidal-server cluster` starts with 3-region topology YAML; all nodes connect and report healthy
- [ ] `GET /cluster/status` returns leader identity, per-region applied events, lag, and partition status
- [ ] `POST /signals` to a follower returns success; signal appears on leader and all followers within 5s
- [ ] `POST /cluster/promote` changes leader; subsequent writes route to new leader
- [ ] `?region=eu-west` on `/feed` returns results from the eu-west follower
- [ ] `docker/cluster/Dockerfile` builds and runs a functional 3-region cluster
- [x] `tidal-server cluster` starts with 3-region topology YAML; all nodes connect (real gRPC) and report healthy
- [x] `GET /cluster/status` returns leader identity, per-region applied events, lag, and partition status
- [x] `POST /signals` returns success; signal appears on all followers within 5s **over real gRPC** (`cluster_grpc.rs`, `cluster_e2e.rs` smoke)
- [x] `POST /cluster/promote` changes leader; subsequent writes route to new leader (`cluster_e2e.rs` promote)
- [x] `?region=eu-west` on `/feed` returns results from the eu-west follower
- [x] `docker/cluster/Dockerfile` opts in (`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) and runs a functional 3-region cluster
> **Resolved (m8p8):** `ClusterState` now wires each follower to a real
> `tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of crossbeam
> channels — closing gap G1. Blocking gRPC ships are offloaded off the axum
> reactor; auto-allocated gRPC ports self-heal a transient bind race. All regions
> still share one process: true multi-process/host isolation (process-crash,
> partition-injection, rolling-upgrade) remains tier-3 / m8p10.
**Depends On:** Phase 7 (network transport)
**Complexity:** XL
@ -2870,8 +2879,8 @@ These phases take the proven in-process primitives and deliver actual multi-node
| Gap | Phase | Severity | Description |
|-----|-------|----------|-------------|
| G1 | m8p8 | Medium | `ClusterState` wraps `SimulatedCluster` using crossbeam channels; spec requires `GrpcTransport` from `tidal-net` for real network replication between cluster nodes |
| G2 | m8p10 | Medium | Tier-3 multi-process test harness not built; tier-2 (same-process gRPC) covers transport correctness but not process crash, network partition injection, or rolling upgrade |
| ~~G1~~ | m8p8 | ~~Medium~~ RESOLVED | `ClusterState` now wires each follower to a real `tidal-net` `GrpcTransport` (self-loop over loopback gRPC), replacing the crossbeam path. Replication between regions traverses real gRPC/TCP. Verified by `cluster_grpc.rs` (in-process) and `cluster_e2e.rs` (multi-process). |
| G2 | m8p10 | Medium | Tier-3 multi-process harness exists (`cluster_e2e.rs`: smoke + promote over OS processes), but iptables/pfctl partition injection, clock-skew, and rolling-upgrade scenarios are still NOT STARTED |
| G3 | m8p9 | Low | `entity_shard()` uses Knuth multiplicative hash while `TenantRouter` uses Jump Consistent Hash for the same entity→shard routing problem; must unify before entity-sharded production |
### Done When (M8 Full)
@ -2888,7 +2897,7 @@ The cluster runbook (`docs/runbooks/cluster.md`) is fully operational against re
---
## Milestone 9: Community Sync & Revocation -- "Join, share, leave, purge"
## Milestone 9: Community Sync & Revocation -- "Join, share, leave, purge" — ✅ COMPLETE (2026-06-06)
### Milestone Thesis
@ -2926,10 +2935,10 @@ Then:
**Acceptance Criteria:**
- [ ] WAL event envelope carries `scope`, `origin`, `share_policy_version`, and idempotency key.
- [ ] Default behavior is local-only; sharing is explicit opt-in.
- [ ] Per-intent share filters supported (`skip_for_now`, `not_for_me`, `low_quality`, `hide/mute/block`, etc.).
- [ ] `tidalctl` can inspect scope distribution and share eligibility.
- [x] WAL event envelope carries `scope`, `origin`, `share_policy_version`, and idempotency key.
- [x] Default behavior is local-only; sharing is explicit opt-in.
- [x] Per-intent share filters supported (`skip_for_now`, `not_for_me`, `low_quality`, `hide/mute/block`, etc.).
- [x] `tidalctl` can inspect scope distribution and share eligibility.
**Depends On:** M8
**Complexity:** L
@ -2940,10 +2949,10 @@ Then:
**Acceptance Criteria:**
- [ ] Membership states: `joined`, `leaving_stop_forward`, `left`, `rejoined`.
- [ ] `leave(stop_forward)` blocks new community contributions in < 1s p99.
- [ ] Rejoin creates a new membership epoch (no ambiguous replay across epochs).
- [ ] Queries expose active membership epoch for debugging and explainability.
- [x] Membership states: `joined`, `leaving_stop_forward`, `left`, `rejoined`.
- [x] `leave(stop_forward)` blocks new community contributions in < 1s p99.
- [x] Rejoin creates a new membership epoch (no ambiguous replay across epochs).
- [x] Queries expose active membership epoch for debugging and explainability.
**Depends On:** Phase 1
**Complexity:** L
@ -2954,10 +2963,10 @@ Then:
**Acceptance Criteria:**
- [ ] `purge_prior_contributions(user_id, community_id, epoch_range)` API implemented.
- [ ] Purge writes tombstones and triggers deterministic rematerialization job.
- [ ] Rebuilt aggregates are identical across repeated replay of same purge operation.
- [ ] Community queries include purge watermark metadata for auditability.
- [x] `purge_prior_contributions(user_id, community_id, epoch_range)` API implemented.
- [x] Purge writes tombstones and triggers deterministic rematerialization job.
- [x] Rebuilt aggregates are identical across repeated replay of same purge operation.
- [x] Community queries include purge watermark metadata for auditability.
**Depends On:** Phase 2
**Complexity:** XL
@ -2968,7 +2977,7 @@ Users can opt into community personalization, leave safely, and purge prior cont
---
## Milestone 10: Governance & Agent Rights -- "Who can influence ranking, and how"
## Milestone 10: Governance & Agent Rights -- "Who can influence ranking, and how" — ✅ COMPLETE (2026-06-06)
### Milestone Thesis
@ -3005,10 +3014,10 @@ Then:
**Acceptance Criteria:**
- [ ] Policy schema includes allowed intents, excluded intents, and weighting constraints.
- [ ] Policy changes are versioned and applied with effective timestamps.
- [ ] Query results can return governing policy version for explanation.
- [ ] Out-of-policy signals are rejected or quarantined by rule.
- [x] Policy schema includes allowed intents, excluded intents, and weighting constraints.
- [x] Policy changes are versioned and applied with effective timestamps.
- [x] Query results can return governing policy version for explanation.
- [x] Out-of-policy signals are rejected or quarantined by rule.
**Depends On:** M9
**Complexity:** L
@ -3019,10 +3028,10 @@ Then:
**Acceptance Criteria:**
- [ ] Agent capability tokens include scope permissions and TTL.
- [ ] Reads/writes outside granted scope return policy errors and audit events.
- [ ] Revocation invalidates capabilities immediately (< 1s p99).
- [ ] `tidalctl` can inspect agent capabilities and revocation history.
- [x] Agent capability tokens include scope permissions and TTL.
- [x] Reads/writes outside granted scope return policy errors and audit events.
- [x] Revocation invalidates capabilities immediately (< 1s p99).
- [x] `tidalctl` can inspect agent capabilities and revocation history.
**Depends On:** Phase 1
**Complexity:** L
@ -3033,10 +3042,10 @@ Then:
**Acceptance Criteria:**
- [ ] Every ranking-affecting signal has provenance metadata (`writer`, `scope`, `policy_version`, `membership_epoch`).
- [ ] `remove_from_personalization(scope=...)` API supports precise, non-global deletion.
- [ ] Explainability endpoint can attribute top-ranked items to policy-allowed signals.
- [ ] Replay/failover preserves remove-by-scope outcomes deterministically.
- [x] Every ranking-affecting signal has provenance metadata (`writer`, `scope`, `policy_version`, `membership_epoch`).
- [x] `remove_from_personalization(scope=...)` API supports precise, non-global deletion.
- [x] Explainability endpoint can attribute top-ranked items to policy-allowed signals.
- [x] Replay/failover preserves remove-by-scope outcomes deterministically.
**Depends On:** Phase 2
**Complexity:** XL
@ -3047,6 +3056,46 @@ Communities and users can control ranking influence with explicit governance and
---
## Implementation Status (M9–M10 as-built) — 2026-06-06
M9 and M10 are implemented in the `tidaldb` engine crate under a new
`tidal/src/governance/` module, plus `db/communities.rs`, `db/capabilities.rs`,
`db/remove_scope.rs`, WAL **format v3**, and the storage `Tag` band `0x0E–0x13`.
| Phase | Status | Key surfaces |
|-------|--------|--------------|
| m9p1 Signal Scope & Share Contract | ✅ | `SignalScope`/`CommunityId`/`SignalProvenance`/`SharePolicy`; WAL v3 `EventRecord` (21→32B, v1/v2 still decode); `TidalDb::signal_scoped`; shipper `community_share_only` filter; `tidalctl scope-stats` |
| m9p2 Membership Lifecycle & Stop-Forward | ✅ | join/leave/rejoin, O(1) atomic stop-forward gate, epochs, `Tag::Membership` persist, gated `signal_for_community` |
| m9p3 Retroactive Purge & Rematerialization | ✅ | per-contributor `CommunityLedger`, `purge_prior_contributions`, deterministic rematerialization (BLAKE3-digest tested), tombstones + watermark, reopen survival |
| m10p1 Community Governance Policy Engine | ✅ | versioned `GovernancePolicy` (monotonic + effective ts, weighting bounds, intent eligibility), write-path enforcement, persist |
| m10p2 Agent Capability & Scope Controls | ✅ | `CapabilityToken`/`ScopeClass`, grant/revoke, atomic <1s revocation, TTL, `agent_signal_for_community`, audit + persist |
| m10p3 Provenance, Explainability, Remove-by-Scope | ✅ | writer-agent provenance, `explain_community_contributions`, `remove_from_personalization` (by user / by agent) |
**Verification:** `tidaldb` lib 1,311 unit tests + 30 new M9/M10 integration tests
(`tests/m9p1_scopes.rs`, `m9p2_membership.rs`, `m9p3_purge.rs`,
`m10p1_governance.rs`, `m10p2_capabilities.rs`, `m10p3_provenance.rs`) pass;
`cargo clippy -p tidaldb --lib -- -D warnings` clean; `cargo build --workspace` green.
**Known deltas vs. the per-phase acceptance text:**
1. **Query metadata.** `Results.policy_metadata` (membership_epoch / purge_watermark /
community_id / policy_version / effective_at_ns) and `QueryStats.membership_epoch`
exist and default cleanly, and every value is reachable via direct read APIs
(`membership_epoch`, `governing_policy_version`, `community_purge_watermark`,
`read_community_decay_score`/`read_community_windowed_count`,
`explain_community_contributions`). Folding these *inline into a single blended
local+community RETRIEVE* is the community-overlay UAT capstone (see *Next*), not
yet wired into the RETRIEVE executor.
2. **tidalctl.** `scope-stats` (WAL-based, lock-free) shipped. Capability/revocation
inspection is via the DB API (`agent_capabilities`, `revocation_history`) rather
than `tidalctl` subcommands, because `tidalctl` is intentionally lock-free and
does not open the items keyspace (consistent with its existing `diagnostics` design).
3. **WAL v3 `writer_agent`.** The community ledger carries the full writer-agent
string for remove-by-agent; the WAL v3 envelope's `writer_agent: u16` is reserved
(interning deferred). Provenance is authoritative in the ledger.
---
## Use Case Coverage Progression
| UC | Description | M1 | M2 | M3 | M4 | M5 | M6 | M7 |
@ -3148,8 +3197,8 @@ m1p1 (Types/Schema) ✓
|
+---> m8p10 (Multi-Node UAT) ✓
M9 phases depend on M8p10 (now unblocked)
M10 phases depend on M9
M9 ✓ COMPLETE (m9p1 scope+share, m9p2 membership+stop-forward, m9p3 purge+rematerialize)
M10 ✓ COMPLETE (m10p1 governance policy, m10p2 agent capabilities, m10p3 provenance+remove-by-scope)
```
**Parallelization opportunities:**

74
tidal-net/BUILD.bazel Normal file
View File

@ -0,0 +1,74 @@
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_proc_macro", "rust_test")
load("@rules_rust//cargo:defs.bzl", "cargo_build_script")
load("@crates//:defs.bzl", "aliases", "all_crate_deps")
load("//bazel:rust_cargo_tests.bzl", "rust_cargo_integration_tests")
cargo_build_script(
name = "build_script",
srcs = [
"build.rs",
],
crate_root = "build.rs",
data = [
"proto/wal_shipping.proto",
],
edition = "2024",
aliases = aliases(build = True),
deps = all_crate_deps(build = True),
proc_macro_deps = all_crate_deps(build_proc_macro = True),
visibility = ["//visibility:private"],
)
rust_library(
name = "tidal_net",
srcs = glob(["src/**/*.rs", "src/*.rs"], allow_empty = True),
crate_root = "src/lib.rs",
crate_name = "tidal_net",
edition = "2024",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(),
deps = all_crate_deps(normal = True) + [
"//tidal:tidaldb",
":build_script",
],
proc_macro_deps = all_crate_deps(proc_macro = True),
visibility = ["//visibility:public"],
)
rust_test(
name = "tidal_net_test",
crate = ":tidal_net",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True),
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
)
rust_cargo_integration_tests(
name_prefix = "tidal_net",
tests = glob(["tests/*.rs"]),
srcs = glob(["tests/**/*.rs"]),
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True) + [
"//tidal:tidaldb",
":tidal_net",
],
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
# mtls.rs does a real mTLS gRPC send/receive over a socket -> requires-network (excluded from the
# default `bazel test //...`, like other network-bound integration tests).
# KNOWN DEFECT it surfaced: tidal-net enables NO rustls crypto provider in its own dep closure
# (cargo tree shows rustls gets only "log" via tonic). Under `cargo test` it works ONLY because the
# whole-workspace shared rustls inherits aws-lc-rs from another workspace crate's reqwest;
# crate_universe resolves the tidal closure narrowly, so the provider is absent and rustls panics
# ("Could not automatically determine the process-level CryptoProvider"). This is a latent fragility
# even in cargo: a standalone tidal-server build would break mTLS at runtime. PROPER FIX (out of
# this build-only review's scope — touches the production crypto path + needs a lock repin): make
# the provider explicit in tidal-net (direct `rustls = { features = ["aws_lc_rs"] }` dep OR
# CryptoProvider::install_default() at startup), then CARGO_BAZEL_REPIN.
requires_network = [
"tests/mtls.rs",
],
)

30
tidal-net/Cargo.toml
View File

@ -6,6 +6,29 @@ rust-version.workspace = true
license.workspace = true
description = "gRPC network transport for tidalDB WAL segment shipping"
# ── tidal-crate lint posture (single source of truth) ──────────────────────
# IDENTICAL block across tidaldb / tidal-net / tidal-server / tidalctl. These
# crates deliberately DO NOT inherit `[workspace.lints]`; they hold the embedded
# recommendation DB + its transport/server/CLI to a stricter correctness bar
# (`unsafe_code = forbid`, `clippy::all = deny`, `unwrap_used = deny`). A network
# transport crate especially must gate unsafe + unwrap.
# `unwrap_used = "deny"` is kept per-crate rather than in `[workspace.lints]`
# because the workspace also hosts the example/consumer crates under
# `applications/` (not held to the engine's bar). Keep these four blocks BYTE-IDENTICAL.
[lints.rust]
unsafe_code = "forbid"
[lints.clippy]
all = { level = "deny", priority = -1 }
pedantic = { level = "warn", priority = -1 }
nursery = { level = "warn", priority = -1 }
# Justified allows (lossy numeric casts are pervasive + intentional in the
# ranking/scoring math; module_name_repetitions is idiomatic for the flat
# module layout documented in CLAUDE.md):
cast_possible_truncation = "allow"
module_name_repetitions = "allow"
unwrap_used = "deny"
[dependencies]
tidaldb = { path = "../tidal" }
tonic = { version = "0.12", features = ["tls", "tls-roots"] }
@ -13,6 +36,13 @@ prost = "0.13"
tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] }
tokio-stream = "0.1"
rustls-pemfile = "2"
# Direct rustls dep with an explicit crypto provider. tonic pulls rustls only
# transitively (no provider feature in tidal-net's own closure), so the
# process-level default CryptoProvider is otherwise absent/ambiguous and the
# mTLS path panics ("Could not automatically determine the process-level
# CryptoProvider"). `transport::ensure_crypto_provider` installs this aws-lc-rs
# provider idempotently at GrpcTransport construction. See BUILD.bazel.
rustls = { version = "0.23", features = ["aws_lc_rs"] }
tracing = "0.1"
thiserror = "2"

24
tidal-net/benches/transport_throughput.rs
View File

@ -1,21 +1,19 @@
//! Benchmark comparing InProcessTransport vs GrpcTransport throughput.
// Benchmark harness: unwrap on known-good setup is idiomatic here.
#![allow(clippy::unwrap_used)]
//! Benchmark comparing `InProcessTransport` vs `GrpcTransport` throughput.
//!
//! Measures segments/sec for both transports to quantify gRPC overhead.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::thread;
use std::time::Duration;
use std::{collections::HashMap, net::SocketAddr, thread, time::Duration};
use criterion::{Criterion, Throughput, criterion_group, criterion_main};
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::in_process::InProcessTransportFactory;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::{Transport, WalSegmentPayload};
use tidal_net::GrpcTransport;
use tidal_net::config::GrpcTransportConfig;
use tidal_net::{GrpcTransport, config::GrpcTransportConfig};
use tidaldb::replication::{
WalSegmentId,
in_process::InProcessTransportFactory,
shard::{RegionId, ShardId},
transport::{Transport, WalSegmentPayload},
};
const SEGMENT_SIZE: usize = 1024; // 1 KB payload per segment

25
tidal-net/src/circuit_breaker.rs
View File

@ -1,11 +1,15 @@
//! Per-peer circuit breaker for gRPC connections.
//!
//! State machine: Closed → Open → HalfOpen → Closed.
//! State machine: Closed → Open → `HalfOpen` → Closed.
//! After `threshold` consecutive failures, the breaker opens for `reset_duration`.
//! The first call after the reset period transitions to HalfOpen and allows one probe.
//! The first call after the reset period transitions to `HalfOpen` and allows one probe.
//!
//! Struct-shaped error: `CircuitOpenError` carries no fields and represents the single "breaker is open" outcome; an enum with one variant would be ceremony with no readability gain.
use std::sync::Mutex;
use std::time::{Duration, Instant};
use std::{
sync::Mutex,
time::{Duration, Instant},
};
/// Error returned when the circuit breaker is open and not yet ready to probe.
#[derive(Debug, Clone, thiserror::Error)]
@ -28,7 +32,8 @@ enum CircuitState {
impl CircuitBreaker {
/// Create a new circuit breaker.
pub fn new(threshold: u32, reset_duration: Duration) -> Self {
#[must_use]
pub const fn new(threshold: u32, reset_duration: Duration) -> Self {
Self {
state: Mutex::new(CircuitState::Closed {
consecutive_failures: 0,
@ -40,9 +45,13 @@ impl CircuitBreaker {
/// Check if a request is allowed.
///
/// Returns `Ok(())` if the breaker is closed or transitioning to half-open.
/// Returns `Err(CircuitOpenError)` if the breaker is open and the reset period has not elapsed.
/// Returns `Ok(())` on lock poisoning (fail-open: allow the request through).
/// Returns `Ok(())` if the breaker is closed or transitioning to half-open,
/// and `Ok(())` on lock poisoning (fail-open: allow the request through).
///
/// # Errors
///
/// Returns [`CircuitOpenError`] if the breaker is open and the reset period
/// has not yet elapsed.
pub fn check(&self) -> Result<(), CircuitOpenError> {
let Ok(mut state) = self.state.lock() else {
tracing::warn!("circuit breaker lock poisoned; failing open");

80
tidal-net/src/client.rs
View File

@ -2,17 +2,16 @@
use std::collections::HashMap;
use tidaldb::replication::{shard::ShardId, transport::WalSegmentPayload};
use tonic::transport::Channel;
use tidaldb::replication::shard::ShardId;
use tidaldb::replication::transport::WalSegmentPayload;
use crate::circuit_breaker::CircuitBreaker;
use crate::config::GrpcTransportConfig;
use crate::error::GrpcTransportError;
use crate::proto::ShipSegmentRequest;
use crate::proto::wal_shipping_client::WalShippingClient;
use crate::tls;
use crate::{
circuit_breaker::CircuitBreaker,
config::GrpcTransportConfig,
error::GrpcTransportError,
proto::{ShipSegmentRequest, wal_shipping_client::WalShippingClient},
tls,
};
/// A connection to a single peer shard.
struct PeerConnection {
@ -27,6 +26,11 @@ pub struct PeerPool {
impl PeerPool {
/// Build a pool with lazy connections to all configured peers.
///
/// # Errors
///
/// Returns [`GrpcTransportError`] if a peer URI is invalid or the client TLS
/// configuration cannot be built.
pub fn new(config: &GrpcTransportConfig) -> Result<Self, GrpcTransportError> {
let mut peers = HashMap::new();
@ -38,24 +42,39 @@ impl PeerPool {
};
let uri = format!("{scheme}://{addr}");
// Build the endpoint with hard connect/request timeouts and HTTP/2
// keep-alive so a stalled peer fails fast (transient error → circuit
// breaker) instead of hanging the single-threaded WAL shipper. These
// apply on both the plaintext and TLS paths.
let mut endpoint = Channel::from_shared(uri)
.map_err(|e| GrpcTransportError::Internal(format!("invalid URI: {e}")))?
.connect_lazy();
.connect_timeout(config.connect_timeout)
.timeout(config.request_timeout)
.tcp_keepalive(Some(config.keep_alive_interval))
.http2_keep_alive_interval(config.keep_alive_interval)
.keep_alive_timeout(config.keep_alive_timeout)
.keep_alive_while_idle(true);
// Configure client TLS if available.
if let Some(ref tls_config) = config.tls {
let client_tls = tls::client_tls_config(tls_config)?;
// We need to rebuild the endpoint with TLS. tonic's connect_lazy
// doesn't support post-hoc TLS config, so rebuild.
let uri_str = format!("{scheme}://{addr}");
endpoint = Channel::from_shared(uri_str)
.map_err(|e| GrpcTransportError::Internal(format!("invalid URI: {e}")))?
endpoint = endpoint
.tls_config(client_tls)
.map_err(GrpcTransportError::TonicTransport)?
.connect_lazy();
.map_err(GrpcTransportError::TonicTransport)?;
}
let client = WalShippingClient::new(endpoint);
let channel = endpoint.connect_lazy();
// Raise tonic's default 4 MiB codec limits to the configured max
// payload on BOTH directions. The client is the ENCODER for the
// outbound ShipSegmentRequest (and the DECODER for any future
// server-streaming response). Without this the client encoder trips
// FIRST on a full-size WAL segment (16 MiB default, up to 64 MiB)
// before bytes ever leave the process, surfacing as a codec error
// the shipper would retry forever. Both ends must agree on the same
// ceiling the application already validates against.
let client = WalShippingClient::new(channel)
.max_encoding_message_size(config.max_payload_bytes)
.max_decoding_message_size(config.max_payload_bytes);
let circuit_breaker = CircuitBreaker::new(
config.circuit_breaker_threshold,
config.circuit_breaker_reset,
@ -74,6 +93,16 @@ impl PeerPool {
}
/// Send a WAL segment to a peer shard.
///
/// # Errors
///
/// Returns [`GrpcTransportError`] if the peer is unknown, the circuit breaker
/// is open, the follower signals backpressure, or the gRPC call fails.
// `peer` is a plain `&PeerConnection` borrow into `self.peers` (no Drop of
// its own) used both before the await (clone the client) and after (record
// success/failure on its circuit breaker), so it legitimately spans the
// whole fn — significant_drop_tightening is a false positive for a borrow.
#[allow(clippy::significant_drop_tightening)]
pub async fn send_to(
&self,
shard: ShardId,
@ -95,9 +124,18 @@ impl PeerPool {
let mut client = peer.client.clone();
match client.ship_segment(request).await {
Ok(_response) => {
peer.circuit_breaker.record_success();
Ok(())
Ok(response) => {
// The follower signals backpressure with `accepted=false` when
// its inbound channel is full: the segment was NOT enqueued.
// Treat this as a transient failure so the shipper does not
// advance its cursor and retries on the next poll.
if response.into_inner().accepted {
peer.circuit_breaker.record_success();
Ok(())
} else {
peer.circuit_breaker.record_failure();
Err(GrpcTransportError::SegmentRejected(shard))
}
}
Err(status) => {
peer.circuit_breaker.record_failure();

28
tidal-net/src/config.rs
View File

@ -1,9 +1,6 @@
//! Configuration for the gRPC transport.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::time::Duration;
use std::{collections::HashMap, net::SocketAddr, path::PathBuf, time::Duration};
use tidaldb::replication::shard::ShardId;
@ -17,7 +14,7 @@ pub struct GrpcTransportConfig {
pub local_shard: ShardId,
/// Address to bind the gRPC server on.
pub listen_addr: SocketAddr,
/// Peer shard addresses (shard_id -> address).
/// Peer shard addresses (`shard_id` -> address).
pub peers: HashMap<ShardId, SocketAddr>,
/// TLS configuration. `None` requires `insecure = true`.
pub tls: Option<TlsConfig>,
@ -31,6 +28,23 @@ pub struct GrpcTransportConfig {
pub circuit_breaker_threshold: u32,
/// Duration the circuit breaker stays open before allowing a probe.
pub circuit_breaker_reset: Duration,
/// Maximum time to establish a TCP/TLS connection to a peer before the
/// attempt fails fast (and the failure trips the circuit breaker). Without
/// this, a peer whose host is up but whose port silently blackholes the
/// SYN would stall the single-threaded shipper indefinitely.
pub connect_timeout: Duration,
/// Maximum time for a single `ship_segment` RPC (connect + send + response)
/// before it is aborted as a transient failure. Bounds the per-call latency
/// the shipper can ever incur on a stalled peer.
pub request_timeout: Duration,
/// HTTP/2 PING keep-alive interval on the client channel. Idle and active
/// connections send a PING this often; a peer that stops responding is
/// detected within `keep_alive_interval + keep_alive_timeout` instead of
/// hanging on a half-open connection.
pub keep_alive_interval: Duration,
/// How long to wait for a keep-alive PING ACK before declaring the
/// connection dead and tearing it down.
pub keep_alive_timeout: Duration,
}
impl Default for GrpcTransportConfig {
@ -45,6 +59,10 @@ impl Default for GrpcTransportConfig {
max_payload_bytes: MAX_PAYLOAD_BYTES,
circuit_breaker_threshold: 5,
circuit_breaker_reset: Duration::from_secs(30),
connect_timeout: Duration::from_secs(5),
request_timeout: Duration::from_secs(10),
keep_alive_interval: Duration::from_secs(10),
keep_alive_timeout: Duration::from_secs(5),
}
}
}

9
tidal-net/src/convert.rs
View File

@ -1,8 +1,10 @@
//! Conversions between protobuf types and tidalDB domain types.
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::WalSegmentPayload;
use tidaldb::replication::{
WalSegmentId,
shard::{RegionId, ShardId},
transport::WalSegmentPayload,
};
use crate::proto;
@ -44,6 +46,7 @@ impl TryFrom<proto::ShipSegmentRequest> for WalSegmentPayload {
}
#[cfg(test)]
#[allow(clippy::unwrap_used)] // test assertions on known-good fixtures
mod tests {
use super::*;

215
tidal-net/src/error.rs
View File

@ -1,19 +1,10 @@
//! Error types for the gRPC transport layer.
use tidaldb::replication::shard::ShardId;
use tidaldb::replication::transport::TransportError;
use tidaldb::replication::{shard::ShardId, transport::TransportError};
/// Errors specific to the gRPC transport.
#[derive(Debug, thiserror::Error)]
pub enum GrpcTransportError {
/// Failed to connect to a peer shard.
#[error("connection to peer {shard} at {addr} failed: {reason}")]
ConnectionFailed {
shard: ShardId,
addr: String,
reason: String,
},
/// Circuit breaker is open for this peer.
#[error("circuit breaker open for peer {0}; will retry after reset period")]
CircuitOpen(ShardId),
@ -26,9 +17,11 @@ pub enum GrpcTransportError {
#[error("TLS configuration error: {0}")]
TlsConfig(String),
/// Inbound channel is full; segment was dropped.
#[error("inbound channel full ({capacity} segments); segment dropped")]
ChannelFull { capacity: usize },
/// The follower replied `accepted=false` (its inbound channel was full).
/// This is follower backpressure: the segment was NOT durably enqueued, so
/// the shipper must treat it as un-shipped and retry on the next poll.
#[error("peer {0} rejected segment (accepted=false); follower backpressure")]
SegmentRejected(ShardId),
/// Internal transport error.
#[error("internal transport error: {0}")]
@ -45,23 +38,195 @@ pub enum GrpcTransportError {
impl From<tonic::Status> for GrpcTransportError {
fn from(status: tonic::Status) -> Self {
GrpcTransportError::Grpc(Box::new(status))
Self::Grpc(Box::new(status))
}
}
impl GrpcTransportError {
/// Whether this failure is *permanent* — re-shipping the same segment will
/// keep failing identically until an operator changes configuration.
///
/// The shipper's retry loop is correct ONLY for transient failures (peer
/// down, circuit open, follower backpressure): it leaves the per-peer HWM
/// untouched and re-sends the same seqno next poll. For a permanent failure
/// that retry is an infinite silent stall — a mutually-distrusted TLS chain,
/// a misconfigured CA, an authentication rejection, or a payload the wire
/// codec will never accept does not become shippable by trying again.
///
/// Classification:
/// - [`TlsConfig`](Self::TlsConfig): local TLS/config material is bad →
/// permanent.
/// - [`Grpc`](Self::Grpc): permanent iff the gRPC status code is one the
/// server will return identically on retry — `Unauthenticated` /
/// `PermissionDenied` (mTLS / authz rejection), `InvalidArgument` /
/// `OutOfRange` (the payload itself is malformed), `Unimplemented` (the
/// RPC does not exist on the peer). Everything else (`Unavailable`,
/// `ResourceExhausted`, `DeadlineExceeded`, codec-size on a transient
/// over-large segment, …) is transient.
/// - [`TonicTransport`](Self::TonicTransport): a connect/handshake error.
/// These are predominantly transient (peer not up yet) and are treated as
/// such; a genuinely permanent handshake fault surfaces as a `Grpc` TLS
/// status on the next attempt.
/// - [`PeerUnreachable`](Self::PeerUnreachable): routing error (unknown
/// shard) — handled separately via `UnknownPeer`, never as a retry.
/// - All other variants (circuit open, segment rejected, internal): transient.
#[must_use]
pub fn is_permanent(&self) -> bool {
match self {
Self::TlsConfig(_) => true,
Self::Grpc(status) => matches!(
status.code(),
tonic::Code::Unauthenticated
| tonic::Code::PermissionDenied
| tonic::Code::InvalidArgument
| tonic::Code::OutOfRange
| tonic::Code::Unimplemented
),
Self::CircuitOpen(_)
| Self::PeerUnreachable(_)
| Self::SegmentRejected(_)
| Self::Internal(_)
| Self::TonicTransport(_) => false,
}
}
}
impl From<GrpcTransportError> for TransportError {
fn from(e: GrpcTransportError) -> Self {
match e {
GrpcTransportError::PeerUnreachable(shard) => TransportError::UnknownPeer(shard),
// Circuit open and connection failures are transient — peer is known but unavailable.
GrpcTransportError::CircuitOpen(_) | GrpcTransportError::ConnectionFailed { .. } => {
TransportError::Closed
}
GrpcTransportError::ChannelFull { .. }
| GrpcTransportError::Internal(_)
| GrpcTransportError::TlsConfig(_)
| GrpcTransportError::Grpc(_)
| GrpcTransportError::TonicTransport(_) => TransportError::Closed,
// Routing failure: the shard is not in our peer map. Surface as the
// dedicated non-retry variant.
if let GrpcTransportError::PeerUnreachable(shard) = e {
return Self::UnknownPeer(shard);
}
// Permanent failures (bad TLS/CA, auth rejection, malformed payload, RPC
// not implemented on the peer) will fail identically on every retry.
// Mapping them to the transient `Closed` variant would make the shipper
// re-ship the same seqno forever — a silent replication stall with no
// actionable signal. Surface them as the dedicated
// `TransportError::Permanent` variant (carrying the human-readable
// cause) so the shipper quarantines the peer and stops advancing it,
// and log LOUDLY at error! so the fault is observable in logs/alerts.
if e.is_permanent() {
tracing::error!(
error = %e,
"gRPC transport PERMANENT failure (bad TLS/CA, auth rejection, malformed payload, \
or unimplemented RPC); retrying this segment will not help — fix peer/config. \
Quarantining the peer via TransportError::Permanent",
);
return Self::Permanent {
reason: e.to_string(),
};
}
// Genuinely transient failures map to Closed: circuit-open and follower
// backpressure are temporary (peer known but unable to accept right
// now); the remaining internal/transport variants are non-routing and
// recoverable. Mapping to Closed keeps the shipper from advancing its
// per-peer cursor so it retries the same seqno, in order, next poll.
Self::Closed
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn tls_config_failure_is_permanent() {
let e = GrpcTransportError::TlsConfig("bad CA".into());
assert!(e.is_permanent(), "bad TLS material never becomes shippable");
}
#[test]
fn auth_and_malformed_grpc_codes_are_permanent() {
// mTLS / authz rejection and malformed-payload codes are permanent:
// the server returns them identically on every retry.
for code in [
tonic::Code::Unauthenticated,
tonic::Code::PermissionDenied,
tonic::Code::InvalidArgument,
tonic::Code::OutOfRange,
tonic::Code::Unimplemented,
] {
let e = GrpcTransportError::Grpc(Box::new(tonic::Status::new(code, "x")));
assert!(e.is_permanent(), "{code:?} must be permanent");
}
}
#[test]
fn unavailable_and_exhausted_grpc_codes_are_transient() {
// A peer that is down or temporarily overloaded WILL accept the same
// segment once it recovers — these must stay transient so the shipper
// keeps retrying.
for code in [
tonic::Code::Unavailable,
tonic::Code::ResourceExhausted,
tonic::Code::DeadlineExceeded,
tonic::Code::Aborted,
] {
let e = GrpcTransportError::Grpc(Box::new(tonic::Status::new(code, "x")));
assert!(!e.is_permanent(), "{code:?} must be transient");
}
}
#[test]
fn circuit_and_backpressure_are_transient() {
assert!(!GrpcTransportError::CircuitOpen(ShardId(1)).is_permanent());
assert!(!GrpcTransportError::SegmentRejected(ShardId(1)).is_permanent());
assert!(!GrpcTransportError::Internal("transient".into()).is_permanent());
}
#[test]
fn peer_unreachable_maps_to_unknown_peer_not_closed() {
let mapped: TransportError = GrpcTransportError::PeerUnreachable(ShardId(9)).into();
assert!(
matches!(mapped, TransportError::UnknownPeer(ShardId(9))),
"routing failure must surface as UnknownPeer, not a retry"
);
}
#[test]
fn permanent_tls_failure_maps_to_permanent() {
// A permanent failure (bad TLS material) must surface as the dedicated
// `Permanent` variant so the shipper quarantines the peer instead of
// re-shipping the same seqno forever. The reason carries the cause for
// logs/alerts.
let mapped: TransportError = GrpcTransportError::TlsConfig("bad CA".into()).into();
match mapped {
TransportError::Permanent { reason } => {
assert!(
reason.contains("bad CA"),
"reason must carry cause: {reason}"
);
}
other => panic!("expected Permanent, got {other:?}"),
}
}
#[test]
fn permanent_grpc_code_maps_to_permanent() {
// Auth/authz and malformed-payload gRPC codes are permanent and must
// map to `Permanent`, never the transient `Closed`.
for code in [
tonic::Code::Unauthenticated,
tonic::Code::PermissionDenied,
tonic::Code::InvalidArgument,
tonic::Code::OutOfRange,
tonic::Code::Unimplemented,
] {
let e = GrpcTransportError::Grpc(Box::new(tonic::Status::new(code, "x")));
let mapped: TransportError = e.into();
assert!(
matches!(mapped, TransportError::Permanent { .. }),
"{code:?} must map to Permanent, got {mapped:?}"
);
}
}
#[test]
fn transient_failure_maps_to_closed() {
let mapped: TransportError = GrpcTransportError::SegmentRejected(ShardId(2)).into();
assert!(matches!(mapped, TransportError::Closed));
}
}

12
tidal-net/src/lib.rs
View File

@ -20,6 +20,18 @@ pub mod tls;
pub mod transport;
/// Generated protobuf types for the WAL shipping service.
///
/// This module is tonic/prost code generation output (`include_proto!`); we do
/// not control its style. The allows below cover exactly the pedantic/nursery
/// lints the generated code trips and apply to nothing hand-written.
#[allow(
clippy::doc_markdown,
clippy::missing_errors_doc,
clippy::derive_partial_eq_without_eq,
clippy::default_trait_access,
clippy::missing_const_for_fn,
clippy::too_many_lines
)]
pub mod proto {
tonic::include_proto!("tidal.replication.v1");
}

94
tidal-net/src/server.rs
View File

@ -1,16 +1,18 @@
//! gRPC server implementing the `WalShipping` service.
use tidaldb::replication::transport::WalSegmentPayload;
use tokio::sync::mpsc;
use tonic::{Request, Response, Status};
use tidaldb::replication::transport::WalSegmentPayload;
use crate::config::GrpcTransportConfig;
use crate::proto::wal_shipping_server::{WalShipping, WalShippingServer};
use crate::proto::{
HeartbeatRequest, HeartbeatResponse, ShipSegmentRequest, ShipSegmentResponse, StreamRequest,
use crate::{
config::GrpcTransportConfig,
proto::{
HeartbeatRequest, HeartbeatResponse, ShipSegmentRequest, ShipSegmentResponse,
StreamRequest,
wal_shipping_server::{WalShipping, WalShippingServer},
},
tls,
};
use crate::tls;
/// The gRPC service that receives WAL segments from peers.
pub struct WalShippingService {
@ -20,7 +22,11 @@ pub struct WalShippingService {
impl WalShippingService {
/// Create a new service that forwards received segments to the given channel.
pub fn new(inbound_tx: mpsc::Sender<WalSegmentPayload>, max_payload_bytes: usize) -> Self {
#[must_use]
pub const fn new(
inbound_tx: mpsc::Sender<WalSegmentPayload>,
max_payload_bytes: usize,
) -> Self {
Self {
inbound_tx,
max_payload_bytes,
@ -75,10 +81,20 @@ impl WalShipping for WalShippingService {
&self,
_request: Request<StreamRequest>,
) -> Result<Response<Self::StreamSegmentsStream>, Status> {
// TODO(m8p8): Implement server-streaming for catch-up replication.
// Current replication uses ShipSegment (unary) via the Transport trait.
// DELIBERATELY UNIMPLEMENTED, not a silent stub: this returns an
// explicit `Status::unimplemented` so a caller that mis-routes to the
// streaming path fails loudly with an actionable code rather than
// hanging or silently no-op'ing.
//
// Production replication ships segments via the unary `ShipSegment` RPC
// (the `Transport` trait surface). Server-streaming is only needed for
// bulk catch-up of a far-behind follower; until that path is built the
// shipper's per-poll re-send of un-acked seqnos covers catch-up at the
// cost of more round-trips. Tracked as a known gap in the M8 cluster
// scope; see docs/specs/14-scale-architecture.md (catch-up replication)
// and CHANGELOG.md "Known gaps".
Err(Status::unimplemented(
"StreamSegments not yet implemented; use ShipSegment for segment delivery",
"StreamSegments not implemented; use the unary ShipSegment RPC for segment delivery",
))
}
@ -86,16 +102,33 @@ impl WalShipping for WalShippingService {
&self,
_request: Request<HeartbeatRequest>,
) -> Result<Response<HeartbeatResponse>, Status> {
// For now, acknowledge heartbeats without forwarding to ControlPlane.
// Full ControlPlane integration will come in m8p8.
// MINIMAL REAL BEHAVIOR, not a silent stub: a heartbeat that reaches
// this handler proves the gRPC server is up, the listener is accepting,
// and (under mTLS) the peer's certificate was accepted — i.e. a genuine
// network-liveness probe. Returning `acknowledged: true` is therefore a
// truthful answer to "are you reachable", which is exactly what the
// failure detector needs from the transport layer.
//
// Forwarding liveness into the engine's ControlPlane health table (so a
// missed heartbeat demotes a peer) is a deferred enrichment, tracked as
// a known gap in the M8 cluster scope; see docs/runbooks/cluster.md
// (health checks) and CHANGELOG.md "Known gaps". It is additive: it does
// not change this handler's contract, only who else observes the result.
Ok(Response::new(HeartbeatResponse { acknowledged: true }))
}
}
/// Start the gRPC server on the given address.
///
/// Returns a `JoinHandle` that resolves when the server stops.
pub async fn start_server(
/// Returns a `JoinHandle` that resolves when the server stops. Must be called
/// from within a tokio runtime context (it `tokio::spawn`s the serve loop).
///
/// # Errors
///
/// Returns [`GrpcTransportError`](crate::error::GrpcTransportError) if TLS is
/// requested but the server TLS config cannot be built, or if TLS is unset and
/// `--insecure` was not opted into.
pub fn start_server(
config: &GrpcTransportConfig,
inbound_tx: mpsc::Sender<WalSegmentPayload>,
) -> Result<
@ -119,11 +152,34 @@ pub async fn start_server(
));
}
// Raise tonic's default 4 MiB codec limits to the configured max payload on
// BOTH directions. The server is the DECODER for inbound ShipSegmentRequest
// (and the ENCODER for any future server-streaming response). With the
// default 4 MiB limit a full-size WAL segment (16 MiB by default, up to
// `max_payload_bytes` = 64 MiB) is rejected inside the codec with a
// "message too large" status, which the shipper would retry forever as a
// transient `Closed`. Pin both limits to the same ceiling the application
// already advertises and validates against (server.rs ship_segment guard,
// transport.rs send_segment guard) so the wire path and the app agree.
let max = config.max_payload_bytes;
let wal_service = WalShippingServer::new(service)
.max_decoding_message_size(max)
.max_encoding_message_size(max);
let handle = tokio::spawn(async move {
server_builder
.add_service(WalShippingServer::new(service))
.serve(addr)
.await
// Observe the serve loop's terminal Result here so a failure AFTER
// successful startup (e.g. the listener dies, a TLS handshake task
// panics, the reactor is torn down) is logged at error! rather than
// silently swallowed when the JoinHandle is dropped or aborted. The
// Result is still propagated out of the task so callers can also
// inspect it via the handle (see `GrpcTransport::server_terminated`).
let result = server_builder.add_service(wal_service).serve(addr).await;
if let Err(ref e) = result {
tracing::error!(error = %e, %addr, "gRPC WAL-shipping serve loop terminated with error");
} else {
tracing::info!(%addr, "gRPC WAL-shipping serve loop stopped");
}
result
});
Ok(handle)

13
tidal-net/src/tls.rs
View File

@ -4,10 +4,14 @@ use std::fs;
use tonic::transport::{Certificate, ClientTlsConfig, Identity, ServerTlsConfig};
use crate::config::TlsConfig;
use crate::error::GrpcTransportError;
use crate::{config::TlsConfig, error::GrpcTransportError};
/// Build a tonic `ServerTlsConfig` from our TLS config.
///
/// # Errors
///
/// Returns [`GrpcTransportError`] if the server cert, server key, or CA cert
/// file cannot be read.
pub fn server_tls_config(tls: &TlsConfig) -> Result<ServerTlsConfig, GrpcTransportError> {
let server_cert = fs::read(&tls.server_cert)
.map_err(|e| GrpcTransportError::TlsConfig(format!("read server cert: {e}")))?;
@ -23,6 +27,11 @@ pub fn server_tls_config(tls: &TlsConfig) -> Result<ServerTlsConfig, GrpcTranspo
}
/// Build a tonic `ClientTlsConfig` from our TLS config.
///
/// # Errors
///
/// Returns [`GrpcTransportError`] if the CA cert (or, when mutual TLS is
/// configured, the client cert/key) file cannot be read.
pub fn client_tls_config(tls: &TlsConfig) -> Result<ClientTlsConfig, GrpcTransportError> {
let ca_cert = fs::read(&tls.ca_cert)
.map_err(|e| GrpcTransportError::TlsConfig(format!("read CA cert: {e}")))?;

91
tidal-net/src/transport.rs
View File

@ -6,18 +6,15 @@
//! callers (WAL shipper and segment receiver) run on `std::thread`, not
//! inside a tokio context.
use std::collections::HashMap;
use std::sync::Mutex;
use std::{collections::HashMap, sync::Mutex};
use tidaldb::replication::{
shard::ShardId,
transport::{Transport, TransportError, WalSegmentPayload},
};
use tokio::sync::mpsc;
use tidaldb::replication::shard::ShardId;
use tidaldb::replication::transport::{Transport, TransportError, WalSegmentPayload};
use crate::client::PeerPool;
use crate::config::GrpcTransportConfig;
use crate::error::GrpcTransportError;
use crate::server;
use crate::{client::PeerPool, config::GrpcTransportConfig, error::GrpcTransportError, server};
/// A gRPC-based transport for WAL segment shipping between tidalDB shards.
///
@ -33,10 +30,30 @@ use crate::server;
/// `runtime.block_on(rx.recv())`, which is safe but wasteful.
pub struct GrpcTransport {
config: GrpcTransportConfig,
runtime: tokio::runtime::Runtime,
/// `Some` for the transport's whole life; taken in [`Drop`] so the runtime
/// can be torn down with the non-blocking [`tokio::runtime::Runtime::shutdown_background`]
/// (which consumes the runtime) instead of the default blocking `Runtime::drop`.
runtime: Option<tokio::runtime::Runtime>,
inbound_rx: Mutex<mpsc::Receiver<WalSegmentPayload>>,
pool: PeerPool,
_server_handle: tokio::task::JoinHandle<Result<(), tonic::transport::Error>>,
server_handle: tokio::task::JoinHandle<Result<(), tonic::transport::Error>>,
}
/// Install a process-wide rustls [`CryptoProvider`] before tonic's TLS
/// builders run. tidal-net pulls rustls only transitively (via tonic), with no
/// provider feature in its own dependency closure, so under a narrow build the
/// process-level default is absent and rustls panics (could not automatically
/// determine the process-level [`CryptoProvider`]); under a wider build two
/// providers can be present, which is equally ambiguous. Installing aws-lc-rs
/// explicitly here — idempotently, ignoring the error when another component
/// (e.g. a reqwest-based client elsewhere in the process) already set a default
/// — removes that fragility for the mTLS tests, a standalone tidal-server, and
/// the replication path alike. See `tidal-net/BUILD.bazel`.
fn ensure_crypto_provider() {
static ONCE: std::sync::Once = std::sync::Once::new();
ONCE.call_once(|| {
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
});
}
impl GrpcTransport {
@ -49,6 +66,8 @@ impl GrpcTransport {
///
/// Returns an error if the server or client pool fails to initialize.
pub fn new(config: GrpcTransportConfig) -> Result<Self, GrpcTransportError> {
ensure_crypto_provider();
let runtime = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
@ -61,21 +80,41 @@ impl GrpcTransport {
// Build server and client pool inside the runtime context.
// TLS setup in tonic requires a tokio reactor to be available.
let (server_handle, pool) = runtime.block_on(async {
let handle = server::start_server(&config, inbound_tx).await?;
let handle = server::start_server(&config, inbound_tx)?;
let pool = PeerPool::new(&config)?;
Ok::<_, GrpcTransportError>((handle, pool))
})?;
Ok(Self {
config,
runtime,
runtime: Some(runtime),
inbound_rx: Mutex::new(inbound_rx),
pool,
_server_handle: server_handle,
server_handle,
})
}
/// Assert we are not inside a tokio runtime (block_on would panic).
/// The embedded tokio runtime. Present until [`Drop`] takes it.
const fn runtime(&self) -> &tokio::runtime::Runtime {
self.runtime
.as_ref()
.expect("runtime is present for the transport's whole lifetime")
}
/// Whether the embedded gRPC serve loop has terminated.
///
/// The serve task runs for the transport's whole lifetime; a `true` here
/// means the listener stopped accepting — either because the transport is
/// shutting down or because the serve loop failed (in which case the failure
/// was already logged at `error!` inside the task; see [`server::start_server`]).
/// Health checks and the cluster control plane poll this so a follower whose
/// receive side has silently died is observable, not a black hole.
#[must_use]
pub fn server_terminated(&self) -> bool {
self.server_handle.is_finished()
}
/// Assert we are not inside a tokio runtime (`block_on` would panic).
#[cfg(debug_assertions)]
fn assert_not_in_async_context() {
debug_assert!(
@ -99,7 +138,7 @@ impl Transport for GrpcTransport {
});
}
self.runtime
self.runtime()
.block_on(self.pool.send_to(to, payload))
.map_err(TransportError::from)
}
@ -115,7 +154,7 @@ impl Transport for GrpcTransport {
poisoned.into_inner()
}
};
self.runtime.block_on(rx.recv())
self.runtime().block_on(rx.recv())
}
fn local_shard(&self) -> ShardId {
@ -123,6 +162,24 @@ impl Transport for GrpcTransport {
}
}
impl Drop for GrpcTransport {
fn drop(&mut self) {
// Abort the server task so its `inbound_tx` is released (unblocking any
// parked `recv_segment`), then tear the runtime down WITHOUT blocking.
//
// A `GrpcTransport` can be dropped from within an async context — e.g.
// cluster shutdown unwinding on the axum reactor thread — where the
// default blocking `Runtime::drop` panics ("Cannot drop a runtime in a
// context where blocking is not allowed"). `shutdown_background` returns
// immediately and reclaims the worker threads off-thread, so `Drop` is
// safe in any context.
self.server_handle.abort();
if let Some(runtime) = self.runtime.take() {
runtime.shutdown_background();
}
}
}
/// Factory for building a set of [`GrpcTransport`] instances, one per shard.
///
/// Analogous to `InProcessTransportFactory` but for gRPC connections.

93
tidal-net/tests/large_payload.rs Normal file
View File

@ -0,0 +1,93 @@
// Integration-test exemptions (same posture as the tidaldb integration tests):
// unwrap on known-good fixtures and short-lived read guards are idiomatic here.
#![allow(clippy::unwrap_used, clippy::significant_drop_tightening)]
//! Regression test for the tonic default 4 MiB codec message-size limit.
//!
//! tonic 0.12 defaults BOTH the encoder and decoder message-size limits to
//! 4 MiB. `GrpcTransportConfig::max_payload_bytes` defaults to 64 MiB and the
//! WAL default sealed segment is 16 MiB, so a full-size segment is 4x the codec
//! default. If the limits are not raised on both ends, the FIRST full-size
//! segment a leader ships fails inside the codec, maps to `TransportError::
//! Closed`, and the shipper retries the same seqno forever (silent replication
//! stall). This test ships a payload strictly between 4 MiB and the configured
//! max and asserts it round-trips byte-for-byte, so the regression can never
//! return undetected.
use std::{collections::HashMap, net::SocketAddr, thread, time::Duration};
use tidal_net::{GrpcTransport, config::GrpcTransportConfig};
use tidaldb::replication::{
WalSegmentId,
shard::{RegionId, ShardId},
transport::{Transport, WalSegmentPayload},
};
/// Get a unique listen address using port 0 (OS-assigned).
fn free_addr() -> SocketAddr {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
listener.local_addr().unwrap()
}
fn make_config(
shard: ShardId,
listen: SocketAddr,
peers: HashMap<ShardId, SocketAddr>,
) -> GrpcTransportConfig {
GrpcTransportConfig {
local_shard: shard,
listen_addr: listen,
peers,
insecure: true,
..Default::default()
}
}
#[test]
fn ships_payload_larger_than_tonic_default_codec_limit() {
let addr0 = free_addr();
let addr1 = free_addr();
let config0 = make_config(ShardId(0), addr0, HashMap::from([(ShardId(1), addr1)]));
let config1 = make_config(ShardId(1), addr1, HashMap::from([(ShardId(0), addr0)]));
// Sanity: the test is only meaningful if the payload exceeds tonic's 4 MiB
// default AND fits inside the configured ceiling.
const FOUR_MIB: usize = 4 * 1024 * 1024;
const EIGHT_MIB: usize = 8 * 1024 * 1024;
assert!(EIGHT_MIB > FOUR_MIB, "payload must exceed tonic default");
assert!(
EIGHT_MIB <= config0.max_payload_bytes,
"payload must fit configured max"
);
let t0 = GrpcTransport::new(config0).expect("transport 0");
let t1 = GrpcTransport::new(config1).expect("transport 1");
// Give the servers a moment to start.
thread::sleep(Duration::from_millis(150));
// A deterministic, verifiable 8 MiB body (a repeating byte pattern keyed on
// index so a silent truncation or corruption is caught, not just length).
let body: Vec<u8> = (0..EIGHT_MIB).map(|i| (i % 251) as u8).collect();
let payload = WalSegmentPayload {
id: WalSegmentId::new(RegionId::SINGLE, ShardId(0), 7),
bytes: body.clone(),
event_count: 3,
};
t0.send_segment(ShardId(1), payload)
.expect("8 MiB segment must ship past the raised codec limit");
let received = t1
.recv_segment()
.expect("follower must receive the 8 MiB segment");
assert_eq!(received.id.seqno, 7);
assert_eq!(received.event_count, 3);
assert_eq!(
received.bytes.len(),
EIGHT_MIB,
"body length must round-trip"
);
assert_eq!(received.bytes, body, "body bytes must round-trip exactly");
}

180
tidal-net/tests/mtls.rs
View File

@ -1,16 +1,19 @@
// Integration-test exemptions (same posture as the tidaldb integration tests):
// unwrap on known-good fixtures and short-lived read guards are idiomatic here.
#![allow(clippy::unwrap_used, clippy::significant_drop_tightening)]
//! Tests for mutual TLS configuration.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::thread;
use std::time::Duration;
use std::{collections::HashMap, net::SocketAddr, thread, time::Duration};
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::{Transport, WalSegmentPayload};
use tidal_net::GrpcTransport;
use tidal_net::config::{GrpcTransportConfig, TlsConfig};
use tidal_net::{
GrpcTransport,
config::{GrpcTransportConfig, TlsConfig},
};
use tidaldb::replication::{
WalSegmentId,
shard::{RegionId, ShardId},
transport::{Transport, WalSegmentPayload},
};
fn free_addr() -> SocketAddr {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
@ -64,6 +67,163 @@ fn generate_certs(dir: &std::path::Path) -> TlsConfig {
}
}
/// Generate a SECOND, independent CA and a client cert/key signed by it,
/// written alongside the legit material. The returned `TlsConfig` carries the
/// legit server-trusted CA (so the client still trusts the server) but a client
/// identity the server's `client_ca_root` does NOT trust — i.e. an untrusted
/// client certificate. Used to prove the mTLS boundary actually rejects
/// foreign client certs.
fn generate_untrusted_client(dir: &std::path::Path, legit: &TlsConfig) -> TlsConfig {
use rcgen::{CertificateParams, KeyPair};
// A rogue CA the server has never heard of.
let rogue_ca_key = KeyPair::generate().unwrap();
let mut rogue_ca_params = CertificateParams::new(vec!["rogue-ca".to_string()]).unwrap();
rogue_ca_params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
let rogue_ca_cert = rogue_ca_params.self_signed(&rogue_ca_key).unwrap();
// A client cert signed by the rogue CA (NOT the server-trusted CA).
let rogue_client_key = KeyPair::generate().unwrap();
let rogue_client_params = CertificateParams::new(vec!["rogue-client".to_string()]).unwrap();
let rogue_client_cert = rogue_client_params
.signed_by(&rogue_client_key, &rogue_ca_cert, &rogue_ca_key)
.unwrap();
let rogue_client_cert_path = dir.join("rogue-client.pem");
let rogue_client_key_path = dir.join("rogue-client-key.pem");
std::fs::write(&rogue_client_cert_path, rogue_client_cert.pem()).unwrap();
std::fs::write(&rogue_client_key_path, rogue_client_key.serialize_pem()).unwrap();
TlsConfig {
// Keep the legit CA so the CLIENT still trusts the server's cert; only
// the client's OWN identity is untrusted by the server.
ca_cert: legit.ca_cert.clone(),
server_cert: legit.server_cert.clone(),
server_key: legit.server_key.clone(),
client_cert: Some(rogue_client_cert_path),
client_key: Some(rogue_client_key_path),
}
}
/// A server config with the legit (trusted) CA and short timeouts so a rejected
/// handshake fails fast in tests.
fn server_config(shard: ShardId, listen: SocketAddr, tls: TlsConfig) -> GrpcTransportConfig {
GrpcTransportConfig {
local_shard: shard,
listen_addr: listen,
peers: HashMap::new(),
tls: Some(tls),
insecure: false,
connect_timeout: Duration::from_millis(500),
request_timeout: Duration::from_millis(500),
..Default::default()
}
}
/// SECURITY BOUNDARY: a client whose certificate is signed by a CA the server
/// does not trust must be REJECTED — the mTLS handshake fails and no segment is
/// ever accepted. This is the negative counterpart to `mtls_send_and_receive`;
/// without it the happy path alone could pass even if the server accepted ANY
/// client cert (or none).
#[test]
fn untrusted_client_cert_is_rejected() {
let tmp = tempfile::tempdir().unwrap();
let legit = generate_certs(tmp.path());
let rogue = generate_untrusted_client(tmp.path(), &legit);
let addr_server = free_addr();
let addr_client = free_addr();
// Server (shard 1) trusts only the legit CA for client auth.
let server = GrpcTransport::new(server_config(ShardId(1), addr_server, legit))
.expect("server with trusted CA");
// Client (shard 0) presents a cert signed by a rogue CA.
let client_cfg = GrpcTransportConfig {
peers: HashMap::from([(ShardId(1), addr_server)]),
..server_config(ShardId(0), addr_client, rogue)
};
let client = GrpcTransport::new(client_cfg).expect("client with rogue cert");
thread::sleep(Duration::from_millis(200));
// The mTLS handshake must fail, so the send must error. Retry a couple of
// times to defeat any single-shot connect race; every attempt must fail.
let mut last_ok = false;
for seq in 0..3u64 {
let payload = WalSegmentPayload {
id: WalSegmentId::new(RegionId::SINGLE, ShardId(0), seq),
bytes: vec![0x11; 32],
event_count: 1,
};
if client.send_segment(ShardId(1), payload).is_ok() {
last_ok = true;
break;
}
thread::sleep(Duration::from_millis(100));
}
assert!(
!last_ok,
"server must REJECT a client cert signed by an untrusted CA"
);
// Keep the server alive until assertions run.
drop(server);
}
/// SECURITY BOUNDARY: a client that presents NO certificate (absent client
/// identity) must be REJECTED by a server that requires client auth
/// (`client_ca_root`). Verifies the server demands mutual auth rather than
/// accepting one-way TLS.
#[test]
fn absent_client_cert_is_rejected() {
let tmp = tempfile::tempdir().unwrap();
let legit = generate_certs(tmp.path());
let addr_server = free_addr();
let addr_client = free_addr();
let server = GrpcTransport::new(server_config(ShardId(1), addr_server, legit.clone()))
.expect("server requiring client auth");
// Client trusts the server's CA but presents NO client identity.
let no_client_id = TlsConfig {
ca_cert: legit.ca_cert.clone(),
server_cert: legit.server_cert.clone(),
server_key: legit.server_key,
client_cert: None,
client_key: None,
};
let client_cfg = GrpcTransportConfig {
peers: HashMap::from([(ShardId(1), addr_server)]),
..server_config(ShardId(0), addr_client, no_client_id)
};
let client = GrpcTransport::new(client_cfg).expect("client without identity");
thread::sleep(Duration::from_millis(200));
let mut last_ok = false;
for seq in 0..3u64 {
let payload = WalSegmentPayload {
id: WalSegmentId::new(RegionId::SINGLE, ShardId(0), seq),
bytes: vec![0x22; 32],
event_count: 1,
};
if client.send_segment(ShardId(1), payload).is_ok() {
last_ok = true;
break;
}
thread::sleep(Duration::from_millis(100));
}
assert!(
!last_ok,
"server requiring client_ca_root must REJECT a client with no certificate"
);
// Keep the server alive until assertions run.
drop(server);
}
#[test]
fn mtls_send_and_receive() {
let tmp = tempfile::tempdir().unwrap();

88
tidal-net/tests/multi_node_uat.rs
View File

@ -1,10 +1,18 @@
// Integration-test exemptions (same posture as the tidaldb integration tests):
// unwrap on known-good fixtures, short-lived read guards, and loop-counter
// casts in throughput math are idiomatic here.
#![allow(
clippy::unwrap_used,
clippy::significant_drop_tightening,
clippy::cast_precision_loss
)]
//! gRPC transport integration tests (tier-2 distributed testing).
//!
//! Proves the full path: signal write → WAL batch encode → gRPC ship →
//! gRPC receive → apply_payload → SignalLedger replication.
//! gRPC receive → `apply_payload` → `SignalLedger` replication.
//!
//! Uses `GrpcTransport` on localhost instead of in-process crossbeam channels.
//! All TidalDb instances run in the same test process — this validates the
//! All `TidalDb` instances run in the same test process — this validates the
//! transport layer's serialization, delivery, and idempotency guarantees.
//!
//! **Not covered here (requires tier-3 multi-process harness):**
@ -14,23 +22,27 @@
//! - Rolling upgrade with mixed binary versions
//! - HTTP runbook endpoint verification against real cluster
use std::collections::HashMap;
use std::net::SocketAddr;
use std::thread;
use std::time::{Duration, Instant};
use std::{
collections::HashMap,
net::SocketAddr,
thread,
time::{Duration, Instant},
};
use tidaldb::TidalDb;
use tidaldb::db::config::{NodeConfig, NodeRole};
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::receiver::apply_payload;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::{Transport, WalSegmentPayload};
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window};
use tidaldb::signals::{NoopWalWriter, SignalLedger};
use tidaldb::wal::format::batch::{EventRecord, encode_batch};
use tidal_net::GrpcTransport;
use tidal_net::config::GrpcTransportConfig;
use tidal_net::{GrpcTransport, config::GrpcTransportConfig};
use tidaldb::{
TidalDb,
db::config::{NodeConfig, NodeRole},
replication::{
WalSegmentId,
receiver::apply_payload,
shard::{RegionId, ShardId},
transport::{Transport, WalSegmentPayload},
},
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
signals::{NoopWalWriter, SignalLedger},
wal::format::batch::{EventRecord, encode_batch},
};
// ── Helpers ────────────────────────────────────────────────────────────────
@ -72,7 +84,7 @@ struct GrpcNode {
transport: GrpcTransport,
}
/// Build a leader + follower pair connected via GrpcTransport.
/// Build a leader + follower pair connected via `GrpcTransport`.
fn build_pair() -> (GrpcNode, GrpcNode, HashMap<String, u8>) {
let schema = m8_schema();
let addr0 = free_addr();
@ -160,12 +172,12 @@ fn write_and_ship(
node.db.signal(signal_type, entity_id, weight, ts).unwrap();
let type_id = *sig_ids.get(signal_type).unwrap();
let events = [EventRecord {
entity_id: entity_id.as_u64(),
signal_type: type_id,
weight: weight as f32,
timestamp_nanos: ts.as_nanos(),
}];
let events = [EventRecord::signal(
entity_id.as_u64(),
type_id,
weight as f32,
ts.as_nanos(),
)];
let bytes = encode_batch(&events, seqno, ts.as_nanos()).unwrap();
let payload = WalSegmentPayload {
@ -186,7 +198,8 @@ fn recv_and_apply(node: &GrpcNode) {
.expect("recv_segment returned None");
let ledger = node.db.ledger().unwrap().clone();
let rep_state = node.db.replication_state().clone();
apply_payload(&payload.bytes, ShardId(0), &ledger, &rep_state).expect("apply_payload failed");
apply_payload(&payload.bytes, ShardId(0), &ledger, &rep_state, None)
.expect("apply_payload failed");
}
// ── UAT Tests ──────────────────────────────────────────────────────────────
@ -238,12 +251,7 @@ fn uat_step2_idempotent_replay() {
let ts = Timestamp::now();
let type_id = *sig_ids.get("view").unwrap();
let events = [EventRecord {
entity_id: 42,
signal_type: type_id,
weight: 1.0,
timestamp_nanos: ts.as_nanos(),
}];
let events = [EventRecord::signal(42, type_id, 1.0, ts.as_nanos())];
let bytes = encode_batch(&events, 1, ts.as_nanos()).unwrap();
// Ship same batch twice.
@ -405,12 +413,7 @@ fn uat_step5_three_node_replication() {
let eid = EntityId::new(seq);
dbs[0].signal("view", eid, 1.0, ts).unwrap();
let events = [EventRecord {
entity_id: seq,
signal_type: view_id,
weight: 1.0,
timestamp_nanos: ts.as_nanos(),
}];
let events = [EventRecord::signal(seq, view_id, 1.0, ts.as_nanos())];
let bytes = encode_batch(&events, seq, ts.as_nanos()).unwrap();
// Ship to follower 1 and 2.
@ -432,7 +435,7 @@ fn uat_step5_three_node_replication() {
.expect("recv failed");
let ledger = dbs[follower_idx].ledger().unwrap().clone();
let rep = dbs[follower_idx].replication_state().clone();
apply_payload(&payload.bytes, ShardId(0), &ledger, &rep).unwrap();
apply_payload(&payload.bytes, ShardId(0), &ledger, &rep, None).unwrap();
}
}
@ -524,12 +527,7 @@ fn partition_heal_convergence() {
leader.db.signal("view", EntityId::new(i), 1.0, ts).unwrap();
let type_id = *sig_ids.get("view").unwrap();
let events = [EventRecord {
entity_id: i,
signal_type: type_id,
weight: 1.0,
timestamp_nanos: ts.as_nanos(),
}];
let events = [EventRecord::signal(i, type_id, 1.0, ts.as_nanos())];
let bytes = encode_batch(&events, i, ts.as_nanos()).unwrap();
missed_payloads.push(WalSegmentPayload {
id: WalSegmentId::new(RegionId::SINGLE, ShardId(0), i),

43
tidal-net/tests/reconnection.rs
View File

@ -1,16 +1,21 @@
//! Tests that GrpcTransport reconnects after a peer restarts.
// Integration-test exemptions (same posture as the tidaldb integration tests):
// unwrap on known-good fixtures and short-lived read guards are idiomatic here.
#![allow(clippy::unwrap_used, clippy::significant_drop_tightening)]
//! Tests that `GrpcTransport` reconnects after a peer restarts.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::thread;
use std::time::{Duration, Instant};
use std::{
collections::HashMap,
net::SocketAddr,
thread,
time::{Duration, Instant},
};
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::{Transport, WalSegmentPayload};
use tidal_net::GrpcTransport;
use tidal_net::config::GrpcTransportConfig;
use tidal_net::{GrpcTransport, config::GrpcTransportConfig};
use tidaldb::replication::{
WalSegmentId,
shard::{RegionId, ShardId},
transport::{Transport, WalSegmentPayload},
};
fn free_addr() -> SocketAddr {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
@ -66,8 +71,8 @@ fn reconnects_after_peer_restart() {
// Send successfully.
sender.send_segment(ShardId(1), make_payload(1)).unwrap();
let received = receiver.recv_segment().unwrap();
assert_eq!(received.id.seqno, 1);
let first_segment = receiver.recv_segment().unwrap();
assert_eq!(first_segment.id.seqno, 1);
// Drop the receiver to simulate crash.
drop(receiver);
@ -92,12 +97,12 @@ fn reconnects_after_peer_restart() {
let start = Instant::now();
let mut delivered = false;
while start.elapsed() < Duration::from_secs(5) {
if sender.send_segment(ShardId(1), make_payload(3)).is_ok() {
if let Some(seg) = receiver2.recv_segment() {
assert_eq!(seg.id.seqno, 3);
delivered = true;
break;
}
if sender.send_segment(ShardId(1), make_payload(3)).is_ok()
&& let Some(seg) = receiver2.recv_segment()
{
assert_eq!(seg.id.seqno, 3);
delivered = true;
break;
}
thread::sleep(Duration::from_millis(200));
}

25
tidal-net/tests/transport_contract.rs
View File

@ -1,21 +1,22 @@
// Integration-test exemptions (same posture as the tidaldb integration tests):
// unwrap/unwrap_err on known-good fixtures and short-lived read guards are
// idiomatic here.
#![allow(clippy::unwrap_used, clippy::significant_drop_tightening)]
//! Transport trait contract tests for `GrpcTransport`.
//!
//! Mirrors the test patterns from `InProcessTransport` but over gRPC on localhost.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::thread;
use std::time::Duration;
use std::{collections::HashMap, net::SocketAddr, thread, time::Duration};
use tidaldb::replication::WalSegmentId;
use tidaldb::replication::shard::{RegionId, ShardId};
use tidaldb::replication::transport::{Transport, TransportError, WalSegmentPayload};
use tidal_net::GrpcTransport;
use tidal_net::config::GrpcTransportConfig;
use tidal_net::{GrpcTransport, config::GrpcTransportConfig};
use tidaldb::replication::{
WalSegmentId,
shard::{RegionId, ShardId},
transport::{Transport, TransportError, WalSegmentPayload},
};
/// Get a unique listen address using port 0 (OS-assigned).
/// Since tonic doesn't support port 0, we bind a TcpListener to find a free port.
/// Since tonic doesn't support port 0, we bind a `TcpListener` to find a free port.
fn free_addr() -> SocketAddr {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
listener.local_addr().unwrap()
@ -43,7 +44,7 @@ fn make_payload(shard: ShardId, seqno: u64) -> WalSegmentPayload {
}
}
/// Build two GrpcTransports on localhost that can talk to each other.
/// Build two `GrpcTransports` on localhost that can talk to each other.
fn build_pair() -> (GrpcTransport, GrpcTransport) {
let addr0 = free_addr();
let addr1 = free_addr();

62
tidal-server/BUILD.bazel Normal file
View File

@ -0,0 +1,62 @@
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_proc_macro", "rust_test")
load("@crates//:defs.bzl", "aliases", "all_crate_deps")
load("//bazel:rust_cargo_tests.bzl", "rust_cargo_integration_tests")
rust_library(
name = "tidal_server",
srcs = glob(["src/**/*.rs", "src/*.rs"], allow_empty = True),
crate_root = "src/lib.rs",
crate_name = "tidal_server",
edition = "2024",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(),
deps = all_crate_deps(normal = True) + [
"//tidal:tidaldb_test_support",
"//tidal-net:tidal_net",
],
proc_macro_deps = all_crate_deps(proc_macro = True),
visibility = ["//visibility:public"],
)
rust_test(
name = "tidal_server_test",
crate = ":tidal_server",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True),
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
)
rust_binary(
name = "tidal-server",
srcs = glob(["src/**/*.rs", "src/*.rs"], allow_empty = True),
crate_root = "src/main.rs",
edition = "2024",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(),
deps = all_crate_deps(normal = True) + [
"//tidal:tidaldb_test_support",
"//tidal-net:tidal_net",
":tidal_server",
],
proc_macro_deps = all_crate_deps(proc_macro = True),
visibility = ["//visibility:public"],
)
rust_cargo_integration_tests(
name_prefix = "tidal_server",
tests = glob(["tests/*.rs"]),
srcs = glob(["tests/**/*.rs"]),
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True) + [
":tidal-server",
],
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True) + [
"//tidal:tidaldb_test_support",
"//tidal-net:tidal_net",
":tidal_server",
],
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
)

31
tidal-server/Cargo.toml
View File

@ -9,6 +9,28 @@ license.workspace = true
name = "tidal_server"
path = "src/lib.rs"
# ── tidal-crate lint posture (single source of truth) ──────────────────────
# IDENTICAL block across tidaldb / tidal-net / tidal-server / tidalctl. These
# crates deliberately DO NOT inherit `[workspace.lints]`; they hold the embedded
# recommendation DB + its transport/server/CLI to a stricter correctness bar
# (`unsafe_code = forbid`, `clippy::all = deny`, `unwrap_used = deny`).
# `unwrap_used = "deny"` is kept per-crate rather than in `[workspace.lints]`
# because the workspace also hosts the example/consumer crates under
# `applications/` (not held to the engine's bar). Keep these four blocks BYTE-IDENTICAL.
[lints.rust]
unsafe_code = "forbid"
[lints.clippy]
all = { level = "deny", priority = -1 }
pedantic = { level = "warn", priority = -1 }
nursery = { level = "warn", priority = -1 }
# Justified allows (lossy numeric casts are pervasive + intentional in the
# ranking/scoring math; module_name_repetitions is idiomatic for the flat
# module layout documented in CLAUDE.md):
cast_possible_truncation = "allow"
module_name_repetitions = "allow"
unwrap_used = "deny"
[dependencies]
axum = "0.8"
clap = { version = "4.5", features = ["derive", "env"] }
@ -17,12 +39,16 @@ tower = { version = "0.5", features = ["limit"] }
tower-http = { version = "0.6", features = ["timeout", "trace", "request-id"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
# serde_yml is the maintained fork of the deprecated/unmaintained serde_yaml
# 0.9 (RUSTSEC-2024-0320). It keeps the same `Value`/`Mapping` API, so the
# schema/topology/profile parsing below is byte-for-byte behaviour-identical.
serde_yml = "0.0.12"
thiserror = "2"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "sync"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tidaldb = { path = "../tidal", features = ["test-utils"] }
tidal-net = { path = "../tidal-net" }
[features]
cluster-e2e = []
@ -31,4 +57,3 @@ cluster-e2e = []
tempfile = "3"
reqwest = { version = "0.12", features = ["json", "blocking"] }
serde_json = "1"
libc = "0.2"

797
tidal-server/src/cluster.rs
View File

File diff suppressed because it is too large Load Diff

182
tidal-server/src/config.rs
View File

@ -1,21 +1,72 @@
use std::fs;
use std::path::Path;
use std::time::Duration;
use std::{
fs,
path::{Path, PathBuf},
time::Duration,
};
use serde::Deserialize;
use tidaldb::ranking::profile::{
Boost, CandidateStrategy, DiversitySpec, Exclude, Gate, Penalty, ProfileDecay, RankingProfile,
SignalAgg, Sort,
use tidaldb::{
ranking::profile::{
Boost, CandidateStrategy, DiversitySpec, Exclude, Gate, Penalty, ProfileDecay,
RankingProfile, SignalAgg, Sort,
},
schema::{DecaySpec, EntityKind, Schema, SchemaBuilder, TextFieldType, Window},
};
use tidaldb::schema::{DecaySpec, EntityKind, Schema, SchemaBuilder, TextFieldType, Window};
use crate::error::{Result, ServerError};
const DEFAULT_SCHEMA_YAML: &str = include_str!("../config/default-schema.yaml");
/// Default schema file name expected inside a `TIDAL_CONFIG` directory.
pub const CONFIG_DIR_SCHEMA_FILE: &str = "default-schema.yaml";
/// Default topology file name expected inside a `TIDAL_CONFIG` directory.
pub const CONFIG_DIR_TOPOLOGY_FILE: &str = "default-cluster.yaml";
/// Resolve a config file path against an optional `TIDAL_CONFIG` directory.
///
/// Resolution order:
/// 1. An explicit `--schema` / `--topology` path always wins.
/// 2. Otherwise, if a config dir is set (via `TIDAL_CONFIG`), look for
/// `<dir>/<file_name>` and use it when present.
/// 3. Otherwise return `None`, so the caller falls back to the compiled-in
/// default YAML.
///
/// # Errors
///
/// Returns an error if a config dir is set and names `<dir>/<file_name>` but
/// the path is not a readable file, so a misconfigured container fails loudly
/// instead of silently serving the compiled-in defaults.
pub fn resolve_config_path(
explicit: Option<&Path>,
config_dir: Option<&Path>,
file_name: &str,
) -> Result<Option<PathBuf>> {
if let Some(path) = explicit {
return Ok(Some(path.to_path_buf()));
}
config_dir.map_or(Ok(None), |dir| {
let candidate = dir.join(file_name);
if candidate.is_file() {
Ok(Some(candidate))
} else {
Err(ServerError::SchemaConfig(format!(
"TIDAL_CONFIG dir '{}' does not contain '{file_name}'",
dir.display()
)))
}
})
}
/// Load the tidalDB schema and ranking profiles from YAML (or the compiled-in
/// default when `path` is `None`).
///
/// # Errors
///
/// Returns [`ServerError`] if the config cannot be read, the YAML fails to
/// parse, or the schema declares no signals.
pub fn load_schema(path: Option<&Path>) -> Result<(Schema, Vec<RankingProfile>)> {
let raw = read_config(path, DEFAULT_SCHEMA_YAML)?;
let spec: SchemaSpec = serde_yaml::from_str(&raw)
let spec: SchemaSpec = serde_yml::from_str(&raw)
.map_err(|e| ServerError::SchemaConfig(format!("parse schema yaml: {e}")))?;
if spec.signals.is_empty() {
@ -79,10 +130,10 @@ pub fn load_schema(path: Option<&Path>) -> Result<(Schema, Vec<RankingProfile>)>
}
fn read_config(path: Option<&Path>, fallback: &str) -> Result<String> {
match path {
Some(p) => fs::read_to_string(p).map_err(|e| ServerError::io(p, e)),
None => Ok(fallback.to_string()),
}
path.map_or_else(
|| Ok(fallback.to_string()),
|p| fs::read_to_string(p).map_err(|e| ServerError::io(p, e)),
)
}
#[derive(Debug, Deserialize)]
@ -223,7 +274,7 @@ struct ProfileSpec {
exploration: f64,
}
fn default_version() -> u32 {
const fn default_version() -> u32 {
1
}
@ -279,7 +330,7 @@ struct DiversitySpecConfig {
#[serde(untagged)]
enum SortSpec {
Simple(String),
Parameterized(std::collections::HashMap<String, serde_yaml::Value>),
Parameterized(std::collections::HashMap<String, serde_yml::Value>),
}
/// Candidate strategy can be a simple string or a map with parameters.
@ -287,7 +338,7 @@ enum SortSpec {
#[serde(untagged)]
enum CandidateStrategySpec {
Simple(String),
Parameterized(std::collections::HashMap<String, serde_yaml::Value>),
Parameterized(std::collections::HashMap<String, serde_yml::Value>),
}
// ── Profile conversion ──────────────────────────────────────────────────────
@ -364,13 +415,13 @@ impl ProfileSpec {
weight: d.weight,
});
let diversity = match &self.diversity {
Some(d) => DiversitySpec {
let diversity = self
.diversity
.as_ref()
.map_or_else(DiversitySpec::default, |d| DiversitySpec {
max_per_creator: d.max_per_creator,
format_mix_max_fraction: d.format_mix_max_fraction,
},
None => DiversitySpec::default(),
};
});
Ok(RankingProfile {
name: self.name.clone(),
@ -489,61 +540,66 @@ fn parse_candidate_strategy(spec: &CandidateStrategySpec) -> Result<CandidateStr
// ── YAML value extraction helpers ───────────────────────────────────────────
fn extract_f64(val: &serde_yaml::Value, field: &str) -> Result<Option<f64>> {
fn extract_f64(val: &serde_yml::Value, field: &str) -> Result<Option<f64>> {
match val {
serde_yaml::Value::Mapping(map) => {
if let Some(v) = map.get(serde_yaml::Value::String(field.to_owned())) {
serde_yml::Value::Mapping(map) => map
.get(serde_yml::Value::String(field.to_owned()))
.map_or(Ok(None), |v| {
v.as_f64()
.or_else(|| v.as_i64().map(|i| i as f64))
// Config integers are small; the i64→f64 widening is exact here.
.or_else(|| {
#[allow(clippy::cast_precision_loss)]
v.as_i64().map(|i| i as f64)
})
.map(Some)
.ok_or_else(|| ServerError::SchemaConfig(format!("'{field}' must be a number")))
} else {
Ok(None)
}
}
}),
_ => Ok(None),
}
}
fn extract_u64(val: &serde_yaml::Value, field: &str) -> Result<Option<u64>> {
fn extract_u64(val: &serde_yml::Value, field: &str) -> Result<Option<u64>> {
match val {
serde_yaml::Value::Mapping(map) => {
if let Some(v) = map.get(serde_yaml::Value::String(field.to_owned())) {
serde_yml::Value::Mapping(map) => map
.get(serde_yml::Value::String(field.to_owned()))
.map_or(Ok(None), |v| {
v.as_u64().map(Some).ok_or_else(|| {
ServerError::SchemaConfig(format!("'{field}' must be a positive integer"))
})
} else {
Ok(None)
}
}
}),
_ => Ok(None),
}
}
fn extract_string(val: &serde_yaml::Value, field: &str) -> Result<String> {
fn extract_string(val: &serde_yml::Value, field: &str) -> Result<String> {
extract_string_field(val, field)
}
fn extract_string_field(val: &serde_yaml::Value, field: &str) -> Result<String> {
fn extract_string_field(val: &serde_yml::Value, field: &str) -> Result<String> {
match val {
serde_yaml::Value::Mapping(map) => {
if let Some(v) = map.get(serde_yaml::Value::String(field.to_owned())) {
v.as_str()
.map(|s| s.to_owned())
.ok_or_else(|| ServerError::SchemaConfig(format!("'{field}' must be a string")))
} else {
Err(ServerError::SchemaConfig(format!(
"missing required field '{field}'"
)))
}
}
serde_yml::Value::Mapping(map) => map
.get(serde_yml::Value::String(field.to_owned()))
.map_or_else(
|| {
Err(ServerError::SchemaConfig(format!(
"missing required field '{field}'"
)))
},
|v| {
v.as_str()
.map(std::borrow::ToOwned::to_owned)
.ok_or_else(|| {
ServerError::SchemaConfig(format!("'{field}' must be a string"))
})
},
),
_ => Err(ServerError::SchemaConfig(format!(
"expected mapping with field '{field}'"
))),
}
}
fn extract_window(val: &serde_yaml::Value, field: &str) -> Result<Window> {
fn extract_window(val: &serde_yml::Value, field: &str) -> Result<Window> {
let s = extract_string_field(val, field)?;
parse_window(&s)
}
@ -552,13 +608,13 @@ fn extract_window(val: &serde_yaml::Value, field: &str) -> Result<Window> {
fn validate_profile_signals(profile: &RankingProfile, signal_names: &[String]) -> Result<()> {
let check = |signal: &str, context: &str| -> Result<()> {
if !signal_names.iter().any(|s| s == signal) {
if signal_names.iter().any(|s| s == signal) {
Ok(())
} else {
Err(ServerError::SchemaConfig(format!(
"profile '{}': {} references unknown signal '{signal}'",
profile.name, context
)))
} else {
Ok(())
}
};
@ -589,7 +645,7 @@ mod tests {
#[test]
fn parse_schema_with_profiles() {
let yaml = r#"
let yaml = r"
signals:
- name: chat
entity: item
@ -621,8 +677,8 @@ profiles:
diversity:
max_per_creator: 2
exploration: 0.1
"#;
let spec: SchemaSpec = serde_yaml::from_str(yaml).unwrap();
";
let spec: SchemaSpec = serde_yml::from_str(yaml).unwrap();
assert!(spec.profiles.is_some());
let profiles = spec.profiles.unwrap();
assert_eq!(profiles.len(), 1);
@ -657,12 +713,12 @@ profiles:
#[test]
fn parse_parameterized_sort() {
let mut map = std::collections::HashMap::new();
let mut inner = serde_yaml::Mapping::new();
let mut inner = serde_yml::Mapping::new();
inner.insert(
serde_yaml::Value::String("window".into()),
serde_yaml::Value::String("7d".into()),
serde_yml::Value::String("window".into()),
serde_yml::Value::String("7d".into()),
);
map.insert("top_window".into(), serde_yaml::Value::Mapping(inner));
map.insert("top_window".into(), serde_yml::Value::Mapping(inner));
let result = parse_sort(&SortSpec::Parameterized(map)).unwrap();
assert!(matches!(
result,
@ -741,7 +797,7 @@ profiles:
#[test]
fn schema_with_no_profiles_section() {
let yaml = r#"
let yaml = r"
signals:
- name: view
entity: item
@ -749,7 +805,7 @@ signals:
exponential:
half_life_seconds: 604800
windows: [24h, 7d]
"#;
";
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(tmp.path(), yaml).unwrap();
let (_, profiles) = load_schema(Some(tmp.path())).unwrap();
@ -758,7 +814,7 @@ signals:
#[test]
fn profile_defaults() {
let yaml = r#"
let yaml = r"
signals:
- name: view
entity: item
@ -774,7 +830,7 @@ profiles:
agg: value
window: all_time
weight: 1.0
"#;
";
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(tmp.path(), yaml).unwrap();
let (_, profiles) = load_schema(Some(tmp.path())).unwrap();

184
tidal-server/src/dto.rs Normal file
View File

@ -0,0 +1,184 @@
//! Shared request/response DTOs and result-mapping for the data routes.
//!
//! The standalone router ([`crate::router`]) and the cluster router
//! ([`crate::cluster`]) expose the SAME data surface (`/items`,
//! `/embeddings`, `/signals`, `/feed`, `/search`). Their request/response
//! shapes and the engine-result → JSON mapping were previously duplicated
//! verbatim across both modules; any drift between the two (e.g. a new field
//! added on one side only) would silently change one mode's API. This module
//! is the single source of truth so the two routers cannot diverge.
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use tidaldb::query::{retrieve::RetrieveResult, search::SearchResultItem};
// ── Request DTOs ─────────────────────────────────────────────────────────────
/// `POST /items` body.
#[derive(Debug, Deserialize)]
pub struct ItemRequest {
pub entity_id: u64,
pub metadata: HashMap<String, String>,
}
/// `POST /embeddings` body.
#[derive(Debug, Deserialize)]
pub struct EmbeddingRequest {
pub entity_id: u64,
pub values: Vec<f32>,
}
/// `POST /signals` body.
///
/// `user_id` / `creator_id` are optional signal context. They are read on the
/// standalone path; the cluster path ignores them (its replication layer does
/// not yet carry per-signal context), so defaulting them keeps both modes'
/// wire contract identical.
#[derive(Debug, Deserialize)]
pub struct SignalRequest {
pub entity_id: u64,
pub signal: String,
pub weight: f64,
#[serde(default)]
pub user_id: Option<u64>,
#[serde(default)]
pub creator_id: Option<u64>,
}
/// `GET /feed` query parameters.
#[derive(Debug, Deserialize)]
pub struct FeedQuery {
#[serde(default)]
pub user_id: Option<u64>,
#[serde(default = "default_profile")]
pub profile: String,
#[serde(default = "default_limit")]
pub limit: u32,
#[serde(default)]
pub region: Option<String>,
}
/// `GET /search` query parameters.
#[derive(Debug, Deserialize)]
pub struct SearchQueryParams {
pub query: String,
#[serde(default)]
pub user_id: Option<u64>,
#[serde(default = "default_limit")]
pub limit: u32,
#[serde(default)]
pub region: Option<String>,
}
/// Default ranking profile when `?profile=` is omitted.
#[must_use]
pub fn default_profile() -> String {
"for_you".into()
}
/// Default page size when `?limit=` is omitted.
#[must_use]
pub const fn default_limit() -> u32 {
20
}
// ── Response DTOs ────────────────────────────────────────────────────────────
/// `GET /feed` response body.
#[derive(Debug, Serialize)]
pub struct FeedResponse {
pub items: Vec<FeedItem>,
pub total_candidates: usize,
pub region: Option<String>,
}
/// One ranked item in a feed response.
#[derive(Debug, Serialize)]
pub struct FeedItem {
pub entity_id: u64,
pub score: f64,
pub rank: usize,
#[serde(skip_serializing_if = "Option::is_none")]
pub signals: Option<Vec<SignalValue>>,
}
/// A signal value snapshot attached to a ranked item, for explainability.
#[derive(Debug, Serialize)]
pub struct SignalValue {
pub name: String,
pub value: f64,
}
/// `GET /search` response body.
#[derive(Debug, Serialize)]
pub struct SearchResponse {
pub items: Vec<SearchItem>,
pub total_candidates: usize,
pub region: Option<String>,
}
/// One ranked item in a search response.
#[derive(Debug, Serialize)]
pub struct SearchItem {
pub entity_id: u64,
pub score: f64,
pub rank: usize,
#[serde(skip_serializing_if = "Option::is_none")]
pub bm25_score: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub semantic_score: Option<f64>,
}
// ── Engine-result → DTO mapping ──────────────────────────────────────────────
/// Map one engine [`RetrieveResult`] into a wire [`FeedItem`].
///
/// An empty signal list serializes as absent (`None`) rather than `[]` so the
/// response stays compact.
#[must_use]
pub fn feed_item(item: &RetrieveResult) -> FeedItem {
let signals = if item.signals.is_empty() {
None
} else {
Some(
item.signals
.iter()
.map(|s| SignalValue {
name: s.name.clone(),
value: s.value,
})
.collect(),
)
};
FeedItem {
entity_id: item.entity_id.as_u64(),
score: item.score,
rank: item.rank,
signals,
}
}
/// Map a slice of engine [`RetrieveResult`]s into wire [`FeedItem`]s.
#[must_use]
pub fn feed_items(items: &[RetrieveResult]) -> Vec<FeedItem> {
items.iter().map(feed_item).collect()
}
/// Map one engine [`SearchResultItem`] into a wire [`SearchItem`].
#[must_use]
pub fn search_item(item: &SearchResultItem) -> SearchItem {
SearchItem {
entity_id: item.entity_id.as_u64(),
score: item.score,
rank: item.rank,
bm25_score: item.bm25_score.map(f64::from),
semantic_score: item.semantic_score.map(f64::from),
}
}
/// Map a slice of engine [`SearchResultItem`]s into wire [`SearchItem`]s.
#[must_use]
pub fn search_items(items: &[SearchResultItem]) -> Vec<SearchItem> {
items.iter().map(search_item).collect()
}

4
tidal-server/src/error.rs
View File

@ -21,6 +21,10 @@ pub enum ServerError {
Http(#[from] std::io::Error),
#[error("bad request: {0}")]
BadRequest(String),
#[error("cluster mode is experimental and disabled: {0}")]
ExperimentalDisabled(String),
#[error("cluster error: {0}")]
Cluster(String),
}
impl ServerError {

12
tidal-server/src/lib.rs
View File

@ -1,9 +1,21 @@
//! tidal-server — standalone gRPC cluster server for tidaldb.
//!
//! Wraps the embedded `tidaldb` engine in an axum HTTP surface plus a
//! tonic gRPC cluster layer (via `tidal-net`) for multi-shard deployments.
//! Carries a Dockerfile + k8s manifests under `tidal/docker/` for standalone
//! deployment; embedding consumers use the `tidaldb` library directly and do
//! NOT need this server.
//!
//! Status: M8 Phase 8 and 10 marked PARTIAL; see `tidal/AGENTS.md` § Known
//! Gaps for G1/G2/G3.
/// Public modules exposed for integration testing.
///
/// The binary (`main.rs`) imports from this lib crate rather than declaring
/// the modules directly, so integration tests in `tests/` can access them.
pub mod cluster;
pub mod config;
pub mod dto;
pub mod error;
pub mod router;
pub mod scatter_gather;

95
tidal-server/src/main.rs
View File

@ -1,13 +1,13 @@
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;
use std::{net::SocketAddr, path::PathBuf, sync::Arc};
use clap::{Args, Parser, Subcommand};
use tidal_server::cluster::{ClusterState, build_cluster_router, load_topology};
use tidal_server::config::load_schema;
use tidal_server::error::{Result, ServerError};
use tidal_server::router::build_router;
use tidal_server::state::ServerState;
use tidal_server::{
cluster::{ClusterState, build_cluster_router, ensure_experimental_enabled, load_topology},
config::{CONFIG_DIR_SCHEMA_FILE, CONFIG_DIR_TOPOLOGY_FILE, load_schema, resolve_config_path},
error::{Result, ServerError},
router::build_router,
state::ServerState,
};
use tidaldb::TidalDb;
#[derive(Parser)]
@ -33,6 +33,20 @@ struct ClusterArgs {
schema: Option<PathBuf>,
#[arg(long)]
topology: Option<PathBuf>,
/// Directory holding `default-schema.yaml` / `default-cluster.yaml`, used
/// when `--schema` / `--topology` are not given. Wired from the container's
/// `TIDAL_CONFIG` env var.
#[arg(long, env = "TIDAL_CONFIG")]
config_dir: Option<PathBuf>,
/// Opt in to the EXPERIMENTAL single-process cluster.
///
/// Cluster mode replicates between regions over the real tidal-net gRPC
/// transport (loopback), but every region runs inside THIS one process —
/// there is no host/process isolation, so it is not production HA. It
/// refuses to start without this flag (or the
/// `TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1` env var).
#[arg(long)]
experimental_cluster: bool,
}
#[derive(Args)]
@ -41,6 +55,10 @@ struct StandaloneArgs {
listen: String,
#[arg(long)]
schema: Option<PathBuf>,
/// Directory holding `default-schema.yaml`, used when `--schema` is not
/// given. Wired from the container's `TIDAL_CONFIG` env var.
#[arg(long, env = "TIDAL_CONFIG")]
config_dir: Option<PathBuf>,
#[arg(long)]
data_dir: Option<PathBuf>,
#[arg(
@ -90,7 +108,12 @@ fn init_tracing() {
}
async fn run_standalone(args: StandaloneArgs) -> Result<()> {
let (schema, profiles) = load_schema(args.schema.as_deref())?;
let schema_path = resolve_config_path(
args.schema.as_deref(),
args.config_dir.as_deref(),
CONFIG_DIR_SCHEMA_FILE,
)?;
let (schema, profiles) = load_schema(schema_path.as_deref())?;
let mut builder = TidalDb::builder()
.with_schema(schema.clone())
@ -115,8 +138,26 @@ async fn run_standalone(args: StandaloneArgs) -> Result<()> {
}
async fn run_cluster(args: ClusterArgs) -> Result<()> {
let (schema, profiles) = load_schema(args.schema.as_deref())?;
let topology = load_topology(args.topology.as_deref())?;
// Honest gate: cluster mode replicates between regions over the real
// tidal-net gRPC transport, but every region runs inside this single
// process (no host/process isolation), so it is not production HA. Refuse
// to start (and emit a loud WARN when permitted) so no operator mistakes it
// for production HA.
ensure_experimental_enabled(args.experimental_cluster)?;
let schema_path = resolve_config_path(
args.schema.as_deref(),
args.config_dir.as_deref(),
CONFIG_DIR_SCHEMA_FILE,
)?;
let topology_path = resolve_config_path(
args.topology.as_deref(),
args.config_dir.as_deref(),
CONFIG_DIR_TOPOLOGY_FILE,
)?;
let (schema, profiles) = load_schema(schema_path.as_deref())?;
let topology = load_topology(topology_path.as_deref())?;
tracing::info!(
regions = topology.regions.len(),
@ -124,7 +165,16 @@ async fn run_cluster(args: ClusterArgs) -> Result<()> {
"building cluster"
);
let state = ClusterState::new(&topology, schema, profiles)?;
// `ClusterState::new` starts the follower gRPC servers via
// `GrpcTransport::new`, which blocks on its own tokio runtime and must not
// run inside this reactor. Build it on a dedicated thread; the constructed
// state (and its embedded runtimes) then lives for the server's lifetime.
let state = std::thread::Builder::new()
.name("cluster-build".into())
.spawn(move || ClusterState::new(&topology, schema, profiles))
.map_err(|e| ServerError::Cluster(format!("spawn cluster builder thread: {e}")))?
.join()
.map_err(|_| ServerError::Cluster("cluster builder thread panicked".into()))??;
let api_key = read_api_key();
serve_cluster(state, &args.listen, api_key).await
@ -143,8 +193,23 @@ async fn serve_cluster(state: ClusterState, addr: &str, api_key: Option<Arc<str>
let shutdown_state = state.clone();
axum::serve(listener, build_cluster_router(state, api_key))
.with_graceful_shutdown(cluster_shutdown_signal(shutdown_state))
.with_graceful_shutdown(cluster_shutdown_signal(shutdown_state.clone()))
.await?;
// axum::serve has returned, so the router (and every `Arc<ClusterState>` it
// held) is dropped. We should now be the sole owner; reclaim ownership and
// shut every node down deterministically (checkpoint + WAL fsync + thread
// join) before the process exits. If an Arc unexpectedly lingers, the
// `Drop` backstop on `ClusterState` still runs the same shutdown.
match Arc::try_unwrap(shutdown_state) {
Ok(mut owned) => owned.shutdown(),
Err(arc) => {
tracing::warn!(
strong = Arc::strong_count(&arc),
"cluster state still shared after serve returned; relying on Drop for shutdown"
);
}
}
Ok(())
}
@ -171,7 +236,7 @@ async fn cluster_shutdown_signal(state: Arc<ClusterState>) {
tracing::warn!("ctrl-c handler error: {err}");
}
}
_ = sigterm => {}
() = sigterm => {}
}
state.set_shutting_down();
@ -240,7 +305,7 @@ async fn shutdown_signal(state: Arc<ServerState>) {
tracing::warn!("ctrl-c handler error: {err}");
}
}
_ = sigterm => {}
() = sigterm => {}
}
// Flip readiness BEFORE axum starts draining

216
tidal-server/src/router.rs
View File

@ -1,31 +1,40 @@
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use axum::Json;
use axum::Router;
use axum::extract::{Query, Request, State};
use axum::http::StatusCode;
use axum::http::header::AUTHORIZATION;
use axum::middleware::{self, Next};
use axum::response::{IntoResponse, Response};
use axum::routing::{get, post};
use serde::{Deserialize, Serialize};
use subtle::ConstantTimeEq;
use tidaldb::query::retrieve::Retrieve;
use tidaldb::query::search::Search;
use tidaldb::schema::EntityId;
use tower::ServiceBuilder;
use tower::limit::ConcurrencyLimitLayer;
use tower_http::request_id::{
MakeRequestId, PropagateRequestIdLayer, RequestId, SetRequestIdLayer,
use std::{
collections::HashMap,
sync::{
Arc,
atomic::{AtomicU64, Ordering},
},
time::Duration,
};
use tower_http::timeout::TimeoutLayer;
use tower_http::trace::TraceLayer;
use crate::error::{Result, ServerError};
use crate::state::ServerState;
use axum::{
Json, Router,
extract::{Query, Request, State},
http::{StatusCode, header::AUTHORIZATION},
middleware::{self, Next},
response::{IntoResponse, Response},
routing::{get, post},
};
use subtle::ConstantTimeEq;
use tidaldb::{
query::{retrieve::Retrieve, search::Search},
schema::EntityId,
};
use tower::{ServiceBuilder, limit::ConcurrencyLimitLayer};
use tower_http::{
request_id::{MakeRequestId, PropagateRequestIdLayer, RequestId, SetRequestIdLayer},
timeout::TimeoutLayer,
trace::TraceLayer,
};
use crate::{
dto::{
EmbeddingRequest, FeedQuery, FeedResponse, ItemRequest, SearchQueryParams, SearchResponse,
SignalRequest, feed_items, search_items,
},
error::{Result, ServerError},
state::ServerState,
};
/// Maximum request body size. Requests exceeding this are rejected with 413
/// before any deserialization occurs.
@ -155,15 +164,12 @@ pub async fn bearer_auth(request: Request, next: Next, expected_key: &str) -> Re
}
});
let authorized = match token {
Some(t) => {
let a = t.as_bytes();
let b = expected_key.as_bytes();
// Length mismatch does not leak token content; short-circuit is safe.
a.len() == b.len() && bool::from(a.ct_eq(b))
}
None => false,
};
let authorized = token.is_some_and(|t| {
let a = t.as_bytes();
let b = expected_key.as_bytes();
// Length mismatch does not leak token content; short-circuit is safe.
a.len() == b.len() && bool::from(a.ct_eq(b))
});
if authorized {
next.run(request).await
@ -177,12 +183,6 @@ pub async fn bearer_auth(request: Request, next: Next, expected_key: &str) -> Re
}
}
#[derive(Deserialize)]
struct ItemRequest {
entity_id: u64,
metadata: HashMap<String, String>,
}
async fn create_item(
State(state): State<Arc<ServerState>>,
Json(req): Json<ItemRequest>,
@ -193,12 +193,6 @@ async fn create_item(
Ok(StatusCode::CREATED)
}
#[derive(Deserialize)]
struct EmbeddingRequest {
entity_id: u64,
values: Vec<f32>,
}
async fn write_embedding(
State(state): State<Arc<ServerState>>,
Json(req): Json<EmbeddingRequest>,
@ -209,17 +203,6 @@ async fn write_embedding(
Ok(StatusCode::NO_CONTENT)
}
#[derive(Deserialize)]
struct SignalRequest {
entity_id: u64,
signal: String,
weight: f64,
#[serde(default)]
user_id: Option<u64>,
#[serde(default)]
creator_id: Option<u64>,
}
async fn write_signal(
State(state): State<Arc<ServerState>>,
Json(req): Json<SignalRequest>,
@ -236,48 +219,6 @@ async fn write_signal(
Ok(StatusCode::NO_CONTENT)
}
#[derive(Deserialize)]
struct FeedQuery {
#[serde(default)]
user_id: Option<u64>,
#[serde(default = "default_profile")]
profile: String,
#[serde(default = "default_limit")]
limit: u32,
#[serde(default)]
region: Option<String>,
}
fn default_profile() -> String {
"for_you".into()
}
fn default_limit() -> u32 {
20
}
#[derive(Serialize)]
struct FeedResponse {
items: Vec<FeedItem>,
total_candidates: usize,
region: Option<String>,
}
#[derive(Serialize)]
struct FeedItem {
entity_id: u64,
score: f64,
rank: usize,
#[serde(skip_serializing_if = "Option::is_none")]
signals: Option<Vec<SignalValue>>,
}
#[derive(Serialize)]
struct SignalValue {
name: String,
value: f64,
}
async fn feed(
State(state): State<Arc<ServerState>>,
Query(query): Query<FeedQuery>,
@ -295,65 +236,13 @@ async fn feed(
.retrieve(query.region.as_deref(), &retrieve)
.map_err(AppError)?;
let mut items = Vec::with_capacity(result.items.len());
for item in result.items {
let signals = if item.signals.is_empty() {
None
} else {
Some(
item.signals
.iter()
.map(|s| SignalValue {
name: s.name.clone(),
value: s.value,
})
.collect(),
)
};
items.push(FeedItem {
entity_id: item.entity_id.as_u64(),
score: item.score,
rank: item.rank,
signals,
});
}
Ok(Json(FeedResponse {
items,
items: feed_items(&result.items),
total_candidates: result.total_candidates,
region: query.region,
}))
}
#[derive(Deserialize)]
struct SearchQueryParams {
query: String,
#[serde(default)]
user_id: Option<u64>,
#[serde(default = "default_limit")]
limit: u32,
#[serde(default)]
region: Option<String>,
}
#[derive(Serialize)]
struct SearchResponse {
items: Vec<SearchItem>,
total_candidates: usize,
region: Option<String>,
}
#[derive(Serialize)]
struct SearchItem {
entity_id: u64,
score: f64,
rank: usize,
#[serde(skip_serializing_if = "Option::is_none")]
bm25_score: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
semantic_score: Option<f64>,
}
async fn search(
State(state): State<Arc<ServerState>>,
Query(query): Query<SearchQueryParams>,
@ -370,20 +259,8 @@ async fn search(
.search(query.region.as_deref(), &search)
.map_err(AppError)?;
let items = result
.items
.into_iter()
.map(|item| SearchItem {
entity_id: item.entity_id.as_u64(),
score: item.score,
rank: item.rank,
bm25_score: item.bm25_score.map(f64::from),
semantic_score: item.semantic_score.map(f64::from),
})
.collect();
Ok(Json(SearchResponse {
items,
items: search_items(&result.items),
total_candidates: result.total_candidates,
region: query.region,
}))
@ -415,7 +292,7 @@ async fn health(
));
}
let region = query.get("region").map(|s| s.as_str());
let region = query.get("region").map(std::string::String::as_str);
let items = state.item_count(region).map_err(AppError)?;
Ok((
@ -433,7 +310,7 @@ struct TidalErrorWrapper(tidaldb::TidalError);
impl From<TidalErrorWrapper> for AppError {
fn from(value: TidalErrorWrapper) -> Self {
AppError(ServerError::Tidal(value.0))
Self(ServerError::Tidal(value.0))
}
}
@ -449,7 +326,8 @@ impl IntoResponse for AppError {
}
}
pub fn status_from_error(err: &ServerError) -> StatusCode {
#[must_use]
pub const fn status_from_error(err: &ServerError) -> StatusCode {
match err {
ServerError::BadRequest(_) | ServerError::SchemaConfig(_) => StatusCode::BAD_REQUEST,
ServerError::Tidal(tidal_err) => match tidal_err {

1007
tidal-server/src/scatter_gather.rs
View File

File diff suppressed because it is too large Load Diff

70
tidal-server/src/state.rs
View File

@ -1,11 +1,16 @@
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::{
collections::HashMap,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
},
};
use tidaldb::TidalDb;
use tidaldb::query::retrieve::Retrieve;
use tidaldb::query::search::Search;
use tidaldb::schema::EntityId;
use tidaldb::{
TidalDb,
query::{retrieve::Retrieve, search::Search},
schema::EntityId,
};
use crate::error::{Result, ServerError};
@ -31,8 +36,18 @@ pub struct ServerState {
impl ServerState {
pub fn new(db: TidalDb) -> Self {
// Wrap the db in the Arc that lives for the whole server lifetime, then
// start the session-TTL sweeper against THAT Arc. This is the long-lived
// handle the sweeper's `Weak` must upgrade to — `builder.open()` cannot
// start it (a by-value `TidalDb` has no stable self-address), so without
// this the sweeper was dead in every persistent deployment
// (CONCURRENCY-1). `start_sweeper` only spawns a thread in persistent
// mode; ephemeral standalone never persists sessions, so the call is a
// cheap no-op there.
let db = Arc::new(db);
TidalDb::start_sweeper(&db);
Self {
db: Arc::new(db),
db,
shutting_down: Arc::new(AtomicBool::new(false)),
}
}
@ -41,10 +56,16 @@ impl ServerState {
self.shutting_down.store(true, Ordering::SeqCst);
}
#[must_use]
pub fn is_shutting_down(&self) -> bool {
self.shutting_down.load(Ordering::SeqCst)
}
/// Write an item and its metadata to the embedded DB.
///
/// # Errors
///
/// Returns [`ServerError`] if the underlying DB write fails.
pub fn write_item(
&self,
entity_id: EntityId,
@ -55,12 +76,22 @@ impl ServerState {
.map_err(ServerError::from)
}
/// Write an item embedding to the embedded DB.
///
/// # Errors
///
/// Returns [`ServerError`] if the underlying DB write fails.
pub fn write_embedding(&self, entity_id: EntityId, embedding: &[f32]) -> Result<()> {
self.db
.write_item_embedding(entity_id, embedding)
.map_err(ServerError::from)
}
/// Record a signal, with optional user/creator context.
///
/// # Errors
///
/// Returns [`ServerError`] if the underlying DB signal write fails.
pub fn signal(
&self,
signal_name: &str,
@ -92,6 +123,12 @@ impl ServerState {
}
}
/// Retrieve and rank items for the (standalone) region.
///
/// # Errors
///
/// Returns [`ServerError`] if a region is specified (standalone mode is
/// single-region) or the underlying DB retrieve fails.
pub fn retrieve(
&self,
region_name: Option<&str>,
@ -107,11 +144,22 @@ impl ServerState {
/// On-disk indexes auto-reload via `OnCommitWithDelay`; ephemeral indexes
/// (the default for standalone mode) use `ReloadPolicy::Manual` and require
/// an explicit reload before each search.
///
/// # Errors
///
/// Returns [`ServerError`] if a region is specified (standalone mode is
/// single-region) or the reader reload fails.
pub fn reload_text_index(&self, region_name: Option<&str>) -> Result<()> {
ensure_standalone(region_name)?;
self.db.reload_text_index().map_err(ServerError::from)
}
/// Full-text search within the (standalone) region.
///
/// # Errors
///
/// Returns [`ServerError`] if a region is specified (standalone mode is
/// single-region) or the underlying DB search fails.
pub fn search(
&self,
region_name: Option<&str>,
@ -121,6 +169,12 @@ impl ServerState {
self.db.search(query).map_err(ServerError::from)
}
/// Count items in the (standalone) region.
///
/// # Errors
///
/// Returns [`ServerError`] if a region is specified (standalone mode is
/// single-region).
pub fn item_count(&self, region_name: Option<&str>) -> Result<u64> {
ensure_standalone(region_name)?;
Ok(self.db.item_count())

86
tidal-server/tests/cluster_e2e.rs
View File

@ -10,12 +10,16 @@
//! cargo test -p tidal-server --features cluster-e2e --test cluster_e2e -- --nocapture
//! ```
#![cfg(feature = "cluster-e2e")]
// Tier-3 E2E harness: unwrap on known-good fixtures is idiomatic test noise.
#![allow(clippy::unwrap_used)]
use std::io::Write;
use std::net::{SocketAddr, TcpListener};
use std::path::PathBuf;
use std::process::{Child, Command};
use std::time::{Duration, Instant};
use std::{
io::Write,
net::{SocketAddr, TcpListener},
path::{Path, PathBuf},
process::{Child, Command},
time::{Duration, Instant},
};
/// A running tidal-server cluster node.
struct NodeHandle {
@ -30,9 +34,11 @@ impl Drop for NodeHandle {
// Best-effort graceful shutdown.
#[cfg(unix)]
{
unsafe {
libc::kill(self.process.id() as i32, libc::SIGTERM);
}
// Send SIGTERM via the `kill` binary so this crate stays
// `unsafe_code = forbid` — no `libc::kill` FFI needed.
let _ = Command::new("kill")
.args(["-TERM", &self.process.id().to_string()])
.status();
// Wait up to 2s for graceful exit.
let deadline = Instant::now() + Duration::from_secs(2);
loop {
@ -74,8 +80,8 @@ impl ClusterHarness {
// Write config files.
let config_dir = tempfile::tempdir().expect("create temp dir");
let topology = Self::write_topology(&config_dir.path().to_path_buf(), &region_names);
let schema = Self::write_schema(&config_dir.path().to_path_buf());
let topology = Self::write_topology(config_dir.path(), &region_names);
let schema = Self::write_schema(config_dir.path());
// Find the tidal-server binary.
let bin = tidal_server_bin();
@ -91,6 +97,10 @@ impl ClusterHarness {
.arg(&schema)
.arg("--topology")
.arg(&topology)
// Cluster mode is experimental (regions replicate over real
// gRPC but share one process — no host/process isolation) and
// refuses to start without this opt-in.
.env("TIDAL_ALLOW_EXPERIMENTAL_CLUSTER", "1")
.env("TIDAL_SERVER_LOG", "warn")
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
@ -109,12 +119,15 @@ impl ClusterHarness {
_config_dir: config_dir,
};
// Wait for all nodes to become healthy.
harness.wait_all_healthy(Duration::from_secs(15));
// Wait for all nodes to become healthy. Each node boots a real gRPC
// transport (its own tokio runtime + server) per follower region on top
// of opening its tidalDB instances, and the suite may start several
// such processes concurrently, so allow a generous tier-3 budget.
harness.wait_all_healthy(Duration::from_secs(30));
harness
}
fn write_topology(dir: &PathBuf, regions: &[String]) -> PathBuf {
fn write_topology(dir: &Path, regions: &[String]) -> PathBuf {
let path = dir.join("topology.yaml");
let mut f = std::fs::File::create(&path).expect("create topology.yaml");
writeln!(f, "regions:").unwrap();
@ -125,11 +138,11 @@ impl ClusterHarness {
path
}
fn write_schema(dir: &PathBuf) -> PathBuf {
fn write_schema(dir: &Path) -> PathBuf {
let path = dir.join("schema.yaml");
std::fs::write(
&path,
r#"signals:
r"signals:
- name: view
entity: item
decay:
@ -151,7 +164,7 @@ embedding_slots:
- name: content_vector
entity: item
dimensions: 4
"#,
",
)
.expect("write schema.yaml");
path
@ -184,16 +197,17 @@ embedding_slots:
.unwrap_or_else(|e| panic!("POST {url} failed: {e}"))
}
/// Poll `/cluster/status` until all follower regions show lag_events == 0, or timeout.
/// Poll `/cluster/status` until all follower regions show `lag_events == 0`, or timeout.
///
/// The leader is excluded because it writes directly (not via replication)
/// and its `applied_events` counter tracks replication from other shards.
fn wait_converged(&self, timeout: Duration) {
let deadline = Instant::now() + timeout;
loop {
if Instant::now() > deadline {
panic!("convergence timeout after {timeout:?}; cluster did not converge");
}
assert!(
Instant::now() <= deadline,
"convergence timeout after {timeout:?}; cluster did not converge"
);
let resp = self.http_get(0, "/cluster/status");
if resp.status().is_success() {
let body: serde_json::Value = resp.json().unwrap();
@ -221,10 +235,21 @@ fn free_addr() -> SocketAddr {
/// Find the tidal-server binary. Build it first if needed.
fn tidal_server_bin() -> PathBuf {
// Try the standard cargo target directory.
let manifest_dir = env!("CARGO_MANIFEST_DIR");
let workspace_root = std::path::Path::new(manifest_dir)
.parent()
// Resolve the workspace root that owns the cargo `target/` dir. tidal-server
// is a workspace member at `<workspace>/tidal-server`, so the workspace root
// is the parent of `CARGO_MANIFEST_DIR`. Probe the parent first, then the
// grandparent, so the harness keeps working regardless of how the crate is
// nested in a workspace.
let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
let candidates = [
manifest_dir.parent(),
manifest_dir.parent().and_then(std::path::Path::parent),
];
let workspace_root = candidates
.into_iter()
.flatten()
.find(|root| root.join("target").is_dir() || root.join("Cargo.toml").is_file())
.or_else(|| manifest_dir.parent())
.expect("workspace root");
// Build the binary to make sure it's up to date.
@ -240,7 +265,11 @@ fn tidal_server_bin() -> PathBuf {
assert!(status.success(), "cargo build -p tidal-server failed");
let bin = workspace_root.join("target/debug/tidal-server");
assert!(bin.exists(), "tidal-server binary not found at {bin:?}");
assert!(
bin.exists(),
"tidal-server binary not found at {}",
bin.display()
);
bin
}
@ -248,9 +277,10 @@ fn tidal_server_bin() -> PathBuf {
fn wait_for_health(addr: &SocketAddr, deadline: Instant) {
let url = format!("http://{addr}/health/startup");
loop {
if Instant::now() > deadline {
panic!("health check timeout for {addr}");
}
assert!(
Instant::now() <= deadline,
"health check timeout for {addr}"
);
match reqwest::blocking::get(&url) {
Ok(resp) if resp.status().is_success() => return,
_ => std::thread::sleep(Duration::from_millis(100)),

210
tidal-server/tests/cluster_grpc.rs Normal file
View File

@ -0,0 +1,210 @@
//! m8p8 in-process gRPC replication tests.
//!
//! Unlike `cluster_e2e.rs` (tier-3, feature-gated, spawns OS processes), these
//! run in the default test build. They prove that `ClusterState` replicates
//! between regions over the **real `tidal-net` gRPC transport** — closing gap
//! G1 (`ClusterState` previously wrapped crossbeam channels) — and that the
//! async HTTP write path offloads its blocking gRPC ship correctly.
#![allow(clippy::unwrap_used, clippy::missing_panics_doc)]
use std::{
sync::Arc,
time::{Duration, Instant},
};
use tidal_server::cluster::{ClusterState, RegionSpec, TopologySpec, build_cluster_router};
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, Schema, SchemaBuilder, Window};
/// Three-region topology (leader us-east) with auto-allocated loopback gRPC
/// ports — the production default path.
fn three_region_topology() -> TopologySpec {
TopologySpec {
regions: vec![
RegionSpec {
name: "us-east".into(),
grpc_addr: None,
},
RegionSpec {
name: "eu-west".into(),
grpc_addr: None,
},
RegionSpec {
name: "ap-south".into(),
grpc_addr: None,
},
],
leader: "us-east".into(),
}
}
fn view_schema() -> Schema {
let mut builder = SchemaBuilder::new();
let _ = builder
.signal(
"view",
EntityKind::Item,
DecaySpec::Exponential {
half_life: Duration::from_secs(7 * 24 * 3600),
},
)
.windows(&[Window::OneHour])
.velocity(false)
.add();
builder.build().unwrap()
}
/// Direct proof: writes to the leader replicate to every follower over gRPC,
/// and the followers' decay scores converge to the leader's.
#[test]
fn grpc_replication_converges_across_followers() {
// Plain (non-async) test thread: ClusterState::new starts the follower gRPC
// servers via GrpcTransport::new, which blocks on its own runtime.
let state = ClusterState::new(&three_region_topology(), view_schema(), Vec::new())
.expect("cluster builds with gRPC transports");
let cluster = state.cluster();
let leader = cluster.leader_region();
let followers: Vec<_> = cluster
.regions()
.into_iter()
.filter(|&r| r != leader)
.collect();
assert_eq!(followers.len(), 2, "expected two follower regions");
// Write 10 signals to the leader; each ships over gRPC to both followers.
for i in 1..=10u64 {
cluster
.write_signal("view", EntityId::new(i), 1.0)
.expect("leader signal write + gRPC ship");
}
// Poll until both followers have applied all 10 leader segments.
let target = cluster.leader_seqno();
assert_eq!(target, 10, "leader should have sequenced 10 writes");
let deadline = Instant::now() + Duration::from_secs(5);
loop {
let converged = followers
.iter()
.all(|&r| cluster.applied_count(r) >= target);
if converged {
break;
}
assert!(
Instant::now() <= deadline,
"followers did not converge over gRPC within 5s: applied = {:?}, target = {target}",
followers
.iter()
.map(|&r| cluster.applied_count(r))
.collect::<Vec<_>>()
);
std::thread::sleep(Duration::from_millis(20));
}
// Decay scores must match the leader's to 6 decimals on every follower.
for i in 1..=10u64 {
let eid = EntityId::new(i);
let l = cluster.read_decay_score(leader, eid, "view").unwrap_or(0.0);
for &f in &followers {
let got = cluster.read_decay_score(f, eid, "view").unwrap_or(0.0);
assert!(
(l - got).abs() < 1e-6,
"entity {i}: leader={l} follower={got}"
);
}
}
}
/// Full HTTP path: POST /signals (which offloads its blocking gRPC ship off the
/// reactor) replicates to followers, and a `?region=` read serves the follower.
#[test]
fn http_signal_replicates_and_region_read_serves_follower() {
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.unwrap();
// Build the cluster off the reactor (we are on a plain test thread).
let state = ClusterState::new(&three_region_topology(), view_schema(), Vec::new())
.expect("cluster builds with gRPC transports");
let router = build_cluster_router(Arc::new(state), None);
// Bind + serve on the runtime; keep the test thread free for blocking HTTP.
let listener = rt
.block_on(tokio::net::TcpListener::bind("127.0.0.1:0"))
.unwrap();
let addr = listener.local_addr().unwrap();
rt.spawn(async move {
let _ = axum::serve(listener, router).await;
});
let base = format!("http://{addr}");
let client = reqwest::blocking::Client::new();
// Broadcast items to all regions, then write signals (replicated to followers).
for i in 1..=5u64 {
let resp = client
.post(format!("{base}/items"))
.json(&serde_json::json!({
"entity_id": i,
"metadata": { "title": format!("item {i}") }
}))
.send()
.unwrap();
assert!(resp.status().is_success(), "POST /items: {}", resp.status());
let resp = client
.post(format!("{base}/signals"))
.json(&serde_json::json!({ "entity_id": i, "signal": "view", "weight": 1.0 }))
.send()
.unwrap();
assert!(
resp.status().is_success(),
"POST /signals: {}",
resp.status()
);
}
// Poll /cluster/status until every follower reports zero lag.
let deadline = Instant::now() + Duration::from_secs(5);
loop {
let status: serde_json::Value = client
.get(format!("{base}/cluster/status"))
.send()
.unwrap()
.json()
.unwrap();
let leader = status["leader"].as_str().unwrap_or("");
let converged = status["regions"]
.as_array()
.unwrap()
.iter()
.filter(|r| r["name"].as_str().unwrap_or("") != leader)
.all(|r| r["lag_events"].as_u64().unwrap_or(u64::MAX) == 0);
if converged {
break;
}
assert!(
Instant::now() <= deadline,
"followers did not converge over gRPC within 5s: {status}"
);
std::thread::sleep(Duration::from_millis(50));
}
// A region-pinned read on a follower must return the replicated items.
let feed: serde_json::Value = client
.get(format!(
"{base}/feed?profile=trending&limit=5&region=eu-west"
))
.send()
.unwrap()
.json()
.unwrap();
let items = feed["items"].as_array().unwrap();
assert!(
!items.is_empty(),
"follower eu-west should serve replicated items, got: {feed}"
);
rt.shutdown_timeout(Duration::from_secs(2));
}

11
tidal-server/tests/middleware.rs
View File

@ -1,3 +1,6 @@
// Integration-test exemption (same posture as the tidaldb integration tests):
// unwrap on known-good fixtures is idiomatic here.
#![allow(clippy::unwrap_used)]
//! Integration tests for the tidal-server middleware stack.
//!
//! Each test spins up an in-process router using `tower::ServiceExt::oneshot`
@ -6,8 +9,10 @@
use std::sync::Arc;
use axum::body::Body;
use axum::http::{Method, Request, StatusCode};
use axum::{
body::Body,
http::{Method, Request, StatusCode},
};
use tidal_server::{router::build_router, state::ServerState};
use tidaldb::TidalDb;
use tower::ServiceExt;
@ -28,7 +33,7 @@ fn make_state() -> Arc<ServerState> {
}
fn make_app(api_key: Option<&str>) -> axum::Router {
let key = api_key.map(|k| Arc::from(k));
let key = api_key.map(Arc::from);
build_router(make_state(), key)
}

98
tidal/AGENTS.md Normal file
View File

@ -0,0 +1,98 @@
# AGENTS.md
Agent instructions for tidalDB.
## Team
| Agent | Identity | Model | Invoke when |
|-------|----------|-------|-------------|
| `@tidal-engineer` | Jon Gjengset — principal Rust database engineer | opus | Implementing features, storage internals, signal system, query engine, debugging correctness |
| `@tidal-visionary` | Spencer Kimball — product and roadmap strategist | opus | Planning milestones, scoping phases, build-vs-defer decisions, roadmap sequencing |
| `@tidal-researcher` | Andy Pavlo — database systems researcher | opus | Prior art surveys, library evaluation, architectural research, producing `docs/research/` docs |
| `@tidal-storyteller` | Marketing and technical writer | sonnet | Marketing site (`site/`), blog posts, public-facing copy |
Agent definitions live in `.claude/agents/`. Full context in `CLAUDE.md §Agents`.
---
<!-- sdlc:start -->
## SDLC
> **Required reading:** `.sdlc/guidance.md` — engineering principles that govern all implementation decisions on this project. <!-- sdlc:guidance -->
This project uses `sdlc` as its SDLC state machine. `sdlc` manages feature lifecycle, artifacts, tasks, and milestones. It emits structured directives via `sdlc next --json` that any consumer (Claude Code, custom scripts, or humans) acts on to decide what to do next.
Consumer scaffolding is installed globally under `~/.claude/commands/`, `~/.gemini/commands/`, `~/.opencode/command/`, and `~/.agents/skills/` — available across all projects. Use `/sdlc-specialize` in Claude Code to generate a project-specific AI team (agents + skills) tailored to this project's tech stack and roles.
### Key Commands
- `sdlc feature create <slug> --title "..."` — create a new feature
- `sdlc next --for <slug> --json` — get the next action directive (JSON)
- `sdlc next` — show all active features and their next actions
- `sdlc artifact approve <slug> <type>` — approve an artifact to advance the phase
- `sdlc state` — show project state
- `sdlc feature list` — list all features and their phases
- `sdlc task list [<slug>]` — list tasks for a feature (or all tasks)
### Lifecycle
draft → specified → planned → ready → implementation → review → audit → qa → merge → released
Treat this lifecycle as the default pathway. You can use explicit manual transitions when needed, but approvals/artifacts are the recommended way to keep quality and traceability.
### Artifact Types
`spec` `design` `tasks` `qa_plan` `review` `audit` `qa_results`
### CRITICAL: Never edit .sdlc/ YAML directly
All state changes go through `sdlc` CLI commands. See §6 of `.sdlc/guidance.md` for the full command reference. Direct YAML edits corrupt state.
### Directive Interface
Use `sdlc next --for <slug> --json` to get the next directive. The JSON output tells the consumer what to do next (action, message, output_path, is_heavy, gates).
### Consumer Commands
- `/sdlc-next <slug>` — execute one step, then stop (human controls cadence)
- `/sdlc-run <slug>` — run autonomously to completion
- `/sdlc-status [<slug>]` — show current state
- `/sdlc-plan` — distribute a plan into milestones, features, and tasks
- `/sdlc-milestone-uat <milestone-slug>` — run the acceptance test for a milestone
- `/sdlc-pressure-test <milestone-slug>` — pressure-test a milestone against user perspectives
- `/sdlc-vision-adjustment [description]` — align all docs, sdlc state, and code to a vision change
- `/sdlc-architecture-adjustment [description]` — align all docs, code, and sdlc state to an architecture change
- `/sdlc-enterprise-readiness [--stage <stage>]` — analyze production readiness
- `/sdlc-setup-quality-gates` — set up pre-commit hooks and quality gates
- `/sdlc-cookbook <milestone-slug>` — create developer-scenario cookbook recipes
- `/sdlc-cookbook-run <milestone-slug>` — execute cookbook recipes and record results
- `/sdlc-ponder [slug]` — open the ideation workspace for exploring and committing ideas
- `/sdlc-ponder-commit <slug>` — crystallize a pondered idea into milestones and features
- `/sdlc-guideline <slug-or-problem>` — build an evidence-backed guideline through five research perspectives and TOC-first distillation
- `/sdlc-suggest` — analyze project state and suggest 3-5 ponder topics to explore next
- `/sdlc-beat [domain | feature:<slug> | --week]` — step back with a senior leadership lens; evaluate if we're building the right thing in the right direction; stores history in `.sdlc/beat.yaml`
- `/sdlc-recruit <role>` — recruit an expert thought partner as a persistent agent
- `/sdlc-empathy <subject>` — deep user perspective interviews before decisions
- `/sdlc-spike <slug> — <need>; [see <ref>]` — research, prototype, validate, and report; produces working prototype + findings in `.sdlc/spikes/<slug>/findings.md`
- `/sdlc-convo-mine [file or text]` — mine conversation dumps for signal; apply 5 perspective lenses, group themes, launch parallel ponder sessions per group
### Tool Suite
<!-- sdlc:tools -->
Project-scoped TypeScript tools in `.sdlc/tools/` — callable by agents and humans
during any lifecycle phase. Read `.sdlc/tools/tools.md` for the full help menu.
- `sdlc tool list` — show installed tools
- `sdlc tool run <name> [args]` — run a tool; pass `--json '{...}'` for complex input
- `sdlc tool sync` — regenerate `tools.md` after adding a custom tool
- `sdlc tool scaffold <name> "desc"` — create a new tool skeleton
**Core tools:** `ama` (codebase Q&A), `quality-check` (runs platform shell gates)
Use `/sdlc-tool-run`, `/sdlc-tool-build`, `/sdlc-tool-audit`, `/sdlc-tool-uat` in Claude Code for guided tool workflows.
<!-- /sdlc:tools -->
Project: tidalDB
<!-- sdlc:end -->

86
tidal/BUILD.bazel Normal file
View File

@ -0,0 +1,86 @@
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_proc_macro", "rust_test")
load("@rules_rust//cargo:defs.bzl", "cargo_build_script")
load("@crates//:defs.bzl", "aliases", "all_crate_deps")
load("//bazel:rust_cargo_tests.bzl", "rust_cargo_integration_tests")
cargo_build_script(
name = "build_script",
srcs = [
"build.rs",
],
crate_root = "build.rs",
edition = "2024",
aliases = aliases(build = True),
deps = all_crate_deps(build = True),
proc_macro_deps = all_crate_deps(build_proc_macro = True),
visibility = ["//visibility:private"],
)
rust_library(
name = "tidaldb",
srcs = glob(["src/**/*.rs", "src/*.rs"], allow_empty = True),
crate_root = "src/lib.rs",
crate_name = "tidaldb",
edition = "2024",
crate_features = [
"default",
"metrics",
],
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(),
deps = all_crate_deps(normal = True) + [
":build_script",
],
proc_macro_deps = all_crate_deps(proc_macro = True),
visibility = ["//visibility:public"],
)
rust_library(
name = "tidaldb_test_support",
srcs = glob(["src/**/*.rs", "src/*.rs"], allow_empty = True),
crate_root = "src/lib.rs",
crate_name = "tidaldb",
edition = "2024",
crate_features = [
"default",
"metrics",
"test-utils",
],
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(),
deps = all_crate_deps(normal = True) + [
":build_script",
],
proc_macro_deps = all_crate_deps(proc_macro = True),
visibility = ["//visibility:public"],
)
rust_test(
name = "tidaldb_test",
crate = ":tidaldb_test_support",
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True),
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
)
rust_cargo_integration_tests(
name_prefix = "tidaldb",
tests = glob(["tests/*.rs"]),
srcs = glob(["tests/**/*.rs"]),
compile_data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
data = glob(["**/*"], exclude = ["**/*.rs", "**/*.bazel", "BUILD", "WORKSPACE"], allow_empty = True),
aliases = aliases(normal_dev = True, proc_macro_dev = True),
deps = all_crate_deps(normal = True, normal_dev = True) + [
":tidaldb_test_support",
],
proc_macro_deps = all_crate_deps(proc_macro = True, proc_macro_dev = True),
# regression_guards reads tidal/CLAUDE.md and docker/standalone/Dockerfile via
# env!("CARGO_MANIFEST_DIR"); the baked compile-sandbox path is gone at run time, so these can't
# resolve under hermetic sandboxing. Passes under `cargo test`; excluded from the default
# `bazel test //...`, still built by `bazel build //...`.
sandbox_incompatible = [
"tests/regression_guards.rs",
],
)

127
tidal/CHANGELOG.md Normal file
View File

@ -0,0 +1,127 @@
# Changelog
All notable changes to tidalDB will be documented in this file.
## [Unreleased]
### Changed
**Cluster mode (m8p8): real gRPC replication**
- `tidal-server`'s `ClusterState` now wires each follower region to a real
`tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
crossbeam channels — replication between regions traverses real gRPC/TCP
(serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
per region) and self-heal a transient bind race by retrying on a fresh port.
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
offloaded to a dedicated thread so they never block the axum reactor.
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
blocks on its own runtime).
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
cluster mode is honest that it replicates over real gRPC but still runs all
regions in one process (no host/process isolation — true multi-process
deployment remains m8p10).
**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
`rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
image serves a functional 3-region cluster (write replicates to both followers
over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
+ promote over real OS processes, feature-gated).
## [0.1.0] - 2026-02-23
### Added
**Core Database Engine**
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration
**Signal System**
- Typed signal recording with exponential decay scoring
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
- Signal velocity tracking
- WAL-backed signal durability with crash recovery
- Periodic signal checkpointing to fjall (every 30s)
- WAL compaction after each checkpoint
**Retrieval (RETRIEVE query)**
- 5-stage pipeline: universe, filter, score, diversify, return
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
- Custom ranking profiles via `SchemaBuilder`
- Diversity enforcement (max N per category/creator)
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more
**Search (SEARCH query)**
- BM25 full-text search via Tantivy
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
- Creator search with `entity_kind(EntityKind::Creator)`
- `similar_to(EntityId)` for content-based recommendations
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
- Autocomplete suggestions via `db.suggest()`
**Entity Model**
- Three built-in entity types: `Item`, `User`, `Creator`
- Metadata storage as `HashMap<String, String>`
- Embedding slots (up to 4 per entity type) via USearch
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`
**Sessions**
- Session lifecycle: `open_session`, `close_session`
- Cross-session preference vector updates (EMA blend)
- Session snapshots with signal state and preference vectors
- Session serialization format v0x03 with backward compatibility
**Social Graph**
- Creator follower/following indexes
- Cohort membership (user segments)
- CoEngagementIndex for co-viewing patterns with LRU eviction
- Social graph filter for "followed creator" content scoping
**Collections**
- Named collections with `Private`, `Shared`, `Public` visibility
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
- `FilterExpr::InCollection` for collection-scoped retrieval
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`
**Observability**
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
- `TidalDb::diagnostics()` -- structured health snapshot
- WAL diagnostics and recovery tools
**Safety**
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
- Export request limit: 500K signals max per request
- `FilterExpr` complexity limit: 256 nodes max
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
- Schema fingerprint persistence detects decay parameter changes on reopen
- Bounded `closed_sessions` cache (10K max, LRU eviction)
- Metrics server non-loopback bind warning
**CLI (`tidalctl`)**
- `tidalctl` binary for database inspection and diagnostics
**RLHF / ML Export**
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics
### Stability
tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.
---
*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*

139
tidal/CLAUDE.md Normal file
View File

@ -0,0 +1,139 @@
> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
# tidalDB
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
**Status:** Implemented (M0–M8 shipped). The engine is the `tidaldb` crate at `tidal/`, with companion crates `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings. ~55K LOC, 1,209 unit tests + 188 integration tests + 5 crash-invariant property tests passing. The API surface is stable for implemented features; breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and known gaps.
## Find Your Guide
| If you need to... | Read this |
|-------------------|-----------|
| **Get started quickly** | [README.md](README.md) → [docs/QUICKSTART.md](docs/QUICKSTART.md) |
| **Understand the vision** | [docs/VISION.md](docs/VISION.md) |
| **See use cases and surfaces** | [docs/USE_CASES.md](docs/USE_CASES.md) |
| **See sequence diagrams** | [docs/SEQUENCE.md](docs/SEQUENCE.md) |
| **Understand the system architecture** | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) |
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
| **Follow coding standards** | [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) |
| **See the API spec** | [docs/API.md](docs/API.md) |
| **Read architectural lessons** | [docs/thoughts.md](docs/thoughts.md) |
| **Read technical research** | [docs/research/](docs/research/) |
## Agents
| Agent | Identity | Use when |
|-------|----------|----------|
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
| **@tidal-distributed** | Kyle Kingsbury | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |
## Skills
### Phase Lifecycle
| Step | Skill | Use when |
|------|-------|----------|
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
### Other Skills
| Skill | Use when |
|-------|----------|
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
| `/develop` | Quick implementation work outside the milestone lifecycle |
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
| `/build-site` | Creating or iterating on the marketing site |
| `/write-blog` | Writing blog posts about progress or architecture |
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |
## Core Domain Model
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
## Ports
Dev servers use port range **59520–59529** (e.g. `site/` on 59520). Any new tidalDB-adjacent service should claim a slot from this band.
## Critical Rules
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
- **Single-node first:** Embeddable. Scales vertically before horizontally.
- **Language:** Rust.
## Repository Structure
This repository is a Cargo workspace. The embeddable database engine is the `tidaldb` crate at `tidal/`, with three companion crates as workspace siblings and example consumers under `applications/`:
```
. (workspace root)
├── Cargo.toml # Workspace manifest
├── tidal/ # The embeddable database engine — package name = "tidaldb"
│ ├── Cargo.toml
│ ├── CLAUDE.md # This file — project instructions
│ ├── README.md # Public README + getting-started paths
│ ├── CHANGELOG.md # Milestone history (M0–M8) + known gaps
│ ├── CONTRIBUTING.md # Contributor workflow
│ ├── AGENTS.md # Agent roster + SDLC block
│ ├── src/ # Flat module layout
│ │ ├── cohort/ # Cohort-scoped signal aggregation
│ │ ├── db/ # Top-level TidalDb + builder
│ │ ├── entities/ # Item / User / Creator entity model
│ │ ├── load/ # Bulk load / ingest paths
│ │ ├── query/ # Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST)
│ │ ├── ranking/ # Profile engine, signal scoring, diversity enforcement
│ │ ├── replication/ # WAL-stream replication for cluster mode
│ │ ├── schema/ # Schema builder, validation, signal/profile defs
│ │ ├── session/ # Session + agent context, policies
│ │ ├── signals/ # Signal types, decay, velocity, windowed aggregation
│ │ ├── storage/ # Entity store, signal ledger, inverted index, HNSW
│ │ ├── testing/ # Shared test harness + fixtures
│ │ ├── text/ # Tantivy full-text indexing
│ │ └── wal/ # Write-ahead log + crash recovery
│ ├── benches/ # Performance benchmarks
│ ├── examples/ # quickstart, axum_embedding, actix_embedding, cli_embedding
│ ├── tests/ # Integration and crash-property tests
│ ├── docker/ # cluster / standalone / deploy Dockerfiles
│ ├── ai-lookup/ # Domain concept reference (index.md)
│ ├── site/ # Public marketing site (Next.js)
│ └── docs/ # Specs, API, guides, research, runbooks, ops
│ ├── VISION.md ARCHITECTURE.md API.md
│ ├── QUICKSTART.md SEQUENCE.md USE_CASES.md
│ ├── CODING_GUIDELINES.md thoughts.md
│ ├── specs/ # 00–14 component specifications
│ ├── research/ # Deep technical research docs
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
│ ├── ops/ # Monitoring, alerts, capacity planning
│ ├── profiling/ # Flamegraph + scale profiling notes
│ └── planning/ # Per-milestone phase/task planning archive
├── tidal-net/ # Network transport primitives
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/ # CLI tool for inspecting persisted databases
└── applications/ # Example consumers (forage, iknowyou)
```
## Pre-commit Hooks
The pre-commit hook runs automatically on staged files:
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
- **site/ (Next.js):** `eslint` (if node_modules installed)
Cargo commands target this crate with `-p tidaldb` from the workspace root (e.g. `cargo test -p tidaldb`, `cargo check -p tidaldb`), or equivalently `--manifest-path tidal/Cargo.toml`. The engine crate lives at `tidal/`.
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.

104
tidal/CONTRIBUTING.md Normal file
View File

@ -0,0 +1,104 @@
# Contributing to tidalDB
## Quick Start
tidalDB is the `tidaldb` crate at `tidal/` in this Cargo workspace. Run cargo
commands from the workspace root, targeting this crate with `-p tidaldb`.
```bash
# From the workspace root:
# Confirm the engine compiles and all tests pass
cargo test -p tidaldb
# Confirm doc tests and examples compile and run
cargo test -p tidaldb --doc
cargo test -p tidaldb --examples
```
## Run Samples Checklist
Before opening a PR that touches public API or examples, verify all samples still work:
```bash
# Doc tests (default features)
cargo test -p tidaldb --doc
# Doc tests with optional features
cargo test -p tidaldb --doc --features test-utils,metrics
# All four examples compile and run
cargo run -p tidaldb --example quickstart
cargo run -p tidaldb --example cli_embedding
cargo run -p tidaldb --example axum_embedding # Ctrl+C to stop
cargo run -p tidaldb --example actix_embedding # Ctrl+C to stop
```
Expected output for `quickstart`:
```
build: dev
uptime: 0.000s
health: ok
tidalDB opened, verified, and closed. M0 complete.
```
## Full Quality Gate
The pre-commit hook enforces these automatically on staged Rust files:
```bash
cargo fmt -p tidaldb -- --check
cargo clippy -p tidaldb -- -D warnings
cargo test -p tidaldb --lib
```
Run the complete gate manually:
```bash
cargo fmt -p tidaldb
cargo clippy -p tidaldb -- -D warnings
cargo test -p tidaldb
cargo bench -p tidaldb --no-run # ensure benches compile
```
## Project Layout
tidalDB is the `tidaldb` crate at `tidal/` in this Cargo workspace. Companion
crates `tidal-net`, `tidal-server`, and `tidalctl` are workspace siblings at
the repository root.
```
tidal/
src/ Flat module layout (engine source)
cohort/ Cohort-scoped signal aggregation
db/ TidalDb handle, builder, config, metrics
entities/ Item / User / Creator entity model
load/ Bulk load / ingest paths
query/ RETRIEVE / SEARCH / SUGGEST parser, planner, executor
ranking/ Profile engine, signal scoring, diversity enforcement
replication/ WAL-stream replication (cluster mode)
schema/ Schema builder, validation, signal/profile defs, error types
session/ Session + agent context, policies
signals/ Signal ledger, decay, velocity, windowed counters, checkpoint
storage/ StorageEngine trait, fjall backend, key encoding
testing/ Shared test harness + fixtures
text/ Tantivy full-text indexing
wal/ Write-ahead log, group commit, crash recovery
benches/ Criterion benchmarks
examples/ Embedding guides (quickstart, axum, actix, cli)
tests/ Integration and crash-property tests
docker/ cluster / standalone / deploy Dockerfiles
site/ Marketing site (Next.js)
docs/ Specs, API, guides, research, runbooks, ops, planning
```
## Coding Standards
See [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) for the full engineering standards.
Key rules:
- `Result<T, LumenError>` everywhere — no panics on recoverable failures
- `#![forbid(unsafe_code)]` — relaxed only at explicit FFI boundaries with `// SAFETY:` comment
- Property tests for invariants, criterion benchmarks for performance claims
- `cargo clippy -D warnings` must pass with zero warnings

48
tidal/Cargo.toml
View File

@ -38,6 +38,19 @@ tempfile = "3"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# ── tidal-crate lint posture (single source of truth) ──────────────────────
# The four tidal crates (tidaldb / tidal-net / tidal-server / tidalctl) carry an
# IDENTICAL, stricter-than-workspace [lints] posture and deliberately DO NOT
# inherit `[workspace.lints]` (`lints.workspace = false` is the Cargo default).
# Rationale: this is the embedded recommendation DB plus its transport, server,
# and CLI — a correctness-critical, mostly self-contained subsystem held to a
# higher bar (`unsafe_code = forbid`, `clippy::all = deny`, `unwrap_used = deny`).
# `unwrap_used = "deny"` is kept per-crate rather than promoted into
# `[workspace.lints]` because the workspace also hosts the example/consumer
# crates under `applications/`, which are not held to the engine's bar.
# Keeping the block per-crate turns an accidental divergence into an
# intentional, documented one.
# Keep these four blocks BYTE-IDENTICAL when changing one.
[lints.rust]
unsafe_code = "forbid"
@ -45,6 +58,9 @@ unsafe_code = "forbid"
all = { level = "deny", priority = -1 }
pedantic = { level = "warn", priority = -1 }
nursery = { level = "warn", priority = -1 }
# Justified allows (lossy numeric casts are pervasive + intentional in the
# ranking/scoring math; module_name_repetitions is idiomatic for the flat
# module layout documented in CLAUDE.md):
cast_possible_truncation = "allow"
module_name_repetitions = "allow"
unwrap_used = "deny"
@ -65,10 +81,38 @@ name = "cli_embedding"
name = "sandboxed_storage"
required-features = ["test-utils"]
[[test]]
name = "m9p1_scopes"
required-features = ["test-utils"]
[[test]]
name = "m9p2_membership"
required-features = ["test-utils"]
[[test]]
name = "m9p3_purge"
required-features = ["test-utils"]
[[test]]
name = "m10p1_governance"
required-features = ["test-utils"]
[[test]]
name = "m10p2_capabilities"
required-features = ["test-utils"]
[[test]]
name = "m10p3_provenance"
required-features = ["test-utils"]
[[test]]
name = "metrics_integration"
required-features = ["metrics"]
[[test]]
name = "m6_crash_surfaces"
required-features = ["test-utils"]
[[test]]
name = "m7_crash_property"
required-features = ["test-utils"]
@ -110,6 +154,10 @@ required-features = ["test-utils"]
[[test]]
name = "vector_usearch"
[[test]]
name = "m0m10_recovery"
required-features = ["test-utils"]
[[bench]]
name = "signals"
harness = false

285
tidal/README.md Normal file
View File

@ -0,0 +1,285 @@
# tidalDB
**An embeddable Rust database for the personalized content ranking problem.**
> Pre-release. API is stabilizing. Not yet recommended for production.
---
Every content platform eventually builds the same distributed system from scratch: Elasticsearch for retrieval, Redis for hot signals, Kafka for event ingestion, a feature store for user profiles, a vector database for semantic search, and a ranking service that stitches them together. The seams between those systems are where correctness dies — stale signals, inconsistent ranking, cache invalidation bugs, ETL lag.
The root cause: existing databases treat ranking as an afterthought. They have no native concept of signals that evolve over time, no understanding of user context, no diversity as a query constraint.
**Ranking is not a feature. It is a primitive.**
tidalDB is a single-node, embeddable Rust library built for one question: *given a user and a context, what content should they see, and in what order?* No server, no network protocol, no client SDK. Link it into your process.
---
## What it looks like
```rust
use std::collections::HashMap;
use std::time::Duration;
use tidaldb::{TidalDb, query::retrieve::Retrieve, schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window}};
// Declare signals with native decay — no application formulas.
let mut schema = SchemaBuilder::new();
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(7 * 24 * 3600),
}).windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
let _ = schema.signal("like", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(30 * 24 * 3600),
}).windows(&[Window::AllTime]).velocity(false).add();
let schema = schema.build()?;
// Open — ephemeral for tests, persistent for production.
let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
// Ingest content with metadata.
let mut meta = HashMap::new();
meta.insert("title".to_string(), "Introduction to Jazz Piano".to_string());
meta.insert("category".to_string(), "music".to_string());
db.write_item_with_metadata(EntityId::new(1), &meta)?;
// Write an embedding (you generate it, tidalDB indexes and ranks over it).
db.write_item_embedding(EntityId::new(1), &your_model.embed("Introduction to Jazz Piano"))?;
// Record engagement — the feedback loop closes here, no ETL required.
db.signal("view", EntityId::new(1), 1.0, Timestamp::now())?;
db.signal_with_context("like", EntityId::new(1), 1.0, Timestamp::now(), Some(user_id), Some(creator_id))?;
// Retrieve a ranked feed. Name the profile. tidalDB executes the pipeline.
let results = db.retrieve(&Retrieve::builder().for_user(user_id).profile("for_you").limit(50).build()?)?;
// Search: BM25 + semantic similarity fused via RRF.
let results = db.search(&Search::builder().query("jazz piano tutorial").for_user(user_id).limit(20).build()?)?;
db.close()?;
```
---
## What it replaces
| System | tidalDB equivalent |
|--------|--------------------|
| Elasticsearch | Tantivy BM25 text index (derived, crash-recoverable) |
| Redis | Lock-free in-memory signal ledger — decay scores, windowed counters |
| Kafka | Write-ahead log — durable, ordered, replayable |
| Feature store | Signal aggregates + user preference vectors (updated at write time) |
| Vector DB | USearch HNSW — embedded, f16 quantized, predicate-filtered ANN |
| Ranking service | 25 named profiles, scored at query time, swappable by name |
---
## Key capabilities
- **Signals with native decay** — declare `view` with a 7-day half-life; the database applies it at query time. No `trending_score_7d` field to maintain.
- **25 built-in ranking profiles** — `trending`, `hot`, `for_you`, `following`, `related`, `hidden_gems`, `top_week`, `shuffle`, `controversial`, and more. Name the profile; the database executes the full pipeline.
- **Hybrid search** — BM25 full-text + ANN semantic similarity, fused via Reciprocal Rank Fusion, personalized by user preference vector.
- **Composable filters** — filter by category, format, duration, language, engagement threshold, location, collection membership, and more — any combination, all composable.
- **Diversity as a query constraint** — `max_per_creator: 2` belongs in the query, not your API layer.
- **Feedback loop in the write path** — a signal write atomically updates the item's ledger, the user's preference vector, and relationship weights. The next ranking query — 100ms later — reflects it.
- **Cold start handled** — new content gets an exploration budget; new users get sensible defaults. No application logic required.
- **Cohort-scoped trending** — "trending among US users aged 18-24 who engage with jazz" is one query, not a pipeline.
- **Embeddable first** — runs in your process. `Arc<TidalDb>` is `Send + Sync`. No operational overhead.
---
## Getting started
Pick the path that matches how you plan to use tidalDB today. Every option below is self-contained and ships in this repo.
### 1. Embed tidalDB inside your Rust service (library mode)
**Setup**
1. Add the dependency. Depend on the `tidaldb` crate (at `tidal/` in this repository) by path or git:
```toml
[dependencies]
tidaldb = { path = "../tidaldb/tidal" } # adjust to the tidal/ crate's location, or use git = "https://github.com/orchard9/tidaldb"
```
2. Define your schema before opening the database (decay, windows, text fields, embeddings). The snippet in **[Quickstart, Step 2](docs/QUICKSTART.md#step-2-define-a-schema)** is a ready-to-copy template.
3. Choose storage mode when building:
```rust
let db = tidaldb::TidalDb::builder()
.with_schema(schema)
.ephemeral() // in-memory for tests
// .with_data_dir("/var/lib/tidaldb") // persistent deployment
.open()?;
```
4. Run the end-to-end sample:
```bash
cargo run -p tidaldb --example quickstart
```
**Usage**
- Call `db.signal(...)`, `db.signal_with_context(...)`, and `db.retrieve(...)` / `db.search(...)` from the same process; no network stack required.
- Wrap the instance in `Arc<TidalDb>` to share it across threads or tasks.
- Persisted deployments can be inspected with the CLI tool: `cargo run -p tidalctl -- status --path /var/lib/tidaldb`.
- Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)** and **[docs/API.md](docs/API.md)**.
### 2. Run the standalone HTTP server (`tidal-server`)
**Why:** you want a ready-to-run HTTP facade without writing Axum/Actix glue.
```bash
cargo run -p tidal-server -- \
standalone \
--listen 127.0.0.1:9400 \
--schema tidal-server/config/default-schema.yaml
```
Options:
- `--data-dir /var/lib/tidaldb` switches to persistent storage.
- Provide your own schema file (YAML) to match your signal mix.
Usage:
```bash
# register metadata + embedding
curl -X POST http://127.0.0.1:9400/items \
-H 'Content-Type: application/json' \
-d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
curl -X POST http://127.0.0.1:9400/embeddings \
-H 'Content-Type: application/json' \
-d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3] }'
# write engagement (supports user/creator context)
curl -X POST http://127.0.0.1:9400/signals \
-H 'Content-Type: application/json' \
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0, "user_id": 42 }'
# query
curl "http://127.0.0.1:9400/feed?user_id=42&profile=for_you&limit=20"
curl "http://127.0.0.1:9400/search?query=jazz%20piano&user_id=42&limit=5"
curl http://127.0.0.1:9400/health
```
The default schema lives at `tidal-server/config/default-schema.yaml`. Edit
it (or provide your own path) to align with your application’s signals,
text fields, and embedding slots.
### 3. Wrap it in an HTTP service you control
Expose tidalDB through your favorite web framework; the repo ships runnable templates.
- **Axum sample (`examples/axum_embedding.rs`)**
```bash
cargo run -p tidaldb --example axum_embedding
```
Usage:
```bash
curl -X POST http://127.0.0.1:3000/signal \
-H 'Content-Type: application/json' \
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
curl "http://127.0.0.1:3000/feed?user_id=42"
curl http://127.0.0.1:3000/health
```
The example handles schema setup, wraps `Arc<TidalDb>` in Axum `State`, and maps `TidalError` to HTTP responses.
- **Actix sample (`examples/actix_embedding.rs`)**
```bash
cargo run -p tidaldb --example actix_embedding
# curl http://127.0.0.1:3001/health
```
Demonstrates sharing `Arc<TidalDb>` through `web::Data` and using Actix’s shutdown hooks.
Use either sample as a starting point for microservices that prefer a client/server boundary.
### 4. Run the cluster server + Docker image
Need a single endpoint that fronts the built-in simulated cluster? Use
`tidal-server` in `cluster` mode. It spins up the multi-region fabric,
ships WAL batches between regions, and exposes `/signals`, `/feed`,
`/search` plus cluster-management routes.
```bash
cargo run -p tidal-server -- \
cluster \
--listen 0.0.0.0:9500 \
--schema tidal-server/config/default-schema.yaml \
--topology tidal-server/config/default-cluster.yaml
```
Key endpoints:
```bash
curl http://127.0.0.1:9500/health
curl -X POST http://127.0.0.1:9500/signals -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
curl "http://127.0.0.1:9500/feed?profile=trending&region=eu-west"
curl http://127.0.0.1:9500/cluster/status
curl -X POST http://127.0.0.1:9500/cluster/promote -d '{ "region": "eu-west" }'
```
Cluster mode currently replicates global signals (no `user_id` /
`creator_id` contexts) so that followers can stay in sync with the leader’s
WAL stream. See **[docs/runbooks/cluster.md](docs/runbooks/cluster.md)** for
operational steps, failure drills, and API references.
Prefer containers? Build the provided image and run it anywhere:
```bash
docker build -f docker/cluster/Dockerfile -t tidal-cluster .
docker run --rm -p 9500:9500 tidal-cluster
```
Mount your own schema/topology files with `-v` if you want different regions
or signal definitions.
### 5. Simulate a multi-region cluster in tests
The raw `SimulatedCluster` harness (no HTTP) remains available for property
tests and fuzzing.
```bash
cargo test --test m8_uat
cargo test --test m8_uat uat_step3 -- --nocapture # run a single scenario
```
Tweak `tests/m8_uat.rs` to script specific replication, failover, and
migration scenarios inside your own test suites.
**MSRV:** Rust 1.91
---
## Documentation
| Document | Contents |
|----------|----------|
| [docs/QUICKSTART.md](docs/QUICKSTART.md) | Step-by-step guide: schema, ingest, signals, ranking, search |
| [docs/API.md](docs/API.md) | Full API reference with code examples |
| [docs/VISION.md](docs/VISION.md) | Problem statement and design thesis |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Storage, signal system, vector index, query pipeline |
| [docs/USE_CASES.md](docs/USE_CASES.md) | 14 content discovery surfaces, filter and sort references |
| [docs/specs/](docs/specs/) | Component specifications (00–14) |
---
## Status
Milestones completed:
- Storage engine, WAL, entity store, signal ledger
- RETRIEVE query: candidate retrieval, filtering, scoring, diversity, pagination
- Vector index (USearch HNSW) with adaptive filtered search
- 25 built-in ranking profiles
- BM25 full-text search (Tantivy) + hybrid RRF fusion
- Creator search and creator profiles
- Cohort-scoped signal aggregation and trending
- Social graph (follows, blocks, following feed)
- Collections, saved searches, autocomplete suggestions
- Session and agent context (short-lived signals, preference decay)
- Crash recovery, graceful degradation, rate limiting, diagnostics
- Scale: tested to 1M items; scale benchmarks passing
The API surface is stable for the implemented features. Breaking changes are possible before 1.0.
---
## License
MIT

34
tidal/ai-lookup/features/filters.md Normal file
View File

@ -0,0 +1,34 @@
# Filters
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
All filter dimensions are composable — any combination produces a valid, efficiently-executed query. Filters are first-class query citizens, not application logic.
**Key Facts:**
- Any filter can combine with any other filter and any sort mode
- Faceted queries are first-class operations
- Filter categories: content attributes, date/time, creator, engagement thresholds, user state, geography
- Multi-select within a dimension uses OR logic (Jazz OR Blues)
- Cross-dimension uses AND logic (Jazz AND short AND this week)
**File Pointer:** `USE_CASES.md:536-628` (Appendix A)
## Filter Categories
| Category | Example Dimensions |
|----------|-------------------|
| Content | category, tag, format, duration, language, resolution, content_rating |
| Date/Time | created_within, created_preset (hour/today/week/month/year) |
| Creator | creator, verified, followed_by_user, new_to_user, follower count range |
| Engagement | min_views, min_likes, min_completion_rate, min_like_ratio |
| User State | seen/unseen, in_progress, saved, liked, downloaded, in_collection |
| Geographic | content_region, trending_in_region, near_location |
| Availability | free, premium, subscriber_only, downloadable, leaving_soon |
| Accessibility | has_subtitles, has_audio_description, has_sign_language |
## Related Topics
- [Query Language](./query-language.md)
- [Sort Modes](./sort-modes.md)

65
tidal/ai-lookup/features/query-language.md Normal file
View File

@ -0,0 +1,65 @@
# Query Language
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
The query interface is a single operation that encapsulates candidate retrieval, filtering, ranking, and diversity enforcement. Three primary operations: RETRIEVE (feed/browse), SEARCH (text+semantic), and SIGNAL (engagement write-back).
**Key Facts:**
- One query replaces what currently requires 6 systems
- RETRIEVE: feed generation, browse, related content
- SEARCH: keyword + semantic + hybrid retrieval
- SIGNAL: engagement event write-back (closes the feedback loop in the same transaction)
- All queries accept: FOR USER, USING PROFILE, FILTER, DIVERSITY, LIMIT
- Filters are composable — any combination is valid
**File Pointer:** `VISION.md:47-57`
## Query Shapes
**Feed retrieval:**
```
RETRIEVE items
FOR USER @user_id
CONTEXT feed
USING PROFILE for_you
FILTER unseen, unblocked, format:video
DIVERSITY max_per_creator:2, format_mix:true
LIMIT 50
```
**Search:**
```
SEARCH items
QUERY "rust tutorial beginner"
VECTOR query_vector
FOR USER @user_id
USING PROFILE search
DIVERSITY max_per_creator:2
LIMIT 20
```
**Signal write:**
```
SIGNAL like
item: @item_id
user: @user_id
timestamp: now()
```
**Related content:**
```
RETRIEVE items
SIMILAR TO @item_id
FOR USER @user_id
USING PROFILE related
FILTER unseen
LIMIT 10
```
## Related Topics
- [Ranking Profiles](../services/ranking-profiles.md)
- [Filters](./filters.md)
- [Sort Modes](./sort-modes.md)

30
tidal/ai-lookup/features/sort-modes.md Normal file
View File

@ -0,0 +1,30 @@
# Sort Modes
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
25+ native sort modes available on any surface. The application names a sort mode; the database executes it. No application-side sorting logic required.
**Key Facts:**
- Sort modes are built-in, not formulas the application implements
- Same sort mode works across different candidate sets (global, category, social graph)
- Windowed top sorts: hour, today, week, month, year, all-time
- Hot uses Reddit-style age decay: score / (age + 2)^gravity
- Trending is pure velocity (rate of change), distinct from Hot (cumulative with decay)
- Controversial maximizes product of positive and negative signals
**File Pointer:** `USE_CASES.md:635-663` (Appendix B)
## Categories
**Quality-based:** relevance, personalized, top_* (windowed), hidden_gems
**Time-based:** new, old, hot, trending, rising
**Engagement-based:** most_viewed, most_liked, most_commented, most_shared
**Format-based:** shortest, longest, alphabetical_asc/desc
**Special:** shuffle (quality-weighted random), live_viewer_count, date_saved, controversial
## Related Topics
- [Ranking Profiles](../services/ranking-profiles.md)
- [Query Language](./query-language.md)

10
tidal/ai-lookup/index.md Normal file
View File

@ -0,0 +1,10 @@
# AI Lookup Index
| Topic | File | Confidence | Updated | Summary |
|-------|------|------------|---------|---------|
| Entities | `services/entities.md` | High | 2026-02-19 | Items, Users, Creators — core data model |
| Signals | `services/signals.md` | High | 2026-02-19 | Typed event streams with decay, velocity, windowed aggregation |
| Ranking Profiles | `services/ranking-profiles.md` | High | 2026-02-19 | Named scoring functions declared in schema |
| Query Language | `features/query-language.md` | High | 2026-02-19 | RETRIEVE/SEARCH/SIGNAL query surface |
| Sort Modes | `features/sort-modes.md` | High | 2026-02-19 | 25+ native sort modes (hot, trending, rising, etc.) |
| Filters | `features/filters.md` | High | 2026-02-19 | Composable filter dimensions across all queries |

28
tidal/ai-lookup/services/entities.md Normal file
View File

@ -0,0 +1,28 @@
# Entities
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
Entities are the nodes of the system. Three types: Items (content), Users, and Creators. Every entity has metadata, a vector embedding slot, and an attached signal ledger.
**Key Facts:**
- Items have metadata, embeddings, and signals — signals are typed timestamped streams, not fields
- Users have preferences, histories, and relationships — living profiles that update continuously
- Creators are linked to Items and have their own embeddings (aggregated from catalog)
- Relationships are first-class edges between entities (weighted, directional, traversable)
**File Pointer:** `VISION.md:36-43`
## How It Works
Items enter via the WRITE path with metadata + embedding. A signal ledger is initialized at zero. Cold start exploration budget is applied automatically. Items are immediately queryable after commit.
Users accumulate implicit preference vectors from engagement history. Preference vectors update on every signal write (like, skip, hide, completion).
Creators are entities with their own embeddings derived from their item catalog. Creator-level signals include engagement rate, posting frequency, and follower count.
## Related Topics
- [Signals](./signals.md)
- [Ranking Profiles](./ranking-profiles.md)

38
tidal/ai-lookup/services/ranking-profiles.md Normal file
View File

@ -0,0 +1,38 @@
# Ranking Profiles
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
Ranking profiles are named, versioned scoring functions declared in schema. They reference signals, relationship weights, recency curves, and diversity rules. Profiles live in the database, are versioned alongside data, and can be swapped at query time by name.
**Key Facts:**
- Profiles are schema-level declarations, not application code
- Each profile defines: primary signals, secondary signals, boosts, gates, and diversity rules
- The same profile can operate on different candidate sets (global vs category vs social graph)
- Profiles are versioned — old versions remain queryable
**File Pointer:** `VISION.md:43-55`
## Built-in Profiles
| Profile | Primary Signal | Use Case |
|---------|---------------|----------|
| `for_you` | preference_match + engagement_velocity | Personalized feed |
| `search` | text_relevance + semantic_similarity | Search results |
| `trending` | share_velocity + view_velocity | Trending surfaces |
| `rising` | velocity relative to baseline, age-boosted | Breakout content |
| `following` | created_at DESC | Subscription feed |
| `related` | semantic_similarity + collaborative_filtering | Up next / related |
| `browse` | quality_score (completion + like_ratio + reach) | Category pages |
| `hot` | score / (age + 2)^gravity | Community frontpages |
| `controversial` | max(positive * negative signals) | Debate surfaces |
| `hidden_gems` | high quality, inverse reach | Discovery |
| `notification` | relationship_strength + item quality | Push prioritization |
| `live` | relationship_weight + viewer_count | Live surfaces |
## Related Topics
- [Signals](./signals.md)
- [Query Language](../features/query-language.md)
- [Sort Modes](../features/sort-modes.md)

33
tidal/ai-lookup/services/signals.md Normal file
View File

@ -0,0 +1,33 @@
# Signals
**Last Updated:** 2026-02-19
**Confidence:** High
## Summary
Signals are typed, timestamped event streams attached to entity signal ledgers. The database natively understands signal semantics: velocity (rate of change), decay (exponential or linear, configurable per type), and windowed aggregation (last hour, day, 7 days, all time).
**Key Facts:**
- Signals are NOT fields — they are streams with temporal semantics
- Decay half-lives are declared in schema, applied at query time
- Velocity is computed natively (rate of new events in a window)
- Windowed aggregation: 1h, 24h, 7d, all-time windows are first-class
- Negative signals (skip, hide, block) are equal citizens with positive signals
- Signal writes are atomic transactions updating item ledger, user preference vector, and relationship weights in one commit
**File Pointer:** `VISION.md:39-41`, `USE_CASES.md:669-711` (Appendix C)
## Signal Categories
| Category | Examples | Decay |
|----------|----------|-------|
| Positive engagement | view, like, share, completion | slow-medium |
| Negative engagement | skip, hide, block, not_interested | fast-permanent |
| Relationship | follow, unfollow, interaction_weight | slow-permanent |
| Quality | completion ratio, dwell_time, replay | slow-medium |
| Recommendation | autoplay_accept/reject, search_click | medium |
| Notification | notification_open, notification_dismiss | slow-medium |
## Related Topics
- [Entities](./entities.md)
- [Ranking Profiles](./ranking-profiles.md)

6
tidal/benches/diversity.rs
View File

@ -10,8 +10,10 @@
//! - No constraints (fast path baseline)
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::ranking::{DiversityConstraints, DiversitySelector, ScoredCandidate};
use tidaldb::schema::EntityId;
use tidaldb::{
ranking::{DiversityConstraints, DiversitySelector, ScoredCandidate},
schema::EntityId,
};
fn make_200_candidates(n_creators: usize) -> Vec<ScoredCandidate> {
(0..200_usize)

9
tidal/benches/fusion.rs
View File

@ -1,13 +1,18 @@
#![allow(clippy::unwrap_used)]
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::query::{HybridFusion, RetrievalMode, route_results};
use tidaldb::schema::EntityId;
use tidaldb::{
query::{HybridFusion, RetrievalMode, route_results},
schema::EntityId,
};
// Benchmark fixtures: rank-score magnitudes only, exactness irrelevant.
#[allow(clippy::cast_precision_loss)]
fn make_bm25(n: u64) -> Vec<(EntityId, f32)> {
(0..n).map(|i| (EntityId::new(i), (n - i) as f32)).collect()
}
#[allow(clippy::cast_precision_loss)]
fn make_ann(n: u64) -> Vec<(EntityId, f32)> {
(500..500 + n)
.enumerate()

23
tidal/benches/query.rs
View File

@ -17,21 +17,19 @@
//! - `retrieve_200_with_diversity`: 200 items, diversity constraints (`max_per_creator`)
//! - `retrieve_200_signal_ranked`: 200 items via signal-ranked candidate generation
use std::sync::RwLock;
use std::time::Duration;
use std::{sync::RwLock, time::Duration};
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use roaring::RoaringBitmap;
use tidaldb::query::executor::RetrieveExecutor;
use tidaldb::query::retrieve::Retrieve;
use tidaldb::ranking::builtins::register_builtins;
use tidaldb::ranking::diversity::DiversityConstraints;
use tidaldb::ranking::registry::ProfileRegistry;
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window};
use tidaldb::signals::{NoopWalWriter, SignalLedger};
use tidaldb::storage::indexes::bitmap::BitmapIndex;
use tidaldb::storage::indexes::filter::FilterExpr;
use tidaldb::storage::indexes::range::RangeIndex;
use tidaldb::{
query::{executor::RetrieveExecutor, retrieve::Retrieve},
ranking::{
builtins::register_builtins, diversity::DiversityConstraints, registry::ProfileRegistry,
},
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
signals::{NoopWalWriter, SignalLedger},
storage::indexes::{bitmap::BitmapIndex, filter::FilterExpr, range::RangeIndex},
};
// ── Helpers ──────────────────────────────────────────────────────────────────
@ -146,7 +144,6 @@ fn make_executor<'a>(
Some(dur),
Some(ts),
Some(universe),
None, // embedding_registry
)
}

9
tidal/benches/ranking.rs
View File

@ -10,10 +10,11 @@
use std::time::Duration;
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::ranking::builtins::register_builtins;
use tidaldb::ranking::{ProfileExecutor, ProfileRegistry};
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window};
use tidaldb::signals::{NoopWalWriter, SignalLedger};
use tidaldb::{
ranking::{ProfileExecutor, ProfileRegistry, builtins::register_builtins},
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
signals::{NoopWalWriter, SignalLedger},
};
#[allow(clippy::cast_precision_loss)]
fn make_ledger_with_200_items() -> SignalLedger {

31
tidal/benches/recovery.rs
View File

@ -26,18 +26,22 @@
//! `#[ignore = "expensive"]` and must be run explicitly:
//!
//! ```bash
//! cargo test --manifest-path tidal/Cargo.toml --test m7_recovery_sla -- --ignored
//! cargo test -p tidaldb --test m7_recovery_sla -- --ignored
//! ```
use std::time::Duration;
use criterion::{Criterion, criterion_group, criterion_main};
use tidaldb::TidalDb;
use tidaldb::replication::ShardId;
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window};
use tidaldb::wal::checkpoint::CheckpointManager;
use tidaldb::wal::format::{EventRecord, MAX_EVENTS_PER_BATCH, encode_batch};
use tidaldb::wal::segment::segment_filename;
use tidaldb::{
TidalDb,
replication::ShardId,
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
wal::{
checkpoint::CheckpointManager,
format::{EventRecord, MAX_EVENTS_PER_BATCH, encode_batch},
segment::segment_filename,
},
};
fn bench_schema() -> tidaldb::schema::Schema {
let mut builder = SchemaBuilder::new();
@ -97,11 +101,14 @@ fn inject_wal_backlog(data_dir: &std::path::Path, base_entity: u64, backlog_coun
// Build event records for the backlog.
let events: Vec<EventRecord> = (1..=backlog_count)
.map(|i| EventRecord {
entity_id: base_entity + i,
signal_type: 0, // "view" is the only signal, assigned ID 0
weight: 1.0,
timestamp_nanos: base_ns + (base_entity + i) * 1_000_000,
.map(|i| {
// "view" is the only signal, assigned ID 0.
EventRecord::signal(
base_entity + i,
0,
1.0,
base_ns + (base_entity + i) * 1_000_000,
)
})
.collect();

19
tidal/benches/scale.rs
View File

@ -19,24 +19,19 @@
//! - 20 categories, 128D random unit vectors
//! - 10% view coverage, 5% like coverage
use std::collections::HashMap;
use std::sync::LazyLock;
use std::time::Duration;
use std::{collections::HashMap, sync::LazyLock, time::Duration};
use criterion::{BatchSize, Criterion, SamplingMode, black_box, criterion_group, criterion_main};
use tidaldb::TidalDb;
use tidaldb::query::retrieve::Retrieve;
use tidaldb::query::search::Search;
use tidaldb::schema::{
DecaySpec, EntityId, EntityKind, SchemaBuilder, TextFieldType, Timestamp, Window,
use tidaldb::{
TidalDb,
query::{retrieve::Retrieve, search::Search},
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, TextFieldType, Timestamp, Window},
storage::indexes::filter::FilterExpr,
};
use tidaldb::storage::indexes::filter::FilterExpr;
const N_ITEMS: u64 = 1_000_000;
const N_CREATORS: u64 = 10_000;
const ITEMS_PER_CREATOR: u64 = 100;
const N_CATEGORIES: u64 = 20;
const DIM: usize = 128;
/// Categories pool for round-robin assignment.
static CATEGORIES: &[&str] = &[
@ -143,7 +138,7 @@ static SCALE_DB: LazyLock<TidalDb> = LazyLock::new(build_scale_db);
// ── RETRIEVE benchmarks ───────────────────────────────────────────────────────
/// RETRIEVE: "for_you" profile — signal-scored ranking over full universe.
/// RETRIEVE: "`for_you`" profile — signal-scored ranking over full universe.
fn bench_retrieve_for_you(c: &mut Criterion) {
let db: &TidalDb = &SCALE_DB;

13
tidal/benches/search.rs
View File

@ -4,14 +4,13 @@
//! Measures end-to-end `db.search()` latency at 10K items to validate the
//! < 50ms target specified in the m5p3 phase acceptance criteria.
use std::collections::HashMap;
use std::time::Duration;
use std::{collections::HashMap, time::Duration};
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::TidalDb;
use tidaldb::query::search::Search;
use tidaldb::schema::{
DecaySpec, EntityId, EntityKind, SchemaBuilder, TextFieldDef, TextFieldType, Timestamp, Window,
use tidaldb::{
TidalDb,
query::search::Search,
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, TextFieldType, Timestamp, Window},
};
fn search_schema() -> tidaldb::schema::Schema {
@ -44,7 +43,7 @@ fn search_schema() -> tidaldb::schema::Schema {
builder.build().unwrap()
}
/// Build a TidalDb with N items indexed for text search.
/// Build a `TidalDb` with N items indexed for text search.
fn make_db(n: u64) -> TidalDb {
let db = TidalDb::builder()
.ephemeral()

11
tidal/benches/session.rs
View File

@ -2,14 +2,13 @@
//! Benchmarks for the session layer: signal writes, snapshot reads,
//! and retrieve queries with active session context.
use std::collections::HashMap;
use std::time::Duration;
use std::{collections::HashMap, time::Duration};
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::TidalDb;
use tidaldb::query::retrieve::{ProfileRef, RetrieveBuilder};
use tidaldb::schema::{
AgentPolicy, DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window,
use tidaldb::{
TidalDb,
query::retrieve::{ProfileRef, RetrieveBuilder},
schema::{AgentPolicy, DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
};
fn session_schema() -> tidaldb::schema::Schema {

6
tidal/benches/signals.rs
View File

@ -3,8 +3,10 @@
use std::time::Duration;
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window};
use tidaldb::signals::{NoopWalWriter, SignalLedger, SignalTypeId};
use tidaldb::{
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window},
signals::{NoopWalWriter, SignalLedger, SignalTypeId},
};
fn view_ledger() -> (SignalLedger, SignalTypeId) {
let mut builder = SchemaBuilder::new();

17
tidal/benches/social.rs
View File

@ -14,9 +14,11 @@
//! - `CoEngagementIndex::score` on a populated index
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
use tidaldb::entities::{CoEngagementIndex, CreatorItemsBitmap, PreferenceVectors, UserStateIndex};
use tidaldb::query::executor::social_filter::{resolve_social_subgraph_users, social_graph_bitmap};
use tidaldb::schema::EntityId;
use tidaldb::{
entities::{CoEngagementIndex, CreatorItemsBitmap, PreferenceVectors, UserStateIndex},
query::executor::social_filter::{resolve_social_subgraph_users, social_graph_bitmap},
schema::EntityId,
};
// ── Social graph bitmap benchmark ─────────────────────────────────────────────
@ -220,7 +222,8 @@ fn bench_preference_merge(c: &mut Criterion) {
*x /= norm;
}
}
prefs.update_with_custom_rate(user_id, &emb, 0.1);
// Setup: dims always match here, so the bool is always true; ignore it.
let _ = prefs.update_with_custom_rate(user_id, &emb, 0.1);
}
let interaction: Vec<f32> = {
@ -239,7 +242,8 @@ fn bench_preference_merge(c: &mut Criterion) {
group.bench_function("single_128d_ema_100k_users", |b| {
let mut user_id = 0u64;
b.iter(|| {
prefs.update_with_custom_rate(
// Benchmarking call cost; dims match so the bool is always true.
let _ = prefs.update_with_custom_rate(
black_box(user_id % 100_000),
black_box(&interaction),
0.1,
@ -252,7 +256,8 @@ fn bench_preference_merge(c: &mut Criterion) {
let mut user_id = 0u64;
b.iter(|| {
for _ in 0..10 {
prefs.update_with_custom_rate(
// Benchmarking call cost; dims match so the bool is always true.
let _ = prefs.update_with_custom_rate(
black_box(user_id % 100_000),
black_box(&interaction),
0.1,

15
tidal/benches/sort.rs
View File

@ -11,20 +11,21 @@
//!
//! Sort modes exercised:
//! - `alphabetical_asc`: packs 8-byte title prefix into u64 for ordering
//! - `alphabetical_desc`: inverse of alphabetical_asc
//! - `alphabetical_desc`: inverse of `alphabetical_asc`
//! - `shortest`: reads "duration" metadata, orders ascending
//! - `longest`: reads "duration" metadata, orders descending
//!
//! Dataset: 500 items, each with a unique title and duration. No signals.
//! The benchmark uses a shared `LazyLock<TidalDb>` to amortize setup.
use std::sync::LazyLock;
use std::time::Duration;
use std::{sync::LazyLock, time::Duration};
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::TidalDb;
use tidaldb::query::retrieve::Retrieve;
use tidaldb::schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Window};
use tidaldb::{
TidalDb,
query::retrieve::Retrieve,
schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Window},
};
const N_ITEMS: u64 = 500;
@ -61,7 +62,7 @@ fn build_sort_db() -> TidalDb {
let item_id = EntityId::new(i + 1);
let mut meta = std::collections::HashMap::new();
// Vary the title so alphabetical sort has real ordering work to do.
meta.insert("title".to_string(), format!("Item {:03}", i));
meta.insert("title".to_string(), format!("Item {i:03}"));
// Duration cycles 1..=500 seconds.
meta.insert("duration".to_string(), (i + 1).to_string());
db.write_item_with_metadata(item_id, &meta).unwrap();

8
tidal/benches/storage.rs
View File

@ -1,9 +1,11 @@
#![allow(clippy::unwrap_used)]
use criterion::{BatchSize, Criterion, criterion_group, criterion_main};
use tidaldb::schema::{EntityId, EntityKind};
use tidaldb::storage::{
FjallStorage, InMemoryBackend, StorageEngine, Tag, WriteBatch, encode_key, entity_prefix,
use tidaldb::{
schema::{EntityId, EntityKind},
storage::{
FjallStorage, InMemoryBackend, StorageEngine, Tag, WriteBatch, encode_key, entity_prefix,
},
};
fn bench_sequential_put(c: &mut Criterion) {

6
tidal/benches/text_index.rs
View File

@ -7,8 +7,10 @@
use std::collections::HashMap;
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use tidaldb::schema::{EntityId, TextFieldDef, TextFieldType};
use tidaldb::text::{AllScoresCollector, TextIndex};
use tidaldb::{
schema::{EntityId, TextFieldDef, TextFieldType},
text::{AllScoresCollector, TextIndex},
};
fn make_index(n: u64) -> TextIndex {
let fields = vec![

62
tidal/benches/vector.rs
View File

@ -2,10 +2,10 @@
//! Criterion benchmarks for the vector index subsystem.
//!
//! Measures ANN search latency across the adaptive query planner's strategy
//! spectrum: unfiltered, in-graph filtered (20%), widened filtered (5%),
//! and pre-filter brute-force (0.5%). Also benchmarks recall@100, single
//! insert, and single delete.
//! Measures ANN search latency across the selectivity spectrum: unfiltered,
//! filtered (20%), widened filtered (5%, ef=400), and high-selectivity
//! filtered (0.5%). Also benchmarks recall@100, single insert, and single
//! delete. Calls the `VectorIndex` search API directly.
//!
//! All setup (index construction, vector insertion) is done OUTSIDE the
//! `b.iter()` closure. Only the search/insert/delete call is measured.
@ -13,8 +13,7 @@
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use rand::Rng;
use tidaldb::storage::vector::{
AdaptiveQueryPlanner, BruteForceIndex, DistanceMetric, QuantizationLevel, VectorId,
VectorIndex, VectorIndexConfig,
BruteForceIndex, DistanceMetric, QuantizationLevel, VectorId, VectorIndex, VectorIndexConfig,
};
// ---------------------------------------------------------------------------
@ -68,22 +67,14 @@ fn bench_ann_search_unfiltered(c: &mut Criterion) {
let dim = 128;
let n = 10_000_u64;
let index = build_brute_index(n, dim);
let planner = AdaptiveQueryPlanner::with_defaults();
let mut rng = rand::rng();
let query = random_unit_vector(dim, &mut rng);
c.bench_function("ann_search_unfiltered_10k", |b| {
b.iter(|| {
planner
.execute(
black_box(&index),
black_box(&query),
black_box(100),
None,
1.0,
None,
)
index
.search(black_box(&query), black_box(100), black_box(200))
.unwrap()
});
});
@ -95,7 +86,6 @@ fn bench_ann_search_filtered_20pct(c: &mut Criterion) {
let dim = 128;
let n = 10_000_u64;
let index = build_brute_index(n, dim);
let planner = AdaptiveQueryPlanner::with_defaults();
let mut rng = rand::rng();
let query = random_unit_vector(dim, &mut rng);
@ -105,15 +95,8 @@ fn bench_ann_search_filtered_20pct(c: &mut Criterion) {
c.bench_function("ann_search_filtered_20pct_10k", |b| {
b.iter(|| {
planner
.execute(
black_box(&index),
black_box(&query),
black_box(100),
Some(black_box(&filter)),
0.20,
None,
)
index
.filtered_search(black_box(&query), black_box(100), black_box(200), &filter)
.unwrap()
});
});
@ -125,25 +108,17 @@ fn bench_ann_search_filtered_5pct(c: &mut Criterion) {
let dim = 128;
let n = 10_000_u64;
let index = build_brute_index(n, dim);
let planner = AdaptiveQueryPlanner::with_defaults();
let mut rng = rand::rng();
let query = random_unit_vector(dim, &mut rng);
// ~5% selectivity: IDs 0..499 pass (5% of 10K).
// ~5% selectivity: IDs 0..499 pass (5% of 10K). Widened beam (ef=400).
let filter = |id: VectorId| id < 500;
c.bench_function("ann_search_filtered_5pct_10k", |b| {
b.iter(|| {
planner
.execute(
black_box(&index),
black_box(&query),
black_box(100),
Some(black_box(&filter)),
0.05,
None,
)
index
.filtered_search(black_box(&query), black_box(100), black_box(400), &filter)
.unwrap()
});
});
@ -155,8 +130,6 @@ fn bench_ann_search_brute_force(c: &mut Criterion) {
let dim = 128;
let n = 10_000_u64;
let index = build_brute_index(n, dim);
let brute = build_brute_index(n, dim);
let planner = AdaptiveQueryPlanner::with_defaults();
let mut rng = rand::rng();
let query = random_unit_vector(dim, &mut rng);
@ -166,15 +139,8 @@ fn bench_ann_search_brute_force(c: &mut Criterion) {
c.bench_function("ann_search_brute_force_10k", |b| {
b.iter(|| {
planner
.execute(
black_box(&index),
black_box(&query),
black_box(100),
Some(black_box(&filter)),
0.005,
Some(black_box(&brute as &dyn VectorIndex)),
)
index
.filtered_search(black_box(&query), black_box(100), black_box(200), &filter)
.unwrap()
});
});

73
tidal/docker/cluster/Dockerfile Normal file
View File

@ -0,0 +1,73 @@
# Multi-region tidalDB cluster image.
#
# Runs a 3-region cluster (us-east, eu-west, ap-south) behind a single HTTP
# surface on port 9500. Regions replicate over the real tidal-net gRPC transport
# (loopback), but all run in this single container process — there is no host or
# process isolation, so it is NOT production HA (true multi-process deployment is
# m8p10). See docs/runbooks/cluster.md for the operational API.
#
# Build context is this repository's root (the workspace `Cargo.toml` /
# `Cargo.lock` and every member crate live there). Build from the repo root:
#
# docker build -f tidal/docker/cluster/Dockerfile -t tidaldb:cluster .
#
# The repo-root allowlist `.dockerignore` strips target/, node_modules/, .git,
# and .env* from the context, so `COPY . .` brings in exactly the workspace
# sources cargo needs to resolve the member graph and build `-p tidal-server`.
# Pin the builder to bookworm so its glibc matches the bookworm-slim runtime
# below. The default `rust:1.91` tracks Debian trixie (glibc 2.41), whose
# auto-vectorized math pulls in `libmvec.so.1` — a library bookworm's glibc does
# not ship, so a trixie-built binary aborts at startup on bookworm with
# "libmvec.so.1: cannot open shared object file".
FROM rust:1.91-bookworm AS builder
ARG DEBIAN_FRONTEND=noninteractive
WORKDIR /app
# Copy the full workspace and build only the server binary. The unified
# workspace requires every member manifest present to resolve metadata, so we
# copy the whole (already-pruned) context rather than a fragile manifest-only
# subset.
COPY . .
# g++ builds the usearch C++ HNSW core; protobuf-compiler (protoc) is needed by
# tidal-net's build script (tonic-build compiles the WAL-shipping .proto), which
# tidal-server now depends on for real gRPC cluster replication.
RUN apt-get update && apt-get install -y --no-install-recommends g++ protobuf-compiler && rm -rf /var/lib/apt/lists/*
RUN cargo build -p tidal-server --release --locked
FROM debian:bookworm-slim
ARG DEBIAN_FRONTEND=noninteractive
WORKDIR /srv
RUN useradd --system --home /srv tidal && \
apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && \
rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/tidal-server /usr/local/bin/tidal-server
COPY --chown=tidal:tidal tidal-server/config /etc/tidal-server
USER tidal
EXPOSE 9500
# Path the server reads its schema/topology from. The binary-side clap wiring
# for this lives in tidal-server (see sibling task T5); keep the name and
# default in sync with it here.
ENV TIDAL_CONFIG=/etc/tidal-server
# Cluster mode is gated as experimental: replication is real gRPC but all
# regions share one process (no host/process isolation), so it refuses to start
# without an explicit opt-in. The image opts in here so `docker run` works; the
# server still logs a loud WARN that this is not production HA.
ENV TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD curl -f http://localhost:9500/health || exit 1
# ENTRYPOINT is just the binary so `docker run tidaldb:cluster <subcommand>`
# overrides work. CMD carries the default cluster invocation against the baked
# schema + topology.
ENTRYPOINT ["tidal-server"]
CMD ["cluster", "--listen", "0.0.0.0:9500", "--schema", "/etc/tidal-server/default-schema.yaml", "--topology", "/etc/tidal-server/default-cluster.yaml"]

64
tidal/docker/deploy/Dockerfile Normal file
View File

@ -0,0 +1,64 @@
# Production single-node tidalDB deploy image (slim toolchain, durable /data).
#
# Build context is this repository's root (the workspace `Cargo.toml` /
# `Cargo.lock` and every member crate live there). Build from the repo root:
#
# docker build -f tidal/docker/deploy/Dockerfile -t tidaldb:deploy .
#
# The repo-root allowlist `.dockerignore` strips target/, node_modules/, .git,
# and .env* from the context, so `COPY . .` brings in exactly the workspace
# sources cargo needs to resolve the member graph and build `-p tidal-server`.
# Pin the builder to bookworm so its glibc matches the bookworm-slim runtime.
# The default slim tag tracks Debian trixie, whose vectorized-math
# `libmvec.so.1` is absent on bookworm and aborts the binary at startup.
FROM rust:1.91-slim-bookworm AS builder
ARG DEBIAN_FRONTEND=noninteractive
WORKDIR /build
# protobuf-compiler (protoc) is required by tidal-net's build script (tonic-build
# compiles the WAL-shipping .proto); tidal-server depends on tidal-net for gRPC
# cluster replication.
RUN apt-get update && apt-get install -y --no-install-recommends pkg-config libssl-dev g++ protobuf-compiler && rm -rf /var/lib/apt/lists/*
# Copy the full workspace and build only the server binary. The unified
# workspace requires every member manifest present to resolve metadata, so we
# copy the whole (already-pruned) context rather than a fragile manifest-only
# subset.
COPY . .
RUN cargo build -p tidal-server --release --locked
FROM debian:bookworm-slim
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && rm -rf /var/lib/apt/lists/*
RUN useradd --system -u 10001 tidal
COPY --from=builder --chown=tidal:tidal /build/target/release/tidal-server /usr/local/bin/tidal-server
COPY --chown=tidal:tidal tidal-server/config /config
# Persistent data lives under /data, owned by the runtime user. Without this
# directory + the `--data-dir /data` flag below the server boots EPHEMERAL and
# loses every write on restart.
RUN mkdir -p /data && chown tidal:tidal /data
VOLUME ["/data"]
USER tidal:tidal
WORKDIR /data
EXPOSE 9400 9091
ENV TIDAL_SERVER_LOG=info
# Path the server reads its schema/topology from. The binary-side clap wiring
# for this lives in tidal-server (see sibling task T5); keep the name and
# default in sync with it here.
ENV TIDAL_CONFIG=/config
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD curl -sf http://localhost:9400/health || exit 1
# ENTRYPOINT is just the binary so `docker run tidaldb:deploy <subcommand>`
# overrides work. CMD carries the durable standalone invocation against the
# baked schema, persisting to the /data volume.
ENTRYPOINT ["tidal-server"]
CMD ["standalone", \
"--listen", "0.0.0.0:9400", \
"--schema", "/config/default-schema.yaml", \
"--data-dir", "/data", \
"--metrics", "0.0.0.0:9091"]

35
tidal/docker/docker-compose.yml Normal file
View File

@ -0,0 +1,35 @@
services:
tidaldb:
# Build context is this repository's root (../.. from this compose file:
# docker -> tidal -> root) so the workspace Cargo.toml/lock and every
# member crate are in scope for `cargo build -p tidal-server`.
build:
context: ../..
dockerfile: tidal/docker/standalone/Dockerfile
ports:
- "9400:9400"
- "9091:9091"
environment:
- TIDAL_API_KEY=${TIDAL_API_KEY}
- TIDAL_SERVER_LOG=info
volumes:
# Matches the standalone image's durable --data-dir + VOLUME ["/data"].
- tidaldb-data:/data
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "-H", "Authorization: Bearer ${TIDAL_API_KEY}", "http://localhost:9400/health"]
interval: 30s
timeout: 5s
retries: 3
prometheus:
image: prom/prometheus:v2.53.0
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
depends_on:
- tidaldb
volumes:
tidaldb-data:

8
tidal/docker/prometheus.yml Normal file
View File

@ -0,0 +1,8 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: tidaldb
static_configs:
- targets: ['tidaldb:9091']

66
tidal/docker/standalone/Dockerfile Normal file
View File

@ -0,0 +1,66 @@
# Single-node tidalDB server image.
#
# Build context is this repository's root (the workspace `Cargo.toml` /
# `Cargo.lock` and every member crate live there). Build from the repo root:
#
# docker build -f tidal/docker/standalone/Dockerfile -t tidaldb:standalone .
#
# tidal-server is a workspace member at `tidal-server/` and depends on the
# `tidal/` engine crate plus `tidal-net/` (gRPC cluster transport). The repo-root
# allowlist `.dockerignore` strips target/, node_modules/, .git, and .env* from
# the context, so `COPY . .` brings in exactly the workspace sources cargo needs
# to resolve the member graph and build `-p tidal-server`.
# Pin the builder to bookworm so its glibc matches the bookworm-slim runtime.
# The default `rust:1.91` tracks Debian trixie, whose vectorized-math
# `libmvec.so.1` is absent on bookworm and aborts the binary at startup.
FROM rust:1.91-bookworm AS builder
ARG DEBIAN_FRONTEND=noninteractive
WORKDIR /app
# g++ builds the usearch C++ HNSW core; protobuf-compiler (protoc) is needed by
# tidal-net's build script (tonic-build compiles the WAL-shipping .proto).
RUN apt-get update && apt-get install -y --no-install-recommends g++ protobuf-compiler && rm -rf /var/lib/apt/lists/*
# Copy the full workspace and build only the server binary. The unified
# workspace requires every member manifest present to resolve metadata, so we
# copy the whole (already-pruned) context rather than a fragile manifest-only
# subset.
COPY . .
RUN cargo build -p tidal-server --release --locked
FROM debian:bookworm-slim
ARG DEBIAN_FRONTEND=noninteractive
WORKDIR /srv
RUN useradd --system --home /srv tidal && \
apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && \
rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/tidal-server /usr/local/bin/tidal-server
COPY --chown=tidal:tidal tidal-server/config /etc/tidal-server
# Persistent data lives under /data, owned by the runtime user. Without this
# directory + the `--data-dir /data` flag below the server boots EPHEMERAL and
# loses every write on restart.
RUN mkdir -p /data && chown tidal:tidal /data
VOLUME ["/data"]
# Path the server reads its schema/topology from. The binary-side clap wiring
# for this lives in tidal-server (see sibling task T5); keep the name and
# default in sync with it here.
ENV TIDAL_CONFIG=/etc/tidal-server
USER tidal
EXPOSE 9400 9091
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD curl -f -H "Authorization: Bearer ${TIDAL_API_KEY:-}" http://localhost:9400/health || exit 1
# ENTRYPOINT is just the binary so `docker run tidaldb:standalone <subcommand>`
# overrides work (e.g. `... cluster ...` or an ad-hoc inspect). CMD carries the
# default standalone invocation, including `--data-dir /data` for durability.
ENTRYPOINT ["tidal-server"]
CMD ["standalone", \
"--listen", "0.0.0.0:9400", \
"--data-dir", "/data", \
"--metrics", "0.0.0.0:9091"]

1072
tidal/docs/API.md Normal file
View File

File diff suppressed because it is too large Load Diff

487
tidal/docs/ARCHITECTURE.md Normal file
View File

@ -0,0 +1,487 @@
# Architecture
tidalDB is a purpose-built ranking database. Its architecture is shaped by a single constraint: every design decision must serve the question "given a user and a context, what content should they see, in what order?" Nothing else.
This document describes how the system is structured, why it is structured that way, and how the major subsystems interact. For the API surface, see [API.md](API.md). For engineering standards, see [CODING_GUIDELINES.md](CODING_GUIDELINES.md). For the research behind specific decisions, see [docs/research/](docs/research/).
---
## Core Thesis
Every content platform (YouTube, TikTok, Reddit, Netflix) builds the same 6-system distributed stack from scratch — Elasticsearch, Redis, Kafka, a feature store, a vector DB, and a ranking service. The seams between those systems are where correctness fails: stale signals, inconsistent ranking, cache invalidation bugs, ETL lag.
The root cause is that existing databases treat ranking as an afterthought. They have no native concept of signals that evolve over time, no understanding of user context, no diversity as a query constraint, and no feedback loop between what users see and what the system learns.
tidalDB treats ranking as a primitive. Signals, decay, velocity, user preferences, relationships, and diversity are first-class schema concepts — not application logic bolted on top.
---
## Domain Model
Six first-class entity types:
| Type | What it represents |
|------|--------------------|
| **Item** | A piece of content (video, article, post) — has metadata, an embedding slot, a signal ledger |
| **User** | A viewer — has attributes, a preference vector, a signal ledger, a seen-item set |
| **Creator** | An author — has attributes, an embedding slot, a signal ledger |
| **Relationship** | A weighted, directional edge between any two entities (follows, blocks, interaction weight) |
| **Cohort** | A named, live predicate over user attributes (e.g. `age_range ∈ {18-24} AND locale = en-US`) |
| **Session / Agent Context** | A short-lived, agent-scoped memory surface binding a user, agent identity, and session metadata (tools, reward hints, policy) |
Six schema-level primitives:
| Primitive | What it captures |
|-----------|-----------------|
| **Signal** | A typed, timestamped event stream (view, like, skip, hide...) with declared decay rate, velocity, and windowed aggregation |
| **Ranking Profile** | A named, versioned scoring function: candidate retrieval strategy, boosts, penalties, quality gates, diversity rules, exploration budget |
| **Relationship** | Weighted edges: follows, blocks, interaction strength — used as ranking inputs |
| **Cohort** | Live predicate membership — enables cohort-scoped signal aggregation and trending |
| **Session** | Agent-scoped conversational context: short-lived signals, reward hints, policy tags, decay curves |
| **Filter** | Composable predicates over entity attributes, signal values, and relationship state |
---
## Module Structure
The dependency chain is strict. No circular dependencies. Each module knows only about modules beneath it.
```
schema/ ← standalone; no dependencies; defines all types
↑
storage/ ← depends on schema; knows nothing about signals or ranking
↑
signals/ ← depends on storage; knows nothing about queries or ranking
↑
agent/ ← depends on signals; manages sessions, policy, agent APIs
↑
query/ ← depends on agent + signals; orchestrates execution
↑
ranking/ ← depends on signals; invoked by the query executor
```
### `schema/`
The type system. Defines `EntityId`, `SignalDef`, `ProfileDef`, `CohortDef`, `TidalError`, and validation logic. No dependencies. Every other module depends on this one.
No external crates except `thiserror` (error derives).
### `storage/`
The persistence layer. Owns:
- **WAL** — the durability boundary. Every write goes here first.
- **Entity store** — item, user, creator metadata. Trait-abstracted: `EntityStore`, `SignalLedgerStore`, `RelationshipStore`.
- **Key encoding** — `[entity_id: u64 BE][0x00][TAG:suffix]` for co-location and range scans.
The storage backend (fjall initially) sits behind a trait. No storage engine types leak into higher modules.
### `signals/`
Signal ingestion and aggregation. Owns:
- **Ingest** — validates, hashes (BLAKE3 for deduplication), writes to WAL, triggers downstream
- **Decay** — forward-decay formula maintenance (`S(t) = S(t_prev) * exp(-λ * dt) + weight`)
- **Aggregation** — windowed counters (SWAG-based), velocity computation
- **Materialization** — background worker that writes pre-computed aggregates to O(1) lookup keys
### `agent/`
Session and policy management for agents. Owns:
- **Session store** — lifecycle of `(user, agent, session_id)` plus short-lived signals
- **Session materializers** — aggressive-decay aggregates agents can query (`last_5m_reward`, “tools used”, etc.)
- **Policy enforcement** — per-agent read/write rules, rate limiting, and isolation guardrails
- **API surface** — typed commands (`start_session`, `append_signal`, `close_session`) used by query/ranking layers
### `query/`
Query parsing and execution. Owns:
- **Parser** — validates `Retrieve`, `Search`, `Suggest` inputs
- **Planner** — selects candidate retrieval strategy (ANN vs. scan vs. cohort-scoped), estimates filter selectivity
- **Executor** — orchestrates retrieval → filter → score → diversity → paginate
### `ranking/`
Scoring and diversity. Owns:
- **Profile engine** — loads named profiles, applies boosts, penalties, gates
- **Signal scoring** — reads decay scores and windowed aggregates from signals/
- **Diversity enforcement** — post-scoring reordering pass enforcing `max_per_creator`, format mix, topic spread
- **Exploration** — injects new-item candidates at the declared exploration rate
---
## Storage Architecture
### WAL as source of truth
Every write — entity, signal, relationship — goes through the Write-Ahead Log before any processing. The entity store, signal aggregates, vector index, and text index are all derived state. If they are lost, they can be rebuilt from the WAL.
```
write_signal(event)
→ hash payload (BLAKE3) // deduplication
→ append to WAL (fsync amortized) // durability boundary
→ update in-memory decay score // hot path, atomic
→ update windowed counter // hot path, lock-free
→ enqueue for materializer // background
```
Signal durability is configurable per signal type:
- `Immediate` — fsync per event (purchases, high-value actions)
- `Batched` — fsync per N events or T ms, whichever comes first (likes, views)
- `Eventual` — OS-buffered (impressions, hover events)
Default: `Batched { max_events: 100, max_delay_ms: 10 }`.
### Key encoding
All keys follow the subject-prefix pattern: `[entity_id: u64 BE][0x00][TAG:suffix]`.
Big-endian encoding ensures byte-lexicographic order matches numeric order — enabling range scans and prefix compression. All data for one entity is co-located.
```
{item_id}\x00SIG:view:1h → windowed aggregate (1-hour view count)
{item_id}\x00SIG:view:decay → running decay score
{item_id}\x00META → entity metadata
{item_id}\x00EMB → embedding vector reference
{user_id}\x00PREF → preference vector
{user_id}\x00SEEN:{item_id} → seen-item record
{user_id}\x00REL:follows:{creator_id} → relationship edge
```
This layout is shard-ready: `entity_id` is a partition key, and range-based partitioning needs no format migration.
### Storage isolation
Item signal ledgers, user preference vectors, and creator profiles occupy separate storage namespaces (column families). A burst of view events on a viral item must not slow down user profile reads.
### Hybrid backend
LSM-tree (fjall) for the signal event log — write-heavy, sequential, FIFO-compacted. The same engine serves entity metadata with prefix bloom filters for point lookups. If a B-tree backend proves faster for entity random reads in benchmarks, the trait abstraction allows substitution without touching higher layers.
---
## Signal System
### Decay model
Decay is declared in schema, applied at query time. The application never computes `trending_score = views / (age_hours + 2)^1.8`.
```
DEFINE SIGNAL view ON item
DECAY exponential HALF_LIFE 7d
WINDOWS 1h, 24h, 7d, 30d, all_time
VELOCITY enabled
```
The forward-decay formula is mathematically exact and O(1) per operation:
```
// Write path (3 exp() ≈ 36ns)
S(t) = S(t_prev) * exp(-λ * dt) + weight
// Read path (1 exp() ≈ 15ns per entity per λ)
current = stored * exp(-λ * dt_since_last)
```
For 200 candidates: ~3-4 µs total. This replaces scanning raw events, which costs 160-1600 µs at 50 events/entity.
Out-of-order events: when `t_event < last_update`, pre-decay the weight: `score += weight * exp(-λ * (last_update - t_event))`. `last_update` is not modified — it already reflects a more recent time.
### Windowed aggregation
Per-signal, per-window counters maintained using a SWAG (Sliding Window Aggregate) structure. The database tracks counts within declared windows (1h, 24h, 7d, 30d) at all times. No re-scan of raw events at query time.
### Materialization
A background materializer continuously pre-computes aggregate values and writes them to O(1) lookup keys:
```
{item_id}\x00SIG:view:vel:1h → view velocity (events/hour over last hour)
{item_id}\x00SIG:like:24h → like count in last 24 hours
{item_id}\x00SIG:completion:all → completion rate, all time
```
Ranking queries read from materialized state on the fast path. If materialized state is stale (background worker lagging), the query falls back to computing from the in-memory decay score and windowed counters — slower but never wrong.
### Cohort-scoped aggregation
When a signal event arrives and cohorts are defined, the signal fans out to per-entity aggregates **and** per-cohort-entity aggregates:
```
signal(view, item: X, user: U)
→ update item X's entity-level aggregates // always
→ for each cohort C where U ∈ C:
update (cohort C, item X) aggregate // fan-out
```
Cohort membership is maintained as RoaringBitmaps — O(1) membership test. The per-cohort-item aggregate is sparse: only active (cohort, item) pairs with at least one signal are stored. Write amplification is ~6x for 5 cohorts per user on average; mitigated by batching.
### Immutable events, mutable aggregates
Signal events (user U liked item I at time T) are immutable facts appended to the WAL. Signal aggregates (item I has 1,247 likes in the last 24h) are mutable derived state maintained in the signal ledger. These layers are kept strictly separate. Aggregates can always be recomputed from events.
---
## Vector Index
**USearch** (Unum Cloud) is the HNSW engine. It is not built from scratch — correct, high-performance, concurrent HNSW with SIMD distance computation is 6-12 months of dedicated work. USearch runs in ScyllaDB, ClickHouse, and DuckDB at scale. The FFI boundary via `cxx` is thin.
### Quantization
f16 by default: 10M vectors at 1536D → ~31.5 GB (f16) vs ~60 GB (float32). Less than 1% recall loss. Float32 only if benchmarks prove f16 is insufficient for a specific embedding model.
Embeddings are normalized to unit length at insertion time. L2 distance is then equivalent to cosine similarity, and more SIMD-friendly. Re-normalization at query time never happens.
### Adaptive filtered search
The query planner estimates filter selectivity from metadata indexes (roaring bitmaps per creator, B-tree for date ranges), then selects a strategy:
| Estimated selectivity | Strategy |
|-----------------------|----------|
| < 2% | Pre-filter via bitmap intersection → brute-force L2 over matched set |
| 2%–100% | `index.filtered_search(vector, k, \|key\| predicate(key))` — USearch evaluates filters inline during HNSW traversal; non-matching nodes are skipped for results but still used for graph navigation |
| Fallback | Widen `ef_search`; if still insufficient, fall back to pre-filter + brute-force |
This matches how ScyllaDB uses USearch in production and how Weaviate and Qdrant handle the same problem.
### Persistence lifecycle
1. Active index in RAM for reads and writes during operation.
2. Periodic `save()` coordinated with WAL checkpointing.
3. On restart: `view()` for immediate read-only mmap serving while a writable copy loads in background.
4. Segment-based management for growing datasets: new inserts go to a new segment; periodic compaction merges segments and reclaims tombstoned space.
### Multi-vector user preference
User interest is not a single vector. Averaging engagement embeddings across topics ("hiking," "cooking," "cars") produces a centroid that represents none of them. Instead, each user's preference is represented as 3-10 interest cluster centroids (PinnerSage-style), maintained by the database as signals arrive. At query time, the planner issues one filtered HNSW query per active cluster and merges results. This requires no special index modifications — standard `filtered_search` per cluster, results deduped by score.
---
## Text Search
**Tantivy** is the full-text / BM25 engine. It is a derived index, not a source of truth.
### Consistency model
The entity store is the source of truth. Tantivy is a materialized view over it. If the Tantivy index is corrupted or lost, it can be rebuilt from the entity store by replaying the entity outbox.
```
write_item(item)
→ write to entity store (within WAL)
→ append to background indexer outbox
→ [async] background indexer → Tantivy
→ on each Tantivy commit, store last-processed WAL sequence number
→ on crash recovery, replay from that sequence number
```
Tantivy's single-writer guarantee is enforced via filesystem lock. Segment merging runs on background threads to avoid query latency spikes.
### Hybrid fusion
Search queries combine BM25 relevance and ANN semantic similarity using Reciprocal Rank Fusion:
```
RRF(d) = 1/(60 + rank_bm25) + 1/(60 + rank_ann)
```
RRF is rank-based — no score normalization required, robust across query types. Graduate to a tuned linear combination `α * bm25 + (1-α) * ann` only after relevance labels exist to set α.
Personalization re-ranks the fused set using the user's preference vector and relationship graph. The order of operations: text retrieval → ANN retrieval → RRF fusion → personalization re-ranking → diversity enforcement.
---
## Query Execution Pipeline
Every `retrieve()` or `search()` call follows this pipeline:
```
1. Parse & validate
└── input types, profile existence, filter validity
2. Plan candidate retrieval
├── ANN (user preference vector → top-k items by embedding similarity)
├── BM25 (text query → top-k items by relevance)
├── Full scan (trending/browse — no user vector required)
├── Graph walk (following feed — reverse-chronological from followed creators)
└── Cohort-scoped (trending/rising within a named cohort)
3. Apply hard filters
└── unseen, unblocked, unhidden, field predicates — eliminate ineligible candidates
└── Negative relationship checks (blocked creators, muted topics)
4. Score candidates
├── Load decay scores and windowed aggregates (from materialized state or computed)
├── Apply profile boosts (signal velocity, relationship weight, social proof)
├── Apply profile penalties (skip count, hide, negative engagement)
├── Apply freshness decay (age-based score reduction)
└── Apply quality gates (minimum completion rate, minimum score threshold)
5. Diversity enforcement (post-scoring reordering pass)
└── max_per_creator, format_mix, topic_diversity
└── Reorders — does not reduce result count
6. Exploration injection
└── Inject new/low-signal items at declared exploration rate (e.g. 10%)
└── New items get exploration budget until signals accumulate
7. Paginate and return
└── Cursor-based, stable across pages
```
### Ranking profiles are data, not code
Profiles are schema-level declarations — parsed, validated, versioned, stored in the database. They are not Rust functions compiled into the binary. Changing a profile weight requires no recompile, no redeploy. The query planner reasons about profile structure to optimize execution (e.g. a profile that only uses velocity signals skips the ANN step).
### Graceful degradation
Under load, the executor degrades in order — never returns errors for well-formed queries:
1. Reduce candidate set size (top_k: 500 → 200)
2. Use coarser signal aggregates (skip velocity, use windowed counts)
3. Skip diversity enforcement
4. Return from materialized ranking cache
---
## Write Path: Single Engagement Signal
Tracing `db.signal(Signal { kind: "like", item: "I", user: "U", ... })`:
```
1. Hash event payload (BLAKE3) → deduplicate
2. Append to WAL → fsync (batched)
3. Update item I's like decay score (atomic CAS)
4. Increment item I's like_count windowed counters (atomic add)
5. Recompute like velocity for item I
6. Update user U → item I relationship weight
7. Increment user U → creator C interaction weight
8. Shift user U's preference vector toward item I's embedding
9. Fan-out to cohort aggregates for each cohort U belongs to
10. Enqueue item I for materializer (windowed aggregate refresh)
```
Steps 3-9 execute atomically in memory. Step 10 is background. A ranking query issued 100ms later sees the updated decay score, relationship weight, and preference vector.
---
## Concurrency Model
### Hot path: lock-free
Signal counters, decay scores, and windowed aggregates use atomic operations exclusively.
- `AtomicU64` with `Relaxed` ordering for monotonic counters (view_count, like_count)
- `AtomicU64` via `f64::to_bits / from_bits` with CAS loops for decay scores
- `Acquire/Release` at synchronization points (checkpoint, materializer flush)
- `DashMap` for concurrent entity state access (sharded, no global lock)
A `like` event increments an atomic. A ranking query reads it. No blocking between writers and readers.
### Cold path: mutex acceptable
Schema changes, profile definitions, background compaction coordination — these happen infrequently and outside the query hot path. Mutexes are acceptable here.
### Hot-path structs: cache-line aligned
Any struct touched during candidate scoring is `#[repr(C, align(64))]` — one L1 cache line. This prevents false sharing under concurrent access and keeps scoring loops cache-friendly.
```rust
#[repr(C, align(64))]
struct EntitySignalState {
entity_id: u64,
decay_scores: [f64; 3], // one per declared decay rate
last_update_ns: u64,
window_counts: BucketedCounter,
// padded to 64-byte boundary
}
```
---
## Performance Targets
These are constraints, not aspirations. Regressions are bugs.
| Operation | Target |
|-----------|--------|
| Signal write (including WAL, amortized) | < 100 µs |
| Decay score read per candidate | ~15 ns |
| 200-candidate scoring pass | < 5 µs |
| ANN retrieval at 1M vectors | < 10 ms p99 |
| BM25 query at 1M documents | < 10 ms |
| End-to-end RETRIEVE query | < 50 ms |
The 200-candidate scoring budget breaks down as: 200 × 15 ns (decay read) + 200 × (boost/penalty application) + 1 diversity pass. Everything else in the pipeline must fit within the remainder of the 50 ms budget.
---
## Dependency Map
```
usearch (C++ FFI via cxx) → vector index
tantivy (pure Rust) → text/BM25 index
fjall (pure Rust) → storage engine (WAL, entity store, signal ledger)
roaring (pure Rust) → bitmap indexes (cohort membership, filter selectivity)
blake3 (pure Rust) → content-addressed signal deduplication
dashmap (pure Rust) → concurrent entity state map
thiserror → typed error derives
tracing → structured spans (embedder provides subscriber)
serde / serde_json → serialization at API boundaries only
criterion → benchmarking (dev dependency)
proptest → property testing (dev dependency)
```
Every dependency must justify its existence against "could we write this in 200 lines?" The approved list above is the complete list. No additions without research justification.
---
## Key Architectural Decisions
| Decision | Choice | Why |
|----------|--------|-----|
| WAL strategy | Append-only, fsync batched | Durability before processing; replay-based recovery; matches Citadel and Engram patterns |
| Storage engine | fjall (LSM-tree) | Pure Rust, embeddable, FIFO compaction for event logs, prefix bloom filters |
| Vector index | USearch | 150x faster than Lucene, predicate callback during HNSW traversal, mmap, quantization; used in ScyllaDB/ClickHouse/DuckDB |
| Quantization | f16 by default | 50% memory savings, <1% recall loss; 10M × 1536D → ~31.5 GB |
| Filtered ANN | Adaptive planner | <2% selectivity: pre-filter + brute-force; 2-100%: USearch predicate callback |
| Text search | Tantivy as derived index | 40K lines of battle-tested Rust; custom Collector for score extraction; DB-primary with background indexer |
| Hybrid fusion | RRF (k=60) | Rank-based, no score normalization, proven better than CombMNZ |
| Decay model | Forward-decay formula | Mathematically exact, O(1) write/read; no raw-event scanning at query time |
| Decay storage | f64 via AtomicU64 | 15 significant digits; sufficient for 528-year precision |
| Timestamps | u64 nanoseconds since Unix epoch | Overflows year 2554; matches ClickHouse/Sonnerie; no external dependency |
| Cohort membership | RoaringBitmap | O(1) membership test; sparse fan-out for signal aggregation |
| Signal deduplication | BLAKE3 content hash | Automatic deduplication of webhook retries and client double-submissions |
| Key encoding | `[entity_id: u64 BE]\x00TAG:suffix` | Co-location, range scans, natural shard boundaries, no migration path needed |
| Ranking profiles | Schema declarations | Swappable at query time by name; A/B testable; no recompile on change |
| Diversity | Post-scoring reordering pass | Does not reduce result count; enforces constraints after scoring is complete |
| Error handling | `thiserror` enum with 6 variants | Typed, actionable errors; used by fjall/tantivy/tikv; no `unwrap()` outside tests |
| Observability | `tracing` spans, embedder provides subscriber | Library crate; never initializes a subscriber; `#[tracing::instrument]` at subsystem boundaries |
---
## What This Replaces
```
Elasticsearch → Tantivy (BM25, derived index)
Redis → In-memory decay scores + windowed counters (lock-free atomics)
Kafka → WAL (durable, ordered, replayable)
Feature store → Signal ledger + materialized aggregates
Vector DB → USearch (HNSW, embedded)
Ranking service → Named profiles, query-time scoring
```
One process. One query interface. One operational model.
The test: this query should execute in under 50 ms, incorporate signals written 100 ms ago, enforce diversity without application logic, handle cold-start items without application intervention:
```rust
db.retrieve(Retrieve {
entity: EntityKind::Item,
for_user: Some("user_123"),
context: Some("feed"),
profile: "for_you",
filters: vec![Filter::unseen(), Filter::not_blocked(), Filter::eq("format", "video")],
diversity: Some(DiversitySpec { max_per_creator: Some(2), format_mix: true }),
limit: 50,
})
```
That is what six systems currently produce. It should be one query here.

366
tidal/docs/CODING_GUIDELINES.md Normal file
View File

@ -0,0 +1,366 @@
# Coding Guidelines
Engineering standards for tidalDB. Derived from the research in `docs/research/`, the architectural patterns in `thoughts.md`, and the roadmap's dependency chain.
These are not aspirational. They are load-bearing constraints. Violating them creates bugs that are expensive to find and painful to fix in a ranking system.
---
## 1. Memory Layout and Performance
### Cache-line alignment on hot-path structs
Any struct touched during candidate scoring must be `#[repr(C, align(64))]` — exactly one L1 cache line. This prevents false sharing under concurrent access and keeps scoring loops cache-friendly.
Hot-path structs include: per-entity signal state, entity metadata summaries, user preference vectors, relationship weights.
```rust
#[repr(C, align(64))]
struct EntitySignalState {
entity_id: u64,
decay_scores: [f64; 3], // one per decay rate
last_update_ns: u64,
window_counts: BucketedCounter,
// ... pad to 64-byte boundary if needed
}
```
### Lock-free on the hot path
Signal counters, decay scores, and windowed aggregates must use atomic operations — never mutexes. A `like` event increments an atomic counter. A ranking query reads it without blocking writers.
- `AtomicU64` with `Relaxed` ordering for counters
- `AtomicF64` (via `AtomicU64` + `f64::from_bits`) with CAS loops for decay scores
- `Acquire/Release` ordering only at synchronization boundaries (checkpoint, flush)
- `DashMap` or sharded maps for concurrent entity state access
Mutexes are acceptable for cold-path operations: schema changes, profile definitions, background compaction coordination.
### Allocation discipline
- Pre-allocate result buffers. Ranking queries should not allocate per-candidate.
- Reuse `Vec` capacity across query executions where possible.
- Avoid `String` in hot-path structs — use interned IDs or `u64` hashes.
- Embedding vectors are `&[f32]` slices backed by mmap or arena, never `Vec<f32>` copies.
---
## 2. Storage Architecture
### WAL is the source of truth
Every write — entity, signal, relationship — goes through the Write-Ahead Log before any processing. The entity store, signal aggregates, and search index are derived state. If they are lost, they can be rebuilt from the WAL.
- Signal events are durably logged (fsync'd) before aggregation occurs
- The aggregation system can crash, restart, and replay from the WAL
- Content-addressed events (BLAKE3 hash of payload) for automatic deduplication of retries
### Trait-abstract the storage backend
The storage engine (fjall initially, potentially RocksDB later) must sit behind a trait boundary. No storage engine types should leak into the signal, query, or ranking modules.
```rust
pub trait EntityStore: Send + Sync {
fn get(&self, id: &EntityId) -> Result<Option<Entity>>;
fn put(&self, entity: &Entity) -> Result<()>;
fn scan_prefix(&self, prefix: &[u8]) -> Result<Box<dyn Iterator<Item = Entity>>>;
}
```
### Per-entity-type storage isolation
Item signal ledgers, user preference vectors, and creator profiles live in separate storage namespaces (column families or keyspaces). A burst of signal events for a viral item must not slow down user profile reads.
### Key encoding
Follow the subject-prefix pattern: `{entity_id}\x00{TAG}:{suffix}`. All data for one entity is co-located. Big-endian encoding so byte-lexicographic ordering matches numeric ordering.
```
[entity_id: u64 BE][0x00][SIG:view:24h] → windowed aggregate
[entity_id: u64 BE][0x00][META] → entity metadata
[entity_id: u64 BE][0x00][REL:follows] → relationship edge
```
---
## 3. Signal System
### Decay is a type, not a formula you call
The application never computes `trending_score = views_24h / (age_hours + 2)^1.8`. That logic lives in a named ranking profile. The application writes `SIGNAL view` and queries `USING PROFILE trending`.
### Running decay scores — O(1) update, O(1) read
Use the forward-decay formula. It is mathematically exact, not an approximation.
**Write:** `S(t) = S(t_prev) * exp(-lambda * dt) + weight`
**Read:** `current = stored * exp(-lambda * dt_since_last)`
Cost: 3 `exp()` calls per write (~36ns), 1 `exp()` per read per entity per lambda (~15ns). For 200 candidates, that's ~3-4 microseconds total.
Do not scan raw events to compute decay at read time. That path costs 160+ microseconds at 50 events/entity and breaks the budget at 500+.
### Out-of-order events are handled correctly
When `t_event < last_update`, pre-decay the weight: `score += weight * exp(-lambda * (last_update - t_event))`. Do not update `last_update` — it already reflects a more recent time.
### Immutable events, mutable aggregates
Signal events (a user liked an item at time T) are immutable facts. Signal aggregates (this item has 1,247 likes in the last 24h) are mutable derived state. Keep these layers distinct. Aggregates can always be recomputed from events.
---
## 4. Vector Index
### USearch is the HNSW engine
Do not build HNSW from scratch. USearch provides 126K+ QPS, predicate callbacks during traversal, mmap persistence, and quantization. The FFI boundary via CXX is thin.
### f16 quantization as default
10M vectors at 1536D: ~31.5 GB (f16) vs ~60 GB (float32). Less than 1% recall loss. Use float32 only when benchmarks prove f16 is insufficient for a specific embedding model.
### Normalize embeddings at insertion time
For cosine similarity, normalize vectors to unit length and use L2 distance (equivalent for unit vectors, more SIMD-friendly). Store normalized vectors — never re-normalize at query time.
### Adaptive filtered search
Never hardcode a single filtering strategy. Estimate selectivity, then branch:
- **<2% selectivity:** Pre-filter (roaring bitmap intersection) then brute-force L2
- **2-100% selectivity:** `filtered_search` with predicate callback (in-graph filtering)
- **Fallback:** Widen ef_search or degrade to pre-filter + brute-force
---
## 5. Text Search
### Tantivy is a derived index, not a source of truth
The entity store is the source of truth. Tantivy is a materialized view. If the Tantivy index is corrupted or lost, it can be rebuilt from the entity store.
Consistency pattern:
1. Write to entity store (within transaction / WAL)
2. Background indexer reads outbox and feeds Tantivy
3. On each Tantivy commit, store last-processed sequence number in commit payload
4. On crash recovery, replay from that sequence number
### Hybrid fusion starts with RRF
`RRF(d) = 1/(60 + rank_bm25) + 1/(60 + rank_ann)`. Rank-based, no score normalization needed, robust across query types. Graduate to tuned linear combination only after relevance labels exist to tune alpha.
---
## 6. Query and Ranking
### Ranking profiles are data, not code
Profiles are schema-level declarations — parsed, validated, versioned, stored in the database. They are not Rust functions compiled into the binary. The query optimizer reasons about profile structure to plan execution.
A profile change should never require recompiling or redeploying.
### Diversity is a post-scoring pass
After candidates are scored, apply diversity constraints as a separate reordering pass. Diversity does not reduce result count — it reorders to enforce constraints (max_per_creator, format_mix) while maintaining the target count.
### Negative signals are structurally equal to positive signals
Skips, hides, blocks, mutes, downvotes are not the absence of engagement. They are data. They carry the same weight, precision, and update immediacy as likes. A hide creates a permanent hard-negative. A skip within 3 seconds is a strong quality signal. The ranking function treats these as first-class inputs.
### Graceful degradation, never failure
Under load, return slightly less precise rankings — not errors. Degrade in this order:
1. Reduce candidate set size (top_k: 500 -> 200)
2. Use coarser signal aggregates (skip velocity, use windowed counts)
3. Skip diversity enforcement
4. Return results from materialized cache
Never return an empty result set or an error for a well-formed query.
---
## 7. Error Handling
### `Result<T>` everywhere, `unwrap()` nowhere
Every fallible operation returns `Result`. No `unwrap()`, no `expect()` outside of tests and initialization. Panics in a database corrupt state.
### Errors are typed and actionable
```rust
pub enum TidalError {
/// Storage engine failure — retry may succeed.
Storage(StorageError),
/// Entity not found — caller should handle.
NotFound { entity: EntityId },
/// Schema violation — caller's fault, fix the input.
Schema(SchemaError),
/// Signal write failed durability check — retry required.
Durability(DurabilityError),
/// Query malformed — parse error with position.
Query(QueryError),
/// Internal invariant violated — this is a bug, log and degrade.
Internal(String),
}
```
`Internal` errors trigger graceful degradation, not crashes. Log them loudly. Return approximate results if possible.
---
## 8. Testing
### Property tests for invariants
Use `proptest` for properties that must hold regardless of input:
- Decay scores monotonically decrease when no new events arrive
- Windowed aggregates equal the sum of events within the window
- Diversity constraints hold in every result set
- WAL replay produces identical state to uninterrupted execution
- Filter composition is commutative (order of filters doesn't change results)
- Blocked/hidden items never appear in query results
### Crash recovery tests
Simulate crashes at every point in the write path:
- Mid-WAL-write
- After WAL commit, before entity store update
- After entity store, before signal aggregation
- After signal aggregation, before Tantivy index
- During background materialization
Verify: the system recovers to a consistent state. No lost events. No phantom state.
### Benchmark from day one
Use `criterion` for micro-benchmarks. Track these numbers continuously:
- Signal write latency (target: <100 microseconds including WAL fsync amortized)
- Decay score read per candidate (target: ~15ns)
- 200-candidate scoring pass (target: <5 microseconds)
- ANN retrieval at 1M vectors (target: <10ms p99)
- BM25 query at 1M documents (target: <10ms)
- End-to-end RETRIEVE query (target: <50ms)
Regressions in these numbers are bugs. Treat them like test failures.
---
## 9. Code Organization
### Module boundaries match the dependency chain
```
storage/ → knows nothing about signals, queries, or ranking
signals/ → depends on storage, knows nothing about queries or ranking
query/ → depends on storage + signals, knows nothing about ranking internals
ranking/ → depends on signals, invoked by query executor
schema/ → standalone, depended on by everything
```
Circular dependencies between these modules are architectural bugs. If ranking needs to call into storage directly, that call goes through a trait the query executor provides.
### Public API is minimal
Expose the smallest possible surface. Internal types stay internal. The public API is:
- `TidalDB::open()`, `TidalDB::shutdown()`
- `define_entity()`, `define_signal()`, `define_profile()`
- `write_item()`, `write_user()`, `write_creator()`
- `write_relationship()`
- `signal()`
- `retrieve()`, `search()`, `suggest()`
Everything else is `pub(crate)` or module-private.
### One concern per file
A file that handles both signal ingestion and signal aggregation will grow into a 2000-line mess. Split early: `signals/ingest.rs`, `signals/decay.rs`, `signals/aggregation.rs`, `signals/materialization.rs`.
---
## 10. Dependencies
### Minimal, intentional, auditable
Every dependency must justify its existence against "could we write this in 200 lines?"
Approved dependencies (from research):
- **fjall** — storage engine (pure Rust, embeddable)
- **usearch** — HNSW vector index (C++ FFI via cxx)
- **tantivy** — full-text search / BM25
- **blake3** — content-addressed hashing
- **roaring** — bitmap indexes for filtered search
- **thiserror** — derive `Display` and `From` for typed error enums; eliminates boilerplate without hiding structure
- **tracing** — structured spans for query execution, WAL writes, and signal ingestion; embedders choose their own subscriber
- **criterion** — benchmarking
- **proptest** — property testing
- **serde** / **serde_json** — serialization (at API boundaries only, not in hot paths)
- **chrono** or **time** — timestamp handling
- **dashmap** — concurrent hash map for hot-path entity state
Do not add dependencies for things the standard library or a 50-line util handles: argument parsing, builder pattern macros, derive-everything crates.
### No `unsafe` without a comment explaining why
Every `unsafe` block must have a `// SAFETY:` comment explaining:
1. What invariant the compiler can't verify
2. Why this specific usage is sound
3. What would make it unsound (for future maintainers)
Prefer `#![forbid(unsafe_code)]` at the crate level where possible. The storage engine and FFI boundaries (USearch) are the only modules that should need `unsafe`.
---
## 11. Observability
### `tracing` spans on every public operation
Every public function that crosses a subsystem boundary gets a `#[tracing::instrument]` attribute. This is non-negotiable — it is how query latency, signal write throughput, and WAL sync times are measured in production without any additional instrumentation work later.
```rust
#[tracing::instrument(skip(self), fields(entity_id = %id))]
pub fn get_entity(&self, id: EntityId) -> Result<Option<Entity>> {
// ...
}
```
The `skip` attribute prevents large or sensitive arguments from being logged by default. Add `fields(...)` to surface the key identifiers that make traces navigable.
### Instrument at subsystem entry points, not every helper
Instrument the public API and the major internal stage boundaries:
- `EntityStore::{get, put, scan_prefix}`
- `SignalLedger::{record, decay_score}`
- `QueryExecutor::execute`
- `RankingEngine::score`
- `Wal::{append, flush}`
Do not add spans to private helpers called within a single instrumented function. The overhead accumulates.
### tidalDB is a library — embedders choose their subscriber
Do not initialize a tracing subscriber anywhere in this crate. The subscriber is the embedder's responsibility. Import `tracing = "0.1"` only; never `tracing-subscriber` in the main crate.
### Error events
Use `tracing::error!` for `TidalError::Internal` (a bug occurred), `tracing::warn!` for recoverable degradation, `tracing::debug!` for query planning decisions, `tracing::trace!` for per-candidate scoring.
Never use `println!` or `eprintln!` in production code.
---
## 12. Commit and Review Standards
### Commits are atomic and purposeful
One logical change per commit. "Add signal decay scoring" is a commit. "Add decay scoring and also fix a typo and refactor entity store" is three commits.
### Every PR must include
- What changed and why (not how — the diff shows how)
- Benchmark results if touching hot-path code
- Property test or crash recovery test if touching write path or state management
- No regressions in existing benchmarks
### No TODO without an issue
`// TODO:` comments are allowed only with a link to a tracking issue. Orphan TODOs rot. If it's worth noting, it's worth tracking.

298
tidal/docs/QUICKSTART.md Normal file
View File

@ -0,0 +1,298 @@
# Quickstart
Get a working ranked feed in 10 minutes.
**Prerequisites:** Rust 1.91+, Cargo. No external services.
---
## Run the example
The fastest path is the included example, which demonstrates the complete loop — schema, ingest, signals, ranking:
```bash
cargo run -p tidaldb --example quickstart
```
The rest of this guide explains what it does and extends it with personalization and search.
---
## Step 1: Add the dependency
Depend on the `tidaldb` crate (at `tidal/` in this repository) by path or git:
```toml
[dependencies]
tidaldb = { path = "../tidaldb/tidal" } # adjust to the tidal/ crate's location, or use git = "https://github.com/orchard9/tidaldb"
```
---
## Step 2: Define a schema
Schema is defined before opening the database. It declares signal types (what events you'll record and how they decay), text fields (for BM25 search), and embedding slots (for vector search).
```rust
use std::time::Duration;
use tidaldb::schema::{SchemaBuilder, EntityKind, DecaySpec, Window, TextFieldType};
let mut schema = SchemaBuilder::new();
// View signal: 7-day half-life, three windows, velocity enabled.
// You declare the decay. tidalDB applies it at query time — no formula to maintain.
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(7 * 24 * 3600),
}).windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
// Like signal: 30-day half-life. Durable engagement decays slowly.
let _ = schema.signal("like", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(30 * 24 * 3600),
}).windows(&[Window::AllTime]).velocity(false).add();
// Share signal: 3-day half-life. Short-lived but strongly trending.
let _ = schema.signal("share", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(3 * 24 * 3600),
}).windows(&[Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
// Skip signal: permanent. A user who skipped should not see it again.
let _ = schema.signal("hide", EntityKind::Item, DecaySpec::Permanent).add();
// Text fields for BM25 full-text search.
schema.text_field("title", TextFieldType::Text);
schema.text_field("category", TextFieldType::Keyword);
// Embedding slot for semantic / vector search (128D in this example).
// In production, use the dimensionality of your embedding model.
schema.embedding_slot("content", EntityKind::Item, 128);
let schema = schema.build()?;
```
**Decay types:**
- `Exponential { half_life }` — weight halves every `half_life`. Use for views, likes, shares.
- `Linear { lifetime }` — weight drops to zero over `lifetime`.
- `Permanent` — never decays. Use for hides, blocks, follows.
---
## Step 3: Open the database
```rust
use tidaldb::TidalDb;
// Ephemeral: in-memory, ideal for tests and this tutorial.
let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
// Persistent: durable storage at a path on disk.
// let db = TidalDb::builder().with_data_dir("/var/lib/myapp/tidaldb").with_schema(schema).open()?;
db.health_check()?;
```
`TidalDb` is `Send + Sync`. Wrap it in `Arc<TidalDb>` to share across threads or tasks.
---
## Step 4: Ingest content
Write items with metadata as `HashMap<String, String>` key-value pairs. Then write their embeddings separately.
**tidalDB does not generate embeddings.** You bring your model; tidalDB handles retrieval and ranking over the vectors you produce.
```rust
use std::collections::HashMap;
use tidaldb::schema::{EntityId, Timestamp};
let tracks = [
(1u64, "Introduction to Jazz Piano", "music", "1320"),
(2, "Rust Async Programming", "tech", "3600"),
(3, "Sourdough Bread Masterclass", "cooking", "2700"),
(4, "Jazz Improvisation Techniques","music", "1800"),
(5, "Building a Compiler in Rust", "tech", "5400"),
(6, "French Pastry Fundamentals", "cooking", "2100"),
(7, "Modal Jazz: Coltrane Changes", "music", "2400"),
(8, "WebAssembly from Scratch", "tech", "2700"),
(9, "Knife Skills for Home Cooks", "cooking", "900"),
(10, "Bebop Piano Vocabulary", "music", "1500"),
];
for (id, title, category, duration) in &tracks {
let mut meta = HashMap::new();
meta.insert("title".to_string(), title.to_string());
meta.insert("category".to_string(), category.to_string());
meta.insert("format".to_string(), "video".to_string());
meta.insert("duration".to_string(), duration.to_string());
meta.insert("created_at".to_string(), Timestamp::now().as_nanos().to_string());
db.write_item_with_metadata(EntityId::new(*id), &meta)?;
// In production: embed the title with your model.
// Here we use random unit vectors for illustration.
let embedding = random_unit_vector(128, &mut rng);
db.write_item_embedding(EntityId::new(*id), &embedding)?;
}
println!("Ingested {} items.", db.item_count());
```
On write, tidalDB:
1. Stores the entity and metadata
2. Indexes text fields into the BM25 index
3. Inserts the embedding into the HNSW vector index
4. Initializes the signal ledger with an exploration budget
5. Makes the item immediately queryable
---
## Step 5: Record engagement signals
When a user engages with content, write a signal. The feedback loop closes at write time — no Kafka consumer to lag, no feature store sync to schedule.
```rust
let now = Timestamp::now();
// Global signals — these update the item's aggregate signal ledger.
db.signal("view", EntityId::new(1), 1.0, now)?; // Jazz Piano viewed
db.signal("view", EntityId::new(4), 1.0, now)?;
db.signal("view", EntityId::new(7), 1.0, now)?; // Modal Jazz viewed
db.signal("like", EntityId::new(4), 1.0, now)?; // Jazz Improv liked
db.signal("share", EntityId::new(7), 1.0, now)?; // Modal Jazz shared
```
For signals with user context, use `signal_with_context`. This also updates the user's preference vector and interaction weights — enabling personalization.
```rust
let user_id = 42u64;
let creator_id = 100u64;
// User 42 viewed item 4. Their preference vector shifts toward jazz content.
db.signal_with_context("view", EntityId::new(4), 1.0, now, Some(user_id), Some(creator_id))?;
db.signal_with_context("like", EntityId::new(7), 1.0, now, Some(user_id), Some(creator_id))?;
// Negative signals are equal citizens.
db.signal("hide", EntityId::new(2), 1.0, now)?; // User hid the Rust video.
```
A ranking query issued 100ms later sees the updated state. No ETL required.
---
## Step 6: Retrieve a ranked feed
tidalDB ships 25 built-in ranking profiles. The application names a profile; the database executes the full scoring pipeline.
```rust
use tidaldb::query::retrieve::Retrieve;
// Global trending: items with the highest share + view velocity.
let query = Retrieve::builder().profile("trending").limit(10).build()?;
let results = db.retrieve(&query)?;
println!("Trending ({} candidates):", results.total_candidates);
for item in &results.items {
let sigs: Vec<_> = item.signals.iter()
.map(|s| format!("{}={:.3}", s.name, s.value))
.collect();
println!(" #{} id={} score={:.4} [{}]",
item.rank, item.entity_id.as_u64(), item.score, sigs.join(", "));
}
```
---
## Step 7: Personalize
Swap the profile to `for_you`. Because user 42 signaled views and likes on jazz content, their results differ from global trending.
```rust
// Personalized feed for user 42.
let query = Retrieve::builder()
.for_user(user_id)
.profile("for_you")
.limit(10)
.build()?;
let results = db.retrieve(&query)?;
println!("For You (user {}):", user_id);
for item in &results.items {
println!(" #{} id={} score={:.4}", item.rank, item.entity_id.as_u64(), item.score);
}
```
Other useful profiles:
- `"hot"` — score with age decay (Reddit model)
- `"following"` — content from followed creators (requires `for_user` + written `follows` relationships)
- `"hidden_gems"` — high completion rate, low reach
- `"top_week"` — cumulative quality over the last 7 days
- `"shuffle"` — random, quality-weighted
---
## Step 8: Search
Search combines BM25 full-text and ANN semantic similarity via Reciprocal Rank Fusion.
```rust
use tidaldb::query::search::Search;
// Flush the text index so recently written items are searchable.
// In production with persistent mode this happens automatically on a ~2s commit cycle.
db.flush_text_index()?;
// Keyword search, personalized for user 42.
let query = Search::builder()
.query("jazz piano")
.for_user(user_id)
.limit(5)
.build()?;
let results = db.search(&query)?;
println!("Search 'jazz piano':");
for item in &results.items {
println!(" #{} id={} bm25={:.3?} semantic={:.3?}",
item.rank,
item.entity_id.as_u64(),
item.bm25_score,
item.semantic_score,
);
}
```
Add a query embedding for hybrid search — text relevance + semantic similarity:
```rust
let query_vector = your_model.embed("jazz piano"); // same model as item embeddings
let query = Search::builder()
.query("jazz piano")
.vector(query_vector)
.for_user(user_id)
.limit(5)
.build()?;
```
---
## Step 9: Close
```rust
db.close()?;
```
This flushes the WAL, checkpoints signal state, and persists indexes. In persistent mode, the next open recovers to the last checkpointed state.
---
## What to explore next
| Topic | Where to look |
|-------|--------------|
| Full API reference | [API.md](API.md) |
| Filters — format, duration, location, engagement thresholds | [API.md — Filters](API.md#filters) |
| Diversity constraints | [API.md — Diversity Constraints](API.md#diversity-constraints) |
| All 25 ranking profiles | [API.md — Sort Modes](API.md#sort-modes) |
| Cohort-scoped trending | [API.md — Cohorts](API.md#cohort-definitions) |
| Collections and saved searches | [API.md — Collections](API.md#collections) |
| Axum embedding example | `examples/axum_embedding.rs` |
| 14 content discovery surfaces | [USE_CASES.md](USE_CASES.md) |
| Architecture and design decisions | [ARCHITECTURE.md](ARCHITECTURE.md) |

438
tidal/docs/SEQUENCE.md Normal file
View File

@ -0,0 +1,438 @@
# Sequence Diagrams
User-perspective flows for every major surface. Each diagram shows what the application sends, what tidalDB does internally, and what signals flow back to close the loop.
---
## Table of Contents
- [UC-01 · For You Feed](#uc-01--for-you-feed)
- [UC-02 · Search with Personalized Ranking](#uc-02--search-with-personalized-ranking)
- [UC-03 · Trending / Popular](#uc-03--trending--popular)
- [UC-04 · Following Feed](#uc-04--following-feed)
- [UC-05 · Related Content / Up Next](#uc-05--related-content--up-next)
- [UC-06 · Browse / Category Discovery](#uc-06--browse--category-discovery)
- [UC-07 · Notification Prioritization](#uc-07--notification-prioritization)
- [UC-15 · Cohort-Scoped Trending](#uc-15--cohort-scoped-trending)
- [Core · Engagement Feedback Loop](#core--engagement-feedback-loop)
- [Write · Content Ingest](#write--content-ingest)
---
## UC-01 · For You Feed
User opens the app. tidalDB retrieves candidates, scores them against the user's preference profile, enforces diversity, and returns a ready-to-render batch. Pagination continues with previously-returned IDs excluded.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Opens app / pulls to refresh
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nCONTEXT feed\nUSING PROFILE for_you\nFILTER unseen, unblocked\nDIVERSITY max_per_creator:2\nLIMIT 50
Note over TidalDB: 1. Load user preference vector\n2. ANN retrieval over item embeddings\n3. Apply seen / blocked filters\n4. Score via for_you profile:\n preference_match\n × engagement_velocity(24h)\n × recency_decay\n × social_proof\n5. Enforce diversity constraints\n6. Return ranked batch
TidalDB-->>App: [{item_id, score, signals_snapshot} × 50]
App-->>User: Feed renders
User->>App: Scrolls to bottom
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nCONTEXT feed\nUSING PROFILE for_you\nFILTER unseen, unblocked\nEXCLUDE [previously_returned_ids]\nLIMIT 50
TidalDB-->>App: Next batch of 50
App-->>User: Feed extends
```
**Signals powering this query:** user preference vector (built from history), item view velocity (24h window), item completion rate, user→creator interaction weight, social graph engagement, item age with decay curve, skip and hide signals.
---
## UC-02 · Search with Personalized Ranking
Text relevance is the floor — an irrelevant result never appears just because the user likes the creator. Personalization reorders within the relevant candidate set. A beginner and an expert searching the same query get different orderings.
```mermaid
sequenceDiagram
actor User
participant App
participant Embed as Embedding Service
participant TidalDB
User->>App: Types "rust tutorial beginner"
App->>Embed: embed("rust tutorial beginner")
Embed-->>App: query_vector[1536]
App->>TidalDB: SEARCH items\nQUERY "rust tutorial beginner"\nVECTOR query_vector\nFOR USER @u123\nUSING PROFILE search\nDIVERSITY max_per_creator:2\nLIMIT 20
Note over TidalDB: 1. BM25 text match → relevance scores\n2. ANN over item embeddings → semantic scores\n3. Merge: text(0.6) + semantic(0.4)\n4. Personalization layer:\n user topic engagement history\n item quality signals (completion, like ratio)\n recency curve (slow decay for tutorials)\n5. Diversity pass\n6. Return ranked results
TidalDB-->>App: [{item_id, relevance_score, rank} × 20]
App-->>User: Search results render
User->>App: Clicks result #3
App->>TidalDB: SIGNAL search_click\nitem: @item_id\nuser: @u123\nquery_context: "rust tutorial beginner"\nrank_at_click: 3
Note over TidalDB: Updates item relevance signal\nfor this query category.\nBoosts user→topic weight.
TidalDB-->>App: ok
User->>App: Watches 80% of video
App->>TidalDB: SIGNAL completion\nitem: @item_id\nuser: @u123\nratio: 0.80
Note over TidalDB: Strong positive on item quality.\nUpdates user preference vector\ntoward this topic and format.
TidalDB-->>App: ok
```
---
## UC-03 · Trending / Popular
Velocity, not volume. A video posted 4 hours ago with a high share rate outranks a video with 10× the total views but slow recent growth. The same profile applies to global, category-scoped, social-graph-scoped, and cohort-scoped trending — only the candidate set and signal scope change.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Taps "Trending" tab
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nWINDOW 24h\nDIVERSITY max_per_creator:1\nLIMIT 25
Note over TidalDB: Profile: trending\nPrimary: share_velocity(6h)\nSecondary: view_velocity(6h)\nBoost: new_user_reach (virality)\nGate: engagement_ratio > 0.03\nNo personalization.\nNo total-count signals.
TidalDB-->>App: [{item_id, trending_score, velocity_snapshot} × 25]
App-->>User: Trending page renders
User->>App: Taps "Jazz" category filter
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nFILTER category: jazz\nWINDOW 24h\nDIVERSITY max_per_creator:1\nLIMIT 25
Note over TidalDB: Same profile. Same velocity signals.\nHard filter on category metadata.\nScoped candidate set.
TidalDB-->>App: Trending in Jazz
App-->>User: "Trending in Jazz" renders
User->>App: Switches to "Among People I Follow"
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nFILTER social_graph: @u123 depth:2\nWINDOW 24h\nLIMIT 25
Note over TidalDB: Same profile.\nCandidates constrained to items\nengaged by users @u123 follows.
TidalDB-->>App: Trending among follows
App-->>User: Social trending renders
User->>App: Switches to "Trending in My Demo"
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nCOHORT locale:en-US, age:18-24\nWINDOW 24h\nLIMIT 25
Note over TidalDB: Same profile.\nSignal aggregation scoped to\nusers matching cohort predicate.
TidalDB-->>App: Cohort-scoped trending
App-->>User: Demographic trending renders
```
**One profile, four scopes.** Global, category, social, and cohort trending are the same ranking profile applied to different signal scopes. No code changes — just query parameters.
---
## UC-04 · Following Feed
The surface where users expect control. Recency-dominant, minimal algorithmic intervention. A creator's worst-performing video still appears here — because the user chose to follow them.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Taps "Following" tab
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nFILTER relationship: follows\nUSING PROFILE following\nFILTER unseen\nLIMIT 50
Note over TidalDB: Profile: following\nPrimary sort: created_at DESC\nTiebreaker: completion_rate\nHard filter: creator in user's follows\nHard filter: unseen by this user\nNo exploration budget.\nNo aggressive personalization.
TidalDB-->>App: Chronological feed from follows
App-->>User: Subscription feed renders
User->>App: Scrolls — catching up from 3 days ago
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nFILTER relationship: follows\nFILTER created_at < @cursor_timestamp\nUSING PROFILE following\nLIMIT 50
TidalDB-->>App: Older batch (cursor pagination)
App-->>User: Older content loads
User->>App: Follows a new creator
App->>TidalDB: WRITE relationship\ntype: follows\nfrom: @u123\nto: @creator_id\ntimestamp: now()
Note over TidalDB: Adds edge to relationship graph.\nUpdates user preference vector\nto include creator's topic signals.\nNew creator appears in next query.
TidalDB-->>App: ok
```
---
## UC-05 · Related Content / Up Next
Anchored to the item just consumed. Blends semantic similarity with collaborative filtering and user taste. The autoplay accept/skip signal strengthens or decays the item-to-item pairing for future recommendations.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Finishes watching @item_abc (Rust tutorial, beginner)
App->>TidalDB: SIGNAL completion\nitem: @item_abc\nuser: @u123\nratio: 0.94
Note over TidalDB: Appends to @item_abc signal ledger.\nUpdates user→item_abc relationship.\nUpdates user preference vector\ntoward rust/programming/tutorials.
TidalDB-->>App: ok
App->>TidalDB: RETRIEVE items\nSIMILAR TO @item_abc\nFOR USER @u123\nUSING PROFILE related\nFILTER unseen\nEXCLUDE creator: @item_abc.creator_id (top 3 only)\nLIMIT 10
Note over TidalDB: Profile: related\n1. ANN: items near @item_abc embedding\n2. Collaborative: items co-engaged with @item_abc\n3. Personalize: re-rank by user preference match\n4. Quality gate: completion_rate > 0.4\n5. Diversity: avoid same creator in top 3\n6. Return
TidalDB-->>App: [{item_id, similarity_score, collab_score} × 10]
App-->>User: Up Next queue renders
User->>App: Autoplay begins on result #1
App->>TidalDB: SIGNAL autoplay_accept\nsource_item: @item_abc\ntarget_item: @item_xyz\nuser: @u123
Note over TidalDB: Strong positive on item_abc → item_xyz pairing.\nStrengthens collaborative edge.
TidalDB-->>App: ok
User->>App: Skips after 8 seconds
App->>TidalDB: SIGNAL skip\nitem: @item_xyz\nuser: @u123\ndwell_ms: 8200\ncontext: autoplay_from @item_abc
Note over TidalDB: Negative on this pairing.\nDecays item_abc → item_xyz\ncollaborative edge slightly.
TidalDB-->>App: ok
```
---
## UC-06 · Browse / Category Discovery
Quality-dominant ranking within a filtered candidate set. Mix of established content and breakout newcomers. Sort mode switches (Top, New, All Time) are different profiles on the same candidate set — no application logic needed.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Taps "Jazz" category
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE browse\nDIVERSITY max_per_creator:2\nLIMIT 20
Note over TidalDB: Profile: browse\nPrimary: quality_score\n completion_rate(0.5)\n + like_ratio(0.3)\n + reach(0.2)\nRecency boost: 30d half-life\nBreakout bonus: age < 14d\n AND velocity_percentile > 0.9
TidalDB-->>App: [{item_id, quality_score} × 20]
App-->>User: Jazz browse page renders
User->>App: Switches sort to "New"
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE new\nLIMIT 20
Note over TidalDB: Profile: new\nSort: created_at DESC\nNo quality gate.
TidalDB-->>App: Latest jazz content
App-->>User: New jazz renders
User->>App: Switches sort to "Top All Time"
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE top_all_time\nDIVERSITY max_per_creator:3\nLIMIT 20
Note over TidalDB: Profile: top_all_time\nSort: total_completion_weighted_views DESC\nNo recency boost. No decay.\nPure historical quality.
TidalDB-->>App: All-time top jazz
App-->>User: Top jazz renders
```
---
## UC-07 · Notification Prioritization
Of all events since the user was last active, tidalDB decides which are worth a push and in what order they appear in the notification center. Open and dismiss signals feed back to adjust future notification priority per creator.
```mermaid
sequenceDiagram
participant Job as Background Job
participant TidalDB
participant Push as Push Service
actor User
Job->>TidalDB: RETRIEVE notifications\nFOR USER @u123\nSINCE @last_seen_timestamp\nUSING PROFILE notification\nLIMIT push:3, inbox:20
Note over TidalDB: Profile: notification\nCandidates: events from followed creators\n + social graph activity\nScore by:\n relationship_strength(user, creator)\n × item engagement_velocity at event time\n × user notification_open_rate\nHard filter: event_age > 48h → suppress\nHard filter: max 1 per creator per day
TidalDB-->>Job: push_candidates[3], inbox_candidates[20]
Job->>Push: Send push notifications (top 3)
Push->>User: Push notification arrives
User->>Push: Taps notification
Push->>Job: opened — item @item_id
Job->>TidalDB: SIGNAL notification_open\nuser: @u123\ncreator: @creator_id\nitem: @item_id
Note over TidalDB: Strong positive on user→creator\nrelationship for notification context.\nIncreases future notification priority\nfor this creator.
TidalDB-->>Job: ok
User->>Push: Swipes to dismiss
Push->>Job: dismissed
Job->>TidalDB: SIGNAL notification_dismiss\nuser: @u123\ncreator: @creator_id
Note over TidalDB: Mild negative on relationship\nfor notification context.\nReduces push frequency\nfrom this creator slightly.
TidalDB-->>Job: ok
```
---
## UC-15 · Cohort-Scoped Trending
User explores what's trending within their audience segment. tidalDB scopes signal aggregation to users matching the cohort predicate, ranks by velocity within that cohort, and supports search composition on top.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Opens "Trending For You"
Note over App: App resolves user's primary cohort<br/>from their attributes:<br/>locale:en-US, age:18-24, interest:music
App->>TidalDB: RETRIEVE items<br/>USING PROFILE trending<br/>COHORT locale:en-US, age:18-24, interest:music<br/>WINDOW 24h<br/>DIVERSITY max_per_creator:1<br/>LIMIT 25
Note over TidalDB: 1. Resolve cohort: users matching predicate<br/>2. Load cohort-scoped signal aggregates<br/> (view_velocity, share_velocity scoped<br/> to signals from cohort members)<br/>3. Rank by cohort-scoped velocity<br/>4. Gate: engagement_ratio > 0.03<br/>5. Enforce diversity<br/>6. Return ranked batch
TidalDB-->>App: [{item_id, cohort_trending_score, velocity_snapshot} × 25]
App-->>User: "Trending in your world" renders
User->>App: Searches "jazz piano" within trending
App->>TidalDB: SEARCH items<br/>QUERY "jazz piano"<br/>WITHIN TRENDING<br/>COHORT locale:en-US, age:18-24, interest:music<br/>WINDOW 24h<br/>LIMIT 20
Note over TidalDB: 1. Cohort-scoped trending candidates<br/>2. BM25 text match within candidates<br/>3. Merge: trending_score × text_relevance<br/>4. Return intersection
TidalDB-->>App: [{item_id, composite_score} × 20]
App-->>User: Search-within-trending results render
User->>App: Switches to "Trending Globally"
App->>TidalDB: RETRIEVE items<br/>USING PROFILE trending<br/>WINDOW 24h<br/>DIVERSITY max_per_creator:1<br/>LIMIT 25
Note over TidalDB: Same profile, no cohort scope.<br/>Global signal aggregates used.<br/>Different results than cohort view.
TidalDB-->>App: Global trending results
App-->>User: "Trending Globally" renders
```
**Three layers, one engine.** Global trending, cohort trending, and search-within-cohort-trending are the same ranking profile applied to different signal scopes. The profile doesn't change — the signal aggregation scope does.
---
## Core · Engagement Feedback Loop
Every interaction closes the loop in a single write transaction. There is no ETL, no Kafka consumer, no feature store sync. The next ranking query — even 100ms later — sees the updated state.
```mermaid
sequenceDiagram
actor User
participant App
participant TidalDB
User->>App: Likes a video
App->>TidalDB: SIGNAL like\nitem: @item_id\nuser: @u123\ntimestamp: now()
Note over TidalDB: Atomic write transaction:\n1. Append event to item signal ledger\n2. Update windowed aggregates:\n like_count_1h, _24h, _7d\n3. Recompute like_velocity\n4. Update user→item relationship weight\n5. Increment user→creator interaction_weight\n6. Update user preference vector\n toward item's topic embedding\n7. Attribute signal to user's cohorts\n (increment cohort-scoped counters)\n8. Commit
TidalDB-->>App: ok
Note over App: Next RETRIEVE for this user\nreflects updated signals immediately.
User->>App: Hides a video ("Not interested")
App->>TidalDB: SIGNAL hide\nitem: @item_id\nuser: @u123
Note over TidalDB: 1. Set permanent hard-negative flag\n on user→item relationship\n2. Decay user→creator interaction_weight\n3. Update user preference vector\n AWAY from item's topic embedding\n4. Item excluded from all future\n queries for this user
TidalDB-->>App: ok
User->>App: Blocks a creator
App->>TidalDB: SIGNAL block\nuser: @u123\ntarget_creator: @creator_id
Note over TidalDB: 1. Set permanent hard block\n on user→creator relationship\n2. All items by @creator_id excluded\n from every query for this user\n3. Existing relationship edges zeroed
TidalDB-->>App: ok
```
---
## Write · Content Ingest
How a new item enters the system and becomes immediately retrievable and rankable. Cold start is handled by the database via an exploration budget — the application does not manage this.
```mermaid
sequenceDiagram
actor Creator
participant App
participant Embed as Embedding Service
participant TidalDB
Creator->>App: Uploads video\n(title, description, tags, category, transcript)
App->>Embed: embed(title + description + transcript)
Embed-->>App: content_vector[1536]
App->>TidalDB: WRITE item\nid: @item_id\ncreator: @creator_id\nmetadata: {title, description, tags, category, duration}\nembedding: content_vector\ncreated_at: now()
Note over TidalDB: 1. Store metadata in entity store\n2. Index text fields into inverted index\n3. Insert vector into ANN index (HNSW)\n4. Initialize signal ledger (all zeros)\n5. Apply new-item exploration budget:\n appears in a small % of for_you feeds\n before signals accumulate\n6. Link to creator entity\n7. Commit — item is immediately queryable
TidalDB-->>App: ok, @item_id indexed
App-->>Creator: "Your content is live"
Note over TidalDB: Item is now retrievable in:\n• Search (text + semantic)\n• Following feeds of creator's followers\n• Browse / category pages\n• Trending (once signals accumulate)\n• For You (exploration budget active)
```
**Cold start is a first-class problem.** New content has no engagement signals. tidalDB handles this natively — new items receive a configurable exploration window proportional to the creator's relationship strength with their existing followers. The application does not manage this.
---
## Signal Reference
| Signal | Type | Decay | Primary Use |
|---|---|---|---|
| `view` | count | slow (7d half-life) | baseline engagement |
| `completion` | ratio 0–1 | very slow | quality signal |
| `like` | count | slow | positive sentiment |
| `share` | count | medium | virality |
| `comment` | count | medium | community |
| `skip` | count | fast (1d half-life) | negative quality |
| `hide` | bool | permanent | hard negative |
| `block` | bool | permanent | hard filter |
| `follow` | bool | permanent | relationship |
| `interaction_weight` | float | slow | relationship strength |
| `dwell_time` | duration | medium | true engagement |
| `autoplay_accept` | bool | medium | recommendation quality |
| `notification_open` | bool | slow | creator notification priority |
| `notification_dismiss` | bool | medium | reduce push frequency |
| `search_click` | count + rank | medium | query relevance signal |

779
tidal/docs/USE_CASES.md Normal file
View File

@ -0,0 +1,779 @@
# Use Cases
Each use case describes a real surface, the query it requires, how signals flow in, and what "correct" looks like. These are the scenarios the database must handle natively and efficiently.
A note on scope: this document is exhaustive by design. Every filtering mode, sort order, and discovery pattern listed here is something a real user has wanted on YouTube, Twitter, Reddit, Pinterest, Netflix, Spotify, or a media library. tidalDB must support all of them without the application building custom ranking logic.
A critical addition to this document is UC-15: Cohort-Scoped Trending. This addresses the requirement that trending, rising, and quality signals must be sliceable by audience segment — not just globally or by category, but by demographic, behavioral, and interest-based cohorts. This capability underpins advertiser-facing trend reports, localized discovery, and the "trending for people like you" surface.
---
## Table of Contents
- [UC-01 · Personalized Feed — For You](#uc-01--personalized-feed--for-you)
- [UC-02 · Search](#uc-02--search)
- [UC-03 · Trending and Rising](#uc-03--trending-and-rising)
- [UC-04 · Following / Subscription Feed](#uc-04--following--subscription-feed)
- [UC-05 · Related Content / Up Next](#uc-05--related-content--up-next)
- [UC-06 · Browse and Category Discovery](#uc-06--browse-and-category-discovery)
- [UC-07 · Notification Prioritization](#uc-07--notification-prioritization)
- [UC-08 · Creator Profile Page](#uc-08--creator-profile-page)
- [UC-09 · User Library and Personal Collections](#uc-09--user-library-and-personal-collections)
- [UC-10 · People and Creator Search](#uc-10--people-and-creator-search)
- [UC-11 · Visual and Semantic Search](#uc-11--visual-and-semantic-search)
- [UC-12 · Live and Scheduled Content](#uc-12--live-and-scheduled-content)
- [UC-13 · Hidden Gems and Breakout Detection](#uc-13--hidden-gems-and-breakout-detection)
- [UC-14 · Controversial and Hot Surfaces](#uc-14--controversial-and-hot-surfaces)
- [UC-15 · Cohort-Scoped Trending](#uc-15--cohort-scoped-trending)
- [Appendix A · Filter Reference](#appendix-a--filter-reference)
- [Appendix B · Sort Mode Reference](#appendix-b--sort-mode-reference)
- [Appendix C · Signal Reference](#appendix-c--signal-reference)
---
## UC-01 · Personalized Feed — For You
**Surface:** The primary content feed. YouTube home, TikTok FYP, Instagram feed, Twitter For You, Reddit home.
**The Question:** Given user U right now, what content should they see that they haven't seen, from creators they'll enjoy, with healthy format and topic diversity?
**Signals Required:**
- User implicit preference vector (built continuously from history)
- Item engagement velocity — views, completions, shares in the last 24h
- User→creator interaction weight (have they engaged with this creator before? how much?)
- Social proof — did people the user follows engage with this?
- Negative signals — skips, hides, mutes, "not interested" taps
- Recency decay on item age
- Completion rate as a quality gate (do not surface content people abandon)
- Format affinity — does this user prefer short-form, long-form, articles, images?
**Ranking Profile:** `for_you` — preference match and social proof weighted heavily, moderate recency decay, completion rate as quality floor, skip signals as strong negative.
**Diversity Constraints:** max 2 items per creator per batch, format mix enforced, minimum 10% exploration budget (creators the user does not follow).
**Feedback Written Back:**
- Viewed → increment view signal, update user→item relationship
- Completed → strong positive on completion signal, boost creator weight
- Skipped in under 3 seconds → strong skip signal, decay creator weight
- Liked or shared → strong positive across all signals
- "Not interested" → permanent negative on this item, decay topic weight
- "Don't recommend this creator" → hard suppress, equivalent to soft block for ranking
**What Correct Looks Like:** Feels like it knows the user without being a hall of mirrors. Some expected, some surprising. No creator dominating. Nothing seen this week resurfaced.
---
## UC-02 · Search
Search is the most complex surface because it has the most dimensions. Every sub-feature listed here is something real users rely on daily.
### 2.1 · Full-Text Keyword Search
**The Question:** User typed a query — return the most relevant results, ranked for this user specifically.
**Query Capabilities:**
- Basic keyword match: `jazz piano tutorial`
- Exact phrase match: `"jazz piano"` returns only items containing that exact sequence
- Boolean operators: `jazz AND piano NOT beginner`, `(jazz OR blues) piano`
- Exclusion: `-beginner`, `NOT beginner` — never show items matching this term
- Wildcard: `jazz pian*` matches piano, pianist, pianos
- Field-scoped search: `title:jazz`, `tag:tutorial`, `creator:username`
- Hashtag search: `#jazz` matches tagged items directly
- Minimum engagement filter inline: `jazz piano min_views:10000`
**Signals Required:**
- BM25 text relevance score (inverted index)
- Semantic similarity — query embedding vs item embedding (catches "intro to jazz" matching "jazz for beginners")
- User topic engagement history — a beginner gets beginner content elevated
- Item quality signals — completion rate, like ratio as secondary ranking
- Recency curve — configurable per content type (news decays fast, tutorials decay slowly)
**Ranking Profile:** `search` — text relevance is the floor, personalization adjusts rank above it. An irrelevant result never surfaces because the user likes the creator.
**Diversity Constraints:** max 2 results per creator in the first 10 results.
**Feedback Written Back:**
- Click at rank N → positive relevance signal, trains query→item affinity
- Immediate back-navigation → negative signal (irrelevant or low quality)
- Long dwell after click → strong positive on item and creator for this topic
- Zero clicks in a session → weak signal that results were poor for this query
### 2.2 · Advanced Search Filters
These are the filters users expect to be able to combine freely. All must be composable — any filter can be combined with any other. See Appendix A for the complete filter reference.
**By Date / Recency:**
- Presets: last hour, today, this week, this month, this year
- Custom range: `uploaded_after:2024-01-01 uploaded_before:2024-06-01`
- Relative: `uploaded_within:30d`
**By Duration:**
- Presets: short (under 4 minutes), medium (4–20 minutes), long (over 20 minutes)
- Custom range: `duration_min:5m duration_max:15m`
**By Format / Content Type:**
- Video, short-form video, live stream, VOD of past live, podcast episode, article, image, image gallery, audio-only, interactive
**By Quality / Technical Specs:**
- Resolution: SD, HD, Full HD, 4K, 8K
- HDR, Dolby Vision, Dolby Atmos, spatial audio
- Subtitles/captions available, audio description available, sign language version available
- Offline/download available
**By Language:**
- Content language, subtitle language available, dubbed version available in language X, original language only
**By Content Rating / Maturity:**
- G, PG, PG-13, R, etc.
- Safe search toggle, age-gated content filter, sensitive topics toggle
**By Creator Attributes:**
- Verified only, minimum follower count, creators the user follows only, creators new to the user, exclude a specific creator, search within one creator's catalog
**By Engagement Thresholds:**
- Minimum views, likes, like ratio, comments — lets users filter to proven content
**By Location / Geography:**
- Content created in a region, content about a location, trending in a region
**By Status / Availability:**
- Live right now, premiering soon, subscriber-only, free only, leaving platform soon, downloadable
**By Community Signals (Reddit / forum-style):**
- Flair filter, awarded/gilded only, minimum score, specific community, post type (text/link/image/video/poll), original only (exclude crossposts)
**By Seen State:**
- Unseen only, already seen (user wants to find something they watched before), in progress, saved/bookmarked
### 2.3 · Search Suggestions and Autocomplete
- Autocomplete on partial query (prefix match on popular queries)
- Trending searches in empty search bar
- Personalized suggestions based on search and watch history
- Creator name autocomplete, hashtag autocomplete
- "Did you mean" typo correction on submitted query
- Related query suggestions below results ("People also search for")
### 2.4 · Saved Searches and Alerts
- User saves a search query — gets a feed of new results matching it over time
- Alert when new content matching a saved search is published
- Search history (personal, clearable)
- Quick access to recent searches
### 2.5 · Search Within Scoped Results (Query Composition)
Search can be composed with other retrieval modes. The application specifies a retrieval scope, and search operates within that candidate set:
- **Search within trending:** "jazz piano" within globally trending items
- **Search within cohort trending:** "jazz piano" within items trending for US users aged 18-24
- **Search within following:** "jazz piano" within items from followed creators
- **Search within category:** "jazz piano" within the Jazz category (this already works via filters, but the composition model generalizes it)
Query composition means SEARCH and RETRIEVE are not separate operations — they can be layered. The database handles the intersection efficiently using its query planner.
---
## UC-03 · Trending and Rising
### 3.1 · Trending
**Surface:** Trending tab, Explore page, "What's happening" sidebar.
**The Question:** What content is gaining real traction right now — not what has the most views historically?
**Signals Required:**
- Share velocity — rate of new shares (strongest trending signal)
- View velocity — rate of new views, not total views
- New-user reach — percentage of viewers new to this creator (measures virality, not fanbase loyalty)
- Engagement ratio — (likes + comments + shares) / views — filters clickbait
- Comment velocity — discussions erupting quickly signal cultural relevance
**Ranking Profile:** `trending` — velocity signals only, windowed 1h/6h/24h. No personalization. Total view count is explicitly not a primary signal.
**Scoping (same profile, different candidate sets):**
- Global trending
- Trending in category/genre
- Trending in my language/region
- Trending among people I follow
- Trending within a specific community
- **Trending within a cohort (demographic/behavioral segment)**
- **Search within cohort-scoped trending**
See UC-15 for full cohort-scoped trending specification. Cohort trending uses the same velocity-based ranking profile but scopes signal aggregation to users matching a cohort predicate.
**Diversity Constraints:** max 1 item per creator in top 10.
### 3.2 · Rising (Hot / Breakout)
**The Question:** What new content is overperforming for its age? Designed to surface content that has not yet reached trending thresholds but is clearly on its way.
This is the Reddit "rising" concept applied broadly.
**Signals Required:**
- Age of content (hard weight — very new content gets a boost)
- Engagement velocity relative to creator's own baseline (a small creator getting 10× their normal engagement is "rising")
- Engagement velocity relative to category baseline
- Share-to-view ratio (high share rate relative to views signals genuine enthusiasm)
**Ranking Profile:** `rising` — age-weighted velocity. A 2-hour-old video with 5k views and a 15% share rate outranks a 2-day-old video with 100k views and a 0.3% share rate.
---
## UC-04 · Following / Subscription Feed
**Surface:** YouTube Subscriptions tab, Twitter Following tab, Substack inbox, Twitch following list.
**The Question:** Show me everything from creators I follow, in the right order, with nothing missing.
**Signals Required:**
- Relationship: user follows creator (hard filter — only followed creators)
- Recency: primary sort signal
- Light quality gate: completion rate as tiebreaker within same-minute posts
- Seen flag: filter already-seen items (optional — some users want all posts regardless)
**Ranking Profile:** `following` — recency-dominant, minimal algorithmic intervention. This is the surface where users feel most strongly that the algorithm should stay out of the way.
**Diversity Constraints:** None by default. If a followed creator posts 10 times in one day, show all 10.
**Modes:**
- Chronological (pure reverse time order)
- Chronological with quality tiebreaker (same timestamp → prefer higher quality)
- Algorithmic following (light ranking — surfaces the most engaging posts from follows first, for users who follow too many to consume everything)
**What Correct Looks Like:** Nothing missing. Nothing reordered dramatically. The user trusts this surface because it reflects their explicit choices.
---
## UC-05 · Related Content / Up Next
**Surface:** YouTube right rail, Spotify Radio, Netflix "More Like This," Pinterest related pins, end-screen recommendations.
**The Question:** Given what the user just consumed, what is the natural next thing?
**Signals Required:**
- Semantic similarity — embedding distance between source item and candidates
- Collaborative filtering — users who engaged with item A also engaged with item B
- User preference match — semantically similar AND matches this user's taste
- Content journey awareness — a user who just watched beginner content should see intermediate next, not more beginner
- Quality gate — completion rate (do not autoplay bad content)
- Novelty — do not recommend items the user has already seen
**Ranking Profile:** `related` — semantic similarity as primary retrieval signal, collaborative filtering as secondary boost, user preference as personalization layer.
**Diversity Constraints:** avoid same creator as source item in first 3 results (unless on a creator profile page).
**Feedback Written Back:**
- Autoplay accepted → strong positive on source→target pairing
- Autoplay skipped under 10 seconds → negative on pairing
- Manual click on sidebar → weaker positive than autoplay accept
- Saved to watch later → strong positive
---
## UC-06 · Browse and Category Discovery
**Surface:** Genre pages, mood pages, topic pages, aesthetic boards (Pinterest).
### 6.1 · Standard Browse
**Sort modes users expect within a category:**
**Top / Best** — all-time quality rank. Completion rate + like ratio + total reach. Stable. Does not change hourly.
**New** — pure reverse chronological. No quality gate. Shows everything. Users use this to find content the algorithm hasn't surfaced yet.
**Hot** — recency + engagement combined. Content decays as it ages regardless of engagement. The Reddit model: score / (age_hours + 2)^gravity. Refreshes meaningfully every hour.
**Rising** — overperforming new content (see UC-03.2).
**Trending** — velocity-based, short time window (see UC-03.1).
**Controversial** — high engagement with polarized sentiment. High comment count, moderate like ratio, high share count. Surfaces content generating strong reactions in both directions.
**Top: This Hour / Today / This Week / This Month / This Year / All Time** — windowed top sort. Lets users choose their recency preference explicitly. All-Time Top is useful for discovering classics. This Week is useful for staying current without hourly noise.
**Shuffle / Random** — random sample weighted by quality score. Useful for music, podcasts, and "surprise me" contexts.
**Alphabetical** — A-Z / Z-A. Useful for structured collections and course curricula.
**Shortest First / Longest First** — sort by duration. Users looking for something quick explicitly want this.
**Highest Rated** — explicit critic or audience score where available, distinct from like ratio.
### 6.2 · Faceted Browse (Multiple Simultaneous Filters)
All filters must be composable simultaneously. Examples of real user behavior:
- Genre: Jazz AND Duration: Short AND New (last 7 days)
- Format: Podcast AND Language: Spanish AND Top: This Month
- Creator: Verified AND Category: Cooking AND Sort: Hot
- Mood: Focus AND Duration: Long AND Unseen only
- Quality: 4K AND Has Subtitles AND Top: All Time
The database must handle arbitrary filter combinations without the application implementing them. Faceted queries are a first-class operation.
### 6.3 · Mood and Aesthetic Filters
Common on music, video, and Pinterest-style platforms. Moods are not categories — they are cross-cutting signals derived from engagement patterns and embeddings.
- Mood: Chill, Energetic, Focus, Sad, Happy, Hype, Romantic, Dark, Nostalgic
- Aesthetic: Minimalist, Maximalist, Vintage, Futuristic, Cottagecore, Brutalist
- Era/Decade: 70s, 80s, 90s, 2000s — useful for music, film, fashion content
These are best modeled as embedding-space regions rather than explicit tags. A "chill" query retrieves items whose embeddings cluster near what users seeking "chill" content engage with.
### 6.4 · Color and Visual Filtering (Pinterest Model)
When items are images or have dominant visual content:
- Filter by dominant color or color palette
- "More like this image" — visual similarity search
- Style similarity — find visually similar items even without shared tags
- Crop-and-search — user selects a region of an image and searches for items similar to that region
These require visual embeddings on items. The application provides the embedding. The database handles retrieval and ranking.
---
## UC-07 · Notification Prioritization
**Surface:** Push notifications, in-app notification center, email digest.
**The Question:** Of all events since the user was last active, which deserve a push? Which deserve inbox prominence?
**Signals Required:**
- Relationship strength — notification from a creator they interact with constantly vs. one they follow but never open
- Quality of the triggering item — a video already performing well is more worth notifying about
- User notification open rate — are they opening notifications lately? If not, reduce frequency (fatigue signal)
- Time since event — events older than 48h are suppressed
- Notification type priority — a reply to the user's comment > a new video from a creator they loosely follow
**Ranking Profile:** `notification` — relationship strength dominant, item quality secondary, strict recency filter.
**Diversity Constraints:** max 1 push per creator per day, max 3 total pushes per day per user, max 1 per topic cluster per hour.
**Feedback Written Back:**
- Opened → strong positive on creator relationship for notification context
- Dismissed → mild negative, reduce future frequency
- Notifications disabled for creator → permanent suppress
- App opened directly (not via notification) → weak positive on all pending notifications
---
## UC-08 · Creator Profile Page
**Surface:** A creator's profile — their complete catalog, browsable by the visitor.
**The Question:** Show this creator's content in the best order for this visitor.
**Modes:**
- **Top** — all-time quality. Best first impression for new visitors.
- **New** — reverse chronological. For fans keeping up.
- **Hot** — currently performing best within the creator's own catalog.
- **For You** — which of this creator's items best matches this visitor's preferences. A jazz fan visiting a multi-genre creator sees jazz content elevated even if the creator's pop content has more total views.
- **Series / Playlists** — items grouped into explicit collections, ordered within collection.
**What Correct Looks Like:** A first-time visitor and a longtime subscriber see different orderings on "For You." The new visitor sees the creator's best all-time content. The subscriber sees what they haven't yet watched from this creator.
---
## UC-09 · User Library and Personal Collections
**Surface:** YouTube Library, Spotify Liked Songs, Instagram Saved, Reddit Saved, Pinterest Boards, Watch History.
### 9.1 · Watch / View History
- Complete chronological history of items the user viewed
- Filterable by: date range, category, creator, format, duration
- Searchable by keyword within history
- Clearable (individual items or full history)
- "Continue Watching" — items viewed but not completed, sorted by most recently viewed
- Resume playback position stored per item
### 9.2 · Saved / Bookmarked Items
- Items explicitly saved (Watch Later, Saved Posts, Bookmarks)
- Sortable by: date saved (default), date published, creator, category, duration
- Filterable by: category, creator, format, unseen vs. seen
- Expiry detection — saved items that have since been deleted or become unavailable
- Bulk management — mark batch as watched, remove batch
### 9.3 · Liked Items
- All items the user has liked
- Sortable by: date liked, creator, category
- Used as a strong signal in preference vector construction
### 9.4 · User-Created Collections / Boards / Playlists
- Named collections the user creates and curates
- Items can belong to multiple collections
- Collections can be private, shared with specific users, or public
- Collections themselves are rankable — popular public playlists surface in browse/search
- Collaborative collections (multiple users contribute — shared boards, Pinterest-style)
### 9.5 · Downloads / Offline
- Items downloaded for offline viewing
- Filterable, sortable, manageable separately from online library
- Download state as a retrievable attribute in queries
---
## UC-10 · People and Creator Search
**Surface:** Search results "People" tab, "Accounts" tab, creator discovery.
**The Question:** User wants to find creators, not content.
**Capabilities:**
- Search by creator name, username, handle
- Search creators by topic: "find creators who post about jazz"
- Filter creators by: follower count range, posting frequency, category, language, location, verified status
- "Creators like [creator X]" — semantic similarity between creator embeddings (built from their catalog)
- "Creators followed by people I follow" — social graph traversal
- "Creators I used to follow" — historical relationship query
- Sort creators by: follower count, posting frequency, engagement rate, recent activity
**Signals Required:**
- Creator-level embedding (derived from their item embeddings, aggregated)
- Creator engagement rate (average engagement ratio across recent catalog)
- Creator posting frequency
- Social graph (who follows them, who follows the current user)
- User's creator affinity history
---
## UC-11 · Visual and Semantic Search
**Surface:** Pinterest visual search, Google Lens-style search, "find more like this image."
### 11.1 · Search by Image
- User uploads an image or selects one from the platform
- Find items whose visual embedding is nearest to the query image embedding
- Crop-and-search — user selects a region of the image to search against
- Combine with text: image embedding + text query vector, merged score
### 11.2 · Semantic / Intent Search
Beyond keyword matching — understanding what the user means, not just what they typed.
- Query: "something relaxing to watch on a rainy day" → system interprets mood/intent, retrieves by embedding similarity to that intent
- Query: "that video about the jazz pianist in new orleans I watched last year" → retrieves from user history using semantic match, not exact title
- Disambiguation: "jaguar" — is the user searching for the car or the animal? User history and query context disambiguate
### 11.3 · Multi-Modal Retrieval
- Text query against image items (find images matching a text description)
- Image query against video items (find videos containing visuals similar to a reference image)
- Audio fingerprint query against audio items — tidalDB handles the embedding retrieval, not the generation
---
## UC-12 · Live and Scheduled Content
**Surface:** Live tab, "Happening Now," event pages, premiere countdowns.
**The Question:** What is live right now that this user would care about?
**Signals Required:**
- Live status flag (boolean, real-time)
- Scheduled start time and end time
- Current viewer count (real-time signal)
- Creator relationship weight (live from a creator they care about > random live)
- Category match with user preferences
- Notification opt-in (did the user set a reminder?)
**Ranking Profile:** `live` — relationship weight dominant, current viewer count as social proof, category match secondary. Recency is not a concept here — everything is happening now.
**Discovery of upcoming:**
- "Premiering in X hours" — scheduled content with countdown
- "Set reminder" → creates a notification relationship between user and item
- Calendar-style browse of upcoming events
**Filtering:**
- Live only (exclude VOD)
- By category within live
- By minimum viewer count
- From followed creators only
---
## UC-13 · Hidden Gems and Breakout Detection
**Surface:** "Underrated," "Staff Picks," "You might have missed," editorial surfaces.
**The Question:** What high-quality content is being overlooked by the algorithm?
Hidden gems are items with high completion rate and like ratio but low total view count relative to those quality signals. Content that performs well with everyone who sees it but hasn't been seen by many people yet.
**Signals Required:**
- Quality signals: completion rate, like ratio — must be high
- Reach signals: total views — must be low relative to quality
- Age of content — recent enough to still be worth surfacing
- Creator follower count — small/new creators get priority
**Ranking Profile:** `hidden_gems` — quality signals as primary, inverse of reach as a boost, creator size as a discovery equity signal.
**What Correct Looks Like:** Content that makes the user think "how have I never seen this before?" Not content that is obscure because it is bad.
---
## UC-14 · Controversial and Hot Surfaces
### 14.1 · Controversial Sort
**Surface:** Reddit "Controversial" sort, comment sections surfacing heated debates.
**The Question:** What content is generating strong reactions in both directions?
Controversial is defined as: high total engagement AND polarized sentiment. High comment count, high share count, but split positive/negative signal ratio. Content people feel strongly about in opposite directions.
**Signals Required:**
- Total engagement (must be high enough to be genuinely controversial, not just unpopular)
- Sentiment polarity — ratio of positive to negative signals
- Comment velocity — discussions growing quickly
- Share count — even content people dislike gets shared for debate
**Ranking Profile:** `controversial` — maximizes the product of positive and negative engagement signals. A post with 1000 upvotes and 1000 downvotes scores higher than one with 1800 upvotes and 200 downvotes.
### 14.2 · Hot Sort (Reddit Model)
**Surface:** Reddit "Hot," Hacker News front page, time-sensitive community surfaces.
**The Question:** What is the best content right now, with age decay applied?
Hot rewards early engagement but punishes age. Formula concept: `score / (age_hours + 2)^gravity`. The database exposes this as a native sort mode — the application does not implement the formula.
**What makes Hot different from Trending:** Trending is pure velocity (rate of change). Hot is cumulative score with age decay. An hour-old post with 500 upvotes scores higher on Hot than a day-old post with 2000 upvotes.
---
## UC-15 · Cohort-Scoped Trending
**Surface:** "Trending for You" (personalized trending), audience-segmented trending dashboards, advertiser-facing trend reports, regional/demographic trend pages.
**The Question:** What content is gaining traction right now among users who match this profile?
This is distinct from global trending (UC-03) and personalized feeds (UC-01). Global trending answers "what's popular everywhere." Personalized feeds answer "what should this specific user see." Cohort trending answers "what's resonating with this type of user" — a question that sits between the two.
**Cohort Definition:**
A cohort is a named predicate over user attributes:
- Demographic: locale, age range, gender
- Interest-based: users who engage with jazz, cooking, tech, etc.
- Behavioral: power users, casual browsers, binge watchers
- Geographic: users in a specific region or timezone
- Composite: US + age 18-24 + interest:jazz (AND logic across dimensions)
**Signals Required:**
- All the same velocity signals as UC-03 (share velocity, view velocity, engagement ratio)
- But scoped to signal events generated by users matching the cohort predicate
- Cohort-scoped view_velocity(24h) = rate of views from cohort members, not global views
**Three-Layer Model:**
1. **Global trending** — same as UC-03, no cohort filter
2. **Cohort trending** — velocity signals filtered to cohort members
3. **Search within cohort trending** — text/semantic search composed with cohort trending
**Ranking Profile:** `cohort_trending` — same velocity-based ranking as `trending`, but candidate set and signal aggregation scoped to cohort.
**Scoping (composable):**
- Cohort: locale:US, age:18-24
- Cohort + category: above AND category:jazz
- Cohort + search: above AND QUERY "piano tutorial"
- Cohort + social: above AND social_graph:@u123
**Diversity Constraints:** max 1 item per creator in top 10.
**What Correct Looks Like:** A 22-year-old in Tokyo and a 45-year-old in Texas see different trending pages. Not because of personalization (individual preference), but because different content is genuinely trending within their respective audience segments. An advertiser can see what's trending among their target demographic. A creator can see what's trending in their niche audience.
---
## Appendix A · Filter Reference
All filters must be composable with each other and with any sort mode. A query combining any subset of these is a valid, first-class database operation.
### Content Attribute Filters
| Filter | Values | Notes |
|---|---|---|
| `category` | string or list | multi-select: Jazz OR Blues OR Soul |
| `tag` | string or list | multi-select |
| `hashtag` | string | exact match |
| `flair` | string | community-specific labels |
| `format` | video, short, live, vod, podcast, article, image, gallery, audio | |
| `duration` | range (min, max) or preset | short / medium / long presets |
| `language` | ISO code | content language |
| `subtitle_language` | ISO code | subtitles available in this language |
| `dubbed_language` | ISO code | dubbed version available |
| `resolution` | SD, HD, FHD, 4K, 8K | |
| `hdr` | bool | |
| `audio_quality` | standard, high, lossless, spatial | |
| `has_subtitles` | bool | |
| `has_audio_description` | bool | accessibility |
| `has_sign_language` | bool | accessibility |
| `content_rating` | G, PG, PG-13, R, etc. | |
| `safe_search` | bool | |
| `sensitive_content` | show, hide, only | |
| `status` | published, live, scheduled, archived | |
| `availability` | free, premium, subscriber_only | |
| `downloadable` | bool | |
| `leaving_soon` | bool or date threshold | availability window ending |
| `award_count` | minimum int | gilded/awarded |
| `has_award` | bool | |
| `post_type` | text, link, image, video, poll, crosspost | |
| `original_only` | bool | exclude crossposts/reposts |
### Date and Time Filters
| Filter | Values |
|---|---|
| `created_after` | ISO date |
| `created_before` | ISO date |
| `created_within` | duration: 7d, 30d, 1y |
| `created_preset` | hour, today, week, month, year |
| `updated_after` | ISO date |
| `event_date` | date range for scheduled/live content |
### Creator Filters
| Filter | Values |
|---|---|
| `creator` | creator_id or handle |
| `exclude_creator` | creator_id or handle |
| `creator_min_followers` | integer |
| `creator_max_followers` | integer |
| `creator_verified` | bool |
| `creator_followed_by_user` | bool |
| `creator_new_to_user` | bool — never seen this creator before |
| `creator_language` | ISO code |
| `creator_location` | region or country |
### Engagement Threshold Filters
| Filter | Values |
|---|---|
| `min_views` | integer |
| `max_views` | integer — for hidden gems |
| `min_likes` | integer |
| `min_like_ratio` | float 0–1 |
| `min_comments` | integer |
| `min_shares` | integer |
| `min_score` | integer — upvotes for forum-style |
| `min_completion_rate` | float 0–1 |
### User State Filters
| Filter | Values |
|---|---|
| `seen` | bool — true = already seen, false = unseen only |
| `in_progress` | bool — partially watched |
| `saved` | bool — in user's saved/bookmarked |
| `liked` | bool — user has liked this |
| `downloaded` | bool — available offline |
| `in_collection` | collection_id |
### Geographic Filters
| Filter | Values |
|---|---|
| `content_region` | country or region code |
| `trending_in_region` | country or region code |
| `creator_region` | country or region code |
| `near_location` | lat/lng + radius |
### Cohort Filters
| Filter | Values | Notes |
|---|---|---|
| `cohort` | cohort_name | Pre-defined named cohort |
| `cohort_locale` | locale code (en-US, ja-JP) | User locale match |
| `cohort_age` | range (18-24, 25-34) | User age range |
| `cohort_interest` | keyword or list | User interest match |
| `cohort_engagement_level` | power, regular, casual | Behavioral segment |
| `cohort_format_preference` | short, long, mixed | Content format preference |
---
## Appendix B · Sort Mode Reference
All sort modes must be available on any surface. The application specifies the sort mode; tidalDB executes it natively without application-side sorting logic.
| Sort Mode | Description | Best For |
|---|---|---|
| `relevance` | Text + semantic match score | Search results |
| `personalized` | User preference match | For You surfaces |
| `new` | `created_at DESC` | Latest content |
| `old` | `created_at ASC` | Archives, chronological viewing |
| `top_all_time` | Cumulative quality score, no decay | Classic / best-of |
| `top_hour` | Quality score, last 1h | Real-time quality |
| `top_today` | Quality score, last 24h | Daily best |
| `top_week` | Quality score, last 7d | Weekly digest |
| `top_month` | Quality score, last 30d | Monthly recap |
| `top_year` | Quality score, last 365d | Annual best |
| `hot` | Score / (age + 2)^gravity — decays with time | Community frontpages |
| `trending` | Pure engagement velocity | Trending tabs |
| `rising` | Velocity relative to baseline, age-boosted | Breakout content |
| `controversial` | max(positive_signals × negative_signals) | Debate/discussion |
| `hidden_gems` | High quality, low reach, inverse boost | Discovery |
| `most_viewed` | Raw view count DESC | All-time popularity |
| `most_liked` | Raw like count DESC | Positive sentiment |
| `most_commented` | Raw comment count DESC | Discussion |
| `most_shared` | Raw share count DESC | Virality |
| `shortest` | `duration ASC` | Quick content |
| `longest` | `duration DESC` | Deep dives |
| `alphabetical_asc` | Title A–Z | Structured catalogs |
| `alphabetical_desc` | Title Z–A | |
| `shuffle` | Random, weighted by quality | Music, "surprise me" |
| `live_viewer_count` | Current viewer count DESC | Live surfaces |
| `date_saved` | When user saved/bookmarked DESC | Personal library |
| `creator_engagement_rate` | Creator's avg engagement ratio | Creator discovery |
---
## Appendix C · Signal Reference
| Signal | Type | Decay | Primary Use |
|---|---|---|---|
| `view` | count | slow (7d half-life) | baseline engagement |
| `unique_view` | count | slow | deduped reach |
| `impression` | count | fast | exposure without engagement |
| `completion` | ratio 0–1 | very slow | quality signal |
| `partial_completion` | float — last position | slow | continue watching |
| `like` | count | slow | positive sentiment |
| `dislike` | count | slow | negative sentiment |
| `share` | count | medium | virality |
| `repost` | count | medium | Twitter RT / reblog equivalent |
| `quote` | count | medium | engaged reshare with commentary |
| `comment` | count | medium | community engagement |
| `reply` | count | medium | discussion depth |
| `upvote` | count | medium | forum positive signal |
| `downvote` | count | medium | forum negative signal |
| `save` | count | slow | intent to return |
| `pin` | count | slow | Pinterest save-equivalent |
| `collection_add` | count | slow | curation signal |
| `download` | count | slow | high-intent engagement |
| `screenshot` | count | slow | save-intent (Pinterest model) |
| `outbound_click` | count | medium | link content engagement |
| `skip` | count | fast (1d half-life) | negative quality |
| `skip_intro` | bool | fast | format preference |
| `hide` | bool | permanent | hard negative |
| `not_interested` | bool | permanent | hard negative on topic |
| `block` | bool | permanent | hard filter |
| `mute` | bool | permanent | soft filter |
| `report` | count | permanent | quality / moderation flag |
| `follow` | bool | permanent | relationship |
| `unfollow` | event | decays follow signal | relationship decay |
| `interaction_weight` | float | slow | relationship strength |
| `dwell_time` | duration | medium | true engagement depth |
| `replay` | count | medium | exceptional content signal |
| `autoplay_accept` | bool | medium | recommendation quality |
| `autoplay_reject` | bool | fast | recommendation failure |
| `notification_open` | bool | slow | creator notification priority |
| `notification_dismiss` | bool | medium | reduce push frequency |
| `reminder_set` | bool | slow | intent signal for scheduled content |
| `search_click` | count + rank | medium | query relevance |
| `search_impression` | count | fast | query exposure |
| `award_given` | count | permanent | community quality endorsement |

225
tidal/docs/VISION.md Normal file
View File

@ -0,0 +1,225 @@
# Vision
## The Problem
Every platform that serves personalized content — a media library, a social feed, a marketplace, a content discovery surface — eventually builds the same distributed system from scratch. Elasticsearch for retrieval. Redis for hot signals. Kafka for event ingestion. A feature store for user profiles. A vector database for semantic search. A ranking service that tries to stitch all of the above together into a single ordered list.
This is not an ecosystem. It is scar tissue. The seams between these systems are where correctness dies — stale signals, inconsistent ranking, cache invalidation bugs, and an operational burden that consumes entire engineering teams.
The root cause is that existing databases were not built with this problem in mind. They treat ranking as an afterthought — a sort clause, a float field, a bolt-on scoring function. They have no concept of a signal that evolves over time, no concept of a user context that shapes relevance, no concept of diversity as a query constraint, no concept of the feedback loop between what a user sees and what the system learns.
Worse: every team building one of these platforms discovers that their users want the same things. Search with typo tolerance and boolean operators. Filter by duration, date, language, format, quality, creator size, and a dozen other dimensions simultaneously. Sort by trending, hot, rising, controversial, top-this-week, hidden gems, shuffle. Personalize the result of all of the above. Apply diversity constraints. Close the feedback loop.
These are not exotic requirements. They are table stakes for any serious content platform. And today, every team builds them from scratch, on top of systems not designed for the task.
## The Thesis
**Ranking is not a feature. It is a primitive.**
A database purpose-built for personalized content delivery should model the world the way this problem actually works:
- Content has metadata, embeddings, and signals. Signals are not fields — they are typed, timestamped streams with native decay, velocity, and windowed aggregation semantics.
- Users have preferences, histories, and relationships. These are not rows — they are living profiles that update continuously as events arrive.
- Personalization has scopes — global, community, and session/agent — and every scope is user-controlled and revocable.
- Agents mediate most interactions. They retrieve context, elicit preferences, and publish structured feedback (reward, tool usage, confidence) as first-class signals. The system must let them read and write memory instantly.
- A query is not "give me items matching these filters sorted by this field." It is "given this user, this context, and this surface — what should they see, in what order, subject to these constraints?"
- Filters, sort modes, and diversity rules are first-class query citizens — not application logic bolted on top.
- Engagement is not application logic that happens to write back into the database. It is a first-class write path that closes the feedback loop natively.
This is the database that models that world.
## What It Is
A single-node-first, embeddable Rust database designed specifically for the **personalized content ranking problem**. It replaces the 6-system stack for this one domain with a single process, a single query interface, and a single operational model.
It is strongly opinionated. It does not try to be a general-purpose database. It does not try to solve problems outside its domain. Every design decision is made in service of one question: **given a user and a context, what content should they see, and in what order?**
### First-Class Primitives
**Entities** are the nodes of the system — Items (content), Users, and Creators. Every entity has metadata, a vector embedding slot, and an attached signal ledger.
**Signals** are typed, timestamped event streams. The database natively understands signal semantics: velocity (rate of change), decay (exponential or linear, configurable per signal type), and windowed aggregation (last hour, last day, last 7 days, all time). You do not pre-compute `trending_score_7d` and store it in a field. You declare a `view` signal type and query its 7-day windowed velocity at ranking time.
**Users** have preferences, histories, relationships, and attributes. Attributes include demographics, locale, interests, and behavioral segments. These attributes are queryable for cohort membership and enable cohort-scoped signal aggregation. Some attributes are application-set (locale, age); others are database-computed from engagement patterns (interest affinity, engagement level, format preferences).
**Relationships** are first-class edges between entities — follows, blocks, interactions, similarity. They are weighted, directional, and traversable in queries.
**Ranking Profiles** are named, versioned scoring functions declared in schema. They reference signals, relationship weights, recency curves, and diversity rules. A profile is not code deployed separately — it lives in the database, is versioned alongside your data, and can be swapped at query time by name.
**Cohorts** are named predicates over user attributes — demographic, behavioral, and interest-based segments. A cohort is not a static list of users — it is a live query over user state. "US users aged 18-24 who engage with jazz content" is a cohort. The database maintains per-cohort signal aggregation so that trending, rising, and quality signals can be scoped to any cohort at query time. This enables the three-layer trending model: global trending, cohort-scoped trending, and search within cohort-scoped trending.
**Personalization Layers** are composable scopes over the same signal model: a user-owned global profile, optional community overlays, and short-lived session/agent context. Ranking can blend them, but ownership stays explicit per layer.
**Sessions / Agent Context** capture in-flight conversations and tool use. They bind a user, an agent, and a session identifier to short-lived signals (preference hints, rewards, critiques) with aggressive decay. Sessions can be forked, merged, and policy-limited so an agent only sees what it is allowed to remember. Users can revoke agent scope and remove agent-contributed signals from specific personalization layers.
**The Query** is a single operation that encapsulates candidate retrieval, filtering, ranking, and diversity enforcement:
```
RETRIEVE items
FOR USER @user_id
FOR SESSION @session_id
CONTEXT feed
USING PROFILE for_you
FILTER unseen, unblocked, format:video, duration:short
DIVERSITY max_per_creator:2, format_mix:true
LIMIT 50
```
This is what 6 systems currently produce. It is one query here.
Cohort scoping and query composition extend this further. Trending scoped to a cohort:
```
RETRIEVE items
USING PROFILE trending
COHORT locale:US, age:18-24, interest:jazz
WINDOW 24h
DIVERSITY max_per_creator:1
LIMIT 25
```
Search within cohort-scoped trending:
```
SEARCH items
QUERY "piano tutorial"
FOR SESSION @session_id
WITHIN TRENDING
COHORT locale:US, age:18-24, interest:jazz
WINDOW 24h
LIMIT 20
```
Three queries, three layers of the same question: what's happening globally, what's happening for people like this, and can I find something specific within that.
### The Full Query Surface
tidalDB is designed to handle every retrieval and ranking pattern a content platform needs. This is the complete surface the database covers natively:
**Retrieval modes:**
- Full-text keyword search with BM25 relevance scoring
- Exact phrase match, boolean operators (AND/OR/NOT), field-scoped search
- Semantic search — query by meaning, not just keywords
- Vector similarity search — ANN over item and creator embeddings
- Visual similarity search — find items near a reference image embedding
- Hybrid search — text relevance + semantic similarity, merged score
- User history search — find something the user previously engaged with
- Collaborative filtering — "users who engaged with X also engaged with Y"
- Social graph traversal — content from or engaged by a user's follows
**Sort modes (all native, no application implementation required):**
- Relevance (text + semantic match)
- Personalized (user preference match)
- New / Old (chronological)
- Hot (score with age decay — Reddit model)
- Trending (pure velocity)
- Rising (velocity relative to creator/category baseline, age-boosted)
- Top: All Time / This Year / This Month / This Week / Today / This Hour
- Controversial (maximizes product of positive and negative signals)
- Hidden Gems (high quality, low reach)
- Most Viewed / Most Liked / Most Commented / Most Shared
- Shortest / Longest (by duration)
- Alphabetical A-Z / Z-A
- Shuffle (random, quality-weighted)
- Live Viewer Count (for live surfaces)
- Date Saved (for personal library)
**Filter dimensions (all composable simultaneously):**
- Content type / format: video, short, live, VOD, podcast, article, image, gallery, audio
- Duration: range or presets (short / medium / long)
- Date range: presets or custom (last hour, today, this week, custom range)
- Category, tag, hashtag, flair (multi-select, OR logic within dimension)
- Language, subtitle language, dubbed language
- Technical quality: SD / HD / 4K / HDR / Dolby / spatial audio
- Accessibility: subtitles available, audio description, sign language
- Content rating / maturity level
- Safe search toggle
- Status: published, live, scheduled, archived
- Availability: free, premium, subscriber-only, downloadable, leaving soon
- Creator: specific creator, exclude creator, verified only, follower count range, new to user, followed by user
- Engagement thresholds: minimum views, likes, like ratio, comments, shares, completion rate
- Community signals: flair, minimum score, award/gilded, post type, original only
- User state: unseen, in progress, saved, liked, downloaded, in collection
- Geography: content region, creator region, near location, trending in region
**Discovery surfaces (all driven by the same underlying query engine):**
- For You personalized feed
- Following / subscription feed
- Trending (global, category-scoped, cohort-scoped, social-graph-scoped, region-scoped)
- Cohort-scoped discovery — "trending for people like you"
- Rising / breakout content
- Browse by category with any sort mode
- Related / up next recommendations
- Hidden gems and underrated content
- Live and scheduled content
- Mood and aesthetic-filtered browse
- Visual similarity browse (Pinterest model)
- Creator discovery ("creators like X")
- Notification prioritization
- Search suggestions and autocomplete
- Saved searches as persistent feeds
Every one of these surfaces is driven by the same underlying query primitives. The application does not implement ranking logic — it specifies profiles, filters, and context.
### The Feedback Loop
When a user engages with content — directly or via an agent — that event is written to the database as a signal. The agent can attach structured metadata (reward, confidence, tool invocation) in the same write. The database updates the item's signal ledger, the user's implicit preference profile, the relationship weight, and the session-scoped memory — automatically, as part of the write transaction. The next ranking or grounding query reflects this immediately. There is no Kafka consumer to lag, no feature store sync to schedule, no cache to invalidate.
Negative signals are equal citizens. A skip, a hide, a block, a "not interested," a downvote — these update the system with the same immediacy and precision as a like or a completion.
Feedback intent is typed (`skip_for_now`, `not_for_me`, `low_quality`, `hide/mute/block`) and scope-aware (local, community, session/agent), so ranking updates preserve meaning rather than collapsing all negatives into one bucket.
## What It Is Not
It is not a general-purpose document store. It is not a replacement for PostgreSQL for your transactional data. It is not trying to win the NewSQL wars or build a distributed OLAP engine.
It is not schema-free. Strong opinions about data shape enable strong guarantees about ranking correctness.
It is not trying to generate embeddings. It accepts vectors — you bring your model, you bring your embeddings, you write them in. The database owns retrieval and ranking over those vectors, not generation.
It is embeddable first — it runs in your process with zero operational overhead. But it is designed for scale from day one. Key encoding, storage isolation, and signal aggregation are all partitioning-ready. The single-node deployment is the first target, not the ceiling. When you outgrow one node, the architecture supports horizontal scaling without a rewrite.
It is not trying to solve moderation, payments, authentication, or content delivery. It solves one problem: given a user and a context, what content should they see, and in what order.
## Design Principles
**Temporal decay is a type, not a formula you write.** Signal half-lives are declared in schema. The database applies them at query time.
**Negative signals are equal citizens.** A skip, a hide, a block, a mute, a downvote — these are not the absence of positive engagement. They are data. They belong in the ranking function with the same weight and precision as a like.
**Feedback intent is explicit.** "Not for me," "low quality," and "skip for now" are different semantics, with different ranking effects and retention policies.
**All sort modes are native.** Trending, hot, rising, controversial, hidden gems, shuffle — these are built-in sort modes, not formulas the application implements and passes in. The application names a sort mode. The database executes it correctly.
**All filters are composable.** Any combination of filter dimensions produces a valid, efficiently-executed query. There is no special-casing for "common" filter combinations. Faceted queries are first-class.
**Diversity is a query constraint, not application logic.** "No more than 2 items per creator" does not belong in your API layer. It belongs in the query.
**The write path and the read path are one system.** Engagement events and ranking queries share a storage model and a signal ledger. There is no ETL between them.
**Cold start is handled by the database.** New content with no signals gets an exploration budget. New users with no history get a sensible default experience. The application does not manage this.
**Cohorts are live queries, not static lists.** A cohort is a predicate over user attributes — demographics, interests, behavioral segments. Users flow in and out of cohorts as their attributes change. Signal aggregation runs per-cohort so trending and quality signals reflect what's happening within any audience segment.
**Agents own managed contexts.** Sessions scope short-lived memory, rewards, and tool usage. Agents can only read/write within their sessions, and policy guards live in schema, not ad-hoc middleware.
**Personalization is user-owned and revocable.** Users can opt into community overlays, scope agent access, and remove scoped contributions from future ranking.
**Correctness over cleverness.** Ranking is already approximate by nature. The database does not need to be more clever than the signals it has. It needs to be fast, consistent, and operationally simple.
## Who This Is For
Engineering teams building any surface where content is ranked for a user — media libraries, social feeds, content discovery, search — who are currently operating a multi-system stack and paying the consistency, latency, and operational cost of the seams between those systems.
The target developer has domain data that fits the entity/signal/relationship model, has immediate use cases that need this in production, and values a single system with sharp opinions over a flexible system with unlimited configuration.
The target scale is platforms serving millions of users across diverse audiences — where "what's trending" means different things to different cohorts, and the ability to slice engagement signals by audience segment is not a nice-to-have but the core product question.
## The Name
**tidalDB** — the tide that surfaces the right content for the right person at the right time. Rising signals, ebbing decay, a natural rhythm of discovery.
*(The idea matters more than the label.)*
---
*This is a focused tool for a focused problem. It will do one thing and do it correctly.*

339
tidal/docs/content-strategy.md Normal file
View File

@ -0,0 +1,339 @@
# Content Strategy
Blog posts mapped to the tidalDB roadmap. Each entry identifies the moment worth writing about, the thesis that makes it shareable, and the type of post it demands.
The audience is engineers who have built or are currently maintaining recommendation and discovery systems -- the people running the 6-system stack this database replaces. They know what Kafka lag feels like at 3am. They know why cache invalidation bugs in the ranking pipeline are the ones that never get root-caused. They will smell marketing language from the first sentence. Respect that.
---
## Publishing Principles
**Write when something is true, not when something is scheduled.** A blog post published the day a milestone passes its UAT is credible. A blog post published before the code works is fiction.
**One insight per post.** The reader should leave with a single idea they did not have before. If the post contains two insights, it is two posts.
**Code proves claims.** Every technical assertion is backed by a code example or a benchmark number from the actual codebase. Not a prototype. Not a plan. The shipped code.
**The title is the thesis.** If the title does not work as a standalone sentence that makes an engineer stop scrolling, the post is not ready.
---
## Content Calendar
### Pre-Implementation (Now)
These posts can be written before the engine is feature-complete. They draw on the vision, architecture research, and the problem space -- not on shipped code.
#### Post 1: "Every content platform builds the same 6 systems from scratch" [PUBLISHED]
- **Type:** Vision / Problem Statement
- **Thesis:** The Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service stack is not an architecture. It is scar tissue. The seams between these systems are where correctness dies.
- **Source material:** VISION.md, thoughts.md (Part VI)
- **When to publish:** Any time. This post defines the problem and does not depend on implementation progress.
- **Why it matters:** This is the foundational narrative. Every subsequent post assumes the reader understands this problem. It also serves as the litmus test for whether the audience cares -- if this post does not resonate, the subsequent ones will not either.
- **Structure:** Problem statement. The 6 systems named and indicted. The seams enumerated (stale signals, ETL lag, cache invalidation, operational burden). The thesis: ranking is not a feature, it is a primitive. End with the one-query vision, not with a product pitch.
---
### Milestone 1: Signal Engine
M1 proves that temporal signals with O(1) decay, velocity, and windowed aggregation work as a database primitive. This is the most technically interesting milestone for blog content because the math is elegant and the performance numbers are dramatic.
#### Post 2: "Running decay scores are O(1) -- here is the math" [PUBLISHED]
- **Type:** Technical Deep Dive
- **Roadmap phase:** m1p4 (Signal Ledger) completion
- **Thesis:** The forward-decay formula `S(t) = S(t_prev) * exp(-lambda * dt) + weight` eliminates raw-event scanning at query time. Three `exp()` calls on write, one on read. 15 nanoseconds per entity. Every platform computing `trending_score = views / (age + 2)^1.8` in application code is doing O(N) work that should be O(1).
- **Source material:** docs/research/tidaldb_signal_ledger.md, ARCHITECTURE.md (Signal System section), m1p4 task docs
- **When to publish:** After m1p4 passes UAT with benchmark numbers in hand.
- **Code to include:** The `EntitySignalState` struct. The forward-decay write path. The out-of-order event correction. Benchmark output showing 200-entity scoring pass under 5 microseconds.
- **Why it matters:** This is the post that demonstrates tidalDB is not vaporware. The math is verifiable. The benchmarks are reproducible. Engineers who have implemented trending scores in Redis will immediately understand the value.
#### Post 3: "What three databases taught us before we wrote a line of code" [PUBLISHED]
- **Type:** Architecture Decision Record
- **Roadmap phase:** m1p1-m1p3 completion (the foundation phases)
- **Thesis:** We studied Engram (cognitive memory), Citadel (append-only logging), and StemeDB (knowledge graph) -- three purpose-built databases in the same codebase -- and stole their best patterns. WAL-first durability from Citadel. Cache-line aligned hot structs from Engram. Subject-prefix key encoding from StemeDB. Background materialization from StemeDB. Here is what converged and what the gaps taught us.
- **Source material:** thoughts.md (all six parts), CODING_GUIDELINES.md
- **When to publish:** After m1p3 (Storage Engine) is complete. The patterns referenced are already implemented.
- **Code to include:** Key encoding format. Cache-line aligned struct. Group commit writer. Side-by-side comparison of the pattern in the source database and in tidalDB.
- **Why it matters:** Engineers respect builders who study prior art. This post establishes technical credibility and shows the architectural foundation is grounded in real patterns, not invented from scratch.
#### Post 4: "Signals wrote 100ms ago. The query sees them now." [PUBLISHED]
- **Type:** Devlog / Milestone Announcement
- **Roadmap phase:** m1p5 (Entity CRUD and Signal Write API) -- M1 complete
- **Thesis:** Milestone 1 is done. A developer can open a tidalDB instance, define signal types with decay rates and windows, write 10,000 engagement events, and read back decay-correct scores that match analytical computation to 6 decimal places. Including after a crash. The UAT scenario passes.
- **Source material:** The m1p5 integration test, benchmark results, git log for the M1 period
- **When to publish:** The day M1 UAT passes.
- **Code to include:** The full UAT scenario (or a clean excerpt). `TidalDB::open()` with schema. Signal write. Decay score read. Before/after crash recovery.
- **Why it matters:** This is the first "it works" post. It converts skeptics from "interesting idea" to "this is real." The UAT code is the proof.
---
### Milestone 2: Ranked Retrieval
M2 proves that a single query can retrieve, filter, score, and enforce diversity over live signals. This is where tidalDB stops being a signal engine and starts being a database.
#### Post 5: "One query. Six systems. Under 50 milliseconds." [PUBLISHED]
- **Type:** Technical Deep Dive / Announcement
- **Roadmap phase:** m2p5 (RETRIEVE Query Executor) -- M2 complete
- **Thesis:** `RETRIEVE items USING PROFILE trending DIVERSITY max_per_creator:1 LIMIT 25` executes in under 50ms on 10K items. It retrieves candidates via ANN, filters by metadata, scores using live decay signals and velocity, enforces diversity, and returns a ranked list. That is what Elasticsearch + Redis + a ranking service produce. It is one query here.
- **Source material:** m2p5 integration test, benchmark results, the dependency DAG showing how all M2 phases compose
- **When to publish:** After M2 UAT passes.
- **Code to include:** The RETRIEVE query. The ranked result with signal snapshots. The trending profile definition. A before/after signal burst showing the ranking change.
- **Why it matters:** This is the money post. The one-query thesis is no longer a vision document -- it is a benchmark. Engineers who operate the 6-system stack will immediately understand what this eliminates.
#### Post 6: "Diversity enforcement in 3 microseconds" [PUBLISHED]
- **Type:** Technical Deep Dive
- **Roadmap phase:** m2p4 (Diversity Enforcement)
- **Thesis:** "No more than 2 items per creator" does not belong in your API layer. It belongs in the query. tidalDB enforces diversity as a post-scoring reordering pass -- it does not reduce result count. The greedy selection algorithm runs in under 3 microseconds for 200 candidates.
- **Source material:** m2p4 task docs, VISION.md (diversity section), benchmark results
- **When to publish:** After m2p4 is complete.
- **Code to include:** The DiversitySpec. The greedy selector. A concrete example showing reordering (creator A dominates pre-diversity, balanced post-diversity). Benchmark numbers.
- **Why it matters:** Every team building a feed implements diversity in the API layer. Showing that it belongs in the database -- and costs 3 microseconds -- is a strong differentiator. This is the kind of post that gets shared in Slack channels.
#### Post 7: "Ranking profiles are data, not code" [PUBLISHED]
- **Type:** Architecture Decision Record
- **Roadmap phase:** m2p3 (Ranking Profile Engine)
- **Thesis:** Changing how content is ranked should not require a code change, a deployment, or a restart. tidalDB treats ranking profiles as versioned schema declarations. Define a profile. Name it. Swap it at query time. A/B test two profiles by name. The database executes the entire pipeline.
- **Source material:** m2p3 task docs, API.md (ranking profiles section), VISION.md
- **When to publish:** After m2p3 is complete.
- **Code to include:** A `trending` profile definition. A `for_you` profile definition. The same RETRIEVE query with two different profile names producing different orderings. The profile versioning API.
- **Why it matters:** This reframes ranking as a database concern. Engineers who maintain ranking services as separate microservices will recognize the operational simplification.
---
### Milestone 3: Personalized Ranking
M3 is where the feedback loop closes. Signal writes update the user's preference vector, the creator's interaction weight, and the item's signal ledger -- atomically, in one write. The "For You" query works.
#### Post 8: "The feedback loop that closes in one write" [PUBLISHED]
- **Type:** Technical Deep Dive
- **Roadmap phase:** m3p2 (Feedback Loop) completion
- **Thesis:** When a user likes an item, the database atomically updates: the item's like count, the user-to-creator interaction weight, and the user's preference vector (shifted toward the item's embedding). One `db.signal("like", ...)` call. No Kafka consumer to lag. No feature store to sync. No cache to invalidate. The next ranking query -- even 100ms later -- reflects the change.
- **Source material:** m3p2 task docs, ARCHITECTURE.md (Write Path section), SEQUENCE.md
- **When to publish:** After m3p2 passes UAT.
- **Code to include:** The signal write. The 10-step atomic update path. A before/after query showing the preference shift. The property test that proves hidden items and blocked creators never surface.
- **Why it matters:** The closed feedback loop is the core architectural thesis of tidalDB. This post proves it works. It is the strongest argument against the 6-system stack, because the stack's primary failure mode is feedback lag.
#### Post 9: "Negative signals are equal citizens" [PUBLISHED]
- **Type:** Architecture Decision Record
- **Roadmap phase:** m3p2 (Feedback Loop)
- **Thesis:** A skip is not the absence of a like. It is data. tidalDB treats negative signals -- skips, hides, blocks, "not interested" -- with the same precision and immediacy as positive signals. A skip within 3 seconds is a strong quality signal. A hide creates a permanent exclusion. A block removes all of a creator's content from all future queries. These are not afterthoughts. They are first-class signal types with their own decay rates, velocity, and ranking weight.
- **Source material:** VISION.md (negative signals section), USE_CASES.md (UC-01 feedback), m3p2 task docs
- **When to publish:** After m3p2 is complete. Can be bundled with or separated from Post 8.
- **Code to include:** Signal type definitions for skip, hide, block. The penalty clause in a ranking profile. The property test: 10,000 random signal sequences never produce a result where a hidden item or blocked creator appears.
- **Why it matters:** Most recommendation systems handle negative feedback as an afterthought -- a manual "not interested" button that writes to a separate blocklist. tidalDB's approach is architecturally different and engineers building these systems will recognize the improvement immediately.
#### Post 10: "Cold start without application logic" [PUBLISHED]
- **Type:** Technical Deep Dive
- **Roadmap phase:** m3p3 (Personalized Ranking Profiles)
- **Thesis:** New items with no signals get an exploration budget. New users with no history get a sensible default from population-level signals. The application does not manage either. The exploration rate decays as signals accumulate. This is declared per ranking profile, not implemented in application code.
- **Source material:** m3p3 task docs, VISION.md (cold start section)
- **When to publish:** After m3p3 is complete.
- **Code to include:** The exploration budget in a profile definition. A new item appearing in a for_you feed despite having zero signals. The decay of exploration as signals arrive.
- **Why it matters:** Cold start is the problem everyone hacks around and no one solves cleanly. Showing a database-native solution is a strong differentiator.
---
### Milestone 5: Hybrid Search
M5 merges full-text search with semantic similarity and signal-ranked results. Search and retrieval become the same system.
#### Post 11: "Search and ranking are the same system" [PUBLISHED]
- **Type:** Technical Deep Dive / Architecture Preview
- **Roadmap phase:** Published during M4. Describes the architecture that M5 will complete.
- **Status:** PUBLISHED. Written as an architectural intent post -- explains the unified pipeline design, what is already built (RETRIEVE, USearch, ranking), and what remains (Tantivy, RRF fusion, SEARCH query). Does not claim SEARCH is shipped.
- **Thesis:** Text retrieval, vector retrieval, and signal-based ranking belong in one pipeline. The data model is already unified. Fusion is arithmetic. The RETRIEVE pipeline, the HNSW index, and the ranking profiles are all in place. Three pieces of wiring remain.
- **Source material:** Published post at `site/content/blog/search-and-ranking.mdx`
- **Why it matters:** Frames the M5 work for the audience before it ships. The architectural argument stands regardless of what's wired today.
#### Post 12: "Tantivy as a derived index, not a source of truth"
- **Type:** Architecture Decision Record
- **Roadmap phase:** m5p1 (Tantivy Integration)
- **Thesis:** The entity store is the source of truth. Tantivy is a materialized view. If the index is corrupted, it can be rebuilt from the entity store. Crash recovery replays from a stored sequence number. Consistency is DB-primary, not two-phase commit. This is simpler, deterministic, and the right model for an embedded database.
- **Source material:** docs/research/tantivy.md, m5p1 task docs (once written), ARCHITECTURE.md
- **When to publish:** After m5p1 is complete.
- **Code to include:** The outbox pattern. The crash recovery sequence number. The background indexer. The consistency model.
- **Why it matters:** This is a useful architectural pattern beyond tidalDB. Engineers building systems with derived indexes will find this directly applicable.
---
### Milestone 6: Full Surface Coverage
M6 completes all 14 use cases. The content here shifts from "how does the engine work" to "what can you build with it."
#### Post 13: "14 use cases, one query engine"
- **Type:** Devlog / Announcement
- **Roadmap phase:** M6 complete
- **Thesis:** For You feeds, trending, search, following, related content, notifications, hidden gems, controversial, live content, creator discovery, user library, cohort-scoped trending -- every surface a content platform needs, driven by the same query primitives. The application specifies profiles, filters, and context. The database executes ranking.
- **Source material:** USE_CASES.md, M6 UAT results
- **When to publish:** After M6 UAT passes.
- **Code to include:** A curated selection of 4-5 queries spanning different surfaces (for_you, trending, search, hidden_gems, cohort_trending). Each with a brief setup and result.
- **Why it matters:** This is the completeness post. It demonstrates that the database is not a toy or a prototype -- it handles the full surface area of a real content platform.
#### Post 14: "Cohort-scoped trending: what is hot for people like you"
- **Type:** Technical Deep Dive
- **Roadmap phase:** M6, cohort-scoped trending phase
- **Thesis:** "What's trending" means different things to different audiences. A 22-year-old in Tokyo and a 45-year-old in Texas see different trending pages -- not because of personalization (individual preference), but because different content is genuinely trending within their respective audience segments. tidalDB maintains per-cohort signal aggregation using RoaringBitmaps for O(1) membership testing and sparse fan-out for storage efficiency.
- **Source material:** USE_CASES.md (UC-15), ARCHITECTURE.md (Cohort-scoped aggregation), API.md (Cohort Definitions)
- **When to publish:** After cohort-scoped trending passes integration tests.
- **Code to include:** Cohort definition. Three-layer query (global trending, cohort trending, search within cohort trending). The fan-out write path. Storage cost analysis.
- **Why it matters:** Cohort-scoped trending is a differentiator. Most systems compute trending globally. Slicing by audience segment is a product feature that usually requires a separate analytics pipeline. tidalDB does it natively.
---
### Production Hardening
The final phase is about trust. The content shifts from "what it does" to "why you can trust it."
#### Post 15: "Kill it at any point. It comes back correct."
- **Type:** Technical Deep Dive
- **Roadmap phase:** Production hardening -- crash recovery hardening phase
- **Thesis:** We injected faults at every write-path stage. Recovery time is under 30 seconds at 1M items. WAL replay produces state identical to pre-crash. No phantom items, no lost signals, no inconsistent aggregates. The WAL is the source of truth. Everything else is derived state that can be rebuilt.
- **Source material:** Crash recovery test results, fault injection methodology, WAL implementation
- **When to publish:** After crash recovery hardening passes.
- **Code to include:** The crash simulation test. Recovery time measurements. The WAL checkpoint and replay sequence.
- **Why it matters:** Trust is the precondition for adoption. Engineers will not embed a database they cannot crash-test. This post is the trust credential.
#### Post 16: "Graceful degradation: less precise, never wrong"
- **Type:** Architecture Decision Record
- **Roadmap phase:** Production hardening -- graceful degradation phase
- **Thesis:** Under 3x overload, tidalDB does not return errors. It reduces candidate set size, uses coarser aggregates, skips diversity enforcement, and serves from materialized cache -- in that order. Results are less precise but never incorrect. The degradation order is documented and configurable.
- **Source material:** Graceful degradation task docs, ARCHITECTURE.md (Graceful degradation)
- **When to publish:** After graceful degradation is complete.
- **Code to include:** The degradation cascade. Load test results at 1x, 2x, 3x. Latency distribution at each level.
- **Why it matters:** This is how production systems should behave. Engineers who have been paged for "ranking service returned 500" will appreciate a system that degrades gracefully instead.
---
### Ongoing / Anytime
These posts are not tied to specific milestones. They can be written whenever the insight is clear.
#### "Why not SQL"
- **Type:** Architecture Decision Record
- **Status:** READY -- code shipped, decision documented
- **Thesis:** The custom query language exists because SQL cannot express ranking semantics without losing optimization opportunities. `FOR USER` means "load this user's preference vector and relationship graph." `USING PROFILE` means "apply this named scoring function." `DIVERSITY` means "enforce post-ranking constraints." These are not WHERE clauses.
- **Source material:** thoughts.md (Part II.4), VISION.md (query examples), API.md
- **Code to read:**
- `tidal/src/query/retrieve.rs` -- `RetrieveBuilder` showing FOR USER, USING PROFILE, DIVERSITY as typed builder methods, not string predicates
- `tidal/src/ranking/profile.rs` -- `RankingProfile` and `CandidateStrategy` showing how profiles express scoring intent that SQL GROUP BY cannot
- `tidal/src/query/executor.rs` -- dispatcher showing that FOR USER loads preference vector and relationship graph -- state SQL has no model for
- **When to publish:** Any time after M1. Best paired with M2 when the RETRIEVE query is functional. Both are complete.
#### "Why we chose fjall over RocksDB (for now)"
- **Type:** Architecture Decision Record
- **Status:** READY -- code shipped, decision documented
- **Thesis:** Pure Rust, `#![forbid(unsafe_code)]`, fast compile times, trait-abstracted for swap. fjall is not the fastest LSM-tree. It is the right one for an embeddable database built by a small team that values correctness over raw throughput, with a trait boundary that makes the decision reversible.
- **Source material:** thoughts.md (Part V.9), CODING_GUIDELINES.md
- **Code to read:**
- `tidal/src/storage/engine.rs` -- the `StorageEngine` trait (six methods, zero fjall imports -- this is the abstraction boundary)
- `tidal/src/storage/fjall.rs` -- `FjallBackend` implementing the trait; note the `fjall::Keyspace` is the only fjall type that crosses the boundary
- `tidal/src/storage/memory.rs` -- `InMemoryBackend` proving the trait is genuinely swappable (used in all tests)
- `tidal/Cargo.toml` -- fjall version pin, no `unsafe` in the crate's feature flags
- **When to publish:** Any time. Code has been shipped since m1p3.
#### "USearch, not from scratch"
- **Type:** Architecture Decision Record
- **Status:** READY -- code shipped, decision documented
- **Thesis:** Correct, high-performance, concurrent HNSW with SIMD distance computation is 6-12 months of dedicated work. We are not a vector database company. USearch runs in ScyllaDB, ClickHouse, and DuckDB. The FFI boundary is thin. Build what differentiates you. Borrow what does not.
- **Source material:** docs/research/ann_for_tidaldb.md, ARCHITECTURE.md (Vector Index)
- **Code to read:**
- `tidal/src/storage/vector/mod.rs` -- `VectorIndex` trait and the module comment explaining the design decisions (VectorId = u64, L2 squared, ef_search uniformity)
- `tidal/src/storage/vector/usearch_index.rs` -- `UsearchIndex` wrapping the USearch FFI; the wrapper is thin by design
- `tidal/src/storage/vector/planner.rs` -- `AdaptiveQueryPlanner` with four strategies (HNSW, in-graph filter, widened beam, pre-filter brute-force) -- this is tidalDB's contribution on top of the borrowed index
- `tidal/src/storage/vector/brute.rs` -- `BruteForceIndex` and `MockVectorIndex` proving the trait boundary is real
- **When to publish:** Any time. Code has been shipped since m2p1.
---
## Post Cadence
| Milestone | Posts | Approximate Pace | Status |
|-----------|-------|-----------------|--------|
| Pre-implementation | 1 | Publish when ready | PUBLISHED |
| M1 (Signal Engine) | 2-3 | One per phase completion | PUBLISHED |
| M2 (Ranked Retrieval) | 3 | One per major phase | PUBLISHED |
| M3 (Personalized Ranking) | 2-3 | One per key insight | PUBLISHED |
| M4 (Agent Session Layer) | 1 (Post 11, architectural preview) | Published during M4 | PUBLISHED |
| M5 (Hybrid Search) | 1 (Post 12) | After m5p1 ships | Blocked on M5 |
| M6 (Full Coverage) | 2 (Posts 13-14) | At milestone boundaries | Blocked on M6 |
| Production Hardening | 2 (Posts 15-16) | At milestone boundaries | Blocked on hardening phase |
| Ongoing / ADRs | 3 (fjall, SQL, USearch) | When the decision is fresh | READY |
**Target: 16-20 posts across the full roadmap.** Not more. Each one earns its place.
---
## What Not to Write
- Progress updates that are changelogs. ("We merged 47 PRs this month.") Nobody cares.
- Posts that announce intent without shipped code. ("We plan to build...") Ship first.
- Posts with titles that are labels. ("Q1 Update," "Phase 3 Complete.") The title is the thesis.
- Posts that explain what a concept is without showing why the reader should care. ("Windowed aggregation is...") Start with the problem.
- Posts that use "we're excited to announce." You are not excited. You are precise.
---
## Reference: Roadmap to Post Mapping
| Roadmap Phase | Post # | Title | Status |
|---------------|--------|-------|--------|
| Pre-implementation | 1 | Every content platform builds the same 6 systems from scratch | PUBLISHED |
| m1p1-m1p3 (Foundation) | 3 | What three databases taught us before we wrote a line of code | PUBLISHED |
| m1p4 (Signal Ledger) | 2 | Running decay scores are O(1) -- here is the math | PUBLISHED |
| m1p5 (M1 Complete) | 4 | Signals wrote 100ms ago. The query sees them now. | PUBLISHED |
| m2p3 (Ranking Profiles) | 7 | Ranking profiles are data, not code | PUBLISHED |
| m2p4 (Diversity) | 6 | Diversity enforcement in 3 microseconds | PUBLISHED |
| m2p5 (M2 Complete) | 5 | One query. Six systems. Under 50 milliseconds. | PUBLISHED |
| m3p2 (Feedback Loop) | 8, 9 | The feedback loop that closes in one write / Negative signals are equal citizens | PUBLISHED |
| m3p3 (Personalized Profiles) | 10 | Cold start without application logic | PUBLISHED |
| M4 Complete (Agent Session Layer) | 11 | Search and ranking are the same system | PUBLISHED |
| Any time | -- | Why we chose fjall over RocksDB (for now) | READY |
| Any time | -- | Why not SQL | READY |
| Any time | -- | USearch, not from scratch | READY |
| M5p1 (Tantivy Integration) | 12 | Tantivy as a derived index, not a source of truth | Blocked on M5p1 |
| M5 Complete | 13, 14 | 14 use cases, one query engine / Cohort-scoped trending | Blocked on M5 |
| m6p1 (Crash Recovery) | 15 | Kill it at any point. It comes back correct. | Blocked on M6 |
| m6p2 (Graceful Degradation) | 16 | Graceful degradation: less precise, never wrong | Blocked on M6 |
---
## Current Queue
**As of M4 complete (Agent Session Layer), posts 1-11 published.**
### Ready to write now
| Post | Status |
|------|--------|
| "Why we chose fjall over RocksDB (for now)" | READY -- m1p3 shipped, decision is documented |
| "Why not SQL" | READY -- any time after M1 |
| "USearch, not from scratch" | READY -- m2p1 shipped |
### Next milestone-gated posts
| Post | Blocked on |
|------|------------|
| Post 12: "Tantivy as a derived index, not a source of truth" | M5p1 (Tantivy integration) |
| Post 13: "14 use cases, one query engine" | M5 complete |
| Post 14: "Cohort-scoped trending" | M5 complete |
| Post 15: "Kill it at any point. It comes back correct." | M6p1 |
| Post 16: "Graceful degradation: less precise, never wrong" | M6p2 |

153
tidal/docs/ops/capacity-planning.md Normal file
View File

@ -0,0 +1,153 @@
# Capacity Planning
This document provides RAM, disk, and startup time estimates for tidalDB deployments. Use these tables to provision hardware before going to production.
All estimates assume a single-node deployment with default configuration (30-second checkpoint interval, f16 vector quantization, DashMap-based hot tier).
---
## RAM Capacity
tidalDB is an in-memory-first database. USearch HNSW indexes, the signal ledger hot tier, and Tantivy reader segments all reside in RAM during operation. There is no swap tolerance for USearch -- if the process is swapped, ANN query latency degrades from microseconds to seconds.
| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 signals) | Tantivy RAM | Total Estimate |
|------:|---------------:|------------:|-------------------------------:|------------:|---------------:|
| 100K | 128D | ~26 MB | ~110 MB | ~50 MB | ~200 MB |
| 100K | 768D | ~154 MB | ~110 MB | ~50 MB | ~320 MB |
| 100K | 1536D | ~307 MB | ~110 MB | ~50 MB | ~470 MB |
| 1M | 128D | ~256 MB | ~1.1 GB | ~200 MB | ~1.6 GB |
| 1M | 768D | ~1.5 GB | ~1.1 GB | ~200 MB | ~2.8 GB |
| 1M | 1536D | ~3.1 GB | ~1.1 GB | ~200 MB | ~4.4 GB |
| 10M | 128D | ~2.6 GB | ~11 GB | ~500 MB | ~14 GB |
| 10M | 768D | ~15 GB | ~11 GB | ~500 MB | ~27 GB |
| 10M | 1536D | ~31 GB | ~11 GB | ~500 MB | ~43 GB |
### Formulas
**USearch HNSW index:**
```
items * dims * 2 bytes (f16 quantization) * 1.2 (HNSW graph overhead)
```
The 20% graph overhead accounts for HNSW neighbor lists (M=16 default, two layers). Actual overhead varies with M and ef_construction parameters.
**Signal ledger hot tier:**
```
items * signal_count * ~1,088 bytes/entry
```
Each `(entity_id, signal_type_id)` entry in the DashMap holds the running decay score, windowed counters (BucketedCounter with minute and hour buckets), velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure was measured in the m7p3 scale benchmarks.
The signal ledger has a memory budget of 5M entries (`DEFAULT_MAX_SIGNAL_ENTRIES`). When exceeded, the checkpoint thread evicts cold entries (oldest `last_update` timestamp). If your workload has more than 5M active `(entity, signal_type)` pairs, cold entries will be served from fjall checkpoints (slower, but correct).
**Tantivy text index:**
Tantivy's RAM usage depends on the number of indexed documents, average document length, and the number of open reader segments. The estimates above assume short metadata fields (title + description, ~200 bytes average). Long-form content indexing will increase RAM proportionally.
### Notes
- Signal ledger RAM is for the in-memory hot tier only. The WAL and fjall checkpoints add disk usage, not RAM.
- The "10 signals" column assumes 10 distinct signal types per entity. Scale linearly for more signal types.
- USearch RAM is the dominant cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), plan for USearch to consume 70%+ of total RAM at 10M items.
---
## Disk Capacity
Disk usage comes from three sources: fjall LSM-tree storage (metadata, relationships, signal checkpoints), WAL segments (append-only signal event log), and Tantivy/USearch index files.
| Items | Metadata Size | Signal Events/Day | Disk/Day (WAL) | Fjall (90 days) | Total (90 days) |
|------:|:----------------|------------------:|----------------:|----------------:|----------------:|
| 100K | small (256B avg) | 50K | ~2 MB | ~1 GB | ~1.2 GB |
| 1M | small | 500K | ~20 MB | ~10 GB | ~11.8 GB |
| 10M | small | 5M | ~200 MB | ~100 GB | ~118 GB |
### Formulas
**WAL daily growth:**
```
signal_events_per_day * ~40 bytes/event
```
Each WAL entry contains: 4-byte magic, 8-byte sequence, 1-byte event type, 8-byte entity ID, 2-byte signal type ID, 8-byte timestamp, 8-byte weight (f64), 32-byte BLAKE3 checksum. WAL segments are compacted after each successful checkpoint (every 30 seconds), so WAL disk usage represents only the uncompacted tail, not cumulative growth.
**Fjall storage:**
```
items * metadata_avg_bytes * 1.5 (LSM write amplification)
```
The 1.5x amplification factor accounts for LSM-tree space amplification (multiple sorted runs before compaction merges them). Actual amplification depends on the compaction strategy and write pattern. Signal checkpoints are also stored in fjall -- add ~100 bytes per active `(entity, signal_type)` pair for the serialized checkpoint data.
**Tantivy and USearch on disk:**
- Tantivy: roughly 1.5-2x the raw text size after indexing (inverted index + postings + term dictionary).
- USearch: saved index files are approximately the same size as the in-memory representation (items * dims * 2 bytes + graph metadata).
### WAL Compaction
WAL segments older than the last successful checkpoint are automatically deleted by the checkpoint thread (every 30 seconds). Under normal operation, WAL disk usage stays bounded at roughly `signal_rate * 40 bytes * 30 seconds`. Monitor `tidaldb_wal_lag_bytes` -- if it grows unbounded, checkpointing may be failing (check `tidaldb_checkpoint_failures_total`).
---
## Startup Time
Startup involves: opening fjall keyspaces, restoring the signal ledger from checkpoint, replaying WAL events since the last checkpoint, rebuilding in-memory indexes (bitmap, range, universe, creator-items, collections, suggestions), and loading USearch vector indexes.
| Items | Vectors | Typical Startup |
|------:|--------:|:----------------|
| 100K | 100K | ~2-5 sec |
| 1M | 1M | ~15-45 sec |
| 10M | 10M | ~3-8 min |
### Dominant Costs
1. **USearch index load** is the dominant startup cost at 1M+ vectors. USearch rebuilds the HNSW graph from its serialized format. Progress is logged every 10K vectors.
2. **Signal ledger restore** reads the checkpoint from fjall (a single prefix scan of `Tag::Sig` keys), then replays any WAL events with sequence numbers higher than the checkpoint's `wal_sequence`. Time is proportional to the number of active signal entries + unreplayed WAL events.
3. **Entity state rebuild** scans the items and users keyspaces to reconstruct creator-items bitmaps, relationship indexes (follows, blocks, hides), and interaction weights. Progress is logged every 10K items.
4. **Suggestion index rebuild** scans all item metadata for "title" fields and indexes terms for autocomplete. This is a sequential scan -- fast for 100K items, noticeable at 10M.
5. **Collection index rebuild** reconstructs collection membership bitmaps from fjall.
### Notes
- Startup time is I/O-bound, not CPU-bound. Fast NVMe storage reduces startup time significantly compared to spinning disk.
- WAL replay time depends on how many signals were written since the last checkpoint (at most ~30 seconds of writes under normal operation).
- Tantivy indexes are opened directly from disk (memory-mapped) and do not require a rebuild step.
---
## Recommended Provisioning
**General rule:** provision 2x the estimated RAM for headroom.
| Scale | Recommended RAM | Recommended Disk | CPU Cores |
|:---------|:----------------|:-----------------|:----------|
| 100K items, 128D | 512 MB | 5 GB SSD | 2 |
| 100K items, 768D | 1 GB | 5 GB SSD | 2 |
| 1M items, 128D | 4 GB | 25 GB SSD | 4 |
| 1M items, 768D | 8 GB | 25 GB SSD | 4 |
| 10M items, 128D | 32 GB | 250 GB NVMe | 8 |
| 10M items, 768D | 64 GB | 250 GB NVMe | 8 |
| 10M items, 1536D | 96 GB | 250 GB NVMe | 16 |
### Why 2x headroom?
- Signal ledger entries grow as new `(entity, signal_type)` pairs are written. The hot tier can hold up to 5M entries before trimming kicks in.
- Tantivy segment merges temporarily double the index size during merge operations.
- USearch does not support incremental resize -- if you approach capacity, you need enough free RAM to hold both the old and new index during a potential rebuild.
- The Rust allocator (jemalloc or system) has its own fragmentation overhead.
### Swap
Do not configure swap for production tidalDB instances. USearch HNSW traversal accesses memory in a random-access pattern that defeats page-level caching. A single swapped page in the HNSW graph can turn a 50-microsecond ANN query into a 50-millisecond disk seek.
### Disk Type
SSD is strongly recommended for all deployments. NVMe is recommended at 10M+ items. The WAL uses synchronous `fsync` on every segment rotation, and fjall's journal uses `persist(SyncAll)` during checkpoint. Spinning disk latency on these operations directly impacts signal write throughput.

523
tidal/docs/ops/grafana-dashboard.json Normal file
View File

@ -0,0 +1,523 @@
{
"uid": "tidaldb-overview",
"title": "tidalDB Overview",
"description": "Operational dashboard covering all 20 tidalDB metrics including retrieve and search latency histograms.",
"schemaVersion": 38,
"version": 2,
"refresh": "30s",
"time": { "from": "now-1h", "to": "now" },
"timepicker": {},
"tags": ["tidaldb"],
"panels": [
{
"id": 1,
"type": "row",
"title": "Health Overview",
"gridPos": { "x": 0, "y": 0, "w": 24, "h": 1 },
"collapsed": false
},
{
"id": 2,
"type": "stat",
"title": "Health",
"gridPos": { "x": 0, "y": 1, "w": 4, "h": 4 },
"targets": [
{
"expr": "tidaldb_health_ok",
"legendFormat": "health_ok"
}
],
"fieldConfig": {
"defaults": {
"mappings": [
{ "type": "value", "options": { "0": { "text": "UNHEALTHY", "color": "red" } } },
{ "type": "value", "options": { "1": { "text": "OK", "color": "green" } } }
],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": 0 },
{ "color": "green", "value": 1 }
]
},
"color": { "mode": "thresholds" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "background" }
},
{
"id": 3,
"type": "stat",
"title": "Uptime",
"gridPos": { "x": 4, "y": 1, "w": 4, "h": 4 },
"targets": [
{
"expr": "tidaldb_uptime_seconds",
"legendFormat": "uptime_seconds"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"color": { "mode": "fixed", "fixedColor": "blue" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "value" }
},
{
"id": 4,
"type": "stat",
"title": "Degradation Level",
"gridPos": { "x": 8, "y": 1, "w": 4, "h": 4 },
"targets": [
{
"expr": "tidaldb_degradation_level",
"legendFormat": "degradation_level"
}
],
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "red", "value": 1 }
]
},
"color": { "mode": "thresholds" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "background" }
},
{
"id": 5,
"type": "stat",
"title": "Version",
"gridPos": { "x": 12, "y": 1, "w": 4, "h": 4 },
"targets": [
{
"expr": "tidaldb_info",
"legendFormat": "{{version}}"
}
],
"fieldConfig": {
"defaults": {
"color": { "mode": "fixed", "fixedColor": "text" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "none", "textMode": "name" }
},
{
"id": 10,
"type": "row",
"title": "Signal Throughput",
"gridPos": { "x": 0, "y": 5, "w": 24, "h": 1 },
"collapsed": false
},
{
"id": 11,
"type": "timeseries",
"title": "Signal Write Rate (per second)",
"gridPos": { "x": 0, "y": 6, "w": 8, "h": 6 },
"targets": [
{
"expr": "rate(tidaldb_signal_writes_total[5m])",
"legendFormat": "writes/s"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 12,
"type": "timeseries",
"title": "Signal Write Latency (µs)",
"gridPos": { "x": 8, "y": 6, "w": 8, "h": 6 },
"targets": [
{
"expr": "tidaldb_signal_write_latency_us",
"legendFormat": "latency_us"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 13,
"type": "gauge",
"title": "Signal Hot Entries",
"gridPos": { "x": 16, "y": 6, "w": 8, "h": 6 },
"targets": [
{
"expr": "tidaldb_signal_hot_entries",
"legendFormat": "hot_entries"
}
],
"fieldConfig": {
"defaults": {
"min": 0,
"max": 5000000,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 4000000 },
{ "color": "red", "value": 5000000 }
]
},
"color": { "mode": "thresholds" }
}
}
},
{
"id": 14,
"type": "timeseries",
"title": "Retrieve Latency Percentiles (µs)",
"gridPos": { "x": 0, "y": 12, "w": 12, "h": 6 },
"targets": [
{
"expr": "histogram_quantile(0.50, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.95, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p95"
},
{
"expr": "histogram_quantile(0.99, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"color": { "mode": "palette-classic" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 500000 }
]
}
}
}
},
{
"id": 15,
"type": "timeseries",
"title": "Search Latency Percentiles (µs)",
"gridPos": { "x": 12, "y": 12, "w": 12, "h": 6 },
"targets": [
{
"expr": "histogram_quantile(0.50, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.95, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p95"
},
{
"expr": "histogram_quantile(0.99, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"color": { "mode": "palette-classic" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 1000000 }
]
}
}
}
},
{
"id": 20,
"type": "row",
"title": "Durability",
"gridPos": { "x": 0, "y": 19, "w": 24, "h": 1 },
"collapsed": false
},
{
"id": 21,
"type": "timeseries",
"title": "Checkpoint Age (seconds)",
"gridPos": { "x": 0, "y": 20, "w": 6, "h": 6 },
"targets": [
{
"expr": "tidaldb_checkpoint_age_seconds",
"legendFormat": "checkpoint_age_seconds"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "red", "value": 300 }
]
},
"color": { "mode": "thresholds" }
}
}
},
{
"id": 22,
"type": "stat",
"title": "Checkpoint Failures",
"gridPos": { "x": 6, "y": 20, "w": 6, "h": 6 },
"targets": [
{
"expr": "tidaldb_checkpoint_failures_total",
"legendFormat": "checkpoint_failures_total"
}
],
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "red", "value": 1 }
]
},
"color": { "mode": "thresholds" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "background" }
},
{
"id": 23,
"type": "timeseries",
"title": "WAL Lag (bytes)",
"gridPos": { "x": 12, "y": 20, "w": 6, "h": 6 },
"targets": [
{
"expr": "tidaldb_wal_lag_bytes",
"legendFormat": "wal_lag_bytes"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 500000000 },
{ "color": "red", "value": 1000000000 }
]
},
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 24,
"type": "timeseries",
"title": "WAL Compacted Segments (rate)",
"gridPos": { "x": 18, "y": 20, "w": 6, "h": 6 },
"targets": [
{
"expr": "rate(tidaldb_wal_compacted_segments_total[5m])",
"legendFormat": "compacted/s"
}
],
"fieldConfig": {
"defaults": {
"unit": "cps",
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 30,
"type": "row",
"title": "Index Health",
"gridPos": { "x": 0, "y": 26, "w": 24, "h": 1 },
"collapsed": false
},
{
"id": 31,
"type": "stat",
"title": "Tantivy Indexed Docs",
"gridPos": { "x": 0, "y": 27, "w": 4, "h": 4 },
"targets": [
{ "expr": "tidaldb_tantivy_indexed_docs", "legendFormat": "indexed_docs" }
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "fixed", "fixedColor": "blue" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
},
{
"id": 32,
"type": "gauge",
"title": "Tantivy Segment Count",
"gridPos": { "x": 4, "y": 27, "w": 4, "h": 4 },
"targets": [
{ "expr": "tidaldb_tantivy_segment_count", "legendFormat": "segment_count" }
],
"fieldConfig": {
"defaults": {
"min": 0,
"max": 50,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 20 },
{ "color": "red", "value": 30 }
]
},
"color": { "mode": "thresholds" }
}
}
},
{
"id": 33,
"type": "stat",
"title": "uSearch Vector Count",
"gridPos": { "x": 8, "y": 27, "w": 4, "h": 4 },
"targets": [
{ "expr": "tidaldb_usearch_vector_count", "legendFormat": "vector_count" }
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "fixed", "fixedColor": "blue" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
},
{
"id": 34,
"type": "stat",
"title": "uSearch Index Size",
"gridPos": { "x": 12, "y": 27, "w": 4, "h": 4 },
"targets": [
{ "expr": "tidaldb_usearch_index_size_bytes", "legendFormat": "index_size_bytes" }
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"color": { "mode": "fixed", "fixedColor": "blue" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
},
{
"id": 35,
"type": "stat",
"title": "Bitmap Index Cardinality",
"gridPos": { "x": 16, "y": 27, "w": 4, "h": 4 },
"targets": [
{ "expr": "tidaldb_bitmap_index_cardinality", "legendFormat": "bitmap_cardinality" }
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "fixed", "fixedColor": "blue" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
},
{
"id": 40,
"type": "row",
"title": "Sessions",
"gridPos": { "x": 0, "y": 31, "w": 24, "h": 1 },
"collapsed": false
},
{
"id": 41,
"type": "timeseries",
"title": "Active Sessions",
"gridPos": { "x": 0, "y": 32, "w": 6, "h": 6 },
"targets": [
{ "expr": "tidaldb_active_sessions", "legendFormat": "active_sessions" }
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 42,
"type": "timeseries",
"title": "Session Close Rate (per second)",
"gridPos": { "x": 6, "y": 32, "w": 6, "h": 6 },
"targets": [
{ "expr": "rate(tidaldb_closed_sessions_total[5m])", "legendFormat": "closes/s" }
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": { "mode": "palette-classic" }
}
}
},
{
"id": 43,
"type": "stat",
"title": "Auto-Closed Sessions",
"gridPos": { "x": 12, "y": 32, "w": 4, "h": 6 },
"targets": [
{ "expr": "tidaldb_session_auto_closed_total", "legendFormat": "auto_closed_total" }
],
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "fixed", "fixedColor": "yellow" }
}
},
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
},
{
"id": 44,
"type": "timeseries",
"title": "Rate Limited (per second)",
"gridPos": { "x": 16, "y": 32, "w": 8, "h": 6 },
"targets": [
{ "expr": "rate(tidaldb_rate_limited_total[5m])", "legendFormat": "rate_limited/s" }
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "red", "value": 100 }
]
},
"color": { "mode": "palette-classic" }
}
}
}
]
}

174
tidal/docs/ops/monitoring.md Normal file
View File

@ -0,0 +1,174 @@
# Monitoring
This document covers tidalDB's built-in Prometheus metrics endpoint, all exposed metrics, and recommended alerting thresholds.
---
## Setup
Enable the metrics HTTP server via the builder:
```rust
let db = TidalDb::builder()
.with_data_dir("/var/lib/tidaldb")
.with_schema(schema)
.enable_metrics("127.0.0.1:9090")
.open()?;
// Discover the bound address (useful when using port 0):
if let Some(addr) = db.metrics_addr() {
println!("metrics at http://{}/metrics", addr);
println!("health at http://{}/healthz", addr);
}
```
**Security:** The metrics endpoint has no authentication. Bind to `127.0.0.1` (loopback) only. If you need to scrape from a remote Prometheus server, use your infrastructure's network controls (SSH tunnel, reverse proxy with auth, or VPN) rather than binding to `0.0.0.0`. tidalDB logs a WARN-level message if you bind to a non-loopback address.
**Feature flag:** The metrics HTTP server requires the `metrics` feature, which is enabled by default. Build with `--no-default-features` to disable the HTTP server entirely. Base metrics (`uptime_seconds`, `health_ok`, `info`, `checkpoint_failures_total`) are always compiled regardless of the feature flag.
---
## Endpoints
| Path | Content-Type | Description |
|:-----|:-------------|:------------|
| `/metrics` | `text/plain` | Prometheus text exposition format |
| `/healthz` | `application/json` | JSON health check: `{"status":"ok","uptime_seconds":123.456,"version":"0.1.0","build_hash":"..."}` |
---
## Prometheus Scrape Configuration
```yaml
scrape_configs:
- job_name: 'tidaldb'
static_configs:
- targets: ['127.0.0.1:9090']
scrape_interval: 15s
```
---
## Metrics Reference
All metrics use the `tidaldb_` prefix. Metrics marked with "(feature-gated)" are only emitted when the `metrics` Cargo feature is enabled (default: enabled).
### Build and Health
| Metric | Type | Description | Labels |
|:-------|:-----|:------------|:-------|
| `tidaldb_uptime_seconds` | gauge | Seconds since the database was opened. Monotonically increasing. | `partition_id="0"` |
| `tidaldb_health_ok` | gauge | Whether the database is healthy. `1` = ok, `0` = degraded or closed. | `partition_id="0"` |
| `tidaldb_info` | gauge | Build and version information. Always `1`. | `version`, `build_hash`, `partition_id="0"` |
**Normal range for `tidaldb_health_ok`:** Always `1` during normal operation. Drops to `0` during shutdown or if an internal health check fails. Alert immediately if `0` during expected uptime.
### Signal System (feature-gated)
| Metric | Type | Unit | Description |
|:-------|:-----|:-----|:------------|
| `tidaldb_signal_writes_total` | counter | count | Total signal writes since database open. Includes all signal types across all entities. |
| `tidaldb_signal_hot_entries` | gauge | count | Number of entries currently in the signal ledger hot tier (DashMap). Each entry is one `(entity_id, signal_type_id)` pair. |
| `tidaldb_signal_write_latency_us` | histogram | microseconds | Signal write latency distribution. Bucket boundaries: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 10000 microseconds. |
**Normal range for `signal_hot_entries`:** Proportional to `active_entities * signal_types_per_entity`. The hot tier is trimmed at 5M entries (`DEFAULT_MAX_SIGNAL_ENTRIES`). Alert if approaching 80% of budget (4M entries).
**Normal range for `signal_write_latency_us`:** p50 should be < 50us, p99 should be < 1ms. If p99 exceeds 5ms, investigate WAL write latency or DashMap contention.
### WAL and Checkpoint (feature-gated)
| Metric | Type | Unit | Description |
|:-------|:-----|:-----|:------------|
| `tidaldb_wal_lag_bytes` | gauge | bytes | Total bytes of WAL segment files not yet compacted. Updated after each checkpoint cycle. |
| `tidaldb_wal_compacted_segments_total` | counter | count | Total WAL segments deleted by compaction since database open. |
| `tidaldb_checkpoint_age_seconds` | gauge | seconds | Seconds since the last successful signal checkpoint. Derived from `last_checkpoint_ns` at render time. |
| `tidaldb_checkpoint_failures_total` | counter | count | Total number of failed periodic signal checkpoints. **Not feature-gated** -- always emitted. |
**Normal range for `checkpoint_age_seconds`:** Should stay below 60 seconds (checkpoint runs every 30 seconds, with some jitter from the 500ms poll interval). Alert if > 300 seconds (5 minutes) -- the checkpoint thread may be stuck or the storage engine is under pressure.
**Normal range for `wal_lag_bytes`:** Depends on signal write rate. At 1K signals/sec, expect ~1.2 MB of WAL per 30-second checkpoint cycle. Alert if > 1 GB -- compaction may be failing.
**Normal range for `checkpoint_failures_total`:** Should be 0. Any non-zero value means signal durability is at risk -- the hot tier is not being persisted. Investigate storage errors (disk full, I/O errors).
### Index Health (feature-gated)
| Metric | Type | Unit | Description |
|:-------|:-----|:-----|:------------|
| `tidaldb_tantivy_segment_count` | gauge | count | Number of Tantivy index segments for the items text index. |
| `tidaldb_tantivy_indexed_docs` | gauge | count | Number of documents indexed in the items Tantivy text index. |
| `tidaldb_usearch_index_size_bytes` | gauge | bytes | Estimated total byte size of all USearch vector index files (f16). |
| `tidaldb_usearch_vector_count` | gauge | count | Number of vectors stored across all USearch indexes. |
| `tidaldb_bitmap_index_cardinality` | gauge | count | Total entity IDs across all four bitmap indexes (category + format + creator + tag). |
Index health metrics are refreshed every 10 seconds by the checkpoint thread (3x more frequently than checkpoints) so operators get near-real-time visibility.
**Normal range for `tantivy_segment_count`:** Should stay below 20 during normal operation. Tantivy merges segments in the background. If segment count grows unbounded, the text syncer thread may have stalled.
**Normal range for `usearch_vector_count`:** Should match the number of entities with embeddings written via `write_item_embedding()` or `write_creator_embedding()`.
### Session Lifecycle (feature-gated)
| Metric | Type | Unit | Description |
|:-------|:-----|:-----|:------------|
| `tidaldb_active_sessions` | gauge | count | Number of currently active agent sessions. |
| `tidaldb_closed_sessions_total` | counter | count | Total agent sessions closed (explicitly or by sweeper) since database open. |
| `tidaldb_session_auto_closed_total` | counter | count | Total sessions auto-closed by the TTL sweeper due to exceeding `max_session_duration`. |
**Normal range for `active_sessions`:** Depends on your application's agent concurrency. Each open session consumes memory for signal state tracking. Alert if this grows unbounded -- agents may be leaking sessions (opening without closing).
### Rate Limiting and Degradation (feature-gated)
| Metric | Type | Unit | Description |
|:-------|:-----|:-----|:------------|
| `tidaldb_rate_limited_total` | counter | count | Total signal write requests rejected due to per-agent rate limits since database open. |
| `tidaldb_degradation_level` | gauge | level | Current graceful degradation level. `0` = full quality, `1` = reduced candidates, `2` = coarse aggregates, `3` = no diversity enforcement. |
**Normal range for `degradation_level`:** Should be `0` during normal operation. Any value > 0 means the load detector has triggered degradation to protect latency. Investigate system load (CPU, memory pressure, I/O saturation).
---
## Recommended Alerts
| Alert Name | Condition | Severity | Meaning |
|:-----------|:----------|:---------|:--------|
| TidalDB Down | `tidaldb_health_ok == 0` | Critical | Database is unhealthy or shut down. Immediate investigation required. |
| Checkpoint Stale | `tidaldb_checkpoint_age_seconds > 300` | Warning | Checkpoint has not run in 5+ minutes. Signal durability at risk. Check storage I/O and disk space. |
| Checkpoint Failures | `tidaldb_checkpoint_failures_total > 0` | Warning | At least one checkpoint has failed. Signal state may not be durable. Check disk space and storage errors. |
| WAL Disk Pressure | `tidaldb_wal_lag_bytes > 1000000000` | Warning | WAL exceeds 1 GB uncompacted. Compaction may be stuck or checkpoint is failing. |
| Signal Backlog | `tidaldb_signal_hot_entries > 4000000` | Warning | Signal ledger over 80% of the 5M entry budget. Cold entry trimming will begin at 5M. |
| Degraded Ranking | `tidaldb_degradation_level > 0` | Warning | Load-based degradation is active. Ranking quality is reduced to protect latency. Scale up or reduce load. |
| Session Leak | `deriv(tidaldb_active_sessions[5m]) > 0.03 AND tidaldb_active_sessions > 100` | Warning | Active session count trending up. `tidaldb_active_sessions` is a gauge, so `deriv()` (slope) is correct — `rate()`/`increase()` are invalid on gauges and never fire. Agents may not be closing sessions. |
| High Rate Limiting | `rate(tidaldb_rate_limited_total[5m]) > 100` | Info | Sustained rate limiting. Review agent rate limit configuration or reduce write volume. |
| Tantivy Segment Bloat | `tidaldb_tantivy_segment_count > 30` | Warning | Tantivy has many unmerged segments. Text syncer may be stalled. |
### Grafana Dashboard Suggestions
**Row 1: Health overview**
- `tidaldb_health_ok` (stat panel, green/red)
- `tidaldb_uptime_seconds` (stat panel)
- `tidaldb_degradation_level` (stat panel, thresholds at 1/2/3)
- `tidaldb_info` labels (stat panel showing version + build hash)
**Row 2: Signal throughput**
- `rate(tidaldb_signal_writes_total[5m])` (time series, signals/sec)
- `tidaldb_signal_write_latency_us` histogram (heatmap or quantile panel)
- `tidaldb_signal_hot_entries` (gauge, threshold at 4M/5M)
**Row 3: Durability**
- `tidaldb_checkpoint_age_seconds` (time series, threshold line at 300)
- `tidaldb_checkpoint_failures_total` (stat panel, should be 0)
- `tidaldb_wal_lag_bytes` (time series)
- `rate(tidaldb_wal_compacted_segments_total[5m])` (time series)
**Row 4: Index health**
- `tidaldb_tantivy_indexed_docs` (stat panel)
- `tidaldb_tantivy_segment_count` (gauge)
- `tidaldb_usearch_vector_count` (stat panel)
- `tidaldb_usearch_index_size_bytes` (stat panel, bytes format)
- `tidaldb_bitmap_index_cardinality` (stat panel)
**Row 5: Sessions**
- `tidaldb_active_sessions` (time series)
- `rate(tidaldb_closed_sessions_total[5m])` (time series)
- `tidaldb_session_auto_closed_total` (stat panel)
- `rate(tidaldb_rate_limited_total[5m])` (time series)

131
tidal/docs/ops/prometheus-alerts.yaml Normal file
View File

@ -0,0 +1,131 @@
# =============================================================================
# DESIGN-REFERENCE RULE SET — NOT LOADED BY ANY ALERTMANAGER TODAY.
# =============================================================================
# This file is reference/design config. It is NOT wired into any running
# alerting pipeline:
# - vmalert (pilot/dev) loads ONLY ops/vmalert/rules/*.yaml
# - prod CRDs live ONLY in infra/k8s/prom/prometheus-rules/*.yaml
# Nothing mounts or imports tidal/docs/ops/prometheus-alerts.yaml.
# Do not read these as live pager rules.
#
# Provenance: every metric referenced below (tidaldb_health_ok,
# tidaldb_checkpoint_age_seconds, tidaldb_checkpoint_failures_total,
# tidaldb_wal_lag_bytes, tidaldb_signal_hot_entries, tidaldb_degradation_level,
# tidaldb_active_sessions, tidaldb_rate_limited_total,
# tidaldb_tantivy_segment_count, tidaldb_retrieve_latency_us_bucket,
# tidaldb_search_latency_us_bucket) IS really emitted today by
# tidal/src/db/metrics/mod.rs. So these rules are promotable — they
# are not orphaned against phantom metrics.
#
# Promotion path (the right long-term fix — do it deliberately, do NOT
# hand-copy these as live pager rules without the steps below):
# 1. Add a vmalert rule file: ops/vmalert/rules/tidaldb.yaml, translating
# each rule and stamping the canonical labels every routed rule carries —
# severity + capability + surface (+ optional component). Map per the
# canonical severity taxonomy in ops/alerting.md:
# labels {severity: critical} = pager class (TidalDBDown only here),
# {severity: warning} = non-paging / business-hours,
# {severity: info} = pure-FYI.
# Add capability: tidaldb and a surface label per rule, and a
# runbook_url annotation:
# https://github.com/orchard9/tidaldb/blob/main/tidal/docs/runbooks/<area>/<slug>.md
# 2. Add the prod CRD twin: infra/k8s/prom/prometheus-rules/tidaldb.yaml,
# kept byte-aligned in expr/threshold with the vmalert file.
# 3. Verify the alerts fire against real metrics (scrape tidaldb, force a
# degraded state) before declaring them on-call-ready.
# Until steps 1–3 land, this file stays a design reference only.
# =============================================================================
groups:
- name: tidaldb
interval: 30s
rules:
- alert: TidalDBDown
expr: tidaldb_health_ok == 0
for: 1m
labels: { severity: critical }
annotations:
summary: "tidalDB is unhealthy"
description: "tidaldb_health_ok is 0 — database is unhealthy or shut down."
- alert: TidalDBCheckpointStale
expr: tidaldb_checkpoint_age_seconds > 300
for: 2m
labels: { severity: warning }
annotations:
summary: "Signal checkpoint not running"
description: "{{ $value }}s since last checkpoint (threshold: 300s). Signal durability at risk."
- alert: TidalDBCheckpointFailures
expr: increase(tidaldb_checkpoint_failures_total[5m]) > 0
labels: { severity: warning }
annotations:
summary: "Signal checkpoint failures detected"
description: "Checkpoint failures in last 5m. Check disk space and storage errors."
- alert: TidalDBWALDiskPressure
expr: tidaldb_wal_lag_bytes > 1000000000
for: 5m
labels: { severity: warning }
annotations:
summary: "WAL disk usage exceeds 1GB"
description: "{{ $value | humanize1024 }}B of WAL uncompacted. Compaction may be stuck."
- alert: TidalDBSignalBacklog
expr: tidaldb_signal_hot_entries > 4000000
for: 5m
labels: { severity: warning }
annotations:
summary: "Signal ledger over 80% of capacity"
description: "{{ $value }} hot entries (threshold: 4M / 80% of 5M budget)."
- alert: TidalDBDegradedRanking
expr: tidaldb_degradation_level > 0
for: 2m
labels: { severity: warning }
annotations:
summary: "Ranking quality degraded"
description: "Degradation level {{ $value }} active. Scale up or reduce load."
- alert: TidalDBSessionLeak
# tidaldb_active_sessions is a GAUGE, so rate()/increase() are invalid
# (those operators only count counter resets). deriv() fits a least-squares
# slope over the window and is the gauge-correct way to detect sustained
# growth: > 0.03 sessions/sec is ~> 9 new sessions over a 5m window.
expr: deriv(tidaldb_active_sessions[5m]) > 0.03 and tidaldb_active_sessions > 100
for: 5m
labels: { severity: warning }
annotations:
summary: "Active session count growing steadily"
description: "{{ $value }} active sessions and trending up (deriv > 0.03/s). Agents may not be closing sessions."
- alert: TidalDBHighRateLimiting
expr: rate(tidaldb_rate_limited_total[5m]) > 100
for: 5m
labels: { severity: info }
annotations:
summary: "Sustained rate limiting"
description: "{{ $value }}/s rate-limited writes. Review agent rate limit config."
- alert: TidalDBTantivySegmentBloat
expr: tidaldb_tantivy_segment_count > 30
for: 10m
labels: { severity: warning }
annotations:
summary: "Tantivy segment count elevated"
description: "{{ $value }} segments (threshold: 30). Text syncer may be stalled."
- alert: TidalDBSlowRetrieve
expr: histogram_quantile(0.95, rate(tidaldb_retrieve_latency_us_bucket[5m])) > 500000
for: 5m
labels: { severity: warning }
annotations:
summary: "Retrieve p95 latency exceeds 500ms"
description: "p95 retrieve latency is {{ $value | humanizeDuration }}. Check signal ledger load and degradation level."
- alert: TidalDBSlowSearch
expr: histogram_quantile(0.95, rate(tidaldb_search_latency_us_bucket[5m])) > 1000000
for: 5m
labels: { severity: warning }
annotations:
summary: "Search p95 latency exceeds 1s"
description: "p95 search latency is {{ $value | humanizeDuration }}. Check Tantivy segment count and ANN index health."

189
tidal/docs/ops/recovery.md Normal file
View File

@ -0,0 +1,189 @@
# Recovery Guide
This document covers error scenarios, their causes, data at risk, and step-by-step recovery procedures for tidalDB.
---
## Error Scenarios
### 1. `StorageError::Corruption` on open
**Error message:** `storage error: data corruption: <details>`
**Cause:** fjall data files are corrupted. This can happen due to bit rot, partial writes from a hard power loss (no UPS), or physical disk failure. fjall uses checksums on its SST (sorted string table) files, so corruption is detected rather than silently returning wrong data.
**Data at risk:** All item metadata and relationships stored in the corrupt keyspace since the last backup.
**Recovery steps:**
1. **Identify which engine is corrupt.** tidalDB uses three fjall keyspaces: `items/`, `users/`, `creators/`. The error message will typically include the path or keyspace name. Check the log output for the specific directory.
2. **If you have a backup:**
- Stop the process.
- Replace the corrupt engine directory (e.g., `{data_dir}/items/`) with the corresponding directory from the backup.
- Reopen tidalDB. The WAL replay (`recover()`) will restore signal history from the backup's checkpoint timestamp forward. Any signals between the backup and the corruption event are preserved in WAL segments.
- Verify data integrity by running queries against known entities.
3. **If you have no backup:**
- Stop the process.
- Delete the corrupt engine directory (e.g., `rm -rf {data_dir}/items/`).
- Reopen tidalDB. It will start fresh for that engine -- all metadata and relationships in that keyspace are lost.
- Signal state in the WAL is independent of fjall and is recoverable. WAL replay will restore signal scores.
- You will need to re-ingest item metadata and embeddings from your upstream data source.
4. **If corruption is in the `items/` keyspace specifically:**
- Signal checkpoints (`Tag::Sig`) are stored in the items keyspace. Losing this keyspace means the signal ledger falls back to full WAL replay (slower startup but no data loss for signals).
- Collection definitions (`Tag::Collection`), cohort definitions (`Tag::CohortDef`), and co-engagement data are also in items. These will be lost.
### 2. `WalError::Corruption` on open
**Error message:** `WAL corruption: <details>` (surfaced as `TidalError::Durability`)
**Cause:** A WAL segment was partially written when the process crashed (e.g., SIGKILL during a signal write). The BLAKE3 checksum on the partial entry does not match, so the WAL reader detects corruption.
**Data at risk:** Signals written after the last successful WAL entry in the corrupt segment. In practice, this is at most one signal event (the one that was mid-write when the crash occurred).
**Automatic recovery:** tidalDB's crash recovery (`recover()`) automatically truncates the corrupt tail of the WAL and continues. The truncated entry is logged at WARN level. No manual action is needed unless `open()` continues to fail after the automatic recovery attempt.
**Manual recovery (if automatic recovery fails):**
1. List WAL segment files: `ls -la {data_dir}/wal/`
2. Identify the segment with the highest sequence number in its filename (e.g., `segment_000042.wal`).
3. Delete that single file: `rm {data_dir}/wal/segment_000042.wal`
4. Reopen tidalDB. It will replay from the remaining segments up to the last complete entry.
5. The signals in the deleted segment that were written after the last checkpoint are lost. Signals before the checkpoint are already materialized in fjall and are safe.
**Prevention:** tidalDB uses `fsync` on WAL segment rotation and BLAKE3 checksums on every entry. The only scenario where corruption occurs is a hard crash (SIGKILL, power loss) during the write of a single entry. Clean shutdowns (`db.close()`) always leave the WAL in a consistent state.
### 3. `TidalError::Config(DataDirLocked)` on open
**Error message:** `config error: data directory is already open by another process: <path>`
**Cause:** Another process has acquired the advisory lock on `{data_dir}/tidaldb.lock`. tidalDB uses file locking to prevent two processes from opening the same data directory simultaneously, which would cause data corruption.
**Data at risk:** None. This error is protective -- no data is accessed or modified.
**Recovery:**
1. Find the other process: `ps aux | grep <your_binary_name>` or `lsof {data_dir}/tidaldb.lock`
2. If the other process is a legitimate tidalDB instance, stop it cleanly (send SIGTERM and wait for graceful shutdown).
3. If the other process crashed and left a stale lock file:
- Verify no process is actually using the directory: `lsof +D {data_dir}`
- Delete the lock file: `rm {data_dir}/tidaldb.lock`
- Reopen tidalDB.
**Note:** The lock file (`tidaldb.lock`) is an advisory lock. Deleting it while another process is running will NOT prevent corruption -- the lock only works if both processes respect it. Always verify no process is running before deleting.
### 4. `TidalError::Schema(UnknownSignalType)` or schema fingerprint mismatch on open
**Error message:** `unknown signal type: '<name>'` or schema fingerprint mismatch
**Cause:** The application's schema definition has changed since the database was created. Signal decay parameters, signal names, or embedding slot dimensions differ from what was used when the data was written.
**Data at risk:** None. This is a protective error. No data is modified.
**Recovery options:**
1. **Revert the schema** to match the one used when the database was created. This is the safest option if the schema change was unintentional.
2. **Add the new signal type alongside the old one.** If you are adding a new signal (e.g., `"share"`), keep all existing signal definitions unchanged and add the new one. Existing signal data is unaffected. Note: this changes the schema fingerprint, so you may need to use a migration path when fingerprint validation is enforced.
3. **Start fresh.** Delete `{data_dir}` entirely and reopen with the new schema. All existing data is lost. Re-ingest from your upstream data source.
**What you must NOT do:** Change decay parameters (half_life) on an existing signal type and force the database open. The existing decay scores were computed with the old half_life. Applying a different decay rate to historical scores produces mathematically incorrect results. If you need a different decay rate, define a new signal type with the new parameters and let the old signal data age out naturally.
### 5. Disk full during operation
**Symptoms:** Signal writes return `TidalError::Durability(...)`. Metadata writes return `TidalError::Storage(StorageError::Io(...))`. Checkpoint thread logs errors. `tidaldb_checkpoint_failures_total` increments.
**Data at risk:** Signals written after the last successful checkpoint. In-memory state remains correct and readable -- queries continue to work against the hot tier.
**Recovery:**
1. Free disk space. Priorities:
- Delete old WAL segments that predate the last checkpoint. Check `{data_dir}/wal/` for segments with low sequence numbers. The checkpoint thread compacts these automatically, but if checkpointing itself failed due to disk pressure, compaction may be stuck.
- If you have a recent backup, you can safely delete all WAL segments and let the database start from the fjall checkpoint on next open.
- Clear temporary files, logs, or other non-tidalDB data from the volume.
2. Once disk space is available, signal writes resume automatically. The WAL writer thread retries on the next signal event. No restart is needed.
3. Verify recovery: check that `tidaldb_checkpoint_failures_total` stops incrementing and `tidaldb_checkpoint_age_seconds` returns to < 60 seconds.
**Prevention:** Monitor `tidaldb_wal_lag_bytes` and set alerts at 80% of your disk capacity. The WAL is the fastest-growing component. At 5M signals/day, the WAL grows ~200 MB/day before compaction.
### 6. `TidalError::Backpressure` during signal writes
**Error message:** `backpressure: WAL queue full, retry after <N>ms`
**Cause:** The WAL writer thread's channel is full. This means signal writes are arriving faster than the WAL can persist them to disk. This is NOT a data loss event -- the signal was never enqueued, so it can be safely retried.
**Recovery:** Retry the signal write after the suggested delay (`retry_after_ms`). If backpressure is sustained:
- Check disk I/O latency (WAL writes are `fsync`-bound).
- Check if another process is competing for disk bandwidth.
- Consider faster storage (NVMe).
### 7. `TidalError::RateLimited` during session signal writes
**Error message:** `rate limited: agent '<id>' at <limit> signals/sec, retry after <N>ms`
**Cause:** An agent has exceeded its configured rate limit for the current session. The signal was NOT written.
**Recovery:** Back off and retry after `retry_after_ms`. If rate limiting is too aggressive, adjust the agent's rate limit in the schema's `AgentPolicy` configuration.
---
## Safe Files to Delete
| File/Directory | Safe to delete? | Notes |
|:---------------|:----------------|:------|
| `tidaldb.lock` | Yes, if no process is running | Advisory lock file. Auto-recreated on next open. Verify with `lsof` first. |
| `wal/segment_*.wal` | Only segments with sequence numbers below the last checkpoint | Never delete the segment with the highest sequence number. To find the checkpoint sequence, check the last `tidaldb-checkpoint` log entry. |
| `items/` | **NO**, not without a backup | Primary fjall keyspace. Contains item metadata, signal checkpoints, collections, cohort definitions, co-engagement data. |
| `users/` | **NO**, not without a backup | Contains user relationship edges (follows, blocks, hides, interaction weights). |
| `creators/` | **NO**, not without a backup | Contains creator metadata and embeddings. |
| `text_index/` | Yes | Tantivy item text index. Rebuilt automatically from item metadata on next open. Rebuild cost is proportional to item count. |
| `creator_text_index/` | Yes | Tantivy creator text index. Same as above but for creators. |
| `cache/` | Yes | Temporary cache directory. Safe to delete at any time. |
---
## Backup and Restore
tidalDB's underlying storage engine (fjall 3.x) does not yet expose a native backup API ([fjall issue #52](https://github.com/fjall-rs/fjall/issues/52)). Until that ships, the recommended backup procedure is quiesce-and-copy.
### Creating a Backup
1. **Stop writes** to the database (either shut down the process or use an application-level write pause).
2. **Flush all state:**
- Call `db.close()` for a clean shutdown, which checkpoints the signal ledger, flushes fjall, and writes a WAL checkpoint marker. OR:
- If keeping the process running: flush text indexes, then wait for the next checkpoint cycle (30 seconds).
3. **Copy the entire data directory** recursively:
```
cp -R {data_dir} {backup_dest}
```
This copies: `items/`, `users/`, `creators/`, `wal/`, `text_index/`, `creator_text_index/`, and `tidaldb.lock`.
4. **Resume writes** (restart the process or unpause).
5. **Do NOT copy while the database is actively writing** without a quiesce step. Partial fjall SST files or WAL segments will produce corruption on restore.
### Restoring from Backup
1. Stop the current tidalDB process if running.
2. Delete or move the current `{data_dir}`.
3. Copy the backup into place: `cp -R {backup_source} {data_dir}`
4. Remove the stale lock file: `rm {data_dir}/tidaldb.lock`
5. Open tidalDB with the same schema. WAL replay will recover any signal events written between the backup's checkpoint and the end of the backup's WAL.
### What Survives a Crash Without Backup
| Component | Survives unclean shutdown? | Recovery mechanism |
|:----------|:--------------------------|:-------------------|
| Item metadata | Yes | Stored in fjall (durable) |
| Relationships | Yes | Stored in fjall (durable) |
| Signal decay scores | Yes (up to last checkpoint + WAL tail) | Checkpoint + WAL replay |
| Windowed counts | Approximately | Checkpoint stores state; WAL replay re-applies events |
| Active sessions | Yes | Session open/signal events in WAL are replayed; sessions restored as active |
| Bitmap/range indexes | Rebuilt on open | Scanned from fjall metadata |
| USearch vectors | Yes | Loaded from saved `.idx` files |
| Tantivy text index | Yes | Opened from on-disk segments |
| Collections | Yes | Rebuilt from fjall on open |
| Suggestions | Rebuilt on open | Scanned from fjall item metadata |

259
tidal/docs/personal-briefing-beachhead.md Normal file
View File

@ -0,0 +1,259 @@
# Use Case: Personal Briefing Feed (Knowledge Workers + Consumers)
**Date:** 2026-02-21
**Author:** @tidal-visionary
**Status:** Proposed beachhead
---
## 1. One-Line Definition
A daily and in-the-moment briefing feed that ranks what matters most for a person right now, adapts immediately to lightweight feedback, and explains why each item is shown.
---
## 2. The User (Not Developers)
### Primary Persona
**Information-overloaded decision maker**
- Works in product, strategy, operations, media, investing, policy, or as a highly engaged consumer.
- Consumes content to make decisions, not just to be entertained.
- Feels overwhelmed by tabs, newsletters, feeds, podcasts, and chatbots.
- Wants control over what appears (`more`, `less`, `hide`, `mute`) without heavy setup.
- Expects trust signals and source quality, not clickbait recirculation.
### Secondary Persona
**Curious consumer with intent**
- Follows multiple topics (career, health, finance, AI, hobbies).
- Wants a short, high-value briefing instead of infinite scroll.
- Will use the product if value appears on day 1 with minimal configuration.
---
## 3. User Job To Be Done
### Functional Job
"Help me understand what matters now in my domains without spending an hour hunting."
### Emotional Job
"Make me feel informed and in control, not behind and overwhelmed."
### Social Job
"Help me sound current and prepared in meetings and conversations."
---
## 4. Two-Sentence Hook
Every morning, get a briefing ranked for what actually matters to you right now, with clear "why this" reasons and no endless scrolling.
Tap `more`, `less`, or `hide` once, and the next refresh immediately adapts so your feed gets smarter in minutes, not weeks.
---
## 5. End-to-End User Experience
### 5.1 Day-0 to Day-1
1. User picks 5-10 interests, desired depth, and hard excludes.
2. User chooses time budget (`5 min`, `10 min`, `20 min`) and preferred formats.
3. Product delivers first `Today Brief` with 10-20 ranked items.
4. Each card shows a short explanation:
- "Trending in your cohort"
- "Matches your finance + AI priority"
- "New source for exploration"
5. User gives 3-5 feedback actions (`more`, `less`, `hide topic`, `mute source`, `save`).
6. Feed refresh shows immediate adaptation.
### 5.2 Daily Loop
1. Morning brief arrives via app/email/push.
2. User scans top cards quickly.
3. User opens selected cards for deeper summary/source context.
4. User gives lightweight feedback.
5. System updates ranking immediately for next retrieval.
6. Midday and evening updates are optional and scoped by time budget.
### 5.3 Session-Aware Interaction
1. User asks: "Only show items relevant to this week's strategy memo."
2. Session context constrains ranking while preserving global user profile.
3. Session expires or is closed; long-term profile remains stable.
---
## 6. Pressure Test (From the User Point of View)
### 6.1 Test Method
Pressure test assumes a skeptical user comparing against current habits:
- Existing feeds (X, LinkedIn, YouTube, Reddit, news apps)
- Newsletters and podcasts
- General AI assistants
The product only wins if the user can feel practical value within 1-3 sessions.
### 6.2 Core User Questions and Required Answers
| User Question | User Risk | Product Must Prove |
|---|---|---|
| "Why not just use my current feeds?" | No reason to switch | Better relevance + less noise + explicit control in one place |
| "Will this take too much setup?" | Onboarding drop-off | First useful briefing in < 3 minutes |
| "Can I trust this?" | Wrong or low-quality items | Clear source quality signals + transparent "why this" |
| "Will it trap me in a bubble?" | Repetitive narrow feed | Enforced diversity + exploration budget |
| "If I say less of this, does it actually change?" | Learned helplessness | Visible adaptation on next refresh |
| "Does it respect my time?" | Fatigue and churn | Time-budget mode with 5-minute high-value brief |
| "Can I safely use this for work decisions?" | Reputation risk | Freshness guarantees, quality gates, and easy source verification |
| "Is my data private?" | Trust barrier | Explicit controls for retention, session scope, and deletion |
### 6.3 Failure Modes That Kill Adoption
| Failure Mode | User Reaction | Severity |
|---|---|---|
| Feed still noisy after 2 days | "This is another feed app." | Critical |
| Feedback actions appear ignored | "I have no control." | Critical |
| Explanations are generic or fake | "This is smoke and mirrors." | High |
| One source dominates repeatedly | "Biased and boring." | High |
| Important updates are stale | "I cannot rely on this." | High |
| Too many notifications | "Annoying, uninstall." | Medium |
| Onboarding asks too much | "Not worth it." | Medium |
### 6.4 Pass Criteria (User-Perceived)
1. User can identify at least 3 briefing items as "actually useful" in first session.
2. User sees obvious feed adaptation after 3 feedback actions.
3. User can explain why each top card appeared using the provided reason labels.
4. User does not feel forced to scroll endlessly; time-budget mode feels real.
5. User returns on Day 2 without a reminder from support or onboarding prompts.
---
## 7. Why This Beachhead Fits tidalDB
This use case directly exercises tidalDB primitives without requiring a broad horizontal platform story on day 1:
- **Signals:** real-time positive and negative feedback.
- **Ranking profiles:** configurable briefing logic by context and time budget.
- **Diversity:** hard constraints to avoid source/topic over-concentration.
- **Cohorts:** "trending for people like me" layer.
- **Sessions:** short-lived task context (memo prep, market scan, exam prep).
- **Closed feedback loop:** next retrieval reflects feedback immediately.
This is materially different from search-first systems and generic chat assistants because ranking quality improves through continuous, explicit user control.
---
## 8. Product Requirements (User-First)
### 8.1 Must-Have V1 Capabilities
1. `Today Brief` ranking with clear reasons.
2. Lightweight controls: `more`, `less`, `hide topic`, `mute source`, `save`.
3. Immediate feedback reflection on next refresh.
4. Time-budget view (`5/10/20` minute mode).
5. Diversity constraints for source and topic spread.
6. Baseline cohort view: "trending for people like you."
7. Source transparency and one-tap source access.
### 8.2 Should-Have V1.5 Capabilities
1. Session-scoped task mode ("for this meeting only").
2. Morning/midday/evening briefing cadence controls.
3. Digest email + mobile app parity.
4. Credibility filters (verified sources, quality thresholds).
### 8.3 Explicit Non-Goals for Beachhead
1. Building a developer platform first.
2. Full social graph product.
3. Monetization optimization surfaces.
4. End-to-end enterprise admin suite in V1.
---
## 9. User-Facing Metrics
### 9.1 Activation
1. First briefing completion rate.
2. Median time to first "useful item saved."
3. Feedback action rate in first session.
### 9.2 Retention
1. D1, D7, D30 return rate.
2. Average sessions per active day.
3. Percentage of users using briefing at least 4 days per week.
### 9.3 Quality
1. "Useful item rate" per session.
2. Repeated-unwanted-item rate after negative feedback.
3. Diversity score by source/topic in top 10 results.
4. Freshness score for time-sensitive domains.
### 9.4 Trust
1. Explanation usefulness rating.
2. Source credibility acceptance rate.
3. Reported "feed felt biased/repetitive" rate.
---
## 10. Launch Plan (Beachhead Scope)
### Phase A: Concierge Pilot (20-50 users)
1. Target one segment: strategy/product/analyst professionals.
2. Run daily brief with strong manual QA on source quality and reasons.
3. Capture explicit user interview feedback after each week.
### Phase B: Productized Beta (200-500 users)
1. Self-serve onboarding under 3 minutes.
2. Reliable immediate feedback loop.
3. Basic cohort view and time-budget mode.
### Phase C: Scaled Consumer Entry
1. Multi-domain templates (finance, tech, health, creator economy).
2. Push/email cadence personalization.
3. Quality and trust controls as default UX, not advanced settings.
---
## 11. Strategic Risks and Mitigations
| Risk | Impact | Mitigation |
|---|---|---|
| Trying to solve too many surfaces at once | Slow execution, weak product feel | Ship one briefing surface first |
| Over-personalization creates bubble | Reduced discovery trust | Enforce exploration/diversity budgets |
| Weak source quality gates | Credibility collapse | Add quality floor and transparent sourcing |
| Slow adaptation to user feedback | Perceived irrelevance | Prioritize immediate write-to-read reflection |
| Too much AI summary, not enough evidence | Trust erosion | Keep source links and quote-backed rationale visible |
---
## 12. Kill Criteria (Be Honest Early)
Stop or pivot if any of the following remain true after two iteration cycles:
1. D7 retention remains below acceptable threshold for target segment.
2. Users do not perceive adaptation after direct feedback actions.
3. "Useful item rate" fails to outperform a simple baseline feed.
4. User interviews repeatedly describe the product as "another noisy feed."
---
## 13. Decision
This is the right forward-looking beachhead for tidalDB if the goal is knowledge workers and consumers rather than developers.
It is narrow enough to ship, painful enough to matter, and aligned with tidalDB's actual architectural advantage: real-time, feedback-aware ranking with explicit user control and transparent reasons.

217
tidal/docs/planning/PRODUCT_ROADMAP.md Normal file
View File

@ -0,0 +1,217 @@
# Product Roadmap: Personal Briefing Feed
**Date:** 2026-02-21
**Owner:** Product track (knowledge workers + consumers)
**Status:** Draft for execution
---
## Vision
Ship a daily briefing product that helps users identify what matters now, adapt the feed instantly with lightweight feedback, and trust the results enough to return habitually.
## Product Thesis
People do not need another infinite feed. They need a controllable, high-signal briefing that respects time, explains relevance, and improves immediately from feedback.
---
## Milestone Summary
| # | Name | Proves | Depends On |
|---|------|--------|------------|
| P0 | Beachhead Validation | Users care enough to return for a daily briefing | M0 + partial M1 |
| P1 | Concierge Alpha | Briefing usefulness and immediate adaptation in a narrow segment | M1 + partial M2 |
| PG1 | Personalization Core Done (Blocking Gate) | Core personalization loop is correct, immediate, and meaningfully better than baseline | P1 + M1/M2/M3 core slices |
| P2 | Productized Beta | Self-serve onboarding, trust UX, and repeatability without manual curation | M2 + partial M3 |
| P3 | Public Launch | Reliability, quality floor, and trust controls at public volume | M3 + M5 core + M6 partial |
| P4 | Scale + Revenue Fit | Sustainable growth and monetization without quality collapse | M6 + M7 |
---
## Current Status
| Phase | Status |
|-------|--------|
| p0: Beachhead Validation | NOT STARTED |
| p1: Concierge Alpha | NOT STARTED |
| pg1: Personalization Core Done gate | NOT STARTED |
| p2: Productized Beta | NOT STARTED |
| p3: Public Launch | NOT STARTED |
| p4: Scale + Revenue Fit | NOT STARTED |
**Current phase:** p0 (Beachhead Validation)
---
## Milestone P0: Beachhead Validation
### Milestone Thesis
Validate that a personal briefing feed solves a painful daily job for target users and creates early repeat usage.
### Done When
- 20-50 users recruited in target segment.
- Daily briefing prototype used for 2+ weeks.
- Median user gives at least one feedback action per session.
- Interview evidence confirms value over baseline feeds.
- D2 and D7 retention meet agreed threshold.
### Phase Plan
- Phase 1: Target Segment and Research Ops
- Phase 2: Concierge Briefing Loop
- Phase 3: Signal and Retention Evaluation
---
## Milestone P1: Concierge Alpha
### Milestone Thesis
Deliver a high-value daily `Today Brief` with transparent reasons and immediate feedback reflection for a narrow cohort.
### Done When
- Ranked briefing cards with source links and reason labels are live.
- `more/less/hide/mute/save` controls are available and used.
- Next refresh visibly reflects negative feedback.
- Time-budget mode (`5/10/20`) is used by pilot users.
- Weekly active usage indicates repeated utility.
### Phase Plan
- Phase 1: Briefing UX and Reason Labels
- Phase 2: Feedback Controls and Real-Time Adaptation
- Phase 3: Quality and Diversity Guardrails
---
## Milestone P2: Productized Beta
### Milestone Thesis
Turn concierge alpha into a self-serve product with stable onboarding and trust-preserving defaults.
### Done When
- Onboarding can be completed in under 3 minutes.
- Cohort layer (`trending for people like you`) is available.
- Explanations are clear and consistent per card.
- D7 retention and useful-item rate beat baseline.
- Manual curation is no longer required for normal operation.
### Blocking Prerequisite
P2 cannot start until **PG1 Personalization Core Done** passes.
### Phase Plan
- Phase 1: Self-Serve Onboarding and Profile Bootstrapping
- Phase 2: Cohort and Context Views
- Phase 3: Trust Controls and Preference Persistence
---
## Milestone P3: Public Launch
### Milestone Thesis
Launch publicly with reliability and trust controls suitable for broader consumer and knowledge-worker usage.
### Done When
- Reliability and latency SLOs for briefing generation are met.
- Freshness, duplicate suppression, and source quality floor are enforced.
- Notification cadence controls prevent fatigue.
- Product support and incident playbook are active.
### Phase Plan
- Phase 1: Reliability and SLO Enforcement
- Phase 2: Operational Quality Controls
- Phase 3: Launch Readiness and Support Ops
---
## Milestone P4: Scale + Revenue Fit
### Milestone Thesis
Scale the product and validate monetization while preserving user trust and briefing quality.
### Done When
- Monetization model validated.
- Revenue and quality metrics monitored together.
- Retention remains stable as volume increases.
- Next segment expansion is backed by product data.
### Phase Plan
- Phase 1: Monetization Experiments
- Phase 2: Quality-Protecting Growth
- Phase 3: Expansion Strategy
---
## Milestone PG1: Personalization Core Done (Blocking Gate)
### Milestone Thesis
Before expanding product breadth, prove the personalization loop is technically correct, immediately responsive, and materially useful to users.
### Must Pass (All Required)
- **Hard negatives are exact:** `hide/mute/block` items and creators never leak after write, restart, or replay.
- **Immediate adaptation is real:** `more/less/skip/save` actions change next-refresh ranking within target latency budget.
- **Replay correctness:** user personalization state (seen-state, preference shifts, relationship weights) rebuilds deterministically from checkpoint + WAL replay.
- **Quality uplift vs baseline:** useful-item rate and repeated-unwanted-item rate beat a non-personalized baseline feed.
- **Diversity with personalization:** relevance holds while source/topic domination stays within guardrails.
### Why This Gate Exists
Without this gate, roadmap execution drifts into breadth (more surfaces, more modes) before the core personalization promise is trustworthy.
---
## Product Metrics
### Activation
- First briefing completion rate
- Time to first useful item saved
- Feedback action rate in first session
### Retention
- D1/D7/D30 retention
- Sessions per active user per week
- Percentage of users with 4+ briefing days per week
### Quality and Trust
- Useful-item rate per briefing
- Repeated-unwanted-item rate after negative feedback
- Diversity score in top 10 items
- Explanation usefulness rating
---
## Dependencies on Engine Track
| Product Capability | Primary Engine Dependencies |
|--------------------|-----------------------------|
| Real-time adaptation from feedback | M1 Signal Engine, M3 Feedback Loop |
| Cohort briefing layer | M3 User model, M6 cohort surface support |
| Search-within-briefing | M5 Hybrid Search |
| Public reliability | M7 Production Hardening |
---
## Planning References
- Use case: `docs/personal-briefing-beachhead.md`
- Engine roadmap: `docs/planning/ROADMAP.md`
- Product milestone execution: `docs/planning/milestone-p/`

3244
tidal/docs/planning/ROADMAP.md Normal file
View File

File diff suppressed because it is too large Load Diff

309
tidal/docs/planning/architecture-review.md Normal file
View File

@ -0,0 +1,309 @@
# Architecture Review: The "Materialized Views Over Event Stream" Reframing
**Date:** 2026-02-20
**Author:** @tidal-visionary (Spencer Kimball)
**Status:** Assessment
---
## Context
After the product owner introduced five new requirements -- cohorts as a first-class primitive, three-layer trending, dynamic cohorts, rich user model, and query composition -- the engineering team produced 14 detailed specifications (01-14) and a revised roadmap (M1-M7). A subsequent architectural review proposed reframing the entire system around a single insight:
> The signal ledger IS a materialized view. Cohort-scoped signals are just more materialized views over the same event stream.
This reframing proposes collapsing 14 specs into 10 subsystems centered on a generalized `Materializer<Scope>` abstraction, where a single event stream feeds multiple materializers keyed by different scopes (Global, Cohort, User, Relationship), and a columnar event store retains events with full user context for GROUP BY operations.
This document assesses whether this reframing holds, how it changes the product, and what I would do differently.
---
## Question 1: Does the "Materialized Views Over Event Stream" Reframing Hold?
**Yes. But with an important qualification: the existing specs already embody this pattern. The reframing names something that is already true, not something that needs to change.**
Let me be precise about what the existing architecture says:
- **Spec 01 (Storage Engine):** "The WAL is the source of truth. Everything else is derived state." This is the event stream.
- **Spec 03 (Signal System), Section 3:** "Immutable events, mutable aggregates." This is the materialized view pattern.
- **Spec 03, Section 7:** The hierarchical dimensional rollup system already materializes the same event stream into Level 0 (global), Level 1 (region/language/age), and Level 2 (behavioral segments) views. Each level is a materialized view keyed by a different scope.
- **Spec 03, Section 9:** The background materializer already performs bucket rotation, rollup generation, checkpointing, and cohort segment recomputation -- exactly the responsibilities a generalized materializer framework would have.
- **Spec 10 (Feedback Loop), Section 2:** The seven-step signal ingestion pipeline shows a single event atomically updating the signal ledger (global view), user preference vector (user view), relationship weight (relationship view), cohort counters (cohort view), and user state (user-item view).
The proposed reframing says "these are all materialized views." The existing specs say "the WAL is truth, everything else is derived, and here is exactly how each derived state is updated." These are the same statement expressed in different vocabularies.
**Where the reframing adds genuine value:**
1. **It names the abstraction.** The existing specs describe five separate update paths (signal ledger, user preference, relationship, cohort, user state) without calling them instances of the same pattern. A `Materializer<Scope>` trait would make the shared structure explicit in code.
2. **It clarifies the extension model.** If someone asks "how do I add a new kind of derived state?" the answer with the reframing is clear: implement `Materializer<YourScope>` and register it with the event stream. Without the reframing, the answer is "find all the places in the seven-step pipeline where state is updated and add another step."
3. **It justifies the columnar event store.** The current specs store signal events with minimal context (item_id, user_id, signal_type, weight, timestamp, context blob). The reframing argues for storing events with full user attributes at write time, enabling retrospective cohort analysis without joining against user state. This is a genuine architectural addition.
**Where the reframing overstates its case:**
The proposal implies the existing architecture is organized wrong ("14 specs vs. 10 subsystems"). In fact, the existing architecture already has one event stream (WAL) feeding multiple derived state stores with different keys and different update logic. The specs are organized by domain concern (signals, entities, relationships, cohorts, text, vectors, queries, ranking, feedback, cold start, concurrency, schema, scale), not by storage topology. This is deliberate and correct -- domain organization is what engineers need when implementing and debugging. Storage topology is an implementation detail that emerges from the domain model.
**Verdict: The reframing is correct as an implementation insight. It should influence the trait design in Rust code. It should not drive a reorganization of the specification documents.**
---
## Question 2: Does the 10-Subsystem Decomposition Make More Sense Than 14 Specs?
**No. The 14 specs are the right organization. The 10-subsystem proposal conflates two different questions.**
The 14 specs answer: "What does each domain concept need to do, and what are its invariants?" The 10-subsystem proposal answers: "How should the code be organized at the module level?" These are different questions with different correct answers.
Here is what the 10-subsystem proposal merges:
| Proposed Subsystem | What It Combines | What Is Lost |
|---|---|---|
| Event Store | WAL (from spec 01) + signal events (from spec 03) | The WAL handles ALL mutations (entities, relationships, schema, signals, checkpoints). Calling it the "event store" and scoping it to signals misses that entity writes and relationship writes also go through the WAL. |
| Materializer Framework | Signal ledger (spec 03) + feedback loop (spec 10) + cohort attribution (spec 05) + background materializer (spec 03) | The feedback loop (spec 10) is not just materialization -- it defines the semantic mapping from signal types to preference vector directions, to relationship weight deltas, to user state transitions. These are domain rules, not materialization mechanics. |
| Entity Store | Entity model (spec 02) | Fine. |
| Cohort Engine | Cohorts (spec 05) | Fine. |
| Text Index | Text retrieval (spec 06) | Fine. |
| Vector Index | Vector retrieval (spec 07) | Fine. |
| Relationship Graph | Relationships (spec 04) | Fine. |
| Ranking Engine | Ranking (spec 09) + cold start (spec 12) | Reasonable merge. |
| Query Engine | Query engine (spec 08) | Fine. |
| Schema System | Schema (spec 11) | Fine. |
**What is lost entirely:**
- **Concurrency spec (13):** The lock-free hot-path design, atomic CAS patterns, memory ordering rationale, and the DashMap sharding strategy for concurrent entity state access. This is not part of any of the 10 proposed subsystems. It is a cross-cutting concern that the 14-spec approach correctly isolates.
- **Scale architecture spec (14):** The four-tier scaling model (Seed/Growth/Scale/Hyperscale), resource estimates, and the single-node ceiling analysis. This is a product strategy document, not a subsystem. It belongs in its own spec.
- **Cold start spec (12):** The exploration budget, the cold-start item injection strategy, and the new-user fallback to population-level signals. The proposal absorbs this into the ranking engine, which is defensible but loses the explicit treatment of a critical product behavior.
**The real issue with the proposal:** It is optimizing for code module count. CockroachDB has far more than 14 packages in its codebase. The question is not "can we reduce the number of specs?" but "does each spec have a coherent responsibility and clear invariants?" The answer for the existing 14 is yes.
**What I would actually do:** Keep the 14 specs as domain-level documentation. In the Rust codebase, organize modules around the materializer insight where it improves code structure. These are compatible. The spec is not the code. The spec is the contract. The code is an implementation that satisfies the contract.
**Specific code structure recommendation:**
```
tidal/src/
wal/ # Spec 01: WAL, segments, crash recovery
storage/ # Spec 01: fjall/redb backend, key encoding
entities/ # Spec 02: Item, User, Creator, embedding slots
signals/ # Spec 03: Signal types, decay, velocity, windowed agg
materializer.rs # The Materializer<Scope> trait lives here
global.rs # GlobalMaterializer (Level 0)
cohort.rs # CohortMaterializer (Levels 1-2)
user.rs # UserPreferenceMaterializer
relationship.rs # RelationshipWeightMaterializer
relationships/ # Spec 04: edges, weights, graph traversal
cohorts/ # Spec 05: predicates, bitmaps, resolution
text/ # Spec 06: Tantivy integration
vectors/ # Spec 07: USearch integration
query/ # Spec 08: parser, planner, executor
ranking/ # Spec 09 + 12: profiles, scoring, diversity, cold start
feedback/ # Spec 10: signal ingestion pipeline orchestration
schema/ # Spec 11: validation, migrations
```
This gives you the materializer abstraction where it matters (inside `signals/`) without reorganizing the domain model.
---
## Question 3: How Does This Change the Roadmap?
**It does not change the milestone order. It adds one phase to Milestone 1 and refines one phase in Milestone 4.**
The existing roadmap (from `docs/planning/ROADMAP.md` as amended by `roadmap-cohort-analysis.md`) already has the right milestone sequence:
- M1: Signal Engine
- M2: Ranked Retrieval
- M3: Personalized Ranking (expanded with rich user model)
- M4: Cohort-Scoped Ranking (new)
- M5: Hybrid Search (expanded with query composition)
- M6: Full Surface Coverage
- M7: Production Hardening
**What changes:**
### M1: Add m1p3a -- Materializer Trait
Insert a small phase between m1p3 (Storage Engine) and m1p4 (Signal Ledger):
**m1p3a: Materializer Trait**
- Defines `Materializer<Scope>` with `on_event(&self, event: &WalEvent) -> Result<()>` and `checkpoint(&self) -> Result<()>` and `restore(&self, checkpoint: &[u8]) -> Result<()>`
- Defines `Scope` enum: `Global`, `User`, `Cohort`, `Relationship`
- `GlobalSignalMaterializer` is the first implementation (used by m1p4)
- The materializer registry is created (initially holding one materializer)
- Complexity: S
This is the "design for distribution from the start" principle applied to the materializer pattern. Building the trait now costs almost nothing. Retrofitting it into m1p4's signal ledger later costs a refactor of every call site.
### M3: m3p2 Becomes a Materializer Implementation
m3p2 (Feedback Loop -- Signal Writes Update User State) is currently specified as a monolithic change to the signal write path. With the materializer insight, this phase implements two new materializers:
- `UserPreferenceMaterializer` (updates preference vector on positive/negative signals)
- `RelationshipWeightMaterializer` (updates user-creator interaction weights)
Both register with the materializer registry. The signal write path does not change -- it calls `registry.on_event()` and all registered materializers are invoked. This is cleaner than the current spec's seven-step pipeline, which hardcodes each update step.
### M4: m4p2 Becomes a Materializer Implementation
m4p2 (Cohort-Scoped Signal Aggregation) -- already identified as XL complexity and the highest-risk phase -- implements `CohortMaterializer`. This materializer receives signal events, resolves the user's cohort memberships, and increments the appropriate dimensional rollup counters.
The materializer trait boundary means m4p2 can be developed and tested in isolation: give it a stream of events with user context, verify it produces correct cohort-scoped counters. It does not need to understand the signal ledger internals or the WAL format -- it receives typed events and produces typed state.
### What Does NOT Change
- M1 UAT is identical. The materializer trait is invisible to the UAT scenario.
- M2 UAT is identical. The materializer trait does not affect query execution.
- M5-M7 are unchanged.
- The milestone order is unchanged.
- The complexity estimates are unchanged (the Materializer trait is S; the cohort materializer remains XL).
**The columnar event store question:**
The reframing proposes retaining signal events with full user context in a columnar format for GROUP BY operations. This is the most substantive architectural addition. Here is my assessment:
- **Defer to M4.** The columnar event store is only needed for retrospective cohort analysis ("recalculate trending for a cohort that was defined after the events occurred"). During M1-M3, signal events are stored in the WAL (which is the event stream) and the cold-tier signal_events CF (which has item_id, timestamp, signal_type, user_id, weight, context). This is sufficient.
- **In M4, add user attributes to the cold-tier event format.** When cohort tracking is activated for an item, the signal write path already looks up `UserCohortMemberships`. Storing these 22 bytes alongside the event in the cold tier enables retrospective analysis without a full columnar store.
- **A full columnar event store (Arrow/Parquet-style) is a post-M7 concern.** The use case is offline analytics, not real-time ranking. The real-time path uses pre-computed dimensional rollups. Adding a columnar engine before M7 violates the "does the UAT require it?" test.
---
## Question 4: What Does This Do to the Product Story?
**The reframing strengthens the product story, but not in the way the proposal suggests.**
The proposal suggests the story becomes "one event stream, multiple materialized views." This is an architecture story. Users do not care about materialized views. They care about what the database does for them.
The real product story is unchanged and already excellent:
> Write a signal event. The database instantly updates the item's trending score, the user's preference vector, the relationship weight, and the cohort-scoped trending metrics. The next query, issued 100ms later, reflects all of these updates. No ETL. No Kafka. No feature store sync. No stale data. One write, six updates, zero application logic.
The materialized view insight strengthens this story by making it extensible:
> And when you define a new cohort, the database starts materializing that cohort's trending signals from the existing event stream. No backfill job. No pipeline reconfiguration. Define the cohort. Query it. Done.
This is the "define a cohort and it works immediately" story, which is genuinely new and powerful. It comes from the materializer framework, but the story is about the user experience, not the implementation pattern.
**The competitive positioning:**
The three-layer trending model (global, cohort, search-within-cohort) is a capability that Algolia, Typesense, Meilisearch, and Elasticsearch cannot offer at all. They have no concept of cohort-scoped signal aggregation. This is the strongest differentiator tidalDB has, and the reframing does not change it -- the existing specs already define it in detail (spec 03 Section 7, spec 05 Section 6).
The product story on the website should emphasize: "Define a cohort. See what is trending for that audience. Search within those trends." The implementation -- materialized views, dimensional rollups, independence estimation -- stays behind the curtain.
---
## Question 5: What Is the Biggest Risk?
**The biggest risk is not architectural. It is scope.**
The 14 specs total approximately 40,000 words of detailed specification. They describe a system that, when fully implemented, handles 15 use cases across 25+ sort modes with 40 signal types, cohort-scoped trending, query composition, cold start, graceful degradation, and crash recovery at 1M items.
This is an enormous amount of functionality for a product that has zero lines of implementation code.
CockroachDB's first release (beta, 2015) was a KV store with Raft consensus and basic SQL parsing. It did not have window functions, JSON support, change data capture, or geographic partitioning. Those came over years of iteration informed by real usage.
**The risk with tidalDB's current trajectory is that the specifications are so detailed and so comprehensive that the team feels obligated to implement all of them before shipping anything.** The specs describe the end state. The roadmap describes the journey. But the specs' level of detail creates pressure to get everything right before writing code.
**Specific risk items, ranked:**
1. **m4p2 (Cohort-Scoped Signal Aggregation) at XL complexity.** This is the longest pole in the roadmap and blocks the most downstream work. The dimensional rollup system with threshold-gated activation, hierarchical Level 0/1/2/3 aggregation, independence estimation for composites, and write amplification management is genuinely hard. The spec (03, Section 7) runs to 3000+ words of detailed design. The risk is that implementation reveals edge cases the spec did not anticipate, and the cohort system ships 2-3 months later than planned.
2. **The warm tier memory model.** Spec 03 Section 3 calculates that the warm tier at full population (10M entities, 6 signal types, 1.8KB per entity per signal) would require 108 GB. The solution is sparse allocation (only active entities). But the active/inactive boundary, eviction policy, and promotion-on-demand strategy are complex to implement correctly under concurrent read/write load. Getting this wrong means either excessive memory consumption or cold-read latency spikes.
3. **The preference vector update.** Spec 10 Section 3 describes shifting a 1536-dimension preference vector on every positive/negative signal. The learning rate, normalization, and convergence properties of this approach are not well-studied for the tidalDB use case. If the preference vector drifts too fast, the For You feed becomes unstable. If it drifts too slowly, it does not reflect recent interests. This is a machine learning tuning problem disguised as a database implementation problem.
4. **Query composition performance.** The three-layer query (`SEARCH items QUERY "piano" WITHIN TRENDING FOR COHORT young_us_jazz WINDOW 24h`) has a 50ms latency budget. Spec 05 Section 6.3 breaks this into: cohort resolution (2ms) + candidate generation (20ms) + text search within candidates (10ms) + ranking (5ms) + diversity (1ms) = 38ms. This is tight. If any step exceeds its budget, the entire query misses the target.
5. **Spec-to-implementation drift.** With 14 specs and 7 milestones, the probability that implementation reveals a design flaw in one spec that forces changes in 2-3 others is high. The specs cross-reference each other extensively (spec 03 Section 7 references spec 05, spec 08 references specs 01-07, spec 10 references specs 01-04). A change in one spec's invariants can cascade.
**The materialized view reframing does not change these risks.** The risks are in the domain complexity, not the code organization.
---
## Question 6: What Would I Change?
### 1. Implement M1 Now. Stop Specifying.
The specs are good enough. They are detailed enough to build from. The marginal value of further specification is negative -- it delays the feedback loop between design and implementation. m1p1 (Core Type System) is S complexity. m1p2 (WAL) is L complexity. m1p3 (Storage Engine) is M complexity. Start writing Rust.
The most valuable thing that can happen right now is discovering, in the first 1000 lines of Rust code, which assumptions in the specs are wrong. This always happens. CockroachDB's first key-value store invalidated several assumptions in the design document. The sooner you find these, the cheaper the corrections.
### 2. Add the Materializer Trait in M1, Not M3
As described in Question 3. This is an S-complexity addition that prevents an M-complexity refactor later. The trait is:
```rust
pub trait Materializer: Send + Sync {
fn on_event(&self, event: &WalEvent) -> Result<()>;
fn checkpoint(&self, writer: &mut dyn Write) -> Result<()>;
fn restore(&self, reader: &mut dyn Read) -> Result<()>;
}
```
Three methods. Implement once for `GlobalSignalMaterializer` in M1. Add `UserPreferenceMaterializer` and `RelationshipWeightMaterializer` in M3. Add `CohortMaterializer` in M4. The trait boundary keeps each materializer testable in isolation.
### 3. Simplify the Warm Tier in M1
The warm tier spec (03, Section 3) describes per-minute and per-hour bucketed counters with SWAG stacks, EWMA smoothing, and Scotty stream-slicing. This is the correct end-state design, but it is too much for M1.
For M1, implement:
- Hot tier: running decay scores (cache-line aligned, atomic CAS). This is the core.
- Simple windowed counters: fixed-size circular buffer of per-minute counts. No SWAG stacks. No EWMA. No Scotty slicing. Just count events per minute, sum the last N minutes for an N-minute window.
This passes the M1 UAT. Windowed count will be exact. Velocity will be count/duration. The M1 integration test does not require EWMA or sub-minute granularity.
Upgrade to the full warm tier design in M2 or M3 when ranking profiles need it. The spec describes the end state. The implementation builds toward it incrementally.
### 4. Defer the Columnar Event Store Indefinitely
The reframing's most substantive proposal -- a columnar event store with full user context for GROUP BY -- is premature. Here is why:
- The real-time ranking path uses pre-computed dimensional rollups, not event scanning.
- Retrospective cohort analysis ("recalculate trending for a newly-defined cohort") can be served by scanning the cold-tier signal_events CF with a user_id join against current user attributes. This is slow (minutes, not milliseconds) but correct, and it is an admin/analytics operation, not a user-facing query.
- A columnar engine (Arrow, DataFusion, Polars) is a significant dependency with its own complexity. Adding it before there is a production workload that demands it is premature optimization of the analytics path at the expense of shipping the ranking path.
Store user cohort memberships (22 bytes) alongside signal events in the cold tier. This enables efficient retrospective filtering. A full columnar store can be added post-M7 when analytics requirements are concrete.
### 5. Reduce the Signal Type Count for M1-M2
Spec 03 Section 11 defines 40 signal types across 5 categories. For M1 and M2, implement 6: view, like, skip, share, completion, dwell_time. These cover the UAT scenarios for both milestones. The remaining 34 signal types add configuration complexity but no new code paths -- they use the same decay/velocity/windowed infrastructure.
Define the signal type registry so that adding new types is a schema operation, not a code change. Then add signal types as use cases demand them.
### 6. Keep 14 Specs, Add an Architecture Overview
The 14 specs are well-organized by domain concern. What is missing is a single document that shows how they connect -- the data flow from signal write to ranking query, touching all 14 specs in sequence. Add a document called `docs/specs/00-architecture-overview.md` that:
- Shows the single event stream (WAL) feeding multiple derived state stores
- Names the materialized view pattern explicitly
- Maps each spec to its role in the overall data flow
- Shows the dependency graph between specs
This gives the reader the forest before the trees. The 14 specs are the trees. Both are needed.
---
## Summary
| Question | Answer |
|----------|--------|
| Does the materialized view reframing hold? | Yes, but the existing specs already embody it. Name the pattern in code (Materializer trait), not in spec reorganization. |
| Is 10 subsystems better than 14 specs? | No. Keep 14 specs for domain documentation. Use the materializer insight for code structure within the signals module. |
| How does this change the roadmap? | Adds one S-complexity phase to M1 (Materializer trait). Refines M3 and M4 phases to use the trait. No milestone order changes. |
| What does this do to the product story? | Strengthens "define a cohort, see trends immediately." Does not change the core "replace 6 systems" narrative. |
| What is the biggest risk? | Scope. 14 detailed specs with zero implementation. The risk is specification paralysis, not architectural incorrectness. |
| What would I change? | Start implementing M1 immediately. Add Materializer trait in M1. Simplify the warm tier for M1. Defer columnar event store. Reduce initial signal types to 6. Add an architecture overview document. |
---
## The CockroachDB Parallel
When we built CockroachDB, the design document described Raft consensus, distributed SQL, range replication, and transaction isolation. The first thing we shipped was a monolithic key-value store that ran on one machine. It did not have Raft. It did not have distributed transactions. It had a RocksDB backend, a key-value API, and a test suite that proved the basics worked.
Every subsequent release added one layer from the design document. But -- and this is the critical point -- every layer was informed by what we learned building the previous one. The Raft implementation looked different from the design document because we discovered things about the key-value layer that the document did not anticipate. The SQL layer looked different because we discovered things about the Raft layer.
tidalDB is in the same position. The specs are the design document. They are good. They are detailed. They describe the right system. Now ship M1 and discover what the specs got wrong. The materialized view reframing is an insight that should live in the code, not in a specification reorganization. The 14 specs are the right documentation structure. The 7-milestone roadmap is the right delivery sequence.
The next step is not another architecture review. It is `cargo init`.

64
tidal/docs/planning/milestone-0/README.md Normal file
View File

@ -0,0 +1,64 @@
# Milestone 0 · Embeddable Runtime (Overview)
Milestone 0 proves that tidalDB can run entirely in-process with zero-configuration defaults, deterministic file layout, and tooling support. Engineers can `cargo add tidaldb`, open an ephemeral database, and run doctested samples before touching the harder ranking work.
## Objective
Ship the runtime shell that every later milestone depends on: builder/config API, sandboxed paths, WAL/metrics plumbing, CLI inspection, and living documentation.
## Deliverables
- `TidalDb::builder()` with `ephemeral()` and `single_process()` shortcuts, eager path validation, deterministic shutdown hooks, and temp-directory helpers used by doctests/integration tests.
- Tooling + diagnostics: `tidalctl` CLI (`status`, `paths`) and optional `/metrics` + `/healthz` endpoints that read the same files as the embedded process.
- Samples + docs: quick start code, doctests, Axum/Actix embedding examples, CONTRIBUTING checklist updates.
## Phase Breakdown
1. **Phase 1 — Embeddable Runtime Skeleton**
- Config + builder API with temp dirs, sandboxed layout, shutdown hooks.
- Deterministic path helper reused everywhere.
2. **Phase 2 — Tooling & Diagnostics**
- `tidalctl` surface plus metrics exporters sharing the same path helpers.
- Health probes expose WAL sequence, config hash, uptime.
3. **Phase 3 — Samples & Docs**
- Quick-start doctest stays green in CI.
- Embedding examples for Axum/Actix/CLI + CONTRIBUTING updates.
Each phase has task files under `docs/planning/milestone-0/phase-*`.
## Acceptance Criteria
- Builder defaults require zero manual config; paths validated up-front.
- Shutdown drains background workers and surfaces errors.
- Temp-directory helper cleans up automatically unless `preserve()` is called.
- `tidalctl status --path <dir>` prints WAL seq, storage layout, config snapshot.
- Metrics endpoint (optional) exposes uptime, WAL queue depth, build hash.
- Docs + samples compile via `cargo test --doc`; doctests fail CI if stale.
## Dependencies & Unblocks
- **Depends on:** None — foundational.
- **Unblocks:** Every milestone that needs storage paths, WAL writers, doctests, or CLI inspection (M1 onward).
## UAT Snapshot
```
let db = TidalDb::builder().ephemeral().with_temp_dir().open().unwrap();
db.health_check();
tidalctl status --path <dir>;
cargo test --doc;
```
All commands succeed without hand configuration.
## Research / Reference Notes
- `docs/research/tidaldb_signal_ledger.md` — running-score math informs future phases; builder must not block that work.
- `docs/research/tidaldb_wal.md` — dictates WAL layout and diagnostics exposed by tooling.
## Risks & Mitigations
- **Risk:** Temp directories linger on CI failures.
**Mitigation:** `TempTidalHome::drop` best-effort cleanup + `preserve()` flag for debugging.
- **Risk:** Tooling diverges from runtime paths.
**Mitigation:** `Paths` helper shared between runtime, CLI, and docs.

15
tidal/docs/planning/milestone-0/phase-1/OVERVIEW.md Normal file
View File

@ -0,0 +1,15 @@
# Milestone 0 · Phase 1 — Embeddable Runtime Skeleton
**Objective:** ship an ergonomic, zero-config builder so engineers can spin up a tidalDB instance inside tests or services without touching the signal stack yet.
**Success criteria**
- `TidalDb::builder()` exposes `ephemeral()` and `single_process(data_dir)` shortcuts with sensible defaults.
- `Config` validates eagerly (missing dirs, bad permissions) so failures happen before WAL threads spin up.
- Builder registers shutdown hooks (`Drop` + explicit `close()`), returning errors if background workers fail to drain.
- Temp-directory helper guarantees deterministic cleanup (used by doctests + integration tests).
**Dependencies:** none — runs before any signal-specific work.
**Blocked by:** n/a
**Unblocks:** M0 Phase 2 (CLI needs stable layout), all later milestones (test harnesses use the builder).

13
tidal/docs/planning/milestone-0/phase-1/task-01-builder-and-config.md Normal file
View File

@ -0,0 +1,13 @@
# Task 01 — Builder + Config API
**Goal:** expose a fluent API that hides all the knobs required to open a single-process tidalDB instance.
## Deliverables
- `TidalDbBuilder` with methods: `ephemeral()`, `with_data_dir(Path)`, `cache_dir(Path)`, `wal_dir(Path)`, `validate()` and `open()`.
- `Config` struct that can be serialized (for CLI) and implements `Default` tuned for embeddable use (single WAL segment, no background threads yet).
- Unit tests proving builder errors when directories are missing/unwritable.
## Acceptance Criteria
- `cargo test builder` passes on macOS + Linux runners.
- Builder docs contain runnable snippet used in Phase 3 doctests.
- No public API exposes fjall/useless internals; consumers only see `Config` + `TidalDbBuilder`.

13
tidal/docs/planning/milestone-0/phase-1/task-02-sandboxed-storage.md Normal file
View File

@ -0,0 +1,13 @@
# Task 02 — Sandboxed Storage Layout
**Goal:** deterministic filesystem layout for embedded instances so tooling/cleanup is reliable.
## Deliverables
- `Paths` helper that derives `{base}/wal`, `{base}/items`, `{base}/users`, etc., and ensures directories exist with correct perms.
- Temp-dir helper (`TempTidalHome`) for tests; implements Drop to delete directories unless `preserve=true`.
- Documentation table describing folder purpose for future CLI use.
## Acceptance Criteria
- Integration test proves two builders pointing to different `TempTidalHome` roots never collide.
- Cleanup confirmed via test that drops the helper and asserts directories removed.
- Paths helper reused by upcoming CLI (Phase 2) — single source of truth.

12
tidal/docs/planning/milestone-0/phase-2/OVERVIEW.md Normal file
View File

@ -0,0 +1,12 @@
# Milestone 0 · Phase 2 — Tooling & Diagnostics
**Objective:** give developers minimal introspection tooling that works even when tidalDB is embedded. This phase adds `tidalctl` plus a metrics endpoint so later milestones can reuse the same plumbing.
**Success criteria**
- `tidalctl status --path <dir>` prints build info, WAL seq, open segments, and builder config snapshot.
- Metrics exporter (text or JSON) exposes uptime, WAL queue depth, and build hash; tag everything with `partition_id=0` to future-proof multi-node rollouts.
- Tooling uses the same `Paths` helper from Phase 1 — no duplicated layout logic.
**Dependencies:** Phase 1 (stable directories + config serialization).
**Unblocks:** future performance debugging + automated tests that assert health via CLI before/after workloads.

443
tidal/docs/planning/milestone-0/phase-2/SCOPING.md Normal file
View File

@ -0,0 +1,443 @@
# Milestone 0, Phase 2: Tooling & Diagnostics -- Scoping Decisions
**Date:** 2026-02-20
**Author:** @tidal-visionary (Spencer Kimball)
**Status:** APPROVED -- ready for implementation
---
## Context
m0p1 (Embeddable Runtime Skeleton) is complete. m1p1-p3 (Type System, WAL, Storage Engine) are also complete. The codebase has:
- `TidalDb` as a thin handle: holds `Config`, has `health_check()`, `close()`, `Drop`
- A full WAL implementation (`WalHandle`, `SegmentWriter`, `CheckpointManager`) that writes segment files (`wal-{seq:020}.seg`) and checkpoint metadata (`checkpoint.meta`) to disk
- No `db.signal()` yet in the public API (deferred to m1p5)
- No WAL writes from the `TidalDb` public API -- the WAL is implemented but not wired to the `TidalDb` facade
- `Config` has no serde derive -- it is a plain struct with no serialization
- Single crate `tidal/`, no workspace
The task documents in `phase-2/` were written before m1p2 and m1p3 shipped. They assumed WAL writes would be accessible from the public API. They are not. This scoping document corrects the task definitions to match reality.
---
## 1. tidalctl Scope at M0
### What tidalctl Can Do
tidalctl is a **cold inspector**. There is no live process to connect to. The CLI reads files from disk and reports what it finds. This is the correct model for an embeddable database -- there is no server process listening on a port. The inspector reads the same files the embedded library writes.
### Commands
#### `tidalctl status --path <dir>`
Reads the tidalDB home directory and prints a JSON report:
```json
{
"version": "0.1.0",
"build_hash": "29400d4",
"status": "ok",
"storage_mode": "persistent",
"wal": {
"segments": 3,
"first_seq": 1,
"last_segment_seq": 201,
"checkpoint_seq": 150,
"checkpoint_ts": "2026-02-20T14:30:00Z",
"wal_dir_bytes": 49152
},
"dirs": {
"base": "/var/lib/tidaldb",
"wal": "/var/lib/tidaldb/wal",
"items": "/var/lib/tidaldb/items",
"users": "/var/lib/tidaldb/users",
"creators": "/var/lib/tidaldb/creators",
"cache": "/var/lib/tidaldb/cache"
}
}
```
**How each field is computed:**
| Field | Source | Notes |
|-------|--------|-------|
| `version` | Compiled into binary via `env!("CARGO_PKG_VERSION")` | Always available |
| `build_hash` | Compiled via `option_env!("GIT_HASH")` or build script | Falls back to `"unknown"` |
| `status` | `"ok"` if dir exists, has wal subdir, and at least one segment | `"empty"` if no WAL segments, `"error"` if dir missing |
| `storage_mode` | Inferred: if WAL dir exists with segments, `"persistent"` | No way to know `ephemeral` from disk -- ephemeral leaves no trace |
| `wal.segments` | `segment::list_segments(&wal_dir)?.len()` | Already implemented in `tidal/src/wal/segment.rs` |
| `wal.first_seq` | First element of `list_segments()` result | 0 if empty |
| `wal.last_segment_seq` | Last element of `list_segments()` result | 0 if empty |
| `wal.checkpoint_seq` | `CheckpointManager::read(&wal_dir)?` | null if no checkpoint file |
| `wal.checkpoint_ts` | Same -- the `ts` field, formatted as ISO 8601 | null if no checkpoint |
| `wal.wal_dir_bytes` | Sum of file sizes in WAL dir | Filesystem stat |
| `dirs.*` | `Paths::new(base)` expanded | Existence checked per dir |
**No config file is written.** tidalctl does not need `TidalDb::open()` to write a `.tidaldb.json` config snapshot. The CLI reports what it can observe on the filesystem. The config is a runtime concept -- it exists in memory while the process runs and is not persisted. This is correct for M0. If future milestones need a config file for operational tooling, that is a separate decision.
**No live process query.** tidalctl reads disk. It does not connect to a running process. No Unix socket, no HTTP, no PID file. This is the right model for an embeddable library.
#### `tidalctl paths --path <dir>`
Prints the resolved directory layout:
```json
{
"base": "/var/lib/tidaldb",
"wal": "/var/lib/tidaldb/wal",
"items": "/var/lib/tidaldb/items",
"users": "/var/lib/tidaldb/users",
"creators": "/var/lib/tidaldb/creators",
"cache": "/var/lib/tidaldb/cache",
"exists": {
"base": true,
"wal": true,
"items": true,
"users": false,
"creators": false,
"cache": false
}
}
```
This uses `Paths::new(dir)` -- the same path helper from m0p1. No duplication.
#### Common Flags
- `--path <dir>` (required): the tidalDB home directory
- `--pretty` (optional): pretty-print JSON output (default: compact)
- `--format json|text` (optional, default `json`): `text` prints human-friendly tabular output
### What tidalctl Does NOT Do at M0
- No `tidalctl init` (creating a fresh tidalDB home) -- the library creates dirs on open
- No `tidalctl repair` (WAL repair) -- crash recovery is automatic in `WalHandle::open()`
- No `tidalctl compact` (storage compaction) -- no compaction exists yet
- No `tidalctl dump` (WAL event dump) -- useful but not needed for the m0p2 UAT
- No live process communication of any kind
---
## 2. Metrics Scope at M0
### The Problem with the Original Task
The original task says: "Integration test hits `/metrics` and asserts counters increment when WAL appends."
At M0, the `TidalDb` public API has no WAL write path. `WalHandle::append()` exists but is not wired to `TidalDb`. There are no signal writes from the public API. A test that asserts "counters increment when WAL appends" cannot be written without either (a) using WAL internals directly or (b) waiting for m1p5.
### Corrected Scope
The metrics surface at M0 serves one purpose: **prove the plumbing works so later milestones can add counters without redesigning the metrics layer.** The counters themselves are scaffolding. The architecture is the deliverable.
### Endpoints
#### `GET /healthz`
```json
{
"status": "ok",
"uptime_seconds": 127.3,
"version": "0.1.0",
"build_hash": "29400d4"
}
```
#### `GET /metrics`
Prometheus text exposition format:
```
# HELP tidaldb_uptime_seconds Seconds since database opened.
# TYPE tidaldb_uptime_seconds gauge
tidaldb_uptime_seconds{partition_id="0"} 127.3
# HELP tidaldb_health_ok Whether the database is healthy. 1 = ok, 0 = degraded.
# TYPE tidaldb_health_ok gauge
tidaldb_health_ok{partition_id="0"} 1
# HELP tidaldb_info Build and version information.
# TYPE tidaldb_info gauge
tidaldb_info{version="0.1.0",build_hash="29400d4",partition_id="0"} 1
```
### Exact Counters at M0
| Counter | Type | Source | Note |
|---------|------|--------|------|
| `tidaldb_uptime_seconds` | Gauge | `Instant::now() - opened_at` | Computed on read |
| `tidaldb_health_ok` | Gauge | `health_check().is_ok() as u8` | 1 or 0 |
| `tidaldb_info` | Gauge (info-pattern) | Build constants | Static, always 1 |
That is the complete set. Three metrics. No WAL counters, no signal counters, no storage counters. Those arrive in m1p5 when the WAL is wired to the public API.
### What the Integration Test Verifies
The integration test at M0 verifies:
1. `TidalDb::builder().ephemeral().enable_metrics("127.0.0.1:0").open()` succeeds (port 0 = OS assigns)
2. `GET /healthz` returns 200 with `status: "ok"` and `uptime_seconds > 0`
3. `GET /metrics` returns 200 with valid Prometheus text format
4. `tidaldb_uptime_seconds` increases between two reads separated by a sleep
5. `tidaldb_health_ok` is 1
6. `db.close()` stops the metrics server cleanly (no leaked threads, no port still bound)
No WAL assertions. No signal assertions. The test proves the HTTP server starts, serves correct responses, and shuts down cleanly.
### What Is Deferred
| Counter | Deferred To | Why |
|---------|-------------|-----|
| `tidaldb_wal_seq` | m1p5 | WAL not wired to public API yet |
| `tidaldb_wal_segments` | m1p5 | Same |
| `tidaldb_wal_bytes_total` | m1p5 | Same |
| `tidaldb_signal_writes_total` | m1p5 | `db.signal()` does not exist yet |
| `tidaldb_signal_read_latency` | m1p5 | Signal reads do not exist yet |
| `tidaldb_query_latency` | m2p5 | Query executor does not exist yet |
| `tidaldb_query_count` | m2p5 | Same |
---
## 3. HTTP Approach: Sync (Option A)
**Chosen: (a) Sync HTTP via `tiny_http` in a background thread.**
Rationale:
1. **Minimal deps is an explicit tidalDB requirement.** Tokio is 200+ transitive dependencies. `tiny_http` is 5. For an embeddable library, dependency weight matters -- every dep is a compile-time cost and an audit surface for every user.
2. **The metrics endpoint does ~2 requests per scrape interval.** This is not a high-throughput server. A single-threaded sync HTTP listener on a background thread handles thousands of req/s. Prometheus scrapes every 15-30s. `tiny_http` handles this with zero contention.
3. **No Tokio runtime conflict.** If the host application uses Tokio (likely for an Axum/Actix service), embedding a second Tokio runtime inside tidalDB creates footguns: nested `block_on`, unexpected thread pools, panic behavior. A background `std::thread` with sync HTTP avoids all of this.
4. **The "Future implementor" spec is wrong for M0.** The original task assumed tidalDB would share the host's async runtime. That is a leaky abstraction. An embeddable library should not assume or require any particular async runtime. A background thread with sync HTTP is the correct primitive.
5. **Feature flag is premature.** Option (c) with feature flags adds compile-time complexity for a surface that serves 3 metrics. Ship sync now. If M7 (Production Hardening) needs async HTTP for high-frequency scraping, add it then. The internal `MetricsRegistry` / counter abstraction is the same either way -- only the HTTP transport changes.
### Implementation Shape
```rust
// Builder API
let db = TidalDb::builder()
.ephemeral()
.enable_metrics("127.0.0.1:9090") // Starts background thread
.open()?;
// Internal: spawns std::thread with tiny_http::Server
// Thread reads from Arc<MetricsState> (uptime, health_ok, build_info)
// Thread exits cleanly when TidalDb::close() sets a shutdown flag
```
### Dependency Addition
```toml
# In tidal/Cargo.toml, behind a feature flag:
[features]
metrics = ["dep:tiny_http"]
[dependencies]
tiny_http = { version = "0.12", optional = true }
```
The `metrics` feature is opt-in. Users who do not need the HTTP endpoint pay zero compile cost. The `MetricsState` struct (atomic counters) exists unconditionally -- only the HTTP server is gated.
---
## 4. Workspace Structure: Workspace with Separate Binary Crate
**Confirmed: workspace layout.**
### Structure
```
tidalDB/
Cargo.toml # [workspace] members = ["tidal", "tidalctl"]
tidal/
Cargo.toml # [package] name = "tidaldb" (the library)
src/
tidalctl/
Cargo.toml # [package] name = "tidalctl" (the binary)
src/
main.rs
```
### Why Workspace, Not `[[bin]]`
1. **Separate dependency trees.** tidalctl needs `clap` for argument parsing. The tidaldb library should not carry `clap` as a dependency -- embeddable libraries do not parse CLI arguments. A `[[bin]]` inside `tidal/` would either make `clap` unconditional or require a feature flag, both of which pollute the library.
2. **Independent versioning path.** tidalctl may version independently from tidaldb. The CLI is a companion tool, not part of the library API surface.
3. **`cargo install tidalctl` works naturally.** Users install the CLI separately from embedding the library. A workspace member with `[[bin]]` in its own crate gives `cargo install --path tidalctl` the right behavior.
4. **Shared dependencies via workspace.** `tidalctl` depends on `tidaldb` (for `Paths`, `WalConfig`, segment parsing, checkpoint reading). The workspace ensures they share the same compiled artifacts.
### tidalctl Dependencies
```toml
[package]
name = "tidalctl"
version = "0.1.0"
edition = "2024"
[dependencies]
tidaldb = { path = "../tidal" }
clap = { version = "4", features = ["derive"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
```
### What This Means for Pre-Commit Hooks and CI
The root `Cargo.toml` becomes the workspace root. All `cargo` commands (`fmt`, `clippy`, `test`) need to run from the workspace root or with `--workspace`. (Resolved: tidalDB is the `tidaldb` crate at `tidal/` in this workspace; commands target it with `-p tidaldb` from the workspace root, or `--manifest-path tidal/Cargo.toml`.)
---
## 5. Deferred Items
### Explicitly NOT in m0p2
| Item | Why Deferred | Arrives In |
|------|-------------|------------|
| Config serialization to disk (`.tidaldb.json`) | tidalctl inspects filesystem artifacts, not config files. Config is a runtime concept. | Revisit in M7 if operational tooling needs it |
| `tidalctl init` command | Library creates dirs on open. A separate init command is redundant. | Possibly never |
| `tidalctl repair` command | Crash recovery is automatic in `WalHandle::open()`. Manual repair is a production concern. | M7 |
| `tidalctl dump` (WAL event dump) | Useful for debugging but not required for m0p2 UAT | M1 or M2 when developers need to debug signal event streams |
| WAL counters in metrics | WAL not wired to public API yet | m1p5 |
| Signal counters in metrics | `db.signal()` does not exist yet | m1p5 |
| Query counters in metrics | Query executor does not exist yet | m2p5 |
| Async HTTP for metrics | Sync HTTP is sufficient for Prometheus scraping | M7 if needed |
| `tidalctl` connecting to live process | Embeddable library has no server process | Possibly never |
| Serde on `Config` | tidalctl does not read a config file. Config serde is needed only if we write a config file, which is deferred. | When needed |
---
## 6. Acceptance Criteria
### Task 1: tidalctl CLI
- [ ] **AC-1:** `tidalctl status --path <dir>` against a directory with WAL segments and checkpoint outputs valid JSON containing `version`, `wal.segments`, `wal.checkpoint_seq`, and `dirs.base`
- [ ] **AC-2:** `tidalctl status --path <dir>` against an empty directory (no WAL, no segments) outputs JSON with `status: "empty"` and `wal.segments: 0`
- [ ] **AC-3:** `tidalctl status --path /nonexistent` exits with non-zero status and prints a JSON error object to stderr
- [ ] **AC-4:** `tidalctl paths --path <dir>` outputs JSON with all six directory paths and existence flags matching actual filesystem state
- [ ] **AC-5:** `--pretty` flag produces indented JSON; absence produces compact JSON
- [ ] **AC-6:** `cargo test -p tidalctl` passes with tests for: valid home, empty home, missing home, pretty flag, paths command
### Task 2: Metrics Surface
- [ ] **AC-7:** `TidalDb::builder().ephemeral().enable_metrics("127.0.0.1:0").open()` starts a background HTTP thread bound to an OS-assigned port
- [ ] **AC-8:** `GET /healthz` returns HTTP 200 with JSON containing `status: "ok"` and `uptime_seconds > 0`
- [ ] **AC-9:** `GET /metrics` returns HTTP 200 with valid Prometheus text format containing `tidaldb_uptime_seconds`, `tidaldb_health_ok`, and `tidaldb_info`
- [ ] **AC-10:** `tidaldb_uptime_seconds` increases monotonically between reads (verified by sleeping 100ms between two fetches)
- [ ] **AC-11:** `TidalDb::close()` stops the metrics HTTP thread; subsequent connection attempts to the port are refused
- [ ] **AC-12:** Building `tidaldb` without the `metrics` feature flag compiles successfully with no `tiny_http` dependency; `enable_metrics()` method is absent or returns a compile error guiding the user to enable the feature
---
## 7. UAT Scenario
### Given
```
A developer has:
- Built the workspace: `cargo build --workspace`
- Created a persistent tidalDB instance that wrote WAL segments:
let home = TempTidalHome::new()?;
let paths = home.paths();
paths.ensure_all()?;
let wal_config = WalConfig { dir: home.path().to_path_buf(), ..Default::default() };
let (wal, _) = WalHandle::open(wal_config)?;
wal.append(event_1)?;
wal.append(event_2)?;
wal.checkpoint(2)?;
wal.shutdown()?;
- Opened a TidalDb with metrics enabled:
let db = TidalDb::builder()
.ephemeral()
.enable_metrics("127.0.0.1:0")
.open()?;
```
### When
```
1. Run: tidalctl status --path <home.path()>
2. Run: tidalctl paths --path <home.path()>
3. HTTP GET /healthz on the metrics port
4. HTTP GET /metrics on the metrics port
5. Sleep 200ms
6. HTTP GET /metrics again
7. db.close()
8. Attempt HTTP GET /healthz on the metrics port
```
### Then
```
Step 1: JSON output with wal.segments >= 1, wal.checkpoint_seq == 2,
status == "ok", version matches Cargo.toml
Step 2: JSON output with dirs.wal == "<home>/wal", exists.wal == true
Step 3: HTTP 200, body contains "status":"ok", uptime_seconds > 0
Step 4: HTTP 200, body contains tidaldb_uptime_seconds,
tidaldb_health_ok 1, tidaldb_info{version="0.1.0"...} 1
Step 5: (sleep)
Step 6: tidaldb_uptime_seconds > value from step 4
Step 7: close() returns Ok(())
Step 8: Connection refused (metrics server stopped)
```
### Pass/Fail Gate
m0p2 is **done** when:
- `cargo test -p tidalctl` passes
- `cargo test -p tidaldb --features metrics` passes (metrics integration tests)
- `cargo build --workspace` succeeds with no warnings under `clippy -D warnings`
- All 12 acceptance criteria above are verified by automated tests
- tidalctl uses `Paths` from the tidaldb crate (no duplicated layout logic)
---
## Implementation Notes
### Build Hash
Use a build script (`tidal/build.rs`) or `option_env!("GIT_HASH")` set by CI. For local builds, fall back to `"dev"`. Both tidalctl and the metrics endpoint use the same constant.
### Metrics State Sharing
```rust
pub(crate) struct MetricsState {
opened_at: Instant,
health_ok: AtomicBool,
// Future milestones add: wal_seq: AtomicU64, signal_writes: AtomicU64, etc.
}
```
This struct is `Arc`-shared between `TidalDb` and the metrics HTTP thread. Adding new counters in future milestones is a one-line addition to this struct plus a one-line addition to the Prometheus renderer. The plumbing is paid for once in m0p2.
### tidalctl WAL Inspection
tidalctl depends on `tidaldb` as a library. It calls:
- `tidaldb::db::Paths::new(dir)` for path resolution
- `tidaldb::wal::segment::list_segments(&wal_dir)` for segment enumeration
- `tidaldb::wal::checkpoint::CheckpointManager::read(&wal_dir)` for checkpoint state
These are all `pub` functions already. No new internal APIs need to be exposed. The WAL module's public surface is sufficient.
### Complexity Estimates
| Task | Complexity | Rationale |
|------|-----------|-----------|
| Workspace setup (root Cargo.toml, pre-commit hook update) | S | Mechanical, no design decisions |
| tidalctl CLI (clap, status, paths) | M | Two commands, JSON output, error handling, tests |
| Metrics surface (tiny_http, feature flag, MetricsState, endpoints) | M | Background thread lifecycle, Prometheus format, integration test |
| Build hash plumbing | S | Build script or env var, shared constant |
**Total phase complexity: M** (two M tasks + two S tasks, all independent after workspace setup)

13
tidal/docs/planning/milestone-0/phase-2/task-01-tidalctl.md Normal file
View File

@ -0,0 +1,13 @@
# Task 01 — `tidalctl` CLI
**Goal:** ship a tiny, dependency-light binary that inspects an embedded tidalDB home.
## Deliverables
- `tidalctl status --path <dir>` and `tidalctl paths --path <dir>` commands.
- Shared crate for serialization so CLI can read builder `Config` dumps atomically.
- Documentation on how to vendor the CLI into host repos (cargo install or binary download).
## Acceptance Criteria
- Works against both temp homes and long-lived dirs.
- Output is JSON by default with `--pretty` option.
- `cargo test -p tidalctl` exercises happy/error paths.

13
tidal/docs/planning/milestone-0/phase-2/task-02-metrics-surface.md Normal file
View File

@ -0,0 +1,13 @@
# Task 02 — Metrics / Health Surface
**Goal:** expose runtime stats (uptime, WAL lag, open handles) without requiring full observability stack.
## Deliverables
- Optional HTTP endpoint (disabled by default) serving `/metrics` (Prometheus text) and `/healthz` (JSON).
- Builder flag `enable_metrics(addr)` + `with_metrics(false)` toggle for tests.
- Metrics counters tagged with `process`, `partition_id`, `build_hash` for future distributed rollouts.
## Acceptance Criteria
- Endpoint can run on the same Tokio runtime as host service (returns `Future` implementor).
- Minimal deps (hyper/tiny_http) to keep embeddable footprint light.
- Integration test hits `/metrics` and asserts counters increment when WAL appends.

12
tidal/docs/planning/milestone-0/phase-3/OVERVIEW.md Normal file
View File

@ -0,0 +1,12 @@
# Milestone 0 · Phase 3 — Samples & Docs
**Objective:** provide living examples so every change to the builder/tooling is exercised in CI. This phase adds doctest-backed quickstarts plus integration tests wiring tidalDB into typical Rust stacks.
**Success criteria**
- Quick-start snippet (For You microdemo) lives in `/examples` and is referenced from docs/site; compiled via `cargo test --doc`.
- Axum/Actix embedding examples show graceful shutdown and metrics wiring.
- CONTRIBUTING section updated with “run the samples” checklist.
**Dependencies:** Phase 1 + 2 (uses builder + CLI output).
**Unblocks:** Developer onboarding, future blog posts, automated regression tests.

12
tidal/docs/planning/milestone-0/phase-3/task-01-quickstart-and-doctests.md Normal file
View File

@ -0,0 +1,12 @@
# Task 01 — Quickstart + Doctests
**Goal:** demonstrate embedding tidalDB in <20 LOC and keep it compiling forever.
## Deliverables
- `examples/quickstart.rs` referenced from README + VISION.
- Corresponding snippet in docs that uses Rust fenced code block with `no_run` + doctest harness.
- CI hook (`cargo test --doc --examples`) wired into main workflow.
## Acceptance Criteria
- Quickstart writes a single signal and reads it back once M1 lands, but for now asserts builder + health_check.
- Fails CI if builder API changes without updating docs.

14
tidal/docs/planning/milestone-0/phase-3/task-02-embedding-guides.md Normal file
View File

@ -0,0 +1,14 @@
# Task 02 — Embedding Guides (Axum/Actix/CLI)
**Goal:** show how to wire tidalDB into real hosts so customers don’t guess.
## Deliverables
- Axum guide: builder in `State`, graceful shutdown via `with_graceful_shutdown`.
- Actix guide: `Data<TidalDb>` example + metric endpoint mounting.
- CLI embedding: minimal binary using builder + tidalctl for smoke test.
- Docs page (mdx) referencing all three.
## Acceptance Criteria
- Each guide lives under `/examples` with README instructions.
- Guides double as integration tests (run during CI).
- Includes “cleanup” guidance referencing TempTidalHome helper.

83
tidal/docs/planning/milestone-1/phase-1/OVERVIEW.md Normal file
View File

@ -0,0 +1,83 @@
# Milestone 1, Phase 1: Core Type System and Schema
## Phase Deliverable
The foundational type system -- entity IDs, signal type definitions, decay rate declarations, window specifications, and the error types that every subsequent module depends on. The schema module that validates and stores signal/entity definitions.
## Acceptance Criteria
- [ ] `EntityId` is a u64 newtype with `Display`, `Hash`, `Eq`, `Ord`
- [ ] `SignalTypeDef` declaration captures: name, decay model (exponential/linear/permanent), half-life duration, enabled windows (1h/24h/7d/30d/all_time), velocity enabled flag
- [ ] `DecayModel::Exponential` stores pre-computed lambda derived from half-life: `lambda = ln(2) / half_life_seconds`
- [ ] `LumenError` enum covers Storage, NotFound, Schema, Durability, Query, Internal variants per CODING_GUIDELINES.md
- [ ] Schema validation rejects: duplicate signal names, zero/negative half-life, empty window list on non-permanent signals, velocity without windows
- [ ] All hot-path numeric types use the precision specified in research (f64 for decay scores, u64 for timestamps in nanoseconds)
## Dependencies
- **Requires:** Nothing -- this is the root of the dependency DAG
- **Blocks:** m1p2 (WAL), m1p3 (Storage/fjall), and transitively all subsequent phases
## Research References
- [docs/research/tidaldb_signal_ledger.md](../../../research/tidaldb_signal_ledger.md) -- decay formula, EntityState struct, running-score approach
- [docs/research/phase1_1_type_system.md](../../../research/phase1_1_type_system.md) -- newtype patterns, Duration handling, error hierarchy, schema validation, f64 precision analysis, Window enum design
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) -- error handling (section 7), module boundaries (section 9), dependencies (section 10)
- [thoughts.md](../../../../thoughts.md) -- Part V.12 (subject-prefix keys), Part II.1 (WAL convergence)
## Spec References
- [docs/specs/03-signal-system.md](../../../specs/03-signal-system.md) -- signal type declaration, decay types and lambda precomputation, window definitions, signal ledger architecture
- [docs/specs/11-schema.md](../../../specs/11-schema.md) -- schema definition API, type system, validation rules, schema versioning
- [docs/specs/02-entity-model.md](../../../specs/02-entity-model.md) -- EntityKind (Item/User/Creator), entity ID encoding, storage representation
- [docs/specs/01-storage-engine.md](../../../specs/01-storage-engine.md) -- key encoding scheme using big-endian EntityId and Timestamp
- [docs/specs/00-architecture-overview.md](../../../specs/00-architecture-overview.md) -- system architecture, code module map showing schema/ layout
## Task Index
| # | Task | Delivers | Depends On | Complexity |
|---|------|----------|------------|------------|
| 01 | Core Identity and Temporal Types | `EntityId`, `EntityKind`, `Timestamp`, `Score` | None | S |
| 02 | Signal Type Definitions | `SignalTypeDef`, `DecayModel`, `DecaySpec`, `Window`, `WindowSet` | Task 01 | S |
| 03 | Error Types and Schema Validation | `LumenError`, `SchemaError`, `Schema`, `SchemaBuilder` | Task 01, Task 02 | S |
## Task Dependency DAG
```
Task 01: Core Identity Types
|
v
Task 02: Signal Type Definitions (uses EntityKind from Task 01)
|
v
Task 03: Error Types + Schema Validation (uses EntityId, SignalTypeDef, DecayModel, Window)
```
Tasks 01 and 02 are technically parallelizable if `EntityKind` is extracted first, but at complexity S each, sequential execution is fine.
## File Layout
```
tidal/src/
lib.rs -- pub mod declarations, Result<T> alias, re-exports
schema/
mod.rs -- pub use re-exports from submodules
entity.rs -- Task 01: EntityId, EntityKind
timestamp.rs -- Task 01: Timestamp newtype
score.rs -- Task 01: Score newtype (finite f64 with Ord)
signal.rs -- Task 02: SignalTypeDef, DecayModel, Window, WindowSet
error.rs -- Task 03: LumenError, SchemaError, sub-error stubs
validation.rs -- Task 03: Schema, SchemaBuilder, DecaySpec, SignalBuilder
signals/mod.rs -- empty (m1p4)
storage/mod.rs -- empty (m1p3)
query/mod.rs -- empty (Milestone 2)
ranking/mod.rs -- empty (Milestone 2)
```
## Open Questions
1. **String vs u64 entity IDs in public API** -- API.md uses string IDs (`"item_abc"`), internal types use `u64`. Resolution: `EntityId` is `u64` internally. String-to-u64 mapping is a m1p5 concern when the public `Lumen` API is built. m1p1 defines only the internal type.
2. **EntityId uniqueness scope** -- globally unique or per-EntityKind? Resolution: signal names are globally unique (no `item.view` vs `user.view`). Entity IDs are scoped per-EntityKind by storage namespace. Different column families isolate the namespaces.
3. **Custom windows** -- `Window::Custom(Duration)` deferred. The five fixed variants cover every sort mode and ranking profile in the spec. Adding custom windows would require dynamic bucket allocation. Revisit if M5 benchmarks demand it.

260
tidal/docs/planning/milestone-1/phase-1/task-01-core-identity-types.md Normal file
View File

@ -0,0 +1,260 @@
# Task 01: Core Identity and Temporal Types
## Context
**Milestone:** 1 -- Signal Engine
**Phase:** m1p1 -- Core Type System and Schema
**Depends On:** None
**Blocks:** Task 02, Task 03
**Complexity:** S
## Objective
Deliver the foundational identity and temporal types that every module in the codebase will import: `EntityId`, `EntityKind`, `Timestamp`, and `Score`. These are the zero-cost newtypes that prevent type confusion (mixing entity IDs with raw u64 values, mixing timestamps with byte counts) and provide the ordering guarantees the storage engine requires (big-endian encoding where byte-lexicographic order matches numeric order).
## Requirements
- `EntityId` must be a u64 newtype with `Display`, `Hash`, `Eq`, `Ord`, `Copy`
- `EntityKind` must enumerate exactly three kinds: Item, User, Creator
- `Timestamp` must store nanoseconds since Unix epoch as u64
- `Score` must wrap f64 with a finiteness invariant (rejects NaN/Infinity) and implement `Ord`
- Big-endian byte encoding on `EntityId` and `Timestamp` must preserve numeric ordering
- All types must be `Send + Sync` (they are, since they contain only primitives)
- No new dependencies -- standard library only
## Technical Design
### Module Structure
```
tidal/src/schema/
entity.rs -- EntityId, EntityKind
timestamp.rs -- Timestamp
score.rs -- Score
```
### Public API
```rust
// === entity.rs ===
/// Unique identifier for any entity. Internally a u64.
/// Does NOT carry EntityKind -- kind is determined by storage namespace.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct EntityId(u64);
impl EntityId {
pub const fn new(id: u64) -> Self;
pub const fn as_u64(self) -> u64;
/// Big-endian bytes for key construction. Byte order matches numeric order.
pub const fn to_be_bytes(self) -> [u8; 8];
}
impl fmt::Display for EntityId { /* formats as the raw number */ }
impl fmt::Debug for EntityId { /* formats as EntityId(N) */ }
impl From<u64> for EntityId { /* zero-cost conversion */ }
/// The three entity kinds. Fixed, not extensible.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum EntityKind {
Item,
User,
Creator,
}
impl fmt::Display for EntityKind { /* "item", "user", "creator" */ }
// === timestamp.rs ===
/// Nanoseconds since Unix epoch. u64 overflows in year 2554.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Timestamp(u64);
impl Timestamp {
pub const fn from_nanos(nanos: u64) -> Self;
/// Current wall-clock time via SystemTime::now().
pub fn now() -> Self;
pub const fn as_nanos(self) -> u64;
/// Seconds elapsed as f64 (for decay math: lambda * dt).
pub fn seconds_since(self, now: Timestamp) -> f64;
/// Duration elapsed since this timestamp.
pub fn elapsed_since(self, now: Timestamp) -> std::time::Duration;
/// Big-endian bytes for key construction.
pub const fn to_be_bytes(self) -> [u8; 8];
}
impl fmt::Display for Timestamp { /* "seconds.nanos" format */ }
impl fmt::Debug for Timestamp { /* Timestamp(Nns) */ }
// === score.rs ===
/// A ranking score. Guaranteed finite (not NaN, not infinite).
/// Implements Ord (unlike raw f64) for use in sorting and priority queues.
/// NOT bounded to [0, 1] -- ranking scores can exceed 1.0 after boosting.
#[derive(Clone, Copy, PartialEq)]
pub struct Score(f64);
impl Score {
pub fn new(value: f64) -> Option<Self>;
pub const ZERO: Self;
pub const fn as_f64(self) -> f64;
}
impl Eq for Score {}
impl Ord for Score { /* uses f64::total_cmp, safe because both values are finite */ }
impl PartialOrd for Score { /* delegates to Ord */ }
impl fmt::Display for Score { /* 6 decimal places */ }
impl fmt::Debug for Score { /* Score(N.NNNNNN) */ }
```
### Internal Design
**EntityId does NOT carry EntityKind.** The kind is always known from context -- which storage namespace (column family) you are reading from, which query target was specified, which signal definition targets which kind. Embedding the kind would waste bits of the u64 and force every ID comparison to also compare kind. The key encoding `{entity_id}\x00{TAG}:{suffix}` already isolates by namespace.
**Timestamp uses u64 nanoseconds, not i64.** All signal events are present-tense engagement events. Pre-epoch timestamps are never needed. u64 gives the full range to year 2554 (vs i64's 2262 limit used by InfluxDB). `seconds_since()` returns f64 for direct use in decay math: `exp(-lambda * dt)` where `dt = self.seconds_since(now)`.
**Score enforces finiteness, not bounds.** NaN breaks `Ord` (the reason f64 doesn't implement it). Score guarantees finiteness at construction, enabling total ordering. It is NOT bounded to [0, 1] because ranking profiles apply boosts (multiplication), penalties (subtraction), and diversity reordering that produce scores outside that range.
### Error Handling
No errors in this task. All constructors either cannot fail (`EntityId::new`, `Timestamp::from_nanos`) or return `Option` (`Score::new`). Error types are defined in Task 03.
## Test Strategy
### Property Tests
```rust
// EntityId big-endian round-trip
proptest! {
#[test]
fn entity_id_roundtrip(id: u64) {
let eid = EntityId::new(id);
let bytes = eid.to_be_bytes();
prop_assert_eq!(id, u64::from_be_bytes(bytes));
}
}
// EntityId ordering matches byte ordering (critical for storage scans)
proptest! {
#[test]
fn entity_id_ordering_matches_bytes(a: u64, b: u64) {
let ea = EntityId::new(a);
let eb = EntityId::new(b);
prop_assert_eq!(ea.cmp(&eb), ea.to_be_bytes().cmp(&eb.to_be_bytes()));
}
}
// Timestamp ordering matches byte ordering
proptest! {
#[test]
fn timestamp_ordering_matches_bytes(a: u64, b: u64) {
let ta = Timestamp::from_nanos(a);
let tb = Timestamp::from_nanos(b);
prop_assert_eq!(ta.cmp(&tb), ta.to_be_bytes().cmp(&tb.to_be_bytes()));
}
}
// Timestamp seconds_since is non-negative
proptest! {
#[test]
fn timestamp_seconds_non_negative(a: u64, b: u64) {
let ta = Timestamp::from_nanos(a);
let tb = Timestamp::from_nanos(b);
prop_assert!(ta.seconds_since(tb) >= 0.0);
}
}
// Score total ordering consistency
proptest! {
#[test]
fn score_ordering_consistent(a in proptest::num::f64::NORMAL, b in proptest::num::f64::NORMAL) {
if let (Some(sa), Some(sb)) = (Score::new(a), Score::new(b)) {
prop_assert_eq!(sa.partial_cmp(&sb), Some(sa.cmp(&sb)));
}
}
}
```
### Unit Tests
```rust
// Score rejects non-finite values
#[test]
fn score_rejects_nan_and_infinity() {
assert!(Score::new(f64::NAN).is_none());
assert!(Score::new(f64::INFINITY).is_none());
assert!(Score::new(f64::NEG_INFINITY).is_none());
assert!(Score::new(0.0).is_some());
assert!(Score::new(-1.5).is_some());
assert!(Score::new(100.0).is_some());
}
// EntityId display format
#[test]
fn entity_id_display() {
assert_eq!(EntityId::new(42).to_string(), "42");
assert_eq!(format!("{:?}", EntityId::new(42)), "EntityId(42)");
}
// EntityKind display format
#[test]
fn entity_kind_display() {
assert_eq!(EntityKind::Item.to_string(), "item");
assert_eq!(EntityKind::User.to_string(), "user");
assert_eq!(EntityKind::Creator.to_string(), "creator");
}
// Timestamp now() returns a reasonable value
#[test]
fn timestamp_now_reasonable() {
let ts = Timestamp::now();
// Must be after 2020-01-01
let min = 1_577_836_800_000_000_000u64; // 2020-01-01 in nanos
assert!(ts.as_nanos() > min);
}
// Timestamp seconds_since arithmetic
#[test]
fn timestamp_seconds_since() {
let t1 = Timestamp::from_nanos(1_000_000_000); // 1 second
let t2 = Timestamp::from_nanos(3_500_000_000); // 3.5 seconds
let dt = t1.seconds_since(t2);
assert!((dt - 2.5).abs() < 1e-9);
}
```
## Acceptance Criteria
- [ ] `EntityId` is a u64 newtype with `Display`, `Hash`, `Eq`, `Ord`, `Copy`, `Clone`, `Debug`
- [ ] `EntityKind` has exactly three variants: `Item`, `User`, `Creator`
- [ ] `Timestamp` stores nanoseconds as u64; `Timestamp::now()` returns current time
- [ ] `Timestamp::seconds_since()` returns f64 delta for decay math
- [ ] `Score` rejects NaN and infinities; implements `Ord`
- [ ] Big-endian byte encoding on `EntityId` and `Timestamp` preserves numeric ordering (property tested)
- [ ] All types live in `tidal/src/schema/` submodules
- [ ] `cargo clippy -- -D warnings` passes
- [ ] All property tests and unit tests pass
## Research References
- [docs/research/tidaldb_signal_ledger.md](../../../research/tidaldb_signal_ledger.md) -- `entity_id: u64`, `last_update_ns: u64` in EntityState struct, f64 precision analysis confirming adequacy through year 18,000
- [docs/research/phase1_1_type_system.md](../../../research/phase1_1_type_system.md) -- Section 1 (EntityId newtype pattern: hand-implement vs derive_more vs nutype), Section 6 (Timestamp precision: u64 nanoseconds, production system survey of InfluxDB/QuestDB/ClickHouse/Sonnerie), Section 5 (f64 for decay scores and atomic operations)
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) -- Section 1 (memory layout), Section 2 (key encoding: big-endian for byte-lexicographic ordering)
- [thoughts.md](../../../../thoughts.md) -- Part V.12 (subject-prefix keys require byte-ordered entity IDs)
## Spec References
- [docs/specs/02-entity-model.md](../../../specs/02-entity-model.md) -- EntityKind (Item/User/Creator), entity ID encoding as u64 big-endian with 0x01/0x02/0x03 kind bytes, storage representation key layout `[entity_kind: u8][entity_id: u64 BE][0x00][TAG]:[suffix]`
- [docs/specs/01-storage-engine.md](../../../specs/01-storage-engine.md) -- Section 5 (key encoding scheme: big-endian entity IDs for lexicographic ordering, NUL separator, tag-based routing), Section 5.5 (byte-level example), Appendix C invariant 9 (big-endian encoding preserves numeric ordering)
- [docs/specs/03-signal-system.md](../../../specs/03-signal-system.md) -- Section 3 (HotSignalState struct using `entity_id: u64` and `last_update_ns: AtomicU64`), Section 4 (Timestamp used in decay computation: `dt = (event_time_ns - prev_time) as f64 / 1e9`)
- [docs/specs/09-ranking-scoring.md](../../../specs/09-ranking-scoring.md) -- Section 8 (Score normalization to [0.0, 1.0] range, confirming Score type must support values outside that range before normalization)
## Implementation Notes
- `#[repr(transparent)]` is NOT needed on newtypes that don't cross FFI boundaries. The compiler optimizes these identically without it.
- The `expect()` in `Timestamp::now()` is acceptable -- a system clock before Unix epoch is a hardware fault, not a recoverable error.
- `Score::ZERO` uses `const` construction. This requires knowing the value is finite at compile time, which 0.0 trivially is.
- Do NOT add `serde` derives yet. Serialization is m1p3's concern when types need to go to disk.
- Do NOT add `#[repr(C, align(64))]` to any type. Cache-line alignment is m1p4's concern for the hot-path `EntitySignalState` struct.

325
tidal/docs/planning/milestone-1/phase-1/task-02-signal-type-definitions.md Normal file
View File

@ -0,0 +1,325 @@
# Task 02: Signal Type Definitions
## Context
**Milestone:** 1 -- Signal Engine
**Phase:** m1p1 -- Core Type System and Schema
**Depends On:** Task 01 (uses `EntityKind`)
**Blocks:** Task 03
**Complexity:** S
## Objective
Deliver the types that declare what a signal IS in schema: `SignalTypeDef`, `DecayModel`, `Window`, and `WindowSet`. These are the *declarations*, not the runtime state. They describe how a signal decays, what windows to maintain, and whether velocity is computed. The actual signal ledger and aggregation logic are m1p4.
The critical design choice: `DecayModel::Exponential` stores the pre-computed lambda (`ln(2) / half_life_seconds`) so that every signal write and every ranking read avoids a division on the hot path. The user specifies `DecaySpec::Exponential { half_life: Duration }` (validated in Task 03). The internal `DecayModel` stores the derived lambda.
## Requirements
- `SignalTypeDef` must capture: name, target entity kind, decay model, windows, velocity flag
- `DecayModel` must support three variants: Exponential (with pre-computed lambda), Linear, Permanent
- Pre-computed lambda for exponential decay: `lambda = ln(2) / half_life_seconds`
- `Window` must enumerate exactly five variants: 1h, 24h, 7d, 30d, AllTime
- `WindowSet` must be an ordered, deduplicated collection of windows
- Signal type fields must be private with getters (constructed only through validated SchemaBuilder in Task 03)
- No new dependencies
## Technical Design
### Module Structure
```
tidal/src/schema/
signal.rs -- SignalTypeDef, DecayModel, Window, WindowSet
```
### Public API
```rust
// === signal.rs ===
/// A named signal type definition declared in schema.
/// This is the *declaration*, not runtime state.
#[derive(Debug, Clone)]
pub struct SignalTypeDef { /* private fields */ }
impl SignalTypeDef {
/// Unique name within the schema (e.g., "view", "like", "skip").
pub fn name(&self) -> &str;
/// Which entity kind this signal targets.
pub fn target(&self) -> EntityKind;
/// How the signal's weight decays over time.
pub fn decay(&self) -> &DecayModel;
/// Which time windows to maintain aggregates for.
pub fn windows(&self) -> &WindowSet;
/// Whether velocity computation is enabled.
pub fn velocity_enabled(&self) -> bool;
}
// pub(crate) constructor -- only callable from validation module
impl SignalTypeDef {
pub(crate) fn new(
name: String,
target: EntityKind,
decay: DecayModel,
windows: WindowSet,
velocity_enabled: bool,
) -> Self;
}
/// How a signal's contribution decays over time.
#[derive(Debug, Clone, PartialEq)]
pub enum DecayModel {
/// Weight halves every `half_life`.
/// Running score formula: S(t) = S(t_prev) * exp(-lambda * dt) + weight
Exponential {
half_life: std::time::Duration,
/// Pre-computed: ln(2) / half_life.as_secs_f64()
lambda: f64,
},
/// Weight drops linearly to zero over `lifetime`.
Linear {
lifetime: std::time::Duration,
},
/// Never decays. Used for permanent flags: hide, block, follow.
Permanent,
}
impl DecayModel {
/// Construct exponential decay with pre-computed lambda.
/// pub(crate): bypasses validation. Use SchemaBuilder for external construction.
pub(crate) fn exponential(half_life: Duration) -> Self;
/// Construct linear decay.
pub(crate) fn linear(lifetime: Duration) -> Self;
/// Returns the lambda value for Exponential, None otherwise.
pub fn lambda(&self) -> Option<f64>;
/// Returns the half-life for Exponential, None otherwise.
pub fn half_life(&self) -> Option<Duration>;
}
/// A time window for signal aggregation.
/// Fixed variants -- not configurable durations.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum Window {
OneHour,
TwentyFourHours,
SevenDays,
ThirtyDays,
AllTime,
}
impl Window {
/// The duration this window spans. AllTime returns Duration::MAX.
pub const fn duration(&self) -> Duration;
/// Duration in seconds as f64 (for velocity: count / duration_secs).
pub const fn duration_secs_f64(&self) -> f64;
/// Short label for display and key encoding ("1h", "24h", "7d", "30d", "all").
pub const fn label(&self) -> &'static str;
}
impl fmt::Display for Window { /* delegates to label() */ }
/// An ordered, deduplicated set of windows.
/// Sorted from finest to coarsest (OneHour < ... < AllTime).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WindowSet { /* private Vec<Window> */ }
impl WindowSet {
/// Construct from a slice. Deduplicates and sorts.
pub fn new(windows: &[Window]) -> Self;
/// Empty set. Valid only for Permanent decay signals.
pub const fn empty() -> Self;
pub fn is_empty(&self) -> bool;
pub fn len(&self) -> usize;
pub fn iter(&self) -> std::slice::Iter<'_, Window>;
pub fn contains(&self, w: &Window) -> bool;
}
```
### Internal Design
**Window is an enum with 5 fixed variants, not configurable durations.** The research doc shows exactly these five windows used across all 14 use cases and 25+ sort modes. The storage engine pre-allocates bucketed counters per window. The materializer schedules rollups at window boundaries. Arbitrary durations would force dynamic allocation and unpredictable rollup schedules. If a sixth window is ever needed, it is a schema migration.
**The Ord derivation on Window sorts by temporal duration.** `OneHour < TwentyFourHours < SevenDays < ThirtyDays < AllTime`. This matches the enum variant declaration order and is relied upon by WindowSet for canonical ordering.
**DecayModel constructors are `pub(crate)`.** External users construct signals through `SchemaBuilder` (Task 03), which validates inputs before calling these constructors. Making them `pub(crate)` prevents construction that bypasses validation.
**SignalTypeDef fields are private with getters.** Once validated and constructed by the SchemaBuilder, signal type definitions are immutable. Private fields + getters enforce this.
**`WindowSet::empty()` uses `const fn` with `Vec::new()`.** As of Rust edition 2024, `Vec::new()` is const. This allows `WindowSet::empty()` to be a const function.
### Error Handling
No errors in this task. All constructors are `pub(crate)` and infallible -- validation happens in the SchemaBuilder (Task 03). `WindowSet::new()` cannot fail (it deduplicates and sorts silently).
## Test Strategy
### Property Tests
```rust
// Lambda is correctly computed from half-life
proptest! {
#[test]
fn decay_lambda_correct(secs in 1u64..=31_536_000u64) {
let half_life = Duration::from_secs(secs);
let model = DecayModel::exponential(half_life);
if let DecayModel::Exponential { lambda, .. } = model {
let expected = std::f64::consts::LN_2 / half_life.as_secs_f64();
prop_assert!((lambda - expected).abs() < 1e-15);
}
}
}
// Lambda * half_life = ln(2) (the defining property)
proptest! {
#[test]
fn lambda_times_halflife_is_ln2(secs in 1u64..=31_536_000u64) {
let half_life = Duration::from_secs(secs);
let model = DecayModel::exponential(half_life);
if let DecayModel::Exponential { lambda, .. } = model {
let product = lambda * half_life.as_secs_f64();
prop_assert!((product - std::f64::consts::LN_2).abs() < 1e-10);
}
}
}
```
### Unit Tests
```rust
// Window ordering
#[test]
fn window_ordering() {
assert!(Window::OneHour < Window::TwentyFourHours);
assert!(Window::TwentyFourHours < Window::SevenDays);
assert!(Window::SevenDays < Window::ThirtyDays);
assert!(Window::ThirtyDays < Window::AllTime);
}
// Window durations are correct
#[test]
fn window_durations() {
assert_eq!(Window::OneHour.duration(), Duration::from_secs(3_600));
assert_eq!(Window::TwentyFourHours.duration(), Duration::from_secs(86_400));
assert_eq!(Window::SevenDays.duration(), Duration::from_secs(604_800));
assert_eq!(Window::ThirtyDays.duration(), Duration::from_secs(2_592_000));
assert_eq!(Window::AllTime.duration(), Duration::MAX);
}
// Window labels for key encoding
#[test]
fn window_labels() {
assert_eq!(Window::OneHour.label(), "1h");
assert_eq!(Window::TwentyFourHours.label(), "24h");
assert_eq!(Window::SevenDays.label(), "7d");
assert_eq!(Window::ThirtyDays.label(), "30d");
assert_eq!(Window::AllTime.label(), "all");
}
// WindowSet deduplication and sorting
#[test]
fn window_set_dedup_and_sort() {
let ws = WindowSet::new(&[Window::SevenDays, Window::OneHour, Window::SevenDays, Window::AllTime]);
assert_eq!(ws.len(), 3);
let windows: Vec<_> = ws.iter().copied().collect();
assert_eq!(windows, vec![Window::OneHour, Window::SevenDays, Window::AllTime]);
}
// WindowSet empty
#[test]
fn window_set_empty() {
let ws = WindowSet::empty();
assert!(ws.is_empty());
assert_eq!(ws.len(), 0);
}
// DecayModel exponential stores half-life and lambda
#[test]
fn decay_model_exponential() {
let model = DecayModel::exponential(Duration::from_secs(604_800)); // 7 days
assert!(matches!(model, DecayModel::Exponential { .. }));
let lambda = model.lambda().unwrap();
let expected = std::f64::consts::LN_2 / 604_800.0;
assert!((lambda - expected).abs() < 1e-20);
}
// DecayModel permanent has no lambda
#[test]
fn decay_model_permanent() {
assert_eq!(DecayModel::Permanent.lambda(), None);
assert_eq!(DecayModel::Permanent.half_life(), None);
}
// SignalTypeDef getters
#[test]
fn signal_type_def_getters() {
let def = SignalTypeDef::new(
"view".into(),
EntityKind::Item,
DecayModel::exponential(Duration::from_secs(604_800)),
WindowSet::new(&[Window::OneHour, Window::AllTime]),
true,
);
assert_eq!(def.name(), "view");
assert_eq!(def.target(), EntityKind::Item);
assert!(def.velocity_enabled());
assert_eq!(def.windows().len(), 2);
assert!(def.decay().lambda().is_some());
}
// Edge case: very small half-life (lambda is very large)
#[test]
fn decay_model_tiny_halflife() {
let model = DecayModel::exponential(Duration::from_nanos(1)); // 1 nanosecond
let lambda = model.lambda().unwrap();
// lambda should be enormous, signals decay instantly
assert!(lambda > 1e8);
}
// Edge case: very large half-life (lambda is very small)
#[test]
fn decay_model_huge_halflife() {
let model = DecayModel::exponential(Duration::from_secs(365 * 24 * 3600)); // 1 year
let lambda = model.lambda().unwrap();
assert!(lambda > 0.0);
assert!(lambda < 1e-6);
}
```
## Acceptance Criteria
- [ ] `SignalTypeDef` stores name, target, decay model, windows, and velocity flag with private fields and pub getters
- [ ] `DecayModel::Exponential` stores pre-computed `lambda = ln(2) / half_life.as_secs_f64()`
- [ ] `DecayModel::Linear` stores lifetime duration
- [ ] `DecayModel::Permanent` has no parameters
- [ ] `Window` has exactly 5 variants with correct durations and labels
- [ ] `WindowSet` deduplicates and sorts windows from finest to coarsest
- [ ] `DecayModel` constructors are `pub(crate)` (external construction through SchemaBuilder only)
- [ ] Property test: `lambda * half_life = ln(2)` for all positive half-life values
- [ ] `cargo clippy -- -D warnings` passes
- [ ] All property tests and unit tests pass
## Research References
- [docs/research/tidaldb_signal_ledger.md](../../../research/tidaldb_signal_ledger.md) -- decay formula (`S(t) = S(t_prev) * exp(-lambda * dt) + weight`), lambda = `ln(2) / half_life_seconds`, EntityState struct showing `decay_scores: [f64; 3]`
- [docs/research/phase1_1_type_system.md](../../../research/phase1_1_type_system.md) -- Section 2 (Duration handling for half-life: `std::time::Duration` vs raw f64, precision analysis), Section 5 (f64 for decay scores and atomic operations), Section 7 (Window enum design: fixed 5-variant enum vs configurable durations, production system survey)
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) -- Section 3 (Signal System: "Decay is a type, not a formula you call", running decay scores O(1) update O(1) read)
- [API.md](../../../../API.md) -- SignalDef struct (Decay::Exponential, Decay::Linear, Decay::Permanent, Window variants)
- [USE_CASES.md](../../../../USE_CASES.md) -- Appendix C (Signal Reference: decay rates per signal type)
## Spec References
- [docs/specs/03-signal-system.md](../../../specs/03-signal-system.md) -- Section 2 (signal type declaration: name, target_kind, decay, windows, velocity_enabled), Section 3 (HotSignalState struct using decay_scores with pre-computed lambda), Section 4 (decay computation: `dt = (event_time_ns - prev_time) as f64 / 1e9`), lambda precomputation table, 40 signal types reference with decay rates and windows
- [docs/specs/11-schema.md](../../../specs/11-schema.md) -- Schema definition API (signal type registration, validation rules), DecaySpec vs DecayModel separation, SchemaBuilder pattern
- [docs/specs/09-ranking-scoring.md](../../../specs/09-ranking-scoring.md) -- Boost types referencing signal windows and decay scores (e.g., `SignalBoost { signal: "view", window: "24h" }`), score normalization pipeline
- [docs/specs/01-storage-engine.md](../../../specs/01-storage-engine.md) -- Section 5 (key encoding: `SIG:{signal_name}:{window_label}` suffix format, window labels used in storage keys)
## Implementation Notes
- The `PartialEq` on `DecayModel` compares `f64` lambda values. This is sound because lambda is deterministically computed from the same half-life Duration -- the same input always produces the same bits. Two DecayModels with the same half-life will have bitwise-equal lambdas.
- `Window::duration_secs_f64()` for `AllTime` should return `f64::MAX` or `f64::INFINITY`. Choose `f64::INFINITY` -- velocity = count / infinity = 0.0, which is correct (all-time counts don't have a meaningful rate).
- Do NOT implement the actual decay computation (`S(t) = S(t_prev) * exp(-lambda * dt) + weight`) here. That is m1p4. This task only stores the lambda value.
- Do NOT add serde derives. Serialization is m1p3+.

508
tidal/docs/planning/milestone-1/phase-1/task-03-error-types-and-schema-validation.md Normal file
View File

@ -0,0 +1,508 @@
# Task 03: Error Types and Schema Validation
## Context
**Milestone:** 1 -- Signal Engine
**Phase:** m1p1 -- Core Type System and Schema
**Depends On:** Task 01 (EntityId for NotFound), Task 02 (SignalTypeDef, DecayModel, Window for validation)
**Blocks:** m1p2 (WAL), m1p3 (Storage/fjall)
**Complexity:** S
## Objective
Deliver the error hierarchy (`LumenError` with 6 variants per CODING_GUIDELINES.md) and the `SchemaBuilder` that validates and produces an immutable `Schema`. The SchemaBuilder is the single construction path for signal type definitions -- it validates inputs, computes derived values (lambda from half-life), and produces an immutable schema that every other module receives.
This task delivers the `DecaySpec` type (user-facing, e.g., `DecaySpec::Exponential { half_life }`) which is separate from `DecayModel` (internal, carries pre-computed lambda). The builder validates the spec and converts it to the model.
## Requirements
- `LumenError` must have exactly 6 variants: Storage, NotFound, Schema, Durability, Query, Internal
- All error types must implement `std::fmt::Display` and `std::error::Error`
- `SchemaError` must have variants for every validation rule
- `Schema` must be immutable after construction
- `SchemaBuilder` must validate:
- No duplicate signal names
- Signal names are valid identifiers (lowercase alphanumeric + underscore)
- Positive half-life for exponential decay
- Positive lifetime for linear decay
- Non-empty windows for non-permanent signals
- No velocity without windows
- `Result<T>` type alias exported from crate root
- `From` impls for ergonomic `?` operator usage
- No new dependencies (hand-implement Display/Error)
## Technical Design
### Module Structure
```
tidal/src/schema/
error.rs -- LumenError, SchemaError, StorageError, DurabilityError, QueryError
validation.rs -- Schema, SchemaBuilder, DecaySpec, SignalBuilder
tidal/src/
lib.rs -- pub type Result<T> = std::result::Result<T, LumenError>;
```
### Public API
```rust
// === error.rs ===
/// Top-level error type. Every public API method returns Result<T, LumenError>.
#[derive(Debug)]
pub enum LumenError {
/// Storage engine failure. Retry may succeed.
Storage(StorageError),
/// Entity not found. Caller should handle.
NotFound { kind: EntityKind, id: EntityId },
/// Schema violation. Caller's fault -- fix the input.
Schema(SchemaError),
/// Signal write failed durability check. Retry required.
Durability(DurabilityError),
/// Query malformed. Parse error with details.
Query(QueryError),
/// Internal invariant violated. This is a bug in Lumen.
Internal(String),
}
impl fmt::Display for LumenError { /* variant-specific messages */ }
impl std::error::Error for LumenError { /* source() delegates to inner errors */ }
/// Schema validation errors. Exhaustive for m1p1.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SchemaError {
DuplicateSignalName(String),
InvalidSignalName(String),
InvalidHalfLife { signal_name: String, half_life_secs: f64 },
InvalidLifetime { signal_name: String, lifetime_secs: f64 },
EmptyWindows { signal_name: String },
VelocityWithoutWindows { signal_name: String },
NoSignalsDefined,
}
impl fmt::Display for SchemaError { /* actionable messages per variant */ }
impl std::error::Error for SchemaError {}
/// Stub for m1p2+. Single message field.
#[derive(Debug)]
pub struct StorageError { pub message: String }
/// Stub for m1p2+.
#[derive(Debug)]
pub struct DurabilityError { pub message: String }
/// Stub for Milestone 2+.
#[derive(Debug)]
pub struct QueryError { pub message: String }
// From impls for ? operator
impl From<SchemaError> for LumenError { ... }
impl From<StorageError> for LumenError { ... }
impl From<DurabilityError> for LumenError { ... }
impl From<QueryError> for LumenError { ... }
// === validation.rs ===
/// A validated, immutable schema.
#[derive(Debug, Clone)]
pub struct Schema { /* HashMap<String, SignalTypeDef> */ }
impl Schema {
pub fn signal(&self, name: &str) -> Option<&SignalTypeDef>;
pub fn signals(&self) -> impl Iterator<Item = &SignalTypeDef>;
pub fn signal_count(&self) -> usize;
}
/// Builder for constructing a validated Schema.
pub struct SchemaBuilder { /* Vec<SignalEntry> */ }
impl SchemaBuilder {
pub fn new() -> Self;
pub fn signal(&mut self, name: &str, target: EntityKind, decay: DecaySpec) -> SignalBuilder<'_>;
pub fn build(self) -> Result<Schema, SchemaError>;
}
impl Default for SchemaBuilder { fn default() -> Self { Self::new() } }
/// User-facing decay specification (before validation computes lambda).
#[derive(Debug, Clone)]
pub enum DecaySpec {
Exponential { half_life: Duration },
Linear { lifetime: Duration },
Permanent,
}
/// Intermediate builder for a single signal type.
pub struct SignalBuilder<'a> { /* &mut SchemaBuilder + SignalEntry */ }
impl<'a> SignalBuilder<'a> {
pub fn windows(self, windows: &[Window]) -> Self;
pub fn velocity(self, enabled: bool) -> Self;
pub fn add(self) -> &'a mut SchemaBuilder;
}
```
### Internal Design
**Validation rules in `SchemaBuilder::build()`:**
1. **Empty schema** -- `self.signals.is_empty()` -> `SchemaError::NoSignalsDefined`
2. **For each signal entry:**
a. **Name validation** -- must be non-empty, ASCII, lowercase alphanumeric + underscore, start with a letter -> `SchemaError::InvalidSignalName`
b. **Duplicate check** -- `signals.contains_key(&name)` -> `SchemaError::DuplicateSignalName`
c. **Half-life validation** -- for `DecaySpec::Exponential`, `half_life.as_secs_f64() <= 0.0 || !finite` -> `SchemaError::InvalidHalfLife`
d. **Lifetime validation** -- for `DecaySpec::Linear`, same check -> `SchemaError::InvalidLifetime`
e. **Window check** -- for non-Permanent decay, empty windows -> `SchemaError::EmptyWindows`
f. **Velocity check** -- `velocity && windows.is_empty()` -> `SchemaError::VelocityWithoutWindows`
3. **Convert** `DecaySpec` to `DecayModel` (computing lambda for exponential)
4. **Construct** `WindowSet` (dedup and sort)
5. **Create** `SignalTypeDef` via `pub(crate)` constructor
6. **Insert** into `HashMap<String, SignalTypeDef>`
**Signal name validation function:**
```rust
fn is_valid_signal_name(name: &str) -> bool {
!name.is_empty()
&& name.is_ascii()
&& name.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'_')
&& name.as_bytes()[0].is_ascii_lowercase()
}
```
Names must be safe for use as key components in storage (`[entity_id]\x00SIG:{name}:{window}`) and as identifiers in the query language.
**`DecaySpec` vs `DecayModel` separation:** Users specify `DecaySpec::Exponential { half_life: Duration::from_secs(604_800) }` -- no lambda. The builder validates the duration and computes `DecayModel::Exponential { half_life, lambda }`. This means users never touch lambda, and the hot-path code never recomputes it.
**Error stubs (`StorageError`, `DurabilityError`, `QueryError`)** have a single `message: String` field. They exist so that `LumenError` can be fully defined now. Later phases replace the stub with detailed variants (e.g., `StorageError::IoError`, `StorageError::Corruption`).
### Error Handling
The `SchemaBuilder::build()` method returns `Result<Schema, SchemaError>`. It does NOT return `LumenError` directly -- the caller converts via `From<SchemaError>`:
```rust
let schema = SchemaBuilder::new()
.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(604_800),
})
.windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays])
.velocity(true)
.add()
.build()?; // SchemaError auto-converts to LumenError via From impl
```
## Test Strategy
### Unit Tests (Validation Logic)
```rust
// Valid schema construction
#[test]
fn valid_schema_round_trip() {
let schema = SchemaBuilder::new()
.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(604_800),
})
.windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays, Window::AllTime])
.velocity(true)
.add()
.signal("hide", EntityKind::Item, DecaySpec::Permanent)
.add()
.build()
.expect("valid schema");
assert_eq!(schema.signal_count(), 2);
let view = schema.signal("view").unwrap();
assert_eq!(view.name(), "view");
assert_eq!(view.target(), EntityKind::Item);
assert!(view.velocity_enabled());
assert_eq!(view.windows().len(), 4);
assert!(view.decay().lambda().is_some());
let hide = schema.signal("hide").unwrap();
assert_eq!(hide.windows().len(), 0);
assert!(!hide.velocity_enabled());
assert_eq!(*hide.decay(), DecayModel::Permanent);
}
// Duplicate signal name rejected
#[test]
fn rejects_duplicate_signal_name() {
let result = SchemaBuilder::new()
.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(604_800),
})
.windows(&[Window::AllTime]).add()
.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(86_400),
})
.windows(&[Window::AllTime]).add()
.build();
assert_eq!(result, Err(SchemaError::DuplicateSignalName("view".into())));
}
// Zero half-life rejected
#[test]
fn rejects_zero_half_life() {
let result = SchemaBuilder::new()
.signal("bad", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::ZERO,
})
.windows(&[Window::AllTime]).add()
.build();
assert!(matches!(result, Err(SchemaError::InvalidHalfLife { .. })));
}
// Zero linear lifetime rejected
#[test]
fn rejects_zero_linear_lifetime() {
let result = SchemaBuilder::new()
.signal("bad", EntityKind::Item, DecaySpec::Linear {
lifetime: Duration::ZERO,
})
.windows(&[Window::AllTime]).add()
.build();
assert!(matches!(result, Err(SchemaError::InvalidLifetime { .. })));
}
// Empty windows on exponential signal rejected
#[test]
fn rejects_empty_windows_on_exponential() {
let result = SchemaBuilder::new()
.signal("bad", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(3600),
})
.add() // no windows
.build();
assert!(matches!(result, Err(SchemaError::EmptyWindows { .. })));
}
// Permanent signal with empty windows accepted (the "hide" pattern)
#[test]
fn accepts_permanent_with_empty_windows() {
let result = SchemaBuilder::new()
.signal("hide", EntityKind::Item, DecaySpec::Permanent)
.add()
.build();
assert!(result.is_ok());
}
// Velocity without windows rejected
#[test]
fn rejects_velocity_without_windows() {
let result = SchemaBuilder::new()
.signal("bad", EntityKind::Item, DecaySpec::Permanent)
.velocity(true)
.add()
.build();
assert!(matches!(result, Err(SchemaError::VelocityWithoutWindows { .. })));
}
// Empty schema rejected
#[test]
fn rejects_empty_schema() {
let result = SchemaBuilder::new().build();
assert_eq!(result, Err(SchemaError::NoSignalsDefined));
}
// Invalid signal names rejected
#[test]
fn rejects_invalid_signal_names() {
// Empty
let r = SchemaBuilder::new()
.signal("", EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(matches!(r, Err(SchemaError::InvalidSignalName(_))));
// Uppercase
let r = SchemaBuilder::new()
.signal("View", EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(matches!(r, Err(SchemaError::InvalidSignalName(_))));
// Starts with digit
let r = SchemaBuilder::new()
.signal("1view", EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(matches!(r, Err(SchemaError::InvalidSignalName(_))));
// Contains space
let r = SchemaBuilder::new()
.signal("view count", EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(matches!(r, Err(SchemaError::InvalidSignalName(_))));
// Contains hyphen
let r = SchemaBuilder::new()
.signal("view-count", EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(matches!(r, Err(SchemaError::InvalidSignalName(_))));
}
// Valid signal names accepted
#[test]
fn accepts_valid_signal_names() {
let names = ["view", "like", "skip", "hide", "search_click", "autoplay_accept", "view_24h"];
for name in names {
let r = SchemaBuilder::new()
.signal(name, EntityKind::Item, DecaySpec::Permanent).add().build();
assert!(r.is_ok(), "should accept signal name '{name}'");
}
}
// LumenError Display formatting
#[test]
fn lumen_error_display() {
let e = LumenError::NotFound {
kind: EntityKind::Item,
id: EntityId::new(42),
};
assert_eq!(e.to_string(), "item 42 not found");
let e = LumenError::Schema(SchemaError::DuplicateSignalName("view".into()));
assert!(e.to_string().contains("duplicate signal name"));
let e = LumenError::Internal("something broke".into());
assert!(e.to_string().contains("internal error"));
}
// Error source chain
#[test]
fn lumen_error_source() {
use std::error::Error;
let e = LumenError::Schema(SchemaError::NoSignalsDefined);
assert!(e.source().is_some());
let e = LumenError::Internal("bug".into());
assert!(e.source().is_none());
}
// From conversions for ? operator
#[test]
fn schema_error_converts_to_lumen_error() {
let schema_err = SchemaError::NoSignalsDefined;
let lumen_err: LumenError = schema_err.into();
assert!(matches!(lumen_err, LumenError::Schema(SchemaError::NoSignalsDefined)));
}
```
### Property Tests
```rust
// Any valid schema can be queried for all its signals
proptest! {
#[test]
fn schema_contains_all_defined_signals(
count in 1usize..10,
) {
let mut builder = SchemaBuilder::new();
let names: Vec<String> = (0..count)
.map(|i| format!("signal_{i}"))
.collect();
for name in &names {
builder = *builder.signal(
name,
EntityKind::Item,
DecaySpec::Permanent,
).add();
}
let schema = builder.build().unwrap();
prop_assert_eq!(schema.signal_count(), count);
for name in &names {
prop_assert!(schema.signal(name).is_some());
}
}
}
```
### Full UAT-Style Schema (Integration Test)
```rust
// Build the exact schema from the Milestone 1 UAT scenario
#[test]
fn milestone_1_uat_schema() {
let schema = SchemaBuilder::new()
.signal("view", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(7 * 24 * 3600), // 7 days
})
.windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays])
.velocity(true)
.add()
.signal("like", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(14 * 24 * 3600), // 14 days
})
.windows(&[Window::TwentyFourHours, Window::SevenDays, Window::AllTime])
.velocity(true)
.add()
.signal("skip", EntityKind::Item, DecaySpec::Exponential {
half_life: Duration::from_secs(24 * 3600), // 1 day
})
.windows(&[Window::OneHour, Window::TwentyFourHours])
.velocity(false)
.add()
.build()
.expect("UAT schema should be valid");
assert_eq!(schema.signal_count(), 3);
// Verify view signal
let view = schema.signal("view").unwrap();
assert_eq!(view.windows().len(), 3);
assert!(view.velocity_enabled());
let lambda = view.decay().lambda().unwrap();
let expected_lambda = std::f64::consts::LN_2 / (7.0 * 24.0 * 3600.0);
assert!((lambda - expected_lambda).abs() < 1e-15);
// Verify like signal
let like = schema.signal("like").unwrap();
assert_eq!(like.windows().len(), 3);
assert!(like.windows().contains(&Window::AllTime));
// Verify skip signal
let skip = schema.signal("skip").unwrap();
assert!(!skip.velocity_enabled());
let skip_lambda = skip.decay().lambda().unwrap();
let expected_skip_lambda = std::f64::consts::LN_2 / (24.0 * 3600.0);
assert!((skip_lambda - expected_skip_lambda).abs() < 1e-15);
}
```
## Acceptance Criteria
- [ ] `LumenError` has exactly 6 variants: Storage, NotFound, Schema, Durability, Query, Internal
- [ ] All error types implement `Display` and `Error`
- [ ] `From<SchemaError>`, `From<StorageError>`, `From<DurabilityError>`, `From<QueryError>` into `LumenError`
- [ ] `Result<T>` alias exported from `tidal/src/lib.rs`
- [ ] `SchemaBuilder::build()` rejects duplicate signal names (`SchemaError::DuplicateSignalName`)
- [ ] `SchemaBuilder::build()` rejects invalid signal names (`SchemaError::InvalidSignalName`)
- [ ] `SchemaBuilder::build()` rejects zero/negative half-life (`SchemaError::InvalidHalfLife`)
- [ ] `SchemaBuilder::build()` rejects zero/negative linear lifetime (`SchemaError::InvalidLifetime`)
- [ ] `SchemaBuilder::build()` rejects empty windows on non-permanent signals (`SchemaError::EmptyWindows`)
- [ ] `SchemaBuilder::build()` rejects velocity without windows (`SchemaError::VelocityWithoutWindows`)
- [ ] `SchemaBuilder::build()` rejects empty schema (`SchemaError::NoSignalsDefined`)
- [ ] `SchemaBuilder::build()` accepts permanent signals with empty windows (the "hide" pattern)
- [ ] Validated `Schema` is immutable and queryable by signal name
- [ ] The M1 UAT schema (view/like/skip from the roadmap) builds successfully
- [ ] `cargo clippy -- -D warnings` passes
- [ ] All unit tests, property tests, and integration tests pass
- [ ] `cargo test --lib` exits cleanly
## Research References
- [docs/research/phase1_1_type_system.md](../../../research/phase1_1_type_system.md) -- Section 3 (error handling: thiserror vs hand-implement, recommendation to hand-implement for <100 lines), Section 4 (schema validation pattern: typestate vs runtime builder vs struct+validate, recommendation for struct-with-validation / builder pattern)
- [docs/research/tidaldb_signal_ledger.md](../../../research/tidaldb_signal_ledger.md) -- validates that lambda = ln(2)/half_life is the correct formula, EntityState struct showing the fields the schema must declare
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) -- Section 7 (Error Handling: `Result<T>` everywhere, typed errors, `LumenError` enum definition with exactly 6 variants)
- [API.md](../../../../API.md) -- Schema Definition section (SchemaBuilder usage pattern, Decay enum, Window constructors)
- [ROADMAP.md](../../ROADMAP.md) -- m1p1 acceptance criteria, Milestone 1 UAT scenario (schema definition)
## Spec References
- [docs/specs/11-schema.md](../../../specs/11-schema.md) -- Schema definition API (type system, validation rules, builder pattern), schema versioning and migration, signal name uniqueness constraints, DecaySpec vs DecayModel separation
- [docs/specs/03-signal-system.md](../../../specs/03-signal-system.md) -- Signal type declaration fields (name, target_kind, decay, windows, velocity_enabled), validation constraints (positive half-life, non-empty windows for non-permanent), 40 signal types reference for UAT validation
- [docs/specs/00-architecture-overview.md](../../../specs/00-architecture-overview.md) -- System architecture showing Schema as input to all subsystems (storage, query, ranking), code module map showing `schema/` layout with error.rs and validation.rs
- [docs/specs/01-storage-engine.md](../../../specs/01-storage-engine.md) -- Section 5 (key encoding: signal names used in `SIG:{name}:{window}` storage keys, constraining valid signal name characters)
- [docs/specs/09-ranking-scoring.md](../../../specs/09-ranking-scoring.md) -- Ranking profiles reference signal names and windows by string, confirming signal names must be valid identifiers for query language use
## Implementation Notes
- **No thiserror.** CODING_GUIDELINES.md Section 10 says "Every dependency must justify its existence against 'could we write this in 200 lines?'" The error types are ~100 lines of hand-implemented Display/Error. Adding thiserror would save ~40 lines but add a compile-time dependency. Hand-implement for now; add thiserror if the error hierarchy grows significantly in later milestones.
- **SchemaError derives PartialEq + Eq** for test assertions. This is unusual for errors but justified: these are validation errors, not I/O errors, so equality comparison is meaningful and deterministic.
- **Signal names are globally unique** regardless of target entity kind. There is no `item.view` vs `user.view`. The query language references signals by name alone (`view.velocity(24h)`). This simplifies the schema, storage keys, and query parser.
- **`Schema` is Clone.** In m1p5, when the `Lumen` struct is built, the schema will be wrapped in `Arc<Schema>` for shared ownership. For now, direct ownership and Clone suffice.
- The builder returns `&mut SchemaBuilder` from `add()`, enabling method chaining. This is a common Rust builder pattern (see `reqwest::ClientBuilder`, `tantivy::SchemaBuilder`).

Some files were not shown because too many files have changed in this diff Show More