tidaldb/tidal/CLAUDE.md

Jon Gjengset: I don't ship what I wouldn't trust at 3am during a production incident. Pay attention to what the user says and follow it. Do not make them repeat themselves.

tidalDB

A single-node-first, embeddable Rust database for the personalized content ranking problem. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.

Status: Implemented (M0–M8 shipped). The engine is the tidaldb crate at tidal/, with companion crates tidal-net/, tidal-server/, and tidalctl/ as workspace siblings. ~55K LOC, 1,209 unit tests + 188 integration tests + 5 crash-invariant property tests passing. The API surface is stable for implemented features; breaking changes are possible before 1.0. See CHANGELOG.md for milestone history and known gaps.

Find Your Guide

If you need to...	Read this
Get started quickly	README.md → docs/QUICKSTART.md
Understand the vision	docs/VISION.md
See use cases and surfaces	docs/USE_CASES.md
See sequence diagrams	docs/SEQUENCE.md
Understand the system architecture	docs/ARCHITECTURE.md
Read the component specs	docs/specs/ (00–14)
Look up domain concepts	ai-lookup/index.md
Follow coding standards	docs/CODING_GUIDELINES.md
See the API spec	docs/API.md
Read architectural lessons	docs/thoughts.md
Read technical research	docs/research/

Agents

Agent	Identity	Use when
@tidal-engineer	Jon Gjengset	Implementing features, designing storage internals, building the signal system, debugging correctness issues
@tidal-visionary	Spencer Kimball	Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions
@tidal-researcher	Andy Pavlo	Investigating best practices, surveying prior art, evaluating libraries, producing research documents
@tidal-distributed	Kyle Kingsbury	Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA
@tidal-storyteller	—	Building the marketing site, writing blog posts, crafting public-facing copy

Skills

Phase Lifecycle

Step	Skill	Use when
1. Plan	/milestone	Planning task documents for a milestone phase (orchestrates all 3 agents)
2. Build	/implement	Executing a planned phase task-by-task (delegates to @tidal-engineer)
3. Review	/review	Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer)
4. Accept	/uat	User acceptance testing a reviewed phase (delegates to @tidal-engineer)

Other Skills

Skill	Use when
/tidal-deliver-task	End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept)
/tidal-verify-completion-to-spec	Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement
/develop	Quick implementation work outside the milestone lifecycle
/research [topic]	Investigating best practices, evaluating approaches (delegates to @tidal-researcher)
/roadmap	Building or updating the milestone roadmap (delegates to @tidal-visionary)
/build-site	Creating or iterating on the marketing site
/write-blog	Writing blog posts about progress or architecture
/distribute	Building multi-node distributed system (network transport, cluster mode, cross-node queries)

Core Domain Model

• Entities: Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
• Signals: Typed, timestamped event streams with native decay, velocity, and windowed aggregation
• Relationships: Weighted, directional edges between entities (follows, blocks, interactions)
• Ranking Profiles: Named, versioned scoring functions declared in schema
• Query: Single operation combining retrieval, filtering, ranking, and diversity enforcement

Ports

Dev servers use port range 59520–59529 (e.g. site/ on 59520). Any new tidalDB-adjacent service should claim a slot from this band.

Critical Rules

• Scope: This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
• Embeddings: The database retrieves and ranks over vectors. It does NOT generate them.
• Signals are primitives: Decay, velocity, and windowed aggregation are native — not application logic.
• Single-node first: Embeddable. Scales vertically before horizontally.
• Language: Rust.

Repository Structure

This repository is a Cargo workspace. The embeddable database engine is the tidaldb crate at tidal/, with three companion crates as workspace siblings and example consumers under applications/:

. (workspace root)
├── Cargo.toml           # Workspace manifest
├── tidal/               # The embeddable database engine — package name = "tidaldb"
│   ├── Cargo.toml
│   ├── CLAUDE.md        # This file — project instructions
│   ├── README.md        # Public README + getting-started paths
│   ├── CHANGELOG.md     # Milestone history (M0–M8) + known gaps
│   ├── CONTRIBUTING.md  # Contributor workflow
│   ├── AGENTS.md        # Agent roster + SDLC block
│   ├── src/             # Flat module layout
│   │   ├── cohort/      # Cohort-scoped signal aggregation
│   │   ├── db/          # Top-level TidalDb + builder
│   │   ├── entities/    # Item / User / Creator entity model
│   │   ├── load/        # Bulk load / ingest paths
│   │   ├── query/       # Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST)
│   │   ├── ranking/     # Profile engine, signal scoring, diversity enforcement
│   │   ├── replication/ # WAL-stream replication for cluster mode
│   │   ├── schema/      # Schema builder, validation, signal/profile defs
│   │   ├── session/     # Session + agent context, policies
│   │   ├── signals/     # Signal types, decay, velocity, windowed aggregation
│   │   ├── storage/     # Entity store, signal ledger, inverted index, HNSW
│   │   ├── testing/     # Shared test harness + fixtures
│   │   ├── text/        # Tantivy full-text indexing
│   │   └── wal/         # Write-ahead log + crash recovery
│   ├── benches/         # Performance benchmarks
│   ├── examples/        # quickstart, axum_embedding, actix_embedding, cli_embedding
│   ├── tests/           # Integration and crash-property tests
│   ├── docker/          # cluster / standalone / deploy Dockerfiles
│   ├── ai-lookup/       # Domain concept reference (index.md)
│   ├── site/            # Public marketing site (Next.js)
│   └── docs/            # Specs, API, guides, research, runbooks, ops
│       ├── VISION.md        ARCHITECTURE.md   API.md
│       ├── QUICKSTART.md    SEQUENCE.md       USE_CASES.md
│       ├── CODING_GUIDELINES.md  thoughts.md
│       ├── specs/       # 00–14 component specifications
│       ├── research/    # Deep technical research docs
│       ├── runbooks/    # Operational runbooks (cluster, recovery)
│       ├── ops/         # Monitoring, alerts, capacity planning
│       ├── profiling/   # Flamegraph + scale profiling notes
│       └── planning/    # Per-milestone phase/task planning archive
├── tidal-net/           # Network transport primitives
├── tidal-server/        # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/            # CLI tool for inspecting persisted databases
└── applications/        # Example consumers (forage, iknowyou)

Pre-commit Hooks

The pre-commit hook runs automatically on staged files:

• Rust: cargo fmt (auto-fix + re-stage), cargo clippy -p tidaldb -D warnings, cargo test -p tidaldb --lib
• site/ (Next.js): eslint (if node_modules installed)

Cargo commands target this crate with -p tidaldb from the workspace root (e.g. cargo test -p tidaldb, cargo check -p tidaldb), or equivalently --manifest-path tidal/Cargo.toml. The engine crate lives at tidal/.

Tests must be fast. Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.