> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
> Pay attention to what the user says and follow it. Do not make them repeat themselves.

# tidalDB

A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.

**Status:** Implemented (M0–M8 shipped). The engine is the `tidaldb` crate at `tidal/`, with companion crates `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings. ~55K LOC, 1,209 unit tests + 188 integration tests + 5 crash-invariant property tests passing. The API surface is stable for implemented features; breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and known gaps.

## Find Your Guide

| If you need to... | Read this |
|-------------------|-----------|
| **Get started quickly** | [README.md](README.md) → [docs/QUICKSTART.md](docs/QUICKSTART.md) |
| **Understand the vision** | [docs/VISION.md](docs/VISION.md) |
| **See use cases and surfaces** | [docs/USE_CASES.md](docs/USE_CASES.md) |
| **See sequence diagrams** | [docs/SEQUENCE.md](docs/SEQUENCE.md) |
| **Understand the system architecture** | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) |
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
| **Follow coding standards** | [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) |
| **See the API spec** | [docs/API.md](docs/API.md) |
| **Read architectural lessons** | [docs/thoughts.md](docs/thoughts.md) |
| **Read technical research** | [docs/research/](docs/research/) |

## Agents

| Agent | Identity | Use when |
|-------|----------|----------|
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
| **@tidal-distributed** | Kyle Kingsbury | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |

## Skills

### Phase Lifecycle

| Step | Skill | Use when |
|------|-------|----------|
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |

### Other Skills

| Skill | Use when |
|-------|----------|
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
| `/develop` | Quick implementation work outside the milestone lifecycle |
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
| `/build-site` | Creating or iterating on the marketing site |
| `/write-blog` | Writing blog posts about progress or architecture |
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |

## Core Domain Model

- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement

## Ports

Dev servers use port range **59520–59529** (e.g. `site/` on 59520). Any new tidalDB-adjacent service should claim a slot from this band.

## Critical Rules

- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
- **Single-node first:** Embeddable. Scales vertically before horizontally.
- **Language:** Rust.

## Repository Structure

This repository is a Cargo workspace. The embeddable database engine is the `tidaldb` crate at `tidal/`, with three companion crates as workspace siblings and example consumers under `applications/`:

```
. (workspace root)
├── Cargo.toml           # Workspace manifest
├── tidal/               # The embeddable database engine — package name = "tidaldb"
│   ├── Cargo.toml
│   ├── CLAUDE.md        # This file — project instructions
│   ├── README.md        # Public README + getting-started paths
│   ├── CHANGELOG.md     # Milestone history (M0–M8) + known gaps
│   ├── CONTRIBUTING.md  # Contributor workflow
│   ├── AGENTS.md        # Agent roster + SDLC block
│   ├── src/             # Flat module layout
│   │   ├── cohort/      # Cohort-scoped signal aggregation
│   │   ├── db/          # Top-level TidalDb + builder
│   │   ├── entities/    # Item / User / Creator entity model
│   │   ├── load/        # Bulk load / ingest paths
│   │   ├── query/       # Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST)
│   │   ├── ranking/     # Profile engine, signal scoring, diversity enforcement
│   │   ├── replication/ # WAL-stream replication for cluster mode
│   │   ├── schema/      # Schema builder, validation, signal/profile defs
│   │   ├── session/     # Session + agent context, policies
│   │   ├── signals/     # Signal types, decay, velocity, windowed aggregation
│   │   ├── storage/     # Entity store, signal ledger, inverted index, HNSW
│   │   ├── testing/     # Shared test harness + fixtures
│   │   ├── text/        # Tantivy full-text indexing
│   │   └── wal/         # Write-ahead log + crash recovery
│   ├── benches/         # Performance benchmarks
│   ├── examples/        # quickstart, axum_embedding, actix_embedding, cli_embedding
│   ├── tests/           # Integration and crash-property tests
│   ├── docker/          # cluster / standalone / deploy Dockerfiles
│   ├── ai-lookup/       # Domain concept reference (index.md)
│   ├── site/            # Public marketing site (Next.js)
│   └── docs/            # Specs, API, guides, research, runbooks, ops
│       ├── VISION.md        ARCHITECTURE.md   API.md
│       ├── QUICKSTART.md    SEQUENCE.md       USE_CASES.md
│       ├── CODING_GUIDELINES.md  thoughts.md
│       ├── specs/       # 00–14 component specifications
│       ├── research/    # Deep technical research docs
│       ├── runbooks/    # Operational runbooks (cluster, recovery)
│       ├── ops/         # Monitoring, alerts, capacity planning
│       ├── profiling/   # Flamegraph + scale profiling notes
│       └── planning/    # Per-milestone phase/task planning archive
├── tidal-net/           # Network transport primitives
├── tidal-server/        # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/            # CLI tool for inspecting persisted databases
└── applications/        # Example consumers (forage, iknowyou)
```

## Pre-commit Hooks

The pre-commit hook runs automatically on staged files:
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
- **site/ (Next.js):** `eslint` (if node_modules installed)

Cargo commands target this crate with `-p tidaldb` from the workspace root (e.g. `cargo test -p tidaldb`, `cargo check -p tidaldb`), or equivalently `--manifest-path tidal/Cargo.toml`. The engine crate lives at `tidal/`.

**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.