> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
> Pay attention to what the user says and follow it. Do not make them repeat themselves.

# tidalDB

A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.

**Status:** Implemented — M0–M10 shipped (embeddable engine + multi-region cluster mode). This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.

## Find Your Guide

| If you need to... | Read this |
|-------------------|-----------|
| **Get started quickly** | [README.md](README.md) → [QUICKSTART.md](QUICKSTART.md) |
| **Understand the vision** | [VISION.md](VISION.md) |
| **See use cases and surfaces** | [USE_CASES.md](USE_CASES.md) |
| **See sequence diagrams** | [SEQUENCE.md](SEQUENCE.md) |
| **Understand the system architecture** | [ARCHITECTURE.md](ARCHITECTURE.md) |
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
| **See the API spec** | [API.md](API.md) |
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
| **Browse all engineering docs** | [docs/README.md](docs/README.md) (index of specs, planning, research, reviews, ops, runbooks, profiling) |
| **See the roadmap / milestone history** | [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md) |
| **Read code-review findings** | [docs/reviews/](docs/reviews/) |
| **Operate / monitor in production** | [docs/ops/](docs/ops/), [docs/runbooks/](docs/runbooks/) |
| **Read technical research** | [docs/research/](docs/research/) |
| **Contribute** | [CONTRIBUTING.md](CONTRIBUTING.md) |

## Agents

This is the canonical agent roster. `AGENTS.md` mirrors it for tools that read that file; keep the two in sync.

| Agent | Identity | Model | Use when |
|-------|----------|-------|----------|
| **@tidal-engineer** | Jon Gjengset | opus | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
| **@tidal-visionary** | Spencer Kimball | opus | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
| **@tidal-researcher** | Andy Pavlo | opus | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
| **@tidal-distributed** | Kyle Kingsbury | opus | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
| **@tidal-storyteller** | — | sonnet | Building the marketing site, writing blog posts, crafting public-facing copy |

**Utility agent:** `@knowledge-librarian` (sonnet) — classifies, cross-references, and maintains the project knowledge base.

**Vendored team (for the `applications/` consumers, not the database):** `@kai-park` (Aeries full-stack engineer), `@kaya-osei` (Aeries product designer), and `@mira-vasquez` (Aeries product visionary) back the `aeries-*` skills. They are scoped to companion-app work, not the tidalDB engine.

## Skills

### Phase Lifecycle

| Step | Skill | Use when |
|------|-------|----------|
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |

### Other Skills

| Skill | Use when |
|-------|----------|
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
| `/develop` | Quick implementation work outside the milestone lifecycle |
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
| `/build-site` | Creating or iterating on the marketing site |
| `/write-blog` | Writing blog posts about progress or architecture |
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |

## Core Domain Model

- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement

## Ports

Dev servers use port range **59520–59529** (e.g. `site/` on 59520).

## Critical Rules

- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
- **Single-node first:** Embeddable. Scales vertically before horizontally.
- **Language:** Rust.
- **Docs have two canonical homes:** top-level `*.md` and `docs/`. Edit the canonical file — never a per-crate mirror. `.sdlc/` is live SDLC tooling and `applications/*/` docs belong to those consumer products; neither is part of the database doc set.

## Repository Structure

This repository is a standalone Cargo workspace (members: `tidal`, `tidal-net`, `tidalctl`,
`tidal-server`, and the `applications/` consumers). **Documentation has exactly two homes:**
the top-level `*.md` files and `docs/`. Do not create per-crate doc mirrors (e.g. `tidal/docs/`,
`tidal/ai-lookup/`, `tidal/site/`) — those were a stale duplicate and were consolidated away.

```
.                    # Workspace root — canonical docs + config
├── Cargo.toml       # Workspace manifest (8 members)
├── CLAUDE.md  AGENTS.md  README.md  CONTRIBUTING.md  CHANGELOG.md
├── VISION.md  USE_CASES.md  SEQUENCE.md  ARCHITECTURE.md
├── API.md  QUICKSTART.md  CODING_GUIDELINES.md  thoughts.md
├── ai-lookup/       # Domain concept reference (index.md + features/ + services/)
├── docs/            # Engineering docs (see docs/README.md for the index)
│   ├── specs/       # 00–14 component specifications
│   ├── planning/    # ROADMAP.md + per-milestone phase/task archive
│   ├── research/    # Deep technical research docs
│   ├── reviews/     # Code-review passes
│   ├── ops/         # Monitoring, alerts, capacity planning
│   ├── runbooks/    # Operational runbooks (cluster, recovery)
│   └── profiling/   # Flamegraph + scale profiling notes
├── .claude/         # Claude Code config — agents/ and skills/ (canonical; .agents/ removed)
├── tidal/           # The `tidaldb` engine crate (CLAUDE.md here is a thin crate pointer)
│   ├── Cargo.toml
│   ├── src/         # cohort, db, entities, governance, load, query, ranking,
│   │                #   replication, schema, session, signals, storage, testing, text, wal
│   ├── benches/  examples/  tests/  docker/
├── tidal-net/       # Network transport primitives (gRPC, WAL shipping)
├── tidal-server/    # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/        # CLI for inspecting persisted databases
├── applications/    # Example consumers (forage, iknowyou)
└── site/            # Public marketing site (Next.js, dev on 59520)
```

## Pre-commit Hooks

The pre-commit hook runs automatically on staged files:
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
- **site/ (Next.js):** `eslint` (if node_modules installed)
- **Docs:** `scripts/check-docs.sh` — fails if a doc mirror reappears (`tidal/docs/`, `.ai/`, `.agents/skills/`), if CLAUDE.md's workspace members drift from `Cargo.toml`, or if a canonical cross-reference breaks.

Cargo commands target the engine crate with `-p tidaldb` from the workspace root (equivalently `--manifest-path tidal/Cargo.toml`).

**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.