tidaldb/CLAUDE.md

> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
> Pay attention to what the user says and follow it. Do not make them repeat themselves.

# tidalDB

A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.

**Status:** Vision and specification phase. No implementation yet.

## Find Your Guide

| If you need to... | Read this |
|-------------------|-----------|
| **Get started quickly** | [README.md](README.md) → [QUICKSTART.md](QUICKSTART.md) |
| **Understand the vision** | [VISION.md](VISION.md) |
| **See use cases and surfaces** | [USE_CASES.md](USE_CASES.md) |
| **See sequence diagrams** | [SEQUENCE.md](SEQUENCE.md) |
| **Understand the system architecture** | [ARCHITECTURE.md](ARCHITECTURE.md) |
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
| **See the API spec** | [API.md](API.md) |
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
| **Read technical research** | [docs/research/](docs/research/) |

## Agents

| Agent | Identity | Use when |
|-------|----------|----------|
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |

## Skills

### Phase Lifecycle

| Step | Skill | Use when |
|------|-------|----------|
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |

### Other Skills

| Skill | Use when |
|-------|----------|
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
| `/develop` | Quick implementation work outside the milestone lifecycle |
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
| `/build-site` | Creating or iterating on the marketing site |
| `/write-blog` | Writing blog posts about progress or architecture |

## Core Domain Model

- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement

## Ports

Dev servers use port range **59520–59529** (e.g. `site/` on 59520).

## Critical Rules

- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
- **Single-node first:** Embeddable. Scales vertically before horizontally.
- **Language:** Rust.

## Repository Structure

```
.                    # Top-level docs and configuration
├── CLAUDE.md        # This file — project instructions
├── VISION.md        # Product vision and thesis
├── USE_CASES.md     # 14 use cases, all discovery surfaces
├── SEQUENCE.md      # Data flow sequence diagrams
├── CODING_GUIDELINES.md  # Engineering standards
├── API.md           # API specification
├── thoughts.md      # Architectural lessons from sister projects
├── ai-lookup/       # Domain concept reference
├── docs/            # Research and documentation
│   └── research/    # Deep technical research docs
├── .claude/         # Claude Code configuration
│   ├── agents/      # Agent definitions
│   └── skills/      # Skill definitions
├── tidal/           # Rust database engine
│   ├── Cargo.toml
│   ├── src/
│   │   ├── storage/ # Entity store, signal ledger, inverted index, HNSW
│   │   ├── query/   # Query parser, planner, executor
│   │   ├── ranking/ # Profile engine, signal scoring, diversity enforcement
│   │   ├── signals/ # Signal types, decay, velocity, windowed aggregation
│   │   └── schema/  # Schema definition, validation, migrations
│   ├── benches/     # Performance benchmarks
│   └── tests/       # Integration and property tests
└── site/            # Public marketing site (Next.js)
```

## Pre-commit Hooks

The pre-commit hook runs automatically on staged files:
- **tidal/ (Rust):** `cargo fmt` (auto-fix + re-stage), `cargo clippy -D warnings`, `cargo test --lib`
- **site/ (Next.js):** `eslint` (if node_modules installed)

All cargo commands use `--manifest-path tidal/Cargo.toml` since the Rust project is not at repo root.

**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.