3bcfb3c576
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
140 lines
8.9 KiB
Markdown
140 lines
8.9 KiB
Markdown
> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
|
||
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
|
||
|
||
# tidalDB
|
||
|
||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||
|
||
**Status:** Implemented (M0–M8 shipped). The engine is the `tidaldb` crate at `tidal/`, with companion crates `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings. ~55K LOC, 1,209 unit tests + 188 integration tests + 5 crash-invariant property tests passing. The API surface is stable for implemented features; breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and known gaps.
|
||
|
||
## Find Your Guide
|
||
|
||
| If you need to... | Read this |
|
||
|-------------------|-----------|
|
||
| **Get started quickly** | [README.md](README.md) → [docs/QUICKSTART.md](docs/QUICKSTART.md) |
|
||
| **Understand the vision** | [docs/VISION.md](docs/VISION.md) |
|
||
| **See use cases and surfaces** | [docs/USE_CASES.md](docs/USE_CASES.md) |
|
||
| **See sequence diagrams** | [docs/SEQUENCE.md](docs/SEQUENCE.md) |
|
||
| **Understand the system architecture** | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) |
|
||
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
|
||
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
|
||
| **Follow coding standards** | [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) |
|
||
| **See the API spec** | [docs/API.md](docs/API.md) |
|
||
| **Read architectural lessons** | [docs/thoughts.md](docs/thoughts.md) |
|
||
| **Read technical research** | [docs/research/](docs/research/) |
|
||
|
||
## Agents
|
||
|
||
| Agent | Identity | Use when |
|
||
|-------|----------|----------|
|
||
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||
| **@tidal-distributed** | Kyle Kingsbury | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||
|
||
## Skills
|
||
|
||
### Phase Lifecycle
|
||
|
||
| Step | Skill | Use when |
|
||
|------|-------|----------|
|
||
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
|
||
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
|
||
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
|
||
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
|
||
|
||
### Other Skills
|
||
|
||
| Skill | Use when |
|
||
|-------|----------|
|
||
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
|
||
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
|
||
| `/develop` | Quick implementation work outside the milestone lifecycle |
|
||
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
|
||
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
|
||
| `/build-site` | Creating or iterating on the marketing site |
|
||
| `/write-blog` | Writing blog posts about progress or architecture |
|
||
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |
|
||
|
||
## Core Domain Model
|
||
|
||
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
|
||
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
|
||
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
|
||
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
|
||
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
|
||
|
||
## Ports
|
||
|
||
Dev servers use port range **59520–59529** (e.g. `site/` on 59520). Any new tidalDB-adjacent service should claim a slot from this band.
|
||
|
||
## Critical Rules
|
||
|
||
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
|
||
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
|
||
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
|
||
- **Single-node first:** Embeddable. Scales vertically before horizontally.
|
||
- **Language:** Rust.
|
||
|
||
## Repository Structure
|
||
|
||
This repository is a Cargo workspace. The embeddable database engine is the `tidaldb` crate at `tidal/`, with three companion crates as workspace siblings and example consumers under `applications/`:
|
||
|
||
```
|
||
. (workspace root)
|
||
├── Cargo.toml # Workspace manifest
|
||
├── tidal/ # The embeddable database engine — package name = "tidaldb"
|
||
│ ├── Cargo.toml
|
||
│ ├── CLAUDE.md # This file — project instructions
|
||
│ ├── README.md # Public README + getting-started paths
|
||
│ ├── CHANGELOG.md # Milestone history (M0–M8) + known gaps
|
||
│ ├── CONTRIBUTING.md # Contributor workflow
|
||
│ ├── AGENTS.md # Agent roster + SDLC block
|
||
│ ├── src/ # Flat module layout
|
||
│ │ ├── cohort/ # Cohort-scoped signal aggregation
|
||
│ │ ├── db/ # Top-level TidalDb + builder
|
||
│ │ ├── entities/ # Item / User / Creator entity model
|
||
│ │ ├── load/ # Bulk load / ingest paths
|
||
│ │ ├── query/ # Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST)
|
||
│ │ ├── ranking/ # Profile engine, signal scoring, diversity enforcement
|
||
│ │ ├── replication/ # WAL-stream replication for cluster mode
|
||
│ │ ├── schema/ # Schema builder, validation, signal/profile defs
|
||
│ │ ├── session/ # Session + agent context, policies
|
||
│ │ ├── signals/ # Signal types, decay, velocity, windowed aggregation
|
||
│ │ ├── storage/ # Entity store, signal ledger, inverted index, HNSW
|
||
│ │ ├── testing/ # Shared test harness + fixtures
|
||
│ │ ├── text/ # Tantivy full-text indexing
|
||
│ │ └── wal/ # Write-ahead log + crash recovery
|
||
│ ├── benches/ # Performance benchmarks
|
||
│ ├── examples/ # quickstart, axum_embedding, actix_embedding, cli_embedding
|
||
│ ├── tests/ # Integration and crash-property tests
|
||
│ ├── docker/ # cluster / standalone / deploy Dockerfiles
|
||
│ ├── ai-lookup/ # Domain concept reference (index.md)
|
||
│ ├── site/ # Public marketing site (Next.js)
|
||
│ └── docs/ # Specs, API, guides, research, runbooks, ops
|
||
│ ├── VISION.md ARCHITECTURE.md API.md
|
||
│ ├── QUICKSTART.md SEQUENCE.md USE_CASES.md
|
||
│ ├── CODING_GUIDELINES.md thoughts.md
|
||
│ ├── specs/ # 00–14 component specifications
|
||
│ ├── research/ # Deep technical research docs
|
||
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
|
||
│ ├── ops/ # Monitoring, alerts, capacity planning
|
||
│ ├── profiling/ # Flamegraph + scale profiling notes
|
||
│ └── planning/ # Per-milestone phase/task planning archive
|
||
├── tidal-net/ # Network transport primitives
|
||
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
|
||
├── tidalctl/ # CLI tool for inspecting persisted databases
|
||
└── applications/ # Example consumers (forage, iknowyou)
|
||
```
|
||
|
||
## Pre-commit Hooks
|
||
|
||
The pre-commit hook runs automatically on staged files:
|
||
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
|
||
- **site/ (Next.js):** `eslint` (if node_modules installed)
|
||
|
||
Cargo commands target this crate with `-p tidaldb` from the workspace root (e.g. `cargo test -p tidaldb`, `cargo check -p tidaldb`), or equivalently `--manifest-path tidal/Cargo.toml`. The engine crate lives at `tidal/`.
|
||
|
||
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
|