1092d34c39
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly
141 lines
9.6 KiB
Markdown
141 lines
9.6 KiB
Markdown
> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
|
||
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
|
||
|
||
# tidalDB
|
||
|
||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||
|
||
**Status:** Implemented — M0–M10 shipped (embeddable engine + multi-region cluster mode). This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
|
||
|
||
## Find Your Guide
|
||
|
||
| If you need to... | Read this |
|
||
|-------------------|-----------|
|
||
| **Get started quickly** | [README.md](README.md) → [QUICKSTART.md](QUICKSTART.md) |
|
||
| **Build a real app (feed)** | [docs/guides/build-a-feed-app.md](docs/guides/build-a-feed-app.md) |
|
||
| **Wire an embedding model** | [docs/guides/embeddings.md](docs/guides/embeddings.md) |
|
||
| **Run / deploy the HTTP server** | [docs/guides/server-deployment.md](docs/guides/server-deployment.md) |
|
||
| **Deploy on Kubernetes** | [docs/runbooks/kubernetes.md](docs/runbooks/kubernetes.md) + [k8s/](k8s/) |
|
||
| **Understand the vision** | [VISION.md](VISION.md) |
|
||
| **See use cases and surfaces** | [USE_CASES.md](USE_CASES.md) |
|
||
| **See sequence diagrams** | [SEQUENCE.md](SEQUENCE.md) |
|
||
| **Understand the system architecture** | [ARCHITECTURE.md](ARCHITECTURE.md) |
|
||
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
|
||
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
|
||
| **See the API spec** | [API.md](API.md) |
|
||
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
|
||
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
|
||
| **Browse all engineering docs** | [docs/README.md](docs/README.md) (index of specs, planning, research, reviews, ops, runbooks, profiling) |
|
||
| **See the roadmap / milestone history** | [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md) |
|
||
| **Read code-review findings** | [docs/reviews/](docs/reviews/) |
|
||
| **Operate / monitor in production** | [docs/ops/](docs/ops/), [docs/runbooks/](docs/runbooks/) |
|
||
| **Read technical research** | [docs/research/](docs/research/) |
|
||
| **Contribute** | [CONTRIBUTING.md](CONTRIBUTING.md) |
|
||
|
||
## Agents
|
||
|
||
This is the canonical agent roster. `AGENTS.md` mirrors it for tools that read that file; keep the two in sync.
|
||
|
||
| Agent | Identity | Model | Use when |
|
||
|-------|----------|-------|----------|
|
||
| **@tidal-engineer** | Jon Gjengset | opus | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||
| **@tidal-visionary** | Spencer Kimball | opus | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||
| **@tidal-researcher** | Andy Pavlo | opus | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||
| **@tidal-distributed** | Kyle Kingsbury | opus | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||
| **@tidal-storyteller** | — | sonnet | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||
|
||
**Utility agent:** `@knowledge-librarian` (sonnet) — classifies, cross-references, and maintains the project knowledge base.
|
||
|
||
**Vendored team (for the `applications/` consumers, not the database):** `@kai-park` (Aeries full-stack engineer), `@kaya-osei` (Aeries product designer), and `@mira-vasquez` (Aeries product visionary) back the `aeries-*` skills. They are scoped to companion-app work, not the tidalDB engine.
|
||
|
||
## Skills
|
||
|
||
### Phase Lifecycle
|
||
|
||
| Step | Skill | Use when |
|
||
|------|-------|----------|
|
||
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
|
||
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
|
||
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
|
||
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
|
||
|
||
### Other Skills
|
||
|
||
| Skill | Use when |
|
||
|-------|----------|
|
||
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
|
||
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
|
||
| `/develop` | Quick implementation work outside the milestone lifecycle |
|
||
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
|
||
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
|
||
| `/build-site` | Creating or iterating on the marketing site |
|
||
| `/write-blog` | Writing blog posts about progress or architecture |
|
||
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |
|
||
|
||
## Core Domain Model
|
||
|
||
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
|
||
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
|
||
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
|
||
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
|
||
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
|
||
|
||
## Ports
|
||
|
||
Dev servers use port range **59520–59529** (e.g. `site/` on 59520).
|
||
|
||
## Critical Rules
|
||
|
||
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
|
||
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
|
||
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
|
||
- **Single-node first:** Embeddable. Scales vertically before horizontally.
|
||
- **Language:** Rust.
|
||
- **Docs have two canonical homes:** top-level `*.md` and `docs/`. Edit the canonical file — never a per-crate mirror. `.sdlc/` is live SDLC tooling and `applications/*/` docs belong to those consumer products; neither is part of the database doc set.
|
||
|
||
## Repository Structure
|
||
|
||
This repository is a standalone Cargo workspace (members: `tidal`, `tidal-net`, `tidalctl`,
|
||
`tidal-server`, and the `applications/` consumers). **Documentation has exactly two homes:**
|
||
the top-level `*.md` files and `docs/`. Do not create per-crate doc mirrors (e.g. `tidal/docs/`,
|
||
`tidal/ai-lookup/`, `tidal/site/`) — those were a stale duplicate and were consolidated away.
|
||
|
||
```
|
||
. # Workspace root — canonical docs + config
|
||
├── Cargo.toml # Workspace manifest (8 members)
|
||
├── CLAUDE.md AGENTS.md README.md CONTRIBUTING.md CHANGELOG.md
|
||
├── VISION.md USE_CASES.md SEQUENCE.md ARCHITECTURE.md
|
||
├── API.md QUICKSTART.md CODING_GUIDELINES.md thoughts.md
|
||
├── ai-lookup/ # Domain concept reference (index.md + features/ + services/)
|
||
├── docs/ # Engineering docs (see docs/README.md for the index)
|
||
│ ├── specs/ # 00–14 component specifications
|
||
│ ├── planning/ # ROADMAP.md + per-milestone phase/task archive
|
||
│ ├── research/ # Deep technical research docs
|
||
│ ├── reviews/ # Code-review passes
|
||
│ ├── ops/ # Monitoring, alerts, capacity planning
|
||
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
|
||
│ └── profiling/ # Flamegraph + scale profiling notes
|
||
├── .claude/ # Claude Code config — agents/ and skills/ (canonical; .agents/ removed)
|
||
├── tidal/ # The `tidaldb` engine crate (CLAUDE.md here is a thin crate pointer)
|
||
│ ├── Cargo.toml
|
||
│ ├── src/ # cohort, db, entities, governance, load, query, ranking,
|
||
│ │ # replication, schema, session, signals, storage, testing, text, wal
|
||
│ ├── benches/ examples/ tests/ docker/
|
||
├── tidal-net/ # Network transport primitives (gRPC, WAL shipping)
|
||
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
|
||
├── tidalctl/ # CLI for inspecting persisted databases
|
||
├── applications/ # Example consumers (forage, iknowyou)
|
||
└── site/ # Public marketing site (Next.js, dev on 59520)
|
||
```
|
||
|
||
## Pre-commit Hooks
|
||
|
||
The pre-commit hook runs automatically on staged files:
|
||
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
|
||
- **site/ (Next.js):** `eslint` (if node_modules installed)
|
||
- **Docs:** `scripts/check-docs.sh` — fails if a doc mirror reappears (`tidal/docs/`, `.ai/`, `.agents/skills/`), if CLAUDE.md's workspace members drift from `Cargo.toml`, or if a canonical cross-reference breaks.
|
||
|
||
Cargo commands target the engine crate with `-p tidaldb` from the workspace root (equivalently `--manifest-path tidal/Cargo.toml`).
|
||
|
||
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
|