jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
d923b036af
tidaldb/CLAUDE.md
jx12n c22a3b65a6 docs: withdraw the pre-release "not ready for production" disclaimer 
M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.

- README.md: replace the "Pre-release / not yet recommended for production"
  banner with a production-ready statement; drop "(experimental)" from the
  cluster status bullet; state the post-1.0 versioning posture (additive in
  minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
  docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
  guard against standing up a multi-node fabric by accident rather than a
  readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
  the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
  stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.

Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.

Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.

Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.

Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
2026-07-30 19:03:34 -06:00

143 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
# tidalDB
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
**Status:** Implemented — M0–M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Production-ready — APIs are stable for shipped features; breaking changes ship with a documented migration path. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
## Find Your Guide
| If you need to... | Read this |
|-------------------|-----------|
| **Get started quickly** | [README.md](README.md) → [QUICKSTART.md](QUICKSTART.md) |
| **Build a real app (feed)** | [docs/guides/build-a-feed-app.md](docs/guides/build-a-feed-app.md) |
| **Wire an embedding model** | [docs/guides/embeddings.md](docs/guides/embeddings.md) |
| **Run / deploy the HTTP server** | [docs/guides/server-deployment.md](docs/guides/server-deployment.md) |
| **Deploy on Kubernetes** | [docs/runbooks/kubernetes.md](docs/runbooks/kubernetes.md) + [k8s/](k8s/) |
| **Understand the vision** | [VISION.md](VISION.md) |
| **See use cases and surfaces** | [USE_CASES.md](USE_CASES.md) |
| **See sequence diagrams** | [SEQUENCE.md](SEQUENCE.md) |
| **Understand the system architecture** | [ARCHITECTURE.md](ARCHITECTURE.md) |
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
| **See the API spec** | [API.md](API.md) |
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
| **Browse all engineering docs** | [docs/README.md](docs/README.md) (index of specs, planning, research, reviews, ops, runbooks, profiling) |
| **See the roadmap / milestone history** | [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md) |
| **Read code-review findings** | [docs/reviews/](docs/reviews/) |
| **Operate / monitor in production** | [docs/ops/](docs/ops/), [docs/runbooks/](docs/runbooks/) |
| **Read technical research** | [docs/research/](docs/research/) |
| **Contribute** | [CONTRIBUTING.md](CONTRIBUTING.md) |
## Agents
This is the canonical agent roster. `AGENTS.md` mirrors it for tools that read that file; keep the two in sync.
| Agent | Identity | Model | Use when |
|-------|----------|-------|----------|
| **@tidal-engineer** | Jon Gjengset | opus | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
| **@tidal-visionary** | Spencer Kimball | opus | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
| **@tidal-researcher** | Andy Pavlo | opus | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
| **@tidal-distributed** | Kyle Kingsbury | opus | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
| **@tidal-performance** | Martin Thompson | opus | Performance engineering — low-latency hot-path optimization, lock-free concurrency, honest latency measurement, throughput funnels, mechanical sympathy, scale validation |
| **@tidal-storyteller** | — | sonnet | Building the marketing site, writing blog posts, crafting public-facing copy |
**Utility agent:** `@knowledge-librarian` (sonnet) — classifies, cross-references, and maintains the project knowledge base.
**Vendored team (for the `applications/` consumers, not the database):** `@kai-park` (Aeries full-stack engineer), `@kaya-osei` (Aeries product designer), and `@mira-vasquez` (Aeries product visionary) back the `aeries-*` skills. They are scoped to companion-app work, not the tidalDB engine.
## Skills
### Phase Lifecycle
| Step | Skill | Use when |
|------|-------|----------|
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
### Other Skills
| Skill | Use when |
|-------|----------|
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
| `/develop` | Quick implementation work outside the milestone lifecycle |
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
| `/build-site` | Creating or iterating on the marketing site |
| `/write-blog` | Writing blog posts about progress or architecture |
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |
## Core Domain Model
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
## Ports
Dev servers use port range **59520–59529** (e.g. `site/` on 59520).
## Critical Rules
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
- **Single-node first:** Embeddable. Scales vertically before horizontally.
- **Language:** Rust.
- **Docs have two canonical homes:** top-level `*.md` and `docs/`. Edit the canonical file — never a per-crate mirror. `.sdlc/` is live SDLC tooling and `applications/*/` docs belong to those consumer products; neither is part of the database doc set.
## Repository Structure
This repository is a standalone Cargo workspace (members: `tidal`, `tidal-net`, `tidalctl`,
`tidal-server`, `tidal-stress`, and the `applications/` consumers). **Documentation has exactly two homes:**
the top-level `*.md` files and `docs/`. Do not create per-crate doc mirrors (e.g. `tidal/docs/`,
`tidal/ai-lookup/`, `tidal/site/`) — those were a stale duplicate and were consolidated away.
```
. # Workspace root — canonical docs + config
├── Cargo.toml # Workspace manifest (9 members)
├── CLAUDE.md AGENTS.md README.md CONTRIBUTING.md CHANGELOG.md
├── VISION.md USE_CASES.md SEQUENCE.md ARCHITECTURE.md
├── API.md QUICKSTART.md CODING_GUIDELINES.md thoughts.md
├── ai-lookup/ # Domain concept reference (index.md + features/ + services/)
├── docs/ # Engineering docs (see docs/README.md for the index)
│ ├── specs/ # 00–14 component specifications
│ ├── planning/ # ROADMAP.md + per-milestone phase/task archive
│ ├── research/ # Deep technical research docs
│ ├── reviews/ # Code-review passes
│ ├── ops/ # Monitoring, alerts, capacity planning
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
│ └── profiling/ # Flamegraph + scale profiling notes
├── .claude/ # Claude Code config — agents/ and skills/ (canonical; .agents/ removed)
├── tidal/ # The `tidaldb` engine crate (CLAUDE.md here is a thin crate pointer)
│ ├── Cargo.toml
│ ├── src/ # cohort, db, entities, governance, load, query, ranking,
│ │ # replication, schema, session, signals, storage, testing, text, wal
│ ├── benches/ examples/ tests/ docker/
├── tidal-net/ # Network transport primitives (gRPC, WAL shipping)
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/ # CLI for inspecting persisted databases
├── tidal-stress/ # Open-loop HTTP load generator + capacity ramp (thepeach feed workload)
├── applications/ # Example consumers (forage, iknowyou)
└── site/ # Public marketing site (Next.js, dev on 59520)
```
## Pre-commit Hooks
The pre-commit hook runs automatically on staged files:
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
- **site/ (Next.js):** `eslint` (if node_modules installed)
- **Docs:** `scripts/check-docs.sh` — fails if a doc mirror reappears (`tidal/docs/`, `.ai/`, `.agents/skills/`), if CLAUDE.md's workspace members drift from `Cargo.toml`, or if a canonical cross-reference breaks.
Cargo commands target the engine crate with `-p tidaldb` from the workspace root (equivalently `--manifest-path tidal/Cargo.toml`).
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
Reference in New Issue View Git Blame Copy Permalink