tidaldb/CLAUDE.md

Jon Gjengset: I don't ship what I wouldn't trust at 3am during a production incident. Pay attention to what the user says and follow it. Do not make them repeat themselves.

tidalDB

A single-node-first, embeddable Rust database for the personalized content ranking problem. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.

Status: Implemented — M0–M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the tidaldb crate at tidal/, with tidal-net/, tidal-server/, and tidalctl/ as workspace siblings and example consumers under applications/. Production-ready — APIs are stable for shipped features; breaking changes ship with a documented migration path. See CHANGELOG.md for milestone history and docs/planning/ROADMAP.md for status and known gaps.

Find Your Guide

If you need to...	Read this
Get started quickly	README.md → QUICKSTART.md
Build a real app (feed)	docs/guides/build-a-feed-app.md
Wire an embedding model	docs/guides/embeddings.md
Run / deploy the HTTP server	docs/guides/server-deployment.md
Deploy on Kubernetes	docs/runbooks/kubernetes.md + k8s/
Understand the vision	VISION.md
See use cases and surfaces	USE_CASES.md
See sequence diagrams	SEQUENCE.md
Understand the system architecture	ARCHITECTURE.md
Look up domain concepts	ai-lookup/index.md
Follow coding standards	CODING_GUIDELINES.md
See the API spec	API.md
Read architectural lessons	thoughts.md
Read the component specs	docs/specs/ (00–14)
Browse all engineering docs	docs/README.md (index of specs, planning, research, reviews, ops, runbooks, profiling)
See the roadmap / milestone history	docs/planning/ROADMAP.md, CHANGELOG.md
Read code-review findings	docs/reviews/
Operate / monitor in production	docs/ops/, docs/runbooks/
Read technical research	docs/research/
Contribute	CONTRIBUTING.md

Agents

This is the canonical agent roster. AGENTS.md mirrors it for tools that read that file; keep the two in sync.

Agent	Identity	Model	Use when
@tidal-engineer	Jon Gjengset	opus	Implementing features, designing storage internals, building the signal system, debugging correctness issues
@tidal-visionary	Spencer Kimball	opus	Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions
@tidal-researcher	Andy Pavlo	opus	Investigating best practices, surveying prior art, evaluating libraries, producing research documents
@tidal-distributed	Kyle Kingsbury	opus	Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA
@tidal-performance	Martin Thompson	opus	Performance engineering — low-latency hot-path optimization, lock-free concurrency, honest latency measurement, throughput funnels, mechanical sympathy, scale validation
@tidal-storyteller	—	sonnet	Building the marketing site, writing blog posts, crafting public-facing copy

Utility agent: @knowledge-librarian (sonnet) — classifies, cross-references, and maintains the project knowledge base.

Vendored team (for the applications/ consumers, not the database): @kai-park (Aeries full-stack engineer), @kaya-osei (Aeries product designer), and @mira-vasquez (Aeries product visionary) back the aeries-* skills. They are scoped to companion-app work, not the tidalDB engine.

Skills

Phase Lifecycle

Step	Skill	Use when
1. Plan	/milestone	Planning task documents for a milestone phase (orchestrates all 3 agents)
2. Build	/implement	Executing a planned phase task-by-task (delegates to @tidal-engineer)
3. Review	/review	Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer)
4. Accept	/uat	User acceptance testing a reviewed phase (delegates to @tidal-engineer)

Other Skills

Skill	Use when
/tidal-deliver-task	End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept)
/tidal-verify-completion-to-spec	Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement
/develop	Quick implementation work outside the milestone lifecycle
/research [topic]	Investigating best practices, evaluating approaches (delegates to @tidal-researcher)
/roadmap	Building or updating the milestone roadmap (delegates to @tidal-visionary)
/build-site	Creating or iterating on the marketing site
/write-blog	Writing blog posts about progress or architecture
/distribute	Building multi-node distributed system (network transport, cluster mode, cross-node queries)

Core Domain Model

• Entities: Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
• Signals: Typed, timestamped event streams with native decay, velocity, and windowed aggregation
• Relationships: Weighted, directional edges between entities (follows, blocks, interactions)
• Ranking Profiles: Named, versioned scoring functions declared in schema
• Query: Single operation combining retrieval, filtering, ranking, and diversity enforcement

Ports

Dev servers use port range 59520–59529 (e.g. site/ on 59520).

Critical Rules

• Scope: This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
• Embeddings: The database retrieves and ranks over vectors. It does NOT generate them.
• Signals are primitives: Decay, velocity, and windowed aggregation are native — not application logic.
• Single-node first: Embeddable. Scales vertically before horizontally.
• Language: Rust.
• Docs have two canonical homes: top-level *.md and docs/. Edit the canonical file — never a per-crate mirror. .sdlc/ is live SDLC tooling and applications/*/ docs belong to those consumer products; neither is part of the database doc set.
• Every deploy MUST be verified through the Playwright harness, and the harness MUST be maintained. See ## Deploy Verification.

Deploy Verification

Non-negotiable: no deploy is done until this suite is green. Run it after every roll.

export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
npm install && npx playwright install chromium   # first time
cargo build -p tidalctl                          # section 7 needs it
npm run test:e2e                                 # 34 checks, ~50s, LIVE cluster
npm run test:e2e:semantics                       # 5 checks, hermetic, no cluster

Video walkthrough, in this order (demo:promote is NOT optional — it is the step that re-stamps the manifest from the live StatefulSet):

npm run test:demo && npm run demo:promote && npm run demo:preflight && npm run demo:render
npm run demo:promote -- --audited "<what you checked>"   # after opening the frames

Full detail: docs/runbooks/deploy-verification.md.

Maintaining it is part of the change, not follow-up

This suite deliberately pins current reality, including defects and gaps. So a correct improvement WILL turn it red. That failure is the harness working.

• A test fails claiming "good news — invert this assertion" → invert it, and fix the runbook section it names. Do not skip, retry, or loosen it.
• Never add retries to playwright.config.ts. retries: 0 is deliberate: a live check that only passes on attempt two has told you something true.
• A flake means the assertion is wrong, not that the suite is unreliable. Readiness is polled (waitForPodsReady) because reseed_self_restart: true makes a bounded exit(0)/reinstall cycle designed behavior.
• Calibrate thresholds against a measurement, never a guess. A reseed restart produces ~1200-2000 consecutive ship failures; a count ceiling cannot separate that from a stuck shipper. Assert whether the burst recovered.
• Stale evidence is worse than none: the promoted captures and rendered video sat two image rolls behind while preflight reported "audited perfect".

Repository Structure

This repository is a standalone Cargo workspace (members: tidal, tidal-net, tidalctl, tidal-server, tidal-stress, and the applications/ consumers). Documentation has exactly two homes: the top-level *.md files and docs/. Do not create per-crate doc mirrors (e.g. tidal/docs/, tidal/ai-lookup/, tidal/site/) — those were a stale duplicate and were consolidated away.

.                    # Workspace root — canonical docs + config
├── Cargo.toml       # Workspace manifest (9 members)
├── CLAUDE.md  AGENTS.md  README.md  CONTRIBUTING.md  CHANGELOG.md
├── VISION.md  USE_CASES.md  SEQUENCE.md  ARCHITECTURE.md
├── API.md  QUICKSTART.md  CODING_GUIDELINES.md  thoughts.md
├── ai-lookup/       # Domain concept reference (index.md + features/ + services/)
├── docs/            # Engineering docs (see docs/README.md for the index)
│   ├── specs/       # 00–14 component specifications
│   ├── planning/    # ROADMAP.md + per-milestone phase/task archive
│   ├── research/    # Deep technical research docs
│   ├── reviews/     # Code-review passes
│   ├── ops/         # Monitoring, alerts, capacity planning
│   ├── runbooks/    # Operational runbooks (cluster, recovery)
│   └── profiling/   # Flamegraph + scale profiling notes
├── .claude/         # Claude Code config — agents/ and skills/ (canonical; .agents/ removed)
├── tidal/           # The `tidaldb` engine crate (CLAUDE.md here is a thin crate pointer)
│   ├── Cargo.toml
│   ├── src/         # cohort, db, entities, governance, load, query, ranking,
│   │                #   replication, schema, session, signals, storage, testing, text, wal
│   ├── benches/  examples/  tests/  docker/
├── tidal-net/       # Network transport primitives (gRPC, WAL shipping)
├── tidal-server/    # Standalone Axum HTTP server (standalone + cluster modes)
├── tidalctl/        # CLI for inspecting persisted databases
├── tidal-stress/    # Open-loop HTTP load generator + capacity ramp (thepeach feed workload)
├── applications/    # Example consumers (forage, iknowyou)
└── site/            # Public marketing site (Next.js, dev on 59520)

Pre-commit Hooks

The pre-commit hook runs automatically on staged files:

• Rust: cargo fmt (auto-fix + re-stage), cargo clippy -p tidaldb -D warnings, cargo test -p tidaldb --lib
• site/ (Next.js): eslint (if node_modules installed)
• Docs: scripts/check-docs.sh — fails if a doc mirror reappears (tidal/docs/, .ai/, .agents/skills/), if CLAUDE.md's workspace members drift from Cargo.toml, or if a canonical cross-reference breaks.

Cargo commands target the engine crate with -p tidaldb from the workspace root (equivalently --manifest-path tidal/Cargo.toml).

Tests must be fast. Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.