4f076c927d
## M0p1 — Embeddable Runtime Skeleton (329 tests)
- TidalDb with builder(), health_check(), close(), and Drop-based cleanup
- TidalDbBuilder fluent API: ephemeral(), with_data_dir(), wal_dir(), cache_dir()
- Config, StorageMode, ConfigError types; Config(ConfigError) variant on LumenError
- Paths: single source of truth for directory layout (wal, items, users, creators, cache)
- TempTidalHome: test isolation helper gated behind #[cfg(test)] / test-utils feature
- 8 integration tests: tests/sandboxed_storage.rs
## M0p2 — Tooling & Diagnostics (349 tests)
- Workspace root Cargo.toml (members: ["tidal", "tidalctl"])
- tidal/build.rs: BUILD_HASH from GIT_HASH with option_env!() fallback to "dev"
- MetricsState: always-compiled Arc-shared atomics (uptime, health_ok)
- MetricsHandle (metrics feature): hand-rolled TcpListener HTTP, zero new deps
- GET /healthz → {"status":"ok","uptime_secs":N}
- GET /metrics → Prometheus text (tidaldb_uptime_seconds, health_ok, info)
- TidalDbBuilder.enable_metrics(addr) starts background metrics thread
- tidalctl binary: status + paths commands, manual std::env::args() parsing
- 7 metrics integration tests, 9 tidalctl CLI tests
## m1p4 Signal Ledger (in-progress)
- SignalLedger: DashMap<(EntityId, SignalTypeId), EntitySignalEntry>, WAL-first writes
- HotSignalState: #[repr(C, align(64))], lock-free CAS decay, out-of-order handling
- BucketedCounter: 60 per-minute + 168 per-hour circular buffers, trigger-based rotation
- CheckpointMeta + serialize/restore: 983-byte fixed records, atomic WriteBatch
- Property tests: running score matches analytical to 1e-6, decay monotonic, non-negative
- Proptest regression: signals/warm.txt
## Documentation and planning
- ROADMAP: m0p1 COMPLETE (329), m0p2 COMPLETE (349), product track milestones
- PRODUCT_ROADMAP: P0-P4 product milestone track (personal briefing beachhead)
- Milestone planning docs: milestone-0 (phases 1-3), milestone-p (phases 1-5)
- docs/research/tidaldb_tooling_and_diagnostics.md
- ARCHITECTURE.md, CLAUDE.md, VISION.md updates
## Site
- Blog: every-platform-builds-the-same-6-systems.mdx (new)
- Blog: why-tidaldb.mdx (updated)
- next.config.ts, layout.tsx, blog/page.tsx updates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
115 lines
6.1 KiB
Markdown
115 lines
6.1 KiB
Markdown
> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
|
||
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
|
||
|
||
# tidalDB
|
||
|
||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||
|
||
**Status:** Vision and specification phase. No implementation yet.
|
||
|
||
## Find Your Guide
|
||
|
||
| If you need to... | Read this |
|
||
|-------------------|-----------|
|
||
| **Understand the vision** | [VISION.md](VISION.md) |
|
||
| **See use cases and surfaces** | [USE_CASES.md](USE_CASES.md) |
|
||
| **See sequence diagrams** | [SEQUENCE.md](SEQUENCE.md) |
|
||
| **Understand the system architecture** | [ARCHITECTURE.md](ARCHITECTURE.md) |
|
||
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
|
||
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
|
||
| **See the API spec** | [API.md](API.md) |
|
||
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
|
||
| **Read technical research** | [docs/research/](docs/research/) |
|
||
|
||
## Agents
|
||
|
||
| Agent | Identity | Use when |
|
||
|-------|----------|----------|
|
||
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||
|
||
## Skills
|
||
|
||
### Phase Lifecycle
|
||
|
||
| Step | Skill | Use when |
|
||
|------|-------|----------|
|
||
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
|
||
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
|
||
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
|
||
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
|
||
|
||
### Other Skills
|
||
|
||
| Skill | Use when |
|
||
|-------|----------|
|
||
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
|
||
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
|
||
| `/develop` | Quick implementation work outside the milestone lifecycle |
|
||
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
|
||
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
|
||
| `/build-site` | Creating or iterating on the marketing site |
|
||
| `/write-blog` | Writing blog posts about progress or architecture |
|
||
|
||
## Core Domain Model
|
||
|
||
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
|
||
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
|
||
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
|
||
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
|
||
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
|
||
|
||
## Ports
|
||
|
||
Dev servers use port range **59520–59529** (e.g. `site/` on 59520).
|
||
|
||
## Critical Rules
|
||
|
||
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
|
||
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
|
||
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
|
||
- **Single-node first:** Embeddable. Scales vertically before horizontally.
|
||
- **Language:** Rust.
|
||
|
||
## Repository Structure
|
||
|
||
```
|
||
. # Top-level docs and configuration
|
||
├── CLAUDE.md # This file — project instructions
|
||
├── VISION.md # Product vision and thesis
|
||
├── USE_CASES.md # 14 use cases, all discovery surfaces
|
||
├── SEQUENCE.md # Data flow sequence diagrams
|
||
├── CODING_GUIDELINES.md # Engineering standards
|
||
├── API.md # API specification
|
||
├── thoughts.md # Architectural lessons from sister projects
|
||
├── ai-lookup/ # Domain concept reference
|
||
├── docs/ # Research and documentation
|
||
│ └── research/ # Deep technical research docs
|
||
├── .claude/ # Claude Code configuration
|
||
│ ├── agents/ # Agent definitions
|
||
│ └── skills/ # Skill definitions
|
||
├── tidal/ # Rust database engine
|
||
│ ├── Cargo.toml
|
||
│ ├── src/
|
||
│ │ ├── storage/ # Entity store, signal ledger, inverted index, HNSW
|
||
│ │ ├── query/ # Query parser, planner, executor
|
||
│ │ ├── ranking/ # Profile engine, signal scoring, diversity enforcement
|
||
│ │ ├── signals/ # Signal types, decay, velocity, windowed aggregation
|
||
│ │ └── schema/ # Schema definition, validation, migrations
|
||
│ ├── benches/ # Performance benchmarks
|
||
│ └── tests/ # Integration and property tests
|
||
└── site/ # Public marketing site (Next.js)
|
||
```
|
||
|
||
## Pre-commit Hooks
|
||
|
||
The pre-commit hook runs automatically on staged files:
|
||
- **tidal/ (Rust):** `cargo fmt` (auto-fix + re-stage), `cargo clippy -D warnings`, `cargo test --lib`
|
||
- **site/ (Next.js):** `eslint` (if node_modules installed)
|
||
|
||
All cargo commands use `--manifest-path tidal/Cargo.toml` since the Rust project is not at repo root.
|
||
|
||
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
|