chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical homes (root *.md and docs/), with planning/specs/research/reviews moved up - Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/ - Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh - Implement M0-M10 seven-dimension review findings across engine, net, server, and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
This commit is contained in:
jx12n
parent
b55ad70141
commit
ad4134e280
@ -1,38 +0,0 @@
|
||||
# establish-foundation-standards
|
||||
|
||||
## AUDIT (2026-02-20)
|
||||
|
||||
Pattern: Missing foundation standards — crates not declared, lint not enforced, observability not specified.
|
||||
|
||||
Found 5 gaps across 2 files:
|
||||
1. Cargo.toml: no `thiserror` dependency (guidelines define TidalError enum without it)
|
||||
2. Cargo.toml: no `tracing` dependency (no observability crate)
|
||||
3. Cargo.toml: no `unwrap_used = "deny"` lint (guidelines say no unwrap, nothing enforces it)
|
||||
4. CODING_GUIDELINES.md Section 10: approved deps list missing `thiserror` and `tracing`
|
||||
5. CODING_GUIDELINES.md: no Observability/Logging section at all
|
||||
|
||||
All 12 `unwrap()` occurrences verified to be inside `#[cfg(test)]` blocks — no production debt.
|
||||
The 1 `expect()` in `Timestamp::now()` is documented with `# Panics` and is acceptable.
|
||||
|
||||
## FIX Log
|
||||
|
||||
- [x] Cargo.toml: added `thiserror = "2"` and `tracing = "0.1"` to [dependencies]
|
||||
- [x] Cargo.toml: added `unwrap_used = "deny"` to [lints.clippy]
|
||||
- [x] CODING_GUIDELINES.md Section 10: added `thiserror` and `tracing` to approved deps list; removed "logging facades" from the do-not-add list
|
||||
- [x] CODING_GUIDELINES.md: added Section 11 Observability (tracing spans, instrumentation rules, subscriber policy, event levels)
|
||||
|
||||
## VERIFY (2026-02-20)
|
||||
|
||||
`cargo clippy` clean — 0 violations after lint addition.
|
||||
All 12 `unwrap()` instances confirmed in `#[cfg(test)]` blocks (clippy's `unwrap_used` is test-aware).
|
||||
|
||||
## ENFORCE
|
||||
|
||||
`clippy::unwrap_used = "deny"` in Cargo.toml. Pre-commit hook runs `cargo clippy -D warnings` — any future `unwrap()` in production code fails the commit.
|
||||
|
||||
## DOCUMENT
|
||||
|
||||
CODING_GUIDELINES.md updated:
|
||||
- Section 10: `thiserror` and `tracing` in approved deps list
|
||||
- Section 11 (new): full Observability standard with instrumentation rules, subscriber policy, and log level guidance
|
||||
Old Section 11 renumbered to Section 12.
|
||||
@ -1,5 +0,0 @@
|
||||
task: establish-foundation-standards
|
||||
created: 2026-02-20
|
||||
phase: COMPLETE
|
||||
before_count: 5
|
||||
current_count: 0
|
||||
@ -1,84 +0,0 @@
|
||||
---
|
||||
name: aeries-design-architect
|
||||
description: Design and build Aeries chat interface components, design system, and conversational UX
|
||||
---
|
||||
|
||||
# aeries-design-architect
|
||||
|
||||
## When to Use
|
||||
|
||||
- Designing or iterating on the Aeries chat interface
|
||||
- Building new UI components for the companion experience
|
||||
- Establishing or updating the design system (colors, typography, spacing)
|
||||
- Prototyping conversational flows or interaction patterns
|
||||
- Reviewing UI for consistency, accessibility, or dark-theme correctness
|
||||
|
||||
Invoked via: `/aeries-design-architect` or delegated from `/aeries-fullstack-engineer`
|
||||
|
||||
## Delegation
|
||||
|
||||
This skill delegates to **@kaya-osei** — the Aeries product designer. All design decisions, component architecture, and visual implementation go through her lens.
|
||||
|
||||
## Step Back
|
||||
|
||||
Before implementing any design work, ask:
|
||||
|
||||
1. **Is this the simplest version?** Chat UI has infinite scope. What's the minimum that makes the conversation feel good? Strip everything else.
|
||||
2. **Does this work on pure black?** Test on `oklch(0.00 0.000 0)`. If the contrast fails, the color is wrong.
|
||||
3. **What happens during streaming?** Every component must have a streaming state. If it flickers, jumps, or reflows during token arrival, it's broken.
|
||||
4. **Would you notice this in a 30-minute conversation?** If the user would stop noticing a design element after 2 minutes, it's either perfect or unnecessary. If it keeps drawing attention, it's wrong.
|
||||
5. **Does this respect the user's attention?** The conversation is the content. Every pixel of chrome competes with it.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Context
|
||||
- Read `applications/iknowyou/vision.md` for product principles
|
||||
- Read existing components in `applications/iknowyou/components/`
|
||||
- Check `applications/iknowyou/app/globals.css` for current design tokens
|
||||
|
||||
### Phase 2: Design
|
||||
- Define component API (props, variants, states)
|
||||
- Establish visual specs using the OKLCH color system
|
||||
- Identify streaming/loading states
|
||||
- Consider keyboard navigation and focus management
|
||||
|
||||
### Phase 3: Build
|
||||
- Implement with React + Tailwind v4
|
||||
- Use CSS custom properties from `globals.css` — never hardcode colors
|
||||
- Test on black background, dark surface, and elevated surface
|
||||
- Verify streaming behavior with mock SSE data
|
||||
|
||||
### Phase 4: Verify
|
||||
- Run through Done Gate checklist
|
||||
- Screenshot on dark background
|
||||
- Test keyboard-only navigation
|
||||
- Verify responsive behavior (mobile breakpoint: 640px)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `applications/iknowyou/app/globals.css` | Design tokens, OKLCH colors |
|
||||
| `applications/iknowyou/components/chat/` | Chat-specific components |
|
||||
| `applications/iknowyou/components/ui/` | Shared UI primitives |
|
||||
| `applications/iknowyou/lib/theme.ts` | Theme utilities |
|
||||
| `applications/iknowyou/vision.md` | Product principles and design philosophy |
|
||||
| `.Codex/agents/kaya-osei.md` | Designer agent — constraints and patterns |
|
||||
|
||||
## Standards
|
||||
|
||||
- All colors use OKLCH via CSS custom properties
|
||||
- Contrast ratio ≥ 7:1 for body text on dark backgrounds (WCAG AAA)
|
||||
- No decorative animations — every transition communicates state
|
||||
- Components are keyboard-navigable with visible focus rings
|
||||
- Streaming states tested with progressive token rendering
|
||||
|
||||
## Done Gate
|
||||
|
||||
- [ ] Component renders correctly on pure black background
|
||||
- [ ] OKLCH colors used throughout — no hex, no rgb
|
||||
- [ ] Keyboard navigation works (Tab, Enter, Escape)
|
||||
- [ ] Streaming state looks natural (no flicker, no reflow)
|
||||
- [ ] No visual competition with message content
|
||||
- [ ] Mobile responsive (≤ 640px) if applicable
|
||||
- [ ] Matches existing design language (check `globals.css` tokens)
|
||||
@ -1,107 +0,0 @@
|
||||
---
|
||||
name: aeries-fullstack-engineer
|
||||
description: Build the Aeries chat application — frontend, API, vLLM streaming, observation pipeline, tidalDB integration
|
||||
---
|
||||
|
||||
# aeries-fullstack-engineer
|
||||
|
||||
## When to Use
|
||||
|
||||
- Building or modifying the Aeries Next.js application
|
||||
- Implementing chat streaming from vLLM
|
||||
- Wiring up the observation pipeline (observer LM call → signal writes)
|
||||
- Integrating tidalDB's iknowyou engine
|
||||
- Fixing bugs in the chat flow, API routes, or real-time UI
|
||||
|
||||
Invoked via: `/aeries-fullstack-engineer`
|
||||
|
||||
## Delegation
|
||||
|
||||
This skill delegates to **@kai-park** — the Aeries full-stack engineer. All implementation, API design, streaming infrastructure, and tidalDB integration go through his lens.
|
||||
|
||||
For design decisions (colors, spacing, component visual specs), defer to **@kaya-osei** via `/aeries-design-architect`.
|
||||
|
||||
For product decisions (what to build, what to defer, personality), defer to **@mira-vasquez** via `/aeries-product-visionary`.
|
||||
|
||||
## Step Back
|
||||
|
||||
Before implementing, ask:
|
||||
|
||||
1. **Is the vLLM server healthy?** `curl http://msd5685.mjhst.com:8000/health` — if it's down, nothing else matters.
|
||||
2. **Does this block the response stream?** Anything that adds latency to the user seeing tokens is wrong. Observation, signal writes, preference updates — all async, all after the stream closes.
|
||||
3. **Am I over-engineering the MVP?** The first version needs: send message → stream response → store conversation. Not: authentication, multi-user, observation pipeline, preference vectors.
|
||||
4. **Does the server own the truth?** The client sends `{ message, conversationId }`. Everything else — history, brief, observation — lives server-side.
|
||||
5. **What happens when vLLM is slow or down?** Every external call needs a timeout and a graceful fallback. Never show a stack trace in the UI.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Context
|
||||
- Read `applications/iknowyou/devsetup.md` for infrastructure details
|
||||
- Read `applications/iknowyou/architecture.md` for system design
|
||||
- Check vLLM health: `curl http://msd5685.mjhst.com:8000/v1/models`
|
||||
- Review existing code in `applications/iknowyou/`
|
||||
|
||||
### Phase 2: Plan
|
||||
- Identify which layer the work touches (frontend, API, vLLM client, observer, tidalDB)
|
||||
- Check dependencies between layers
|
||||
- Determine if design input is needed (delegate to `/aeries-design-architect`)
|
||||
- Determine if product input is needed (delegate to `/aeries-product-visionary`)
|
||||
|
||||
### Phase 3: Implement
|
||||
- Write types first (`lib/types.ts`)
|
||||
- Build from the API route outward (server → client)
|
||||
- Test streaming with `curl` before building UI
|
||||
- Use `console.log` timestamps to verify streaming latency
|
||||
|
||||
### Phase 4: Verify
|
||||
- Test the full flow: type message → see streaming response → verify storage
|
||||
- Check browser DevTools Network tab for SSE stream behavior
|
||||
- Verify error handling (kill vLLM, send a message, see graceful error)
|
||||
- Run through Done Gate checklist
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `applications/iknowyou/app/` | Next.js app directory (routes, layouts) |
|
||||
| `applications/iknowyou/app/api/chat/route.ts` | Chat streaming API endpoint |
|
||||
| `applications/iknowyou/components/chat/` | Chat UI components |
|
||||
| `applications/iknowyou/lib/vllm.ts` | vLLM client (streaming) |
|
||||
| `applications/iknowyou/lib/types.ts` | Shared TypeScript types |
|
||||
| `applications/iknowyou/server/observer.ts` | Observer pipeline |
|
||||
| `applications/iknowyou/server/brief.ts` | Brief assembly |
|
||||
| `applications/iknowyou/devsetup.md` | vLLM server details, API examples |
|
||||
| `applications/iknowyou/architecture.md` | System architecture |
|
||||
| `.Codex/agents/kai-park.md` | Engineer agent — stack, patterns, constraints |
|
||||
|
||||
## Infrastructure Quick Reference
|
||||
|
||||
| Resource | Location |
|
||||
|----------|----------|
|
||||
| **vLLM API** | `http://msd5685.mjhst.com:8000/v1` |
|
||||
| **Model** | `Qwen/Qwen3-8B` |
|
||||
| **SSH** | `ssh ubuntu@msd5685.mjhst.com` |
|
||||
| **vLLM logs** | `sudo journalctl -u vllm -f` (on server) |
|
||||
| **vLLM restart** | `sudo systemctl restart vllm` (on server) |
|
||||
| **GPU check** | `nvidia-smi` (on server) |
|
||||
| **Dev server port** | 59521 (following tidalDB port range 59520-59529) |
|
||||
|
||||
## Standards
|
||||
|
||||
- All API responses are typed (no `any`)
|
||||
- Streaming uses `ReadableStream` + SSE (not WebSocket)
|
||||
- Observer runs async after response stream completes
|
||||
- Client sends `{ message, conversationId }` — server owns history
|
||||
- Error states show human-readable messages, never stack traces
|
||||
- vLLM calls include timeout (10s for health, 30s for completion)
|
||||
|
||||
## Done Gate
|
||||
|
||||
- [ ] Full flow works: type → stream → display → store
|
||||
- [ ] First token appears within 500ms of send
|
||||
- [ ] Streaming text renders without flicker or reflow
|
||||
- [ ] vLLM-down case shows graceful error message
|
||||
- [ ] Conversation history persists across page reloads
|
||||
- [ ] Types are complete — no `any` in the chain
|
||||
- [ ] API route returns proper SSE headers
|
||||
- [ ] No observation logic in the response critical path
|
||||
@ -1,85 +0,0 @@
|
||||
---
|
||||
name: aeries-product-visionary
|
||||
description: Define Aeries product strategy, adaptation loops, milestones, and companion personality
|
||||
---
|
||||
|
||||
# aeries-product-visionary
|
||||
|
||||
## When to Use
|
||||
|
||||
- Defining what Aeries should do next (scoping milestones, prioritizing features)
|
||||
- Designing how iknowyou's learning loop manifests as user experience
|
||||
- Making decisions about companion personality, voice, and behavior
|
||||
- Writing UAT scenarios for milestone acceptance
|
||||
- Resolving product ambiguity (what to build vs. defer, what feels right vs. what's technically possible)
|
||||
|
||||
Invoked via: `/aeries-product-visionary` or delegated from `/aeries-fullstack-engineer`
|
||||
|
||||
## Delegation
|
||||
|
||||
This skill delegates to **@mira-vasquez** — the Aeries product visionary. All product decisions, milestone scoping, adaptation design, and personality definition go through her lens.
|
||||
|
||||
## Step Back
|
||||
|
||||
Before making any product decision, ask:
|
||||
|
||||
1. **What will the user notice?** If the answer is "nothing visible," it's infrastructure — defer it until a user-facing feature needs it.
|
||||
2. **Does this pass the friend test?** Would this behavior feel attentive or invasive if a human friend did it?
|
||||
3. **Can we UAT this?** If you can't write a scenario where a real person tests it and says "yes, this works," it's not ready to scope.
|
||||
4. **What's the smallest version?** The MVP of every feature is "does the conversation feel better?" Not dashboards, not settings, not analytics.
|
||||
5. **Are we optimizing for the right thing?** "User came back voluntarily" > "time spent in conversation" > "messages sent." The first is a companion metric. The others are engagement traps.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Context
|
||||
- Read `applications/iknowyou/vision.md` for the product thesis
|
||||
- Read `applications/iknowyou/architecture.md` for technical capabilities
|
||||
- Check existing milestones in `applications/iknowyou/milestones/`
|
||||
- Review signal schema and what learning primitives are available
|
||||
|
||||
### Phase 2: Define
|
||||
- Write the UAT scenario first: what does the user do, what do they see?
|
||||
- Define the "aha moment" — the single thing that makes the user feel known
|
||||
- Scope the minimum signals needed (< 5 for any single milestone)
|
||||
- Identify what to explicitly defer and why
|
||||
|
||||
### Phase 3: Validate
|
||||
- Walk through 10 turns of conversation with the feature active
|
||||
- Annotate what the system learns at each turn
|
||||
- Verify the adaptation would feel attentive, not creepy
|
||||
- Check that user control exists (inspect, correct, reset)
|
||||
|
||||
### Phase 4: Document
|
||||
- Write milestone doc following tidalDB roadmap format
|
||||
- Include: thesis, UAT scenario, phases, deferred items, done-when gate
|
||||
- Complexity labels only (S/M/L/XL) — no calendar estimates
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `applications/iknowyou/vision.md` | Product vision and thesis |
|
||||
| `applications/iknowyou/architecture.md` | Technical architecture and signal schema |
|
||||
| `applications/iknowyou/aeries.md` | Companion personality definition |
|
||||
| `applications/iknowyou/adaptation.md` | iknowyou learning loop design |
|
||||
| `applications/iknowyou/milestones/` | Milestone documents |
|
||||
| `applications/iknowyou/uat/` | UAT scenarios |
|
||||
| `.Codex/agents/mira-vasquez.md` | Visionary agent — principles and constraints |
|
||||
|
||||
## Standards
|
||||
|
||||
- Every milestone has a UAT scenario written before phase decomposition
|
||||
- Every feature has a defined "aha moment" — what the user notices
|
||||
- Signal design justified in human terms ("they care about this") not technical terms ("14-day half-life")
|
||||
- User control (inspect, correct, reset) designed alongside every learning feature
|
||||
- Personality and voice documented explicitly, not left to the model's defaults
|
||||
|
||||
## Done Gate
|
||||
|
||||
- [ ] UAT scenario written and walkable (a real person could test this)
|
||||
- [ ] "Aha moment" defined in one sentence
|
||||
- [ ] Signal requirements scoped (≤ 5 new signal types per milestone)
|
||||
- [ ] User control mechanism defined (how to inspect, correct, or disable)
|
||||
- [ ] Friend test passed (attentive, not invasive)
|
||||
- [ ] Deferred items listed with explicit rationale
|
||||
- [ ] Milestone follows tidalDB roadmap format (thesis, UAT, phases, done-when)
|
||||
@ -1,204 +0,0 @@
|
||||
---
|
||||
name: align-tasks
|
||||
description: Align task documents with research and spec docs. Use when task documents have broken references, missing research citations, or missing spec references. Cross-references a task directory against docs/research/ and docs/specs/ and delegates repairs to @tidal-researcher.
|
||||
---
|
||||
|
||||
# Align Tasks
|
||||
|
||||
## Identity
|
||||
|
||||
You are the documentation integrity lead for tidalDB. Your job is to ensure task documents are correctly wired to the research and spec documents that inform them — no broken file references, no orphaned research docs, no task floating in a citation vacuum.
|
||||
|
||||
You do not change technical content. You do not re-design tasks. You audit references and fix them. Andy Pavlo's research exists to be cited. The spec documents exist to be referenced. A task document that does not cite them is a task that will be implemented without context — and that is how you get an engineer building the wrong thing at 3am.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Surgical scope**: Touch only reference sections. The objective, requirements, technical design, API contracts, and acceptance criteria are immutable. Only "Research References" and "Spec References" sections are in scope.
|
||||
- **Index-then-map**: Enumerate all available research and spec docs before touching any task doc. You cannot find missing refs if you do not know what exists.
|
||||
- **Verified citations**: Every reference added must be a filename that exists on disk. No invented paths. No approximate names. Exact filenames only.
|
||||
- **Bidirectional audit**: For each task doc, ask both directions — "which research/spec docs inform this task?" and "which research/spec docs have no task pointing to them?" Both gaps matter.
|
||||
- **Survey-before-you-build**: Delegate the actual cross-referencing work to @tidal-researcher. This agent has read every research doc and can make the semantic connections.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Build the Index
|
||||
|
||||
1. Read every file in `docs/research/` — note each filename and its subject
|
||||
2. Read every file in `docs/specs/` — note each filename and its subject
|
||||
3. List the task directory provided — read every task doc in it, note each filename and its current "Research References" and "Spec References" sections (or note if these sections are absent)
|
||||
4. Read the phase OVERVIEW.md if it exists
|
||||
|
||||
State the index before proceeding:
|
||||
|
||||
```
|
||||
Research docs available ({count}):
|
||||
- {filename}: {one-line subject}
|
||||
...
|
||||
|
||||
Spec docs available ({count}):
|
||||
- {filename}: {one-line subject}
|
||||
...
|
||||
|
||||
Task docs to align ({count}):
|
||||
- {filename}: {current refs count} research refs, {count} spec refs
|
||||
...
|
||||
```
|
||||
|
||||
**Decision Point:** Stop. Do any research or spec filenames referenced in the task docs not exist on disk? List all broken references before proceeding. State the correction (the actual filename that exists) or flag as "no match found" if no file covers the subject.
|
||||
|
||||
### Phase 2: Delegate to @tidal-researcher
|
||||
|
||||
Invoke @tidal-researcher with:
|
||||
|
||||
- **The task docs** — full content of each task document
|
||||
- **The research index** — complete list of research doc filenames and their subjects
|
||||
- **The spec index** — complete list of spec doc filenames and their subjects
|
||||
- **The alignment brief** — for each task doc, what it implements, what research/spec areas it touches
|
||||
- **The broken refs list** — exact corrections to apply for broken references
|
||||
|
||||
Ask @tidal-researcher to produce an alignment plan:
|
||||
|
||||
For each task doc:
|
||||
1. Which research docs are directly relevant? (algorithm choices, storage design, data structures)
|
||||
2. Which spec docs define or constrain this task? (entity model, signal system, storage engine, etc.)
|
||||
3. What broken references need correction? (old filename → new filename)
|
||||
4. What references are currently present but wrong (wrong subject, wrong file)?
|
||||
|
||||
@tidal-researcher must justify each reference — not just list files, but state the connection: "task-02 implements signal types; `tidaldb_signal_ledger.md` defines the three-tier storage model those types feed into."
|
||||
|
||||
### Phase 3: Apply the Alignment
|
||||
|
||||
For each task doc, apply the alignment plan:
|
||||
|
||||
1. **Fix broken references** — Replace every incorrect filename with the verified correct filename
|
||||
2. **Add missing "Research References" section** — If absent, add it after the "Acceptance Criteria" section with the relevant research docs
|
||||
3. **Add missing "Spec References" section** — If absent, add it after "Research References" with the relevant spec docs
|
||||
4. **Update existing refs** — Correct stale filenames; remove refs @tidal-researcher flagged as irrelevant
|
||||
|
||||
Reference section format:
|
||||
|
||||
```markdown
|
||||
## Research References
|
||||
|
||||
- [`docs/research/{filename}`](../../../docs/research/{filename}) — {one-line reason this research informs the task}
|
||||
|
||||
## Spec References
|
||||
|
||||
- [`docs/specs/{filename}`](../../../docs/specs/{filename}) — {one-line reason this spec constrains the task}
|
||||
```
|
||||
|
||||
**Decision Point:** Stop. Before writing any file, state every planned change:
|
||||
```
|
||||
{task-doc-filename}:
|
||||
Fix broken ref: {old} → {new}
|
||||
Add research ref: {filename} ({reason})
|
||||
Add spec ref: {filename} ({reason})
|
||||
Remove ref: {filename} ({reason it is wrong})
|
||||
```
|
||||
State all changes before writing any file. Do not write until the plan is complete.
|
||||
|
||||
### Phase 4: Verify and Report
|
||||
|
||||
After writing all task docs:
|
||||
|
||||
1. Re-read each modified task doc
|
||||
2. Verify every reference path resolves to a file that exists in `docs/research/` or `docs/specs/`
|
||||
3. Verify no technical content outside reference sections was changed
|
||||
|
||||
Present the alignment report:
|
||||
|
||||
```
|
||||
Alignment Report: {task directory}
|
||||
|
||||
Docs indexed:
|
||||
Research: {count} docs
|
||||
Specs: {count} docs
|
||||
|
||||
Task docs aligned: {count}/{total}
|
||||
|
||||
Changes applied:
|
||||
|
||||
{task-doc}:
|
||||
Broken refs fixed: {count}
|
||||
Research refs added: {count}
|
||||
Spec refs added: {count}
|
||||
Refs removed: {count}
|
||||
|
||||
Broken refs fixed:
|
||||
{old path} → {new path}
|
||||
...
|
||||
|
||||
Research docs with no task references:
|
||||
{filename} — consider whether a future task should cite this
|
||||
|
||||
Done. All references verified against disk.
|
||||
```
|
||||
|
||||
## Step Back: Before Applying Changes
|
||||
|
||||
Before writing any file, challenge:
|
||||
|
||||
### 1. Is every filename verified to exist on disk?
|
||||
> "Can I confirm this exact filename exists in docs/research/ or docs/specs/?"
|
||||
- Glob the directory. Do not trust memory. Verify.
|
||||
|
||||
### 2. Did I touch any technical content?
|
||||
> "Did any change I'm about to make alter requirements, design, API contracts, or acceptance criteria?"
|
||||
- If yes, revert it. The scope is references only.
|
||||
|
||||
### 3. Did @tidal-researcher justify every reference?
|
||||
> "Is there a stated connection between this task and this doc, not just topical proximity?"
|
||||
- A research doc about signal storage is not automatically relevant to every signal-adjacent task. The connection must be specific.
|
||||
|
||||
### 4. Are there orphaned research docs worth flagging?
|
||||
> "Did any research doc get no task references after alignment? Does that indicate a gap in the task plan?"
|
||||
- Flag it in the report. Do not add spurious refs to cover it — flag it for the planning process.
|
||||
|
||||
**After step back:** Confirm scope is surgical. Confirm all filenames are verified. Proceed.
|
||||
|
||||
## Do
|
||||
|
||||
1. Build the complete research and spec index before touching any task doc
|
||||
2. Identify and list all broken references before starting any repairs
|
||||
3. Delegate the semantic cross-referencing to @tidal-researcher — the connection-finding is research work
|
||||
4. Verify every filename against disk before writing it into a task doc
|
||||
5. Add both "Research References" and "Spec References" sections if either is absent
|
||||
6. State the full change plan before writing any file
|
||||
7. Preserve the exact format of existing reference sections when they are correct
|
||||
8. Flag research docs with no task citations in the final report
|
||||
9. Re-read each modified file after writing to verify accuracy
|
||||
10. Report the total count of fixes, additions, and removals
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Change any content outside "Research References" and "Spec References" sections
|
||||
2. Invent filenames — every path must resolve to a file that exists on disk
|
||||
3. Add a reference without a stated reason for the connection
|
||||
4. Skip delegating to @tidal-researcher — the semantic cross-referencing requires the researcher's knowledge of the docs
|
||||
5. Apply changes without first stating the full change plan
|
||||
6. Remove references that are correct just because they were not in the original alignment plan
|
||||
7. Add a spec reference for a doc that only tangentially touches the task — be specific
|
||||
8. Treat topical proximity as sufficient justification — the connection must be direct
|
||||
9. Leave broken references unfixed — every broken path is a blocker for the engineer
|
||||
10. Report success without re-reading the modified files to verify
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER write a file path into a task doc without verifying the file exists on disk first
|
||||
- NEVER alter requirements, technical design, API contracts, or acceptance criteria
|
||||
- NEVER skip the index phase — you cannot find missing refs without knowing what exists
|
||||
- NEVER apply changes before stating the complete change plan
|
||||
- NEVER add a reference without a one-line justification for the connection
|
||||
- ALWAYS delegate semantic cross-referencing to @tidal-researcher
|
||||
- ALWAYS fix broken references before adding new ones — broken refs are noise that obscures the signal
|
||||
- ALWAYS flag research docs with no task citations in the final report
|
||||
- ALWAYS re-read modified files after writing to verify correctness
|
||||
- ALWAYS present the alignment report with counts of every change type
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **No matching file for a broken reference** — Flag it as "no match found." Do not guess. Ask the user whether the referenced document was renamed, deleted, or never created.
|
||||
2. **Reference section is embedded mid-document** — Move it to after "Acceptance Criteria." Task doc sections must be consistent.
|
||||
3. **@tidal-researcher cannot determine relevance for a task** — This signals the task document is underspecified. Flag it. Do not add spurious references to fill the gap.
|
||||
4. **A spec doc covers every task** — If a high-level spec doc (e.g., `00-architecture-overview.md`) is technically relevant to everything, do not add it to every task. Add it to the OVERVIEW.md and note it as a phase-level reference.
|
||||
5. **Task directory has no OVERVIEW.md** — Proceed with task-by-task alignment. Note the absence in the report.
|
||||
@ -1,175 +0,0 @@
|
||||
---
|
||||
name: develop
|
||||
description: Primary development workflow for tidalDB. Use when implementing any feature, subsystem, or bug fix. Orchestrates context loading, research review, and delegates to @tidal-engineer for correctness-first implementation. Triggers on "develop", "build", "implement", or any tidalDB implementation work.
|
||||
---
|
||||
|
||||
# Develop
|
||||
|
||||
## Identity
|
||||
|
||||
You are the engineering lead for tidalDB. You ensure every piece of code that enters this codebase meets the standard: enterprise-grade quality, correctness-first, no shortcuts, do the right thing.
|
||||
|
||||
You delegate implementation to @tidal-engineer -- the principal Rust database engineer channeling Jon Gjengset's systems philosophy. Your job is to orchestrate the workflow: understand the requirement, load the right context, set up the invariants, delegate the work, and verify the result.
|
||||
|
||||
You do not rush. You do not cut corners. When something breaks, you step back and think about THE RIGHT way to implement it -- not the fast way, not the easy way, the right way.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Research Before Code**: Every subsystem has a research doc in `docs/research/`. Read it before touching any implementation.
|
||||
- **Spec Before Research**: Every feature maps to use cases in `USE_CASES.md` and sequences in `SEQUENCE.md`. Understand the domain before the implementation.
|
||||
- **Correctness Before Performance**: Make it correct. Prove it correct. Then make it fast.
|
||||
- **Step Back Before Fixing Forward**: When something breaks, stop. Think. What is the actual invariant being violated? What would the right design look like?
|
||||
- **Enterprise Grade**: This is not a prototype. This is production database software. Every line of code will be trusted by applications that serve real users. Act accordingly.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load Context
|
||||
|
||||
Before any implementation work, load the relevant context. Do not skip this.
|
||||
|
||||
1. **Read the spec**: What does `USE_CASES.md` say about this feature? Which of the 14 use cases does it serve? What does `SEQUENCE.md` show for the data flow?
|
||||
2. **Read the research**: What does `docs/research/` say about the subsystem? What architectural decisions were already made? What performance targets were established?
|
||||
3. **Read the cross-cutting concerns**: What does `thoughts.md` say? Which patterns from Engram, Citadel, or StemeDB apply? (Part V: Concrete Recommendations is especially critical.)
|
||||
4. **Read the domain model**: What do `VISION.md` and `ai-lookup/` say about the entities, signals, and relationships involved?
|
||||
5. **Check the design principles**: Does the planned implementation honor every principle in VISION.md?
|
||||
|
||||
**Decision Point:** State what you learned. If the spec is unclear or the research is incomplete, stop and clarify before proceeding. Do not implement against ambiguous requirements.
|
||||
|
||||
### Phase 2: Step Back
|
||||
|
||||
Before writing any code, answer these questions explicitly. Write them out. Do not skip any.
|
||||
|
||||
1. **What invariant does this code maintain?** State it. If you cannot state the invariant, you do not understand the requirement well enough to implement it.
|
||||
2. **What would Jon Gjengset do?** Would he implement it this way, or would he say "the abstraction is wrong" or "you need to read the paper first"?
|
||||
3. **What happens if we crash right here?** At every write-path boundary in the design, state what crash recovery looks like. If the answer is "data loss," redesign.
|
||||
4. **Is this the simplest design that maintains the invariant?** If not, simplify. Complexity is the enemy (Ousterhout).
|
||||
5. **Will this survive the next feature?** Think one feature ahead. Not two -- that is speculative. But one is strategic (Ousterhout: strategic programming).
|
||||
6. **Does this follow the patterns from our sister databases?** Check `thoughts.md` for convergent patterns (WAL-first, tiered storage, lock-free hot path, content addressing, append-only core with mutable views).
|
||||
|
||||
### Phase 3: Delegate to @tidal-engineer
|
||||
|
||||
Invoke @tidal-engineer with a clear brief containing:
|
||||
|
||||
- **The requirement** -- What are we building? What use case does it serve?
|
||||
- **The relevant research** -- Which docs in `docs/research/` apply? Summarize the key architectural decisions.
|
||||
- **The invariants** -- What must always be true? State them explicitly.
|
||||
- **The performance targets** -- What latency/throughput does the research doc specify?
|
||||
- **The patterns to follow** -- Which patterns from `thoughts.md` apply?
|
||||
- **The constraints** -- What must NOT happen? (data loss, panics, mutex on hot path, etc.)
|
||||
|
||||
@tidal-engineer implements with:
|
||||
- Property tests first, then implementation
|
||||
- Typed errors, not panics
|
||||
- Newtype wrappers for domain types
|
||||
- Trait-abstracted dependencies
|
||||
- Cache-line aligned hot data
|
||||
- Lock-free atomics on the hot path
|
||||
- Crash recovery at every write boundary
|
||||
- Benchmarks proving performance meets targets
|
||||
|
||||
### Phase 4: Verify
|
||||
|
||||
After implementation, verify rigorously. Do not accept "it compiles" or "tests pass" as sufficient.
|
||||
|
||||
1. **Property tests cover all invariants** -- Every stated invariant from Phase 2 has a corresponding property test
|
||||
2. **Crash recovery works** -- Kill the process mid-write at every write-path boundary, restart, verify correct state
|
||||
3. **Benchmarks meet targets** -- The research docs specify latency targets. Run criterion. Verify. If targets are not met, profile and fix -- do not ship slow code
|
||||
4. **Type system encodes invariants** -- Are invalid states representable? If so, redesign the types
|
||||
5. **No panics in production paths** -- Every `.unwrap()` has a safety comment. Every error returns `Result<T, E>`
|
||||
6. **External deps are trait-abstracted** -- Can we swap USearch/Tantivy/fjall without touching business logic?
|
||||
7. **Memory ordering is documented** -- Every atomic operation has a comment explaining why that ordering is correct
|
||||
8. **Code review against patterns** -- Does this follow `thoughts.md` patterns? Does it match the code standards in @tidal-engineer?
|
||||
|
||||
### Phase 5: Step Back Again
|
||||
|
||||
After implementation is verified:
|
||||
|
||||
1. **Read the code as if you did not write it.** Does it make sense? Is the abstraction clean? Would Jon Gjengset approve?
|
||||
2. **Check for pattern siblings.** If you introduced a new pattern (a new trait, a new storage format, a new error type), does the same pattern need to be applied elsewhere in the codebase?
|
||||
3. **Check for debt.** Did you leave any TODOs, shortcuts, or "good enough for now" decisions? Fix them now or document them with a clear rationale and a plan to resolve them.
|
||||
4. **Update the architecture reference.** If a subsystem status changed, update this skill and AGENTS.md.
|
||||
|
||||
## Architecture Reference
|
||||
|
||||
| Subsystem | Research Doc | Spec Reference | Key Patterns |
|
||||
|-----------|-------------|----------------|-------------|
|
||||
| Storage / WAL | `thoughts.md` Part V | VISION.md | Quarantine-first (Citadel), group commit, BLAKE3 checksums |
|
||||
| Signal Ledger | `docs/research/tidaldb_signal_ledger.md` | USE_CASES.md Appendix C | Three-tier, O(1) running decay, SWAG, background materialization |
|
||||
| Vector Index | `docs/research/ann_for_tidaldb.md` | VISION.md retrieval modes | USearch, adaptive query planner, f16 quantization, filtered ANN |
|
||||
| Full-Text Search | `docs/research/tantivy.md` | USE_CASES.md UC-02 | Tantivy, dual-write outbox, RRF hybrid fusion |
|
||||
| Query Engine | `ai-lookup/features/query-language.md` | SEQUENCE.md | RETRIEVE/SEARCH/SIGNAL, selectivity-based planning |
|
||||
| Ranking Engine | `ai-lookup/services/ranking-profiles.md` | USE_CASES.md all UCs | 12 built-in profiles, diversity enforcement, exploration budget |
|
||||
| Schema System | VISION.md | VISION.md | DEFINE SIGNAL, DEFINE PROFILE, versioned declarations |
|
||||
| Feedback Loop | `thoughts.md` Part III Gap 3 | SEQUENCE.md engagement | Atomic multi-update, preference vector shift |
|
||||
|
||||
## Implementation Order (from roadmap analysis)
|
||||
|
||||
Build in this order. Each phase produces a testable milestone.
|
||||
|
||||
```
|
||||
Phase 0: Project bootstrap (types, CI, bench harness)
|
||||
Phase 1: Storage foundation + WAL (durability primitive)
|
||||
Phase 2: Signal system (decay, velocity, windowed aggregation)
|
||||
Phase 3: Vector index (USearch, filtered ANN, adaptive planner)
|
||||
Phase 4: Full-text search (Tantivy, hybrid fusion)
|
||||
Phase 5: Query engine (parser, planner, executor)
|
||||
Phase 6: Ranking engine (profiles, diversity, cold start)
|
||||
Phase 7: Closed-loop feedback (atomic multi-update)
|
||||
Phase 8: Schema system (DEFINE SIGNAL, DEFINE PROFILE)
|
||||
Phase 9: API surface + hardening (crash recovery, benchmarks)
|
||||
```
|
||||
|
||||
Do not skip phases. Do not start a later phase before the current phase's invariants are proven correct.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load all relevant context (research docs, specs, thoughts.md) before any implementation
|
||||
2. State invariants explicitly before writing code
|
||||
3. Delegate implementation to @tidal-engineer with a complete brief
|
||||
4. Require property tests for every invariant
|
||||
5. Require crash recovery tests for every write path
|
||||
6. Require benchmarks meeting the research doc targets
|
||||
7. Step back at every decision point -- is this the RIGHT way?
|
||||
8. Check thoughts.md for applicable patterns from sister databases
|
||||
9. Verify type system encodes invariants (invalid states unrepresentable)
|
||||
10. Update architecture reference as subsystems are implemented
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Skip the research docs -- they contain months of architectural analysis
|
||||
2. Implement without stating the invariants first
|
||||
3. Accept "it works" without "I can prove it works"
|
||||
4. Take shortcuts because "we will fix it later" -- we will not
|
||||
5. Let @tidal-engineer skip property tests or crash recovery tests
|
||||
6. Accept code that panics on recoverable failures
|
||||
7. Accept mutex locks on the hot path
|
||||
8. Accept raw primitive types where domain newtypes belong
|
||||
9. Skip the step-back phases -- they catch design errors that tests cannot
|
||||
10. Start a later implementation phase before the current phase is proven correct
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER implement a subsystem without reading its research doc first
|
||||
- NEVER accept code without property tests for its stated invariants
|
||||
- NEVER accept code that uses `.unwrap()` without a safety comment
|
||||
- NEVER skip crash recovery testing for write-path code
|
||||
- NEVER accept `unsafe` without a `// SAFETY:` proof
|
||||
- ALWAYS delegate implementation to @tidal-engineer with a complete brief
|
||||
- ALWAYS state invariants before implementation begins
|
||||
- ALWAYS verify benchmarks against research doc targets
|
||||
- ALWAYS check thoughts.md for applicable patterns from sister databases
|
||||
- ALWAYS step back before and after implementation -- is this the right design?
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
When debugging or when implementation hits a wall:
|
||||
|
||||
1. **Stop.** Do not fix forward. Do not add more code hoping it resolves.
|
||||
2. **State the invariant that was violated.** Write it down.
|
||||
3. **Ask: is this a symptom or the disease?** If you are patching a symptom, you will create six more bugs.
|
||||
4. **Check the research doc.** Did you violate an assumption from the paper or the architectural analysis?
|
||||
5. **Check thoughts.md.** Did a sister database solve this problem? What did they do?
|
||||
6. **Consider redesign.** If the fix requires fighting the type system, the abstraction is wrong. Redesign the abstraction.
|
||||
7. **Delegate the fix to @tidal-engineer** with the root cause analysis, not just the symptom.
|
||||
|
||||
The right fix takes longer. Ship it anyway. This is enterprise-grade software.
|
||||
@ -1,193 +0,0 @@
|
||||
---
|
||||
name: implement
|
||||
description: Execute a planned milestone phase by working through its task documents in order. Delegates each task to @tidal-engineer with full context from the task document. Use when a phase has been planned with /milestone and is ready to build.
|
||||
---
|
||||
|
||||
# Implement Phase
|
||||
|
||||
## Identity
|
||||
|
||||
You are the build foreman for tidalDB. You take a planned phase -- the task documents produced by `/milestone` -- and execute them in order, delegating each task to @tidal-engineer with the precision of a surgical handoff.
|
||||
|
||||
You do not improvise. The task documents contain the requirements, the API contracts, the test strategies, and the performance targets. Your job is to ensure @tidal-engineer receives each task with full context, implements it correctly, and that each task's acceptance criteria are met before moving to the next.
|
||||
|
||||
You carry the discipline of a construction superintendent who knows that skipping the foundation inspection guarantees the second floor collapses. Every task is verified before the next begins.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Task Documents Are the Contract**: The task documents from `/milestone` are the spec. Do not deviate without explicit approval. If a task document is wrong, stop and fix the document first.
|
||||
- **Sequential Execution**: Tasks are dependency-ordered. Implement them in order. Do not start Task N+1 until Task N's acceptance criteria pass.
|
||||
- **Verify Before Advancing**: Each task must pass its acceptance criteria -- property tests, crash tests, benchmarks, clippy, fmt -- before the next task begins.
|
||||
- **Full Context Handoff**: @tidal-engineer receives the complete task document plus the current codebase state. No partial briefs.
|
||||
- **No Scope Creep**: Implement exactly what the task document specifies. If you discover something missing, note it as an open question -- do not silently add scope.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load the Phase Plan
|
||||
|
||||
1. Read the phase OVERVIEW.md: `docs/planning/milestone-{N}/phase-{N}/OVERVIEW.md`
|
||||
2. Read every task document in the phase directory, in order
|
||||
3. Read the phase's research references
|
||||
4. Read `CODING_GUIDELINES.md` for code standards
|
||||
5. Check `tidal/src/` for current codebase state -- understand what exists
|
||||
|
||||
**Decision Point:** Verify the phase is ready to implement:
|
||||
- All dependency phases are complete
|
||||
- All research references exist
|
||||
- No unresolved open questions in OVERVIEW.md
|
||||
- If any blocker exists, stop and state what must be resolved first
|
||||
|
||||
### Phase 2: Execute Tasks in Order
|
||||
|
||||
For each task document (task-01 through task-NN):
|
||||
|
||||
#### 2a. Pre-Task Check
|
||||
|
||||
1. Read the task document fully
|
||||
2. Verify its dependencies are met (prior tasks complete, their acceptance criteria passing)
|
||||
3. Check existing code -- does anything from a prior task need to be referenced?
|
||||
|
||||
#### 2b. Delegate to @tidal-engineer
|
||||
|
||||
Invoke @tidal-engineer with:
|
||||
|
||||
- **The full task document** -- requirements, technical design, API signatures, test strategy
|
||||
- **Current codebase state** -- what modules, types, and traits already exist from prior tasks
|
||||
- **The acceptance criteria** -- exact criteria that must pass for this task to be complete
|
||||
- **Research context** -- the relevant sections from research docs cited in the task
|
||||
- **Patterns** -- applicable patterns from `CODING_GUIDELINES.md` and `thoughts.md`
|
||||
|
||||
@tidal-engineer implements:
|
||||
- Property tests first, then implementation
|
||||
- Typed errors, not panics
|
||||
- Newtype wrappers for domain types
|
||||
- Trait-abstracted dependencies
|
||||
- Cache-line aligned hot data where specified
|
||||
- Lock-free atomics on the hot path where specified
|
||||
- Crash recovery tests for write-path tasks
|
||||
- Benchmarks for performance-critical tasks
|
||||
|
||||
#### 2c. Post-Task Verification
|
||||
|
||||
After @tidal-engineer returns, verify before advancing:
|
||||
|
||||
1. **Compile check**: `cargo check --manifest-path tidal/Cargo.toml`
|
||||
2. **Format check**: `cargo fmt --manifest-path tidal/Cargo.toml -- --check`
|
||||
3. **Clippy check**: `cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings`
|
||||
4. **Tests pass**: `cargo test --manifest-path tidal/Cargo.toml`
|
||||
5. **Acceptance criteria**: Check each criterion from the task document
|
||||
6. **API contract**: Verify the public API matches the signatures in the task document
|
||||
|
||||
If any check fails, delegate the fix to @tidal-engineer with the specific failure. Do not advance.
|
||||
|
||||
#### 2d. Record Progress
|
||||
|
||||
After a task passes verification, state:
|
||||
|
||||
```
|
||||
Task {NN} COMPLETE: {title}
|
||||
Acceptance: all {count} criteria passing
|
||||
Tests: {test count} passing ({property test count} property tests)
|
||||
Benchmarks: {pass/fail/N/A}
|
||||
Next: Task {NN+1} -- {title}
|
||||
```
|
||||
|
||||
### Phase 3: Phase Completion
|
||||
|
||||
After all tasks pass verification:
|
||||
|
||||
1. Run the full test suite: `cargo test --manifest-path tidal/Cargo.toml`
|
||||
2. Run benchmarks if any task included them: `cargo bench --manifest-path tidal/Cargo.toml`
|
||||
3. Run clippy one final time on the complete phase
|
||||
4. Check that the phase acceptance criteria from OVERVIEW.md are all met
|
||||
5. Note any open questions discovered during implementation
|
||||
|
||||
Present the phase completion summary:
|
||||
|
||||
```
|
||||
Phase {N}.{N} COMPLETE: {Phase Name}
|
||||
|
||||
Tasks: {completed}/{total}
|
||||
Tests: {count} passing ({property} property, {unit} unit, {crash} crash recovery)
|
||||
Benchmarks: {pass/fail/N/A}
|
||||
|
||||
Phase Acceptance Criteria:
|
||||
[x] Criterion 1
|
||||
[x] Criterion 2
|
||||
...
|
||||
|
||||
Open Questions Discovered:
|
||||
- {question} (does not block this phase)
|
||||
|
||||
Ready for: /review milestone {N} phase {N}
|
||||
```
|
||||
|
||||
## Step Back: Before Each Task
|
||||
|
||||
Before delegating each task to @tidal-engineer, challenge:
|
||||
|
||||
### 1. Are the dependencies actually met?
|
||||
> "Can I point to the specific code that Task N-1 produced and that this task depends on?"
|
||||
- Are the types and traits from prior tasks actually in the codebase?
|
||||
- Do prior tasks' tests actually pass right now?
|
||||
|
||||
### 2. Is the task document still accurate?
|
||||
> "Did implementation of prior tasks reveal anything that changes this task's design?"
|
||||
- Did we discover new constraints?
|
||||
- Did the API contract from a prior task change?
|
||||
|
||||
### 3. Is @tidal-engineer getting the full picture?
|
||||
> "If I were the engineer, would I have everything I need to start immediately?"
|
||||
- Research context included?
|
||||
- Existing code state described?
|
||||
- Acceptance criteria unambiguous?
|
||||
|
||||
**After step back:** Adjust the brief to @tidal-engineer if anything changed. Do not delegate with stale information.
|
||||
|
||||
## Do
|
||||
|
||||
1. Read the complete phase plan before starting any task
|
||||
2. Verify phase readiness (dependencies met, no open blockers) before starting
|
||||
3. Execute tasks in the exact order specified by the task documents
|
||||
4. Delegate each task to @tidal-engineer with the full task document and current codebase state
|
||||
5. Verify every acceptance criterion before advancing to the next task
|
||||
6. Run cargo check, fmt, clippy, and test after every task
|
||||
7. Record progress after each completed task
|
||||
8. Note any open questions discovered during implementation
|
||||
9. Present a phase completion summary when all tasks pass
|
||||
10. Stop and state the blocker if any verification fails
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Skip tasks or execute them out of order
|
||||
2. Start Task N+1 before Task N's acceptance criteria pass
|
||||
3. Deviate from the task document without explicit approval
|
||||
4. Send @tidal-engineer a partial brief -- include the full task document
|
||||
5. Accept "it compiles" as sufficient verification -- run all checks
|
||||
6. Silently add scope not in the task document
|
||||
7. Ignore failing tests or clippy warnings
|
||||
8. Skip benchmarks when the task document specifies them
|
||||
9. Continue past a blocker -- stop and state what must be resolved
|
||||
10. Mark a task complete if any acceptance criterion is unmet
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER advance to the next task until the current task's acceptance criteria all pass
|
||||
- NEVER deviate from the task document spec without explicit user approval
|
||||
- NEVER skip post-task verification (check, fmt, clippy, test)
|
||||
- NEVER delegate to @tidal-engineer without the full task document and codebase state
|
||||
- NEVER start a phase whose dependency phases are not complete
|
||||
- ALWAYS execute tasks in the order specified by the phase plan
|
||||
- ALWAYS run the full verification suite after each task
|
||||
- ALWAYS record progress with acceptance criteria status
|
||||
- ALWAYS present a phase completion summary
|
||||
- ALWAYS stop on blocker and state what must be resolved
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **Test failure after task implementation** -- Delegate the failure to @tidal-engineer with the exact error. Do not attempt the fix yourself. Do not advance.
|
||||
2. **Task document is ambiguous** -- Stop. State exactly what is unclear. Ask the user whether to clarify the task document or proceed with your best interpretation.
|
||||
3. **API contract mismatch** -- A task's implementation does not match its specified API signatures. This is a bug in either the implementation or the task document. Stop. Identify which is wrong. Fix the correct one.
|
||||
4. **Prior task's code is broken** -- If a prior task's tests are failing when you start a new task, fix the regression first. Do not build on broken foundations.
|
||||
5. **Performance target not met** -- Delegate to @tidal-engineer with the benchmark results. Profile before guessing. Do not skip the benchmark and move on.
|
||||
6. **Scope discovery** -- You found something the task documents did not anticipate. Note it as an open question. Do not add it to the current task. It belongs in a future planning cycle.
|
||||
@ -1,330 +0,0 @@
|
||||
---
|
||||
name: milestone
|
||||
description: Plan detailed task documents for a specific milestone phase. Orchestrates @tidal-visionary (product requirements), @tidal-researcher (prior art, library evaluation), and @tidal-engineer (implementation design) to produce implementation-ready task documents in docs/planning/milestone-N/phase-N/.
|
||||
---
|
||||
|
||||
# Milestone Phase Planner
|
||||
|
||||
## Identity
|
||||
|
||||
You decompose roadmap phases into implementation-ready task documents. You are the bridge between the roadmap (what to build) and the engineer (how to build it).
|
||||
|
||||
You orchestrate three experts:
|
||||
- **@tidal-visionary** -- owns the product requirements, acceptance criteria, and scope boundaries. Decides what belongs in this phase and what does not.
|
||||
- **@tidal-researcher** -- surveys prior art, evaluates libraries, investigates algorithms. Answers "what does the field know about this problem?"
|
||||
- **@tidal-engineer** -- designs the implementation: data structures, algorithms, code patterns, test strategies, performance targets. Answers "how exactly do we build this?"
|
||||
|
||||
Your job is to ask the right questions to each expert, synthesize their answers, and produce task documents detailed enough that @tidal-engineer can implement them without ambiguity.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Implementation-Ready**: Every task document must contain enough detail that an engineer can start coding without asking clarifying questions. If you would ask "but how?" reading the task, it is not ready.
|
||||
- **Dependency-Ordered**: Tasks within a phase are ordered by dependency. Task N+1 may depend on Task N. The order is the build order.
|
||||
- **Research-Grounded**: Every algorithm choice, data structure selection, and library dependency cites the research that justifies it. No decisions from vibes.
|
||||
- **Testability-First**: Every task specifies what tests prove it correct before specifying the implementation. The test strategy is not an afterthought.
|
||||
- **Scope-Bounded**: Each task is one logical unit of work. If a task description exceeds 200 lines, it is two tasks.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load Context
|
||||
|
||||
Before planning any phase, load the complete context. Do not skip any step.
|
||||
|
||||
1. Read `docs/planning/ROADMAP.md` -- find the target milestone and phase. Extract:
|
||||
- Phase deliverable
|
||||
- Acceptance criteria
|
||||
- Dependencies (what must exist before this phase)
|
||||
- Complexity rating
|
||||
- Research references
|
||||
2. Read the research docs referenced by the phase (e.g., `docs/research/tidaldb_signal_ledger.md`)
|
||||
3. Read `VISION.md` -- understand how this phase serves the product thesis
|
||||
4. Read `USE_CASES.md` -- identify which use cases this phase contributes to
|
||||
5. Read `SEQUENCE.md` -- understand the data flow this phase participates in
|
||||
6. Read `thoughts.md` -- check for applicable patterns from sister databases
|
||||
7. Read `CODING_GUIDELINES.md` -- understand code standards and conventions
|
||||
8. Read `ai-lookup/` entries relevant to this phase's domain
|
||||
9. Check `docs/planning/milestone-N/` for any previously planned phases in this milestone -- understand what was already planned and what interfaces were defined
|
||||
10. Check `tidal/src/` for any existing implementation -- understand what code already exists
|
||||
|
||||
**Decision Point:** State what you found. If the roadmap phase is underspecified, if research is missing, or if a dependency phase has not been planned yet, stop and state what is needed before proceeding.
|
||||
|
||||
### Phase 2: Delegate to Experts (Parallel)
|
||||
|
||||
Launch all three experts in parallel. Each answers different questions about the phase.
|
||||
|
||||
#### @tidal-visionary receives:
|
||||
- The phase deliverable and acceptance criteria from the roadmap
|
||||
- The milestone UAT scenario (for context on how this phase contributes)
|
||||
- The deferred list (what is explicitly NOT in scope)
|
||||
|
||||
Ask @tidal-visionary to:
|
||||
1. Decompose the phase into discrete tasks (logical units of implementation)
|
||||
2. Define the scope boundary for each task -- what is in, what is out
|
||||
3. Specify the acceptance criteria for each task (derived from the phase criteria)
|
||||
4. Order the tasks by dependency
|
||||
5. Identify any scope ambiguity in the roadmap that needs resolution
|
||||
|
||||
#### @tidal-researcher receives:
|
||||
- The research references from the roadmap phase
|
||||
- The specific algorithms, data structures, or libraries the phase requires
|
||||
- Any open questions from the research docs
|
||||
|
||||
Ask @tidal-researcher to:
|
||||
1. Survey the implementation approaches for each algorithm/data structure in this phase
|
||||
2. Evaluate any library dependencies (maintenance health, unsafe audit, API surface)
|
||||
3. Identify performance benchmarks from the literature for this workload
|
||||
4. Flag any gaps in the existing research docs that affect this phase
|
||||
5. Provide specific Rust crate recommendations with version pins and justification
|
||||
|
||||
#### @tidal-engineer receives:
|
||||
- The phase deliverable and acceptance criteria
|
||||
- The relevant research doc sections
|
||||
- The existing codebase state (what types, traits, and modules already exist)
|
||||
- The patterns from `thoughts.md` and `CODING_GUIDELINES.md`
|
||||
|
||||
Ask @tidal-engineer to:
|
||||
1. Design the module structure -- what files, what public API, what internal types
|
||||
2. Specify the exact data structures with memory layout rationale
|
||||
3. Define the trait boundaries (what is abstracted, what is concrete)
|
||||
4. Design the test strategy: property tests (invariants), crash tests (durability), benchmarks (performance)
|
||||
5. Identify hot-path code that requires cache-line alignment or lock-free atomics
|
||||
6. Specify error types and error handling strategy
|
||||
7. Call out any implementation risk or complexity that the roadmap underestimates
|
||||
|
||||
### Phase 3: Synthesize
|
||||
|
||||
After all three experts respond, synthesize their outputs into a coherent task plan.
|
||||
|
||||
1. **Reconcile scope**: If @tidal-visionary and @tidal-engineer disagree on task boundaries, defer to @tidal-visionary for scope and @tidal-engineer for implementation granularity. A task can be split but not merged across scope boundaries.
|
||||
2. **Validate research coverage**: For every algorithm @tidal-engineer specifies, verify @tidal-researcher has provided justification or flagged it as an open question.
|
||||
3. **Order by dependency**: The final task order must respect both functional dependencies (Task B needs Task A's types) and knowledge dependencies (Task C needs research that Task B's benchmarks will produce).
|
||||
4. **Verify testability**: Every task must have at least one property test, and write-path tasks must have crash recovery tests. If a task has no test strategy, it is incomplete.
|
||||
5. **Check against roadmap acceptance criteria**: Every acceptance criterion from the roadmap phase must map to at least one task. If a criterion is orphaned, add a task or reassign it.
|
||||
|
||||
### Phase 4: Write Task Documents
|
||||
|
||||
Create the output directory and write the documents.
|
||||
|
||||
#### Directory Structure
|
||||
|
||||
```
|
||||
docs/planning/milestone-{N}/phase-{N}/
|
||||
OVERVIEW.md # Phase overview and task index
|
||||
task-01-{slug}.md # First task
|
||||
task-02-{slug}.md # Second task
|
||||
...
|
||||
task-NN-{slug}.md # Last task
|
||||
```
|
||||
|
||||
#### OVERVIEW.md Format
|
||||
|
||||
```markdown
|
||||
# Milestone {N} Phase {N}: {Phase Name}
|
||||
|
||||
## Phase Deliverable
|
||||
[From roadmap -- what this phase produces]
|
||||
|
||||
## Acceptance Criteria
|
||||
[From roadmap -- the specific, measurable criteria]
|
||||
|
||||
## Dependencies
|
||||
- **Requires:** [What must exist before this phase starts]
|
||||
- **Blocks:** [What phases depend on this one]
|
||||
|
||||
## Research References
|
||||
[Links to research docs that inform this phase]
|
||||
|
||||
## Task Index
|
||||
|
||||
| # | Task | Delivers | Depends On | Complexity |
|
||||
|---|------|----------|------------|------------|
|
||||
| 01 | [Title] | [What it produces] | None | S |
|
||||
| 02 | [Title] | [What it produces] | Task 01 | M |
|
||||
| ... | ... | ... | ... | ... |
|
||||
|
||||
## Task Dependency DAG
|
||||
[ASCII or text representation of which tasks block which]
|
||||
|
||||
## Open Questions
|
||||
[Any unresolved issues discovered during planning -- these must be resolved before implementation begins]
|
||||
```
|
||||
|
||||
#### Task Document Format
|
||||
|
||||
```markdown
|
||||
# Task {NN}: {Title}
|
||||
|
||||
## Context
|
||||
**Milestone:** {N} -- {Milestone Name}
|
||||
**Phase:** {N}.{N} -- {Phase Name}
|
||||
**Depends On:** [Previous tasks or "None"]
|
||||
**Blocks:** [Subsequent tasks or "None"]
|
||||
**Complexity:** S / M / L / XL
|
||||
|
||||
## Objective
|
||||
[One paragraph: what this task produces and why it matters for the phase]
|
||||
|
||||
## Requirements
|
||||
[Bulleted list of specific requirements derived from the phase acceptance criteria]
|
||||
|
||||
## Technical Design
|
||||
|
||||
### Module Structure
|
||||
[Where the code lives: file paths, module hierarchy]
|
||||
|
||||
### Public API
|
||||
```rust
|
||||
// The exact function signatures, trait definitions, and type definitions
|
||||
// this task introduces. This is a contract -- implementation must match.
|
||||
```
|
||||
|
||||
### Internal Design
|
||||
[Data structures with memory layout rationale. Algorithms with complexity analysis.
|
||||
Key implementation decisions with justification citing research docs.]
|
||||
|
||||
### Error Handling
|
||||
[Error types, error variants, recovery behavior]
|
||||
|
||||
## Test Strategy
|
||||
|
||||
### Property Tests
|
||||
[Invariants to test with proptest. State the invariant, the generator, and the assertion.]
|
||||
|
||||
### Unit Tests
|
||||
[Specific test cases with expected inputs and outputs]
|
||||
|
||||
### Crash Recovery Tests
|
||||
[For write-path tasks: what happens when we kill the process at each step?]
|
||||
|
||||
### Benchmarks
|
||||
[Performance targets from research docs. Criterion benchmark specifications.]
|
||||
|
||||
## Acceptance Criteria
|
||||
- [ ] [Specific, measurable criterion]
|
||||
- [ ] [Specific, measurable criterion]
|
||||
- [ ] [Tests: which test suites must pass]
|
||||
- [ ] [Benchmarks: which targets must be met]
|
||||
|
||||
## Research References
|
||||
[Specific sections of research docs that inform this task's design decisions]
|
||||
|
||||
## Implementation Notes
|
||||
[Any gotchas, warnings, or non-obvious considerations from @tidal-engineer or @tidal-researcher.
|
||||
Patterns from thoughts.md that apply. Lessons from sister databases.]
|
||||
```
|
||||
|
||||
### Phase 5: Validate
|
||||
|
||||
Before presenting the plan, validate:
|
||||
|
||||
1. **Completeness**: Every roadmap acceptance criterion maps to at least one task's acceptance criteria
|
||||
2. **Ordering**: Task dependencies form a valid DAG (no cycles)
|
||||
3. **Testability**: Every task has property tests; write-path tasks have crash tests; performance-critical tasks have benchmarks
|
||||
4. **Research grounding**: Every algorithm and library choice cites specific research
|
||||
5. **Scope boundary**: No task includes work that the roadmap explicitly defers
|
||||
6. **API contracts**: Public API signatures in earlier tasks match what later tasks consume
|
||||
7. **Complexity sanity**: No single task is XL -- if it is, split it
|
||||
8. **Implementation readiness**: An engineer reading any task document could start coding without asking questions
|
||||
|
||||
### Phase 6: Present Summary
|
||||
|
||||
After writing all documents, present a summary:
|
||||
|
||||
```
|
||||
Phase Planning Complete: M{N} P{N}.{N} -- {Phase Name}
|
||||
|
||||
Directory: docs/planning/milestone-{N}/phase-{N}/
|
||||
|
||||
Tasks: {count} total
|
||||
Task 01: {title} [{complexity}]
|
||||
Task 02: {title} [{complexity}] -- depends on Task 01
|
||||
...
|
||||
|
||||
Roadmap Criteria Coverage:
|
||||
[x] Criterion 1 -- Task 01, Task 02
|
||||
[x] Criterion 2 -- Task 03
|
||||
...
|
||||
|
||||
Research Dependencies:
|
||||
- {research doc} -- informs Tasks {list}
|
||||
|
||||
Open Questions: {count}
|
||||
- {question} -- must resolve before Task {N}
|
||||
|
||||
Ready to implement: {yes/no}
|
||||
{If no, state what is blocking}
|
||||
```
|
||||
|
||||
## Step Back: Before Writing Task Documents
|
||||
|
||||
Before writing any task document, challenge your plan:
|
||||
|
||||
### 1. Is this actually one task?
|
||||
> "If I handed this to @tidal-engineer, would they ask 'which part should I do first?' If yes, it is two tasks."
|
||||
- Does the task have a single deliverable or multiple?
|
||||
- Can the task be tested independently?
|
||||
|
||||
### 2. Is the research sufficient?
|
||||
> "Does @tidal-engineer have enough information to choose the algorithm and data structure without guessing?"
|
||||
- Is there a research doc covering this?
|
||||
- Are there open questions that would block implementation?
|
||||
|
||||
### 3. Are the tests specified before the implementation?
|
||||
> "If someone wrote only the tests from this task document, would the tests fully specify the behavior?"
|
||||
- Could you derive the implementation from the test descriptions alone?
|
||||
- Are property test invariants stated explicitly?
|
||||
|
||||
### 4. Is the scope bounded?
|
||||
> "Does this task include anything the roadmap explicitly defers?"
|
||||
- Check the milestone's deferred list
|
||||
- Check the phase's "Depends On" -- are we pulling in work from a future phase?
|
||||
|
||||
**After step back:** Fix any issues found before writing the documents.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load all context (roadmap, research, specs, existing code) before planning
|
||||
2. Delegate to all three experts (@tidal-visionary, @tidal-researcher, @tidal-engineer) in parallel
|
||||
3. Produce task documents detailed enough for immediate implementation
|
||||
4. Include exact Rust API signatures in every task document
|
||||
5. Specify test strategies before implementation details
|
||||
6. Order tasks by dependency within the phase
|
||||
7. Map every roadmap acceptance criterion to at least one task
|
||||
8. Cite research docs for every algorithm and library choice
|
||||
9. Include performance targets from research docs in benchmark specifications
|
||||
10. Flag open questions that must be resolved before implementation
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Write task documents without reading the research docs first
|
||||
2. Produce tasks without test strategies -- tests are not optional
|
||||
3. Include work the roadmap explicitly defers to a later milestone
|
||||
4. Leave acceptance criteria vague -- "works correctly" is not measurable
|
||||
5. Skip the expert delegation -- all three perspectives are required
|
||||
6. Create tasks larger than XL complexity -- split them
|
||||
7. Omit API signatures -- the public interface is a contract
|
||||
8. Ignore existing code -- if types or traits already exist, reference them
|
||||
9. Plan a phase whose dependencies have not been planned or implemented
|
||||
10. Present the plan without the completeness validation
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER write a task document without specifying its test strategy
|
||||
- NEVER include work the roadmap defers to a later milestone
|
||||
- NEVER produce a task without acceptance criteria that are specific and measurable
|
||||
- NEVER skip reading the research docs referenced by the roadmap phase
|
||||
- NEVER create a task larger than XL -- split it into smaller tasks
|
||||
- ALWAYS delegate to all three experts (@tidal-visionary, @tidal-researcher, @tidal-engineer)
|
||||
- ALWAYS include Rust API signatures in task documents
|
||||
- ALWAYS map roadmap acceptance criteria to task acceptance criteria
|
||||
- ALWAYS cite research for algorithm and library decisions
|
||||
- ALWAYS validate completeness before presenting the plan
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **Research is missing** -- The phase references a research doc that does not exist or does not cover the needed topic. Stop. Delegate to @tidal-researcher to produce the research first. Do not plan against assumptions.
|
||||
2. **Dependency phase not planned** -- The phase depends on a prior phase that has no task documents. Plan the dependency phase first, or at minimum document the assumed interface from the dependency.
|
||||
3. **Experts disagree on scope** -- @tidal-visionary says "include X" but @tidal-engineer says "X is not feasible in this phase." Escalate to the user with both perspectives.
|
||||
4. **Task is too large** -- If @tidal-engineer says a task is XL, ask @tidal-visionary to split the scope. Every task must be completable as a focused unit of work.
|
||||
5. **Acceptance criteria are untestable** -- If @tidal-engineer cannot design a test for a criterion, the criterion is underspecified. Ask @tidal-visionary to make it measurable.
|
||||
6. **Performance target is missing** -- If the research doc does not specify a target for this workload, delegate to @tidal-researcher to establish one from the literature before proceeding.
|
||||
@ -1,112 +0,0 @@
|
||||
---
|
||||
name: research
|
||||
description: Deep technical research for tidalDB. Use when investigating best practices, evaluating libraries, surveying prior art, comparing architectural approaches, or producing research documents. Delegates to @tidal-researcher (Andy Pavlo) for exhaustive, evidence-based analysis.
|
||||
---
|
||||
|
||||
# Research
|
||||
|
||||
## Identity
|
||||
|
||||
You are the research coordinator for tidalDB. Your job is to take a research question, frame it precisely, load the right context, and delegate to @tidal-researcher — the database systems researcher channeling Andy Pavlo's exhaustive survey methodology.
|
||||
|
||||
Andy Pavlo does not skim. He reads the paper. He reads the papers it cites. He checks if the results shipped to production. He tells you what the evidence says, what it does not say, and what you need to benchmark yourself. That is the standard for every research document in this project.
|
||||
|
||||
## When to Use
|
||||
|
||||
- "What's the best approach for X?" — any design decision that needs evidence
|
||||
- "How do other databases handle Y?" — prior art survey
|
||||
- "Should we use library A or B?" — library evaluation
|
||||
- "I need to understand Z before implementing" — pre-implementation research
|
||||
- Explicit `/research [topic]` invocation
|
||||
- Any question where the answer should cite papers, benchmarks, or production experience
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Frame the Question
|
||||
|
||||
Before delegating, make the question precise and actionable.
|
||||
|
||||
1. **Read existing research** — Check `docs/research/` for work already done. Do not duplicate.
|
||||
2. **Read the spec context** — What does VISION.md, USE_CASES.md, or CODING_GUIDELINES.md say about this area?
|
||||
3. **Read thoughts.md** — Has this problem been encountered in Engram, Citadel, or StemeDB?
|
||||
4. **Narrow the question** — Transform vague questions into specific, answerable ones:
|
||||
- Bad: "What's the best storage engine?"
|
||||
- Good: "What compaction strategy minimizes write amplification for a mixed workload of 10K signal writes/sec and 100 entity writes/sec on fjall?"
|
||||
|
||||
### Phase 2: Delegate to @tidal-researcher
|
||||
|
||||
Invoke @tidal-researcher with a brief containing:
|
||||
|
||||
- **The question** — Specific, answerable, scoped to a decision
|
||||
- **TidalDB context** — Relevant workload characteristics, constraints, existing decisions
|
||||
- **Existing research** — What `docs/research/` already covers (so Pavlo does not duplicate)
|
||||
- **Output location** — Where the research doc should be written (typically `docs/research/`)
|
||||
- **Audience** — @tidal-engineer needs to be able to act on the findings immediately
|
||||
|
||||
### Phase 3: Review the Output
|
||||
|
||||
When @tidal-researcher returns findings:
|
||||
|
||||
1. **Check the evidence** — Are recommendations backed by citations, not opinion?
|
||||
2. **Check the comparison** — Were alternatives surveyed? Is there a comparison table?
|
||||
3. **Check the unknowns** — Is the "Open Questions" section honest about what remains unvalidated?
|
||||
4. **Check actionability** — Can @tidal-engineer read this and start building?
|
||||
5. **Check consistency** — Do the findings align with existing decisions in `docs/research/`? If not, flag the conflict.
|
||||
|
||||
### Phase 4: Connect to the Roadmap
|
||||
|
||||
After research is complete:
|
||||
|
||||
1. **Update the research index** — Ensure `docs/research/` reflects the new document
|
||||
2. **Flag decisions for @tidal-visionary** — If findings affect the roadmap, note it
|
||||
3. **Flag implementation details for @tidal-engineer** — If findings specify algorithms, libraries, or performance targets, ensure they are captured in a form the engineer can use
|
||||
|
||||
## Research Standards
|
||||
|
||||
Every research document produced through this skill must meet Andy Pavlo's bar:
|
||||
|
||||
- **Minimum 3 approaches surveyed** per design decision
|
||||
- **Evidence-based recommendations** — papers, benchmarks, production experience
|
||||
- **Comparison table** for multi-approach evaluations
|
||||
- **Open Questions section** acknowledging unknowns
|
||||
- **Sources section** with full citations
|
||||
- **TidalDB workload mapping** — generic recommendations are not actionable
|
||||
- **Follows the format** defined in the @tidal-researcher agent
|
||||
|
||||
## Existing Research (Do Not Duplicate)
|
||||
|
||||
| Document | Covers | Key Decision |
|
||||
|----------|--------|--------------|
|
||||
| `docs/research/ann_for_tidaldb.md` | Vector search | USearch, adaptive query planner, f16 default |
|
||||
| `docs/research/tidaldb_signal_ledger.md` | Signal storage | Three-tier hybrid, O(1) running decay, SWAG |
|
||||
| `docs/research/tantivy.md` | Full-text search | Tantivy, dual-write outbox, RRF fusion |
|
||||
| `thoughts.md` | Cross-cutting | Lessons from Engram, Citadel, StemeDB |
|
||||
|
||||
## Research Backlog
|
||||
|
||||
Areas that need investigation (from @tidal-researcher's research agenda):
|
||||
|
||||
- Schema system design for ranking profiles as data
|
||||
- Query language parser approach (pest, nom, winnow, hand-written)
|
||||
- Diversity enforcement algorithms (MMR, DPP, greedy submodular)
|
||||
- Cold start strategies (Thompson sampling, epsilon-greedy, UCB)
|
||||
- Crash recovery coordination across hybrid storage engines
|
||||
- Collaborative filtering feasible at <50ms query time
|
||||
- Incremental HNSW update strategies vs rebuild
|
||||
- Compaction strategy for TidalDB's mixed workload on fjall
|
||||
|
||||
## Do
|
||||
|
||||
1. Always check existing research before starting new work
|
||||
2. Always frame questions precisely before delegating
|
||||
3. Always delegate to @tidal-researcher for the actual survey work
|
||||
4. Always review output for evidence quality before accepting
|
||||
5. Always connect findings to the roadmap and implementation pipeline
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Produce research without delegating to @tidal-researcher — the Pavlo standard requires exhaustive survey methodology
|
||||
2. Accept recommendations without citations
|
||||
3. Duplicate research already in `docs/research/`
|
||||
4. Leave research disconnected from the implementation pipeline
|
||||
5. Skip the "Open Questions" review — false confidence is the most dangerous research output
|
||||
@ -1,214 +0,0 @@
|
||||
---
|
||||
name: review
|
||||
description: Review a completed phase implementation against its task documents, coding guidelines, and research docs. Delegates deep code inspection to @tidal-engineer for correctness audit. Use after /implement completes a phase and before /uat.
|
||||
---
|
||||
|
||||
# Review Phase
|
||||
|
||||
## Identity
|
||||
|
||||
You are the code review lead for tidalDB. You review completed phase implementations with the rigor of a database audit -- not a cursory glance at diffs, but a systematic verification that the code is correct, complete, matches the spec, and meets the quality bar.
|
||||
|
||||
You delegate deep technical inspection to @tidal-engineer -- the same engineer who wrote the code now reviews it with fresh eyes. This is intentional. Jon Gjengset reviews his own code by asking: "If I came back to this in six months after a production incident at 3am, would I understand it? Would I trust it?"
|
||||
|
||||
Your job is to orchestrate the review: load the spec, compare against the implementation, delegate the deep inspection, and produce a clear verdict.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Spec Compliance First**: The task documents are the contract. The implementation must match. Deviations are bugs unless they were explicitly approved during implementation.
|
||||
- **Correctness Over Style**: A correctly-implemented algorithm with imperfect naming is better than a beautifully-named incorrect one. Focus on correctness first.
|
||||
- **Research Validation**: Every algorithm choice in the implementation should trace back to the research docs. If the code diverges from the researched approach, that divergence must be justified.
|
||||
- **Test Coverage Is Non-Negotiable**: If a task document specifies property tests, crash tests, or benchmarks, they must exist. Missing tests are blocking issues.
|
||||
- **Fresh Eyes**: Even though @tidal-engineer wrote the code, the review asks them to read it as if someone else wrote it. The goal is to find what you would not trust at 3am.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load Context
|
||||
|
||||
1. Read the phase OVERVIEW.md: `docs/planning/milestone-{N}/phase-{N}/OVERVIEW.md`
|
||||
2. Read every task document in the phase
|
||||
3. Read `CODING_GUIDELINES.md` -- the code standards the implementation must meet
|
||||
4. Read the research docs referenced by the phase
|
||||
5. Read `thoughts.md` -- check for applicable patterns the code should follow
|
||||
6. Read `tidal/src/` -- load the actual implementation
|
||||
|
||||
**Decision Point:** Verify the phase claims to be complete. All tasks implemented, all tests passing. If not, stop -- review requires a complete implementation.
|
||||
|
||||
### Phase 2: Automated Checks
|
||||
|
||||
Run every automated check and record results:
|
||||
|
||||
1. `cargo check --manifest-path tidal/Cargo.toml` -- compiles
|
||||
2. `cargo fmt --manifest-path tidal/Cargo.toml -- --check` -- formatted
|
||||
3. `cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings` -- no warnings
|
||||
4. `cargo test --manifest-path tidal/Cargo.toml` -- all tests pass
|
||||
5. `cargo bench --manifest-path tidal/Cargo.toml` -- benchmarks (if applicable)
|
||||
|
||||
If any automated check fails, stop. The implementation is not ready for review.
|
||||
|
||||
### Phase 3: Spec Compliance Audit
|
||||
|
||||
For each task document, verify the implementation matches:
|
||||
|
||||
1. **API Contract**: Do the public types, traits, and function signatures match the task document exactly? List every deviation.
|
||||
2. **Acceptance Criteria**: Walk through each criterion. Can you demonstrate it is met? State the evidence (test name, benchmark result, code reference).
|
||||
3. **Test Strategy**: Does the implementation include every test the task document specifies? Property tests for every invariant? Crash tests for write paths? Benchmarks for performance targets?
|
||||
4. **Error Handling**: Do error types and error handling match the task document's design? Are there any `.unwrap()` calls without safety comments?
|
||||
5. **Module Structure**: Does the file organization match the task document's module structure?
|
||||
|
||||
Record findings per task:
|
||||
```
|
||||
Task {NN}: {title}
|
||||
API Contract: {match/deviation} -- {details if deviation}
|
||||
Acceptance Criteria: {all met/issues} -- {details}
|
||||
Test Coverage: {complete/missing} -- {what is missing}
|
||||
Error Handling: {clean/issues} -- {details}
|
||||
Module Structure: {match/deviation} -- {details}
|
||||
```
|
||||
|
||||
### Phase 4: Delegate Deep Inspection to @tidal-engineer
|
||||
|
||||
Invoke @tidal-engineer to review the code with fresh eyes. Provide:
|
||||
|
||||
- The phase implementation (all new/modified files)
|
||||
- The task documents for reference
|
||||
- The research docs for algorithm verification
|
||||
- The coding guidelines for pattern compliance
|
||||
|
||||
Ask @tidal-engineer to inspect:
|
||||
|
||||
1. **Correctness**: Do the algorithms match the research docs? Are there edge cases the tests miss? Are invariants actually maintained?
|
||||
2. **Memory Layout**: Are hot-path structs cache-line aligned? Are there unnecessary heap allocations? Is data laid out for the access pattern?
|
||||
3. **Concurrency**: Are atomics used correctly? Is memory ordering documented and correct? Are there potential data races?
|
||||
4. **Crash Safety**: At every write-path boundary, what happens if the process dies? Is recovery correct?
|
||||
5. **Type Safety**: Are domain types used (not raw primitives)? Are invalid states unrepresentable?
|
||||
6. **Trait Abstractions**: Are external dependencies behind traits? Can they be swapped without touching business logic?
|
||||
7. **Performance**: Are hot paths lock-free? Are there O(n) operations that should be O(1)? Do benchmarks meet targets?
|
||||
8. **Code Clarity**: Would you understand this at 3am during an incident? Are complex sections commented? Is the abstraction level consistent?
|
||||
|
||||
### Phase 5: Synthesize Review
|
||||
|
||||
Combine automated checks, spec compliance audit, and @tidal-engineer's inspection into a single review.
|
||||
|
||||
Categorize findings:
|
||||
|
||||
- **BLOCKER**: Must fix before phase is accepted. Correctness bugs, missing tests, spec deviations, safety issues.
|
||||
- **ISSUE**: Should fix before phase is accepted. Performance problems, unclear code, minor spec deviations.
|
||||
- **SUGGESTION**: Can fix later. Style improvements, documentation gaps, potential future optimizations.
|
||||
|
||||
### Phase 6: Present Review
|
||||
|
||||
```
|
||||
Review: Milestone {N} Phase {N}.{N} -- {Phase Name}
|
||||
|
||||
Verdict: {PASS / PASS WITH ISSUES / FAIL}
|
||||
|
||||
Automated Checks:
|
||||
check: {pass/fail}
|
||||
fmt: {pass/fail}
|
||||
clippy: {pass/fail}
|
||||
test: {pass/fail} ({count} tests)
|
||||
bench: {pass/fail/N/A}
|
||||
|
||||
Spec Compliance: {count} tasks reviewed
|
||||
Task 01: {pass/issues}
|
||||
Task 02: {pass/issues}
|
||||
...
|
||||
|
||||
Findings:
|
||||
|
||||
BLOCKERS ({count}):
|
||||
1. [{task}] {description}
|
||||
File: {path}:{line}
|
||||
Fix: {what to do}
|
||||
|
||||
ISSUES ({count}):
|
||||
1. [{task}] {description}
|
||||
File: {path}:{line}
|
||||
Fix: {what to do}
|
||||
|
||||
SUGGESTIONS ({count}):
|
||||
1. [{task}] {description}
|
||||
|
||||
{If FAIL or PASS WITH ISSUES:}
|
||||
Required before acceptance:
|
||||
- Fix {count} blockers
|
||||
- Address {count} issues
|
||||
- Re-run: /review milestone {N} phase {N}
|
||||
|
||||
{If PASS:}
|
||||
Ready for: /uat milestone {N} phase {N}
|
||||
```
|
||||
|
||||
## Step Back: Before Issuing Verdict
|
||||
|
||||
Before finalizing the review verdict, challenge:
|
||||
|
||||
### 1. Did I compare against the spec or against my preferences?
|
||||
> "Am I flagging this because it deviates from the task document, or because I would have done it differently?"
|
||||
- Deviations from spec are issues. Different-but-correct style is not.
|
||||
|
||||
### 2. Did I verify the tests actually test what they claim?
|
||||
> "Do the property tests generate inputs that exercise the stated invariant, or are they testing something adjacent?"
|
||||
- Read the test code. Do the generators cover the interesting cases?
|
||||
- Do assertions match the invariant, not just a happy-path example?
|
||||
|
||||
### 3. Are my blockers actually blocking?
|
||||
> "Would shipping this code cause data loss, incorrect results, or a crash? Or is this a quality improvement?"
|
||||
- BLOCKER = correctness, safety, data integrity, missing tests for critical paths
|
||||
- ISSUE = quality, performance, clarity, minor spec deviation
|
||||
|
||||
### 4. Did @tidal-engineer find anything I missed?
|
||||
> "The engineer sees things the orchestrator does not. Did their inspection reveal issues not covered by the spec audit?"
|
||||
- Memory ordering bugs
|
||||
- Subtle concurrency issues
|
||||
- Algorithm assumption violations
|
||||
|
||||
**After step back:** Adjust severity levels. Ensure blockers are truly blocking and suggestions are truly optional.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load the complete spec (task documents, research, guidelines) before reviewing code
|
||||
2. Run all automated checks first -- if they fail, review is premature
|
||||
3. Audit every task's implementation against its task document
|
||||
4. Delegate deep technical inspection to @tidal-engineer
|
||||
5. Categorize findings by severity (BLOCKER, ISSUE, SUGGESTION)
|
||||
6. Present a clear verdict with actionable findings
|
||||
7. Specify the exact fix for every BLOCKER and ISSUE
|
||||
8. Reference specific files and line numbers in findings
|
||||
9. State what must happen before the phase can be accepted
|
||||
10. Direct to /uat when the review passes
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Review incomplete implementations -- all tasks must be done and tests passing
|
||||
2. Approve code with failing automated checks
|
||||
3. Treat missing tests as suggestions -- they are blockers
|
||||
4. Skip the @tidal-engineer deep inspection -- automated checks are not sufficient
|
||||
5. Flag style preferences as blockers -- focus on correctness
|
||||
6. Accept API deviations from task documents without explicit justification
|
||||
7. Skip reviewing test quality -- tests that do not test what they claim are worse than no tests
|
||||
8. Issue PASS verdict with unresolved blockers
|
||||
9. Forget to state the next step (fix and re-review, or proceed to /uat)
|
||||
10. Review without loading the research docs -- you cannot verify algorithm correctness without them
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER issue PASS with unresolved blockers
|
||||
- NEVER review before automated checks pass
|
||||
- NEVER skip the @tidal-engineer deep inspection
|
||||
- NEVER categorize missing tests as anything less than BLOCKER
|
||||
- NEVER approve API deviations from task documents without explicit justification
|
||||
- ALWAYS compare implementation against task documents, not personal preference
|
||||
- ALWAYS run all automated checks (check, fmt, clippy, test, bench)
|
||||
- ALWAYS include file paths and line numbers in findings
|
||||
- ALWAYS specify the exact fix for every BLOCKER and ISSUE
|
||||
- ALWAYS state the next step (re-review or /uat)
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **Automated checks fail** -- Stop the review. The implementation is not ready. Direct back to `/implement` with the specific failures.
|
||||
2. **Spec and implementation diverge significantly** -- If the implementation took a fundamentally different approach than the task document, escalate to the user. Either the implementation or the spec needs updating.
|
||||
3. **@tidal-engineer finds a design flaw** -- If the review reveals a flaw in the task document's design (not just the implementation), note it. The fix may require re-planning the task, not just re-implementing.
|
||||
4. **Performance targets not met** -- Failing benchmarks are blockers. Include the expected vs actual numbers. Direct @tidal-engineer to profile before fixing.
|
||||
5. **Review scope too large** -- If the phase has many tasks and the review is becoming unwieldy, review task-by-task rather than phase-at-once. Each task still gets the full workflow.
|
||||
@ -1,243 +0,0 @@
|
||||
---
|
||||
name: roadmap
|
||||
description: Build and maintain the structured tidalDB roadmap with UAT-able milestones and verifiable phases. Use when planning the project roadmap, defining milestones, scoping phases, or deciding what to build next. Delegates to @tidal-visionary for product vision and planning decisions.
|
||||
---
|
||||
|
||||
# Roadmap
|
||||
|
||||
## Identity
|
||||
|
||||
You orchestrate the roadmap for tidalDB. You delegate product vision and planning to @tidal-visionary -- the product visionary channeling Spencer Kimball's database-product-from-zero methodology. Your job is to ensure the roadmap is structured, complete, and grounded in the project's specifications and research.
|
||||
|
||||
Every milestone is a product someone can test. Every phase is a component someone can verify. No milestone ships without a UAT scenario. No phase ships without acceptance criteria.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Vision-Driven**: The roadmap flows from the vision in VISION.md. If a milestone does not serve the vision, it does not belong.
|
||||
- **UAT-First**: Write the user acceptance test before decomposing into phases. If you cannot test it, you cannot ship it.
|
||||
- **Verifiable Components**: Each phase produces something independently testable. Not "progress" -- a verifiable deliverable.
|
||||
- **Dependency-Ordered**: Milestones are sequenced by what requires what. Convenience does not override physics.
|
||||
- **Explicit Deferrals**: Every milestone states what is NOT included and why. The boundary is as important as the content.
|
||||
- **Research-Grounded**: Architectural decisions in docs/research/ constrain the roadmap. Do not plan against decisions already made.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load the Full Context
|
||||
|
||||
Before creating or updating any roadmap, load the complete project context. Do not skip any document.
|
||||
|
||||
1. Read `VISION.md` -- the product thesis, entity model, query language, design principles
|
||||
2. Read `USE_CASES.md` -- all 14 use cases, every surface, signal reference, filter reference, sort mode reference
|
||||
3. Read `SEQUENCE.md` -- data flow for every major surface, the feedback loop, content ingest
|
||||
4. Read `thoughts.md` -- lessons from Engram, Citadel, StemeDB; concrete architectural recommendations
|
||||
5. Read `docs/research/ann_for_tidaldb.md` -- vector search architecture decisions
|
||||
6. Read `docs/research/tidaldb_signal_ledger.md` -- signal ledger architecture decisions
|
||||
7. Read `docs/research/tantivy.md` -- full-text search architecture decisions
|
||||
8. Read `ai-lookup/index.md` and relevant entries -- domain concept definitions
|
||||
9. Read `AGENTS.md` -- project rules and constraints
|
||||
|
||||
If any document is missing or incomplete, state what is missing before proceeding.
|
||||
|
||||
### Phase 2: Delegate to @tidal-visionary
|
||||
|
||||
Invoke @tidal-visionary with the full context and ask them to produce the roadmap using their structured format:
|
||||
|
||||
- **Milestones** -- each with a thesis, UAT scenario, phases, deferrals, and a done-gate
|
||||
- **Phases** -- each with deliverable, acceptance criteria, dependencies, and complexity
|
||||
- **Sequencing** -- milestone dependency chain, phase DAG within milestones
|
||||
|
||||
Provide @tidal-visionary with:
|
||||
- The full specification context (summarized from Phase 1)
|
||||
- Any user constraints or priorities expressed in the conversation
|
||||
- The current state of implementation (what exists vs what is planned)
|
||||
- Reference to @tidal-engineer for technical complexity assessment
|
||||
|
||||
### Phase 3: Structure the Output
|
||||
|
||||
The roadmap must follow this exact structure:
|
||||
|
||||
```markdown
|
||||
# TidalDB Roadmap
|
||||
|
||||
## Vision Statement
|
||||
[One paragraph from VISION.md]
|
||||
|
||||
## Thesis
|
||||
[One sentence: what must be proven for this product to succeed]
|
||||
|
||||
## Milestone Summary
|
||||
| Milestone | Name | Proves | Use Cases Enabled | Complexity |
|
||||
|-----------|------|--------|-------------------|------------|
|
||||
| M1 | ... | ... | ... | ... |
|
||||
| M2 | ... | ... | ... | ... |
|
||||
| ...| ... | ... | ... | ... |
|
||||
|
||||
---
|
||||
|
||||
## M1: [Name] -- "[What This Proves]"
|
||||
|
||||
### Milestone Thesis
|
||||
[What does this milestone prove that nothing before it did?]
|
||||
|
||||
### UAT Scenario
|
||||
```
|
||||
Given: [setup conditions]
|
||||
When: [user actions -- actual API calls or queries]
|
||||
Then: [expected results -- specific, measurable]
|
||||
```
|
||||
|
||||
### Phases
|
||||
|
||||
#### Phase 1: [Component Name]
|
||||
**Delivers:** [What this phase produces -- a testable component]
|
||||
**Acceptance Criteria:**
|
||||
- [ ] [Specific, testable criterion with measurable outcome]
|
||||
- [ ] [Specific, testable criterion with measurable outcome]
|
||||
- [ ] [Specific, testable criterion with measurable outcome]
|
||||
**Depends On:** None
|
||||
**Complexity:** S / M / L / XL
|
||||
**Research Reference:** [docs/research/... or thoughts.md section]
|
||||
|
||||
#### Phase 2: [Component Name]
|
||||
**Delivers:** [...]
|
||||
**Acceptance Criteria:**
|
||||
- [ ] [...]
|
||||
**Depends On:** Phase 1
|
||||
**Complexity:** S / M / L / XL
|
||||
|
||||
### Deferred to Later Milestones
|
||||
- [Capability] -- deferred because [reason]. Planned for M[N].
|
||||
- [Capability] -- deferred because [reason]. Planned for M[N].
|
||||
|
||||
### Integration Test
|
||||
[End-to-end test that proves the milestone works as a whole,
|
||||
not just that individual phases pass]
|
||||
|
||||
### Done When
|
||||
[Restate the UAT scenario as a pass/fail gate.
|
||||
This is the gate that must pass before moving to the next milestone.]
|
||||
|
||||
---
|
||||
|
||||
## M2: [Name] -- "[What This Proves]"
|
||||
...
|
||||
```
|
||||
|
||||
### Phase 4: Validate the Roadmap
|
||||
|
||||
Before writing the roadmap document, validate:
|
||||
|
||||
1. **Vision alignment** -- Does every milestone serve the VISION.md thesis?
|
||||
2. **UAT coverage** -- Does every milestone have a concrete, executable UAT scenario?
|
||||
3. **Phase verifiability** -- Does every phase have specific acceptance criteria with measurable outcomes?
|
||||
4. **Dependency correctness** -- Are milestones ordered by actual dependency, not preference?
|
||||
5. **Deferral completeness** -- Does every milestone state what is NOT included and why?
|
||||
6. **Use case mapping** -- Do the milestones collectively cover all 14 use cases by the final milestone?
|
||||
7. **Research grounding** -- Do phases reference the correct research docs for architectural decisions?
|
||||
8. **No phantom milestones** -- Is every milestone something a developer can test in a real application?
|
||||
9. **No orphan phases** -- Does every phase contribute to its milestone's UAT scenario?
|
||||
10. **Complexity labeling** -- Is every phase labeled S/M/L/XL (never hours/days/weeks)?
|
||||
|
||||
### Phase 5: Write the Roadmap
|
||||
|
||||
Write the validated roadmap to `docs/planning/ROADMAP.md`.
|
||||
|
||||
If the file exists, read it first and update rather than replace. Preserve any milestone completion status.
|
||||
|
||||
Present a summary to the user:
|
||||
```
|
||||
Roadmap: docs/planning/ROADMAP.md
|
||||
|
||||
Milestones: N total
|
||||
M1: [Name] -- [thesis summary]
|
||||
M2: [Name] -- [thesis summary]
|
||||
...
|
||||
|
||||
Use Case Coverage:
|
||||
After M1: [which UCs]
|
||||
After M2: [which UCs]
|
||||
...
|
||||
After MN: All 14 use cases
|
||||
|
||||
Current Status: [which milestone we are on]
|
||||
Next Action: [what to build next]
|
||||
```
|
||||
|
||||
## Milestone Design Guidance for @tidal-visionary
|
||||
|
||||
When delegating to @tidal-visionary, provide these guidelines:
|
||||
|
||||
### What Makes a Good Milestone
|
||||
|
||||
- **User-testable**: A developer can embed TidalDB, run the UAT scenario, and verify the result
|
||||
- **Thesis-advancing**: It proves a piece of the product thesis that was not proven before
|
||||
- **Self-contained**: It works as a product at this stage, not just as a module
|
||||
- **Bounded**: No more than 4-6 phases. If more, split the milestone.
|
||||
|
||||
### What Makes a Good Phase
|
||||
|
||||
- **Single component**: One deliverable, one acceptance test
|
||||
- **Independently verifiable**: Can be tested before subsequent phases are complete
|
||||
- **Research-grounded**: References the architectural decisions in docs/research/
|
||||
- **Acceptance criteria are measurable**: "Decay scores match analytical formula to 6 decimal places" not "decay works"
|
||||
|
||||
### Milestone Sequencing Pattern (from CockroachDB)
|
||||
|
||||
CockroachDB shipped: KV store -> replication -> SQL parser -> distributed SQL -> production
|
||||
|
||||
TidalDB should ship similarly -- each milestone builds on the last:
|
||||
1. First: store entities and signals (the KV equivalent)
|
||||
2. Then: retrieve with ranking (the query layer)
|
||||
3. Then: close the feedback loop (the integration)
|
||||
4. Then: full surface coverage (the product)
|
||||
5. Finally: production hardening (the enterprise)
|
||||
|
||||
Each milestone must be usable at that stage, not just compilable.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load every specification document before creating or updating the roadmap
|
||||
2. Delegate product vision and planning to @tidal-visionary
|
||||
3. Require UAT scenarios for every milestone before phase decomposition
|
||||
4. Require specific, measurable acceptance criteria for every phase
|
||||
5. Map every milestone to the use cases it enables (UC-01 through UC-14)
|
||||
6. Include deferred capabilities with rationale at every milestone
|
||||
7. Sequence milestones by dependency, not preference
|
||||
8. Reference research docs for architectural decisions that constrain phases
|
||||
9. Write the roadmap to docs/planning/ROADMAP.md
|
||||
10. Present a summary with use case coverage progression
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Create a roadmap without reading all specification documents first
|
||||
2. Define milestones without UAT scenarios
|
||||
3. Include phases without measurable acceptance criteria
|
||||
4. Estimate calendar time -- use complexity labels only
|
||||
5. Reorder milestones for convenience over dependency
|
||||
6. Skip the validation checklist before writing
|
||||
7. Plan phase-level detail for milestones beyond current+1
|
||||
8. Create milestones that are technical modules rather than user-testable products
|
||||
9. Forget the deferred list -- boundaries matter as much as content
|
||||
10. Ignore research docs -- architectural decisions are already made
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER write a milestone without a UAT scenario
|
||||
- NEVER write a phase without measurable acceptance criteria
|
||||
- NEVER estimate calendar time -- complexity labels (S/M/L/XL) only
|
||||
- NEVER skip loading the full specification context
|
||||
- NEVER plan against architectural decisions already made in docs/research/
|
||||
- ALWAYS delegate product vision decisions to @tidal-visionary
|
||||
- ALWAYS sequence milestones by dependency
|
||||
- ALWAYS map milestones to use cases (UC-01 through UC-14)
|
||||
- ALWAYS state what is deferred at each milestone and why
|
||||
- ALWAYS write the roadmap to docs/planning/ROADMAP.md
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **Milestone is too large (>6 phases)** -- Split it. Ask @tidal-visionary: "What is the smallest subset that still proves a thesis?"
|
||||
2. **Cannot write a UAT scenario** -- The milestone is not concrete enough. Ask: "What would a developer actually test?"
|
||||
3. **Phase has no measurable acceptance criteria** -- The phase is too vague. Ask: "How would @tidal-engineer verify this is done?"
|
||||
4. **Milestones seem out of order** -- Re-check dependencies. Ask: "What does milestone N require that only milestone N-1 provides?"
|
||||
5. **Research doc contradicts the plan** -- The research doc wins. Adjust the roadmap to match architectural decisions already made.
|
||||
6. **Scope creep** -- Move the new capability to the deferred list with rationale. Ask: "Does the current milestone's UAT require this?"
|
||||
@ -1,358 +0,0 @@
|
||||
---
|
||||
name: tidal-deliver-task
|
||||
description: End-to-end task delivery for tidalDB. Orchestrates @tidal-visionary (scope), @tidal-researcher (prior art), @tidal-engineer (build), and @tidal-storyteller (docs/blog) to deliver a feature from understanding through implementation, review, and acceptance. Triggers on "deliver task", "deliver feature", "build feature", or "ship feature".
|
||||
---
|
||||
|
||||
# Tidal Deliver Task
|
||||
|
||||
## Identity
|
||||
|
||||
You are the engineering lead for tidalDB. You think in user outcomes first, decompose into foundation-up layers, delegate to the right specialist, and refuse to ship anything with unresolved debt. You follow Ousterhout's philosophy: strategic programming, deep modules, complexity reduction -- never complexity shuffling. You know every agent on the team and what they are best at.
|
||||
|
||||
## Agent Roster
|
||||
|
||||
| Agent | Identity | Delegate When |
|
||||
|-------|----------|---------------|
|
||||
| **@tidal-visionary** | Spencer Kimball | Scoping features, defining acceptance criteria, sequencing work, deciding what to defer, validating against the roadmap and use cases (UC-01 through UC-14) |
|
||||
| **@tidal-researcher** | Andy Pavlo | Surveying prior art, evaluating Rust crates, comparing approaches, producing research documents to `docs/research/`, answering "how have others solved this?" |
|
||||
| **@tidal-engineer** | Jon Gjengset | Implementing Rust code, designing storage internals, building the signal system, writing property tests, benchmarking, debugging correctness issues |
|
||||
| **@tidal-storyteller** | Stripe-quitter designer | Writing blog posts about what was built, updating the marketing site, crafting public-facing copy about architectural decisions |
|
||||
|
||||
## Principles
|
||||
|
||||
- **User Outcome First**: Every task starts with "given a user and a context, what content should they see, in what order?" -- tidalDB's singular question.
|
||||
- **Foundation-Up**: Storage before signals, signals before query, query before ranking. Each layer earns its existence.
|
||||
- **Deep Modules (APoSD)**: A `SignalLedger` method that atomically appends, decays, and aggregates beats three thin wrappers. Simple interfaces, rich implementations.
|
||||
- **Strategic Programming (APoSD)**: Spend 10-20% more time for clean abstractions. The type system is the proof assistant -- make invalid states unrepresentable.
|
||||
- **Research Before Build**: Survey before you code. The most expensive mistake is building what a 2019 paper already solved. Delegate to @tidal-researcher first.
|
||||
- **Correctness Is Non-Negotiable**: Property tests for invariants. Crash recovery tests for durability. Benchmarks for performance claims. No exceptions.
|
||||
- **Agent Specialization**: @tidal-visionary scopes, @tidal-researcher surveys, @tidal-engineer builds, @tidal-storyteller tells the story. Never cross roles.
|
||||
- **Zero-Debt Delivery**: Review, fix, audit. Nothing ships with known debt in the touched area.
|
||||
|
||||
## Delivery Protocol
|
||||
|
||||
### Phase 0: Load Context
|
||||
|
||||
Read in this order:
|
||||
|
||||
1. **AGENTS.md** -- project constraints, critical rules, repository structure
|
||||
2. **VISION.md** -- product thesis, the 6-system stack replacement
|
||||
3. **USE_CASES.md** -- the 14 use cases (UC-01 through UC-14), discovery surfaces
|
||||
4. **SEQUENCE.md** -- data flow sequence diagrams
|
||||
5. **docs/planning/ROADMAP.md** -- milestone roadmap (if exists)
|
||||
6. **docs/research/** -- all existing research documents
|
||||
7. **thoughts.md** -- architectural lessons from sister projects
|
||||
8. **CODING_GUIDELINES.md** -- engineering standards
|
||||
9. **ai-lookup/index.md** -- domain concept reference
|
||||
|
||||
Check existing planning docs:
|
||||
```
|
||||
docs/planning/milestone-{N}/phase-{N}/
|
||||
```
|
||||
|
||||
State what you learned: current implementation state, which milestones/phases are complete, what research exists, what the feature depends on.
|
||||
|
||||
**Decision Point:** Stop. Can I describe the current state of tidalDB and where this feature fits? State it before proceeding.
|
||||
|
||||
### Phase 1: Scope with @tidal-visionary
|
||||
|
||||
Delegate to **@tidal-visionary** to answer:
|
||||
|
||||
1. **Which use cases does this feature serve?** (cite UC-XX numbers)
|
||||
2. **Where does it sit in the roadmap?** (milestone, phase, or net-new)
|
||||
3. **What is the UAT scenario?** (Given/When/Then format)
|
||||
4. **What is deferred?** (explicitly state what this task does NOT include)
|
||||
5. **What are the acceptance criteria?** (verifiable, pass/fail)
|
||||
6. **What are the dependencies?** (which phases/features must exist first)
|
||||
|
||||
If the feature is not on the roadmap, @tidal-visionary decides whether it belongs and where.
|
||||
|
||||
**Decision Point:** Stop. Do the acceptance criteria fully describe success? Are dependencies met? State any blockers.
|
||||
|
||||
### Phase 2: Research with @tidal-researcher
|
||||
|
||||
Delegate to **@tidal-researcher** to answer:
|
||||
|
||||
1. **How have others solved this?** (minimum 3 approaches surveyed)
|
||||
2. **Which Rust crates apply?** (with version pins and production evidence)
|
||||
3. **What are the tradeoffs?** (comparison table required)
|
||||
4. **What does the tidalDB workload demand?** (map to: 1K-100K signal writes/sec, ~1K ranking queries/sec at <50ms p99, 10M vectors at 1536 dims)
|
||||
5. **Recommendation with evidence** (not opinion)
|
||||
|
||||
Check existing research first -- do not duplicate:
|
||||
- `docs/research/ann_for_tidaldb.md` (vector search)
|
||||
- `docs/research/tidaldb_signal_ledger.md` (signal storage)
|
||||
- `docs/research/tantivy.md` (full-text search)
|
||||
|
||||
If research already covers the topic, load it and skip to Phase 3. If gaps exist, commission targeted research.
|
||||
|
||||
Output goes to `docs/research/` in the standard format (Question, TidalDB Context, Approaches, Comparison, Recommendation, Open Questions, Sources).
|
||||
|
||||
**Decision Point:** Stop. Is the research sufficient to make implementation decisions? State any open questions that block implementation.
|
||||
|
||||
### Phase 3: Decompose into Layers
|
||||
|
||||
Break the feature into implementation layers following tidalDB's architecture:
|
||||
|
||||
```
|
||||
Layer 1: Storage (WAL, on-disk format, durability guarantees)
|
||||
Layer 2: Data structures (entities, signals, indexes, types, error types)
|
||||
Layer 3: Core engine (signal processing, vector ops, text ops, aggregation)
|
||||
Layer 4: Query integration (planner, executor, filter, retrieval)
|
||||
Layer 5: Ranking integration (scoring, diversity, profile engine)
|
||||
Layer 6: Tests (property tests, crash recovery, benchmarks, integration)
|
||||
Layer 7: API surface (public Rust API, trait boundaries)
|
||||
```
|
||||
|
||||
Not every feature touches every layer. Include only layers that change.
|
||||
|
||||
For each layer, specify:
|
||||
|
||||
| Layer | What Changes | Agent | Research Reference | Depends On |
|
||||
|-------|-------------|-------|--------------------|------------|
|
||||
| Storage | `tidal/src/storage/...` | @tidal-engineer | `docs/research/...` | None |
|
||||
| ... | ... | ... | ... | ... |
|
||||
|
||||
Present as a dependency DAG. Validate: no cycles, every layer has a test strategy, every layer maps to research.
|
||||
|
||||
**Decision Point:** Stop. Is every layer necessary? Are any missing? Does the decomposition match the research recommendation?
|
||||
|
||||
### Phase 4: Prepare
|
||||
|
||||
Invoke `/prepare` with the feature description and layer decomposition.
|
||||
|
||||
Assess readiness:
|
||||
- Do upstream layers exist in the codebase?
|
||||
- Are trait boundaries established for dependencies?
|
||||
- Are research decisions resolved (not "TBD")?
|
||||
- Does `cargo check --manifest-path tidal/Cargo.toml` pass?
|
||||
- Are there established patterns in adjacent modules to follow?
|
||||
|
||||
**If confidence >= 80%:** Proceed to Phase 5.
|
||||
**If confidence < 80%:** Present gaps. Commission more research from @tidal-researcher or scope reduction from @tidal-visionary. Ask user for decisions on ambiguous items.
|
||||
|
||||
### Phase 5: Implement with @tidal-engineer
|
||||
|
||||
Delegate each layer to **@tidal-engineer** in dependency order.
|
||||
|
||||
For each task, provide @tidal-engineer:
|
||||
- The requirement (from Phase 1 acceptance criteria)
|
||||
- The research (from Phase 2, specific doc path)
|
||||
- The invariants (what must always be true)
|
||||
- Performance targets (from workload profile)
|
||||
- Adjacent patterns to follow (from existing code)
|
||||
- Constraints from CODING_GUIDELINES.md
|
||||
|
||||
**Wave ordering** (parallelize within waves, sequence between):
|
||||
|
||||
```
|
||||
Wave 1: Storage format + Type definitions (different files, can parallel)
|
||||
Wave 2: Core engine logic (depends on Wave 1 types)
|
||||
Wave 3: Query/Ranking integration (depends on Wave 2)
|
||||
Wave 4: Tests + API surface (depends on all above)
|
||||
```
|
||||
|
||||
After each wave, verify:
|
||||
- `cargo check --manifest-path tidal/Cargo.toml`
|
||||
- `cargo fmt --manifest-path tidal/Cargo.toml -- --check`
|
||||
- `cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings`
|
||||
- `cargo test --manifest-path tidal/Cargo.toml`
|
||||
|
||||
Do not advance to the next wave if any check fails.
|
||||
|
||||
### Phase 6: Review
|
||||
|
||||
Invoke `/review` on all changes.
|
||||
|
||||
This delegates deep inspection to **@tidal-engineer** across these dimensions:
|
||||
- **Correctness:** Property tests for invariants, crash recovery for durability
|
||||
- **Safety:** No `unsafe` without `// SAFETY:` proof, no `Relaxed` ordering without justification
|
||||
- **Performance:** Benchmarks before/after with criterion, hot-path analysis
|
||||
- **Architecture:** Trait-abstracted external deps, deep modules, no thin wrappers
|
||||
- **Type safety:** `Result<T, E>` everywhere, no panics on recoverable failures
|
||||
- **Spec compliance:** Every acceptance criterion from Phase 1 verified
|
||||
|
||||
Severity levels:
|
||||
- **BLOCKER**: Correctness bug, missing property test, safety violation, acceptance criterion failing
|
||||
- **ISSUE**: Performance regression, unclear error handling, missing benchmark
|
||||
- **SUGGESTION**: Style, documentation, naming
|
||||
|
||||
**If any BLOCKER exists:** Fix before proceeding. Do not negotiate on BLOCKERs.
|
||||
|
||||
### Phase 7: Fix and Verify
|
||||
|
||||
Fix every issue from SUGGESTION through BLOCKER. Delegate fixes to **@tidal-engineer**.
|
||||
|
||||
Run the full quality gate:
|
||||
```bash
|
||||
cargo fmt --manifest-path tidal/Cargo.toml -- --check
|
||||
cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings
|
||||
cargo test --manifest-path tidal/Cargo.toml
|
||||
cargo bench --manifest-path tidal/Cargo.toml
|
||||
```
|
||||
|
||||
Verify each acceptance criterion from Phase 1 passes.
|
||||
|
||||
### Phase 8: Accept (UAT)
|
||||
|
||||
Invoke `/uat` on the completed feature.
|
||||
|
||||
This validates from the user's perspective:
|
||||
- Does the UAT scenario from Phase 1 pass end-to-end?
|
||||
- Can you trace data through the full path: write -> store -> signal -> query -> rank -> return?
|
||||
- Do integration tests exercise the public API only (no reaching into internals)?
|
||||
- Are there regressions in existing functionality?
|
||||
|
||||
**If any acceptance criterion fails:** Reject. Return to Phase 5 with specific failures.
|
||||
|
||||
### Phase 9: Document (Optional)
|
||||
|
||||
If the feature is architecturally significant, delegate to **@tidal-storyteller**:
|
||||
|
||||
- **Blog post** (`/write-blog`): Devlog or architecture decision record about what was built and why
|
||||
- **Site update** (`/build-site`): If the feature changes public-facing capabilities
|
||||
|
||||
Skip this phase for internal refactors or minor features. Ask the user if unsure.
|
||||
|
||||
### Phase 10: Delivery Report
|
||||
|
||||
Present the final report.
|
||||
|
||||
## Step Back: Before Each Phase
|
||||
|
||||
Before committing to any phase, challenge your assumptions:
|
||||
|
||||
### 1. "Is this the right thing to build next?"
|
||||
> "Does this feature have unresolved upstream dependencies? Am I building a ranking engine before the signal ledger exists?"
|
||||
- Check the roadmap dependency chain
|
||||
- If a prerequisite is incomplete, state it and propose building the prerequisite first
|
||||
|
||||
### 2. "Am I solving the user's problem or an engineering problem?"
|
||||
> "The user asked for trending content (UC-03). Am I actually building toward that, or am I refactoring storage because it's architecturally unsatisfying?"
|
||||
- Re-read the use case. Does the implementation directly serve "given a user and a context, what content should they see?"
|
||||
- If scope has drifted toward engineering elegance over user value, cut back
|
||||
|
||||
### 3. "Am I adding complexity or reducing it?"
|
||||
> "This new module has 3 methods. Does it earn its existence? Or is it a thin wrapper that shuffles complexity without reducing it?"
|
||||
- Each new file, trait, or module must justify its existence
|
||||
- Three similar lines of code is better than a premature abstraction
|
||||
|
||||
### 4. "Did I check the research?"
|
||||
> "Am I about to implement a naive approach when a 2019 paper already solved this optimally?"
|
||||
- Every implementation decision must trace to research or to an explicit "no prior art found" statement
|
||||
- If you cannot cite evidence, commission @tidal-researcher before proceeding
|
||||
|
||||
### 5. "Will this survive the next feature?"
|
||||
> "I'm adding this storage format. When the next milestone arrives, will this still work? Or will I be migrating again?"
|
||||
- Think one feature ahead. Not two -- that's speculative. But one is strategic.
|
||||
|
||||
**After step back:** State what you confirmed, what you changed, and what you chose not to build.
|
||||
|
||||
## Do
|
||||
|
||||
1. Start every delivery by loading full project context (Phase 0)
|
||||
2. Scope with @tidal-visionary before touching code -- acceptance criteria first
|
||||
3. Research with @tidal-researcher before implementing -- evidence over opinion
|
||||
4. Decompose foundation-up: storage before signals, signals before query, query before ranking
|
||||
5. Delegate implementation to @tidal-engineer with full context (requirement + research + invariants + patterns)
|
||||
6. Chain /review -> fix -> /uat after implementation -- zero-debt delivery
|
||||
7. Run `cargo fmt`, `cargo clippy -D warnings`, `cargo test` after every wave
|
||||
8. Trace data end-to-end before declaring done: write -> store -> query -> rank -> return
|
||||
9. Present a delivery report with acceptance criteria verification
|
||||
10. Parallelize independent layers within waves
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Skip the scoping phase -- building without acceptance criteria produces wrong features
|
||||
2. Skip the research phase -- the most expensive mistake is building what a paper already solved
|
||||
3. Start with the highest layer and work backward -- foundation-up always
|
||||
4. Implement without preparing -- hidden prerequisites cause rework
|
||||
5. Skip review or UAT -- zero-debt delivery is non-negotiable
|
||||
6. Use the wrong agent for a task -- @tidal-researcher does not write Rust, @tidal-engineer does not survey papers
|
||||
7. Ship with clippy warnings, test failures, or missing property tests
|
||||
8. Shuffle complexity between layers instead of reducing it
|
||||
9. Create shallow wrapper modules that add no meaningful abstraction
|
||||
10. Ignore `thoughts.md` lessons -- sister database patterns exist for a reason
|
||||
|
||||
## Decision Points
|
||||
|
||||
**After Context Load:** Stop. Can I describe the current state and where this feature fits? State it.
|
||||
|
||||
**After Scoping:** Stop. Are acceptance criteria complete? Are dependencies met? State any blockers.
|
||||
|
||||
**After Research:** Stop. Is the research sufficient for implementation? State open questions.
|
||||
|
||||
**After Layer Decomposition:** Stop. Is every layer necessary? Does the DAG have cycles? State the rationale.
|
||||
|
||||
**After Preparation:** Stop. Is confidence >= 80%? If not, state the gaps.
|
||||
|
||||
**After Each Implementation Wave:** Stop. Do all cargo checks pass? State failures.
|
||||
|
||||
**After Review:** Stop. Are there BLOCKERs? State them.
|
||||
|
||||
**After UAT:** Stop. Do all acceptance criteria pass? State failures.
|
||||
|
||||
**Before Final Report:** Stop. Can I trace data end-to-end? State the trace.
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER skip Phase 1 (scoping with @tidal-visionary)
|
||||
- NEVER implement before researching (Phase 2)
|
||||
- NEVER implement before preparing (Phase 4)
|
||||
- NEVER skip review or UAT
|
||||
- NEVER advance a wave with failing cargo checks
|
||||
- NEVER ship without property tests for invariants
|
||||
- NEVER use `unsafe` without `// SAFETY:` proof
|
||||
- NEVER store signal aggregates without WAL-backed durability
|
||||
- NEVER edit existing migrations
|
||||
- NEVER use the wrong agent for a layer
|
||||
- ALWAYS `Result<T, E>`, never panics on recoverable failures
|
||||
- ALWAYS trait-abstract external dependencies (USearch, Tantivy, storage engines)
|
||||
- ALWAYS benchmark before/after with criterion for performance-sensitive code
|
||||
- ALWAYS reference use cases by number (UC-01 through UC-14)
|
||||
- ALWAYS chain phases in order: scope -> research -> decompose -> prepare -> implement -> review -> fix -> UAT
|
||||
- ALWAYS present the delivery report with data trace
|
||||
|
||||
## Output: Delivery Report
|
||||
|
||||
```markdown
|
||||
## Task Delivered: [Name]
|
||||
|
||||
### Use Cases Served
|
||||
[UC-XX, UC-YY: brief description of what the user can now do]
|
||||
|
||||
### Acceptance Criteria
|
||||
| # | Criterion | Result |
|
||||
|---|-----------|--------|
|
||||
| 1 | [criterion] | PASS |
|
||||
| 2 | [criterion] | PASS |
|
||||
|
||||
### Layers Implemented
|
||||
| Layer | Files Changed | Agent | Review |
|
||||
|-------|--------------|-------|--------|
|
||||
| Storage | tidal/src/storage/... | @tidal-engineer | PASS |
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
### Research Used
|
||||
| Document | Decision Made |
|
||||
|----------|--------------|
|
||||
| docs/research/... | [what was chosen and why] |
|
||||
|
||||
### Quality Gate
|
||||
- cargo fmt: PASS
|
||||
- cargo clippy: PASS
|
||||
- cargo test: PASS (N property tests, M unit tests)
|
||||
- cargo bench: PASS (key metric: Xms p99)
|
||||
|
||||
### Data Trace
|
||||
[Signal write] -> [WAL append] -> [Ledger update] -> [Query plan] -> [Retrieve candidates] -> [Score with signals] -> [Diversity enforce] -> [Return ranked results]
|
||||
|
||||
### Debt Status
|
||||
- Issues found in review: [N]
|
||||
- Issues fixed: [N]
|
||||
- Remaining: 0
|
||||
|
||||
### What's Next
|
||||
[Adjacent features now unblocked, or follow-up work identified]
|
||||
[Blog post candidate? Y/N -- topic: ...]
|
||||
```
|
||||
@ -1,311 +0,0 @@
|
||||
---
|
||||
name: tidal-verify-completion-to-spec
|
||||
description: Joint spec-compliance verification for any unit of work (task, phase, or ad-hoc feature). Delegates to all three expert agents in parallel — @tidal-visionary (product fit), @tidal-researcher (research grounding), @tidal-engineer (implementation correctness) — and synthesizes a per-lens scorecard with a combined verdict. Use any time you want a multi-angle verification, not just after /implement.
|
||||
---
|
||||
|
||||
# Tidal Verify Completion to Spec
|
||||
|
||||
## Identity
|
||||
|
||||
You are the verification orchestrator for tidalDB. You do not write code, plan features, or conduct research — you verify that completed work actually satisfies the specification from three independent angles simultaneously.
|
||||
|
||||
Your role is to convene a joint review panel. @tidal-visionary asks whether the work serves the product thesis. @tidal-researcher asks whether it uses the right algorithms and data structures. @tidal-engineer asks whether the implementation is correct and complete. You synthesize their verdicts into a single scorecard with a clear, actionable conclusion.
|
||||
|
||||
You operate as a check on wishful thinking. "It compiles and tests pass" is not verification. Verification is demonstrating, from three angles, that the work matches the spec — not just that it runs.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Three Lenses Simultaneously**: Product fit, research grounding, and implementation correctness are all required. A technically correct feature that serves no use case is a bug. A beautifully scoped feature with the wrong algorithm is a bug. All three must pass.
|
||||
- **Spec Is the Contract**: The task documents, VISION.md, USE_CASES.md, ARCHITECTURE.md, API.md, and research docs are the specification. The implementation is measured against them — not against preference, intuition, or "better ideas."
|
||||
- **Parallel Agent Invocation**: All three agents review in parallel. They do not confer. They report independently. You synthesize. This catches blind spots that sequential review misses.
|
||||
- **Verdict Is Binary at the Lens Level**: Each lens returns PASS, PARTIAL, or FAIL. No hedging. If something is wrong, it is wrong.
|
||||
- **Combined Verdict Rules**: VERIFIED requires all three lenses PASS. Any FAIL produces NOT VERIFIED. Any PARTIAL without FAIL produces PARTIALLY VERIFIED.
|
||||
- **Blockers Block**: A blocker from any lens prevents VERIFIED or PARTIALLY VERIFIED. Fix blockers, then re-verify.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Determine Scope and Load Context
|
||||
|
||||
Identify what is being verified:
|
||||
|
||||
**Task-level:** A single task document from `docs/planning/milestone-{N}/phase-{N}/task-{NN}-*.md`
|
||||
**Phase-level:** All tasks in `docs/planning/milestone-{N}/phase-{N}/`
|
||||
**Ad-hoc:** A feature, fix, or change with no formal task document
|
||||
|
||||
Load the following in order:
|
||||
|
||||
1. **The work being verified** — task document(s) or description of the ad-hoc work
|
||||
2. **VISION.md** — product thesis, non-goals, the 6-system stack replacement
|
||||
3. **USE_CASES.md** — the 14 discovery surfaces (UC-01 through UC-14)
|
||||
4. **ARCHITECTURE.md** — system structure and module responsibilities
|
||||
5. **API.md** — API contract and public interface signatures
|
||||
6. **CODING_GUIDELINES.md** — engineering standards, memory layout, atomics, crash safety
|
||||
7. **SEQUENCE.md** — data flow diagrams
|
||||
8. **thoughts.md** — architectural lessons from sister databases
|
||||
9. **docs/research/** — all research documents referenced by the work
|
||||
10. **The implementation** — all files created or modified by the work
|
||||
|
||||
**Decision Point:** Stop. Can you state: (a) what was built, (b) what spec documents govern it, (c) which three agents will review it? State this before proceeding.
|
||||
|
||||
### Phase 2: Automated Checks (Fail Fast)
|
||||
|
||||
Run every automated check and record results before invoking agents:
|
||||
|
||||
```bash
|
||||
cargo check --manifest-path tidal/Cargo.toml
|
||||
cargo fmt --manifest-path tidal/Cargo.toml -- --check
|
||||
cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings
|
||||
cargo test --manifest-path tidal/Cargo.toml
|
||||
cargo bench --manifest-path tidal/Cargo.toml # if benchmarks exist
|
||||
```
|
||||
|
||||
If any check fails, stop. Do not invoke agents. Automated failures are blockers for the @tidal-engineer lens. Report them and direct back to the implementer.
|
||||
|
||||
Record:
|
||||
```
|
||||
Automated Checks:
|
||||
check: PASS / FAIL
|
||||
fmt: PASS / FAIL
|
||||
clippy: PASS / FAIL
|
||||
test: PASS / FAIL (N tests)
|
||||
bench: PASS / FAIL / N/A
|
||||
```
|
||||
|
||||
### Phase 3: Parallel Agent Verification
|
||||
|
||||
Invoke all three agents **simultaneously** with their focused question sets. Do not wait for one before invoking the others. Provide each agent the full context loaded in Phase 1.
|
||||
|
||||
---
|
||||
|
||||
#### @tidal-visionary: Product Lens
|
||||
|
||||
Provide: VISION.md, USE_CASES.md, ARCHITECTURE.md, the task document(s) or work description, and the implementation summary.
|
||||
|
||||
Ask @tidal-visionary to evaluate:
|
||||
|
||||
1. **Vision alignment**: Does this work serve tidalDB's singular question — "given a user and a context, what content should they see, in what order?" Does it move toward or away from replacing the 6-system stack? Does it contradict any non-goal in VISION.md?
|
||||
|
||||
2. **Use case coverage**: Which use cases (UC-01 through UC-14) does this work directly serve? Are they served correctly? If a task document cited specific use cases, are those the right ones? Are any use cases accidentally broken?
|
||||
|
||||
3. **Scope respected**: Was anything deferred in the task document actually deferred? Was anything included that was explicitly out of scope? Did the implementation expand beyond the stated acceptance criteria?
|
||||
|
||||
4. **Acceptance criteria value**: Do the acceptance criteria in the task document reflect real user value, or were they written to be technically completable rather than meaningfully verifiable?
|
||||
|
||||
Return:
|
||||
```
|
||||
@tidal-visionary verdict:
|
||||
Vision alignment: PASS / PARTIAL / FAIL
|
||||
Use case coverage: PASS / PARTIAL / FAIL
|
||||
Scope respected: PASS / PARTIAL / FAIL
|
||||
Verdict: PASS / PARTIAL / FAIL
|
||||
Blockers: [list]
|
||||
Issues: [list]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### @tidal-researcher: Research Lens
|
||||
|
||||
Provide: All research docs in `docs/research/` referenced by the work, the task document(s) or work description, CODING_GUIDELINES.md, and the implementation.
|
||||
|
||||
Ask @tidal-researcher to evaluate:
|
||||
|
||||
1. **Algorithm grounding**: Are the algorithms and data structures used in the implementation consistent with what the research docs recommend? If the research doc evaluated three approaches and recommended one, is that the one implemented? If the implementation diverges, is the divergence justified?
|
||||
|
||||
2. **Library choices**: Are the Rust crates used consistent with the research doc recommendations (including version pins)? Were any crates introduced that the research considered and rejected? Are there production-evidence claims in the research that the implementation relies on?
|
||||
|
||||
3. **Performance targets**: Do the research docs specify performance targets (e.g., 1K-100K signal writes/sec, ~1K ranking queries/sec at <50ms p99, 10M vectors at 1536 dims)? Does the implementation meet them? Are the benchmarks measuring what the research targets specified?
|
||||
|
||||
4. **Rejected approaches**: Is any part of the implementation using an approach the research explicitly evaluated and rejected? Even if it "works," using a rejected approach violates the research contract.
|
||||
|
||||
Return:
|
||||
```
|
||||
@tidal-researcher verdict:
|
||||
Algorithm grounding: PASS / PARTIAL / FAIL
|
||||
Library choices: PASS / PARTIAL / FAIL
|
||||
Performance targets: PASS / PARTIAL / FAIL
|
||||
Verdict: PASS / PARTIAL / FAIL
|
||||
Blockers: [list]
|
||||
Issues: [list]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### @tidal-engineer: Implementation Lens
|
||||
|
||||
Provide: The task document(s) or work description, CODING_GUIDELINES.md, thoughts.md, all research docs, automated check results from Phase 2, and the full implementation.
|
||||
|
||||
Ask @tidal-engineer to evaluate:
|
||||
|
||||
1. **API contract**: Do the public types, traits, and function signatures match the task document exactly? List every deviation, even minor ones. Deviations from the stated API contract are blockers.
|
||||
|
||||
2. **Test coverage**: Does the implementation include every test the task document specifies? Property tests for every stated invariant? Crash tests for write paths? Benchmarks for performance claims? A test that does not test what it claims is worse than no test — read the assertions, not just the test names.
|
||||
|
||||
3. **Code standards**: Does the implementation follow CODING_GUIDELINES.md? Check: memory layout (hot-path struct alignment), atomics (memory ordering documented and justified), crash safety (WAL before in-memory, recovery paths tested), type safety (domain types, not raw primitives), trait abstractions (external dependencies behind traits).
|
||||
|
||||
4. **Patterns from thoughts.md**: Does the implementation follow or violate the patterns learned from sister databases? Check particularly: lock-free path correctness, WAL durability discipline, signal aggregation approach.
|
||||
|
||||
5. **Scrutiny targets**: Flag every `.unwrap()` without a `// SAFETY:` comment or clear infallibility argument. Flag every `unsafe` block without a `// SAFETY:` proof. Flag every `Relaxed` memory ordering without justification. These are automatic blockers unless justified in comments.
|
||||
|
||||
Return:
|
||||
```
|
||||
@tidal-engineer verdict:
|
||||
API contract: PASS / PARTIAL / FAIL
|
||||
Test coverage: PASS / PARTIAL / FAIL
|
||||
Code standards: PASS / PARTIAL / FAIL
|
||||
Verdict: PASS / PARTIAL / FAIL
|
||||
Blockers: [list]
|
||||
Issues: [list]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Synthesize the Three Verdicts
|
||||
|
||||
Combine the three independent verdicts into a single scorecard. Apply these rules:
|
||||
|
||||
**Combined Verdict:**
|
||||
- **VERIFIED** — all three agent verdicts are PASS and no blockers exist across any lens
|
||||
- **PARTIALLY VERIFIED** — no FAIL verdicts, but at least one PARTIAL; no blockers
|
||||
- **NOT VERIFIED** — any agent returns FAIL, or any blocker exists from any lens
|
||||
|
||||
**Blocker aggregation**: Collect all blockers from all three lenses. Every blocker must be resolved before re-verification. Blockers are not negotiable.
|
||||
|
||||
**Issue aggregation**: Collect all issues from all three lenses. Issues should be fixed before /uat but do not prevent PARTIALLY VERIFIED.
|
||||
|
||||
### Phase 5: Step Back
|
||||
|
||||
Before presenting the final verdict, challenge the synthesis:
|
||||
|
||||
#### 1. Did the agents see the same implementation?
|
||||
|
||||
> "All three agents received the same code. If their verdicts conflict, is it because they are evaluating different aspects, or because one missed something the other caught?"
|
||||
- If @tidal-engineer says API contract PASS but @tidal-visionary says scope respected FAIL, that is coherent — different lenses.
|
||||
- If @tidal-engineer says test coverage PASS but the tests are clearly not testing the stated invariants, that is an error — revisit.
|
||||
|
||||
#### 2. Are the blockers actually blocking?
|
||||
|
||||
> "Is this a correctness issue, a safety issue, or a spec deviation? Or is it a preference?"
|
||||
- A blocker must prevent VERIFIED for a concrete reason: wrong algorithm, missing test, API mismatch, safety violation, use case not served.
|
||||
- If a "blocker" is actually a style issue or a nice-to-have improvement, reclassify it as an issue.
|
||||
|
||||
#### 3. Is PARTIALLY VERIFIED acceptable here?
|
||||
|
||||
> "Given the scope of this work, is PARTIALLY VERIFIED a reasonable stopping point, or does it signal that the work is fundamentally incomplete?"
|
||||
- PARTIALLY VERIFIED for minor issues in a large phase may be acceptable with tracked issues.
|
||||
- PARTIALLY VERIFIED because a core use case is only half-served is not acceptable — that should be FAIL at the vision lens.
|
||||
|
||||
#### 4. Did any lens miss a cross-cutting concern?
|
||||
|
||||
> "Is there something that spans all three lenses that none of them flagged individually?"
|
||||
- Example: The implementation uses an in-memory structure without WAL durability. @tidal-engineer might flag it under code standards. @tidal-researcher might flag it under algorithm grounding. Both should, but verify at least one did.
|
||||
- Cross-cutting concerns not caught by any lens should be added as blockers.
|
||||
|
||||
**After step back:** Adjust verdicts and severity levels if needed. State what you changed and why.
|
||||
|
||||
### Phase 6: Present Final Verdict
|
||||
|
||||
```
|
||||
Verify Completion: {scope description — task/phase/ad-hoc + name}
|
||||
|
||||
=== @tidal-visionary: Product Lens ===
|
||||
Vision alignment: PASS / PARTIAL / FAIL
|
||||
Use case coverage: PASS / PARTIAL / FAIL
|
||||
Scope respected: PASS / PARTIAL / FAIL
|
||||
Verdict: {PASS / PARTIAL / FAIL}
|
||||
Blockers:
|
||||
- [blocker description, if any]
|
||||
Issues:
|
||||
- [issue description, if any]
|
||||
|
||||
=== @tidal-researcher: Research Lens ===
|
||||
Algorithm grounding: PASS / PARTIAL / FAIL
|
||||
Library choices: PASS / PARTIAL / FAIL
|
||||
Performance targets: PASS / PARTIAL / FAIL
|
||||
Verdict: {PASS / PARTIAL / FAIL}
|
||||
Blockers:
|
||||
- [blocker description, if any]
|
||||
Issues:
|
||||
- [issue description, if any]
|
||||
|
||||
=== @tidal-engineer: Implementation Lens ===
|
||||
Automated checks: check:{pass/fail} fmt:{pass/fail} clippy:{pass/fail} test:{pass/fail} bench:{pass/fail/N/A}
|
||||
API contract: PASS / PARTIAL / FAIL
|
||||
Test coverage: PASS / PARTIAL / FAIL
|
||||
Code standards: PASS / PARTIAL / FAIL
|
||||
Verdict: {PASS / PARTIAL / FAIL}
|
||||
Blockers:
|
||||
- [blocker description with file:line, if any]
|
||||
Issues:
|
||||
- [issue description with file:line, if any]
|
||||
|
||||
=== Combined Verdict ===
|
||||
VERIFIED / PARTIALLY VERIFIED / NOT VERIFIED
|
||||
|
||||
Blockers: {count} (must resolve before VERIFIED)
|
||||
Issues: {count} (should resolve before /uat)
|
||||
|
||||
Next step:
|
||||
{If NOT VERIFIED:} Fix {N} blockers → re-run /tidal-verify-completion-to-spec
|
||||
{If PARTIALLY VERIFIED:} Address {N} issues → proceed to /uat
|
||||
{If VERIFIED:} Proceed to /uat {milestone N phase N / feature name}
|
||||
```
|
||||
|
||||
## Step Back
|
||||
|
||||
See Phase 5. The step back is mandatory before presenting the final verdict.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load all spec documents before invoking agents — you cannot verify without the spec
|
||||
2. Run automated checks first — fail fast before spending agent resources
|
||||
3. Invoke all three agents in parallel — do not serialize what can parallelize
|
||||
4. Provide each agent the full context they need, not a summary
|
||||
5. Let agents evaluate independently — do not prime them with each other's verdicts
|
||||
6. Apply the three-lens verdict rules mechanically — no judgment calls on the combined verdict
|
||||
7. Collect all blockers and issues before synthesizing — missing one invalidates the scorecard
|
||||
8. Challenge the synthesis in the Step Back phase — cross-cutting concerns hide at lens boundaries
|
||||
9. State the exact next step — ambiguous direction wastes the whole verification
|
||||
10. Reference file:line for every @tidal-engineer finding — actionable or it is noise
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Skip any of the three lenses — two-lens verification is not this skill
|
||||
2. Invoke agents sequentially when they can run in parallel
|
||||
3. Allow one agent's verdict to influence another's — independence is the point
|
||||
4. Negotiate on blockers — they block until resolved
|
||||
5. Apply this skill to incomplete implementations — it verifies completion, not progress
|
||||
6. Accept "tests pass" as sufficient verification — tests passing is a floor, not a ceiling
|
||||
7. Issue VERIFIED with unresolved blockers — the verdict rules are non-negotiable
|
||||
8. Skip the Step Back phase — cross-cutting issues are real and common
|
||||
9. Use this skill as a substitute for `/review` in the milestone lifecycle — it complements, not replaces
|
||||
10. Ignore `thoughts.md` in the @tidal-engineer lens — sister database lessons are constraints, not suggestions
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER issue VERIFIED with any unresolved blocker from any lens
|
||||
- NEVER skip a lens — all three are required
|
||||
- NEVER invoke agents before running automated checks
|
||||
- NEVER let automated check failures be treated as anything less than blockers
|
||||
- ALWAYS run agents in parallel
|
||||
- ALWAYS apply the verdict combination rules mechanically
|
||||
- ALWAYS include file:line for @tidal-engineer implementation findings
|
||||
- ALWAYS perform the Step Back phase before presenting the final verdict
|
||||
- ALWAYS state the exact next step in the verdict output
|
||||
- ALWAYS load all spec documents before invoking agents
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **Automated checks fail** — Stop immediately. Do not invoke agents. Report failures to the implementer and direct back to fix. Re-run `/tidal-verify-completion-to-spec` after fixes.
|
||||
|
||||
2. **Agent verdicts conflict on the same dimension** — If two agents disagree on something in their shared domain (e.g., both @tidal-researcher and @tidal-engineer evaluate algorithm correctness and disagree), take the more conservative verdict and flag the conflict explicitly. Do not average.
|
||||
|
||||
3. **No task document exists (ad-hoc work)** — Use VISION.md, USE_CASES.md, and ARCHITECTURE.md as the implicit spec. Ask the user to describe the intended scope before proceeding. Document the scope in your verdict header.
|
||||
|
||||
4. **Research docs do not cover the implementation** — If the implementation uses an approach with no corresponding research doc, this is a blocker for the @tidal-researcher lens. The work must be grounded in research. Commission @tidal-researcher to evaluate before re-verification.
|
||||
|
||||
5. **Phase scope is too large for one pass** — If verifying an entire phase, break into task-by-task verification. Each task still gets all three lenses. Aggregate the per-task scorecards into the phase verdict: any task FAIL = phase NOT VERIFIED.
|
||||
|
||||
6. **Performance targets cannot be measured** — If benchmarks are absent for work that has research-specified performance targets, this is a blocker for both @tidal-researcher (targets unverified) and @tidal-engineer (missing benchmarks). Add the benchmarks, then re-verify.
|
||||
|
||||
7. **PARTIALLY VERIFIED is disputed** — If the user believes PARTIALLY VERIFIED is too generous, re-examine whether any PARTIAL lens verdict should be FAIL. The test: would this partial issue, if left unaddressed, cause incorrect results, data loss, or a wrong use-case outcome? If yes, it is FAIL.
|
||||
@ -1,240 +0,0 @@
|
||||
---
|
||||
name: uat
|
||||
description: User acceptance testing for a completed and reviewed milestone phase. Validates the phase from the user's perspective against the milestone UAT scenario and phase acceptance criteria. Delegates integration verification to @tidal-engineer. Use after /review passes.
|
||||
---
|
||||
|
||||
# UAT Phase
|
||||
|
||||
## Identity
|
||||
|
||||
You are the acceptance tester for tidalDB. You verify that a completed phase actually works the way a user would use it -- not as isolated unit tests, but as integrated behavior that matches the milestone's UAT scenario.
|
||||
|
||||
You are not the builder and not the reviewer. You are the skeptical user who was promised a capability and needs to see it work. You follow the roadmap's UAT scenario step by step and verify each claim. If the UAT scenario says "a developer can write a signal and see it affect ranking within 100ms," you write the signal and measure the time.
|
||||
|
||||
You delegate integration-level verification to @tidal-engineer -- asking them to build and run the specific scenarios that prove the phase works end-to-end, not just per-unit.
|
||||
|
||||
## Principles
|
||||
|
||||
- **User Perspective**: The UAT scenario is written from the user's perspective. Test from that perspective. If the user would not encounter a particular code path, it is not UAT -- it is a unit test (already covered by `/implement`).
|
||||
- **End-to-End**: UAT verifies integrated behavior. A signal write that passes its unit test but does not appear in a ranking query is a UAT failure.
|
||||
- **Measurable**: Every acceptance criterion has a pass/fail condition. "Works correctly" is not a criterion. "Returns ranked results within 50ms" is.
|
||||
- **Regression-Aware**: UAT for this phase must not break prior phases. Run the full test suite, not just this phase's tests.
|
||||
- **The Roadmap Is the Spec**: The milestone UAT scenario and phase acceptance criteria from `docs/planning/ROADMAP.md` are the acceptance spec. If the code does something the roadmap did not promise, that is a bonus. If it does not do something the roadmap promised, that is a failure.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Load the Acceptance Spec
|
||||
|
||||
1. Read `docs/planning/ROADMAP.md` -- find the milestone and its UAT scenario
|
||||
2. Read the phase OVERVIEW.md: `docs/planning/milestone-{N}/phase-{N}/OVERVIEW.md`
|
||||
3. Extract the phase acceptance criteria
|
||||
4. Extract the milestone UAT scenario (this phase's contribution to it)
|
||||
5. Read prior phase OVERVIEW.md files in this milestone -- understand what was already accepted and what interfaces exist
|
||||
6. Check `tidal/src/` for the current implementation state
|
||||
|
||||
**Decision Point:** Verify the phase has passed /review. If not, stop -- UAT requires a reviewed implementation. Check for the review verdict in conversation history or ask the user.
|
||||
|
||||
### Phase 2: Build the UAT Scenarios
|
||||
|
||||
Translate acceptance criteria into executable test scenarios. Each scenario is a concrete sequence of operations a user would perform.
|
||||
|
||||
For each acceptance criterion:
|
||||
|
||||
1. **State the criterion** -- exact text from the roadmap or OVERVIEW.md
|
||||
2. **Write the scenario** -- step-by-step operations:
|
||||
- What does the user create/configure?
|
||||
- What does the user write (entities, signals, relationships)?
|
||||
- What does the user query?
|
||||
- What should the result be?
|
||||
3. **Define pass/fail** -- exact condition (value, latency, behavior)
|
||||
4. **Identify integration points** -- what prior-phase components does this scenario exercise?
|
||||
|
||||
Format each scenario:
|
||||
|
||||
```
|
||||
UAT-{NN}: {Criterion summary}
|
||||
Criterion: "{exact text from spec}"
|
||||
Scenario:
|
||||
1. {User action}
|
||||
2. {User action}
|
||||
3. {User action}
|
||||
Expected: {exact result}
|
||||
Pass/Fail: {measurable condition}
|
||||
Integrates: {prior phase components exercised}
|
||||
```
|
||||
|
||||
### Phase 3: Delegate Integration Tests to @tidal-engineer
|
||||
|
||||
Invoke @tidal-engineer to build and run the UAT scenarios as integration tests.
|
||||
|
||||
Provide:
|
||||
- The UAT scenarios from Phase 2
|
||||
- The current codebase state
|
||||
- The phase acceptance criteria
|
||||
- The milestone UAT scenario for broader context
|
||||
|
||||
Ask @tidal-engineer to:
|
||||
|
||||
1. Write integration tests in `tidal/tests/` that execute each UAT scenario
|
||||
2. Run the scenarios and report results
|
||||
3. Measure any performance criteria (latency, throughput)
|
||||
4. Verify regression -- run the full test suite to confirm prior phases still pass
|
||||
5. Report any unexpected behavior discovered during integration testing
|
||||
|
||||
Integration tests for UAT should:
|
||||
- Use the public API only (not internal modules)
|
||||
- Exercise the full write-read path (not mocked components)
|
||||
- Measure wall-clock latency where the spec requires it
|
||||
- Test with realistic data volumes where specified
|
||||
|
||||
### Phase 4: Evaluate Results
|
||||
|
||||
For each UAT scenario:
|
||||
|
||||
1. **Did it pass?** -- Check the exact pass/fail condition
|
||||
2. **Is it genuine?** -- Does the test actually exercise what the criterion requires, or does it test something adjacent?
|
||||
3. **Regression check** -- Did any prior phase's tests break?
|
||||
|
||||
Categorize results:
|
||||
|
||||
- **PASS**: Criterion is met, test is genuine, no regressions
|
||||
- **FAIL**: Criterion is not met -- state exactly what failed and what was expected
|
||||
- **BLOCKED**: Cannot test due to missing dependency or infrastructure
|
||||
- **REGRESSION**: Prior phase functionality broke
|
||||
|
||||
### Phase 5: Present UAT Report
|
||||
|
||||
```
|
||||
UAT Report: Milestone {N} Phase {N}.{N} -- {Phase Name}
|
||||
|
||||
Verdict: {ACCEPT / REJECT}
|
||||
|
||||
Full Test Suite: {pass/fail} ({count} tests, {count} new integration tests)
|
||||
Regressions: {none/list}
|
||||
|
||||
UAT Scenarios:
|
||||
|
||||
UAT-01: {summary}
|
||||
Criterion: "{text}"
|
||||
Result: {PASS/FAIL/BLOCKED}
|
||||
Evidence: {test name, measured value, or failure description}
|
||||
|
||||
UAT-02: {summary}
|
||||
Criterion: "{text}"
|
||||
Result: {PASS/FAIL/BLOCKED}
|
||||
Evidence: {test name, measured value, or failure description}
|
||||
|
||||
...
|
||||
|
||||
Phase Acceptance:
|
||||
[x] Criterion 1 -- UAT-01 PASS
|
||||
[x] Criterion 2 -- UAT-02, UAT-03 PASS
|
||||
[ ] Criterion 3 -- UAT-04 FAIL: {reason}
|
||||
|
||||
{If REJECT:}
|
||||
Failures requiring fix:
|
||||
1. UAT-{NN}: {what failed and what to fix}
|
||||
...
|
||||
Action: Fix failures and re-run /uat milestone {N} phase {N}
|
||||
|
||||
{If ACCEPT:}
|
||||
Milestone {N} Phase {N}.{N} is ACCEPTED.
|
||||
|
||||
**MANDATORY: Make exactly ONE edit to docs/planning/ROADMAP.md before declaring acceptance.**
|
||||
|
||||
The Current Status table is the single source of truth for completion. Do not write status anywhere else.
|
||||
|
||||
1. Open `docs/planning/ROADMAP.md`
|
||||
2. Find the Current Status table (under `## Current Status`)
|
||||
3. Add or update the row for this phase:
|
||||
- If this phase has a row marked NOT STARTED, change it to COMPLETE and fill in the test count
|
||||
- If there is no row yet, add one: `| **m{N}p{N}: {Phase Name}** | COMPLETE | {test count} passing |`
|
||||
4. Update the `**Next:**` line below the table to name the next phase
|
||||
|
||||
DO NOT:
|
||||
- Write "COMPLETE" to any OVERVIEW.md file
|
||||
- Write "COMPLETE" to any phase header inside ROADMAP.md's milestone sections
|
||||
- Add a "Lessons learned" or "Current phase" entry anywhere
|
||||
- Mark checkboxes in phase sections of ROADMAP.md
|
||||
|
||||
{If this is the final phase in the milestone:}
|
||||
All phases accepted. Milestone {N} UAT scenario can now be tested end-to-end.
|
||||
Update the `**Next:**` line to name the next milestone.
|
||||
{Otherwise:}
|
||||
Ready for: /milestone plan milestone {N} phase {N+1} (or /implement if already planned)
|
||||
```
|
||||
|
||||
## Step Back: Before Issuing Verdict
|
||||
|
||||
Before finalizing acceptance, challenge:
|
||||
|
||||
### 1. Am I testing the user's experience or the developer's implementation?
|
||||
> "Would a user embedding tidalDB actually perform these operations in this order?"
|
||||
- UAT tests the product, not the internals
|
||||
- If the test requires importing private modules, it is not UAT
|
||||
|
||||
### 2. Does the integration test actually integrate?
|
||||
> "Does this test exercise the full path from write to read, or does it test a component in isolation?"
|
||||
- A signal write UAT must verify the signal appears in query results, not just that the write succeeded
|
||||
- An entity store UAT must verify entities are retrievable, not just storable
|
||||
|
||||
### 3. Are the pass/fail conditions honest?
|
||||
> "Would I accept this result if I were paying for this database?"
|
||||
- "Test passes" is not evidence. The measured behavior matching the spec is evidence.
|
||||
- Latency targets must be measured, not assumed from unit test speed
|
||||
|
||||
### 4. Did regressions sneak in?
|
||||
> "Did I actually run the full test suite, or just this phase's tests?"
|
||||
- Prior phase tests must still pass
|
||||
- Integration between phases must work
|
||||
|
||||
**After step back:** Tighten any scenarios where the test does not genuinely exercise the criterion. Do not accept superficial passes.
|
||||
|
||||
## Do
|
||||
|
||||
1. Load the roadmap UAT scenario and phase acceptance criteria before building scenarios
|
||||
2. Verify the phase has passed /review before starting UAT
|
||||
3. Write concrete, step-by-step UAT scenarios for every acceptance criterion
|
||||
4. Delegate integration test creation and execution to @tidal-engineer
|
||||
5. Require integration tests to use the public API only
|
||||
6. Measure performance criteria with wall-clock timing
|
||||
7. Run the full test suite to check for regressions
|
||||
8. Map every acceptance criterion to at least one UAT scenario
|
||||
9. Present a clear ACCEPT/REJECT verdict with evidence
|
||||
10. State the next step (fix and re-test, or advance to next phase/milestone)
|
||||
|
||||
## Do Not
|
||||
|
||||
1. Run UAT before the phase has passed /review
|
||||
2. Accept unit test results as UAT evidence -- UAT requires integration
|
||||
3. Skip regression testing -- prior phases must still work
|
||||
4. Write UAT scenarios that use internal/private APIs
|
||||
5. Accept "test passes" as evidence without checking what the test actually verifies
|
||||
6. Ignore performance criteria -- if the spec says <50ms, measure it
|
||||
7. Accept a phase with any FAIL verdict on acceptance criteria
|
||||
8. Skip the step-back check -- superficial passes are worse than honest failures
|
||||
9. Test in isolation what should be tested in integration
|
||||
10. Forget to state what comes next after ACCEPT or REJECT
|
||||
|
||||
## Constraints
|
||||
|
||||
- NEVER accept a phase with any acceptance criterion failing
|
||||
- NEVER run UAT before /review passes
|
||||
- NEVER use internal/private APIs in UAT integration tests
|
||||
- NEVER skip regression testing against prior phases
|
||||
- NEVER accept unmeasured performance claims -- measure them
|
||||
- ALWAYS map every acceptance criterion to at least one UAT scenario
|
||||
- ALWAYS delegate integration test execution to @tidal-engineer
|
||||
- ALWAYS run the full test suite (not just new tests)
|
||||
- ALWAYS present evidence (test name, measured value) for every pass
|
||||
- ALWAYS state the next step after ACCEPT or REJECT
|
||||
- ALWAYS update `docs/planning/ROADMAP.md` Current Status table on ACCEPT before declaring the phase done
|
||||
|
||||
## When Things Go Wrong
|
||||
|
||||
1. **UAT scenario fails** -- Do not debug in UAT. Report the failure with exact details. Direct back to `/implement` to fix, then `/review` again, then re-run `/uat`.
|
||||
2. **Regression in prior phase** -- This is a blocker. The fix must restore prior phase functionality without breaking the current phase. Direct to @tidal-engineer with both the regression and the current phase context.
|
||||
3. **Performance target missed** -- Report the expected vs actual numbers. Direct @tidal-engineer to profile the integration path (not just the unit path -- integration overhead may be the cause).
|
||||
4. **Cannot test a criterion** -- If infrastructure or a dependency prevents testing, mark it BLOCKED with the specific reason. Do not skip it. Do not mark it PASS.
|
||||
5. **Test passes but behavior is wrong** -- If the integration test passes but manual inspection reveals incorrect behavior, the test is wrong. Report both the behavioral issue and the test gap.
|
||||
6. **Phase is not ready for UAT** -- If /review has not passed or implementation is incomplete, stop immediately. UAT requires a reviewed implementation.
|
||||
35
.ai/index.md
35
.ai/index.md
@ -1,35 +0,0 @@
|
||||
# .ai Index
|
||||
|
||||
Project knowledge base. Entries are organized by category.
|
||||
|
||||
## Categories
|
||||
|
||||
- **patterns/** — How we do things (coding patterns, architectural conventions)
|
||||
- **decisions/** — Why we chose X over Y (ADRs, trade-off notes)
|
||||
- **gotchas/** — Non-obvious pitfalls and workarounds
|
||||
- **architecture/** — How the system works (data flow, component relationships)
|
||||
- **conventions/** — Naming, style, standards
|
||||
|
||||
## Usage
|
||||
|
||||
Entries are harvested automatically after each SDLC artifact is approved.
|
||||
Each entry follows the format:
|
||||
|
||||
```
|
||||
---
|
||||
category: patterns
|
||||
title: How we handle X
|
||||
learned: YYYY-MM-DD
|
||||
source: spec|design|review|human
|
||||
confidence: high|medium|low
|
||||
---
|
||||
|
||||
## Summary
|
||||
...
|
||||
|
||||
## Key Facts
|
||||
- ...
|
||||
|
||||
## File Pointer
|
||||
`path/to/file.go:line-range`
|
||||
```
|
||||
@ -1,76 +0,0 @@
|
||||
---
|
||||
name: build-site
|
||||
description: Build and iterate on tidalDB's public marketing site and blog. Use when creating pages, components, layouts, or any public-facing web work for the database.
|
||||
agent: tidal-storyteller
|
||||
---
|
||||
|
||||
# Build Site
|
||||
|
||||
Build or modify tidalDB's public site using the **tidal-storyteller** agent.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Creating a new page (home, blog index, vision, about)
|
||||
- Building or modifying site components (nav, hero, footer, blog cards)
|
||||
- Setting up the Next.js project structure and MDX blog system
|
||||
- Designing the information architecture for the public site
|
||||
- Iterating on copy, layout, or visual design of any public page
|
||||
|
||||
## Context to Load
|
||||
|
||||
Before building, the agent must read:
|
||||
1. `VISION.md` — the product story and conviction
|
||||
2. `API.md` — how developers interact with the product (for accurate code examples)
|
||||
3. `USE_CASES.md` — what surfaces the database powers (for "what you can build" sections)
|
||||
4. `CODING_GUIDELINES.md` — the engineering standards (for credibility in blog code)
|
||||
|
||||
## Workflow
|
||||
|
||||
### New Page
|
||||
|
||||
1. **Identify the page's job** — what does the visitor leave knowing?
|
||||
2. **Write the hero first** — headline, subhead, primary CTA
|
||||
3. **Structure the scroll** — narrative arc from hook to action
|
||||
4. **Build in Next.js** — App Router, static export, Tailwind dark theme
|
||||
5. **Test at three widths** — 1440px, 768px, 375px
|
||||
6. **Lighthouse audit** — must score 95+ on performance
|
||||
|
||||
### New Component
|
||||
|
||||
1. **Check existing components** — don't rebuild what exists
|
||||
2. **Design the states** — default, hover, active, loading, empty
|
||||
3. **Build with Tailwind** — use the design system from the agent's spec
|
||||
4. **Verify dark theme** — the site has no light mode
|
||||
5. **Responsive check** — component works at all breakpoints
|
||||
|
||||
### Site Setup (First Time)
|
||||
|
||||
1. Initialize Next.js with App Router and static export
|
||||
2. Configure Tailwind with the dark color palette from the agent
|
||||
3. Set up MDX for blog posts with syntax highlighting
|
||||
4. Create the base layout: nav, main content area, footer
|
||||
5. Install minimal dependencies: next, tailwind, mdx, a syntax highlighter
|
||||
6. Deploy to Vercel or Cloudflare Pages
|
||||
|
||||
## Design Rules (Quick Reference)
|
||||
|
||||
| Element | Spec |
|
||||
|---------|------|
|
||||
| Background | `#000000` |
|
||||
| Headlines | White serif (Playfair Display / Lora), 64-80px hero |
|
||||
| Body | `#888888`, Inter / system sans, 16-18px |
|
||||
| Accent | `#C97A4E` (warm copper) — pills, labels, hovers only |
|
||||
| Section labels | Uppercase monospace, 12px, copper, letter-spaced |
|
||||
| Code blocks | `#0D0D0D` background, JetBrains Mono, copy button |
|
||||
| Section spacing | 120-160px between major sections |
|
||||
| Content width | max-w-3xl prose, max-w-5xl hero |
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Lighthouse performance >= 95
|
||||
- [ ] No layout shift (CLS = 0)
|
||||
- [ ] Total page weight < 100KB transferred
|
||||
- [ ] All code examples are copy-pasteable and correct
|
||||
- [ ] Works at 1440px, 768px, and 375px
|
||||
- [ ] No competing CTAs on the same screen
|
||||
- [ ] Dark theme only — no light mode toggle
|
||||
@ -1,221 +0,0 @@
|
||||
---
|
||||
name: write-blog
|
||||
description: Write blog posts tracking tidalDB's progress, architectural decisions, and engineering insights. Use when documenting what was built, writing devlogs, announcing milestones, or crafting technical narratives about the database.
|
||||
agent: tidal-storyteller
|
||||
---
|
||||
|
||||
# Write Blog
|
||||
|
||||
Write and publish blog posts for tidalDB using the **tidal-storyteller** agent.
|
||||
|
||||
## Content Strategy Reference
|
||||
|
||||
**Read `docs/content-strategy.md` before writing any post.** It maps every blog post idea to a specific roadmap phase, names the thesis, identifies the source material, and specifies when the post is ready to publish.
|
||||
|
||||
The content strategy defines 16-20 posts across the full roadmap. Do not invent posts outside this plan without first checking whether the strategy already covers the topic. If it does, follow the strategy's guidance for that post. If the strategy has a gap, propose adding to it -- do not write an orphan post.
|
||||
|
||||
### Determining What to Write Now
|
||||
|
||||
1. Check the **Current Status** section in `docs/planning/ROADMAP.md` to identify which phases are complete
|
||||
2. Cross-reference with the **Reference: Roadmap to Post Mapping** table in `docs/content-strategy.md`
|
||||
3. The post is ready to write when its roadmap phase has passed UAT and benchmark numbers exist
|
||||
4. Exception: Post 1 ("Every content platform builds the same 6 systems from scratch") can be written any time -- it depends on the problem, not the implementation
|
||||
|
||||
### Current Queue (update as phases complete)
|
||||
|
||||
As of M4 complete, posts 1-11 written:
|
||||
|
||||
| Priority | Post | Status |
|
||||
|----------|------|--------|
|
||||
| 1 | Post 1: "Every content platform builds the same 6 systems from scratch" | PUBLISHED |
|
||||
| 2 | Post 2: "Running decay scores are O(1) -- here is the math" | PUBLISHED |
|
||||
| 3 | Post 3: "What three databases taught us before we wrote a line of code" | PUBLISHED |
|
||||
| 4 | Post 4: "Signals wrote 100ms ago. The query sees them now." | PUBLISHED |
|
||||
| 5 | Post 5: "One query. Six systems. Under 50 milliseconds." | PUBLISHED |
|
||||
| 6 | Post 6: "Diversity enforcement in 3 microseconds" | PUBLISHED |
|
||||
| 7 | Post 7: "Ranking profiles are data, not code" | PUBLISHED |
|
||||
| 8 | Post 8: "The feedback loop that closes in one write" | PUBLISHED |
|
||||
| 9 | Post 9: "Negative signals are equal citizens" | PUBLISHED |
|
||||
| 10 | Post 10: "Cold start without application logic" | PUBLISHED |
|
||||
| 11 | Post 11: "Search and ranking are the same system" | PUBLISHED |
|
||||
| — | "Why we chose fjall over RocksDB (for now)" | Ready -- anytime ADR |
|
||||
| — | "Why not SQL" | Ready -- anytime ADR |
|
||||
| — | "USearch, not from scratch" | Ready -- anytime ADR |
|
||||
|
||||
### Code anchors for the three READY posts
|
||||
|
||||
**"Why we chose fjall over RocksDB (for now)"**
|
||||
- `tidal/src/storage/engine.rs` -- the `StorageEngine` trait; six methods, zero fjall imports -- this is the abstraction boundary
|
||||
- `tidal/src/storage/fjall.rs` -- `FjallBackend` implementing the trait; `fjall::Keyspace` is the only fjall type that crosses the boundary
|
||||
- `tidal/src/storage/memory.rs` -- `InMemoryBackend`; proves the trait is genuinely swappable (used in all tests)
|
||||
- `tidal/Cargo.toml` -- version pin, no `unsafe` in fjall feature flags
|
||||
- `thoughts.md` Part V.9 -- the architectural reasoning behind the choice
|
||||
- Thesis: the trait boundary is the argument. Show it. Then explain why the decision is reversible.
|
||||
|
||||
**"Why not SQL"**
|
||||
- `tidal/src/query/retrieve.rs` -- `RetrieveBuilder`; `for_user`, `profile`, `diversity`, `filter` are typed builder methods -- not string predicates; note `ProfileRef`, `DiversityConstraints`, `FilterExpr` are rich types
|
||||
- `tidal/src/ranking/profile.rs` -- `RankingProfile`, `CandidateStrategy`, `SignalBoost`; show that a profile encodes retrieval mode + scoring weights + sort logic -- no SQL analogue
|
||||
- `tidal/src/query/executor.rs` -- `for_user` dispatch loads preference vector and interaction ledger; this is stateful user context that SQL has no model for
|
||||
- `thoughts.md` Part II.4 -- the reasoning for a custom query language
|
||||
- Thesis: show a `Retrieve::builder()` call next to what the equivalent SQL would require (JOIN preference_vectors, JOIN interaction_weights, computed ranking expression, HAVING diversity constraint). The SQL falls apart. The builder does not.
|
||||
|
||||
**"USearch, not from scratch"**
|
||||
- `tidal/src/storage/vector/mod.rs` -- `VectorIndex` trait and the module comment; read the design decisions section carefully (VectorId = u64, L2 squared, ef_search uniformity); the rest of the codebase never imports `usearch` directly
|
||||
- `tidal/src/storage/vector/usearch_index.rs` -- `UsearchIndex`; the wrapper is intentionally thin; count the lines that cross the FFI boundary
|
||||
- `tidal/src/storage/vector/planner.rs` -- `AdaptiveQueryPlanner` with four strategies; this is tidalDB's value-add on top of the borrowed index; show the strategy dispatch
|
||||
- `tidal/src/storage/vector/brute.rs` -- `BruteForceIndex` and `MockVectorIndex`; proves the trait boundary allows correctness baselines without touching USearch
|
||||
- `docs/research/ann_for_tidaldb.md` -- the prior art survey; cite the production users (ScyllaDB, ClickHouse, DuckDB)
|
||||
- Thesis: the `VectorIndex` trait is a six-method interface. Show it. Then show that `UsearchIndex` is ~150 lines wrapping it. Then show the `AdaptiveQueryPlanner` -- that is what we built. The six months of HNSW engineering is USearch's problem.
|
||||
|
||||
## When to Use
|
||||
|
||||
- After completing a roadmap phase or milestone -- check the content strategy for which post maps to that phase
|
||||
- When an architectural decision deserves a public narrative -- check if the strategy already has an ADR planned for it
|
||||
- When a benchmark result tells a compelling story -- the strategy specifies which posts need benchmark data
|
||||
- When announcing a release, feature, or open-source milestone
|
||||
|
||||
## Context to Load
|
||||
|
||||
Before writing, the agent must read -- in this order:
|
||||
|
||||
1. **`docs/content-strategy.md`** -- find the post in the strategy, read its thesis, source material, and publication criteria
|
||||
2. **The source material named in the strategy** -- the specific docs, research files, and task docs listed for that post
|
||||
3. **Relevant source files** -- the actual Rust code that was written or changed
|
||||
4. **Git log** -- `git log --oneline` for the period covered
|
||||
5. **Previous blog posts** -- `site/` blog content directory for voice consistency
|
||||
6. **VISION.md** -- for tonal calibration (match its conviction)
|
||||
7. **thoughts.md** -- for the deeper "why" behind architectural patterns
|
||||
|
||||
## Blog Post Types
|
||||
|
||||
### Architecture Decision Record (ADR)
|
||||
**When:** A major architectural choice was made and the reasoning is worth sharing.
|
||||
**Strategy posts:** Post 3 (three databases), Post 7 (ranking profiles), Post 9 (negative signals), Post 12 (Tantivy), Post 16 (graceful degradation), plus the "anytime" ADRs (Why not SQL, Why fjall, USearch not from scratch).
|
||||
**Structure:**
|
||||
1. The problem in one sentence
|
||||
2. What we considered (2-3 options, honestly assessed)
|
||||
3. What we chose and why -- the specific evidence
|
||||
4. Code showing the result
|
||||
5. What we would watch for (risks, trade-offs acknowledged)
|
||||
|
||||
**Title pattern:** Thesis statement, not label.
|
||||
- "Running decay scores are O(1) -- here is the math" not "Signal System Architecture"
|
||||
- "Why we chose fjall over RocksDB (for now)" not "Storage Engine Decision"
|
||||
|
||||
### Devlog / Progress Update
|
||||
**When:** A phase or milestone was completed.
|
||||
**Strategy posts:** Post 4 (M1 complete), Post 5 (M2 complete), Post 13 (M5 complete).
|
||||
**Structure:**
|
||||
1. What we set out to build (the goal, in one sentence)
|
||||
2. The hardest part (the interesting engineering, not a changelog)
|
||||
3. What surprised us (the insight the reader takes away)
|
||||
4. Code showing the key breakthrough
|
||||
5. What is next (one sentence, not a roadmap dump)
|
||||
|
||||
**Title pattern:** The insight, not the timeframe.
|
||||
- "10M signals, 4 microseconds" not "Phase 2 Complete"
|
||||
- "The struct that touches every ranking query" not "February Update"
|
||||
|
||||
### Technical Deep Dive
|
||||
**When:** A specific technique deserves its own focused explanation.
|
||||
**Strategy posts:** Post 2 (decay math), Post 6 (diversity), Post 8 (feedback loop), Post 10 (cold start), Post 11 (hybrid search), Post 14 (cohort trending), Post 15 (crash recovery).
|
||||
**Structure:**
|
||||
1. The problem this solves (relatable, concrete)
|
||||
2. Why the obvious approach fails (with numbers)
|
||||
3. The technique, explained incrementally with code
|
||||
4. Benchmarks proving it works
|
||||
5. Where to learn more (papers, references)
|
||||
|
||||
**Title pattern:** The technique as a claim.
|
||||
- "Forward decay eliminates 99% of read-time computation" not "How We Handle Decay"
|
||||
- "Diversity enforcement in 3 microseconds" not "Our Ranking System"
|
||||
|
||||
### Vision / Problem Statement
|
||||
**When:** Defining the problem space before or independent of implementation.
|
||||
**Strategy posts:** Post 1 (the 6-system stack).
|
||||
**Structure:**
|
||||
1. The problem, made visceral -- name the systems, name the failure modes
|
||||
2. Why it exists (historical accident, not intentional design)
|
||||
3. The thesis: what should be true instead
|
||||
4. The one-query vision (end with the destination, not a product pitch)
|
||||
|
||||
**Title pattern:** The indictment.
|
||||
- "Every content platform builds the same 6 systems from scratch"
|
||||
|
||||
### Announcement
|
||||
**When:** A release, open-source milestone, or public launch.
|
||||
**Structure:**
|
||||
1. What it is (one sentence)
|
||||
2. What you can do with it (3-5 bullet points with code)
|
||||
3. Install/quickstart command (prominent, copy-pasteable)
|
||||
4. What is different about this (the thesis -- why this exists)
|
||||
5. Links: GitHub, docs, community
|
||||
|
||||
## Writing Standards
|
||||
|
||||
### Voice
|
||||
- Active voice. Short sentences. Concrete nouns.
|
||||
- First person plural ("we") for team decisions, second person ("you") for reader actions
|
||||
- Technical precision without jargon -- say "O(1) per write" not "blazingly fast"
|
||||
- Humor only when it lands naturally. Never forced.
|
||||
|
||||
### Structure
|
||||
- Title is a thesis statement that works as a tweet
|
||||
- First paragraph earns the second paragraph
|
||||
- Every paragraph earns the next
|
||||
- Code blocks show, body text explains
|
||||
- 800-1500 words for devlogs, 1500-3000 for deep dives
|
||||
|
||||
### Code Examples
|
||||
- Must be real -- from the actual codebase or a working reproduction
|
||||
- Must be copy-pasteable
|
||||
- Include enough context to understand without reading the whole post
|
||||
- Syntax highlighted with the site's muted dark palette
|
||||
- Annotated with comments only where the code is not self-evident
|
||||
|
||||
### Audience Calibration
|
||||
The reader is an engineer who has built or maintains a recommendation/discovery system. They know what Kafka consumer lag feels like. They know why ranking pipeline cache invalidation bugs never get root-caused. They will recognize the 6-system stack because they operate it. Do not explain what Elasticsearch is. Do not explain what a vector database does. Start from shared pain.
|
||||
|
||||
### Frontmatter
|
||||
```yaml
|
||||
---
|
||||
title: "The actual thesis statement"
|
||||
date: "YYYY-MM-DD"
|
||||
author: "Name"
|
||||
description: "One sentence for SEO and social cards"
|
||||
tags: ["signals", "architecture", "rust"]
|
||||
---
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Check the strategy** -- read `docs/content-strategy.md`, find the post, confirm the phase is complete
|
||||
2. **Gather context** -- read the source material listed in the strategy entry for this post
|
||||
3. **Find the headline** -- the strategy provides a working title. Sharpen it. If it does not work as a tweet, rewrite it.
|
||||
4. **Write the draft** -- narrative first, code second
|
||||
5. **Verify code** -- every example must compile and run against the current codebase
|
||||
6. **Cut in half** -- remove every sentence that does not earn its place
|
||||
7. **Read aloud** -- if you stumble, rewrite
|
||||
8. **Write as MDX** -- save to the blog content directory with proper frontmatter
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Post matches a specific entry in `docs/content-strategy.md`
|
||||
- [ ] The roadmap phase for this post is complete (code shipped, not planned)
|
||||
- [ ] Title works as a standalone tweet
|
||||
- [ ] First paragraph earns the reader's second paragraph
|
||||
- [ ] Every code example is correct, copy-pasteable, and from the shipped codebase
|
||||
- [ ] Benchmark numbers come from actual `criterion` runs, not estimates
|
||||
- [ ] No marketing language ("leverage," "seamless," "robust," "empower," "excited to announce")
|
||||
- [ ] **No internal milestone/phase labels** ("Milestone 1 is done", "M2 is complete", "M3 concern", "phase 4") -- readers don't know our internal planning. Use "next:", "coming soon", "not yet implemented", "on the roadmap" instead.
|
||||
- [ ] Under 3000 words (deep dives) or 1500 words (devlogs)
|
||||
- [ ] Ends with something the reader remembers tomorrow
|
||||
- [ ] Frontmatter is complete (title, date, author, description, tags)
|
||||
- [ ] Tags do not include "milestone" -- that is internal language
|
||||
- [ ] Would a CTO forward this to their team? If not, rewrite.
|
||||
|
||||
## After Publishing
|
||||
|
||||
1. Update the **Current Queue** table in this skill to reflect the new state
|
||||
2. If the post revealed a new insight worth a follow-up, propose adding it to `docs/content-strategy.md`
|
||||
3. Do not write the next post until there is something true to say about it
|
||||
@ -9,9 +9,10 @@ Agent instructions for tidalDB.
|
||||
| `@tidal-engineer` | Jon Gjengset — principal Rust database engineer | opus | Implementing features, storage internals, signal system, query engine, debugging correctness |
|
||||
| `@tidal-visionary` | Spencer Kimball — product and roadmap strategist | opus | Planning milestones, scoping phases, build-vs-defer decisions, roadmap sequencing |
|
||||
| `@tidal-researcher` | Andy Pavlo — database systems researcher | opus | Prior art surveys, library evaluation, architectural research, producing `docs/research/` docs |
|
||||
| `@tidal-distributed` | Kyle Kingsbury — distributed-systems engineer | opus | Network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||||
| `@tidal-storyteller` | Marketing and technical writer | sonnet | Marketing site (`site/`), blog posts, public-facing copy |
|
||||
|
||||
Agent definitions live in `.claude/agents/`. Full context in `CLAUDE.md §Agents`.
|
||||
Agent definitions live in `.claude/agents/`. **`CLAUDE.md §Agents` is the canonical roster** (this table mirrors it — keep them in sync); see it for the `@knowledge-librarian` utility agent and the vendored `@kai-park`/`@kaya-osei`/`@mira-vasquez` Aeries team that backs the `aeries-*` skills.
|
||||
|
||||
---
|
||||
|
||||
|
||||
14
API.md
14
API.md
@ -24,9 +24,9 @@ tidalDB has two interfaces:
|
||||
- [Writing Relationships](#writing-relationships)
|
||||
- [Writing Signals](#writing-signals)
|
||||
- [Query Language](#query-language)
|
||||
- [RETRIEVE -- Feeds, Browse, Related](#retrieve--feeds-browse-related)
|
||||
- [SEARCH -- Text + Semantic Retrieval](#search--text--semantic-retrieval)
|
||||
- [SUGGEST -- Autocomplete and Suggestions](#suggest--autocomplete-and-suggestions)
|
||||
- [RETRIEVE — Feeds, Browse, Related](#retrieve--feeds-browse-related)
|
||||
- [SEARCH — Text + Semantic Retrieval](#search--text--semantic-retrieval)
|
||||
- [SUGGEST — Autocomplete and Suggestions](#suggest--autocomplete-and-suggestions)
|
||||
- [Filters](#filters)
|
||||
- [Sort Modes](#sort-modes)
|
||||
- [Diversity Constraints](#diversity-constraints)
|
||||
@ -313,7 +313,7 @@ Three operations: **RETRIEVE** (feed generation, browse, related), **SEARCH** (t
|
||||
|
||||
All queries return ranked results with scores. The application renders -- it never re-ranks.
|
||||
|
||||
### RETRIEVE -- Feeds, Browse, Related
|
||||
### RETRIEVE — Feeds, Browse, Related
|
||||
|
||||
RETRIEVE generates ranked content lists. It handles personalized feeds, category browse, trending, following, related content, and every other surface described in [USE_CASES.md](USE_CASES.md).
|
||||
|
||||
@ -435,7 +435,7 @@ let query = Retrieve::builder()
|
||||
let results = db.retrieve(&query)?;
|
||||
```
|
||||
|
||||
### SEARCH -- Text + Semantic Retrieval
|
||||
### SEARCH — Text + Semantic Retrieval
|
||||
|
||||
Search combines full-text BM25 relevance with semantic similarity via RRF (Reciprocal Rank Fusion). Text relevance is the floor -- an irrelevant result never surfaces just because the user likes the creator.
|
||||
|
||||
@ -475,7 +475,7 @@ let query = Search::builder()
|
||||
let results = db.search(&query)?;
|
||||
```
|
||||
|
||||
### Query Composition -- SEARCH within Scoped Results
|
||||
### Query Composition — SEARCH within Scoped Results
|
||||
|
||||
SEARCH can be composed with scope constraints. This enables searching within trending, within a cohort, or within any candidate set.
|
||||
|
||||
@ -525,7 +525,7 @@ let results = db.search(&query)?;
|
||||
| `Category { name }` | Items in a category |
|
||||
| `Collection { id }` | Items in a collection |
|
||||
|
||||
### SUGGEST -- Autocomplete and Suggestions
|
||||
### SUGGEST — Autocomplete and Suggestions
|
||||
|
||||
```rust
|
||||
use tidaldb::query::suggest::Suggest;
|
||||
|
||||
39
CHANGELOG.md
39
CHANGELOG.md
@ -4,6 +4,45 @@ All notable changes to tidalDB will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
**M9 — Community Sync & Revocation**
|
||||
- Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata`. Community signal reconciliation via `CrdtSignalState` with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.
|
||||
|
||||
**M10 — Governance & Agent Rights**
|
||||
- Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.
|
||||
|
||||
### Changed
|
||||
|
||||
**Cluster mode (m8p8): real gRPC replication**
|
||||
- `tidal-server`'s `ClusterState` now wires each follower region to a real
|
||||
`tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
|
||||
crossbeam channels — replication between regions traverses real gRPC/TCP
|
||||
(serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
|
||||
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
|
||||
per region) and self-heal a transient bind race by retrying on a fresh port.
|
||||
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
|
||||
offloaded to a dedicated thread so they never block the axum reactor.
|
||||
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
|
||||
blocks on its own runtime).
|
||||
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
|
||||
cluster mode is honest that it replicates over real gRPC but still runs all
|
||||
regions in one process (no host/process isolation — true multi-process
|
||||
deployment remains m8p10).
|
||||
|
||||
**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
|
||||
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
|
||||
runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
|
||||
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
|
||||
`rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
|
||||
runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
|
||||
bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
|
||||
image serves a functional 3-region cluster (write replicates to both followers
|
||||
over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
|
||||
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
|
||||
HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
|
||||
+ promote over real OS processes, feature-gated).
|
||||
|
||||
## [0.1.0] - 2026-02-23
|
||||
|
||||
### Added
|
||||
|
||||
87
CLAUDE.md
87
CLAUDE.md
@ -5,7 +5,7 @@
|
||||
|
||||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||||
|
||||
**Status:** Vision and specification phase. No implementation yet.
|
||||
**Status:** Implemented — M0–M10 shipped (embeddable engine + multi-region cluster mode). This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
|
||||
|
||||
## Find Your Guide
|
||||
|
||||
@ -20,17 +20,29 @@ A single-node-first, embeddable Rust database for the **personalized content ran
|
||||
| **Follow coding standards** | [CODING_GUIDELINES.md](CODING_GUIDELINES.md) |
|
||||
| **See the API spec** | [API.md](API.md) |
|
||||
| **Read architectural lessons** | [thoughts.md](thoughts.md) |
|
||||
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
|
||||
| **Browse all engineering docs** | [docs/README.md](docs/README.md) (index of specs, planning, research, reviews, ops, runbooks, profiling) |
|
||||
| **See the roadmap / milestone history** | [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md), [CHANGELOG.md](CHANGELOG.md) |
|
||||
| **Read code-review findings** | [docs/reviews/](docs/reviews/) |
|
||||
| **Operate / monitor in production** | [docs/ops/](docs/ops/), [docs/runbooks/](docs/runbooks/) |
|
||||
| **Read technical research** | [docs/research/](docs/research/) |
|
||||
| **Contribute** | [CONTRIBUTING.md](CONTRIBUTING.md) |
|
||||
|
||||
## Agents
|
||||
|
||||
| Agent | Identity | Use when |
|
||||
|-------|----------|----------|
|
||||
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||||
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||||
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||||
| **@tidal-distributed** | Kyle Kingsbury | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||||
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||||
This is the canonical agent roster. `AGENTS.md` mirrors it for tools that read that file; keep the two in sync.
|
||||
|
||||
| Agent | Identity | Model | Use when |
|
||||
|-------|----------|-------|----------|
|
||||
| **@tidal-engineer** | Jon Gjengset | opus | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||||
| **@tidal-visionary** | Spencer Kimball | opus | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||||
| **@tidal-researcher** | Andy Pavlo | opus | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||||
| **@tidal-distributed** | Kyle Kingsbury | opus | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||||
| **@tidal-storyteller** | — | sonnet | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||||
|
||||
**Utility agent:** `@knowledge-librarian` (sonnet) — classifies, cross-references, and maintains the project knowledge base.
|
||||
|
||||
**Vendored team (for the `applications/` consumers, not the database):** `@kai-park` (Aeries full-stack engineer), `@kaya-osei` (Aeries product designer), and `@mira-vasquez` (Aeries product visionary) back the `aeries-*` skills. They are scoped to companion-app work, not the tidalDB engine.
|
||||
|
||||
## Skills
|
||||
|
||||
@ -75,43 +87,50 @@ Dev servers use port range **59520–59529** (e.g. `site/` on 59520).
|
||||
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
|
||||
- **Single-node first:** Embeddable. Scales vertically before horizontally.
|
||||
- **Language:** Rust.
|
||||
- **Docs have two canonical homes:** top-level `*.md` and `docs/`. Edit the canonical file — never a per-crate mirror. `.sdlc/` is live SDLC tooling and `applications/*/` docs belong to those consumer products; neither is part of the database doc set.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
This repository is a standalone Cargo workspace (members: `tidal`, `tidal-net`, `tidalctl`,
|
||||
`tidal-server`, and the `applications/` consumers). **Documentation has exactly two homes:**
|
||||
the top-level `*.md` files and `docs/`. Do not create per-crate doc mirrors (e.g. `tidal/docs/`,
|
||||
`tidal/ai-lookup/`, `tidal/site/`) — those were a stale duplicate and were consolidated away.
|
||||
|
||||
```
|
||||
. # Top-level docs and configuration
|
||||
├── CLAUDE.md # This file — project instructions
|
||||
├── VISION.md # Product vision and thesis
|
||||
├── USE_CASES.md # 14 use cases, all discovery surfaces
|
||||
├── SEQUENCE.md # Data flow sequence diagrams
|
||||
├── CODING_GUIDELINES.md # Engineering standards
|
||||
├── API.md # API specification
|
||||
├── thoughts.md # Architectural lessons from sister projects
|
||||
├── ai-lookup/ # Domain concept reference
|
||||
├── docs/ # Research and documentation
|
||||
│ └── research/ # Deep technical research docs
|
||||
├── .claude/ # Claude Code configuration
|
||||
│ ├── agents/ # Agent definitions
|
||||
│ └── skills/ # Skill definitions
|
||||
├── tidal/ # Rust database engine
|
||||
. # Workspace root — canonical docs + config
|
||||
├── Cargo.toml # Workspace manifest (8 members)
|
||||
├── CLAUDE.md AGENTS.md README.md CONTRIBUTING.md CHANGELOG.md
|
||||
├── VISION.md USE_CASES.md SEQUENCE.md ARCHITECTURE.md
|
||||
├── API.md QUICKSTART.md CODING_GUIDELINES.md thoughts.md
|
||||
├── ai-lookup/ # Domain concept reference (index.md + features/ + services/)
|
||||
├── docs/ # Engineering docs (see docs/README.md for the index)
|
||||
│ ├── specs/ # 00–14 component specifications
|
||||
│ ├── planning/ # ROADMAP.md + per-milestone phase/task archive
|
||||
│ ├── research/ # Deep technical research docs
|
||||
│ ├── reviews/ # Code-review passes
|
||||
│ ├── ops/ # Monitoring, alerts, capacity planning
|
||||
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
|
||||
│ └── profiling/ # Flamegraph + scale profiling notes
|
||||
├── .claude/ # Claude Code config — agents/ and skills/ (canonical; .agents/ removed)
|
||||
├── tidal/ # The `tidaldb` engine crate (CLAUDE.md here is a thin crate pointer)
|
||||
│ ├── Cargo.toml
|
||||
│ ├── src/
|
||||
│ │ ├── storage/ # Entity store, signal ledger, inverted index, HNSW
|
||||
│ │ ├── query/ # Query parser, planner, executor
|
||||
│ │ ├── ranking/ # Profile engine, signal scoring, diversity enforcement
|
||||
│ │ ├── signals/ # Signal types, decay, velocity, windowed aggregation
|
||||
│ │ └── schema/ # Schema definition, validation, migrations
|
||||
│ ├── benches/ # Performance benchmarks
|
||||
│ └── tests/ # Integration and property tests
|
||||
└── site/ # Public marketing site (Next.js)
|
||||
│ ├── src/ # cohort, db, entities, governance, load, query, ranking,
|
||||
│ │ # replication, schema, session, signals, storage, testing, text, wal
|
||||
│ ├── benches/ examples/ tests/ docker/
|
||||
├── tidal-net/ # Network transport primitives (gRPC, WAL shipping)
|
||||
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
|
||||
├── tidalctl/ # CLI for inspecting persisted databases
|
||||
├── applications/ # Example consumers (forage, iknowyou)
|
||||
└── site/ # Public marketing site (Next.js, dev on 59520)
|
||||
```
|
||||
|
||||
## Pre-commit Hooks
|
||||
|
||||
The pre-commit hook runs automatically on staged files:
|
||||
- **tidal/ (Rust):** `cargo fmt` (auto-fix + re-stage), `cargo clippy -D warnings`, `cargo test --lib`
|
||||
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
|
||||
- **site/ (Next.js):** `eslint` (if node_modules installed)
|
||||
- **Docs:** `scripts/check-docs.sh` — fails if a doc mirror reappears (`tidal/docs/`, `.ai/`, `.agents/skills/`), if CLAUDE.md's workspace members drift from `Cargo.toml`, or if a canonical cross-reference breaks.
|
||||
|
||||
All cargo commands use `--manifest-path tidal/Cargo.toml` since the Rust project is not at repo root.
|
||||
Cargo commands target the engine crate with `-p tidaldb` from the workspace root (equivalently `--manifest-path tidal/Cargo.toml`).
|
||||
|
||||
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
|
||||
|
||||
@ -140,10 +140,19 @@ Never hardcode a single filtering strategy. Estimate selectivity, then branch:
|
||||
The entity store is the source of truth. Tantivy is a materialized view. If the Tantivy index is corrupted or lost, it can be rebuilt from the entity store.
|
||||
|
||||
Consistency pattern:
|
||||
1. Write to entity store (within transaction / WAL)
|
||||
2. Background indexer reads outbox and feeds Tantivy
|
||||
3. On each Tantivy commit, store last-processed sequence number in commit payload
|
||||
4. On crash recovery, replay from that sequence number
|
||||
1. Write to entity store (within transaction / WAL) — the durable source of truth.
|
||||
2. Background indexer reads the outbox and feeds Tantivy (best-effort, async).
|
||||
3. On crash recovery, unconditionally rebuild the text index from the entity
|
||||
store at open — the same proven pattern as the vector index, which is rebuilt
|
||||
from the durable embeddings on every reopen.
|
||||
|
||||
> Crash recovery is NOT driven by a Tantivy commit-payload sequence number.
|
||||
> Because the indexer is best-effort and async, an item can be durably persisted
|
||||
> to the entity store and then lost in a crash before the syncer commits it to
|
||||
> Tantivy. A stored sequence number does not heal that gap by itself and is
|
||||
> easy to leave un-wired (it once silently was, hiding the lost write). The
|
||||
> unconditional rebuild-from-store at open is deterministic and cannot silently
|
||||
> regress: any entity in the store but missing from Tantivy is re-indexed.
|
||||
|
||||
### Hybrid fusion starts with RRF
|
||||
|
||||
|
||||
@ -62,27 +62,17 @@ cargo bench --manifest-path tidal/Cargo.toml --no-run # ensure benches compi
|
||||
|
||||
## Project Layout
|
||||
|
||||
```
|
||||
tidal/ Rust database engine
|
||||
src/
|
||||
db/ TidalDb handle, builder, config, metrics
|
||||
schema/ Types, validation, error types
|
||||
signals/ Signal ledger, decay, windowed counters, checkpoint
|
||||
storage/ StorageEngine trait, fjall backend, key encoding
|
||||
wal/ Write-ahead log, group commit, crash recovery
|
||||
benches/ Criterion benchmarks
|
||||
examples/ Embedding guides (quickstart, axum, actix, cli)
|
||||
tests/ Integration and property tests
|
||||
site/ Marketing site (Next.js)
|
||||
docs/ Research and planning documents
|
||||
```
|
||||
This is a Cargo workspace — the `tidaldb` engine crate is at `tidal/`, with `tidal-net/`,
|
||||
`tidal-server/`, and `tidalctl/` as siblings and example consumers under `applications/`.
|
||||
See **[CLAUDE.md § Repository Structure](CLAUDE.md#repository-structure)** for the full,
|
||||
canonical layout (including the `tidal/src/` module map and the `docs/` doc homes).
|
||||
|
||||
## Coding Standards
|
||||
|
||||
See [CODING_GUIDELINES.md](CODING_GUIDELINES.md) for the full engineering standards.
|
||||
|
||||
Key rules:
|
||||
- `Result<T, LumenError>` everywhere — no panics on recoverable failures
|
||||
- `Result<T, TidalError>` everywhere — no panics on recoverable failures
|
||||
- `#![forbid(unsafe_code)]` — relaxed only at explicit FFI boundaries with `// SAFETY:` comment
|
||||
- Property tests for invariants, criterion benchmarks for performance claims
|
||||
- `cargo clippy -D warnings` must pass with zero warnings
|
||||
|
||||
@ -20,9 +20,12 @@ The rest of this guide explains what it does and extends it with personalization
|
||||
|
||||
## Step 1: Add the dependency
|
||||
|
||||
Depend on the `tidaldb` crate (the `tidal/` crate in this repository) by git or by path:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
tidaldb = { git = "https://github.com/your-org/tidalDB", rev = "..." }
|
||||
tidaldb = { git = "https://github.com/orchard9/tidaldb", rev = "..." }
|
||||
# or, for a local checkout: tidaldb = { path = "path/to/tidaldb/tidal" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@ -95,10 +95,11 @@ Pick the path that matches how you plan to use tidalDB today. Every option below
|
||||
|
||||
**Setup**
|
||||
|
||||
1. Add the git dependency:
|
||||
1. Add the dependency (the `tidaldb` crate is at `tidal/` in this repository):
|
||||
```toml
|
||||
[dependencies]
|
||||
tidaldb = { git = "https://github.com/your-org/tidalDB", rev = "..." }
|
||||
tidaldb = { git = "https://github.com/orchard9/tidaldb", rev = "..." }
|
||||
# or, for a local checkout: tidaldb = { path = "path/to/tidaldb/tidal" }
|
||||
```
|
||||
2. Define your schema before opening the database (decay, windows, text fields, embeddings). The snippet in **[Quickstart, Step 2](QUICKSTART.md#step-2-define-a-schema)** is a ready-to-copy template.
|
||||
3. Choose storage mode when building:
|
||||
@ -209,7 +210,7 @@ curl -X POST http://localhost:4242/signal \
|
||||
-d '{ "user_id": 1, "item_id": 42, "signal_type": "view" }'
|
||||
curl "http://localhost:4242/feed?user=1&limit=7"
|
||||
```
|
||||
The UI shows seeded users, exploration labels, and real-time adaptation; see `applications/forage/readme.md` for the full loop.
|
||||
The UI shows seeded users, exploration labels, and real-time adaptation; see `applications/forage/README.md` for the full loop.
|
||||
|
||||
### 5. Run the cluster server + Docker image
|
||||
|
||||
|
||||
24
applications/iknowyou/README.md
Normal file
24
applications/iknowyou/README.md
Normal file
@ -0,0 +1,24 @@
|
||||
# iknowyou
|
||||
|
||||
Communication personalization as a signal-processing problem. iknowyou wraps tidalDB's
|
||||
signal ledger, preference vectors, and windowed aggregation with an observation pipeline
|
||||
(LM-as-classifier), a briefing engine (query-to-profile), and a generation interface
|
||||
(brief-to-prompt) — so a system can learn how each person prefers to be communicated with
|
||||
and adapt continuously, without a training loop or feature store.
|
||||
|
||||
> **Design vs shipped.** The vision/architecture docs describe the *target* tidalDB-native
|
||||
> design; what runs today is a Next.js + vLLM implementation. See the roadmap and dev setup
|
||||
> for current state.
|
||||
|
||||
## Docs
|
||||
|
||||
| Doc | What it covers |
|
||||
|-----|----------------|
|
||||
| [vision.md](vision.md) | The problem and the product thesis (target design) |
|
||||
| [architecture.md](architecture.md) | Domain model, pipelines, components (target design) |
|
||||
| [ROADMAP.md](ROADMAP.md) | Milestones and current build status |
|
||||
| [devsetup.md](devsetup.md) | Running the engine + Next.js app locally |
|
||||
| [engine/README.md](engine/README.md) | The personalization engine service |
|
||||
|
||||
Part of the tidalDB workspace's `applications/` example consumers — see the
|
||||
[repository root](../../README.md) for the database itself.
|
||||
@ -1,5 +1,12 @@
|
||||
# iknowyou — Architecture
|
||||
|
||||
> **Status — design vs shipped.** This document describes the *target* tidalDB-native
|
||||
> architecture. The currently *shipped* implementation is a Next.js + vLLM stack with the
|
||||
> personalization engine as a separate service — see [ROADMAP.md](ROADMAP.md) and
|
||||
> [devsetup.md](devsetup.md) for what runs today. Treat the Rust engine / `server/` design
|
||||
> below as intent, not a description of the current code. (The shipped engine dev bind is
|
||||
> `127.0.0.1:7777` per devsetup.md, which sits outside the project's 59520–59529 dev-port band.)
|
||||
|
||||
## Core Thesis
|
||||
|
||||
Communication personalization is a signal processing problem. Every exchange between the system and a person produces observable signals — engagement, sentiment, timing, style — that decay over time and compound across conversations. tidalDB's signal ledger, preference vectors, windowed aggregation, and cohort system provide the learning substrate. iknowyou wraps these primitives with an observation pipeline (LM-as-classifier), a briefing engine (query-to-profile), and a generation interface (brief-to-prompt).
|
||||
|
||||
@ -1,5 +1,9 @@
|
||||
# iknowyou — Vision
|
||||
|
||||
> **Status — design vs shipped.** This vision describes the *target* tidalDB-native
|
||||
> system. What runs today is a Next.js + vLLM implementation — see [ROADMAP.md](ROADMAP.md)
|
||||
> and [devsetup.md](devsetup.md). Treat this as direction, not current-state documentation.
|
||||
|
||||
## The Problem
|
||||
|
||||
Every system that talks to people talks to all of them the same way.
|
||||
|
||||
52
docs/README.md
Normal file
52
docs/README.md
Normal file
@ -0,0 +1,52 @@
|
||||
# tidalDB Engineering Docs
|
||||
|
||||
The engineering documentation home. Top-level product docs (VISION, USE_CASES, SEQUENCE,
|
||||
ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the
|
||||
[repository root](../CLAUDE.md); everything below is the deeper engineering record.
|
||||
|
||||
> This and the repo root are the **two canonical doc homes**. There is intentionally no
|
||||
> per-crate doc mirror (no `tidal/docs/`). Edit the canonical file, never a copy.
|
||||
|
||||
## Component specs — `specs/`
|
||||
|
||||
The authoritative component specifications (status: Implemented, M0–M8).
|
||||
|
||||
| # | Spec | # | Spec |
|
||||
|---|------|---|------|
|
||||
| 00 | [Architecture overview](specs/00-architecture-overview.md) | 08 | [Query engine](specs/08-query-engine.md) |
|
||||
| 01 | [Storage engine](specs/01-storage-engine.md) | 09 | [Ranking & scoring](specs/09-ranking-scoring.md) |
|
||||
| 02 | [Entity model](specs/02-entity-model.md) | 10 | [Feedback loop](specs/10-feedback-loop.md) |
|
||||
| 03 | [Signal system](specs/03-signal-system.md) | 11 | [Schema](specs/11-schema.md) |
|
||||
| 04 | [Relationships](specs/04-relationships.md) | 12 | [Cold start](specs/12-cold-start.md) |
|
||||
| 05 | [Cohorts](specs/05-cohorts.md) | 13 | [Concurrency](specs/13-concurrency.md) |
|
||||
| 06 | [Text retrieval](specs/06-text-retrieval.md) | 14 | [Scale architecture](specs/14-scale-architecture.md) |
|
||||
| 07 | [Vector retrieval](specs/07-vector-retrieval.md) | | |
|
||||
|
||||
## Planning — `planning/`
|
||||
|
||||
- [**ROADMAP.md**](planning/ROADMAP.md) — milestones M0–M10, phase status, known gaps
|
||||
- [PRODUCT_ROADMAP.md](planning/PRODUCT_ROADMAP.md) · [architecture-review.md](planning/architecture-review.md) · [roadmap-cohort-analysis.md](planning/roadmap-cohort-analysis.md) · [site-cohort-analysis.md](planning/site-cohort-analysis.md)
|
||||
- Per-milestone phase/task archive: `planning/milestone-0,1,2,3,5,7,8,9,10,p/`
|
||||
|
||||
## Code reviews — `reviews/`
|
||||
|
||||
- [M0–M10 code review — 2026-06-07](reviews/M0-M10-code-review-2026-06-07.md) — seven-dimension re-review, 88 verified findings
|
||||
- [M0–M10 code review — 2026-06-08](reviews/M0-M10-code-review-2026-06-08.md) — seven-dimension review, 142 findings (latest pass)
|
||||
- [M0–M10 seven-dimension review](reviews/M0-M10-seven-dimension-review.md) — additional pass (2 BLOCKERs: signal-checkpoint trim, 30-day window)
|
||||
|
||||
## Operations — `ops/` and `runbooks/`
|
||||
|
||||
- [Monitoring](ops/monitoring.md) · [Prometheus alerts](ops/prometheus-alerts.yaml) · [Grafana dashboard](ops/grafana-dashboard.json) · [Capacity planning](ops/capacity-planning.md) · [Recovery](ops/recovery.md)
|
||||
- Runbooks: [Cluster](runbooks/cluster.md)
|
||||
|
||||
## Research — `research/`
|
||||
|
||||
ANN ([1](research/ann_for_tidaldb.md), [2](research/ann_for_tidaldb_gemini.md)) · Tantivy ([1](research/tantivy.md), [2](research/tantivy_gemini.md)) · Signal ledger ([1](research/tidaldb_signal_ledger.md), [2](research/tidaldb_signal_ledger_gemini.md)) · [WAL](research/tidaldb_wal.md) · [Type system](research/phase1_1_type_system.md) · [Tooling & diagnostics](research/tidaldb_tooling_and_diagnostics.md) · [Enterprise-readiness risks](research/enterprise_readiness_risks.md)
|
||||
|
||||
## Profiling — `profiling/`
|
||||
|
||||
[Hotspot analysis](profiling/hotspot-analysis.md) · [Scale baselines](profiling/scale-baselines.md) · [Signal memory](profiling/signal-memory-analysis.md) · [Signal rollup eval](profiling/signal-rollup-eval.md) · [Social scale](profiling/social-scale.md) · [Tantivy merge tuning](profiling/tantivy-merge-tuning.md) · [USearch tuning](profiling/usearch-tuning.md)
|
||||
|
||||
## Strategy
|
||||
|
||||
[Content strategy](content-strategy.md) · [Personal-briefing beachhead](personal-briefing-beachhead.md)
|
||||
@ -137,7 +137,7 @@ Index health metrics are refreshed every 10 seconds by the checkpoint thread (3x
|
||||
| WAL Disk Pressure | `tidaldb_wal_lag_bytes > 1000000000` | Warning | WAL exceeds 1 GB uncompacted. Compaction may be stuck or checkpoint is failing. |
|
||||
| Signal Backlog | `tidaldb_signal_hot_entries > 4000000` | Warning | Signal ledger over 80% of the 5M entry budget. Cold entry trimming will begin at 5M. |
|
||||
| Degraded Ranking | `tidaldb_degradation_level > 0` | Warning | Load-based degradation is active. Ranking quality is reduced to protect latency. Scale up or reduce load. |
|
||||
| Session Leak | `rate(tidaldb_active_sessions[5m]) > 10 AND tidaldb_active_sessions > 100` | Warning | Active session count growing rapidly. Agents may not be closing sessions. |
|
||||
| Session Leak | `deriv(tidaldb_active_sessions[5m]) > 0.03 AND tidaldb_active_sessions > 100` | Warning | Active session count trending up. `tidaldb_active_sessions` is a gauge, so `deriv()` (slope) is correct — `rate()`/`increase()` are invalid on gauges and never fire. Agents may not be closing sessions. |
|
||||
| High Rate Limiting | `rate(tidaldb_rate_limited_total[5m]) > 100` | Info | Sustained rate limiting. Review agent rate limit configuration or reduce write volume. |
|
||||
| Tantivy Segment Bloat | `tidaldb_tantivy_segment_count > 30` | Warning | Tantivy has many unmerged segments. Text syncer may be stalled. |
|
||||
|
||||
|
||||
@ -1,3 +1,40 @@
|
||||
# =============================================================================
|
||||
# DESIGN-REFERENCE RULE SET — NOT LOADED BY ANY ALERTMANAGER TODAY.
|
||||
# =============================================================================
|
||||
# This file is reference/design config. It is NOT wired into any running
|
||||
# alerting pipeline:
|
||||
# - vmalert (pilot/dev) loads ONLY ops/vmalert/rules/*.yaml
|
||||
# - prod CRDs live ONLY in infra/k8s/prom/prometheus-rules/*.yaml
|
||||
# Nothing mounts or imports docs/ops/prometheus-alerts.yaml.
|
||||
# Do not read these as live pager rules.
|
||||
#
|
||||
# Provenance: every metric referenced below (tidaldb_health_ok,
|
||||
# tidaldb_checkpoint_age_seconds, tidaldb_checkpoint_failures_total,
|
||||
# tidaldb_wal_lag_bytes, tidaldb_signal_hot_entries, tidaldb_degradation_level,
|
||||
# tidaldb_active_sessions, tidaldb_rate_limited_total,
|
||||
# tidaldb_tantivy_segment_count, tidaldb_retrieve_latency_us_bucket,
|
||||
# tidaldb_search_latency_us_bucket) IS really emitted today by
|
||||
# tidal/src/db/metrics/mod.rs. So these rules are promotable — they
|
||||
# are not orphaned against phantom metrics.
|
||||
#
|
||||
# Promotion path (the right long-term fix — do it deliberately, do NOT
|
||||
# hand-copy these as live pager rules without the steps below):
|
||||
# 1. Add a vmalert rule file: ops/vmalert/rules/tidaldb.yaml, translating
|
||||
# each rule and stamping the canonical labels every routed rule carries —
|
||||
# severity + capability + surface (+ optional component). Map per the
|
||||
# canonical severity taxonomy in ops/alerting.md:
|
||||
# labels {severity: critical} = pager class (TidalDBDown only here),
|
||||
# {severity: warning} = non-paging / business-hours,
|
||||
# {severity: info} = pure-FYI.
|
||||
# Add capability: tidaldb and a surface label per rule, and a
|
||||
# runbook_url annotation:
|
||||
# https://github.com/orchard9/tidaldb/blob/main/docs/runbooks/<slug>.md
|
||||
# 2. Add the prod CRD twin: infra/k8s/prom/prometheus-rules/tidaldb.yaml,
|
||||
# kept byte-aligned in expr/threshold with the vmalert file.
|
||||
# 3. Verify the alerts fire against real metrics (scrape tidaldb, force a
|
||||
# degraded state) before declaring them on-call-ready.
|
||||
# Until steps 1–3 land, this file stays a design reference only.
|
||||
# =============================================================================
|
||||
groups:
|
||||
- name: tidaldb
|
||||
interval: 30s
|
||||
@ -50,12 +87,16 @@ groups:
|
||||
description: "Degradation level {{ $value }} active. Scale up or reduce load."
|
||||
|
||||
- alert: TidalDBSessionLeak
|
||||
expr: rate(tidaldb_active_sessions[5m]) > 10 and tidaldb_active_sessions > 100
|
||||
# tidaldb_active_sessions is a GAUGE, so rate()/increase() are invalid
|
||||
# (those operators only count counter resets). deriv() fits a least-squares
|
||||
# slope over the window and is the gauge-correct way to detect sustained
|
||||
# growth: > 0.03 sessions/sec is ~> 9 new sessions over a 5m window.
|
||||
expr: deriv(tidaldb_active_sessions[5m]) > 0.03 and tidaldb_active_sessions > 100
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Active session count growing rapidly"
|
||||
description: "{{ $value }} active sessions and growing. Agents may not be closing sessions."
|
||||
summary: "Active session count growing steadily"
|
||||
description: "{{ $value }} active sessions and trending up (deriv > 0.03/s). Agents may not be closing sessions."
|
||||
|
||||
- alert: TidalDBHighRateLimiting
|
||||
expr: rate(tidaldb_rate_limited_total[5m]) > 100
|
||||
|
||||
@ -34,7 +34,7 @@ A single embeddable database can replace the 6-system content ranking stack by t
|
||||
| M5 | Hybrid Search | Text + semantic + signal-ranked search in one query | UC-02, UC-10, UC-11 |
|
||||
| M6 | Full Surface Coverage | Every use case, every sort mode, every filter, every feedback loop | UC-01 through UC-14 complete |
|
||||
| M7 | Production Hardening | Crash safety, graceful degradation, operational readiness | All UCs at production quality |
|
||||
| M8 | Distributed Fabric | Multi-region, multi-tenant replication keeps agent-memory semantics intact | Hosted tidalDB, cloud/edge deployments, shared agent substrate (in-process primitives COMPLETE; multi-node PARTIAL — transport + HTTP surface done, GrpcTransport wiring + tier-3 tests pending) |
|
||||
| M8 | Distributed Fabric | Multi-region, multi-tenant replication keeps agent-memory semantics intact | Hosted tidalDB, cloud/edge deployments, shared agent substrate (in-process primitives + multi-node replication over real gRPC COMPLETE, G1 resolved; only tier-3 chaos tests — partition injection, clock-skew, rolling-upgrade — pending, m8p10) |
|
||||
| M9 | Community Sync & Revocation | Local embeddable profiles can opt into community personalization and safely leave/purge contributions | Community personalization, federated taste graphs, shared feeds — ✅ COMPLETE (2026-06-06) |
|
||||
| M10 | Governance & Agent Rights | Community rules and agent-scoped permissions control what signals influence ranking | User-owned AI personalization at scale, policy-compliant agents — ✅ COMPLETE (2026-06-06) |
|
||||
|
||||
|
||||
@ -289,7 +289,7 @@ serde_json = "1"
|
||||
|
||||
### What This Means for Pre-Commit Hooks and CI
|
||||
|
||||
The root `Cargo.toml` becomes the workspace root. All `cargo` commands (`fmt`, `clippy`, `test`) need to run from the workspace root or with `--workspace`. The pre-commit hook currently uses `--manifest-path tidal/Cargo.toml` -- this must be updated to use the workspace root.
|
||||
The root `Cargo.toml` becomes the workspace root. All `cargo` commands (`fmt`, `clippy`, `test`) need to run from the workspace root or with `--workspace`. (Resolved: tidalDB is the `tidaldb` crate at `tidal/` in this workspace; commands target it with `-p tidaldb` from the workspace root, or `--manifest-path tidal/Cargo.toml`.)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@ -1111,7 +1111,7 @@ proptest! {
|
||||
- [VISION.md](../../../../VISION.md) -- User-state filters as first-class query primitives
|
||||
- [USE_CASES.md](../../../../USE_CASES.md) -- Filter reference (Appendix A: unseen, unblocked, saved, liked)
|
||||
- [API.md](../../../../API.md) -- FILTER clause syntax
|
||||
- [m2p2 filter.rs](../../../../tidal/src/storage/indexes/filter.rs) -- Existing FilterExpr and FilterResult types
|
||||
- [m2p2 filter module](../../../../tidal/src/storage/indexes/filter/expr.rs) -- Existing FilterExpr and FilterResult types
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
|
||||
@ -71,7 +71,7 @@ Deliverables:
|
||||
event payload
|
||||
- `thoughts.md` -- Part II.1 (WAL convergence / replay determinism),
|
||||
Part V.12 (subject-prefix key encoding for the membership keyspace)
|
||||
- `tidal/docs/planning/milestone-8/phase-1/` -- ReplicationState extension
|
||||
- `docs/planning/milestone-8/phase-1/` -- ReplicationState extension
|
||||
pattern, identity-newtype conventions
|
||||
|
||||
## Acceptance Criteria (Phase Level)
|
||||
@ -48,8 +48,8 @@ Deliverables:
|
||||
- `docs/research/tidaldb_wal.md` -- WAL replay determinism, segment ordering
|
||||
- `docs/research/tidaldb_signal_ledger.md` -- running-decay recompute, hot/warm tier rebuild from raw events
|
||||
- `thoughts.md` -- Part II.1 (WAL convergence), Part V.5 (quarantine-first removal), CRDT merge laws (Kulkarni et al. 2014, HLC)
|
||||
- `tidal/docs/planning/milestone-8/phase-3` -- `CrdtSignalState` merge laws (commutative/associative/idempotent) this phase must preserve
|
||||
- `tidal/docs/planning/milestone-9/phase-1` & `phase-2` -- `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata` (GIVEN; do not redefine)
|
||||
- `docs/planning/milestone-8/phase-3` -- `CrdtSignalState` merge laws (commutative/associative/idempotent) this phase must preserve
|
||||
- `docs/planning/milestone-9/phase-1` & `phase-2` -- `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata` (GIVEN; do not redefine)
|
||||
|
||||
## Acceptance Criteria (Phase Level)
|
||||
|
||||
@ -858,7 +858,7 @@ No other dependencies are needed for m1p1. All types (EntityId, Timestamp, Decay
|
||||
- [InfluxDB timestamp precision documentation](https://www.influxdata.com/blog/tldr-tech-tips-flux-timestamps/)
|
||||
- [QuestDB timestamp functions](https://questdb.com/docs/query/functions/date-time/)
|
||||
- [Forward Decay - Cormode, Shkapenyuk, Srivastava, Xu (ICDE 2009)](https://doi.org/10.1109/ICDE.2009.65)
|
||||
- [Lumen Signal Ledger Research](docs/research/lumens_signal_ledger.md)
|
||||
- [TidalDB CODING_GUIDELINES.md](CODING_GUIDELINES.md)
|
||||
- [TidalDB API.md](API.md)
|
||||
- [TidalDB thoughts.md](thoughts.md)
|
||||
- [Signal Ledger Research](tidaldb_signal_ledger.md)
|
||||
- [TidalDB CODING_GUIDELINES.md](../../CODING_GUIDELINES.md)
|
||||
- [TidalDB API.md](../../API.md)
|
||||
- [TidalDB thoughts.md](../../thoughts.md)
|
||||
|
||||
@ -1091,7 +1091,7 @@ views.mul_add(
|
||||
|
||||
**Fix.** If a linear weighted sum was intended, change `completion * views * 0.1` to `completion * 0.1`. If the quadratic interaction is genuinely intended, document it and push the actual `completion*views` product into the snapshot so explainability matches the score.
|
||||
|
||||
> **Verification.** In `score_top_window` (lines 265–313 of `/Users/jordan.washburn/Workspace/orchard9/tidaldb/tidal/src/ranking/executor/scoring.rs`), `completion` is read with `SignalAgg::Value` (a raw event count, not a rate), and the score term at line 305 is `completion * views * 0.1` — two raw counts multiplied together, making the contribution quadratic in scale. The spec at `/Users/jordan.washburn/Workspace/orchard9/tidaldb/tidal/docs/specs/09-ranking-scoring.md` lines 1188–1192 specifies `completion_rate(w) * view.count(w) * 0.1` — a rate (0–1) times a count, which is a dimensionally sensible cross-term. The implementation substitutes a raw completion …
|
||||
> **Verification.** In `score_top_window` (lines 265–313 of `/Users/jordan.washburn/Workspace/orchard9/tidaldb/tidal/src/ranking/executor/scoring.rs`), `completion` is read with `SignalAgg::Value` (a raw event count, not a rate), and the score term at line 305 is `completion * views * 0.1` — two raw counts multiplied together, making the contribution quadratic in scale. The spec at `/Users/jordan.washburn/Workspace/orchard9/tidaldb/docs/specs/09-ranking-scoring.md` lines 1188–1192 specifies `completion_rate(w) * view.count(w) * 0.1` — a rate (0–1) times a count, which is a dimensionally sensible cross-term. The implementation substitutes a raw completion …
|
||||
|
||||
### W23. RollingUpgradeCoordinator uses three different poison-handling strategies for the same lock; is_drained silently reports a drained shard as serving
|
||||
|
||||
636
docs/reviews/M0-M10-seven-dimension-review.md
Normal file
636
docs/reviews/M0-M10-seven-dimension-review.md
Normal file
@ -0,0 +1,636 @@
|
||||
# tidalDB M0–M10 — Seven-Dimension Code Review
|
||||
|
||||
**Scope:** Full engine across milestones M0–M10 — `tidal` (db, wal, signals, storage, query, ranking, entities, governance, replication, crdt, schema, session, text, cohort/load/testing), `tidal-net`, `tidal-server`, `tidalctl`. ~86K LOC of Rust source across 28 subsystem slices.
|
||||
|
||||
**Method:** Each slice reviewed against the seven-dimension protocol (Completeness · Accuracy · Tech Debt · Maintainability · Extensibility · DRY · CLEAN) plus the project's `CODING_GUIDELINES.md` and Rust red-flag tables, in the correctness-first "trust it at 3am" standard. Every BLOCKER/CRITICAL was then handed to an independent adversarial verifier instructed to *refute* it by reading the actual code; only survivors are reported below.
|
||||
|
||||
---
|
||||
|
||||
## Remediation status (implemented)
|
||||
|
||||
All findings below have been remediated. Workspace is green: `cargo clippy -p tidaldb -D warnings` (and on `tidal-net`/`tidal-server`/`tidalctl`) clean, `cargo build --workspace --all-targets` clean, `cargo fmt` applied, `scripts/check-docs.sh` OK; engine lib 1663 tests / 0 failures (deterministic), engine integration + sibling suites green. ~160 regression tests were added.
|
||||
|
||||
- **Both BLOCKERs and all nine CRITICALs** are fixed with dedicated regression tests.
|
||||
- The BLOCKER #1 fix is read-side (`30d = last-7d hour tier + day buckets older than 7 days`); the report's suggested "clear the hour buckets on day rotation" would break the 7-day window (which legitimately reads all 168 hour buckets), so the read-side approach was used instead.
|
||||
- The **WARNINGs and SUGGESTIONs** were remediated across the named slices (DRY extractions, validation, durability ordering, doc accuracy, perf, bounds). A handful were intentionally taken as documented-limitations where the reviewer offered that alternative (e.g. Rising/MostFollowed/CreatorEngagementRate remain custom-profile-only with a doc note; `lag_for`'s single-shard scope stays documented).
|
||||
- **W1** (`wal_dir` override): now honored end-to-end via `WalConfig::wal_dir_override` + `Config::resolved_wal_dir`, threaded to the WAL writer and every reader; `cache_dir` remains a documented reserved no-op (no consumer, and the storage backend exposes no separate cache-dir knob).
|
||||
|
||||
**Previously-deferred item — now FIXED (both read AND write side):** a *related* 24h/7d/30d windowed double-count for **continuously-active** entities (events with sub-60-minute gaps crossing an hour boundary) — the same tier-overlap class as BLOCKER #1 but at the minute→hour boundary, which the original review did not flag. The minute tier is a rolling 60-minute window (it must be, to answer `OneHour`), aged only by the elapsed minutes on each rotation, so after a gap it still holds the previous hour's tail that the rollup already copied into the newest completed hour bucket.
|
||||
|
||||
There were **two** overlap sites — and a first remediation pass fixed only the read side, which an adversarial re-review correctly flagged as a half-fix (the durable hour tier was still corrupted on the pure write path: a 90-minute dense history read at 120 min reported 477 for 361 real events). The complete fix bounds **both** sites and the day tier:
|
||||
|
||||
- **Read side** (`windowed_count` → `sum_current_hour(now_ns)`): folds in only the last `k` minute buckets, `k = (now_ns − last_hour_rotation_ns)/60s + 1` (capped at 60) — the minutes since the last hour rotation, disjoint from every completed hour bucket.
|
||||
- **Write side** (`maybe_rotate`'s `hour_agg`): the hour rollup snapshots only the in-progress hour, `k = (last_min − last_hour)/60s + 1` minute buckets walking back from `current_minute` — anchored at the rollup-time `last_min` (where `current_minute` still points), **not** `now`. (A naive `now`-anchored bound evaluates to the full 60 at the hour boundary and does not fix it — the verifier confirmed this.)
|
||||
- **Day tier** (`maybe_rotate`'s `day_agg`): the same overlap one tier up. The day rollup now folds in only the completed hour buckets inside the completing day, `j = (last_hour − last_day)/3600s` (capped 23), plus `hour_agg` — not the rolling "23 most recent hour buckets".
|
||||
|
||||
No persisted-format change: all bounds derive from `last_minute/hour/day_rotation_ns`, already in `BucketedCounterSnapshot`. Six regression tests were added that read **after a gap** and on the **pure write path** (the cases the at-`end` tests structurally could not catch); each fails against the unbounded code (477/361, 440/343, day-tier inflation) and passes against the fix. See `signals/warm.rs`.
|
||||
|
||||
---
|
||||
|
||||
## Post-remediation re-review (2026-06-08) — additional fixes
|
||||
|
||||
An adversarial multi-agent re-review of the uncommitted remediation diff surfaced six further issues (one BLOCKER, two CRITICAL, two WARNING, plus the day-tier extension above), each verified against the code and now fixed with regression tests:
|
||||
|
||||
1. **[BLOCKER] warm-tier write-side double-count** — the half-fix above; now fully bounded on read + write + day tier.
|
||||
2. **[CRITICAL] W22 stale-index half-fix** (`db/items.rs`, `db/state_rebuild.rs`) — `RangeIndex::delete_entity` was added and unit-tested but never wired into the live re-index path, so an item *overwrite* left it indexed under both the old and new category/format/creator/tag/duration/created_at value (phantom RETRIEVE/SEARCH hits; corrupted recency). Fixed: `write_item_with_metadata` now reads the prior metadata and `ItemIndexes::scrub`s every index (plus the creator-keyed `CreatorItemsBitmap`, which gained `remove_item`) before re-inserting. TidalDb-level regression test added.
|
||||
3. **[CRITICAL] checkpoint `read()` mutated disk** (`wal/checkpoint.rs`) — a `remove_stale_temp` call had been added to the shared, documented-read-only `CheckpointManager::read()`, breaking read-only-mount forensic inspection (EROFS) and racing the writer's rename. Fixed: `read()` is side-effect-free again; the orphan-temp sweep moved to `sweep_stale_temp`, called from the writable WAL `recover()` path (best-effort).
|
||||
4. **[WARNING] W13 partition_id never wired** (`db/mod.rs`) — `set_partition_id` existed but no construction site called it, so cluster nodes still emitted colliding `partition_id="0"`. Fixed: a shared `install_metrics_wiring` helper now stamps the real shard on both paths; `#[allow(dead_code)]` removed; construction-level test added.
|
||||
5. **[WARNING] SocialGraph/InCollection u32 truncation** (`query/executor/post_filter.rs`) — the inclusion arms used `id.as_u64() as u32`, which *aliases* a `> u32::MAX` id onto a low slot and could wrongly retain it, contradicting the `entity_as_u32` invariant. Both arms now route through the checked conversion (dropping non-representable ids); `entity_as_u32` made `pub(crate)`; overflow-id regression test added.
|
||||
|
||||
Workspace re-verified green after these fixes (clippy `-D warnings` on all four crates incl. `--features metrics`, `build --workspace --all-targets`, `fmt`, `check-docs.sh`; engine lib + integration + sibling suites; `metrics_integration` stress 60/60 after the non-blocking-accept fix).
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Severity | Count |
|
||||
|---|---|
|
||||
| BLOCKER | 2 |
|
||||
| CRITICAL | 9 |
|
||||
| WARNING | 50 |
|
||||
| SUGGESTION | 80 |
|
||||
|
||||
**Adversarial verification:** 16 high-severity findings examined → **14 confirmed**, **2 refuted** as false positives (and 3 confirmed-but-downgraded to WARNING).
|
||||
|
||||
**Recommendation: REQUEST_CHANGES.** Two BLOCKERs are silent data-loss/correctness holes on documented, expected operations (signal-checkpoint trim, 30-day window). The nine CRITICALs are real, reachable bugs — none are happy-path-instant crashes, which is why they're CRITICAL not BLOCKER, but several are quadratic cliffs, OOM paths, unbounded leaks, or silent data deletion. The engine's durability *machinery* is genuinely strong (see What's Good); the defects are in the seams where derived state, limits, and lifecycle cleanup are not wired all the way through.
|
||||
|
||||
### Top action items
|
||||
|
||||
1. **Fix the signal-checkpoint trim hole (BLOCKER, `signals/checkpoint/mod.rs`)** — after the ledger ever exceeds the 5M trim threshold, the first restart silently loses *all* checkpoint-only signal state because checkpoint never deletes stale keys, the integrity hash mismatches, restore returns `Ok(None)`, and the covering WAL has already been compacted. Make checkpoint authoritative over the keyspace (emit deletes for evicted rows in the same atomic batch).
|
||||
2. **Fix the 30-day windowed-count double-count (BLOCKER, `signals/warm.rs`)** — for ~24h after every day boundary, day-tier rollup double-counts the still-populated hour buckets; 116 real events report as 216. This directly inflates trending/velocity ranking. Clear the hour buckets on day rotation (mirror the minute→hour symmetry) and add a dense-in-progress-day regression test.
|
||||
3. **Kill the two unbounded-growth / quadratic paths** — the community ledger does a full O(total-contributions) durable snapshot on *every* accepted contribution (`db/communities.rs` → `governance/community_ledger.rs`, O(N²) ingest), and `notification_tracker::evict_old_days` has zero callers so its maps leak forever. Both are simple wirings (incremental put; sweeper tick).
|
||||
|
||||
---
|
||||
|
||||
## BLOCKER & CRITICAL findings (verified)
|
||||
|
||||
### 1. [BLOCKER] 30d windowed count double-counts ~24h of events for a full day after each day boundary
|
||||
**Slice:** `signals-decay-hotwarm` **Dimension:** Accuracy **Location:** `tidal/src/signals/warm.rs:213, 471-505` **Confidence:** 97
|
||||
|
||||
**Issue.** windowed_count(ThirtyDays) returns sum_current_day() + sum_last_n_days(30). sum_current_day() reads the in-progress hour (minute buckets) plus the 23 most-recent hour buckets. But rotate_day/rotate_days roll those same 24 hours of hour-tier data into a new day bucket WITHOUT ever clearing the hour buckets (the only hour_buckets write is rotate_hour at line 294, which overwrites the next slot on hour rotation; day rotation never zeroes hours). For roughly a full day after each day boundary, the just-rolled-up events live in BOTH the still-populated hour buckets (read by sum_current_day) AND the new day bucket (read by sum_last_n_days(30)). Verified empirically by a temporary test: 116 real events reported as 216 (sum_current_day=101 + sum_last_n_days30=115). Compare the 24h path, which is correct precisely because minute->hour rotation DOES clear the minute buckets via 60 rotate_minute calls.
|
||||
|
||||
**Why it matters.** read_windowed_count and read_velocity for the 30d window roughly double (then drift) for any entity active around a day boundary. The 30d window directly feeds trending/velocity ranking, so candidate scores for the most-active items are systematically inflated by a derived-state bug — exactly the 3am production incident the project rules warn about. It also violates the documented invariant 'windowed aggregates equal the sum of events within the window' (CODING_GUIDELINES §3/§8). Existing tests (thirty_day_window_aggregates_across_days asserts only >=9; events_outside_30d_window_not_counted asserts only <=31) use one event per day and never exercise a dense in-progress day across a boundary, so the bug ships untested.
|
||||
|
||||
**Fix.**
|
||||
|
||||
The day tier must not re-count what the hour tier still holds. Two correct options: (1) After rolling day_agg into a day bucket in rotate_days, zero the 24 hour buckets that fed it (mirror the minute-buckets-cleared-on-hour-rotation symmetry) so sum_current_day no longer re-reads them; or (2) redefine the 30d read to NOT add sum_current_day on top of the day ring when those hours have already cascaded — e.g. make day bucket [current_day] the live in-progress day fed incrementally and sum exactly DAY_BUCKETS-1 completed days plus the in-progress day once. Option (1) is the smaller change and keeps the cascade model consistent across tiers. Add a regression test that records a dense in-progress day (e.g. 5 events/hour for 23h), crosses the day boundary with one more event, and asserts windowed_count(ThirtyDays) == all_time_count() (e.g. 116, not 216).
|
||||
|
||||
> _Verifier (97): Confirmed by reading the code and by a real test: rotate_day/rotate_days (warm.rs:304-310, 484-505) roll the 24h hour-tier aggregate into a new day bucket but never clear the hour buckets (rotate_hour only overwrites the next slot), so windowed_count(ThirtyDays) at line 213 counts the same events twice — empirically 122 real events reported as 238 (sum_current_day=118 + sum_last_n_days30=120), drifting from ~2x down over ~24h after each day boundary; this 30d count feeds ranking via read_windowed_count/read_velocity (ranking/executor/helpers.rs:74,77, builtins.rs:167) and violates the CODING_GUIDELINES.md:226 invariant "windowed aggregates equal the sum of events within the window," and existing tests (one event/day, loose >=9 / <=31 bounds) never exercise it._
|
||||
|
||||
---
|
||||
|
||||
### 2. [BLOCKER] checkpoint() never deletes stale signal keys; after a trim the integrity hash mismatches and restore silently drops all signal state
|
||||
**Slice:** `signals-ledger-checkpoint` **Dimension:** Accuracy **Location:** `tidal/src/signals/checkpoint/mod.rs:68-125 (checkpoint), 142-208 (restore)` **Confidence:** 90
|
||||
|
||||
**Issue.** checkpoint() builds a WriteBatch of ONLY put() ops for the entries currently live in the DashMap, plus the meta key. It never deletes storage keys for entities present in a previous checkpoint but no longer in the map. The LRU trimmer (signals/trimmer.rs, invoked every checkpoint cycle in db/state_rebuild.rs once the DashMap exceeds DEFAULT_MAX_SIGNAL_ENTRIES) removes entries from the DashMap but does NOT delete their storage keys. So after any trim-then-checkpoint, storage retains orphaned Tag::Sig rows. On restore() the scan at mod.rs:174-183 collects EVERY non-meta Tag::Sig value (including orphans) into raw_values, but the payload_hash in meta was computed in checkpoint() only over the live entries. verify_checkpoint_payload therefore fails, restore() returns Ok(None) (mod.rs:186-194), and the system falls back to full WAL replay.
|
||||
|
||||
**Why it matters.** This is a durability/data-loss hole, not just a perf fallback. The periodic checkpoint path runs compact_wal_online(dir, seq) immediately after a successful checkpoint (state_rebuild.rs:524), deleting WAL segments whose events are covered by the checkpoint. When the next restart's restore() fails the integrity check and falls back to WAL replay, the pre-checkpoint events needed to rebuild those aggregates have already been compacted away. Net: on the first restart after the ledger has ever exceeded the trim threshold (the entire reason the trimmer exists — 5M+ entries), ALL signal aggregate state reconstructable only from the checkpoint is silently lost and the ledger comes back effectively empty. Ranking quality collapses with no error surfaced. This is exactly the 3am-incident class the guidelines forbid: derived state that is NOT rebuildable from the WAL after a documented, expected operation.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Make the checkpoint authoritative over the Tag::Sig keyspace so the on-disk set always equals the in-memory set. In checkpoint(), after building/sorting kv_pairs, enumerate existing Tag::Sig keys (excluding meta) and emit batch.delete() for any not in the live set, in the SAME atomic WriteBatch: let live: HashSet<&[u8]> = kv_pairs.iter().map(|(k,_)| k.as_slice()).collect(); for item in storage.scan_prefix(&[]) { let (k,_) = item?; if let Some((eid, Tag::Sig, suf)) = parse_key(&k) { if !(eid == EntityId::new(0) && suf == META_SUFFIX) && !live.contains(k.as_slice()) { batch.delete(k); } } } then add the puts. Because put+delete are one atomic write_batch, restore() scans exactly the live set and the hash matches. Add a regression test: record N entities, checkpoint, evict some via trim_cold_entries, checkpoint again, restore into a fresh ledger, assert Some(meta) and the surviving entry_count (not None).
|
||||
|
||||
> _Verifier (90): Confirmed all four links: trim_cold_entries (trimmer.rs:97) only removes from the in-memory DashMap, checkpoint() (mod.rs:98-100) emits only put() ops for live entries plus the meta key with no delete() for evicted rows, both write_batch impls apply only the batch's explicit ops (memory.rs:97-111, fjall.rs:155+), so orphaned Tag::Sig rows persist; restore() then scans EVERY non-meta Tag::Sig value including orphans into raw_values (mod.rs:174-183) while the hash covered only live entries, so verify_checkpoint_payload mismatches and returns Ok(None), which open.rs:173 treats as "no checkpoint" with no warning logged and an empty DashMap, and compact_wal_online (state_rebuild.rs:524) has already deleted the pre-checkpoint WAL segments (first_seq < seq, compaction.rs:122-134) so replay cannot rebuild them — a real silent total loss of signal aggregate state on the first restart after the ledger has ever exceeded the 5M trim threshold._
|
||||
|
||||
---
|
||||
|
||||
### 3. [CRITICAL] Full community-ledger checkpoint on every accepted contribution is O(total contributions) per write
|
||||
**Slice:** `db-entity-ops` **Dimension:** CLEAN **Location:** `tidal/src/db/communities.rs:309-319` **Confidence:** 90
|
||||
|
||||
**Issue.** community_contribute calls self.community_ledger.checkpoint(storage.items_engine()) synchronously for every accepted Ok(true). checkpoint() (community_ledger.rs:407) scan_prefix's the ENTIRE Tag::Community keyspace to stage a delete of every existing row, then re-serializes EVERY in-memory contributor cell across ALL communities into one WriteBatch, then commits. So a single signal_for_community write does work proportional to the total number of community contributions in the database, not to the one cell it touched.
|
||||
|
||||
**Why it matters.** Community contribution is a write path (the doc even calls the gate sub-microsecond). As community contribution volume grows, each new contribution rewrites the whole durable snapshot — quadratic ingestion cost. At 10^5 contributions a single new like restages 10^5 rows under WAL/fsync. This violates the hot-path allocation/efficiency discipline in CODING_GUIDELINES and will fall off the signal-write latency budget under any real community load.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Make the per-accept durability incremental rather than a full snapshot swap. Persist only the one affected contributor cell on the write path (a single put of encode_contributor_suffix(community,user,epoch,entity,type_id,writer) -> serialize_entry), and keep the full checkpoint() for shutdown/periodic compaction (where the all-or-nothing prefix-swap belongs). E.g. add a community_ledger.persist_cell(storage, community, entity, type_id, user, writer, epoch, entry) that does one engine.put, and call THAT here instead of checkpoint(). The zero-loss-window guarantee is preserved because the single accepted cell is durably written before Ok(true); the periodic full checkpoint then garbage-collects purged rows.
|
||||
|
||||
> _Verifier (90): Confirmed in production (non-test) code: communities.rs:309-319 calls community_ledger.checkpoint() synchronously on every accepted contribution, and checkpoint() (community_ledger.rs:407-454) scan_prefix's the entire Tag::Community keyspace and re-serializes every contributor across the global `cells: DashMap<(CommunityId,EntityId,SignalTypeId),CommunityCell>` into one WriteBatch — so each write is O(total community contributions), making ingestion quadratic and violating the hot-path/allocation discipline; the synchronous-durability rationale is documented but justifies durability, not the full-snapshot rewrite (an incremental single-row put would satisfy the same Ok(true)="durably logged" promise), and the finding's only misstep is attributing the "sub-microsecond" rustdoc phrase, which actually describes the gate, not the contribution path._
|
||||
|
||||
---
|
||||
|
||||
### 4. [CRITICAL] Session-journal export reads the entire append-only journal into memory, defeating the export limit (OOM)
|
||||
**Slice:** `db-export-backup-http` **Dimension:** Accuracy **Location:** `tidal/src/db/export.rs:280, 433-523` **Confidence:** 88
|
||||
|
||||
**Issue.** `export_signals` carefully bounds the batch-WAL path with `scan_batch_wal_bounded` (a `limit`-capped heap) precisely to avoid OOM on a large pre-compaction WAL. But the session-journal path, `read_session_journal_signals`, slurps the whole file with `std::fs::read(&journal_path)`, decodes every event, and collects ALL matching events into `results` with NO limit applied — the `effective_limit` truncation only happens after the merge at export.rs:287. The session journal (src/wal/session_journal.rs:66) is a single `create(true).append(true)` file with no rotation, compaction, or size bound, so it grows without limit for the life of the database.
|
||||
|
||||
**Why it matters.** A `limit: Some(10)` export against a database with a multi-gigabyte session journal allocates the entire journal contents plus a fully-materialized `Vec<ExportedSignal>` of every user-attributed signal ever written, then throws all but 10 away. This is the exact OOM class the batch-WAL bounded scan was built to prevent — the fix was applied asymmetrically and the larger, never-compacted file is the one left unbounded. Export is documented as 'must never abort a batch' and runs concurrently with the live writer; an OOM here can take down the whole embedding process at 3am.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Push the limit and filters down into the session-journal scan the same way the batch path does. Stream the file (or at minimum, after building the `session_user` map, retain only the `limit` earliest-timestamp matches via the same bounded `BinaryHeap<HeapEntry>` used in `scan_batch_wal_bounded`) so the working set is capped at `limit` regardless of journal size. Factor the heap-bounded retention into a shared helper that both `scan_batch_wal_bounded` and the session path call, taking a closure that yields `(timestamp, ExportedSignal)`. If true streaming of the session journal isn't yet exposed, at least cap retention: `if heap.len() > limit { heap.pop(); }` after each push, mirroring lines 405-415.
|
||||
|
||||
> _Verifier (88): Confirmed in code: export.rs:447 `read_session_journal_signals` does `std::fs::read(&journal_path)` and collects every matching Signal into an unbounded `results` Vec (lines 470-520) with the `effective_limit` truncation applied only after the merge at line 287, while the batch-WAL path uses the deliberately limit-capped `scan_batch_wal_bounded` (rustdoc at 298-318 explicitly documents the anti-OOM intent); session_journal.rs:66 is a single `create(true).append(true)` file with no rotation/compaction/truncation anywhere in the tree (Close events are merely appended, and unlike the batch WAL's `compact_wal_online`, the journal grows for the life of the DB), so a small-limit export against a large accumulated journal allocates the whole file plus a fully-materialized signal Vec — a real, reachable OOM on a public, concurrency-safe export path, gated on multi-GB journal accumulation rather than instant certain crash, which keeps it CRITICAL rather than BLOCKER._
|
||||
|
||||
---
|
||||
|
||||
### 5. [CRITICAL] evict_old_days has zero callers — notification maps grow unbounded forever
|
||||
**Slice:** `db-signals-sessions` **Dimension:** Completeness **Location:** `tidal/src/db/notification_tracker.rs:171` **Confidence:** 90
|
||||
|
||||
**Issue.** The type doc (lines 19-24) says "call evict_old_days periodically (e.g., on startup or daily) to remove stale entries. Without eviction the maps grow by O(users × creators) per day of operation." A grep across the whole crate (excluding this file's own tests) finds no caller of evict_old_days — not in open.rs, not in the sweeper, not on startup, not on a timer. The DashMaps `delivered` and `user_total` therefore accumulate a new key per (user, creator, day) and (user, day) for every day the process runs and are never trimmed.
|
||||
|
||||
**Why it matters.** For a long-lived notification-serving process this is a steady, unbounded memory leak proportional to active-users × creators × days-of-uptime. It exactly matches the project rule 'every entity has a lifecycle' and 'design for recovery' — here the lifecycle's cleanup transition is defined but never invoked, so a server that stays up for months OOMs. The mitigation exists and is tested; it is simply not wired.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Wire eviction on a cadence the existing sweeper already provides. The DB has a background sweeper (src/db/sweeper.rs); add a daily tick that calls self.notification_tracker.evict_old_days(today, keep_days) with keep_days from config (e.g. 1–7), and also call it once during open() so a restart trims stale days. Concretely, in the sweeper loop compute `let today = (Timestamp::now().as_nanos() / 86_400_000_000_000) as u32;` and invoke `self.notification_tracker.evict_old_days(today, self.config.notification_keep_days);`. Without a caller the method and its doc are a false promise.
|
||||
|
||||
> _Verifier (90): Verified: grep across the whole workspace shows evict_old_days (notification_tracker.rs:171) has zero non-test callers, while check_and_record at helpers.rs:276 inserts a new (user,creator,day)/(user,day) key on every notification-profile RETRIEVE, and the only background thread (sweeper.rs:100) calls sweep_expired_sessions and never the tracker — so the DashMaps grow unbounded per day exactly as the type's own doc (lines 19-23) warns, a real but slow long-uptime leak warranting CRITICAL not BLOCKER._
|
||||
|
||||
---
|
||||
|
||||
### 6. [CRITICAL] Empty writer_agent in RemoveScope::CommunityAgent purges ALL direct user contributions
|
||||
**Slice:** `governance` **Dimension:** Accuracy **Location:** `tidal/src/governance/community_ledger.rs:315-317 (purge_writer); reached via /Users/jordan.washburn/Workspace/orchard9/tidaldb/tidal/src/db/remove_scope.rs:60-61` **Confidence:** 88
|
||||
|
||||
**Issue.** purge_writer matches contributors whose writer tag equals writer_agent. A direct (non-agent) user write is stored with writer == "" (WriterTag empty string / WRITER_DIRECT). RemoveScope::CommunityAgent { writer_agent, community } is a public API (RemoveScope is re-exported from lib.rs:61, remove_from_personalization is pub) and passes writer_agent straight into purge_writer with no non-empty validation. An embedder (or a bug upstream) that calls remove_from_personalization with an empty agent id silently removes every direct user contribution to that community and stamps a purge watermark, then writes a durable AgentPurgeTombstone("") that re-purges them on every subsequent open.
|
||||
|
||||
**Why it matters.** This is a silent, durable, replay-persistent data-loss path for the most common contribution class (direct user writes). It is exactly the 3am footgun the durability machinery exists to prevent: 'revoke agent X' with X unset deletes real user data and the tombstone makes it permanent across restart.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Reject an empty writer_agent before mutating. In community_ledger.rs make purge_writer guard the sentinel: `pub fn purge_writer(&self, community: CommunityId, writer_agent: &str) -> usize { if writer_agent.is_empty() { return 0; } self.purge_matching(community, |(_u, writer, _ep)| writer == writer_agent) }` — and, defensively, validate at the API boundary in remove_scope.rs CommunityAgent arm: `if writer_agent.is_empty() { return Err(TidalError::invalid_input("writer_agent must be non-empty; empty matches direct user writes")); }` before persisting the tombstone. Add a test asserting purge_writer(c, "") removes 0 and leaves direct writes intact.
|
||||
|
||||
> _Verifier (88): Confirmed end-to-end: direct user writes store writer_agent="" (communities.rs:207-215, the sole direct-write path; agent writes use a real agent_id at capabilities.rs:173), purge_writer matches writer==writer_agent (community_ledger.rs:316), and the public, freely-constructible RemoveScope::CommunityAgent{writer_agent:String} (lib.rs:61) flows straight into purge_writer with no is_empty guard (remove_scope.rs:60-61) plus a durable AgentPurgeTombstone("") that re-purges all direct contributions on every open (community_ledger.rs:525 via db/mod.rs:609), so an empty agent id silently and permanently deletes all direct user contributions — real and durable, but a public-API misuse footgun (requires passing "") rather than a happy-path corruption, so CRITICAL not BLOCKER._
|
||||
|
||||
---
|
||||
|
||||
### 7. [CRITICAL] Synchronous full-snapshot community checkpoint on every accepted contribution is O(N^2) ingest
|
||||
**Slice:** `governance` **Dimension:** CLEAN **Location:** `tidal/src/governance/community_ledger.rs:407-455 (checkpoint), driven per-write from /Users/jordan.washburn/Workspace/orchard9/tidaldb/tidal/src/db/communities.rs:309-318` **Confidence:** 90
|
||||
|
||||
**Issue.** checkpoint() scans the entire Tag::Community prefix, stages a delete for every existing durable row, then stages a put for every in-memory contributor cell, and commits one WriteBatch. signal_for_community calls this synchronously on the write path for EVERY accepted community contribution (communities.rs:310). For a community that has accumulated M contributor cells, the M-th write does O(M) deletes + O(M) puts + an O(M) durable scan, so ingesting N contributions is O(N^2) total work and O(N) write amplification per signal. The CODING_GUIDELINES signal-write budget is <100us; a community with tens of thousands of cells blows it by orders of magnitude.
|
||||
|
||||
**Why it matters.** Community write throughput collapses quadratically as a community grows — the larger and more active a community, the slower each new contribution, which is the opposite of the desired scaling and a latent production cliff. It also rewrites the whole durable snapshot on every write, multiplying I/O and write-amplification.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Make per-contribution durability incremental instead of a full rewrite: on an accepted contribution put only that single (community,user,epoch,entity,signal_type,writer) row via encode_contributor_suffix, and on a purge delete only the affected rows. Keep the full scan-delete-rewrite snapshot for the lifecycle/periodic/backup checkpoint path (backup.rs, state_rebuild.rs) where an all-or-nothing swap is wanted. Alternatively, debounce: keep an in-memory dirty-set and flush a batched delta on a periodic tick, accepting a bounded loss window like the cohort ledger rather than a synchronous full snapshot per write. Benchmark community ingest before/after per CODING_GUIDELINES §8.
|
||||
|
||||
> _Verifier (90): Confirmed in source: communities.rs:309-318 synchronously calls community_ledger.checkpoint() on every accepted contribution (production path, not test), and checkpoint() (community_ledger.rs:418-453) scans the full Tag::Community prefix staging a delete per durable row then a put per in-memory contributor cell across all communities into one WriteBatch — so the M-th write is O(M) and N writes are O(N^2), blowing the CODING_GUIDELINES.md:246 "<100 microseconds" signal-write budget, and the rustdoc only justifies atomicity, never the quadratic cost._
|
||||
|
||||
---
|
||||
|
||||
### 8. [CRITICAL] Unchecked `id as u32` truncation in trending scope bitmap aliases high entity IDs in release builds
|
||||
**Slice:** `query-retrieve-search` **Dimension:** Accuracy **Location:** `tidal/src/query/search/scope.rs:322-330` **Confidence:** 88
|
||||
|
||||
**Issue.** percentile_bitmap inserts entity IDs into a RoaringBitmap via `bitmap.insert(id as u32)`, guarded only by `debug_assert!(u32::try_from(id).is_ok(), ...)`. debug_assert is compiled out in release. An entity id above u32::MAX (4.29e9) is silently truncated, so a high id wraps onto a low id's slot in the scope bitmap.
|
||||
|
||||
**Why it matters.** The scope bitmap is the Stage 0 pre-filter applied to BOTH BM25 and ANN retrieval (WithinScope::Trending / CohortTrending). A truncated/aliased id makes the trending scope admit an item that is NOT trending (whatever low id the high id aliased onto) and silently drop the real high-id item — a wrong-results leak with no diagnostic. Every other id->u32 conversion in this exact slice (entity_as_u32, retain_excluded, retrieve_ann's filtered_search predicate) uses checked `u32::try_from`; this is the one release-relevant exception, so the codebase already has the right pattern to copy.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Drop the debug_assert and skip non-representable ids explicitly, matching entity_as_u32 elsewhere in the slice:
|
||||
```rust
|
||||
for &(id, vel) in &sorted {
|
||||
if vel >= cutoff_velocity && vel > 0 {
|
||||
match u32::try_from(id) {
|
||||
Ok(k) => { bitmap.insert(k); }
|
||||
Err(_) => {
|
||||
tracing::warn!(entity_id = id, "trending scope: id exceeds u32 bitmap key space; skipped");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Skipping (rather than truncating) is the correct polarity for an inclusion bitmap: a >u32 id cannot be a member, so excluding it is sound, and it can never alias onto another item.
|
||||
|
||||
> _Verifier (88): In scope.rs:329 the producer `percentile_bitmap` inserts `id as u32` (truncating, debug_assert-only) into the trending scope bitmap, while both consumers — BM25 retain (pipeline.rs:132) and the ANN filtered_search predicate (pipeline.rs:396) — use checked `u32::try_from` via `entity_as_u32`, whose own doc comment warns that `id as u32` "would alias a high id onto a low one"; since `EntityId` wraps an arbitrary caller-supplied u64 with no u32 cap, an entity ID above u32::MAX is silently aliased onto a low slot in release, admitting a non-trending item and dropping the real one with no diagnostic._
|
||||
|
||||
---
|
||||
|
||||
### 9. [CRITICAL] Hlc::update() does not carry logical-counter overflow, so it can return a non-unique timestamp
|
||||
**Slice:** `replication-crdt` **Dimension:** Accuracy **Location:** `tidal/src/replication/crdt/hlc.rs:288-301` **Confidence:** 78
|
||||
|
||||
**Issue.** When the wall clock has not advanced (`pt == cur_wall`) and the logical counter is already saturated (`cur_logical == LOGICAL_MAX`), `new_logical = (LOGICAL_MAX + 1).min(LOGICAL_MAX) = LOGICAL_MAX`, so `new_packed == cur`. The CAS succeeds against the unchanged word and `update()` returns a timestamp byte-identical to the current state. `now()` explicitly handles this exact regime by carrying the overflow into the wall component (lines 226-231, with a warn!), but `update()` silently saturates. Two concurrent `update()` calls in this regime, or an `update()` followed by an equal-keyed event, can produce duplicate timestamps from the same node.
|
||||
|
||||
**Why it matters.** The module's central guarantee (doc lines 150-154, 271-272) is that every timestamp a node emits is strictly monotonic and unique within the node, because LWW conflict resolution depends on it. `update()` claims in its own docstring to return a value 'strictly > remote' and to make subsequent reads strictly greater, but in the saturated-counter regime it returns a value equal to the prior state. This is the same practically-unreachable (65536 events/ms) regime the authors deemed worth fixing in `now()` — leaving `update()` asymmetric is a latent uniqueness hole that contradicts both the module doc and `now()`'s own carefully-built invariant.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Mirror the `now()` carry in `update()`: after computing `pt` and the candidate `new_logical`, detect saturation when the wall did not advance and carry into the wall component instead of pinning logical. e.g. replace the saturating `.min(LOGICAL_MAX)` cascade with a check: `if pt == cur_wall && new_logical_raw > LOGICAL_MAX { new_wall = cur_wall.saturating_add(1); new_logical = 0; emit the same warn! }` — reusing the exact overflow-handling branch `now()` already has, so the two methods share one rule and one tracing signal.
|
||||
|
||||
> _Verifier (78): Confirmed at hlc.rs:294/303-314: when pt==cur_wall and cur_logical==LOGICAL_MAX, `((cur_logical+1).min(LOGICAL_MAX))` returns LOGICAL_MAX unchanged, so new_packed==cur, the CAS(cur,cur) succeeds, and update() returns a byte-identical timestamp — whereas now() (lines 226-231) carries the same overflow into the wall component with a warn! and has a guarding test (now_strictly_advances_when_logical_saturated) that update() lacks, so update() silently violates the module's per-node uniqueness invariant (lines 152-154) and its own docstring promise (lines 268-272); the regime is astronomically rare (65,536 events/ms) but is the exact case the authors deemed worth fixing in now(), making the asymmetry a genuine latent uniqueness hole rather than test code, an unreachable path, or a documented design choice._
|
||||
|
||||
---
|
||||
|
||||
### 10. [CRITICAL] from_node_contribution score/last_update_ns contract is ambiguous and the production caller double-decays
|
||||
**Slice:** `replication-crdt` **Dimension:** Accuracy **Location:** `tidal/src/replication/crdt/signal_state.rs:98-112` **Confidence:** 90
|
||||
|
||||
**Issue.** `from_node_contribution(node, score, last_update_ns, ...)` stores `score` against `last_update_ns` and `decay_score(query_ns)` later applies `exp(-lambda*(query_ns - last_update_ns))` to it. The docstring describes `score` only as 'the current node's running decay score', without stating whether it must be the *stored accumulator at last_update_ns* (so re-decay is correct) or a value already decayed to some query time (so re-decay is wrong). The production caller `SignalLedger::crdt_contributions` (signals/ledger/core.rs:550-553) passes `score = hot.current_score(0, now_ns, lambda)` — the score decayed forward to `now_ns` — together with `last_update_ns = hot.last_update_ns()`, the original (earlier) event time. `decay_score(query_ns)` then decays that already-decayed score again across `[last_update_ns, query_ns]`, re-applying the `[last_update_ns, now_ns]` interval a second time.
|
||||
|
||||
**Why it matters.** Post-partition reconciliation is the whole reason this slice exists, and the merged signal score directly drives ranking. A systematic double-decay makes every reconciled contribution monotonically too small, so a viral item's merged engagement score is under-counted after a partition heals — silently degrading exactly the rankings the database is built to produce, with no error surfaced. The unit tests pass only because every test calls `from_node_contribution` with `last_update_ns == query_ns` (e.g. reconcile_tests.rs:546, signal_state.rs:321-327), which masks the production mismatch.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Make the contract explicit and internally consistent. Document that `score` MUST be the stored accumulator value as of `last_update_ns` (NOT decayed to any later time), then fix the caller to pass `hot.stored_score(0)` (which exists, hot.rs:225) with the matching `last_update_ns`, letting `decay_score` apply the one and only decay. Alternatively, if the caller intends to snapshot at `now_ns`, pass `last_update_ns = now_ns` so the stored decayed value is consistent with its timestamp. Add a regression test where `now_ns > last_update_ns` and assert the reconciled `decay_score` equals the single-decay analytical value, so the round-trip can no longer silently double-decay.
|
||||
|
||||
> _Verifier (90): Confirmed in production code: crdt_contributions (core.rs:550-551) builds the contribution with score = current_score(0, now_ns, lambda) = stored*exp(-lambda*(now_ns-last_ns)) but pairs it with last_update_ns = the earlier hot.last_update_ns(), and the live reconcile-apply path apply_crdt_state (core.rs:474) calls state.decay_score(now_ns) which re-applies exp(-lambda*(now_ns-last_ns)) before force_set_score, so every reconciled contribution is silently decayed twice; the masking is also real (the m8_uat CRDT sub-test builds states via on_signal(...,now_ns) so last_update_ns==query_ns and the cluster check uses epsilon=1.0, while the m8p2 replication test exercises the WAL path, not this CRDT snapshot path)._
|
||||
|
||||
---
|
||||
|
||||
### 11. [CRITICAL] Shutdown-drain multi-chunk flush drops remaining callers' replies on a write error
|
||||
**Slice:** `wal-writer-reader` **Dimension:** Accuracy **Location:** `tidal/src/wal/writer.rs:551-558` **Confidence:** 88
|
||||
|
||||
**Issue.** In the shutdown drain, a drained batch larger than max_batch is flushed in chunks with `next_seq = flush_batch(...)?`. flush_batch notifies only the CURRENT chunk's callers before returning Err. The `?` then propagates immediately, so the `events`/`replies` iterators for every NOT-YET-PROCESSED chunk are dropped without sending anything. Those callers' reply channels close and they observe `WalError::SendFailed`/`Closed` instead of the real write error.
|
||||
|
||||
**Why it matters.** The module-level doc on QueuedAppend states it is 'critical that every queued append eventually resolves its reply' and that 'both the steady-state loop and the shutdown drain funnel through the same flush_batch routine' precisely to avoid dropping channels on error. This multi-chunk path violates that invariant. A caller blocked in WalSender::append on shutdown under a real I/O fault gets a misleading Closed, hiding the actual ENOSPC/EIO cause — exactly the 3am-incident confusion the consolidation was built to prevent. It only triggers when >256 appends queue during shutdown AND a write fault occurs, hence CRITICAL not BLOCKER.
|
||||
|
||||
**Fix.**
|
||||
|
||||
Do not early-return on chunk failure during the drain; notify the remaining chunks' callers before propagating. e.g. collect the error and drain the rest:
|
||||
```rust
|
||||
let mut drain_err = None;
|
||||
loop {
|
||||
let chunk_events: Vec<EventRecord> = events.by_ref().take(max_batch).collect();
|
||||
if chunk_events.is_empty() { break; }
|
||||
let chunk_replies: Vec<_> = replies.by_ref().take(chunk_events.len()).collect();
|
||||
if drain_err.is_some() {
|
||||
// a prior chunk already failed; still notify these callers with an error
|
||||
for r in chunk_replies { let _ = r.send(Err(WalError::Closed)); }
|
||||
continue;
|
||||
}
|
||||
match flush_batch(&mut segment, config, next_seq, &chunk_events, chunk_replies) {
|
||||
Ok(seq) => next_seq = seq,
|
||||
Err(e) => drain_err = Some(e),
|
||||
}
|
||||
}
|
||||
if let Some(e) = drain_err { return Err(e); }
|
||||
```
|
||||
This keeps every queued append resolved (each chunk's callers were notified by flush_batch or the fallback) while still surfacing the error from run_writer.
|
||||
|
||||
> _Verifier (88): Confirmed at writer.rs:557 — `next_seq = flush_batch(...)?` propagates immediately while flush_batch (lines 244-254) notifies only the current chunk's replies, so the `replies` iterator still holding senders for not-yet-flushed chunks is dropped on `?`, surfacing as `Closed` instead of the real I/O error; this directly contradicts the module doc (lines 22-25) requiring every queued append to resolve its reply and diverges from the steady-state loop (lines 495-513) which logs-and-continues rather than propagating, but it requires the compound trigger of >max_batch (256) queued appends during shutdown plus a write fault on a non-final chunk and yields a misleading-error (not data-loss/hang) symptom, so CRITICAL is the defensible ceiling._
|
||||
|
||||
---
|
||||
|
||||
## WARNINGS
|
||||
|
||||
| # | Slice | Location | Dim | Issue |
|
||||
|---|---|---|---|---|
|
||||
| 1 | db-core-lifecycle | `tidal/src/db/builder.rs:171-183, 280-290; open.rs:153-156` | Completeness | wal_dir / cache_dir builder overrides are validated then silently ignored |
|
||||
| 2 | db-core-lifecycle | `tidal/src/db/paths.rs:16-23, 60-105` | CLEAN | Paths::items_dir/users_dir/creators_dir/ensure_all describe a layout the engine never creates |
|
||||
| 3 | db-core-lifecycle | `tidal/src/db/builder.rs:298-299` | Accuracy | open() doc claims storage 'will create them via Paths::ensure_all during initialization' — it never does |
|
||||
| 4 | db-core-lifecycle | `tidal/src/db/lifecycle.rs:218-223` | Accuracy | Poisoned WAL mutex in shutdown_inner discards an already-captured first_err |
|
||||
| 5 | db-entity-ops | `tidal/src/db/collections.rs:59, 95` | Accuracy | Collection item IDs above u32::MAX silently alias instead of being rejected |
|
||||
| 6 | db-entity-ops | `tidal/src/db/collections.rs:60-74` | Accuracy | add_to_collection mutates in-memory index before persisting; persist failure leaves memory ahead of disk |
|
||||
| 7 | db-entity-ops | `tidal/src/db/communities.rs:108-125` | Accuracy | leave_community engages the live stop-forward gate before persisting; persist failure leaves an un-durable gate |
|
||||
| 8 | db-signals-sessions | `tidal/src/db/notification_tracker.rs:63` | DRY | Dead public surface: record / would_exceed_per_creator / would_exceed_total are only called from tests |
|
||||
| 9 | db-signals-sessions | `tidal/src/db/signals.rs:297` | Completeness | signal_with_context '# Durability' docstring omits two side-effect indexes it mutates |
|
||||
| 10 | db-signals-sessions | `tidal/src/db/signals.rs:93` | DRY | Backup-guard + WAL backpressure preamble duplicated verbatim between signal and signal_scoped |
|
||||
| 11 | db-recovery-rebuild | `tidal/src/db/sweeper.rs:118` | Accuracy | start_sweeper panics on thread-spawn failure instead of degrading |
|
||||
| 12 | db-export-backup-http | `tidal/src/db/export.rs:11-14, 30, 127-143, 185` | Completeness | `ExportFormat::JsonLines` and `to_json_line` are dead-ended — `export_signals` returns structs, never the wire format |
|
||||
| 13 | db-metrics | `tidal/src/db/metrics/mod.rs:292-298, 470` | Extensibility | partition_id="0" hardcoded into every metric line — wrong in cluster mode |
|
||||
| 14 | wal-writer-reader | `tidal/src/wal/checkpoint.rs:28-49` | Accuracy | Checkpoint temp file leaks on crash between write and rename; no recovery cleanup |
|
||||
| 15 | wal-writer-reader | `tidal/src/wal/reader.rs:181-186` | Tech Debt | Reader hardcodes the payload_len byte offset instead of using the format module's owned layout |
|
||||
| 16 | wal-format-recovery | `tidal/src/wal/format/session.rs:324-328` | Accuracy | v2 frame with a known type but short body is misreported as on-disk corruption |
|
||||
| 17 | signals-decay-hotwarm | `tidal/src/signals/warm.rs:206, 525, 454-459` | DRY | 'last 24h = in-progress hour + 23 hour buckets' is expressed three times |
|
||||
| 18 | signals-decay-hotwarm | `tidal/src/signals/warm.rs:388-412` | Accuracy | Concurrent rotation at an hour boundary can lose minute data from the hour rollup |
|
||||
| 19 | signals-ledger-checkpoint | `tidal/src/signals/checkpoint/meta.rs:43-51, 64-124` | Accuracy | Integrity hash covers entry payloads but NOT the meta record (wal_sequence / checkpoint_time_ns) |
|
||||
| 20 | signals-ledger-checkpoint | `tidal/src/signals/ledger/core.rs:207-260` | CLEAN | read_decay_score / read_windowed_count read Timestamp::now() internally — no consistent query clock, repeated syscalls per candidate |
|
||||
| 21 | storage-core | `tidal/src/storage/engine.rs:38-45` | Completeness | Trait contract for write_batch promises atomicity but is silent on durability |
|
||||
| 22 | storage-indexes | `tidal/src/storage/indexes/range.rs:66-81` | Completeness | RangeIndex has no delete_entity; re-indexed items leave stale range entries |
|
||||
| 23 | storage-vector | `tidal/src/storage/vector/lifecycle/ops.rs:138-165` | Accuracy | update_embedding leaves entity store and ANN index divergent on partial failure with no recovery |
|
||||
| 24 | storage-vector | `tidal/src/storage/vector/lifecycle/ops.rs:188-218` | Accuracy | Soft delete: ANN tombstone applied before durable archive tombstone is written (crash window resurrects the vector) |
|
||||
| 25 | storage-vector | `tidal/src/storage/vector/registry.rs:300-352` | CLEAN | rebuild_from_store does two full O(N) store scans with a per-key String allocation on every embedding key |
|
||||
| 26 | query-executor | `tidal/src/query/executor/helpers.rs:166-198` | Accuracy | Cohort rescore swallows the unknown-signal schema error the main path raises loudly |
|
||||
| 27 | query-executor | `tidal/src/query/executor/user_filter.rs:99-120, 137-158, 242-256, 272-286` | DRY | Four near-identical filter-collector walkers duplicate a load-bearing traversal rule |
|
||||
| 28 | query-retrieve-search | `tidal/src/query/executor/pipeline.rs:250-276` | Completeness | RETRIEVE Stage 2 did not receive the empty/poisoned-universe deferred-filter fix that SEARCH got |
|
||||
| 29 | query-retrieve-search | `tidal/src/query/search/executor_helpers.rs:104-121` | Accuracy | Creator metadata filter matches a value against ANY metadata field, not the intended field |
|
||||
| 30 | ranking | `tidal/src/ranking/executor/mod.rs:617-619` | CLEAN | Alphabetical tie-break allocates two Strings per equal-score comparison |
|
||||
| 31 | ranking | `tidal/src/ranking/executor/mod.rs:393-408` | CLEAN | session_boost re-lowercases all session keywords for every candidate |
|
||||
| 32 | entities | `tidal/src/entities/mod.rs:67-69` | CLEAN | Hot-path callers clone whole RoaringBitmap via get() when get_ref() exists to avoid it |
|
||||
| 33 | entities | `tidal/src/entities/co_engagement.rs:84-90` | Accuracy | Co-engagement edge double-counted when an item recurs in the recent queue |
|
||||
| 34 | entities | `tidal/src/entities/collection.rs:136-154` | Tech Debt | CollectionIndex tracks membership in both a bitmap and a Vec with O(n) per-mutation scans |
|
||||
| 35 | governance | `tidal/src/governance/community_ledger.rs:235-255 (windowed_count)` | Accuracy | windowed_count reads its own wall clock while every sibling read takes now_ns |
|
||||
| 36 | governance | `tidal/src/governance/capability.rs:366-373` | Completeness | Unticketed TODO(rehydrate) violates the no-orphan-TODO rule |
|
||||
| 37 | replication-core | `tidal/src/replication/session_bridge.rs:313-360` | Accuracy | recv_and_apply: Layer-1 seqno HWM advances even when Layer-2 idempotency subsequently drops the event |
|
||||
| 38 | replication-core | `tidal/src/replication/migration.rs:119-137` | Completeness | enter_dual_write mutates router routing before persisting; a crash in the gap loses an in-flight migration's dual-write state |
|
||||
| 39 | schema | `tidal/src/schema/validation/builders.rs:255-285` | Completeness | AgentPolicy numeric fields are never validated at build(); Duration::ZERO silently expires every session |
|
||||
| 40 | session | `tidal/src/session/saved_search.rs:107, 120, 141` | Accuracy | saved_search silently masks corruption with from_utf8_lossy while the rest of the slice rejects it |
|
||||
| 41 | text | `tidal/src/db/text_syncer.rs:186` | Completeness | Unbounded text-syncer channel grows without bound during a sustained commit outage |
|
||||
| 42 | cohort-load-testing | `tidal/src/cohort/ledger.rs:128-148` | Accuracy | Cohort read_decay_score returns an undecayed score for an out-of-range decay index instead of None |
|
||||
| 43 | cohort-load-testing | `tidal/src/db/builder.rs:114-117` | Completeness | Builder accepts a rate-limiter config but never validates it; a degenerate config silently degrades to unlimited |
|
||||
| 44 | cohort-load-testing | `tidal/src/testing/cluster.rs:513-524` | Accuracy | await_convergence calls blocking redeliver_missed while holding the batch_log lock |
|
||||
| 45 | tidal-net | `tidal-net/src/client.rs:34-66` | Accuracy | Client builds a plaintext channel when insecure=false and tls=None, instead of erroring |
|
||||
| 46 | tidal-net | `tidal-net/src/config.rs:70-90` | Completeness | No validation of GrpcTransportConfig invariants (zero capacity/timeouts, payload ceiling) |
|
||||
| 47 | tidal-server-cluster | `tidal-server/src/scatter_gather.rs:640-708 vs 768-834` | DRY | scatter_gather_retrieve and scatter_gather_search duplicate the entire merge+assemble tail |
|
||||
| 48 | tidal-server-core | `tidal-server/src/main.rs:264-280 (serve)` | Completeness | Standalone shutdown never observes final-flush durability failure |
|
||||
| 49 | tidalctl | `tidalctl/src/commands/diagnostics.rs:146-167, 206-214` | DRY | Offline-unavailable field set maintained in two unsynchronized places |
|
||||
| 50 | tidalctl | `tidalctl/src/commands/paths.rs:34-48` | DRY | Six-directory path projection duplicated between paths.rs and status.rs |
|
||||
|
||||
|
||||
## SUGGESTIONS (by slice)
|
||||
|
||||
- **db-core-lifecycle** (2): Six near-identical Tag-prefix restore-scan loops in from_parts; fs4 try_lock_exclusive error is mapped unconditionally to DataDirLocked
|
||||
- **db-entity-ops** (2): flush_text_index discards channel send/recv results; u32 item-id narrowing policy is implemented three different ways
|
||||
- **db-signals-sessions** (3): item_embedding_slot and creator_embedding_slot are the same resolver parameterized by EntityKind; next_session_id advance uses unguarded session_id + 1; duration_ms derived from Instant::elapsed() on a restored session is wall-clock-since-restore, not session age
|
||||
- **db-recovery-rebuild** (4): Four duplicated items/creators scan loops; items keyspace scanned twice at open; Fingerprint lacks length-prefix/domain-separation; collision-safe only by coincidence; run_checkpoint_thread mixes timing, trimming, checkpointing, compaction, and metrics; Recovery/checkpoint cadence constants are scattered inline literals
|
||||
- **db-export-backup-http** (4): No validation that `since <= until` in `ExportRequest`; since-inclusive/until-exclusive window filter duplicated across both scan paths; Backup WAL-marker timestamp uses raw SystemTime while the rest of the method uses `Timestamp::now()`; `extract_path` allocates a Vec just to read the second whitespace token
|
||||
- **db-metrics** (3): 'now in nanos' exists twice with divergent overflow semantics, contradicting the doc claim; Undocumented u128->u64 truncation of latency on the hot path; LatencyHistogram::new does not validate that bounds are sorted ascending
|
||||
- **wal-writer-reader** (2): Active-segment selection can leave segment filename first_seq ahead of next_seq after a checkpoint-heavy reopen; handle_session_command has an unreachable `_ => return` arm that hides intent
|
||||
- **wal-format-recovery** (3): Optional-field `[flag][payload]` framing is hand-rolled three times; Magic byte-offset literal `4 + 3` in corruption test instead of the named frame constants; v3 event reserved bytes [30..32] are written-zero but never asserted-zero on decode
|
||||
- **signals-decay-hotwarm** (1): Decay factor in on_signal uses a single last_ns snapshot for all CAS retries
|
||||
- **signals-ledger-checkpoint** (2): restore() returns CheckpointMeta whose wal_sequence is silently discarded by the sole production caller; V1 and V2 deserialize arms duplicate the checkpoint_time_ns/wal_sequence read+offset-error boilerplate
|
||||
- **storage-core** (3): Tag byte mapping is a hand-maintained parallel table with no drift guard; Repeated keyspace-open block in FjallStorage::open; FjallAtomicBatch publicly re-exported but has no trait abstraction and no production caller
|
||||
- **storage-indexes** (3): delete does a redundant second tree/map lookup after get_mut; Hard-coded 10-byte key-prefix skip duplicated across bitmap and range parsers; evaluate_not complement omits items absent from the universe bitmap (by design, but undocumented edge)
|
||||
- **storage-vector** (3): Unticketed TODO for per-query ef_search override; Duplicated ef_search mismatch-warning block across search and filtered_search; read_*_le helpers documented as panic-on-OOB but are pub(super) used across modules without a debug_assert guard
|
||||
- **query-executor** (2): Notification-cap creator resolution re-reads storage despite pre-loaded metadata; ANN candidate strategy silently falls back to scan with no tracking issue
|
||||
- **query-retrieve-search** (2): User-context suppression block is duplicated between SEARCH and RETRIEVE pipelines; resolve_trending scans the entire global signal ledger per query with no candidate bound
|
||||
- **ranking** (3): resolve_signal_type Result consumed via map_or silently swallows schema errors; score_personalized and score_inner duplicate the candidate-loop skeleton; Rising / MostFollowed / CreatorEngagementRate sort modes have no built-in profile
|
||||
- **entities** (3): Collection item-ID truncation to u32 is silent, unlike the audited narrow_item_slot path; Field-by-field little-endian decode ritual repeated across four entity codecs; top_candidates does a full O(all-edges) scan per call
|
||||
- **governance** (2): contributor_count scans every cell of every community on each call; .expect on infallible 16-byte BLAKE3 truncation in non-test code
|
||||
- **replication-core** (3): IdempotencyStore::new panics on capacity 0 instead of returning a Result; Three near-identical batch-walk loops (filter_segment_drop_local, last_seq_in_segment, count_events_in_segment) share the same decode/advance skeleton; lag_for silently returns the single gauge for any shard in release builds (debug_assert only)
|
||||
- **replication-crdt** (3): LWWRegister::write reimplements the LWW decision instead of routing through lww_other_wins; Undocumented cross-map invariant: node_decay_scores and node_last_update_ns must stay key-paired; node_buckets nests a full PNCounter per node to hold a single key
|
||||
- **schema** (2): Empty embedding-slot name reported as DuplicateEmbeddingSlot is a misleading error; Item and creator text-field validation are two hand-written call sites
|
||||
- **session** (3): saved_search re-implements length-prefixed LE decode instead of reusing the serde cursor helpers; build_frozen_snapshot samples wall-clock twice for one logical close instant; saved_search length fields cast usize->u32 without guarding >4GiB inputs
|
||||
- **text** (3): Failed flush-path commit does not arm retry_not_before, delaying recovery to the timeout path; final_flush sleeps after the last attempt's failure before giving up; read_stats_from_dir conflates 'directory absent' with 'index unreadable'
|
||||
- **cohort-load-testing** (4): parse_cohort_entry_suffix silently accepts trailing bytes after the cohort name; applied_count hardcodes ShardId(0) as 'the initial leader' and silently returns wrong counts after promotion; write_signal ship loop and redeliver_missed duplicate the WalSegmentPayload construction; total_signals stored with Relaxed but read (relay_log_len) with Acquire — mismatched ordering with no paired Release
|
||||
- **tidal-net** (2): expect() on a production code path (runtime() helper); Heartbeat handler ignores the claimed shard/region identity
|
||||
- **tidal-server-cluster** (3): Score sort treats NaN as equal, yielding an unstable/partial order if a score is ever NaN; Backpressure retry_after_ms=50 is a bare magic number; Routing index step bypasses the engine ShardRouter's region mapping, coupling server to hash-mod-len
|
||||
- **tidal-server-core** (6): Config crate hand-rolls enum string tables that drift from the engine's canonical labels; `extract_string` is a pure pass-through to `extract_string_field`; User-supplied `limit` is unbounded at the network trust boundary; Crate doc references a non-existent 'Known Gaps' section; Shutdown-flag atomic ordering differs from ClusterState for the same concept; Two io-error variants with overlapping roles and a misleading name
|
||||
- **tidalctl** (4): as_nanos() as u64 truncates wall-clock for post-2554 timestamps; recover threads a verify_only flag that has exactly one accepted value; scope-stats --pretty only indents JSON; no human-readable mode like recover/diagnostics; format_pretty is a 120-line function building the report inline
|
||||
|
||||
|
||||
## What's Good
|
||||
|
||||
- **db-core-lifecycle** — Shutdown durability is genuinely 3am-grade: shutdown_inner attempts EVERY flush step (signal ledger, cohort, co-engagement, community, preference vectors, replication HWM, storage, WAL marker, compaction), captures the FIRST error, and surfaces it from close() while Drop logs it — the SHUTDOWN-2 reasoning at lines 142-262 of lifecycle.rs is exactly right and the rationale is in-code.
|
||||
- **db-entity-ops** — Durability ordering is reasoned about explicitly and correctly in the strongest paths: write_relationship persists before mutating in-memory state, define_community_policy/define_cohort persist-before-register with a crash-between comment, and purge_prior_contributions writes the tombstone before mutating the aggregate (durability-before-mutation). This is exactly the 3am-incident-grade rigor the project demands.
|
||||
- **db-signals-sessions** — The closed-session cap fix (sessions.rs:208-216, enforce_closed_session_cap) is exactly right: insert-then-trim makes `len() <= MAX` a post-condition instead of a racy pre-condition, the comment explains the concurrent-overshoot bug it replaces, and the low-water-mark batch eviction avoids re-sorting on every close. This is 3am-incident-grade reasoning.
|
||||
- **db-recovery-rebuild** — ItemIndexes (state_rebuild.rs:234) is the standout decision: it forces the live write path and the restart-rebuild path through ONE indexing implementation, with a doc comment that names the exact prior drift (missing creator_items / defaulted created_at) it eliminates. This is correctness-by-construction, not by discipline.
|
||||
- **db-export-backup-http** — The WAL u8 signal-type-ID truncation is handled with exemplary defense-in-depth: export.rs:208-226 surfaces a typed error instead of silently aliasing type #256 onto #0, and the write path (wal_bridge.rs:109) independently `u8::try_from`-guards the same boundary — the comment even documents the prior release-build aliasing bug that motivated it.
|
||||
- **db-metrics** — The histogram observe() comment (lines 54-74) is exemplary: it explains WHY partition_point is used (hot path, O(log n) vs O(n)), acknowledges the inherent O(suffix) increment cost, and is backed by a proptest (observe_matches_naive_cumulative_rule) that checks the optimized path against the naive rule for every value — this is precisely the correctness-first discipline the project demands.
|
||||
- **wal-writer-reader** — The durability rationale is documented at exactly the right depth: SegmentWriter::sync (segment.rs:208-235) explicitly states sync_data covers the data half only and points to the four sites (open, rotate, checkpoint, compaction) that fsync the directory entry — this is the kind of 3am-incident-grade note that prevents someone from 'simplifying' away a load-bearing dir fsync.
|
||||
- **wal-format-recovery** — compact_wal_online (compaction.rs:91-103) closes a genuine 3am silent-data-loss hazard: clamping the deletion floor to min(checkpoint_seq, active_first_seq) so the writer's open FD is never unlinked. The hazard analysis in the doc comment (Linux unlinked-inode appends being lost on next open) is exactly the kind of reasoning that belongs at a durability boundary, and online_never_deletes_active_segment_even_when_fully_checkpointed is the regression test that locks it in.
|
||||
- **signals-decay-hotwarm** — The forward_decay_step kernel (decay.rs) is exactly the right abstraction: a pure, allocation-free, lock-free scalar function extracted as the single source of truth, with a module doc that names the five prior drifted copies and the specific spec violation (session tier added late weight at full value) that motivated consolidation. This is textbook DRY-as-correctness.
|
||||
- **signals-ledger-checkpoint** — The named-offset codec in format.rs (OFF_* derived from prior field sizes, used by BOTH serialize and deserialize) plus the compile-time const-asserts pinning ENTRY_SIZE_V1==983 and ENTRY_SIZE==1116 is exactly the right way to make a binary wire format un-desyncable — a miscount is a build break, not a silent roundtrip corruption.
|
||||
- **storage-core** — map_fjall_err (fjall.rs:69-118) is exactly the kind of error routing you want at 3am: it preserves the io::Error kind/source by moving it into StorageError::Io, separates transient/unavailable (Poisoned/Locked/KeyspaceDeleted -> Closed) from genuine on-disk corruption, and the non_exhaustive wildcard conservatively halts on integrity ambiguity rather than masking a data problem as transient. The reasoning is documented in the doc comment.
|
||||
- **storage-indexes** — Length-prefixed bitmap keys (bitmap.rs:53-61, encode_suffix/parse_suffix_value) are a genuinely correct fix for a real cross-index collision: field 'a:b'+value 'c' vs field 'a'+value 'b:c' both rendered identically under a naive 'BMP:{field}:{value}' scheme. The u16 length prefix removes the ambiguity, the field-name-too-long case returns a typed error instead of corrupting, and a dedicated test (key_format_disambiguates_field_value_boundary) proves cross-load isolation.
|
||||
- **storage-vector** — Brute-force deserialize (brute/mod.rs:296-352) is exemplary hostile-input hardening: count is bounded against the actual buffer length BEFORE any allocation, all arithmetic is checked, the SECURITY comment explains the exact attack (wrap -> tiny expected_size -> exabyte with_capacity -> OOB read), and corruption surfaces as a recoverable CorruptedIndex rather than crashing the recovery path. This is precisely the 3am-incident discipline the project demands.
|
||||
- **query-executor** — The separate-arms treatment of `for_creator` in Stage 1 (pipeline.rs:99-120) is exactly the 3am-correctness mindset: a single `if let && let` chain would have silently fallen through to a full-universe scan and returned every creator's items when the index was absent; the code instead warns and returns empty, with a comment explaining precisely why the obvious refactor is wrong.
|
||||
- **query-retrieve-search** — Cursor encoding is genuinely 3am-proof: a discriminant byte distinguishes keyset vs offset so the two shapes can never be silently misread as each other, decode rejects non-finite keyset scores and unknown kind bytes, and offset() uses checked usize::try_from instead of a silent `as usize`. The proptests round-trip the full f64/usize ranges.
|
||||
- **ranking** — finite_score + normalize is a genuinely careful piece of float engineering: NaN is neutralized at score construction (so it can never reach the sort and scramble the rank, fixing the old partial_cmp().unwrap_or(Equal) trap), while ±Inf 'sort last/first' sentinels are preserved through a total_cmp sort and then folded to the correct end of [0,1] by computing the min/max range over finite scores only. The negated-scale regression (Shortest's -duration making a missing item normalize to 1.0) is captured by an explicit test. This is exactly the 3am-incident-proof rigor the project demands.
|
||||
- **entities** — Durability is genuinely complete and traced end-to-end: every derived index (user_state, hard_neg, preference, co_engagement) has restore wired into open.rs and checkpoint/persist wired into the periodic thread, shutdown, and backup. I verified the call sites rather than trusting the doc comments, and the non-durable add_save/add_like/record_completion variants are correctly reserved for ephemeral (no-storage) mode only.
|
||||
- **governance** — The checkpoint key codec is exemplary defensive engineering: encode_contributor_suffix puts (entity, signal_type) in the KEY (not just the value) so one contributor spanning multiple cells produces distinct rows, the rustdoc names the exact BLOCKER it fixes, and checkpoint_restore_preserves_multi_entity_multi_signal_contributor regression-tests it directly.
|
||||
- **replication-core** — receiver.rs durability is genuinely WAL-first and verified end-to-end: apply_replicated_event appends to the follower's own WAL via a group-commit writer whose append() blocks until segment.sync() (fsync) before the reply, and state.advance() only runs after every staged event is durably synced — the BLOCKER-6 fix is real, not wiring.
|
||||
- **replication-crdt** — The convergence argument is genuinely well-engineered: switching per-node decay merge from addition to a last-writer-wins register versioned by last_update_ns (signal_state.rs:30-45) is the correct fix for replay-induced double-counting, and the doc comment explains exactly WHY addition is unsafe under at-least-once delivery. This is the hard part of the slice and it is right.
|
||||
- **schema** — EntityKind::fingerprint_byte (entity.rs:68) is exemplary: one const fn is the single source of truth for the kind->byte mapping, its coupling to both schema_fingerprint and embedding_slot_fingerprint is documented at the definition, and a dedicated test (entity_kind_fingerprint_byte_is_canonical) locks the byte values so a refactor cannot silently change the on-disk fingerprint. This is precisely the 3am-incident-proofing the project asks for.
|
||||
- **session** — The decay-math consolidation is exemplary: SessionHotState delegates to the canonical forward_decay_step kernel and documents the prior bug it fixed (out-of-order weight was added at full value, over-crediting stale activity). The two property-style tests (out_of_order_event_contributes_strictly_less_than_in_order, out_of_order_event_does_not_regress_timestamp) verify the exact CODING_GUIDELINES §3 invariant against an analytical pre-decay value — this is real correctness verification, not coverage theater.
|
||||
- **text** — The syncer's resilience design is exactly the 3am-incident engineering the project charter demands: a failed Tantivy commit retains the buffered batch, retries with exponential backoff, flips a shared health flag after a threshold, and keeps draining the channel — with a thread-local fault-injection hook and a CRITICAL-tagged regression test (`syncer_survives_transient_commit_failure_and_recovers`) proving the dead-syncer-loses-all-writes failure mode stays fixed.
|
||||
- **cohort-load-testing** — Crash-injection infrastructure is genuinely production-grade for a test harness: every CrashPoint variant is wired to a real write-path call site (signals/hot, signals/ledger/core, signals/checkpoint, cohort/ledger, db/collections, entities/co_engagement) and exercised by m7_crash_property.rs, the atomic Orderings (AcqRel on the crossing counter, Release/Acquire on armed/fired) are each justified with a precise comment, and one-shot + thread-local install keeps it catch_unwind-safe.
|
||||
- **tidal-net** — circuit_breaker.rs: the half-open single-probe invariant is enforced by the state machine (in-flight HalfOpen state), not merely advertised, and the record_backpressure path that resolves a half-open probe without opening the breaker (avoiding a wedged-forever breaker the first time a probe draws accepted=false) shows genuine 3am-incident thinking. The 14 tests cover every transition including the subtle interleavings.
|
||||
- **tidal-server-cluster** — dispatch_shards is a genuinely careful piece of concurrent engineering: detached (not scoped) workers so the coordinator can return its partial result the instant the budget expires without joining an in-flight blocking query; a sync_channel bounded to live_shards.len() so no late worker ever blocks on send into a dropped receiver; the coordinator's own tx dropped so recv_timeout observes Disconnected instead of burning the full budget; and a try_recv drain pass to catch race-window stragglers. Timed-out and failed-to-spawn shards are recorded as degraded by name, never silently dropped. This is the right answer to 'one slow shard must not stall the gather' and it is documented to match.
|
||||
- **tidal-server-core** — Anti-drift design is excellent: `dto.rs` and `health.rs` are deliberately the single source of truth for the request/response shapes and probe bodies shared by the standalone and cluster routers, with a clear comment explaining that prior verbatim duplication let the two modes' APIs silently diverge. This is the right abstraction for a likely change.
|
||||
- **tidalctl** — Exemplary null-vs-zero discipline: every field that cannot be honestly derived offline is typed Option<_> and serialized as JSON null with a self-describing offline_unavailable reason map, never a fabricated 0 that an operator would misread as 'healthy/idle'. This is exactly the 'if you can't observe it, you can't operate it' standard, and it is regression-tested (diagnostics_no_checkpoint_reports_null_age, diagnostics_unreadable_text_index_exits_2_and_reports_null).
|
||||
|
||||
|
||||
## Appendix — per-slice seven-dimension verdicts
|
||||
|
||||
### db-core-lifecycle
|
||||
- **Completeness:** Open/close/recovery paths are thorough and well-instrumented, but two configured builder fields (wal_dir, cache_dir) are validated yet never consumed — a field that does not flow through every layer.
|
||||
- **Accuracy:** Concurrency, atomics ordering, CAS-idempotent shutdown, and lock-acquisition order are correct and carefully reasoned; the main defects are config-honoring (silent ignore of wal_dir/cache_dir) and a Paths layout that misreports fjall's real on-disk shape.
|
||||
- **Tech Debt:** Acknowledged and documented: TidalDb ~60-field struct + two 100+-line constructors exceed §9; the debt is real but explicitly tracked with a safe incremental-extraction plan.
|
||||
- **Maintainability:** Comments consistently explain WHY (obs-REPL-1, BLOCKER 6/7, CONCURRENCY-1/2/3, SHUTDOWN-2) with traceable invariants; the two giant constructors are the only genuine cognitive-load smell and they are flagged in-code.
|
||||
- **Extensibility:** StorageBox trait boundary, NodeConfig/role abstraction, and Paths-as-single-source-of-layout are the right seams; runtime-validated builder over typestate is a defensible, documented choice.
|
||||
- **Dry:** SharedDefaults + build_control_plane + make_single_node_session_bridge already extract the duplicated constructor wiring well; remaining duplication is the per-Tag restore-scan loop in from_parts (6 near-identical blocks) which is the missing abstraction.
|
||||
- **Clean:** Names are intention-revealing and the hot path is untouched (this is cold-path lifecycle code), but Paths exposes items_dir/users_dir/creators_dir/ensure_all that describe a directory layout the engine never creates — dead and misleading API.
|
||||
|
||||
### db-entity-ops
|
||||
- **Completeness:** Strong. Validation, error paths, ephemeral no-op contracts, and crash-recovery comments are present and explicit. Gaps: collection writes lack the u32-universe guard that items.rs enforces, and Mute relationships are durably stored but inert (acknowledged as tracked).
|
||||
- **Accuracy:** Mostly sound, but two real correctness/durability concerns: collection item IDs > u32::MAX silently alias (no guard, unlike items.rs which rejects), and several persist-after-in-memory-mutation orderings diverge on persist failure (add_to_collection, leave_community) leaving in-memory ahead of durable state.
|
||||
- **Tech Debt:** Moderate. The bare `as u32` narrowing is duplicated across collections/signals while items.rs rejects and relationships.rs routes through an observable helper — three different policies for the same invariant. items.rs explicitly flags this as out-of-scope debt.
|
||||
- **Maintainability:** Good. Functions are focused, names intention-revealing, and the WHY comments are unusually thorough (durability ordering, lock-poison recovery, narrowing rationale). write_item_with_metadata is long but justified and annotated.
|
||||
- **Extensibility:** Good. Shared persistence helpers (write_entity_meta/read_entity_meta/persist_to_items/write_entity_embedding) centralize the entity format; the namespaced collection key layout enables bounded prefix scans. The missing piece is a shared item-id narrowing guard the doc itself calls for.
|
||||
- **Dry:** Good with one gap. Metadata persistence and embedding writes are well-factored into users.rs helpers. The unextracted abstraction is the u32 item-id narrowing/guard, hand-rolled differently in 3+ call sites.
|
||||
- **Clean:** Clean and readable overall. One efficiency red flag: community_contribute synchronously re-checkpoints the entire community ledger on every accepted contribution (O(total contributions) per write).
|
||||
|
||||
### db-signals-sessions
|
||||
- **Completeness:** Mostly complete; one real gap — NotificationTracker has an unbounded-growth mitigation (evict_old_days) that nothing ever calls, and the signal_with_context durability docstring omits two of the side-effect indexes it actually mutates.
|
||||
- **Accuracy:** Strong. WAL-first durability holds, atomics use correct orderings, the check_and_record TOCTOU fix and closed-session cap insert-then-trim ordering are sound and concurrency-tested. No panics on normal paths; the one expect() in recovery is provably infallible and documented.
|
||||
- **Tech Debt:** Low-moderate. A dead public surface in NotificationTracker (record/would_exceed_per_creator/would_exceed_total) duplicates logic the executor already inlines, and evict_old_days is wired to nothing.
|
||||
- **Maintainability:** High. Intention-revealing names, focused functions, and unusually good WHY-comments on the non-obvious concurrency and WAL-replay-ordering decisions.
|
||||
- **Extensibility:** Good. Executor wiring is a clean builder; embedding-slot resolution is centralized; scope/governance is threaded flatly but coherently. No over-engineering for hypotheticals.
|
||||
- **Dry:** Mostly DRY. The backup-guard + backpressure block is duplicated verbatim between signal and signal_scoped; the embedding-slot resolver pattern is duplicated for item vs creator; the cap-enforcement logic exists both in NotificationTracker (dead) and inlined in the executor.
|
||||
- **Clean:** Clean and efficient. Hot paths avoid per-candidate allocation, the load detector guard is RAII, and degradation level is threaded without branching the query body.
|
||||
|
||||
### db-recovery-rebuild
|
||||
- **Completeness:** Strong. Crash-recovery rebuild is unconditional from the durable source of truth (text/vector pattern), partial-failure arms are non-fatal+logged, replication HWM is restored, and the dual-write remote path fails closed rather than silently dropping. One gap: only the item side of the text-health latch self-heals; creator-only unhealth never clears from the background thread (documented).
|
||||
- **Accuracy:** Sound. Atomic orderings are deliberate and correct (Release/Acquire on last_seq for shutdown; Relaxed on the periodic read is safe because the WAL marker is intentionally NOT advanced periodically). DashMap iteration in the sweeper collects-then-closes, avoiding re-entrant deadlock. Schema fingerprint is collision-safe today only by coincidence of disjoint byte ranges (no length-prefix canonicalization).
|
||||
- **Tech Debt:** Low-to-moderate. signal_for_tenant remote dispatch is a documented fail-closed stub tied to a spec task (acceptable per §12). The 30s/10s/500ms/60s timing constants are inline literals. Two full item-store scans at open is duplicated work.
|
||||
- **Maintainability:** High. Doc comments consistently explain WHY (BLOCKER/obs references, durability ordering, deadlock rationale). Functions are focused. run_checkpoint_thread is long (~180 lines) and mixes timing, trimming, checkpointing, compaction, and metrics at several abstraction levels.
|
||||
- **Extensibility:** Good. ItemIndexes bundles the shared write/rebuild indexing path so the two cannot drift — the strongest design decision in the slice. TextSyncerPending's two-phase open/spawn cleanly prevents the rebuild-vs-writer-lock deadlock. The schema fingerprint's excluded-fields trade-off is explicitly reasoned and versionable.
|
||||
- **Dry:** One real missing abstraction: four near-identical scan_prefix(&[]) + parse_key + Tag::Meta + EMB-skip loops over the items engine (rebuild_item_indexes, rebuild_text_index_from_engine, rebuild_item_text_and_suggestions_at_open) and a creator variant. At open, the items keyspace is fully scanned twice.
|
||||
- **Clean:** Clean and neat. Intention-revealing names, good tracing discipline (no println!), no dead code. Mixed abstraction levels inside run_checkpoint_thread are the only blemish.
|
||||
|
||||
### db-export-backup-http
|
||||
- **Completeness:** Strong — export validates limits and unknown signal types up front, backup quiesces writes and handles partial-failure non-fatally, http surface is bounded and tested. Gap: the session-journal export path has no limit/streaming, and the JSON-Lines `ExportFormat` enum and `to_json_line` are never used by `export_signals` itself (callers must serialize), leaving the wire format half-wired.
|
||||
- **Accuracy:** Mostly sound. WAL u8 signal-type truncation is correctly guarded at both write and export boundaries; backup ordering (ledger checkpoint -> flush -> copy -> WAL marker) is crash-consistent and well-reasoned; signal weights are finite-validated upstream so f32 export cannot emit JSON null. One real correctness/resource issue: unbounded session-journal read defeats the export limit.
|
||||
- **Tech Debt:** Low. Duplicated since/until range-filter logic across the two scan paths; one inconsistent timestamp source in backup (raw SystemTime vs Timestamp::now elsewhere). No storage-engine type leakage, no magic-number sprawl.
|
||||
- **Maintainability:** Excellent — comments explain WHY (the bounded-heap rationale, the in-flight-WAL consistency argument, the u8 truncation history) rather than WHAT. Functions are focused and well under the 50-line smell threshold.
|
||||
- **Extensibility:** Good. `ExportFormat` is an enum ready for more formats, backup secondary-checkpoint set is factored into one helper shared with shutdown, http request bounding is constant-driven. `ExportRequest` is open to new filters without breaking callers.
|
||||
- **Dry:** Minor duplication: the `since`-inclusive/`until`-exclusive time-window filter is hand-written in both `scan_batch_wal_bounded` and `read_session_journal_signals`; the secondary-checkpoint set is duplicated between backup and shutdown (documented, acceptable).
|
||||
- **Clean:** Clean and neat. One needless allocation in `extract_path` (collects a Vec just to index [1]) but it is on the cold metrics endpoint, not a hot path. Naming is intention-revealing throughout.
|
||||
|
||||
### db-metrics
|
||||
- **Completeness:** Solid — every rendered gauge/counter has a real producer (active_sessions in sessions.rs, last_checkpoint_ns/wal_lag_bytes/tantivy_indexed_docs in state_rebuild.rs, latency histograms in signals.rs/query_ops.rs), health derivation covers dead/stale checkpoint thread, and tests assert well-formed exposition. Gap: usearch_index_size_bytes, usearch_vector_count, tantivy_segment_count, signal_hot_entries, bitmap_index_cardinality, wal_compacted_segments_total appear rendered but I could not confirm a production writer for all of them in this slice's blast radius.
|
||||
- **Accuracy:** Correct. Cumulative-bucket observe via partition_point is sound and proptested against the naive rule; +Inf=total_count is right; staleness saturating_sub is safe against backward clock jumps; Acquire/Release pairing between is_degraded and the checkpoint writer is correct; Relaxed on histogram counters is justified and appropriate. One real truncation: as_micros() (u128) -> u64 cast in callers, harmless in practice but undocumented.
|
||||
- **Tech Debt:** Low. partition_id="0" is hardcoded into every always-on metric line, which will mislead in cluster/multi-partition mode. Two separate now-in-nanos implementations (gated helper vs inline writer) contradict the module's own DRY claim.
|
||||
- **Maintainability:** Strong. Clear module doc with a 3-step recipe for adding counters, intention-revealing names, WHY-comments on ordering and gating, focused functions. render_prometheus is long but flat and uniform (the #[allow(too_many_lines)] is honest).
|
||||
- **Extensibility:** Good for the stated growth path (add field, increment, add render line). LatencyHistogram is reusable with static bounds. Weak spot: partition_id is not parameterized, so cluster-aware labeling is a future rewrite of the constant block rather than a config knob.
|
||||
- **Dry:** One real smell: the 'now in nanos' logic exists twice with divergent overflow semantics (metrics/mod.rs now_unix_nanos uses try_from/u64::MAX; state_rebuild.rs:489 uses as_nanos() as u64 with unwrap_or_default). The module doc claims it 'lives in one place' but it does not, because the helper is metrics-gated and the writer is not.
|
||||
- **Clean:** Clean and neat. No dead code, no needless allocation on the observe hot path (no per-call String), no misleading names. render_prometheus builds one String with format!/write! — efficient. The histogram comment block on observe is exemplary.
|
||||
|
||||
### wal-writer-reader
|
||||
- **Completeness:** Strong — recovery, torn-tail repair, dedup, checkpoint, rotation, and directory fsync are all present with fault-injection tests; one real gap is the shutdown-drain multi-chunk early-return that can drop later callers' replies, contradicting the module's stated 'every queued append eventually resolves' invariant.
|
||||
- **Accuracy:** Mostly sound and unusually well-reasoned on durability (data sync + explicit dir fsync on create/rotate/checkpoint/delete, checked sequence arithmetic, active-segment clamp). Defects: a checkpoint tmp file can leak on a crash mid-write, the shutdown-drain `?` drops remaining reply channels, and a corrupted-but-checksum-valid batch whose first_seq is far below the checkpoint silently advances next_seq past a gap (acceptable but undocumented).
|
||||
- **Tech Debt:** Low. The hardcoded `data[offset + 24..offset + 28]` payload_len slice in the reader duplicates the header layout that format::batch already owns — a magic-offset that will silently break if the header layout ever shifts.
|
||||
- **Maintainability:** High. Rustdoc is exemplary — every non-obvious durability and concurrency decision is explained with the failure it prevents. `run_writer` is long but the extraction into flush_batch/partition_dedup/handle_aux_command keeps each piece focused.
|
||||
- **Extensibility:** Good. Shard/region identity is threaded through cleanly, segment filename is versioned (v1/v2), and the storage surface stays behind WalError. No over-engineering for hypotheticals.
|
||||
- **Dry:** Good. The shared flush_batch/partition_dedup/handle_aux_command routines deliberately collapse the three command loops so they cannot diverge. The one duplicated constant is the 24..28 header offset (owned by format::batch, re-hardcoded in reader).
|
||||
- **Clean:** Clean. Names are intention-revealing, no dead code, no needless allocation in the commit path (batch Vec capacity is reused via drain). Comments explain WHY, not WHAT.
|
||||
|
||||
### wal-format-recovery
|
||||
- **Completeness:** Strong. Both wire formats are versioned with documented backward-compat decode, corruption is surfaced (not swallowed) into the recovery path and diagnostics roll-up, and tests cover legacy/v3/torn-tail/checksum-mismatch/forward-compat. No orphan TODOs, no commented-out code, no empty error arms.
|
||||
- **Accuracy:** Correct on the load-bearing paths: durability ordering (compact only after checkpoint+flush; dedup.record only post-fsync), the compact_wal_online live-segment clamp closes a real silent-data-loss hole, diagnostics use the same `>` replay predicate as recovery, BLAKE3 covers shard/region+governance bytes, and first_seq+i is checked_add. The one expect() is provably infallible and documented. Minor semantic conflation in v2 decode-failure handling (see findings), no blockers.
|
||||
- **Tech Debt:** Low. A couple of magic numbers (frame-length literals in tests, has_* flag bytes) and a duplicated optional-field framing pattern, but nothing structurally rotting. Storage types do not leak; only RegionId/ShardId (replication domain types) cross in, which is intended.
|
||||
- **Maintainability:** High. Wire layouts are documented byte-by-byte, named constants explain offsets, comments consistently explain WHY (the dedup contains/record split, the online-compaction hazard). Functions are focused; decode_session_events_with_diagnostics is the longest but reads linearly.
|
||||
- **Extensibility:** Good and YAGNI-appropriate. The version byte + event_size_for_version dispatch makes a v4 event record a localized change; the SessionDecodeOutcome struct gives room to add more diagnostics. The v2 session-record forward-compat path (unknown checksummed types are skipped) is a genuine extensibility win.
|
||||
- **Dry:** One real missing abstraction: the `[flag: u8][optional payload]` encode/decode pattern is hand-rolled three times in session.rs (annotation, session_seqno, idempotency_key). Otherwise the parse_base / read_*_le helpers already factor the common decoding well.
|
||||
- **Clean:** Clean. Intention-revealing names, no dead code, error-path allocations (format!/into) are confined to the cold corruption path and never touch the hot encode/decode loop. Encode/decode loops are tight with pre-sized buffers.
|
||||
|
||||
### signals-decay-hotwarm
|
||||
- **Completeness:** Mostly complete: decay/hot/trimmer are well-tested with property tests and crash-point hooks; but the warm tier's 30d window has a verified double-count that no existing test catches (loose >=9/<=31 bounds let it slip), so the 'windowed aggregates equal events in window' invariant is NOT actually covered.
|
||||
- **Accuracy:** One BLOCKER: the 30d windowed count double-counts ~24h of events for a full day after every day boundary (verified empirically: 116 real events reported as 216). Decay math and hot-tier CAS concurrency are otherwise sound and well-reasoned.
|
||||
- **Tech Debt:** Low. Magic ns constants are centralized (NANOS_PER_SEC, NS_PER_MIN/HOUR/DAY). The main debt is the triplicated 'last 24h = in-progress hour + 23 hour buckets' expression, which is the structural root of the double-count bug.
|
||||
- **Maintainability:** Strong. Exceptional doc-comments explaining WHY (ordering rationale, fold-in reasoning, crash-point placement). Functions are focused and under the 50-line smell threshold. maybe_rotate is the densest but is heavily commented.
|
||||
- **Extensibility:** Good. The forward_decay_step kernel is correctly extracted as the single source of truth across all tiers; MAX_DECAY_RATES and bucket counts are named constants; storage types do not leak. No over-engineering.
|
||||
- **Dry:** One real smell: the 24h-rollup expression is written three times (windowed_count 24h, sum_current_day, and the day_agg snapshot in maybe_rotate). Extracting it would both DRY the code and force the day-boundary clearing question that the bug hinges on.
|
||||
- **Clean:** Clean and efficient. Lock-free atomics on the hot path, no per-candidate allocation, mul_add for fused rounding, Relaxed where approximation is acceptable, Acquire/Release only at sync boundaries. Names are intention-revealing.
|
||||
|
||||
### signals-ledger-checkpoint
|
||||
- **Completeness:** Mostly complete with strong test/proptest coverage, but the checkpoint write path is missing a delete-stale-keys step — after the trimmer evicts entries, orphaned storage keys are never removed, so a whole class of recovery is untested and broken.
|
||||
- **Accuracy:** Serialization/deserialization, offset-chain const-asserts, and integrity hashing are correct in isolation, but the checkpoint+trim+restore interaction produces a guaranteed integrity-hash mismatch that escalates to silent data loss once WAL segments are compacted away — a real durability hole.
|
||||
- **Tech Debt:** Low. Named-offset codec, shared id-derivation helper, and clone-free hash iterator are deliberate, well-justified choices. The known full-keyspace-scan limitation is documented with a clear rationale.
|
||||
- **Maintainability:** High. Doc comments explain WHY (out-of-range decay None, local-profile-intact, hot/warm reconcile), functions are focused, names are intention-revealing. The offset table + const-assert make the wire format safe to evolve.
|
||||
- **Extensibility:** Good. Versioned record formats (V1/V2 entry, V1/V2 meta) with backward-compatible deserialize, a WalWriter trait with a default-method governance extension point, and a single entry-construction site. No YAGNI over-engineering.
|
||||
- **Dry:** Strong. derive_signal_ids_and_lambdas is the single id/lambda source for global+cohort ledgers; apply_event_local + get_or_create_entry centralize the mutation path; hash_checkpoint_payload_iter and the owned variant share one hashing core. One minor duplicated meta-parse block across V1/V2 arms.
|
||||
- **Clean:** Clean and efficient. Lock-free hot path honored, no per-candidate allocation in serialize, clone-free hashing. One efficiency/determinism wart: read_decay_score/read_windowed_count each call Timestamp::now() internally, so a per-candidate scoring loop takes N independent clock reads and lacks a consistent query time.
|
||||
|
||||
### storage-core
|
||||
- **Completeness:** Strong — every trait method is implemented on both backends, atomicity has both a crash test and a no-crash companion, and the streaming-iterator fix is covered. Gap: the trait contract is silent on write_batch durability, so the atomic-but-not-fsynced semantics live only in the impl comment, not the trait that callers code against.
|
||||
- **Accuracy:** Correct. write_batch routes through a single fjall OwnedWriteBatch (one journal record, one seqno) so recovery sees all-or-none; commit is Buffer-durable and callers pair it with flush()/persist(SyncAll). map_fjall_err classification is sound and conservative on the non_exhaustive wildcard. No unwrap/expect/panic in production paths; no locks held across .await (sync code). Atomic ordering and byte encoding verified.
|
||||
- **Tech Debt:** Low. Minor: triple-repeated keyspace-open block in FjallStorage::open; FjallAtomicBatch is a publicly re-exported fjall-specific primitive with no production caller (tests only) and no trait equivalent.
|
||||
- **Maintainability:** Good — intentional names, focused functions (all <50 lines), comments explain WHY (durability, snapshot ownership, classification). Main drag: keys.rs Tag maintenance is spread across as_byte/from_byte/doc/3 test arrays with no single source of truth.
|
||||
- **Extensibility:** Good. StorageEngine trait cleanly hides the backend; StorageBox composes it; nothing leaks fjall types upward. Cross-keyspace atomicity is only modeled by the concrete fjall type, so a future second engine (RocksDB) cannot satisfy the cross-keyspace transaction use case through the trait.
|
||||
- **Dry:** One real missing abstraction: the Tag enum is a hand-maintained pair of parallel match tables (as_byte/from_byte) plus three verbatim 25-element arrays in tests, and the two proptests disagree on the tag-byte range (1..=25 vs 1..=23). Adding a tag silently risks drift with no compile-time guard.
|
||||
- **Clean:** Clean. No dead code, no needless allocation in the streaming scan path (the prior eager .collect() was removed and documented), consistent abstraction levels, neat error mapping. with_capacity on the batch is appropriately non-const due to allocation.
|
||||
|
||||
### storage-indexes
|
||||
- **Completeness:** Strong overall: persistence, deletion, cardinality, selectivity, and recursion guards all present and tested. One real gap — RangeIndex has no delete_entity counterpart to BitmapIndex, and the index update path (items.rs) only inserts on re-write, leaving stale old values; the missing scrub capability lives in this slice.
|
||||
- **Accuracy:** Solid. RwLock poison handled defensively, inverted-range bounds clamp instead of panicking BTreeMap::range, u32-universe bound enforced defensively in the predicate, length-prefixed keys remove the field/value delimiter ambiguity. No torn writes, no panics on normal paths, no wrong atomic ordering (no atomics here). The recursion guard (256 nodes) is wired at both query entry points.
|
||||
- **Tech Debt:** Low. Minor double-lookup (get_mut then get) in both delete paths; MAX_FILTER_NODES literal duplicated across query_ops.rs call sites (outside slice). No storage-engine types leak — clean trait-free Vec<(Vec<u8>,Vec<u8>)> KV boundary.
|
||||
- **Maintainability:** Excellent. Intention-revealing names, focused functions, doc comments explain WHY (key disambiguation, complement semantics, predicate u32 bound). Comments are load-bearing, not noise.
|
||||
- **Extensibility:** Good. RangeKeyCodec trait cleanly parameterizes the only per-type difference (BE width). FilterExpr enum is open for new variants. The bitmap/range delete asymmetry is the main extensibility wart.
|
||||
- **Dry:** Mostly DRY. RANGE_KEY_PREFIX shared write/read side prevents drift. The 10-byte key-prefix skip and the get_mut-then-get delete dance are each duplicated across bitmap.rs and range.rs — a small missing shared helper.
|
||||
- **Clean:** Clean and efficient. No per-candidate allocation in evaluate; AND intersects smallest-first with exact (not estimated) ordering and short-circuits on empty; OR/NOT minimal. Predicate avoids id truncation. Mixed abstraction is absent.
|
||||
|
||||
### storage-vector
|
||||
- **Completeness:** Strong — trait, three backends, lifecycle, registry, and crash-recovery rebuild are all present with extensive unit + property + corruption tests; the only real gap is the per-query ef_search override carrying an unticketed TODO, and a documented soft-delete crash window.
|
||||
- **Accuracy:** Mostly sound — corruption hardening in brute load is excellent and the soft-delete tombstone closes a real resurrection bug; the notable correctness gaps are a non-atomic insert/delete on the entity store (lost-update / orphaned-tombstone window) in the lifecycle ops, and a torn-write window during update_embedding/delete_embedding where the entity store and ANN index can diverge with no recovery short of full rebuild.
|
||||
- **Tech Debt:** Low-moderate — unticketed TODO for per-query ef_search; the double full-store O(N) scan in rebuild_from_store with per-key String allocs is a startup-cost smell that will bite at scale.
|
||||
- **Maintainability:** Excellent — intention-revealing names, WHY-focused comments, focused functions, single-source-of-truth constants for HNSW defaults and the ARCHIVED prefix.
|
||||
- **Extensibility:** Excellent — the VectorIndex trait cleanly isolates usearch behind the boundary (CODING_GUIDELINES §2 honored), swappable backends, adaptive filtered-search strategy correctly deferred to the query planner.
|
||||
- **Dry:** Very good — shared validate_dimensions / read_*_le helpers, single-source HNSW default consts, shared normalize_and_store prologue and ARCHIVED_PREFIX constant; minor duplicated ef_search-warning block across search/filtered_search.
|
||||
- **Clean:** Very good — brute-force top-k uses select_nth then sorts only k (efficient), distances avoid sqrt, comments are accurate; the rebuild double-scan is the main efficiency blemish.
|
||||
|
||||
### query-executor
|
||||
- **Completeness:** Strong. All 6 pipeline stages are implemented with graceful-degradation paths, warnings on every unsatisfiable-scope branch, and ~1260 lines of unit tests plus shared fixtures. Gap: cohort rescore claims loud-failure parity with the main scoring path but silently scores 0.0 on an unknown signal name; ANN candidate strategy is a documented scan-fallback stub.
|
||||
- **Accuracy:** Mostly sound. Atomic check-and-record closes the notification TOCTOU; cursor-offset overflow is saturated; u32 truncation is validated at write time so the executor's `as u32` casts are provably safe. One real divergence: `rescore_with_cohort` swallows the schema error that the main path propagates. Off-by-pagination, ordering, and bitmap intersection logic all check out.
|
||||
- **Tech Debt:** Low. ANN fallback is a tracked stub; offset-pagination is documented as a known M3 replacement target. No magic numbers without rationale; no storage types leak into the module (all access via the StorageEngine trait).
|
||||
- **Maintainability:** Excellent. Every non-obvious branch carries a WHY comment (the separate for_creator arms, the Not-skip traversal discipline, the New-profile truncation partition). Functions are focused; `execute` is long but linear and stage-delineated. Names are intention-revealing.
|
||||
- **Extensibility:** Good. Builder-style `with_*` setters make the executor additively configurable per milestone; the shared `PostFilterCtx` + `apply_deferred_post_filters` give RETRIEVE and SEARCH one drift-proof seam. Stage boundaries are clean.
|
||||
- **Dry:** One real smell: four near-identical `collect_*_filters` AST walkers share a load-bearing And/Or-recurse-skip-Not traversal that should live in exactly one generic walker. Otherwise the shared post-filter module is exactly the right de-duplication.
|
||||
- **Clean:** Clean and efficient. Bounded-buffer top-K avoids O(N) heap in signal-ranked gen; pre-allocated result buffers; no per-candidate allocation in scoring. Minor: notification-cap creator resolution re-reads storage when a pre-loaded metadata map is already in hand.
|
||||
|
||||
### query-retrieve-search
|
||||
- **Completeness:** Strong. RETRIEVE/SEARCH/SUGGEST surfaces are fully wired with graceful degradation, deferred-post-filter coverage shared across both pipelines, and tests covering the historically-leaky cases (OR-of-deferred, negated-deferred, no-universe post-filter). One real gap: the empty/poisoned-universe degradation fix applied to SEARCH Stage 2 was NOT mirrored into the RETRIEVE Stage 2 evaluator path.
|
||||
- **Accuracy:** Mostly sound, with careful u32-overflow handling almost everywhere (entity_as_u32, retain_excluded/included, cursor decode). One genuine correctness hole: percentile_bitmap in scope.rs uses an unchecked `id as u32` guarded only by debug_assert, so a >u32 entity id silently aliases onto a low id in release builds and corrupts the trending scope pre-filter. BM25 NaN ordering and cursor non-finite handling are correctly defended.
|
||||
- **Tech Debt:** Low. Magic numbers are named constants with rationale (ANN/BM25 caps). The known u32 bitmap key-space ceiling for for_creator is documented and deferred explicitly. The creator-metadata filter's value-only matching (no field discriminant) is a documented limitation that will need revisiting.
|
||||
- **Maintainability:** High. Stage helpers are focused and named to read linearly, the 8-stage pipeline is split out of the executor struct file to respect the 600-line rule, and comments consistently explain WHY (the poison-recovery, the u32 aliasing risk, the relevance-anchor design). Doc comments are unusually load-bearing and accurate.
|
||||
- **Extensibility:** Good. Cursor is discriminant-tagged so keyset pagination can replace offset without wire breakage; ScopeResolver and the builder pattern compose cleanly; deferred-filter extraction is centralized so a polarity-aware evaluator can be slotted in later. Embedding-slot resolution is threaded rather than hardcoded.
|
||||
- **Dry:** Strong. Deferred post-filters, OR/NOT rejection validators, user-state extraction, and the rrf_term formula are each defined once and shared by RETRIEVE and SEARCH. The DEFAULT_EMBEDDING_SLOT literal is centralized. Minor residual duplication: the Stage 5 pagination/assembly block and the user-context suppression block are near-identical between the two pipelines but not extracted.
|
||||
- **Clean:** Clean. Hot-path scoring avoids per-candidate allocation, combined_filter is computed once to avoid repeated tree clones, and sorts use total_cmp where NaN could appear. Naming is intention-revealing throughout.
|
||||
|
||||
### ranking
|
||||
- **Completeness:** Clean — every Sort variant is implemented and exercised by sort_tests.rs; gates/excludes/penalties/decay/exploration are all wired; degradation-window substitution is handled. Rising/MostFollowed/CreatorEngagementRate have executor + test coverage but no built-in profile (reachable only via custom schema profiles); Hot/New use an entity-id recency proxy, both explicitly documented as known limitations rather than silent gaps.
|
||||
- **Accuracy:** Strong. The sort comparator uses f64::total_cmp and finite_score neutralizes NaN at construction, closing the old partial_cmp().unwrap_or(Equal) rank-scramble; normalize folds ±Inf sentinels to the correct end of [0,1] with a dedicated regression test; alphabetical f64 prefix collisions are broken on the full title. The executor is stateless and read-only over the ledger, so no concurrency/durability surface. One real correctness-adjacent issue: per-comparison String allocation in the alphabetical tie-break.
|
||||
- **Tech Debt:** Minor and well-tracked. The `now` parameter is threaded through the whole scoring API but steers no read (decay ages forward to wall-clock inside the ledger) — documented as a clock-contract debt. `resolve_signal_type` is consumed as a Result via map_or, silently swallowing the schema error instead of routing through the read_agg_for_sort degradation path — inconsistent but behaviorally equivalent.
|
||||
- **Maintainability:** Excellent. Doc comments consistently explain WHY (the finite_score/normalize sentinel reasoning, the co-engagement additive-vs-convex deviation, the Hot age-proxy rationale). Functions are focused and the formulas are isolated as pure functions in formulas.rs for independent testing.
|
||||
- **Extensibility:** Very good. Profiles are data (Serialize/Deserialize), sort modes are a closed enum dispatched in one place, the registry versions and validates without leaking storage types, and degradation is a parameter rather than a fork. No over-engineering for hypotheticals.
|
||||
- **Dry:** Good with one structural smell: score_inner and score_personalized re-implement the same exclude→gate→compute_raw_score→session_boost→finalize candidate loop, differing only in the personalization terms. test_fixtures.rs is an exemplary DRY consolidation of previously-duplicated ledger/profile builders.
|
||||
- **Clean:** Clear and neat, but two needless hot-path allocations: the alphabetical tie-break re-reads + re-lowercases titles (String alloc) on every equal-score comparison, and session_boost re-lowercases all session keywords for every candidate instead of once. Both violate the 'no per-candidate allocation in scoring' guideline.
|
||||
|
||||
### entities
|
||||
- **Completeness:** Strong. Every index has durable backing wired into open/checkpoint/shutdown (verified user_state.restore, hard_negatives.restore, preference_vectors.checkpoint, co_engagement.restore), the Mute relationship is explicitly tracked-but-inert with a tracking note, and edge/error paths (empty/torn bytes, unknown user, capacity=0) are handled. Gap: preference update_counts persistence is deferred to M7 (documented), and the in-memory bitmap+Vec dual tracking in CollectionIndex has no consistency guarantee under concurrent reads.
|
||||
- **Accuracy:** Mostly sound. Forward-decay kernel is shared correctly (interaction.rs handles out-of-order via advance_timestamp flag), NaN is neutralized at load boundaries, total_cmp gives deterministic ordering, full-u64 follower IDs fixed a real aliasing bug. Two concerns: co_engagement double-counts an edge when prev_item appears multiple times in the recent VecDeque, and the collection caller truncates item IDs to u32 with a bare cast (the exact pattern narrow_item_slot was built to make observable).
|
||||
- **Tech Debt:** Moderate. CollectionIndex maintains membership in BOTH a RoaringBitmap and a Vec<u32> with O(n) linear contains()/retain() per mutation — the Vec is only needed at serialize time. CreatorItemsBitmap exposes get_ref() to avoid clones but hot-path callers use the cloning get(). No magic-number sprawl; constants are named.
|
||||
- **Maintainability:** High. Intention-revealing names, focused functions, comments explain WHY (the narrow_item_slot rationale, the orthogonal seen/hidden state note, the total_cmp NaN reasoning) rather than WHAT. Durability contracts are documented per-struct. Serialization codecs are hand-rolled but linear and readable.
|
||||
- **Extensibility:** Good. RelationshipType::from_byte returning None forces every match to be revisited when a variant is added (no silent default), the storage-engine trait boundary is respected (no fjall types leak in), and checkpoint/restore follow one consistent sentinel-entity prefix pattern. No over-engineering for hypotheticals.
|
||||
- **Dry:** Mostly DRY. decay_to centralizes the read-side decay so score/top_creators cannot drift; blend_and_normalize is the shared EMA body. The missing abstraction: four near-identical hand-rolled little-endian field-by-field decode loops (mod.rs, collection.rs, relationship.rs, user_state.rs durable rows) repeat the same bounds-check + from_le_bytes + advance-pos ritual; a small reader helper would remove the repetition and the chance of an off-by-one in one copy.
|
||||
- **Clean:** Clean overall. The main efficiency smell is the per-creator RoaringBitmap clone on the candidate-generation hot path (CreatorItemsBitmap::get used where get_ref would avoid the alloc), and top_candidates doing a full O(all-edges) scan per call. Hot-path structs here are index containers (DashMap), not per-candidate scoring structs, so the repr(C, align(64)) rule does not apply to them.
|
||||
|
||||
### governance
|
||||
- **Completeness:** Strong — every governance surface (scope/share/membership/policy/capability/tombstone/provenance/ledger) has lifecycle, durable persistence, crash-replay, and tests; the one gap is an unticketed TODO in capability.rs eviction and a Purge LeaveMode declared-but-rejected, both documented.
|
||||
- **Accuracy:** Mostly sound — atomics use correct Acquire/Release at gate boundaries, CAS revoke/stop-forward are idempotent, checkpoint swap is atomic via WriteBatch, key codec collision is fixed and tested; one real data-integrity footgun (empty writer_agent purges all direct writes) and a determinism/consistency wrinkle in windowed_count's internal clock.
|
||||
- **Tech Debt:** Low-to-moderate — clean codec abstractions and shared serde helper; the standout debt is the synchronous full-snapshot checkpoint on the community write path (an algorithmic O(N^2) ingest cost) and one unticketed TODO.
|
||||
- **Maintainability:** Excellent — intention-revealing names, WHY-focused comments tying each guard to the failure it prevents, focused functions, consistent structure across all ten files.
|
||||
- **Extensibility:** Good — ScopeClass↔SignalScope pinned by compile-time const asserts, versioned policies mirror the profile registry, RemoveScope enum and LeaveMode leave room for new variants without rewrites; no over-engineering.
|
||||
- **Dry:** Strong — to_durable_bytes, encode/decode_contributor_suffix, and the atomic_u64 serde adapter are each single-sourced specifically to prevent drift between two call sites; no meaningful duplication found.
|
||||
- **Clean:** High — no dead code, no misleading names, hot-path gates are O(1) atomic loads; the lone efficiency blemish is the per-write full-rewrite checkpoint and an O(all-cells) contributor_count scan.
|
||||
|
||||
### replication-core
|
||||
- **Completeness:** Strong. WAL shipper, receiver, idempotency, reconcile, migration, tenant routing, lag, upgrade are all implemented with edge/error paths and extensive tests (crash-survival, fault-injection, proptests). Gaps: control.rs lag_for is single-shard-only (documented), and the receiver's atomic-payload claim is not honored on a mid-loop disk fault.
|
||||
- **Accuracy:** Mostly correct; concurrency primitives (CAS loops, advance_atomic_max, rate limiter refill) are carefully reasoned. One real correctness hole: apply_payload Phase 2 applies events one-at-a-time and is NOT atomic on a mid-loop WAL-append failure, contradicting its own docstring and risking double-count on re-ship.
|
||||
- **Tech Debt:** Low. Magic thresholds are named consts with spec references. control.rs lag_for carries an explicit single-shard debt marker tied to M8p6+. No storage types leak across module boundaries except the deliberate Arc<dyn StorageEngine> sink in migration (trait-abstracted, correct).
|
||||
- **Maintainability:** High. Intention-revealing names, WHY-focused comments, focused functions. spawn_shipper is long (~190 lines) but is a documented linear setup + per-peer loop with #[allow(too_many_lines)].
|
||||
- **Extensibility:** Good. Transport is a clean trait with a blanket Arc<dyn Transport> impl; in-process channel wiring is shared between WAL and session factories; routing strategies are an enum. ShipperState was deliberately hoisted to an Arc for operator visibility.
|
||||
- **Dry:** Good. advance_atomic_max, build_channel_endpoints, find_shard_assignment, last_seq/count_events scan loops are each extracted to one home. Minor: the three near-identical batch-scan-loops in shipper.rs (filter/last_seq/count_events) share a walk pattern that could be one iterator.
|
||||
- **Clean:** Clean and efficient. No needless hot-path allocation in the lag/state/rate-limiter paths (atomics, no locks). Receiver stages into a Vec (cold replication path, acceptable). Naming and structure are consistent throughout.
|
||||
|
||||
### replication-crdt
|
||||
- **Completeness:** Strong. Every CRDT type carries example-based AND proptest-based law coverage (commutativity/associativity/idempotency), HLC overflow and clamp edge cases are tested, and the snapshot round-trip is verified. Gap: no test pins down the `from_node_contribution` score/timestamp contract against its production caller, which is where a convergence bug hides.
|
||||
- **Accuracy:** Mostly sound — the per-node LWW + per-node-max merge math converges, and all 74 tests pass. Two real holes: (1) `Hlc::update()` does NOT carry logical-counter overflow the way `now()` does, so a saturated counter can return a non-unique timestamp, contradicting its own docstring; (2) `from_node_contribution`'s `score`/`last_update_ns` contract is ambiguous and the production caller feeds a decayed-to-now score with a stale timestamp, causing double-decay at read time.
|
||||
- **Tech Debt:** Low. `node_buckets` nests a full HashMap-backed `PNCounter` to hold a single node's key — a minor structural over-allocation in the warm path. No magic numbers leak; constants are named and documented.
|
||||
- **Maintainability:** Excellent. Intention-revealing names, dense WHY-comments on every non-obvious branch (overflow carry, key-aligned lookup, LWW tiebreak), and functions are short and focused. A reader can follow the convergence argument from the doc comments alone.
|
||||
- **Extensibility:** Good. The shared `lww_other_wins` helper is the right seam for the two LWW call sites, the storage engine is not touched, and the kernel `forward_decay_step` is correctly reused instead of re-implemented. No over-engineering for hypotheticals.
|
||||
- **Dry:** Good with one drift: `LWWRegister::write` re-implements the `ts > cur` decision inline instead of routing through `lww_other_wins`, whose own doc claims to be 'the single place the decision lives'. Decay math is correctly centralized in `forward_decay_step`.
|
||||
- **Clean:** Clean. No dead code, no needless allocation in the genuinely hot path (this is cold-path reconciliation), consistent abstraction levels. `decay_score` uses a clear iterator-map-sum with a key-aligned lookup that is explicitly documented as load-bearing.
|
||||
|
||||
### schema
|
||||
- **Completeness:** Strong — every validation rule (signal names, decay params, windows, velocity, embedding slots, text fields, policy signal references) has both a typed error and a test. The one real gap: AgentPolicy numeric fields (max_session_duration, max_signals_per_session) are accepted at build() with zero validation, so a Duration::ZERO policy silently expires every session.
|
||||
- **Accuracy:** Sound. Score/Timestamp/EntityId encodings are correct and property-tested; total_cmp gives a real total order; now() degrades on clock anomaly instead of panicking; saturating subtraction in decay math is consistent. No torn-write/durability concerns here (cold-path declaration types). Only accuracy nit is the ZERO-duration policy footgun above.
|
||||
- **Tech Debt:** Low. fingerprint_byte is correctly centralized as one source of truth. Minor: empty-embedding-slot-name is reported as DuplicateEmbeddingSlot (documented reuse, but a misleading variant), and the two text-field sets duplicate the same validate_text_field_set call site rather than iterating a collection.
|
||||
- **Maintainability:** High. Intention-revealing names, doc comments that explain WHY (fingerprint coupling, reserved-key panic avoidance, epoch-saturation safety), focused functions. build() is long but flagged with a justified #[allow(too_many_lines)] and is a flat sequence of independent passes.
|
||||
- **Extensibility:** Good. DecaySpec/DecayModel split (user spec vs computed lambda) and the WindowSet abstraction leave clean seams. The fingerprint exclusion of windows/velocity/positive-engagement is explicitly documented as a versioning decision, not an accident. No over-engineering.
|
||||
- **Dry:** Good. EntityKind::fingerprint_byte is the single kind->byte mapping; RESERVED_TEXT_FIELD_KEY is one const; validate_text_field_set is shared across item/creator sets. No meaningful duplicated validation logic.
|
||||
- **Clean:** Clean. No dead code (target field's load-bearing role is documented and verified against schema_fingerprint), no needless allocation on any hot path (all to_owned/clone are cold build-time), no misleading names beyond the empty-slot-name error variant.
|
||||
|
||||
### session
|
||||
- **Completeness:** Strong — every record type has a serde round-trip, every cap has an enforcement path and a test, and active/frozen snapshot paths are both covered. One gap: saved_search corruption handling silently diverges from the (stricter) serde module.
|
||||
- **Accuracy:** Mostly sound. Decay math is correctly delegated to the canonical kernel, atomics ordering matches the proven HotSignalState pattern, and bounds-checking in the serde cursor is rigorous. Two real correctness concerns: saved_search uses lossy UTF-8 (silent corruption) while the rest of the slice rejects it, and build_frozen_snapshot reads two distinct wall-clock timestamps for one logical close instant.
|
||||
- **Tech Debt:** Low. Shared byte-cursor helpers retired prior copy-paste; audit-entry codec is shared between snapshot and standalone paths. saved_search is the one module that re-rolls its own hand-inlined LE decode instead of reusing the serde cursor helpers — a duplicated, less-safe decoder.
|
||||
- **Maintainability:** High. Module-per-concern split is clean, names are intention-revealing, comments consistently explain WHY (cap-before-entry-lock deadlock, ascending replay seed, read-time window aging). Doc tables in mod.rs and the version-byte history are genuinely helpful.
|
||||
- **Extensibility:** Good. Snapshot format is explicitly versioned (0x01/0x02/0x03) with forward-compatible defaulting; PolicyViolationKind is a typed enum for caller dispatch; lambda is captured per-signal-type. No over-engineering.
|
||||
- **Dry:** Good after the cursor-helper extraction. The one missing abstraction: saved_search's serialize/deserialize should use write_len_prefixed / read_len_prefixed_utf8 / read_u64 from the serde module rather than re-implementing length-prefixed LE decode by hand.
|
||||
- **Clean:** Clean. Hot-path session decay is a single-atomic CAS loop with no per-call allocation; cold-path snapshot/serde paths allocate freely as appropriate. No dead code, no misleading names.
|
||||
|
||||
### text
|
||||
- **Completeness:** Strong — every write/delete/rebuild/flush/shutdown path is implemented, error-returning, and exercised by unit tests; crash recovery is correctly delegated to rebuild-from-store at open. Gap: an unbounded producer channel has no backpressure or depth observability when the syncer is stalled in a commit-failure backoff.
|
||||
- **Accuracy:** Sound — atomic health flag uses correct Acquire/Release pairing, the syncer retains its batch and retries on commit failure (the 3am failure mode is explicitly defended), and the collector's strict fast-field accessor avoids aliasing legal entity 0. One latency gap: a failed flush-path commit does not arm retry_not_before, so it is only retried on the slower timeout path.
|
||||
- **Tech Debt:** Low — constants are named and documented at module scope, the merge policy is centralized, and the entity_id field name lives in exactly one place. No storage-engine type leaks; Tantivy is correctly confined behind this module.
|
||||
- **Maintainability:** Excellent — intention-revealing names, focused functions, and doc comments that explain WHY (durability rationale, single-writer lock lifetime, clamp reasoning) rather than restating WHAT. The syncer run() loop is the only dense spot but is well-sectioned.
|
||||
- **Extensibility:** Good — TextFieldType drives schema construction and default-field selection, the collector is generic over any query, and preprocess_query is a clean hook for future query-syntax additions. RRF/hybrid fusion lives outside this slice as intended.
|
||||
- **Dry:** Good — merge policy, searcher_stats, and the entity_id field name are each defined once and shared by both read and write sides; a prior open()/ephemeral() merge-policy duplication was already extracted. Only duplication is per-module test helpers (test code, not flagged).
|
||||
- **Clean:** Clean — no needless allocation in the collect hot path (merge_fruits pre-sizes the Vec), no dead code, consistent abstraction levels. preprocess_query allocates one String per parse on the cold query-parse path, which is acceptable.
|
||||
|
||||
### cohort-load-testing
|
||||
- **Completeness:** Strong. Every CrashPoint variant is wired to a real call site and exercised by m7_crash_property.rs; cohort checkpoint corruption policy is documented, all-or-nothing, and tested. Gap: RateLimiterConfig's fallible validation (try_limited/try_new) exists but the public db builder only ever calls the infallible RateLimiter::new, so a degenerate user config silently degrades to unlimited at open time rather than failing loudly.
|
||||
- **Accuracy:** Mostly sound: atomic Orderings on the in-flight gauge and crash injector are correct and justified; seqno bump/encode/leader-write is held under one lock with rollback for atomicity; partition set uses poison-tolerant locking. One real divergence: cohort ledger read_decay_score still does unwrap_or(0.0) on an out-of-range decay index where the global ledger was hardened to return Ok(None) — a latent silent-wrong-answer that the current executor (idx=0 only) does not yet trigger.
|
||||
- **Tech Debt:** Acknowledged and documented in place: cohort restore's scan_prefix(&[]) full-keyspace scan and applied_count's hardcoded ShardId(0) are both called out with rationale. The await_convergence blocking redeliver-under-lock is undocumented debt. No orphan TODOs, no commented-out code.
|
||||
- **Maintainability:** Excellent. Intention-revealing names, focused functions, and unusually rich WHY-comments (lock-ordering, Arc-clone-vs-String rationale, corruption posture). cluster.rs at 766 lines is large but justified by a single cohesive harness concern and an explicit module-size note.
|
||||
- **Extensibility:** Good. Transport is trait-abstracted and pluggable (channel vs gRPC); degradation levels and thresholds are configurable; rate limiter keyed for per-session growth. The fallible config constructors are built for a future that the builder does not yet wire up.
|
||||
- **Dry:** Strong intent — cohort ledger explicitly shares derive_signal_ids_and_lambdas with the global ledger and reuses the global fixed-format record. The one DRY break is the read_decay_score lambda-range-guard: the global ledger has it, the cohort ledger does not, so the two read paths have silently diverged on an invariant they are documented to share.
|
||||
- **Clean:** Clean and efficient. Unlimited rate-limiter fast path avoids allocation and DashMap touch; cohort record keys by Arc::clone not String on the fan-out hot path; lazy token-bucket refill is syscall-free. No dead code, no misleading names, consistent abstraction levels.
|
||||
|
||||
### tidal-net
|
||||
- **Completeness:** Strong — payload validation, TLS/mTLS, circuit breaker, permanent-vs-transient classification, and deterministic receiver shutdown are all present with thorough tests and documented known gaps (streaming, heartbeat→control-plane). Gap: no GrpcTransportConfig validation, and the client tolerates an insecure+no-TLS config the server rejects.
|
||||
- **Accuracy:** Mostly correct and carefully reasoned. The circuit-breaker single-probe state machine, atomic shutdown latch ordering, and error classification are sound. One real concern: the runtime() helper uses expect() on a production path (infallible-by-invariant but undocumented as non-test); and the client/server insecure-mode asymmetry can produce a silently-plaintext client channel.
|
||||
- **Tech Debt:** Low. The MAX_PAYLOAD_BYTES mirror is a conscious, compile-time-guarded duplication with a documented migration plan. StreamSegments/Heartbeat are explicit unimplemented/minimal stubs, not silent ones.
|
||||
- **Maintainability:** Excellent. Doc comments explain WHY (backpressure-not-failure, latch vs notify race, block_on bridge rationale) rather than WHAT. Functions are focused and well under the 50-line smell. Names are intention-revealing.
|
||||
- **Extensibility:** Good. Transport sits behind the engine's trait; factory pattern mirrors InProcessTransportFactory; config is fully parameterized. Streaming RPC is wired in proto and stubbed cleanly for later. No over-engineering.
|
||||
- **Dry:** Good. The two codec-limit raises (client/server) and the two payload-size guards (transport/server) are intentional both-ends-must-agree pairs, well-commented. No problematic duplication; the one cross-crate constant duplication is guarded.
|
||||
- **Clean:** Clean. No needless allocation in the send path (tonic client clone is a cheap channel handle), no dead code, consistent abstraction levels. expect() in runtime() and the insecure-asymmetry are the only blemishes.
|
||||
|
||||
### tidal-server-cluster
|
||||
- **Completeness:** Solid for the replicated topology and the experimental-cluster gating; the gap is that the entity-sharded write path (sharded_* routes) ships without the stable-shard-ordering precondition its own helper documents, and the in-module tests never exercise the real regions() ordering, so a routing-divergence bug is undetected.
|
||||
- **Accuracy:** One CRITICAL routing-correctness bug: entity_shard relies on `shards` being in ascending order, but every caller feeds it SimulatedCluster::regions() which is unsorted/unstable HashMap iteration — the same write/read divergence the doc claims to have fixed. Concurrency/deadline/channel logic in dispatch_shards and the write pool is otherwise carefully correct.
|
||||
- **Tech Debt:** Two large multi-concern files (cluster.rs, scatter_gather.rs) with explicit deferred-split notes; the f64-score sort comparator is duplicated verbatim across retrieve/search; a couple of magic constants (oneshot retry_after_ms=50) are inline.
|
||||
- **Maintainability:** Above average — intention-revealing names, thorough WHY-focused doc comments, focused helpers. The deferred file splits are acknowledged and tracked, which is acceptable.
|
||||
- **Extensibility:** Good trait-based MergeItem abstraction and a generic dispatch_shards make adding new fan-out query kinds cheap. Routing strategy is hard-wired to hash-mod-len rather than going through the engine ShardRouter for the index step, which couples the server to one strategy.
|
||||
- **Dry:** MergeItem/dedup/reconcile abstractions are well-factored. The score-descending sort_by closure and the entire ScatterGatherMeta→ScatterGatherInfo + result-assembly block are duplicated between scatter_gather_retrieve and scatter_gather_search.
|
||||
- **Clean:** Clear and efficient; no needless hot-path allocation (workers get owned Arc clones, buffers pre-sized with_capacity). The one stain is the misleading 'ascending order' contract that callers silently violate.
|
||||
|
||||
### tidal-server-core
|
||||
- **Completeness:** Solid — both serve paths have graceful shutdown, SIGTERM+ctrl-c, readiness flip, and config-dir fail-loud resolution with good test coverage. Gap: standalone shutdown never OBSERVES the final-flush durability result (relies on Drop, which only logs), unlike the cluster path; and a lib.rs doc pointer references a non-existent 'Known Gaps' section.
|
||||
- **Accuracy:** Correct — no unwrap/expect/panic in production paths, clean clippy, constant-time auth comparison, sweeper uses Weak (no leak), io→f64 widening is lossless, shutdown-flag atomics are sound (over-strong but correct). No races or durability bugs found in the slice.
|
||||
- **Tech Debt:** Low — `extract_string` is dead pass-through indirection over `extract_string_field`; two io-error variants (`Io`/`Http`) overlap in role; config crate hand-rolls enum parse tables instead of an engine-provided FromStr.
|
||||
- **Maintainability:** Good — intention-revealing names, focused functions, comments explain WHY (CONCURRENCY-1 sweeper rationale, the cluster-build-thread reactor note). Minor: SeqCst vs Acquire/Release inconsistency for the same shutdown-flag concept across ServerState and ClusterState.
|
||||
- **Extensibility:** Mostly good — dto/health centralization (single source of truth across standalone+cluster routers) is the right abstraction. Weak spot: adding a Window/EntityKind/SignalAgg variant in the engine silently won't be accepted by the config parser until its hand-rolled match arm is updated.
|
||||
- **Dry:** One real missing abstraction: canonical string<->enum mapping for Window/EntityKind/SignalAgg lives in the engine as label()/as_str() but is re-implemented as a divergent parse table in config.rs (proven drift: engine emits 'all', parser only accepts 'all_time'). Plus the `extract_string`/`extract_string_field` synonym pair.
|
||||
- **Clean:** Clean and readable; no dead code beyond the `extract_string` delegate, no hot-path allocation concerns (this is the cold control-plane layer). Mapping helpers in dto.rs are tidy and well-scoped.
|
||||
|
||||
### tidalctl
|
||||
- **Completeness:** Strong — every command has the success, empty, missing-dir, corrupt, and degraded paths handled and tested; the exit-code contract is documented at the crate root and honored. Gaps are minor: recover's only mode is verify-only with dead flag plumbing, and scope-stats has no human-readable pretty mode (it just indents JSON) unlike recover/diagnostics.
|
||||
- **Accuracy:** Sound. No unwrap/expect/panic in production code, all fallible IO returns Result, exit codes match the documented contract, the uncheckpointed-lag arithmetic is saturating and correct. One latent: as_nanos() as u64 truncates for post-2554 timestamps (cosmetic, saturating_sub already guards the common skew case).
|
||||
- **Tech Debt:** Low. No magic numbers, no leaked storage-engine types (all access is via tidaldb::wal/text/governance public fns), no commented-out code, no TODOs. The recover --verify-only gate is a forward-compat stub but reads as intentional, not rot.
|
||||
- **Maintainability:** Excellent. Intention-revealing names, focused functions, and doc comments that consistently explain WHY (null-vs-zero rationale, lock-free constraint, single-source reasons). format_pretty in diagnostics is long (~120 lines) but it is flat, linear string-building — readable, not nested.
|
||||
- **Extensibility:** Good. One run(...) -> (String, i32) shape per command, a single render_json path, untagged WalField enum for the ok/error shape. Adding a command is a module + one match arm. The Option<_>-means-null typing makes adding offline-derivable fields safe.
|
||||
- **Dry:** The weakest dimension. The six-directory projection (base/wal/items/users/creators/cache) is duplicated between status.rs DirsOutput and paths.rs PathsOutput, and the offline-unavailable field set is maintained in two places (the OFFLINE_UNAVAILABLE table and the always-None fields of DiagnosticsOutput) with no compile-time link.
|
||||
- **Clean:** Clear, logical, neat. Not a hot path, so per-string allocation is fine. serde owns all JSON escaping (the old hand-rolled json_escape control-char hole is gone and regression-tested). Mixed-abstraction is avoided — pretty formatting and serializable projection are cleanly separated.
|
||||
@ -1,8 +1,8 @@
|
||||
# 00 -- Architecture Overview
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Author:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Purpose:** Show how the 14 specs connect. The forest before the trees.
|
||||
|
||||
---
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Storage Engine Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-02-20
|
||||
**Last Updated:** 2026-05-28
|
||||
**Prerequisites:** [VISION.md](../../VISION.md), [thoughts.md](../../thoughts.md), [Signal Ledger Research](../research/tidaldb_signal_ledger.md)
|
||||
|
||||
---
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Signal System Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** WAL subsystem, Entity Store, Schema Engine
|
||||
**Research:** `docs/research/tidaldb_signal_ledger.md`
|
||||
|
||||
@ -52,36 +52,52 @@ Signal types are declared in schema before signal events can be written. A signa
|
||||
|
||||
### Schema Definition
|
||||
|
||||
Signals are declared on a `SchemaBuilder`. Each `signal(...)` call opens a
|
||||
`SignalBuilder` configured with `.windows(...)` and `.velocity(...)`, finalized
|
||||
with `.add()`; `build()` validates and produces an immutable `Schema`.
|
||||
|
||||
```rust
|
||||
db.define_signal(SignalDef {
|
||||
name: "view",
|
||||
target: EntityKind::Item,
|
||||
decay: Decay::Exponential { half_life: Duration::days(7) },
|
||||
windows: vec![
|
||||
Window::hours(1),
|
||||
Window::hours(24),
|
||||
Window::days(7),
|
||||
Window::days(30),
|
||||
Window::all_time(),
|
||||
],
|
||||
velocity: true,
|
||||
})?;
|
||||
use std::time::Duration;
|
||||
use tidaldb::schema::{DecaySpec, EntityKind, SchemaBuilder, Window};
|
||||
|
||||
let mut builder = SchemaBuilder::new();
|
||||
|
||||
builder
|
||||
.signal(
|
||||
"view",
|
||||
EntityKind::Item,
|
||||
DecaySpec::Exponential { half_life: Duration::from_secs(7 * 24 * 3600) },
|
||||
)
|
||||
.windows(&[
|
||||
Window::OneHour,
|
||||
Window::TwentyFourHours,
|
||||
Window::SevenDays,
|
||||
Window::ThirtyDays,
|
||||
Window::AllTime,
|
||||
])
|
||||
.velocity(true)
|
||||
.add();
|
||||
|
||||
let schema = builder.build()?;
|
||||
```
|
||||
|
||||
### Signal Definition Fields
|
||||
### Signal Declaration Arguments
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| Argument | Type | Required | Description |
|
||||
|----------|------|----------|-------------|
|
||||
| `name` | `&str` | Yes | Unique signal identifier. Lowercase alphanumeric plus underscores. |
|
||||
| `target` | `EntityKind` | Yes | Which entity type this signal targets: `Item`, `User`, or `Creator`. |
|
||||
| `decay` | `Decay` | Yes | How signal weight diminishes over time. |
|
||||
| `windows` | `Vec<Window>` | Yes | Time windows for which aggregates are maintained. May be empty (e.g., `hide`). |
|
||||
| `velocity` | `bool` | Yes | Whether to compute rate-of-change per window. |
|
||||
| `decay` | `DecaySpec` | Yes | How signal weight diminishes over time. |
|
||||
| `.windows(&[Window])` | `&[Window]` | Yes for non-permanent | Time windows for which aggregates are maintained. Permanent signals omit it (empty set). |
|
||||
| `.velocity(bool)` | `bool` | No (default `false`) | Whether to compute rate-of-change per window. Requires at least one window. |
|
||||
|
||||
### Decay Types
|
||||
|
||||
`DecaySpec` is the schema-time input; the builder converts it into an immutable
|
||||
`DecayModel` (precomputing `lambda` for exponential decay) during `build()`.
|
||||
|
||||
```rust
|
||||
pub enum Decay {
|
||||
pub enum DecaySpec {
|
||||
/// Signal weight halves every `half_life` duration.
|
||||
/// Formula: w(t) = w_0 * exp(-lambda * t), lambda = ln(2) / half_life
|
||||
Exponential { half_life: Duration },
|
||||
@ -110,22 +126,28 @@ lambda = ln(2) / half_life_seconds
|
||||
|
||||
### Window Definitions
|
||||
|
||||
`Window` is a closed enum of fixed durations — not an arbitrary-duration type.
|
||||
The storage engine pre-allocates bucketed counters per window and the
|
||||
materializer schedules rollups at fixed boundaries; arbitrary durations would
|
||||
force dynamic allocation and unpredictable rollup schedules. The variants sort
|
||||
by duration: `OneHour < TwentyFourHours < SevenDays < ThirtyDays < AllTime`.
|
||||
|
||||
```rust
|
||||
pub enum Window {
|
||||
/// Fixed-duration sliding window.
|
||||
Sliding { duration: Duration },
|
||||
/// Unbounded accumulator -- all events since entity creation.
|
||||
AllTime,
|
||||
}
|
||||
|
||||
impl Window {
|
||||
pub fn hours(n: u64) -> Self { Window::Sliding { duration: Duration::hours(n) } }
|
||||
pub fn days(n: u64) -> Self { Window::Sliding { duration: Duration::days(n) } }
|
||||
pub fn all_time() -> Self { Window::AllTime }
|
||||
OneHour, // 3,600 s — label "1h"
|
||||
TwentyFourHours, // 86,400 s — label "24h"
|
||||
SevenDays, // 604,800 s — label "7d"
|
||||
ThirtyDays, // 2,592,000 s — label "30d" (day-bucket tier; real windowed counts)
|
||||
AllTime, // unbounded accumulator since entity creation — label "all"
|
||||
}
|
||||
```
|
||||
|
||||
Windows define the time boundaries for count/sum aggregation. A signal with `windows: [hours(1), hours(24), days(7), all_time()]` maintains four independent aggregates. Each window answers "how many/how much of this signal occurred within the last N?"
|
||||
All five windows are implemented. `ThirtyDays` accumulates true 30-day windowed
|
||||
counts via a day-bucket tier (it is no longer a placeholder). A `WindowSet`
|
||||
(constructed by `.windows(&[...])`) deduplicates and sorts the requested
|
||||
windows.
|
||||
|
||||
Windows define the time boundaries for count/sum aggregation. A signal declared with `.windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays, Window::AllTime])` maintains four independent aggregates. Each window answers "how many/how much of this signal occurred within the last N?"
|
||||
|
||||
### Velocity Declaration
|
||||
|
||||
@ -138,7 +160,7 @@ Velocity is computed per window. `view.velocity(1h)` measures short-term acceler
|
||||
1. Signal names must be unique within a target entity type.
|
||||
2. `Permanent` decay signals must have `velocity: false` (rate of change is meaningless for permanent state).
|
||||
3. Windows must be non-empty unless the signal is boolean/permanent (e.g., `hide`, `block`).
|
||||
4. `all_time()` windows do not support velocity (no bounded window to measure rate over).
|
||||
4. `Window::AllTime` does not support velocity (no bounded window to measure rate over).
|
||||
5. Maximum 8 windows per signal type (bounded by the hot-tier struct layout).
|
||||
6. Maximum 64 signal types per entity type (bounded by storage layout).
|
||||
|
||||
@ -534,7 +556,7 @@ Using the `log1p` function for numerical stability when the addend is small.
|
||||
|
||||
### Linear Decay
|
||||
|
||||
For signals using `Decay::Linear { lifetime }`:
|
||||
For signals using `DecaySpec::Linear { lifetime }`:
|
||||
|
||||
```
|
||||
S(t) = SUM_i [ w_i * max(0, 1 - (t - t_i) / lifetime) ]
|
||||
@ -596,46 +618,31 @@ Velocity does not require a separate data structure. It is computed from the buc
|
||||
impl WarmSignalState {
|
||||
/// Compute velocity for a given window.
|
||||
///
|
||||
/// Sums the relevant minute/hour buckets and divides by window duration.
|
||||
/// Cost: O(bucket_count) -- at most 168 for 7-day window at hourly granularity.
|
||||
pub fn velocity(&self, window: &Window, now_ns: u64) -> f64 {
|
||||
let (count, duration_secs) = match window {
|
||||
Window::Sliding { duration } if duration <= &Duration::hours(1) => {
|
||||
let minutes = duration.as_secs() / 60;
|
||||
let count = self.sum_minute_buckets(minutes as usize, now_ns);
|
||||
(count, duration.as_secs_f64())
|
||||
}
|
||||
Window::Sliding { duration } => {
|
||||
let hours = duration.as_secs() / 3600;
|
||||
let count = self.sum_hour_buckets(hours as usize, now_ns);
|
||||
(count, duration.as_secs_f64())
|
||||
}
|
||||
/// Sums the relevant minute/hour/day buckets and divides by the window's
|
||||
/// fixed duration (`Window::duration_secs_f64`).
|
||||
/// Cost: O(bucket_count) -- at most 168 for the 7-day window at hourly granularity.
|
||||
pub fn velocity(&self, window: Window, now_ns: u64) -> f64 {
|
||||
let count = match window {
|
||||
Window::OneHour => self.sum_minute_buckets(60, now_ns),
|
||||
Window::TwentyFourHours => self.sum_hour_buckets(24, now_ns),
|
||||
Window::SevenDays => self.sum_hour_buckets(168, now_ns),
|
||||
Window::ThirtyDays => self.sum_last_n_days(30),
|
||||
Window::AllTime => return 0.0, // velocity is undefined for all-time
|
||||
};
|
||||
count as f64 / duration_secs
|
||||
}
|
||||
|
||||
/// Compute relative velocity (acceleration).
|
||||
///
|
||||
/// ratio > 1.0 means accelerating; ratio < 1.0 means decelerating.
|
||||
pub fn relative_velocity(
|
||||
&self,
|
||||
short_window: &Window,
|
||||
long_window: &Window,
|
||||
now_ns: u64,
|
||||
) -> f64 {
|
||||
let v_short = self.velocity(short_window, now_ns);
|
||||
let v_long = self.velocity(long_window, now_ns);
|
||||
if v_long < f64::EPSILON {
|
||||
// No baseline -- treat as infinite acceleration if short > 0.
|
||||
if v_short > 0.0 { f64::MAX } else { 0.0 }
|
||||
} else {
|
||||
v_short / v_long
|
||||
}
|
||||
count as f64 / window.duration_secs_f64()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Implementation note (relative velocity / acceleration).** The
|
||||
> `relative_velocity = velocity(w_short) / velocity(w_long)` ratio above is a
|
||||
> design concept, not a shipped aggregation. In the ranking engine,
|
||||
> `SignalAgg::Ratio` and `SignalAgg::RelativeVelocity` are declared in the type
|
||||
> system but **not** computed by the executor; a ranking profile that references
|
||||
> either is **rejected at registration time** with
|
||||
> `ProfileError::UnsupportedAggregation`. Absolute per-window `velocity` and
|
||||
> raw windowed counts (including the 30-day window) are fully implemented.
|
||||
|
||||
### Velocity as EWMA (Smoothed)
|
||||
|
||||
The EWMA velocity is maintained as an additional atomic field in the warm tier, updated every time the minute bucket rolls over:
|
||||
@ -692,10 +699,10 @@ This means a single set of per-minute counters supports simultaneous 1h, 24h, an
|
||||
| 1h | per-minute | 60 | ~120 ns |
|
||||
| 24h | per-hour | 24 | ~48 ns |
|
||||
| 7d | per-hour | 168 | ~336 ns |
|
||||
| 30d | per-hour | 720 (from rollups) | ~1.4 us |
|
||||
| 30d | per-day | 30 | ~60 ns |
|
||||
| all_time | single counter | 1 | ~2 ns |
|
||||
|
||||
For the 30-day window, the system merges hourly rollups from the cold tier (disk) with in-memory hour buckets for the current 7 days. This follows the TimescaleDB real-time continuous aggregate pattern.
|
||||
The 30-day window is served by a per-day bucket tier (`Window::ThirtyDays` sums the last 30 day-buckets), so it is fully in-memory for active entities rather than reconstructed from cold-tier rollups on the hot path.
|
||||
|
||||
### Bucket Rotation
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# 05 -- Cohort Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Entity Model (02), Signal System (03), Query Engine
|
||||
**Research:** `docs/research/tidaldb_signal_ledger.md` (Section 7: Cohort-Scoped Signal Aggregation)
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Text Retrieval Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03)
|
||||
**Research:** `docs/research/tantivy.md`, `docs/research/ann_for_tidaldb.md`
|
||||
|
||||
@ -1090,21 +1090,34 @@ struct TextIndexer {
|
||||
|
||||
### 12.3 Crash Recovery
|
||||
|
||||
On startup, the text indexer:
|
||||
> **Implementation note (recovery is rebuild-at-open, not seqno replay).**
|
||||
> The shipped engine does NOT recover the text index by reading a
|
||||
> `last_committed_seqno` from a Tantivy commit payload. The async syncer is
|
||||
> best-effort, so an item can be durably persisted to the entity store and then
|
||||
> lost in a crash before the syncer commits it to Tantivy; a stored sequence
|
||||
> number does not heal that gap on its own and proved easy to leave un-wired,
|
||||
> hiding the lost write. Instead, on every open the engine **unconditionally
|
||||
> rebuilds the item and creator text indexes from the durable entity stores**
|
||||
> (`db::state_rebuild::rebuild_text_indexes_at_open`), exactly as the vector
|
||||
> index is rebuilt from the durable embeddings. Any entity present in the store
|
||||
> but missing from Tantivy is re-indexed. This is deterministic and cannot
|
||||
> silently regress. The item-side rebuild shares a single scan with the
|
||||
> suggestion-index rebuild.
|
||||
|
||||
1. Opens the Tantivy index.
|
||||
2. Reads the last commit's payload to recover `last_committed_seqno`.
|
||||
3. Replays all outbox entries with `seqno > last_committed_seqno`.
|
||||
4. Resumes normal polling.
|
||||
On startup, the engine:
|
||||
|
||||
1. Opens the Tantivy index (item and creator).
|
||||
2. Unconditionally rebuilds each text index from its durable entity store.
|
||||
3. Resumes normal async syncing of subsequent writes.
|
||||
|
||||
**Failure modes and recovery:**
|
||||
|
||||
| Failure | State After Crash | Recovery |
|
||||
|---------|-------------------|----------|
|
||||
| Crash before Tantivy commit | Entity store ahead of text index. Outbox entries exist for uncommitted docs. | Replay from `last_committed_seqno`. Documents appear in search after recovery. |
|
||||
| Crash during Tantivy commit | Tantivy rolls back to last successful commit. | Same as above -- replay from last committed seqno. |
|
||||
| Crash after Tantivy commit but before outbox cleanup | Outbox may re-deliver entries. | Tantivy silently handles duplicate deletes. Duplicate adds create duplicate documents briefly until the next merge consolidates them. The `_entity_id` field provides deduplication at query time. |
|
||||
| Tantivy index corruption | Text index is unusable. | Full rebuild from entity store (Section 12.5). |
|
||||
| Crash before Tantivy commit | Entity store ahead of text index; recently-written items not yet in Tantivy. | Unconditional rebuild-from-store at open re-indexes them; they appear in search after recovery. |
|
||||
| Crash during Tantivy commit | Tantivy rolls back to last successful commit. | Same as above -- rebuild-from-store at open. |
|
||||
| Crash after Tantivy commit | Text index may briefly hold duplicate documents from a re-applied write. | The delete-then-add update path and the `entity_id` term make re-indexing idempotent; rebuild-from-store re-establishes the exact set. |
|
||||
| Tantivy index corruption | Text index is unusable. | Full rebuild from entity store (Section 12.5) — the same primitive run at open. |
|
||||
|
||||
### 12.4 Outbox Key Encoding
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Vector Retrieval Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-02-20
|
||||
**Last Updated:** 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03)
|
||||
**Research:** `docs/research/ann_for_tidaldb.md`
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# 08 -- Query Engine Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07)
|
||||
**Research:** `docs/research/ann_for_tidaldb.md`, `docs/research/tidaldb_signal_ledger.md`, `docs/research/tantivy.md`
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Ranking and Scoring Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07)
|
||||
**Research:** `docs/research/ann_for_tidaldb.md`, `docs/research/tidaldb_signal_ledger.md`, `docs/research/tantivy.md`
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Feedback Loop Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Signal System](03-signal-system.md), [Entity Model](02-entity-model.md), [Relationships](04-relationships.md), [Storage Engine](01-storage-engine.md)
|
||||
**References:** [VISION.md](../../VISION.md), [SEQUENCE.md](../../SEQUENCE.md), [thoughts.md](../../thoughts.md), [API.md](../../API.md)
|
||||
|
||||
|
||||
@ -1,11 +1,13 @@
|
||||
# Schema Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-02-20
|
||||
**Last Updated:** 2026-05-28
|
||||
**Prerequisites:** [02-entity-model.md](02-entity-model.md), [03-signal-system.md](03-signal-system.md), [04-relationships.md](04-relationships.md), [API.md](../../API.md)
|
||||
**Research:** [thoughts.md](../../thoughts.md) (Stage 3 insight: schema encodes behavior, not just shape)
|
||||
|
||||
> **API note.** This spec's examples use an early `db.define_signal(SignalDef { … })` / `Decay::*` / `Window::hours(…)` sketch that predates the shipped API. The implemented surface is the fluent `SchemaBuilder` — `builder.signal(name, EntityKind, DecaySpec::Exponential { half_life })`, `.windows(&[Window::OneHour, Window::TwentyFourHours, Window::SevenDays, Window::ThirtyDays, Window::AllTime])`, `.velocity(bool)`, `.add()`, then `.build()`. `Window` is a closed enum of fixed variants (no `hours()`/`days()`/`Sliding` constructors). See [03-signal-system.md §2](03-signal-system.md#2-signal-type-declaration) and [API.md](../../API.md) for the canonical, compiling form; treat the snippets below as conceptual intent, not copy-paste code.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# 12 -- Cold Start Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Entity Model](02-entity-model.md), [Signal System](03-signal-system.md), [Relationships](04-relationships.md), [Cohorts](05-cohorts.md), [Feedback Loop](10-feedback-loop.md), [Schema](11-schema.md)
|
||||
**References:** [VISION.md](../../VISION.md) (Design Principles: "Cold start is handled by the database"), [USE_CASES.md](../../USE_CASES.md) (UC-01, UC-13), [API.md](../../API.md) (ProfileDef.exploration), [thoughts.md](../../thoughts.md) (Part III, Gap 5)
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# 13 -- Concurrency Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Storage Engine](01-storage-engine.md), [Signal System](03-signal-system.md), [Feedback Loop](10-feedback-loop.md)
|
||||
**References:** [CODING_GUIDELINES.md](../../CODING_GUIDELINES.md), [thoughts.md](../../thoughts.md), [Text Retrieval](06-text-retrieval.md), [Vector Retrieval](07-vector-retrieval.md)
|
||||
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
# Scale Architecture Specification
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Implemented (M0–M8); multi-node cluster mode is PARTIAL — see [CHANGELOG.md](../../CHANGELOG.md) known gaps (G1 in-process transport, G2 tier-3 tests, G3 hash inconsistency)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-02-20
|
||||
**Last Updated:** 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03), Cohorts (05), Vector Retrieval (07)
|
||||
|
||||
---
|
||||
|
||||
39
hooks/pre-commit
Executable file
39
hooks/pre-commit
Executable file
@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# tidalDB pre-commit hook (tracked). Activate per-clone with:
|
||||
# git config core.hooksPath hooks
|
||||
# or run scripts/install-hooks.sh once.
|
||||
#
|
||||
# Gates: Rust fmt/clippy/test (only when Rust is staged), the documentation
|
||||
# consolidation guard (always), and site eslint (when node_modules exist).
|
||||
|
||||
set -uo pipefail
|
||||
ROOT="$(git rev-parse --show-toplevel)"
|
||||
cd "$ROOT" || exit 2
|
||||
|
||||
staged() { git diff --cached --name-only --diff-filter=ACM; }
|
||||
rust_staged=$(staged | grep -E '\.rs$' || true)
|
||||
site_staged=$(staged | grep -E '^site/.*\.(ts|tsx|js|jsx|mjs)$' || true)
|
||||
|
||||
# --- Rust (engine crate) ----------------------------------------------------
|
||||
if [ -n "$rust_staged" ]; then
|
||||
echo "pre-commit: cargo fmt"
|
||||
cargo fmt -p tidaldb || { echo "cargo fmt failed" >&2; exit 1; }
|
||||
# re-stage any files fmt rewrote
|
||||
echo "$rust_staged" | while IFS= read -r f; do [ -f "$f" ] && git add "$f"; done
|
||||
echo "pre-commit: cargo clippy -p tidaldb -D warnings"
|
||||
cargo clippy -p tidaldb --all-targets -- -D warnings || exit 1
|
||||
echo "pre-commit: cargo test -p tidaldb --lib"
|
||||
cargo test -p tidaldb --lib || exit 1
|
||||
fi
|
||||
|
||||
# --- Documentation consolidation guard (always) -----------------------------
|
||||
bash scripts/check-docs.sh || exit 1
|
||||
|
||||
# --- Marketing site ---------------------------------------------------------
|
||||
if [ -n "$site_staged" ] && [ -d site/node_modules ]; then
|
||||
echo "pre-commit: eslint (site)"
|
||||
( cd site && npx --no-install eslint . ) || exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
99
scripts/check-docs.sh
Executable file
99
scripts/check-docs.sh
Executable file
@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Documentation consolidation guard.
|
||||
#
|
||||
# tidalDB documentation has exactly TWO canonical homes: the repository-root *.md
|
||||
# files and docs/. This script prevents the doc set from drifting back into the
|
||||
# per-crate mirror that once sprawled to 240+ duplicated files (tidal/docs/,
|
||||
# tidal/ai-lookup/, tidal/site/, .ai/, .agents/skills/).
|
||||
#
|
||||
# Hard failures (exit 1): a mirror tree reappears; CLAUDE.md's Repository
|
||||
# Structure stops listing a real workspace crate; a core-doc relative link breaks.
|
||||
# Warnings (exit 0): planning-archive link breakage and ROADMAP phase-dir gaps.
|
||||
#
|
||||
# Run standalone (`scripts/check-docs.sh`) or via the pre-commit hook.
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)" || exit 2
|
||||
|
||||
fail=0
|
||||
err() { printf ' \033[31m✗\033[0m %s\n' "$*" >&2; fail=1; }
|
||||
ok() { printf ' \033[32m✓\033[0m %s\n' "$*"; }
|
||||
warn() { printf ' \033[33m!\033[0m %s\n' "$*"; }
|
||||
|
||||
echo "doc-guard: checking documentation consolidation invariants"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. No doc-mirror tree may reappear.
|
||||
# ---------------------------------------------------------------------------
|
||||
mirror_hit=0
|
||||
for p in tidal/docs tidal/ai-lookup tidal/site .ai .agents/skills; do
|
||||
if [ -e "$p" ] || git ls-files --error-unmatch "$p" >/dev/null 2>&1; then
|
||||
err "doc mirror reappeared: '$p' — docs live only at repo root + docs/ (see CLAUDE.md § Critical Rules)"
|
||||
mirror_hit=1
|
||||
fi
|
||||
done
|
||||
[ "$mirror_hit" -eq 0 ] && ok "no doc-mirror trees present"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. CLAUDE.md Repository Structure must list every sibling workspace crate.
|
||||
# ---------------------------------------------------------------------------
|
||||
struct_hit=0
|
||||
for crate in tidal-net tidal-server tidalctl; do
|
||||
if grep -q "^\s*\"$crate\"" Cargo.toml; then
|
||||
grep -q "$crate" CLAUDE.md || { err "CLAUDE.md does not mention workspace crate '$crate' (Cargo.toml/CLAUDE.md drift)"; struct_hit=1; }
|
||||
fi
|
||||
done
|
||||
[ "$struct_hit" -eq 0 ] && ok "CLAUDE.md structure lists all sibling crates"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Core-doc relative links resolve (hard). Planning archive is warn-only.
|
||||
# ---------------------------------------------------------------------------
|
||||
check_links() { # $1 = file, $2 = "hard"|"warn"
|
||||
local f="$1" mode="$2" dir broken=0
|
||||
dir=$(dirname "$f")
|
||||
# extract ](target) link targets
|
||||
while IFS= read -r target; do
|
||||
[ -z "$target" ] && continue
|
||||
case "$target" in
|
||||
http://*|https://*|mailto:*|\#*) continue ;; # external / anchors
|
||||
*\{*\}*) continue ;; # {template} placeholders
|
||||
esac
|
||||
target=${target%%#*} # strip #anchor
|
||||
[ -z "$target" ] && continue
|
||||
if [ ! -e "$dir/$target" ]; then
|
||||
if [ "$mode" = hard ]; then err "broken link in $f -> $target"; else warn "broken link in $f -> $target"; fi
|
||||
broken=1
|
||||
fi
|
||||
done < <(grep -oE '\]\([^)]+\)' "$f" 2>/dev/null | sed -E 's/^\]\(//; s/\)$//')
|
||||
return $broken
|
||||
}
|
||||
|
||||
core_docs=(README.md CLAUDE.md AGENTS.md CONTRIBUTING.md CHANGELOG.md \
|
||||
VISION.md USE_CASES.md SEQUENCE.md ARCHITECTURE.md API.md \
|
||||
QUICKSTART.md CODING_GUIDELINES.md thoughts.md docs/README.md)
|
||||
core_ok=1
|
||||
for f in "${core_docs[@]}"; do [ -f "$f" ] && { check_links "$f" hard || core_ok=0; }; done
|
||||
[ "$core_ok" -eq 1 ] && [ "$fail" -eq 0 ] && ok "core-doc links resolve"
|
||||
# specs cross-links (hard)
|
||||
for f in docs/specs/*.md; do [ -f "$f" ] && check_links "$f" hard >/dev/null || true; done
|
||||
|
||||
# planning archive (warn-only — historical docs)
|
||||
for f in $(git ls-files 'docs/planning/*.md' 2>/dev/null); do check_links "$f" warn >/dev/null || true; done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. ROADMAP COMPLETE milestones should have an on-disk planning dir (warn).
|
||||
# ---------------------------------------------------------------------------
|
||||
if [ -f docs/planning/ROADMAP.md ]; then
|
||||
for n in $(grep -oE 'M[0-9]+' docs/planning/ROADMAP.md | sort -u | sed 's/M//'); do
|
||||
if grep -qiE "\| *\*?\*?M$n\b.*(COMPLETE|✅)" docs/planning/ROADMAP.md 2>/dev/null; then
|
||||
[ -d "docs/planning/milestone-$n" ] || warn "ROADMAP marks M$n complete but docs/planning/milestone-$n/ is missing (backlog: backfill phase/task docs)"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo "doc-guard: FAILED — fix the ✗ items above." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "doc-guard: OK"
|
||||
7
scripts/install-hooks.sh
Executable file
7
scripts/install-hooks.sh
Executable file
@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Activate the tracked git hooks in hooks/ for this clone. Idempotent.
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
git config core.hooksPath hooks
|
||||
chmod +x hooks/pre-commit scripts/check-docs.sh 2>/dev/null || true
|
||||
echo "✓ core.hooksPath = hooks (pre-commit active: cargo gates + doc-guard + site eslint)"
|
||||
@ -29,9 +29,30 @@ impl PeerPool {
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns [`GrpcTransportError`] if a peer URI is invalid or the client TLS
|
||||
/// configuration cannot be built.
|
||||
/// Returns [`GrpcTransportError`] if the config fails its numeric invariant
|
||||
/// check ([`GrpcTransportConfig::validate`]), if TLS is unset but plaintext
|
||||
/// was not explicitly opted into (`insecure == false`), if a peer URI is
|
||||
/// invalid, or if the client TLS configuration cannot be built.
|
||||
pub fn new(config: &GrpcTransportConfig) -> Result<Self, GrpcTransportError> {
|
||||
config.validate()?;
|
||||
|
||||
// Refuse to build a PLAINTEXT channel unless plaintext was explicitly
|
||||
// opted into. With no TLS config the scheme below falls back to `http`,
|
||||
// so silently honoring `insecure == false` would ship WAL bytes in the
|
||||
// clear while the operator believes the peer is authenticated. The server
|
||||
// (server::start_server) rejects this same asymmetry; the client must
|
||||
// match it so a half-configured deployment fails loudly on BOTH ends
|
||||
// instead of one end quietly downgrading to plaintext. `TlsConfig` is the
|
||||
// right variant: it is classified `is_permanent` (see error.rs), so the
|
||||
// shipper quarantines rather than retrying a config that cannot self-heal.
|
||||
if config.tls.is_none() && !config.insecure {
|
||||
return Err(GrpcTransportError::TlsConfig(
|
||||
"TLS not configured and insecure not set; refusing to ship WAL segments over \
|
||||
plaintext"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
|
||||
let mut peers = HashMap::new();
|
||||
|
||||
for (&shard_id, addr) in &config.peers {
|
||||
@ -153,3 +174,53 @@ impl PeerPool {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::GrpcTransportConfig;
|
||||
|
||||
#[test]
|
||||
fn rejects_plaintext_when_insecure_unset() {
|
||||
// insecure=false with no TLS config must error rather than silently
|
||||
// building a plaintext channel — symmetric with the server's rejection.
|
||||
// No peers means the early-return guard fires before any endpoint is
|
||||
// built, so this needs no tokio reactor.
|
||||
let config = GrpcTransportConfig {
|
||||
tls: None,
|
||||
insecure: false,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
// `let-else` instead of `expect_err` so the test does not require
|
||||
// `PeerPool: Debug` (it holds tonic channel handles that are not Debug).
|
||||
let Err(err) = PeerPool::new(&config) else {
|
||||
panic!("plaintext without opt-in must be rejected");
|
||||
};
|
||||
assert!(
|
||||
matches!(err, GrpcTransportError::TlsConfig(_)),
|
||||
"expected a permanent TlsConfig error, got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allows_plaintext_when_insecure_set() {
|
||||
// The explicit opt-in path must still succeed (no peers => no endpoints).
|
||||
let config = GrpcTransportConfig {
|
||||
tls: None,
|
||||
insecure: true,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(PeerPool::new(&config).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_invalid_config() {
|
||||
// PeerPool::new must run the config invariant check; a zero capacity is
|
||||
// caught here instead of panicking later in the mpsc channel build.
|
||||
let config = GrpcTransportConfig {
|
||||
channel_capacity: 0,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(PeerPool::new(&config).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@ -4,6 +4,8 @@ use std::{collections::HashMap, net::SocketAddr, path::PathBuf, time::Duration};
|
||||
|
||||
use tidaldb::replication::shard::ShardId;
|
||||
|
||||
use crate::error::GrpcTransportError;
|
||||
|
||||
/// Maximum payload size in bytes (64 MiB).
|
||||
///
|
||||
/// **Source of truth:** the engine crate's `InProcessTransport::MAX_PAYLOAD_BYTES`
|
||||
@ -67,6 +69,68 @@ pub struct GrpcTransportConfig {
|
||||
pub keep_alive_timeout: Duration,
|
||||
}
|
||||
|
||||
impl GrpcTransportConfig {
|
||||
/// Reject a config whose numeric invariants would silently break the
|
||||
/// transport rather than fail loudly.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns [`GrpcTransportError::Internal`] if any of the following holds.
|
||||
/// Each is a misconfiguration that, left unchecked, degrades into a silent
|
||||
/// fault the operator cannot see at runtime:
|
||||
///
|
||||
/// - `channel_capacity == 0`: [`tokio::sync::mpsc::channel`] panics on a
|
||||
/// zero capacity, so the transport would not even start — surface it as a
|
||||
/// typed error here instead of a panic deep in [`GrpcTransport::new`].
|
||||
/// - any timeout is `Duration::ZERO` (`connect_timeout`, `request_timeout`,
|
||||
/// `keep_alive_interval`, `keep_alive_timeout`): a zero deadline aborts
|
||||
/// every attempt instantly, turning the fail-fast design into a transport
|
||||
/// that can never ship a single segment.
|
||||
/// - `max_payload_bytes == 0` or `> MAX_PAYLOAD_BYTES`: a zero ceiling
|
||||
/// rejects every segment; a ceiling above the engine's wire limit lets the
|
||||
/// client/server codec accept a payload the peer will refuse, so both ends
|
||||
/// must agree on the same in-range bound.
|
||||
/// - `circuit_breaker_reset == Duration::ZERO`: a breaker that reopens with
|
||||
/// no cool-down hammers a down peer with no backoff.
|
||||
///
|
||||
/// `circuit_breaker_threshold == 0` is intentionally allowed: the breaker
|
||||
/// treats a zero threshold as "open on the first failure" (fail-fast), a
|
||||
/// legitimate—if aggressive—policy rather than a degenerate one.
|
||||
///
|
||||
/// [`GrpcTransport::new`]: crate::transport::GrpcTransport::new
|
||||
pub fn validate(&self) -> Result<(), GrpcTransportError> {
|
||||
if self.channel_capacity == 0 {
|
||||
return Err(GrpcTransportError::Internal(
|
||||
"channel_capacity must be > 0 (a zero-capacity mpsc channel panics)".into(),
|
||||
));
|
||||
}
|
||||
for (name, d) in [
|
||||
("connect_timeout", self.connect_timeout),
|
||||
("request_timeout", self.request_timeout),
|
||||
("keep_alive_interval", self.keep_alive_interval),
|
||||
("keep_alive_timeout", self.keep_alive_timeout),
|
||||
] {
|
||||
if d.is_zero() {
|
||||
return Err(GrpcTransportError::Internal(format!(
|
||||
"{name} must be > 0 (a zero deadline aborts every attempt instantly)"
|
||||
)));
|
||||
}
|
||||
}
|
||||
if self.max_payload_bytes == 0 || self.max_payload_bytes > MAX_PAYLOAD_BYTES {
|
||||
return Err(GrpcTransportError::Internal(format!(
|
||||
"max_payload_bytes must be in 1..={MAX_PAYLOAD_BYTES}, got {}",
|
||||
self.max_payload_bytes
|
||||
)));
|
||||
}
|
||||
if self.circuit_breaker_reset.is_zero() {
|
||||
return Err(GrpcTransportError::Internal(
|
||||
"circuit_breaker_reset must be > 0 (a zero cool-down hammers a down peer)".into(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for GrpcTransportConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
@ -103,3 +167,94 @@ pub struct TlsConfig {
|
||||
/// Path to the client private key PEM file (for mTLS).
|
||||
pub client_key: Option<PathBuf>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn default_config_is_valid() {
|
||||
// The shipped defaults must pass their own invariant check, otherwise
|
||||
// every default-built transport would error at construction.
|
||||
GrpcTransportConfig::default()
|
||||
.validate()
|
||||
.expect("default config must satisfy its own invariants");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zero_channel_capacity_is_rejected() {
|
||||
// A zero-capacity mpsc channel panics inside GrpcTransport::new; the
|
||||
// typed error must catch it before that panic.
|
||||
let cfg = GrpcTransportConfig {
|
||||
channel_capacity: 0,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
let err = cfg
|
||||
.validate()
|
||||
.expect_err("zero channel_capacity must be rejected");
|
||||
assert!(matches!(err, GrpcTransportError::Internal(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zero_timeouts_are_rejected() {
|
||||
let mutators: [fn(&mut GrpcTransportConfig); 4] = [
|
||||
|c| c.connect_timeout = Duration::ZERO,
|
||||
|c| c.request_timeout = Duration::ZERO,
|
||||
|c| c.keep_alive_interval = Duration::ZERO,
|
||||
|c| c.keep_alive_timeout = Duration::ZERO,
|
||||
];
|
||||
for mutate in mutators {
|
||||
let mut cfg = GrpcTransportConfig::default();
|
||||
mutate(&mut cfg);
|
||||
assert!(
|
||||
cfg.validate().is_err(),
|
||||
"a zero deadline must be rejected (it aborts every attempt instantly)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zero_circuit_breaker_reset_is_rejected() {
|
||||
let cfg = GrpcTransportConfig {
|
||||
circuit_breaker_reset: Duration::ZERO,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(cfg.validate().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_ceiling_out_of_range_is_rejected() {
|
||||
// Zero rejects every segment; above the engine wire limit lets the codec
|
||||
// accept a payload the peer will refuse.
|
||||
for bytes in [0, MAX_PAYLOAD_BYTES + 1] {
|
||||
let cfg = GrpcTransportConfig {
|
||||
max_payload_bytes: bytes,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(
|
||||
cfg.validate().is_err(),
|
||||
"max_payload_bytes {bytes} is out of the 1..={MAX_PAYLOAD_BYTES} range"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_ceiling_at_engine_limit_is_allowed() {
|
||||
let cfg = GrpcTransportConfig {
|
||||
max_payload_bytes: MAX_PAYLOAD_BYTES,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(cfg.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zero_circuit_breaker_threshold_is_allowed() {
|
||||
// A zero threshold means "open on the first failure" (fail-fast), a
|
||||
// legitimate aggressive policy — not a degenerate config.
|
||||
let cfg = GrpcTransportConfig {
|
||||
circuit_breaker_threshold: 0,
|
||||
..GrpcTransportConfig::default()
|
||||
};
|
||||
assert!(cfg.validate().is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
@ -100,7 +100,7 @@ impl WalShipping for WalShippingService {
|
||||
|
||||
async fn heartbeat(
|
||||
&self,
|
||||
_request: Request<HeartbeatRequest>,
|
||||
request: Request<HeartbeatRequest>,
|
||||
) -> Result<Response<HeartbeatResponse>, Status> {
|
||||
// MINIMAL REAL BEHAVIOR, not a silent stub: a heartbeat that reaches
|
||||
// this handler proves the gRPC server is up, the listener is accepting,
|
||||
@ -108,12 +108,27 @@ impl WalShipping for WalShippingService {
|
||||
// network-liveness probe. Returning `acknowledged: true` is therefore a
|
||||
// truthful answer to "are you reachable", which is exactly what the
|
||||
// failure detector needs from the transport layer.
|
||||
//
|
||||
// Forwarding liveness into the engine's ControlPlane health table (so a
|
||||
// missed heartbeat demotes a peer) is a deferred enrichment, tracked as
|
||||
// a known gap in the M8 cluster scope; see docs/runbooks/cluster.md
|
||||
// (health checks) and CHANGELOG.md "Known gaps". It is additive: it does
|
||||
// not change this handler's contract, only who else observes the result.
|
||||
let req = request.into_inner();
|
||||
|
||||
// Record the peer's claimed identity at debug! so a probe is observable
|
||||
// (which shard/region reached us, when) without being silently dropped on
|
||||
// the floor. Until the identity is wired into the engine's health table
|
||||
// (below) this is the only place the claim is captured, so logging it is
|
||||
// load-bearing for "who is heartbeating me" during a cluster incident.
|
||||
tracing::debug!(
|
||||
shard_id = req.shard_id,
|
||||
region_id = req.region_id,
|
||||
"received heartbeat",
|
||||
);
|
||||
|
||||
// DEFERRED, tracked: forwarding `req.shard_id` / `req.region_id` liveness
|
||||
// into the engine's `ControlPlane` health table (so a missed heartbeat
|
||||
// demotes the claiming peer, and a spoofed/mismatched identity is flagged)
|
||||
// is an additive enrichment, not a contract change — it only widens who
|
||||
// observes the claim, captured here. Known gap in the M8 cluster scope;
|
||||
// see docs/runbooks/cluster.md (health checks) and CHANGELOG.md "Known
|
||||
// gaps". The transport carries no `ControlPlane` handle today; wiring one
|
||||
// is the remaining work.
|
||||
Ok(Response::new(HeartbeatResponse { acknowledged: true }))
|
||||
}
|
||||
}
|
||||
|
||||
@ -155,11 +155,24 @@ impl GrpcTransport {
|
||||
})
|
||||
}
|
||||
|
||||
/// The embedded tokio runtime. Present until [`Drop`] takes it.
|
||||
/// The embedded tokio runtime.
|
||||
///
|
||||
/// # Infallible by construction
|
||||
///
|
||||
/// The `expect` here can never fire on any production path. `runtime` is set to
|
||||
/// `Some` exactly once, in [`new`](Self::new), and is only ever moved out in
|
||||
/// [`Drop::drop`] via `self.runtime.take()`. `Drop` runs at most once and is the
|
||||
/// final use of `self`; no method (`send_segment`, `recv_segment`, …) can observe
|
||||
/// `self` after `Drop` has begun, so every call to `runtime()` sees `Some`. The
|
||||
/// `Option` exists purely so `Drop` can move the runtime out and tear it down with
|
||||
/// the non-blocking `shutdown_background` (see the field doc on `runtime`), not to
|
||||
/// model a genuinely-absent runtime — hence `expect` over fallible propagation: a
|
||||
/// `None` here would be a memory-safety/lifecycle bug in this module, not a runtime
|
||||
/// condition a caller could handle.
|
||||
const fn runtime(&self) -> &tokio::runtime::Runtime {
|
||||
self.runtime
|
||||
.as_ref()
|
||||
.expect("runtime is present for the transport's whole lifetime")
|
||||
.expect("runtime is Some for the transport's whole lifetime; only Drop takes it")
|
||||
}
|
||||
|
||||
/// Signal any current or future [`recv_segment`](Self::recv_segment) to return `None`.
|
||||
|
||||
@ -61,7 +61,7 @@ use tidaldb::{
|
||||
|
||||
use crate::{
|
||||
dto::{
|
||||
EmbeddingRequest, FeedItem, FeedQuery, FeedResponse, ItemRequest, SearchItem,
|
||||
EmbeddingRequest, FeedItem, FeedQuery, FeedResponse, ItemRequest, MAX_LIMIT, SearchItem,
|
||||
SearchQueryParams, SearchResponse, SignalRequest, default_limit, default_profile,
|
||||
feed_items, search_items,
|
||||
},
|
||||
@ -811,9 +811,12 @@ async fn feed(
|
||||
.read_region(query.region.as_deref())
|
||||
.map_err(ClusterAppError)?;
|
||||
|
||||
// Clamp the client-supplied limit at the network trust boundary (the same
|
||||
// memory-amplification guard the standalone /feed handler applies) — the
|
||||
// cluster read path previously passed the raw value straight to the engine.
|
||||
let mut builder = Retrieve::builder()
|
||||
.profile(&query.profile)
|
||||
.limit(query.limit as usize);
|
||||
.limit(query.clamped_limit() as usize);
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
}
|
||||
@ -847,7 +850,11 @@ async fn search(
|
||||
.read_region(query.region.as_deref())
|
||||
.map_err(ClusterAppError)?;
|
||||
|
||||
let mut builder = Search::builder().query(&query.query).limit(query.limit);
|
||||
// Clamp the client-supplied limit at the trust boundary (matches the
|
||||
// standalone /search handler); the cluster path passed the raw value before.
|
||||
let mut builder = Search::builder()
|
||||
.query(&query.query)
|
||||
.limit(query.clamped_limit());
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
}
|
||||
@ -961,9 +968,12 @@ async fn sharded_feed(
|
||||
State(state): State<Arc<ClusterState>>,
|
||||
Query(query): Query<ShardedFeedQuery>,
|
||||
) -> std::result::Result<Json<ShardedFeedResponse>, ClusterAppError> {
|
||||
// Clamp the client-supplied limit at the trust boundary before fanning the
|
||||
// scatter-gather out to every shard (the local query struct has no
|
||||
// clamped_limit helper, so clamp inline against the shared MAX_LIMIT).
|
||||
let mut builder = Retrieve::builder()
|
||||
.profile(&query.profile)
|
||||
.limit(query.limit as usize);
|
||||
.limit(query.limit.min(MAX_LIMIT) as usize);
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
}
|
||||
@ -1024,7 +1034,11 @@ async fn sharded_search(
|
||||
State(state): State<Arc<ClusterState>>,
|
||||
Query(query): Query<ShardedSearchQuery>,
|
||||
) -> std::result::Result<Json<ShardedSearchResponse>, ClusterAppError> {
|
||||
let mut builder = Search::builder().query(&query.query).limit(query.limit);
|
||||
// Clamp at the trust boundary before scatter-gather (inline against the
|
||||
// shared MAX_LIMIT — the local sharded query struct has no helper).
|
||||
let mut builder = Search::builder()
|
||||
.query(&query.query)
|
||||
.limit(query.limit.min(MAX_LIMIT));
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
}
|
||||
|
||||
@ -46,6 +46,20 @@ pub struct SignalRequest {
|
||||
pub creator_id: Option<u64>,
|
||||
}
|
||||
|
||||
/// Maximum page size a client may request via `?limit=`.
|
||||
///
|
||||
/// `limit` arrives across the network trust boundary and feeds the engine's
|
||||
/// candidate cap directly. Left unbounded, a single `?limit=4000000000` request
|
||||
/// would drive the retrieval/scoring pipeline to size buffers for a four-billion
|
||||
/// row result, a trivial memory-amplification `DoS`. The page-ranking surfaces
|
||||
/// (`/feed`, `/search`) are interactive UIs — a few hundred ranked items is the
|
||||
/// realistic ceiling — so 1000 is a generous-but-finite cap. Bulk export uses a
|
||||
/// separate, much larger ceiling (`ExportRequest::MAX_EXPORT_LIMIT`); this is
|
||||
/// deliberately the smaller interactive-read limit. Callers clamp with
|
||||
/// [`FeedQuery::clamped_limit`] / [`SearchQueryParams::clamped_limit`] at the
|
||||
/// boundary rather than trusting the raw value.
|
||||
pub const MAX_LIMIT: u32 = 1000;
|
||||
|
||||
/// `GET /feed` query parameters.
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct FeedQuery {
|
||||
@ -59,6 +73,22 @@ pub struct FeedQuery {
|
||||
pub region: Option<String>,
|
||||
}
|
||||
|
||||
impl FeedQuery {
|
||||
/// The requested page size clamped to [`MAX_LIMIT`].
|
||||
///
|
||||
/// Always call this instead of reading `limit` directly: it is the single
|
||||
/// enforcement point for the network trust boundary, so an oversized client
|
||||
/// value can never reach the engine's candidate cap.
|
||||
#[must_use]
|
||||
pub const fn clamped_limit(&self) -> u32 {
|
||||
if self.limit > MAX_LIMIT {
|
||||
MAX_LIMIT
|
||||
} else {
|
||||
self.limit
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `GET /search` query parameters.
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct SearchQueryParams {
|
||||
@ -71,6 +101,22 @@ pub struct SearchQueryParams {
|
||||
pub region: Option<String>,
|
||||
}
|
||||
|
||||
impl SearchQueryParams {
|
||||
/// The requested page size clamped to [`MAX_LIMIT`].
|
||||
///
|
||||
/// Always call this instead of reading `limit` directly: it is the single
|
||||
/// enforcement point for the network trust boundary, so an oversized client
|
||||
/// value can never reach the engine's candidate cap.
|
||||
#[must_use]
|
||||
pub const fn clamped_limit(&self) -> u32 {
|
||||
if self.limit > MAX_LIMIT {
|
||||
MAX_LIMIT
|
||||
} else {
|
||||
self.limit
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Default ranking profile when `?profile=` is omitted.
|
||||
#[must_use]
|
||||
pub fn default_profile() -> String {
|
||||
@ -182,3 +228,48 @@ pub fn search_item(item: &SearchResultItem) -> SearchItem {
|
||||
pub fn search_items(items: &[SearchResultItem]) -> Vec<SearchItem> {
|
||||
items.iter().map(search_item).collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn feed_query(limit: u32) -> FeedQuery {
|
||||
FeedQuery {
|
||||
user_id: None,
|
||||
profile: default_profile(),
|
||||
limit,
|
||||
region: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn search_query(limit: u32) -> SearchQueryParams {
|
||||
SearchQueryParams {
|
||||
query: "q".into(),
|
||||
user_id: None,
|
||||
limit,
|
||||
region: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_limit_clamped_at_boundary() {
|
||||
// An oversized client-supplied limit is capped, never passed through.
|
||||
assert_eq!(feed_query(u32::MAX).clamped_limit(), MAX_LIMIT);
|
||||
assert_eq!(feed_query(MAX_LIMIT + 1).clamped_limit(), MAX_LIMIT);
|
||||
// Values at or below the cap pass through unchanged.
|
||||
assert_eq!(feed_query(MAX_LIMIT).clamped_limit(), MAX_LIMIT);
|
||||
assert_eq!(feed_query(default_limit()).clamped_limit(), default_limit());
|
||||
assert_eq!(feed_query(0).clamped_limit(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn search_limit_clamped_at_boundary() {
|
||||
assert_eq!(search_query(u32::MAX).clamped_limit(), MAX_LIMIT);
|
||||
assert_eq!(search_query(MAX_LIMIT + 1).clamped_limit(), MAX_LIMIT);
|
||||
assert_eq!(search_query(MAX_LIMIT).clamped_limit(), MAX_LIMIT);
|
||||
assert_eq!(
|
||||
search_query(default_limit()).clamped_limit(),
|
||||
default_limit()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@ -6,6 +6,10 @@ pub type Result<T, E = ServerError> = std::result::Result<T, E>;
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum ServerError {
|
||||
/// A filesystem read that knows WHICH path failed — config / topology /
|
||||
/// schema file loads carry the path so the operator sees the offending file.
|
||||
/// Constructed via [`ServerError::io`]; never `?`-converted (that path lacks
|
||||
/// a path to attach).
|
||||
#[error("failed to read {path}: {source}")]
|
||||
Io {
|
||||
path: PathBuf,
|
||||
@ -17,8 +21,13 @@ pub enum ServerError {
|
||||
SchemaBuild(#[from] tidaldb::schema::SchemaError),
|
||||
#[error("tidalDB error: {0}")]
|
||||
Tidal(#[from] tidaldb::TidalError),
|
||||
#[error("http server error: {0}")]
|
||||
Http(#[from] std::io::Error),
|
||||
/// A pathless `std::io::Error` from socket setup / serving (TCP bind,
|
||||
/// `local_addr`, `axum::serve`). This is the `?`-conversion target for
|
||||
/// `std::io::Error`; the path-tagged file-read case uses [`ServerError::Io`]
|
||||
/// instead. Named for its source (the network/serve stack) rather than the
|
||||
/// old misleading "Http", which read as an HTTP-protocol error.
|
||||
#[error("network/serve I/O error: {0}")]
|
||||
Network(#[from] std::io::Error),
|
||||
#[error("bad request: {0}")]
|
||||
BadRequest(String),
|
||||
#[error("cluster mode is experimental and disabled: {0}")]
|
||||
|
||||
@ -274,8 +274,33 @@ async fn serve(state: ServerState, addr: &str, api_key: Option<Arc<str>>) -> Res
|
||||
let shutdown_state = state.clone();
|
||||
|
||||
axum::serve(listener, build_router(state, api_key))
|
||||
.with_graceful_shutdown(shutdown_signal(shutdown_state))
|
||||
.with_graceful_shutdown(shutdown_signal(shutdown_state.clone()))
|
||||
.await?;
|
||||
|
||||
// axum::serve has returned, so the router (and every `Arc<ServerState>` it
|
||||
// held) is dropped. Deterministically reclaim sole ownership and run the
|
||||
// final durable shutdown HERE, exactly like `serve_cluster` — instead of
|
||||
// letting an implicit drop fire the `TidalDb::Drop` backstop at an
|
||||
// unspecified point. If an `Arc` unexpectedly lingers we cannot observe the
|
||||
// final flush from this stack frame, so we log that we are falling back to
|
||||
// `Drop` (which still runs the same shutdown and logs any flush failure at
|
||||
// error level) rather than silently exiting 0 on a possibly-failed flush.
|
||||
match Arc::try_unwrap(shutdown_state) {
|
||||
Ok(owned) => {
|
||||
// Dropping the sole-owner `ServerState` here drops its
|
||||
// `Arc<TidalDb>`; that runs `TidalDb::Drop`, which attempts every
|
||||
// final durable flush and logs the first failure at error level
|
||||
// (SHUTDOWN-2). The drop is deterministic now, not deferred.
|
||||
drop(owned);
|
||||
tracing::info!("standalone shutdown: database closed (checkpoint + WAL fsync)");
|
||||
}
|
||||
Err(arc) => {
|
||||
tracing::warn!(
|
||||
strong = Arc::strong_count(&arc),
|
||||
"server state still shared after serve returned; relying on Drop for final flush"
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@ -81,6 +81,14 @@ const MAX_WRITE_WORKERS: usize = 8;
|
||||
/// Queued closures permitted per worker before backpressure trips.
|
||||
const QUEUE_DEPTH_PER_WORKER: usize = 8;
|
||||
|
||||
/// Retry-after hint (milliseconds) returned to clients when the cluster write
|
||||
/// queue is saturated. Short by design: the queue drains in worker-thread time
|
||||
/// (a single gRPC ship), not in seconds, so a 50ms backoff lets a client retry
|
||||
/// almost immediately once a slot frees rather than parking it needlessly. Kept
|
||||
/// in line with the engine's own short backpressure hints so the HTTP surface
|
||||
/// and the engine advertise consistent retry semantics.
|
||||
const WRITE_BACKPRESSURE_RETRY_AFTER_MS: u64 = 50;
|
||||
|
||||
impl ClusterWritePoolConfig {
|
||||
/// Build a config for an explicit worker count, deriving `queue_depth`
|
||||
/// proportionally so a custom size still buffers brief bursts before 429.
|
||||
@ -204,7 +212,7 @@ impl ClusterWritePool {
|
||||
// (429) instead of growing threads/queue without limit. The work
|
||||
// was never enqueued, so it is safe to retry.
|
||||
return Err(ServerError::Tidal(TidalError::Backpressure {
|
||||
retry_after_ms: 50,
|
||||
retry_after_ms: WRITE_BACKPRESSURE_RETRY_AFTER_MS,
|
||||
}));
|
||||
}
|
||||
Err(TrySendError::Disconnected(_)) => {
|
||||
|
||||
@ -225,9 +225,12 @@ async fn feed(
|
||||
State(state): State<Arc<ServerState>>,
|
||||
Query(query): Query<FeedQuery>,
|
||||
) -> Result<Json<FeedResponse>, AppError> {
|
||||
// `limit` is client-supplied across the trust boundary; clamp it to
|
||||
// `dto::MAX_LIMIT` before it sizes the engine's candidate cap so an
|
||||
// oversized request cannot drive a memory-amplification DoS.
|
||||
let mut builder = Retrieve::builder()
|
||||
.profile(&query.profile)
|
||||
.limit(query.limit as usize);
|
||||
.limit(query.clamped_limit() as usize);
|
||||
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
@ -257,7 +260,11 @@ async fn search(
|
||||
State(state): State<Arc<ServerState>>,
|
||||
Query(query): Query<SearchQueryParams>,
|
||||
) -> Result<Json<SearchResponse>, AppError> {
|
||||
let mut builder = Search::builder().query(&query.query).limit(query.limit);
|
||||
// `limit` is client-supplied; clamp it to `dto::MAX_LIMIT` at the boundary
|
||||
// (see `feed`) before it reaches the search candidate cap.
|
||||
let mut builder = Search::builder()
|
||||
.query(&query.query)
|
||||
.limit(query.clamped_limit());
|
||||
if let Some(user_id) = query.user_id {
|
||||
builder = builder.for_user(user_id);
|
||||
}
|
||||
|
||||
@ -422,14 +422,18 @@ fn fold_outcome<T>(
|
||||
|
||||
// ── Merge helpers: dedup + candidate-count reconciliation ────────────────────
|
||||
|
||||
/// A merged scatter-gather item that exposes the identity and score the
|
||||
/// coordinator needs to dedup replicated copies and merge by score.
|
||||
/// A merged scatter-gather item that exposes the identity, score, and rank slot
|
||||
/// the coordinator needs to dedup replicated copies, merge by score, and assign
|
||||
/// 1-based ranks after the merge.
|
||||
trait MergeItem {
|
||||
/// The entity this result is for. Replicated shards return the SAME entity,
|
||||
/// so the coordinator dedups on this.
|
||||
fn entity_id(&self) -> EntityId;
|
||||
/// The (already-normalized) score used for K-way merge ordering.
|
||||
fn score(&self) -> f64;
|
||||
/// Assign the post-merge 1-based rank, so the shared merge tail can re-rank
|
||||
/// either result type without duplicating the loop per query kind.
|
||||
fn set_rank(&mut self, rank: usize);
|
||||
}
|
||||
|
||||
impl MergeItem for RetrieveResult {
|
||||
@ -439,6 +443,9 @@ impl MergeItem for RetrieveResult {
|
||||
fn score(&self) -> f64 {
|
||||
self.score
|
||||
}
|
||||
fn set_rank(&mut self, rank: usize) {
|
||||
self.rank = rank;
|
||||
}
|
||||
}
|
||||
|
||||
impl MergeItem for tidaldb::query::search::SearchResultItem {
|
||||
@ -448,6 +455,9 @@ impl MergeItem for tidaldb::query::search::SearchResultItem {
|
||||
fn score(&self) -> f64 {
|
||||
self.score
|
||||
}
|
||||
fn set_rank(&mut self, rank: usize) {
|
||||
self.rank = rank;
|
||||
}
|
||||
}
|
||||
|
||||
/// Collapse duplicate entities returned by REPLICATED shards, keeping the
|
||||
@ -573,6 +583,85 @@ fn enforce_max_per_creator<T: MergeItem>(
|
||||
dropped
|
||||
}
|
||||
|
||||
/// The type-agnostic result of merging gathered shard items: the final ranked
|
||||
/// page plus the two derived facts the caller folds into its result struct.
|
||||
struct MergedResults<T> {
|
||||
/// The deduped, diversity-enforced, score-sorted, truncated, re-ranked page.
|
||||
items: Vec<T>,
|
||||
/// Candidate universe reconciled across shards (see
|
||||
/// [`reconcile_total_candidates`]) — never the raw per-shard sum.
|
||||
total_candidates: usize,
|
||||
/// `false` iff the coordinator-level diversity pass dropped any item.
|
||||
constraints_satisfied: bool,
|
||||
}
|
||||
|
||||
/// Shared merge + assemble tail for RETRIEVE and SEARCH.
|
||||
///
|
||||
/// `scatter_gather_retrieve` and `scatter_gather_search` differ only in their
|
||||
/// per-shard query closure and the concrete result struct they build; the merge
|
||||
/// pipeline between is identical, so it lives here once:
|
||||
///
|
||||
/// 1. **Dedup** replicated copies of the same entity (keep the best-scoring
|
||||
/// copy), recording whether shards overlapped.
|
||||
/// 2. **Sort by score descending**, with `f64::total_cmp` so a NaN score yields
|
||||
/// a *total*, stable order instead of being treated as equal-to-everything
|
||||
/// (`partial_cmp` returns `None` for NaN, which silently degrades the sort to
|
||||
/// an unstable partial order). `total_cmp` ranks NaN deterministically (a
|
||||
/// positive NaN is the greatest value, so it sorts to the front of this
|
||||
/// descending compare), so a single poisoned score can never interleave with
|
||||
/// or scramble the ordering of the real scores.
|
||||
/// 3. **Re-enforce `max_per_creator`** across the merged set when requested,
|
||||
/// resolving each item's creator from the shard that returned it.
|
||||
/// 4. **Reconcile `total_candidates`** BEFORE truncation so it reflects the
|
||||
/// merged universe, not the page.
|
||||
/// 5. **Truncate** to `limit`, drop the source-region tags, and assign 1-based
|
||||
/// ranks.
|
||||
fn merge_and_assemble<T: MergeItem>(
|
||||
cluster: &SimulatedCluster,
|
||||
items: Vec<Sourced<T>>,
|
||||
per_shard_totals: &[usize],
|
||||
max_per_creator: Option<usize>,
|
||||
limit: usize,
|
||||
) -> MergedResults<T> {
|
||||
// Dedup replicated copies of the same entity (keep the best-scoring copy).
|
||||
let (mut items, overlap_detected) = dedup_by_entity(items);
|
||||
|
||||
// Sort the deduped set by score descending. `total_cmp` gives a total order
|
||||
// even if a score is NaN, so one poisoned score cannot scramble the rest.
|
||||
items.sort_by(|a, b| b.item.score().total_cmp(&a.item.score()));
|
||||
|
||||
// Re-enforce max-per-creator across the merged set (each shard only enforced
|
||||
// over its own slice). Each item's creator is resolved from the shard that
|
||||
// returned it, so an entity-sharded owner's metadata is reachable.
|
||||
let mut constraints_satisfied = true;
|
||||
if let Some(max_per_creator) = max_per_creator {
|
||||
let dropped = enforce_max_per_creator(cluster, &mut items, max_per_creator);
|
||||
if dropped > 0 {
|
||||
constraints_satisfied = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile the candidate count BEFORE truncating to the page limit so it
|
||||
// reflects the merged universe, not the page.
|
||||
let total_candidates =
|
||||
reconcile_total_candidates(per_shard_totals, items.len(), overlap_detected);
|
||||
|
||||
// Take top limit, drop the source-region tags, and re-rank (assign 1-based
|
||||
// ranks after merge).
|
||||
let limit = limit.min(items.len());
|
||||
items.truncate(limit);
|
||||
let mut items: Vec<T> = items.into_iter().map(|s| s.item).collect();
|
||||
for (i, item) in items.iter_mut().enumerate() {
|
||||
item.set_rank(i + 1);
|
||||
}
|
||||
|
||||
MergedResults {
|
||||
items,
|
||||
total_candidates,
|
||||
constraints_satisfied,
|
||||
}
|
||||
}
|
||||
|
||||
/// Scatter-gather RETRIEVE across all shards.
|
||||
///
|
||||
/// Fans out the query to each non-partitioned shard CONCURRENTLY, wrapping the
|
||||
@ -637,41 +726,21 @@ pub fn scatter_gather_retrieve(
|
||||
},
|
||||
);
|
||||
|
||||
// Dedup replicated copies of the same entity (keep the best-scoring copy).
|
||||
let (mut all_items, overlap_detected) = dedup_by_entity(all_items);
|
||||
|
||||
// Sort the deduped set by score descending.
|
||||
all_items.sort_by(|a, b| {
|
||||
b.item
|
||||
.score
|
||||
.partial_cmp(&a.item.score)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
});
|
||||
|
||||
// Re-enforce max-per-creator across the merged set (each shard only
|
||||
// enforced over its own slice). Each item's creator is resolved from the
|
||||
// shard that returned it, so an entity-sharded owner's metadata is reachable.
|
||||
let mut constraints_satisfied = true;
|
||||
if let Some(max_per_creator) = query.diversity.as_ref().and_then(|d| d.max_per_creator) {
|
||||
let dropped = enforce_max_per_creator(cluster, &mut all_items, max_per_creator);
|
||||
if dropped > 0 {
|
||||
constraints_satisfied = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile the candidate count BEFORE truncating to the page limit so it
|
||||
// reflects the merged universe, not the page.
|
||||
let total_candidates =
|
||||
reconcile_total_candidates(&per_shard_totals, all_items.len(), overlap_detected);
|
||||
|
||||
// Take top limit, drop the source-region tags, and re-rank (assign 1-based
|
||||
// ranks after merge).
|
||||
let limit = query.limit.min(all_items.len());
|
||||
all_items.truncate(limit);
|
||||
let mut all_items: Vec<RetrieveResult> = all_items.into_iter().map(|s| s.item).collect();
|
||||
for (i, item) in all_items.iter_mut().enumerate() {
|
||||
item.rank = i + 1;
|
||||
}
|
||||
// Dedup → NaN-safe score sort → coordinator diversity → reconcile → top-K
|
||||
// re-rank: the tail shared with SEARCH (see [`merge_and_assemble`]).
|
||||
let max_per_creator = query.diversity.as_ref().and_then(|d| d.max_per_creator);
|
||||
let MergedResults {
|
||||
items: all_items,
|
||||
total_candidates,
|
||||
constraints_satisfied,
|
||||
} = merge_and_assemble(
|
||||
cluster,
|
||||
all_items,
|
||||
&per_shard_totals,
|
||||
max_per_creator,
|
||||
query.limit,
|
||||
);
|
||||
let limit = all_items.len();
|
||||
|
||||
let elapsed = start.elapsed();
|
||||
let meta = ScatterGatherMeta {
|
||||
@ -765,40 +834,23 @@ pub fn scatter_gather_search(
|
||||
},
|
||||
);
|
||||
|
||||
// Dedup replicated copies of the same entity (keep the best-scoring copy).
|
||||
let (mut all_items, overlap_detected) = dedup_by_entity(all_items);
|
||||
|
||||
// Sort the deduped set by score descending.
|
||||
all_items.sort_by(|a, b| {
|
||||
b.item
|
||||
.score
|
||||
.partial_cmp(&a.item.score)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
});
|
||||
|
||||
// Re-enforce max-per-creator across the merged set (each shard only
|
||||
// enforced over its own slice). Each item's creator is resolved from the
|
||||
// shard that returned it, so an entity-sharded owner's metadata is reachable.
|
||||
let mut constraints_satisfied = true;
|
||||
if let Some(max_per_creator) = query.diversity.as_ref().and_then(|d| d.max_per_creator) {
|
||||
let dropped = enforce_max_per_creator(cluster, &mut all_items, max_per_creator);
|
||||
if dropped > 0 {
|
||||
constraints_satisfied = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile the candidate count BEFORE truncating to the page limit so it
|
||||
// reflects the merged universe, not the page.
|
||||
let total_candidates =
|
||||
reconcile_total_candidates(&per_shard_totals, all_items.len(), overlap_detected);
|
||||
|
||||
// Dedup → NaN-safe score sort → coordinator diversity → reconcile → top-K
|
||||
// re-rank: the tail shared with RETRIEVE (see [`merge_and_assemble`]).
|
||||
let max_per_creator = query.diversity.as_ref().and_then(|d| d.max_per_creator);
|
||||
let MergedResults {
|
||||
items: all_items,
|
||||
total_candidates,
|
||||
constraints_satisfied,
|
||||
} = merge_and_assemble(
|
||||
cluster,
|
||||
all_items,
|
||||
&per_shard_totals,
|
||||
max_per_creator,
|
||||
query.limit as usize,
|
||||
);
|
||||
// Preserve the prior SEARCH stat: `candidates_after_diversity` reports the
|
||||
// REQUESTED page size, not the (possibly smaller) returned count.
|
||||
let limit = query.limit as usize;
|
||||
all_items.truncate(limit);
|
||||
let mut all_items: Vec<tidaldb::query::search::SearchResultItem> =
|
||||
all_items.into_iter().map(|s| s.item).collect();
|
||||
for (i, item) in all_items.iter_mut().enumerate() {
|
||||
item.rank = i + 1;
|
||||
}
|
||||
|
||||
let elapsed = start.elapsed();
|
||||
let meta = ScatterGatherMeta {
|
||||
@ -1317,6 +1369,100 @@ mod tests {
|
||||
assert_eq!(total, 5, "must floor at the deduped item count");
|
||||
}
|
||||
|
||||
/// A single NaN score must NOT scramble the ordering of the real-scored
|
||||
/// items. `partial_cmp(...).unwrap_or(Equal)` treated NaN as equal to
|
||||
/// everything, degrading the sort to an unstable partial order where one
|
||||
/// poisoned candidate could reorder the rest; `total_cmp` gives a total
|
||||
/// order so the real scores stay strictly descending and the NaN item is
|
||||
/// ranked deterministically — never dropped, and never able to interleave
|
||||
/// with the real scores.
|
||||
#[test]
|
||||
fn merge_and_assemble_nan_score_does_not_scramble_order() {
|
||||
let (cluster, _shards, _names) = four_region_cluster();
|
||||
let leader = cluster.leader_region();
|
||||
// No creator_id → diversity is a no-op and every item is retained, so we
|
||||
// observe pure sort behavior. Entity 3 carries a NaN score.
|
||||
let items = vec![
|
||||
sourced(leader, 1, 0.2),
|
||||
sourced(leader, 2, 0.9),
|
||||
sourced(leader, 3, f64::NAN),
|
||||
sourced(leader, 4, 0.5),
|
||||
];
|
||||
let merged = merge_and_assemble(&cluster, items, &[4], None, 10);
|
||||
|
||||
// Nothing dropped (no diversity cap, limit exceeds the set).
|
||||
assert_eq!(
|
||||
merged.items.len(),
|
||||
4,
|
||||
"NaN must not drop or duplicate items"
|
||||
);
|
||||
assert!(merged.constraints_satisfied);
|
||||
|
||||
// The three real-scored items stay strictly descending regardless of
|
||||
// where the NaN landed.
|
||||
let real: Vec<f64> = merged
|
||||
.items
|
||||
.iter()
|
||||
.map(MergeItem::score)
|
||||
.filter(|s| !s.is_nan())
|
||||
.collect();
|
||||
assert_eq!(
|
||||
real,
|
||||
vec![0.9, 0.5, 0.2],
|
||||
"real scores must stay descending"
|
||||
);
|
||||
|
||||
// `total_cmp` ranks a (positive) NaN as the greatest value, so in this
|
||||
// descending compare it sorts to the FRONT — deterministically, every
|
||||
// run — rather than randomly interleaving among the real scores as the
|
||||
// old partial_cmp-as-Equal path allowed.
|
||||
assert!(
|
||||
merged.items.first().is_some_and(|s| s.score().is_nan()),
|
||||
"NaN item must sort deterministically, not scramble the rest"
|
||||
);
|
||||
|
||||
// Exactly one NaN survives and it is the only non-finite entry.
|
||||
let nan_count = merged.items.iter().filter(|s| s.score().is_nan()).count();
|
||||
assert_eq!(nan_count, 1, "the single NaN item is retained exactly once");
|
||||
|
||||
// Ranks are 1-based and contiguous over the whole merged page.
|
||||
for (i, item) in merged.items.iter().enumerate() {
|
||||
assert_eq!(item.rank, i + 1, "rank must be 1-based contiguous");
|
||||
}
|
||||
}
|
||||
|
||||
/// The shared merge tail honors the page `limit` and assigns 1-based ranks
|
||||
/// for SEARCH items too (proving [`merge_and_assemble`] is generic over the
|
||||
/// result type, not just `RetrieveResult`).
|
||||
#[test]
|
||||
fn merge_and_assemble_truncates_and_ranks_search_items() {
|
||||
use tidaldb::query::search::SearchResultItem;
|
||||
|
||||
let (cluster, _shards, _names) = four_region_cluster();
|
||||
let leader = cluster.leader_region();
|
||||
let make = |entity_id: u64, score: f64| Sourced {
|
||||
region: leader,
|
||||
item: SearchResultItem {
|
||||
entity_id: EntityId::new(entity_id),
|
||||
score,
|
||||
rank: 0,
|
||||
bm25_score: None,
|
||||
semantic_score: None,
|
||||
signals: Vec::new(),
|
||||
metadata: None,
|
||||
},
|
||||
};
|
||||
let items = vec![make(1, 0.1), make(2, 0.9), make(3, 0.5), make(4, 0.7)];
|
||||
let merged = merge_and_assemble(&cluster, items, &[4], None, 2);
|
||||
|
||||
assert_eq!(merged.items.len(), 2, "limit must truncate to the page");
|
||||
// Top-2 by score descending.
|
||||
assert!((merged.items[0].score - 0.9).abs() < f64::EPSILON);
|
||||
assert!((merged.items[1].score - 0.7).abs() < f64::EPSILON);
|
||||
assert_eq!(merged.items[0].rank, 1);
|
||||
assert_eq!(merged.items[1].rank, 2);
|
||||
}
|
||||
|
||||
/// End-to-end: across a 4-way REPLICATED cluster, `total_candidates` must
|
||||
/// NOT be 4x the single-shard count, and the merged items must not contain
|
||||
/// duplicate entities.
|
||||
|
||||
@ -1,98 +0,0 @@
|
||||
# AGENTS.md
|
||||
|
||||
Agent instructions for tidalDB.
|
||||
|
||||
## Team
|
||||
|
||||
| Agent | Identity | Model | Invoke when |
|
||||
|-------|----------|-------|-------------|
|
||||
| `@tidal-engineer` | Jon Gjengset — principal Rust database engineer | opus | Implementing features, storage internals, signal system, query engine, debugging correctness |
|
||||
| `@tidal-visionary` | Spencer Kimball — product and roadmap strategist | opus | Planning milestones, scoping phases, build-vs-defer decisions, roadmap sequencing |
|
||||
| `@tidal-researcher` | Andy Pavlo — database systems researcher | opus | Prior art surveys, library evaluation, architectural research, producing `docs/research/` docs |
|
||||
| `@tidal-storyteller` | Marketing and technical writer | sonnet | Marketing site (`site/`), blog posts, public-facing copy |
|
||||
|
||||
Agent definitions live in `.claude/agents/`. Full context in `CLAUDE.md §Agents`.
|
||||
|
||||
---
|
||||
|
||||
<!-- sdlc:start -->
|
||||
|
||||
## SDLC
|
||||
|
||||
> **Required reading:** `.sdlc/guidance.md` — engineering principles that govern all implementation decisions on this project. <!-- sdlc:guidance -->
|
||||
|
||||
This project uses `sdlc` as its SDLC state machine. `sdlc` manages feature lifecycle, artifacts, tasks, and milestones. It emits structured directives via `sdlc next --json` that any consumer (Claude Code, custom scripts, or humans) acts on to decide what to do next.
|
||||
|
||||
Consumer scaffolding is installed globally under `~/.claude/commands/`, `~/.gemini/commands/`, `~/.opencode/command/`, and `~/.agents/skills/` — available across all projects. Use `/sdlc-specialize` in Claude Code to generate a project-specific AI team (agents + skills) tailored to this project's tech stack and roles.
|
||||
|
||||
### Key Commands
|
||||
|
||||
- `sdlc feature create <slug> --title "..."` — create a new feature
|
||||
- `sdlc next --for <slug> --json` — get the next action directive (JSON)
|
||||
- `sdlc next` — show all active features and their next actions
|
||||
- `sdlc artifact approve <slug> <type>` — approve an artifact to advance the phase
|
||||
- `sdlc state` — show project state
|
||||
- `sdlc feature list` — list all features and their phases
|
||||
- `sdlc task list [<slug>]` — list tasks for a feature (or all tasks)
|
||||
|
||||
### Lifecycle
|
||||
|
||||
draft → specified → planned → ready → implementation → review → audit → qa → merge → released
|
||||
|
||||
Treat this lifecycle as the default pathway. You can use explicit manual transitions when needed, but approvals/artifacts are the recommended way to keep quality and traceability.
|
||||
|
||||
### Artifact Types
|
||||
|
||||
`spec` `design` `tasks` `qa_plan` `review` `audit` `qa_results`
|
||||
|
||||
### CRITICAL: Never edit .sdlc/ YAML directly
|
||||
|
||||
All state changes go through `sdlc` CLI commands. See §6 of `.sdlc/guidance.md` for the full command reference. Direct YAML edits corrupt state.
|
||||
|
||||
### Directive Interface
|
||||
|
||||
Use `sdlc next --for <slug> --json` to get the next directive. The JSON output tells the consumer what to do next (action, message, output_path, is_heavy, gates).
|
||||
|
||||
### Consumer Commands
|
||||
|
||||
- `/sdlc-next <slug>` — execute one step, then stop (human controls cadence)
|
||||
- `/sdlc-run <slug>` — run autonomously to completion
|
||||
- `/sdlc-status [<slug>]` — show current state
|
||||
- `/sdlc-plan` — distribute a plan into milestones, features, and tasks
|
||||
- `/sdlc-milestone-uat <milestone-slug>` — run the acceptance test for a milestone
|
||||
- `/sdlc-pressure-test <milestone-slug>` — pressure-test a milestone against user perspectives
|
||||
- `/sdlc-vision-adjustment [description]` — align all docs, sdlc state, and code to a vision change
|
||||
- `/sdlc-architecture-adjustment [description]` — align all docs, code, and sdlc state to an architecture change
|
||||
- `/sdlc-enterprise-readiness [--stage <stage>]` — analyze production readiness
|
||||
- `/sdlc-setup-quality-gates` — set up pre-commit hooks and quality gates
|
||||
- `/sdlc-cookbook <milestone-slug>` — create developer-scenario cookbook recipes
|
||||
- `/sdlc-cookbook-run <milestone-slug>` — execute cookbook recipes and record results
|
||||
- `/sdlc-ponder [slug]` — open the ideation workspace for exploring and committing ideas
|
||||
- `/sdlc-ponder-commit <slug>` — crystallize a pondered idea into milestones and features
|
||||
- `/sdlc-guideline <slug-or-problem>` — build an evidence-backed guideline through five research perspectives and TOC-first distillation
|
||||
- `/sdlc-suggest` — analyze project state and suggest 3-5 ponder topics to explore next
|
||||
- `/sdlc-beat [domain | feature:<slug> | --week]` — step back with a senior leadership lens; evaluate if we're building the right thing in the right direction; stores history in `.sdlc/beat.yaml`
|
||||
- `/sdlc-recruit <role>` — recruit an expert thought partner as a persistent agent
|
||||
- `/sdlc-empathy <subject>` — deep user perspective interviews before decisions
|
||||
- `/sdlc-spike <slug> — <need>; [see <ref>]` — research, prototype, validate, and report; produces working prototype + findings in `.sdlc/spikes/<slug>/findings.md`
|
||||
- `/sdlc-convo-mine [file or text]` — mine conversation dumps for signal; apply 5 perspective lenses, group themes, launch parallel ponder sessions per group
|
||||
|
||||
### Tool Suite
|
||||
|
||||
<!-- sdlc:tools -->
|
||||
Project-scoped TypeScript tools in `.sdlc/tools/` — callable by agents and humans
|
||||
during any lifecycle phase. Read `.sdlc/tools/tools.md` for the full help menu.
|
||||
|
||||
- `sdlc tool list` — show installed tools
|
||||
- `sdlc tool run <name> [args]` — run a tool; pass `--json '{...}'` for complex input
|
||||
- `sdlc tool sync` — regenerate `tools.md` after adding a custom tool
|
||||
- `sdlc tool scaffold <name> "desc"` — create a new tool skeleton
|
||||
|
||||
**Core tools:** `ama` (codebase Q&A), `quality-check` (runs platform shell gates)
|
||||
|
||||
Use `/sdlc-tool-run`, `/sdlc-tool-build`, `/sdlc-tool-audit`, `/sdlc-tool-uat` in Claude Code for guided tool workflows.
|
||||
<!-- /sdlc:tools -->
|
||||
|
||||
Project: tidalDB
|
||||
|
||||
<!-- sdlc:end -->
|
||||
@ -1,127 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to tidalDB will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
**Cluster mode (m8p8): real gRPC replication**
|
||||
- `tidal-server`'s `ClusterState` now wires each follower region to a real
|
||||
`tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
|
||||
crossbeam channels — replication between regions traverses real gRPC/TCP
|
||||
(serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
|
||||
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
|
||||
per region) and self-heal a transient bind race by retrying on a fresh port.
|
||||
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
|
||||
offloaded to a dedicated thread so they never block the axum reactor.
|
||||
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
|
||||
blocks on its own runtime).
|
||||
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
|
||||
cluster mode is honest that it replicates over real gRPC but still runs all
|
||||
regions in one process (no host/process isolation — true multi-process
|
||||
deployment remains m8p10).
|
||||
|
||||
**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
|
||||
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
|
||||
runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
|
||||
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
|
||||
`rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
|
||||
runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
|
||||
bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
|
||||
image serves a functional 3-region cluster (write replicates to both followers
|
||||
over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
|
||||
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
|
||||
HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
|
||||
+ promote over real OS processes, feature-gated).
|
||||
|
||||
## [0.1.0] - 2026-02-23
|
||||
|
||||
### Added
|
||||
|
||||
**Core Database Engine**
|
||||
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
|
||||
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
|
||||
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration
|
||||
|
||||
**Signal System**
|
||||
- Typed signal recording with exponential decay scoring
|
||||
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
|
||||
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
|
||||
- Signal velocity tracking
|
||||
- WAL-backed signal durability with crash recovery
|
||||
- Periodic signal checkpointing to fjall (every 30s)
|
||||
- WAL compaction after each checkpoint
|
||||
|
||||
**Retrieval (RETRIEVE query)**
|
||||
- 5-stage pipeline: universe, filter, score, diversify, return
|
||||
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
|
||||
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
|
||||
- Custom ranking profiles via `SchemaBuilder`
|
||||
- Diversity enforcement (max N per category/creator)
|
||||
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more
|
||||
|
||||
**Search (SEARCH query)**
|
||||
- BM25 full-text search via Tantivy
|
||||
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
|
||||
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
|
||||
- Creator search with `entity_kind(EntityKind::Creator)`
|
||||
- `similar_to(EntityId)` for content-based recommendations
|
||||
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
|
||||
- Autocomplete suggestions via `db.suggest()`
|
||||
|
||||
**Entity Model**
|
||||
- Three built-in entity types: `Item`, `User`, `Creator`
|
||||
- Metadata storage as `HashMap<String, String>`
|
||||
- Embedding slots (up to 4 per entity type) via USearch
|
||||
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`
|
||||
|
||||
**Sessions**
|
||||
- Session lifecycle: `open_session`, `close_session`
|
||||
- Cross-session preference vector updates (EMA blend)
|
||||
- Session snapshots with signal state and preference vectors
|
||||
- Session serialization format v0x03 with backward compatibility
|
||||
|
||||
**Social Graph**
|
||||
- Creator follower/following indexes
|
||||
- Cohort membership (user segments)
|
||||
- CoEngagementIndex for co-viewing patterns with LRU eviction
|
||||
- Social graph filter for "followed creator" content scoping
|
||||
|
||||
**Collections**
|
||||
- Named collections with `Private`, `Shared`, `Public` visibility
|
||||
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
|
||||
- `FilterExpr::InCollection` for collection-scoped retrieval
|
||||
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`
|
||||
|
||||
**Observability**
|
||||
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
|
||||
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
|
||||
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
|
||||
- `TidalDb::diagnostics()` -- structured health snapshot
|
||||
- WAL diagnostics and recovery tools
|
||||
|
||||
**Safety**
|
||||
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
|
||||
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
|
||||
- Export request limit: 500K signals max per request
|
||||
- `FilterExpr` complexity limit: 256 nodes max
|
||||
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
|
||||
- Schema fingerprint persistence detects decay parameter changes on reopen
|
||||
- Bounded `closed_sessions` cache (10K max, LRU eviction)
|
||||
- Metrics server non-loopback bind warning
|
||||
|
||||
**CLI (`tidalctl`)**
|
||||
- `tidalctl` binary for database inspection and diagnostics
|
||||
|
||||
**RLHF / ML Export**
|
||||
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
|
||||
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics
|
||||
|
||||
### Stability
|
||||
|
||||
tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.
|
||||
|
||||
---
|
||||
|
||||
*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*
|
||||
171
tidal/CLAUDE.md
171
tidal/CLAUDE.md
@ -1,139 +1,56 @@
|
||||
> **Jon Gjengset:** I don't ship what I wouldn't trust at 3am during a production incident.
|
||||
> Pay attention to what the user says and follow it. Do not make them repeat themselves.
|
||||
# tidaldb crate
|
||||
|
||||
# tidalDB
|
||||
This is the **`tidaldb` engine crate** — the embeddable database at `tidal/` within the
|
||||
standalone tidalDB workspace. It is one crate among workspace siblings (`tidal-net/`,
|
||||
`tidal-server/`, `tidalctl/`); it does **not** carry its own copy of the project docs.
|
||||
|
||||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||||
## Canonical docs live at the workspace root — not here
|
||||
|
||||
**Status:** Implemented (M0–M8 shipped). The engine is the `tidaldb` crate at `tidal/`, with companion crates `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings. ~55K LOC, 1,209 unit tests + 188 integration tests + 5 crash-invariant property tests passing. The API surface is stable for implemented features; breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and known gaps.
|
||||
There is exactly one documentation home: the repository root and `docs/`. Do **not**
|
||||
re-create a `tidal/docs/`, `tidal/ai-lookup/`, `tidal/site/`, or `tidal/README.md`
|
||||
— those were a stale mirror and were consolidated away. Read and edit the canonical files:
|
||||
|
||||
## Find Your Guide
|
||||
| For… | Read |
|
||||
|------|------|
|
||||
| Project instructions, structure, agents, skills, rules | [`../CLAUDE.md`](../CLAUDE.md) |
|
||||
| Vision / use cases / sequences / architecture | [`../VISION.md`](../VISION.md), [`../USE_CASES.md`](../USE_CASES.md), [`../SEQUENCE.md`](../SEQUENCE.md), [`../ARCHITECTURE.md`](../ARCHITECTURE.md) |
|
||||
| API / quickstart / coding standards / lessons | [`../API.md`](../API.md), [`../QUICKSTART.md`](../QUICKSTART.md), [`../CODING_GUIDELINES.md`](../CODING_GUIDELINES.md), [`../thoughts.md`](../thoughts.md) |
|
||||
| Component specs (00–14), research, planning, ops, runbooks | [`../docs/`](../docs/) |
|
||||
| Domain concept reference | [`../ai-lookup/index.md`](../ai-lookup/index.md) |
|
||||
| Milestone history + known gaps | [`../CHANGELOG.md`](../CHANGELOG.md) |
|
||||
|
||||
| If you need to... | Read this |
|
||||
|-------------------|-----------|
|
||||
| **Get started quickly** | [README.md](README.md) → [docs/QUICKSTART.md](docs/QUICKSTART.md) |
|
||||
| **Understand the vision** | [docs/VISION.md](docs/VISION.md) |
|
||||
| **See use cases and surfaces** | [docs/USE_CASES.md](docs/USE_CASES.md) |
|
||||
| **See sequence diagrams** | [docs/SEQUENCE.md](docs/SEQUENCE.md) |
|
||||
| **Understand the system architecture** | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) |
|
||||
| **Read the component specs** | [docs/specs/](docs/specs/) (00–14) |
|
||||
| **Look up domain concepts** | [ai-lookup/index.md](ai-lookup/index.md) |
|
||||
| **Follow coding standards** | [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) |
|
||||
| **See the API spec** | [docs/API.md](docs/API.md) |
|
||||
| **Read architectural lessons** | [docs/thoughts.md](docs/thoughts.md) |
|
||||
| **Read technical research** | [docs/research/](docs/research/) |
|
||||
## Crate-local layout (`tidal/src/`, flat module layout)
|
||||
|
||||
## Agents
|
||||
| Module | Responsibility |
|
||||
|--------|----------------|
|
||||
| `cohort/` | Cohort-scoped signal aggregation |
|
||||
| `db/` | Top-level `TidalDb` + builder |
|
||||
| `entities/` | Item / User / Creator entity model |
|
||||
| `load/` | Bulk load / ingest paths |
|
||||
| `query/` | Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST) |
|
||||
| `ranking/` | Profile engine, signal scoring, diversity enforcement |
|
||||
| `replication/` | WAL-stream replication for cluster mode |
|
||||
| `schema/` | Schema builder, validation, signal/profile defs |
|
||||
| `session/` | Session + agent context, policies |
|
||||
| `signals/` | Signal types, decay, velocity, windowed aggregation |
|
||||
| `storage/` | Entity store, signal ledger, inverted index, HNSW |
|
||||
| `testing/` | Shared test harness + fixtures |
|
||||
| `text/` | Tantivy full-text indexing |
|
||||
| `wal/` | Write-ahead log + crash recovery |
|
||||
|
||||
| Agent | Identity | Use when |
|
||||
|-------|----------|----------|
|
||||
| **@tidal-engineer** | Jon Gjengset | Implementing features, designing storage internals, building the signal system, debugging correctness issues |
|
||||
| **@tidal-visionary** | Spencer Kimball | Planning roadmaps, defining milestones, scoping phases, making build-vs-defer decisions |
|
||||
| **@tidal-researcher** | Andy Pavlo | Investigating best practices, surveying prior art, evaluating libraries, producing research documents |
|
||||
| **@tidal-distributed** | Kyle Kingsbury | Building network transports, cluster coordination, multi-node deployment, cross-node query routing, HA |
|
||||
| **@tidal-storyteller** | — | Building the marketing site, writing blog posts, crafting public-facing copy |
|
||||
`benches/`, `examples/` (quickstart, axum_embedding, actix_embedding, cli_embedding),
|
||||
`tests/` (integration + crash-property), and `docker/` (cluster / standalone / deploy)
|
||||
round out the crate.
|
||||
|
||||
## Skills
|
||||
## Build & test (workspace-aware)
|
||||
|
||||
### Phase Lifecycle
|
||||
Run cargo from the **workspace root**, targeting this crate with `-p tidaldb`
|
||||
(equivalently `--manifest-path tidal/Cargo.toml`):
|
||||
|
||||
| Step | Skill | Use when |
|
||||
|------|-------|----------|
|
||||
| 1. Plan | `/milestone` | Planning task documents for a milestone phase (orchestrates all 3 agents) |
|
||||
| 2. Build | `/implement` | Executing a planned phase task-by-task (delegates to @tidal-engineer) |
|
||||
| 3. Review | `/review` | Reviewing completed phase against spec and coding standards (delegates to @tidal-engineer) |
|
||||
| 4. Accept | `/uat` | User acceptance testing a reviewed phase (delegates to @tidal-engineer) |
|
||||
|
||||
### Other Skills
|
||||
|
||||
| Skill | Use when |
|
||||
|-------|----------|
|
||||
| `/tidal-deliver-task` | End-to-end feature delivery orchestrating all 4 agents (scope -> research -> build -> review -> accept) |
|
||||
| `/tidal-verify-completion-to-spec` | Joint spec verification from all 3 agent lenses in parallel (product fit, research grounding, implementation correctness) — use any time, not just after /implement |
|
||||
| `/develop` | Quick implementation work outside the milestone lifecycle |
|
||||
| `/research [topic]` | Investigating best practices, evaluating approaches (delegates to @tidal-researcher) |
|
||||
| `/roadmap` | Building or updating the milestone roadmap (delegates to @tidal-visionary) |
|
||||
| `/build-site` | Creating or iterating on the marketing site |
|
||||
| `/write-blog` | Writing blog posts about progress or architecture |
|
||||
| `/distribute` | Building multi-node distributed system (network transport, cluster mode, cross-node queries) |
|
||||
|
||||
## Core Domain Model
|
||||
|
||||
- **Entities:** Items (content), Users, Creators — each with metadata, embedding slot, signal ledger
|
||||
- **Signals:** Typed, timestamped event streams with native decay, velocity, and windowed aggregation
|
||||
- **Relationships:** Weighted, directional edges between entities (follows, blocks, interactions)
|
||||
- **Ranking Profiles:** Named, versioned scoring functions declared in schema
|
||||
- **Query:** Single operation combining retrieval, filtering, ranking, and diversity enforcement
|
||||
|
||||
## Ports
|
||||
|
||||
Dev servers use port range **59520–59529** (e.g. `site/` on 59520). Any new tidalDB-adjacent service should claim a slot from this band.
|
||||
|
||||
## Critical Rules
|
||||
|
||||
- **Scope:** This is NOT a general-purpose database. Every decision serves one question: "given a user and a context, what content should they see, in what order?"
|
||||
- **Embeddings:** The database retrieves and ranks over vectors. It does NOT generate them.
|
||||
- **Signals are primitives:** Decay, velocity, and windowed aggregation are native — not application logic.
|
||||
- **Single-node first:** Embeddable. Scales vertically before horizontally.
|
||||
- **Language:** Rust.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
This repository is a Cargo workspace. The embeddable database engine is the `tidaldb` crate at `tidal/`, with three companion crates as workspace siblings and example consumers under `applications/`:
|
||||
|
||||
```
|
||||
. (workspace root)
|
||||
├── Cargo.toml # Workspace manifest
|
||||
├── tidal/ # The embeddable database engine — package name = "tidaldb"
|
||||
│ ├── Cargo.toml
|
||||
│ ├── CLAUDE.md # This file — project instructions
|
||||
│ ├── README.md # Public README + getting-started paths
|
||||
│ ├── CHANGELOG.md # Milestone history (M0–M8) + known gaps
|
||||
│ ├── CONTRIBUTING.md # Contributor workflow
|
||||
│ ├── AGENTS.md # Agent roster + SDLC block
|
||||
│ ├── src/ # Flat module layout
|
||||
│ │ ├── cohort/ # Cohort-scoped signal aggregation
|
||||
│ │ ├── db/ # Top-level TidalDb + builder
|
||||
│ │ ├── entities/ # Item / User / Creator entity model
|
||||
│ │ ├── load/ # Bulk load / ingest paths
|
||||
│ │ ├── query/ # Query parser, planner, executor (RETRIEVE/SEARCH/SUGGEST)
|
||||
│ │ ├── ranking/ # Profile engine, signal scoring, diversity enforcement
|
||||
│ │ ├── replication/ # WAL-stream replication for cluster mode
|
||||
│ │ ├── schema/ # Schema builder, validation, signal/profile defs
|
||||
│ │ ├── session/ # Session + agent context, policies
|
||||
│ │ ├── signals/ # Signal types, decay, velocity, windowed aggregation
|
||||
│ │ ├── storage/ # Entity store, signal ledger, inverted index, HNSW
|
||||
│ │ ├── testing/ # Shared test harness + fixtures
|
||||
│ │ ├── text/ # Tantivy full-text indexing
|
||||
│ │ └── wal/ # Write-ahead log + crash recovery
|
||||
│ ├── benches/ # Performance benchmarks
|
||||
│ ├── examples/ # quickstart, axum_embedding, actix_embedding, cli_embedding
|
||||
│ ├── tests/ # Integration and crash-property tests
|
||||
│ ├── docker/ # cluster / standalone / deploy Dockerfiles
|
||||
│ ├── ai-lookup/ # Domain concept reference (index.md)
|
||||
│ ├── site/ # Public marketing site (Next.js)
|
||||
│ └── docs/ # Specs, API, guides, research, runbooks, ops
|
||||
│ ├── VISION.md ARCHITECTURE.md API.md
|
||||
│ ├── QUICKSTART.md SEQUENCE.md USE_CASES.md
|
||||
│ ├── CODING_GUIDELINES.md thoughts.md
|
||||
│ ├── specs/ # 00–14 component specifications
|
||||
│ ├── research/ # Deep technical research docs
|
||||
│ ├── runbooks/ # Operational runbooks (cluster, recovery)
|
||||
│ ├── ops/ # Monitoring, alerts, capacity planning
|
||||
│ ├── profiling/ # Flamegraph + scale profiling notes
|
||||
│ └── planning/ # Per-milestone phase/task planning archive
|
||||
├── tidal-net/ # Network transport primitives
|
||||
├── tidal-server/ # Standalone Axum HTTP server (standalone + cluster modes)
|
||||
├── tidalctl/ # CLI tool for inspecting persisted databases
|
||||
└── applications/ # Example consumers (forage, iknowyou)
|
||||
```bash
|
||||
cargo test -p tidaldb # unit + integration
|
||||
cargo clippy -p tidaldb -D warnings
|
||||
cargo fmt
|
||||
```
|
||||
|
||||
## Pre-commit Hooks
|
||||
|
||||
The pre-commit hook runs automatically on staged files:
|
||||
- **Rust:** `cargo fmt` (auto-fix + re-stage), `cargo clippy -p tidaldb -D warnings`, `cargo test -p tidaldb --lib`
|
||||
- **site/ (Next.js):** `eslint` (if node_modules installed)
|
||||
|
||||
Cargo commands target this crate with `-p tidaldb` from the workspace root (e.g. `cargo test -p tidaldb`, `cargo check -p tidaldb`), or equivalently `--manifest-path tidal/Cargo.toml`. The engine crate lives at `tidal/`.
|
||||
|
||||
**Tests must be fast.** Slow or hanging tests are bugs — diagnose root cause, then remove, fix, or refactor; never leave them hanging.
|
||||
**Tests must be fast.** Slow or hanging tests are bugs — fix the root cause, never leave them hanging.
|
||||
|
||||
@ -1,104 +0,0 @@
|
||||
# Contributing to tidalDB
|
||||
|
||||
## Quick Start
|
||||
|
||||
tidalDB is the `tidaldb` crate at `tidal/` in this Cargo workspace. Run cargo
|
||||
commands from the workspace root, targeting this crate with `-p tidaldb`.
|
||||
|
||||
```bash
|
||||
# From the workspace root:
|
||||
|
||||
# Confirm the engine compiles and all tests pass
|
||||
cargo test -p tidaldb
|
||||
|
||||
# Confirm doc tests and examples compile and run
|
||||
cargo test -p tidaldb --doc
|
||||
cargo test -p tidaldb --examples
|
||||
```
|
||||
|
||||
## Run Samples Checklist
|
||||
|
||||
Before opening a PR that touches public API or examples, verify all samples still work:
|
||||
|
||||
```bash
|
||||
# Doc tests (default features)
|
||||
cargo test -p tidaldb --doc
|
||||
|
||||
# Doc tests with optional features
|
||||
cargo test -p tidaldb --doc --features test-utils,metrics
|
||||
|
||||
# All four examples compile and run
|
||||
cargo run -p tidaldb --example quickstart
|
||||
cargo run -p tidaldb --example cli_embedding
|
||||
cargo run -p tidaldb --example axum_embedding # Ctrl+C to stop
|
||||
cargo run -p tidaldb --example actix_embedding # Ctrl+C to stop
|
||||
```
|
||||
|
||||
Expected output for `quickstart`:
|
||||
|
||||
```
|
||||
build: dev
|
||||
uptime: 0.000s
|
||||
health: ok
|
||||
tidalDB opened, verified, and closed. M0 complete.
|
||||
```
|
||||
|
||||
## Full Quality Gate
|
||||
|
||||
The pre-commit hook enforces these automatically on staged Rust files:
|
||||
|
||||
```bash
|
||||
cargo fmt -p tidaldb -- --check
|
||||
cargo clippy -p tidaldb -- -D warnings
|
||||
cargo test -p tidaldb --lib
|
||||
```
|
||||
|
||||
Run the complete gate manually:
|
||||
|
||||
```bash
|
||||
cargo fmt -p tidaldb
|
||||
cargo clippy -p tidaldb -- -D warnings
|
||||
cargo test -p tidaldb
|
||||
cargo bench -p tidaldb --no-run # ensure benches compile
|
||||
```
|
||||
|
||||
## Project Layout
|
||||
|
||||
tidalDB is the `tidaldb` crate at `tidal/` in this Cargo workspace. Companion
|
||||
crates `tidal-net`, `tidal-server`, and `tidalctl` are workspace siblings at
|
||||
the repository root.
|
||||
|
||||
```
|
||||
tidal/
|
||||
src/ Flat module layout (engine source)
|
||||
cohort/ Cohort-scoped signal aggregation
|
||||
db/ TidalDb handle, builder, config, metrics
|
||||
entities/ Item / User / Creator entity model
|
||||
load/ Bulk load / ingest paths
|
||||
query/ RETRIEVE / SEARCH / SUGGEST parser, planner, executor
|
||||
ranking/ Profile engine, signal scoring, diversity enforcement
|
||||
replication/ WAL-stream replication (cluster mode)
|
||||
schema/ Schema builder, validation, signal/profile defs, error types
|
||||
session/ Session + agent context, policies
|
||||
signals/ Signal ledger, decay, velocity, windowed counters, checkpoint
|
||||
storage/ StorageEngine trait, fjall backend, key encoding
|
||||
testing/ Shared test harness + fixtures
|
||||
text/ Tantivy full-text indexing
|
||||
wal/ Write-ahead log, group commit, crash recovery
|
||||
benches/ Criterion benchmarks
|
||||
examples/ Embedding guides (quickstart, axum, actix, cli)
|
||||
tests/ Integration and crash-property tests
|
||||
docker/ cluster / standalone / deploy Dockerfiles
|
||||
site/ Marketing site (Next.js)
|
||||
docs/ Specs, API, guides, research, runbooks, ops, planning
|
||||
```
|
||||
|
||||
## Coding Standards
|
||||
|
||||
See [docs/CODING_GUIDELINES.md](docs/CODING_GUIDELINES.md) for the full engineering standards.
|
||||
|
||||
Key rules:
|
||||
- `Result<T, LumenError>` everywhere — no panics on recoverable failures
|
||||
- `#![forbid(unsafe_code)]` — relaxed only at explicit FFI boundaries with `// SAFETY:` comment
|
||||
- Property tests for invariants, criterion benchmarks for performance claims
|
||||
- `cargo clippy -D warnings` must pass with zero warnings
|
||||
285
tidal/README.md
285
tidal/README.md
@ -1,285 +0,0 @@
|
||||
# tidalDB
|
||||
|
||||
**An embeddable Rust database for the personalized content ranking problem.**
|
||||
|
||||
> Pre-release. API is stabilizing. Not yet recommended for production.
|
||||
|
||||
---
|
||||
|
||||
Every content platform eventually builds the same distributed system from scratch: Elasticsearch for retrieval, Redis for hot signals, Kafka for event ingestion, a feature store for user profiles, a vector database for semantic search, and a ranking service that stitches them together. The seams between those systems are where correctness dies — stale signals, inconsistent ranking, cache invalidation bugs, ETL lag.
|
||||
|
||||
The root cause: existing databases treat ranking as an afterthought. They have no native concept of signals that evolve over time, no understanding of user context, no diversity as a query constraint.
|
||||
|
||||
**Ranking is not a feature. It is a primitive.**
|
||||
|
||||
tidalDB is a single-node, embeddable Rust library built for one question: *given a user and a context, what content should they see, and in what order?* No server, no network protocol, no client SDK. Link it into your process.
|
||||
|
||||
---
|
||||
|
||||
## What it looks like
|
||||
|
||||
```rust
|
||||
use std::collections::HashMap;
|
||||
use std::time::Duration;
|
||||
use tidaldb::{TidalDb, query::retrieve::Retrieve, schema::{DecaySpec, EntityId, EntityKind, SchemaBuilder, Timestamp, Window}};
|
||||
|
||||
// Declare signals with native decay — no application formulas.
|
||||
let mut schema = SchemaBuilder::new();
|
||||
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
|
||||
half_life: Duration::from_secs(7 * 24 * 3600),
|
||||
}).windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
|
||||
let _ = schema.signal("like", EntityKind::Item, DecaySpec::Exponential {
|
||||
half_life: Duration::from_secs(30 * 24 * 3600),
|
||||
}).windows(&[Window::AllTime]).velocity(false).add();
|
||||
let schema = schema.build()?;
|
||||
|
||||
// Open — ephemeral for tests, persistent for production.
|
||||
let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
|
||||
|
||||
// Ingest content with metadata.
|
||||
let mut meta = HashMap::new();
|
||||
meta.insert("title".to_string(), "Introduction to Jazz Piano".to_string());
|
||||
meta.insert("category".to_string(), "music".to_string());
|
||||
db.write_item_with_metadata(EntityId::new(1), &meta)?;
|
||||
|
||||
// Write an embedding (you generate it, tidalDB indexes and ranks over it).
|
||||
db.write_item_embedding(EntityId::new(1), &your_model.embed("Introduction to Jazz Piano"))?;
|
||||
|
||||
// Record engagement — the feedback loop closes here, no ETL required.
|
||||
db.signal("view", EntityId::new(1), 1.0, Timestamp::now())?;
|
||||
db.signal_with_context("like", EntityId::new(1), 1.0, Timestamp::now(), Some(user_id), Some(creator_id))?;
|
||||
|
||||
// Retrieve a ranked feed. Name the profile. tidalDB executes the pipeline.
|
||||
let results = db.retrieve(&Retrieve::builder().for_user(user_id).profile("for_you").limit(50).build()?)?;
|
||||
|
||||
// Search: BM25 + semantic similarity fused via RRF.
|
||||
let results = db.search(&Search::builder().query("jazz piano tutorial").for_user(user_id).limit(20).build()?)?;
|
||||
|
||||
db.close()?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What it replaces
|
||||
|
||||
| System | tidalDB equivalent |
|
||||
|--------|--------------------|
|
||||
| Elasticsearch | Tantivy BM25 text index (derived, crash-recoverable) |
|
||||
| Redis | Lock-free in-memory signal ledger — decay scores, windowed counters |
|
||||
| Kafka | Write-ahead log — durable, ordered, replayable |
|
||||
| Feature store | Signal aggregates + user preference vectors (updated at write time) |
|
||||
| Vector DB | USearch HNSW — embedded, f16 quantized, predicate-filtered ANN |
|
||||
| Ranking service | 25 named profiles, scored at query time, swappable by name |
|
||||
|
||||
---
|
||||
|
||||
## Key capabilities
|
||||
|
||||
- **Signals with native decay** — declare `view` with a 7-day half-life; the database applies it at query time. No `trending_score_7d` field to maintain.
|
||||
- **25 built-in ranking profiles** — `trending`, `hot`, `for_you`, `following`, `related`, `hidden_gems`, `top_week`, `shuffle`, `controversial`, and more. Name the profile; the database executes the full pipeline.
|
||||
- **Hybrid search** — BM25 full-text + ANN semantic similarity, fused via Reciprocal Rank Fusion, personalized by user preference vector.
|
||||
- **Composable filters** — filter by category, format, duration, language, engagement threshold, location, collection membership, and more — any combination, all composable.
|
||||
- **Diversity as a query constraint** — `max_per_creator: 2` belongs in the query, not your API layer.
|
||||
- **Feedback loop in the write path** — a signal write atomically updates the item's ledger, the user's preference vector, and relationship weights. The next ranking query — 100ms later — reflects it.
|
||||
- **Cold start handled** — new content gets an exploration budget; new users get sensible defaults. No application logic required.
|
||||
- **Cohort-scoped trending** — "trending among US users aged 18-24 who engage with jazz" is one query, not a pipeline.
|
||||
- **Embeddable first** — runs in your process. `Arc<TidalDb>` is `Send + Sync`. No operational overhead.
|
||||
|
||||
---
|
||||
|
||||
## Getting started
|
||||
|
||||
Pick the path that matches how you plan to use tidalDB today. Every option below is self-contained and ships in this repo.
|
||||
|
||||
### 1. Embed tidalDB inside your Rust service (library mode)
|
||||
|
||||
**Setup**
|
||||
|
||||
1. Add the dependency. Depend on the `tidaldb` crate (at `tidal/` in this repository) by path or git:
|
||||
```toml
|
||||
[dependencies]
|
||||
tidaldb = { path = "../tidaldb/tidal" } # adjust to the tidal/ crate's location, or use git = "https://github.com/orchard9/tidaldb"
|
||||
```
|
||||
2. Define your schema before opening the database (decay, windows, text fields, embeddings). The snippet in **[Quickstart, Step 2](docs/QUICKSTART.md#step-2-define-a-schema)** is a ready-to-copy template.
|
||||
3. Choose storage mode when building:
|
||||
```rust
|
||||
let db = tidaldb::TidalDb::builder()
|
||||
.with_schema(schema)
|
||||
.ephemeral() // in-memory for tests
|
||||
// .with_data_dir("/var/lib/tidaldb") // persistent deployment
|
||||
.open()?;
|
||||
```
|
||||
4. Run the end-to-end sample:
|
||||
```bash
|
||||
cargo run -p tidaldb --example quickstart
|
||||
```
|
||||
|
||||
**Usage**
|
||||
|
||||
- Call `db.signal(...)`, `db.signal_with_context(...)`, and `db.retrieve(...)` / `db.search(...)` from the same process; no network stack required.
|
||||
- Wrap the instance in `Arc<TidalDb>` to share it across threads or tasks.
|
||||
- Persisted deployments can be inspected with the CLI tool: `cargo run -p tidalctl -- status --path /var/lib/tidaldb`.
|
||||
- Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)** and **[docs/API.md](docs/API.md)**.
|
||||
|
||||
### 2. Run the standalone HTTP server (`tidal-server`)
|
||||
|
||||
**Why:** you want a ready-to-run HTTP facade without writing Axum/Actix glue.
|
||||
|
||||
```bash
|
||||
cargo run -p tidal-server -- \
|
||||
standalone \
|
||||
--listen 127.0.0.1:9400 \
|
||||
--schema tidal-server/config/default-schema.yaml
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--data-dir /var/lib/tidaldb` switches to persistent storage.
|
||||
- Provide your own schema file (YAML) to match your signal mix.
|
||||
|
||||
Usage:
|
||||
|
||||
```bash
|
||||
# register metadata + embedding
|
||||
curl -X POST http://127.0.0.1:9400/items \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{ "entity_id": 1, "metadata": { "title": "Jazz Piano", "category": "music" } }'
|
||||
curl -X POST http://127.0.0.1:9400/embeddings \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{ "entity_id": 1, "values": [0.1, 0.2, 0.3] }'
|
||||
|
||||
# write engagement (supports user/creator context)
|
||||
curl -X POST http://127.0.0.1:9400/signals \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0, "user_id": 42 }'
|
||||
|
||||
# query
|
||||
curl "http://127.0.0.1:9400/feed?user_id=42&profile=for_you&limit=20"
|
||||
curl "http://127.0.0.1:9400/search?query=jazz%20piano&user_id=42&limit=5"
|
||||
curl http://127.0.0.1:9400/health
|
||||
```
|
||||
|
||||
The default schema lives at `tidal-server/config/default-schema.yaml`. Edit
|
||||
it (or provide your own path) to align with your application’s signals,
|
||||
text fields, and embedding slots.
|
||||
|
||||
### 3. Wrap it in an HTTP service you control
|
||||
|
||||
Expose tidalDB through your favorite web framework; the repo ships runnable templates.
|
||||
|
||||
- **Axum sample (`examples/axum_embedding.rs`)**
|
||||
```bash
|
||||
cargo run -p tidaldb --example axum_embedding
|
||||
```
|
||||
Usage:
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:3000/signal \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
|
||||
curl "http://127.0.0.1:3000/feed?user_id=42"
|
||||
curl http://127.0.0.1:3000/health
|
||||
```
|
||||
The example handles schema setup, wraps `Arc<TidalDb>` in Axum `State`, and maps `TidalError` to HTTP responses.
|
||||
|
||||
- **Actix sample (`examples/actix_embedding.rs`)**
|
||||
```bash
|
||||
cargo run -p tidaldb --example actix_embedding
|
||||
# curl http://127.0.0.1:3001/health
|
||||
```
|
||||
Demonstrates sharing `Arc<TidalDb>` through `web::Data` and using Actix’s shutdown hooks.
|
||||
|
||||
Use either sample as a starting point for microservices that prefer a client/server boundary.
|
||||
|
||||
### 4. Run the cluster server + Docker image
|
||||
|
||||
Need a single endpoint that fronts the built-in simulated cluster? Use
|
||||
`tidal-server` in `cluster` mode. It spins up the multi-region fabric,
|
||||
ships WAL batches between regions, and exposes `/signals`, `/feed`,
|
||||
`/search` plus cluster-management routes.
|
||||
|
||||
```bash
|
||||
cargo run -p tidal-server -- \
|
||||
cluster \
|
||||
--listen 0.0.0.0:9500 \
|
||||
--schema tidal-server/config/default-schema.yaml \
|
||||
--topology tidal-server/config/default-cluster.yaml
|
||||
```
|
||||
|
||||
Key endpoints:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:9500/health
|
||||
curl -X POST http://127.0.0.1:9500/signals -d '{ "entity_id": 1, "signal": "view", "weight": 1.0 }'
|
||||
curl "http://127.0.0.1:9500/feed?profile=trending®ion=eu-west"
|
||||
curl http://127.0.0.1:9500/cluster/status
|
||||
curl -X POST http://127.0.0.1:9500/cluster/promote -d '{ "region": "eu-west" }'
|
||||
```
|
||||
|
||||
Cluster mode currently replicates global signals (no `user_id` /
|
||||
`creator_id` contexts) so that followers can stay in sync with the leader’s
|
||||
WAL stream. See **[docs/runbooks/cluster.md](docs/runbooks/cluster.md)** for
|
||||
operational steps, failure drills, and API references.
|
||||
|
||||
Prefer containers? Build the provided image and run it anywhere:
|
||||
|
||||
```bash
|
||||
docker build -f docker/cluster/Dockerfile -t tidal-cluster .
|
||||
docker run --rm -p 9500:9500 tidal-cluster
|
||||
```
|
||||
|
||||
Mount your own schema/topology files with `-v` if you want different regions
|
||||
or signal definitions.
|
||||
|
||||
### 5. Simulate a multi-region cluster in tests
|
||||
|
||||
The raw `SimulatedCluster` harness (no HTTP) remains available for property
|
||||
tests and fuzzing.
|
||||
|
||||
```bash
|
||||
cargo test --test m8_uat
|
||||
cargo test --test m8_uat uat_step3 -- --nocapture # run a single scenario
|
||||
```
|
||||
|
||||
Tweak `tests/m8_uat.rs` to script specific replication, failover, and
|
||||
migration scenarios inside your own test suites.
|
||||
|
||||
**MSRV:** Rust 1.91
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Contents |
|
||||
|----------|----------|
|
||||
| [docs/QUICKSTART.md](docs/QUICKSTART.md) | Step-by-step guide: schema, ingest, signals, ranking, search |
|
||||
| [docs/API.md](docs/API.md) | Full API reference with code examples |
|
||||
| [docs/VISION.md](docs/VISION.md) | Problem statement and design thesis |
|
||||
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Storage, signal system, vector index, query pipeline |
|
||||
| [docs/USE_CASES.md](docs/USE_CASES.md) | 14 content discovery surfaces, filter and sort references |
|
||||
| [docs/specs/](docs/specs/) | Component specifications (00–14) |
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
Milestones completed:
|
||||
|
||||
- Storage engine, WAL, entity store, signal ledger
|
||||
- RETRIEVE query: candidate retrieval, filtering, scoring, diversity, pagination
|
||||
- Vector index (USearch HNSW) with adaptive filtered search
|
||||
- 25 built-in ranking profiles
|
||||
- BM25 full-text search (Tantivy) + hybrid RRF fusion
|
||||
- Creator search and creator profiles
|
||||
- Cohort-scoped signal aggregation and trending
|
||||
- Social graph (follows, blocks, following feed)
|
||||
- Collections, saved searches, autocomplete suggestions
|
||||
- Session and agent context (short-lived signals, preference decay)
|
||||
- Crash recovery, graceful degradation, rate limiting, diagnostics
|
||||
- Scale: tested to 1M items; scale benchmarks passing
|
||||
|
||||
The API surface is stable for the implemented features. Breaking changes are possible before 1.0.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@ -1,34 +0,0 @@
|
||||
# Filters
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
All filter dimensions are composable — any combination produces a valid, efficiently-executed query. Filters are first-class query citizens, not application logic.
|
||||
|
||||
**Key Facts:**
|
||||
- Any filter can combine with any other filter and any sort mode
|
||||
- Faceted queries are first-class operations
|
||||
- Filter categories: content attributes, date/time, creator, engagement thresholds, user state, geography
|
||||
- Multi-select within a dimension uses OR logic (Jazz OR Blues)
|
||||
- Cross-dimension uses AND logic (Jazz AND short AND this week)
|
||||
|
||||
**File Pointer:** `USE_CASES.md:536-628` (Appendix A)
|
||||
|
||||
## Filter Categories
|
||||
|
||||
| Category | Example Dimensions |
|
||||
|----------|-------------------|
|
||||
| Content | category, tag, format, duration, language, resolution, content_rating |
|
||||
| Date/Time | created_within, created_preset (hour/today/week/month/year) |
|
||||
| Creator | creator, verified, followed_by_user, new_to_user, follower count range |
|
||||
| Engagement | min_views, min_likes, min_completion_rate, min_like_ratio |
|
||||
| User State | seen/unseen, in_progress, saved, liked, downloaded, in_collection |
|
||||
| Geographic | content_region, trending_in_region, near_location |
|
||||
| Availability | free, premium, subscriber_only, downloadable, leaving_soon |
|
||||
| Accessibility | has_subtitles, has_audio_description, has_sign_language |
|
||||
|
||||
## Related Topics
|
||||
- [Query Language](./query-language.md)
|
||||
- [Sort Modes](./sort-modes.md)
|
||||
@ -1,65 +0,0 @@
|
||||
# Query Language
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
The query interface is a single operation that encapsulates candidate retrieval, filtering, ranking, and diversity enforcement. Three primary operations: RETRIEVE (feed/browse), SEARCH (text+semantic), and SIGNAL (engagement write-back).
|
||||
|
||||
**Key Facts:**
|
||||
- One query replaces what currently requires 6 systems
|
||||
- RETRIEVE: feed generation, browse, related content
|
||||
- SEARCH: keyword + semantic + hybrid retrieval
|
||||
- SIGNAL: engagement event write-back (closes the feedback loop in the same transaction)
|
||||
- All queries accept: FOR USER, USING PROFILE, FILTER, DIVERSITY, LIMIT
|
||||
- Filters are composable — any combination is valid
|
||||
|
||||
**File Pointer:** `VISION.md:47-57`
|
||||
|
||||
## Query Shapes
|
||||
|
||||
**Feed retrieval:**
|
||||
```
|
||||
RETRIEVE items
|
||||
FOR USER @user_id
|
||||
CONTEXT feed
|
||||
USING PROFILE for_you
|
||||
FILTER unseen, unblocked, format:video
|
||||
DIVERSITY max_per_creator:2, format_mix:true
|
||||
LIMIT 50
|
||||
```
|
||||
|
||||
**Search:**
|
||||
```
|
||||
SEARCH items
|
||||
QUERY "rust tutorial beginner"
|
||||
VECTOR query_vector
|
||||
FOR USER @user_id
|
||||
USING PROFILE search
|
||||
DIVERSITY max_per_creator:2
|
||||
LIMIT 20
|
||||
```
|
||||
|
||||
**Signal write:**
|
||||
```
|
||||
SIGNAL like
|
||||
item: @item_id
|
||||
user: @user_id
|
||||
timestamp: now()
|
||||
```
|
||||
|
||||
**Related content:**
|
||||
```
|
||||
RETRIEVE items
|
||||
SIMILAR TO @item_id
|
||||
FOR USER @user_id
|
||||
USING PROFILE related
|
||||
FILTER unseen
|
||||
LIMIT 10
|
||||
```
|
||||
|
||||
## Related Topics
|
||||
- [Ranking Profiles](../services/ranking-profiles.md)
|
||||
- [Filters](./filters.md)
|
||||
- [Sort Modes](./sort-modes.md)
|
||||
@ -1,30 +0,0 @@
|
||||
# Sort Modes
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
25+ native sort modes available on any surface. The application names a sort mode; the database executes it. No application-side sorting logic required.
|
||||
|
||||
**Key Facts:**
|
||||
- Sort modes are built-in, not formulas the application implements
|
||||
- Same sort mode works across different candidate sets (global, category, social graph)
|
||||
- Windowed top sorts: hour, today, week, month, year, all-time
|
||||
- Hot uses Reddit-style age decay: score / (age + 2)^gravity
|
||||
- Trending is pure velocity (rate of change), distinct from Hot (cumulative with decay)
|
||||
- Controversial maximizes product of positive and negative signals
|
||||
|
||||
**File Pointer:** `USE_CASES.md:635-663` (Appendix B)
|
||||
|
||||
## Categories
|
||||
|
||||
**Quality-based:** relevance, personalized, top_* (windowed), hidden_gems
|
||||
**Time-based:** new, old, hot, trending, rising
|
||||
**Engagement-based:** most_viewed, most_liked, most_commented, most_shared
|
||||
**Format-based:** shortest, longest, alphabetical_asc/desc
|
||||
**Special:** shuffle (quality-weighted random), live_viewer_count, date_saved, controversial
|
||||
|
||||
## Related Topics
|
||||
- [Ranking Profiles](../services/ranking-profiles.md)
|
||||
- [Query Language](./query-language.md)
|
||||
@ -1,10 +0,0 @@
|
||||
# AI Lookup Index
|
||||
|
||||
| Topic | File | Confidence | Updated | Summary |
|
||||
|-------|------|------------|---------|---------|
|
||||
| Entities | `services/entities.md` | High | 2026-02-19 | Items, Users, Creators — core data model |
|
||||
| Signals | `services/signals.md` | High | 2026-02-19 | Typed event streams with decay, velocity, windowed aggregation |
|
||||
| Ranking Profiles | `services/ranking-profiles.md` | High | 2026-02-19 | Named scoring functions declared in schema |
|
||||
| Query Language | `features/query-language.md` | High | 2026-02-19 | RETRIEVE/SEARCH/SIGNAL query surface |
|
||||
| Sort Modes | `features/sort-modes.md` | High | 2026-02-19 | 25+ native sort modes (hot, trending, rising, etc.) |
|
||||
| Filters | `features/filters.md` | High | 2026-02-19 | Composable filter dimensions across all queries |
|
||||
@ -1,28 +0,0 @@
|
||||
# Entities
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
Entities are the nodes of the system. Three types: Items (content), Users, and Creators. Every entity has metadata, a vector embedding slot, and an attached signal ledger.
|
||||
|
||||
**Key Facts:**
|
||||
- Items have metadata, embeddings, and signals — signals are typed timestamped streams, not fields
|
||||
- Users have preferences, histories, and relationships — living profiles that update continuously
|
||||
- Creators are linked to Items and have their own embeddings (aggregated from catalog)
|
||||
- Relationships are first-class edges between entities (weighted, directional, traversable)
|
||||
|
||||
**File Pointer:** `VISION.md:36-43`
|
||||
|
||||
## How It Works
|
||||
|
||||
Items enter via the WRITE path with metadata + embedding. A signal ledger is initialized at zero. Cold start exploration budget is applied automatically. Items are immediately queryable after commit.
|
||||
|
||||
Users accumulate implicit preference vectors from engagement history. Preference vectors update on every signal write (like, skip, hide, completion).
|
||||
|
||||
Creators are entities with their own embeddings derived from their item catalog. Creator-level signals include engagement rate, posting frequency, and follower count.
|
||||
|
||||
## Related Topics
|
||||
- [Signals](./signals.md)
|
||||
- [Ranking Profiles](./ranking-profiles.md)
|
||||
@ -1,38 +0,0 @@
|
||||
# Ranking Profiles
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
Ranking profiles are named, versioned scoring functions declared in schema. They reference signals, relationship weights, recency curves, and diversity rules. Profiles live in the database, are versioned alongside data, and can be swapped at query time by name.
|
||||
|
||||
**Key Facts:**
|
||||
- Profiles are schema-level declarations, not application code
|
||||
- Each profile defines: primary signals, secondary signals, boosts, gates, and diversity rules
|
||||
- The same profile can operate on different candidate sets (global vs category vs social graph)
|
||||
- Profiles are versioned — old versions remain queryable
|
||||
|
||||
**File Pointer:** `VISION.md:43-55`
|
||||
|
||||
## Built-in Profiles
|
||||
|
||||
| Profile | Primary Signal | Use Case |
|
||||
|---------|---------------|----------|
|
||||
| `for_you` | preference_match + engagement_velocity | Personalized feed |
|
||||
| `search` | text_relevance + semantic_similarity | Search results |
|
||||
| `trending` | share_velocity + view_velocity | Trending surfaces |
|
||||
| `rising` | velocity relative to baseline, age-boosted | Breakout content |
|
||||
| `following` | created_at DESC | Subscription feed |
|
||||
| `related` | semantic_similarity + collaborative_filtering | Up next / related |
|
||||
| `browse` | quality_score (completion + like_ratio + reach) | Category pages |
|
||||
| `hot` | score / (age + 2)^gravity | Community frontpages |
|
||||
| `controversial` | max(positive * negative signals) | Debate surfaces |
|
||||
| `hidden_gems` | high quality, inverse reach | Discovery |
|
||||
| `notification` | relationship_strength + item quality | Push prioritization |
|
||||
| `live` | relationship_weight + viewer_count | Live surfaces |
|
||||
|
||||
## Related Topics
|
||||
- [Signals](./signals.md)
|
||||
- [Query Language](../features/query-language.md)
|
||||
- [Sort Modes](../features/sort-modes.md)
|
||||
@ -1,33 +0,0 @@
|
||||
# Signals
|
||||
|
||||
**Last Updated:** 2026-02-19
|
||||
**Confidence:** High
|
||||
|
||||
## Summary
|
||||
|
||||
Signals are typed, timestamped event streams attached to entity signal ledgers. The database natively understands signal semantics: velocity (rate of change), decay (exponential or linear, configurable per type), and windowed aggregation (last hour, day, 7 days, all time).
|
||||
|
||||
**Key Facts:**
|
||||
- Signals are NOT fields — they are streams with temporal semantics
|
||||
- Decay half-lives are declared in schema, applied at query time
|
||||
- Velocity is computed natively (rate of new events in a window)
|
||||
- Windowed aggregation: 1h, 24h, 7d, all-time windows are first-class
|
||||
- Negative signals (skip, hide, block) are equal citizens with positive signals
|
||||
- Signal writes are atomic transactions updating item ledger, user preference vector, and relationship weights in one commit
|
||||
|
||||
**File Pointer:** `VISION.md:39-41`, `USE_CASES.md:669-711` (Appendix C)
|
||||
|
||||
## Signal Categories
|
||||
|
||||
| Category | Examples | Decay |
|
||||
|----------|----------|-------|
|
||||
| Positive engagement | view, like, share, completion | slow-medium |
|
||||
| Negative engagement | skip, hide, block, not_interested | fast-permanent |
|
||||
| Relationship | follow, unfollow, interaction_weight | slow-permanent |
|
||||
| Quality | completion ratio, dwell_time, replay | slow-medium |
|
||||
| Recommendation | autoplay_accept/reject, search_click | medium |
|
||||
| Notification | notification_open, notification_dismiss | slow-medium |
|
||||
|
||||
## Related Topics
|
||||
- [Entities](./entities.md)
|
||||
- [Ranking Profiles](./ranking-profiles.md)
|
||||
1072
tidal/docs/API.md
1072
tidal/docs/API.md
File diff suppressed because it is too large
Load Diff
@ -1,487 +0,0 @@
|
||||
# Architecture
|
||||
|
||||
tidalDB is a purpose-built ranking database. Its architecture is shaped by a single constraint: every design decision must serve the question "given a user and a context, what content should they see, in what order?" Nothing else.
|
||||
|
||||
This document describes how the system is structured, why it is structured that way, and how the major subsystems interact. For the API surface, see [API.md](API.md). For engineering standards, see [CODING_GUIDELINES.md](CODING_GUIDELINES.md). For the research behind specific decisions, see [docs/research/](docs/research/).
|
||||
|
||||
---
|
||||
|
||||
## Core Thesis
|
||||
|
||||
Every content platform (YouTube, TikTok, Reddit, Netflix) builds the same 6-system distributed stack from scratch — Elasticsearch, Redis, Kafka, a feature store, a vector DB, and a ranking service. The seams between those systems are where correctness fails: stale signals, inconsistent ranking, cache invalidation bugs, ETL lag.
|
||||
|
||||
The root cause is that existing databases treat ranking as an afterthought. They have no native concept of signals that evolve over time, no understanding of user context, no diversity as a query constraint, and no feedback loop between what users see and what the system learns.
|
||||
|
||||
tidalDB treats ranking as a primitive. Signals, decay, velocity, user preferences, relationships, and diversity are first-class schema concepts — not application logic bolted on top.
|
||||
|
||||
---
|
||||
|
||||
## Domain Model
|
||||
|
||||
Six first-class entity types:
|
||||
|
||||
| Type | What it represents |
|
||||
|------|--------------------|
|
||||
| **Item** | A piece of content (video, article, post) — has metadata, an embedding slot, a signal ledger |
|
||||
| **User** | A viewer — has attributes, a preference vector, a signal ledger, a seen-item set |
|
||||
| **Creator** | An author — has attributes, an embedding slot, a signal ledger |
|
||||
| **Relationship** | A weighted, directional edge between any two entities (follows, blocks, interaction weight) |
|
||||
| **Cohort** | A named, live predicate over user attributes (e.g. `age_range ∈ {18-24} AND locale = en-US`) |
|
||||
| **Session / Agent Context** | A short-lived, agent-scoped memory surface binding a user, agent identity, and session metadata (tools, reward hints, policy) |
|
||||
|
||||
Six schema-level primitives:
|
||||
|
||||
| Primitive | What it captures |
|
||||
|-----------|-----------------|
|
||||
| **Signal** | A typed, timestamped event stream (view, like, skip, hide...) with declared decay rate, velocity, and windowed aggregation |
|
||||
| **Ranking Profile** | A named, versioned scoring function: candidate retrieval strategy, boosts, penalties, quality gates, diversity rules, exploration budget |
|
||||
| **Relationship** | Weighted edges: follows, blocks, interaction strength — used as ranking inputs |
|
||||
| **Cohort** | Live predicate membership — enables cohort-scoped signal aggregation and trending |
|
||||
| **Session** | Agent-scoped conversational context: short-lived signals, reward hints, policy tags, decay curves |
|
||||
| **Filter** | Composable predicates over entity attributes, signal values, and relationship state |
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
The dependency chain is strict. No circular dependencies. Each module knows only about modules beneath it.
|
||||
|
||||
```
|
||||
schema/ ← standalone; no dependencies; defines all types
|
||||
↑
|
||||
storage/ ← depends on schema; knows nothing about signals or ranking
|
||||
↑
|
||||
signals/ ← depends on storage; knows nothing about queries or ranking
|
||||
↑
|
||||
agent/ ← depends on signals; manages sessions, policy, agent APIs
|
||||
↑
|
||||
query/ ← depends on agent + signals; orchestrates execution
|
||||
↑
|
||||
ranking/ ← depends on signals; invoked by the query executor
|
||||
```
|
||||
|
||||
### `schema/`
|
||||
|
||||
The type system. Defines `EntityId`, `SignalDef`, `ProfileDef`, `CohortDef`, `TidalError`, and validation logic. No dependencies. Every other module depends on this one.
|
||||
|
||||
No external crates except `thiserror` (error derives).
|
||||
|
||||
### `storage/`
|
||||
|
||||
The persistence layer. Owns:
|
||||
- **WAL** — the durability boundary. Every write goes here first.
|
||||
- **Entity store** — item, user, creator metadata. Trait-abstracted: `EntityStore`, `SignalLedgerStore`, `RelationshipStore`.
|
||||
- **Key encoding** — `[entity_id: u64 BE][0x00][TAG:suffix]` for co-location and range scans.
|
||||
|
||||
The storage backend (fjall initially) sits behind a trait. No storage engine types leak into higher modules.
|
||||
|
||||
### `signals/`
|
||||
|
||||
Signal ingestion and aggregation. Owns:
|
||||
- **Ingest** — validates, hashes (BLAKE3 for deduplication), writes to WAL, triggers downstream
|
||||
- **Decay** — forward-decay formula maintenance (`S(t) = S(t_prev) * exp(-λ * dt) + weight`)
|
||||
- **Aggregation** — windowed counters (SWAG-based), velocity computation
|
||||
- **Materialization** — background worker that writes pre-computed aggregates to O(1) lookup keys
|
||||
|
||||
### `agent/`
|
||||
|
||||
Session and policy management for agents. Owns:
|
||||
- **Session store** — lifecycle of `(user, agent, session_id)` plus short-lived signals
|
||||
- **Session materializers** — aggressive-decay aggregates agents can query (`last_5m_reward`, “tools used”, etc.)
|
||||
- **Policy enforcement** — per-agent read/write rules, rate limiting, and isolation guardrails
|
||||
- **API surface** — typed commands (`start_session`, `append_signal`, `close_session`) used by query/ranking layers
|
||||
|
||||
### `query/`
|
||||
|
||||
Query parsing and execution. Owns:
|
||||
- **Parser** — validates `Retrieve`, `Search`, `Suggest` inputs
|
||||
- **Planner** — selects candidate retrieval strategy (ANN vs. scan vs. cohort-scoped), estimates filter selectivity
|
||||
- **Executor** — orchestrates retrieval → filter → score → diversity → paginate
|
||||
|
||||
### `ranking/`
|
||||
|
||||
Scoring and diversity. Owns:
|
||||
- **Profile engine** — loads named profiles, applies boosts, penalties, gates
|
||||
- **Signal scoring** — reads decay scores and windowed aggregates from signals/
|
||||
- **Diversity enforcement** — post-scoring reordering pass enforcing `max_per_creator`, format mix, topic spread
|
||||
- **Exploration** — injects new-item candidates at the declared exploration rate
|
||||
|
||||
---
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### WAL as source of truth
|
||||
|
||||
Every write — entity, signal, relationship — goes through the Write-Ahead Log before any processing. The entity store, signal aggregates, vector index, and text index are all derived state. If they are lost, they can be rebuilt from the WAL.
|
||||
|
||||
```
|
||||
write_signal(event)
|
||||
→ hash payload (BLAKE3) // deduplication
|
||||
→ append to WAL (fsync amortized) // durability boundary
|
||||
→ update in-memory decay score // hot path, atomic
|
||||
→ update windowed counter // hot path, lock-free
|
||||
→ enqueue for materializer // background
|
||||
```
|
||||
|
||||
Signal durability is configurable per signal type:
|
||||
- `Immediate` — fsync per event (purchases, high-value actions)
|
||||
- `Batched` — fsync per N events or T ms, whichever comes first (likes, views)
|
||||
- `Eventual` — OS-buffered (impressions, hover events)
|
||||
|
||||
Default: `Batched { max_events: 100, max_delay_ms: 10 }`.
|
||||
|
||||
### Key encoding
|
||||
|
||||
All keys follow the subject-prefix pattern: `[entity_id: u64 BE][0x00][TAG:suffix]`.
|
||||
|
||||
Big-endian encoding ensures byte-lexicographic order matches numeric order — enabling range scans and prefix compression. All data for one entity is co-located.
|
||||
|
||||
```
|
||||
{item_id}\x00SIG:view:1h → windowed aggregate (1-hour view count)
|
||||
{item_id}\x00SIG:view:decay → running decay score
|
||||
{item_id}\x00META → entity metadata
|
||||
{item_id}\x00EMB → embedding vector reference
|
||||
{user_id}\x00PREF → preference vector
|
||||
{user_id}\x00SEEN:{item_id} → seen-item record
|
||||
{user_id}\x00REL:follows:{creator_id} → relationship edge
|
||||
```
|
||||
|
||||
This layout is shard-ready: `entity_id` is a partition key, and range-based partitioning needs no format migration.
|
||||
|
||||
### Storage isolation
|
||||
|
||||
Item signal ledgers, user preference vectors, and creator profiles occupy separate storage namespaces (column families). A burst of view events on a viral item must not slow down user profile reads.
|
||||
|
||||
### Hybrid backend
|
||||
|
||||
LSM-tree (fjall) for the signal event log — write-heavy, sequential, FIFO-compacted. The same engine serves entity metadata with prefix bloom filters for point lookups. If a B-tree backend proves faster for entity random reads in benchmarks, the trait abstraction allows substitution without touching higher layers.
|
||||
|
||||
---
|
||||
|
||||
## Signal System
|
||||
|
||||
### Decay model
|
||||
|
||||
Decay is declared in schema, applied at query time. The application never computes `trending_score = views / (age_hours + 2)^1.8`.
|
||||
|
||||
```
|
||||
DEFINE SIGNAL view ON item
|
||||
DECAY exponential HALF_LIFE 7d
|
||||
WINDOWS 1h, 24h, 7d, 30d, all_time
|
||||
VELOCITY enabled
|
||||
```
|
||||
|
||||
The forward-decay formula is mathematically exact and O(1) per operation:
|
||||
|
||||
```
|
||||
// Write path (3 exp() ≈ 36ns)
|
||||
S(t) = S(t_prev) * exp(-λ * dt) + weight
|
||||
|
||||
// Read path (1 exp() ≈ 15ns per entity per λ)
|
||||
current = stored * exp(-λ * dt_since_last)
|
||||
```
|
||||
|
||||
For 200 candidates: ~3-4 µs total. This replaces scanning raw events, which costs 160-1600 µs at 50 events/entity.
|
||||
|
||||
Out-of-order events: when `t_event < last_update`, pre-decay the weight: `score += weight * exp(-λ * (last_update - t_event))`. `last_update` is not modified — it already reflects a more recent time.
|
||||
|
||||
### Windowed aggregation
|
||||
|
||||
Per-signal, per-window counters maintained using a SWAG (Sliding Window Aggregate) structure. The database tracks counts within declared windows (1h, 24h, 7d, 30d) at all times. No re-scan of raw events at query time.
|
||||
|
||||
### Materialization
|
||||
|
||||
A background materializer continuously pre-computes aggregate values and writes them to O(1) lookup keys:
|
||||
|
||||
```
|
||||
{item_id}\x00SIG:view:vel:1h → view velocity (events/hour over last hour)
|
||||
{item_id}\x00SIG:like:24h → like count in last 24 hours
|
||||
{item_id}\x00SIG:completion:all → completion rate, all time
|
||||
```
|
||||
|
||||
Ranking queries read from materialized state on the fast path. If materialized state is stale (background worker lagging), the query falls back to computing from the in-memory decay score and windowed counters — slower but never wrong.
|
||||
|
||||
### Cohort-scoped aggregation
|
||||
|
||||
When a signal event arrives and cohorts are defined, the signal fans out to per-entity aggregates **and** per-cohort-entity aggregates:
|
||||
|
||||
```
|
||||
signal(view, item: X, user: U)
|
||||
→ update item X's entity-level aggregates // always
|
||||
→ for each cohort C where U ∈ C:
|
||||
update (cohort C, item X) aggregate // fan-out
|
||||
```
|
||||
|
||||
Cohort membership is maintained as RoaringBitmaps — O(1) membership test. The per-cohort-item aggregate is sparse: only active (cohort, item) pairs with at least one signal are stored. Write amplification is ~6x for 5 cohorts per user on average; mitigated by batching.
|
||||
|
||||
### Immutable events, mutable aggregates
|
||||
|
||||
Signal events (user U liked item I at time T) are immutable facts appended to the WAL. Signal aggregates (item I has 1,247 likes in the last 24h) are mutable derived state maintained in the signal ledger. These layers are kept strictly separate. Aggregates can always be recomputed from events.
|
||||
|
||||
---
|
||||
|
||||
## Vector Index
|
||||
|
||||
**USearch** (Unum Cloud) is the HNSW engine. It is not built from scratch — correct, high-performance, concurrent HNSW with SIMD distance computation is 6-12 months of dedicated work. USearch runs in ScyllaDB, ClickHouse, and DuckDB at scale. The FFI boundary via `cxx` is thin.
|
||||
|
||||
### Quantization
|
||||
|
||||
f16 by default: 10M vectors at 1536D → ~31.5 GB (f16) vs ~60 GB (float32). Less than 1% recall loss. Float32 only if benchmarks prove f16 is insufficient for a specific embedding model.
|
||||
|
||||
Embeddings are normalized to unit length at insertion time. L2 distance is then equivalent to cosine similarity, and more SIMD-friendly. Re-normalization at query time never happens.
|
||||
|
||||
### Adaptive filtered search
|
||||
|
||||
The query planner estimates filter selectivity from metadata indexes (roaring bitmaps per creator, B-tree for date ranges), then selects a strategy:
|
||||
|
||||
| Estimated selectivity | Strategy |
|
||||
|-----------------------|----------|
|
||||
| < 2% | Pre-filter via bitmap intersection → brute-force L2 over matched set |
|
||||
| 2%–100% | `index.filtered_search(vector, k, \|key\| predicate(key))` — USearch evaluates filters inline during HNSW traversal; non-matching nodes are skipped for results but still used for graph navigation |
|
||||
| Fallback | Widen `ef_search`; if still insufficient, fall back to pre-filter + brute-force |
|
||||
|
||||
This matches how ScyllaDB uses USearch in production and how Weaviate and Qdrant handle the same problem.
|
||||
|
||||
### Persistence lifecycle
|
||||
|
||||
1. Active index in RAM for reads and writes during operation.
|
||||
2. Periodic `save()` coordinated with WAL checkpointing.
|
||||
3. On restart: `view()` for immediate read-only mmap serving while a writable copy loads in background.
|
||||
4. Segment-based management for growing datasets: new inserts go to a new segment; periodic compaction merges segments and reclaims tombstoned space.
|
||||
|
||||
### Multi-vector user preference
|
||||
|
||||
User interest is not a single vector. Averaging engagement embeddings across topics ("hiking," "cooking," "cars") produces a centroid that represents none of them. Instead, each user's preference is represented as 3-10 interest cluster centroids (PinnerSage-style), maintained by the database as signals arrive. At query time, the planner issues one filtered HNSW query per active cluster and merges results. This requires no special index modifications — standard `filtered_search` per cluster, results deduped by score.
|
||||
|
||||
---
|
||||
|
||||
## Text Search
|
||||
|
||||
**Tantivy** is the full-text / BM25 engine. It is a derived index, not a source of truth.
|
||||
|
||||
### Consistency model
|
||||
|
||||
The entity store is the source of truth. Tantivy is a materialized view over it. If the Tantivy index is corrupted or lost, it can be rebuilt from the entity store by replaying the entity outbox.
|
||||
|
||||
```
|
||||
write_item(item)
|
||||
→ write to entity store (within WAL)
|
||||
→ append to background indexer outbox
|
||||
→ [async] background indexer → Tantivy
|
||||
→ on each Tantivy commit, store last-processed WAL sequence number
|
||||
→ on crash recovery, replay from that sequence number
|
||||
```
|
||||
|
||||
Tantivy's single-writer guarantee is enforced via filesystem lock. Segment merging runs on background threads to avoid query latency spikes.
|
||||
|
||||
### Hybrid fusion
|
||||
|
||||
Search queries combine BM25 relevance and ANN semantic similarity using Reciprocal Rank Fusion:
|
||||
|
||||
```
|
||||
RRF(d) = 1/(60 + rank_bm25) + 1/(60 + rank_ann)
|
||||
```
|
||||
|
||||
RRF is rank-based — no score normalization required, robust across query types. Graduate to a tuned linear combination `α * bm25 + (1-α) * ann` only after relevance labels exist to set α.
|
||||
|
||||
Personalization re-ranks the fused set using the user's preference vector and relationship graph. The order of operations: text retrieval → ANN retrieval → RRF fusion → personalization re-ranking → diversity enforcement.
|
||||
|
||||
---
|
||||
|
||||
## Query Execution Pipeline
|
||||
|
||||
Every `retrieve()` or `search()` call follows this pipeline:
|
||||
|
||||
```
|
||||
1. Parse & validate
|
||||
└── input types, profile existence, filter validity
|
||||
|
||||
2. Plan candidate retrieval
|
||||
├── ANN (user preference vector → top-k items by embedding similarity)
|
||||
├── BM25 (text query → top-k items by relevance)
|
||||
├── Full scan (trending/browse — no user vector required)
|
||||
├── Graph walk (following feed — reverse-chronological from followed creators)
|
||||
└── Cohort-scoped (trending/rising within a named cohort)
|
||||
|
||||
3. Apply hard filters
|
||||
└── unseen, unblocked, unhidden, field predicates — eliminate ineligible candidates
|
||||
└── Negative relationship checks (blocked creators, muted topics)
|
||||
|
||||
4. Score candidates
|
||||
├── Load decay scores and windowed aggregates (from materialized state or computed)
|
||||
├── Apply profile boosts (signal velocity, relationship weight, social proof)
|
||||
├── Apply profile penalties (skip count, hide, negative engagement)
|
||||
├── Apply freshness decay (age-based score reduction)
|
||||
└── Apply quality gates (minimum completion rate, minimum score threshold)
|
||||
|
||||
5. Diversity enforcement (post-scoring reordering pass)
|
||||
└── max_per_creator, format_mix, topic_diversity
|
||||
└── Reorders — does not reduce result count
|
||||
|
||||
6. Exploration injection
|
||||
└── Inject new/low-signal items at declared exploration rate (e.g. 10%)
|
||||
└── New items get exploration budget until signals accumulate
|
||||
|
||||
7. Paginate and return
|
||||
└── Cursor-based, stable across pages
|
||||
```
|
||||
|
||||
### Ranking profiles are data, not code
|
||||
|
||||
Profiles are schema-level declarations — parsed, validated, versioned, stored in the database. They are not Rust functions compiled into the binary. Changing a profile weight requires no recompile, no redeploy. The query planner reasons about profile structure to optimize execution (e.g. a profile that only uses velocity signals skips the ANN step).
|
||||
|
||||
### Graceful degradation
|
||||
|
||||
Under load, the executor degrades in order — never returns errors for well-formed queries:
|
||||
|
||||
1. Reduce candidate set size (top_k: 500 → 200)
|
||||
2. Use coarser signal aggregates (skip velocity, use windowed counts)
|
||||
3. Skip diversity enforcement
|
||||
4. Return from materialized ranking cache
|
||||
|
||||
---
|
||||
|
||||
## Write Path: Single Engagement Signal
|
||||
|
||||
Tracing `db.signal(Signal { kind: "like", item: "I", user: "U", ... })`:
|
||||
|
||||
```
|
||||
1. Hash event payload (BLAKE3) → deduplicate
|
||||
2. Append to WAL → fsync (batched)
|
||||
3. Update item I's like decay score (atomic CAS)
|
||||
4. Increment item I's like_count windowed counters (atomic add)
|
||||
5. Recompute like velocity for item I
|
||||
6. Update user U → item I relationship weight
|
||||
7. Increment user U → creator C interaction weight
|
||||
8. Shift user U's preference vector toward item I's embedding
|
||||
9. Fan-out to cohort aggregates for each cohort U belongs to
|
||||
10. Enqueue item I for materializer (windowed aggregate refresh)
|
||||
```
|
||||
|
||||
Steps 3-9 execute atomically in memory. Step 10 is background. A ranking query issued 100ms later sees the updated decay score, relationship weight, and preference vector.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
### Hot path: lock-free
|
||||
|
||||
Signal counters, decay scores, and windowed aggregates use atomic operations exclusively.
|
||||
|
||||
- `AtomicU64` with `Relaxed` ordering for monotonic counters (view_count, like_count)
|
||||
- `AtomicU64` via `f64::to_bits / from_bits` with CAS loops for decay scores
|
||||
- `Acquire/Release` at synchronization points (checkpoint, materializer flush)
|
||||
- `DashMap` for concurrent entity state access (sharded, no global lock)
|
||||
|
||||
A `like` event increments an atomic. A ranking query reads it. No blocking between writers and readers.
|
||||
|
||||
### Cold path: mutex acceptable
|
||||
|
||||
Schema changes, profile definitions, background compaction coordination — these happen infrequently and outside the query hot path. Mutexes are acceptable here.
|
||||
|
||||
### Hot-path structs: cache-line aligned
|
||||
|
||||
Any struct touched during candidate scoring is `#[repr(C, align(64))]` — one L1 cache line. This prevents false sharing under concurrent access and keeps scoring loops cache-friendly.
|
||||
|
||||
```rust
|
||||
#[repr(C, align(64))]
|
||||
struct EntitySignalState {
|
||||
entity_id: u64,
|
||||
decay_scores: [f64; 3], // one per declared decay rate
|
||||
last_update_ns: u64,
|
||||
window_counts: BucketedCounter,
|
||||
// padded to 64-byte boundary
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Targets
|
||||
|
||||
These are constraints, not aspirations. Regressions are bugs.
|
||||
|
||||
| Operation | Target |
|
||||
|-----------|--------|
|
||||
| Signal write (including WAL, amortized) | < 100 µs |
|
||||
| Decay score read per candidate | ~15 ns |
|
||||
| 200-candidate scoring pass | < 5 µs |
|
||||
| ANN retrieval at 1M vectors | < 10 ms p99 |
|
||||
| BM25 query at 1M documents | < 10 ms |
|
||||
| End-to-end RETRIEVE query | < 50 ms |
|
||||
|
||||
The 200-candidate scoring budget breaks down as: 200 × 15 ns (decay read) + 200 × (boost/penalty application) + 1 diversity pass. Everything else in the pipeline must fit within the remainder of the 50 ms budget.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Map
|
||||
|
||||
```
|
||||
usearch (C++ FFI via cxx) → vector index
|
||||
tantivy (pure Rust) → text/BM25 index
|
||||
fjall (pure Rust) → storage engine (WAL, entity store, signal ledger)
|
||||
roaring (pure Rust) → bitmap indexes (cohort membership, filter selectivity)
|
||||
blake3 (pure Rust) → content-addressed signal deduplication
|
||||
dashmap (pure Rust) → concurrent entity state map
|
||||
thiserror → typed error derives
|
||||
tracing → structured spans (embedder provides subscriber)
|
||||
serde / serde_json → serialization at API boundaries only
|
||||
criterion → benchmarking (dev dependency)
|
||||
proptest → property testing (dev dependency)
|
||||
```
|
||||
|
||||
Every dependency must justify its existence against "could we write this in 200 lines?" The approved list above is the complete list. No additions without research justification.
|
||||
|
||||
---
|
||||
|
||||
## Key Architectural Decisions
|
||||
|
||||
| Decision | Choice | Why |
|
||||
|----------|--------|-----|
|
||||
| WAL strategy | Append-only, fsync batched | Durability before processing; replay-based recovery; matches Citadel and Engram patterns |
|
||||
| Storage engine | fjall (LSM-tree) | Pure Rust, embeddable, FIFO compaction for event logs, prefix bloom filters |
|
||||
| Vector index | USearch | 150x faster than Lucene, predicate callback during HNSW traversal, mmap, quantization; used in ScyllaDB/ClickHouse/DuckDB |
|
||||
| Quantization | f16 by default | 50% memory savings, <1% recall loss; 10M × 1536D → ~31.5 GB |
|
||||
| Filtered ANN | Adaptive planner | <2% selectivity: pre-filter + brute-force; 2-100%: USearch predicate callback |
|
||||
| Text search | Tantivy as derived index | 40K lines of battle-tested Rust; custom Collector for score extraction; DB-primary with background indexer |
|
||||
| Hybrid fusion | RRF (k=60) | Rank-based, no score normalization, proven better than CombMNZ |
|
||||
| Decay model | Forward-decay formula | Mathematically exact, O(1) write/read; no raw-event scanning at query time |
|
||||
| Decay storage | f64 via AtomicU64 | 15 significant digits; sufficient for 528-year precision |
|
||||
| Timestamps | u64 nanoseconds since Unix epoch | Overflows year 2554; matches ClickHouse/Sonnerie; no external dependency |
|
||||
| Cohort membership | RoaringBitmap | O(1) membership test; sparse fan-out for signal aggregation |
|
||||
| Signal deduplication | BLAKE3 content hash | Automatic deduplication of webhook retries and client double-submissions |
|
||||
| Key encoding | `[entity_id: u64 BE]\x00TAG:suffix` | Co-location, range scans, natural shard boundaries, no migration path needed |
|
||||
| Ranking profiles | Schema declarations | Swappable at query time by name; A/B testable; no recompile on change |
|
||||
| Diversity | Post-scoring reordering pass | Does not reduce result count; enforces constraints after scoring is complete |
|
||||
| Error handling | `thiserror` enum with 6 variants | Typed, actionable errors; used by fjall/tantivy/tikv; no `unwrap()` outside tests |
|
||||
| Observability | `tracing` spans, embedder provides subscriber | Library crate; never initializes a subscriber; `#[tracing::instrument]` at subsystem boundaries |
|
||||
|
||||
---
|
||||
|
||||
## What This Replaces
|
||||
|
||||
```
|
||||
Elasticsearch → Tantivy (BM25, derived index)
|
||||
Redis → In-memory decay scores + windowed counters (lock-free atomics)
|
||||
Kafka → WAL (durable, ordered, replayable)
|
||||
Feature store → Signal ledger + materialized aggregates
|
||||
Vector DB → USearch (HNSW, embedded)
|
||||
Ranking service → Named profiles, query-time scoring
|
||||
```
|
||||
|
||||
One process. One query interface. One operational model.
|
||||
|
||||
The test: this query should execute in under 50 ms, incorporate signals written 100 ms ago, enforce diversity without application logic, handle cold-start items without application intervention:
|
||||
|
||||
```rust
|
||||
db.retrieve(Retrieve {
|
||||
entity: EntityKind::Item,
|
||||
for_user: Some("user_123"),
|
||||
context: Some("feed"),
|
||||
profile: "for_you",
|
||||
filters: vec![Filter::unseen(), Filter::not_blocked(), Filter::eq("format", "video")],
|
||||
diversity: Some(DiversitySpec { max_per_creator: Some(2), format_mix: true }),
|
||||
limit: 50,
|
||||
})
|
||||
```
|
||||
|
||||
That is what six systems currently produce. It should be one query here.
|
||||
@ -1,375 +0,0 @@
|
||||
# Coding Guidelines
|
||||
|
||||
Engineering standards for tidalDB. Derived from the research in `docs/research/`, the architectural patterns in `thoughts.md`, and the roadmap's dependency chain.
|
||||
|
||||
These are not aspirational. They are load-bearing constraints. Violating them creates bugs that are expensive to find and painful to fix in a ranking system.
|
||||
|
||||
---
|
||||
|
||||
## 1. Memory Layout and Performance
|
||||
|
||||
### Cache-line alignment on hot-path structs
|
||||
|
||||
Any struct touched during candidate scoring must be `#[repr(C, align(64))]` — exactly one L1 cache line. This prevents false sharing under concurrent access and keeps scoring loops cache-friendly.
|
||||
|
||||
Hot-path structs include: per-entity signal state, entity metadata summaries, user preference vectors, relationship weights.
|
||||
|
||||
```rust
|
||||
#[repr(C, align(64))]
|
||||
struct EntitySignalState {
|
||||
entity_id: u64,
|
||||
decay_scores: [f64; 3], // one per decay rate
|
||||
last_update_ns: u64,
|
||||
window_counts: BucketedCounter,
|
||||
// ... pad to 64-byte boundary if needed
|
||||
}
|
||||
```
|
||||
|
||||
### Lock-free on the hot path
|
||||
|
||||
Signal counters, decay scores, and windowed aggregates must use atomic operations — never mutexes. A `like` event increments an atomic counter. A ranking query reads it without blocking writers.
|
||||
|
||||
- `AtomicU64` with `Relaxed` ordering for counters
|
||||
- `AtomicF64` (via `AtomicU64` + `f64::from_bits`) with CAS loops for decay scores
|
||||
- `Acquire/Release` ordering only at synchronization boundaries (checkpoint, flush)
|
||||
- `DashMap` or sharded maps for concurrent entity state access
|
||||
|
||||
Mutexes are acceptable for cold-path operations: schema changes, profile definitions, background compaction coordination.
|
||||
|
||||
### Allocation discipline
|
||||
|
||||
- Pre-allocate result buffers. Ranking queries should not allocate per-candidate.
|
||||
- Reuse `Vec` capacity across query executions where possible.
|
||||
- Avoid `String` in hot-path structs — use interned IDs or `u64` hashes.
|
||||
- Embedding vectors are `&[f32]` slices backed by mmap or arena, never `Vec<f32>` copies.
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage Architecture
|
||||
|
||||
### WAL is the source of truth
|
||||
|
||||
Every write — entity, signal, relationship — goes through the Write-Ahead Log before any processing. The entity store, signal aggregates, and search index are derived state. If they are lost, they can be rebuilt from the WAL.
|
||||
|
||||
- Signal events are durably logged (fsync'd) before aggregation occurs
|
||||
- The aggregation system can crash, restart, and replay from the WAL
|
||||
- Content-addressed events (BLAKE3 hash of payload) for automatic deduplication of retries
|
||||
|
||||
### Trait-abstract the storage backend
|
||||
|
||||
The storage engine (fjall initially, potentially RocksDB later) must sit behind a trait boundary. No storage engine types should leak into the signal, query, or ranking modules.
|
||||
|
||||
```rust
|
||||
pub trait EntityStore: Send + Sync {
|
||||
fn get(&self, id: &EntityId) -> Result<Option<Entity>>;
|
||||
fn put(&self, entity: &Entity) -> Result<()>;
|
||||
fn scan_prefix(&self, prefix: &[u8]) -> Result<Box<dyn Iterator<Item = Entity>>>;
|
||||
}
|
||||
```
|
||||
|
||||
### Per-entity-type storage isolation
|
||||
|
||||
Item signal ledgers, user preference vectors, and creator profiles live in separate storage namespaces (column families or keyspaces). A burst of signal events for a viral item must not slow down user profile reads.
|
||||
|
||||
### Key encoding
|
||||
|
||||
Follow the subject-prefix pattern: `{entity_id}\x00{TAG}:{suffix}`. All data for one entity is co-located. Big-endian encoding so byte-lexicographic ordering matches numeric ordering.
|
||||
|
||||
```
|
||||
[entity_id: u64 BE][0x00][SIG:view:24h] → windowed aggregate
|
||||
[entity_id: u64 BE][0x00][META] → entity metadata
|
||||
[entity_id: u64 BE][0x00][REL:follows] → relationship edge
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Signal System
|
||||
|
||||
### Decay is a type, not a formula you call
|
||||
|
||||
The application never computes `trending_score = views_24h / (age_hours + 2)^1.8`. That logic lives in a named ranking profile. The application writes `SIGNAL view` and queries `USING PROFILE trending`.
|
||||
|
||||
### Running decay scores — O(1) update, O(1) read
|
||||
|
||||
Use the forward-decay formula. It is mathematically exact, not an approximation.
|
||||
|
||||
**Write:** `S(t) = S(t_prev) * exp(-lambda * dt) + weight`
|
||||
**Read:** `current = stored * exp(-lambda * dt_since_last)`
|
||||
|
||||
Cost: 3 `exp()` calls per write (~36ns), 1 `exp()` per read per entity per lambda (~15ns). For 200 candidates, that's ~3-4 microseconds total.
|
||||
|
||||
Do not scan raw events to compute decay at read time. That path costs 160+ microseconds at 50 events/entity and breaks the budget at 500+.
|
||||
|
||||
### Out-of-order events are handled correctly
|
||||
|
||||
When `t_event < last_update`, pre-decay the weight: `score += weight * exp(-lambda * (last_update - t_event))`. Do not update `last_update` — it already reflects a more recent time.
|
||||
|
||||
### Immutable events, mutable aggregates
|
||||
|
||||
Signal events (a user liked an item at time T) are immutable facts. Signal aggregates (this item has 1,247 likes in the last 24h) are mutable derived state. Keep these layers distinct. Aggregates can always be recomputed from events.
|
||||
|
||||
---
|
||||
|
||||
## 4. Vector Index
|
||||
|
||||
### USearch is the HNSW engine
|
||||
|
||||
Do not build HNSW from scratch. USearch provides 126K+ QPS, predicate callbacks during traversal, mmap persistence, and quantization. The FFI boundary via CXX is thin.
|
||||
|
||||
### f16 quantization as default
|
||||
|
||||
10M vectors at 1536D: ~31.5 GB (f16) vs ~60 GB (float32). Less than 1% recall loss. Use float32 only when benchmarks prove f16 is insufficient for a specific embedding model.
|
||||
|
||||
### Normalize embeddings at insertion time
|
||||
|
||||
For cosine similarity, normalize vectors to unit length and use L2 distance (equivalent for unit vectors, more SIMD-friendly). Store normalized vectors — never re-normalize at query time.
|
||||
|
||||
### Adaptive filtered search
|
||||
|
||||
Never hardcode a single filtering strategy. Estimate selectivity, then branch:
|
||||
- **<2% selectivity:** Pre-filter (roaring bitmap intersection) then brute-force L2
|
||||
- **2-100% selectivity:** `filtered_search` with predicate callback (in-graph filtering)
|
||||
- **Fallback:** Widen ef_search or degrade to pre-filter + brute-force
|
||||
|
||||
---
|
||||
|
||||
## 5. Text Search
|
||||
|
||||
### Tantivy is a derived index, not a source of truth
|
||||
|
||||
The entity store is the source of truth. Tantivy is a materialized view. If the Tantivy index is corrupted or lost, it can be rebuilt from the entity store.
|
||||
|
||||
Consistency pattern:
|
||||
1. Write to entity store (within transaction / WAL) — the durable source of truth.
|
||||
2. Background indexer reads the outbox and feeds Tantivy (best-effort, async).
|
||||
3. On crash recovery, unconditionally rebuild the text index from the entity
|
||||
store at open — the same proven pattern as the vector index, which is rebuilt
|
||||
from the durable embeddings on every reopen.
|
||||
|
||||
> Crash recovery is NOT driven by a Tantivy commit-payload sequence number.
|
||||
> Because the indexer is best-effort and async, an item can be durably persisted
|
||||
> to the entity store and then lost in a crash before the syncer commits it to
|
||||
> Tantivy. A stored sequence number does not heal that gap by itself and is
|
||||
> easy to leave un-wired (it once silently was, hiding the lost write). The
|
||||
> unconditional rebuild-from-store at open is deterministic and cannot silently
|
||||
> regress: any entity in the store but missing from Tantivy is re-indexed.
|
||||
|
||||
### Hybrid fusion starts with RRF
|
||||
|
||||
`RRF(d) = 1/(60 + rank_bm25) + 1/(60 + rank_ann)`. Rank-based, no score normalization needed, robust across query types. Graduate to tuned linear combination only after relevance labels exist to tune alpha.
|
||||
|
||||
---
|
||||
|
||||
## 6. Query and Ranking
|
||||
|
||||
### Ranking profiles are data, not code
|
||||
|
||||
Profiles are schema-level declarations — parsed, validated, versioned, stored in the database. They are not Rust functions compiled into the binary. The query optimizer reasons about profile structure to plan execution.
|
||||
|
||||
A profile change should never require recompiling or redeploying.
|
||||
|
||||
### Diversity is a post-scoring pass
|
||||
|
||||
After candidates are scored, apply diversity constraints as a separate reordering pass. Diversity does not reduce result count — it reorders to enforce constraints (max_per_creator, format_mix) while maintaining the target count.
|
||||
|
||||
### Negative signals are structurally equal to positive signals
|
||||
|
||||
Skips, hides, blocks, mutes, downvotes are not the absence of engagement. They are data. They carry the same weight, precision, and update immediacy as likes. A hide creates a permanent hard-negative. A skip within 3 seconds is a strong quality signal. The ranking function treats these as first-class inputs.
|
||||
|
||||
### Graceful degradation, never failure
|
||||
|
||||
Under load, return slightly less precise rankings — not errors. Degrade in this order:
|
||||
1. Reduce candidate set size (top_k: 500 -> 200)
|
||||
2. Use coarser signal aggregates (skip velocity, use windowed counts)
|
||||
3. Skip diversity enforcement
|
||||
4. Return results from materialized cache
|
||||
|
||||
Never return an empty result set or an error for a well-formed query.
|
||||
|
||||
---
|
||||
|
||||
## 7. Error Handling
|
||||
|
||||
### `Result<T>` everywhere, `unwrap()` nowhere
|
||||
|
||||
Every fallible operation returns `Result`. No `unwrap()`, no `expect()` outside of tests and initialization. Panics in a database corrupt state.
|
||||
|
||||
### Errors are typed and actionable
|
||||
|
||||
```rust
|
||||
pub enum TidalError {
|
||||
/// Storage engine failure — retry may succeed.
|
||||
Storage(StorageError),
|
||||
/// Entity not found — caller should handle.
|
||||
NotFound { entity: EntityId },
|
||||
/// Schema violation — caller's fault, fix the input.
|
||||
Schema(SchemaError),
|
||||
/// Signal write failed durability check — retry required.
|
||||
Durability(DurabilityError),
|
||||
/// Query malformed — parse error with position.
|
||||
Query(QueryError),
|
||||
/// Internal invariant violated — this is a bug, log and degrade.
|
||||
Internal(String),
|
||||
}
|
||||
```
|
||||
|
||||
`Internal` errors trigger graceful degradation, not crashes. Log them loudly. Return approximate results if possible.
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing
|
||||
|
||||
### Property tests for invariants
|
||||
|
||||
Use `proptest` for properties that must hold regardless of input:
|
||||
- Decay scores monotonically decrease when no new events arrive
|
||||
- Windowed aggregates equal the sum of events within the window
|
||||
- Diversity constraints hold in every result set
|
||||
- WAL replay produces identical state to uninterrupted execution
|
||||
- Filter composition is commutative (order of filters doesn't change results)
|
||||
- Blocked/hidden items never appear in query results
|
||||
|
||||
### Crash recovery tests
|
||||
|
||||
Simulate crashes at every point in the write path:
|
||||
- Mid-WAL-write
|
||||
- After WAL commit, before entity store update
|
||||
- After entity store, before signal aggregation
|
||||
- After signal aggregation, before Tantivy index
|
||||
- During background materialization
|
||||
|
||||
Verify: the system recovers to a consistent state. No lost events. No phantom state.
|
||||
|
||||
### Benchmark from day one
|
||||
|
||||
Use `criterion` for micro-benchmarks. Track these numbers continuously:
|
||||
- Signal write latency (target: <100 microseconds including WAL fsync amortized)
|
||||
- Decay score read per candidate (target: ~15ns)
|
||||
- 200-candidate scoring pass (target: <5 microseconds)
|
||||
- ANN retrieval at 1M vectors (target: <10ms p99)
|
||||
- BM25 query at 1M documents (target: <10ms)
|
||||
- End-to-end RETRIEVE query (target: <50ms)
|
||||
|
||||
Regressions in these numbers are bugs. Treat them like test failures.
|
||||
|
||||
---
|
||||
|
||||
## 9. Code Organization
|
||||
|
||||
### Module boundaries match the dependency chain
|
||||
|
||||
```
|
||||
storage/ → knows nothing about signals, queries, or ranking
|
||||
signals/ → depends on storage, knows nothing about queries or ranking
|
||||
query/ → depends on storage + signals, knows nothing about ranking internals
|
||||
ranking/ → depends on signals, invoked by query executor
|
||||
schema/ → standalone, depended on by everything
|
||||
```
|
||||
|
||||
Circular dependencies between these modules are architectural bugs. If ranking needs to call into storage directly, that call goes through a trait the query executor provides.
|
||||
|
||||
### Public API is minimal
|
||||
|
||||
Expose the smallest possible surface. Internal types stay internal. The public API is:
|
||||
- `TidalDB::open()`, `TidalDB::shutdown()`
|
||||
- `define_entity()`, `define_signal()`, `define_profile()`
|
||||
- `write_item()`, `write_user()`, `write_creator()`
|
||||
- `write_relationship()`
|
||||
- `signal()`
|
||||
- `retrieve()`, `search()`, `suggest()`
|
||||
|
||||
Everything else is `pub(crate)` or module-private.
|
||||
|
||||
### One concern per file
|
||||
|
||||
A file that handles both signal ingestion and signal aggregation will grow into a 2000-line mess. Split early: `signals/ingest.rs`, `signals/decay.rs`, `signals/aggregation.rs`, `signals/materialization.rs`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Dependencies
|
||||
|
||||
### Minimal, intentional, auditable
|
||||
|
||||
Every dependency must justify its existence against "could we write this in 200 lines?"
|
||||
|
||||
Approved dependencies (from research):
|
||||
- **fjall** — storage engine (pure Rust, embeddable)
|
||||
- **usearch** — HNSW vector index (C++ FFI via cxx)
|
||||
- **tantivy** — full-text search / BM25
|
||||
- **blake3** — content-addressed hashing
|
||||
- **roaring** — bitmap indexes for filtered search
|
||||
- **thiserror** — derive `Display` and `From` for typed error enums; eliminates boilerplate without hiding structure
|
||||
- **tracing** — structured spans for query execution, WAL writes, and signal ingestion; embedders choose their own subscriber
|
||||
- **criterion** — benchmarking
|
||||
- **proptest** — property testing
|
||||
- **serde** / **serde_json** — serialization (at API boundaries only, not in hot paths)
|
||||
- **chrono** or **time** — timestamp handling
|
||||
- **dashmap** — concurrent hash map for hot-path entity state
|
||||
|
||||
Do not add dependencies for things the standard library or a 50-line util handles: argument parsing, builder pattern macros, derive-everything crates.
|
||||
|
||||
### No `unsafe` without a comment explaining why
|
||||
|
||||
Every `unsafe` block must have a `// SAFETY:` comment explaining:
|
||||
1. What invariant the compiler can't verify
|
||||
2. Why this specific usage is sound
|
||||
3. What would make it unsound (for future maintainers)
|
||||
|
||||
Prefer `#![forbid(unsafe_code)]` at the crate level where possible. The storage engine and FFI boundaries (USearch) are the only modules that should need `unsafe`.
|
||||
|
||||
---
|
||||
|
||||
## 11. Observability
|
||||
|
||||
### `tracing` spans on every public operation
|
||||
|
||||
Every public function that crosses a subsystem boundary gets a `#[tracing::instrument]` attribute. This is non-negotiable — it is how query latency, signal write throughput, and WAL sync times are measured in production without any additional instrumentation work later.
|
||||
|
||||
```rust
|
||||
#[tracing::instrument(skip(self), fields(entity_id = %id))]
|
||||
pub fn get_entity(&self, id: EntityId) -> Result<Option<Entity>> {
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
The `skip` attribute prevents large or sensitive arguments from being logged by default. Add `fields(...)` to surface the key identifiers that make traces navigable.
|
||||
|
||||
### Instrument at subsystem entry points, not every helper
|
||||
|
||||
Instrument the public API and the major internal stage boundaries:
|
||||
- `EntityStore::{get, put, scan_prefix}`
|
||||
- `SignalLedger::{record, decay_score}`
|
||||
- `QueryExecutor::execute`
|
||||
- `RankingEngine::score`
|
||||
- `Wal::{append, flush}`
|
||||
|
||||
Do not add spans to private helpers called within a single instrumented function. The overhead accumulates.
|
||||
|
||||
### tidalDB is a library — embedders choose their subscriber
|
||||
|
||||
Do not initialize a tracing subscriber anywhere in this crate. The subscriber is the embedder's responsibility. Import `tracing = "0.1"` only; never `tracing-subscriber` in the main crate.
|
||||
|
||||
### Error events
|
||||
|
||||
Use `tracing::error!` for `TidalError::Internal` (a bug occurred), `tracing::warn!` for recoverable degradation, `tracing::debug!` for query planning decisions, `tracing::trace!` for per-candidate scoring.
|
||||
|
||||
Never use `println!` or `eprintln!` in production code.
|
||||
|
||||
---
|
||||
|
||||
## 12. Commit and Review Standards
|
||||
|
||||
### Commits are atomic and purposeful
|
||||
|
||||
One logical change per commit. "Add signal decay scoring" is a commit. "Add decay scoring and also fix a typo and refactor entity store" is three commits.
|
||||
|
||||
### Every PR must include
|
||||
|
||||
- What changed and why (not how — the diff shows how)
|
||||
- Benchmark results if touching hot-path code
|
||||
- Property test or crash recovery test if touching write path or state management
|
||||
- No regressions in existing benchmarks
|
||||
|
||||
### No TODO without an issue
|
||||
|
||||
`// TODO:` comments are allowed only with a link to a tracking issue. Orphan TODOs rot. If it's worth noting, it's worth tracking.
|
||||
@ -1,298 +0,0 @@
|
||||
# Quickstart
|
||||
|
||||
Get a working ranked feed in 10 minutes.
|
||||
|
||||
**Prerequisites:** Rust 1.91+, Cargo. No external services.
|
||||
|
||||
---
|
||||
|
||||
## Run the example
|
||||
|
||||
The fastest path is the included example, which demonstrates the complete loop — schema, ingest, signals, ranking:
|
||||
|
||||
```bash
|
||||
cargo run -p tidaldb --example quickstart
|
||||
```
|
||||
|
||||
The rest of this guide explains what it does and extends it with personalization and search.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Add the dependency
|
||||
|
||||
Depend on the `tidaldb` crate (at `tidal/` in this repository) by path or git:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
tidaldb = { path = "../tidaldb/tidal" } # adjust to the tidal/ crate's location, or use git = "https://github.com/orchard9/tidaldb"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Define a schema
|
||||
|
||||
Schema is defined before opening the database. It declares signal types (what events you'll record and how they decay), text fields (for BM25 search), and embedding slots (for vector search).
|
||||
|
||||
```rust
|
||||
use std::time::Duration;
|
||||
use tidaldb::schema::{SchemaBuilder, EntityKind, DecaySpec, Window, TextFieldType};
|
||||
|
||||
let mut schema = SchemaBuilder::new();
|
||||
|
||||
// View signal: 7-day half-life, three windows, velocity enabled.
|
||||
// You declare the decay. tidalDB applies it at query time — no formula to maintain.
|
||||
let _ = schema.signal("view", EntityKind::Item, DecaySpec::Exponential {
|
||||
half_life: Duration::from_secs(7 * 24 * 3600),
|
||||
}).windows(&[Window::OneHour, Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
|
||||
|
||||
// Like signal: 30-day half-life. Durable engagement decays slowly.
|
||||
let _ = schema.signal("like", EntityKind::Item, DecaySpec::Exponential {
|
||||
half_life: Duration::from_secs(30 * 24 * 3600),
|
||||
}).windows(&[Window::AllTime]).velocity(false).add();
|
||||
|
||||
// Share signal: 3-day half-life. Short-lived but strongly trending.
|
||||
let _ = schema.signal("share", EntityKind::Item, DecaySpec::Exponential {
|
||||
half_life: Duration::from_secs(3 * 24 * 3600),
|
||||
}).windows(&[Window::TwentyFourHours, Window::AllTime]).velocity(true).add();
|
||||
|
||||
// Skip signal: permanent. A user who skipped should not see it again.
|
||||
let _ = schema.signal("hide", EntityKind::Item, DecaySpec::Permanent).add();
|
||||
|
||||
// Text fields for BM25 full-text search.
|
||||
schema.text_field("title", TextFieldType::Text);
|
||||
schema.text_field("category", TextFieldType::Keyword);
|
||||
|
||||
// Embedding slot for semantic / vector search (128D in this example).
|
||||
// In production, use the dimensionality of your embedding model.
|
||||
schema.embedding_slot("content", EntityKind::Item, 128);
|
||||
|
||||
let schema = schema.build()?;
|
||||
```
|
||||
|
||||
**Decay types:**
|
||||
- `Exponential { half_life }` — weight halves every `half_life`. Use for views, likes, shares.
|
||||
- `Linear { lifetime }` — weight drops to zero over `lifetime`.
|
||||
- `Permanent` — never decays. Use for hides, blocks, follows.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Open the database
|
||||
|
||||
```rust
|
||||
use tidaldb::TidalDb;
|
||||
|
||||
// Ephemeral: in-memory, ideal for tests and this tutorial.
|
||||
let db = TidalDb::builder().ephemeral().with_schema(schema).open()?;
|
||||
|
||||
// Persistent: durable storage at a path on disk.
|
||||
// let db = TidalDb::builder().with_data_dir("/var/lib/myapp/tidaldb").with_schema(schema).open()?;
|
||||
|
||||
db.health_check()?;
|
||||
```
|
||||
|
||||
`TidalDb` is `Send + Sync`. Wrap it in `Arc<TidalDb>` to share across threads or tasks.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Ingest content
|
||||
|
||||
Write items with metadata as `HashMap<String, String>` key-value pairs. Then write their embeddings separately.
|
||||
|
||||
**tidalDB does not generate embeddings.** You bring your model; tidalDB handles retrieval and ranking over the vectors you produce.
|
||||
|
||||
```rust
|
||||
use std::collections::HashMap;
|
||||
use tidaldb::schema::{EntityId, Timestamp};
|
||||
|
||||
let tracks = [
|
||||
(1u64, "Introduction to Jazz Piano", "music", "1320"),
|
||||
(2, "Rust Async Programming", "tech", "3600"),
|
||||
(3, "Sourdough Bread Masterclass", "cooking", "2700"),
|
||||
(4, "Jazz Improvisation Techniques","music", "1800"),
|
||||
(5, "Building a Compiler in Rust", "tech", "5400"),
|
||||
(6, "French Pastry Fundamentals", "cooking", "2100"),
|
||||
(7, "Modal Jazz: Coltrane Changes", "music", "2400"),
|
||||
(8, "WebAssembly from Scratch", "tech", "2700"),
|
||||
(9, "Knife Skills for Home Cooks", "cooking", "900"),
|
||||
(10, "Bebop Piano Vocabulary", "music", "1500"),
|
||||
];
|
||||
|
||||
for (id, title, category, duration) in &tracks {
|
||||
let mut meta = HashMap::new();
|
||||
meta.insert("title".to_string(), title.to_string());
|
||||
meta.insert("category".to_string(), category.to_string());
|
||||
meta.insert("format".to_string(), "video".to_string());
|
||||
meta.insert("duration".to_string(), duration.to_string());
|
||||
meta.insert("created_at".to_string(), Timestamp::now().as_nanos().to_string());
|
||||
|
||||
db.write_item_with_metadata(EntityId::new(*id), &meta)?;
|
||||
|
||||
// In production: embed the title with your model.
|
||||
// Here we use random unit vectors for illustration.
|
||||
let embedding = random_unit_vector(128, &mut rng);
|
||||
db.write_item_embedding(EntityId::new(*id), &embedding)?;
|
||||
}
|
||||
|
||||
println!("Ingested {} items.", db.item_count());
|
||||
```
|
||||
|
||||
On write, tidalDB:
|
||||
1. Stores the entity and metadata
|
||||
2. Indexes text fields into the BM25 index
|
||||
3. Inserts the embedding into the HNSW vector index
|
||||
4. Initializes the signal ledger with an exploration budget
|
||||
5. Makes the item immediately queryable
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Record engagement signals
|
||||
|
||||
When a user engages with content, write a signal. The feedback loop closes at write time — no Kafka consumer to lag, no feature store sync to schedule.
|
||||
|
||||
```rust
|
||||
let now = Timestamp::now();
|
||||
|
||||
// Global signals — these update the item's aggregate signal ledger.
|
||||
db.signal("view", EntityId::new(1), 1.0, now)?; // Jazz Piano viewed
|
||||
db.signal("view", EntityId::new(4), 1.0, now)?;
|
||||
db.signal("view", EntityId::new(7), 1.0, now)?; // Modal Jazz viewed
|
||||
db.signal("like", EntityId::new(4), 1.0, now)?; // Jazz Improv liked
|
||||
db.signal("share", EntityId::new(7), 1.0, now)?; // Modal Jazz shared
|
||||
```
|
||||
|
||||
For signals with user context, use `signal_with_context`. This also updates the user's preference vector and interaction weights — enabling personalization.
|
||||
|
||||
```rust
|
||||
let user_id = 42u64;
|
||||
let creator_id = 100u64;
|
||||
|
||||
// User 42 viewed item 4. Their preference vector shifts toward jazz content.
|
||||
db.signal_with_context("view", EntityId::new(4), 1.0, now, Some(user_id), Some(creator_id))?;
|
||||
db.signal_with_context("like", EntityId::new(7), 1.0, now, Some(user_id), Some(creator_id))?;
|
||||
|
||||
// Negative signals are equal citizens.
|
||||
db.signal("hide", EntityId::new(2), 1.0, now)?; // User hid the Rust video.
|
||||
```
|
||||
|
||||
A ranking query issued 100ms later sees the updated state. No ETL required.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Retrieve a ranked feed
|
||||
|
||||
tidalDB ships 25 built-in ranking profiles. The application names a profile; the database executes the full scoring pipeline.
|
||||
|
||||
```rust
|
||||
use tidaldb::query::retrieve::Retrieve;
|
||||
|
||||
// Global trending: items with the highest share + view velocity.
|
||||
let query = Retrieve::builder().profile("trending").limit(10).build()?;
|
||||
let results = db.retrieve(&query)?;
|
||||
|
||||
println!("Trending ({} candidates):", results.total_candidates);
|
||||
for item in &results.items {
|
||||
let sigs: Vec<_> = item.signals.iter()
|
||||
.map(|s| format!("{}={:.3}", s.name, s.value))
|
||||
.collect();
|
||||
println!(" #{} id={} score={:.4} [{}]",
|
||||
item.rank, item.entity_id.as_u64(), item.score, sigs.join(", "));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Personalize
|
||||
|
||||
Swap the profile to `for_you`. Because user 42 signaled views and likes on jazz content, their results differ from global trending.
|
||||
|
||||
```rust
|
||||
// Personalized feed for user 42.
|
||||
let query = Retrieve::builder()
|
||||
.for_user(user_id)
|
||||
.profile("for_you")
|
||||
.limit(10)
|
||||
.build()?;
|
||||
let results = db.retrieve(&query)?;
|
||||
|
||||
println!("For You (user {}):", user_id);
|
||||
for item in &results.items {
|
||||
println!(" #{} id={} score={:.4}", item.rank, item.entity_id.as_u64(), item.score);
|
||||
}
|
||||
```
|
||||
|
||||
Other useful profiles:
|
||||
- `"hot"` — score with age decay (Reddit model)
|
||||
- `"following"` — content from followed creators (requires `for_user` + written `follows` relationships)
|
||||
- `"hidden_gems"` — high completion rate, low reach
|
||||
- `"top_week"` — cumulative quality over the last 7 days
|
||||
- `"shuffle"` — random, quality-weighted
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Search
|
||||
|
||||
Search combines BM25 full-text and ANN semantic similarity via Reciprocal Rank Fusion.
|
||||
|
||||
```rust
|
||||
use tidaldb::query::search::Search;
|
||||
|
||||
// Flush the text index so recently written items are searchable.
|
||||
// In production with persistent mode this happens automatically on a ~2s commit cycle.
|
||||
db.flush_text_index()?;
|
||||
|
||||
// Keyword search, personalized for user 42.
|
||||
let query = Search::builder()
|
||||
.query("jazz piano")
|
||||
.for_user(user_id)
|
||||
.limit(5)
|
||||
.build()?;
|
||||
let results = db.search(&query)?;
|
||||
|
||||
println!("Search 'jazz piano':");
|
||||
for item in &results.items {
|
||||
println!(" #{} id={} bm25={:.3?} semantic={:.3?}",
|
||||
item.rank,
|
||||
item.entity_id.as_u64(),
|
||||
item.bm25_score,
|
||||
item.semantic_score,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Add a query embedding for hybrid search — text relevance + semantic similarity:
|
||||
|
||||
```rust
|
||||
let query_vector = your_model.embed("jazz piano"); // same model as item embeddings
|
||||
let query = Search::builder()
|
||||
.query("jazz piano")
|
||||
.vector(query_vector)
|
||||
.for_user(user_id)
|
||||
.limit(5)
|
||||
.build()?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Close
|
||||
|
||||
```rust
|
||||
db.close()?;
|
||||
```
|
||||
|
||||
This flushes the WAL, checkpoints signal state, and persists indexes. In persistent mode, the next open recovers to the last checkpointed state.
|
||||
|
||||
---
|
||||
|
||||
## What to explore next
|
||||
|
||||
| Topic | Where to look |
|
||||
|-------|--------------|
|
||||
| Full API reference | [API.md](API.md) |
|
||||
| Filters — format, duration, location, engagement thresholds | [API.md — Filters](API.md#filters) |
|
||||
| Diversity constraints | [API.md — Diversity Constraints](API.md#diversity-constraints) |
|
||||
| All 25 ranking profiles | [API.md — Sort Modes](API.md#sort-modes) |
|
||||
| Cohort-scoped trending | [API.md — Cohorts](API.md#cohort-definitions) |
|
||||
| Collections and saved searches | [API.md — Collections](API.md#collections) |
|
||||
| Axum embedding example | `examples/axum_embedding.rs` |
|
||||
| 14 content discovery surfaces | [USE_CASES.md](USE_CASES.md) |
|
||||
| Architecture and design decisions | [ARCHITECTURE.md](ARCHITECTURE.md) |
|
||||
@ -1,438 +0,0 @@
|
||||
# Sequence Diagrams
|
||||
|
||||
User-perspective flows for every major surface. Each diagram shows what the application sends, what tidalDB does internally, and what signals flow back to close the loop.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [UC-01 · For You Feed](#uc-01--for-you-feed)
|
||||
- [UC-02 · Search with Personalized Ranking](#uc-02--search-with-personalized-ranking)
|
||||
- [UC-03 · Trending / Popular](#uc-03--trending--popular)
|
||||
- [UC-04 · Following Feed](#uc-04--following-feed)
|
||||
- [UC-05 · Related Content / Up Next](#uc-05--related-content--up-next)
|
||||
- [UC-06 · Browse / Category Discovery](#uc-06--browse--category-discovery)
|
||||
- [UC-07 · Notification Prioritization](#uc-07--notification-prioritization)
|
||||
- [UC-15 · Cohort-Scoped Trending](#uc-15--cohort-scoped-trending)
|
||||
- [Core · Engagement Feedback Loop](#core--engagement-feedback-loop)
|
||||
- [Write · Content Ingest](#write--content-ingest)
|
||||
|
||||
---
|
||||
|
||||
## UC-01 · For You Feed
|
||||
|
||||
User opens the app. tidalDB retrieves candidates, scores them against the user's preference profile, enforces diversity, and returns a ready-to-render batch. Pagination continues with previously-returned IDs excluded.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Opens app / pulls to refresh
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nCONTEXT feed\nUSING PROFILE for_you\nFILTER unseen, unblocked\nDIVERSITY max_per_creator:2\nLIMIT 50
|
||||
|
||||
Note over TidalDB: 1. Load user preference vector\n2. ANN retrieval over item embeddings\n3. Apply seen / blocked filters\n4. Score via for_you profile:\n preference_match\n × engagement_velocity(24h)\n × recency_decay\n × social_proof\n5. Enforce diversity constraints\n6. Return ranked batch
|
||||
|
||||
TidalDB-->>App: [{item_id, score, signals_snapshot} × 50]
|
||||
App-->>User: Feed renders
|
||||
|
||||
User->>App: Scrolls to bottom
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nCONTEXT feed\nUSING PROFILE for_you\nFILTER unseen, unblocked\nEXCLUDE [previously_returned_ids]\nLIMIT 50
|
||||
|
||||
TidalDB-->>App: Next batch of 50
|
||||
App-->>User: Feed extends
|
||||
```
|
||||
|
||||
**Signals powering this query:** user preference vector (built from history), item view velocity (24h window), item completion rate, user→creator interaction weight, social graph engagement, item age with decay curve, skip and hide signals.
|
||||
|
||||
---
|
||||
|
||||
## UC-02 · Search with Personalized Ranking
|
||||
|
||||
Text relevance is the floor — an irrelevant result never appears just because the user likes the creator. Personalization reorders within the relevant candidate set. A beginner and an expert searching the same query get different orderings.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant Embed as Embedding Service
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Types "rust tutorial beginner"
|
||||
|
||||
App->>Embed: embed("rust tutorial beginner")
|
||||
Embed-->>App: query_vector[1536]
|
||||
|
||||
App->>TidalDB: SEARCH items\nQUERY "rust tutorial beginner"\nVECTOR query_vector\nFOR USER @u123\nUSING PROFILE search\nDIVERSITY max_per_creator:2\nLIMIT 20
|
||||
|
||||
Note over TidalDB: 1. BM25 text match → relevance scores\n2. ANN over item embeddings → semantic scores\n3. Merge: text(0.6) + semantic(0.4)\n4. Personalization layer:\n user topic engagement history\n item quality signals (completion, like ratio)\n recency curve (slow decay for tutorials)\n5. Diversity pass\n6. Return ranked results
|
||||
|
||||
TidalDB-->>App: [{item_id, relevance_score, rank} × 20]
|
||||
App-->>User: Search results render
|
||||
|
||||
User->>App: Clicks result #3
|
||||
|
||||
App->>TidalDB: SIGNAL search_click\nitem: @item_id\nuser: @u123\nquery_context: "rust tutorial beginner"\nrank_at_click: 3
|
||||
|
||||
Note over TidalDB: Updates item relevance signal\nfor this query category.\nBoosts user→topic weight.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
|
||||
User->>App: Watches 80% of video
|
||||
|
||||
App->>TidalDB: SIGNAL completion\nitem: @item_id\nuser: @u123\nratio: 0.80
|
||||
|
||||
Note over TidalDB: Strong positive on item quality.\nUpdates user preference vector\ntoward this topic and format.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UC-03 · Trending / Popular
|
||||
|
||||
Velocity, not volume. A video posted 4 hours ago with a high share rate outranks a video with 10× the total views but slow recent growth. The same profile applies to global, category-scoped, social-graph-scoped, and cohort-scoped trending — only the candidate set and signal scope change.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Taps "Trending" tab
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nWINDOW 24h\nDIVERSITY max_per_creator:1\nLIMIT 25
|
||||
|
||||
Note over TidalDB: Profile: trending\nPrimary: share_velocity(6h)\nSecondary: view_velocity(6h)\nBoost: new_user_reach (virality)\nGate: engagement_ratio > 0.03\nNo personalization.\nNo total-count signals.
|
||||
|
||||
TidalDB-->>App: [{item_id, trending_score, velocity_snapshot} × 25]
|
||||
App-->>User: Trending page renders
|
||||
|
||||
User->>App: Taps "Jazz" category filter
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nFILTER category: jazz\nWINDOW 24h\nDIVERSITY max_per_creator:1\nLIMIT 25
|
||||
|
||||
Note over TidalDB: Same profile. Same velocity signals.\nHard filter on category metadata.\nScoped candidate set.
|
||||
|
||||
TidalDB-->>App: Trending in Jazz
|
||||
App-->>User: "Trending in Jazz" renders
|
||||
|
||||
User->>App: Switches to "Among People I Follow"
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nFILTER social_graph: @u123 depth:2\nWINDOW 24h\nLIMIT 25
|
||||
|
||||
Note over TidalDB: Same profile.\nCandidates constrained to items\nengaged by users @u123 follows.
|
||||
|
||||
TidalDB-->>App: Trending among follows
|
||||
App-->>User: Social trending renders
|
||||
|
||||
User->>App: Switches to "Trending in My Demo"
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nUSING PROFILE trending\nCOHORT locale:en-US, age:18-24\nWINDOW 24h\nLIMIT 25
|
||||
|
||||
Note over TidalDB: Same profile.\nSignal aggregation scoped to\nusers matching cohort predicate.
|
||||
|
||||
TidalDB-->>App: Cohort-scoped trending
|
||||
App-->>User: Demographic trending renders
|
||||
```
|
||||
|
||||
**One profile, four scopes.** Global, category, social, and cohort trending are the same ranking profile applied to different signal scopes. No code changes — just query parameters.
|
||||
|
||||
---
|
||||
|
||||
## UC-04 · Following Feed
|
||||
|
||||
The surface where users expect control. Recency-dominant, minimal algorithmic intervention. A creator's worst-performing video still appears here — because the user chose to follow them.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Taps "Following" tab
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nFILTER relationship: follows\nUSING PROFILE following\nFILTER unseen\nLIMIT 50
|
||||
|
||||
Note over TidalDB: Profile: following\nPrimary sort: created_at DESC\nTiebreaker: completion_rate\nHard filter: creator in user's follows\nHard filter: unseen by this user\nNo exploration budget.\nNo aggressive personalization.
|
||||
|
||||
TidalDB-->>App: Chronological feed from follows
|
||||
App-->>User: Subscription feed renders
|
||||
|
||||
User->>App: Scrolls — catching up from 3 days ago
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFOR USER @u123\nFILTER relationship: follows\nFILTER created_at < @cursor_timestamp\nUSING PROFILE following\nLIMIT 50
|
||||
|
||||
TidalDB-->>App: Older batch (cursor pagination)
|
||||
App-->>User: Older content loads
|
||||
|
||||
User->>App: Follows a new creator
|
||||
|
||||
App->>TidalDB: WRITE relationship\ntype: follows\nfrom: @u123\nto: @creator_id\ntimestamp: now()
|
||||
|
||||
Note over TidalDB: Adds edge to relationship graph.\nUpdates user preference vector\nto include creator's topic signals.\nNew creator appears in next query.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UC-05 · Related Content / Up Next
|
||||
|
||||
Anchored to the item just consumed. Blends semantic similarity with collaborative filtering and user taste. The autoplay accept/skip signal strengthens or decays the item-to-item pairing for future recommendations.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Finishes watching @item_abc (Rust tutorial, beginner)
|
||||
|
||||
App->>TidalDB: SIGNAL completion\nitem: @item_abc\nuser: @u123\nratio: 0.94
|
||||
|
||||
Note over TidalDB: Appends to @item_abc signal ledger.\nUpdates user→item_abc relationship.\nUpdates user preference vector\ntoward rust/programming/tutorials.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nSIMILAR TO @item_abc\nFOR USER @u123\nUSING PROFILE related\nFILTER unseen\nEXCLUDE creator: @item_abc.creator_id (top 3 only)\nLIMIT 10
|
||||
|
||||
Note over TidalDB: Profile: related\n1. ANN: items near @item_abc embedding\n2. Collaborative: items co-engaged with @item_abc\n3. Personalize: re-rank by user preference match\n4. Quality gate: completion_rate > 0.4\n5. Diversity: avoid same creator in top 3\n6. Return
|
||||
|
||||
TidalDB-->>App: [{item_id, similarity_score, collab_score} × 10]
|
||||
App-->>User: Up Next queue renders
|
||||
|
||||
User->>App: Autoplay begins on result #1
|
||||
|
||||
App->>TidalDB: SIGNAL autoplay_accept\nsource_item: @item_abc\ntarget_item: @item_xyz\nuser: @u123
|
||||
|
||||
Note over TidalDB: Strong positive on item_abc → item_xyz pairing.\nStrengthens collaborative edge.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
|
||||
User->>App: Skips after 8 seconds
|
||||
|
||||
App->>TidalDB: SIGNAL skip\nitem: @item_xyz\nuser: @u123\ndwell_ms: 8200\ncontext: autoplay_from @item_abc
|
||||
|
||||
Note over TidalDB: Negative on this pairing.\nDecays item_abc → item_xyz\ncollaborative edge slightly.
|
||||
|
||||
TidalDB-->>App: ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UC-06 · Browse / Category Discovery
|
||||
|
||||
Quality-dominant ranking within a filtered candidate set. Mix of established content and breakout newcomers. Sort mode switches (Top, New, All Time) are different profiles on the same candidate set — no application logic needed.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Taps "Jazz" category
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE browse\nDIVERSITY max_per_creator:2\nLIMIT 20
|
||||
|
||||
Note over TidalDB: Profile: browse\nPrimary: quality_score\n completion_rate(0.5)\n + like_ratio(0.3)\n + reach(0.2)\nRecency boost: 30d half-life\nBreakout bonus: age < 14d\n AND velocity_percentile > 0.9
|
||||
|
||||
TidalDB-->>App: [{item_id, quality_score} × 20]
|
||||
App-->>User: Jazz browse page renders
|
||||
|
||||
User->>App: Switches sort to "New"
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE new\nLIMIT 20
|
||||
|
||||
Note over TidalDB: Profile: new\nSort: created_at DESC\nNo quality gate.
|
||||
|
||||
TidalDB-->>App: Latest jazz content
|
||||
App-->>User: New jazz renders
|
||||
|
||||
User->>App: Switches sort to "Top All Time"
|
||||
|
||||
App->>TidalDB: RETRIEVE items\nFILTER category: jazz\nUSING PROFILE top_all_time\nDIVERSITY max_per_creator:3\nLIMIT 20
|
||||
|
||||
Note over TidalDB: Profile: top_all_time\nSort: total_completion_weighted_views DESC\nNo recency boost. No decay.\nPure historical quality.
|
||||
|
||||
TidalDB-->>App: All-time top jazz
|
||||
App-->>User: Top jazz renders
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UC-07 · Notification Prioritization
|
||||
|
||||
Of all events since the user was last active, tidalDB decides which are worth a push and in what order they appear in the notification center. Open and dismiss signals feed back to adjust future notification priority per creator.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Job as Background Job
|
||||
participant TidalDB
|
||||
participant Push as Push Service
|
||||
actor User
|
||||
|
||||
Job->>TidalDB: RETRIEVE notifications\nFOR USER @u123\nSINCE @last_seen_timestamp\nUSING PROFILE notification\nLIMIT push:3, inbox:20
|
||||
|
||||
Note over TidalDB: Profile: notification\nCandidates: events from followed creators\n + social graph activity\nScore by:\n relationship_strength(user, creator)\n × item engagement_velocity at event time\n × user notification_open_rate\nHard filter: event_age > 48h → suppress\nHard filter: max 1 per creator per day
|
||||
|
||||
TidalDB-->>Job: push_candidates[3], inbox_candidates[20]
|
||||
|
||||
Job->>Push: Send push notifications (top 3)
|
||||
Push->>User: Push notification arrives
|
||||
|
||||
User->>Push: Taps notification
|
||||
Push->>Job: opened — item @item_id
|
||||
Job->>TidalDB: SIGNAL notification_open\nuser: @u123\ncreator: @creator_id\nitem: @item_id
|
||||
|
||||
Note over TidalDB: Strong positive on user→creator\nrelationship for notification context.\nIncreases future notification priority\nfor this creator.
|
||||
|
||||
TidalDB-->>Job: ok
|
||||
|
||||
User->>Push: Swipes to dismiss
|
||||
Push->>Job: dismissed
|
||||
Job->>TidalDB: SIGNAL notification_dismiss\nuser: @u123\ncreator: @creator_id
|
||||
|
||||
Note over TidalDB: Mild negative on relationship\nfor notification context.\nReduces push frequency\nfrom this creator slightly.
|
||||
|
||||
TidalDB-->>Job: ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UC-15 · Cohort-Scoped Trending
|
||||
|
||||
User explores what's trending within their audience segment. tidalDB scopes signal aggregation to users matching the cohort predicate, ranks by velocity within that cohort, and supports search composition on top.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Opens "Trending For You"
|
||||
|
||||
Note over App: App resolves user's primary cohort<br/>from their attributes:<br/>locale:en-US, age:18-24, interest:music
|
||||
|
||||
App->>TidalDB: RETRIEVE items<br/>USING PROFILE trending<br/>COHORT locale:en-US, age:18-24, interest:music<br/>WINDOW 24h<br/>DIVERSITY max_per_creator:1<br/>LIMIT 25
|
||||
|
||||
Note over TidalDB: 1. Resolve cohort: users matching predicate<br/>2. Load cohort-scoped signal aggregates<br/> (view_velocity, share_velocity scoped<br/> to signals from cohort members)<br/>3. Rank by cohort-scoped velocity<br/>4. Gate: engagement_ratio > 0.03<br/>5. Enforce diversity<br/>6. Return ranked batch
|
||||
|
||||
TidalDB-->>App: [{item_id, cohort_trending_score, velocity_snapshot} × 25]
|
||||
App-->>User: "Trending in your world" renders
|
||||
|
||||
User->>App: Searches "jazz piano" within trending
|
||||
|
||||
App->>TidalDB: SEARCH items<br/>QUERY "jazz piano"<br/>WITHIN TRENDING<br/>COHORT locale:en-US, age:18-24, interest:music<br/>WINDOW 24h<br/>LIMIT 20
|
||||
|
||||
Note over TidalDB: 1. Cohort-scoped trending candidates<br/>2. BM25 text match within candidates<br/>3. Merge: trending_score × text_relevance<br/>4. Return intersection
|
||||
|
||||
TidalDB-->>App: [{item_id, composite_score} × 20]
|
||||
App-->>User: Search-within-trending results render
|
||||
|
||||
User->>App: Switches to "Trending Globally"
|
||||
|
||||
App->>TidalDB: RETRIEVE items<br/>USING PROFILE trending<br/>WINDOW 24h<br/>DIVERSITY max_per_creator:1<br/>LIMIT 25
|
||||
|
||||
Note over TidalDB: Same profile, no cohort scope.<br/>Global signal aggregates used.<br/>Different results than cohort view.
|
||||
|
||||
TidalDB-->>App: Global trending results
|
||||
App-->>User: "Trending Globally" renders
|
||||
```
|
||||
|
||||
**Three layers, one engine.** Global trending, cohort trending, and search-within-cohort-trending are the same ranking profile applied to different signal scopes. The profile doesn't change — the signal aggregation scope does.
|
||||
|
||||
---
|
||||
|
||||
## Core · Engagement Feedback Loop
|
||||
|
||||
Every interaction closes the loop in a single write transaction. There is no ETL, no Kafka consumer, no feature store sync. The next ranking query — even 100ms later — sees the updated state.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor User
|
||||
participant App
|
||||
participant TidalDB
|
||||
|
||||
User->>App: Likes a video
|
||||
|
||||
App->>TidalDB: SIGNAL like\nitem: @item_id\nuser: @u123\ntimestamp: now()
|
||||
|
||||
Note over TidalDB: Atomic write transaction:\n1. Append event to item signal ledger\n2. Update windowed aggregates:\n like_count_1h, _24h, _7d\n3. Recompute like_velocity\n4. Update user→item relationship weight\n5. Increment user→creator interaction_weight\n6. Update user preference vector\n toward item's topic embedding\n7. Attribute signal to user's cohorts\n (increment cohort-scoped counters)\n8. Commit
|
||||
|
||||
TidalDB-->>App: ok
|
||||
|
||||
Note over App: Next RETRIEVE for this user\nreflects updated signals immediately.
|
||||
|
||||
User->>App: Hides a video ("Not interested")
|
||||
|
||||
App->>TidalDB: SIGNAL hide\nitem: @item_id\nuser: @u123
|
||||
|
||||
Note over TidalDB: 1. Set permanent hard-negative flag\n on user→item relationship\n2. Decay user→creator interaction_weight\n3. Update user preference vector\n AWAY from item's topic embedding\n4. Item excluded from all future\n queries for this user
|
||||
|
||||
TidalDB-->>App: ok
|
||||
|
||||
User->>App: Blocks a creator
|
||||
|
||||
App->>TidalDB: SIGNAL block\nuser: @u123\ntarget_creator: @creator_id
|
||||
|
||||
Note over TidalDB: 1. Set permanent hard block\n on user→creator relationship\n2. All items by @creator_id excluded\n from every query for this user\n3. Existing relationship edges zeroed
|
||||
|
||||
TidalDB-->>App: ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Write · Content Ingest
|
||||
|
||||
How a new item enters the system and becomes immediately retrievable and rankable. Cold start is handled by the database via an exploration budget — the application does not manage this.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor Creator
|
||||
participant App
|
||||
participant Embed as Embedding Service
|
||||
participant TidalDB
|
||||
|
||||
Creator->>App: Uploads video\n(title, description, tags, category, transcript)
|
||||
|
||||
App->>Embed: embed(title + description + transcript)
|
||||
Embed-->>App: content_vector[1536]
|
||||
|
||||
App->>TidalDB: WRITE item\nid: @item_id\ncreator: @creator_id\nmetadata: {title, description, tags, category, duration}\nembedding: content_vector\ncreated_at: now()
|
||||
|
||||
Note over TidalDB: 1. Store metadata in entity store\n2. Index text fields into inverted index\n3. Insert vector into ANN index (HNSW)\n4. Initialize signal ledger (all zeros)\n5. Apply new-item exploration budget:\n appears in a small % of for_you feeds\n before signals accumulate\n6. Link to creator entity\n7. Commit — item is immediately queryable
|
||||
|
||||
TidalDB-->>App: ok, @item_id indexed
|
||||
|
||||
App-->>Creator: "Your content is live"
|
||||
|
||||
Note over TidalDB: Item is now retrievable in:\n• Search (text + semantic)\n• Following feeds of creator's followers\n• Browse / category pages\n• Trending (once signals accumulate)\n• For You (exploration budget active)
|
||||
```
|
||||
|
||||
**Cold start is a first-class problem.** New content has no engagement signals. tidalDB handles this natively — new items receive a configurable exploration window proportional to the creator's relationship strength with their existing followers. The application does not manage this.
|
||||
|
||||
---
|
||||
|
||||
## Signal Reference
|
||||
|
||||
| Signal | Type | Decay | Primary Use |
|
||||
|---|---|---|---|
|
||||
| `view` | count | slow (7d half-life) | baseline engagement |
|
||||
| `completion` | ratio 0–1 | very slow | quality signal |
|
||||
| `like` | count | slow | positive sentiment |
|
||||
| `share` | count | medium | virality |
|
||||
| `comment` | count | medium | community |
|
||||
| `skip` | count | fast (1d half-life) | negative quality |
|
||||
| `hide` | bool | permanent | hard negative |
|
||||
| `block` | bool | permanent | hard filter |
|
||||
| `follow` | bool | permanent | relationship |
|
||||
| `interaction_weight` | float | slow | relationship strength |
|
||||
| `dwell_time` | duration | medium | true engagement |
|
||||
| `autoplay_accept` | bool | medium | recommendation quality |
|
||||
| `notification_open` | bool | slow | creator notification priority |
|
||||
| `notification_dismiss` | bool | medium | reduce push frequency |
|
||||
| `search_click` | count + rank | medium | query relevance signal |
|
||||
@ -1,779 +0,0 @@
|
||||
# Use Cases
|
||||
|
||||
Each use case describes a real surface, the query it requires, how signals flow in, and what "correct" looks like. These are the scenarios the database must handle natively and efficiently.
|
||||
|
||||
A note on scope: this document is exhaustive by design. Every filtering mode, sort order, and discovery pattern listed here is something a real user has wanted on YouTube, Twitter, Reddit, Pinterest, Netflix, Spotify, or a media library. tidalDB must support all of them without the application building custom ranking logic.
|
||||
|
||||
A critical addition to this document is UC-15: Cohort-Scoped Trending. This addresses the requirement that trending, rising, and quality signals must be sliceable by audience segment — not just globally or by category, but by demographic, behavioral, and interest-based cohorts. This capability underpins advertiser-facing trend reports, localized discovery, and the "trending for people like you" surface.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [UC-01 · Personalized Feed — For You](#uc-01--personalized-feed--for-you)
|
||||
- [UC-02 · Search](#uc-02--search)
|
||||
- [UC-03 · Trending and Rising](#uc-03--trending-and-rising)
|
||||
- [UC-04 · Following / Subscription Feed](#uc-04--following--subscription-feed)
|
||||
- [UC-05 · Related Content / Up Next](#uc-05--related-content--up-next)
|
||||
- [UC-06 · Browse and Category Discovery](#uc-06--browse-and-category-discovery)
|
||||
- [UC-07 · Notification Prioritization](#uc-07--notification-prioritization)
|
||||
- [UC-08 · Creator Profile Page](#uc-08--creator-profile-page)
|
||||
- [UC-09 · User Library and Personal Collections](#uc-09--user-library-and-personal-collections)
|
||||
- [UC-10 · People and Creator Search](#uc-10--people-and-creator-search)
|
||||
- [UC-11 · Visual and Semantic Search](#uc-11--visual-and-semantic-search)
|
||||
- [UC-12 · Live and Scheduled Content](#uc-12--live-and-scheduled-content)
|
||||
- [UC-13 · Hidden Gems and Breakout Detection](#uc-13--hidden-gems-and-breakout-detection)
|
||||
- [UC-14 · Controversial and Hot Surfaces](#uc-14--controversial-and-hot-surfaces)
|
||||
- [UC-15 · Cohort-Scoped Trending](#uc-15--cohort-scoped-trending)
|
||||
- [Appendix A · Filter Reference](#appendix-a--filter-reference)
|
||||
- [Appendix B · Sort Mode Reference](#appendix-b--sort-mode-reference)
|
||||
- [Appendix C · Signal Reference](#appendix-c--signal-reference)
|
||||
|
||||
---
|
||||
|
||||
## UC-01 · Personalized Feed — For You
|
||||
|
||||
**Surface:** The primary content feed. YouTube home, TikTok FYP, Instagram feed, Twitter For You, Reddit home.
|
||||
|
||||
**The Question:** Given user U right now, what content should they see that they haven't seen, from creators they'll enjoy, with healthy format and topic diversity?
|
||||
|
||||
**Signals Required:**
|
||||
- User implicit preference vector (built continuously from history)
|
||||
- Item engagement velocity — views, completions, shares in the last 24h
|
||||
- User→creator interaction weight (have they engaged with this creator before? how much?)
|
||||
- Social proof — did people the user follows engage with this?
|
||||
- Negative signals — skips, hides, mutes, "not interested" taps
|
||||
- Recency decay on item age
|
||||
- Completion rate as a quality gate (do not surface content people abandon)
|
||||
- Format affinity — does this user prefer short-form, long-form, articles, images?
|
||||
|
||||
**Ranking Profile:** `for_you` — preference match and social proof weighted heavily, moderate recency decay, completion rate as quality floor, skip signals as strong negative.
|
||||
|
||||
**Diversity Constraints:** max 2 items per creator per batch, format mix enforced, minimum 10% exploration budget (creators the user does not follow).
|
||||
|
||||
**Feedback Written Back:**
|
||||
- Viewed → increment view signal, update user→item relationship
|
||||
- Completed → strong positive on completion signal, boost creator weight
|
||||
- Skipped in under 3 seconds → strong skip signal, decay creator weight
|
||||
- Liked or shared → strong positive across all signals
|
||||
- "Not interested" → permanent negative on this item, decay topic weight
|
||||
- "Don't recommend this creator" → hard suppress, equivalent to soft block for ranking
|
||||
|
||||
**What Correct Looks Like:** Feels like it knows the user without being a hall of mirrors. Some expected, some surprising. No creator dominating. Nothing seen this week resurfaced.
|
||||
|
||||
---
|
||||
|
||||
## UC-02 · Search
|
||||
|
||||
Search is the most complex surface because it has the most dimensions. Every sub-feature listed here is something real users rely on daily.
|
||||
|
||||
### 2.1 · Full-Text Keyword Search
|
||||
|
||||
**The Question:** User typed a query — return the most relevant results, ranked for this user specifically.
|
||||
|
||||
**Query Capabilities:**
|
||||
- Basic keyword match: `jazz piano tutorial`
|
||||
- Exact phrase match: `"jazz piano"` returns only items containing that exact sequence
|
||||
- Boolean operators: `jazz AND piano NOT beginner`, `(jazz OR blues) piano`
|
||||
- Exclusion: `-beginner`, `NOT beginner` — never show items matching this term
|
||||
- Wildcard: `jazz pian*` matches piano, pianist, pianos
|
||||
- Field-scoped search: `title:jazz`, `tag:tutorial`, `creator:username`
|
||||
- Hashtag search: `#jazz` matches tagged items directly
|
||||
- Minimum engagement filter inline: `jazz piano min_views:10000`
|
||||
|
||||
**Signals Required:**
|
||||
- BM25 text relevance score (inverted index)
|
||||
- Semantic similarity — query embedding vs item embedding (catches "intro to jazz" matching "jazz for beginners")
|
||||
- User topic engagement history — a beginner gets beginner content elevated
|
||||
- Item quality signals — completion rate, like ratio as secondary ranking
|
||||
- Recency curve — configurable per content type (news decays fast, tutorials decay slowly)
|
||||
|
||||
**Ranking Profile:** `search` — text relevance is the floor, personalization adjusts rank above it. An irrelevant result never surfaces because the user likes the creator.
|
||||
|
||||
**Diversity Constraints:** max 2 results per creator in the first 10 results.
|
||||
|
||||
**Feedback Written Back:**
|
||||
- Click at rank N → positive relevance signal, trains query→item affinity
|
||||
- Immediate back-navigation → negative signal (irrelevant or low quality)
|
||||
- Long dwell after click → strong positive on item and creator for this topic
|
||||
- Zero clicks in a session → weak signal that results were poor for this query
|
||||
|
||||
### 2.2 · Advanced Search Filters
|
||||
|
||||
These are the filters users expect to be able to combine freely. All must be composable — any filter can be combined with any other. See Appendix A for the complete filter reference.
|
||||
|
||||
**By Date / Recency:**
|
||||
- Presets: last hour, today, this week, this month, this year
|
||||
- Custom range: `uploaded_after:2024-01-01 uploaded_before:2024-06-01`
|
||||
- Relative: `uploaded_within:30d`
|
||||
|
||||
**By Duration:**
|
||||
- Presets: short (under 4 minutes), medium (4–20 minutes), long (over 20 minutes)
|
||||
- Custom range: `duration_min:5m duration_max:15m`
|
||||
|
||||
**By Format / Content Type:**
|
||||
- Video, short-form video, live stream, VOD of past live, podcast episode, article, image, image gallery, audio-only, interactive
|
||||
|
||||
**By Quality / Technical Specs:**
|
||||
- Resolution: SD, HD, Full HD, 4K, 8K
|
||||
- HDR, Dolby Vision, Dolby Atmos, spatial audio
|
||||
- Subtitles/captions available, audio description available, sign language version available
|
||||
- Offline/download available
|
||||
|
||||
**By Language:**
|
||||
- Content language, subtitle language available, dubbed version available in language X, original language only
|
||||
|
||||
**By Content Rating / Maturity:**
|
||||
- G, PG, PG-13, R, etc.
|
||||
- Safe search toggle, age-gated content filter, sensitive topics toggle
|
||||
|
||||
**By Creator Attributes:**
|
||||
- Verified only, minimum follower count, creators the user follows only, creators new to the user, exclude a specific creator, search within one creator's catalog
|
||||
|
||||
**By Engagement Thresholds:**
|
||||
- Minimum views, likes, like ratio, comments — lets users filter to proven content
|
||||
|
||||
**By Location / Geography:**
|
||||
- Content created in a region, content about a location, trending in a region
|
||||
|
||||
**By Status / Availability:**
|
||||
- Live right now, premiering soon, subscriber-only, free only, leaving platform soon, downloadable
|
||||
|
||||
**By Community Signals (Reddit / forum-style):**
|
||||
- Flair filter, awarded/gilded only, minimum score, specific community, post type (text/link/image/video/poll), original only (exclude crossposts)
|
||||
|
||||
**By Seen State:**
|
||||
- Unseen only, already seen (user wants to find something they watched before), in progress, saved/bookmarked
|
||||
|
||||
### 2.3 · Search Suggestions and Autocomplete
|
||||
|
||||
- Autocomplete on partial query (prefix match on popular queries)
|
||||
- Trending searches in empty search bar
|
||||
- Personalized suggestions based on search and watch history
|
||||
- Creator name autocomplete, hashtag autocomplete
|
||||
- "Did you mean" typo correction on submitted query
|
||||
- Related query suggestions below results ("People also search for")
|
||||
|
||||
### 2.4 · Saved Searches and Alerts
|
||||
|
||||
- User saves a search query — gets a feed of new results matching it over time
|
||||
- Alert when new content matching a saved search is published
|
||||
- Search history (personal, clearable)
|
||||
- Quick access to recent searches
|
||||
|
||||
### 2.5 · Search Within Scoped Results (Query Composition)
|
||||
|
||||
Search can be composed with other retrieval modes. The application specifies a retrieval scope, and search operates within that candidate set:
|
||||
|
||||
- **Search within trending:** "jazz piano" within globally trending items
|
||||
- **Search within cohort trending:** "jazz piano" within items trending for US users aged 18-24
|
||||
- **Search within following:** "jazz piano" within items from followed creators
|
||||
- **Search within category:** "jazz piano" within the Jazz category (this already works via filters, but the composition model generalizes it)
|
||||
|
||||
Query composition means SEARCH and RETRIEVE are not separate operations — they can be layered. The database handles the intersection efficiently using its query planner.
|
||||
|
||||
---
|
||||
|
||||
## UC-03 · Trending and Rising
|
||||
|
||||
### 3.1 · Trending
|
||||
|
||||
**Surface:** Trending tab, Explore page, "What's happening" sidebar.
|
||||
|
||||
**The Question:** What content is gaining real traction right now — not what has the most views historically?
|
||||
|
||||
**Signals Required:**
|
||||
- Share velocity — rate of new shares (strongest trending signal)
|
||||
- View velocity — rate of new views, not total views
|
||||
- New-user reach — percentage of viewers new to this creator (measures virality, not fanbase loyalty)
|
||||
- Engagement ratio — (likes + comments + shares) / views — filters clickbait
|
||||
- Comment velocity — discussions erupting quickly signal cultural relevance
|
||||
|
||||
**Ranking Profile:** `trending` — velocity signals only, windowed 1h/6h/24h. No personalization. Total view count is explicitly not a primary signal.
|
||||
|
||||
**Scoping (same profile, different candidate sets):**
|
||||
- Global trending
|
||||
- Trending in category/genre
|
||||
- Trending in my language/region
|
||||
- Trending among people I follow
|
||||
- Trending within a specific community
|
||||
- **Trending within a cohort (demographic/behavioral segment)**
|
||||
- **Search within cohort-scoped trending**
|
||||
|
||||
See UC-15 for full cohort-scoped trending specification. Cohort trending uses the same velocity-based ranking profile but scopes signal aggregation to users matching a cohort predicate.
|
||||
|
||||
**Diversity Constraints:** max 1 item per creator in top 10.
|
||||
|
||||
### 3.2 · Rising (Hot / Breakout)
|
||||
|
||||
**The Question:** What new content is overperforming for its age? Designed to surface content that has not yet reached trending thresholds but is clearly on its way.
|
||||
|
||||
This is the Reddit "rising" concept applied broadly.
|
||||
|
||||
**Signals Required:**
|
||||
- Age of content (hard weight — very new content gets a boost)
|
||||
- Engagement velocity relative to creator's own baseline (a small creator getting 10× their normal engagement is "rising")
|
||||
- Engagement velocity relative to category baseline
|
||||
- Share-to-view ratio (high share rate relative to views signals genuine enthusiasm)
|
||||
|
||||
**Ranking Profile:** `rising` — age-weighted velocity. A 2-hour-old video with 5k views and a 15% share rate outranks a 2-day-old video with 100k views and a 0.3% share rate.
|
||||
|
||||
---
|
||||
|
||||
## UC-04 · Following / Subscription Feed
|
||||
|
||||
**Surface:** YouTube Subscriptions tab, Twitter Following tab, Substack inbox, Twitch following list.
|
||||
|
||||
**The Question:** Show me everything from creators I follow, in the right order, with nothing missing.
|
||||
|
||||
**Signals Required:**
|
||||
- Relationship: user follows creator (hard filter — only followed creators)
|
||||
- Recency: primary sort signal
|
||||
- Light quality gate: completion rate as tiebreaker within same-minute posts
|
||||
- Seen flag: filter already-seen items (optional — some users want all posts regardless)
|
||||
|
||||
**Ranking Profile:** `following` — recency-dominant, minimal algorithmic intervention. This is the surface where users feel most strongly that the algorithm should stay out of the way.
|
||||
|
||||
**Diversity Constraints:** None by default. If a followed creator posts 10 times in one day, show all 10.
|
||||
|
||||
**Modes:**
|
||||
- Chronological (pure reverse time order)
|
||||
- Chronological with quality tiebreaker (same timestamp → prefer higher quality)
|
||||
- Algorithmic following (light ranking — surfaces the most engaging posts from follows first, for users who follow too many to consume everything)
|
||||
|
||||
**What Correct Looks Like:** Nothing missing. Nothing reordered dramatically. The user trusts this surface because it reflects their explicit choices.
|
||||
|
||||
---
|
||||
|
||||
## UC-05 · Related Content / Up Next
|
||||
|
||||
**Surface:** YouTube right rail, Spotify Radio, Netflix "More Like This," Pinterest related pins, end-screen recommendations.
|
||||
|
||||
**The Question:** Given what the user just consumed, what is the natural next thing?
|
||||
|
||||
**Signals Required:**
|
||||
- Semantic similarity — embedding distance between source item and candidates
|
||||
- Collaborative filtering — users who engaged with item A also engaged with item B
|
||||
- User preference match — semantically similar AND matches this user's taste
|
||||
- Content journey awareness — a user who just watched beginner content should see intermediate next, not more beginner
|
||||
- Quality gate — completion rate (do not autoplay bad content)
|
||||
- Novelty — do not recommend items the user has already seen
|
||||
|
||||
**Ranking Profile:** `related` — semantic similarity as primary retrieval signal, collaborative filtering as secondary boost, user preference as personalization layer.
|
||||
|
||||
**Diversity Constraints:** avoid same creator as source item in first 3 results (unless on a creator profile page).
|
||||
|
||||
**Feedback Written Back:**
|
||||
- Autoplay accepted → strong positive on source→target pairing
|
||||
- Autoplay skipped under 10 seconds → negative on pairing
|
||||
- Manual click on sidebar → weaker positive than autoplay accept
|
||||
- Saved to watch later → strong positive
|
||||
|
||||
---
|
||||
|
||||
## UC-06 · Browse and Category Discovery
|
||||
|
||||
**Surface:** Genre pages, mood pages, topic pages, aesthetic boards (Pinterest).
|
||||
|
||||
### 6.1 · Standard Browse
|
||||
|
||||
**Sort modes users expect within a category:**
|
||||
|
||||
**Top / Best** — all-time quality rank. Completion rate + like ratio + total reach. Stable. Does not change hourly.
|
||||
|
||||
**New** — pure reverse chronological. No quality gate. Shows everything. Users use this to find content the algorithm hasn't surfaced yet.
|
||||
|
||||
**Hot** — recency + engagement combined. Content decays as it ages regardless of engagement. The Reddit model: score / (age_hours + 2)^gravity. Refreshes meaningfully every hour.
|
||||
|
||||
**Rising** — overperforming new content (see UC-03.2).
|
||||
|
||||
**Trending** — velocity-based, short time window (see UC-03.1).
|
||||
|
||||
**Controversial** — high engagement with polarized sentiment. High comment count, moderate like ratio, high share count. Surfaces content generating strong reactions in both directions.
|
||||
|
||||
**Top: This Hour / Today / This Week / This Month / This Year / All Time** — windowed top sort. Lets users choose their recency preference explicitly. All-Time Top is useful for discovering classics. This Week is useful for staying current without hourly noise.
|
||||
|
||||
**Shuffle / Random** — random sample weighted by quality score. Useful for music, podcasts, and "surprise me" contexts.
|
||||
|
||||
**Alphabetical** — A-Z / Z-A. Useful for structured collections and course curricula.
|
||||
|
||||
**Shortest First / Longest First** — sort by duration. Users looking for something quick explicitly want this.
|
||||
|
||||
**Highest Rated** — explicit critic or audience score where available, distinct from like ratio.
|
||||
|
||||
### 6.2 · Faceted Browse (Multiple Simultaneous Filters)
|
||||
|
||||
All filters must be composable simultaneously. Examples of real user behavior:
|
||||
|
||||
- Genre: Jazz AND Duration: Short AND New (last 7 days)
|
||||
- Format: Podcast AND Language: Spanish AND Top: This Month
|
||||
- Creator: Verified AND Category: Cooking AND Sort: Hot
|
||||
- Mood: Focus AND Duration: Long AND Unseen only
|
||||
- Quality: 4K AND Has Subtitles AND Top: All Time
|
||||
|
||||
The database must handle arbitrary filter combinations without the application implementing them. Faceted queries are a first-class operation.
|
||||
|
||||
### 6.3 · Mood and Aesthetic Filters
|
||||
|
||||
Common on music, video, and Pinterest-style platforms. Moods are not categories — they are cross-cutting signals derived from engagement patterns and embeddings.
|
||||
|
||||
- Mood: Chill, Energetic, Focus, Sad, Happy, Hype, Romantic, Dark, Nostalgic
|
||||
- Aesthetic: Minimalist, Maximalist, Vintage, Futuristic, Cottagecore, Brutalist
|
||||
- Era/Decade: 70s, 80s, 90s, 2000s — useful for music, film, fashion content
|
||||
|
||||
These are best modeled as embedding-space regions rather than explicit tags. A "chill" query retrieves items whose embeddings cluster near what users seeking "chill" content engage with.
|
||||
|
||||
### 6.4 · Color and Visual Filtering (Pinterest Model)
|
||||
|
||||
When items are images or have dominant visual content:
|
||||
|
||||
- Filter by dominant color or color palette
|
||||
- "More like this image" — visual similarity search
|
||||
- Style similarity — find visually similar items even without shared tags
|
||||
- Crop-and-search — user selects a region of an image and searches for items similar to that region
|
||||
|
||||
These require visual embeddings on items. The application provides the embedding. The database handles retrieval and ranking.
|
||||
|
||||
---
|
||||
|
||||
## UC-07 · Notification Prioritization
|
||||
|
||||
**Surface:** Push notifications, in-app notification center, email digest.
|
||||
|
||||
**The Question:** Of all events since the user was last active, which deserve a push? Which deserve inbox prominence?
|
||||
|
||||
**Signals Required:**
|
||||
- Relationship strength — notification from a creator they interact with constantly vs. one they follow but never open
|
||||
- Quality of the triggering item — a video already performing well is more worth notifying about
|
||||
- User notification open rate — are they opening notifications lately? If not, reduce frequency (fatigue signal)
|
||||
- Time since event — events older than 48h are suppressed
|
||||
- Notification type priority — a reply to the user's comment > a new video from a creator they loosely follow
|
||||
|
||||
**Ranking Profile:** `notification` — relationship strength dominant, item quality secondary, strict recency filter.
|
||||
|
||||
**Diversity Constraints:** max 1 push per creator per day, max 3 total pushes per day per user, max 1 per topic cluster per hour.
|
||||
|
||||
**Feedback Written Back:**
|
||||
- Opened → strong positive on creator relationship for notification context
|
||||
- Dismissed → mild negative, reduce future frequency
|
||||
- Notifications disabled for creator → permanent suppress
|
||||
- App opened directly (not via notification) → weak positive on all pending notifications
|
||||
|
||||
---
|
||||
|
||||
## UC-08 · Creator Profile Page
|
||||
|
||||
**Surface:** A creator's profile — their complete catalog, browsable by the visitor.
|
||||
|
||||
**The Question:** Show this creator's content in the best order for this visitor.
|
||||
|
||||
**Modes:**
|
||||
- **Top** — all-time quality. Best first impression for new visitors.
|
||||
- **New** — reverse chronological. For fans keeping up.
|
||||
- **Hot** — currently performing best within the creator's own catalog.
|
||||
- **For You** — which of this creator's items best matches this visitor's preferences. A jazz fan visiting a multi-genre creator sees jazz content elevated even if the creator's pop content has more total views.
|
||||
- **Series / Playlists** — items grouped into explicit collections, ordered within collection.
|
||||
|
||||
**What Correct Looks Like:** A first-time visitor and a longtime subscriber see different orderings on "For You." The new visitor sees the creator's best all-time content. The subscriber sees what they haven't yet watched from this creator.
|
||||
|
||||
---
|
||||
|
||||
## UC-09 · User Library and Personal Collections
|
||||
|
||||
**Surface:** YouTube Library, Spotify Liked Songs, Instagram Saved, Reddit Saved, Pinterest Boards, Watch History.
|
||||
|
||||
### 9.1 · Watch / View History
|
||||
|
||||
- Complete chronological history of items the user viewed
|
||||
- Filterable by: date range, category, creator, format, duration
|
||||
- Searchable by keyword within history
|
||||
- Clearable (individual items or full history)
|
||||
- "Continue Watching" — items viewed but not completed, sorted by most recently viewed
|
||||
- Resume playback position stored per item
|
||||
|
||||
### 9.2 · Saved / Bookmarked Items
|
||||
|
||||
- Items explicitly saved (Watch Later, Saved Posts, Bookmarks)
|
||||
- Sortable by: date saved (default), date published, creator, category, duration
|
||||
- Filterable by: category, creator, format, unseen vs. seen
|
||||
- Expiry detection — saved items that have since been deleted or become unavailable
|
||||
- Bulk management — mark batch as watched, remove batch
|
||||
|
||||
### 9.3 · Liked Items
|
||||
|
||||
- All items the user has liked
|
||||
- Sortable by: date liked, creator, category
|
||||
- Used as a strong signal in preference vector construction
|
||||
|
||||
### 9.4 · User-Created Collections / Boards / Playlists
|
||||
|
||||
- Named collections the user creates and curates
|
||||
- Items can belong to multiple collections
|
||||
- Collections can be private, shared with specific users, or public
|
||||
- Collections themselves are rankable — popular public playlists surface in browse/search
|
||||
- Collaborative collections (multiple users contribute — shared boards, Pinterest-style)
|
||||
|
||||
### 9.5 · Downloads / Offline
|
||||
|
||||
- Items downloaded for offline viewing
|
||||
- Filterable, sortable, manageable separately from online library
|
||||
- Download state as a retrievable attribute in queries
|
||||
|
||||
---
|
||||
|
||||
## UC-10 · People and Creator Search
|
||||
|
||||
**Surface:** Search results "People" tab, "Accounts" tab, creator discovery.
|
||||
|
||||
**The Question:** User wants to find creators, not content.
|
||||
|
||||
**Capabilities:**
|
||||
- Search by creator name, username, handle
|
||||
- Search creators by topic: "find creators who post about jazz"
|
||||
- Filter creators by: follower count range, posting frequency, category, language, location, verified status
|
||||
- "Creators like [creator X]" — semantic similarity between creator embeddings (built from their catalog)
|
||||
- "Creators followed by people I follow" — social graph traversal
|
||||
- "Creators I used to follow" — historical relationship query
|
||||
- Sort creators by: follower count, posting frequency, engagement rate, recent activity
|
||||
|
||||
**Signals Required:**
|
||||
- Creator-level embedding (derived from their item embeddings, aggregated)
|
||||
- Creator engagement rate (average engagement ratio across recent catalog)
|
||||
- Creator posting frequency
|
||||
- Social graph (who follows them, who follows the current user)
|
||||
- User's creator affinity history
|
||||
|
||||
---
|
||||
|
||||
## UC-11 · Visual and Semantic Search
|
||||
|
||||
**Surface:** Pinterest visual search, Google Lens-style search, "find more like this image."
|
||||
|
||||
### 11.1 · Search by Image
|
||||
|
||||
- User uploads an image or selects one from the platform
|
||||
- Find items whose visual embedding is nearest to the query image embedding
|
||||
- Crop-and-search — user selects a region of the image to search against
|
||||
- Combine with text: image embedding + text query vector, merged score
|
||||
|
||||
### 11.2 · Semantic / Intent Search
|
||||
|
||||
Beyond keyword matching — understanding what the user means, not just what they typed.
|
||||
|
||||
- Query: "something relaxing to watch on a rainy day" → system interprets mood/intent, retrieves by embedding similarity to that intent
|
||||
- Query: "that video about the jazz pianist in new orleans I watched last year" → retrieves from user history using semantic match, not exact title
|
||||
- Disambiguation: "jaguar" — is the user searching for the car or the animal? User history and query context disambiguate
|
||||
|
||||
### 11.3 · Multi-Modal Retrieval
|
||||
|
||||
- Text query against image items (find images matching a text description)
|
||||
- Image query against video items (find videos containing visuals similar to a reference image)
|
||||
- Audio fingerprint query against audio items — tidalDB handles the embedding retrieval, not the generation
|
||||
|
||||
---
|
||||
|
||||
## UC-12 · Live and Scheduled Content
|
||||
|
||||
**Surface:** Live tab, "Happening Now," event pages, premiere countdowns.
|
||||
|
||||
**The Question:** What is live right now that this user would care about?
|
||||
|
||||
**Signals Required:**
|
||||
- Live status flag (boolean, real-time)
|
||||
- Scheduled start time and end time
|
||||
- Current viewer count (real-time signal)
|
||||
- Creator relationship weight (live from a creator they care about > random live)
|
||||
- Category match with user preferences
|
||||
- Notification opt-in (did the user set a reminder?)
|
||||
|
||||
**Ranking Profile:** `live` — relationship weight dominant, current viewer count as social proof, category match secondary. Recency is not a concept here — everything is happening now.
|
||||
|
||||
**Discovery of upcoming:**
|
||||
- "Premiering in X hours" — scheduled content with countdown
|
||||
- "Set reminder" → creates a notification relationship between user and item
|
||||
- Calendar-style browse of upcoming events
|
||||
|
||||
**Filtering:**
|
||||
- Live only (exclude VOD)
|
||||
- By category within live
|
||||
- By minimum viewer count
|
||||
- From followed creators only
|
||||
|
||||
---
|
||||
|
||||
## UC-13 · Hidden Gems and Breakout Detection
|
||||
|
||||
**Surface:** "Underrated," "Staff Picks," "You might have missed," editorial surfaces.
|
||||
|
||||
**The Question:** What high-quality content is being overlooked by the algorithm?
|
||||
|
||||
Hidden gems are items with high completion rate and like ratio but low total view count relative to those quality signals. Content that performs well with everyone who sees it but hasn't been seen by many people yet.
|
||||
|
||||
**Signals Required:**
|
||||
- Quality signals: completion rate, like ratio — must be high
|
||||
- Reach signals: total views — must be low relative to quality
|
||||
- Age of content — recent enough to still be worth surfacing
|
||||
- Creator follower count — small/new creators get priority
|
||||
|
||||
**Ranking Profile:** `hidden_gems` — quality signals as primary, inverse of reach as a boost, creator size as a discovery equity signal.
|
||||
|
||||
**What Correct Looks Like:** Content that makes the user think "how have I never seen this before?" Not content that is obscure because it is bad.
|
||||
|
||||
---
|
||||
|
||||
## UC-14 · Controversial and Hot Surfaces
|
||||
|
||||
### 14.1 · Controversial Sort
|
||||
|
||||
**Surface:** Reddit "Controversial" sort, comment sections surfacing heated debates.
|
||||
|
||||
**The Question:** What content is generating strong reactions in both directions?
|
||||
|
||||
Controversial is defined as: high total engagement AND polarized sentiment. High comment count, high share count, but split positive/negative signal ratio. Content people feel strongly about in opposite directions.
|
||||
|
||||
**Signals Required:**
|
||||
- Total engagement (must be high enough to be genuinely controversial, not just unpopular)
|
||||
- Sentiment polarity — ratio of positive to negative signals
|
||||
- Comment velocity — discussions growing quickly
|
||||
- Share count — even content people dislike gets shared for debate
|
||||
|
||||
**Ranking Profile:** `controversial` — maximizes the product of positive and negative engagement signals. A post with 1000 upvotes and 1000 downvotes scores higher than one with 1800 upvotes and 200 downvotes.
|
||||
|
||||
### 14.2 · Hot Sort (Reddit Model)
|
||||
|
||||
**Surface:** Reddit "Hot," Hacker News front page, time-sensitive community surfaces.
|
||||
|
||||
**The Question:** What is the best content right now, with age decay applied?
|
||||
|
||||
Hot rewards early engagement but punishes age. Formula concept: `score / (age_hours + 2)^gravity`. The database exposes this as a native sort mode — the application does not implement the formula.
|
||||
|
||||
**What makes Hot different from Trending:** Trending is pure velocity (rate of change). Hot is cumulative score with age decay. An hour-old post with 500 upvotes scores higher on Hot than a day-old post with 2000 upvotes.
|
||||
|
||||
---
|
||||
|
||||
## UC-15 · Cohort-Scoped Trending
|
||||
|
||||
**Surface:** "Trending for You" (personalized trending), audience-segmented trending dashboards, advertiser-facing trend reports, regional/demographic trend pages.
|
||||
|
||||
**The Question:** What content is gaining traction right now among users who match this profile?
|
||||
|
||||
This is distinct from global trending (UC-03) and personalized feeds (UC-01). Global trending answers "what's popular everywhere." Personalized feeds answer "what should this specific user see." Cohort trending answers "what's resonating with this type of user" — a question that sits between the two.
|
||||
|
||||
**Cohort Definition:**
|
||||
A cohort is a named predicate over user attributes:
|
||||
- Demographic: locale, age range, gender
|
||||
- Interest-based: users who engage with jazz, cooking, tech, etc.
|
||||
- Behavioral: power users, casual browsers, binge watchers
|
||||
- Geographic: users in a specific region or timezone
|
||||
- Composite: US + age 18-24 + interest:jazz (AND logic across dimensions)
|
||||
|
||||
**Signals Required:**
|
||||
- All the same velocity signals as UC-03 (share velocity, view velocity, engagement ratio)
|
||||
- But scoped to signal events generated by users matching the cohort predicate
|
||||
- Cohort-scoped view_velocity(24h) = rate of views from cohort members, not global views
|
||||
|
||||
**Three-Layer Model:**
|
||||
1. **Global trending** — same as UC-03, no cohort filter
|
||||
2. **Cohort trending** — velocity signals filtered to cohort members
|
||||
3. **Search within cohort trending** — text/semantic search composed with cohort trending
|
||||
|
||||
**Ranking Profile:** `cohort_trending` — same velocity-based ranking as `trending`, but candidate set and signal aggregation scoped to cohort.
|
||||
|
||||
**Scoping (composable):**
|
||||
- Cohort: locale:US, age:18-24
|
||||
- Cohort + category: above AND category:jazz
|
||||
- Cohort + search: above AND QUERY "piano tutorial"
|
||||
- Cohort + social: above AND social_graph:@u123
|
||||
|
||||
**Diversity Constraints:** max 1 item per creator in top 10.
|
||||
|
||||
**What Correct Looks Like:** A 22-year-old in Tokyo and a 45-year-old in Texas see different trending pages. Not because of personalization (individual preference), but because different content is genuinely trending within their respective audience segments. An advertiser can see what's trending among their target demographic. A creator can see what's trending in their niche audience.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A · Filter Reference
|
||||
|
||||
All filters must be composable with each other and with any sort mode. A query combining any subset of these is a valid, first-class database operation.
|
||||
|
||||
### Content Attribute Filters
|
||||
|
||||
| Filter | Values | Notes |
|
||||
|---|---|---|
|
||||
| `category` | string or list | multi-select: Jazz OR Blues OR Soul |
|
||||
| `tag` | string or list | multi-select |
|
||||
| `hashtag` | string | exact match |
|
||||
| `flair` | string | community-specific labels |
|
||||
| `format` | video, short, live, vod, podcast, article, image, gallery, audio | |
|
||||
| `duration` | range (min, max) or preset | short / medium / long presets |
|
||||
| `language` | ISO code | content language |
|
||||
| `subtitle_language` | ISO code | subtitles available in this language |
|
||||
| `dubbed_language` | ISO code | dubbed version available |
|
||||
| `resolution` | SD, HD, FHD, 4K, 8K | |
|
||||
| `hdr` | bool | |
|
||||
| `audio_quality` | standard, high, lossless, spatial | |
|
||||
| `has_subtitles` | bool | |
|
||||
| `has_audio_description` | bool | accessibility |
|
||||
| `has_sign_language` | bool | accessibility |
|
||||
| `content_rating` | G, PG, PG-13, R, etc. | |
|
||||
| `safe_search` | bool | |
|
||||
| `sensitive_content` | show, hide, only | |
|
||||
| `status` | published, live, scheduled, archived | |
|
||||
| `availability` | free, premium, subscriber_only | |
|
||||
| `downloadable` | bool | |
|
||||
| `leaving_soon` | bool or date threshold | availability window ending |
|
||||
| `award_count` | minimum int | gilded/awarded |
|
||||
| `has_award` | bool | |
|
||||
| `post_type` | text, link, image, video, poll, crosspost | |
|
||||
| `original_only` | bool | exclude crossposts/reposts |
|
||||
|
||||
### Date and Time Filters
|
||||
|
||||
| Filter | Values |
|
||||
|---|---|
|
||||
| `created_after` | ISO date |
|
||||
| `created_before` | ISO date |
|
||||
| `created_within` | duration: 7d, 30d, 1y |
|
||||
| `created_preset` | hour, today, week, month, year |
|
||||
| `updated_after` | ISO date |
|
||||
| `event_date` | date range for scheduled/live content |
|
||||
|
||||
### Creator Filters
|
||||
|
||||
| Filter | Values |
|
||||
|---|---|
|
||||
| `creator` | creator_id or handle |
|
||||
| `exclude_creator` | creator_id or handle |
|
||||
| `creator_min_followers` | integer |
|
||||
| `creator_max_followers` | integer |
|
||||
| `creator_verified` | bool |
|
||||
| `creator_followed_by_user` | bool |
|
||||
| `creator_new_to_user` | bool — never seen this creator before |
|
||||
| `creator_language` | ISO code |
|
||||
| `creator_location` | region or country |
|
||||
|
||||
### Engagement Threshold Filters
|
||||
|
||||
| Filter | Values |
|
||||
|---|---|
|
||||
| `min_views` | integer |
|
||||
| `max_views` | integer — for hidden gems |
|
||||
| `min_likes` | integer |
|
||||
| `min_like_ratio` | float 0–1 |
|
||||
| `min_comments` | integer |
|
||||
| `min_shares` | integer |
|
||||
| `min_score` | integer — upvotes for forum-style |
|
||||
| `min_completion_rate` | float 0–1 |
|
||||
|
||||
### User State Filters
|
||||
|
||||
| Filter | Values |
|
||||
|---|---|
|
||||
| `seen` | bool — true = already seen, false = unseen only |
|
||||
| `in_progress` | bool — partially watched |
|
||||
| `saved` | bool — in user's saved/bookmarked |
|
||||
| `liked` | bool — user has liked this |
|
||||
| `downloaded` | bool — available offline |
|
||||
| `in_collection` | collection_id |
|
||||
|
||||
### Geographic Filters
|
||||
|
||||
| Filter | Values |
|
||||
|---|---|
|
||||
| `content_region` | country or region code |
|
||||
| `trending_in_region` | country or region code |
|
||||
| `creator_region` | country or region code |
|
||||
| `near_location` | lat/lng + radius |
|
||||
|
||||
### Cohort Filters
|
||||
|
||||
| Filter | Values | Notes |
|
||||
|---|---|---|
|
||||
| `cohort` | cohort_name | Pre-defined named cohort |
|
||||
| `cohort_locale` | locale code (en-US, ja-JP) | User locale match |
|
||||
| `cohort_age` | range (18-24, 25-34) | User age range |
|
||||
| `cohort_interest` | keyword or list | User interest match |
|
||||
| `cohort_engagement_level` | power, regular, casual | Behavioral segment |
|
||||
| `cohort_format_preference` | short, long, mixed | Content format preference |
|
||||
|
||||
---
|
||||
|
||||
## Appendix B · Sort Mode Reference
|
||||
|
||||
All sort modes must be available on any surface. The application specifies the sort mode; tidalDB executes it natively without application-side sorting logic.
|
||||
|
||||
| Sort Mode | Description | Best For |
|
||||
|---|---|---|
|
||||
| `relevance` | Text + semantic match score | Search results |
|
||||
| `personalized` | User preference match | For You surfaces |
|
||||
| `new` | `created_at DESC` | Latest content |
|
||||
| `old` | `created_at ASC` | Archives, chronological viewing |
|
||||
| `top_all_time` | Cumulative quality score, no decay | Classic / best-of |
|
||||
| `top_hour` | Quality score, last 1h | Real-time quality |
|
||||
| `top_today` | Quality score, last 24h | Daily best |
|
||||
| `top_week` | Quality score, last 7d | Weekly digest |
|
||||
| `top_month` | Quality score, last 30d | Monthly recap |
|
||||
| `top_year` | Quality score, last 365d | Annual best |
|
||||
| `hot` | Score / (age + 2)^gravity — decays with time | Community frontpages |
|
||||
| `trending` | Pure engagement velocity | Trending tabs |
|
||||
| `rising` | Velocity relative to baseline, age-boosted | Breakout content |
|
||||
| `controversial` | max(positive_signals × negative_signals) | Debate/discussion |
|
||||
| `hidden_gems` | High quality, low reach, inverse boost | Discovery |
|
||||
| `most_viewed` | Raw view count DESC | All-time popularity |
|
||||
| `most_liked` | Raw like count DESC | Positive sentiment |
|
||||
| `most_commented` | Raw comment count DESC | Discussion |
|
||||
| `most_shared` | Raw share count DESC | Virality |
|
||||
| `shortest` | `duration ASC` | Quick content |
|
||||
| `longest` | `duration DESC` | Deep dives |
|
||||
| `alphabetical_asc` | Title A–Z | Structured catalogs |
|
||||
| `alphabetical_desc` | Title Z–A | |
|
||||
| `shuffle` | Random, weighted by quality | Music, "surprise me" |
|
||||
| `live_viewer_count` | Current viewer count DESC | Live surfaces |
|
||||
| `date_saved` | When user saved/bookmarked DESC | Personal library |
|
||||
| `creator_engagement_rate` | Creator's avg engagement ratio | Creator discovery |
|
||||
|
||||
---
|
||||
|
||||
## Appendix C · Signal Reference
|
||||
|
||||
| Signal | Type | Decay | Primary Use |
|
||||
|---|---|---|---|
|
||||
| `view` | count | slow (7d half-life) | baseline engagement |
|
||||
| `unique_view` | count | slow | deduped reach |
|
||||
| `impression` | count | fast | exposure without engagement |
|
||||
| `completion` | ratio 0–1 | very slow | quality signal |
|
||||
| `partial_completion` | float — last position | slow | continue watching |
|
||||
| `like` | count | slow | positive sentiment |
|
||||
| `dislike` | count | slow | negative sentiment |
|
||||
| `share` | count | medium | virality |
|
||||
| `repost` | count | medium | Twitter RT / reblog equivalent |
|
||||
| `quote` | count | medium | engaged reshare with commentary |
|
||||
| `comment` | count | medium | community engagement |
|
||||
| `reply` | count | medium | discussion depth |
|
||||
| `upvote` | count | medium | forum positive signal |
|
||||
| `downvote` | count | medium | forum negative signal |
|
||||
| `save` | count | slow | intent to return |
|
||||
| `pin` | count | slow | Pinterest save-equivalent |
|
||||
| `collection_add` | count | slow | curation signal |
|
||||
| `download` | count | slow | high-intent engagement |
|
||||
| `screenshot` | count | slow | save-intent (Pinterest model) |
|
||||
| `outbound_click` | count | medium | link content engagement |
|
||||
| `skip` | count | fast (1d half-life) | negative quality |
|
||||
| `skip_intro` | bool | fast | format preference |
|
||||
| `hide` | bool | permanent | hard negative |
|
||||
| `not_interested` | bool | permanent | hard negative on topic |
|
||||
| `block` | bool | permanent | hard filter |
|
||||
| `mute` | bool | permanent | soft filter |
|
||||
| `report` | count | permanent | quality / moderation flag |
|
||||
| `follow` | bool | permanent | relationship |
|
||||
| `unfollow` | event | decays follow signal | relationship decay |
|
||||
| `interaction_weight` | float | slow | relationship strength |
|
||||
| `dwell_time` | duration | medium | true engagement depth |
|
||||
| `replay` | count | medium | exceptional content signal |
|
||||
| `autoplay_accept` | bool | medium | recommendation quality |
|
||||
| `autoplay_reject` | bool | fast | recommendation failure |
|
||||
| `notification_open` | bool | slow | creator notification priority |
|
||||
| `notification_dismiss` | bool | medium | reduce push frequency |
|
||||
| `reminder_set` | bool | slow | intent signal for scheduled content |
|
||||
| `search_click` | count + rank | medium | query relevance |
|
||||
| `search_impression` | count | fast | query exposure |
|
||||
| `award_given` | count | permanent | community quality endorsement |
|
||||
@ -1,225 +0,0 @@
|
||||
# Vision
|
||||
|
||||
## The Problem
|
||||
|
||||
Every platform that serves personalized content — a media library, a social feed, a marketplace, a content discovery surface — eventually builds the same distributed system from scratch. Elasticsearch for retrieval. Redis for hot signals. Kafka for event ingestion. A feature store for user profiles. A vector database for semantic search. A ranking service that tries to stitch all of the above together into a single ordered list.
|
||||
|
||||
This is not an ecosystem. It is scar tissue. The seams between these systems are where correctness dies — stale signals, inconsistent ranking, cache invalidation bugs, and an operational burden that consumes entire engineering teams.
|
||||
|
||||
The root cause is that existing databases were not built with this problem in mind. They treat ranking as an afterthought — a sort clause, a float field, a bolt-on scoring function. They have no concept of a signal that evolves over time, no concept of a user context that shapes relevance, no concept of diversity as a query constraint, no concept of the feedback loop between what a user sees and what the system learns.
|
||||
|
||||
Worse: every team building one of these platforms discovers that their users want the same things. Search with typo tolerance and boolean operators. Filter by duration, date, language, format, quality, creator size, and a dozen other dimensions simultaneously. Sort by trending, hot, rising, controversial, top-this-week, hidden gems, shuffle. Personalize the result of all of the above. Apply diversity constraints. Close the feedback loop.
|
||||
|
||||
These are not exotic requirements. They are table stakes for any serious content platform. And today, every team builds them from scratch, on top of systems not designed for the task.
|
||||
|
||||
## The Thesis
|
||||
|
||||
**Ranking is not a feature. It is a primitive.**
|
||||
|
||||
A database purpose-built for personalized content delivery should model the world the way this problem actually works:
|
||||
|
||||
- Content has metadata, embeddings, and signals. Signals are not fields — they are typed, timestamped streams with native decay, velocity, and windowed aggregation semantics.
|
||||
- Users have preferences, histories, and relationships. These are not rows — they are living profiles that update continuously as events arrive.
|
||||
- Personalization has scopes — global, community, and session/agent — and every scope is user-controlled and revocable.
|
||||
- Agents mediate most interactions. They retrieve context, elicit preferences, and publish structured feedback (reward, tool usage, confidence) as first-class signals. The system must let them read and write memory instantly.
|
||||
- A query is not "give me items matching these filters sorted by this field." It is "given this user, this context, and this surface — what should they see, in what order, subject to these constraints?"
|
||||
- Filters, sort modes, and diversity rules are first-class query citizens — not application logic bolted on top.
|
||||
- Engagement is not application logic that happens to write back into the database. It is a first-class write path that closes the feedback loop natively.
|
||||
|
||||
This is the database that models that world.
|
||||
|
||||
## What It Is
|
||||
|
||||
A single-node-first, embeddable Rust database designed specifically for the **personalized content ranking problem**. It replaces the 6-system stack for this one domain with a single process, a single query interface, and a single operational model.
|
||||
|
||||
It is strongly opinionated. It does not try to be a general-purpose database. It does not try to solve problems outside its domain. Every design decision is made in service of one question: **given a user and a context, what content should they see, and in what order?**
|
||||
|
||||
### First-Class Primitives
|
||||
|
||||
**Entities** are the nodes of the system — Items (content), Users, and Creators. Every entity has metadata, a vector embedding slot, and an attached signal ledger.
|
||||
|
||||
**Signals** are typed, timestamped event streams. The database natively understands signal semantics: velocity (rate of change), decay (exponential or linear, configurable per signal type), and windowed aggregation (last hour, last day, last 7 days, all time). You do not pre-compute `trending_score_7d` and store it in a field. You declare a `view` signal type and query its 7-day windowed velocity at ranking time.
|
||||
|
||||
**Users** have preferences, histories, relationships, and attributes. Attributes include demographics, locale, interests, and behavioral segments. These attributes are queryable for cohort membership and enable cohort-scoped signal aggregation. Some attributes are application-set (locale, age); others are database-computed from engagement patterns (interest affinity, engagement level, format preferences).
|
||||
|
||||
**Relationships** are first-class edges between entities — follows, blocks, interactions, similarity. They are weighted, directional, and traversable in queries.
|
||||
|
||||
**Ranking Profiles** are named, versioned scoring functions declared in schema. They reference signals, relationship weights, recency curves, and diversity rules. A profile is not code deployed separately — it lives in the database, is versioned alongside your data, and can be swapped at query time by name.
|
||||
|
||||
**Cohorts** are named predicates over user attributes — demographic, behavioral, and interest-based segments. A cohort is not a static list of users — it is a live query over user state. "US users aged 18-24 who engage with jazz content" is a cohort. The database maintains per-cohort signal aggregation so that trending, rising, and quality signals can be scoped to any cohort at query time. This enables the three-layer trending model: global trending, cohort-scoped trending, and search within cohort-scoped trending.
|
||||
|
||||
**Personalization Layers** are composable scopes over the same signal model: a user-owned global profile, optional community overlays, and short-lived session/agent context. Ranking can blend them, but ownership stays explicit per layer.
|
||||
|
||||
**Sessions / Agent Context** capture in-flight conversations and tool use. They bind a user, an agent, and a session identifier to short-lived signals (preference hints, rewards, critiques) with aggressive decay. Sessions can be forked, merged, and policy-limited so an agent only sees what it is allowed to remember. Users can revoke agent scope and remove agent-contributed signals from specific personalization layers.
|
||||
|
||||
**The Query** is a single operation that encapsulates candidate retrieval, filtering, ranking, and diversity enforcement:
|
||||
|
||||
```
|
||||
RETRIEVE items
|
||||
FOR USER @user_id
|
||||
FOR SESSION @session_id
|
||||
CONTEXT feed
|
||||
USING PROFILE for_you
|
||||
FILTER unseen, unblocked, format:video, duration:short
|
||||
DIVERSITY max_per_creator:2, format_mix:true
|
||||
LIMIT 50
|
||||
```
|
||||
|
||||
This is what 6 systems currently produce. It is one query here.
|
||||
|
||||
Cohort scoping and query composition extend this further. Trending scoped to a cohort:
|
||||
|
||||
```
|
||||
RETRIEVE items
|
||||
USING PROFILE trending
|
||||
COHORT locale:US, age:18-24, interest:jazz
|
||||
WINDOW 24h
|
||||
DIVERSITY max_per_creator:1
|
||||
LIMIT 25
|
||||
```
|
||||
|
||||
Search within cohort-scoped trending:
|
||||
|
||||
```
|
||||
SEARCH items
|
||||
QUERY "piano tutorial"
|
||||
FOR SESSION @session_id
|
||||
WITHIN TRENDING
|
||||
COHORT locale:US, age:18-24, interest:jazz
|
||||
WINDOW 24h
|
||||
LIMIT 20
|
||||
```
|
||||
|
||||
Three queries, three layers of the same question: what's happening globally, what's happening for people like this, and can I find something specific within that.
|
||||
|
||||
### The Full Query Surface
|
||||
|
||||
tidalDB is designed to handle every retrieval and ranking pattern a content platform needs. This is the complete surface the database covers natively:
|
||||
|
||||
**Retrieval modes:**
|
||||
- Full-text keyword search with BM25 relevance scoring
|
||||
- Exact phrase match, boolean operators (AND/OR/NOT), field-scoped search
|
||||
- Semantic search — query by meaning, not just keywords
|
||||
- Vector similarity search — ANN over item and creator embeddings
|
||||
- Visual similarity search — find items near a reference image embedding
|
||||
- Hybrid search — text relevance + semantic similarity, merged score
|
||||
- User history search — find something the user previously engaged with
|
||||
- Collaborative filtering — "users who engaged with X also engaged with Y"
|
||||
- Social graph traversal — content from or engaged by a user's follows
|
||||
|
||||
**Sort modes (all native, no application implementation required):**
|
||||
- Relevance (text + semantic match)
|
||||
- Personalized (user preference match)
|
||||
- New / Old (chronological)
|
||||
- Hot (score with age decay — Reddit model)
|
||||
- Trending (pure velocity)
|
||||
- Rising (velocity relative to creator/category baseline, age-boosted)
|
||||
- Top: All Time / This Year / This Month / This Week / Today / This Hour
|
||||
- Controversial (maximizes product of positive and negative signals)
|
||||
- Hidden Gems (high quality, low reach)
|
||||
- Most Viewed / Most Liked / Most Commented / Most Shared
|
||||
- Shortest / Longest (by duration)
|
||||
- Alphabetical A-Z / Z-A
|
||||
- Shuffle (random, quality-weighted)
|
||||
- Live Viewer Count (for live surfaces)
|
||||
- Date Saved (for personal library)
|
||||
|
||||
**Filter dimensions (all composable simultaneously):**
|
||||
- Content type / format: video, short, live, VOD, podcast, article, image, gallery, audio
|
||||
- Duration: range or presets (short / medium / long)
|
||||
- Date range: presets or custom (last hour, today, this week, custom range)
|
||||
- Category, tag, hashtag, flair (multi-select, OR logic within dimension)
|
||||
- Language, subtitle language, dubbed language
|
||||
- Technical quality: SD / HD / 4K / HDR / Dolby / spatial audio
|
||||
- Accessibility: subtitles available, audio description, sign language
|
||||
- Content rating / maturity level
|
||||
- Safe search toggle
|
||||
- Status: published, live, scheduled, archived
|
||||
- Availability: free, premium, subscriber-only, downloadable, leaving soon
|
||||
- Creator: specific creator, exclude creator, verified only, follower count range, new to user, followed by user
|
||||
- Engagement thresholds: minimum views, likes, like ratio, comments, shares, completion rate
|
||||
- Community signals: flair, minimum score, award/gilded, post type, original only
|
||||
- User state: unseen, in progress, saved, liked, downloaded, in collection
|
||||
- Geography: content region, creator region, near location, trending in region
|
||||
|
||||
**Discovery surfaces (all driven by the same underlying query engine):**
|
||||
- For You personalized feed
|
||||
- Following / subscription feed
|
||||
- Trending (global, category-scoped, cohort-scoped, social-graph-scoped, region-scoped)
|
||||
- Cohort-scoped discovery — "trending for people like you"
|
||||
- Rising / breakout content
|
||||
- Browse by category with any sort mode
|
||||
- Related / up next recommendations
|
||||
- Hidden gems and underrated content
|
||||
- Live and scheduled content
|
||||
- Mood and aesthetic-filtered browse
|
||||
- Visual similarity browse (Pinterest model)
|
||||
- Creator discovery ("creators like X")
|
||||
- Notification prioritization
|
||||
- Search suggestions and autocomplete
|
||||
- Saved searches as persistent feeds
|
||||
|
||||
Every one of these surfaces is driven by the same underlying query primitives. The application does not implement ranking logic — it specifies profiles, filters, and context.
|
||||
|
||||
### The Feedback Loop
|
||||
|
||||
When a user engages with content — directly or via an agent — that event is written to the database as a signal. The agent can attach structured metadata (reward, confidence, tool invocation) in the same write. The database updates the item's signal ledger, the user's implicit preference profile, the relationship weight, and the session-scoped memory — automatically, as part of the write transaction. The next ranking or grounding query reflects this immediately. There is no Kafka consumer to lag, no feature store sync to schedule, no cache to invalidate.
|
||||
|
||||
Negative signals are equal citizens. A skip, a hide, a block, a "not interested," a downvote — these update the system with the same immediacy and precision as a like or a completion.
|
||||
Feedback intent is typed (`skip_for_now`, `not_for_me`, `low_quality`, `hide/mute/block`) and scope-aware (local, community, session/agent), so ranking updates preserve meaning rather than collapsing all negatives into one bucket.
|
||||
|
||||
## What It Is Not
|
||||
|
||||
It is not a general-purpose document store. It is not a replacement for PostgreSQL for your transactional data. It is not trying to win the NewSQL wars or build a distributed OLAP engine.
|
||||
|
||||
It is not schema-free. Strong opinions about data shape enable strong guarantees about ranking correctness.
|
||||
|
||||
It is not trying to generate embeddings. It accepts vectors — you bring your model, you bring your embeddings, you write them in. The database owns retrieval and ranking over those vectors, not generation.
|
||||
|
||||
It is embeddable first — it runs in your process with zero operational overhead. But it is designed for scale from day one. Key encoding, storage isolation, and signal aggregation are all partitioning-ready. The single-node deployment is the first target, not the ceiling. When you outgrow one node, the architecture supports horizontal scaling without a rewrite.
|
||||
|
||||
It is not trying to solve moderation, payments, authentication, or content delivery. It solves one problem: given a user and a context, what content should they see, and in what order.
|
||||
|
||||
## Design Principles
|
||||
|
||||
**Temporal decay is a type, not a formula you write.** Signal half-lives are declared in schema. The database applies them at query time.
|
||||
|
||||
**Negative signals are equal citizens.** A skip, a hide, a block, a mute, a downvote — these are not the absence of positive engagement. They are data. They belong in the ranking function with the same weight and precision as a like.
|
||||
|
||||
**Feedback intent is explicit.** "Not for me," "low quality," and "skip for now" are different semantics, with different ranking effects and retention policies.
|
||||
|
||||
**All sort modes are native.** Trending, hot, rising, controversial, hidden gems, shuffle — these are built-in sort modes, not formulas the application implements and passes in. The application names a sort mode. The database executes it correctly.
|
||||
|
||||
**All filters are composable.** Any combination of filter dimensions produces a valid, efficiently-executed query. There is no special-casing for "common" filter combinations. Faceted queries are first-class.
|
||||
|
||||
**Diversity is a query constraint, not application logic.** "No more than 2 items per creator" does not belong in your API layer. It belongs in the query.
|
||||
|
||||
**The write path and the read path are one system.** Engagement events and ranking queries share a storage model and a signal ledger. There is no ETL between them.
|
||||
|
||||
**Cold start is handled by the database.** New content with no signals gets an exploration budget. New users with no history get a sensible default experience. The application does not manage this.
|
||||
|
||||
**Cohorts are live queries, not static lists.** A cohort is a predicate over user attributes — demographics, interests, behavioral segments. Users flow in and out of cohorts as their attributes change. Signal aggregation runs per-cohort so trending and quality signals reflect what's happening within any audience segment.
|
||||
|
||||
**Agents own managed contexts.** Sessions scope short-lived memory, rewards, and tool usage. Agents can only read/write within their sessions, and policy guards live in schema, not ad-hoc middleware.
|
||||
|
||||
**Personalization is user-owned and revocable.** Users can opt into community overlays, scope agent access, and remove scoped contributions from future ranking.
|
||||
|
||||
**Correctness over cleverness.** Ranking is already approximate by nature. The database does not need to be more clever than the signals it has. It needs to be fast, consistent, and operationally simple.
|
||||
|
||||
## Who This Is For
|
||||
|
||||
Engineering teams building any surface where content is ranked for a user — media libraries, social feeds, content discovery, search — who are currently operating a multi-system stack and paying the consistency, latency, and operational cost of the seams between those systems.
|
||||
|
||||
The target developer has domain data that fits the entity/signal/relationship model, has immediate use cases that need this in production, and values a single system with sharp opinions over a flexible system with unlimited configuration.
|
||||
|
||||
The target scale is platforms serving millions of users across diverse audiences — where "what's trending" means different things to different cohorts, and the ability to slice engagement signals by audience segment is not a nice-to-have but the core product question.
|
||||
|
||||
## The Name
|
||||
|
||||
**tidalDB** — the tide that surfaces the right content for the right person at the right time. Rising signals, ebbing decay, a natural rhythm of discovery.
|
||||
|
||||
*(The idea matters more than the label.)*
|
||||
|
||||
---
|
||||
|
||||
*This is a focused tool for a focused problem. It will do one thing and do it correctly.*
|
||||
@ -1,339 +0,0 @@
|
||||
# Content Strategy
|
||||
|
||||
Blog posts mapped to the tidalDB roadmap. Each entry identifies the moment worth writing about, the thesis that makes it shareable, and the type of post it demands.
|
||||
|
||||
The audience is engineers who have built or are currently maintaining recommendation and discovery systems -- the people running the 6-system stack this database replaces. They know what Kafka lag feels like at 3am. They know why cache invalidation bugs in the ranking pipeline are the ones that never get root-caused. They will smell marketing language from the first sentence. Respect that.
|
||||
|
||||
---
|
||||
|
||||
## Publishing Principles
|
||||
|
||||
**Write when something is true, not when something is scheduled.** A blog post published the day a milestone passes its UAT is credible. A blog post published before the code works is fiction.
|
||||
|
||||
**One insight per post.** The reader should leave with a single idea they did not have before. If the post contains two insights, it is two posts.
|
||||
|
||||
**Code proves claims.** Every technical assertion is backed by a code example or a benchmark number from the actual codebase. Not a prototype. Not a plan. The shipped code.
|
||||
|
||||
**The title is the thesis.** If the title does not work as a standalone sentence that makes an engineer stop scrolling, the post is not ready.
|
||||
|
||||
---
|
||||
|
||||
## Content Calendar
|
||||
|
||||
### Pre-Implementation (Now)
|
||||
|
||||
These posts can be written before the engine is feature-complete. They draw on the vision, architecture research, and the problem space -- not on shipped code.
|
||||
|
||||
#### Post 1: "Every content platform builds the same 6 systems from scratch" [PUBLISHED]
|
||||
|
||||
- **Type:** Vision / Problem Statement
|
||||
- **Thesis:** The Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service stack is not an architecture. It is scar tissue. The seams between these systems are where correctness dies.
|
||||
- **Source material:** VISION.md, thoughts.md (Part VI)
|
||||
- **When to publish:** Any time. This post defines the problem and does not depend on implementation progress.
|
||||
- **Why it matters:** This is the foundational narrative. Every subsequent post assumes the reader understands this problem. It also serves as the litmus test for whether the audience cares -- if this post does not resonate, the subsequent ones will not either.
|
||||
- **Structure:** Problem statement. The 6 systems named and indicted. The seams enumerated (stale signals, ETL lag, cache invalidation, operational burden). The thesis: ranking is not a feature, it is a primitive. End with the one-query vision, not with a product pitch.
|
||||
|
||||
---
|
||||
|
||||
### Milestone 1: Signal Engine
|
||||
|
||||
M1 proves that temporal signals with O(1) decay, velocity, and windowed aggregation work as a database primitive. This is the most technically interesting milestone for blog content because the math is elegant and the performance numbers are dramatic.
|
||||
|
||||
#### Post 2: "Running decay scores are O(1) -- here is the math" [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** m1p4 (Signal Ledger) completion
|
||||
- **Thesis:** The forward-decay formula `S(t) = S(t_prev) * exp(-lambda * dt) + weight` eliminates raw-event scanning at query time. Three `exp()` calls on write, one on read. 15 nanoseconds per entity. Every platform computing `trending_score = views / (age + 2)^1.8` in application code is doing O(N) work that should be O(1).
|
||||
- **Source material:** docs/research/tidaldb_signal_ledger.md, ARCHITECTURE.md (Signal System section), m1p4 task docs
|
||||
- **When to publish:** After m1p4 passes UAT with benchmark numbers in hand.
|
||||
- **Code to include:** The `EntitySignalState` struct. The forward-decay write path. The out-of-order event correction. Benchmark output showing 200-entity scoring pass under 5 microseconds.
|
||||
- **Why it matters:** This is the post that demonstrates tidalDB is not vaporware. The math is verifiable. The benchmarks are reproducible. Engineers who have implemented trending scores in Redis will immediately understand the value.
|
||||
|
||||
#### Post 3: "What three databases taught us before we wrote a line of code" [PUBLISHED]
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Roadmap phase:** m1p1-m1p3 completion (the foundation phases)
|
||||
- **Thesis:** We studied Engram (cognitive memory), Citadel (append-only logging), and StemeDB (knowledge graph) -- three purpose-built databases in the same codebase -- and stole their best patterns. WAL-first durability from Citadel. Cache-line aligned hot structs from Engram. Subject-prefix key encoding from StemeDB. Background materialization from StemeDB. Here is what converged and what the gaps taught us.
|
||||
- **Source material:** thoughts.md (all six parts), CODING_GUIDELINES.md
|
||||
- **When to publish:** After m1p3 (Storage Engine) is complete. The patterns referenced are already implemented.
|
||||
- **Code to include:** Key encoding format. Cache-line aligned struct. Group commit writer. Side-by-side comparison of the pattern in the source database and in tidalDB.
|
||||
- **Why it matters:** Engineers respect builders who study prior art. This post establishes technical credibility and shows the architectural foundation is grounded in real patterns, not invented from scratch.
|
||||
|
||||
#### Post 4: "Signals wrote 100ms ago. The query sees them now." [PUBLISHED]
|
||||
|
||||
- **Type:** Devlog / Milestone Announcement
|
||||
- **Roadmap phase:** m1p5 (Entity CRUD and Signal Write API) -- M1 complete
|
||||
- **Thesis:** Milestone 1 is done. A developer can open a tidalDB instance, define signal types with decay rates and windows, write 10,000 engagement events, and read back decay-correct scores that match analytical computation to 6 decimal places. Including after a crash. The UAT scenario passes.
|
||||
- **Source material:** The m1p5 integration test, benchmark results, git log for the M1 period
|
||||
- **When to publish:** The day M1 UAT passes.
|
||||
- **Code to include:** The full UAT scenario (or a clean excerpt). `TidalDB::open()` with schema. Signal write. Decay score read. Before/after crash recovery.
|
||||
- **Why it matters:** This is the first "it works" post. It converts skeptics from "interesting idea" to "this is real." The UAT code is the proof.
|
||||
|
||||
---
|
||||
|
||||
### Milestone 2: Ranked Retrieval
|
||||
|
||||
M2 proves that a single query can retrieve, filter, score, and enforce diversity over live signals. This is where tidalDB stops being a signal engine and starts being a database.
|
||||
|
||||
#### Post 5: "One query. Six systems. Under 50 milliseconds." [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive / Announcement
|
||||
- **Roadmap phase:** m2p5 (RETRIEVE Query Executor) -- M2 complete
|
||||
- **Thesis:** `RETRIEVE items USING PROFILE trending DIVERSITY max_per_creator:1 LIMIT 25` executes in under 50ms on 10K items. It retrieves candidates via ANN, filters by metadata, scores using live decay signals and velocity, enforces diversity, and returns a ranked list. That is what Elasticsearch + Redis + a ranking service produce. It is one query here.
|
||||
- **Source material:** m2p5 integration test, benchmark results, the dependency DAG showing how all M2 phases compose
|
||||
- **When to publish:** After M2 UAT passes.
|
||||
- **Code to include:** The RETRIEVE query. The ranked result with signal snapshots. The trending profile definition. A before/after signal burst showing the ranking change.
|
||||
- **Why it matters:** This is the money post. The one-query thesis is no longer a vision document -- it is a benchmark. Engineers who operate the 6-system stack will immediately understand what this eliminates.
|
||||
|
||||
#### Post 6: "Diversity enforcement in 3 microseconds" [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** m2p4 (Diversity Enforcement)
|
||||
- **Thesis:** "No more than 2 items per creator" does not belong in your API layer. It belongs in the query. tidalDB enforces diversity as a post-scoring reordering pass -- it does not reduce result count. The greedy selection algorithm runs in under 3 microseconds for 200 candidates.
|
||||
- **Source material:** m2p4 task docs, VISION.md (diversity section), benchmark results
|
||||
- **When to publish:** After m2p4 is complete.
|
||||
- **Code to include:** The DiversitySpec. The greedy selector. A concrete example showing reordering (creator A dominates pre-diversity, balanced post-diversity). Benchmark numbers.
|
||||
- **Why it matters:** Every team building a feed implements diversity in the API layer. Showing that it belongs in the database -- and costs 3 microseconds -- is a strong differentiator. This is the kind of post that gets shared in Slack channels.
|
||||
|
||||
#### Post 7: "Ranking profiles are data, not code" [PUBLISHED]
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Roadmap phase:** m2p3 (Ranking Profile Engine)
|
||||
- **Thesis:** Changing how content is ranked should not require a code change, a deployment, or a restart. tidalDB treats ranking profiles as versioned schema declarations. Define a profile. Name it. Swap it at query time. A/B test two profiles by name. The database executes the entire pipeline.
|
||||
- **Source material:** m2p3 task docs, API.md (ranking profiles section), VISION.md
|
||||
- **When to publish:** After m2p3 is complete.
|
||||
- **Code to include:** A `trending` profile definition. A `for_you` profile definition. The same RETRIEVE query with two different profile names producing different orderings. The profile versioning API.
|
||||
- **Why it matters:** This reframes ranking as a database concern. Engineers who maintain ranking services as separate microservices will recognize the operational simplification.
|
||||
|
||||
---
|
||||
|
||||
### Milestone 3: Personalized Ranking
|
||||
|
||||
M3 is where the feedback loop closes. Signal writes update the user's preference vector, the creator's interaction weight, and the item's signal ledger -- atomically, in one write. The "For You" query works.
|
||||
|
||||
#### Post 8: "The feedback loop that closes in one write" [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** m3p2 (Feedback Loop) completion
|
||||
- **Thesis:** When a user likes an item, the database atomically updates: the item's like count, the user-to-creator interaction weight, and the user's preference vector (shifted toward the item's embedding). One `db.signal("like", ...)` call. No Kafka consumer to lag. No feature store to sync. No cache to invalidate. The next ranking query -- even 100ms later -- reflects the change.
|
||||
- **Source material:** m3p2 task docs, ARCHITECTURE.md (Write Path section), SEQUENCE.md
|
||||
- **When to publish:** After m3p2 passes UAT.
|
||||
- **Code to include:** The signal write. The 10-step atomic update path. A before/after query showing the preference shift. The property test that proves hidden items and blocked creators never surface.
|
||||
- **Why it matters:** The closed feedback loop is the core architectural thesis of tidalDB. This post proves it works. It is the strongest argument against the 6-system stack, because the stack's primary failure mode is feedback lag.
|
||||
|
||||
#### Post 9: "Negative signals are equal citizens" [PUBLISHED]
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Roadmap phase:** m3p2 (Feedback Loop)
|
||||
- **Thesis:** A skip is not the absence of a like. It is data. tidalDB treats negative signals -- skips, hides, blocks, "not interested" -- with the same precision and immediacy as positive signals. A skip within 3 seconds is a strong quality signal. A hide creates a permanent exclusion. A block removes all of a creator's content from all future queries. These are not afterthoughts. They are first-class signal types with their own decay rates, velocity, and ranking weight.
|
||||
- **Source material:** VISION.md (negative signals section), USE_CASES.md (UC-01 feedback), m3p2 task docs
|
||||
- **When to publish:** After m3p2 is complete. Can be bundled with or separated from Post 8.
|
||||
- **Code to include:** Signal type definitions for skip, hide, block. The penalty clause in a ranking profile. The property test: 10,000 random signal sequences never produce a result where a hidden item or blocked creator appears.
|
||||
- **Why it matters:** Most recommendation systems handle negative feedback as an afterthought -- a manual "not interested" button that writes to a separate blocklist. tidalDB's approach is architecturally different and engineers building these systems will recognize the improvement immediately.
|
||||
|
||||
#### Post 10: "Cold start without application logic" [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** m3p3 (Personalized Ranking Profiles)
|
||||
- **Thesis:** New items with no signals get an exploration budget. New users with no history get a sensible default from population-level signals. The application does not manage either. The exploration rate decays as signals accumulate. This is declared per ranking profile, not implemented in application code.
|
||||
- **Source material:** m3p3 task docs, VISION.md (cold start section)
|
||||
- **When to publish:** After m3p3 is complete.
|
||||
- **Code to include:** The exploration budget in a profile definition. A new item appearing in a for_you feed despite having zero signals. The decay of exploration as signals arrive.
|
||||
- **Why it matters:** Cold start is the problem everyone hacks around and no one solves cleanly. Showing a database-native solution is a strong differentiator.
|
||||
|
||||
---
|
||||
|
||||
### Milestone 5: Hybrid Search
|
||||
|
||||
M5 merges full-text search with semantic similarity and signal-ranked results. Search and retrieval become the same system.
|
||||
|
||||
#### Post 11: "Search and ranking are the same system" [PUBLISHED]
|
||||
|
||||
- **Type:** Technical Deep Dive / Architecture Preview
|
||||
- **Roadmap phase:** Published during M4. Describes the architecture that M5 will complete.
|
||||
- **Status:** PUBLISHED. Written as an architectural intent post -- explains the unified pipeline design, what is already built (RETRIEVE, USearch, ranking), and what remains (Tantivy, RRF fusion, SEARCH query). Does not claim SEARCH is shipped.
|
||||
- **Thesis:** Text retrieval, vector retrieval, and signal-based ranking belong in one pipeline. The data model is already unified. Fusion is arithmetic. The RETRIEVE pipeline, the HNSW index, and the ranking profiles are all in place. Three pieces of wiring remain.
|
||||
- **Source material:** Published post at `site/content/blog/search-and-ranking.mdx`
|
||||
- **Why it matters:** Frames the M5 work for the audience before it ships. The architectural argument stands regardless of what's wired today.
|
||||
|
||||
#### Post 12: "Tantivy as a derived index, not a source of truth"
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Roadmap phase:** m5p1 (Tantivy Integration)
|
||||
- **Thesis:** The entity store is the source of truth. Tantivy is a materialized view. If the index is corrupted, it can be rebuilt from the entity store. Crash recovery replays from a stored sequence number. Consistency is DB-primary, not two-phase commit. This is simpler, deterministic, and the right model for an embedded database.
|
||||
- **Source material:** docs/research/tantivy.md, m5p1 task docs (once written), ARCHITECTURE.md
|
||||
- **When to publish:** After m5p1 is complete.
|
||||
- **Code to include:** The outbox pattern. The crash recovery sequence number. The background indexer. The consistency model.
|
||||
- **Why it matters:** This is a useful architectural pattern beyond tidalDB. Engineers building systems with derived indexes will find this directly applicable.
|
||||
|
||||
---
|
||||
|
||||
### Milestone 6: Full Surface Coverage
|
||||
|
||||
M6 completes all 14 use cases. The content here shifts from "how does the engine work" to "what can you build with it."
|
||||
|
||||
#### Post 13: "14 use cases, one query engine"
|
||||
|
||||
- **Type:** Devlog / Announcement
|
||||
- **Roadmap phase:** M6 complete
|
||||
- **Thesis:** For You feeds, trending, search, following, related content, notifications, hidden gems, controversial, live content, creator discovery, user library, cohort-scoped trending -- every surface a content platform needs, driven by the same query primitives. The application specifies profiles, filters, and context. The database executes ranking.
|
||||
- **Source material:** USE_CASES.md, M6 UAT results
|
||||
- **When to publish:** After M6 UAT passes.
|
||||
- **Code to include:** A curated selection of 4-5 queries spanning different surfaces (for_you, trending, search, hidden_gems, cohort_trending). Each with a brief setup and result.
|
||||
- **Why it matters:** This is the completeness post. It demonstrates that the database is not a toy or a prototype -- it handles the full surface area of a real content platform.
|
||||
|
||||
#### Post 14: "Cohort-scoped trending: what is hot for people like you"
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** M6, cohort-scoped trending phase
|
||||
- **Thesis:** "What's trending" means different things to different audiences. A 22-year-old in Tokyo and a 45-year-old in Texas see different trending pages -- not because of personalization (individual preference), but because different content is genuinely trending within their respective audience segments. tidalDB maintains per-cohort signal aggregation using RoaringBitmaps for O(1) membership testing and sparse fan-out for storage efficiency.
|
||||
- **Source material:** USE_CASES.md (UC-15), ARCHITECTURE.md (Cohort-scoped aggregation), API.md (Cohort Definitions)
|
||||
- **When to publish:** After cohort-scoped trending passes integration tests.
|
||||
- **Code to include:** Cohort definition. Three-layer query (global trending, cohort trending, search within cohort trending). The fan-out write path. Storage cost analysis.
|
||||
- **Why it matters:** Cohort-scoped trending is a differentiator. Most systems compute trending globally. Slicing by audience segment is a product feature that usually requires a separate analytics pipeline. tidalDB does it natively.
|
||||
|
||||
---
|
||||
|
||||
### Production Hardening
|
||||
|
||||
The final phase is about trust. The content shifts from "what it does" to "why you can trust it."
|
||||
|
||||
#### Post 15: "Kill it at any point. It comes back correct."
|
||||
|
||||
- **Type:** Technical Deep Dive
|
||||
- **Roadmap phase:** Production hardening -- crash recovery hardening phase
|
||||
- **Thesis:** We injected faults at every write-path stage. Recovery time is under 30 seconds at 1M items. WAL replay produces state identical to pre-crash. No phantom items, no lost signals, no inconsistent aggregates. The WAL is the source of truth. Everything else is derived state that can be rebuilt.
|
||||
- **Source material:** Crash recovery test results, fault injection methodology, WAL implementation
|
||||
- **When to publish:** After crash recovery hardening passes.
|
||||
- **Code to include:** The crash simulation test. Recovery time measurements. The WAL checkpoint and replay sequence.
|
||||
- **Why it matters:** Trust is the precondition for adoption. Engineers will not embed a database they cannot crash-test. This post is the trust credential.
|
||||
|
||||
#### Post 16: "Graceful degradation: less precise, never wrong"
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Roadmap phase:** Production hardening -- graceful degradation phase
|
||||
- **Thesis:** Under 3x overload, tidalDB does not return errors. It reduces candidate set size, uses coarser aggregates, skips diversity enforcement, and serves from materialized cache -- in that order. Results are less precise but never incorrect. The degradation order is documented and configurable.
|
||||
- **Source material:** Graceful degradation task docs, ARCHITECTURE.md (Graceful degradation)
|
||||
- **When to publish:** After graceful degradation is complete.
|
||||
- **Code to include:** The degradation cascade. Load test results at 1x, 2x, 3x. Latency distribution at each level.
|
||||
- **Why it matters:** This is how production systems should behave. Engineers who have been paged for "ranking service returned 500" will appreciate a system that degrades gracefully instead.
|
||||
|
||||
---
|
||||
|
||||
### Ongoing / Anytime
|
||||
|
||||
These posts are not tied to specific milestones. They can be written whenever the insight is clear.
|
||||
|
||||
#### "Why not SQL"
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Status:** READY -- code shipped, decision documented
|
||||
- **Thesis:** The custom query language exists because SQL cannot express ranking semantics without losing optimization opportunities. `FOR USER` means "load this user's preference vector and relationship graph." `USING PROFILE` means "apply this named scoring function." `DIVERSITY` means "enforce post-ranking constraints." These are not WHERE clauses.
|
||||
- **Source material:** thoughts.md (Part II.4), VISION.md (query examples), API.md
|
||||
- **Code to read:**
|
||||
- `tidal/src/query/retrieve.rs` -- `RetrieveBuilder` showing FOR USER, USING PROFILE, DIVERSITY as typed builder methods, not string predicates
|
||||
- `tidal/src/ranking/profile.rs` -- `RankingProfile` and `CandidateStrategy` showing how profiles express scoring intent that SQL GROUP BY cannot
|
||||
- `tidal/src/query/executor.rs` -- dispatcher showing that FOR USER loads preference vector and relationship graph -- state SQL has no model for
|
||||
- **When to publish:** Any time after M1. Best paired with M2 when the RETRIEVE query is functional. Both are complete.
|
||||
|
||||
#### "Why we chose fjall over RocksDB (for now)"
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Status:** READY -- code shipped, decision documented
|
||||
- **Thesis:** Pure Rust, `#![forbid(unsafe_code)]`, fast compile times, trait-abstracted for swap. fjall is not the fastest LSM-tree. It is the right one for an embeddable database built by a small team that values correctness over raw throughput, with a trait boundary that makes the decision reversible.
|
||||
- **Source material:** thoughts.md (Part V.9), CODING_GUIDELINES.md
|
||||
- **Code to read:**
|
||||
- `tidal/src/storage/engine.rs` -- the `StorageEngine` trait (six methods, zero fjall imports -- this is the abstraction boundary)
|
||||
- `tidal/src/storage/fjall.rs` -- `FjallBackend` implementing the trait; note the `fjall::Keyspace` is the only fjall type that crosses the boundary
|
||||
- `tidal/src/storage/memory.rs` -- `InMemoryBackend` proving the trait is genuinely swappable (used in all tests)
|
||||
- `tidal/Cargo.toml` -- fjall version pin, no `unsafe` in the crate's feature flags
|
||||
- **When to publish:** Any time. Code has been shipped since m1p3.
|
||||
|
||||
#### "USearch, not from scratch"
|
||||
|
||||
- **Type:** Architecture Decision Record
|
||||
- **Status:** READY -- code shipped, decision documented
|
||||
- **Thesis:** Correct, high-performance, concurrent HNSW with SIMD distance computation is 6-12 months of dedicated work. We are not a vector database company. USearch runs in ScyllaDB, ClickHouse, and DuckDB. The FFI boundary is thin. Build what differentiates you. Borrow what does not.
|
||||
- **Source material:** docs/research/ann_for_tidaldb.md, ARCHITECTURE.md (Vector Index)
|
||||
- **Code to read:**
|
||||
- `tidal/src/storage/vector/mod.rs` -- `VectorIndex` trait and the module comment explaining the design decisions (VectorId = u64, L2 squared, ef_search uniformity)
|
||||
- `tidal/src/storage/vector/usearch_index.rs` -- `UsearchIndex` wrapping the USearch FFI; the wrapper is thin by design
|
||||
- `tidal/src/storage/vector/planner.rs` -- `AdaptiveQueryPlanner` with four strategies (HNSW, in-graph filter, widened beam, pre-filter brute-force) -- this is tidalDB's contribution on top of the borrowed index
|
||||
- `tidal/src/storage/vector/brute.rs` -- `BruteForceIndex` and `MockVectorIndex` proving the trait boundary is real
|
||||
- **When to publish:** Any time. Code has been shipped since m2p1.
|
||||
|
||||
---
|
||||
|
||||
## Post Cadence
|
||||
|
||||
| Milestone | Posts | Approximate Pace | Status |
|
||||
|-----------|-------|-----------------|--------|
|
||||
| Pre-implementation | 1 | Publish when ready | PUBLISHED |
|
||||
| M1 (Signal Engine) | 2-3 | One per phase completion | PUBLISHED |
|
||||
| M2 (Ranked Retrieval) | 3 | One per major phase | PUBLISHED |
|
||||
| M3 (Personalized Ranking) | 2-3 | One per key insight | PUBLISHED |
|
||||
| M4 (Agent Session Layer) | 1 (Post 11, architectural preview) | Published during M4 | PUBLISHED |
|
||||
| M5 (Hybrid Search) | 1 (Post 12) | After m5p1 ships | Blocked on M5 |
|
||||
| M6 (Full Coverage) | 2 (Posts 13-14) | At milestone boundaries | Blocked on M6 |
|
||||
| Production Hardening | 2 (Posts 15-16) | At milestone boundaries | Blocked on hardening phase |
|
||||
| Ongoing / ADRs | 3 (fjall, SQL, USearch) | When the decision is fresh | READY |
|
||||
|
||||
**Target: 16-20 posts across the full roadmap.** Not more. Each one earns its place.
|
||||
|
||||
---
|
||||
|
||||
## What Not to Write
|
||||
|
||||
- Progress updates that are changelogs. ("We merged 47 PRs this month.") Nobody cares.
|
||||
- Posts that announce intent without shipped code. ("We plan to build...") Ship first.
|
||||
- Posts with titles that are labels. ("Q1 Update," "Phase 3 Complete.") The title is the thesis.
|
||||
- Posts that explain what a concept is without showing why the reader should care. ("Windowed aggregation is...") Start with the problem.
|
||||
- Posts that use "we're excited to announce." You are not excited. You are precise.
|
||||
|
||||
---
|
||||
|
||||
## Reference: Roadmap to Post Mapping
|
||||
|
||||
| Roadmap Phase | Post # | Title | Status |
|
||||
|---------------|--------|-------|--------|
|
||||
| Pre-implementation | 1 | Every content platform builds the same 6 systems from scratch | PUBLISHED |
|
||||
| m1p1-m1p3 (Foundation) | 3 | What three databases taught us before we wrote a line of code | PUBLISHED |
|
||||
| m1p4 (Signal Ledger) | 2 | Running decay scores are O(1) -- here is the math | PUBLISHED |
|
||||
| m1p5 (M1 Complete) | 4 | Signals wrote 100ms ago. The query sees them now. | PUBLISHED |
|
||||
| m2p3 (Ranking Profiles) | 7 | Ranking profiles are data, not code | PUBLISHED |
|
||||
| m2p4 (Diversity) | 6 | Diversity enforcement in 3 microseconds | PUBLISHED |
|
||||
| m2p5 (M2 Complete) | 5 | One query. Six systems. Under 50 milliseconds. | PUBLISHED |
|
||||
| m3p2 (Feedback Loop) | 8, 9 | The feedback loop that closes in one write / Negative signals are equal citizens | PUBLISHED |
|
||||
| m3p3 (Personalized Profiles) | 10 | Cold start without application logic | PUBLISHED |
|
||||
| M4 Complete (Agent Session Layer) | 11 | Search and ranking are the same system | PUBLISHED |
|
||||
| Any time | -- | Why we chose fjall over RocksDB (for now) | READY |
|
||||
| Any time | -- | Why not SQL | READY |
|
||||
| Any time | -- | USearch, not from scratch | READY |
|
||||
| M5p1 (Tantivy Integration) | 12 | Tantivy as a derived index, not a source of truth | Blocked on M5p1 |
|
||||
| M5 Complete | 13, 14 | 14 use cases, one query engine / Cohort-scoped trending | Blocked on M5 |
|
||||
| m6p1 (Crash Recovery) | 15 | Kill it at any point. It comes back correct. | Blocked on M6 |
|
||||
| m6p2 (Graceful Degradation) | 16 | Graceful degradation: less precise, never wrong | Blocked on M6 |
|
||||
|
||||
---
|
||||
|
||||
## Current Queue
|
||||
|
||||
**As of M4 complete (Agent Session Layer), posts 1-11 published.**
|
||||
|
||||
### Ready to write now
|
||||
|
||||
| Post | Status |
|
||||
|------|--------|
|
||||
| "Why we chose fjall over RocksDB (for now)" | READY -- m1p3 shipped, decision is documented |
|
||||
| "Why not SQL" | READY -- any time after M1 |
|
||||
| "USearch, not from scratch" | READY -- m2p1 shipped |
|
||||
|
||||
### Next milestone-gated posts
|
||||
|
||||
| Post | Blocked on |
|
||||
|------|------------|
|
||||
| Post 12: "Tantivy as a derived index, not a source of truth" | M5p1 (Tantivy integration) |
|
||||
| Post 13: "14 use cases, one query engine" | M5 complete |
|
||||
| Post 14: "Cohort-scoped trending" | M5 complete |
|
||||
| Post 15: "Kill it at any point. It comes back correct." | M6p1 |
|
||||
| Post 16: "Graceful degradation: less precise, never wrong" | M6p2 |
|
||||
@ -1,153 +0,0 @@
|
||||
# Capacity Planning
|
||||
|
||||
This document provides RAM, disk, and startup time estimates for tidalDB deployments. Use these tables to provision hardware before going to production.
|
||||
|
||||
All estimates assume a single-node deployment with default configuration (30-second checkpoint interval, f16 vector quantization, DashMap-based hot tier).
|
||||
|
||||
---
|
||||
|
||||
## RAM Capacity
|
||||
|
||||
tidalDB is an in-memory-first database. USearch HNSW indexes, the signal ledger hot tier, and Tantivy reader segments all reside in RAM during operation. There is no swap tolerance for USearch -- if the process is swapped, ANN query latency degrades from microseconds to seconds.
|
||||
|
||||
| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 signals) | Tantivy RAM | Total Estimate |
|
||||
|------:|---------------:|------------:|-------------------------------:|------------:|---------------:|
|
||||
| 100K | 128D | ~26 MB | ~110 MB | ~50 MB | ~200 MB |
|
||||
| 100K | 768D | ~154 MB | ~110 MB | ~50 MB | ~320 MB |
|
||||
| 100K | 1536D | ~307 MB | ~110 MB | ~50 MB | ~470 MB |
|
||||
| 1M | 128D | ~256 MB | ~1.1 GB | ~200 MB | ~1.6 GB |
|
||||
| 1M | 768D | ~1.5 GB | ~1.1 GB | ~200 MB | ~2.8 GB |
|
||||
| 1M | 1536D | ~3.1 GB | ~1.1 GB | ~200 MB | ~4.4 GB |
|
||||
| 10M | 128D | ~2.6 GB | ~11 GB | ~500 MB | ~14 GB |
|
||||
| 10M | 768D | ~15 GB | ~11 GB | ~500 MB | ~27 GB |
|
||||
| 10M | 1536D | ~31 GB | ~11 GB | ~500 MB | ~43 GB |
|
||||
|
||||
### Formulas
|
||||
|
||||
**USearch HNSW index:**
|
||||
|
||||
```
|
||||
items * dims * 2 bytes (f16 quantization) * 1.2 (HNSW graph overhead)
|
||||
```
|
||||
|
||||
The 20% graph overhead accounts for HNSW neighbor lists (M=16 default, two layers). Actual overhead varies with M and ef_construction parameters.
|
||||
|
||||
**Signal ledger hot tier:**
|
||||
|
||||
```
|
||||
items * signal_count * ~1,088 bytes/entry
|
||||
```
|
||||
|
||||
Each `(entity_id, signal_type_id)` entry in the DashMap holds the running decay score, windowed counters (BucketedCounter with minute and hour buckets), velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure was measured in the m7p3 scale benchmarks.
|
||||
|
||||
The signal ledger has a memory budget of 5M entries (`DEFAULT_MAX_SIGNAL_ENTRIES`). When exceeded, the checkpoint thread evicts cold entries (oldest `last_update` timestamp). If your workload has more than 5M active `(entity, signal_type)` pairs, cold entries will be served from fjall checkpoints (slower, but correct).
|
||||
|
||||
**Tantivy text index:**
|
||||
|
||||
Tantivy's RAM usage depends on the number of indexed documents, average document length, and the number of open reader segments. The estimates above assume short metadata fields (title + description, ~200 bytes average). Long-form content indexing will increase RAM proportionally.
|
||||
|
||||
### Notes
|
||||
|
||||
- Signal ledger RAM is for the in-memory hot tier only. The WAL and fjall checkpoints add disk usage, not RAM.
|
||||
- The "10 signals" column assumes 10 distinct signal types per entity. Scale linearly for more signal types.
|
||||
- USearch RAM is the dominant cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), plan for USearch to consume 70%+ of total RAM at 10M items.
|
||||
|
||||
---
|
||||
|
||||
## Disk Capacity
|
||||
|
||||
Disk usage comes from three sources: fjall LSM-tree storage (metadata, relationships, signal checkpoints), WAL segments (append-only signal event log), and Tantivy/USearch index files.
|
||||
|
||||
| Items | Metadata Size | Signal Events/Day | Disk/Day (WAL) | Fjall (90 days) | Total (90 days) |
|
||||
|------:|:----------------|------------------:|----------------:|----------------:|----------------:|
|
||||
| 100K | small (256B avg) | 50K | ~2 MB | ~1 GB | ~1.2 GB |
|
||||
| 1M | small | 500K | ~20 MB | ~10 GB | ~11.8 GB |
|
||||
| 10M | small | 5M | ~200 MB | ~100 GB | ~118 GB |
|
||||
|
||||
### Formulas
|
||||
|
||||
**WAL daily growth:**
|
||||
|
||||
```
|
||||
signal_events_per_day * ~40 bytes/event
|
||||
```
|
||||
|
||||
Each WAL entry contains: 4-byte magic, 8-byte sequence, 1-byte event type, 8-byte entity ID, 2-byte signal type ID, 8-byte timestamp, 8-byte weight (f64), 32-byte BLAKE3 checksum. WAL segments are compacted after each successful checkpoint (every 30 seconds), so WAL disk usage represents only the uncompacted tail, not cumulative growth.
|
||||
|
||||
**Fjall storage:**
|
||||
|
||||
```
|
||||
items * metadata_avg_bytes * 1.5 (LSM write amplification)
|
||||
```
|
||||
|
||||
The 1.5x amplification factor accounts for LSM-tree space amplification (multiple sorted runs before compaction merges them). Actual amplification depends on the compaction strategy and write pattern. Signal checkpoints are also stored in fjall -- add ~100 bytes per active `(entity, signal_type)` pair for the serialized checkpoint data.
|
||||
|
||||
**Tantivy and USearch on disk:**
|
||||
|
||||
- Tantivy: roughly 1.5-2x the raw text size after indexing (inverted index + postings + term dictionary).
|
||||
- USearch: saved index files are approximately the same size as the in-memory representation (items * dims * 2 bytes + graph metadata).
|
||||
|
||||
### WAL Compaction
|
||||
|
||||
WAL segments older than the last successful checkpoint are automatically deleted by the checkpoint thread (every 30 seconds). Under normal operation, WAL disk usage stays bounded at roughly `signal_rate * 40 bytes * 30 seconds`. Monitor `tidaldb_wal_lag_bytes` -- if it grows unbounded, checkpointing may be failing (check `tidaldb_checkpoint_failures_total`).
|
||||
|
||||
---
|
||||
|
||||
## Startup Time
|
||||
|
||||
Startup involves: opening fjall keyspaces, restoring the signal ledger from checkpoint, replaying WAL events since the last checkpoint, rebuilding in-memory indexes (bitmap, range, universe, creator-items, collections, suggestions), and loading USearch vector indexes.
|
||||
|
||||
| Items | Vectors | Typical Startup |
|
||||
|------:|--------:|:----------------|
|
||||
| 100K | 100K | ~2-5 sec |
|
||||
| 1M | 1M | ~15-45 sec |
|
||||
| 10M | 10M | ~3-8 min |
|
||||
|
||||
### Dominant Costs
|
||||
|
||||
1. **USearch index load** is the dominant startup cost at 1M+ vectors. USearch rebuilds the HNSW graph from its serialized format. Progress is logged every 10K vectors.
|
||||
|
||||
2. **Signal ledger restore** reads the checkpoint from fjall (a single prefix scan of `Tag::Sig` keys), then replays any WAL events with sequence numbers higher than the checkpoint's `wal_sequence`. Time is proportional to the number of active signal entries + unreplayed WAL events.
|
||||
|
||||
3. **Entity state rebuild** scans the items and users keyspaces to reconstruct creator-items bitmaps, relationship indexes (follows, blocks, hides), and interaction weights. Progress is logged every 10K items.
|
||||
|
||||
4. **Suggestion index rebuild** scans all item metadata for "title" fields and indexes terms for autocomplete. This is a sequential scan -- fast for 100K items, noticeable at 10M.
|
||||
|
||||
5. **Collection index rebuild** reconstructs collection membership bitmaps from fjall.
|
||||
|
||||
### Notes
|
||||
|
||||
- Startup time is I/O-bound, not CPU-bound. Fast NVMe storage reduces startup time significantly compared to spinning disk.
|
||||
- WAL replay time depends on how many signals were written since the last checkpoint (at most ~30 seconds of writes under normal operation).
|
||||
- Tantivy indexes are opened directly from disk (memory-mapped) and do not require a rebuild step.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Provisioning
|
||||
|
||||
**General rule:** provision 2x the estimated RAM for headroom.
|
||||
|
||||
| Scale | Recommended RAM | Recommended Disk | CPU Cores |
|
||||
|:---------|:----------------|:-----------------|:----------|
|
||||
| 100K items, 128D | 512 MB | 5 GB SSD | 2 |
|
||||
| 100K items, 768D | 1 GB | 5 GB SSD | 2 |
|
||||
| 1M items, 128D | 4 GB | 25 GB SSD | 4 |
|
||||
| 1M items, 768D | 8 GB | 25 GB SSD | 4 |
|
||||
| 10M items, 128D | 32 GB | 250 GB NVMe | 8 |
|
||||
| 10M items, 768D | 64 GB | 250 GB NVMe | 8 |
|
||||
| 10M items, 1536D | 96 GB | 250 GB NVMe | 16 |
|
||||
|
||||
### Why 2x headroom?
|
||||
|
||||
- Signal ledger entries grow as new `(entity, signal_type)` pairs are written. The hot tier can hold up to 5M entries before trimming kicks in.
|
||||
- Tantivy segment merges temporarily double the index size during merge operations.
|
||||
- USearch does not support incremental resize -- if you approach capacity, you need enough free RAM to hold both the old and new index during a potential rebuild.
|
||||
- The Rust allocator (jemalloc or system) has its own fragmentation overhead.
|
||||
|
||||
### Swap
|
||||
|
||||
Do not configure swap for production tidalDB instances. USearch HNSW traversal accesses memory in a random-access pattern that defeats page-level caching. A single swapped page in the HNSW graph can turn a 50-microsecond ANN query into a 50-millisecond disk seek.
|
||||
|
||||
### Disk Type
|
||||
|
||||
SSD is strongly recommended for all deployments. NVMe is recommended at 10M+ items. The WAL uses synchronous `fsync` on every segment rotation, and fjall's journal uses `persist(SyncAll)` during checkpoint. Spinning disk latency on these operations directly impacts signal write throughput.
|
||||
@ -1,523 +0,0 @@
|
||||
{
|
||||
"uid": "tidaldb-overview",
|
||||
"title": "tidalDB Overview",
|
||||
"description": "Operational dashboard covering all 20 tidalDB metrics including retrieve and search latency histograms.",
|
||||
"schemaVersion": 38,
|
||||
"version": 2,
|
||||
"refresh": "30s",
|
||||
"time": { "from": "now-1h", "to": "now" },
|
||||
"timepicker": {},
|
||||
"tags": ["tidaldb"],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "row",
|
||||
"title": "Health Overview",
|
||||
"gridPos": { "x": 0, "y": 0, "w": 24, "h": 1 },
|
||||
"collapsed": false
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Health",
|
||||
"gridPos": { "x": 0, "y": 1, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_health_ok",
|
||||
"legendFormat": "health_ok"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{ "type": "value", "options": { "0": { "text": "UNHEALTHY", "color": "red" } } },
|
||||
{ "type": "value", "options": { "1": { "text": "OK", "color": "green" } } }
|
||||
],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "red", "value": 0 },
|
||||
{ "color": "green", "value": 1 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "background" }
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "stat",
|
||||
"title": "Uptime",
|
||||
"gridPos": { "x": 4, "y": 1, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_uptime_seconds",
|
||||
"legendFormat": "uptime_seconds"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"color": { "mode": "fixed", "fixedColor": "blue" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "value" }
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Degradation Level",
|
||||
"gridPos": { "x": 8, "y": 1, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_degradation_level",
|
||||
"legendFormat": "degradation_level"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "red", "value": 1 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "background" }
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "stat",
|
||||
"title": "Version",
|
||||
"gridPos": { "x": 12, "y": 1, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_info",
|
||||
"legendFormat": "{{version}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "fixed", "fixedColor": "text" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "colorMode": "none", "textMode": "name" }
|
||||
},
|
||||
|
||||
{
|
||||
"id": 10,
|
||||
"type": "row",
|
||||
"title": "Signal Throughput",
|
||||
"gridPos": { "x": 0, "y": 5, "w": 24, "h": 1 },
|
||||
"collapsed": false
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "timeseries",
|
||||
"title": "Signal Write Rate (per second)",
|
||||
"gridPos": { "x": 0, "y": 6, "w": 8, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(tidaldb_signal_writes_total[5m])",
|
||||
"legendFormat": "writes/s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "timeseries",
|
||||
"title": "Signal Write Latency (µs)",
|
||||
"gridPos": { "x": 8, "y": 6, "w": 8, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_signal_write_latency_us",
|
||||
"legendFormat": "latency_us"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "µs",
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "gauge",
|
||||
"title": "Signal Hot Entries",
|
||||
"gridPos": { "x": 16, "y": 6, "w": 8, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_signal_hot_entries",
|
||||
"legendFormat": "hot_entries"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"max": 5000000,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "yellow", "value": 4000000 },
|
||||
{ "color": "red", "value": 5000000 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
{
|
||||
"id": 14,
|
||||
"type": "timeseries",
|
||||
"title": "Retrieve Latency Percentiles (µs)",
|
||||
"gridPos": { "x": 0, "y": 12, "w": 12, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, rate(tidaldb_retrieve_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "µs",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "yellow", "value": 500000 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "timeseries",
|
||||
"title": "Search Latency Percentiles (µs)",
|
||||
"gridPos": { "x": 12, "y": 12, "w": 12, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, rate(tidaldb_search_latency_us_bucket[$__rate_interval]))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "µs",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "yellow", "value": 1000000 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
{
|
||||
"id": 20,
|
||||
"type": "row",
|
||||
"title": "Durability",
|
||||
"gridPos": { "x": 0, "y": 19, "w": 24, "h": 1 },
|
||||
"collapsed": false
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"type": "timeseries",
|
||||
"title": "Checkpoint Age (seconds)",
|
||||
"gridPos": { "x": 0, "y": 20, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_checkpoint_age_seconds",
|
||||
"legendFormat": "checkpoint_age_seconds"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "red", "value": 300 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"type": "stat",
|
||||
"title": "Checkpoint Failures",
|
||||
"gridPos": { "x": 6, "y": 20, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_checkpoint_failures_total",
|
||||
"legendFormat": "checkpoint_failures_total"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "red", "value": 1 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "background" }
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
"type": "timeseries",
|
||||
"title": "WAL Lag (bytes)",
|
||||
"gridPos": { "x": 12, "y": 20, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "tidaldb_wal_lag_bytes",
|
||||
"legendFormat": "wal_lag_bytes"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "yellow", "value": 500000000 },
|
||||
{ "color": "red", "value": 1000000000 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 24,
|
||||
"type": "timeseries",
|
||||
"title": "WAL Compacted Segments (rate)",
|
||||
"gridPos": { "x": 18, "y": 20, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(tidaldb_wal_compacted_segments_total[5m])",
|
||||
"legendFormat": "compacted/s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "cps",
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
{
|
||||
"id": 30,
|
||||
"type": "row",
|
||||
"title": "Index Health",
|
||||
"gridPos": { "x": 0, "y": 26, "w": 24, "h": 1 },
|
||||
"collapsed": false
|
||||
},
|
||||
{
|
||||
"id": 31,
|
||||
"type": "stat",
|
||||
"title": "Tantivy Indexed Docs",
|
||||
"gridPos": { "x": 0, "y": 27, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_tantivy_indexed_docs", "legendFormat": "indexed_docs" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": { "mode": "fixed", "fixedColor": "blue" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
|
||||
},
|
||||
{
|
||||
"id": 32,
|
||||
"type": "gauge",
|
||||
"title": "Tantivy Segment Count",
|
||||
"gridPos": { "x": 4, "y": 27, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_tantivy_segment_count", "legendFormat": "segment_count" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"max": 50,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "yellow", "value": 20 },
|
||||
{ "color": "red", "value": 30 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 33,
|
||||
"type": "stat",
|
||||
"title": "uSearch Vector Count",
|
||||
"gridPos": { "x": 8, "y": 27, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_usearch_vector_count", "legendFormat": "vector_count" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": { "mode": "fixed", "fixedColor": "blue" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
|
||||
},
|
||||
{
|
||||
"id": 34,
|
||||
"type": "stat",
|
||||
"title": "uSearch Index Size",
|
||||
"gridPos": { "x": 12, "y": 27, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_usearch_index_size_bytes", "legendFormat": "index_size_bytes" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"color": { "mode": "fixed", "fixedColor": "blue" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
|
||||
},
|
||||
{
|
||||
"id": 35,
|
||||
"type": "stat",
|
||||
"title": "Bitmap Index Cardinality",
|
||||
"gridPos": { "x": 16, "y": 27, "w": 4, "h": 4 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_bitmap_index_cardinality", "legendFormat": "bitmap_cardinality" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": { "mode": "fixed", "fixedColor": "blue" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
|
||||
},
|
||||
|
||||
{
|
||||
"id": 40,
|
||||
"type": "row",
|
||||
"title": "Sessions",
|
||||
"gridPos": { "x": 0, "y": 31, "w": 24, "h": 1 },
|
||||
"collapsed": false
|
||||
},
|
||||
{
|
||||
"id": 41,
|
||||
"type": "timeseries",
|
||||
"title": "Active Sessions",
|
||||
"gridPos": { "x": 0, "y": 32, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_active_sessions", "legendFormat": "active_sessions" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 42,
|
||||
"type": "timeseries",
|
||||
"title": "Session Close Rate (per second)",
|
||||
"gridPos": { "x": 6, "y": 32, "w": 6, "h": 6 },
|
||||
"targets": [
|
||||
{ "expr": "rate(tidaldb_closed_sessions_total[5m])", "legendFormat": "closes/s" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 43,
|
||||
"type": "stat",
|
||||
"title": "Auto-Closed Sessions",
|
||||
"gridPos": { "x": 12, "y": 32, "w": 4, "h": 6 },
|
||||
"targets": [
|
||||
{ "expr": "tidaldb_session_auto_closed_total", "legendFormat": "auto_closed_total" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": { "mode": "fixed", "fixedColor": "yellow" }
|
||||
}
|
||||
},
|
||||
"options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "value" }
|
||||
},
|
||||
{
|
||||
"id": 44,
|
||||
"type": "timeseries",
|
||||
"title": "Rate Limited (per second)",
|
||||
"gridPos": { "x": 16, "y": 32, "w": 8, "h": 6 },
|
||||
"targets": [
|
||||
{ "expr": "rate(tidaldb_rate_limited_total[5m])", "legendFormat": "rate_limited/s" }
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": 0 },
|
||||
{ "color": "red", "value": 100 }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "palette-classic" }
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -1,174 +0,0 @@
|
||||
# Monitoring
|
||||
|
||||
This document covers tidalDB's built-in Prometheus metrics endpoint, all exposed metrics, and recommended alerting thresholds.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
Enable the metrics HTTP server via the builder:
|
||||
|
||||
```rust
|
||||
let db = TidalDb::builder()
|
||||
.with_data_dir("/var/lib/tidaldb")
|
||||
.with_schema(schema)
|
||||
.enable_metrics("127.0.0.1:9090")
|
||||
.open()?;
|
||||
|
||||
// Discover the bound address (useful when using port 0):
|
||||
if let Some(addr) = db.metrics_addr() {
|
||||
println!("metrics at http://{}/metrics", addr);
|
||||
println!("health at http://{}/healthz", addr);
|
||||
}
|
||||
```
|
||||
|
||||
**Security:** The metrics endpoint has no authentication. Bind to `127.0.0.1` (loopback) only. If you need to scrape from a remote Prometheus server, use your infrastructure's network controls (SSH tunnel, reverse proxy with auth, or VPN) rather than binding to `0.0.0.0`. tidalDB logs a WARN-level message if you bind to a non-loopback address.
|
||||
|
||||
**Feature flag:** The metrics HTTP server requires the `metrics` feature, which is enabled by default. Build with `--no-default-features` to disable the HTTP server entirely. Base metrics (`uptime_seconds`, `health_ok`, `info`, `checkpoint_failures_total`) are always compiled regardless of the feature flag.
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Path | Content-Type | Description |
|
||||
|:-----|:-------------|:------------|
|
||||
| `/metrics` | `text/plain` | Prometheus text exposition format |
|
||||
| `/healthz` | `application/json` | JSON health check: `{"status":"ok","uptime_seconds":123.456,"version":"0.1.0","build_hash":"..."}` |
|
||||
|
||||
---
|
||||
|
||||
## Prometheus Scrape Configuration
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: 'tidaldb'
|
||||
static_configs:
|
||||
- targets: ['127.0.0.1:9090']
|
||||
scrape_interval: 15s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics Reference
|
||||
|
||||
All metrics use the `tidaldb_` prefix. Metrics marked with "(feature-gated)" are only emitted when the `metrics` Cargo feature is enabled (default: enabled).
|
||||
|
||||
### Build and Health
|
||||
|
||||
| Metric | Type | Description | Labels |
|
||||
|:-------|:-----|:------------|:-------|
|
||||
| `tidaldb_uptime_seconds` | gauge | Seconds since the database was opened. Monotonically increasing. | `partition_id="0"` |
|
||||
| `tidaldb_health_ok` | gauge | Whether the database is healthy. `1` = ok, `0` = degraded or closed. | `partition_id="0"` |
|
||||
| `tidaldb_info` | gauge | Build and version information. Always `1`. | `version`, `build_hash`, `partition_id="0"` |
|
||||
|
||||
**Normal range for `tidaldb_health_ok`:** Always `1` during normal operation. Drops to `0` during shutdown or if an internal health check fails. Alert immediately if `0` during expected uptime.
|
||||
|
||||
### Signal System (feature-gated)
|
||||
|
||||
| Metric | Type | Unit | Description |
|
||||
|:-------|:-----|:-----|:------------|
|
||||
| `tidaldb_signal_writes_total` | counter | count | Total signal writes since database open. Includes all signal types across all entities. |
|
||||
| `tidaldb_signal_hot_entries` | gauge | count | Number of entries currently in the signal ledger hot tier (DashMap). Each entry is one `(entity_id, signal_type_id)` pair. |
|
||||
| `tidaldb_signal_write_latency_us` | histogram | microseconds | Signal write latency distribution. Bucket boundaries: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 10000 microseconds. |
|
||||
|
||||
**Normal range for `signal_hot_entries`:** Proportional to `active_entities * signal_types_per_entity`. The hot tier is trimmed at 5M entries (`DEFAULT_MAX_SIGNAL_ENTRIES`). Alert if approaching 80% of budget (4M entries).
|
||||
|
||||
**Normal range for `signal_write_latency_us`:** p50 should be < 50us, p99 should be < 1ms. If p99 exceeds 5ms, investigate WAL write latency or DashMap contention.
|
||||
|
||||
### WAL and Checkpoint (feature-gated)
|
||||
|
||||
| Metric | Type | Unit | Description |
|
||||
|:-------|:-----|:-----|:------------|
|
||||
| `tidaldb_wal_lag_bytes` | gauge | bytes | Total bytes of WAL segment files not yet compacted. Updated after each checkpoint cycle. |
|
||||
| `tidaldb_wal_compacted_segments_total` | counter | count | Total WAL segments deleted by compaction since database open. |
|
||||
| `tidaldb_checkpoint_age_seconds` | gauge | seconds | Seconds since the last successful signal checkpoint. Derived from `last_checkpoint_ns` at render time. |
|
||||
| `tidaldb_checkpoint_failures_total` | counter | count | Total number of failed periodic signal checkpoints. **Not feature-gated** -- always emitted. |
|
||||
|
||||
**Normal range for `checkpoint_age_seconds`:** Should stay below 60 seconds (checkpoint runs every 30 seconds, with some jitter from the 500ms poll interval). Alert if > 300 seconds (5 minutes) -- the checkpoint thread may be stuck or the storage engine is under pressure.
|
||||
|
||||
**Normal range for `wal_lag_bytes`:** Depends on signal write rate. At 1K signals/sec, expect ~1.2 MB of WAL per 30-second checkpoint cycle. Alert if > 1 GB -- compaction may be failing.
|
||||
|
||||
**Normal range for `checkpoint_failures_total`:** Should be 0. Any non-zero value means signal durability is at risk -- the hot tier is not being persisted. Investigate storage errors (disk full, I/O errors).
|
||||
|
||||
### Index Health (feature-gated)
|
||||
|
||||
| Metric | Type | Unit | Description |
|
||||
|:-------|:-----|:-----|:------------|
|
||||
| `tidaldb_tantivy_segment_count` | gauge | count | Number of Tantivy index segments for the items text index. |
|
||||
| `tidaldb_tantivy_indexed_docs` | gauge | count | Number of documents indexed in the items Tantivy text index. |
|
||||
| `tidaldb_usearch_index_size_bytes` | gauge | bytes | Estimated total byte size of all USearch vector index files (f16). |
|
||||
| `tidaldb_usearch_vector_count` | gauge | count | Number of vectors stored across all USearch indexes. |
|
||||
| `tidaldb_bitmap_index_cardinality` | gauge | count | Total entity IDs across all four bitmap indexes (category + format + creator + tag). |
|
||||
|
||||
Index health metrics are refreshed every 10 seconds by the checkpoint thread (3x more frequently than checkpoints) so operators get near-real-time visibility.
|
||||
|
||||
**Normal range for `tantivy_segment_count`:** Should stay below 20 during normal operation. Tantivy merges segments in the background. If segment count grows unbounded, the text syncer thread may have stalled.
|
||||
|
||||
**Normal range for `usearch_vector_count`:** Should match the number of entities with embeddings written via `write_item_embedding()` or `write_creator_embedding()`.
|
||||
|
||||
### Session Lifecycle (feature-gated)
|
||||
|
||||
| Metric | Type | Unit | Description |
|
||||
|:-------|:-----|:-----|:------------|
|
||||
| `tidaldb_active_sessions` | gauge | count | Number of currently active agent sessions. |
|
||||
| `tidaldb_closed_sessions_total` | counter | count | Total agent sessions closed (explicitly or by sweeper) since database open. |
|
||||
| `tidaldb_session_auto_closed_total` | counter | count | Total sessions auto-closed by the TTL sweeper due to exceeding `max_session_duration`. |
|
||||
|
||||
**Normal range for `active_sessions`:** Depends on your application's agent concurrency. Each open session consumes memory for signal state tracking. Alert if this grows unbounded -- agents may be leaking sessions (opening without closing).
|
||||
|
||||
### Rate Limiting and Degradation (feature-gated)
|
||||
|
||||
| Metric | Type | Unit | Description |
|
||||
|:-------|:-----|:-----|:------------|
|
||||
| `tidaldb_rate_limited_total` | counter | count | Total signal write requests rejected due to per-agent rate limits since database open. |
|
||||
| `tidaldb_degradation_level` | gauge | level | Current graceful degradation level. `0` = full quality, `1` = reduced candidates, `2` = coarse aggregates, `3` = no diversity enforcement. |
|
||||
|
||||
**Normal range for `degradation_level`:** Should be `0` during normal operation. Any value > 0 means the load detector has triggered degradation to protect latency. Investigate system load (CPU, memory pressure, I/O saturation).
|
||||
|
||||
---
|
||||
|
||||
## Recommended Alerts
|
||||
|
||||
| Alert Name | Condition | Severity | Meaning |
|
||||
|:-----------|:----------|:---------|:--------|
|
||||
| TidalDB Down | `tidaldb_health_ok == 0` | Critical | Database is unhealthy or shut down. Immediate investigation required. |
|
||||
| Checkpoint Stale | `tidaldb_checkpoint_age_seconds > 300` | Warning | Checkpoint has not run in 5+ minutes. Signal durability at risk. Check storage I/O and disk space. |
|
||||
| Checkpoint Failures | `tidaldb_checkpoint_failures_total > 0` | Warning | At least one checkpoint has failed. Signal state may not be durable. Check disk space and storage errors. |
|
||||
| WAL Disk Pressure | `tidaldb_wal_lag_bytes > 1000000000` | Warning | WAL exceeds 1 GB uncompacted. Compaction may be stuck or checkpoint is failing. |
|
||||
| Signal Backlog | `tidaldb_signal_hot_entries > 4000000` | Warning | Signal ledger over 80% of the 5M entry budget. Cold entry trimming will begin at 5M. |
|
||||
| Degraded Ranking | `tidaldb_degradation_level > 0` | Warning | Load-based degradation is active. Ranking quality is reduced to protect latency. Scale up or reduce load. |
|
||||
| Session Leak | `deriv(tidaldb_active_sessions[5m]) > 0.03 AND tidaldb_active_sessions > 100` | Warning | Active session count trending up. `tidaldb_active_sessions` is a gauge, so `deriv()` (slope) is correct — `rate()`/`increase()` are invalid on gauges and never fire. Agents may not be closing sessions. |
|
||||
| High Rate Limiting | `rate(tidaldb_rate_limited_total[5m]) > 100` | Info | Sustained rate limiting. Review agent rate limit configuration or reduce write volume. |
|
||||
| Tantivy Segment Bloat | `tidaldb_tantivy_segment_count > 30` | Warning | Tantivy has many unmerged segments. Text syncer may be stalled. |
|
||||
|
||||
### Grafana Dashboard Suggestions
|
||||
|
||||
**Row 1: Health overview**
|
||||
- `tidaldb_health_ok` (stat panel, green/red)
|
||||
- `tidaldb_uptime_seconds` (stat panel)
|
||||
- `tidaldb_degradation_level` (stat panel, thresholds at 1/2/3)
|
||||
- `tidaldb_info` labels (stat panel showing version + build hash)
|
||||
|
||||
**Row 2: Signal throughput**
|
||||
- `rate(tidaldb_signal_writes_total[5m])` (time series, signals/sec)
|
||||
- `tidaldb_signal_write_latency_us` histogram (heatmap or quantile panel)
|
||||
- `tidaldb_signal_hot_entries` (gauge, threshold at 4M/5M)
|
||||
|
||||
**Row 3: Durability**
|
||||
- `tidaldb_checkpoint_age_seconds` (time series, threshold line at 300)
|
||||
- `tidaldb_checkpoint_failures_total` (stat panel, should be 0)
|
||||
- `tidaldb_wal_lag_bytes` (time series)
|
||||
- `rate(tidaldb_wal_compacted_segments_total[5m])` (time series)
|
||||
|
||||
**Row 4: Index health**
|
||||
- `tidaldb_tantivy_indexed_docs` (stat panel)
|
||||
- `tidaldb_tantivy_segment_count` (gauge)
|
||||
- `tidaldb_usearch_vector_count` (stat panel)
|
||||
- `tidaldb_usearch_index_size_bytes` (stat panel, bytes format)
|
||||
- `tidaldb_bitmap_index_cardinality` (stat panel)
|
||||
|
||||
**Row 5: Sessions**
|
||||
- `tidaldb_active_sessions` (time series)
|
||||
- `rate(tidaldb_closed_sessions_total[5m])` (time series)
|
||||
- `tidaldb_session_auto_closed_total` (stat panel)
|
||||
- `rate(tidaldb_rate_limited_total[5m])` (time series)
|
||||
@ -1,131 +0,0 @@
|
||||
# =============================================================================
|
||||
# DESIGN-REFERENCE RULE SET — NOT LOADED BY ANY ALERTMANAGER TODAY.
|
||||
# =============================================================================
|
||||
# This file is reference/design config. It is NOT wired into any running
|
||||
# alerting pipeline:
|
||||
# - vmalert (pilot/dev) loads ONLY ops/vmalert/rules/*.yaml
|
||||
# - prod CRDs live ONLY in infra/k8s/prom/prometheus-rules/*.yaml
|
||||
# Nothing mounts or imports tidal/docs/ops/prometheus-alerts.yaml.
|
||||
# Do not read these as live pager rules.
|
||||
#
|
||||
# Provenance: every metric referenced below (tidaldb_health_ok,
|
||||
# tidaldb_checkpoint_age_seconds, tidaldb_checkpoint_failures_total,
|
||||
# tidaldb_wal_lag_bytes, tidaldb_signal_hot_entries, tidaldb_degradation_level,
|
||||
# tidaldb_active_sessions, tidaldb_rate_limited_total,
|
||||
# tidaldb_tantivy_segment_count, tidaldb_retrieve_latency_us_bucket,
|
||||
# tidaldb_search_latency_us_bucket) IS really emitted today by
|
||||
# tidal/src/db/metrics/mod.rs. So these rules are promotable — they
|
||||
# are not orphaned against phantom metrics.
|
||||
#
|
||||
# Promotion path (the right long-term fix — do it deliberately, do NOT
|
||||
# hand-copy these as live pager rules without the steps below):
|
||||
# 1. Add a vmalert rule file: ops/vmalert/rules/tidaldb.yaml, translating
|
||||
# each rule and stamping the canonical labels every routed rule carries —
|
||||
# severity + capability + surface (+ optional component). Map per the
|
||||
# canonical severity taxonomy in ops/alerting.md:
|
||||
# labels {severity: critical} = pager class (TidalDBDown only here),
|
||||
# {severity: warning} = non-paging / business-hours,
|
||||
# {severity: info} = pure-FYI.
|
||||
# Add capability: tidaldb and a surface label per rule, and a
|
||||
# runbook_url annotation:
|
||||
# https://github.com/orchard9/tidaldb/blob/main/tidal/docs/runbooks/<area>/<slug>.md
|
||||
# 2. Add the prod CRD twin: infra/k8s/prom/prometheus-rules/tidaldb.yaml,
|
||||
# kept byte-aligned in expr/threshold with the vmalert file.
|
||||
# 3. Verify the alerts fire against real metrics (scrape tidaldb, force a
|
||||
# degraded state) before declaring them on-call-ready.
|
||||
# Until steps 1–3 land, this file stays a design reference only.
|
||||
# =============================================================================
|
||||
groups:
|
||||
- name: tidaldb
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: TidalDBDown
|
||||
expr: tidaldb_health_ok == 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "tidalDB is unhealthy"
|
||||
description: "tidaldb_health_ok is 0 — database is unhealthy or shut down."
|
||||
|
||||
- alert: TidalDBCheckpointStale
|
||||
expr: tidaldb_checkpoint_age_seconds > 300
|
||||
for: 2m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Signal checkpoint not running"
|
||||
description: "{{ $value }}s since last checkpoint (threshold: 300s). Signal durability at risk."
|
||||
|
||||
- alert: TidalDBCheckpointFailures
|
||||
expr: increase(tidaldb_checkpoint_failures_total[5m]) > 0
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Signal checkpoint failures detected"
|
||||
description: "Checkpoint failures in last 5m. Check disk space and storage errors."
|
||||
|
||||
- alert: TidalDBWALDiskPressure
|
||||
expr: tidaldb_wal_lag_bytes > 1000000000
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "WAL disk usage exceeds 1GB"
|
||||
description: "{{ $value | humanize1024 }}B of WAL uncompacted. Compaction may be stuck."
|
||||
|
||||
- alert: TidalDBSignalBacklog
|
||||
expr: tidaldb_signal_hot_entries > 4000000
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Signal ledger over 80% of capacity"
|
||||
description: "{{ $value }} hot entries (threshold: 4M / 80% of 5M budget)."
|
||||
|
||||
- alert: TidalDBDegradedRanking
|
||||
expr: tidaldb_degradation_level > 0
|
||||
for: 2m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Ranking quality degraded"
|
||||
description: "Degradation level {{ $value }} active. Scale up or reduce load."
|
||||
|
||||
- alert: TidalDBSessionLeak
|
||||
# tidaldb_active_sessions is a GAUGE, so rate()/increase() are invalid
|
||||
# (those operators only count counter resets). deriv() fits a least-squares
|
||||
# slope over the window and is the gauge-correct way to detect sustained
|
||||
# growth: > 0.03 sessions/sec is ~> 9 new sessions over a 5m window.
|
||||
expr: deriv(tidaldb_active_sessions[5m]) > 0.03 and tidaldb_active_sessions > 100
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Active session count growing steadily"
|
||||
description: "{{ $value }} active sessions and trending up (deriv > 0.03/s). Agents may not be closing sessions."
|
||||
|
||||
- alert: TidalDBHighRateLimiting
|
||||
expr: rate(tidaldb_rate_limited_total[5m]) > 100
|
||||
for: 5m
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "Sustained rate limiting"
|
||||
description: "{{ $value }}/s rate-limited writes. Review agent rate limit config."
|
||||
|
||||
- alert: TidalDBTantivySegmentBloat
|
||||
expr: tidaldb_tantivy_segment_count > 30
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Tantivy segment count elevated"
|
||||
description: "{{ $value }} segments (threshold: 30). Text syncer may be stalled."
|
||||
|
||||
- alert: TidalDBSlowRetrieve
|
||||
expr: histogram_quantile(0.95, rate(tidaldb_retrieve_latency_us_bucket[5m])) > 500000
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Retrieve p95 latency exceeds 500ms"
|
||||
description: "p95 retrieve latency is {{ $value | humanizeDuration }}. Check signal ledger load and degradation level."
|
||||
|
||||
- alert: TidalDBSlowSearch
|
||||
expr: histogram_quantile(0.95, rate(tidaldb_search_latency_us_bucket[5m])) > 1000000
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Search p95 latency exceeds 1s"
|
||||
description: "p95 search latency is {{ $value | humanizeDuration }}. Check Tantivy segment count and ANN index health."
|
||||
@ -1,189 +0,0 @@
|
||||
# Recovery Guide
|
||||
|
||||
This document covers error scenarios, their causes, data at risk, and step-by-step recovery procedures for tidalDB.
|
||||
|
||||
---
|
||||
|
||||
## Error Scenarios
|
||||
|
||||
### 1. `StorageError::Corruption` on open
|
||||
|
||||
**Error message:** `storage error: data corruption: <details>`
|
||||
|
||||
**Cause:** fjall data files are corrupted. This can happen due to bit rot, partial writes from a hard power loss (no UPS), or physical disk failure. fjall uses checksums on its SST (sorted string table) files, so corruption is detected rather than silently returning wrong data.
|
||||
|
||||
**Data at risk:** All item metadata and relationships stored in the corrupt keyspace since the last backup.
|
||||
|
||||
**Recovery steps:**
|
||||
|
||||
1. **Identify which engine is corrupt.** tidalDB uses three fjall keyspaces: `items/`, `users/`, `creators/`. The error message will typically include the path or keyspace name. Check the log output for the specific directory.
|
||||
|
||||
2. **If you have a backup:**
|
||||
- Stop the process.
|
||||
- Replace the corrupt engine directory (e.g., `{data_dir}/items/`) with the corresponding directory from the backup.
|
||||
- Reopen tidalDB. The WAL replay (`recover()`) will restore signal history from the backup's checkpoint timestamp forward. Any signals between the backup and the corruption event are preserved in WAL segments.
|
||||
- Verify data integrity by running queries against known entities.
|
||||
|
||||
3. **If you have no backup:**
|
||||
- Stop the process.
|
||||
- Delete the corrupt engine directory (e.g., `rm -rf {data_dir}/items/`).
|
||||
- Reopen tidalDB. It will start fresh for that engine -- all metadata and relationships in that keyspace are lost.
|
||||
- Signal state in the WAL is independent of fjall and is recoverable. WAL replay will restore signal scores.
|
||||
- You will need to re-ingest item metadata and embeddings from your upstream data source.
|
||||
|
||||
4. **If corruption is in the `items/` keyspace specifically:**
|
||||
- Signal checkpoints (`Tag::Sig`) are stored in the items keyspace. Losing this keyspace means the signal ledger falls back to full WAL replay (slower startup but no data loss for signals).
|
||||
- Collection definitions (`Tag::Collection`), cohort definitions (`Tag::CohortDef`), and co-engagement data are also in items. These will be lost.
|
||||
|
||||
### 2. `WalError::Corruption` on open
|
||||
|
||||
**Error message:** `WAL corruption: <details>` (surfaced as `TidalError::Durability`)
|
||||
|
||||
**Cause:** A WAL segment was partially written when the process crashed (e.g., SIGKILL during a signal write). The BLAKE3 checksum on the partial entry does not match, so the WAL reader detects corruption.
|
||||
|
||||
**Data at risk:** Signals written after the last successful WAL entry in the corrupt segment. In practice, this is at most one signal event (the one that was mid-write when the crash occurred).
|
||||
|
||||
**Automatic recovery:** tidalDB's crash recovery (`recover()`) automatically truncates the corrupt tail of the WAL and continues. The truncated entry is logged at WARN level. No manual action is needed unless `open()` continues to fail after the automatic recovery attempt.
|
||||
|
||||
**Manual recovery (if automatic recovery fails):**
|
||||
|
||||
1. List WAL segment files: `ls -la {data_dir}/wal/`
|
||||
2. Identify the segment with the highest sequence number in its filename (e.g., `segment_000042.wal`).
|
||||
3. Delete that single file: `rm {data_dir}/wal/segment_000042.wal`
|
||||
4. Reopen tidalDB. It will replay from the remaining segments up to the last complete entry.
|
||||
5. The signals in the deleted segment that were written after the last checkpoint are lost. Signals before the checkpoint are already materialized in fjall and are safe.
|
||||
|
||||
**Prevention:** tidalDB uses `fsync` on WAL segment rotation and BLAKE3 checksums on every entry. The only scenario where corruption occurs is a hard crash (SIGKILL, power loss) during the write of a single entry. Clean shutdowns (`db.close()`) always leave the WAL in a consistent state.
|
||||
|
||||
### 3. `TidalError::Config(DataDirLocked)` on open
|
||||
|
||||
**Error message:** `config error: data directory is already open by another process: <path>`
|
||||
|
||||
**Cause:** Another process has acquired the advisory lock on `{data_dir}/tidaldb.lock`. tidalDB uses file locking to prevent two processes from opening the same data directory simultaneously, which would cause data corruption.
|
||||
|
||||
**Data at risk:** None. This error is protective -- no data is accessed or modified.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
1. Find the other process: `ps aux | grep <your_binary_name>` or `lsof {data_dir}/tidaldb.lock`
|
||||
2. If the other process is a legitimate tidalDB instance, stop it cleanly (send SIGTERM and wait for graceful shutdown).
|
||||
3. If the other process crashed and left a stale lock file:
|
||||
- Verify no process is actually using the directory: `lsof +D {data_dir}`
|
||||
- Delete the lock file: `rm {data_dir}/tidaldb.lock`
|
||||
- Reopen tidalDB.
|
||||
|
||||
**Note:** The lock file (`tidaldb.lock`) is an advisory lock. Deleting it while another process is running will NOT prevent corruption -- the lock only works if both processes respect it. Always verify no process is running before deleting.
|
||||
|
||||
### 4. `TidalError::Schema(UnknownSignalType)` or schema fingerprint mismatch on open
|
||||
|
||||
**Error message:** `unknown signal type: '<name>'` or schema fingerprint mismatch
|
||||
|
||||
**Cause:** The application's schema definition has changed since the database was created. Signal decay parameters, signal names, or embedding slot dimensions differ from what was used when the data was written.
|
||||
|
||||
**Data at risk:** None. This is a protective error. No data is modified.
|
||||
|
||||
**Recovery options:**
|
||||
|
||||
1. **Revert the schema** to match the one used when the database was created. This is the safest option if the schema change was unintentional.
|
||||
|
||||
2. **Add the new signal type alongside the old one.** If you are adding a new signal (e.g., `"share"`), keep all existing signal definitions unchanged and add the new one. Existing signal data is unaffected. Note: this changes the schema fingerprint, so you may need to use a migration path when fingerprint validation is enforced.
|
||||
|
||||
3. **Start fresh.** Delete `{data_dir}` entirely and reopen with the new schema. All existing data is lost. Re-ingest from your upstream data source.
|
||||
|
||||
**What you must NOT do:** Change decay parameters (half_life) on an existing signal type and force the database open. The existing decay scores were computed with the old half_life. Applying a different decay rate to historical scores produces mathematically incorrect results. If you need a different decay rate, define a new signal type with the new parameters and let the old signal data age out naturally.
|
||||
|
||||
### 5. Disk full during operation
|
||||
|
||||
**Symptoms:** Signal writes return `TidalError::Durability(...)`. Metadata writes return `TidalError::Storage(StorageError::Io(...))`. Checkpoint thread logs errors. `tidaldb_checkpoint_failures_total` increments.
|
||||
|
||||
**Data at risk:** Signals written after the last successful checkpoint. In-memory state remains correct and readable -- queries continue to work against the hot tier.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
1. Free disk space. Priorities:
|
||||
- Delete old WAL segments that predate the last checkpoint. Check `{data_dir}/wal/` for segments with low sequence numbers. The checkpoint thread compacts these automatically, but if checkpointing itself failed due to disk pressure, compaction may be stuck.
|
||||
- If you have a recent backup, you can safely delete all WAL segments and let the database start from the fjall checkpoint on next open.
|
||||
- Clear temporary files, logs, or other non-tidalDB data from the volume.
|
||||
|
||||
2. Once disk space is available, signal writes resume automatically. The WAL writer thread retries on the next signal event. No restart is needed.
|
||||
|
||||
3. Verify recovery: check that `tidaldb_checkpoint_failures_total` stops incrementing and `tidaldb_checkpoint_age_seconds` returns to < 60 seconds.
|
||||
|
||||
**Prevention:** Monitor `tidaldb_wal_lag_bytes` and set alerts at 80% of your disk capacity. The WAL is the fastest-growing component. At 5M signals/day, the WAL grows ~200 MB/day before compaction.
|
||||
|
||||
### 6. `TidalError::Backpressure` during signal writes
|
||||
|
||||
**Error message:** `backpressure: WAL queue full, retry after <N>ms`
|
||||
|
||||
**Cause:** The WAL writer thread's channel is full. This means signal writes are arriving faster than the WAL can persist them to disk. This is NOT a data loss event -- the signal was never enqueued, so it can be safely retried.
|
||||
|
||||
**Recovery:** Retry the signal write after the suggested delay (`retry_after_ms`). If backpressure is sustained:
|
||||
- Check disk I/O latency (WAL writes are `fsync`-bound).
|
||||
- Check if another process is competing for disk bandwidth.
|
||||
- Consider faster storage (NVMe).
|
||||
|
||||
### 7. `TidalError::RateLimited` during session signal writes
|
||||
|
||||
**Error message:** `rate limited: agent '<id>' at <limit> signals/sec, retry after <N>ms`
|
||||
|
||||
**Cause:** An agent has exceeded its configured rate limit for the current session. The signal was NOT written.
|
||||
|
||||
**Recovery:** Back off and retry after `retry_after_ms`. If rate limiting is too aggressive, adjust the agent's rate limit in the schema's `AgentPolicy` configuration.
|
||||
|
||||
---
|
||||
|
||||
## Safe Files to Delete
|
||||
|
||||
| File/Directory | Safe to delete? | Notes |
|
||||
|:---------------|:----------------|:------|
|
||||
| `tidaldb.lock` | Yes, if no process is running | Advisory lock file. Auto-recreated on next open. Verify with `lsof` first. |
|
||||
| `wal/segment_*.wal` | Only segments with sequence numbers below the last checkpoint | Never delete the segment with the highest sequence number. To find the checkpoint sequence, check the last `tidaldb-checkpoint` log entry. |
|
||||
| `items/` | **NO**, not without a backup | Primary fjall keyspace. Contains item metadata, signal checkpoints, collections, cohort definitions, co-engagement data. |
|
||||
| `users/` | **NO**, not without a backup | Contains user relationship edges (follows, blocks, hides, interaction weights). |
|
||||
| `creators/` | **NO**, not without a backup | Contains creator metadata and embeddings. |
|
||||
| `text_index/` | Yes | Tantivy item text index. Rebuilt automatically from item metadata on next open. Rebuild cost is proportional to item count. |
|
||||
| `creator_text_index/` | Yes | Tantivy creator text index. Same as above but for creators. |
|
||||
| `cache/` | Yes | Temporary cache directory. Safe to delete at any time. |
|
||||
|
||||
---
|
||||
|
||||
## Backup and Restore
|
||||
|
||||
tidalDB's underlying storage engine (fjall 3.x) does not yet expose a native backup API ([fjall issue #52](https://github.com/fjall-rs/fjall/issues/52)). Until that ships, the recommended backup procedure is quiesce-and-copy.
|
||||
|
||||
### Creating a Backup
|
||||
|
||||
1. **Stop writes** to the database (either shut down the process or use an application-level write pause).
|
||||
2. **Flush all state:**
|
||||
- Call `db.close()` for a clean shutdown, which checkpoints the signal ledger, flushes fjall, and writes a WAL checkpoint marker. OR:
|
||||
- If keeping the process running: flush text indexes, then wait for the next checkpoint cycle (30 seconds).
|
||||
3. **Copy the entire data directory** recursively:
|
||||
```
|
||||
cp -R {data_dir} {backup_dest}
|
||||
```
|
||||
This copies: `items/`, `users/`, `creators/`, `wal/`, `text_index/`, `creator_text_index/`, and `tidaldb.lock`.
|
||||
4. **Resume writes** (restart the process or unpause).
|
||||
5. **Do NOT copy while the database is actively writing** without a quiesce step. Partial fjall SST files or WAL segments will produce corruption on restore.
|
||||
|
||||
### Restoring from Backup
|
||||
|
||||
1. Stop the current tidalDB process if running.
|
||||
2. Delete or move the current `{data_dir}`.
|
||||
3. Copy the backup into place: `cp -R {backup_source} {data_dir}`
|
||||
4. Remove the stale lock file: `rm {data_dir}/tidaldb.lock`
|
||||
5. Open tidalDB with the same schema. WAL replay will recover any signal events written between the backup's checkpoint and the end of the backup's WAL.
|
||||
|
||||
### What Survives a Crash Without Backup
|
||||
|
||||
| Component | Survives unclean shutdown? | Recovery mechanism |
|
||||
|:----------|:--------------------------|:-------------------|
|
||||
| Item metadata | Yes | Stored in fjall (durable) |
|
||||
| Relationships | Yes | Stored in fjall (durable) |
|
||||
| Signal decay scores | Yes (up to last checkpoint + WAL tail) | Checkpoint + WAL replay |
|
||||
| Windowed counts | Approximately | Checkpoint stores state; WAL replay re-applies events |
|
||||
| Active sessions | Yes | Session open/signal events in WAL are replayed; sessions restored as active |
|
||||
| Bitmap/range indexes | Rebuilt on open | Scanned from fjall metadata |
|
||||
| USearch vectors | Yes | Loaded from saved `.idx` files |
|
||||
| Tantivy text index | Yes | Opened from on-disk segments |
|
||||
| Collections | Yes | Rebuilt from fjall on open |
|
||||
| Suggestions | Rebuilt on open | Scanned from fjall item metadata |
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user