192c473f55
- M5p1: BM25 text indexing via Tantivy with background syncer (0.26ms @ 10K docs) - M5p2: RRF fusion layer combining BM25 + ANN scores (46µs @ 1K candidates) - M5p3: unified Search query API (8-stage pipeline, BM25 + vector + ranking) - M5p4: creator text + vector indexing and creator search executor (< 20ms @ 200 creators) - Refactor db/mod.rs into focused sub-modules (creators, items, sessions, signals, etc.) - Decompose monolithic files into directory modules (query/executor, ranking/diversity, etc.) - Split brute.rs → brute/mod.rs + brute/tests.rs; extract search executor helpers - Add benches: fusion, search, session, text_index - Add M5 UAT test suites (m5_uat, m5_search, m5p4_creator_search, text_index) - Update blog posts, roadmap, content strategy, and M5 planning docs - Add tmp/ and .claude/worktrees/ to .gitignore Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
85 lines
5.1 KiB
Markdown
85 lines
5.1 KiB
Markdown
# Milestone 1, Phase 3: Storage Engine Trait and fjall Backend
|
|
|
|
## Phase Deliverable
|
|
|
|
The `StorageEngine` trait abstraction and two implementations: `FjallBackend` (fjall 3 LSM-tree) for production and `InMemoryBackend` (BTreeMap + RwLock) for deterministic testing. Key encoding follows the subject-prefix pattern with a `Tag` discriminant. `FjallStorage` coordinates three keyspaces per entity kind. `FjallAtomicBatch` provides cross-keyspace atomic writes.
|
|
|
|
This phase is the durable entity store — where metadata, signal checkpoints, and index data live. It is separate from the WAL (m1p2): the WAL is the signal event source of truth; the storage engine is where derived entity state is persisted.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] `StorageEngine` trait with `get`, `put`, `delete`, `scan_prefix`, `write_batch`, `flush` operations
|
|
- [x] Key encoding: `[entity_id: 8 bytes BE][0x00][Tag: 1 byte][suffix...]` with `Tag` enum (`Evt`=0x01, `Sig`=0x02, `Meta`=0x03, `Rel`=0x04, `Mv`=0x05, `Idx`=0x06)
|
|
- [x] `encode_key`, `parse_key` roundtrip correctly for all tag variants and arbitrary suffixes
|
|
- [x] `entity_prefix` (9 bytes) and `entity_tag_prefix` (10 bytes) for scoped prefix scans
|
|
- [x] Byte-lexicographic key ordering matches numeric entity ID ordering (property tested)
|
|
- [x] `FjallBackend` wraps a single fjall `Keyspace`, implements `StorageEngine`
|
|
- [x] `FjallStorage` owns a fjall `Database` with three keyspaces: "items", "users", "creators" (one per `EntityKind`)
|
|
- [x] `FjallStorage::backend(EntityKind)` routes to the correct keyspace backend
|
|
- [x] Entity kind isolation: same key written to different entity kinds does not collide
|
|
- [x] `FjallAtomicBatch` provides cross-keyspace atomic writes via `fjall::OwnedWriteBatch`
|
|
- [x] Data persists across close and reopen (`flush_all` + reopen test)
|
|
- [x] `InMemoryBackend` uses `BTreeMap` + `RwLock` for deterministic, sorted, concurrent testing
|
|
- [x] `WriteBatch` and `BatchOp` types for atomic multi-operation writes
|
|
- [x] `PrefixIterator` type alias for boxed prefix scan iterators
|
|
- [x] Property tests with proptest: encode/parse roundtrip, prefix ordering, prefix containment
|
|
- [x] Criterion benchmarks passing
|
|
- [x] `cargo fmt` clean, `cargo clippy -D warnings` clean, all 140 tests pass (128 unit + 12 integration)
|
|
|
|
## Dependencies
|
|
|
|
- **Requires:** m1p1 (types: `EntityId`, `EntityKind` — used in key encoding; `StorageError` references `LumenError` error hierarchy)
|
|
- **Blocks:** m1p4 (Signal Ledger checkpoints via `StorageEngine`), m1p5 (Entity CRUD via `StorageEngine`)
|
|
|
|
## Research References
|
|
|
|
- [thoughts.md](../../../../thoughts.md) — Part V.9 (hybrid storage: fjall for entity metadata, WAL for signals), Part V.12 (subject-prefix keys: `[entity_id][NUL][TAG][suffix]` for co-location and prefix scan efficiency)
|
|
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) — Section 2 (key encoding: big-endian entity IDs for lexicographic ordering), Section 10 (fjall as the primary storage backend)
|
|
|
|
## Task Index
|
|
|
|
| # | Task | Delivers | Depends On | Complexity |
|
|
|---|------|----------|------------|------------|
|
|
| 01 | StorageEngine Trait and Key Encoding | `StorageEngine`, `Tag`, `encode_key`, `parse_key`, `entity_prefix`, `entity_tag_prefix`, `WriteBatch`, `BatchOp`, `PrefixIterator`, `StorageError` | None | M |
|
|
| 02 | FjallBackend | `FjallBackend`, `FjallStorage`, `FjallAtomicBatch`, persistence tests | Task 01 | M |
|
|
| 03 | InMemoryBackend | `InMemoryBackend`, property tests, benchmarks | Task 01 | S |
|
|
|
|
## Task Dependency DAG
|
|
|
|
```
|
|
Task 01: StorageEngine Trait + Key Encoding
|
|
|
|
|
+---------------------------+
|
|
| |
|
|
v v
|
|
Task 02: FjallBackend Task 03: InMemoryBackend
|
|
```
|
|
|
|
Tasks 02 and 03 are fully parallelizable after Task 01's trait and key encoding are defined.
|
|
|
|
## File Layout
|
|
|
|
```
|
|
tidal/src/
|
|
storage/
|
|
mod.rs -- pub use re-exports
|
|
engine.rs -- Task 01: StorageEngine trait
|
|
keys.rs -- Task 01: Tag, encode_key, parse_key, entity_prefix, entity_tag_prefix
|
|
batch.rs -- Task 01: WriteBatch, BatchOp
|
|
iterator.rs -- Task 01: PrefixIterator type alias
|
|
error.rs -- Task 01: StorageError
|
|
fjall.rs -- Task 02: FjallBackend, FjallStorage, FjallAtomicBatch
|
|
memory.rs -- Task 03: InMemoryBackend
|
|
lib.rs -- pub mod storage (already present)
|
|
```
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Keyspaces are per `EntityKind`**, not per data category. The `Tag` enum provides data-category namespace within each entity-kind keyspace. This means `FjallStorage` has three keyspaces: "items", "users", "creators". A `Tag::Meta` key in the "items" keyspace is distinct from `Tag::Meta` in the "users" keyspace.
|
|
|
|
2. **MSRV bumped to 1.91** for fjall 3 compatibility. Documented in `tidal/Cargo.toml`.
|
|
|
|
3. **`LumenError` name** is a legacy artifact from the predecessor project (Engram/Lumen). Will be renamed to `TidalError` when convenient but does not block m1p3 progress.
|
|
|
|
4. **`FjallAtomicBatch`** provides cross-keyspace atomicity via `fjall::OwnedWriteBatch`. This is the mechanism for m1p4 checkpoint writes that touch multiple entity kinds atomically.
|