# Milestone 1, Phase 3: Storage Engine Trait and fjall Backend

## Phase Deliverable

The `StorageEngine` trait abstraction and two implementations: `FjallBackend` (fjall 3 LSM-tree) for production and `InMemoryBackend` (BTreeMap + RwLock) for deterministic testing. Key encoding follows the subject-prefix pattern with a `Tag` discriminant. `FjallStorage` coordinates three keyspaces per entity kind. `FjallAtomicBatch` provides cross-keyspace atomic writes.

This phase is the durable entity store — where metadata, signal checkpoints, and index data live. It is separate from the WAL (m1p2): the WAL is the signal event source of truth; the storage engine is where derived entity state is persisted.

## Acceptance Criteria

- [x] `StorageEngine` trait with `get`, `put`, `delete`, `scan_prefix`, `write_batch`, `flush` operations
- [x] Key encoding: `[entity_id: 8 bytes BE][0x00][Tag: 1 byte][suffix...]` with `Tag` enum (`Evt`=0x01, `Sig`=0x02, `Meta`=0x03, `Rel`=0x04, `Mv`=0x05, `Idx`=0x06)
- [x] `encode_key`, `parse_key` roundtrip correctly for all tag variants and arbitrary suffixes
- [x] `entity_prefix` (9 bytes) and `entity_tag_prefix` (10 bytes) for scoped prefix scans
- [x] Byte-lexicographic key ordering matches numeric entity ID ordering (property tested)
- [x] `FjallBackend` wraps a single fjall `Keyspace`, implements `StorageEngine`
- [x] `FjallStorage` owns a fjall `Database` with three keyspaces: "items", "users", "creators" (one per `EntityKind`)
- [x] `FjallStorage::backend(EntityKind)` routes to the correct keyspace backend
- [x] Entity kind isolation: same key written to different entity kinds does not collide
- [x] `FjallAtomicBatch` provides cross-keyspace atomic writes via `fjall::OwnedWriteBatch`
- [x] Data persists across close and reopen (`flush_all` + reopen test)
- [x] `InMemoryBackend` uses `BTreeMap` + `RwLock` for deterministic, sorted, concurrent testing
- [x] `WriteBatch` and `BatchOp` types for atomic multi-operation writes
- [x] `PrefixIterator` type alias for boxed prefix scan iterators
- [x] Property tests with proptest: encode/parse roundtrip, prefix ordering, prefix containment
- [x] Criterion benchmarks passing
- [x] `cargo fmt` clean, `cargo clippy -D warnings` clean, all 140 tests pass (128 unit + 12 integration)

## Dependencies

- **Requires:** m1p1 (types: `EntityId`, `EntityKind` — used in key encoding; `StorageError` references `LumenError` error hierarchy)
- **Blocks:** m1p4 (Signal Ledger checkpoints via `StorageEngine`), m1p5 (Entity CRUD via `StorageEngine`)

## Research References

- [thoughts.md](../../../../thoughts.md) — Part V.9 (hybrid storage: fjall for entity metadata, WAL for signals), Part V.12 (subject-prefix keys: `[entity_id][NUL][TAG][suffix]` for co-location and prefix scan efficiency)
- [CODING_GUIDELINES.md](../../../../CODING_GUIDELINES.md) — Section 2 (key encoding: big-endian entity IDs for lexicographic ordering), Section 10 (fjall as the primary storage backend)

## Task Index

| # | Task | Delivers | Depends On | Complexity |
|---|------|----------|------------|------------|
| 01 | StorageEngine Trait and Key Encoding | `StorageEngine`, `Tag`, `encode_key`, `parse_key`, `entity_prefix`, `entity_tag_prefix`, `WriteBatch`, `BatchOp`, `PrefixIterator`, `StorageError` | None | M |
| 02 | FjallBackend | `FjallBackend`, `FjallStorage`, `FjallAtomicBatch`, persistence tests | Task 01 | M |
| 03 | InMemoryBackend | `InMemoryBackend`, property tests, benchmarks | Task 01 | S |

## Task Dependency DAG

```
Task 01: StorageEngine Trait + Key Encoding
    |
    +---------------------------+
    |                           |
    v                           v
Task 02: FjallBackend       Task 03: InMemoryBackend
```

Tasks 02 and 03 are fully parallelizable after Task 01's trait and key encoding are defined.

## File Layout

```
tidal/src/
  storage/
    mod.rs          -- pub use re-exports
    engine.rs       -- Task 01: StorageEngine trait
    keys.rs         -- Task 01: Tag, encode_key, parse_key, entity_prefix, entity_tag_prefix
    batch.rs        -- Task 01: WriteBatch, BatchOp
    iterator.rs     -- Task 01: PrefixIterator type alias
    error.rs        -- Task 01: StorageError
    fjall.rs        -- Task 02: FjallBackend, FjallStorage, FjallAtomicBatch
    memory.rs       -- Task 03: InMemoryBackend
  lib.rs            -- pub mod storage (already present)
```

## Lessons Learned

1. **Keyspaces are per `EntityKind`**, not per data category. The `Tag` enum provides data-category namespace within each entity-kind keyspace. This means `FjallStorage` has three keyspaces: "items", "users", "creators". A `Tag::Meta` key in the "items" keyspace is distinct from `Tag::Meta` in the "users" keyspace.

2. **MSRV bumped to 1.91** for fjall 3 compatibility. Documented in `tidal/Cargo.toml`.

3. **`LumenError` name** is a legacy artifact from the predecessor project (Engram/Lumen). Will be renamed to `TidalError` when convenient but does not block m1p3 progress.

4. **`FjallAtomicBatch`** provides cross-keyspace atomicity via `fjall::OwnedWriteBatch`. This is the mechanism for m1p4 checkpoint writes that touch multiple entity kinds atomically.