3bcfb3c576
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather
72 lines
4.0 KiB
Markdown
72 lines
4.0 KiB
Markdown
# m5p1: Tantivy Integration
|
|
|
|
## Delivers
|
|
|
|
Tantivy embedded as a derived index for full-text search. DB-primary consistency pattern: entity store is source of truth, Tantivy is a materialized view updated via an outbox sequence. BM25 scoring exposed via custom Collector and the Weight/Scorer seek pattern. Schema text fields (title, description, tags) automatically indexed. Crash recovery replays from the last committed sequence number stored in Tantivy's commit payload.
|
|
|
|
## Dependencies
|
|
|
|
- m1p3 (storage engine, key encoding, `StorageEngine` trait, `scan_prefix`)
|
|
- m1p5 (entity write API, WAL sequence numbers)
|
|
- m2p2 (metadata fields used for field-scoped queries)
|
|
- m4 (full TidalDb API with sessions and agents — all complete)
|
|
|
|
## Research References
|
|
|
|
- `docs/research/tantivy.md` — Collector API, consistency pattern, seek scoring, commit model, single-writer lock, segment merge
|
|
- `CODING_GUIDELINES.md` Section 5 — Text Search guidelines
|
|
- `CODING_GUIDELINES.md` Section 7 — Error handling
|
|
|
|
## Acceptance Criteria (Phase Level)
|
|
|
|
- [ ] `TextIndex` struct wraps Tantivy `Index`, `IndexWriter` (behind `Mutex`), and `IndexReader` with auto-reload
|
|
- [ ] Tantivy schema created from tidalDB schema text field definitions: `text` fields get full-text tokenization; `keyword` fields get raw indexing
|
|
- [ ] `TextIndexWriter::index_item(entity_id, metadata)` adds or updates a document in Tantivy; `delete_item(entity_id)` removes via `delete_term`
|
|
- [ ] Background indexer: `TextIndexSyncer` reads entity store writes (via WAL sequence tracking) and feeds Tantivy writer; commit interval configurable (default: every 1000 docs or 2 seconds)
|
|
- [ ] Each Tantivy `commit()` stores the last-processed WAL sequence number in the commit payload via `set_payload()`; crash recovery replays from that sequence number
|
|
- [ ] Custom `AllScoresCollector` implementing Tantivy's `Collector` trait returns all matching `(EntityId, f32)` pairs with BM25 scores; `requires_scoring()` returns `true`
|
|
- [ ] `ScoredCandidateCollector` implementing Tantivy's `Collector` trait accepts a pre-sorted candidate set and returns BM25 scores via `DocSet::seek()`
|
|
- [ ] External `EntityId -> DocAddress` mapping maintained via a fast field (`entity_id_field`) on every Tantivy document
|
|
- [ ] Boolean query parsing: AND, OR, NOT operators; exact phrase (`"..."`); field-scoped (`title:jazz`); exclusion (`-beginner`); wildcard prefix (`pian*`)
|
|
- [ ] Index rebuild from entity store: `text_index.rebuild_from(storage)` scans all items and rebuilds Tantivy index
|
|
- [ ] BM25 query latency < 10ms at 10K documents (Criterion benchmarked)
|
|
- [ ] Tantivy `IndexWriter` heap budget set to 50MB
|
|
- [ ] `LogMergePolicy` configured with defaults; `wait_merging_threads()` called on shutdown
|
|
- [ ] `TextIndex` is `Send + Sync` — safe to share across threads behind `Arc`
|
|
|
|
## Task Execution Order
|
|
|
|
```
|
|
task-01 (TextIndex Core)
|
|
|
|
|
v
|
|
task-02 (Document Write/Delete)
|
|
| | |
|
|
v v v
|
|
task-03 task-04 task-05
|
|
(Syncer) (Collectors) (Query Parser)
|
|
```
|
|
|
|
Tasks 01-02 are sequential. Tasks 03, 04, 05 can parallelize after task-02 completes.
|
|
|
|
## Module Location
|
|
|
|
New module: `tidal/src/text/` with submodules:
|
|
- `mod.rs` — public re-exports
|
|
- `index.rs` — `TextIndex`, `TextIndexConfig`
|
|
- `writer.rs` — `TextIndexWriter` (write/delete operations)
|
|
- `syncer.rs` — `TextIndexSyncer` (background indexing)
|
|
- `collectors.rs` — `AllScoresCollector`, `ScoredCandidateCollector`
|
|
- `query.rs` — `TextQueryParser`
|
|
|
|
## Notes
|
|
|
|
- `tantivy` must be added to `tidal/Cargo.toml` as a dependency
|
|
- Text field definitions must be added to `Schema` / `SchemaBuilder`
|
|
- The `unsafe_code = "forbid"` lint is crate-level — `tantivy` itself uses unsafe but we do not need unsafe in our wrapper code
|
|
- `tantivy` crate itself has `forbid(unsafe_code)` in some modules but not all — the FFI is contained within their crate
|
|
|
|
## Done When
|
|
|
|
All 14 acceptance criteria above pass. Tests pass with `cargo test`. The `text_index` bench shows BM25 query < 10ms at 10K documents.
|