tidaldb/tidal/docs/planning/milestone-5/phase-1/OVERVIEW.md

m5p1: Tantivy Integration

Delivers

Tantivy embedded as a derived index for full-text search. DB-primary consistency pattern: entity store is source of truth, Tantivy is a materialized view updated via an outbox sequence. BM25 scoring exposed via custom Collector and the Weight/Scorer seek pattern. Schema text fields (title, description, tags) automatically indexed. Crash recovery replays from the last committed sequence number stored in Tantivy's commit payload.

Dependencies

• m1p3 (storage engine, key encoding, StorageEngine trait, scan_prefix)
• m1p5 (entity write API, WAL sequence numbers)
• m2p2 (metadata fields used for field-scoped queries)
• m4 (full TidalDb API with sessions and agents — all complete)

Research References

• docs/research/tantivy.md — Collector API, consistency pattern, seek scoring, commit model, single-writer lock, segment merge
• CODING_GUIDELINES.md Section 5 — Text Search guidelines
• CODING_GUIDELINES.md Section 7 — Error handling

Acceptance Criteria (Phase Level)

• TextIndex struct wraps Tantivy Index, IndexWriter (behind Mutex), and IndexReader with auto-reload
• Tantivy schema created from tidalDB schema text field definitions: text fields get full-text tokenization; keyword fields get raw indexing
• TextIndexWriter::index_item(entity_id, metadata) adds or updates a document in Tantivy; delete_item(entity_id) removes via delete_term
• Background indexer: TextIndexSyncer reads entity store writes (via WAL sequence tracking) and feeds Tantivy writer; commit interval configurable (default: every 1000 docs or 2 seconds)
• Each Tantivy commit() stores the last-processed WAL sequence number in the commit payload via set_payload(); crash recovery replays from that sequence number
• Custom AllScoresCollector implementing Tantivy's Collector trait returns all matching (EntityId, f32) pairs with BM25 scores; requires_scoring() returns true
• ScoredCandidateCollector implementing Tantivy's Collector trait accepts a pre-sorted candidate set and returns BM25 scores via DocSet::seek()
• External EntityId -> DocAddress mapping maintained via a fast field (entity_id_field) on every Tantivy document
• Boolean query parsing: AND, OR, NOT operators; exact phrase ("..."); field-scoped (title:jazz); exclusion (-beginner); wildcard prefix (pian*)
• Index rebuild from entity store: text_index.rebuild_from(storage) scans all items and rebuilds Tantivy index
• BM25 query latency < 10ms at 10K documents (Criterion benchmarked)
• Tantivy IndexWriter heap budget set to 50MB
• LogMergePolicy configured with defaults; wait_merging_threads() called on shutdown
• TextIndex is Send + Sync — safe to share across threads behind Arc

Task Execution Order

task-01 (TextIndex Core)
    |
    v
task-02 (Document Write/Delete)
    |            |              |
    v            v              v
task-03       task-04        task-05
(Syncer)    (Collectors)  (Query Parser)

Tasks 01-02 are sequential. Tasks 03, 04, 05 can parallelize after task-02 completes.

Module Location

New module: tidal/src/text/ with submodules:

• mod.rs — public re-exports
• index.rs — TextIndex, TextIndexConfig
• writer.rs — TextIndexWriter (write/delete operations)
• syncer.rs — TextIndexSyncer (background indexing)
• collectors.rs — AllScoresCollector, ScoredCandidateCollector
• query.rs — TextQueryParser

Notes

• tantivy must be added to tidal/Cargo.toml as a dependency
• Text field definitions must be added to Schema / SchemaBuilder
• The unsafe_code = "forbid" lint is crate-level — tantivy itself uses unsafe but we do not need unsafe in our wrapper code
• tantivy crate itself has forbid(unsafe_code) in some modules but not all — the FFI is contained within their crate

Done When

All 14 acceptance criteria above pass. Tests pass with cargo test. The text_index bench shows BM25 query < 10ms at 10K documents.