# Task 04: BM25 Scoring Collectors

## Delivers

`AllScoresCollector` for returning all matching `(EntityId, f32)` pairs with BM25 scores, and `ScoredCandidateCollector` for scoring a pre-sorted candidate set using `DocSet::seek()`. Entity ID resolution from the `entity_id` fast field.

## Complexity: M

## Dependencies

- Task 01 complete: `TextIndex`, `TantivyFields` with `entity_id` fast field
- Task 02 complete: documents have `entity_id` fast field populated

## Technical Design

From `docs/research/tantivy.md` — the Collector API is the recommended approach:

### AllScoresCollector

Captures every `(DocAddress, Score)` pair matching a query. `requires_scoring()` must return `true` or BM25 is skipped.

```rust
// tidal/src/text/collectors.rs

use std::collections::HashMap;
use tantivy::{DocAddress, DocId, Score, SegmentOrdinal, SegmentReader};
use tantivy::collector::{Collector, SegmentCollector};
use tantivy::fastfield::FastFieldReader;
use crate::schema::EntityId;

// ── AllScoresCollector ────────────────────────────────────────────────────────

/// Tantivy Collector that captures all matching documents with their BM25 scores.
///
/// Returns `Vec<(EntityId, f32)>` — every matched item with its BM25 relevance score.
/// `requires_scoring()` must return `true` for BM25 to be computed.
pub struct AllScoresCollector {
    pub entity_id_field: tantivy::schema::Field,
}

pub struct AllScoresSegmentCollector {
    segment_ord: SegmentOrdinal,
    entity_id_reader: tantivy::fastfield::Column<u64>,
    results: Vec<(EntityId, f32)>,
}

impl Collector for AllScoresCollector {
    type Fruit = Vec<(EntityId, f32)>;
    type Child = AllScoresSegmentCollector;

    fn for_segment(
        &self,
        segment_local_id: SegmentOrdinal,
        segment: &SegmentReader,
    ) -> tantivy::Result<Self::Child> {
        let entity_id_reader = segment
            .fast_fields()
            .u64(self.entity_id_field)?;
        Ok(AllScoresSegmentCollector {
            segment_ord: segment_local_id,
            entity_id_reader,
            results: Vec::new(),
        })
    }

    fn requires_scoring(&self) -> bool {
        true  // CRITICAL: must be true for BM25 computation
    }

    fn merge_fruits(
        &self,
        segment_fruits: Vec<Vec<(EntityId, f32)>>,
    ) -> tantivy::Result<Self::Fruit> {
        Ok(segment_fruits.into_iter().flatten().collect())
    }
}

impl SegmentCollector for AllScoresSegmentCollector {
    type Fruit = Vec<(EntityId, f32)>;

    fn collect(&mut self, doc: DocId, score: Score) {
        if let Ok(entity_id_val) = self.entity_id_reader.get_val(doc) {
            self.results.push((EntityId::new(entity_id_val), score));
        }
    }

    fn harvest(self) -> Self::Fruit {
        self.results
    }
}
```

### ScoredCandidateCollector

Scores a pre-sorted candidate set using `DocSet::seek()`. From the research doc:
> Seek advances to the first doc ≥ target; if it returns exactly the target, the document matches the query, and `scorer.score()` gives its BM25 score.

For tidalDB's use case, we need to:
1. Map `EntityId` → `(SegmentOrdinal, DocId)` — requires maintaining a lookup table
2. Sort candidates by `(segment_ord, doc_id)` for ascending seek
3. For each candidate, seek to its `DocId` and read the score

The entity_id → DocAddress mapping needs to be maintained in `TextIndex`. After each commit, the reader reloads and we can rebuild the mapping by scanning the fast field.

```rust
/// Scores a pre-sorted candidate set via BM25 (seek-based).
///
/// For tidalDB's hybrid search: given ANN results as a candidate set,
/// get BM25 scores for those specific documents.
///
/// `candidates` must be sorted by (segment_ord, doc_id) ascending for seek to work.
pub fn score_candidates(
    searcher: &tantivy::Searcher,
    query: &dyn tantivy::query::Query,
    candidates: &[(u32, u32, EntityId)],  // (segment_ord, doc_id, entity_id)
) -> crate::Result<Vec<(EntityId, f32)>> {
    use tantivy::query::EnableScoring;

    let weight = query
        .weight(EnableScoring::enabled_from_searcher(searcher))
        .map_err(|e| crate::TidalError::Internal(format!("tantivy weight: {e}")))?;

    let mut results = Vec::with_capacity(candidates.len());

    // Group candidates by segment
    let mut by_segment: HashMap<u32, Vec<(u32, EntityId)>> = HashMap::new();
    for &(seg_ord, doc_id, entity_id) in candidates {
        by_segment.entry(seg_ord).or_default().push((doc_id, entity_id));
    }

    for (seg_ord, mut docs) in by_segment {
        docs.sort_by_key(|(doc_id, _)| *doc_id); // must be ascending for seek

        let segment_reader = searcher
            .segment_reader(seg_ord)
            .ok_or_else(|| crate::TidalError::Internal("segment not found".into()))?;

        let mut scorer = weight
            .scorer(segment_reader, 1.0)
            .map_err(|e| crate::TidalError::Internal(format!("tantivy scorer: {e}")))?;

        for (doc_id, entity_id) in docs {
            use tantivy::DocSet;
            let reached = scorer.seek(doc_id);
            if reached == doc_id {
                results.push((entity_id, scorer.score()));
            }
        }
    }

    Ok(results)
}
```

### EntityId → DocAddress Mapping

Maintain a `DashMap<u64, (u32, u32)>` — entity_id → (segment_ord, doc_id). This mapping:
- Is rebuilt after each commit (by scanning the fast field across all segments)
- Is used by `score_candidates()` to convert EntityId candidates to Tantivy addresses
- Can be rebuilt from the index at any time (derived state)

Add to `TextIndex`:
```rust
/// Entity ID → (segment_ord, doc_id) mapping.
/// Rebuilt after each commit by scanning the entity_id fast field.
pub(crate) entity_map: Arc<DashMap<u64, (u32, u32)>>,
```

Add `TextIndex::rebuild_entity_map()` method that scans all segments.

## Acceptance Criteria

- [ ] `AllScoresCollector` implements Tantivy's `Collector` trait
- [ ] `AllScoresCollector::requires_scoring()` returns `true`
- [ ] `AllScoresCollector::for_segment()` creates a segment collector with fast field reader
- [ ] `AllScoresSegmentCollector::collect()` reads the `entity_id` fast field value and pushes `(EntityId, f32)` to results
- [ ] `AllScoresCollector::merge_fruits()` flattens segment results into a single `Vec<(EntityId, f32)>`
- [ ] `score_candidates()` function groups candidates by segment, sorts by doc_id ascending, seeks through posting lists, returns scores for matching documents
- [ ] `TextIndex::entity_map: Arc<DashMap<u64, (u32, u32)>>` maintained
- [ ] `TextIndex::rebuild_entity_map()` scans all segments and populates the entity map
- [ ] `rebuild_entity_map()` called after each commit in the syncer
- [ ] Unit tests: `all_scores_collector_captures_bm25`, `all_scores_requires_scoring`, `score_candidates_seek_based`, `entity_map_rebuilds_after_commit`, `missing_candidate_skipped`
- [ ] Property test: for any set of indexed documents, `AllScoresCollector` returns exactly the matching documents with positive scores
- [ ] `cargo check`, `cargo fmt`, `cargo clippy -D warnings` all pass

## Test Strategy

```rust
#[test]
fn all_scores_collector_captures_bm25() {
    let fields = vec![TextFieldDef { key: "title".into(), field_type: TextFieldType::Text }];
    let idx = TextIndex::ephemeral(&fields).unwrap();

    // Index 3 documents
    let mut w = idx.writer_guard().unwrap();
    for (i, title) in [(1u64, "jazz piano"), (2u64, "rock guitar"), (3u64, "jazz violin")] {
        let mut m = HashMap::new();
        m.insert("title".into(), title.into());
        w.index_item(EntityId::new(i), &m).unwrap();
    }
    w.commit(3).unwrap();

    // Force reader reload
    idx.reader.reload().unwrap();
    let searcher = idx.reader.searcher();

    // Search for "jazz"
    let qp = tantivy::query::QueryParser::for_index(&idx.index, vec![idx.fields().text_fields[0].1]);
    let query = qp.parse_query("jazz").unwrap();

    let collector = AllScoresCollector { entity_id_field: idx.fields().entity_id };
    let results = searcher.search(&query, &collector).unwrap();

    // Should find entities 1 and 3 (jazz), not 2 (rock)
    let ids: Vec<u64> = results.iter().map(|(id, _)| id.get()).collect();
    assert!(ids.contains(&1));
    assert!(ids.contains(&3));
    assert!(!ids.contains(&2));

    // All BM25 scores should be positive
    for (_, score) in &results {
        assert!(*score > 0.0);
    }
}

#[test]
fn score_candidates_seek_based() {
    // Index 5 docs, rebuild entity_map, score only candidates [1, 3] via seek
}
```
