tidaldb/docs/planning/milestone-7/phase-4/task-03-index-health-metrics.md

# Task 03: Index Health Metrics

## Delivers

Prometheus gauges for all secondary index health: Tantivy segment count and indexed document count, USearch vector count and byte size, bitmap index total cardinality. These metrics let operators detect stale derived indexes, growing segment fragmentation, and index size anomalies before they affect query latency.

## Complexity: M

## Dependencies

- task-01 complete (establishes instrumentation pattern)
- `tidal/src/db/metrics.rs` -- `MetricsState` to extend
- `tidal/src/text/index.rs` -- `TextIndex` wraps Tantivy `Index` and `IndexReader`
- `tidal/src/storage/vector/registry.rs` -- `EmbeddingSlotRegistry` owns USearch indexes
- `tidal/src/storage/indexes/bitmap.rs` -- `BitmapIndex` with `RoaringBitmap` values

## Technical Design

### 1. Add atomic gauges to MetricsState

In `tidal/src/db/metrics.rs`:

```rust
pub struct MetricsState {
    // ... existing + task-02 fields ...

    // ── Index health metrics (m7p4) ────────────────────────────────────
    /// Number of Tantivy segments for the items text index.
    #[cfg(feature = "metrics")]
    pub(crate) tantivy_segment_count: AtomicU64,
    /// Number of documents indexed in the items text index.
    #[cfg(feature = "metrics")]
    pub(crate) tantivy_indexed_docs: AtomicU64,
    /// Total byte size of the USearch index files on disk (or in-memory estimate).
    #[cfg(feature = "metrics")]
    pub(crate) usearch_index_size_bytes: AtomicU64,
    /// Number of vectors stored in the USearch index.
    #[cfg(feature = "metrics")]
    pub(crate) usearch_vector_count: AtomicU64,
    /// Total cardinality across all bitmap index entries (category + format + creator + tag).
    #[cfg(feature = "metrics")]
    pub(crate) bitmap_index_cardinality: AtomicU64,
}
```

### 2. Expose index introspection methods

#### TextIndex

Add a method to `TextIndex` for segment and document count:

```rust
impl TextIndex {
    /// Return the number of segments and total indexed documents.
    ///
    /// Reads from the current IndexReader snapshot. Thread-safe.
    #[must_use]
    pub fn index_stats(&self) -> (usize, u64) {
        let searcher = self.reader.searcher();
        let segment_count = searcher.segment_readers().len();
        let doc_count = searcher
            .segment_readers()
            .iter()
            .map(|r| u64::from(r.num_docs()))
            .sum();
        (segment_count, doc_count)
    }
}
```

#### EmbeddingSlotRegistry

Add a method to report total vector count and estimated byte size:

```rust
impl EmbeddingSlotRegistry {
    /// Return the total vector count and estimated byte size across all slots.
    #[must_use]
    pub fn index_stats(&self) -> (u64, u64) {
        let mut total_vectors: u64 = 0;
        let mut total_bytes: u64 = 0;
        for slot in self.slots.values() {
            let count = slot.index.size() as u64;
            // USearch reports serialized size; use dimensions * sizeof(f16) * count as estimate
            let dim = slot.dimensions as u64;
            let bytes = count * dim * 2; // f16 = 2 bytes
            total_vectors += count;
            total_bytes += bytes;
        }
        (total_vectors, total_bytes)
    }
}
```

If USearch provides a `serialized_length()` method, prefer that over the estimate. The estimate is a lower bound (excludes HNSW graph overhead).

#### BitmapIndex

Add a method to report total cardinality:

```rust
impl BitmapIndex {
    /// Total number of entity IDs across all bitmap entries.
    #[must_use]
    pub fn total_cardinality(&self) -> u64 {
        self.entries.iter().map(|e| e.value().len()).sum()
    }
}
```

### 3. Periodic metrics refresh

In the checkpoint thread or a dedicated metrics-refresh interval (reuse the pattern from task-02), collect index stats:

```rust
#[cfg(feature = "metrics")]
fn refresh_index_metrics(db: &TidalDb) {
    // Tantivy
    if let Some(text_index) = &db.text_index {
        let (segments, docs) = text_index.index_stats();
        db.metrics.tantivy_segment_count.store(segments as u64, Ordering::Relaxed);
        db.metrics.tantivy_indexed_docs.store(docs, Ordering::Relaxed);
    }

    // USearch
    if let Ok(registry) = db.embedding_registry.read() {
        let (vectors, bytes) = registry.index_stats();
        db.metrics.usearch_vector_count.store(vectors, Ordering::Relaxed);
        db.metrics.usearch_index_size_bytes.store(bytes, Ordering::Relaxed);
    }

    // Bitmap indexes
    let cardinality = db.category_index.total_cardinality()
        + db.format_index.total_cardinality()
        + db.creator_index.total_cardinality()
        + db.tag_index.total_cardinality();
    db.metrics.bitmap_index_cardinality.store(cardinality, Ordering::Relaxed);
}
```

Call this function every 10 seconds from the checkpoint thread's periodic loop. Index stats are not hot-path -- 10-second staleness is acceptable for monitoring.

### 4. Render in Prometheus format

Extend `MetricsState::render_prometheus()`:

```rust
// Tantivy
write_gauge(&mut out, "tidaldb_tantivy_segment_count",
    "Number of Tantivy index segments",
    self.tantivy_segment_count.load(Ordering::Relaxed) as f64);

write_gauge(&mut out, "tidaldb_tantivy_indexed_docs",
    "Number of documents indexed in Tantivy",
    self.tantivy_indexed_docs.load(Ordering::Relaxed) as f64);

// USearch
write_gauge(&mut out, "tidaldb_usearch_index_size_bytes",
    "Estimated byte size of USearch vector indexes",
    self.usearch_index_size_bytes.load(Ordering::Relaxed) as f64);

write_gauge(&mut out, "tidaldb_usearch_vector_count",
    "Number of vectors stored in USearch indexes",
    self.usearch_vector_count.load(Ordering::Relaxed) as f64);

// Bitmap
write_gauge(&mut out, "tidaldb_bitmap_index_cardinality",
    "Total entity IDs across all bitmap indexes",
    self.bitmap_index_cardinality.load(Ordering::Relaxed) as f64);
```

### 5. Metric names (string literals)

| Metric name | Type | Description |
|---|---|---|
| `tidaldb_tantivy_segment_count` | gauge | Number of Tantivy index segments |
| `tidaldb_tantivy_indexed_docs` | gauge | Number of documents indexed in Tantivy |
| `tidaldb_usearch_index_size_bytes` | gauge | Estimated byte size of USearch vector indexes |
| `tidaldb_usearch_vector_count` | gauge | Number of vectors stored in USearch indexes |
| `tidaldb_bitmap_index_cardinality` | gauge | Total entity IDs across all bitmap indexes |

## Acceptance Criteria

- [ ] `TextIndex::index_stats()` returns `(segment_count, doc_count)` correctly
- [ ] `EmbeddingSlotRegistry::index_stats()` returns `(vector_count, byte_size)`
- [ ] `BitmapIndex::total_cardinality()` sums across all entries
- [ ] `MetricsState` extended with 5 atomic gauges, all `#[cfg(feature = "metrics")]`
- [ ] Metrics refreshed periodically (every 10 seconds in checkpoint thread)
- [ ] `/metrics` endpoint renders all 5 new metrics in valid Prometheus format
- [ ] Metrics reflect actual index state after writes (verified in integration test)
- [ ] `cargo clippy -D warnings` and `cargo fmt --check` pass

## Test Strategy

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn text_index_stats_empty() {
        let fields = vec![TextFieldDef { key: "title".into(), field_type: TextFieldType::Text }];
        let idx = TextIndex::ephemeral(&fields).unwrap();
        let (segments, docs) = idx.index_stats();
        assert_eq!(docs, 0);
        // Tantivy may report 0 or 1 segments for an empty index
        assert!(segments <= 1);
    }

    #[test]
    fn bitmap_total_cardinality_empty() {
        let idx = BitmapIndex::new("test");
        assert_eq!(idx.total_cardinality(), 0);
    }

    #[test]
    fn bitmap_total_cardinality_after_inserts() {
        let idx = BitmapIndex::new("test");
        idx.insert("jazz", 1);
        idx.insert("jazz", 2);
        idx.insert("rock", 3);
        assert_eq!(idx.total_cardinality(), 3);
    }
}
```

Integration test:

```rust
#[test]
fn index_metrics_reflect_writes() {
    let db = make_test_db_with_text_schema();
    // Write items with metadata
    for i in 0..10 {
        db.write_item_with_metadata(
            EntityId::new(i),
            &HashMap::from([
                ("title".to_string(), format!("Item {i}")),
                ("category".to_string(), "jazz".to_string()),
            ]),
        ).unwrap();
    }
    db.flush_text_index().unwrap();

    let metrics = db.metrics();
    let prom = metrics.render_prometheus();
    assert!(prom.contains("tidaldb_tantivy_indexed_docs"));
    assert!(prom.contains("tidaldb_bitmap_index_cardinality"));
}
```