# Task 03: Composable Filter Engine

## Context

**Milestone:** 2 -- Ranked Retrieval
**Phase:** m2p2 -- Metadata Indexes and Filter Engine
**Depends On:** Task 01 (BitmapIndex), Task 02 (RangeIndex)
**Blocks:** m2p1 Task 04 (adaptive query planner's `SelectivityEstimator`), m2p5 (RETRIEVE executor)
**Complexity:** M

## Objective

Deliver `FilterExpr`, `FilterEvaluator`, and `FilterResult` -- the composable filter evaluation engine that sits between the query parser (m2p5) and the index structures (Tasks 01 and 02). The filter engine takes an arbitrary boolean combination of metadata predicates, evaluates them against bitmap and range indexes, and produces either a `RoaringBitmap` (for pre-filtering ANN candidates) or a predicate closure `Fn(u64) -> bool` (for in-graph filtering during HNSW traversal).

This is the component that makes `FILTER category:jazz, format:video, duration_min:5m, created_within:7d` work as a single composable expression. The query parser (m2p5) builds a `FilterExpr` tree. The executor passes it to `FilterEvaluator` which resolves each leaf against the appropriate index, composes the results via bitmap algebra (AND = intersection, OR = union, NOT = complement), and returns a `FilterResult` that can be used either as a bitmap or a predicate closure.

The filter engine also provides selectivity estimation for the adaptive query planner (m2p1 Task 04, Spec 07 Section 9). Before executing the ANN search, the planner calls `evaluator.selectivity(&expr)` to estimate what fraction of the corpus matches the filter. This determines the search strategy: brute-force for <1%, widened HNSW for 1-20%, in-graph predicate for >20%, unfiltered for 100%.

## Requirements

- `FilterExpr` enum: the AST of filter conditions (categorical equality, range predicates, boolean composition)
- `FilterEvaluator` struct: holds references to `BitmapIndex` and `RangeIndex` instances, evaluates filter expressions
- `FilterResult` enum: either `Bitmap(RoaringBitmap)` or `Predicate(Box<dyn Fn(u64) -> bool + Send + Sync>)`
- `evaluator.evaluate(expr) -> FilterResult` -- evaluates a filter expression against indexes
- `evaluator.selectivity(expr) -> f64` -- estimates the fraction of entities matching the filter
- `result.to_bitmap() -> RoaringBitmap` -- extracts or computes the bitmap representation
- `result.to_predicate() -> Box<dyn Fn(u64) -> bool + Send + Sync>` -- extracts or wraps as a predicate closure
- AND composition: bitmap intersection for two bitmap results
- OR composition: bitmap union for two bitmap results
- NOT composition: bitmap complement (requires the "universe" bitmap of all entity IDs)
- Short-circuit evaluation: sort AND operands by ascending selectivity, abort on empty (Spec 08 Section 7.4)
- Compound AND selectivity: product of individual selectivities (independence assumption)
- Compound OR selectivity: `1 - product(1 - s_i)` (inclusion-exclusion approximation)
- Criterion benchmarks for filter evaluation and predicate-per-candidate performance

## Technical Design

### Module Structure

```
tidal/src/storage/
  indexes/
    filter.rs    -- FilterExpr, FilterEvaluator, FilterResult (this task)
tidal/benches/
  filters.rs     -- Criterion benchmarks (this task)
```

### Public API

```rust
// === storage/indexes/filter.rs ===

use roaring::RoaringBitmap;
use std::time::Duration;

use super::bitmap::BitmapIndex;
use super::range::RangeIndex;
use super::IndexError;

/// A filter expression AST node.
///
/// Built by the query parser (m2p5) from the `FILTER` clause
/// of a RETRIEVE or SEARCH query. Evaluated by `FilterEvaluator`
/// against bitmap and range indexes.
///
/// # Composition
///
/// Filters compose naturally:
/// - `And(vec![CategoryEq("jazz"), FormatEq("video")])` = jazz videos
/// - `Or(vec![CategoryEq("jazz"), CategoryEq("blues")])` = jazz or blues
/// - `And(vec![CategoryEq("jazz"), DurationMin(300)])` = jazz items >= 5 min
/// - `Not(Box::new(CreatorEq(77)))` = not by creator 77
///
/// The query parser enforces valid structure: AND across dimensions,
/// OR within a dimension. The evaluator handles any structure.
#[derive(Debug, Clone, PartialEq)]
pub enum FilterExpr {
    /// Exact equality on a categorical field: `category:jazz`
    CategoryEq(String),
    /// Exact equality on format field: `format:video`
    FormatEq(String),
    /// Exact equality on creator: `creator:@id`
    CreatorEq(u32),
    /// Tag match (multi-value field): `tag:classical`
    Tag(String),
    /// Minimum duration in seconds: `duration_min:300` (5 minutes)
    DurationMin(u32),
    /// Maximum duration in seconds: `duration_max:1800` (30 minutes)
    DurationMax(u32),
    /// Created after a timestamp (nanos): `created_after:...`
    CreatedAfter(u64),
    /// Created before a timestamp (nanos): `created_before:...`
    CreatedBefore(u64),
    /// AND composition: all sub-expressions must match.
    And(Vec<FilterExpr>),
    /// OR composition: at least one sub-expression must match.
    Or(Vec<FilterExpr>),
    /// NOT composition: the sub-expression must NOT match.
    /// Requires a universe bitmap for complement computation.
    Not(Box<FilterExpr>),
}

/// The result of evaluating a filter expression.
///
/// Can be materialized as either a `RoaringBitmap` (for pre-filtering)
/// or a predicate closure (for in-graph filtering).
pub enum FilterResult {
    /// A bitmap of matching entity IDs. Used for pre-filtering ANN
    /// candidates or intersecting with other filter results.
    Bitmap(RoaringBitmap),
    /// A predicate closure for in-graph filtering when bitmap
    /// materialization is deferred or unnecessary.
    Predicate(Box<dyn Fn(u64) -> bool + Send + Sync>),
}

impl FilterResult {
    /// Extract the bitmap. This is the primary representation.
    pub fn into_bitmap(self) -> RoaringBitmap;

    /// Convert to a predicate closure that checks bitmap containment.
    ///
    /// The predicate captures the bitmap and returns `true` if the
    /// entity ID is present. This is the closure passed to
    /// `VectorIndex::filtered_search()` for in-graph filtering.
    ///
    /// Performance: bitmap containment check is O(1) amortized,
    /// approximately 10-50 nanoseconds per check.
    pub fn into_predicate(self) -> Box<dyn Fn(u64) -> bool + Send + Sync>;

    /// Number of entities matching the filter.
    pub fn cardinality(&self) -> u64;

    /// Whether no entities match.
    pub fn is_empty(&self) -> bool;
}

/// Evaluates filter expressions against bitmap and range indexes.
///
/// The evaluator holds references to all indexes needed for filter
/// resolution. It is constructed once per query and evaluates the
/// filter expression tree.
///
/// # Selectivity Estimation
///
/// The `selectivity()` method estimates the fraction of entities
/// matching a filter WITHOUT materializing the full bitmap. This
/// is used by the adaptive query planner (m2p1) to choose the
/// ANN search strategy before executing the search.
///
/// For leaf predicates, selectivity uses `BitmapIndex::cardinality()`
/// or `RangeIndex::selectivity()`. For compound predicates:
/// - AND: product of individual selectivities (independence assumption)
/// - OR: `1 - product(1 - s_i)` (inclusion-exclusion approximation)
/// - NOT: `1 - selectivity(inner)`
pub struct FilterEvaluator<'a> {
    category_index: &'a BitmapIndex,
    format_index: &'a BitmapIndex,
    creator_index: &'a BitmapIndex,
    tag_index: &'a BitmapIndex,
    duration_index: &'a RangeIndex<u32>,
    created_at_index: &'a RangeIndex<u64>,
    /// The universe bitmap: all entity IDs currently in the database.
    /// Used for NOT (complement) operations.
    universe: &'a RoaringBitmap,
}

impl<'a> FilterEvaluator<'a> {
    /// Create a new evaluator with references to all indexes.
    pub fn new(
        category_index: &'a BitmapIndex,
        format_index: &'a BitmapIndex,
        creator_index: &'a BitmapIndex,
        tag_index: &'a BitmapIndex,
        duration_index: &'a RangeIndex<u32>,
        created_at_index: &'a RangeIndex<u64>,
        universe: &'a RoaringBitmap,
    ) -> Self;

    /// Evaluate a filter expression and return the matching entity set.
    ///
    /// Recursively evaluates the expression tree:
    /// - Leaf nodes resolve to index lookups.
    /// - AND nodes intersect child bitmaps.
    /// - OR nodes union child bitmaps.
    /// - NOT nodes compute the complement against the universe.
    ///
    /// # Short-Circuit Optimization (Spec 08, Section 7.4)
    ///
    /// For AND nodes, children are sorted by estimated selectivity
    /// (ascending). After each child evaluation, if the running
    /// intersection is empty, evaluation stops immediately.
    pub fn evaluate(&self, expr: &FilterExpr) -> FilterResult;

    /// Estimate the fraction of entities matching a filter expression.
    ///
    /// This is a cheap computation that does NOT materialize bitmaps.
    /// It uses index cardinalities and the independence assumption
    /// for compound predicates.
    ///
    /// Returns a value in [0.0, 1.0].
    ///
    /// Used by the adaptive query planner to choose the ANN strategy:
    /// - selectivity < 0.01 -> pre-filter + brute-force
    /// - selectivity 0.01 - 0.20 -> widened HNSW (ACORN-1)
    /// - selectivity > 0.20 -> in-graph predicate filter
    /// - selectivity == 1.0 -> unfiltered search
    pub fn selectivity(&self, expr: &FilterExpr) -> f64;
}
```

### Internal Design

**Leaf evaluation:**

Each `FilterExpr` leaf resolves to an index lookup:

| FilterExpr Variant | Index | Method |
|-------------------|-------|--------|
| `CategoryEq(v)` | `category_index` | `get(v) -> RoaringBitmap` |
| `FormatEq(v)` | `format_index` | `get(v) -> RoaringBitmap` |
| `CreatorEq(id)` | `creator_index` | `get(&id.to_string()) -> RoaringBitmap` |
| `Tag(v)` | `tag_index` | `get(v) -> RoaringBitmap` |
| `DurationMin(secs)` | `duration_index` | `gte(&secs) -> RoaringBitmap` |
| `DurationMax(secs)` | `duration_index` | `lte(&secs) -> RoaringBitmap` |
| `CreatedAfter(ns)` | `created_at_index` | `gt(&ns) -> RoaringBitmap` |
| `CreatedBefore(ns)` | `created_at_index` | `lt(&ns) -> RoaringBitmap` |

When a leaf lookup returns `None` (no entities match), an empty `RoaringBitmap` is used. This ensures AND short-circuit works: `empty & X = empty`.

**AND evaluation with short-circuit:**

```rust
fn evaluate_and(&self, children: &[FilterExpr]) -> RoaringBitmap {
    if children.is_empty() {
        return self.universe.clone();
    }

    // Sort children by estimated selectivity (ascending) for early termination
    let mut ordered: Vec<_> = children.iter()
        .map(|c| (self.selectivity(c), c))
        .collect();
    ordered.sort_by(|a, b| a.0.partial_cmp(&b.0).unwrap_or(std::cmp::Ordering::Equal));

    let mut result = self.evaluate(&ordered[0].1).into_bitmap();
    for &(_, child) in &ordered[1..] {
        // Short-circuit: empty ∩ X = empty regardless of remaining children,
        // including NOT sub-expressions. AND identity: ∅ & X = ∅.
        if result.is_empty() {
            return FilterResult::Bitmap(result);
        }
        let child_bitmap = self.evaluate(child).into_bitmap();
        result &= &child_bitmap;
    }
    result
}
```

**NOT evaluation:**

```rust
fn evaluate_not(&self, inner: &FilterExpr) -> RoaringBitmap {
    let inner_bitmap = self.evaluate(inner).into_bitmap();
    let mut complement = self.universe.clone();
    complement -= &inner_bitmap;
    complement
}
```

The NOT operation requires the "universe" bitmap (all valid entity IDs). This is maintained by the entity store and passed to the `FilterEvaluator` at construction time. Without it, complement is undefined.

**Selectivity estimation without bitmap materialization:**

```rust
fn selectivity(&self, expr: &FilterExpr) -> f64 {
    let total = self.universe.len() as f64;
    if total == 0.0 {
        return 0.0;
    }

    match expr {
        FilterExpr::CategoryEq(v) =>
            self.category_index.cardinality(v) as f64 / total,
        FilterExpr::FormatEq(v) =>
            self.format_index.cardinality(v) as f64 / total,
        FilterExpr::CreatorEq(id) =>
            self.creator_index.cardinality(&id.to_string()) as f64 / total,
        FilterExpr::Tag(v) =>
            self.tag_index.cardinality(v) as f64 / total,
        FilterExpr::DurationMin(secs) =>
            self.duration_index.selectivity(
                Bound::Included(secs), Bound::Unbounded,
                self.universe.len(),
            ),
        FilterExpr::DurationMax(secs) =>
            self.duration_index.selectivity(
                Bound::Unbounded, Bound::Included(secs),
                self.universe.len(),
            ),
        FilterExpr::CreatedAfter(ns) =>
            self.created_at_index.selectivity(
                Bound::Excluded(ns), Bound::Unbounded,
                self.universe.len(),
            ),
        FilterExpr::CreatedBefore(ns) =>
            self.created_at_index.selectivity(
                Bound::Unbounded, Bound::Excluded(ns),
                self.universe.len(),
            ),
        FilterExpr::And(children) => {
            // Independence assumption: P(A AND B) = P(A) * P(B)
            children.iter()
                .map(|c| self.selectivity(c))
                .product::<f64>()
                .clamp(0.0, 1.0)
        }
        FilterExpr::Or(children) => {
            // Inclusion-exclusion approximation:
            // P(A OR B) = 1 - (1 - P(A)) * (1 - P(B))
            let complement_product: f64 = children.iter()
                .map(|c| 1.0 - self.selectivity(c))
                .product();
            (1.0 - complement_product).clamp(0.0, 1.0)
        }
        FilterExpr::Not(inner) => {
            (1.0 - self.selectivity(inner)).clamp(0.0, 1.0)
        }
    }
}
```

The independence assumption (AND selectivity = product of individual selectivities) is known to be inaccurate for correlated filters (e.g., `category:jazz AND format:audio`). Spec 07 Section 3 acknowledges this and proposes a correlation cache for frequently co-occurring filter pairs. This cache is deferred to M5+. For M2, the independence assumption is sufficient because the query planner's strategy selection has wide threshold bands (1%, 20%) that tolerate estimation error.

**`FilterResult::into_predicate()`:**

```rust
impl FilterResult {
    pub fn into_predicate(self) -> Box<dyn Fn(u64) -> bool + Send + Sync> {
        match self {
            FilterResult::Bitmap(bitmap) => {
                Box::new(move |id: u64| {
                    debug_assert!(id <= u64::from(u32::MAX), "EntityId out of u32 range");
                    bitmap.contains(id as u32)
                })
            }
            FilterResult::Predicate(f) => f,
        }
    }
}
```

The predicate captures the `RoaringBitmap` by move. `RoaringBitmap` is `Send + Sync`, so the closure is `Send + Sync`. The bitmap containment check (`bitmap.contains(id)`) is O(1) amortized via the roaring bitmap's hierarchical container structure. Performance: ~10-50 nanoseconds per check, well under the 1 microsecond target.

### Error Handling

- `evaluate()` is infallible. Missing index values produce empty bitmaps.
- `selectivity()` is infallible. Returns 0.0 for empty universe or missing values.
- `into_bitmap()` and `into_predicate()` are infallible.

## Test Strategy

### Property Tests

```rust
use proptest::prelude::*;
use roaring::RoaringBitmap;

// AND of two filters equals bitmap intersection.
proptest! {
    #[test]
    fn and_equals_intersection(
        a_ids in prop::collection::hash_set(0u32..10_000, 1..200),
        b_ids in prop::collection::hash_set(0u32..10_000, 1..200),
    ) {
        let cat_index = BitmapIndex::new("category");
        let fmt_index = BitmapIndex::new("format");
        for &id in &a_ids { cat_index.insert(id, "jazz"); }
        for &id in &b_ids { fmt_index.insert(id, "video"); }

        // Build universe
        let mut universe = RoaringBitmap::new();
        for &id in a_ids.iter().chain(b_ids.iter()) { universe.insert(id); }

        let evaluator = build_evaluator(&cat_index, &fmt_index, &universe);
        let result = evaluator.evaluate(&FilterExpr::And(vec![
            FilterExpr::CategoryEq("jazz".into()),
            FilterExpr::FormatEq("video".into()),
        ]));

        let expected = {
            let a: RoaringBitmap = a_ids.iter().copied().collect();
            let b: RoaringBitmap = b_ids.iter().copied().collect();
            a & b
        };
        prop_assert_eq!(result.into_bitmap(), expected);
    }
}

// OR of two filters equals bitmap union.
proptest! {
    #[test]
    fn or_equals_union(
        a_ids in prop::collection::hash_set(0u32..10_000, 1..200),
        b_ids in prop::collection::hash_set(0u32..10_000, 1..200),
    ) {
        let cat_index = BitmapIndex::new("category");
        for &id in &a_ids { cat_index.insert(id, "jazz"); }
        for &id in &b_ids { cat_index.insert(id, "blues"); }

        let mut universe = RoaringBitmap::new();
        for &id in a_ids.iter().chain(b_ids.iter()) { universe.insert(id); }

        let evaluator = build_evaluator_simple(&cat_index, &universe);
        let result = evaluator.evaluate(&FilterExpr::Or(vec![
            FilterExpr::CategoryEq("jazz".into()),
            FilterExpr::CategoryEq("blues".into()),
        ]));

        let expected = {
            let a: RoaringBitmap = a_ids.iter().copied().collect();
            let b: RoaringBitmap = b_ids.iter().copied().collect();
            a | b
        };
        prop_assert_eq!(result.into_bitmap(), expected);
    }
}

// NOT produces the complement.
proptest! {
    #[test]
    fn not_is_complement(
        all_ids in prop::collection::hash_set(0u32..10_000, 50..500),
        match_ids in prop::collection::hash_set(0u32..10_000, 1..200),
    ) {
        let cat_index = BitmapIndex::new("category");
        for &id in &match_ids { cat_index.insert(id, "jazz"); }

        let universe: RoaringBitmap = all_ids.iter().copied().collect();
        let evaluator = build_evaluator_simple(&cat_index, &universe);

        let result = evaluator.evaluate(&FilterExpr::Not(
            Box::new(FilterExpr::CategoryEq("jazz".into()))
        ));
        let result_bitmap = result.into_bitmap();

        // The result should be universe \ jazz
        let jazz_bitmap: RoaringBitmap = match_ids.iter()
            .filter(|id| all_ids.contains(id))
            .copied()
            .collect();
        let expected = &universe - &jazz_bitmap;

        prop_assert_eq!(result_bitmap, expected);
    }
}

// Selectivity is in [0.0, 1.0] for any expression.
proptest! {
    #[test]
    fn selectivity_in_unit_range(
        n_items in 1u32..10_000,
        n_jazz in 0u32..5_000,
        n_video in 0u32..5_000,
    ) {
        let cat_index = BitmapIndex::new("category");
        let fmt_index = BitmapIndex::new("format");
        let n_jazz = n_jazz.min(n_items);
        let n_video = n_video.min(n_items);
        for id in 0..n_jazz { cat_index.insert(id, "jazz"); }
        for id in 0..n_video { fmt_index.insert(id, "video"); }

        let universe: RoaringBitmap = (0..n_items).collect();
        let evaluator = build_evaluator(&cat_index, &fmt_index, &universe);

        // Test various expressions
        let exprs = vec![
            FilterExpr::CategoryEq("jazz".into()),
            FilterExpr::FormatEq("video".into()),
            FilterExpr::And(vec![
                FilterExpr::CategoryEq("jazz".into()),
                FilterExpr::FormatEq("video".into()),
            ]),
            FilterExpr::Or(vec![
                FilterExpr::CategoryEq("jazz".into()),
                FilterExpr::FormatEq("video".into()),
            ]),
            FilterExpr::Not(Box::new(FilterExpr::CategoryEq("jazz".into()))),
        ];

        for expr in &exprs {
            let sel = evaluator.selectivity(expr);
            prop_assert!(sel >= 0.0 && sel <= 1.0,
                "selectivity {sel} out of range for {expr:?}");
        }
    }
}

// Predicate closure agrees with bitmap containment.
proptest! {
    #[test]
    fn predicate_matches_bitmap(
        ids in prop::collection::hash_set(0u32..10_000, 1..200),
        test_ids in prop::collection::vec(0u32..10_000, 1..100),
    ) {
        let cat_index = BitmapIndex::new("category");
        for &id in &ids { cat_index.insert(id, "jazz"); }

        let universe: RoaringBitmap = (0..10_000u32).collect();
        let evaluator = build_evaluator_simple(&cat_index, &universe);
        let result = evaluator.evaluate(&FilterExpr::CategoryEq("jazz".into()));
        let bitmap = result.into_bitmap();
        let bitmap_clone = bitmap.clone();
        let predicate = FilterResult::Bitmap(bitmap).into_predicate();

        for &test_id in &test_ids {
            prop_assert_eq!(
                predicate(test_id),
                bitmap_clone.contains(test_id),
                "predicate disagreed with bitmap for id {test_id}"
            );
        }
    }
}
```

### Unit Tests

```rust
#[test]
fn evaluate_single_category() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::CategoryEq("jazz".into()));
    let bitmap = result.into_bitmap();
    assert!(!bitmap.is_empty());
    // All IDs in the bitmap should be jazz items
}

#[test]
fn evaluate_and_two_filters() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
    ]));
    let bitmap = result.into_bitmap();
    // Should be the intersection of jazz items and video items
}

#[test]
fn evaluate_or_two_categories() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::Or(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::CategoryEq("blues".into()),
    ]));
    let bitmap = result.into_bitmap();
    // Should be the union of jazz and blues items
}

#[test]
fn evaluate_not_creator() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::Not(
        Box::new(FilterExpr::CreatorEq(77))
    ));
    let bitmap = result.into_bitmap();
    // Should exclude all items by creator 77
    assert!(!bitmap.contains(77)); // assuming creator 77's items use creator_id as entity_id for test simplicity
}

#[test]
fn evaluate_duration_min() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::DurationMin(300)); // >= 5 min
    let bitmap = result.into_bitmap();
    // Should include items with duration >= 300 seconds
}

#[test]
fn evaluate_created_after() {
    let (evaluator, _indexes) = setup_test_indexes();
    let seven_days_ago = now_ns() - 7 * 86_400_000_000_000;
    let result = evaluator.evaluate(&FilterExpr::CreatedAfter(seven_days_ago));
    let bitmap = result.into_bitmap();
    // Should include items created within the last 7 days
}

#[test]
fn evaluate_complex_compound() {
    // category:jazz AND format:video AND duration_min:5m AND created_within:7d
    let (evaluator, _indexes) = setup_test_indexes();
    let seven_days_ago = now_ns() - 7 * 86_400_000_000_000;
    let result = evaluator.evaluate(&FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
        FilterExpr::DurationMin(300),
        FilterExpr::CreatedAfter(seven_days_ago),
    ]));
    let bitmap = result.into_bitmap();
    // Result should be the intersection of all four conditions
}

#[test]
fn evaluate_nonexistent_category_short_circuits() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::And(vec![
        FilterExpr::CategoryEq("nonexistent".into()),
        FilterExpr::FormatEq("video".into()),
    ]));
    assert!(result.is_empty());
}

#[test]
fn selectivity_single_category() {
    let (evaluator, _indexes) = setup_test_indexes_with_known_counts();
    // If 100 of 1000 items are jazz, selectivity = 0.1
    let sel = evaluator.selectivity(&FilterExpr::CategoryEq("jazz".into()));
    assert!((sel - 0.1).abs() < 0.01);
}

#[test]
fn selectivity_and_independence() {
    let (evaluator, _indexes) = setup_test_indexes_with_known_counts();
    // jazz=10%, video=20%, AND selectivity = 10% * 20% = 2%
    let sel = evaluator.selectivity(&FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
    ]));
    assert!((sel - 0.02).abs() < 0.01);
}

#[test]
fn selectivity_or_inclusion_exclusion() {
    let (evaluator, _indexes) = setup_test_indexes_with_known_counts();
    // jazz=10%, blues=15%, OR selectivity = 1 - (0.9 * 0.85) = 0.235
    let sel = evaluator.selectivity(&FilterExpr::Or(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::CategoryEq("blues".into()),
    ]));
    assert!((sel - 0.235).abs() < 0.02);
}

#[test]
fn selectivity_not() {
    let (evaluator, _indexes) = setup_test_indexes_with_known_counts();
    // jazz=10%, NOT jazz = 90%
    let sel = evaluator.selectivity(&FilterExpr::Not(
        Box::new(FilterExpr::CategoryEq("jazz".into()))
    ));
    assert!((sel - 0.9).abs() < 0.01);
}

#[test]
fn selectivity_empty_universe() {
    // With no entities, selectivity is always 0.0
    let universe = RoaringBitmap::new();
    let cat_index = BitmapIndex::new("category");
    let evaluator = build_evaluator_with_empty(&cat_index, &universe);
    let sel = evaluator.selectivity(&FilterExpr::CategoryEq("jazz".into()));
    assert!((sel - 0.0).abs() < f64::EPSILON);
}

#[test]
fn into_predicate_checks_bitmap_containment() {
    let mut bitmap = RoaringBitmap::new();
    bitmap.insert(1);
    bitmap.insert(42);
    bitmap.insert(999);

    let result = FilterResult::Bitmap(bitmap);
    let predicate = result.into_predicate();

    assert!(predicate(1));
    assert!(predicate(42));
    assert!(predicate(999));
    assert!(!predicate(0));
    assert!(!predicate(2));
    assert!(!predicate(1000));
}

#[test]
fn empty_and_returns_universe() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::And(vec![]));
    // AND of nothing = everything (identity element for intersection)
    let bitmap = result.into_bitmap();
    assert!(!bitmap.is_empty());
}

#[test]
fn empty_or_returns_empty() {
    let (evaluator, _indexes) = setup_test_indexes();
    let result = evaluator.evaluate(&FilterExpr::Or(vec![]));
    // OR of nothing = nothing (identity element for union)
    assert!(result.is_empty());
}
```

### Benchmarks

```rust
// === tidal/benches/filters.rs ===

use criterion::{black_box, criterion_group, criterion_main, Criterion};
use roaring::RoaringBitmap;

fn bench_filter_single_category(c: &mut Criterion) {
    // Setup: 10K items, 10 categories (~1K items per category)
    let (evaluator, _indexes) = setup_benchmark_indexes(10_000);
    let expr = FilterExpr::CategoryEq("jazz".into());

    c.bench_function("filter_single_category", |b| {
        b.iter(|| {
            let result = evaluator.evaluate(black_box(&expr));
            black_box(result.cardinality());
        });
    });
}

fn bench_filter_compound_and(c: &mut Criterion) {
    // Setup: 10K items, category:jazz AND format:video AND duration_min:300
    let (evaluator, _indexes) = setup_benchmark_indexes(10_000);
    let expr = FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
        FilterExpr::DurationMin(300),
    ]);

    c.bench_function("filter_compound_and_3", |b| {
        b.iter(|| {
            let result = evaluator.evaluate(black_box(&expr));
            black_box(result.cardinality());
        });
    });
}

fn bench_filter_predicate_per_candidate(c: &mut Criterion) {
    // Setup: build a filter result, then check 10K candidate IDs
    let (evaluator, _indexes) = setup_benchmark_indexes(10_000);
    let expr = FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
    ]);
    let result = evaluator.evaluate(&expr);
    let predicate = result.into_predicate();

    c.bench_function("filter_predicate_per_candidate_10k", |b| {
        b.iter(|| {
            for id in 0..10_000u32 {
                black_box(predicate(black_box(id)));
            }
        });
    });
}

fn bench_selectivity_estimation(c: &mut Criterion) {
    // Setup: 10K items
    let (evaluator, _indexes) = setup_benchmark_indexes(10_000);
    let expr = FilterExpr::And(vec![
        FilterExpr::CategoryEq("jazz".into()),
        FilterExpr::FormatEq("video".into()),
        FilterExpr::DurationMin(300),
    ]);

    c.bench_function("selectivity_estimation_compound", |b| {
        b.iter(|| {
            black_box(evaluator.selectivity(black_box(&expr)));
        });
    });
}

criterion_group!(
    benches,
    bench_filter_single_category,
    bench_filter_compound_and,
    bench_filter_predicate_per_candidate,
    bench_selectivity_estimation,
);
criterion_main!(benches);
```

## Acceptance Criteria

- [ ] `FilterExpr` enum covers: `CategoryEq`, `FormatEq`, `CreatorEq`, `Tag`, `DurationMin`, `DurationMax`, `CreatedAfter`, `CreatedBefore`, `And`, `Or`, `Not`
- [ ] `FilterEvaluator` resolves each leaf `FilterExpr` variant against the correct index (`BitmapIndex` or `RangeIndex`)
- [ ] `evaluate()` produces correct results for single-leaf, AND, OR, NOT, and nested compound expressions (property tested)
- [ ] AND evaluation uses bitmap intersection; empty AND (no children) returns the universe
- [ ] OR evaluation uses bitmap union; empty OR (no children) returns empty
- [ ] NOT evaluation uses bitmap complement against the universe
- [ ] AND evaluation sorts children by ascending selectivity and short-circuits on empty intersection (Spec 08 Section 7.4)
- [ ] `selectivity()` returns a value in [0.0, 1.0] for all inputs (property tested)
- [ ] AND selectivity uses product of individual selectivities (independence assumption)
- [ ] OR selectivity uses `1 - product(1 - s_i)` (inclusion-exclusion approximation)
- [ ] NOT selectivity = `1 - inner_selectivity`
- [ ] `FilterResult::into_bitmap()` extracts the bitmap
- [ ] `FilterResult::into_predicate()` returns a closure that checks bitmap containment; closure is `Send + Sync`
- [ ] Predicate per-candidate evaluation < 1 microsecond (benchmarked; target ~10-50 nanoseconds via roaring containment check)
- [ ] Criterion benchmarks pass: `bench_filter_single_category`, `bench_filter_compound_and`, `bench_filter_predicate_per_candidate`, `bench_selectivity_estimation`
- [ ] `cargo clippy -- -D warnings` passes
- [ ] All property tests and unit tests pass

## Research References

- [docs/research/ann_for_tidaldb.md](../../../research/ann_for_tidaldb.md) -- "estimate selectivity via metadata indexes" -> select ANN strategy; "resolve filter predicates to roaring bitmaps" -> pre-filter for brute-force; predicate callback for in-graph filtering
- [docs/specs/07-vector-retrieval.md](../../../specs/07-vector-retrieval.md) -- Section 3 (selectivity estimation: independence assumption for AND, correlation cache deferred; keyword equality from bitmap cardinality; numeric range from sorted index), Section 9 (adaptive query planner decision tree: threshold reference table)

## Spec References

- [docs/specs/08-query-engine.md](../../../specs/08-query-engine.md) -- Section 7.1 (bitmap-based filter architecture), Section 7.2 (filter push-down: bitmap passed to `VectorIndex::filtered_search()` as predicate callback), Section 7.3 (`Filter` enum with all variant types), Section 7.4 (short-circuit evaluation: sort by ascending cardinality, abort on empty intersection), Section 5.3 (Stage 2: filter evaluation reduces candidate set)
- [docs/specs/09-ranking-scoring.md](../../../specs/09-ranking-scoring.md) -- Section 3.1 (filter interaction with ANN: "pre-filters are applied as predicate callbacks during HNSW traversal when selectivity is 2-100%"), Section 4 Stage 3 (filter evaluation: "AND-composed across dimensions; OR-composed within a dimension")

## Implementation Notes

- `FilterExpr` implements `Clone` and `PartialEq` for testing. It does NOT need to be `Serialize`/`Deserialize` -- it exists only as an in-memory AST between the parser and the evaluator.
- The `FilterEvaluator` borrows all indexes by reference (`&'a`). It is constructed per-query and does not outlive the query execution. This avoids cloning indexes.
- `FilterResult` only has a `Bitmap` variant in M2. A future `Predicate` variant (wrapping an arbitrary closure without materializing a bitmap) is deferred because at M2 scale (10K items), bitmap materialization is always cheap. At M7 scale, if filter evaluation on 10M items is too slow to materialize as a bitmap, add a `Predicate` variant that evaluates lazily.
- The `into_predicate()` method moves the bitmap into the closure. This means the `FilterResult` is consumed. If both the bitmap and the predicate are needed, clone the bitmap first.
- Add `roaring = "0.10"` to `[dependencies]` in `tidal/Cargo.toml` (may already be added by Task 01).
- Add `[[bench]] name = "filters" harness = false` to `tidal/Cargo.toml`.
- Do NOT implement user-state filters (`Unseen`, `NotBlocked`, `Relationship`, `SocialGraph`, `InCollection`) in this task. These require user entities (M3). The `FilterExpr` enum deliberately omits them. They are added in m3p4 (User State Filters).
- Do NOT implement the correlation cache for co-occurring filter pairs. The independence assumption is sufficient for M2's selectivity estimation.
- Do NOT implement filter push-down into the ANN search in this task. The filter engine produces the bitmap/predicate; the RETRIEVE executor (m2p5) is responsible for passing it to the vector index.
