tidaldb/docs/planning/milestone-2/phase-1/task-04-adaptive-query-planner.md

Task 04: Adaptive Query Planner + Benchmarks

Context

Milestone: 2 -- Ranked Retrieval Phase: m2p1 -- Vector Index Integration (USearch) Depends On: Task 01 (VectorIndex trait, BruteForceIndex, types), Task 02 (UsearchIndex for benchmarking) Blocks: m2p5 (RETRIEVE executor calls the planner to select ANN strategy) Complexity: M

Objective

Deliver the AdaptiveQueryPlanner that evaluates filter selectivity before each ANN query and routes to the optimal strategy. The planner eliminates the single most common failure mode in filtered vector search: using HNSW in-graph filtering on extremely selective predicates (< 1% matching) where recall collapses, or using brute-force on broad predicates (> 20% matching) where linear scan is too slow.

The planner implements the decision tree from Spec 07, Section 9:

• No filter (100%): Standard HNSW search() -- fastest path, highest recall.
• Broad filter (> 20%): In-graph predicate filter via filtered_search() -- predicate evaluated during graph traversal, non-matching nodes used for navigation.
• Danger zone (1-20%): filtered_search() with widened ef_search (2-3x normal) -- ACORN-1 approximation to maintain recall under moderate selectivity.

ef_search concurrency caveat: Changing ef_search per query via index.change_expansion_search(ef) mutates global USearch state. For concurrent queries using different strategies, this requires a mutex around the (change_expansion_search, search) sequence. For M2, accept this limitation and document it. The AdaptiveQueryPlanner should take a Mutex<Box<dyn VectorIndex>> or wrap per-query ef_search changes in a lock. Alternatively, set ef_search conservatively high at construction time (e.g., 400) and skip per-query override for M2. Defer true per-query ef_search to M7 after benchmarking.

• Very selective (< 1%): Pre-filter to bitmap, then brute-force L2 scan over the small matched set -- exact results, fast on small sets.

This task also delivers the Criterion benchmarks for the entire vector subsystem, establishing the baseline performance measurements that all future milestones track.

For M2, the SelectivityEstimator is a placeholder that accepts an externally provided selectivity value. The real estimator (reading metadata bitmap cardinalities) is wired up when m2p2 (Metadata Indexes and Filter Engine) is implemented. This decoupling allows the planner to be tested and benchmarked independently.

Requirements

• AnnStrategy enum: Unfiltered, InGraphFilter, WidenedFilter, PreFilterBruteForce
• AdaptiveQueryPlanner selects strategy based on estimated selectivity
• Selectivity thresholds: < 1% brute-force, 1-20% widened filter, > 20% standard filter, 100% unfiltered
• ef_search widening: 2x for 5-20% selectivity, 3x for 1-5% selectivity
• SelectivityEstimator trait with a placeholder implementation returning caller-provided values
• AnnQueryStats struct for per-query observability: estimated selectivity, actual selectivity, strategy, latency, results count
• PlannerConfig for threshold tuning: in_graph_min_selectivity, brute_force_max_selectivity, ef_search_multiplier_moderate, ef_search_multiplier_low
• Criterion benchmarks: unfiltered search, filtered search at 20% and 5% selectivity, brute-force search, recall@100
• No unsafe code

Technical Design

Module Structure

tidal/src/storage/vector/
  planner.rs  -- AdaptiveQueryPlanner, SelectivityEstimator, AnnQueryStats, PlannerConfig, AnnStrategy

tidal/benches/
  vector.rs   -- Criterion benchmarks

Public API

// === storage/vector/planner.rs ===

use std::time::{Duration, Instant};
use super::{VectorIndex, VectorId, VectorSearchResult, VectorError, VectorIndexConfig};

/// The ANN strategy selected by the query planner.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AnnStrategy {
    /// No filter active. Standard HNSW search.
    Unfiltered,
    /// Filter selectivity > 20%. Standard in-graph predicate filter.
    InGraphFilter,
    /// Filter selectivity 1-20%. In-graph filter with widened ef_search.
    WidenedFilter {
        /// The widened ef_search value (2-3x normal).
        ef_search: usize,
    },
    /// Filter selectivity < 1%. Pre-filter to candidate set, then brute-force.
    PreFilterBruteForce,
}

/// Configuration for the adaptive query planner's selectivity thresholds.
///
/// These thresholds determine which ANN strategy is selected based on
/// estimated filter selectivity. They can be tuned based on runtime
/// statistics from `AnnQueryStats`.
#[derive(Debug, Clone)]
pub struct PlannerConfig {
    /// Minimum selectivity for standard in-graph filtering.
    /// Below this, use widened filter or brute-force.
    /// Default: 0.20 (20%). Range: [0.05, 0.50].
    pub in_graph_min_selectivity: f64,

    /// Maximum selectivity for pre-filter + brute-force.
    /// Above this, use widened filter instead.
    /// Default: 0.01 (1%). Range: [0.001, 0.05].
    pub brute_force_max_selectivity: f64,

    /// ef_search multiplier for moderate selectivity (5-20%).
    /// Default: 2.0.
    pub ef_search_multiplier_moderate: f64,

    /// ef_search multiplier for low selectivity (1-5%).
    /// Default: 3.0.
    pub ef_search_multiplier_low: f64,

    /// Default ef_search when no override is specified.
    /// Default: 200.
    pub default_ef_search: usize,
}

impl Default for PlannerConfig {
    fn default() -> Self {
        Self {
            in_graph_min_selectivity: 0.20,
            brute_force_max_selectivity: 0.01,
            ef_search_multiplier_moderate: 2.0,
            ef_search_multiplier_low: 3.0,
            default_ef_search: 200,
        }
    }
}

/// Trait for selectivity estimation.
///
/// The real implementation (m2p2) reads metadata bitmap cardinalities.
/// For m2p1, a placeholder implementation returns caller-provided values.
pub trait SelectivityEstimator: Send + Sync {
    /// Estimate the fraction of items matching the given filter.
    ///
    /// Returns a value in [0.0, 1.0]:
    /// - 1.0 means no filter (all items match).
    /// - 0.01 means ~1% of items match.
    /// - 0.0 means nothing matches (empty result guaranteed).
    fn estimate_selectivity(&self, filter: &dyn Fn(VectorId) -> bool) -> f64;
}

/// Placeholder estimator that always returns a fixed selectivity.
///
/// Used for m2p1 testing before metadata indexes (m2p2) exist.
/// Callers set the selectivity directly.
pub struct FixedSelectivityEstimator {
    selectivity: f64,
}

impl FixedSelectivityEstimator {
    pub fn new(selectivity: f64) -> Self {
        Self { selectivity: selectivity.clamp(0.0, 1.0) }
    }

    /// Update the fixed selectivity value.
    pub fn set_selectivity(&mut self, selectivity: f64) {
        self.selectivity = selectivity.clamp(0.0, 1.0);
    }
}

impl SelectivityEstimator for FixedSelectivityEstimator {
    fn estimate_selectivity(&self, _filter: &dyn Fn(VectorId) -> bool) -> f64 {
        self.selectivity
    }
}

/// Statistics collected per ANN query for planner observability.
#[derive(Debug, Clone)]
pub struct AnnQueryStats {
    /// Estimated selectivity before execution.
    pub estimated_selectivity: f64,
    /// Strategy selected by the planner.
    pub strategy: AnnStrategy,
    /// Number of results returned.
    pub results_returned: usize,
    /// Requested K.
    pub requested_k: usize,
    /// Wall clock time for the ANN query.
    pub latency: Duration,
}

/// The adaptive query planner for filtered ANN search.
///
/// Evaluates filter selectivity and selects the optimal ANN strategy
/// for each query. Logs the plan at DEBUG level for observability.
///
/// # Strategy Selection
///
/// ```text
/// selectivity = 100% (no filter) -> Unfiltered
/// selectivity > 20%              -> InGraphFilter (standard ef_search)
/// selectivity 5-20%              -> WidenedFilter (2x ef_search)
/// selectivity 1-5%               -> WidenedFilter (3x ef_search)
/// selectivity < 1%               -> PreFilterBruteForce
/// ```
pub struct AdaptiveQueryPlanner {
    config: PlannerConfig,
}

impl AdaptiveQueryPlanner {
    pub fn new(config: PlannerConfig) -> Self {
        Self { config }
    }

    pub fn with_defaults() -> Self {
        Self::new(PlannerConfig::default())
    }

    /// Select the ANN strategy for a query based on estimated selectivity.
    ///
    /// If `selectivity` is 1.0, returns `Unfiltered`.
    /// Otherwise, applies the threshold decision tree.
    pub fn select_strategy(&self, selectivity: f64) -> AnnStrategy {
        if (selectivity - 1.0).abs() < f64::EPSILON || selectivity > 1.0 {
            return AnnStrategy::Unfiltered;
        }
        if selectivity >= self.config.in_graph_min_selectivity {
            return AnnStrategy::InGraphFilter;
        }
        if selectivity >= self.config.brute_force_max_selectivity {
            let multiplier = if selectivity >= 0.05 {
                self.config.ef_search_multiplier_moderate
            } else {
                self.config.ef_search_multiplier_low
            };
            let ef = (self.config.default_ef_search as f64 * multiplier) as usize;
            return AnnStrategy::WidenedFilter { ef_search: ef };
        }
        AnnStrategy::PreFilterBruteForce
    }

    /// Execute an ANN query using the selected strategy.
    ///
    /// This is the top-level entry point called by the RETRIEVE executor.
    /// It estimates selectivity, selects a strategy, executes the search,
    /// and returns results with query statistics.
    ///
    /// # Arguments
    ///
    /// * `index` -- The HNSW index to search.
    /// * `query` -- The query vector (L2-normalized).
    /// * `k` -- Number of results to return.
    /// * `filter` -- Optional filter predicate. If `None`, unfiltered search.
    /// * `selectivity` -- Estimated selectivity (provided by estimator or caller).
    /// * `brute_force_index` -- Optional brute-force index for pre-filter fallback.
    ///   If `None` and strategy is `PreFilterBruteForce`, falls back to `WidenedFilter`.
    pub fn execute(
        &self,
        index: &dyn VectorIndex,
        query: &[f32],
        k: usize,
        filter: Option<&dyn Fn(VectorId) -> bool>,
        selectivity: f64,
        brute_force_index: Option<&dyn VectorIndex>,
    ) -> Result<(Vec<VectorSearchResult>, AnnQueryStats), VectorError> {
        let strategy = match &filter {
            None => AnnStrategy::Unfiltered,
            Some(_) => self.select_strategy(selectivity),
        };

        let start = Instant::now();

        let results = match (&strategy, filter) {
            (AnnStrategy::Unfiltered, _) => {
                index.search(query, k, self.config.default_ef_search)?
            }
            (AnnStrategy::InGraphFilter, Some(f)) => {
                index.filtered_search(query, k, self.config.default_ef_search, f)?
            }
            (AnnStrategy::WidenedFilter { ef_search }, Some(f)) => {
                index.filtered_search(query, k, *ef_search, f)?
            }
            (AnnStrategy::PreFilterBruteForce, Some(f)) => {
                match brute_force_index {
                    Some(bf) => bf.filtered_search(query, k, 0, f)?,
                    None => {
                        // Fallback: use widened filter if no brute-force index
                        let ef = (self.config.default_ef_search as f64
                            * self.config.ef_search_multiplier_low) as usize;
                        index.filtered_search(query, k, ef, f)?
                    }
                }
            }
            _ => {
                // Filter is None but strategy is not Unfiltered -- should not happen.
                // Defensive: run unfiltered.
                index.search(query, k, self.config.default_ef_search)?
            }
        };

        let latency = start.elapsed();

        let stats = AnnQueryStats {
            estimated_selectivity: selectivity,
            strategy,
            results_returned: results.len(),
            requested_k: k,
            latency,
        };

        Ok((results, stats))
    }

    /// Get the current planner configuration.
    pub fn config(&self) -> &PlannerConfig {
        &self.config
    }
}

Error Handling

• All errors propagate from the underlying VectorIndex methods.
• If PreFilterBruteForce is selected but no brute-force index is available, the planner falls back to WidenedFilter with ef_search_multiplier_low. This is logged at WARN level.
• If a filtered search returns fewer than k results (recall underflow), this is captured in AnnQueryStats::results_returned < requested_k. The planner does not automatically retry with a different strategy -- the caller (RETRIEVE executor) decides whether to retry.

Criterion Benchmarks

// === tidal/benches/vector.rs ===

use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId};
use tidaldb::storage::vector::*;
use rand::Rng;

fn random_unit_vector(dim: usize, rng: &mut impl Rng) -> Vec<f32> {
    let v: Vec<f32> = (0..dim).map(|_| rng.gen::<f32>() - 0.5).collect();
    let norm: f32 = v.iter().map(|x| x * x).sum::<f32>().sqrt();
    v.iter().map(|x| x / norm).collect()
}

/// Benchmark: unfiltered ANN search at 10K vectors.
fn bench_ann_search_unfiltered(c: &mut Criterion) {
    let dim = 128; // Use 128d for CI-friendly benchmarks. 1536d for nightly.
    let n = 10_000;
    let k = 100;

    let config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };

    let index = UsearchIndex::new(config).unwrap();
    index.reserve(n * 2).unwrap();

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let query = random_unit_vector(dim, &mut rng);

    c.bench_function("ann_search_unfiltered_10k", |b| {
        b.iter(|| {
            index.search(&query, k, 200).unwrap()
        })
    });
}

/// Benchmark: filtered ANN search at 20% selectivity.
fn bench_ann_search_filtered_20pct(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;
    let k = 100;

    let config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };

    let index = UsearchIndex::new(config).unwrap();
    index.reserve(n * 2).unwrap();

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let query = random_unit_vector(dim, &mut rng);
    // 20% selectivity: IDs divisible by 5
    let predicate = |id: VectorId| id % 5 == 0;

    c.bench_function("ann_search_filtered_20pct_10k", |b| {
        b.iter(|| {
            index.filtered_search(&query, k, 200, &predicate).unwrap()
        })
    });
}

/// Benchmark: filtered ANN search at 5% selectivity (danger zone, widened ef).
fn bench_ann_search_filtered_5pct(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;
    let k = 100;

    let config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };

    let index = UsearchIndex::new(config).unwrap();
    index.reserve(n * 2).unwrap();

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let query = random_unit_vector(dim, &mut rng);
    // 5% selectivity: IDs divisible by 20
    let predicate = |id: VectorId| id % 20 == 0;

    c.bench_function("ann_search_filtered_5pct_10k", |b| {
        b.iter(|| {
            index.filtered_search(&query, k, 400, &predicate).unwrap()
        })
    });
}

/// Benchmark: brute-force search over filtered candidate set.
fn bench_ann_search_brute_force(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;
    let k = 100;

    let config = VectorIndexConfig {
        dimensions: dim,
        ..VectorIndexConfig::default()
    };

    let index = BruteForceIndex::new(config);

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let query = random_unit_vector(dim, &mut rng);
    // 0.5% selectivity: ~50 candidates from 10K
    let predicate = |id: VectorId| id % 200 == 0;

    c.bench_function("ann_brute_force_0_5pct_10k", |b| {
        b.iter(|| {
            index.filtered_search(&query, k, 0, &predicate).unwrap()
        })
    });
}

/// Benchmark: measure recall@100 (not a latency benchmark -- measures quality).
fn bench_ann_recall_at_100(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;
    let k = 100;

    let usearch_config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };
    let brute_config = VectorIndexConfig {
        dimensions: dim,
        ..VectorIndexConfig::default()
    };

    let usearch_index = UsearchIndex::new(usearch_config).unwrap();
    usearch_index.reserve(n * 2).unwrap();
    let brute_index = BruteForceIndex::new(brute_config);

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        usearch_index.insert(id, &v).unwrap();
        brute_index.insert(id, &v).unwrap();
    }

    // Generate 10 queries
    let queries: Vec<Vec<f32>> = (0..10)
        .map(|_| random_unit_vector(dim, &mut rng))
        .collect();

    c.bench_function("ann_recall_at_100_10k", |b| {
        b.iter(|| {
            let mut total_recall = 0.0;
            for query in &queries {
                let exact = brute_index.search(query, k, 0).unwrap();
                let approx = usearch_index.search(query, k, 0).unwrap();
                let exact_ids: std::collections::HashSet<u64> =
                    exact.iter().map(|r| r.id).collect();
                let approx_ids: std::collections::HashSet<u64> =
                    approx.iter().map(|r| r.id).collect();
                total_recall += exact_ids.intersection(&approx_ids).count() as f64 / k as f64;
            }
            total_recall / queries.len() as f64
        })
    });
}

/// Benchmark: insert one f16 vector into a 10K-vector index, pre-reserved capacity.
fn bench_ann_insert_single(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;

    let config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };

    let index = UsearchIndex::new(config).unwrap();
    index.reserve(n * 2).unwrap();

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let mut next_id = n as u64;

    c.bench_function("ann_insert_single_10k", |b| {
        b.iter(|| {
            let v = random_unit_vector(dim, &mut rng);
            index.insert(next_id, &v).unwrap();
            next_id += 1;
        })
    });
}

/// Benchmark: tombstone-delete one vector from a 10K-vector index.
fn bench_ann_delete_single(c: &mut Criterion) {
    let dim = 128;
    let n = 10_000;

    let config = VectorIndexConfig {
        dimensions: dim,
        quantization: QuantizationLevel::F16,
        ..VectorIndexConfig::default()
    };

    let index = UsearchIndex::new(config).unwrap();
    index.reserve(n * 2).unwrap();

    let mut rng = rand::thread_rng();
    for id in 0..n as u64 {
        let v = random_unit_vector(dim, &mut rng);
        index.insert(id, &v).unwrap();
    }

    let mut delete_id = 0u64;

    c.bench_function("ann_delete_single_10k", |b| {
        b.iter(|| {
            // Delete and re-insert to keep the bench iterable
            let _ = index.delete(delete_id);
            let v = random_unit_vector(dim, &mut rng);
            index.insert(delete_id, &v).unwrap();
            delete_id = (delete_id + 1) % n as u64;
        })
    });
}

criterion_group!(
    benches,
    bench_ann_search_unfiltered,
    bench_ann_search_filtered_20pct,
    bench_ann_search_filtered_5pct,
    bench_ann_search_brute_force,
    bench_ann_recall_at_100,
    bench_ann_insert_single,
    bench_ann_delete_single,
);
criterion_main!(benches);

Test Strategy

Unit Tests

#[test]
fn strategy_unfiltered_at_100pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    assert_eq!(planner.select_strategy(1.0), AnnStrategy::Unfiltered);
}

#[test]
fn strategy_in_graph_above_20pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    assert_eq!(planner.select_strategy(0.50), AnnStrategy::InGraphFilter);
    assert_eq!(planner.select_strategy(0.25), AnnStrategy::InGraphFilter);
    assert_eq!(planner.select_strategy(0.20), AnnStrategy::InGraphFilter);
}

#[test]
fn strategy_widened_moderate_5_to_20pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    let strategy = planner.select_strategy(0.10);
    match strategy {
        AnnStrategy::WidenedFilter { ef_search } => {
            // 10% is in the moderate range (5-20%), so 2x multiplier
            assert_eq!(ef_search, 400, "ef_search should be 2x default (200*2=400)");
        }
        _ => panic!("expected WidenedFilter, got {strategy:?}"),
    }
}

#[test]
fn strategy_widened_low_1_to_5pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    let strategy = planner.select_strategy(0.03);
    match strategy {
        AnnStrategy::WidenedFilter { ef_search } => {
            // 3% is in the low range (1-5%), so 3x multiplier
            assert_eq!(ef_search, 600, "ef_search should be 3x default (200*3=600)");
        }
        _ => panic!("expected WidenedFilter, got {strategy:?}"),
    }
}

#[test]
fn strategy_brute_force_below_1pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    assert_eq!(planner.select_strategy(0.005), AnnStrategy::PreFilterBruteForce);
    assert_eq!(planner.select_strategy(0.001), AnnStrategy::PreFilterBruteForce);
    assert_eq!(planner.select_strategy(0.0), AnnStrategy::PreFilterBruteForce);
}

#[test]
fn strategy_boundary_at_20pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    // Exactly at 20%: in-graph filter
    assert_eq!(planner.select_strategy(0.20), AnnStrategy::InGraphFilter);
    // Just below 20%: widened filter
    let strategy = planner.select_strategy(0.19);
    assert!(matches!(strategy, AnnStrategy::WidenedFilter { .. }));
}

#[test]
fn strategy_boundary_at_1pct() {
    let planner = AdaptiveQueryPlanner::with_defaults();
    // Exactly at 1%: widened filter
    let strategy = planner.select_strategy(0.01);
    assert!(matches!(strategy, AnnStrategy::WidenedFilter { .. }));
    // Just below 1%: brute-force
    assert_eq!(planner.select_strategy(0.009), AnnStrategy::PreFilterBruteForce);
}

#[test]
fn custom_thresholds() {
    let config = PlannerConfig {
        in_graph_min_selectivity: 0.30,
        brute_force_max_selectivity: 0.02,
        ef_search_multiplier_moderate: 2.5,
        ef_search_multiplier_low: 4.0,
        default_ef_search: 100,
    };
    let planner = AdaptiveQueryPlanner::new(config);

    // 25%: below 30% threshold, widened filter
    let strategy = planner.select_strategy(0.25);
    assert!(matches!(strategy, AnnStrategy::WidenedFilter { .. }));

    // 35%: above 30% threshold, in-graph
    assert_eq!(planner.select_strategy(0.35), AnnStrategy::InGraphFilter);

    // 1.5%: below 2% threshold, brute-force
    assert_eq!(planner.select_strategy(0.015), AnnStrategy::PreFilterBruteForce);
}

#[test]
fn execute_unfiltered() {
    let config = VectorIndexConfig { dimensions: 3, ..VectorIndexConfig::default() };
    let index = BruteForceIndex::new(config);
    index.insert(1, &[1.0, 0.0, 0.0]).unwrap();
    index.insert(2, &[0.0, 1.0, 0.0]).unwrap();

    let planner = AdaptiveQueryPlanner::with_defaults();
    let (results, stats) = planner.execute(
        &index, &[1.0, 0.0, 0.0], 2, None, 1.0, None,
    ).unwrap();

    assert_eq!(results.len(), 2);
    assert_eq!(stats.strategy, AnnStrategy::Unfiltered);
    assert_eq!(stats.results_returned, 2);
    assert_eq!(stats.requested_k, 2);
}

#[test]
fn execute_filtered_in_graph() {
    let config = VectorIndexConfig { dimensions: 3, ..VectorIndexConfig::default() };
    let index = BruteForceIndex::new(config);
    for id in 0..10u64 {
        index.insert(id, &[1.0, 0.0, 0.0]).unwrap();
    }

    let planner = AdaptiveQueryPlanner::with_defaults();
    let filter = |id: VectorId| id % 2 == 0;
    let (results, stats) = planner.execute(
        &index, &[1.0, 0.0, 0.0], 5, Some(&filter), 0.50, None,
    ).unwrap();

    assert!(results.iter().all(|r| r.id % 2 == 0));
    assert_eq!(stats.strategy, AnnStrategy::InGraphFilter);
}

#[test]
fn execute_brute_force_fallback_without_brute_index() {
    let config = VectorIndexConfig { dimensions: 3, ..VectorIndexConfig::default() };
    let index = BruteForceIndex::new(config);
    for id in 0..10u64 {
        index.insert(id, &[1.0, 0.0, 0.0]).unwrap();
    }

    let planner = AdaptiveQueryPlanner::with_defaults();
    let filter = |id: VectorId| id == 0;
    // Selectivity 0.005 triggers PreFilterBruteForce, but no brute index provided
    // Should fall back to WidenedFilter
    let (results, stats) = planner.execute(
        &index, &[1.0, 0.0, 0.0], 1, Some(&filter), 0.005, None,
    ).unwrap();

    // Should still return results (fallback works)
    assert!(!results.is_empty());
}

#[test]
fn execute_brute_force_with_brute_index() {
    let config = VectorIndexConfig { dimensions: 3, ..VectorIndexConfig::default() };
    let hnsw_index = BruteForceIndex::new(config.clone());
    let brute_index = BruteForceIndex::new(config);
    for id in 0..100u64 {
        let v = [1.0, 0.0, 0.0];
        hnsw_index.insert(id, &v).unwrap();
        brute_index.insert(id, &v).unwrap();
    }

    let planner = AdaptiveQueryPlanner::with_defaults();
    let filter = |id: VectorId| id < 1; // 1% selectivity
    let (results, stats) = planner.execute(
        &hnsw_index, &[1.0, 0.0, 0.0], 1, Some(&filter), 0.005, Some(&brute_index),
    ).unwrap();

    assert_eq!(stats.strategy, AnnStrategy::PreFilterBruteForce);
    assert!(results.iter().all(|r| r.id < 1));
}

#[test]
fn ann_query_stats_captures_latency() {
    let config = VectorIndexConfig { dimensions: 3, ..VectorIndexConfig::default() };
    let index = BruteForceIndex::new(config);
    index.insert(1, &[1.0, 0.0, 0.0]).unwrap();

    let planner = AdaptiveQueryPlanner::with_defaults();
    let (_, stats) = planner.execute(
        &index, &[1.0, 0.0, 0.0], 1, None, 1.0, None,
    ).unwrap();

    // Latency should be non-zero
    assert!(stats.latency.as_nanos() > 0, "latency should be > 0");
}

#[test]
fn fixed_selectivity_estimator() {
    let estimator = FixedSelectivityEstimator::new(0.15);
    assert!((estimator.estimate_selectivity(&|_| true) - 0.15).abs() < f64::EPSILON);

    let mut estimator = FixedSelectivityEstimator::new(2.0); // clamped to 1.0
    assert!((estimator.estimate_selectivity(&|_| true) - 1.0).abs() < f64::EPSILON);

    estimator.set_selectivity(-0.5); // clamped to 0.0
    assert!((estimator.estimate_selectivity(&|_| true) - 0.0).abs() < f64::EPSILON);
}

#[test]
fn planner_config_defaults() {
    let config = PlannerConfig::default();
    assert!((config.in_graph_min_selectivity - 0.20).abs() < f64::EPSILON);
    assert!((config.brute_force_max_selectivity - 0.01).abs() < f64::EPSILON);
    assert!((config.ef_search_multiplier_moderate - 2.0).abs() < f64::EPSILON);
    assert!((config.ef_search_multiplier_low - 3.0).abs() < f64::EPSILON);
    assert_eq!(config.default_ef_search, 200);
}

Acceptance Criteria

• AnnStrategy enum with 4 variants: Unfiltered, InGraphFilter, WidenedFilter { ef_search }, PreFilterBruteForce
• AdaptiveQueryPlanner::select_strategy() correctly routes: < 1% to brute-force, 1-5% to widened(3x), 5-20% to widened(2x), > 20% to in-graph, 100% to unfiltered
• AdaptiveQueryPlanner::execute() dispatches to the correct VectorIndex method based on selected strategy
• execute() falls back to WidenedFilter when PreFilterBruteForce is selected but no brute-force index is available
• AnnQueryStats captures: estimated_selectivity, strategy, results_returned, requested_k, latency
• PlannerConfig allows threshold tuning with correct defaults
• FixedSelectivityEstimator returns caller-provided selectivity, clamped to [0.0, 1.0]
• SelectivityEstimator trait defined for future m2p2 integration
• Criterion benchmarks implemented: bench_ann_search_unfiltered, bench_ann_search_filtered_20pct, bench_ann_search_filtered_5pct, bench_ann_search_brute_force, bench_ann_recall_at_100
• All benchmarks compile and produce results (performance targets are tracked, not gated)
• ANN retrieval latency < 10ms at 10K vectors (benchmark report)
• ANN recall@100 > 0.95 at 10K vectors (benchmark report)
• No unsafe code in planner.rs
• cargo clippy -- -D warnings passes
• All unit tests pass

Research References

• docs/research/ann_for_tidaldb.md -- "The critical insight across all systems: at extreme selectivity (<1-2%), everyone falls back to pre-filter + brute-force", ACORN-1 two-hop expansion, adaptive query planner architecture, selectivity estimation via metadata indexes, brute-force breakeven at ef_search * 10 nodes

Spec References

• docs/specs/07-vector-retrieval.md -- Section 3 (three filtered ANN strategies: in-graph, pre-filter brute-force, ACORN-1 widened), Section 9 (adaptive query planner: decision tree, threshold reference table, runtime statistics AnnQueryStats, threshold adjustment bounds, query plan logging), Section 12 (performance targets: < 10ms unfiltered, < 15ms filtered > 20%, < 25ms filtered 1-20%, < 10ms brute-force < 1%; recall targets: > 97% unfiltered, > 95% filtered > 20%, > 90% filtered 1-20%, 100% brute-force; benchmark definitions)

Implementation Notes

• Add [[bench]] name = "vector" harness = false to tidal/Cargo.toml.
• Add rand = "0.9" to [dev-dependencies] if not already present (shared with Task 02 tests).
• The benchmarks use 128-dimensional vectors for CI speed. Add a separate #[cfg(feature = "nightly-bench")] set at 1536 dimensions for nightly performance regression tracking.
• The execute() method takes Option<&dyn Fn(VectorId) -> bool> for the filter, not a &dyn Fn. When None, the planner always selects Unfiltered regardless of the selectivity parameter. This is a convenience for callers that do not have a filter.
• The brute_force_index parameter in execute() is Option<&dyn VectorIndex>. In practice, the RETRIEVE executor holds both the HNSW index and a reference to the entity store embeddings that can be loaded for brute-force scan. For M2, the BruteForceIndex is pre-populated alongside the HNSW index for small datasets. At scale, brute-force operates by loading embeddings from the entity store on demand.
• Do NOT implement threshold self-tuning based on AnnQueryStats in this task. The stats are collected for observability; automatic threshold adjustment is an M7 optimization. The thresholds are fixed per PlannerConfig.
• Do NOT implement the ACORN-1 two-hop expansion as a separate strategy. The WidenedFilter with increased ef_search achieves a similar effect. True ACORN-1 requires modifying the HNSW traversal algorithm inside USearch, which is not exposed via the current API. Deferred to M7 if widened ef_search proves insufficient.