tidaldb/docs/planning/milestone-7/phase-3/OVERVIEW.md

m7p3: Performance at Scale

Delivers

Benchmarks and optimization at 1M items, 100K users, 10K creators. USearch parameter tuning for recall/latency tradeoff. Tantivy segment management for steady-state read/write performance. Signal state memory footprint analysis with LRU trimming if exceeding 10 GB. Signal rollup evaluation for 30d+ windows if bucketed counters exceed the 50ms budget at scale. Flamegraph profiling of RETRIEVE and SEARCH hot paths with optimization of any hotspot exceeding 10% of total time. CoEngagementIndex LRU correctness at 2x capacity and social graph filter verification at 1M items.

Dependencies

• m7p1 complete (provides the baseline entity/signal/relationship infrastructure at current scale)

Research References

• docs/research/ann_for_tidaldb.md -- USearch parameter guidance (M, ef_construction, recall/latency tradeoff at 1536D)
• docs/research/tantivy.md -- LogMergePolicy, segment management, concurrent read/write latency
• docs/research/tidaldb_signal_ledger.md -- Three-tier hybrid, per-entity memory model, rollup architecture

Acceptance Criteria (Phase Level)

• Criterion benchmark suite: RETRIEVE p99 < 50ms, SEARCH p99 < 100ms, signal write p99 < 100us amortized at 1M items
• USearch M={8,16,32} x ef_construction={100,200,400} benchmarked; optimal config documented and applied; ANN recall@10 > 0.95
• Tantivy LogMergePolicy tuned; segment count < 20 at steady state; background merge does not block reads
• Signal state memory < 10 GB for 1M items x 10 signal types x 5 windows; LRU trimming if exceeded
• Signal rollup evaluation: benchmark 30d windowed count; implement if p99 > 50ms; document result
• Flamegraph profiling; top-3 hotspots documented; any > 10% of total time optimized
• CoEngagementIndex eviction at 2x capacity: memory stays bounded; LRU ordering correct
• Cross-session preference at scale: 100K users x 10 sessions; merge < 1ms per merge

Task Execution Order

task-01 (Scale Benchmark Suite)
    |
    +---> task-02 (USearch Parameter Tuning)
    +---> task-03 (Tantivy Merge Policy)
    +---> task-04 (Signal State Memory Analysis)
    +---> task-05 (Signal Rollup Evaluation)
    +---> task-06 (Flamegraph Profiling)
    +---> task-07 (CoEngagement LRU + Social Scale)

task-01 establishes baselines that all other tasks depend on. Tasks 02-07 are independent and can execute in parallel after task-01 completes.

Module Location

• New bench file: tidal/benches/scale.rs -- 1M-item benchmark suite
• Existing bench files extended: tidal/benches/vector.rs, tidal/benches/search.rs, tidal/benches/social.rs
• Configuration changes: tidal/src/storage/vector/ (USearch params), tidal/src/text/ (Tantivy merge policy)
• Potential new module: tidal/src/signals/trimmer.rs (LRU trimming, if memory exceeds budget)

Notes

• The 1M-item benchmark setup will take minutes to construct. Use criterion::BenchmarkGroup::sample_size(10) and measurement_time(Duration::from_secs(60)) for these large-scale benches to keep CI wall time manageable.
• USearch parameter tuning requires building separate indexes per (M, ef_construction) pair. Build these once in setup, not per iteration.
• Tantivy segment count observation requires a write-then-search steady-state loop, not a single query benchmark.
• Signal memory analysis is a measurement task, not a benchmark. Use std::mem::size_of and DashMap entry counting rather than Criterion.
• Flamegraph profiling uses cargo flamegraph (requires flamegraph crate or cargo install flamegraph). Not a Criterion bench -- produces SVG artifacts.

Done When

All 8 phase-level acceptance criteria pass. The scale bench target is registered in Cargo.toml. Flamegraph SVGs are committed to docs/profiling/. USearch optimal config is applied to VectorIndexConfig::default(). Tantivy merge policy is configured in TextIndex construction. Signal trimming is implemented if the memory threshold is exceeded, or a document explains why it is not needed.