# m7p3: Performance at Scale

## Delivers

Benchmarks and optimization at 1M items, 100K users, 10K creators. USearch parameter tuning for recall/latency tradeoff. Tantivy segment management for steady-state read/write performance. Signal state memory footprint analysis with LRU trimming if exceeding 10 GB. Signal rollup evaluation for 30d+ windows if bucketed counters exceed the 50ms budget at scale. Flamegraph profiling of RETRIEVE and SEARCH hot paths with optimization of any hotspot exceeding 10% of total time. CoEngagementIndex LRU correctness at 2x capacity and social graph filter verification at 1M items.

## Dependencies

- m7p1 complete (provides the baseline entity/signal/relationship infrastructure at current scale)

## Research References

- `docs/research/ann_for_tidaldb.md` -- USearch parameter guidance (M, ef_construction, recall/latency tradeoff at 1536D)
- `docs/research/tantivy.md` -- LogMergePolicy, segment management, concurrent read/write latency
- `docs/research/tidaldb_signal_ledger.md` -- Three-tier hybrid, per-entity memory model, rollup architecture

## Acceptance Criteria (Phase Level)

- [ ] Criterion benchmark suite: RETRIEVE p99 < 50ms, SEARCH p99 < 100ms, signal write p99 < 100us amortized at 1M items
- [ ] USearch M={8,16,32} x ef_construction={100,200,400} benchmarked; optimal config documented and applied; ANN recall@10 > 0.95
- [ ] Tantivy LogMergePolicy tuned; segment count < 20 at steady state; background merge does not block reads
- [ ] Signal state memory < 10 GB for 1M items x 10 signal types x 5 windows; LRU trimming if exceeded
- [ ] Signal rollup evaluation: benchmark 30d windowed count; implement if p99 > 50ms; document result
- [ ] Flamegraph profiling; top-3 hotspots documented; any > 10% of total time optimized
- [ ] CoEngagementIndex eviction at 2x capacity: memory stays bounded; LRU ordering correct
- [ ] Cross-session preference at scale: 100K users x 10 sessions; merge < 1ms per merge

## Task Execution Order

```
task-01 (Scale Benchmark Suite)
    |
    +---> task-02 (USearch Parameter Tuning)
    +---> task-03 (Tantivy Merge Policy)
    +---> task-04 (Signal State Memory Analysis)
    +---> task-05 (Signal Rollup Evaluation)
    +---> task-06 (Flamegraph Profiling)
    +---> task-07 (CoEngagement LRU + Social Scale)
```

task-01 establishes baselines that all other tasks depend on. Tasks 02-07 are independent and can execute in parallel after task-01 completes.

## Module Location

- New bench file: `tidal/benches/scale.rs` -- 1M-item benchmark suite
- Existing bench files extended: `tidal/benches/vector.rs`, `tidal/benches/search.rs`, `tidal/benches/social.rs`
- Configuration changes: `tidal/src/storage/vector/` (USearch params), `tidal/src/text/` (Tantivy merge policy)
- Potential new module: `tidal/src/signals/trimmer.rs` (LRU trimming, if memory exceeds budget)

## Notes

- The 1M-item benchmark setup will take minutes to construct. Use `criterion::BenchmarkGroup::sample_size(10)` and `measurement_time(Duration::from_secs(60))` for these large-scale benches to keep CI wall time manageable.
- USearch parameter tuning requires building separate indexes per (M, ef_construction) pair. Build these once in setup, not per iteration.
- Tantivy segment count observation requires a write-then-search steady-state loop, not a single query benchmark.
- Signal memory analysis is a measurement task, not a benchmark. Use `std::mem::size_of` and DashMap entry counting rather than Criterion.
- Flamegraph profiling uses `cargo flamegraph` (requires `flamegraph` crate or `cargo install flamegraph`). Not a Criterion bench -- produces SVG artifacts.

## Done When

All 8 phase-level acceptance criteria pass. The `scale` bench target is registered in `Cargo.toml`. Flamegraph SVGs are committed to `docs/profiling/`. USearch optimal config is applied to `VectorIndexConfig::default()`. Tantivy merge policy is configured in `TextIndex` construction. Signal trimming is implemented if the memory threshold is exceeded, or a document explains why it is not needed.