61 lines
4.0 KiB
Markdown
61 lines
4.0 KiB
Markdown
# m7p3: Performance at Scale
|
|
|
|
## Delivers
|
|
|
|
Benchmarks and optimization at 1M items, 100K users, 10K creators. USearch parameter tuning for recall/latency tradeoff. Tantivy segment management for steady-state read/write performance. Signal state memory footprint analysis with LRU trimming if exceeding 10 GB. Signal rollup evaluation for 30d+ windows if bucketed counters exceed the 50ms budget at scale. Flamegraph profiling of RETRIEVE and SEARCH hot paths with optimization of any hotspot exceeding 10% of total time. CoEngagementIndex LRU correctness at 2x capacity and social graph filter verification at 1M items.
|
|
|
|
## Dependencies
|
|
|
|
- m7p1 complete (provides the baseline entity/signal/relationship infrastructure at current scale)
|
|
|
|
## Research References
|
|
|
|
- `docs/research/ann_for_tidaldb.md` -- USearch parameter guidance (M, ef_construction, recall/latency tradeoff at 1536D)
|
|
- `docs/research/tantivy.md` -- LogMergePolicy, segment management, concurrent read/write latency
|
|
- `docs/research/tidaldb_signal_ledger.md` -- Three-tier hybrid, per-entity memory model, rollup architecture
|
|
|
|
## Acceptance Criteria (Phase Level)
|
|
|
|
- [ ] Criterion benchmark suite: RETRIEVE p99 < 50ms, SEARCH p99 < 100ms, signal write p99 < 100us amortized at 1M items
|
|
- [ ] USearch M={8,16,32} x ef_construction={100,200,400} benchmarked; optimal config documented and applied; ANN recall@10 > 0.95
|
|
- [ ] Tantivy LogMergePolicy tuned; segment count < 20 at steady state; background merge does not block reads
|
|
- [ ] Signal state memory < 10 GB for 1M items x 10 signal types x 5 windows; LRU trimming if exceeded
|
|
- [ ] Signal rollup evaluation: benchmark 30d windowed count; implement if p99 > 50ms; document result
|
|
- [ ] Flamegraph profiling; top-3 hotspots documented; any > 10% of total time optimized
|
|
- [ ] CoEngagementIndex eviction at 2x capacity: memory stays bounded; LRU ordering correct
|
|
- [ ] Cross-session preference at scale: 100K users x 10 sessions; merge < 1ms per merge
|
|
|
|
## Task Execution Order
|
|
|
|
```
|
|
task-01 (Scale Benchmark Suite)
|
|
|
|
|
+---> task-02 (USearch Parameter Tuning)
|
|
+---> task-03 (Tantivy Merge Policy)
|
|
+---> task-04 (Signal State Memory Analysis)
|
|
+---> task-05 (Signal Rollup Evaluation)
|
|
+---> task-06 (Flamegraph Profiling)
|
|
+---> task-07 (CoEngagement LRU + Social Scale)
|
|
```
|
|
|
|
task-01 establishes baselines that all other tasks depend on. Tasks 02-07 are independent and can execute in parallel after task-01 completes.
|
|
|
|
## Module Location
|
|
|
|
- New bench file: `tidal/benches/scale.rs` -- 1M-item benchmark suite
|
|
- Existing bench files extended: `tidal/benches/vector.rs`, `tidal/benches/search.rs`, `tidal/benches/social.rs`
|
|
- Configuration changes: `tidal/src/storage/vector/` (USearch params), `tidal/src/text/` (Tantivy merge policy)
|
|
- Potential new module: `tidal/src/signals/trimmer.rs` (LRU trimming, if memory exceeds budget)
|
|
|
|
## Notes
|
|
|
|
- The 1M-item benchmark setup will take minutes to construct. Use `criterion::BenchmarkGroup::sample_size(10)` and `measurement_time(Duration::from_secs(60))` for these large-scale benches to keep CI wall time manageable.
|
|
- USearch parameter tuning requires building separate indexes per (M, ef_construction) pair. Build these once in setup, not per iteration.
|
|
- Tantivy segment count observation requires a write-then-search steady-state loop, not a single query benchmark.
|
|
- Signal memory analysis is a measurement task, not a benchmark. Use `std::mem::size_of` and DashMap entry counting rather than Criterion.
|
|
- Flamegraph profiling uses `cargo flamegraph` (requires `flamegraph` crate or `cargo install flamegraph`). Not a Criterion bench -- produces SVG artifacts.
|
|
|
|
## Done When
|
|
|
|
All 8 phase-level acceptance criteria pass. The `scale` bench target is registered in `Cargo.toml`. Flamegraph SVGs are committed to `docs/profiling/`. USearch optimal config is applied to `VectorIndexConfig::default()`. Tantivy merge policy is configured in `TextIndex` construction. Signal trimming is implemented if the memory threshold is exceeded, or a document explains why it is not needed.
|