45 lines
2.0 KiB
Markdown
45 lines
2.0 KiB
Markdown
# Baseline Comparison Study — Design
|
|
|
|
## Architecture
|
|
|
|
Pure backend/library feature. No UI. Three components added:
|
|
|
|
1. **`experiment` module** (`tidal/src/experiment/`) — types, assignment function, report builder
|
|
2. **`db/experiment.rs`** — `TidalDb` impl block wiring experiment methods to existing infrastructure
|
|
3. **`chronological` builtin profile** — registered in `ranking/builtins.rs`
|
|
|
|
## Module Layout
|
|
|
|
```
|
|
tidal/src/
|
|
├── experiment/
|
|
│ ├── mod.rs # ExperimentConfig, ExperimentGroup, ExperimentReport,
|
|
│ │ # GroupMetrics, MetricLift, assign_group()
|
|
│ └── report.rs # ReportBuilder: scans UserSignalIndex per group
|
|
├── db/
|
|
│ └── experiment.rs # TidalDb::experiment_group/profile/report
|
|
└── ranking/
|
|
└── builtins.rs # + chronological() profile
|
|
```
|
|
|
|
## Assignment
|
|
|
|
FNV-1a hash over `experiment_id.as_bytes()` concatenated with `user_id.to_le_bytes()`. Bucket = hash % 10,000. Treatment if bucket < threshold. Pure function, no I/O, deterministic across platforms.
|
|
|
|
## Report
|
|
|
|
`ReportBuilder` takes `&Arc<SignalLedger>` and `&UserSignalIndex`. Partitions users via `assign_group`, then for each group iterates users calling `user_signal_count()` (new method on `UserSignalIndex` that sums AllTime counts across all entities for a user+signal_type pair) and `user_activity_split()` for return rate.
|
|
|
|
## Key Design Decisions
|
|
|
|
1. **No new storage** — all metrics derived from existing `UserSignalIndex` DashMap scans
|
|
2. **New UserSignalIndex methods** — `user_signal_count(user_id, type_id)` and `user_activity_split(user_id, boundary_ns)` added to support per-user aggregation across all entities
|
|
3. **chronological profile** — uses existing `Sort::New`, no new sort variant needed
|
|
4. **Thread safety** — all types are `Send + Sync`, `assign_group` is pure, report takes `&self`
|
|
|
|
## Error Handling
|
|
|
|
- `InvalidInput`: treatment_fraction out of range, empty signal lists
|
|
- `Internal`: no schema/ledger available
|
|
- No new error variants needed
|