52 lines
2.8 KiB
Markdown
52 lines
2.8 KiB
Markdown
# Baseline Comparison Study — Tasks
|
|
|
|
## T1: Register chronological builtin profile
|
|
- Add `chronological()` function to `tidal/src/ranking/builtins.rs`
|
|
- Uses `Sort::New` for pure reverse-chronological ordering, no personalization signals
|
|
- Register in `register_builtins()`
|
|
- Add unit test verifying sort mode and no boosts
|
|
- **Acceptance:** `cargo test --manifest-path tidal/Cargo.toml --lib` passes; profile available via retrieve
|
|
|
|
## T2: Create experiment types module
|
|
- New file `tidal/src/experiment/mod.rs`
|
|
- `ExperimentConfig` struct: experiment_id, treatment_fraction, treatment/control profile names, click/completion signal names, return_window
|
|
- `ExperimentGroup` enum: Treatment, Control (PartialEq, Eq, Debug, Clone)
|
|
- `assign_group(user_id, experiment_id, treatment_fraction) -> ExperimentGroup` using FNV-1a hash for deterministic assignment
|
|
- `GroupMetrics` struct: views, clicks, completions, ctr, completion_rate, return_rate
|
|
- `MetricLift` struct with `compute()` and `relative_lift()`
|
|
- `ExperimentReport` struct aggregating both groups
|
|
- Unit tests for hash determinism, balanced split, metric lift math
|
|
- **Acceptance:** `cargo check --manifest-path tidal/Cargo.toml` clean; 16 unit tests pass
|
|
|
|
## T3: Build report aggregation
|
|
- New file `tidal/src/experiment/report.rs`
|
|
- `ReportBuilder` takes `&Arc<SignalLedger>` and `&UserSignalIndex`
|
|
- `aggregate_group_metrics()` resolves signal type IDs, iterates users calling `user_signal_count()` and `user_activity_split()`
|
|
- Add `user_signal_count(user_id, type_id) -> u64` to `UserSignalIndex`
|
|
- Add `user_activity_split(user_id, window_boundary_ns) -> (bool, bool)` to `UserSignalIndex`
|
|
- **Acceptance:** Report builder compiles and aggregates correctly in unit tests
|
|
|
|
## T4: Wire experiment API onto TidalDb
|
|
- New file `tidal/src/db/experiment.rs`
|
|
- `experiment_group(user_id, &ExperimentConfig) -> Result<ExperimentGroup>`
|
|
- `experiment_profile(user_id, &ExperimentConfig) -> Result<&str>`
|
|
- `experiment_report(&ExperimentConfig, &[u64]) -> Result<ExperimentReport>`
|
|
- Add `mod experiment;` to `db/mod.rs`
|
|
- **Acceptance:** Methods callable from integration tests
|
|
|
|
## T5: Integration tests
|
|
- New file `tidal/tests/pg1_baseline.rs`
|
|
- Test chronological profile returns reverse created_at order
|
|
- Test chronological is deterministic across users
|
|
- Test experiment group assignment is deterministic
|
|
- Test experiment profile returns correct name
|
|
- Test balanced 50/50 split across 10k users
|
|
- Test experiment report with synthetic signals (treatment > control CTR)
|
|
- Test experiment report is reproducible
|
|
- **Acceptance:** All 7 integration tests pass
|
|
|
|
## T6: Wire re-exports in lib.rs
|
|
- Add `pub mod experiment;` to `tidal/src/lib.rs`
|
|
- Re-export `ExperimentConfig`, `ExperimentGroup`, `ExperimentReport`, `GroupMetrics`, `MetricLift`
|
|
- **Acceptance:** Types accessible as `tidaldb::ExperimentConfig` etc.
|