Baseline Comparison Study — QA Plan
Scope
Verify the A/B experiment framework: chronological profile, deterministic assignment, metric aggregation, lift calculation, and integration test coverage.
Test Matrix
1. Chronological Profile
| # |
Scenario |
Method |
Expected |
| 1.1 |
Reverse created_at ordering |
Integration test |
Items returned newest-first |
| 1.2 |
Deterministic across users |
Integration test |
Same query produces identical order |
| 1.3 |
Profile registered as builtin |
Unit test |
chronological in ProfileRegistry |
2. Experiment Assignment
| # |
Scenario |
Method |
Expected |
| 2.1 |
Deterministic assignment |
Integration test |
Same user_id + config always yields same group |
| 2.2 |
Balanced 50/50 split |
Integration test |
10k users within 4750–5250 treatment |
| 2.3 |
Profile name matches group |
Integration test |
Treatment -> treatment_profile, Control -> control_profile |
| 2.4 |
FNV-1a hash quality |
Unit test |
No systematic bias across user_id ranges |
3. Report Aggregation
| # |
Scenario |
Method |
Expected |
| 3.1 |
Synthetic signal report |
Integration test |
Treatment CTR > Control CTR with known signal pattern |
| 3.2 |
Group sizes sum to total |
Integration test |
treatment_users + control_users == user_ids.len() |
| 3.3 |
Reproducible report |
Integration test |
Two calls with same inputs yield identical output |
| 3.4 |
Zero-signal edge case |
Unit test |
GroupMetrics with 0 views yields 0.0 rates (no NaN/panic) |
| 3.5 |
CTR lift positive |
Integration test |
lift.ctr_lift > 0.0 when treatment has more clicks |
4. UserSignalIndex Extensions
| # |
Scenario |
Method |
Expected |
| 4.1 |
user_signal_count aggregation |
Unit/Integration |
Counts across all entities for a user+signal_type |
| 4.2 |
user_activity_split detection |
Unit/Integration |
Returns (has_any, has_recent) correctly |
Pass Criteria
- All 7 integration tests in
pg1_baseline.rs pass
- All 16 unit tests in
experiment/mod.rs pass
cargo check --manifest-path tidal/Cargo.toml clean (no warnings from experiment code)- No panics on edge cases (zero views, empty user list)
