43 lines
2.2 KiB
Markdown
43 lines
2.2 KiB
Markdown
# Baseline Comparison Study — QA Plan
|
||
|
||
## Scope
|
||
Verify the A/B experiment framework: chronological profile, deterministic assignment, metric aggregation, lift calculation, and integration test coverage.
|
||
|
||
## Test Matrix
|
||
|
||
### 1. Chronological Profile
|
||
| # | Scenario | Method | Expected |
|
||
|---|----------|--------|----------|
|
||
| 1.1 | Reverse created_at ordering | Integration test | Items returned newest-first |
|
||
| 1.2 | Deterministic across users | Integration test | Same query produces identical order |
|
||
| 1.3 | Profile registered as builtin | Unit test | `chronological` in ProfileRegistry |
|
||
|
||
### 2. Experiment Assignment
|
||
| # | Scenario | Method | Expected |
|
||
|---|----------|--------|----------|
|
||
| 2.1 | Deterministic assignment | Integration test | Same user_id + config always yields same group |
|
||
| 2.2 | Balanced 50/50 split | Integration test | 10k users within 4750–5250 treatment |
|
||
| 2.3 | Profile name matches group | Integration test | Treatment -> treatment_profile, Control -> control_profile |
|
||
| 2.4 | FNV-1a hash quality | Unit test | No systematic bias across user_id ranges |
|
||
|
||
### 3. Report Aggregation
|
||
| # | Scenario | Method | Expected |
|
||
|---|----------|--------|----------|
|
||
| 3.1 | Synthetic signal report | Integration test | Treatment CTR > Control CTR with known signal pattern |
|
||
| 3.2 | Group sizes sum to total | Integration test | treatment_users + control_users == user_ids.len() |
|
||
| 3.3 | Reproducible report | Integration test | Two calls with same inputs yield identical output |
|
||
| 3.4 | Zero-signal edge case | Unit test | GroupMetrics with 0 views yields 0.0 rates (no NaN/panic) |
|
||
| 3.5 | CTR lift positive | Integration test | lift.ctr_lift > 0.0 when treatment has more clicks |
|
||
|
||
### 4. UserSignalIndex Extensions
|
||
| # | Scenario | Method | Expected |
|
||
|---|----------|--------|----------|
|
||
| 4.1 | user_signal_count aggregation | Unit/Integration | Counts across all entities for a user+signal_type |
|
||
| 4.2 | user_activity_split detection | Unit/Integration | Returns (has_any, has_recent) correctly |
|
||
|
||
## Pass Criteria
|
||
- All 7 integration tests in `pg1_baseline.rs` pass
|
||
- All 16 unit tests in `experiment/mod.rs` pass
|
||
- `cargo check --manifest-path tidal/Cargo.toml` clean (no warnings from experiment code)
|
||
- No panics on edge cases (zero views, empty user list)
|