tidaldb/.sdlc/features/pg1-baseline-comparison/qa-results.md

Baseline Comparison Study — QA Results

Test Execution Summary

Suite	Tests	Passed	Failed	Duration
Unit tests (experiment::tests)	17	17	0	0.00s
Builtin tests (chronological)	2	2	0	0.00s
Integration tests (pg1_baseline)	7	7	0	0.04s
Total	26	26	0	< 1s

Compilation

• cargo check --manifest-path tidal/Cargo.toml --lib — CLEAN (no errors, no warnings from experiment code)
• cargo test --manifest-path tidal/Cargo.toml --lib -- experiment — 17 passed
• cargo test --manifest-path tidal/Cargo.toml --test pg1_baseline — 7 passed

QA Plan Coverage

1. Chronological Profile

#	Scenario	Result	Notes
1.1	Reverse created_at ordering	PASS	chronological_profile_returns_reverse_created_at — newest item (id=10) first, oldest (id=1) last
1.2	Deterministic across users	PASS	chronological_produces_same_order_for_different_users — same query, identical order
1.3	Profile registered as builtin	PASS	all_twentyseven_builtins_registered — 27 builtins including chronological

2. Experiment Assignment

#	Scenario	Result	Notes
2.1	Deterministic assignment	PASS	experiment_group_assignment_is_deterministic — 100 iterations same result
2.2	Balanced 50/50 split	PASS	experiment_group_balanced_split — 10k users within 4750-5250
2.3	Profile name matches group	PASS	experiment_profile_returns_correct_name — Treatment=for_you, Control=chronological
2.4	FNV-1a hash quality	PASS	treatment_fraction_0_5_produces_balanced_split + treatment_fraction_0_1_produces_10_percent_split

3. Report Aggregation

#	Scenario	Result	Notes
3.1	Synthetic signal report	PASS	experiment_report_with_synthetic_signals — treatment CTR > control CTR
3.2	Group sizes sum to total	PASS	Verified: treatment_users + control_users == 100
3.3	Reproducible report	PASS	experiment_report_is_reproducible — two calls identical
3.4	Zero-signal edge case	PASS	GroupMetrics::zero() returns 0.0 rates; relative_lift(0.0, 0.0) returns 0.0
3.5	CTR lift positive	PASS	report.lift.ctr_lift > 0.0 verified in integration test

4. UserSignalIndex Extensions

#	Scenario	Result	Notes
4.1	user_signal_count aggregation	PASS	Used in report builder; verified via synthetic signal patterns
4.2	user_activity_split detection	PASS	Used in return rate calculation; verified implicitly

Issues Found During QA

1. signal_with_context signature change: The method now takes 6 arguments (added creator_id: Option<u64> from uncommitted M10 work). Fixed integration test to pass None for the 6th parameter. This is a pre-existing API change, not a defect in the experiment code.

Verdict

PASSED. All 26 tests pass. All QA plan scenarios verified. One minor test fix applied for API compatibility with uncommitted M10 changes. No defects found in the experiment framework itself.