99 lines
4.7 KiB
Markdown
99 lines
4.7 KiB
Markdown
# Baseline Comparison Study — Specification
|
|
|
|
## Overview
|
|
|
|
This feature adds an A/B comparison framework to tidalDB that measures the lift of personalized ranking against a non-personalized baseline feed. The framework runs within a single TidalDb instance, using the existing session and query infrastructure to produce statistically meaningful measurements of click-through rate (CTR), content completion rate, and return rate across treatment (personalized) and control (baseline) groups.
|
|
|
|
## Problem Statement
|
|
|
|
tidalDB's personalization loop (signal write -> decay -> ranking profile scoring) is implemented through M8, but there is no built-in mechanism to prove it produces better outcomes than a simple non-personalized feed. Without this evidence:
|
|
|
|
1. Operators cannot justify the complexity cost of personalization.
|
|
2. There is no baseline to measure regressions against.
|
|
3. The PG1 gate ("measurably better than baseline") cannot pass.
|
|
|
|
## Goals
|
|
|
|
1. **Baseline profile**: Register a "chronological" ranking profile that returns items in reverse-created_at order with no signal boosts, no personalization, no diversity enforcement — the simplest possible non-personalized feed.
|
|
2. **Experiment assignment**: Deterministic, stable assignment of users to control (baseline) or treatment (personalized) groups based on user_id hash, configurable split ratio.
|
|
3. **Metric collection**: Per-group aggregation of CTR, completion rate, and return rate from existing signal data — no new storage format.
|
|
4. **Comparison report**: A structured `ExperimentReport` that computes per-metric lift (treatment vs. control) with sample sizes, so callers can evaluate statistical significance externally.
|
|
|
|
## Non-Goals
|
|
|
|
- Statistical significance testing (p-values, confidence intervals) — callers bring their own stats library.
|
|
- Multi-armed bandits or adaptive assignment — this is a fixed A/B split.
|
|
- UI or API endpoint — this is a library-level feature; tidal-server can expose it later.
|
|
- Cross-node experiment coordination — single-node only (consistent with tidalDB's single-node-first principle).
|
|
|
|
## Design Constraints
|
|
|
|
- Must use existing `RankingProfile` and `ProfileRegistry` infrastructure — no parallel ranking system.
|
|
- Must use existing `SignalLedger` reads for metric computation — no new storage tier.
|
|
- Assignment must be deterministic and stable: same user_id + experiment_id always maps to the same group. Users must not flip groups during an experiment.
|
|
- The baseline profile must be a real registered `RankingProfile` usable with `db.retrieve()` — not a special code path.
|
|
|
|
## Detailed Requirements
|
|
|
|
### R1: Baseline Ranking Profile
|
|
|
|
Register a new builtin profile named `"chronological"`:
|
|
- `CandidateStrategy::Scan { sort_field: "created_at" }`
|
|
- `sort: Some(Sort::New)` — reverse chronological
|
|
- No boosts, no decay, no gates, no penalties, no diversity constraints
|
|
- `exploration: 0.0`
|
|
- `is_builtin: true`
|
|
|
|
### R2: Experiment Configuration
|
|
|
|
```rust
|
|
pub struct ExperimentConfig {
|
|
pub experiment_id: String,
|
|
pub treatment_fraction: f64,
|
|
pub treatment_profile: String,
|
|
pub control_profile: String,
|
|
pub click_signals: Vec<String>,
|
|
pub completion_signals: Vec<String>,
|
|
pub return_window: Duration,
|
|
}
|
|
```
|
|
|
|
### R3: User Assignment
|
|
|
|
Deterministic FNV-1a hash over `experiment_id` bytes + `user_id` LE bytes. Bucket modulo 10,000 compared to `treatment_fraction * 10,000`. Pure function, no state.
|
|
|
|
### R4: Metric Aggregation
|
|
|
|
Per-group metrics computed from `UserSignalIndex` reads:
|
|
- **CTR**: `total_clicks / total_views`
|
|
- **Completion rate**: `total_completions / total_views`
|
|
- **Return rate**: fraction of users with both old and recent activity
|
|
|
|
### R5: Experiment Report
|
|
|
|
`ExperimentReport` with `GroupMetrics` (ctr, completion_rate, return_rate, totals) and `MetricLift` (relative lift per metric).
|
|
|
|
### R6: TidalDb API Surface
|
|
|
|
- `experiment_group(&self, user_id, config) -> Result<ExperimentGroup>`
|
|
- `experiment_profile(&self, user_id, config) -> Result<&str>`
|
|
- `experiment_report(&self, config, user_ids) -> Result<ExperimentReport>`
|
|
|
|
## Acceptance Criteria
|
|
|
|
1. `chronological` profile returns items in reverse-created_at order.
|
|
2. `assign_group` is deterministic (1000 calls, same result).
|
|
3. 50/50 split within 4750-5250 for 10K users.
|
|
4. CTR = clicks / views (not clicks / users).
|
|
5. Lift edge cases: 0/0 = 0.0, positive/0 = INFINITY.
|
|
6. Synthetic workload shows positive CTR lift for treatment.
|
|
7. Unit tests + integration test covering end-to-end flow.
|
|
|
|
## File Placement
|
|
|
|
- `tidal/src/experiment/mod.rs` — types + `assign_group()`
|
|
- `tidal/src/experiment/report.rs` — `ReportBuilder`
|
|
- `tidal/src/db/experiment.rs` — `TidalDb` impl block
|
|
- `tidal/src/ranking/builtins.rs` — `chronological()` profile
|
|
- `tidal/tests/pg1_baseline.rs` — integration test
|