# Baseline Comparison Study — Specification

## Overview

This feature adds an A/B comparison framework to tidalDB that measures the lift of personalized ranking against a non-personalized baseline feed. The framework runs within a single TidalDb instance, using the existing session and query infrastructure to produce statistically meaningful measurements of click-through rate (CTR), content completion rate, and return rate across treatment (personalized) and control (baseline) groups.

## Problem Statement

tidalDB's personalization loop (signal write -> decay -> ranking profile scoring) is implemented through M8, but there is no built-in mechanism to prove it produces better outcomes than a simple non-personalized feed. Without this evidence:

1. Operators cannot justify the complexity cost of personalization.
2. There is no baseline to measure regressions against.
3. The PG1 gate ("measurably better than baseline") cannot pass.

## Goals

1. **Baseline profile**: Register a "chronological" ranking profile that returns items in reverse-created_at order with no signal boosts, no personalization, no diversity enforcement — the simplest possible non-personalized feed.
2. **Experiment assignment**: Deterministic, stable assignment of users to control (baseline) or treatment (personalized) groups based on user_id hash, configurable split ratio.
3. **Metric collection**: Per-group aggregation of CTR, completion rate, and return rate from existing signal data — no new storage format.
4. **Comparison report**: A structured `ExperimentReport` that computes per-metric lift (treatment vs. control) with sample sizes, so callers can evaluate statistical significance externally.

## Non-Goals

- Statistical significance testing (p-values, confidence intervals) — callers bring their own stats library.
- Multi-armed bandits or adaptive assignment — this is a fixed A/B split.
- UI or API endpoint — this is a library-level feature; tidal-server can expose it later.
- Cross-node experiment coordination — single-node only (consistent with tidalDB's single-node-first principle).

## Design Constraints

- Must use existing `RankingProfile` and `ProfileRegistry` infrastructure — no parallel ranking system.
- Must use existing `SignalLedger` reads for metric computation — no new storage tier.
- Assignment must be deterministic and stable: same user_id + experiment_id always maps to the same group. Users must not flip groups during an experiment.
- The baseline profile must be a real registered `RankingProfile` usable with `db.retrieve()` — not a special code path.

## Detailed Requirements

### R1: Baseline Ranking Profile

Register a new builtin profile named `"chronological"`:
- `CandidateStrategy::Scan { sort_field: "created_at" }`
- `sort: Some(Sort::New)` — reverse chronological
- No boosts, no decay, no gates, no penalties, no diversity constraints
- `exploration: 0.0`
- `is_builtin: true`

### R2: Experiment Configuration

```rust
pub struct ExperimentConfig {
    pub experiment_id: String,
    pub treatment_fraction: f64,
    pub treatment_profile: String,
    pub control_profile: String,
    pub click_signals: Vec<String>,
    pub completion_signals: Vec<String>,
    pub return_window: Duration,
}
```

### R3: User Assignment

Deterministic FNV-1a hash over `experiment_id` bytes + `user_id` LE bytes. Bucket modulo 10,000 compared to `treatment_fraction * 10,000`. Pure function, no state.

### R4: Metric Aggregation

Per-group metrics computed from `UserSignalIndex` reads:
- **CTR**: `total_clicks / total_views`
- **Completion rate**: `total_completions / total_views`
- **Return rate**: fraction of users with both old and recent activity

### R5: Experiment Report

`ExperimentReport` with `GroupMetrics` (ctr, completion_rate, return_rate, totals) and `MetricLift` (relative lift per metric).

### R6: TidalDb API Surface

- `experiment_group(&self, user_id, config) -> Result<ExperimentGroup>`
- `experiment_profile(&self, user_id, config) -> Result<&str>`
- `experiment_report(&self, config, user_ids) -> Result<ExperimentReport>`

## Acceptance Criteria

1. `chronological` profile returns items in reverse-created_at order.
2. `assign_group` is deterministic (1000 calls, same result).
3. 50/50 split within 4750-5250 for 10K users.
4. CTR = clicks / views (not clicks / users).
5. Lift edge cases: 0/0 = 0.0, positive/0 = INFINITY.
6. Synthetic workload shows positive CTR lift for treatment.
7. Unit tests + integration test covering end-to-end flow.

## File Placement

- `tidal/src/experiment/mod.rs` — types + `assign_group()`
- `tidal/src/experiment/report.rs` — `ReportBuilder`
- `tidal/src/db/experiment.rs` — `TidalDb` impl block
- `tidal/src/ranking/builtins.rs` — `chronological()` profile
- `tidal/tests/pg1_baseline.rs` — integration test