tidaldb/.sdlc/features/pg1-baseline-comparison/spec.md

Baseline Comparison Study — Specification

Overview

This feature adds an A/B comparison framework to tidalDB that measures the lift of personalized ranking against a non-personalized baseline feed. The framework runs within a single TidalDb instance, using the existing session and query infrastructure to produce statistically meaningful measurements of click-through rate (CTR), content completion rate, and return rate across treatment (personalized) and control (baseline) groups.

Problem Statement

tidalDB's personalization loop (signal write -> decay -> ranking profile scoring) is implemented through M8, but there is no built-in mechanism to prove it produces better outcomes than a simple non-personalized feed. Without this evidence:

1. Operators cannot justify the complexity cost of personalization.
2. There is no baseline to measure regressions against.
3. The PG1 gate ("measurably better than baseline") cannot pass.

Goals

1. Baseline profile: Register a "chronological" ranking profile that returns items in reverse-created_at order with no signal boosts, no personalization, no diversity enforcement — the simplest possible non-personalized feed.
2. Experiment assignment: Deterministic, stable assignment of users to control (baseline) or treatment (personalized) groups based on user_id hash, configurable split ratio.
3. Metric collection: Per-group aggregation of CTR, completion rate, and return rate from existing signal data — no new storage format.
4. Comparison report: A structured ExperimentReport that computes per-metric lift (treatment vs. control) with sample sizes, so callers can evaluate statistical significance externally.

Non-Goals

• Statistical significance testing (p-values, confidence intervals) — callers bring their own stats library.
• Multi-armed bandits or adaptive assignment — this is a fixed A/B split.
• UI or API endpoint — this is a library-level feature; tidal-server can expose it later.
• Cross-node experiment coordination — single-node only (consistent with tidalDB's single-node-first principle).

Design Constraints

• Must use existing RankingProfile and ProfileRegistry infrastructure — no parallel ranking system.
• Must use existing SignalLedger reads for metric computation — no new storage tier.
• Assignment must be deterministic and stable: same user_id + experiment_id always maps to the same group. Users must not flip groups during an experiment.
• The baseline profile must be a real registered RankingProfile usable with db.retrieve() — not a special code path.

Detailed Requirements

R1: Baseline Ranking Profile

Register a new builtin profile named "chronological":

• CandidateStrategy::Scan { sort_field: "created_at" }
• sort: Some(Sort::New) — reverse chronological
• No boosts, no decay, no gates, no penalties, no diversity constraints
• exploration: 0.0
• is_builtin: true

R2: Experiment Configuration

pub struct ExperimentConfig {
    pub experiment_id: String,
    pub treatment_fraction: f64,
    pub treatment_profile: String,
    pub control_profile: String,
    pub click_signals: Vec<String>,
    pub completion_signals: Vec<String>,
    pub return_window: Duration,
}

R3: User Assignment

Deterministic FNV-1a hash over experiment_id bytes + user_id LE bytes. Bucket modulo 10,000 compared to treatment_fraction * 10,000. Pure function, no state.

R4: Metric Aggregation

Per-group metrics computed from UserSignalIndex reads:

• CTR: total_clicks / total_views
• Completion rate: total_completions / total_views
• Return rate: fraction of users with both old and recent activity

R5: Experiment Report

ExperimentReport with GroupMetrics (ctr, completion_rate, return_rate, totals) and MetricLift (relative lift per metric).

R6: TidalDb API Surface

• experiment_group(&self, user_id, config) -> Result<ExperimentGroup>
• experiment_profile(&self, user_id, config) -> Result<&str>
• experiment_report(&self, config, user_ids) -> Result<ExperimentReport>

Acceptance Criteria

1. chronological profile returns items in reverse-created_at order.
2. assign_group is deterministic (1000 calls, same result).
3. 50/50 split within 4750-5250 for 10K users.
4. CTR = clicks / views (not clicks / users).
5. Lift edge cases: 0/0 = 0.0, positive/0 = INFINITY.
6. Synthetic workload shows positive CTR lift for treatment.
7. Unit tests + integration test covering end-to-end flow.

File Placement

• tidal/src/experiment/mod.rs — types + assign_group()
• tidal/src/experiment/report.rs — ReportBuilder
• tidal/src/db/experiment.rs — TidalDb impl block
• tidal/src/ranking/builtins.rs — chronological() profile
• tidal/tests/pg1_baseline.rs — integration test