tidaldb/.sdlc/features/p0-validation-readout/spec.md

p0-validation-readout: Validation Readout

Problem Statement

The P0 pilot produces behavioral events, feedback actions, retention counters, and interview transcripts, but nothing converts them into a decision. Without a pre-registered analysis, the P1 Concierge Alpha build decision degrades into advocacy: whoever ran the pilot narrates the numbers that flatter it, kill criteria (docs/personal-briefing-beachhead.md §12) are re-litigated after the fact, and ambiguous cases ("D2 was close") resolve as GO by default.

This feature defines the analysis, the thresholds, and the readout artifact before the pilot data is read, so that the GO/NO-GO decision is a computation over a results table rather than a judgement call. It must also answer the questions the numbers alone cannot: how many operator interventions were required to keep briefing quality acceptable, and what evidence would falsify the conclusion.

Goals

1. Deterministic verdict -- One of GO / NO-GO / EXTEND derived mechanically from the pilot inputs.
2. Pre-registration -- Every threshold and computation rule fixed and committed before outcome data is read.
3. Traceability -- Each threshold cites a ROADMAP P0 acceptance criterion or a beachhead §9/§12 metric, or is explicitly TBD (owner: product).
4. Honest cost accounting -- The manual QA and operator intervention required to sustain quality is reported as a first-class result, not a footnote.
5. Closed loop into the roadmap -- A NO-GO or EXTEND verdict lands as a roadmap change, not a dropped thread.

Non-Goals

• Recruiting the pilot cohort (p0-target-segment-recruitment).
• Operating the daily briefing or conducting the interviews (p0-concierge-pilot-loop).
• Building any P1 surface, ranking change, or engine feature.
• Deciding P2/P3 scope, pricing, or go-to-market.

Functional Requirements

FR-1: Frozen Input Snapshot

Consume from p0-concierge-pilot-loop FR-9, as a single immutable snapshot frozen at day 14 -- no rows added, corrected, or recoded after handoff: (a) the raw timestamped interaction stream (every open, card open, feedback action, and close, each with briefing_day and actor) as the primitive, plus the derived session records and the frozen inactivity timeout value, (b) per-session feedback action counts by type (more/less/hide/mute/save), (c) D2 retention counters from the pg1 metrics pipeline (daily /diagnostics JSON snapshots plus the interaction stream), (d) the operator-intervention log and the missed-briefing-day log, (e) interview transcripts. Record a content hash per input file in the readout. Analysis reads only the snapshot; later pilot data does not amend a published verdict.

FR-2: Session and Cohort Definitions

A session is the session boundary emitted by p0-concierge-pilot-loop FR-5 (opens on first open of that day's briefing; closes on explicit close/navigation-away or the inactivity timeout that feature freezes before day 1); this analysis defines no timeout of its own. A re-open within the same briefing_day is a distinct session (beachhead §9.2 sessions-per-active-day). Sessions carrying actor=operator are excluded from every cohort metric. The pilot cohort denominator is participants who completed activation: first daily briefing delivered and opened. Enrolled-but-never-activated participants are reported separately and excluded from retention and feedback metrics.

FR-3: D2 Retention Computation

Each participant has a personal Day-0 = the briefing_day of their first opened briefing. D2 retention = fraction of activated participants with >= 1 actor=participant session on briefing_day Day-0 + 1. Because the metric counts distinct briefing_day values rather than session objects, it is invariant to the FR-2 inactivity timeout; only FR-4 is timeout-sensitive. Rules, applied in order:

1. Missed briefing. If no briefing was delivered on a participant's Day-0 + 1 (missed-briefing-day log), that day is void: Day-0 re-anchors to their next delivered-and-opened briefing. Count and report re-anchored participants.
2. Reminders. Product-cadence push/email (beachhead §5.2) counts as a normal return. A session that follows an operator or support nudge within the same day does not count, per beachhead §6.4.5 ("returns on Day 2 without a reminder from support or onboarding prompts").
3. Dropouts. Intent-to-treat: a participant who stops using the product stays in the denominator as a non-return. Withdrawals are excluded only with a named reason in the operator-intervention log, and the metric is reported both with and without exclusions (see FR-6).
4. Partial observation. A participant whose Day-0 + 1 falls after pilot end is excluded from D2 and reported as unobserved.

FR-4: Feedback-Action Rate Computation

For each activated participant, compute their median feedback action count per session across all their observed sessions (zero-action sessions included). The cohort statistic is the median of those per-participant medians. The ROADMAP criterion "at least one meaningful feedback action per session for the median user" is met iff that cohort statistic is >= 1. Report the distribution, not only the median, and report per-action-type counts so a cohort that only ever taps save is visible.

Because this statistic counts sessions, it is sensitive to the inactivity timeout frozen by p0-concierge-pilot-loop FR-5. Record that timeout value in the pre-registration block, then re-derive sessions from the FR-1 raw interaction stream under a stated alternative timeout and recompute this statistic as part of the FR-6 sensitivity check: if G2 flips, the verdict is EXTEND, never GO. Re-sessionization uses the frozen snapshot only -- it never triggers re-collection.

FR-5: Interview Coding

Code every transcript against the beachhead §6.2 required answers. For each of the eight rows, mark proven / not proven / not raised, with a verbatim quote required for proven. Separately, code the three ROADMAP value axes against the participant's own named baseline feed (§6.1: existing feeds, newsletters, AI assistants) as a forced choice per axis -- better / same / worse:

• less noise, more useful, saves time.

A participant counts as value-confirmed only when all three axes are better. Also code the §12.4 kill frame: does the participant describe the product as "another noisy feed" or equivalent. Every transcript is double-coded; unresolved disagreement resolves to the conservative code (same, and kill-frame present).

FR-6: Pre-Registered Decision Rule

Thresholds are fixed in the readout's pre-registration block before FR-1 data is read. Named parameters:

Param	Meaning	Value	Source
N_MIN	activated participants required	20	ROADMAP P0 ("Recruit 20-50 target users")
T_FEEDBACK	cohort median actions/session	1	ROADMAP P0
T_D2	D2 retention floor	TBD (owner: product)	ROADMAP P0 ("agreed threshold"); beachhead §9.2 names no number
T_VALUE	fraction of interviewed participants value-confirmed	TBD (owner: product)	ROADMAP P0; beachhead §9.3 names no number
T_NOISE	max fraction coding the §12.4 kill frame	TBD (owner: product)	beachhead §12.4

Gates: G1 activated >= N_MIN. G2 FR-4 statistic >= T_FEEDBACK. G3 value-confirmed fraction >= T_VALUE. G4 D2 retention >= T_D2. G5 kill-frame fraction < T_NOISE. Verdict, evaluated top-down, first match wins:

1. NO-GO -- G2 or G5 fails (kill-class: beachhead §12.2 feedback ignored, §12.4 another noisy feed).
2. NO-GO -- this is the second iteration cycle (beachhead §12) and any gate fails.
3. EXTEND -- first cycle and only G1, G3, or G4 fails. One additional cycle maximum.
4. EXTEND -- all gates pass but any gate flips under the sensitivity re-run: dropout exclusions applied vs not applied (FR-3 rule 3), and the alternative inactivity timeout (FR-4).
5. GO -- all gates pass and the sensitivity re-run agrees.

Verdicts are computed with G1 evaluated first for reporting: when G1 fails, G2-G5 are still computed and published, labelled provisional.

FR-7: Readout Artifact

Publish docs/planning/p0-validation-readout.md containing: pre-registration block (parameters, commit hash, timestamp preceding first outcome read); input snapshot hashes; evidence table with one row per gate (gate, definition, threshold, observed, PASS/FAIL, source doc); the verdict and the rule clause that produced it; participant accounting (enrolled, activated, re-anchored, withdrawn, unobserved); the intervention ledger -- every manual source-QA and operator action required to keep briefing quality acceptable, with a per-participant-day rate; the §6.2 coding matrix with quotes; and a Falsification section stating explicitly what observation would overturn the conclusion (e.g. for GO: which gate is nearest its threshold and what cohort composition change would flip it).

FR-8: Roadmap Feedback

The verdict must land in the roadmap in the same change as the readout. GO: P0 section marked complete with a link to the readout; P1 unblocked. EXTEND: the P0 ROADMAP section gains the failing gates and the scope of the one permitted additional cycle; P1 stays blocked. NO-GO: the P0 section records the kill-class failure, links the readout, and states the pivot question for the next ponder; P1 Concierge Alpha is not entered. In every case docs/planning/PRODUCT_ROADMAP.md and the ROADMAP P0 block cite the readout, and P1 does not enter preparation while the verdict is EXTEND or NO-GO.

Non-Functional Requirements

• NFR-1: The analysis is a committed script over the frozen snapshot -- no spreadsheet-only steps, no hand-entered aggregates.
• NFR-2: Interview transcripts are stored de-identified; the readout carries participant codes, never names, employers, or contact details.
• NFR-3: The readout states its own limits: cohort of 20-50 self-selected participants over 2 weeks supports a directional GO/NO-GO, not a population estimate.
• NFR-4: All five gates are reported even when an earlier gate already forces the verdict.

Test Strategy

• Reproducibility: run the analysis script twice against the same input snapshot hashes; the gate table and verdict must be byte-identical. A third-party re-run from the snapshot plus the script must reach the same verdict without consulting the analyst.
• Instrumentation trust: reconcile the D2 retention counters against the raw event stream independently; a discrepancy above 0 participants blocks the readout until the source of the divergence is identified (this check is inherited from p0-concierge-pilot-loop's pre-recruitment instrumentation verification).
• Pre-registration check: verify the commit containing the parameter table predates the first commit or access that reads outcome data. If it does not, the verdict is downgraded to EXTEND and the cycle re-run with parameters fixed.
• Coding soundness: every transcript double-coded on the §6.2 rows and the three value axes; report raw agreement per axis; conservative tie-break applied and counted.
• Fixture verdicts: exercise the decision rule against synthetic results tables covering each clause of FR-6 (kill-class fail, second-cycle fail, first-cycle G4-only fail, sensitivity flip, all-pass) and confirm the expected verdict.
• Bias review: a named reviewer who did not operate the pilot argues the NO-GO case in writing against the assembled evidence before the verdict is published; their objections and the responses are appended to the readout. Recommended split: @tidal-researcher argues NO-GO, @tidal-visionary owns the verdict, the pilot operator does not vote.

Dependencies

• p0-target-segment-recruitment -- defines the pilot cohort and its segment criteria; supplies enrollment records and the activation denominator.
• p0-concierge-pilot-loop -- supplies every FR-1 input: behavioral events, feedback action counts, D2 retention counters, operator-intervention log, interview transcripts.
• pg1-instrumented-metrics -- /diagnostics JSON endpoint and retention/feedback counters the behavioral analysis reads.
• docs/planning/ROADMAP.md P0 acceptance criteria and docs/personal-briefing-beachhead.md §6.2, §9, §12 -- the only permitted sources of thresholds.