tidaldb/.sdlc/features/p0-concierge-pilot-loop/spec.md

p0-concierge-pilot-loop: Concierge Pilot Loop

Problem Statement

P0 must answer "do users care enough to return?" (docs/planning/ROADMAP.md, P0 milestone thesis). That answer requires a running daily briefing in front of a real pilot cohort for long enough to observe return behavior, plus a record trustworthy enough to defend a GO/NO-GO decision for P1.

The pilot is a concierge operation, not a product. docs/personal-briefing-beachhead.md §10 Phase A specifies "daily brief with strong manual QA on source quality and reasons," and the ROADMAP P0 acceptance criterion states the daily briefing prototype "can include manual source QA." Human intervention is therefore permitted — but unrecorded intervention silently inflates quality results and destroys the readout. Without a defined loop, a fixed briefing contract, verified instrumentation, and an intervention ledger, this pilot produces anecdotes rather than evidence.

Goals

1. Run a 2-week daily briefing for the pilot cohort (20-50 users, per ROADMAP P0) at a fixed daily cutoff, every day, with no skipped days.
2. Gate every briefing on manual source QA performed by a named operator before delivery, per ROADMAP P0 ("can include manual source QA").
3. Ship a briefing artifact each user can act on: ranked items, reason labels, source links, and the five feedback action controls (more/less/hide/mute/save).
4. Instrument each session so median feedback actions per session and D2 retention are computable from recorded events, not reconstructed after the fact.
5. Interview every cohort member weekly against a fixed question set drawn from beachhead §6.2.
6. Record every operator intervention so quality results can be re-read with concierge assistance discounted.
7. Hand p0-validation-readout a complete dataset covering all 14 pilot days.

Non-Goals

• Recruiting, screening, or onboarding the pilot cohort — owned by p0-target-segment-recruitment.
• Computing the GO/NO-GO decision or comparing results to thresholds — owned by p0-validation-readout.
• Building new engine capability. The pilot consumes existing surfaces (signal writes, reason labels, /diagnostics); anything missing is served manually by the operator and logged as intervention.
• Self-serve onboarding, time-budget mode, and cohort view — P1/P2 scope.
• Automating source ingestion or QA. Manual is explicitly in scope for P0.

Functional Requirements

FR-1: Daily Loop and Fixed Cutoff

Each pilot day runs: source ingestion -> candidate ranking -> manual source QA -> delivery -> session capture. Delivery occurs at one fixed local-morning cutoff time, identical every day for all users (cutoff clock time: TBD (owner: product)). Beachhead §5.2 defines the morning brief as the daily loop; midday/evening updates are out of scope. A briefing not delivered by cutoff is a missed day (FR-8), never a late day.

FR-2: Manual Source QA Gate

Before delivery, a named operator reviews the ranked candidate set and (a) removes items failing source quality, (b) verifies each reason label is specific and true, (c) verifies no single source dominates the top 10 (beachhead §6.3 "one source dominates repeatedly"). No briefing is delivered without a recorded QA pass. QA is authorized by ROADMAP P0; each QA edit is an intervention under FR-7.

FR-3: Briefing Artifact Contract

Every daily briefing carries 10-20 ranked items (beachhead §5.1.3). Every card carries: rank position, item title, a reason label drawn from the fixed label set (e.g. "Trending in your cohort", "Matches your priority", "New source for exploration" — beachhead §5.1.4), a one-tap source link, and controls for all five feedback actions. more/less apply to topic affinity, hide to the item's topic, mute to the source, save to the user's library. Each control emits exactly one feedback action event bound to (user, briefing_day, rank_position, item, action).

FR-4: Feedback Action Capture

Each feedback action is written as a typed signal with the action name as its signal type, so the pilot reads counts directly off the existing pg1 surface tidaldb_signal_writes_by_type{signal_type="more"|"less"|"hide"|"mute"|"save"} (pg1-instrumented-metrics FR-2). No parallel counting path is built. Per-user recency comes from the existing user signal timestamp map (pg1 FR-6); loop closure evidence comes from the existing feedback-loop latency histogram (pg1 FR-4).

FR-5: Per-Session Instrumentation and D2 Retention

Session boundary (normative — p0-validation-readout FR-2 defers to this definition). A session opens when a user first opens that day's briefing, and closes at the earlier of an explicit close/navigation-away or an inactivity timeout of TBD (owner: product), frozen before day 1 and never changed mid-run. A re-open after close within the same briefing_day is a new session, so a user may have multiple sessions per briefing day (beachhead §9.2 "average sessions per active day"). Every session carries actor=participant or actor=operator; operator and QA sessions are tagged at open and excluded from every cohort metric.

Per session the pilot records: user_id, briefing_day, actor, session open timestamp, session close timestamp, items opened, and the ordered list of feedback actions. Sessions are a derived view, not the primitive: every interaction (open, card open, each feedback action, close) is stored with its own raw timestamp, so the whole run can be re-sessionized under an alternative inactivity timeout without re-collecting data (p0-validation-readout FR-4 sensitivity re-run). Two derived measures are produced daily and are the only measures the pilot itself computes:

• Feedback actions per session — median across the pilot cohort, against the ROADMAP P0 bar of >= 1 per session for the median user.
• D2 retention — fraction of users with >= 1 participant session on briefing day N who also have >= 1 participant session on day N+1, computed per day-pair. Counted on distinct briefing_day values, so D2 retention is independent of the inactivity timeout; only the per-session median above is timeout-sensitive.

A /diagnostics JSON snapshot (pg1 FR-5) is captured immediately after each daily cutoff and stored with the day's session records, giving an independent counter read per pilot day.

FR-6: Interview Cadence and Fixed Question Set

Each cohort member is interviewed twice: at the end of week 1 and at the end of week 2 (beachhead §10 Phase A, "after each week"). Interviews use one fixed question set derived from beachhead §6.2, asked verbatim and in order: why not just current feeds; was setup too much; can this be trusted; does it feel repetitive or narrow; when you said less, did anything actually change; does it respect your time; would you use it for a work decision; are you comfortable with the data it holds. Responses are recorded verbatim; no interpretation happens during the interview.

FR-7: Operator Runbook and Intervention Ledger

The operator runbook covers, per day: ingestion start, QA pass, delivery confirmation, and end-of-day session-record reconciliation. Every deviation from the automated output is an intervention with a ledger row: timestamp, pilot day, affected users, intervention class (source_removed, reason_label_rewritten, item_reordered, diversity_forced, briefing_hand_assembled, delivery_manual), and free-text cause. When QA finds a bad source, the source is muted for the remainder of the pilot, the affected cards are removed, and the removal is logged — so that day's quality numbers can be recomputed with operator-touched cards excluded.

FR-8: Abort Conditions

The pilot stops early and escalates to p0-validation-readout if a beachhead §6.3 Critical failure mode is confirmed cohort-wide: (a) feedback actions demonstrably not reflected in the next refresh, or (b) the feed remains noisy after 2 days with no measurable improvement. It also stops if instrumentation loss makes FR-5 uncomputable for more than 2 pilot days, or if >= 3 of 14 days are missed days. An aborted pilot still hands over its partial dataset, labeled aborted with the triggering condition.

FR-9: Readout Dataset Handoff

At day 14 the dataset is frozen — no rows added, corrected, or recoded after handoff — and p0-validation-readout receives: the roster and cohort segment tags (from p0-target-segment-recruitment); all 14 daily briefing manifests; the raw timestamped interaction stream; all session records derived from it under the FR-5 session boundary, carrying actor tags; all feedback action events with per-session counts by type (more/less/hide/mute/save); the per-day-pair D2 retention counters; the daily /diagnostics snapshots; the intervention ledger; the missed-day log; and all interview transcripts with their coding. The frozen inactivity timeout is handed over with the data so the readout can re-sessionize under an alternative value.

Non-Functional Requirements

• NFR-1: Instrumentation is verified end-to-end before day 1; no event schema changes during the 2-week run.
• NFR-2: Session records and the intervention ledger are append-only; corrections are new rows, never edits.
• NFR-3: Interview transcripts are stored under participant IDs only — no participant names or contact details in pilot artifacts.
• NFR-4: Operator time per day is bounded and logged, so the concierge cost of one briefing is a known input to the GO/NO-GO decision.

Test Strategy

• Pre-pilot dry run: 2 consecutive days with the operator team as stand-in users. Exercise the full loop including QA gate, delivery, and all five feedback actions. Dry-run data is discarded and never merged into pilot results.
• Instrumentation verification before day 1: fire one of each feedback action and confirm each increments its tidaldb_signal_writes_by_type counter and appears in the /diagnostics snapshot; confirm a session open and close emit the FR-5 boundary events with the correct actor tag, that an inactivity-timeout close and a re-open produce two distinct sessions on one briefing_day, and that a synthetic two-day session pattern yields the expected D2 retention value. Recruiting does not release the pilot cohort until this passes.
• Interview coding: two coders independently tag each transcript against the §6.2 question set and the §6.3 failure modes; disagreements are resolved by re-reading the verbatim answer, not by discussion. Coding scheme is fixed before the first interview.
• Intervention accounting: each day, every operator-touched card is reconcilable to a ledger row. Every reported quality figure is produced twice — all cards, and operator-touched cards excluded. A divergence between the two is a finding, not an error to smooth over.
• Missed-day handling: a day with no delivered briefing is recorded as a missed day with cause, and enters retention math as a no-briefing day for every user. Missed days are never dropped, backfilled, or interpolated.
• Pre-registered before day 1: cohort size 20-50 (ROADMAP), duration 14 days, median >= 1 feedback action per session (ROADMAP), the FR-5 inactivity timeout, and D2 retention threshold TBD (owner: product) — the ROADMAP states "agreed threshold" without a value.

Dependencies

• p0-target-segment-recruitment (upstream) — supplies the enrolled pilot cohort, segment tags, and interest configuration; the pilot cannot start day 1 without it.
• p0-validation-readout (downstream) — consumes the FR-9 dataset and issues the GO/NO-GO decision for P1.
• pg1-instrumented-metrics — tidaldb_signal_writes_by_type counters (FR-2), feedback-loop latency histogram (FR-4), /diagnostics JSON endpoint (FR-5), user signal timestamp map (FR-6).
• docs/personal-briefing-beachhead.md §5.2 daily loop, §6.2 core user questions, §6.3 failure modes, §10 Phase A.
• docs/planning/ROADMAP.md P0 acceptance criteria.