0634d2f4dc
P0 was the only milestone gating the product track and all three of its features sat in `draft` with no spec, while M9/M10/P1/PG1 are released. Engine work was running ahead of the validation that decides whether any of it is wanted. - p0-target-segment-recruitment: screening criteria per beachhead persona, a funnel sized to yield the 20-50 pilot cohort, outreach limits (no accuracy or onboarding promise the prototype cannot meet), consent and data handling, opaque participant ids only, segment balance, and a pre-pilot baseline-feed question so the readout has a control. - p0-concierge-pilot-loop: the 14-day daily loop with the manual source-QA gate the ROADMAP permits, the briefing card contract, the normative session boundary, instrumentation bound to existing pg1 surfaces (signal-type counters, feedback-loop histogram, /diagnostics snapshots) rather than a new counting path, weekly interviews on the fixed beachhead question set, an intervention ledger so concierge help cannot silently inflate quality, abort conditions, and the frozen handoff dataset. - p0-validation-readout: pre-registered GO/NO-GO/EXTEND rule over five gates with an explicit dropout/missed-day/partial-observation policy, double-coded interviews against the beachhead required answers, a sensitivity re-run that downgrades GO to EXTEND if the verdict flips, a falsification section, and a named reviewer who must argue the NO-GO case before publication. Every threshold traces to a ROADMAP P0 acceptance criterion or the beachhead doc; the three that neither document fixes (D2 retention floor, value-confirmed fraction, noise kill-frame ceiling) are marked TBD (owner: product) instead of being invented. Next directive for all three is create_design.
95 lines
12 KiB
Markdown
95 lines
12 KiB
Markdown
# p0-concierge-pilot-loop: Concierge Pilot Loop
|
|
|
|
## Problem Statement
|
|
|
|
P0 must answer "do users care enough to return?" (`docs/planning/ROADMAP.md`, P0 milestone thesis). That answer requires a running daily briefing in front of a real pilot cohort for long enough to observe return behavior, plus a record trustworthy enough to defend a GO/NO-GO decision for P1.
|
|
|
|
The pilot is a concierge operation, not a product. `docs/personal-briefing-beachhead.md` §10 Phase A specifies "daily brief with strong manual QA on source quality and reasons," and the ROADMAP P0 acceptance criterion states the daily briefing prototype "can include manual source QA." Human intervention is therefore permitted — but unrecorded intervention silently inflates quality results and destroys the readout. Without a defined loop, a fixed briefing contract, verified instrumentation, and an intervention ledger, this pilot produces anecdotes rather than evidence.
|
|
|
|
## Goals
|
|
|
|
1. **Run a 2-week daily briefing** for the pilot cohort (20-50 users, per ROADMAP P0) at a fixed daily cutoff, every day, with no skipped days.
|
|
2. **Gate every briefing on manual source QA** performed by a named operator before delivery, per ROADMAP P0 ("can include manual source QA").
|
|
3. **Ship a briefing artifact** each user can act on: ranked items, reason labels, source links, and the five feedback action controls (`more`/`less`/`hide`/`mute`/`save`).
|
|
4. **Instrument each session** so median feedback actions per session and D2 retention are computable from recorded events, not reconstructed after the fact.
|
|
5. **Interview every cohort member weekly** against a fixed question set drawn from beachhead §6.2.
|
|
6. **Record every operator intervention** so quality results can be re-read with concierge assistance discounted.
|
|
7. **Hand `p0-validation-readout` a complete dataset** covering all 14 pilot days.
|
|
|
|
## Non-Goals
|
|
|
|
- Recruiting, screening, or onboarding the pilot cohort — owned by `p0-target-segment-recruitment`.
|
|
- Computing the GO/NO-GO decision or comparing results to thresholds — owned by `p0-validation-readout`.
|
|
- Building new engine capability. The pilot consumes existing surfaces (signal writes, reason labels, `/diagnostics`); anything missing is served manually by the operator and logged as intervention.
|
|
- Self-serve onboarding, time-budget mode, and cohort view — P1/P2 scope.
|
|
- Automating source ingestion or QA. Manual is explicitly in scope for P0.
|
|
|
|
## Functional Requirements
|
|
|
|
### FR-1: Daily Loop and Fixed Cutoff
|
|
|
|
Each pilot day runs: source ingestion -> candidate ranking -> manual source QA -> delivery -> session capture. Delivery occurs at one fixed local-morning cutoff time, identical every day for all users (cutoff clock time: TBD (owner: product)). Beachhead §5.2 defines the morning brief as the daily loop; midday/evening updates are out of scope. A briefing not delivered by cutoff is a missed day (FR-8), never a late day.
|
|
|
|
### FR-2: Manual Source QA Gate
|
|
|
|
Before delivery, a named operator reviews the ranked candidate set and (a) removes items failing source quality, (b) verifies each reason label is specific and true, (c) verifies no single source dominates the top 10 (beachhead §6.3 "one source dominates repeatedly"). No briefing is delivered without a recorded QA pass. QA is authorized by ROADMAP P0; each QA edit is an intervention under FR-7.
|
|
|
|
### FR-3: Briefing Artifact Contract
|
|
|
|
Every daily briefing carries 10-20 ranked items (beachhead §5.1.3). Every card carries: rank position, item title, a reason label drawn from the fixed label set (e.g. "Trending in your cohort", "Matches your <topic> priority", "New source for exploration" — beachhead §5.1.4), a one-tap source link, and controls for all five feedback actions. `more`/`less` apply to topic affinity, `hide` to the item's topic, `mute` to the source, `save` to the user's library. Each control emits exactly one feedback action event bound to `(user, briefing_day, rank_position, item, action)`.
|
|
|
|
### FR-4: Feedback Action Capture
|
|
|
|
Each feedback action is written as a typed signal with the action name as its signal type, so the pilot reads counts directly off the existing pg1 surface `tidaldb_signal_writes_by_type{signal_type="more"|"less"|"hide"|"mute"|"save"}` (`pg1-instrumented-metrics` FR-2). No parallel counting path is built. Per-user recency comes from the existing user signal timestamp map (pg1 FR-6); loop closure evidence comes from the existing feedback-loop latency histogram (pg1 FR-4).
|
|
|
|
### FR-5: Per-Session Instrumentation and D2 Retention
|
|
|
|
**Session boundary (normative — `p0-validation-readout` FR-2 defers to this definition).** A session opens when a user first opens that day's briefing, and closes at the earlier of an explicit close/navigation-away or an inactivity timeout of TBD (owner: product), frozen before day 1 and never changed mid-run. A re-open after close within the same `briefing_day` is a new session, so a user may have multiple sessions per briefing day (beachhead §9.2 "average sessions per active day"). Every session carries `actor=participant` or `actor=operator`; operator and QA sessions are tagged at open and excluded from every cohort metric.
|
|
|
|
Per session the pilot records: `user_id`, `briefing_day`, `actor`, session open timestamp, session close timestamp, items opened, and the ordered list of feedback actions. Sessions are a derived view, not the primitive: every interaction (open, card open, each feedback action, close) is stored with its own raw timestamp, so the whole run can be re-sessionized under an alternative inactivity timeout without re-collecting data (`p0-validation-readout` FR-4 sensitivity re-run). Two derived measures are produced daily and are the only measures the pilot itself computes:
|
|
|
|
- **Feedback actions per session** — median across the pilot cohort, against the ROADMAP P0 bar of >= 1 per session for the median user.
|
|
- **D2 retention** — fraction of users with >= 1 participant session on briefing day N who also have >= 1 participant session on day N+1, computed per day-pair. Counted on distinct `briefing_day` values, so D2 retention is independent of the inactivity timeout; only the per-session median above is timeout-sensitive.
|
|
|
|
A `/diagnostics` JSON snapshot (pg1 FR-5) is captured immediately after each daily cutoff and stored with the day's session records, giving an independent counter read per pilot day.
|
|
|
|
### FR-6: Interview Cadence and Fixed Question Set
|
|
|
|
Each cohort member is interviewed twice: at the end of week 1 and at the end of week 2 (beachhead §10 Phase A, "after each week"). Interviews use one fixed question set derived from beachhead §6.2, asked verbatim and in order: why not just current feeds; was setup too much; can this be trusted; does it feel repetitive or narrow; when you said `less`, did anything actually change; does it respect your time; would you use it for a work decision; are you comfortable with the data it holds. Responses are recorded verbatim; no interpretation happens during the interview.
|
|
|
|
### FR-7: Operator Runbook and Intervention Ledger
|
|
|
|
The operator runbook covers, per day: ingestion start, QA pass, delivery confirmation, and end-of-day session-record reconciliation. Every deviation from the automated output is an intervention with a ledger row: timestamp, pilot day, affected users, intervention class (`source_removed`, `reason_label_rewritten`, `item_reordered`, `diversity_forced`, `briefing_hand_assembled`, `delivery_manual`), and free-text cause. When QA finds a bad source, the source is muted for the remainder of the pilot, the affected cards are removed, and the removal is logged — so that day's quality numbers can be recomputed with operator-touched cards excluded.
|
|
|
|
### FR-8: Abort Conditions
|
|
|
|
The pilot stops early and escalates to `p0-validation-readout` if a beachhead §6.3 Critical failure mode is confirmed cohort-wide: (a) feedback actions demonstrably not reflected in the next refresh, or (b) the feed remains noisy after 2 days with no measurable improvement. It also stops if instrumentation loss makes FR-5 uncomputable for more than 2 pilot days, or if >= 3 of 14 days are missed days. An aborted pilot still hands over its partial dataset, labeled aborted with the triggering condition.
|
|
|
|
### FR-9: Readout Dataset Handoff
|
|
|
|
At day 14 the dataset is frozen — no rows added, corrected, or recoded after handoff — and `p0-validation-readout` receives: the roster and cohort segment tags (from `p0-target-segment-recruitment`); all 14 daily briefing manifests; the raw timestamped interaction stream; all session records derived from it under the FR-5 session boundary, carrying `actor` tags; all feedback action events with per-session counts by type (`more`/`less`/`hide`/`mute`/`save`); the per-day-pair D2 retention counters; the daily `/diagnostics` snapshots; the intervention ledger; the missed-day log; and all interview transcripts with their coding. The frozen inactivity timeout is handed over with the data so the readout can re-sessionize under an alternative value.
|
|
|
|
## Non-Functional Requirements
|
|
|
|
- **NFR-1**: Instrumentation is verified end-to-end before day 1; no event schema changes during the 2-week run.
|
|
- **NFR-2**: Session records and the intervention ledger are append-only; corrections are new rows, never edits.
|
|
- **NFR-3**: Interview transcripts are stored under participant IDs only — no participant names or contact details in pilot artifacts.
|
|
- **NFR-4**: Operator time per day is bounded and logged, so the concierge cost of one briefing is a known input to the GO/NO-GO decision.
|
|
|
|
## Test Strategy
|
|
|
|
- **Pre-pilot dry run**: 2 consecutive days with the operator team as stand-in users. Exercise the full loop including QA gate, delivery, and all five feedback actions. Dry-run data is discarded and never merged into pilot results.
|
|
- **Instrumentation verification before day 1**: fire one of each feedback action and confirm each increments its `tidaldb_signal_writes_by_type` counter and appears in the `/diagnostics` snapshot; confirm a session open and close emit the FR-5 boundary events with the correct `actor` tag, that an inactivity-timeout close and a re-open produce two distinct sessions on one `briefing_day`, and that a synthetic two-day session pattern yields the expected D2 retention value. Recruiting does not release the pilot cohort until this passes.
|
|
- **Interview coding**: two coders independently tag each transcript against the §6.2 question set and the §6.3 failure modes; disagreements are resolved by re-reading the verbatim answer, not by discussion. Coding scheme is fixed before the first interview.
|
|
- **Intervention accounting**: each day, every operator-touched card is reconcilable to a ledger row. Every reported quality figure is produced twice — all cards, and operator-touched cards excluded. A divergence between the two is a finding, not an error to smooth over.
|
|
- **Missed-day handling**: a day with no delivered briefing is recorded as a missed day with cause, and enters retention math as a no-briefing day for every user. Missed days are never dropped, backfilled, or interpolated.
|
|
- Pre-registered before day 1: cohort size 20-50 (ROADMAP), duration 14 days, median >= 1 feedback action per session (ROADMAP), the FR-5 inactivity timeout, and D2 retention threshold TBD (owner: product) — the ROADMAP states "agreed threshold" without a value.
|
|
|
|
## Dependencies
|
|
|
|
- `p0-target-segment-recruitment` (upstream) — supplies the enrolled pilot cohort, segment tags, and interest configuration; the pilot cannot start day 1 without it.
|
|
- `p0-validation-readout` (downstream) — consumes the FR-9 dataset and issues the GO/NO-GO decision for P1.
|
|
- `pg1-instrumented-metrics` — `tidaldb_signal_writes_by_type` counters (FR-2), feedback-loop latency histogram (FR-4), `/diagnostics` JSON endpoint (FR-5), user signal timestamp map (FR-6).
|
|
- `docs/personal-briefing-beachhead.md` §5.2 daily loop, §6.2 core user questions, §6.3 failure modes, §10 Phase A.
|
|
- `docs/planning/ROADMAP.md` P0 acceptance criteria.
|