tidaldb/applications/forage/plan.md

Forage — Build Plan

What We Are Proving

Each phase proves something specific. Do not build phase N+1 until phase N has proven its thesis.

Phase	Proves	Delivers
P0	The loop closes — signal in, re-rank out, observable in real time	Local server + seed data + Claude observes interactions
P1	The Chrome extension can drive the entire signal surface from real web pages	Extension posts signals automatically from browsing behavior
P2	Semantic search works over content Forage finds on the real web	Embedding service + real web crawl
P3	The MAB sharpens — exploration items hit more often over time	Adaptive exploration budget, centroid tracking, exploration-hit instrumentation
P4	The surprise moment — cross-centroid discoveries emerge naturally	Multi-session preference evolution, intersection surfacing

---

Phase 0 — Close the Loop (MVP Demo)

Goal: A running demo where a user interacts with a local feed page, signals are posted by the page itself, and Claude's Chrome extension observes visible ranking shifts. No real web crawl. No real embeddings. Proves the feedback loop.

What we build:

forage-engine (library crate)

The reusable core. Wraps tidalDB with the foraging-specific schema, seed corpus, MAB layer, and public API. This is what transfers to other applications.

pub struct ForageEngine { db: TidalDb }

impl ForageEngine {
    pub fn ephemeral() -> Result<Self>
    pub fn persistent(data_dir: &Path) -> Result<Self>
    pub fn seed_default_corpus(&self) -> Result<()>
    pub fn signal(&self, user: u64, item: u64, kind: SignalKind) -> Result<()>
    pub fn signal_dwell(&self, user: u64, item: u64, duration_ms: u64) -> Result<()>
    pub fn feed(&self, user: u64, limit: usize) -> Result<Vec<ForageItem>>
    pub fn all_items(&self) -> &[SeedItem]
    pub fn add_item(&self, item: ForageItemInput) -> Result<u64>  // P1
}

forage-server (Axum binary)

A thin HTTP wrapper over forage-engine. Serves:

POST /signal    { user_id, item_id, signal_type, duration_ms? }
GET  /feed      ?user=X&limit=7
GET  /items     (all items, for page render)
GET  /          (serves the feed HTML page)

Runtime mode:

• Persistent by default (~/.forage/data)
• Optional --ephemeral mode for throwaway demo runs

Schema on startup:

• 100 seed items across 8 categories (tech, music, jazz, cooking, fitness, travel, science, literature)
• Each item: title, url (placeholder), category, source, reading_time, description
• Seeded RNG — same items every run, deterministic
• 3 pre-built users: cold (no signals), explorer (light signals), convergent (heavy signals in 2 categories)

Signal types registered:

• view — half-life 7d, AllTime + 24h windows
• dwell — half-life 3d (reading time is stronger signal than click)
• save — half-life 30d (strong intent)
• skip — half-life 1d (mild negative, decays fast)
• share — half-life 14d (strongest positive)

Ranking profiles:

• forage_default — personalized, 14% exploration (~1 in 7), max_per_category:2
• forage_explore — heavy exploration, weighted toward underexplored categories
• forage_converge — pure exploitation, no exploration

MAB layer (thin wrapper over tidalDB query):

candidate_pool = RETRIEVE items FOR USER @u USING PROFILE forage_default LIMIT 20
exploit = candidate_pool[0..6]  // top 6 by score
explore = candidate_pool filtered by (category_signal_count < 5) // pick 1 from underexplored
final = interleave(exploit, explore, ratio=0.14)  // ~1 in 7

Item labels returned in feed response:

• "match" — near a known centroid
• "exploring" — from exploration budget
• "trending" — high velocity regardless of personalization
• "resurfaced" — user had prior engagement, decayed, being re-checked

Feed Page (/)

Static HTML + minimal JS. No framework.

• Grid of 7 item cards
• Each card: title, source, category chip, reading time, description, label badge
• Click card → POST /signal {signal_type: "view"}, open URL in new tab
• Hover for >3s → POST /signal {signal_type: "dwell", duration_ms: N}
• "Skip" button on card → POST /signal {signal_type: "skip"}
• "Save" button → POST /signal {signal_type: "save"}
• Auto-refresh feed every 5s (or on any signal write)
• Visual: ranking shift animation when feed re-orders

What Claude Does in P0

Claude uses the Chrome extension lightly — as an observer, not a puppeteer. The feed page handles signal posting itself via JS fetch(). Claude's role:

1. navigate to localhost:4242 — one call
2. read_page to snapshot the initial feed state — one call
3. Wait while a human (or scripted JS) interacts with the feed for 10+ interactions
4. read_page again to snapshot the final feed state — one call
5. Report: what categories dominated before vs. after, which exploration items appeared, how the labels shifted

Three MCP tool calls per session. That is the ceiling. The interesting loop — signal → re-rank → new feed — runs entirely in the browser without Claude's involvement. Claude observes the outcome, it does not produce it.

This is the demo. This is the proof-of-concept that makes the thesis visible.

Deliverables:

• applications/forage/engine/ — ForageEngine library crate (tidalDB + MAB + schema)
• applications/forage/server/ — thin Axum binary wrapping the engine
• applications/forage/server/static/index.html — feed page (plain HTML/JS, signals via fetch())
• CORS headers on the server so the feed page can post signals without browser errors

---

Phase 1 — Real Signal Surface

Goal: The Chrome extension captures signals from real browsing behavior, not just the demo feed page.

What changes:

Claude uses javascript_tool to inject a lightweight signal collector on pages it navigates to:

// injected on each visited page via javascript_tool
const title = document.title;
const url = location.href;
const readingTime = Math.round(document.body.innerText.split(/\s+/).length / 200);
// POST to forage-server: add item if unknown, write "view" signal
fetch('http://localhost:4242/signal', { method: 'POST', ... });
// After 30s dwell, fire "dwell" signal
setTimeout(() => fetch(...), 30_000);

ForageEngine gains an add_item method — engine API extends to:

pub fn add_item(&self, item: ForageItemInput) -> Result<u64>  // returns item_id

The feed page now shows a mix of:

• Seed items (known corpus)
• Items the user actually visited (added via add_item)

No publishable Chrome extension is built. Claude is the browsing agent. The signal injection is Claude executing JS on pages it visits.

Proves: tidalDB can serve as a memory layer for real browsing behavior, not just a demo corpus.

---

Phase 2 — Real Embeddings

Goal: Semantic search and similarity-based recommendations over content Forage actually finds.

What changes:

A thin embedding sidecar (separate process, any language):

POST /embed  { text: string } → { vector: f32[1536] }

Default: OpenAI text-embedding-3-small. Swappable. Forage calls this when writing new items.

With real embeddings:

• SearchBuilder::semantic("jazz theory") works for real
• SearchBuilder::similar_to(item_id) produces genuine similarity
• Preference vectors actually mean something — they are in embedding space

The feed profile adds:

• semantic_boost: 0.3 — items semantically near preference centroid score higher
• similar_to_saved: true — items near saved items get boosted

Proves: The preference vector is not just a signal frequency map — it is a semantic model of what the user cares about, queryable by meaning.

---

Phase 3 — Adaptive MAB

Goal: The exploration budget adapts per-user based on their exploration-hit history.

What changes:

Track per-user: exploration_hits / exploration_total → hit rate.

if hit_rate > 0.5:  exploration_ratio = 0.25  (adventurous user)
if hit_rate < 0.2:  exploration_ratio = 0.10  (convergent user)
else:               exploration_ratio = 0.14  (default)

UCB1 bonus on underexplored categories:

ucb_bonus = sqrt(2 * ln(total_signals) / category_signal_count)

Categories with few signals get a score boost, naturally surfacing exploration candidates higher.

Instrumentation persists per-user exploration outcomes (exploration_hits, exploration_total) and feeds adaptation logic.

Proves: The MAB is not static noise — it learns the user's exploration tolerance and adjusts. Power users feel the system getting bolder with them.

---

Phase 4 — The Surprise Moment

Goal: Cross-centroid discoveries emerge. Users find things at the intersection of two interests they did not know were related.

What changes:

Centroid intersection query:

centroids = top_2_active_centroids(user)
midpoint = (centroid_a.vector + centroid_b.vector) / 2
intersection_candidates = ANN(midpoint, limit=5)
inject 1 intersection candidate into every 7-item feed
label it: "bridge: {category_a} × {category_b}"

Over time, if intersection items hit consistently, they form a new centroid — the user's interests have genuinely merged into a new territory.

Proves: This is not a feature that could be added to a recommendation system after the fact. It is the natural consequence of having a semantic preference model that updates with the feedback loop. The "surprise moment" is an emergent property of the system working correctly.

---

What We Are Not Building

• A Chrome extension we publish to the Chrome Web Store (P0 is Claude-driven, not user-installed)
• A mobile app
• Multi-user / server-hosted version (single user, local process)
• Content moderation, NSFW filtering, language filtering
• Payment, accounts, authentication
• A scraper that violates robots.txt or rate limits

---

Phase 0 Acceptance Criteria

The P0 demo is complete when:

1. cargo run -p forage-server --manifest-path applications/forage/server/Cargo.toml starts a server at localhost:4242
2. The feed page loads with 7 items across ≥3 categories
3. A user generates 10+ signals from the feed page (mix of views, skips, saves) while Claude observes before/after state
4. After 10 signals, the feed has visibly shifted toward the signaled categories
5. At least 1 item in the feed is labeled exploring (from the exploration budget)
6. The signal-to-re-rank latency is < 200ms (measured by feed refresh after POST /signal)
7. A second user (?user=2) with no signals gets a different, more exploratory feed than a user with 20+ signals

If these 7 criteria are met, the loop is closed and the thesis is proven.