8.4 KiB
Forage — Vision
"The Situationists called it a dérive — a purposeful drift through a city with no destination, arriving somewhere you did not expect but were glad to find. The entire internet is built for search. Nothing is built for drift."
The Problem With Information Today
Every tool for consuming information on the internet operates in one of two modes:
Pull. You know what you want. You type it into a search box. You get back what you asked for. Google, Wikipedia, Stack Overflow — these are pull systems. They are excellent at answering questions you have already formed. They are useless for questions you have not formed yet.
Push. An algorithm decides what you should see. YouTube, TikTok, Twitter, Reddit — these are push systems. They are optimized for retention, not for you. They know what keeps you watching, not what makes you think. They surface the popular, not the rare. They flatten your interests toward the mean.
There is a third mode: forage. You wander. You follow threads. You find things adjacent to what you know you care about, and occasionally land somewhere genuinely unexpected. Browsing a used bookstore. Walking through a museum with no map. Reading the footnotes. This mode has always been the most rewarding way to learn, and it has never been digitized — because no system could close the feedback loop fast enough to make it feel responsive rather than random.
Until now.
What Forage Is
Forage is a foraging engine. It is not a search engine. It is not a recommendation feed. It is not a news aggregator or an RSS reader or a content briefing. It is an agent that browses the web on your behalf — the long-tail web, the web you would never reach on your own — learns what you actually care about from how you respond, and gets sharply better at finding the right things with each session.
The Chrome extension browses while you live your life. Quietly. A gentle pulse in the sidebar every 30 seconds, indicating it is out there, wandering. When you check back, there are seven items. Not fifty. Seven. Each is a card: title, source, one paragraph explaining why this specific thing might matter to you specifically. Sometimes they land. Sometimes they miss. Every response — click, dwell, save, ignore, skip — is a signal. The next session is better.
The loop:
- Forage browses a new corner of the web, extracts content, embeds it, writes it to tidalDB as a candidate
- You open the sidebar and see seven items, ranked and filtered by your live preference model
- You engage or you do not — both are signals, written instantly to tidalDB
- Your preference vectors shift, the next browsing session targets updated centroids
- Repeat. The system sharpens.
This loop closes in under 100ms. There is no Kafka consumer lagging. No feature store sync to schedule. No cache to invalidate. The signal is written. The model updates. The next query reflects it.
The Insight
Forage is not possible on the 6-system stack.
The feedback loop is too tight. The cold-start problem is too pervasive. Every item Forage finds is new — it has never been in the database before, has never been seen by this user before, has zero engagement signals. A system that cannot handle cold-start at this volume cannot power this product. A system with seams between signal ingestion and ranking cannot close the loop in the time it takes for the user to click one thing and look at the next recommendation.
tidalDB was built for exactly this: write a signal, update the user model, rank the next candidates — in the same process, in under 50ms. Forage is the application that makes this invisible advantage visible. It is the demo that makes the thesis undeniable.
What It Understands
Forage does not build a social graph. It does not track your purchases, your location, or your health. It builds one thing: a map of your curiosity.
Not your consumption habits (Netflix has that). Not your search history (Google has that). Not your social graph (Twitter has that). Your curiosity — what you explore when nothing is telling you what to look at, what you dwell on, what makes you save something for later, what makes you share something with a friend.
This map has structure. tidalDB stores it as:
- 5–8 interest centroids with different activation levels — some hot, some dormant, some emerging
- Relationship weights between you and the creators and sources Forage has introduced you to
- Signal history showing which centroids are growing, which are decaying, which are new
- Cross-centroid connections — patterns that appear at the intersection of two interests you did not know were related
Over time, Forage surfaces things at the intersection of your interests that you did not know intersected. A distributed systems engineer who also cares about jazz finds an article about Brian Eno's use of cybernetic feedback loops in ambient music composition. That is not serendipity. That is the exploration budget doing its job.
The Exploration Model
Forage uses tidalDB's native exploration budget as a multi-armed bandit:
- 86% exploitation — content near active interest centroids, ranked by the
forage_defaultprofile - 14% exploration — content in the gap between centroids, or territory the user has never visited
The exploration arms are not random. They are informed by which unexplored areas are adjacent to confirmed interests, and by which exploration items have historically hit (engaged) vs. missed (ignored) for this user. Users with high exploration-hit rates get a wider budget. Users who prefer depth over breadth get a narrower one. The ratio starts at 14% in P0 and adapts over time.
Exploration items are labeled. The user can see which items are "from the known map" and which are "exploring." Over time, exploration items that hit become part of the known map. Interests that go unengaged decay and eventually become dormant. New interests form from clusters of exploration hits that the user did not see coming.
What It Is Not
Forage is not a research assistant. It does not answer questions. It does not take instructions. You do not tell it what to find.
Forage is not a briefing. It does not summarize the news. It is not trying to keep you informed about the world. It is trying to expand the edges of what you care about.
Forage is not a social product. There is no feed of what your friends found. There is no sharing mechanic (though you might share something yourself). There are no likes, no comments, no metrics of how much other people engaged with something.
Forage is not trying to maximize your time on the app. It is trying to maximize the quality of what you find in the minimum time spent looking for it. Seven items. Not fifty.
The Pattern, Not Just The App
Forage is not a product. It is a proof that a pattern works — and that the pattern can be extracted and embedded in other applications.
The reusable thing is forage-engine: a library crate wrapping tidalDB with the foraging-specific schema, MAB layer, and signal types. Any application that wants a "things you'd never find on your own, getting better as you respond to them" loop can embed forage-engine directly:
- A bookmark manager that surfaces saved items you have forgotten about
- A research tool that finds adjacent papers as you read
- A reading app that learns your depth vs. breadth preferences
- A note-taking tool that resurfaces old notes when you write something related
The forage-server — the Axum HTTP server with the feed page — is the first instantiation. It proves the engine works. It is the demo. The engine is the deliverable.
The Chrome Extension's Role
In P0, the Chrome extension is a light observer. The feed page handles its own signals — every click, skip, and save posts directly to the server via browser fetch(). Claude uses the extension for three things only:
- Navigate to the feed page
- Read the page state before and after interactions
- Report what changed
That is it for P0. The feedback loop runs without the extension, and the extension witnesses it. In later phases, the extension can expand to capture real browsing signals automatically.
The Name
Forage. To search for something, especially food, by roaming widely. To wander and gather. To find what is there rather than what you asked for.
The engine that powers it is tidalDB. The pattern it enables is forage.