docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality
- API.md: document `similar_to`/`region`/`unavailable_shards` on /feed and /search, the new POST /vector_search k-NN probe, and the cluster-node-only routes (/cluster/*, /sharded/*, /hardnegs) - CHANGELOG.md: M12 entries — multi-vector preference + ANN candidate-gen, idle-readiness + TLS scale-up (m12p5/p6), sharded ingestion (m12p4) - ROADMAP.md: mark M11 + M12 COMPLETE; restate the v1.0 bar (30-day-green nightly calendar + Ref-A/k3s throughput re-runs) - prometheus-alerts.yaml: add ship-stall, quorum-lag, divergence-quarantine, reseed-pending, and snapshot-pin-force-drop cluster alerts - check-docs.sh: self-updating milestone-status freshness guard derived from ROADMAP's latest COMPLETE milestone - refresh specs (00-14), ai-lookup, guides, and runbooks to M0-M12
This commit is contained in:
jx12n
parent
6a937fc4bc
commit
4051077cff
100
API.md
100
API.md
@ -970,10 +970,13 @@ Retrieve a ranked feed.
|
||||
|---|---|---|---|
|
||||
| `user_id` | `u64` | — | User for personalization (optional) |
|
||||
| `profile` | `string` | `"for_you"` | Ranking profile name |
|
||||
| `limit` | `u32` | `20` | Max results |
|
||||
| `limit` | `u32` | `20` | Max results (clamped to 1000) |
|
||||
| `similar_to` | `u64` | — | Seed item for "more like this": with `profile=related`, sources candidates by ANN nearest-neighbor over this item's embedding |
|
||||
| `region` | `string` | — | Target region (cluster mode only; rejected with `400` in standalone) |
|
||||
|
||||
```
|
||||
GET /feed?user_id=123&profile=trending&limit=25
|
||||
GET /feed?profile=related&similar_to=42&limit=20
|
||||
```
|
||||
|
||||
**Response:**
|
||||
@ -991,11 +994,18 @@ GET /feed?user_id=123&profile=trending&limit=25
|
||||
]
|
||||
}
|
||||
],
|
||||
"total_candidates": 1000
|
||||
"total_candidates": 1000,
|
||||
"region": null,
|
||||
"unavailable_shards": ["group-2"]
|
||||
}
|
||||
```
|
||||
|
||||
The `signals` field is omitted when empty.
|
||||
The `signals` field is omitted when empty. `region` is the region the feed was
|
||||
served from in cluster mode (`null` standalone). `unavailable_shards` is present
|
||||
**only** when a cross-shard read was degraded — it lists the shard groups that
|
||||
could not be reached, so a partial page (fewer items, lower `total_candidates`)
|
||||
is never silently indistinguishable from a complete one. It is omitted on a
|
||||
complete read.
|
||||
|
||||
#### `GET /search`
|
||||
|
||||
@ -1005,7 +1015,8 @@ Text search with optional personalization.
|
||||
|---|---|---|---|
|
||||
| `query` | `string` | — | Search text (**required**) |
|
||||
| `user_id` | `u64` | — | User for personalization (optional) |
|
||||
| `limit` | `u32` | `20` | Max results |
|
||||
| `limit` | `u32` | `20` | Max results (clamped to 1000) |
|
||||
| `region` | `string` | — | Target region (cluster mode only; rejected with `400` in standalone) |
|
||||
|
||||
```
|
||||
GET /search?query=jazz+piano&user_id=123&limit=10
|
||||
@ -1024,11 +1035,59 @@ GET /search?query=jazz+piano&user_id=123&limit=10
|
||||
"semantic_score": 0.92
|
||||
}
|
||||
],
|
||||
"total_candidates": 50
|
||||
"total_candidates": 50,
|
||||
"region": null,
|
||||
"unavailable_shards": ["group-2"]
|
||||
}
|
||||
```
|
||||
|
||||
`bm25_score` and `semantic_score` are omitted when not applicable.
|
||||
`bm25_score` and `semantic_score` are omitted when not applicable. `region` and
|
||||
`unavailable_shards` behave exactly as on [`GET /feed`](#get-feed): `region` is
|
||||
the serving region in cluster mode (`null` standalone), and `unavailable_shards`
|
||||
is present only when a cross-shard read was degraded (partial page) and omitted
|
||||
on a complete read.
|
||||
|
||||
#### `POST /vector_search`
|
||||
|
||||
Pure k-NN ANN probe over the item content vector slot — **no** profile scoring,
|
||||
fusion, or diversity. This is the recall-measurement / raw nearest-neighbor
|
||||
surface (used by the `tidal-stress --verify-recall` harness).
|
||||
|
||||
**Request body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"vector": [0.013, -0.41, 0.22],
|
||||
"k": 10,
|
||||
"ef_search": 200
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `vector` | `f32[]` | — | Dense query vector; must match the item content slot's dimensionality (**required**) |
|
||||
| `k` | `u32` | `10` | Number of nearest neighbors to return (clamped to 1000) |
|
||||
| `ef_search` | `u32` | — | Optional per-request HNSW beam-width override (the recall/latency knob); omitted = the slot's configured default |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{"entity_id": 42, "distance": 0.018},
|
||||
{"entity_id": 17, "distance": 0.041}
|
||||
],
|
||||
"region": null,
|
||||
"unavailable_shards": ["group-2"]
|
||||
}
|
||||
```
|
||||
|
||||
Each match carries `entity_id` and `distance` (L2-squared distance from the query
|
||||
vector, lower = more similar; for the L2-normalized vectors tidalDB stores this
|
||||
lies in `[0.0, 4.0]` and is monotonic with cosine distance), ordered closest-first.
|
||||
`region` is always `null` here (the probe serves locally — standalone, or merged
|
||||
across a node's hosted shard groups in cluster mode). `unavailable_shards` is
|
||||
present only when a cross-shard probe was degraded (partial nearest-set).
|
||||
|
||||
### Health Endpoints
|
||||
|
||||
@ -1055,6 +1114,35 @@ The cluster server serves a superset document that also includes the
|
||||
`/cluster/*` and `/sharded/*` routes. See
|
||||
[docs/guides/server-deployment.md](docs/guides/server-deployment.md).
|
||||
|
||||
### Cluster Endpoints
|
||||
|
||||
These routes exist **only on cluster (region) nodes** — they are absent from the
|
||||
standalone server. Status routes are read-only; the rebalance verbs require an
|
||||
admin API key. Full operational detail is in
|
||||
[docs/guides/server-deployment.md](docs/guides/server-deployment.md) and
|
||||
[docs/runbooks/kubernetes.md](docs/runbooks/kubernetes.md).
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `GET` | `/cluster/status` | Aggregated cluster view — leader, relay high-water-mark, per-region lag, and a per-shard-group `shards` array |
|
||||
| `GET` | `/cluster/status/local` | This node's local view — its hosted shard groups and which leaderships it holds |
|
||||
| `POST` | `/cluster/shards/{id}/transfer` | Transfer shard group `{id}`'s leadership to a named replica (fenced promote; body `{"region": "<name>"}`) |
|
||||
| `POST` | `/cluster/shards/{id}/replicas` | Add or remove a replica of shard group `{id}` (body `{"action": "add"\|"remove", "name": ..., "grpc_addr": ..., "http_addr": ...}`; addrs required for `add`) |
|
||||
|
||||
The cluster node also exposes the membership/recovery verbs `POST /cluster/join`,
|
||||
`/cluster/promote`, `/cluster/members{,/remove}`, `/cluster/partition`, `/cluster/heal`,
|
||||
`/cluster/catchup`, `/cluster/reseed`, and `/cluster/reconcile`, plus the sharded
|
||||
data surface `POST /sharded/{items,embeddings,signals}` and `GET /sharded/{feed,search}`.
|
||||
|
||||
### Cluster-Node-Only Data Endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `POST` | `/hardnegs` | Record a hide hard-negative for `(user, item)` (body `{"user_id": ..., "item_id": ...}`); `204 No Content`. Cluster-node-only — converges via the `/cluster/reconcile` LWW snapshot |
|
||||
|
||||
> `POST /vector_search` (documented above) is served by **both** the standalone
|
||||
> and cluster servers.
|
||||
|
||||
**`GET /health` response:**
|
||||
|
||||
```json
|
||||
|
||||
@ -255,7 +255,7 @@ This matches how ScyllaDB uses USearch in production and how Weaviate and Qdrant
|
||||
|
||||
User interest is not a single vector. Averaging engagement embeddings across topics ("hiking," "cooking," "cars") produces a centroid that represents none of them (PinnerSage, KDD 2020). Instead, each user's preference is represented as up to `K_MAX=10` interest cluster centroids, maintained online by the database as signals arrive.
|
||||
|
||||
**As built** (`entities/multi_preference.rs`, design in `docs/research/multi-vector-preference.md`): clusters are maintained by **online sequential k-means with a DP-means threshold split** — a new interaction either updates its nearest cluster (per-cluster adaptive EMA, `alpha = base / (1 + ln(count + 1))`) or, if it exceeds the split threshold and `K_MAX` is not reached, opens a new cluster; at the cap the nearest cluster absorbs it (no eviction). Per-cluster *importance* composes the canonical forward-decay kernel (`signals/decay.rs`), anchored to each engagement's event timestamp, so stale interests fade. At query time the `for_you` path selects the top-`M` clusters by current importance, issues `M` ANN queries **sequentially** (`candidate_gen::ann_candidates_multi`; the loop is parallelizable but not yet parallelized), and merges by **best (min) distance** — not by score; the personalization boost is the max cosine over **all** the user's clusters. Stage-3 re-ranking is unchanged. Users below `COLD_START_N=5` interactions fall back to a **single** adaptive-LR vector (`entities/preference.rs`) — the documented cold-start tier.
|
||||
**As built** (`entities/multi_preference.rs`, design in `docs/research/multi-vector-preference.md`): clusters are maintained by **online sequential k-means with a DP-means threshold split** — a new interaction either updates its nearest cluster (per-cluster adaptive EMA, `alpha = base / (1 + ln(count + 1))`) or, if it exceeds the split threshold and `K_MAX` is not reached, opens a new cluster; at the cap the nearest cluster absorbs it (no eviction). Per-cluster *importance* composes the canonical forward-decay kernel (`signals/decay.rs`), anchored to each engagement's event timestamp, so stale interests fade. At query time the `for_you` path selects the top-`M` clusters by current importance, issues `M` ANN queries **sequentially** by design (`candidate_gen::ann_candidates_multi`; parallelizable — that optimization is deferred), and merges by **best (min) distance** — not by score; the personalization boost is the max cosine over **all** the user's clusters. Stage-3 re-ranking is unchanged. Users below `COLD_START_N=5` interactions fall back to a **single** adaptive-LR vector (`entities/preference.rs`) — the documented cold-start tier.
|
||||
|
||||
**Aspirational:** PinnerSage *proper* uses **medoids** (actual item embeddings, not maintained centroids) computed by **offline Ward hierarchical clustering**. That batch tier is disqualified for an embeddable single-node DB today. The shipped `Tag::Preference` per-cluster row already carries the `anchor_ts` / `importance_at_anchor` decay state a periodic in-process medoid recluster would reset, so *that* field-level change needs no migration; the recluster's other input — a bounded per-user interaction-embedding window — lands as an additive `Tag::PreferenceWindow` row (reserved in `storage/keys.rs`, not yet populated), never a rewrite of existing rows.
|
||||
|
||||
|
||||
39
CHANGELOG.md
39
CHANGELOG.md
@ -6,6 +6,45 @@ All notable changes to tidalDB will be documented in this file.
|
||||
|
||||
### Added
|
||||
|
||||
**Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`**
|
||||
|
||||
- **Online preference clustering (`entities/multi_preference.rs`).** A warm user
|
||||
(≥ `COLD_START_N = 5` interactions) maintains up to `K_MAX` preference clusters
|
||||
built by online sequential k-means with a DP-means threshold split: a new
|
||||
engagement updates its nearest cluster (per-cluster adaptive EMA) or, past the
|
||||
split threshold and under the cap, opens a new cluster; at the cap the nearest
|
||||
cluster absorbs it. Per-cluster *importance* composes the canonical forward-decay
|
||||
kernel anchored to each engagement's timestamp, so stale interests fade.
|
||||
- **Top-M ANN fan-out.** At query time `for_you` selects the top-`M` clusters by
|
||||
current importance, issues `M` ANN queries (`candidate_gen::ann_candidates_multi`),
|
||||
and merges by best (min) distance; the personalization boost is the max cosine
|
||||
over all clusters. Users below the cold-start threshold keep the single
|
||||
adaptive-LR vector (`entities/preference.rs`). Design: `docs/research/multi-vector-preference.md`.
|
||||
|
||||
**Idle-readiness + TLS scale-up (m12p5–m12p6) — followers converge readiness on an idle cluster, and elasticity is proven over REAL mTLS on k8s**
|
||||
|
||||
- **Idle-readiness convergence (m12p5).** The leader heartbeat now carries its live
|
||||
frontier (`leader_last_seq`), so a caught-up follower flips `/health` Ready on an
|
||||
idle cluster instead of stalling until the next status poll or ship. Cert SAN
|
||||
wildcard widened for scale-to-5.
|
||||
- **TLS scale-up (m12p6).** A real `kubectl scale 3→5` exercised seed-join over
|
||||
mTLS on k8s (kind) for the first time: joiners flip Ready in ~13s via the
|
||||
idle-readiness heartbeat, auto-promote to Voter, reach full content parity at
|
||||
lag 0, with zero acked DATA loss across scale-down. Fixes span a six-bug chain —
|
||||
`https://` seed scheme, rustls `CryptoProvider` install order, headless seed
|
||||
Service, cold-handshake poll timeout, two-tier cert-manager PKI, and the
|
||||
`grpc_tls_for` CA fallback for a not-yet-in-topology joiner. Persists the HNSW
|
||||
graph and skips a suspect graph on reseed-pending close.
|
||||
|
||||
**Sharded ingestion (m12p4) — scatter-gather across shard groups with cross-shard unified reads**
|
||||
|
||||
- **Scatter-gather pool + cross-shard reads.** Writes hash-route across a 3-group
|
||||
`shards:` topology; reads unify across groups (L4). Ran REAL on kind with a
|
||||
2-generator load job. Fixed an HTTP/2 204 forward-relay bug (a synthesized JSON
|
||||
body on a 204 relay triggered an h2 `RST_STREAM`). Confirmed with data: at fixed
|
||||
per-pod CPU, full-placement sharding scales failover, not write throughput;
|
||||
the ≥2.5×-and-≥5,000/s scaling target remains Ref-A/k3s-pending.
|
||||
|
||||
**Index tuning + recall/memory at the production shape (m12p3) — the G2 work: per-query `ef_search` is now honored, the brute-force crossover scales with dimensionality, and the HNSW recall/latency/memory frontier is measured at 1536-D with a real exact oracle**
|
||||
|
||||
- **Per-query `ef_search` override — now real.** `UsearchIndex::search` /
|
||||
|
||||
@ -5,7 +5,7 @@
|
||||
|
||||
A single-node-first, embeddable Rust database for the **personalized content ranking problem**. Replaces the 6-system stack (Elasticsearch + Redis + Kafka + feature store + vector DB + ranking service) with a single process, single query interface, and single operational model.
|
||||
|
||||
**Status:** Implemented — M0–M10 shipped (embeddable engine + multi-region cluster mode). This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
|
||||
**Status:** Implemented — M0–M12 shipped (embeddable engine + multi-region cluster mode). M11 (Enterprise-Grade Cluster) closed all nine phases 2026-06-13; M12 (Vector Retrieval at production shape) shipped the recall/ANN/index-tuning G1/G2 work, sharded ingestion, cluster elasticity (idle-readiness, TLS scale-up), and multi-vector preference modeling. This repository is a standalone Cargo workspace: the engine is the `tidaldb` crate at `tidal/`, with `tidal-net/`, `tidal-server/`, and `tidalctl/` as workspace siblings and example consumers under `applications/`. Pre-1.0 — APIs are stable for shipped features, but breaking changes are possible before 1.0. See [CHANGELOG.md](CHANGELOG.md) for milestone history and [docs/planning/ROADMAP.md](docs/planning/ROADMAP.md) for status and known gaps.
|
||||
|
||||
## Find Your Guide
|
||||
|
||||
|
||||
@ -300,7 +300,8 @@ Milestones completed:
|
||||
|
||||
- Storage engine, WAL, entity store, signal ledger
|
||||
- RETRIEVE query: candidate retrieval, filtering, scoring, diversity, pagination
|
||||
- Vector index (USearch HNSW) with adaptive filtered search
|
||||
- Vector index (USearch HNSW) with adaptive filtered search; ANN candidate generation in RETRIEVE with honored per-query `ef_search`
|
||||
- Multi-vector user preference modeling (per-user interest clusters with decayed importance)
|
||||
- 25 built-in ranking profiles
|
||||
- BM25 full-text search (Tantivy) + hybrid RRF fusion
|
||||
- Creator search and creator profiles
|
||||
|
||||
@ -13,6 +13,7 @@ The query interface is a single operation that encapsulates candidate retrieval,
|
||||
- SEARCH: keyword + semantic + hybrid retrieval
|
||||
- SIGNAL: engagement event write-back (closes the feedback loop in the same transaction)
|
||||
- All queries accept: FOR USER, USING PROFILE, FILTER, DIVERSITY, LIMIT
|
||||
- RETRIEVE also accepts SIMILAR TO (`similar_to` — a seed item for "more like this") and an optional `ef_search` ANN-recall/latency override
|
||||
- Filters are composable — any combination is valid
|
||||
|
||||
**File Pointer:** `VISION.md:47-57`
|
||||
@ -56,6 +57,7 @@ SIMILAR TO @item_id
|
||||
FOR USER @user_id
|
||||
USING PROFILE related
|
||||
FILTER unseen
|
||||
EF_SEARCH 128 # optional: tune ANN recall vs. latency per query
|
||||
LIMIT 10
|
||||
```
|
||||
|
||||
|
||||
@ -10,6 +10,7 @@ Entities are the nodes of the system. Three types: Items (content), Users, and C
|
||||
**Key Facts:**
|
||||
- Items have metadata, embeddings, and signals — signals are typed timestamped streams, not fields
|
||||
- Users have preferences, histories, and relationships — living profiles that update continuously
|
||||
- A user's taste is **two-tier**: cold-start users (`< 5` interactions) carry a single adaptive-LR **preference vector** (K=1); warm users carry **multiple preference clusters** (one centroid per coherent interest)
|
||||
- Creators are linked to Items and have their own embeddings (aggregated from catalog)
|
||||
- Relationships are first-class edges between entities (weighted, directional, traversable)
|
||||
|
||||
@ -19,7 +20,12 @@ Entities are the nodes of the system. Three types: Items (content), Users, and C
|
||||
|
||||
Items enter via the WRITE path with metadata + embedding. A signal ledger is initialized at zero. Cold start exploration budget is applied automatically. Items are immediately queryable after commit.
|
||||
|
||||
Users accumulate implicit preference vectors from engagement history. Preference vectors update on every signal write (like, skip, hide, completion).
|
||||
Users accumulate implicit taste from engagement history, updated on every positive-engagement signal write (like, completion, etc.). The model has two tiers, gated on the user's total positive interaction count (`COLD_START_N = 5`):
|
||||
|
||||
- **Cold start (`< 5` interactions):** a single **preference vector** (`entities/preference.rs`) blended via an adaptive learning rate (`alpha = base / (1 + ln(1 + count))`) and L2-normalized — the K=1 case.
|
||||
- **Warm (≥ 5 interactions):** multiple **preference clusters** (`entities/multi_preference.rs`). Each positive engagement is assigned to its nearest centroid by cosine; a DP-means threshold (τ, default `0.55`) opens a new cluster when nothing is similar enough, capped at `K_MAX = 10` (over the cap, the engagement is assigned to the nearest centroid, never evicted). Each cluster carries its own adaptive LR and a forward-decayed importance (default half-life 30 days). On crossing the threshold the single vector seeds cluster 0, so the cold taste is never discarded.
|
||||
|
||||
This keeps a user who engages with hiking, cooking, and cars from collapsing into one averaged centroid that represents none of them.
|
||||
|
||||
Creators are entities with their own embeddings derived from their item catalog. Creator-level signals include engagement rate, posting frequency, and follower count.
|
||||
|
||||
|
||||
@ -11,7 +11,7 @@ A ranking profile is a named, versioned bundle that fully specifies how a query
|
||||
- Profiles are schema-level declarations, not application code. The app names a profile; the database executes the whole pipeline.
|
||||
- The same profile operates over different candidate sets (global / category / social graph / cohort) depending on its candidate strategy and the query's filters.
|
||||
- Personalization (`for_you`, `following`, `related`, `notification`, `date_saved`) requires `FOR USER` context — pass `.for_user(uid)` on the query. Without it the executor cannot read the user's preference vector / saved-state and either degrades to non-personalized scoring or, for `date_saved`, returns an error.
|
||||
- **Preference-vector personalization only updates from signals declared `.positive_engagement(true)`.** `signal_with_context(..)` folds the item embedding into the user's taste vector only for positive-engagement signal types; negative signals (skip/hide/block) update seen-state and hard-negatives but do not pull the taste vector toward the item. Declare engagement signals accordingly or `for_you` will not learn.
|
||||
- **Preference personalization only updates from signals declared `.positive_engagement(true)`.** `signal_with_context(..)` folds the item embedding into the user's taste only for positive-engagement signal types; negative signals (skip/hide/block) update seen-state and hard-negatives but do not pull the taste toward the item. Declare engagement signals accordingly or `for_you` will not learn. For a **warm user** (≥ 5 interactions) the engagement is assigned to the **nearest preference cluster** (or opens a new one past the DP-means threshold); for a cold-start user it blends the single preference vector. At query time `for_you` fans out over the user's **top-M clusters by current decayed importance** (`DEFAULT_TOP_M ≈ 3`), issues one ANN query per cluster centroid, and merges by best (min) distance — see [Entities](./entities.md).
|
||||
- Built-in defaults below are read straight from `tidal/src/ranking/builtins.rs`. Where a value is intentionally application-tuning territory, it is described qualitatively rather than invented.
|
||||
|
||||
**File Pointers:** `tidal/src/ranking/builtins.rs` (the 25 built-ins), `tidal/src/ranking/profile.rs` (`RankingProfile`, `DiversitySpec`, `Sort`, `Boost`, `CandidateStrategy`), `VISION.md:43-55`.
|
||||
@ -51,6 +51,7 @@ A ranking profile is a named, versioned bundle that fully specifies how a query
|
||||
### Notes on specific built-ins
|
||||
|
||||
- **`trending` / `cohort_trending` and the `view`/`share` boosts.** The `Sort::Trending` formula (`view_vel + 2·share_vel` over 24h) is the ordering authority on the global path and *replaces* the boost loop, so the executor skips boosts there (no double-count). The same `view`/`share` velocity boosts are not dead weight: the cohort rescore path (`rescore_with_cohort`) consumes `profile.boosts` directly to re-score against a cohort's ledger. Keep the boost weights mirroring the formula so cohort ordering matches global ordering.
|
||||
- **`SignalRanked` candidate generation is served from a cached per-signal-type top-K** (`signals/ledger/hot_top_k.rs`). Because the same decay `lambda` multiplies every entity of a signal type by the same factor, time decay never changes the relative order — so the cached top-K stays valid until the **next signal write**, not the next clock tick. The O(N) ledger rebuild is throttled (sub-millisecond and always fresh under 50k entries; at most one rebuild per second above that) to keep the read p99 off the scan path.
|
||||
- **`for_you` boosts vs. preference match.** Beyond the `Hot { gravity = 1.5 }` recency curve, `for_you` boosts decayed `view` (weight 1.0), decayed `like` (weight 2.0), and 24h `share` velocity (weight 1.5), layered on top of the user-preference-vector match. The 10% exploration injection is what keeps it from collapsing into a filter bubble.
|
||||
- **`search` has `sort = None` on purpose.** The fused RRF (BM25 + ANN) score from the search executor is the primary ordering signal; the profile only adds a small `view`/`like` quality overlay (weights 0.5 / 0.8) so relevance stays in charge. Exploration is `0.0` so results are deterministic for a given query. Set diversity yourself via `SearchBuilder::diversity(..)`.
|
||||
- **Three sort modes have no built-in profile shortcut.** `MostFollowed` and `CreatorEngagementRate` rank **creators** (not items) and need a creator-scoped candidate strategy plus a `follow` signal the generic schema does not guarantee; `Rising` (1h/24h view-velocity acceleration ratio) overlaps `trending`/`hot` and is sensitive to the exact window pair. All three are reachable via a custom schema profile (`SchemaBuilder::ranking_profile(..).sort(..)`); only the named-convenience built-in is omitted.
|
||||
@ -63,7 +64,7 @@ The candidate strategy decides *which* items the profile even considers before s
|
||||
|----------|--------------|-----------------------|
|
||||
| `Scan` (`sort_field`) | Population scan over the item keyspace; default for built-ins (`sort_field = "created_at"`) | most population + sort-coverage profiles |
|
||||
| `Relationship` | Sources candidates from the user's relationship graph (followed creators) | `following`, `notification` |
|
||||
| `Ann { slot, limit }` | k-NN over an embedding slot (set via the query's `.similar_to` / `Search`'s `.vector`) | custom; `related` uses the seed embedding via the query |
|
||||
| `Ann { slot, limit }` | k-NN over an embedding slot (set via the query's `.similar_to` / `Search`'s `.vector`); the per-query `ef_search` override (m12p3) trades recall for latency on this path | custom; `related` uses the seed embedding via the query, `for_you` over the user's preference clusters |
|
||||
| `SignalRanked { signal, window }` | Pre-orders candidates by a signal's windowed aggregate | custom |
|
||||
| `Hybrid` | Fuses lexical + vector retrieval | the `search` query path (RRF fusion in the search executor) |
|
||||
| `CohortTrending` | Cohort-scoped signal aggregation | reached via `.cohort("name")` with `cohort_trending` |
|
||||
|
||||
@ -9,7 +9,7 @@ ARCHITECTURE, API, QUICKSTART, CODING_GUIDELINES, thoughts) live at the
|
||||
|
||||
## Component specs — `specs/`
|
||||
|
||||
The authoritative component specifications (status: Implemented, M0–M8).
|
||||
The authoritative component specifications (status: Implemented, M0–M12).
|
||||
|
||||
| # | Spec | # | Spec |
|
||||
|---|------|---|------|
|
||||
@ -24,8 +24,8 @@ The authoritative component specifications (status: Implemented, M0–M8).
|
||||
|
||||
## Planning — `planning/`
|
||||
|
||||
- [**ROADMAP.md**](planning/ROADMAP.md) — milestones M0–M11, phase status, known gaps
|
||||
- [**roadmap-to-cluster.md**](roadmap-to-cluster.md) — adopted M11 plan: gap analysis + phase specs taking the multi-process cluster from experimental to enterprise-grade (m11p1–p5 ✅; p6–p9 planned), grounded in the 2026-06-10 live stress-test baselines
|
||||
- [**ROADMAP.md**](planning/ROADMAP.md) — milestones M0–M12, phase status, known gaps
|
||||
- [**roadmap-to-cluster.md**](roadmap-to-cluster.md) — adopted M11 plan: gap analysis + phase specs taking the multi-process cluster from experimental to enterprise-grade (all nine phases m11p1–p9 ✅, 2026-06-13), grounded in the 2026-06-10 live stress-test baselines
|
||||
- [PRODUCT_ROADMAP.md](planning/PRODUCT_ROADMAP.md) · [architecture-review.md](planning/architecture-review.md) · [roadmap-cohort-analysis.md](planning/roadmap-cohort-analysis.md) · [site-cohort-analysis.md](planning/site-cohort-analysis.md)
|
||||
- Per-milestone phase/task archive: `planning/milestone-0,1,2,3,5,7,8,9,10,11,p/`
|
||||
|
||||
|
||||
@ -602,9 +602,11 @@ always-current HTTP reference is the served spec at `GET /openapi.json` (unauthe
|
||||
> **On cluster mode — be honest with yourself.** tidalDB's cluster mode is
|
||||
> **experimental**. It now has a **multi-process** mode (`--region`, one process
|
||||
> per region, real gRPC peering, real process isolation), plus the single-process
|
||||
> dev/demo default. Even multi-process mode is *not* quorum-acked HA: writes are
|
||||
> leader-durable only and failover is operator-driven (`/cluster/promote`), not
|
||||
> automatic. It also replicates **global signals only** — `user_id` / `creator_id`
|
||||
> dev/demo default. Multi-process mode is quorum-acked HA: writes are
|
||||
> **quorum-acked durable by default** (majority-committed before ack) and leadership
|
||||
> failover is **automatic** (Raft-style election — no operator intervention), with
|
||||
> **elastic seed-join** membership for adding nodes without editing the topology. It
|
||||
> also replicates **global signals only** — `user_id` / `creator_id`
|
||||
> contexts are rejected on the cluster `/signals` path, so the personalized swipe
|
||||
> loop (section 4) is a **single-node** feature today (the cross-region escape
|
||||
> hatch for user-scoped hides is `/hardnegs` + `/cluster/reconcile`). Ship the For
|
||||
|
||||
@ -287,6 +287,7 @@ You can load `/openapi.json` into any OpenAPI viewer (Swagger UI, Redoc, Stoplig
|
||||
| `POST /signals` | yes | `204` | Body: `{ "entity_id", "signal", "weight", "user_id"?, "creator_id"? }`. |
|
||||
| `GET /feed` | yes | `200` | `?user_id=&profile=for_you&limit=20`. |
|
||||
| `GET /search` | yes | `200` | `?query=&user_id=&limit=20`. |
|
||||
| `POST /vector_search` | yes | `200` | Pure k-NN. Body: `{ "vector": [<f32>, ...], "k"?, "ef_search"? }`. |
|
||||
|
||||
`limit` is clamped to `1000` at the trust boundary. Request middleware: 30 s timeout (`408`), 100 max in-flight (`429`), 2 MB body cap (`413`), and an `x-request-id` echoed on every response.
|
||||
|
||||
|
||||
@ -210,3 +210,54 @@ groups:
|
||||
annotations:
|
||||
summary: "Self-driving heal not converging"
|
||||
description: "{{ $value }} peer(s) have been mid-heal for 10m. The heal loop retries through breaker resets automatically; a peer stuck this long is a real partition or a dead node — investigate the link, do NOT re-issue heal by hand."
|
||||
|
||||
- alert: TidalDBClusterPeerShipStall
|
||||
# A peer stops accepting batches (its acked frontier flatlines) while
|
||||
# flushed events queue behind it: deriv() is the gauge-correct way to
|
||||
# detect a flat acked seqno (rate()/increase() are invalid on gauges).
|
||||
expr: deriv(tidaldb_cluster_peer_acked_seqno[2m]) == 0 and tidaldb_cluster_peer_ship_queue_depth > 0
|
||||
for: 2m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "Replication to a peer has stalled with events queued"
|
||||
description: "Peer {{ $labels.peer_shard }} stopped accepting batches while events queue behind it (partition, dead peer, or paused sender). Check /cluster/status and the self-heal loop."
|
||||
|
||||
- alert: TidalDBClusterQuorumLag
|
||||
expr: (tidaldb_cluster_relay_last_seq - tidaldb_cluster_relay_durable_seq) > 10000
|
||||
for: 2m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "Quorum commit index is far behind the leader frontier"
|
||||
description: "{{ $value }} events between the leader's flushed frontier and the quorum commit index (threshold 10000). A majority of the replica set is not confirming durability (down/partitioned followers, or follower apply throughput exhausted). ack=quorum writes will 503; the bodies name the laggards."
|
||||
|
||||
- alert: TidalDBClusterDivergenceQuarantine
|
||||
# 1 = the node fenced itself from the data plane after detecting a
|
||||
# divergent suffix. Since m11p5 it auto-reseeds on next boot; a persistent
|
||||
# latch means the reseed is not completing.
|
||||
expr: tidaldb_cluster_divergence_quarantined == 1
|
||||
for: 2m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "A node has quarantined itself for a divergent suffix"
|
||||
description: "A node fenced itself from the data plane after detecting a divergent suffix. Since m11p5 it auto-reseeds on its next boot; if the latch persists, the reseed is not completing — check tidaldb_cluster_reseed_required and the snapshot path."
|
||||
|
||||
- alert: TidalDBClusterReseedPending
|
||||
# Durable reseed-marker latch. Expected briefly after a quarantine or a
|
||||
# behind-a-compacted-leader restart; a persistent latch means the snapshot
|
||||
# fetch is failing.
|
||||
expr: tidaldb_cluster_reseed_required == 1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "A node has a reseed pending but has not completed it"
|
||||
description: "A node has latched the reseed marker but has not completed a snapshot reseed in 10m. Expected briefly after a quarantine or a behind-a-compacted-leader restart; a persistent latch means the snapshot fetch is failing (no reachable leader, capability gate, or staging fault)."
|
||||
|
||||
- alert: TidalDBClusterSnapshotPinForceDrop
|
||||
# A staged-snapshot retention pin was force-dropped past the hard cap — a
|
||||
# joiner started a reseed and never released (died mid-fetch). The dropped
|
||||
# pin protects compaction.
|
||||
expr: increase(tidaldb_cluster_snapshot_pin_force_drops_total[1h]) > 0
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "A staged-snapshot retention pin was force-dropped"
|
||||
description: "A staged-snapshot retention pin was force-dropped past the hard cap in the last hour — a joiner started a reseed and never released (died mid-fetch). The dropped pin protects compaction; the stranded joiner must be re-driven or removed."
|
||||
|
||||
@ -20,7 +20,7 @@ This is the routing table `kubernetes.md` points operators to for **cluster** in
|
||||
| **Read-SLA collapse** | p99 read latency blows past 10 ms; reads hang. | Was CPU oversubscription, fixed in **rc12** (SEARCH_GATE + parallel scatter + cpu limit 2→3). Confirm running image is rc12+ and that load is spread across **all 3** pods (region-pinned reads, not pinned to one). See [cluster.md](../runbooks/cluster.md). |
|
||||
| **PVC loss / corruption on one pod** | One pod's data dir is corrupt or its PVC is gone. | Delete that pod's **PVC and pod**; it reseeds fresh from the quorum (snapshot install) and converges to lag=0. On the live image the corrupt-PVC case is recoverable this way — no full-cluster action needed. See [cluster.md — reseed](../runbooks/cluster.md#membership-verbs-m11p5--online-add--remove--inspect--reseed). |
|
||||
| **Full-cluster loss / rebuild from backup** | Quorum cannot be restored from surviving pods (multi-node data loss). | Rebuild from object store with `tidalctl restore` (BLAKE3-manifested, per-shard `checkpoint_seq`, proven against real S3). See [disaster-recovery.md](../runbooks/disaster-recovery.md). |
|
||||
| **Point-in-time recovery (PITR)** | Need to restore the corpus to a specific point in time. | Use the `tidalctl` + DR runbook PITR procedure. See [disaster-recovery.md](../runbooks/disaster-recovery.md). |
|
||||
| **Restore to a backup's checkpoint** | Need to roll the corpus back to a captured backup. | Arbitrary point-in-time recovery (`tidalctl replay --until <seq>`) is **not shipped**. Restore lands you at the backup's `checkpoint_seq` + whatever WAL tail was captured — not an arbitrary instant. Use `tidalctl backup` / `tidalctl restore`. See [disaster-recovery.md — PITR posture](../runbooks/disaster-recovery.md#pitr-posture). |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@ -37,7 +37,8 @@ A single embeddable database can replace the 6-system content ranking stack by t
|
||||
| M8 | Distributed Fabric | Multi-region, multi-tenant replication keeps agent-memory semantics intact | Hosted tidalDB, cloud/edge deployments, shared agent substrate — **✅ COMPLETE**: in-process primitives + multi-node replication over real gRPC + true multi-process cluster mode (one process per region, real process isolation) with full tier-3 UAT (partition injection via TCP-proxy, clock-skew, rolling-upgrade, runbook verification); G1 + G2 resolved. Post-M8 follow-ups: quorum-ack writes, automatic failure detection / leader election |
|
||||
| M9 | Community Sync & Revocation | Local embeddable profiles can opt into community personalization and safely leave/purge contributions | Community personalization, federated taste graphs, shared feeds — ✅ COMPLETE (2026-06-06) |
|
||||
| M10 | Governance & Agent Rights | Community rules and agent-scoped permissions control what signals influence ranking | User-owned AI personalization at scale, policy-compliant agents — ✅ COMPLETE (2026-06-06) |
|
||||
| M11 | Enterprise-Grade Cluster | The multi-process cluster becomes a system of record: fast unified log, quorum-durable, self-healing, horizontally scalable, secured, observable, chaos-tested | Paying-customer cluster deployments — **IN PROGRESS**: m11p1 ✅ + m11p2 ✅ + m11p3 ✅ + m11p4 ✅ (automatic failover: Raft-style election + term fencing + divergence quarantine — closes **G5**; v0.9 "Credible HA" wave complete 2026-06-12) + m11p5 ✅ (membership/discovery/elasticity: DNS peers, snapshot+stream reseed, kind-4 membership records, seed join, `k8s/cluster/` — 2026-06-12) + m11p7 ✅ (security hardening: gRPC mTLS default + zero-drop cert rotation, inter-node HTTP TLS + per-node signed tokens, admin audit log, per-principal rate limit — 2026-06-13) + m11p8 ✅ (observability + operations: completed `tidaldb_cluster_*` set incl. breaker/forwards/self-heal on the per-node `/metrics` listener + Grafana cluster row + alert group, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR archival + `tidalctl` backup/restore, rolling-upgrade version handshake + Woodpecker release gate — closes **G-O** — 2026-06-13) + m11p9 ✅ (continuous correctness: new fault classes (disk-full / slow-fsync / asymmetric partition) as REAL faults behind a production-compiled-out `fault-injection` feature, first-class invariant checkers, `tidal-stress` soak with regression gates + JSON summary, a Woodpecker cron nightly chaos+soak pipeline, and a guarantee→test traceability matrix — closes the **G-C** apparatus; the 30-day-green calendar accrues nightly — 2026-06-13) + m11p6 ✅ (sharding × replication + rebalancing: ONE hash-routed + replicated write surface across S shard groups each at RF with its own elected leader; per-group rebalancing verbs (`/cluster/shards/{id}/transfer` + `/replicas`) and a `?shard=` admin selector; tier-3 3×3 kill-node exit gate — only the dead node's leaderships move, reads never stop, zero acked loss; ≥5,000/s + 2.5× scaling Ref-A-pending — 2026-06-13). **ALL NINE PHASES COMPLETE**; the v1.0 bar now waits only on the 30-day-green nightly calendar (m11p9) and the standing Ref-A/k3s throughput re-runs |
|
||||
| M11 | Enterprise-Grade Cluster | The multi-process cluster becomes a system of record: fast unified log, quorum-durable, self-healing, horizontally scalable, secured, observable, chaos-tested | Paying-customer cluster deployments — **✅ COMPLETE (all nine phases, 2026-06-13)**: m11p1 ✅ + m11p2 ✅ + m11p3 ✅ + m11p4 ✅ (automatic failover: Raft-style election + term fencing + divergence quarantine — closes **G5**; v0.9 "Credible HA" wave complete 2026-06-12) + m11p5 ✅ (membership/discovery/elasticity: DNS peers, snapshot+stream reseed, kind-4 membership records, seed join, `k8s/cluster/` — 2026-06-12) + m11p7 ✅ (security hardening: gRPC mTLS default + zero-drop cert rotation, inter-node HTTP TLS + per-node signed tokens, admin audit log, per-principal rate limit — 2026-06-13) + m11p8 ✅ (observability + operations: completed `tidaldb_cluster_*` set incl. breaker/forwards/self-heal on the per-node `/metrics` listener + Grafana cluster row + alert group, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR archival + `tidalctl` backup/restore, rolling-upgrade version handshake + Woodpecker release gate — closes **G-O** — 2026-06-13) + m11p9 ✅ (continuous correctness: new fault classes (disk-full / slow-fsync / asymmetric partition) as REAL faults behind a production-compiled-out `fault-injection` feature, first-class invariant checkers, `tidal-stress` soak with regression gates + JSON summary, a Woodpecker cron nightly chaos+soak pipeline, and a guarantee→test traceability matrix — closes the **G-C** apparatus; the 30-day-green calendar accrues nightly — 2026-06-13) + m11p6 ✅ (sharding × replication + rebalancing: ONE hash-routed + replicated write surface across S shard groups each at RF with its own elected leader; per-group rebalancing verbs (`/cluster/shards/{id}/transfer` + `/replicas`) and a `?shard=` admin selector; tier-3 3×3 kill-node exit gate — only the dead node's leaderships move, reads never stop, zero acked loss; ≥5,000/s + 2.5× scaling Ref-A-pending — 2026-06-13). **ALL NINE PHASES COMPLETE**; the v1.0 bar now waits only on the 30-day-green nightly calendar (m11p9) and the standing Ref-A/k3s throughput re-runs |
|
||||
| M12 | Vector Retrieval at Production Shape | Ranking-at-scale steers on measured numbers, not absent ones: ANN candidate generation in RETRIEVE, honored per-query `ef_search`, a recall/latency/memory frontier measured at 1536-D against an exact oracle, sharded ingestion, cluster elasticity, and multi-vector user preference | Relevant, bounded feeds as the corpus grows past the scan cap — **✅ COMPLETE (2026-06-14 / 2026-06-23)**: m12p1 ✅ (read-recall harness: `vector_search_items` k-NN probe + `POST /vector_search` + `tidal-stress --verify-recall` brute-force oracle) + m12p2 ✅ (ANN candidate-gen in RETRIEVE: `for_you` via preference vector, `related` via seed embedding, cached per-signal-type top-K for `trending`; graceful scan-fallback) + m12p3 ✅ (index tuning: per-query `ef_search` honored, dimension-aware brute→HNSW crossover, F16 validated <1%, Int8 rejected) + m12p4 ✅ (sharded ingestion: scatter-gather pool + cross-shard unified reads, 3-group `shards:`) + m12p5 ✅ (idle-readiness: leader heartbeat carries live frontier so followers converge readiness on idle) + m12p6 ✅ (TLS scale-up: real mTLS `kubectl scale 3→5` seed-join over k8s, two-tier cert-manager PKI) + multi-vector user preference (online k-means clusters with DP-means split, per-cluster decayed importance, top-M ANN fan-out; cold-start single-vector fallback). v1.0 bar shared with M11: 30-day-green nightly calendar + standing Ref-A/k3s throughput re-runs |
|
||||
|
||||
### Embeddable → Distributed Path
|
||||
|
||||
@ -153,9 +154,9 @@ The roadmap now has two tracks:
|
||||
|
||||
**iknowyou / Aeries: IN PROGRESS (as of 2026-02-24)** — M1–M4 complete. M5 (Communication Brief) is in progress with core implementation live; acceptance validation pending.
|
||||
|
||||
**Engine status:** M0–M10 **COMPLETE**. M9 (Community Sync & Revocation) and M10 (Governance & Agent Rights) shipped 2026-06-06 — see *Implementation Status (M9–M10 as-built)* below.
|
||||
**Engine status:** M0–M12 **COMPLETE**. M9/M10 shipped 2026-06-06; M11 (Enterprise-Grade Cluster, all nine phases) closed 2026-06-13; M12 (Vector Retrieval at production shape) closed 2026-06-14 / 2026-06-23 — see the M11/M12 milestone rows above and *Implementation Status* below.
|
||||
|
||||
**Next (engine):** the cross-cutting community-overlay UAT surface — a single RETRIEVE that blends local + community layers and folds membership-epoch / policy-version / purge-watermark inline into `Results.policy_metadata` (the per-phase mechanisms and direct read APIs all exist; this is the unifying query surface). Then a multi-node M9/M10 UAT harness, and the deferred `writer_agent` u16 interning on the WAL v3 envelope.
|
||||
**Next (engine):** close the v1.0 bar — the 30-day-green nightly chaos+soak calendar (m11p9) and the standing Ref-A/k3s throughput re-runs (≥5,000/s + ≥2.5× single-shard scaling). Deferred follow-ups: the `writer_agent` u16 interning on the WAL v3 envelope and the offline medoid-recluster tier for multi-vector preference.
|
||||
**Next (product):** iknowyou M5 acceptance pass, then M6 Closed Loop (session lifecycle + preference drift validation).
|
||||
|
||||
---
|
||||
|
||||
@ -70,7 +70,8 @@ are **accurate but DESIGN-REFERENCE — they are not yet loaded by the live
|
||||
Prometheus.** The rules already cover the must-watch signals
|
||||
(`TidalDBClusterBreakerOpen` on `peer_breaker_state == 1`,
|
||||
`TidalDBClusterCommitIndexStall`, `TidalDBClusterElectionChurn`,
|
||||
`TidalDBClusterQuorumTimeouts`, `DivergenceQuarantine`, `ReseedPending`).
|
||||
`TidalDBClusterQuorumTimeouts`, `TidalDBClusterDivergenceQuarantine`,
|
||||
`TidalDBClusterReseedPending`).
|
||||
|
||||
**Open infra step:** promote them into the observability stack as a `PrometheusRule`
|
||||
(or vmalert rule file) so they actually page. Until then, on-call watches the
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 00 -- Architecture Overview
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Author:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Purpose:** Show how the 14 specs connect. The forest before the trees.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Storage Engine Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-05-28
|
||||
**Prerequisites:** [VISION.md](../../VISION.md), [thoughts.md](../../thoughts.md), [Signal Ledger Research](../research/tidaldb_signal_ledger.md)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Signal System Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** WAL subsystem, Entity Store, Schema Engine
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 05 -- Cohort Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Entity Model (02), Signal System (03), Query Engine
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Text Retrieval Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Vector Retrieval Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 08 -- Query Engine Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Ranking and Scoring Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** Signal System (03), Relationships (04), Cohorts (05), Text Retrieval (06), Vector Retrieval (07)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Feedback Loop Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Signal System](03-signal-system.md), [Entity Model](02-entity-model.md), [Relationships](04-relationships.md), [Storage Engine](01-storage-engine.md)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Schema Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-05-28
|
||||
**Prerequisites:** [02-entity-model.md](02-entity-model.md), [03-signal-system.md](03-signal-system.md), [04-relationships.md](04-relationships.md), [API.md](../../API.md)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 12 -- Cold Start Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Entity Model](02-entity-model.md), [Signal System](03-signal-system.md), [Relationships](04-relationships.md), [Cohorts](05-cohorts.md), [Feedback Loop](10-feedback-loop.md), [Schema](11-schema.md)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 13 -- Concurrency Specification
|
||||
|
||||
**Status:** Implemented (M0–M8)
|
||||
**Status:** Implemented (M0–M12)
|
||||
**Authors:** tidalDB Engineering
|
||||
**Date:** 2026-02-20 (spec) · Implemented as of 2026-05-28
|
||||
**Depends on:** [Storage Engine](01-storage-engine.md), [Signal System](03-signal-system.md), [Feedback Loop](10-feedback-loop.md)
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Scale Architecture Specification
|
||||
|
||||
**Status:** Implemented (M0–M8); multi-node cluster mode shipped through M11 — quorum-durable writes (m11p3), automatic election/fencing (m11p4), dynamic membership (m11p5), and **sharding × replication (m11p6): S shard groups, each a replication group at RF with its own elected leader, any gateway hash-routing writes (the §4 Option-C shape), per-group rebalancing verbs.** The cluster `ShardReplica` (`tidal-server/src/cluster/node.rs`) is one group's machinery; a `ClusterNode` hosts a `BTreeMap<ShardId, Arc<ShardReplica>>`. See [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) and [docs/planning/milestone-11/phase-6.md](../planning/milestone-11/phase-6.md). Original M8 known gaps G3 (shard-routing hash unification) and G6 (embedding 500→400) remain open.
|
||||
**Status:** Implemented (M0–M12); multi-node cluster mode shipped through M11 — quorum-durable writes (m11p3), automatic election/fencing (m11p4), dynamic membership (m11p5), and **sharding × replication (m11p6): S shard groups, each a replication group at RF with its own elected leader, any gateway hash-routing writes (the §4 Option-C shape), per-group rebalancing verbs.** The cluster `ShardReplica` (`tidal-server/src/cluster/node.rs`) is one group's machinery; a `ClusterNode` hosts a `BTreeMap<ShardId, Arc<ShardReplica>>`. See [docs/roadmap-to-cluster.md](../roadmap-to-cluster.md) and [docs/planning/milestone-11/phase-6.md](../planning/milestone-11/phase-6.md). Original M8 known gaps G3 (shard-routing hash unification) and G6 (embedding 500→400) remain open.
|
||||
**Author:** tidalDB Engineering
|
||||
**Last Updated:** 2026-05-28
|
||||
**Depends on:** Storage Engine (01), Entity Model (02), Signal System (03), Cohorts (05), Vector Retrieval (07)
|
||||
|
||||
@ -103,6 +103,38 @@ if [ -f docs/planning/ROADMAP.md ]; then
|
||||
done
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. Milestone-status freshness (hard). A core doc must not advertise a status
|
||||
# bound behind ROADMAP's latest COMPLETE milestone (catches the "M0–M10
|
||||
# shipped" / "Implemented, M0–M8" drift). Only status-context lines are
|
||||
# checked, so historical path lists ("M0–M2 Embed & prove primitives")
|
||||
# never trip it. Self-updating: the bound is derived from ROADMAP, not pinned.
|
||||
# ---------------------------------------------------------------------------
|
||||
stale_ms=0
|
||||
latest=0
|
||||
if [ -f docs/planning/ROADMAP.md ]; then
|
||||
for n in $(grep -oE 'M[0-9]+' docs/planning/ROADMAP.md | sed 's/M//' | sort -un); do
|
||||
if grep -qiE "\| *\*?\*?M$n\b.*(COMPLETE|✅)" docs/planning/ROADMAP.md 2>/dev/null; then
|
||||
[ "$n" -gt "$latest" ] && latest=$n
|
||||
fi
|
||||
done
|
||||
if [ "$latest" -gt 0 ]; then
|
||||
for f in CLAUDE.md README.md docs/README.md; do
|
||||
[ -f "$f" ] || continue
|
||||
while IFS=: read -r ln text; do
|
||||
echo "$text" | grep -qiE 'status|shipped|implemented|milestones?' || continue
|
||||
k=$(echo "$text" | grep -oE 'M0(–|-)M[0-9]+' | grep -oE '[0-9]+$' | head -1)
|
||||
[ -z "$k" ] && continue
|
||||
if [ "$k" -lt "$latest" ]; then
|
||||
err "$f:$ln advertises status bound M0–M$k but ROADMAP's latest COMPLETE milestone is M$latest — update the status line"
|
||||
stale_ms=1
|
||||
fi
|
||||
done < <(grep -nE 'M0(–|-)M[0-9]+' "$f")
|
||||
done
|
||||
fi
|
||||
fi
|
||||
[ "$stale_ms" -eq 0 ] && ok "core-doc milestone-status lines current (latest COMPLETE = M$latest)"
|
||||
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo "doc-guard: FAILED — fix the ✗ items above." >&2
|
||||
exit 1
|
||||
|
||||
Loading…
Reference in New Issue
Block a user