a588f01f63
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL,
and a CHANGELOG that named profiles that do not exist. All verified before fixing.
BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a
missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0.
So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable
`created_at` outranked the genuinely newest item and `normalize` reported its
score as 1.0.
The codebase had already ruled on this and gone the other way. `score_shortest`,
`score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and
`helpers.rs` carries a regression test named for the exact anomaly --
`normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because
`Shortest` hit it first: "a pre-clamped 0.0 floated above them and the
last-ranked (missing) item reported the highest score 1.0". I reintroduced it on
a new negated scale.
Three reachable data classes, none hypothetical: a legacy row written before
`created_at` was materialized (`state_rebuild.rs` documents the class), an empty
map from `deserialize_metadata` on a short or corrupt row, and an entity dropped
by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT
STORAGE READ ERROR could promote an item to the top of the feed.
Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole
set ties, unchanged. Map present but this entity has no usable `created_at` ->
NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because
that is a real value that legitimately means "newest"; an absent one carries no
recency claim at all.
Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports
`needs_item_metadata`, so the map is loaded on essentially every Hot query and the
per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a
single undated row the freshest divisor in the set: against a year-old cohort at
the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with
two views outranked correctly-dated items with tens of thousands. Across all four
Hot builtins. Now scored at Hot's floor.
BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_
newest` ranked candidates against the newest ids of the WHOLE universe, but
`scan_candidates` iterates the universe bitmap ASCENDING and breaks at
`(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a
catalog whose ids are assigned in creation order -- which is exactly what
`metadata_with_created_at`'s `Timestamp::now()` default produces -- the global
newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets
stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a
NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by`
I replaced kept the newest available. Strictly worse than the bug I set out to fix,
and only visible when degraded.
The key is now composite -- recency rank, then DESCENDING id for anything the
oversample did not cover -- so where index and candidate window overlap the
survivors are genuinely newest, and where they do not it degrades to the documented
pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_
key`: this path runs ONLY when the load shedder has already decided the node cannot
afford the work, so it must stay a linear partition, and only membership of
`[0, cap)` matters because Stage 3 re-orders the survivors anyway.
Mutation-proven, and the numbers show why the old test was blind: rank-only key
returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the
pre-existing 150-item test PASSES under that same mutation -- its fixture had the
oversample covering the whole universe AND the newest items at the low ids, the one
configuration where the defect cannot appear. New test uses a 3000-item
time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200
so the guard skips), limit=25 does.
Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING
entity id, and I had made newest-first coincide with it, so any mutation that
merely TIED the set still produced the asserted vector. The gate mutation proves
it: reverting `needs_metadata_for_sort` to its old four-variant form yielded
`left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the
suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`,
which is the exact drift this work exists to prevent. Every ordering fixture is now
non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and
ascending id (the tie collapse).
CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits
repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not
New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real
blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them
carry a diversity block so they already loaded item metadata and change behaviour
the instant this image rolls. Replaced with a verified table. Also cited the spec
conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified
`Sort::New` as "created_at DESC".
created_at units are now trusted with a warning instead of silently. A
seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides
by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores
-496731 against boost sums in single digits -- re-entering through DATA the exact
"recency annihilates every boost" defect this work removed -- and Hot buries the
item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was
inert; this work made it live. The repo's OWN fixtures made that mistake in three
places, which is the proof it is the natural one.
`metadata_with_created_at` now warns when the value is too small to be nanoseconds,
logging entity id, value, age_hours and age_years so an operator can act. It does
NOT rewrite the value -- guessing the unit would corrupt what the `created_at`
range index already reads as nanoseconds -- and does NOT reject the write, which
would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond
timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible
date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server:
seconds, millis and micros each warn with age_years=56; a real nanosecond value is
silent. The nanosecond contract is now documented on the public DTO and propagates
to the OpenAPI schema, where it was invisible before.
Stale docs corrected: two published blog posts and two claim-verification ledgers
were certifying "entity recency (higher ID = newer)"; the e2e fixture contract
justified an interleaving decision with reasoning that is now false (its conclusion
still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:"
rationale cited behaviour this work removed, so it is now marked PENDING
RE-MEASUREMENT with the three specific measurements named rather than left reading
as justified.
Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors.
Fast integration suites all green. Real-server e2e re-verified: `new` and `hot`
both return newest-first, `new`'s scores now evenly spaced across evenly spaced
ages.
1296 lines
86 KiB
Markdown
1296 lines
86 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to tidalDB will be documented in this file.
|
||
|
||
## [Unreleased]
|
||
|
||
### Stability
|
||
|
||
tidalDB is **production-ready**. M0–M12 are shipped and the pre-release
|
||
"not yet recommended for production" disclaimer has been withdrawn from the
|
||
documentation set. The API surface and on-disk data format are stable for
|
||
shipped features: additive changes ship in minor releases, and any breaking
|
||
change to a public API or a persisted format ships with a documented migration
|
||
path in this file. This supersedes the `0.1.0` "no stability guarantees" note
|
||
below.
|
||
|
||
### Breaking
|
||
|
||
**`Sort::New` orders by real `created_at`, not by entity ID (wire-visible)**
|
||
|
||
`Sort::New` used `entity_id as f64` as its score. That was wrong twice over. It
|
||
assumed IDs are assigned in creation order, which caller-supplied `u64` IDs do not
|
||
guarantee; and it used the ID's *magnitude* as the base score, so on a catalog of
|
||
N items the sort term contributed ~N against a boost sum in the single digits.
|
||
Recency did not merely participate in the ranking, it annihilated every boost, and
|
||
`normalize()` runs on the final sum so it could not rescue the ratio. The score is
|
||
now negated age in hours (newer = closer to 0 = higher), which preserves
|
||
"newest first" on a scale comparable to a boost sum.
|
||
|
||
Ordering changes wherever entity IDs were not monotonic with creation time. The
|
||
age-derived sorts are set by **seven** built-in profiles, four `Hot` and three
|
||
`New`, plus any custom profile using either:
|
||
|
||
| Profile | Sort | Defined at | Diversity block | Loaded item metadata *before* this change? |
|
||
|---------|------|-----------|-----------------|--------------------------------------------|
|
||
| `hot` | `Hot { gravity: 1.8 }` | `builtins.rs:188` | yes (`:197-200`) | yes |
|
||
| `for_you` | `Hot { gravity: 1.5 }` | `builtins.rs:312` | yes (`:333-336`) | yes |
|
||
| `related` | `Hot { gravity: 1.2 }` | `builtins.rs:381` | yes (`:396-399`) | yes |
|
||
| `brief` | `Hot { gravity: 1.5 }` | `builtins/extras.rs:39` | yes (`:74-77`) | yes |
|
||
| `following` | `New` | `builtins.rs:349` | yes (`:356-359`) | yes |
|
||
| `new` | `New` | `builtins.rs:206` | none | no |
|
||
| `chronological` | `New` | `builtins/extras.rs:91` | none | no |
|
||
|
||
The last column matters for rollout. The pre-change metadata pre-load was gated on
|
||
`session_context.is_some() || needs_metadata_for_creator_grouping`
|
||
(`query::executor::mod.rs:472-478`), and a diversity block satisfies the second
|
||
arm. So the five profiles carrying one were **already** loading the
|
||
`EntityId -> metadata` map and start scoring by real `created_at` the instant the
|
||
image rolls, with no configuration change. `new` and `chronological` carry no
|
||
diversity block and no session; they are newly enabled by the widened
|
||
`needs_metadata_for_sort` arm described under *Fixed* below.
|
||
|
||
Items with no usable `created_at` are handled by cause, not lumped together. When
|
||
the metadata map was never loaded at all, every candidate scores `0.0` and the set
|
||
ties. When the map *is* loaded and this specific entity has no parseable
|
||
`created_at`, it scores `f64::NEG_INFINITY` and sorts last — an undated row is not
|
||
silently promoted to "brand new" alongside genuinely fresh ones.
|
||
|
||
**Wire consequence of a tie, stated plainly:** for a corpus where *no* row carries
|
||
`created_at`, every candidate holds the same `Sort::New` score, so the result order
|
||
falls through to `finalize`'s tie-break 2, **ascending entity id**
|
||
(`ranking/executor/mod.rs:942-943`). That is the exact reverse of the pre-commit
|
||
output, which ranked on `entity_id as f64` descending. A caller reading an undated
|
||
corpus sees its page flip end to end. This is a correction, not a regression:
|
||
`docs/specs/09-ranking-scoring.md:1214` specifies `Sort::New` as `created_at DESC`,
|
||
"Pure chronological, no scoring", and `docs/specs/11-schema.md:1409` specifies the
|
||
`following` profile's sort semantics as "`created_at DESC` (pure chronological)".
|
||
Neither spec licenses entity id as a recency proxy; the implementation now conforms
|
||
to both.
|
||
|
||
Under `ReducedCandidates` load the candidate cap now selects survivors via the
|
||
`created_at` index read newest-first, not by descending entity ID. The old key was
|
||
only correct while `Sort::New` itself ranked by ID; left alone it would have
|
||
discarded the genuinely newest items *before* scoring, leaving the ranking wrong
|
||
only when degraded. Candidates the index oversample does not cover fall back to
|
||
**descending entity id** rather than to scan order, so the degraded path keeps a
|
||
deterministic, recency-shaped tail. `RangeIndex::top_n_descending` is new and
|
||
public for this.
|
||
|
||
### Fixed
|
||
|
||
**`Sort::Hot` is age-aware; it was ranking on view count alone**
|
||
|
||
`score_hot` hardcoded `age_hours = 24.0` for every candidate, so the divisor in
|
||
`log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate
|
||
set and `Sort::Hot` reduced **exactly** to `log10(max(views, 1))` — a view-count
|
||
ranking wearing a recency sort's name. Anyone tuning `gravity` was tuning a no-op.
|
||
All four built-in `Hot` profiles were affected: `hot` (gravity 1.8), `for_you`
|
||
(1.5), `related` (1.2) and the `brief` extra (1.5). `following` sets `Sort::New`,
|
||
not `Sort::Hot`; see the profile table under *Breaking* for the full set.
|
||
|
||
The in-code comment justified this by saying a per-entity `created_at` lookup would
|
||
need an `EntityId -> created_at_ns` reverse map that "is not built". That was stale:
|
||
`created_at` has been materialized **into item metadata** on every write since
|
||
`Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata`
|
||
map since M6p3, and the WAL record ships the materialized map so replicas agree. No
|
||
new index, storage change, schema change or migration was required — the scorer
|
||
reads the map it already had, the same way `read_duration` does.
|
||
|
||
Measured on a real server, 10 items with equal view counts and ages spanning
|
||
2–20 days: before, every item scored `0.5` (an all-equal set folded to the
|
||
normalizer's neutral midpoint) and the feed returned oldest-first forever; after,
|
||
scores run `1.0 → 0.0` strictly descending, newest first.
|
||
|
||
Two distinct "no age available" cases are kept distinct here too. If the metadata
|
||
map was never loaded, `score_hot` falls back to the documented
|
||
`DEFAULT_HOT_AGE_HOURS` constant, so the whole set shares one divisor and ranks by
|
||
view count as before. If the map *is* loaded and this entity has no parseable
|
||
`created_at`, the score returns Hot's floor of `0.0` instead — an undated row is
|
||
not handed the freshest possible divisor and floated to the top of a dated corpus.
|
||
|
||
**LIMIT, deliberately not papered over:** `hot`'s numerator is
|
||
`log10(max(views, 1))`, which is exactly `0.0` for 0 **or** 1 views, so the age
|
||
divisor cannot differentiate a zero-signal corpus — every candidate still scores
|
||
`0.0` and ties. Age-awareness takes effect from the second view onward. Making a
|
||
cold-start corpus rank newest-first requires recency to be **additive** rather than
|
||
a pure divisor, which changes ordering for every existing `Hot` consumer, so it is
|
||
a separate decision and is not folded in here. `sort_hot_zero_view_corpus_still_
|
||
ties_regardless_of_age` pins the current behaviour so the limit cannot be
|
||
rediscovered by accident.
|
||
|
||
**A wrong-unit `created_at` is now visible instead of silent**
|
||
|
||
`Items::metadata_with_created_at` defaults `created_at` only when the key is
|
||
absent or unparseable as `u64`. A **seconds**-unit value parses fine and is stored
|
||
verbatim, and `read_age_hours` divides by nanoseconds-per-hour — so
|
||
`"1700000000"` reads as **496,731 hours (~56 years)** old. Until this release both
|
||
age sorts ignored the value and the mistake was inert. Now it is live and severe:
|
||
`Sort::New` scores `-496_731` against boost sums in single digits, re-entering
|
||
through *data* the exact "recency annihilates every boost" defect the entity-id fix
|
||
above removed, and `Sort::Hot` at 10 views and gravity 1.8 falls from `2.838e-3`
|
||
(a correctly-dated 24-hour-old item) to `5.584e-11`, burying the item permanently.
|
||
The repo's own `tidal-server` fixtures wrote seconds, which is the evidence that
|
||
this is the natural caller mistake rather than a hypothetical; they now write
|
||
nanoseconds.
|
||
|
||
A write whose `created_at` parses but is below `6e17` ns (1989-01-05) now logs a
|
||
`WARN` naming the entity id, the offending value, and the resulting age in hours
|
||
and years. The threshold works because the plausible unit ranges cannot overlap: a
|
||
nanosecond timestamp for any date after 1990 is at least `6.31e17`, while seconds,
|
||
millis and micros values stay below `~1e16` even for dates centuries out.
|
||
|
||
The value is **not** rewritten and the write is **not** rejected. Guessing a unit
|
||
would corrupt a number the `created_at` range index already reads as nanoseconds,
|
||
and rejecting would break an API that accepts it today. `ItemRequest::metadata` now
|
||
documents the nanosecond unit, what it drives, and this failure mode, and the
|
||
description propagates into the OpenAPI schema.
|
||
|
||
**Every metadata-based sort was dead on the SEARCH path**
|
||
|
||
`alphabetical_asc`, `alphabetical_desc`, `shortest` and `longest` silently tied on
|
||
`GET /search` for any query without a session context. Two independent causes, both
|
||
in `query::search::executor::pipeline`: the metadata pre-load was gated on
|
||
`session_context.is_some()` alone and never consulted `profile.sort`, and the
|
||
`ProfileExecutor` it built never had `with_item_metadata` called at all — the map it
|
||
did compute was passed only as the keyword-hint argument, which the sort scorers do
|
||
not read. So `shortest`/`longest` scored `NEG_INFINITY` and the alphabetical sorts
|
||
scored the missing-title sentinel, for every candidate.
|
||
|
||
Both query executors now derive the decision from `Sort::needs_item_metadata`, an
|
||
exhaustive match on the enum itself. The knowledge lived in a `matches!` inside one
|
||
executor while a second executor had its own, different copy — which is precisely
|
||
how a metadata-reading sort came to be omitted from both. A new variant now has one
|
||
place to declare this and the compiler forces the author to visit it.
|
||
|
||
**`Sort::Hot`/`Sort::New` were missing from `needs_metadata_for_sort`**
|
||
|
||
On the RETRIEVE path the metadata point-read is skipped unless the profile needs it.
|
||
Both age-derived sorts were absent from that list, so a profile with no session and
|
||
no diversity block never loaded the map the fix above depends on. Measured cost of
|
||
including them, 2,000 candidates (the realistic ceiling — `scan_candidates` caps at
|
||
`max(limit * 10, 200)` and `limit > 500` is rejected): **7.25 ms, 3.6 µs per
|
||
candidate**, guarded by a test that fails above 250 ms.
|
||
|
||
|
||
|
||
`POST /sharded/{items,embeddings,signals}` hash-partitions by `entity_id` and
|
||
applies the write to the **owning region's local store with no WAL append**. It
|
||
does not ride the leader relay, so the data has **redundancy 1 regardless of the
|
||
replication factor** — an RF3 cluster configured `ack: quorum` does not replicate
|
||
it. That is intentional (parallel write throughput across shard owners: 3,669
|
||
signals/s measured, vs ~90/s on the replicated path), but the endpoint answered
|
||
`201`/`204` with nothing at the call site, in the response, or in the OpenAPI
|
||
saying so. An operator probed with `/sharded/embeddings`, found each embedding on
|
||
exactly one of three nodes, and filed a durability incident that had to be
|
||
retracted.
|
||
|
||
The three write routes now return **400** unless the request carries
|
||
`x-tidal-ack: local`, with a body naming the header and pointing at the
|
||
replicating alternative. The existing `x-tidal-ack` header is reused — no second
|
||
durability knob — and `local` remains rejected on `/items` / `/embeddings` /
|
||
`/signals`, which always replicate. `GET /sharded/{feed,search}` are reads and are
|
||
**not** affected. Nothing was rerouted: a `/sharded/*` write that opts in is
|
||
byte-for-byte as before, and the surface never silently replicates.
|
||
|
||
*Migration:* add `-H 'x-tidal-ack: local'` to acknowledge single-copy durability,
|
||
or move the write to `POST /items` / `/embeddings` / `/signals` for a replicated
|
||
one. Live traffic showed **0 requests** to `/sharded/*` over ~2h of production
|
||
load, and the two in-repo callers were migrated with this change.
|
||
|
||
### Changed
|
||
|
||
**`/cluster/status` `regions[]` frontiers may now be `null` (wire-visible)**
|
||
|
||
`applied_events` and `lag_events` are now `Option<u64>` — JSON `null` when the
|
||
aggregating node has no frontier report for a region. Previously an unreachable
|
||
peer was reported as `applied_events: 0, lag_events: <leader high-water-mark>`;
|
||
because lag is `hwm − applied`, a 500 ms probe timeout was rendered as the
|
||
leader's entire history as a deficit. Measured live: every node reported both
|
||
peers `partitioned: true, reachable: false, lag_events: 13322258` while the **same
|
||
response's** `shards[]` array showed all three replicas converged at identical
|
||
frontiers with lag 0. An operator reading the obvious field concludes the cluster
|
||
is dead.
|
||
|
||
The root cause was a type that could not express "unknown", so the fix is at the
|
||
type: `null` where the value is not known, and no value is derived from one that
|
||
is not. That covers three former fabrications — the unreachable peer, a leader
|
||
with no ack mark for a peer, and a region whose lag was computed against a leader
|
||
high-water-mark the aggregator never learned. `reachable` and `partitioned` are
|
||
unchanged and still distinguish a genuinely partitioned peer from an unknown one;
|
||
`shards[]` is unchanged (it was the trustworthy half throughout).
|
||
|
||
*Migration:* clients typed against `applied_events` / `lag_events` must accept
|
||
`null`. `tidalctl` renders it as `NO REPORT` / `?` and no longer treats it as a
|
||
deficit, so `tidalctl cluster-status && deploy` is usable as a gate again — a
|
||
known deficit, an unreachable region, a `partitioned: true` region, a pending
|
||
reseed, or a local shard lag still exit 2.
|
||
|
||
### Fixed
|
||
|
||
**Embeddings are validated BEFORE the WAL append; malformed vectors now 400
|
||
(wire-visible)** — *fleet-hardening task 17*
|
||
|
||
`POST /embeddings` appended the blob to the leader's WAL **first** and validated
|
||
it **second**. A probe posted a 128-dimension vector to a slot the live schema
|
||
declares at 1536: the client got a `500`, but the record was already durable, so
|
||
it shipped to both followers, neither could apply it, and both **halted their
|
||
receivers**. Shard 1 froze (leader `13540698`, followers pinned at
|
||
`13540694`/`13540693`, `lag` growing) and writes to the group returned **503** —
|
||
a quorum-write outage from one malformed HTTP request. Neither a restart (boot
|
||
self-heal re-pulls the same record) nor `POST /cluster/reseed` escaped it: the
|
||
snapshot is captured at the leader's applied frontier, which is itself *behind*
|
||
the poison, so the reseeded follower replayed straight back into it. Only forcing
|
||
a leader election recovered the group.
|
||
|
||
Fixed on both sides of the durability boundary:
|
||
|
||
- **Write path.** The dimension check now runs before `wal_blob_first`, against
|
||
the schema-declared slot width and this node's registered width. A mismatched,
|
||
empty, non-finite, or zero-norm vector is rejected **400**
|
||
(`TidalError::invalid_input`) and appends **nothing** — it cannot ship, so no
|
||
follower can be poisoned by it. The comparison is
|
||
`storage::vector::validate_dimensions`, which the apply path's
|
||
`normalize_and_store` also calls: one rule, one definition, because it is the
|
||
apply-path copy that halts replication when the two disagree.
|
||
- **Receive path.** A replicated record that is unapplicable on **every** replica
|
||
— its verdict reads only the record's own bytes and the shared schema — is now
|
||
**skipped** rather than halted on: dropped before it enters this node's own log,
|
||
counted once per record on
|
||
`tidaldb_cluster_blobs_apply_failed_total`, logged at ERROR with the seqno
|
||
range, and the round's remaining records still apply so the frontier advances
|
||
past it. Failures that might apply later still halt, deliberately: a node-local
|
||
disk/lock fault, an unknown batch kind a newer binary understands, or a
|
||
malformed term/membership record whose loss would diverge the roster. Skipping
|
||
those would trade an availability bug for silent data loss.
|
||
|
||
`ReplicatedBlobApplier::apply_blobs` now returns `BlobApplyOutcome { rejected }`
|
||
instead of `()` (in-crate trait; no HTTP or on-disk surface changes).
|
||
|
||
*Migration:* a client that treated a `500` from `/embeddings` as retryable must
|
||
treat the `400` as terminal and fix the vector width. Runbook §5 documented the
|
||
`500` as "a known wart, tracked post-M8" — it was not cosmetic, and §5 plus the
|
||
new §16.6 (halted receiver: recovery by leadership change, and why reseed cannot
|
||
escape a poison record) now say so.
|
||
|
||
**`/feed` and `/search` `rank` is now dense and ascending (wire-visible)**
|
||
|
||
Under full placement — the production shape, every shard group on every node —
|
||
`missing_groups()` is empty and both handlers return straight out of
|
||
`scatter_merge`, which sorted and truncated but never re-stamped `rank`. Each
|
||
hosted group ranks its own slice `1..k` locally, the slices were concatenated and
|
||
score-sorted, and the per-group counters reached the wire as-is: live
|
||
`/search?query=verification` returned ranks **`1, 1, 2`**. The sibling
|
||
`merge_cross_shard` already re-stamped for the partial-placement path and
|
||
documented exactly why; full placement returned before reaching it.
|
||
|
||
`scatter_merge` now takes the same `set_rank` closure and stamps after `truncate`,
|
||
so there is one rank-stamping mechanism rather than two. **Ordering is unchanged**
|
||
— the stamp is a renumbering, not a re-sort, and scores were always correctly
|
||
descending. `/vector_search` carries no rank field and passes a no-op. The `S=1`
|
||
fast path still returns the engine's own already-dense ranks; both shapes are now
|
||
asserted.
|
||
|
||
*Migration:* a client that deduplicated or keyed on `rank` while working around
|
||
the duplicates can drop the workaround. `rank` is `1..n` over the returned page.
|
||
|
||
**`vector_search` distances now honor the documented `[0.0, 4.0]` contract (wire-visible)**
|
||
|
||
The read path passed the caller's raw query vector straight to the index while the
|
||
write path L2-normalized every stored vector, so the two sides lived in different
|
||
spaces. With unit `v`, `d = |q|² − 2q·v + 1`: a non-unit query shifted and scaled
|
||
every distance by `|q|²`. Measured on a live RF3 cluster, `/vector_search` returned
|
||
distances of **591–1174** against a contract promising `≤ 4`, and an exact match
|
||
scored `|q|² − 1` instead of ~0.
|
||
|
||
`db::query_ops::vector_search_items` now normalizes the query with the canonical
|
||
`storage::vector::l2_normalize` before searching. A zero-norm query — which has no
|
||
direction, so "nearest by cosine" is undefined — is rejected with **400** instead of
|
||
returning an arbitrary ranking.
|
||
|
||
**Ranking is unchanged.** `|q|²` and `1` are constant across candidates, so the
|
||
ordering was already correct cosine order; that is why this went unnoticed. Absolute
|
||
uses of the number were the casualties: distance thresholding, dedup-by-distance,
|
||
cross-query comparison, and recall measurement.
|
||
|
||
*Migration:* any client comparing distances against a hard-coded threshold derived
|
||
from the old unnormalized values must re-derive it. An exact match now scores ~0
|
||
(`2.4e-7` measured; the f16 quantization floor is ~`1e-3`, so compare with a
|
||
tolerance, never equality) and all distances fall in `[0.0, 4.0]`.
|
||
|
||
**Blob replication is now observable (`tidaldb_cluster_blobs_*`)**
|
||
|
||
The blob path — item metadata, embeddings, term markers, cluster membership — had
|
||
**zero** instrumentation: `lag_events` counts WAL apply, so a blob that never shipped
|
||
or never applied moved no number anywhere. Added per-kind counters
|
||
`tidaldb_cluster_blobs_originated_total`, `_applied_total` and `_apply_failed_total`,
|
||
labelled by record kind with cardinality fixed at 4 by construction. Comparing
|
||
`originated` on the writing node against `applied` on each peer localises a
|
||
replication gap to enqueue, ship, or apply in one query.
|
||
|
||
**Every co-located shard group now reports its vector count**
|
||
|
||
`tidaldb_usearch_vector_count` was rendered only for the node's metrics-owner group,
|
||
so on a 3-group node two thirds of the corpus had no vector-count series and could
|
||
diverge unobserved. Co-located groups now render
|
||
`tidaldb_usearch_vector_count{shard="N"}`; the owner's series stays unlabeled for wire
|
||
compatibility, so an alert grouped `by (shard)` buckets each replica set separately
|
||
without double-counting.
|
||
|
||
**A converged, unquarantined, voting pod no longer sits permanently `503`, and
|
||
`/health` no longer names a cause it never checked** — *fleet-hardening task 18*
|
||
|
||
During the 2026-08-31 staged roll, `tidaldb-2` logged `latched: 2` reseed markers
|
||
(`from_seqno` `13322280` and `13540699`) and `discharged: 1`, then answered
|
||
`503 {"cause":"joiner boot not yet converged"}` indefinitely while
|
||
`/cluster/status/local` reported `reseed_required=false`, `quarantined=false`,
|
||
`membership_role=voter` and `lag=0` on all three shards — every group having
|
||
already logged *"first-converged against a KNOWN leader frontier ... readiness is
|
||
now sticky-ready"*. It happened on all three pods and each needed a manual
|
||
`kubectl delete pod`.
|
||
|
||
Three defects, all in `tidal-server/src/cluster/node.rs`:
|
||
|
||
- **The latch and the discharge were not the same shape.** A shard group held one
|
||
readiness flag plus one durable marker slot that a re-latch **overwrote**, so a
|
||
group that latched at two different frontiers (a term join at one, a later
|
||
`snapshot-required` refusal at another) let one completed catch-up pull decide
|
||
both — clearing readiness with the lower gap still a hole in one latch order,
|
||
and clearing nothing in the other. Outstanding gaps are now a set keyed by
|
||
`(shard, from_seqno)`, each discharged only by evidence about **itself** (the
|
||
unchanged `ReseedMarker::discharged_by_served_range` — never a frontier
|
||
comparison), and the durable slot is re-derived from what remains as the
|
||
conservative collapse: lowest resume point, least-dischargeable reason. The
|
||
latch WARN now fires once per distinct gap and the discharge INFO once per gap
|
||
cleared, so `latched` and `discharged` count the same population.
|
||
- **A refused `reseed_self_restart` was never re-evaluated.** The §2.4 quorum
|
||
refusal is correct and unchanged — it fired during the roll with
|
||
`alive_voters_excluding_self=1 < majority=2` and is why the cluster stayed up —
|
||
but its only re-drive was *"a future latch re-evaluates"*, and a node whose gap
|
||
is real gets no future latch once the catch-up retry stops refusing. It waited
|
||
forever on a quorum event nobody polled; `kubectl delete pod` performed by hand
|
||
exactly the restart the node had refused. A held refusal now re-evaluates every
|
||
15s until the gap heals, the refusal clears, or the exit fires.
|
||
- **`/health` reported a cause it had not tested.** The 503 ladder tested three
|
||
gates and printed `"joiner boot not yet converged"` for everything else —
|
||
including the latched reseed marker, which was the gate that actually held.
|
||
Every gate now has its own arm; the reseed arm names the outstanding gaps
|
||
(`reseed_marker_latched: shard group 1 holds an unhealed reseed gap. Outstanding
|
||
node-wide: [s1@13540699(snapshot_required)] ...`) and says when a self-restart
|
||
is refused; and the terminal arm reports `unready: reason unavailable` instead
|
||
of inventing a state. A probe that lies about its reason is worse than one that
|
||
admits it has none.
|
||
|
||
The readiness gate itself is unchanged in spirit: a node holding stale data it is
|
||
about to discard still drains from the client VIP (the property that stopped a
|
||
PVC-wiped `tidaldb-0` serving an empty corpus). The bug was that it never stopped
|
||
draining.
|
||
|
||
*Wire-visible:* `/cluster/status/local` and each `shards[]` row gain
|
||
`reseed_gaps` (the outstanding frontiers, lowest first), and the flat/per-group
|
||
`reseed_required` now reports the live gap set rather than the marker file — so it
|
||
can no longer answer `false` while `/health` answers 503 for a latched marker.
|
||
|
||
### Added
|
||
|
||
**Multi-vector user preference modeling + ANN candidate-gen (M12) — a warm user is many interests, not one averaged vector: per-user preference clusters drive a top-M ANN fan-out in `for_you`**
|
||
|
||
- **Online preference clustering (`entities/multi_preference.rs`).** A warm user
|
||
(≥ `COLD_START_N = 5` interactions) maintains up to `K_MAX` preference clusters
|
||
built by online sequential k-means with a DP-means threshold split: a new
|
||
engagement updates its nearest cluster (per-cluster adaptive EMA) or, past the
|
||
split threshold and under the cap, opens a new cluster; at the cap the nearest
|
||
cluster absorbs it. Per-cluster *importance* composes the canonical forward-decay
|
||
kernel anchored to each engagement's timestamp, so stale interests fade.
|
||
- **Top-M ANN fan-out.** At query time `for_you` selects the top-`M` clusters by
|
||
current importance, issues `M` ANN queries (`candidate_gen::ann_candidates_multi`),
|
||
and merges by best (min) distance; the personalization boost is the max cosine
|
||
over all clusters. Users below the cold-start threshold keep the single
|
||
adaptive-LR vector (`entities/preference.rs`). Design: `docs/research/multi-vector-preference.md`.
|
||
|
||
**Idle-readiness + TLS scale-up (m12p5–m12p6) — followers converge readiness on an idle cluster, and elasticity is proven over REAL mTLS on k8s**
|
||
|
||
- **Idle-readiness convergence (m12p5).** The leader heartbeat now carries its live
|
||
frontier (`leader_last_seq`), so a caught-up follower flips `/health` Ready on an
|
||
idle cluster instead of stalling until the next status poll or ship. Cert SAN
|
||
wildcard widened for scale-to-5.
|
||
- **TLS scale-up (m12p6).** A real `kubectl scale 3→5` exercised seed-join over
|
||
mTLS on k8s (kind) for the first time: joiners flip Ready in ~13s via the
|
||
idle-readiness heartbeat, auto-promote to Voter, reach full content parity at
|
||
lag 0, with zero acked DATA loss across scale-down. Fixes span a six-bug chain —
|
||
`https://` seed scheme, rustls `CryptoProvider` install order, headless seed
|
||
Service, cold-handshake poll timeout, two-tier cert-manager PKI, and the
|
||
`grpc_tls_for` CA fallback for a not-yet-in-topology joiner. Persists the HNSW
|
||
graph and skips a suspect graph on reseed-pending close.
|
||
|
||
**Sharded ingestion (m12p4) — scatter-gather across shard groups with cross-shard unified reads**
|
||
|
||
- **Scatter-gather pool + cross-shard reads.** Writes hash-route across a 3-group
|
||
`shards:` topology; reads unify across groups (L4). Ran REAL on kind with a
|
||
2-generator load job. Fixed an HTTP/2 204 forward-relay bug (a synthesized JSON
|
||
body on a 204 relay triggered an h2 `RST_STREAM`). Confirmed with data: at fixed
|
||
per-pod CPU, full-placement sharding scales failover, not write throughput;
|
||
the ≥2.5×-and-≥5,000/s scaling target remains Ref-A/k3s-pending.
|
||
|
||
**Index tuning + recall/memory at the production shape (m12p3) — the G2 work: per-query `ef_search` is now honored, the brute-force crossover scales with dimensionality, and the HNSW recall/latency/memory frontier is measured at 1536-D with a real exact oracle**
|
||
|
||
- **Per-query `ef_search` override — now real.** `UsearchIndex::search` /
|
||
`filtered_search` honor a per-request `ef_search` instead of silently dropping
|
||
it to the index default (pre-m12p3 behaviour: accepted for trait compliance,
|
||
logged a warning, ignored — `USearch` 2.24 has no per-call beam argument). The
|
||
override is race-free via an `RwLock` epoch guard (`with_expansion`): searches
|
||
that agree on `ef_search` run in parallel under a shared guard; only a query
|
||
that changes the live beam width takes the exclusive guard for its
|
||
`(set, search)` window — not a per-search mutex. The knob was already plumbed
|
||
end-to-end in m12p1 (`vector_search_items(.., ef_search)`, the `/vector_search`
|
||
`ef_search` field, `tidal-stress --recall-ef-search`); m12p3 makes it move
|
||
recall. `ef_search=0` selects the slot default.
|
||
- **Dimension-aware brute-force → HNSW crossover.** The exact `BruteForceIndex`
|
||
scans every vector under a read lock at `count × dim` cost, so a fixed 10,000
|
||
crossover meant a 15.4M-FMA scan at 1536-D (tens of ms, blocking writers).
|
||
`usearch_min_vectors(dim)` now keeps a brute-force scan within ~4M FMAs:
|
||
≈10,000 at/under 128-D (byte-compatible with pre-m12p3), ≈2,600 at 1536-D —
|
||
flipping high-dim mid-size slots to HNSW before the scan blows the SLA.
|
||
- **`memory_usage()`** exposed on `UsearchIndex` (the true graph + vector
|
||
footprint from `USearch`, not the `index_stats` lower bound) for pod sizing.
|
||
- **Grid-search harness.** `cargo run --release --example ann_grid_search` builds
|
||
a `UsearchIndex` + an exact `BruteForceIndex` oracle over the same
|
||
deterministic id-keyed corpus and reports, per `(M, ef_construction, ef_search,
|
||
quantization)` point, measured recall@10 vs the oracle, mean/p99 search
|
||
latency, build time, and the true footprint — the tool that produces the
|
||
documented `M`/`ef` and the F32/F16/Int8 recall+memory numbers.
|
||
- **Measured at 1536-D (100k clustered corpus, real exact oracle).** The
|
||
production default (M=16, ef_c=400, F16) **clears G1 and G2**: recall@10 0.997
|
||
at p99 ≈ 1.4 ms raw ANN; ef_search is the latency lever (recall saturates by
|
||
ef_s=128 → p99 ≈ 1.0 ms). F16 costs only 0.25% recall vs F32 for half the RAM
|
||
(≈ 5.2 GB/1M true footprint incl. graph); **Int8 rejected** at 1536-D (recall
|
||
0.715, −28%). Live `tidal-stress --verify-recall` against a real server gave
|
||
recall@10 = 1.0000 at 20k/1536-D, default beam and `--recall-ef-search 400`.
|
||
Finding: the recall corpus is now **clustered** (Gaussian mixture) in both the
|
||
grid harness and `tidal-stress` (`recall::embedding_for`) — uniform-random
|
||
high-dim vectors are pathological for recall@k (≈0.97 at 10k → ≈0.54 at 100k, a
|
||
measurement artifact, not an index regression).
|
||
Full frontier + the 1M command in
|
||
[docs/profiling/usearch-tuning.md](docs/profiling/usearch-tuning.md) and
|
||
[docs/profiling/scale-baselines.md](docs/profiling/scale-baselines.md).
|
||
`docs/specs/07-vector-retrieval.md` updated: per-query `ef_search` is
|
||
IMPLEMENTED, not deferred.
|
||
|
||
**ANN candidate generation in RETRIEVE (m12p2) — the G1 unblock: `for_you`/`related` source candidates by nearest-neighbour, `trending` by a cached per-signal-type top-K, so the feed stays relevant AND bounded as the corpus grows past the scan cap**
|
||
|
||
- **ANN in retrieve.** `CandidateStrategy::Ann` is now wired into the RETRIEVE
|
||
executor (it previously fell back to a scan with a warning). The db layer
|
||
resolves the query vector — the user's **preference vector** for `for_you`, the
|
||
**seed item's embedding** (`similar_to`) for `related` — and Stage 1 runs an
|
||
`O(ef_search)` HNSW search over the item content slot instead of scanning an
|
||
arbitrary low-id slice of the universe. `for_you` and `related` are now `Ann`
|
||
profiles. Graceful: no registry / no preference vector / no seed ⇒ it degrades
|
||
to a scan (anonymous reads and cold-start users still serve), so every
|
||
embedding-less schema and pre-m12p2 caller is unchanged.
|
||
- **Cached `SignalRanked`.** The O(N) ledger scan behind `SignalRanked` is now a
|
||
cached per-signal-type top-K (`signals/ledger/hot_top_k.rs`): O(K) on the
|
||
served path, with a bounded O(N) rebuild only when stale. Decay preserves
|
||
relative order (same λ), so a cache is valid until the next write; small
|
||
ledgers rebuild on any write (always fresh), large ledgers throttle the rebuild
|
||
off the read hot path (1s). `trending` now uses `SignalRanked(view)`, so it
|
||
ranks the actually-viewed corpus at any id — not the low-id scan slice.
|
||
- **`related` over HTTP.** `GET /feed?profile=related&similar_to=<id>` resolves
|
||
the seed's embedding and runs ANN — "more like this" on the read surface
|
||
(`similar_to` added to `FeedQuery`, threaded through all three feed handlers).
|
||
- **Harness `/feed` measurement.** `tidal-stress` gains `--feed-profile <name>`
|
||
(force every feed read to one profile, for per-profile retrieve p99) and
|
||
`--seed-preferences` (build a preference vector per user so `for_you` exercises
|
||
ANN, not the scan fallback).
|
||
- **Verified real** against a 1536-dim standalone server: **trending retrieve
|
||
p99 = 3.5–7.7ms** (under the 10ms G1 target) under concurrent writes, via the
|
||
cached top-K; `for_you` retrieve is ANN-backed (preference vectors built) at
|
||
p99 ≈ 24ms, dominated by the Stage-3 preference-boost recompute (per-candidate
|
||
embedding read) — flagged for m12p3's index/score tuning (the risk register's
|
||
"materialized-score layer"). ANN candidate recall is the m12p1 `/vector_search`
|
||
probe (0.9997). New engine tests prove ANN/`related`/`trending` reach the
|
||
relevant items at high ids a scan can never reach, plus cache freshness.
|
||
|
||
**Read-recall harness — measurement truth for ranking-at-scale (m12p1): recall@k + true p99 at the production shape, so G1/G2 steer on real numbers instead of absent ones**
|
||
|
||
- **Pure k-NN probe.** `TidalDb::vector_search_items(query, k, ef_search)` returns
|
||
the raw HNSW nearest neighbours over the item content embedding slot — NO
|
||
profile scoring, fusion, or diversity — so the result is the ANN index quality
|
||
in isolation (the G2 metric). Exposed as `POST /vector_search` on the standalone
|
||
router and the multi-process region node (merge-by-distance across hosted shard
|
||
groups); a dimension-mismatched query is a 400, not a 500.
|
||
- **The oracle.** `tidal-stress --verify-recall` seeds the corpus with
|
||
**deterministic, id-keyed** embeddings (reproducible with `--skip-seed`), holds
|
||
a **brute-force cosine ground truth** in RAM, and ramps `/vector_search` probes
|
||
**open-loop** (coordinated-omission corrected). It reports per-stage **true p99**
|
||
(a genuine tail, not a closed-loop mean) AND **mean recall@k**, plus the
|
||
**read-knee** — the highest sustained QPS where `p99 ≤ target AND recall@k ≥
|
||
target` both hold — with a machine-readable JSON summary and `--fail-on-knee`
|
||
PASS/FAIL exit. Knobs: `--recall-k`, `--recall-queries`, `--read-p99-target-ms`,
|
||
`--recall-target`, `--recall-ef-search`.
|
||
- **Verified end-to-end** against a real standalone server at 1536-dim:
|
||
**recall@10 = 0.9997** at 20k items (HNSW M=16/ef=400/F16 vs brute-force cosine,
|
||
far above the 0.95 G2 target); the harness also exercises the read-knee verdict
|
||
(both branches) and gate exit codes. The 100k/1M exit-gate runs use the same
|
||
harness on the k3s cluster (the brute-force oracle needs ~6 GB RAM at 1M).
|
||
- **No more mean-as-p99.** Repaired the fabricated p99 column in
|
||
`docs/profiling/social-scale.md` (the `social` bench is Criterion = mean only)
|
||
and the `scale.rs` / scale-baselines framing; every closed-loop number is now
|
||
labelled an *isolated per-op mean (regression tripwire)*, with the p99/recall
|
||
tail SLOs signed off only by the open-loop harness. Added a 1536-dim
|
||
HNSW-vs-brute `recall@10` bench (`benches/vector.rs`) and a recall-harness
|
||
section to `docs/profiling/scale-baselines.md`.
|
||
|
||
**Sharding × replication + rebalancing (m11p6) — the "replicated XOR sharded" split is over: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes**
|
||
|
||
- **One write surface.** `/items`//`/embeddings`//`/signals` now hash-route the
|
||
entity to the owning shard group's leader (the engine's FNV-1a `ShardRouter`)
|
||
AND replicate at RF — sharding and replication at once. The old
|
||
`/sharded/*`-vs-leader split in `tidal-stress` (`WritePath::Leader|Sharded`) is
|
||
gone; it drives the one unified path, spreading writes round-robin across
|
||
gateways (or pinning one with `--leader-url`). `x-tidal-ack`/`x-tidal-seq`,
|
||
quorum await, and `NotLeader`/`QuorumTimeout` are per-group; `NotLeader` names
|
||
the group.
|
||
- **Rebalancing verbs (L3).** `POST /cluster/shards/{id}/transfer` moves one
|
||
group's leadership (the m11p4 fenced transfer scoped to the group);
|
||
`POST /cluster/shards/{id}/replicas` adds/removes a replica (the m11p5 join /
|
||
fenced-removal per group). A `?shard=` selector threads through every per-shard
|
||
admin verb (promote/heal/partition/catchup/reseed/members/join) and is
|
||
PROPAGATED on every intra-group forward/broadcast (`ShardReplica::admin_path`)
|
||
so the receiving sibling targets the same group. **S=1 is byte-for-byte** (no
|
||
selector emitted, no shard in the `NotLeader` body).
|
||
- **Tier-3 exit gate** (`cluster_sharding.rs`): 3 nodes × 3 shards × RF=3 over
|
||
real OS processes — SIGKILL a node under `ack=quorum` load → ONLY its
|
||
shard-leaderships re-elect (survivor groups keep theirs), reads never stop, and
|
||
every acked write is present on its shard's new leader (zero acked loss), across
|
||
random kill points. Plus a rebalance-verb test (transfer + `?shard=` promote
|
||
move exactly one group). Harness: `MultiProcCluster::start_sharded`
|
||
(per-(node,shard) ports, `shards:` emission, `agreed_shard_leaders`).
|
||
- **Throughput.** Local 3×3 release cluster sustains 3,000 quorum signal-writes/s
|
||
at 0% error with per-node CPU ≈30% and replication lag ~0 — generator-bound,
|
||
not engine-bound. The ≥5,000/s + ≥2.5×-single-shard scaling is Ref-A (k3s
|
||
Linux/`fdatasync`/multi-generator) — the standing access caveat since p1.
|
||
- Known follow-up (tracked): per-group-aware node readiness (today `is_ready` is
|
||
node-global across co-hosted groups) and cross-node read fan-out under PARTIAL
|
||
placement; the exit gate runs full placement, which these do not touch.
|
||
|
||
**Continuous correctness (m11p9) — new fault classes as REAL faults, first-class invariant checkers, soak regression gates, a Woodpecker nightly chaos+soak pipeline, and a guarantee→test matrix**
|
||
|
||
- **Fault injection compiled out of production.** A non-default `fault-injection`
|
||
cargo feature (tidaldb + a tidal-server passthrough) adds two WAL hooks
|
||
(`tidal/src/fault.rs`): **slow-fsync** (`TIDAL_FAULT_FSYNC_DELAY_MS` sleeps
|
||
before each durable fsync) and **disk-full** (`TIDAL_FAULT_DISK_FULL_AFTER_BYTES`
|
||
returns a real `ENOSPC`/errno-28 once cumulative segment bytes cross the
|
||
threshold). The production image build never passes the feature, so the hooks
|
||
are compiled out entirely — a disk/fsync fault a stray env var could trip in
|
||
prod is a 3am footgun we refuse to ship; the safety is structural. Inert until
|
||
armed even when compiled in. The tier-3 harness builds the spawned binary with
|
||
the feature.
|
||
- **First-class invariant checkers** (`tidal-server/tests/support/invariants.rs`):
|
||
the no-acked-loss `AckLedger` (frontier + content, extracted from the m11p3
|
||
ledger gate, which now consumes it), cross-replica `assert_feed_parity` /
|
||
`feed_item_ids`, `assert_single_leader_per_term` (membership safety, reads the
|
||
m11p6 `shards[]` rows), and `MonotonicCounters` (per-node applied frontier +
|
||
leader commit index never regress, with legitimate epoch resets forgiven).
|
||
- **New fault-class suite** (`tidal-server/tests/cluster_faults.rs`, tier-3): a
|
||
disk-full follower degrades gracefully (receiver halts, node alive, zero acked
|
||
loss, restart recovers to parity); a slow-fsync follower lags then converges
|
||
while the fast follower supplies quorum; both followers slow → `ack=quorum`
|
||
returns a retryable 503 naming the laggards while `ack=leader` is unaffected;
|
||
an asymmetric partition (inbound severed, outbound up) causes no split brain
|
||
(pre-vote + check-quorum) and no loss. 4/4 green.
|
||
- **Soak + regression gates** in `tidal-stress`: `--json-summary <path>` (a
|
||
machine-readable per-stage p99/throughput/error roll-up for trend lines) and
|
||
`--max-p99-ms` / `--max-error-pct` / `--fail-on-knee` gates that exit non-zero
|
||
on a regression (the tool always exited 0 before).
|
||
- **Nightly pipeline** (`.woodpecker.yaml`): a cron `nightly` flow (chaos suites
|
||
with elevated kill-points + the `fault-injection` feature, then the gated
|
||
`tidal-stress` soak) beside the existing push release gate, event-routed by
|
||
per-step `when`. Woodpecker, never GitHub Actions.
|
||
- **Guarantee traceability** (`docs/planning/milestone-11/guarantee-traceability.md`):
|
||
every roadmap §2 guarantee mapped to its named automated test(s). Closes the
|
||
**G-C** apparatus; the 30-consecutive-days-green half of the GA bar is a
|
||
calendar criterion the nightly pipeline accrues.
|
||
|
||
**Observability + operations (m11p8) — complete cluster metric set on a per-node `/metrics` listener, request-id/tracing across hops, truthful status, self-driving heal, WAL PITR + backup/restore, rolling-upgrade release gate**
|
||
|
||
- **Metrics.** Completed the `tidaldb_cluster_*` set with the two members the
|
||
roadmap named that were missing — **breaker** (`tidaldb_cluster_peer_breaker_state`
|
||
0/1/2 + `tidaldb_cluster_breaker_opens_total`, surfaced read-only from the
|
||
`tidal-net` circuit breaker via a new `Transport::peer_breaker_state`) and
|
||
**forwards** (`tidaldb_cluster_forwards_total` / `_forward_failures_total`,
|
||
instrumented at the gateway forward path) — plus the self-heal series
|
||
(`heal_attempts/successes/noops_total`, `healing_peers`). When several shard
|
||
groups co-locate on one node (m11p6) the metrics-owner serves the single
|
||
`/metrics` listener and the siblings register their series under a `shard="N"`
|
||
label (`TidalDb::register_metrics_sibling`); a single-shard node is
|
||
byte-identical to before. A 12-panel "Cluster Replication" Grafana row +
|
||
an 8-rule `tidaldb-cluster` Prometheus alert group ship beside the standalone
|
||
ones.
|
||
- **Request-id + tracing.** Both cluster routers (single- and multi-process) now
|
||
carry the standalone router's `SetRequestId` + `PropagateRequestId` +
|
||
`TraceLayer` stack (extracted to `router::with_request_id_tracing`); the id
|
||
rides the follower→leader forward hop verbatim, so the leader's span shares the
|
||
gateway's `x-request-id`. (Ships are off-request-path and batched, so they
|
||
correlate by seqno, not a request-id — by design.)
|
||
- **Truthful status.** Fixed the leader's own `applied_events` reading 0 (a leader
|
||
writes its WAL directly and never advances its own applied frontier — now
|
||
reports its flushed frontier) and the single-process post-promote `ShardId(0)`
|
||
lag-keying undercount (now keys on the current leader's shard).
|
||
- **Self-driving heal.** A standing leader duty re-arms the backlog re-ship for a
|
||
stuck (breaker-open, behind) peer every ~3s, so it converges through breaker
|
||
resets with no operator `/cluster/heal` loop — closing the §1.4-3 footgun.
|
||
Observable via `healing_peers` + the `TidalDBClusterHealNotConverging` alert.
|
||
- **WAL PITR archival.** `wal.archive_dir` (topology + builder): the online
|
||
compaction copies each sealed segment to the archive — durably, before deletion,
|
||
refusing to delete if archival fails — so the archive is a gap-free PITR record.
|
||
- **Backup/restore.** `tidalctl backup` / `restore`: offline data-dir backup with
|
||
a BLAKE3 `BACKUP_MANIFEST.json` (+ the WAL checkpoint cursor); restore verifies
|
||
every file's hash before writing and refuses a non-empty target. Coordinated
|
||
cluster backup = back up one committed replica per shard group (the runbook
|
||
drill).
|
||
- **Rolling upgrade.** A wire version handshake (`HeartbeatRequest.build_version`,
|
||
stamped at the transport boundary; `>= 2`-major skew WARNs, never rejects) +
|
||
`version` on `/cluster/status`. `mp_rolling_upgrade_no_loss_no_stall` promoted
|
||
to the FIRST step of `.woodpecker.yaml` (the release gate; a failure blocks the
|
||
image build). See [milestone-11/phase-8.md](docs/planning/milestone-11/phase-8.md).
|
||
|
||
**Security hardening (m11p7) — mTLS by default + zero-drop cert rotation, per-node identity, admin audit log, per-principal rate limit**
|
||
- **The cluster stops trusting the network.** gRPC replication mTLS is now the
|
||
intended posture: the inbound server is served over a custom `tokio-rustls`
|
||
acceptor (not tonic's fixed `.tls_config()`) fed a `DynamicCertResolver`
|
||
(`ArcSwap<CertifiedKey>`), preserving mutual TLS exactly (a `WebPkiClientVerifier`
|
||
over the cluster CA — a foreign/absent client cert fails the handshake before
|
||
any RPC). Plaintext is an explicit `insecure: true` with a loud startup WARN.
|
||
- **Cert + bearer rotation WITHOUT restart.** A content-hash poller re-reads the
|
||
cert files (k8s `..data` symlink swaps that inotify misses) and atomically
|
||
swaps the resolver's cert; in-flight TLS sessions keep their negotiated keys,
|
||
so a rotation **drops zero requests** (verified under concurrent load). Outbound
|
||
peer channels rebuild from the refreshed files. The bearer and a new shared
|
||
cluster key live behind `ArcSwap`, read per request and reloaded from
|
||
`TIDAL_API_KEY_FILE` / `TIDAL_CLUSTER_KEY_FILE`.
|
||
- **Authenticated inter-node HTTP with per-node identity.** The axum listener
|
||
serves TLS (reusing the same hot-swappable resolver — one rotation covers both
|
||
planes); forwards/broadcasts/scatter/status/seed-join dial `https://` with the
|
||
cluster CA. A forwarding node mints an `x-tidal-node-token` (keyed-BLAKE3 MAC
|
||
over node-id + expiry under the cluster key — no new crypto dependency) so a
|
||
foreign pod cannot forge a sibling identity. The `x-tidal-internal` marker is
|
||
now honored ONLY from a verified sibling (marker without a valid node token →
|
||
403): the marker stays a routing hint, never an authorization bypass. All
|
||
opt-in via `grpc_tls` / the cluster key — absent ⇒ pre-m11p7 behavior, so every
|
||
existing deployment and test is byte-for-byte unchanged.
|
||
- **Admin-verb audit log.** promote / partition / heal / join / member-remove /
|
||
reseed each emit one structured record (principal, term, target, outcome) to a
|
||
`tidal_audit` tracing target + an optional append-only JSONL file
|
||
(`TIDAL_AUDIT_LOG`), on the operator-originated leg only (no double-audit on the
|
||
forwarded re-apply).
|
||
- **Per-principal HTTP rate limit.** The engine's token-bucket `RateLimiter`
|
||
(now re-exported from the crate root) gates all three routers keyed by
|
||
principal; verified sibling nodes are exempt (replication is never throttled);
|
||
a deny is 429 + `Retry-After`. Off by default (`TIDAL_RATE_LIMIT_RPS`).
|
||
- **Reference deployment + tooling.** `k8s/cluster/` gains cert-manager
|
||
Issuer/Certificate, the cert + cluster-key Secret mounts, and the per-region
|
||
`grpc_tls` topology block; `scripts/gen-cluster-certs.sh` (openssl) provisions
|
||
the same Secret shape without cert-manager. Exit gate verified: foreign pod
|
||
rejected (gRPC handshake + HTTP), zero-drop rotation under load, zero plaintext
|
||
inter-node links. See [docs/planning/milestone-11/phase-7.md](docs/planning/milestone-11/phase-7.md).
|
||
|
||
**Membership, discovery, elasticity (m11p5) — DNS peers, snapshot reseed, conf-changes on the one log, seed join, k8s reference**
|
||
- **Nodes are cattle; topology is data, not files.** `grpc_addr` is now an
|
||
**advertised** address — a hostname or an IP — split from a new optional
|
||
per-region `grpc_bind` (the local SocketAddr to bind). A literal `grpc_addr`
|
||
binds itself byte-for-byte as before (every existing topology keeps working);
|
||
a hostname binds `0.0.0.0:<port>`. `tidal-net`'s peer map retypes from
|
||
`SocketAddr` to `String`, so `Channel::from_shared` makes hyper **re-resolve
|
||
DNS on every reconnect** — the pod-rescheduled-onto-a-new-IP case the
|
||
`SocketAddr::parse`-only constraint made structurally impossible is now fixed
|
||
by construction. One shared topology file can name every region by its per-pod
|
||
DNS name (the per-pod static-ClusterIP ConfigMap hack dies). DNS peer names
|
||
require DNS-SAN certs under mTLS (SNI follows the URI host; documented, no code
|
||
change). Proven by `cluster_membership.rs::mp_dns_hostname_topology_replicates`
|
||
(a hostname topology a pre-p5 binary would have refused to parse boots,
|
||
replicates, and survives SIGKILL+restart) plus unit derivation-table tests.
|
||
- **A new node joins via snapshot + stream and serves quorum in minutes.**
|
||
`FetchSnapshot` is a new server-streaming RPC in the `WalShipping` service:
|
||
the leader stages `TidalDb::create_backup` under `<data_dir>/snapshots/` and
|
||
streams it term-stamped + term-fenced exactly like `StreamSegments`
|
||
(manifest + file chunks, each file BLAKE3-verified, identify-or-refuse end to
|
||
end). The handler answers a **no-snapshot-needed** header when the WAL can
|
||
still serve the puller's range and streams the artifact only when it cannot —
|
||
the `needed` decision is **baseline-aware** (`from_seqno ≤ stream_baseline OR
|
||
< earliest WAL seq`), because a "the WAL still covers it" answer is a lie at
|
||
or below a baseline the stream clamp will never serve, and looped marker boots
|
||
forever. Install is a **boot-time** operation (never a live data-dir swap):
|
||
fetch into a sibling staging dir, copy the node's own identity files
|
||
(`election_state`, `membership`) INTO staging, write a `COMPLETE` sentinel,
|
||
then rename — every crash window is an idempotent redo. A retention pin taken
|
||
at backup **start** keeps the segment chain covering the artifact alive while
|
||
any joiner streams, with a hard cap + `tidaldb_cluster_snapshot_pin_force_drops_total`
|
||
so a dead joiner can't freeze compaction forever.
|
||
- **Reseed is self-healing — no operator verb, no `wipe_data_dir`.** A running
|
||
follower that hits a **typed** snapshot-required refusal (the `StreamSegments`
|
||
trailer is now structured — `x-tidal-catchup: snapshot-required | rejoin |
|
||
stepping-down` — so an ordinary election term-mismatch never latches reseed
|
||
markers fleet-wide; the marker latches **only** on `snapshot-required`), or the
|
||
m11p4 divergence quarantine, durably latches `reseed_required` (ElectionStore
|
||
file discipline), surfaces it in `/cluster/status/local` and the
|
||
`tidaldb_cluster_reseed_required` gauge, and keeps serving degraded with
|
||
**voting enabled** (a reseed needs a leader, and the leader may need this
|
||
node's vote — reseed-blocks-voting would deadlock the cluster). The reseed
|
||
runs on the next boot; `replication.reseed_self_restart: true` (default
|
||
**false**; k8s sets it `true`) drains and clean-exits once the marker latches —
|
||
**refused** when the remaining voters can't sustain quorum without this node.
|
||
A successful reseed boot clears the quarantine latch and
|
||
`tidaldb_cluster_divergence_quarantined` — closing the m11p4 carried hazard
|
||
that quarantine clearing "rides m11p5's reseed machinery." `cluster_reseed.rs`
|
||
proves the full quarantine → marker → restart → converged → gauges-cleared
|
||
loop with no `wipe_data_dir` in the test.
|
||
- **The three-way term-join rule closes p4's last carried hazard** (pre-baseline
|
||
history was unreachable through a term's stream for followers behind at the
|
||
transfer). The first reseed drill exposed that such a follower jumps its
|
||
frontier past pre-baseline history with `lag=0` and never asks for the gap —
|
||
silently missing data with nothing to fire the refusal. The heartbeat now
|
||
carries the leader's election-time position in the previous stream's numbering
|
||
(`prev_log`), and the join check is three-way: `own > prev_log` → divergent
|
||
suffix → quarantine (p4); `own < prev_log` → genuinely missing committed-era
|
||
history → **latch `reseed_required`** (new); `own == prev_log` (or within-term
|
||
rejoin) → clean. A snapshot-installed node joins clean by construction (its WAL
|
||
is the leader's copy, so `tail_term` equals the leader's term).
|
||
- **Membership is data on the one log: kind-4 records, learner → voter.** A new
|
||
`MembershipRecord` WAL blob kind (kind-4, beside kind-0 signals, kind-1/2
|
||
item/embedding blobs, kind-3 term markers) is journaled by the leader,
|
||
replicated through the normal stream, and folded by followers into a
|
||
`ClusterMembership` cell (the `WalTermMark` pattern). Records carry the FULL
|
||
roster (latest record wins; no merge logic); membership **epoch 0 = the
|
||
topology file**, so a cluster with no kind-4 record behaves byte-for-byte as
|
||
today. `POST /cluster/join {name, grpc_addr, http_addr}` on any node forwards
|
||
to the leader, which assigns `id = max(all ids ever) + 1`, appends a **Learner**
|
||
record, and answers only after it is quorum-committed (idempotent by name);
|
||
member ids are **permanent** — removal tombstones (a `Removed` record) keep ids
|
||
burned, never renumbered or reused. **Auto-promotion is a standing leader duty**
|
||
(re-armed on every activation and apply, evaluated on commit-index publishes,
|
||
driven solely from the applied `ClusterMembership` — it survives the
|
||
joining-era leader's death) that promotes a learner within
|
||
`replication.learner_promote_lag` (default 1024) of the flushed frontier, or
|
||
the Raft "rounds stop shrinking" criterion. Conf-changes are Raft single-server
|
||
(one at a time, each same-term-quorum-commit-gated); on every leadership
|
||
activation the new leader **re-appends its full current membership** immediately
|
||
after the kind-3 term marker (the Raft no-op-entry analogue), which closes both
|
||
the vacuous-commit-gate and the baseline-jump-skip blockers at once. Verbs:
|
||
`GET /cluster/members`, `POST /cluster/members/remove`, `POST /cluster/reseed`.
|
||
- **The even-voter-count `majority()` arithmetic was a latent bug — fixed.**
|
||
`ElectionConfig::majority()` was `peers.len()/2 + 1`: correct at n=3, **wrong
|
||
for every even voter count** (n=4 → 2-of-4, so disjoint quorums {A,B} and {C,D}
|
||
could elect two leaders in one term; n=2 → 1-of-2, a follower self-elects with
|
||
zero RPCs). `CommitIndex` already used the correct `div_ceil` form — the two
|
||
formulas in one subsystem disagreed. Since p5 makes even sizes mandatory
|
||
transit states (3→4→5→4→3), this landed **first**, with even-n property tests:
|
||
`majority() = (peers.len() + 1).div_ceil(2) + 1` (true `floor(n/2)+1` over the
|
||
full voter set), spelled to read identically to the commit-index site so they
|
||
can never drift again.
|
||
- **Learner marks never count toward quorum.** A role-blind `CommitIndex` that
|
||
saw learner marks would treat two learners "committing" a write no voter holds
|
||
as acked — acked-write loss. `CommitIndex` is now role-aware: voter marks feed
|
||
the k-th-largest selection; learner marks live in a side map (promotion input,
|
||
transfer-wait input) and are excluded from `needed`. In-flight `ack=quorum`
|
||
waits are **re-evaluated** against a new config on every conf-change (a shrink
|
||
may satisfy waiters instantly), never failed; a report from an unknown id logs
|
||
at WARN with a counter instead of being silently dropped. `ElectionState`,
|
||
`ShipQueue`, and `PeerPool` reconfigure behind one fenced apply path so the
|
||
four peer-set copies can never disagree about the roster.
|
||
- **Seed-join boot.** `tidal-server cluster --region <name> --seed
|
||
http://host:port (repeatable) --advertise-grpc host:port --advertise-http
|
||
host:port --metrics addr`: the joiner skips the "every region declared"
|
||
topology gate, learns its roster + assigned id + current term from any
|
||
reachable seed, persists them to a durable membership cache + `election_state`
|
||
(persist-before-act), and installs a snapshot when behind. A restart boots from
|
||
the cache without the seed. A `--seed` boot **still requires the local
|
||
topology/config file** for the behavioral knob blocks (`replication:`, `wal:`,
|
||
`election:`, `timeouts:`, `grpc_tls`) — a bare `--seed` with neither
|
||
`--topology` nor `TIDAL_CONFIG` refuses to boot naming the rule (so a joiner
|
||
never silently inherits the wrong ack default or election timing).
|
||
- **Kubernetes reference: one StatefulSet.** `k8s/cluster/` (namespace
|
||
`tidaldb-cluster`, `replicas: 3`, `podManagementPolicy: Parallel`,
|
||
topology-spread, PDB `maxUnavailable: 1`) with ONE shared bootstrap topology
|
||
ConfigMap (per-pod DNS advertised addresses, `0.0.0.0` binds). Scaling past 3
|
||
does NOT edit it — pod N≥3 boots `--seed` + the same mounted file and joins as
|
||
a learner; rolling node replace is `kubectl delete pod` (PVC retained → boot
|
||
catch-up) or PVC-delete + pod-delete (fresh reseed via snapshot). Readiness:
|
||
`converged` (boot catch-up completed at least once AND `lag ≤
|
||
learner_promote_lag` — hysteresis, never `lag == 0` which an open-loop load
|
||
keeps perpetually false); a restarted existing voter is Ready on today's terms.
|
||
- Exit-gate suites (tier-3, `cluster-e2e`): `cluster_membership.rs`
|
||
(`mp_seed_join_snapshot_catchup`, `mp_scale_3_5_3_under_load_zero_loss` —
|
||
`lost=0`, max acked seq 1467, p99 impact <2× across the 3→5→3 joins,
|
||
`mp_dns_hostname_topology_replicates`) and `cluster_reseed.rs` (quarantine →
|
||
marker → restart → converged, gauges cleared). Localhost-loopback catch-up:
|
||
5000 heavy items join→converged 26.4s (large headroom under the ≤5-min budget);
|
||
the 100k-item Ref-A figure and the k8s pod-reschedule drill remain Ref-A line
|
||
items (k3s access still pending — the standing M11 caveat).
|
||
- **Mixed-version / downgrade caveats.** A kind-4 record shipped to a **pre-p5
|
||
follower** is an unknown batch kind → `WalError::Corruption` → its receiver's
|
||
torn-state halt latch, **permanent across restarts** (both followers halted =
|
||
quorum-write outage). So `HeartbeatResponse`/`ReportApplied` carry a
|
||
`capabilities` bit-field (proto3 zero-default = pre-p5 = incapable) and the
|
||
leader **refuses `JoinCluster` and every conf-change until all current voters
|
||
report kind-4 capability** — "complete the binary upgrade before the first
|
||
conf-change" is structurally enforced, not operator discipline. Pre-p5 peers
|
||
answer `Unimplemented` to `JoinCluster`/`FetchSnapshot` (the joiner reports it
|
||
loudly and retries the next seed). Downgrade rule (kind-3 precedent verbatim):
|
||
once any kind-4 record is in a node's WAL, **downgrade below p5 requires a
|
||
reseed**.
|
||
|
||
**Automatic failover: failure detection, Raft-style election, term fencing (m11p4) — closes ROADMAP gap G5**
|
||
- **"A machine died" is a non-event.** Every multi-process cluster node runs a
|
||
purpose-built election-only Raft (pre-vote + vote + check-quorum + fenced
|
||
leadership transfer) over the existing `tidal-net` transport: leader
|
||
heartbeats every 300ms (config `election.heartbeat_interval_ms`); a follower
|
||
that hears no leader for a randomized 1500–3000ms starts a pre-vote; a
|
||
majority elects. SIGKILL the leader under `ack=quorum` load → a survivor is
|
||
elected and writes resume in well under a second locally, with **zero
|
||
acknowledged-write loss across repeated random kill points** (tier-3 gate
|
||
`cluster_election.rs::mp_auto_failover_writes_resume_zero_acked_loss`).
|
||
No raft crate: the WAL is already the replicated log (m11p2) and the quorum
|
||
commit index (m11p3) already proves majority durability — the election adds
|
||
only the tiny consensus state `(term, leader)` on top of them.
|
||
- **The WAL carries its own election history.** An elected leader's FIRST log
|
||
entry is a kind-3 **term-marker record** that replicates like any record, so
|
||
every replica's `(lastLogTerm, lastLogIndex)` — Raft's vote restriction — is
|
||
derived from one fsync stream and can never disagree across a crash. The
|
||
frontier half is compared in the **last joined term's stream numbering**
|
||
(a reseeded node's own WAL numbering diverges from the stream's after a
|
||
baseline jump; comparing raw local frontiers across nodes would let a
|
||
behind node win). Hard state `(current_term, voted_for)` persists in
|
||
`data_dir/election_state` (magic + version + checksum; corrupt → refuse to
|
||
boot; deleted-with-a-WAL-present → forced follower) and is fsynced BEFORE
|
||
any vote reply or leadership claim leaves the node.
|
||
- **Term fencing everywhere (kills incident §1.4-1).** Every replication RPC
|
||
carries the sender's term: stale-term ships/chunks/heartbeats/frontier
|
||
reports are rejected, the quorum commit index folds only reports stamped
|
||
with its activation term (race-free, under the index's own lock), and a
|
||
restarted ex-leader boots as a FOLLOWER from its durable state — never from
|
||
the topology file. A partitioned ex-leader that restarts cannot accept a
|
||
single write (`mp_fenced_ex_leader_restart_cannot_write`); a leader that
|
||
loses majority contact steps down within `election.leader_lease_ms`
|
||
(default 900; validated `lease + heartbeat < election_timeout_min` so a
|
||
deposed leader stops before any successor can exist).
|
||
- **Divergent suffixes quarantine instead of lying.** A node whose log
|
||
extends past what the elected leadership subsumed (leader-acked,
|
||
never-quorum-acked writes on a dead leader) detects it at term-join — the
|
||
heartbeat carries the leader's election-time log position — and fences
|
||
itself from the data plane (status `quarantined: true`, metric
|
||
`tidaldb_cluster_divergence_quarantined`, ERROR naming the reseed runbook)
|
||
while still voting. Followers apply on receipt, so un-applying is not a
|
||
thing: reseed is the honest recovery (m11p5 snapshots automate it).
|
||
- **`/cluster/promote` is now a fenced transfer**: with a live leader it
|
||
drains (waits for the target to hold the flushed prefix, breaker-immune via
|
||
the commit index's marks) then sanctions an immediate election; with a dead
|
||
leader the target campaigns. The election refuses a target whose log lags —
|
||
the m11p3 "promote the max-applied survivor" operator rule is now enforced
|
||
by the protocol. The legacy term-0 fan-out survives only on clusters that
|
||
have never elected (mixed-version rollouts; the chaos drills' deliberate
|
||
isolated-node override) and is permanently retired per node at its first
|
||
joined election. `election.auto_election: false` preserves the full
|
||
pre-m11p4 operator posture (no auto elections, no check-quorum step-down).
|
||
- **Bounded churn under flapping links**: the pre-vote (no term inflation
|
||
without a majority probe) plus the leader-freshness lease absorb short
|
||
flaps entirely and bound terms under long ones
|
||
(`mp_flapping_links_bounded_churn`). Election observability:
|
||
`tidaldb_cluster_election_term/_role/_elections_started_total/
|
||
_leader_changes_total`, and `/cluster/status/local` grows `term`, `role`,
|
||
`quarantined`, `prev_log_term/seq`.
|
||
- New tier-3 exit-gate suite `cluster_election.rs` (auto-failover ledger,
|
||
fencing under partition+restart, bounded churn); `cluster_quorum.rs` and
|
||
`cluster_lifecycle.rs` pin `auto_election: false` (they validate the manual
|
||
drill, which remains supported); the partition harness gained
|
||
bidirectional isolation (`isolate_region`) — severing only a node's inbound
|
||
edges leaves its outbound heartbeats/votes flowing, which silently defeats
|
||
partition scenarios.
|
||
|
||
**Catch-up self-healing + WAL segment format versioning (m11p4) — timer-retried pulls, `TSEG` segment header, structured "snapshot required"**
|
||
- **Failed catch-up pulls retry on a timer.** The pull trigger was event-only:
|
||
a follower whose `StreamSegments` pull failed (e.g. the leader's gRPC server
|
||
not yet ready during a rolling restart) waited for the next PUSHED segment
|
||
to re-expose the gap — in an idle cluster that push never comes, and the
|
||
follower stayed lagged forever (the 2026-06-11 p3 rollout: both followers
|
||
stuck at lag=136507). A failed pull now arms a one-shot timer
|
||
(`replication.catchup_retry_ms`, default 30000; transport
|
||
`catchup_retry_interval`) that re-pulls from the CURRENT applied frontier.
|
||
Pulls stay single-flight and rate-limited; the timer's wake-up re-arms when
|
||
consumed by the rate limit or an in-flight pull, so the gap always keeps a
|
||
standing wake-up until a pull completes. Verified over real sockets:
|
||
`catchup_retry.rs` reproduces the incident (pull fails, leader appears,
|
||
zero pushes) and proves timer-only self-heal — and that a clean completion
|
||
arms nothing.
|
||
- **WAL segment files are format-versioned.** New segments open with an
|
||
8-byte header (`TSEG` magic + version byte + reserved); pre-m11p4
|
||
headerless segments stay readable as implicit version 0 — no migration. A
|
||
segment this binary cannot identify (unknown header version, unrecognized
|
||
leading bytes, unparseable `.seg` filename) surfaces as the new
|
||
`WalError::SegmentFormatUnknown` at open — previously it scanned as
|
||
empty (`segments=0`) and recovery's torn-tail repair could TRUNCATE the
|
||
foreign file to zero. Foreign-format files are never repaired, truncated,
|
||
or skipped. Downgrade across m11p4 requires a WAL reseed (runbook §8).
|
||
- **Unservable catch-up is a structured refusal.** `SegmentSource::collect_from`
|
||
returns typed `SegmentReadError::{Unavailable,Failed}`;
|
||
`TidalDb::read_wal_batches` returns the typed `WalError` (was stringified
|
||
`TidalError`). The `StreamSegments` handler maps `Unavailable` to
|
||
`FAILED_PRECONDITION` — `"segments not available from seq N; snapshot
|
||
required"` — and the follower logs it distinctly (*catch-up unservable …
|
||
needs a snapshot (m11p5) or an operator reseed*) instead of burying it as a
|
||
transient. The on-disk segment format is now documented
|
||
(`tidal/src/wal/segment.rs` module docs + spec 01 §2.2).
|
||
|
||
**Quorum-acked writes (m11p3) — `ack=leader|quorum`, durable ship acks, commit index, zero-acked-loss ledger gate (closes G4)**
|
||
- **`ack=quorum`** is an opt-in durability contract for every replicated write
|
||
(`/signals`, `/items`, `/embeddings`): success means a **majority of the
|
||
replica set durably holds the write** (leader + `floor(n/2)` followers, each
|
||
storage-applied and own-WAL-fsynced), so an acked write survives the
|
||
permanent loss of any single node — including the leader. Deployment default
|
||
via topology `replication.ack`; per-request override via the **`x-tidal-ack`**
|
||
header (forwarded verbatim by gateways). `ack=leader` (the default) is the
|
||
m0–m11p2 contract unchanged.
|
||
- **Durable frontier reports.** Followers PUSH their durably-applied frontier
|
||
to the leader once per apply round (new `ReportApplied` RPC, fired by the
|
||
segment receiver through the new `Transport::notify_applied`) — batch-level
|
||
and fully decoupled from ship acks, so the commit index stays fresh even
|
||
when outbound ships stall (gap-parked follower, pull-based catch-up, quiet
|
||
leader). Ship acks keep their m11p2 instant floor-hint semantics — both
|
||
inputs are durable-true because a follower's frontier only ever advances
|
||
after its storage apply + own-WAL fsync. (The first design held each ship
|
||
ack until its segment's apply; measured under open-loop load, that couples
|
||
ship cadence to apply latency and one gap-parked follower spirals into
|
||
total quorum collapse — the report push is what shipped.)
|
||
- **Follower blob applies are group-committed.** m11p2 applied replicated
|
||
items/embeddings one record at a time — one solo follower fsync per item,
|
||
capping item apply at the fsync floor (~100/s on macOS) and stalling the
|
||
quorum frontier behind any item burst. The receiver now hands each apply
|
||
round's blob records to the engine as ONE batch (`apply_replicated_blobs`:
|
||
validate all → stage all WAL appends → wait all → upsert storage), and the
|
||
WAL writer flushes queued blobs under ONE group fsync. Measured: corpus
|
||
seeding 2,000 items + embeddings 39.3s → 1.8s (22×).
|
||
- **Commit index.** The leader folds durable acks (and heal resumes) into
|
||
per-peer durable marks; the commit index is the k-th largest (k =
|
||
`floor(n/2)`), leadership-scoped (promote resets it to the stream baseline;
|
||
demotion fails in-flight waiters — a demoted leader never claims quorum).
|
||
Handlers await it through a watch-channel bridge — **fully async, zero
|
||
threads parked per waiter** (the thread-per-wait design measurably collapsed
|
||
at 1k rps open-loop by exhausting the blocking pool and starving the very
|
||
completions that advance the index). Quorum capacity measured on a real
|
||
3-process localhost cluster (release, writes mix): **3,600 quorum
|
||
signal-writes/s within SLO** (p50 ~45ms, zero errors, replication lag ≤3
|
||
events at ramp end; knee not reached) — 79% of m11p1's 4,534/s leader-ack
|
||
figure, vs the ≥50% gate.
|
||
- **Honest timeout semantics.** Quorum not confirmed within
|
||
`replication.quorum_timeout_ms` (default 2000) → a **retryable 503** naming
|
||
the laggard regions, the commit index, and needed/confirmed counts. The
|
||
write is in the leader's log and may still commit: retries are
|
||
at-least-once (items/embeddings retries are idempotent upserts; signal
|
||
retries can double-count — decided in-phase: no idempotency-key machinery,
|
||
documented in runbook §8 with the session-write precedent for callers that
|
||
need exact-once).
|
||
- **`x-tidal-seq`** on every cluster write response: the write's seqno in the
|
||
replicated log (relayed through forwards) — an exact durability cursor
|
||
against `commit_index` in `/cluster/status/local` (which also gains `ack`).
|
||
`tidaldb_cluster_relay_durable_seq` now reports the commit index
|
||
(`relay_last_seq − relay_durable_seq` = quorum lag); new counter
|
||
`tidaldb_cluster_quorum_timeouts_total`.
|
||
- **The ledger gate (exit gate, run for real):** tier-3
|
||
`mp_quorum_ledger_zero_acked_loss_across_killpoints` SIGKILLs the leader
|
||
under concurrent quorum load and proves zero acknowledged loss on the
|
||
promoted max-applied survivor — frontier invariant (max acked seq ≤
|
||
survivor applied) plus per-item content probes. **167/167 kill points
|
||
passed on the final design** (batches of 64 + 91 + 12; kill timings spread
|
||
120–598ms across fresh 3-process clusters; an earlier 100/100 run had
|
||
validated the superseded ack-holding design before it was replaced — see
|
||
docs/planning/milestone-11/phase-3.md). Plus tier-3 partition semantics
|
||
(one follower down: quorum commits; both down: fast 503 naming laggards
|
||
while `ack=leader` flows; heal: recovers) and in-process gRPC coverage
|
||
(override headers, forwarded quorum writes, blob writes, 400 on bad mode).
|
||
- **Rolling-upgrade order (mixed-version caveat):** a pre-m11p3 leader
|
||
neither serves the `ReportApplied` RPC nor recognizes `x-tidal-ack` — it
|
||
silently applies LEADER-ack semantics to a `quorum` request (a durability
|
||
downgrade the caller cannot see). Upgrade the leader first: `ack=quorum`
|
||
is then honored immediately (commit-index freshness rides the m11p2
|
||
ship-ack floor hints until the followers upgrade too). Replication and
|
||
heal are unaffected by either order. Runbook §8 records the procedure.
|
||
|
||
**One replicated log (m11p2) — items/embeddings ride the WAL, `StreamSegments` catch-up, HTTP broadcast deleted**
|
||
- **The leader's WAL is now THE replicated log.** The group-commit writer hands
|
||
every fsynced batch to a bounded in-memory **ship feed** (`wal::feed::WalShipFeed`);
|
||
the ship queue pushes those already-encoded bytes verbatim — byte-identical on
|
||
the leader's disk, the wire, and the follower's apply path — and stream seqnos
|
||
are WAL seqnos, so they **survive restarts** (the m8p10 relay-reset hazard is
|
||
gone). The m11p1 in-memory relay log, its durable frontier, and its poisoning
|
||
machinery left the server write path entirely (`/signals` stages straight
|
||
through the engine's group commit; a WAL fsync failure now surfaces per-write
|
||
exactly like single-node).
|
||
- **Item metadata and embeddings are replicated mutations.** `/items` and
|
||
`/embeddings` journal kind-1/2 **blob records** (WAL header `flags` byte =
|
||
batch kind; one record, one seqno) BEFORE storage, on the same stream as
|
||
signals. Followers apply them kind-aware — WAL-first into their own log, then
|
||
idempotent storage upserts — and recovery replays them. The m8p10 HTTP
|
||
item/embedding broadcast (marker-gated fan-out, O(items) heal re-broadcast,
|
||
authed side-POSTs — the source of both 2026-06-10 live bugs) is **deleted**;
|
||
those bug classes are now impossible by construction. Cluster `/items` returns
|
||
a plain 201 and `/embeddings` a plain 204 (no broadcast-report body).
|
||
- **`StreamSegments` implemented — catch-up is follower-pulled.** The ship feed's
|
||
tail is bounded; a peer that falls behind it is skipped ahead, and the
|
||
follower pulls the hole itself via the (previously declared-unimplemented)
|
||
server-streaming RPC over the leader's durable, BLAKE3-verified segments.
|
||
Pulls trigger on detected gaps, on follower boot (self-driving restart
|
||
catch-up), and on the leader's heal nudge (`POST /cluster/catchup`, internal,
|
||
forwards the operator's own bearer credential). Pulled chunks flow through the
|
||
same inbound apply path as live ships. Ship acks piggyback the follower's
|
||
applied seqno (`ShipSegmentResponse.applied_seqno`), so retries of
|
||
already-applied data prune and heal is `resume_from` + nudge — no redelivery
|
||
scan, no O(items) traffic.
|
||
- **Promote carries a stream baseline.** A promoted leader's stream starts at its
|
||
promote-time flushed frontier (persisted in `data_dir/stream_baseline`); the
|
||
fan-out body and every catch-up chunk announce it, so peers jump their
|
||
frontier past pre-stream history instead of parking on a phantom gap. Ship
|
||
queues are leadership-gated (`activate_from`/`deactivate`): a follower's
|
||
replicated applies never echo back at its peers.
|
||
- **Multi-process cluster mode now requires `--data-dir`** (validated at
|
||
startup): the durable WAL is the replication stream. Standalone single-node
|
||
deployments are untouched — blob journaling and the ship feed are gated on
|
||
cluster peers, so single-node item writes keep fjall-only durability with
|
||
zero extra fsyncs.
|
||
- New tier-3 suite `mp_items_ride_the_log_and_catchup_stream`: items written on
|
||
the leader AND through a follower gateway converge everywhere via the log
|
||
(feed parity 1e-6), and a follower stopped through item+embedding+signal
|
||
writes restarts and converges via its boot-time `StreamSegments` pull with no
|
||
heal verb and no HTTP item traffic.
|
||
|
||
**Cluster replication performance floor (m11p1) — ack/ship decoupled, batched + windowed shipping, first `tidaldb_cluster_*` metrics**
|
||
- The replicated `/signals` write path no longer serializes every writer onto a
|
||
solo group-commit fsync nor ships to followers on the request path. Writes are
|
||
**staged** (seqno + WAL submission + relay log push, microseconds, atomic with
|
||
rollback) and **completed** (shared group-commit fsync + in-memory fold) in two
|
||
phases, so concurrent writers coalesce into one fsync; follower shipping moved
|
||
to per-peer sender threads that coalesce contiguous runs into multi-event
|
||
batches with a windowed in-flight budget (topology knobs
|
||
`replication.{batch_max_events,window,retry_ms}`). The 204 contract is
|
||
unchanged (leader durability only) — it now returns at leader fsync. Measured
|
||
on a real 3-process localhost cluster (release build, thepeach mix):
|
||
**4,534 replicated signal-writes/s sustained within SLO** vs ~90/s before
|
||
(~50×), replication lag bounded at ≤377 events (~80ms) through the whole ramp.
|
||
- **Durable-frontier shipping + relay poisoning.** Senders only ship the
|
||
leader's contiguous fsynced prefix (an event a follower holds but the leader
|
||
could lose is silent divergence); a staged write whose fsync fails **poisons**
|
||
the relay — further cluster writes are rejected and the ship frontier freezes
|
||
(the CockroachDB/Postgres fsync-failure posture).
|
||
- **Follower group-commit coalescing.** The segment receiver drains its inbound
|
||
backlog (`Transport::try_recv_segment`) and applies it through ONE shared
|
||
group commit (`SignalLedger::apply_replicated_events`), in range-disjoint
|
||
groups so duplicate/subset re-ships cannot double-fold. Without this the
|
||
follower apply ceiling was ~`events-per-segment / fsync-cost` (~1.8k events/s
|
||
measured) and lag grew without bound under m11p1 leader rates.
|
||
- **First cluster metrics + cluster `/metrics` listener.** Cluster mode
|
||
previously had no metrics endpoint at all. New per-region topology
|
||
`metrics_addr` wires the engine's Prometheus listener; new `tidaldb_cluster_*`
|
||
series: ship RTT + batch-size histograms, per-peer ship counters/gauges
|
||
(`peer_shard` labels), WAL fsync latency + group-commit fill histograms,
|
||
write-pool depth/rejections, relay committed + durable frontiers. WAL
|
||
group-commit knobs are deployment config (`wal.{batch_size,batch_timeout_ms}`
|
||
topology block; `wal_batch_size`/`wal_batch_timeout` builder methods).
|
||
- Engine API: `TidalDb::signal_staged`/`StagedSignal::wait`,
|
||
`SignalRelay::{stage_write,complete_write,durable_seq,snapshot_range}`,
|
||
`ShipQueue` (per-peer windowed batch senders with pause/resume),
|
||
`WalWriter::append_signal_staged`, `WalSender::append_record_staged`,
|
||
`range_payload`/`encode_run`. Relay log entries are now `RelayEvent` (raw
|
||
event records, re-encoded deterministically at ship time) instead of
|
||
pre-encoded single-event bytes.
|
||
- Ship-sender failure logging is transition-based (first + every 50th
|
||
consecutive failure WARN with the running count, recovery INFO) — the
|
||
per-retry WARN flood could fill an undrained log pipe and stall the process.
|
||
The multiproc test harness now discards child logs via `/dev/null` (a piped
|
||
fd nobody drains deadlocks the node once the kernel buffer fills), with
|
||
`TIDAL_TEST_NODE_LOGS=inherit` to stream them while debugging.
|
||
|
||
**M9 — Community Sync & Revocation**
|
||
- Local embeddable profiles can opt into community personalization and safely leave/purge their contributions. New types: `SignalScope`, `CommunityId`, `Membership`, `MembershipEpoch`, `PolicyMetadata`. Community signal reconciliation via `CrdtSignalState` with commutative/associative/idempotent merge laws; membership-epoch revocation purges a departed member's contributed signals.
|
||
|
||
**M10 — Governance & Agent Rights**
|
||
- Community rules and agent-scoped permissions control what signals influence ranking: policy-metadata enforcement and agent-rights scoping wired into the signal-write and ranking paths.
|
||
|
||
**Cluster mode (m8p10): true multi-process region nodes + full tier-3 UAT — M8 COMPLETE**
|
||
- Multi-process cluster mode: `tidal-server cluster --region <name> [--data-dir <p>]`
|
||
(env `TIDAL_REGION`) runs **one process per region** (`RegionClusterState`), each
|
||
owning one `TidalDb` and one `GrpcTransport` that binds this region's `grpc_addr`
|
||
and dials every sibling's real `grpc_addr` — real process/host isolation. The
|
||
topology requires per-region `grpc_addr` AND `http_addr` in this mode;
|
||
single-process mode (no `--region`) is unchanged as the dev/demo default. Both
|
||
modes stay behind the experimental gate (`--experimental-cluster` /
|
||
`TIDAL_ALLOW_EXPERIMENTAL_CLUSTER=1`) with mode-specific WARN text.
|
||
- New routes / behaviors on the multi-process surface: `GET /cluster/status/local`
|
||
(per-node status), `GET /cluster/status` aggregates ALL regions with a `reachable`
|
||
field (unreachable ⇒ `reachable:false` + worst-case lag), `POST /cluster/reconcile
|
||
{region}` (cross-process CRDT snapshot exchange; idempotent — a repeat is an exact
|
||
no-op on scores), `POST /hardnegs {user_id,item_id}` (user-scoped hide; converges
|
||
via reconcile, filtered from `/feed?user_id`). Writes on a non-leader **forward to
|
||
the leader**; `?region=` reads forward to the owning region; default reads are
|
||
**LOCAL** in multi-process mode. Items/embeddings are leader-applied + HTTP-broadcast
|
||
to peers with a per-peer report: `POST /items` → 201 `{replicated_to, failed}`,
|
||
`POST /embeddings` → **200** with the same report on the leader path (NOT 204 — a
|
||
204 cannot carry the report; the forwarded/internal path stays 204). `/sharded/*`
|
||
fans out across processes (degraded semantics preserved). `promote` fans out to all
|
||
peers (`{ok, leader, acked, failed}`).
|
||
- `/cluster/heal` is the single recovery verb: redelivers missed signal segments
|
||
(gap-aware) AND re-broadcasts item metadata + embeddings to the healed region
|
||
(idempotent upserts). After a partition the per-peer gRPC circuit breaker (threshold
|
||
5, reset 30s) is open, so re-issue `/cluster/heal` until `/cluster/status` shows
|
||
lag 0.
|
||
- `TIDAL_HLC_SKEW_MS` (multi-process): signed ms offset applied to the process's HLC
|
||
(reconcile LWW stamping only — not signal-decay timestamps); a test/ops escape hatch.
|
||
- Tier-3 UAT over real OS processes (feature `cluster-e2e`): `cluster_multiproc` (5),
|
||
`cluster_chaos` (3, REAL network-partition injection via a root-free in-harness TCP
|
||
relay proxy — the toxiproxy-style alternative the ROADMAP sanctions; iptables/pfctl
|
||
remain an operator option), `cluster_lifecycle` (2, ±500ms clock skew + rolling
|
||
upgrade with zero acknowledged-write loss), `cluster_runbook` (9, every
|
||
`docs/runbooks/cluster.md` §5–§11 operation), plus `cluster_e2e` (2, single-process
|
||
smoke). Measured (localhost): replication p99 ~110–133ms (< 2s SLA), failover
|
||
~31–34ms (< 10s SLA), reconcile 0–1ms/side (< 100ms SLA).
|
||
|
||
### Fixed
|
||
|
||
**Four production bugs surfaced and fixed by the m8p10 tier-3 UAT**
|
||
- **Silent data loss on out-of-order ships.** The replication applied high-water-mark
|
||
swallowed sequence gaps when eager ships arrived out of order. The applied frontier
|
||
is now contiguous with a bounded ahead-buffer, so a gap can never be skipped.
|
||
- **Non-idempotent reconcile (score creep).** `take_crdt_snapshot` attributed replicated
|
||
signal streams per-node, so reconciling already-converged nodes crept the decayed
|
||
scores (0.5 → 0.375 → …). Contributions are now attributed to one canonical
|
||
replication shard (`ShardId::SINGLE`), making `merge` idempotent — reconcile of
|
||
converged nodes is an exact fixpoint.
|
||
- **Lag gauge conflated leader streams across a promotion.** A converged node reported a
|
||
permanent phantom lag after a `/cluster/promote` moved leadership to a different shard.
|
||
The lag gauge now tracks the leader high-water-mark per source shard
|
||
(`ReplicationLagGauge::leader_seqno_for`), computing lag against the current leader.
|
||
- **Items missed during a broadcast were never backfilled.** A node down/partitioned
|
||
during an item/embedding broadcast was permanently missing that data even at signal
|
||
lag 0. `/cluster/heal` now re-broadcasts item metadata + embeddings to the healed
|
||
region (idempotent upserts), making heal the single recovery verb.
|
||
|
||
### Changed
|
||
|
||
**Cluster mode (m8p8): real gRPC replication**
|
||
- `tidal-server`'s `ClusterState` now wires each follower region to a real
|
||
`tidal-net` `GrpcTransport` (self-loop over loopback gRPC) instead of in-process
|
||
crossbeam channels — replication between regions traverses real gRPC/TCP
|
||
(serialization, circuit breaker, HTTP/2). Closes M8 gap G1.
|
||
- Follower gRPC ports are auto-allocated from the topology (`grpc_addr` optional
|
||
per region) and self-heal a transient bind race by retrying on a fresh port.
|
||
- Blocking gRPC ships triggered by `POST /signals` and `POST /cluster/heal` are
|
||
offloaded to a dedicated thread so they never block the axum reactor.
|
||
- `ClusterState` is constructed off the async reactor (`GrpcTransport::new`
|
||
blocks on its own runtime).
|
||
- The experimental opt-in gate and `docker/cluster/Dockerfile` are updated:
|
||
single-process cluster mode is honest that it replicates over real gRPC but
|
||
runs all regions in one process (no host/process isolation). (True
|
||
multi-process region nodes shipped subsequently in m8p10 — see the m8p10 entry
|
||
above.)
|
||
|
||
**Docker build fixes** (all three images now that `tidal-server` pulls `tidal-net`)
|
||
- Install `protobuf-compiler` in the builder stage — `tidal-net`'s build script
|
||
runs `tonic-build`, which needs `protoc` to compile the WAL-shipping `.proto`.
|
||
- Pin the builder base to `bookworm` (`rust:1.91-bookworm` /
|
||
`rust:1.91-slim-bookworm`) so its glibc matches the `debian:bookworm-slim`
|
||
runtime; the default trixie base emitted a `libmvec.so.1` dependency absent on
|
||
bookworm, aborting the binary at startup. Verified: `docker run` of the cluster
|
||
image serves a functional 3-region cluster (write replicates to both followers
|
||
over gRPC; region-pinned reads serve replicated data; SIGTERM exits 0).
|
||
- New tests: `tidal-server/tests/cluster_grpc.rs` (in-process gRPC replication +
|
||
HTTP offload path) and a hardened tier-3 `cluster_e2e.rs` (multi-process smoke
|
||
+ promote over real OS processes, feature-gated).
|
||
|
||
## [0.1.0] - 2026-02-23
|
||
|
||
### Added
|
||
|
||
**Core Database Engine**
|
||
- `TidalDb` embeddable database with `ephemeral()` and `with_data_dir()` open modes
|
||
- `SchemaBuilder` for defining signal types, decay parameters, and ranking profiles
|
||
- `TidalDbBuilder` fluent builder with schema, data directory, metrics, and rate limiter configuration
|
||
|
||
**Signal System**
|
||
- Typed signal recording with exponential decay scoring
|
||
- Hot-tier (DashMap) and warm-tier (BucketedCounter) signal storage
|
||
- Windowed aggregation: `OneHour`, `TwentyFourHours`, `SevenDays`, `AllTime`
|
||
- Signal velocity tracking
|
||
- WAL-backed signal durability with crash recovery
|
||
- Periodic signal checkpointing to fjall (every 30s)
|
||
- WAL compaction after each checkpoint
|
||
|
||
**Retrieval (RETRIEVE query)**
|
||
- 5-stage pipeline: universe, filter, score, diversify, return
|
||
- Filter expressions: `Eq`, `In`, `Gt`, `Lt`, `And`, `Or`, `Not`, `InCollection`, `InProgress`, `MinSignal`, `MaxSignal`, `NearLocation`
|
||
- Built-in ranking profiles: `trending`, `for_you`, `new`, `popular`, `recent`, and 20+ more
|
||
- Custom ranking profiles via `SchemaBuilder`
|
||
- Diversity enforcement (max N per category/creator)
|
||
- Sort modes: `Relevance`, `Trending`, `Newest`, `MostLiked`, `MostViewed`, `MostFollowed`, `AlphabeticalAsc/Desc`, `Shortest/Longest`, `LiveViewerCount`, `DateSaved`, and more
|
||
|
||
**Search (SEARCH query)**
|
||
- BM25 full-text search via Tantivy
|
||
- Approximate nearest-neighbor (ANN) semantic search via USearch HNSW
|
||
- Reciprocal Rank Fusion (RRF) combining BM25 + ANN scores
|
||
- Creator search with `entity_kind(EntityKind::Creator)`
|
||
- `similar_to(EntityId)` for content-based recommendations
|
||
- Scope pre-filters: `Trending`, `CohortTrending`, `Following`, `Category`, `Collection`
|
||
- Autocomplete suggestions via `db.suggest()`
|
||
|
||
**Entity Model**
|
||
- Three built-in entity types: `Item`, `User`, `Creator`
|
||
- Metadata storage as `HashMap<String, String>`
|
||
- Embedding slots (up to 4 per entity type) via USearch
|
||
- Relationships: `Follows`, `Blocks`, `Hide`, `Mute`, `InteractionWeight`
|
||
|
||
**Sessions**
|
||
- Session lifecycle: `open_session`, `close_session`
|
||
- Cross-session preference vector updates (EMA blend)
|
||
- Session snapshots with signal state and preference vectors
|
||
- Session serialization format v0x03 with backward compatibility
|
||
|
||
**Social Graph**
|
||
- Creator follower/following indexes
|
||
- Cohort membership (user segments)
|
||
- CoEngagementIndex for co-viewing patterns with LRU eviction
|
||
- Social graph filter for "followed creator" content scoping
|
||
|
||
**Collections**
|
||
- Named collections with `Private`, `Shared`, `Public` visibility
|
||
- `create_collection`, `add_to_collection`, `remove_from_collection`, `list_collections`
|
||
- `FilterExpr::InCollection` for collection-scoped retrieval
|
||
- Saved searches with `save_search`, `list_saved_searches`, `retrieve_saved_search`
|
||
|
||
**Observability**
|
||
- `enable_metrics(addr)` -- Prometheus-format `/metrics` endpoint + `/healthz` JSON
|
||
- 15+ metrics: signal writes, WAL lag, checkpoint age, degradation level, index health
|
||
- `tidaldb_checkpoint_failures_total` counter for checkpoint monitoring
|
||
- `TidalDb::diagnostics()` -- structured health snapshot
|
||
- WAL diagnostics and recovery tools
|
||
|
||
**Safety**
|
||
- Signal weight NaN/Inf validation (returns `TidalError::InvalidInput`)
|
||
- Metadata size bounds: 64 keys max, 8KB value max, 64KB total max
|
||
- Export request limit: 500K signals max per request
|
||
- `FilterExpr` complexity limit: 256 nodes max
|
||
- Data directory lock (`tidaldb.lock`) prevents dual-process corruption
|
||
- Schema fingerprint persistence detects decay parameter changes on reopen
|
||
- Bounded `closed_sessions` cache (10K max, LRU eviction)
|
||
- Metrics server non-loopback bind warning
|
||
|
||
**CLI (`tidalctl`)**
|
||
- `tidalctl` binary for database inspection and diagnostics
|
||
|
||
**RLHF / ML Export**
|
||
- `db.export_signals(ExportRequest)` -- WAL-based signal export for training data
|
||
- `db.user_session_summary(user_id, since_ns)` -- aggregated session statistics
|
||
|
||
### Stability
|
||
|
||
tidalDB `0.1.0` is pre-1.0. **No API or data format stability guarantees** are made for `0.x` releases. Upgrade guides will be provided for each minor version bump. Do not upgrade `0.x` to `0.y` on a live data directory without reading the release notes.
|
||
|
||
---
|
||
|
||
*Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)*
|