`score_hot` hardcoded `age_hours = 24.0`, so the divisor in
`log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate
set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count
ranking wearing a recency sort's name. Four built-in profiles use it (`hot`,
`for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op.
The in-code comment justified this by saying a per-entity `created_at` lookup
needs an `EntityId -> created_at_ns` reverse map that "is not built". That was
stale, and it was the load-bearing claim: `created_at` has been materialized INTO
item metadata on every write since `Items::metadata_with_created_at`, the executor
has held an `EntityId -> metadata` map since M6p3, and the replication record
carries the materialized map so replicas cannot diverge. No index, storage change,
schema change or migration -- the scorer reads the map it already had, exactly the
way `read_duration` does three lines away.
`Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in
creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog
of N items the sort contributed ~N against a boost sum in single digits. Recency
did not participate in the ranking, it annihilated every boost. Now negated age in
hours -- same ordering, boost-comparable scale.
Three more instances of the same defect class, found by auditing rather than
assuming the report was complete:
1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with
no session and no diversity never loaded the map the fix depends on.
2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was
gated on `session_context.is_some()` and never consulted `profile.sort`, AND
the `ProfileExecutor` it built never had `with_item_metadata` called at all --
the map it did compute went only to the keyword-hint argument, which the sort
scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the
alphabetical sorts the missing-title sentinel, for every candidate, silently.
3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs,
correct only while `Sort::New` meant "highest ID". Left alone it would discard
the genuinely newest items BEFORE scoring -- wrong only when degraded, the
hardest case to notice. Now keyed off the `created_at` index via the new
`RangeIndex::top_n_descending`.
The decision "which sorts read item metadata" now lives on `Sort` itself as an
exhaustive match. It was a `matches!` in one executor while a second executor had
its own different copy, which is precisely how a metadata-reading sort came to be
omitted from both.
MEASURED, not inferred:
- Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5
(all-equal set folded to the normalizer's midpoint) and the feed returned
oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first.
- `new` with zero signals returns the exact REVERSE of candidate-scan order.
- `alphabetical_asc`, `shortest`, `longest` verified end to end with title and
duration order both opposing entity id.
- Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates`
caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per
candidate. Guarded at 250ms.
THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed
this fix lets a zero-signal corpus rank newest-first so a consumer could delete
its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is
exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every
candidate still ties -- confirmed on the live server, all ten scores 0.5.
Age-awareness begins at the second view. Fixing cold-start needs recency to be
ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer,
so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_
age` pins the limit so it cannot be rediscovered by accident.
Three existing tests asserted the old entity-ID behaviour. Inverted to assert real
recency, not loosened -- and each fixture now makes id order and creation order
DISAGREE, because an ordering assertion where the two candidate orderings agree is
satisfied by the defect too. Three of my own new tests were vacuous for exactly
that reason and were caught by mutation-testing; one was also flaky (it passed in
a 12-test run and failed run alone, because retrieval order for exactly-tied
vectors is not deterministic). Every new assertion is mutation-proven against the
implementation it replaces.
Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added.
Fixes the two defects a malformed probe exposed on the live cluster, plus the
coverage gap that let a stale assertion survive the same day it was falsified.
TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim
slot was appended to the WAL FIRST, then validated, then answered 500 — so an
already-durable, unapplicable record shipped to both followers, halted both
receivers, and put shard 1 into a quorum-write outage. Validation now runs before
the append and returns 400 via invalid_input; nothing enters the log.
`storage::vector::validate_dimensions` is now the single comparison, replacing an
inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a
dimension check drift, and the apply-path copy is the one that halts replication
when it disagrees.
The receiver's halt-vs-skip decision is now explicit instead of "halt on
anything". A record whose failure is deterministic and node-independent (schema
width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the
frontier advances; a record that could become applicable after a binary upgrade
(unknown batch kind, capability skew) still halts, because skipping那 would
silently drop replicated data. Both branches are proven reachable by tests.
TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups
latched a marker per group but discharged on a single seqno, so two latches meant
permanent 503 on a node whose every shard read lag 0 — it hit all three pods
during the roll and each needed a manual delete. Gaps are now tracked per group
in a ReseedGapSet and cleared on evidence about themselves; a REFUSED
reseed_self_restart re-evaluates every 15s instead of waiting for a latch that
never arrives. /health's cause ladder was also lying: it printed "joiner boot not
yet converged" for a node whose groups had all converged, because the fallback
asserted a state it never tested. It now names the outstanding gaps, gained the
decommissioned-by-signal arm that is_ready checked but the ladder did not, and
its terminal arm says "reason unavailable" rather than inventing one.
COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical:
cluster_routes still asserted the wire fabrication removed hours earlier
(applied_events == 0 with a lag derived from it) and nothing caught it because
nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search
(distance contract) and cluster_poison_embedding (task 17's own gate) were in the
same position, so those guards would have rotted identically. Every suite now has
a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs
FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the
nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing
before being scheduled.
Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13
minutes in — it demanded an unreachable peer report worst-case lag, i.e. it
required the fabrication task 04a deleted.
Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on
touched trees at 431340f); lib 2115 passed; all 8 fast suites green;
cluster_chaos 5, cluster_sharding 5, cluster_poison_embedding 1,
cluster_cross_shard_reads 2, cluster_graph_persistence 1, cluster_multiproc 5,
cluster_e2e 2; doc-guard OK.
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).
Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.
The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.
Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
/search returned 1,1,2 under full placement. Reuses merge_cross_shard's
existing stamp; asserted on BOTH the multi-group merge path and the
single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
derived from it. applied_events/lag_events are now Option<u64>, null on the
wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
the LEADER computed 0 - applied = 0 for every region and reported a converged
cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
actively hiding the PVC-wipe shape: a measured zero with a real deficit
rendered as "no report" instead of BEHIND. Now read off the wire; converged
exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
existing invalid_input path. Six call sites migrated, not the two this
roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
operators to run a quorum-write probe via POST /sharded/items. That probe
cannot verify quorum: the surface applies locally with no WAL append. It was
used as the safety check between every step of a staged deploy earlier today.
Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.
Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.
Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.
Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.
Three real defects, plus a retracted fourth that was a probe artifact.
P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized
every stored vector; the read path passed the caller's raw query straight to the
index, so the two sides lived in different spaces. With unit v,
d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by
|q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match
scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the
canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is
undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant
across candidates - which is why it went unnoticed; what broke was every absolute
use of the number. WIRE-VISIBLE, recorded in CHANGELOG.
P2 (fixed) - the blob path had zero instrumentation. Added per-kind
tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality
is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind
is now the ONE exhaustive match over the variants (kind() derives from it), so a
new variant is a compile error in one place instead of a silent zero in three.
Only the live apply path is counted - boot replay would inflate applied past
originated on every restart.
Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics
owner's shard group, so on a 3-group node two thirds of the corpus had no
vector-count series at all. Co-located groups now render shard="N"; the owner
stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets
each replica set separately without double-counting.
P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe
wrote through the /sharded/ surface, which hash-partitions and applies to the
owning region's local store with no WAL append, and therefore does not replicate
BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus
/embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six
reach exactly one node. Both are now pinned by tests. See
tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml
entry RETRACTED_blob_replication_rf1_2026_08_30.
Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling
(the project's existing path-attribute convention), and the three hand-rolled
l2_normalize copies collapsed to one. The two entity copies used a zero threshold
about 2900x looser than the canonical one; normalize_centroid now names the
centroid zero-tolerance policy once, and a test pins the tightened behavior.
Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the
P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in
cluster_sharding are now serialized - four concurrent 3-node clusters made the
pre-existing failover test miss its 10s budget.
M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.
- README.md: replace the "Pre-release / not yet recommended for production"
banner with a production-ready statement; drop "(experimental)" from the
cluster status bullet; state the post-1.0 versioning posture (additive in
minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
guard against standing up a multi-node fabric by accident rather than a
readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.
Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.
Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.
Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.
Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
