be37c72b12
64 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
a588f01f63 |
ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Three parallel reviews of
|
||
|
jordan
|
6385425a92 |
ranking: make Hot and New age-aware; fix the same gap in three more places
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
`score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added. |
||
|
jordan
|
a6f663f002 |
harden: validate embeddings before the WAL, fix the reseed-latch leak, run every test suite
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Fixes the two defects a malformed probe exposed on the live cluster, plus the
coverage gap that let a stale assertion survive the same day it was falsified.
TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim
slot was appended to the WAL FIRST, then validated, then answered 500 — so an
already-durable, unapplicable record shipped to both followers, halted both
receivers, and put shard 1 into a quorum-write outage. Validation now runs before
the append and returns 400 via invalid_input; nothing enters the log.
`storage::vector::validate_dimensions` is now the single comparison, replacing an
inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a
dimension check drift, and the apply-path copy is the one that halts replication
when it disagrees.
The receiver's halt-vs-skip decision is now explicit instead of "halt on
anything". A record whose failure is deterministic and node-independent (schema
width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the
frontier advances; a record that could become applicable after a binary upgrade
(unknown batch kind, capability skew) still halts, because skipping那 would
silently drop replicated data. Both branches are proven reachable by tests.
TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups
latched a marker per group but discharged on a single seqno, so two latches meant
permanent 503 on a node whose every shard read lag 0 — it hit all three pods
during the roll and each needed a manual delete. Gaps are now tracked per group
in a ReseedGapSet and cleared on evidence about themselves; a REFUSED
reseed_self_restart re-evaluates every 15s instead of waiting for a latch that
never arrives. /health's cause ladder was also lying: it printed "joiner boot not
yet converged" for a node whose groups had all converged, because the fallback
asserted a state it never tested. It now names the outstanding gaps, gained the
decommissioned-by-signal arm that is_ready checked but the ladder did not, and
its terminal arm says "reason unavailable" rather than inventing one.
COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical:
cluster_routes still asserted the wire fabrication removed hours earlier
(applied_events == 0 with a lag derived from it) and nothing caught it because
nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search
(distance contract) and cluster_poison_embedding (task 17's own gate) were in the
same position, so those guards would have rotted identically. Every suite now has
a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs
FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the
nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing
before being scheduled.
Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13
minutes in — it demanded an unreachable peer report worst-case lag, i.e. it
required the fabrication task 04a deleted.
Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on
touched trees at
|
||
|
jordan
|
fe8d0c87e7 |
harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|
jordan
|
8aa1fbb414 |
vector search: normalize the query, instrument the blob path, expose per-group vector counts
Three real defects, plus a retracted fourth that was a probe artifact. P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized every stored vector; the read path passed the caller's raw query straight to the index, so the two sides lived in different spaces. With unit v, d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by |q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant across candidates - which is why it went unnoticed; what broke was every absolute use of the number. WIRE-VISIBLE, recorded in CHANGELOG. P2 (fixed) - the blob path had zero instrumentation. Added per-kind tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind is now the ONE exhaustive match over the variants (kind() derives from it), so a new variant is a compile error in one place instead of a silent zero in three. Only the live apply path is counted - boot replay would inflate applied past originated on every restart. Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics owner's shard group, so on a 3-group node two thirds of the corpus had no vector-count series at all. Co-located groups now render shard="N"; the owner stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets each replica set separately without double-counting. P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe wrote through the /sharded/ surface, which hash-partitions and applies to the owning region's local store with no WAL append, and therefore does not replicate BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus /embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six reach exactly one node. Both are now pinned by tests. See tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml entry RETRACTED_blob_replication_rf1_2026_08_30. Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling (the project's existing path-attribute convention), and the three hand-rolled l2_normalize copies collapsed to one. The two entity copies used a zero threshold about 2900x looser than the canonical one; normalize_centroid now names the centroid zero-tolerance policy once, and a test pins the tightened behavior. Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in cluster_sharding are now serialized - four concurrent 3-node clusters made the pre-existing failover test miss its 10s budget. |
||
|
jordan
|
7c1c80dd90 |
fix(vector): grow the HNSW graph on insert; it could never accept a new vector once full
Measured on the live RF3 cluster 2026-08-30 while verifying it end to end. Every
NEW embedding returned HTTP 500:
POST /sharded/embeddings -> 500
{"error":"... [op=write_item_embedding] backend error:
USearch insert failed: Reserve capacity ahead of insertions!"}
while POST /sharded/items returned 201, POST /sharded/signals 204, text search
and feed 200, and re-embedding an ALREADY-INDEXED entity returned 204. So the
store had silently become read/update-only for vectors: a consumer could write
items and signals all day and only its embeddings would fail, with nothing
alerting on it.
Cause: USearch's add() cannot grow the graph. build_slot_index() reserves the
rebuild's expected count once and its comment claimed 'the write path grows it
further as needed' - the write path never reserved anything. insert() went
straight to add() for a new key, so the moment size() reached the reservation
every new key failed permanently. The upsert path kept working because remove()
tombstones and size() excludes tombstones, so the re-add lands in the slot just
freed - which is exactly why this looked healthy from the outside.
insert() now reserves before adding a new key at capacity, growing by
max(size/8, 1024) so reserve's reallocation is amortised rather than per-insert.
The regression test drives off the REPORTED capacity, not the requested one:
USearch rounds a reservation up (reserve(4) reported 64), so a hardcoded insert
count sits inside the reservation and exercises nothing. Verified against the
pre-fix source it fails with the production error at insert 64 of capacity 64,
and it asserts the grown graph still answers searches and still upserts.
storage::vector 101 passed, db::items 7 passed.
|
||
|
jordan
|
4766f566de |
feat(observability): HTTP metrics, structured logs, dashboard, live tidalctl
There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.
1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
timeout and rate-limit layers so it sees the status actually returned to the
client. Cardinality is the whole design: the route label is axum's MatchedPath
TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
bucket so a 404 flood cannot mint series. A hard cap folds anything past it
into an overflow bucket while established series keep counting.
The engine owns the /metrics listener but must not learn what a route or a
status code is, so it gained one registration hook
(MetricsState::set_extra_renderer) and tidal-server publishes through it. One
scrape target per node, not two.
2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
produced two real defects: ANSI escapes leaked into collected logs, and every
line failed the collector's JSON parse and was stamped level=info — so
`level:error` matched NOTHING and errors were invisible to the log platform
while being collected. JSON_LOGS=1 emits the collector's exact wire format
(ts/level/service/env/msg), span fields are lifted so request_id lands on every
line of a request, and ANSI is off unconditionally in both formats.
Verified against the running binary, which caught a defect no unit test would
have: dependencies logging through the `log` crate arrived with target="log"
and four log.* metadata fields (absolute cargo registry paths, indexed
forever). The real module is now lifted into target and the bridge metadata
pruned.
3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
as a grafana-database-dashboards key. Every metric name was checked against a
live endpoint and all 26 PromQL expressions were executed against the live
TSDB before commit, because a dashboard full of "No data" is worse than none.
Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).
4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
--url and talk to a running server, with --ca/--insecure because a cluster's
client port is served with the INTERNAL cluster CA. Exit codes follow the crate
contract, so `tidalctl cluster-status && deploy` gates on convergence.
Its first real run immediately found a reporting defect: the aggregated
/cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
lag, having derived lag against an uninitialised applied=0, while every node's
own status reported lag=0, reseed=false and identical frontiers, with
pod-to-pod connectivity open and nothing logged. cluster-status now names that
signature "NO REPORT (aggregated view; query the node directly)" instead of
repeating it as replication lag; a genuine non-zero-applied lag still reports
BEHIND. The underlying gap is documented as open work in
docs/ops/observability.md.
Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
|
||
|
jordan
|
925a616cda |
feat(cluster): expose every tracked replication stream key's position
The remaining reseed defect cannot be diagnosed from the current status surface. A stream key is a per-LEADER-REGION id (`shard_of_region`), not a shard group, so a group accumulates one key per leadership it has followed — but every status field reports only the CURRENT leader's key. A position retained from a previous leadership is therefore invisible, while the receiver's gap check (`receiver.rs`: `request_catchup(key, applied + 1)`) will chase ANY key that received data this round. That is the blind spot: live tidaldb-0 pulls `from_seqno=13540653`, so some key sits at 13540652, while the group it reports on converged at 13540661 — and nothing in the status can say which key that is. Adds `ReplicationState::applied_by_key` and surfaces it as `applied_by_key` on the status response. Instrument only: no behavioural change. Same instrument-first move that turned the previous two defects into one-run diagnoses instead of speculation. |
||
|
jordan
|
afdda7cc0f |
fix(cluster): discharge a reseed marker on served evidence, never on a frontier
|
||
|
jordan
|
da736b8eb2 |
fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail
A follower that latched `reseed_required` from a genuine `snapshot-required` refusal could clear its own marker ~200ms later and so never run the boot reseed that was the only way to close the gap. With `replication.reseed_self_restart: true` it exit-looped: latch -> clear -> exit(0) -> boot with no marker -> re-latch. Production tidaldb-0 did this 196 times in 21h on 2026-08-20 while the cluster ran on 2 of 3 voters. `clear_stale_reseed_marker_if_caught_up(applied >= leader_last_seq)` compared the applied frontier against the LEADER'S TAIL and documented the invariant "a node genuinely behind a COMPACTED gap never reaches caught_up". That is false: on a quiet shard any node meets the leader's tail, including one missing committed history it can never refetch. The clear also reset `tidaldb_cluster_reseed_required`, so the gauge flapped 1->0 every 30s and `TidalDBClusterReseedPending` (`== 1 for 10m`) could never fire - the code path that broke the reseed also erased the signal that would have reported it. The discharge decision now belongs to the marker. `ReseedMarker::discharged_by` requires a stream-dischargeable reason AND an applied frontier that reached the marker's own `from_seqno` - the very entry whose absence latched it. A compacted gap can never satisfy that, so the reseed runs; a node merely behind a shippable tail satisfies it as soon as the stream serves that entry, so the m12 false-alarm self-heal still works (and now clears sooner, since it no longer waits to meet a moving leader tail). The two conditions previously shared `ReseedReason::SnapshotRequired`, so reason alone could not discriminate. The term-join arm's `frontier > baseline` case deliberately sets `from_seqno = baseline`, BELOW the node's own frontier, so a bare `applied >= from_seqno` would discharge it instantly - it holds divergent post-baseline data only a snapshot can discard. It gets its own never-lag- dischargeable reason, `DivergentPostBaseline = 3`. Adding a discriminant is the sanctioned forward-only extension; a downgrade that meets one refuses to decode it, per the existing kind-3/kind-4 precedent. The election-won call site passed a hardcoded `true`; it now passes the leader's durable flushed frontier (`applied_seqno` never advances on a leader), and a `DivergentPostBaseline` node is not campaign-suppressed so it can reach there. Tests: three deterministic predicate tests pinning the incident's exact seqnos (13540653 vs earliest-available 13540657), the false-alarm discharge, and the never-discharge of every structural reason. Pre-existing and NOT introduced here: cluster_reseed's `mp_follower_reseeds_via_snapshot_after_compaction` and `mp_graceful_rolling_restart_under_load_no_reseed` fail on baseline main (verified by stashing this change). The first is the owner-test for this exact mechanism - its leader compaction no longer forces a `snapshot-required`, so it never reached the clear path and never guarded it. Tracked separately. |
||
|
jordan
|
12c7edc374 |
fix(cluster): the frontier pair has one writer, and 0 is not a commit index
`TidalDBClusterQuorumLag` sat CRITICAL all session against the live three-voter cluster while every region reported lag 0 and every per-peer ship queue was empty. Two independent defects fed it: - `observe_ship` bumped `relay_last_seq` on every batch ship while `relay_durable_seq` only moved when a signal write completed. The two are documented as a subtractable pair, so a shipping-but-not-committing node reported the whole relay log (13.3M events) as quorum lag. The ship path now feeds only its own per-peer queue-depth gauge; the pair has one writer. - `set_frontier_gauges` published `CommitIndex::committed()` verbatim, but that returns 0 as a SENTINEL for "no quorum information in this term yet". It now publishes both halves or neither, and every satisfied `await_quorum` -- not just signal writes -- refreshes them, so item and embedding workloads keep the pair live. Regression test asserts a busy ship loop leaves both halves at 0 (lag 0, not 13.3M) and that the single writer still moves them together. Also excludes the `tmp` emptyDir from velero fs-backup: three 0-byte PodVolumeBackups a night whose only other outcome is failing the whole fleet backup when a scratch file vanishes mid-snapshot. |
||
|
jordan
|
897c6086f5 |
fix(cluster): size the read fan-out budget for the transport it crosses
Restoring the three-node cluster for a first production consumer surfaced this
immediately: EVERY cross-shard read came back
{"items":[...],"scatter_gather":{"degraded":true,
"unavailable_shards":["tidaldb-0","tidaldb-2"],"shards_queried":1,
"elapsed_ms":50,"shard_deadline_ms":45}}
HTTP 200, one shard of three, partial results. Replication itself was healthy -
/cluster/status showed all three regions reachable, lag_events 0, 13.3M events
applied each - so nothing in the quorum, election, or ship metrics moved.
Measured on the live cluster: a COLD peer fetch (TCP + TLS handshake + remote
1536-D search) takes ~50ms; a warm one takes ~1ms. DEFAULT_DEADLINE_MS is 50
(spec §7.4) and NETWORK_OVERHEAD_MS is 5, leaving a 45ms per-shard budget -
just under the cold cost. Proven by parameter sweep against one pod:
deadline_ms=50 -> degraded, 1/3 shards, 0 items
deadline_ms=250 -> healthy, 3/3 shards, elapsed 51ms
deadline_ms=1000 -> healthy, 3/3 shards, elapsed 1ms (warm)
The 50ms spec figure budgets a shard READ, not establishing a connection to
another pod. m11p7 put TLS on that hop and the default never followed, so the
first query after any rollout, idle period, or pod restart answered from a third
of the corpus. Fixed with a transport-aware default: 50ms in-process,
TLS_DEFAULT_DEADLINE_MS (250ms) once inter-node TLS is configured. An explicit
`?deadline_ms=` still wins in both directions, and MAX_DEADLINE_MS is unchanged.
The worse half was silence. A degraded fan-out is the one cluster failure that
answers 200 OK: the caller gets a ranked list assembled from a subset of the
corpus with `degraded: true` buried in response metadata. Nothing incremented,
so no alert could exist - a feed quietly ranking over one third of its
candidates looked identical to a healthy one. Added
tidaldb_cluster_scatter_degraded_total and
tidaldb_cluster_scatter_shard_unavailable_total, emitted from both HTTP fan-out
paths, so partial answers are now a countable correctness signal.
Also sizes the cluster StatefulSet for a consumer instead of the endurance gate:
requests 2 cores -> 300m per voter (limit 2 cores). The 2-core reservation was
the 200 rps soak envelope and needed 6,000m plus 2,000m free on each of three
PV-pinned nodes; the fleet is 82-91% committed, so that contract could not be
placed and the cluster stayed parked for a gate nobody is waiting on. 300m is
what the tightest pinned node can reserve, with the quorum/write-pool alerts as
the detector if real load outgrows it.
Tests: default_read_budget_covers_a_cold_inter_node_tls_hop pins the budget
against the measured cold hop and the explicit-override path; the cluster-metrics
render test covers both new counters.
|
||
|
jordan
|
e5fd19eb73 |
fix(vector): load a persisted slot graph by its own backend, not an assumed one
Found by running the standalone server end to end: after a clean shutdown that
logged "persisted HNSW graphs to disk (next boot loads instead of rebuilding)",
the next boot logged
WARN persisted HNSW graph failed to load; falling back to rebuild
slot="content_vector" error=USearch load failed: Failed to read vectors
and rebuilt the index. Every boot. Correct results, no data loss, a WARN nobody
reads - and the optimization was dead.
Cause: `checkpoint_graphs` saves whichever index the slot holds, and
`build_slot_index` holds a BRUTE-FORCE index below the dimension-aware
crossover. Both write to the same `<kind>__<slot>.usearch` path, so a small slot
persisted a `BFVI` file that `load_persisted_slot` then handed to the USearch
reader. Observed on a 6-item, 128-D store; at production scale the slot is
USearch-backed, which is why the cluster never surfaced it - but every dev,
staging, and small-tenant instance pays a full index rebuild on every start, and
that rebuild is what the 20-minute startup probe budget exists for.
Fix: sniff the file's magic (brute-force `MAGIC` is now `pub(crate)` so the
reader and writer cannot drift) and dispatch to the matching loader. A
brute-force graph is still rejected when `expected_count` has grown past the
crossover, so the rebuild upgrades the backend to HNSW rather than pinning a
linear scan forever. Log lines now say "vector graph" and name the backend
instead of claiming HNSW for both.
Tests: `brute_force_slot_graph_round_trips_through_registry_persistence` covers
the case that was broken (the existing round-trip test forces USearch, which is
why it passed throughout), and
`grown_corpus_rejects_a_brute_force_graph_so_the_rebuild_upgrades_it` pins the
upgrade path. Verified live: the same data directory that produced the WARN now
logs `loaded persisted vector graph from disk (skipped full rebuild)
backend="brute-force" count=6`, with feed, text search, and vector search all
returning the pre-restart results.
|
||
|
jordan
|
c97aaa8e5b |
fleet remediation: make the workspace gate runnable, then fix what it caught
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|
jordan
|
cdbe9cb453 |
Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10
Reconciles two independently-developed lines from base
|
||
|
jx12n
|
6a937fc4bc |
feat(m12): multi-vector user preference modeling + ANN candidate-gen
Add multi-vector preference entity (per-signal-type preference vectors with event-time decay) feeding ANN candidate generation in the query executor. - entities: multi_preference vectors + event-time-aware preference updates - query/executor: ANN candidate-gen + personalization/pipeline integration - storage/keys, db ops, state_rebuild: persist & rebuild multi-vector prefs - ranking: profile + builtins support for multi-vector scoring - tidal-server/config: expose multi-preference knobs - tests/bench: m12_preference_event_time integration + multi_preference bench - docs: multi-vector-preference research, ROADMAP/ARCHITECTURE refresh, legal/tidaldb-patent-proposal - .codex/agents: codex agent definitions - chore: gitignore tool-regenerated .agents/ mirror (doc-guard rejects it) |
||
|
jx12n
|
c8ea05b032 |
fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator)
Root cause: after a checkpoint-based snapshot install the engine WAL is empty, so wal_term_mark() reports tail_term=0. decide_join compares (tail_term, frontier) lexicographically — tail_term FIRST — so 0 < leader_term classifies the reseeded shard ReseedRequired on EVERY boot regardless of the correctly-seeded frontier, re-latching the marker and self-restarting forever. Observed live on tidaldb-2: 30 CrashLoopBackOff restarts, leader tidaldb-1 term 5, baseline=536647, the frontier seeded correctly (from_seqno=536647) yet the loop persists because the (tail_term, frontier) compare never reaches the frontier. Fix 1 (already in tree): seed the post-open frontier from sentinel.snapshot_seq, not last_wal_seq() (which a checkpoint restore leaves at 0). Fix 2 (loop-breaker): durably synthesize the artifact's kind-3 TERM_MARKER WAL record in the post-open reseed seed, at the artifact's captured term + the reseed-leader region (threaded through an extended 18-byte install sentinel, back-compat with 10/8-byte). Makes wal_term_mark() truthful on this boot AND every reboot (blob records are NOT checkpoint-filtered on recovery), so decide_join returns Clean. Truthful, not a bypass: the artifact IS the leader's authoritative state at (term, seq); a genuinely-divergent node (no install sentinel) still surfaces tail_term > term -> Quarantine. Crash-idempotent via a monotonic-by-term guard. Fix 3: node-level reseed-restart coordinator — the single process-wide exit fires once, only after every hosted shard requests a restart or a bounded grace elapses, so one shard's self-restart never aborts a co-hosted sibling's in-flight install (S>1). No-op on the S=1 production topology. Fix 4: is_ready() returns 503 while any reseed marker (SnapshotRequired or Quarantine) is latched, closing the plain-restart serve-while-behind gap; readiness is bounded staleness, not "ready the instant the process is up". Tests: decide_join loop/fix/bounded-reseed unit; install-sentinel 18-byte round-trip + back-compat; engine durability (term marker survives a checkpoint advanced past it + crash-reopen); reseed-restart gate (5 cases); reseed_install carries the term. Verified: cluster_reseed 4/4 (zero-loss rolling restart x2, quarantine reseed, failover oracle), reseed_install 3/3, m12_reseed_term_marker 4/4, cluster_membership mp_idle/mp_dns/mp_remove x2, tidal-server lib 154/154. mp_scale_3_5_3 and mp_seed_join_snapshot_catchup OOM on this host (22GB colima VM); their /health/startup failure is process-down, not the is_ready path Fix 4 touches. |
||
|
jx12n
|
580142df49 |
feat(m12): election-divergence-fix + soak-eval streak + release tooling
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno acked under `ack=leader` (journal-only, un-replicated); `decide_join` now quarantines on THIS node's own frontier rather than comparing stream numbers across stream boundaries — eliminates false-quarantine churn on rolling restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum before step-down so the next leader inherits a clean prefix. New `load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains the divergence-fix regression suite; `replication_ops.rs` threads the signal. Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the 30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh` gates the election-divergence suite before any k8s push. Release tooling: `docker/release/` multi-stage Dockerfile + DR image; `scripts/build-release.sh` single repeatable cross-compile+buildx path. |
||
|
jx12n
|
a946c6128c |
fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests |
||
|
jx12n
|
44ec87871e |
fix(m12p6): 7th-edge — correct reseed seqno + skip suspect HNSW graph on reseed-pending close (rc11)
Fixes the rc9 over-correction: forcing baseline for ALL nodes (including caught-up ones) caused needless reseed cascades. Now only divergent nodes (frontier > baseline) use baseline as the reseed seqno; at/below-baseline nodes use frontier+1 so the leader picks cheap catch-up vs snapshot. Also skips the HNSW graph checkpoint on SIGTERM when the shard is reseed- pending: the in-memory index reflects suspect/divergent data the next boot discards, so saving it risks a "Failed to read vectors" failure on the post-reseed open. Durable checkpoints and WAL flush still run. close_shared() gains a save_graphs bool; shutdown_inner_impl() is the shared implementation; node.rs passes !reseed_pending. |
||
|
jx12n
|
727fbfcb6b |
fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9)
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9: 1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region` visits each hosted group's own shard subdir; per-group leader discovery appends `?shard=N` so a divergent shard heals from its own leader (not shard-0's WAL/term — cross-shard contamination). 2. leader self-join term (Bug 4): `become_leader_for_term` now calls `note_self_won_term` so the elected shard's `joined_term` is set and `cluster_promote` routes rebalances correctly (was: topology-era mis-read → legacy fenced promote → 500). 3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate prevents a node pulling its own stream (its stream isn't a registered peer) → eliminates the `PeerUnreachable(self)` loop. 4. scatter-merge degraded partial (Bug 1): failed shard logs + continues instead of `?`-failing the whole read; bounded read-admission semaphore (`offload.rs`) sheds as 429 instead of piling into a 36s p99. 5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4` most-recent sealed segments; online path gets the same retention clamp. Prevents brief-restart forced-reseed. 6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers` returns `needed=true` and the snapshot installs instead of looping. Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM (HNSW graph was not saved when request-scoped Arc clones were alive at shutdown); updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs. |
||
|
jx12n
|
a0399550d6 |
feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
|
||
|
jx12n
|
da5d2d4d53 |
fix(m12p2): invalidate SignalRanked top-K cache on CRDT reconciliation
apply_crdt_state force-sets accumulated scores outside apply_event_local, so it bypassed the single note_write() chokepoint that keeps the trending top-K cache fresh. A node going quiescent right after a partition heal kept serving the pre-reconciliation candidate set indefinitely. Invalidate the cache in apply_crdt_state too, with a regression test that reconciles a new high-score entity and requires it in the next candidate read. Also: extract the for_you/related ANN candidate cap to a named constant (ANN_PROFILE_CANDIDATE_LIMIT), keep the recall test's ef_search in lockstep with its construction default (honored since m12p3), and clarify the vector_search region doc (always null until m12p4 cross-shard reads). |
||
|
jx12n
|
bb21e69ae6 |
feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning
m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe + POST /vector_search (standalone + region node, merge-by-distance) + tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit). Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs. Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force. m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference vector, related=seed embedding (similar_to), graceful scan-fallback. Cached per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains --feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms. m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion, shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs. Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%). Recall corpus is now clustered (Gaussian mixture) in grid + harness. |
||
|
jx12n
|
1265140e28 |
feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline
- fault-injection cargo feature (compiled OUT of prod): slow-fsync + disk-full WAL hooks in tidal/src/fault.rs, inert until armed, tier-3 builds with feature - first-class invariant checkers (tests/support/invariants.rs): AckLedger no-acked-loss (now consumed by m11p3 gate), feed parity, single-leader-per-term, monotonic frontiers - cluster_faults.rs tier-3 suite 4/4: disk-full degrade+recover, slow-fsync lag+converge, both-slow quorum 503, asymmetric partition no-split-brain - tidal-stress soak gates: --json-summary + --max-p99-ms/--max-error-pct/ --fail-on-knee → non-zero exit on regression - Woodpecker cron nightly flow (chaos + gated soak), event-routed, not GH Actions - guarantee-traceability.md: roadmap §2 guarantees → named tests (closes G-C apparatus; 30-day-green is a calendar criterion) |
||
|
jx12n
|
005e292cbb |
fix(m11): review remediation + tidal-stress perf sweep + perf wave 2
Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review: - tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write - corrupt-WAL checkpoint_seq guard; PITR archive-before-delete - cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx - mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff - per-principal rate-limit + node-token marker-pinning tests - self-heal tier-3 coverage; 5 router-auth tests tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact feed-over-SLO verdict, shed annotation) + typed Body, workload.next 184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache; new benches/hotpath.rs + lib.rs. perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type ranking pre-pass. |
||
|
jx12n
|
d5d1e7d81a |
feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2
m11p8 closes G-O + §1.4-3: - Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N") - Grafana cluster row + 8-rule Prometheus alert group - Request-id / TraceLayer on both cluster routers; id rides forward hop - Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix - Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s - WAL PITR: wal.archive_dir, archive-before-delete gap-free - tidalctl backup/restore with BLAKE3 content-hash verification - Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor - signal_values.rs pre-fetches all signal kinds before scoring loop - Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes - Byte-identical output verified with A/B test harness |
||
|
jx12n
|
8673723319 |
perf(m11): kill signal_snapshot allocation cascade (perf-sweep wave 2 T1)
Replace ScoredCandidate.signal_snapshot Vec<(String,f64)> with
SmallVec<[(SignalKey,f64); 4]> where SignalKey is Static(&'static str)
| Owned(Arc<str>):
- Compile-time-constant labels (sort bases, relevance, co_engagement,
preference_affinity) -> Static: zero allocation, pointer-copy clone.
- Dynamic {signal}_boost/_penalty/_decay labels built once per query in a
RuleLabels hoist (was format! per-candidate-per-rule), shared by Arc.
Cohort rescore hoisted the same way.
- scored accumulator pre-sized to candidates.len().
- Owned Strings rebuilt only at the two response-assembly sites (<= limit).
Byte-identical output: full 1896-test lib suite green. Measured win
(cargo bench --bench ranking, committed-base vs working-tree):
score_200_hot 23.89->21.04us (-11.9%), score_200_trending
27.66->25.85us (-6.5%), score_200_full_pipeline 27.88->26.97us (-3.3%).
smallvec promoted from the lock to a direct dep (no new dependency
surface). perf-sweep doc updated; T2 (per-term DashMap collapse) next.
|
||
|
jx12n
|
6651c14adc |
feat(m11): cluster security (m11p7) + perf instrumentation floor
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
(HTTP foreign + zero-drop rotation under load), 7 security unit tests
perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)
new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
|
||
|
jx12n
|
bf57be18e1 | feat(m11): membership, snapshot install, and reseed (m11p5) | ||
|
jx12n
|
95461d3cf8 |
feat(m11): Raft leader election over WAL stream (m11p4)
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers, heartbeat-only divergence detection + quarantine, and fenced promote. Elections converge in 0.6–1.0s; zero acked-write loss across all kill points. Closes G5 (leaderless recovery) from the v0.9 wave. |
||
|
jx12n
|
d0a52e4530 |
feat(m11): catch-up timer retry + TSEG segment version header (m11p4)
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
|
||
|
jx12n
|
5ed2edb211 |
feat(m11): quorum-acked writes — ack=leader|quorum, commit index, durable frontier reports (m11p3)
ack=quorum gates replicated writes on a majority of the replica set durably holding them: followers push their durably-applied frontier (ReportApplied, once per apply round, decoupled from ship acks), the leader folds frontier reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex (k-th-largest durable mark), and handlers await it through an async watch-channel bridge (zero parked threads per waiter). Honest timeouts: retryable 503 naming the laggards; x-tidal-seq on every cluster write. Follower blob applies are batched under group-commit fsyncs (22x seeding). Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss. Seven-dimension review pass (all confirmed findings fixed): - WAL blob drain now ABORTS on the first write failure instead of reusing the failed seqno mid-drain (a torn record buried mid-segment would truncate every later acked record on replay) - apply_replicated_blobs waits every staged append even after a mid-batch failure, parses metadata once, and moves records into Arcs shared with the WAL writer (no deep clone per record on the follower apply path) - CommitIndex: zero-peer fast path now respects demotion (active checked under lock before the single-replica return), k-th-largest uses select_nth over a reused scratch buffer - await_quorum: re-reads the index once after the deadline fires (no false 503 for a write that committed in the race window), warns when the commit-watch bridge dies outside shutdown, zero-peer path checks active - notify_applied report failures: WARN on the first failure of a streak, INFO on recovery (a silently stalling frontier reads as unexplained quorum 503s); receiver skips re-notifying unadvanced frontiers - x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed through forwards) so durability cursors can tell dedup from no-seqno - docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade order (leader first — a pre-m11p3 leader silently downgrades quorum requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for report-loss diagnosis on the quorum-timeout alert Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process quorum suite green (8/8 kill points, zero acked loss, partition gate/recover). |
||
|
jx12n
|
225751d34d |
feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2)
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push, microseconds) separate from group-commit fsync; ShipQueue batches+windows outbound segments; receiver coalesces inbound chunks before applying. Adds first tidaldb_cluster_* metrics. m11p2 — leader WAL is now THE replicated log: fsynced batches feed a bounded WalShipFeed and ship byte-identical to followers; WAL seqnos survive restarts (relay-reset hazard gone). Item metadata and embeddings journal kind-1/2 blob records on the same stream as signals; the m8p10 HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via server-streaming RPC, triggered on gap detection, follower boot, and leader heal nudge. Promote carries a stream baseline so peers skip pre-stream history. |
||
|
jx12n
|
8a0950260f |
feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps. |
||
|
jx12n
|
0c99709990 |
fix(replication): redeliver_missed shipped to the wrong shard over gRPC
Partition recovery (heal_region) and await_convergence never worked over real GrpcTransports: redeliver_missed sent to entry.source_shard (the LEADER's shard), but a follower's transport only knows its own shard as a peer (self-loop wiring in tidal-server::cluster), so every recovery ship failed the peer lookup — and the error was swallowed by 'let _ ='. ChannelTransport ignores the destination, which is why the in-process suite never caught it. Found live on k3s: POST /cluster/partition + writes + /cluster/heal left the region at lag=3 forever, no log line. Fix: ship to transport.local_shard() (matching the eager path's send_segment(ShardId(region.0), ...)); keep entry.source_shard in the payload (receiver advances applied_seqno per SOURCE shard). WARN on ship failure instead of dropping it. Adds a RecordingTransport regression test that locks dest == local_shard + payload source == leader — the exact case channels cannot catch. cargo test: 263 replication + 13 testing + tidal-server all pass. |
||
|
jx12n
|
1092d34c39 |
feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly |
||
|
jx12n
|
9728194f16 |
fix: M0-M10 code-review pass2 remediation — all 91 findings
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md across the engine, network, server, and CLI crates: session restore, replication/CRDT, WAL format and recovery, storage indexes, query/ranking executors, cohort/community governance, and scatter-gather routing. Adds regression tests: - review_pass2_creator_search_filter - review_pass2_d_replication - review_pass2_query_for_session - review_pass2_storage_indexes_bitmap_cache - review_pass2_zone_a_sessions Verified: cargo clippy -D warnings and full test suite green across all crates. |
||
|
jx12n
|
ad4134e280 |
chore: doc consolidation, seven-dimension review fixes, and commit hooks
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical homes (root *.md and docs/), with planning/specs/research/reviews moved up - Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/ - Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh - Implement M0-M10 seven-dimension review findings across engine, net, server, and tidalctl (durability, replication, query, WAL, storage, CLI hardening) |
||
|
jx12n
|
b55ad70141 |
fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across the engine, server, net, and CLI surfaces: - WAL/session-journal durability, checkpoint format, and crash-recovery hardening - Replication shipper/receiver, tenant isolation, and migration paths - Cluster scatter-gather, router, standalone server + health/offload endpoints - tidalctl refactored into command modules with JSON output and WAL-state tooling - Cohort, governance, signal-ledger, and vector-registry correctness fixes - Expanded UAT/integration/durability test coverage across all milestones |
||
|
jx12n
|
69ae6e64a1 |
fix: M0-M10 remediation — deferred post-filters, WAL hardening, docker build fixes
- Extract shared deferred post-filter (InCollection/MinSignal/MaxSignal/ NearLocation/SocialGraph) into query/executor/post_filter.rs so RETRIEVE and SEARCH apply identical predicates; SEARCH previously ignored four variants - Harden WAL writer/compaction/dedup paths - Brute-force vector store fixes + tests - Governance community ledger updates - Pin docker builders to bookworm (glibc/libmvec match) and add protobuf-compiler for tidal-net's tonic-build - Add M0-M10 code-review document |
||
|
jx12n
|
3bcfb3c576 |
feat: Bazel build, crate docs/ai-lookup, docker images, and engine hardening
- Add BUILD.bazel across tidal, tidal-net, tidal-server, tidalctl for bzlmod build - Add tidal/ crate docs (README, CHANGELOG, CONTRIBUTING, AGENTS, CLAUDE, API, ARCHITECTURE) and ai-lookup reference - Add docker standalone/cluster/deploy images, compose, and prometheus config - Harden WAL (batch format, writer, dedup, diagnostics), text syncer/collectors, and vector registry - Expand tidalctl CLI and tests; restructure WAL/visibility integration test suites - Refine tidal-net transport/client/server and tidal-server cluster/scatter-gather |
||
|
jordan.washburn
|
b16025b8b2 |
feat: pluggable cluster transport + multi-process E2E test harness
Three changes closing M8 gaps identified during verification: 1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE). Added Known Gaps table (G1: in-process transport, G2: tier-3 tests, G3: hash inconsistency). 2. SimulatedCluster transport now pluggable via ClusterConfig.transports. Default (None) uses new ChannelTransport (crossbeam, same behavior). When Some, accepts external transports (e.g., GrpcTransport from tidal-net). Updated redeliver_missed to use &dyn Transport. Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged. 3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs). ClusterHarness spawns real tidal-server cluster OS processes, allocates dynamic ports, generates topology YAML, polls health, and cleans up via SIGTERM. Two tests: smoke (write + converge + verify follower reads) and promote (leader change + continued writes). Feature-gated behind cluster-e2e. |
||
|
jordan.washburn
|
fe711870be |
feat: M8 phases 7-10 — gRPC transport, cluster server, scatter-gather, multi-node UAT
Delivers the distributed fabric's network layer and HTTP cluster surface: **m8p7: tidal-net crate (gRPC transport)** - GrpcTransport implementing Transport trait via tonic 0.12 - Per-peer circuit breaker (Closed/Open/HalfOpen), mutual TLS via rustls - Boxed error types (clippy-clean), graceful mutex recovery, debug_assert against calling block_on from tokio context - Proto: WalShipping service (ShipSegment, StreamSegments stub, Heartbeat) - 19 tests: contract, mTLS, reconnection, multi-node UAT, benchmarks **m8p8: cluster subcommand + HTTP routes** - ClusterState wrapping SimulatedCluster with region name mapping - Routes: /health, /cluster/status, /cluster/promote, /partition, /heal - Data routes: /items, /embeddings, /signals, /feed, /search (region-aware) - Ranking profiles wired through ClusterConfig to all cluster nodes - Topology YAML config, docker/cluster/Dockerfile (ENTRYPOINT+CMD, non-root) **m8p9: scatter-gather query routing** - Entity-sharded writes via Knuth multiplicative hash - Scatter-gather RETRIEVE and SEARCH with deadline propagation (50ms-5ms) - Partial failure: degraded=true with unavailable_shards metadata - 6 tests: distribution, determinism, multi-shard retrieve, degraded partial results, deadline propagation, scatter-gather search **m8p10: gRPC transport integration tests** - 8 tests over real gRPC: replication convergence, idempotent replay, mixed signals, 3-node fan-out, partition/heal, degraded follower, perf - Documented as tier-2 (in-process+gRPC); tier-3 multi-process pending |
||
|
jordan
|
9eecb3ef0b | Merge remote-tracking branch 'origin/main' | ||
|
jordan
|
d8e408350c | feat(m9/m10/p1): community policy engine, signal revocation, agent capability boundaries, feedback loop, metrics instrumentation | ||
|
jordan
|
006d3d058a |
fix: populate signal_snapshot in ranking results and fix quickstart demo
- Score and boost contributions now captured per candidate in signal_snapshot:
score_by_sort() returns (f64, Vec<(String, f64)>) with named signal values
(e.g. view_velocity, share_velocity, view, like). Boost contributions are
appended as `{signal}_boost`. compute_raw_score() threads the snapshot
through; score_inner() and score_personalized() write it onto ScoredCandidate
instead of always initialising to vec![].
- Quickstart switched from `trending` to `hot` profile. The trending profile
scores by 24h velocity, which requires signals to arrive over real elapsed
time to populate hour-level bucket aggregates — impossible in a self-contained
demo. The hot profile (AllTime view count + age decay) works with any timestamp
and produces clearly differentiated scores. Signals now use different counts
per item to drive meaningful ranking output.
|
||
|
jordan
|
c4e0e49d6a | Merge branch 'main' of github.com:orchard9/tidalDB | ||
|
Alan Kahn
|
1d826c87b2 |
feat: schema-level ranking profile definitions
Add support for defining ranking profiles in schema YAML, allowing deployments to override builtin profiles with deployment-specific signal names and tuning parameters. - Add override_register() to ProfileRegistry for clean builtin replacement - Add with_profiles() to TidalDbBuilder to thread schema profiles - Parse profiles section in config.rs with full sort/strategy/agg support - Validate that profile signal references exist in schema at startup - Change load_schema() to return (Schema, Vec<RankingProfile>) This closes the gap where the builtin for_you profile referenced signals (view, like, share) that don't exist in deployment schemas, causing all feed scores to normalize to 1.0. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
jordan
|
6f26d03c77 |
feat(m9): implement purge re-materialization engine
Adds the M9 purge re-materialization feature: a WAL-replay background
engine that rebuilds community cohort aggregates for a (user, community)
pair after retroactive signal purge, restoring ranking correctness without
modifying the immutable WAL.
Key additions:
- cohort::rematerialization module: PurgeJobQueue, RematerializationEngine,
WAL replay, atomic CohortSignalLedger swap, BLAKE3 audit log, metrics counters
- TidalDb::{submit_purge_job, purge_job_status, rematerialization_metrics}
public API (db/rematerialization.rs)
- Engine auto-starts in persistent mode; clean shutdown before WAL teardown
- 6 integration tests in tests/m9_purge_remat.rs covering ephemeral and
persistent modes, job lifecycle, and multi-job independence
- Split oversized files to comply with 600-line limit: db/mod.rs →
db/from_parts.rs, entities/revocation.rs → revocation/{mod,tests}.rs,
schema/validation/builders.rs → builders/{mod,tests}.rs,
signals/warm.rs → warm/{mod,tests,proptests}.rs
- Fix pre-existing bootstrap errors: export AuditKind from session module,
add overrides_rejected to SessionSnapshot deserialization
|