20 Commits

Author	SHA1	Message	Date
jordan	a588f01f63	ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units ... Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL, and a CHANGELOG that named profiles that do not exist. All verified before fixing. BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0. So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable `created_at` outranked the genuinely newest item and `normalize` reported its score as 1.0. The codebase had already ruled on this and gone the other way. `score_shortest`, `score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and `helpers.rs` carries a regression test named for the exact anomaly -- `normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because `Shortest` hit it first: "a pre-clamped 0.0 floated above them and the last-ranked (missing) item reported the highest score 1.0". I reintroduced it on a new negated scale. Three reachable data classes, none hypothetical: a legacy row written before `created_at` was materialized (`state_rebuild.rs` documents the class), an empty map from `deserialize_metadata` on a short or corrupt row, and an entity dropped by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT STORAGE READ ERROR could promote an item to the top of the feed. Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole set ties, unchanged. Map present but this entity has no usable `created_at` -> NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because that is a real value that legitimately means "newest"; an absent one carries no recency claim at all. Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports `needs_item_metadata`, so the map is loaded on essentially every Hot query and the per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a single undated row the freshest divisor in the set: against a year-old cohort at the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with two views outranked correctly-dated items with tens of thousands. Across all four Hot builtins. Now scored at Hot's floor. BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_ newest` ranked candidates against the newest ids of the WHOLE universe, but `scan_candidates` iterates the universe bitmap ASCENDING and breaks at `(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a catalog whose ids are assigned in creation order -- which is exactly what `metadata_with_created_at`'s `Timestamp::now()` default produces -- the global newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by` I replaced kept the newest available. Strictly worse than the bug I set out to fix, and only visible when degraded. The key is now composite -- recency rank, then DESCENDING id for anything the oversample did not cover -- so where index and candidate window overlap the survivors are genuinely newest, and where they do not it degrades to the documented pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_ key`: this path runs ONLY when the load shedder has already decided the node cannot afford the work, so it must stay a linear partition, and only membership of `[0, cap)` matters because Stage 3 re-orders the survivors anyway. Mutation-proven, and the numbers show why the old test was blind: rank-only key returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the pre-existing 150-item test PASSES under that same mutation -- its fixture had the oversample covering the whole universe AND the newest items at the low ids, the one configuration where the defect cannot appear. New test uses a 3000-item time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200 so the guard skips), limit=25 does. Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING entity id, and I had made newest-first coincide with it, so any mutation that merely TIED the set still produced the asserted vector. The gate mutation proves it: reverting `needs_metadata_for_sort` to its old four-variant form yielded `left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`, which is the exact drift this work exists to prevent. Every ordering fixture is now non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and ascending id (the tie collapse). CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them carry a diversity block so they already loaded item metadata and change behaviour the instant this image rolls. Replaced with a verified table. Also cited the spec conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified `Sort::New` as "created_at DESC". created_at units are now trusted with a warning instead of silently. A seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores -496731 against boost sums in single digits -- re-entering through DATA the exact "recency annihilates every boost" defect this work removed -- and Hot buries the item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was inert; this work made it live. The repo's OWN fixtures made that mistake in three places, which is the proof it is the natural one. `metadata_with_created_at` now warns when the value is too small to be nanoseconds, logging entity id, value, age_hours and age_years so an operator can act. It does NOT rewrite the value -- guessing the unit would corrupt what the `created_at` range index already reads as nanoseconds -- and does NOT reject the write, which would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server: seconds, millis and micros each warn with age_years=56; a real nanosecond value is silent. The nanosecond contract is now documented on the public DTO and propagates to the OpenAPI schema, where it was invisible before. Stale docs corrected: two published blog posts and two claim-verification ledgers were certifying "entity recency (higher ID = newer)"; the e2e fixture contract justified an interleaving decision with reasoning that is now false (its conclusion still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:" rationale cited behaviour this work removed, so it is now marked PENDING RE-MEASUREMENT with the three specific measurements named rather than left reading as justified. Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors. Fast integration suites all green. Real-server e2e re-verified: `new` and `hot` both return newest-first, `new`'s scores now evenly spaced across evenly spaced ages.	2026-08-31 21:31:31 -06:00
jordan	6385425a92	ranking: make Hot and New age-aware; fix the same gap in three more places ... `score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added.	2026-08-31 19:58:01 -06:00
jordan	a6f663f002	harden: validate embeddings before the WAL, fix the reseed-latch leak, run every test suite ... Fixes the two defects a malformed probe exposed on the live cluster, plus the coverage gap that let a stale assertion survive the same day it was falsified. TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim slot was appended to the WAL FIRST, then validated, then answered 500 — so an already-durable, unapplicable record shipped to both followers, halted both receivers, and put shard 1 into a quorum-write outage. Validation now runs before the append and returns 400 via invalid_input; nothing enters the log. `storage::vector::validate_dimensions` is now the single comparison, replacing an inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a dimension check drift, and the apply-path copy is the one that halts replication when it disagrees. The receiver's halt-vs-skip decision is now explicit instead of "halt on anything". A record whose failure is deterministic and node-independent (schema width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the frontier advances; a record that could become applicable after a binary upgrade (unknown batch kind, capability skew) still halts, because skipping那 would silently drop replicated data. Both branches are proven reachable by tests. TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups latched a marker per group but discharged on a single seqno, so two latches meant permanent 503 on a node whose every shard read lag 0 — it hit all three pods during the roll and each needed a manual delete. Gaps are now tracked per group in a ReseedGapSet and cleared on evidence about themselves; a REFUSED reseed_self_restart re-evaluates every 15s instead of waiting for a latch that never arrives. /health's cause ladder was also lying: it printed "joiner boot not yet converged" for a node whose groups had all converged, because the fallback asserted a state it never tested. It now names the outstanding gaps, gained the decommissioned-by-signal arm that is_ready checked but the ladder did not, and its terminal arm says "reason unavailable" rather than inventing one. COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical: cluster_routes still asserted the wire fabrication removed hours earlier (applied_events == 0 with a lag derived from it) and nothing caught it because nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search (distance contract) and cluster_poison_embedding (task 17's own gate) were in the same position, so those guards would have rotted identically. Every suite now has a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing before being scheduled. Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13 minutes in — it demanded an unreachable peer report worst-case lag, i.e. it required the fabrication task 04a deleted. Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on touched trees at 431340f); lib 2115 passed; all 8 fast suites green; cluster_chaos 5, cluster_sharding 5, cluster_poison_embedding 1, cluster_cross_shard_reads 2, cluster_graph_persistence 1, cluster_multiproc 5, cluster_e2e 2; doc-guard OK.	2026-08-31 00:46:00 -06:00
jordan	fe8d0c87e7	harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path ... Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.	2026-08-30 20:55:58 -06:00
jordan	8aa1fbb414	vector search: normalize the query, instrument the blob path, expose per-group vector counts ... Three real defects, plus a retracted fourth that was a probe artifact. P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized every stored vector; the read path passed the caller's raw query straight to the index, so the two sides lived in different spaces. With unit v, d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by |q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant across candidates - which is why it went unnoticed; what broke was every absolute use of the number. WIRE-VISIBLE, recorded in CHANGELOG. P2 (fixed) - the blob path had zero instrumentation. Added per-kind tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind is now the ONE exhaustive match over the variants (kind() derives from it), so a new variant is a compile error in one place instead of a silent zero in three. Only the live apply path is counted - boot replay would inflate applied past originated on every restart. Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics owner's shard group, so on a 3-group node two thirds of the corpus had no vector-count series at all. Co-located groups now render shard="N"; the owner stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets each replica set separately without double-counting. P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe wrote through the /sharded/ surface, which hash-partitions and applies to the owning region's local store with no WAL append, and therefore does not replicate BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus /embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six reach exactly one node. Both are now pinned by tests. See tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml entry RETRACTED_blob_replication_rf1_2026_08_30. Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling (the project's existing path-attribute convention), and the three hand-rolled l2_normalize copies collapsed to one. The two entity copies used a zero threshold about 2900x looser than the canonical one; normalize_centroid now names the centroid zero-tolerance policy once, and a test pins the tightened behavior. Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in cluster_sharding are now serialized - four concurrent 3-node clusters made the pre-existing failover test miss its 10s budget.	2026-08-30 13:57:36 -06:00
jx12n	c22a3b65a6	docs: withdraw the pre-release "not ready for production" disclaimer ... M0-M12 are shipped and the HA cluster runs in production on k3s, so the pre-release disclaimer no longer describes the project. Removes it from the canonical doc set and corrects the readiness text that had gone stale. - README.md: replace the "Pre-release / not yet recommended for production" banner with a production-ready statement; drop "(experimental)" from the cluster status bullet; state the post-1.0 versioning posture (additive in minor releases, breaking changes get a documented migration path). - CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md / docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a guard against standing up a multi-node fabric by accident rather than a readiness warning. - CHANGELOG.md: record the stability posture under [Unreleased], superseding the historical 0.1.0 "no stability guarantees" note (left intact as history). - k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead. Also corrects text that was factually wrong since m11p3/m11p4: the multi-process cluster gate, its CLI help, and the served OpenAPI description all still claimed quorum-ack writes and automatic failure detection did not exist. They do. Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries, the kubernetes.md rc7 fix note) are left unchanged. Verified against a running binary, not just the build: the opt-in gate's refusal message, the startup WARN, /health 200, and the served /openapi.json description all carry the new text. cargo fmt clean; clippy -D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155 server lib tests pass; scripts/check-docs.sh OK. Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP	2026-07-30 19:03:34 -06:00
jx12n	4051077cff	docs(m12): refresh API, specs, ops, and roadmap to the shipped M12 reality ... - API.md: document `similar_to`/`region`/`unavailable_shards` on /feed and /search, the new POST /vector_search k-NN probe, and the cluster-node-only routes (/cluster/*, /sharded/*, /hardnegs) - CHANGELOG.md: M12 entries — multi-vector preference + ANN candidate-gen, idle-readiness + TLS scale-up (m12p5/p6), sharded ingestion (m12p4) - ROADMAP.md: mark M11 + M12 COMPLETE; restate the v1.0 bar (30-day-green nightly calendar + Ref-A/k3s throughput re-runs) - prometheus-alerts.yaml: add ship-stall, quorum-lag, divergence-quarantine, reseed-pending, and snapshot-pin-force-drop cluster alerts - check-docs.sh: self-updating milestone-status freshness guard derived from ROADMAP's latest COMPLETE milestone - refresh specs (00-14), ai-lookup, guides, and runbooks to M0-M12	2026-06-23 21:39:55 -06:00
jx12n	bb21e69ae6	feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning ... m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe + POST /vector_search (standalone + region node, merge-by-distance) + tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit). Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs. Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force. m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference vector, related=seed embedding (similar_to), graceful scan-fallback. Cached per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains --feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms. m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion, shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs. Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%). Recall corpus is now clustered (Gaussian mixture) in grid + harness.	2026-06-14 11:07:09 -06:00
jx12n	44b768b8c6	feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) ... End the "replicated XOR sharded" split: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes. - One unified write surface: /items,/embeddings,/signals hash-route to the owning shard group's leader (ShardRouter FNV-1a) AND replicate at RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are per-group; NotLeader names the group. - Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced leadership move) + /cluster/shards/{id}/replicas (add/remove replica). A ?shard= selector threads through every per-shard admin verb and is propagated on intra-group forwards (ShardReplica::admin_path). S=1 is byte-for-byte (no selector, no shard in NotLeader body). - Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over real OS processes — SIGKILL a node under ack=quorum load → only its shard-leaderships re-elect, reads never stop, zero acked loss across random kill points; plus a rebalance-verb test. Harness: MultiProcCluster::start_sharded. - tidal-stress drives the single path (WritePath::Leader|Sharded gone), spreading writes round-robin across gateways or pinning --leader-url. - Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err, ~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A. Known follow-up (tracked): per-group-aware node readiness and cross-node read fan-out under PARTIAL placement.	2026-06-13 18:23:43 -06:00
jx12n	1265140e28	feat(m11): continuous correctness (m11p9) — fault classes, invariant checkers, soak gates, nightly pipeline ... - fault-injection cargo feature (compiled OUT of prod): slow-fsync + disk-full WAL hooks in tidal/src/fault.rs, inert until armed, tier-3 builds with feature - first-class invariant checkers (tests/support/invariants.rs): AckLedger no-acked-loss (now consumed by m11p3 gate), feed parity, single-leader-per-term, monotonic frontiers - cluster_faults.rs tier-3 suite 4/4: disk-full degrade+recover, slow-fsync lag+converge, both-slow quorum 503, asymmetric partition no-split-brain - tidal-stress soak gates: --json-summary + --max-p99-ms/--max-error-pct/ --fail-on-knee → non-zero exit on regression - Woodpecker cron nightly flow (chaos + gated soak), event-routed, not GH Actions - guarantee-traceability.md: roadmap §2 guarantees → named tests (closes G-C apparatus; 30-day-green is a calendar criterion)	2026-06-13 15:23:59 -06:00
jx12n	d5d1e7d81a	feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 ... m11p8 closes G-O + §1.4-3: - Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N") - Grafana cluster row + 8-rule Prometheus alert group - Request-id / TraceLayer on both cluster routers; id rides forward hop - Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix - Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s - WAL PITR: wal.archive_dir, archive-before-delete gap-free - tidalctl backup/restore with BLAKE3 content-hash verification - Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor - signal_values.rs pre-fetches all signal kinds before scoring loop - Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes - Byte-identical output verified with A/B test harness	2026-06-13 09:17:49 -06:00
jx12n	6651c14adc	feat(m11): cluster security (m11p7) + perf instrumentation floor ... m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte): - gRPC replication mTLS by default via a custom tokio-rustls acceptor + DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap, no pod restart, no inotify) - inter-node HTTP TLS sharing the same resolver (one rotation, both planes) + per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403 - admin audit log (operator-leg only) + per-principal rate limit (engine RateLimiter; sibling nodes exempt) - k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback; secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable) - exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs (HTTP foreign + zero-drop rotation under load), 7 security unit tests perf — instrument floor (sweep Wave 1): - new tidal/benches/wal.rs + tidal-server/benches/scatter.rs - p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md - add @tidal-performance agent (Martin Thompson) new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs, docs/planning/milestone-11/phase-7.md	2026-06-13 01:25:35 -06:00
jx12n	bf57be18e1	feat(m11): membership, snapshot install, and reseed (m11p5)	2026-06-12 19:55:54 -06:00
jx12n	95461d3cf8	feat(m11): Raft leader election over WAL stream (m11p4) ... Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers, heartbeat-only divergence detection + quarantine, and fenced promote. Elections converge in 0.6–1.0s; zero acked-write loss across all kill points. Closes G5 (leaderless recovery) from the v0.9 wave.	2026-06-11 23:30:24 -06:00
jx12n	d0a52e4530	feat(m11): catch-up timer retry + TSEG segment version header (m11p4) ... WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved) prepended to every new segment. Legacy headerless segments (m0-m11p3) read as implicit v0 — no migration. Unknown magic/version surfaces as WalError::SegmentFormatUnknown at open time; foreign files are never repaired or truncated (fixes the silent data-loss path from the p3 rollout incident where torn-tail repair zeroed a follower's unreadable segments). Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream errors that skip the shard now arm a timer retry (re-arm-on-skip is the load-bearing liveness fix — without it a skipped pull never re-fires and the follower stays permanently behind). Single retry pending per shard; CatchupRunner owns the Arc'd state shared between the retry tasks and the transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path. Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.	2026-06-11 17:05:20 -06:00
jx12n	5ed2edb211	feat(m11): quorum-acked writes — ack=leader|quorum, commit index, durable frontier reports (m11p3) ... ack=quorum gates replicated writes on a majority of the replica set durably holding them: followers push their durably-applied frontier (ReportApplied, once per apply round, decoupled from ship acks), the leader folds frontier reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex (k-th-largest durable mark), and handlers await it through an async watch-channel bridge (zero parked threads per waiter). Honest timeouts: retryable 503 naming the laggards; x-tidal-seq on every cluster write. Follower blob applies are batched under group-commit fsyncs (22x seeding). Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss. Seven-dimension review pass (all confirmed findings fixed): - WAL blob drain now ABORTS on the first write failure instead of reusing the failed seqno mid-drain (a torn record buried mid-segment would truncate every later acked record on replay) - apply_replicated_blobs waits every staged append even after a mid-batch failure, parses metadata once, and moves records into Arcs shared with the WAL writer (no deep clone per record on the follower apply path) - CommitIndex: zero-peer fast path now respects demotion (active checked under lock before the single-replica return), k-th-largest uses select_nth over a reused scratch buffer - await_quorum: re-reads the index once after the deadline fires (no false 503 for a write that committed in the race window), warns when the commit-watch bridge dies outside shutdown, zero-peer path checks active - notify_applied report failures: WARN on the first failure of a streak, INFO on recovery (a silently stalling frontier reads as unexplained quorum 503s); receiver skips re-notifying unadvanced frontiers - x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed through forwards) so durability cursors can tell dedup from no-seqno - docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade order (leader first — a pre-m11p3 leader silently downgrades quorum requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for report-loss diagnosis on the quorum-timeout alert Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).	2026-06-11 13:28:08 -06:00
jx12n	225751d34d	feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2) ... m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push, microseconds) separate from group-commit fsync; ShipQueue batches+windows outbound segments; receiver coalesces inbound chunks before applying. Adds first tidaldb_cluster_* metrics. m11p2 — leader WAL is now THE replicated log: fsynced batches feed a bounded WalShipFeed and ship byte-identical to followers; WAL seqnos survive restarts (relay-reset hazard gone). Item metadata and embeddings journal kind-1/2 blob records on the same stream as signals; the m8p10 HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via server-streaming RPC, triggered on gap detection, follower boot, and leader heal nudge. Promote carries a stream baseline so peers skip pre-stream history.	2026-06-11 09:10:06 -06:00
jx12n	8a0950260f	feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites ... Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.	2026-06-10 14:07:33 -06:00
jx12n	ad4134e280	chore: doc consolidation, seven-dimension review fixes, and commit hooks ... - Eliminate the tidal/ self-contained doc mirror; docs now have two canonical homes (root *.md and docs/), with planning/specs/research/reviews moved up - Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/ - Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh - Implement M0-M10 seven-dimension review findings across engine, net, server, and tidalctl (durability, replication, query, WAL, storage, CLI hardening)	2026-06-08 22:46:28 -06:00
jordan	213b8efcca	feat: complete M6-M7 + Enterprise Readiness milestones; split oversized source files per CODING_GUIDELINES §9 ... Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-02-23 22:41:16 -07:00