64 Commits

Author	SHA1	Message	Date
jordan	67a175e19a	fix(cluster): /cluster/status reported a total partition on a healthy fleet ... The status aggregator probed every peer's /cluster/status/local with NO credential. That route is token-gated, so on any cluster with TIDAL_API_KEY set each peer answered 401, and every peer row collapsed to the honest-unknown placeholder: reachable false, partitioned true, applied_events null, lag_events null, version "". Only the OWN region survived, because it is served in-process with no HTTP hop. The result: the one surface an operator reads to clear the N/N+1 version skew before a rolling upgrade - and the deploy runbook's own step 4 - reported the whole cluster partitioned while it was perfectly healthy, with every peer's version blank so the skew check was blind. Observed on the GKE cluster: all three pods answered /health 200 under leader tidaldb-1, replication applied, and a curl between the exact same pod FQDNs returned 200, while /cluster/status insisted both siblings were unreachable. security::bearer_from_env documents this precise trap - a node that "dials an authenticated peer with NO credential" - and count_alive_other_voters already attaches the bearer. This forwards the CALLER's Authorization header instead of reaching for creds.bearer(), matching the relayed-operator-hop convention that /cluster/promote already uses, so a weakly-authenticated caller cannot borrow the node's own credential to read peers it could not read directly. Why it escaped: every other multi-process test runs with no TIDAL_API_KEY, where a credential-less probe succeeds - including cluster_multiproc's all-reachable assertion. The new test carries the key. Verified differential: it fails on the reverted code with exactly the observed shape (us-east reachable, both peers null/false/empty) and passes with the fix. cluster_multiproc still 5/5.	2026-09-16 00:51:57 -06:00
jordan	6ad8c51cfa	fix(cluster): stop discarding signal context on every clustered write ... `POST /signals` on a cluster node applied (signal, entity, weight) and dropped `user_id`/`creator_id` while still answering 204. Everything user-scoped was silently lost: hard negatives, seen tracking, (user, creator) interaction weight, preference vector, per-user trending index, cohort attribution, community forwarding. A clustered deployment looked healthy, accepted every behavioural signal, and learned nothing — with no wire evidence of the loss. This was known and worked around rather than fixed: `dto.rs` documented the fields as "(standalone path only)", and thepeach's staging tofu pinned its discover corpus to a STANDALONE instance citing this exact defect (infra/tofu/envs/staging/svc_api.tf). thepeach's client does populate both fields (crates/tidaldb-client/src/lib.rs), so the loss was live, not theoretical. Cause: `stage_signal_local` called `signal_staged`, which takes no context. `signal_with_context` (standalone) takes both and runs the side effects after the base write. Fix, engine side. Extract the two halves of `signal_with_context` that were not reusable: `validate_context_entity` (the u32 item-slot guard, which must reject BEFORE anything is written — a truncated id in a durable Tag::HardNeg row is a permanent cross-item collision that survives restart) and `apply_signal_context` (every side effect). `StagedSignal` now optionally carries the context and applies it in `wait()` AFTER durability, so the ordering matches the synchronous path and a crash cannot leave a side effect whose base signal was never logged. `signal_with_context_staged` is the new entry point; both staged constructors share one admission sequence. Not routed through the synchronous `signal_with_context` on purpose: that would serialise every context-carrying write on its own fsync, and since essentially every real signal carries a user, it would have cost the whole m11p1 group-commit win on the hot path. Fix, server side. `stage_signal_local` takes the two ids and ALWAYS uses the context-carrying stage — it degrades to the plain staged write when both are None, so there is no second path to keep in step. Both production handlers (`write_signal`, `sharded_write_signal` in cluster/node.rs, served by `build_region_router`, which is what `run_seed_join_cluster` and `run_region_cluster` boot — the deployed RF3 topology) now pass them. The experimental single-process router (cluster/routes.rs, `SimulatedCluster` + scatter_gather) genuinely cannot honour context: its relay applies (signal, entity, weight). It now REFUSES such a request with 400 naming the supported route, rather than accepting and discarding. A 204 over a dropped user_id is the failure mode that caused this. Verified, not assumed: reverted the fix to context-always-None and the new differential tests fail exactly as the bug did — observed (false, false, 0.0) against the synchronous path's (false, true, 1.0), i.e. no seen bit and no interaction weight. Tests are differential (staged vs synchronous end state, with positive controls so two empty states cannot agree) precisely because that is the check the original bug got past. Existing suites: 2144 engine + 197 server, zero failures.	2026-09-15 17:48:28 -06:00
jordan	936da3c520	feat: add exact qualified-hot ranking API	2026-09-09 09:34:26 -06:00
jordan	a588f01f63	ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units ... Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL, and a CHANGELOG that named profiles that do not exist. All verified before fixing. BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0. So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable `created_at` outranked the genuinely newest item and `normalize` reported its score as 1.0. The codebase had already ruled on this and gone the other way. `score_shortest`, `score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and `helpers.rs` carries a regression test named for the exact anomaly -- `normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because `Shortest` hit it first: "a pre-clamped 0.0 floated above them and the last-ranked (missing) item reported the highest score 1.0". I reintroduced it on a new negated scale. Three reachable data classes, none hypothetical: a legacy row written before `created_at` was materialized (`state_rebuild.rs` documents the class), an empty map from `deserialize_metadata` on a short or corrupt row, and an entity dropped by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT STORAGE READ ERROR could promote an item to the top of the feed. Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole set ties, unchanged. Map present but this entity has no usable `created_at` -> NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because that is a real value that legitimately means "newest"; an absent one carries no recency claim at all. Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports `needs_item_metadata`, so the map is loaded on essentially every Hot query and the per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a single undated row the freshest divisor in the set: against a year-old cohort at the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with two views outranked correctly-dated items with tens of thousands. Across all four Hot builtins. Now scored at Hot's floor. BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_ newest` ranked candidates against the newest ids of the WHOLE universe, but `scan_candidates` iterates the universe bitmap ASCENDING and breaks at `(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a catalog whose ids are assigned in creation order -- which is exactly what `metadata_with_created_at`'s `Timestamp::now()` default produces -- the global newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by` I replaced kept the newest available. Strictly worse than the bug I set out to fix, and only visible when degraded. The key is now composite -- recency rank, then DESCENDING id for anything the oversample did not cover -- so where index and candidate window overlap the survivors are genuinely newest, and where they do not it degrades to the documented pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_ key`: this path runs ONLY when the load shedder has already decided the node cannot afford the work, so it must stay a linear partition, and only membership of `[0, cap)` matters because Stage 3 re-orders the survivors anyway. Mutation-proven, and the numbers show why the old test was blind: rank-only key returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the pre-existing 150-item test PASSES under that same mutation -- its fixture had the oversample covering the whole universe AND the newest items at the low ids, the one configuration where the defect cannot appear. New test uses a 3000-item time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200 so the guard skips), limit=25 does. Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING entity id, and I had made newest-first coincide with it, so any mutation that merely TIED the set still produced the asserted vector. The gate mutation proves it: reverting `needs_metadata_for_sort` to its old four-variant form yielded `left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`, which is the exact drift this work exists to prevent. Every ordering fixture is now non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and ascending id (the tie collapse). CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them carry a diversity block so they already loaded item metadata and change behaviour the instant this image rolls. Replaced with a verified table. Also cited the spec conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified `Sort::New` as "created_at DESC". created_at units are now trusted with a warning instead of silently. A seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores -496731 against boost sums in single digits -- re-entering through DATA the exact "recency annihilates every boost" defect this work removed -- and Hot buries the item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was inert; this work made it live. The repo's OWN fixtures made that mistake in three places, which is the proof it is the natural one. `metadata_with_created_at` now warns when the value is too small to be nanoseconds, logging entity id, value, age_hours and age_years so an operator can act. It does NOT rewrite the value -- guessing the unit would corrupt what the `created_at` range index already reads as nanoseconds -- and does NOT reject the write, which would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server: seconds, millis and micros each warn with age_years=56; a real nanosecond value is silent. The nanosecond contract is now documented on the public DTO and propagates to the OpenAPI schema, where it was invisible before. Stale docs corrected: two published blog posts and two claim-verification ledgers were certifying "entity recency (higher ID = newer)"; the e2e fixture contract justified an interleaving decision with reasoning that is now false (its conclusion still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:" rationale cited behaviour this work removed, so it is now marked PENDING RE-MEASUREMENT with the three specific measurements named rather than left reading as justified. Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors. Fast integration suites all green. Real-server e2e re-verified: `new` and `hot` both return newest-first, `new`'s scores now evenly spaced across evenly spaced ages.	2026-08-31 21:31:31 -06:00
jordan	a6f663f002	harden: validate embeddings before the WAL, fix the reseed-latch leak, run every test suite ... Fixes the two defects a malformed probe exposed on the live cluster, plus the coverage gap that let a stale assertion survive the same day it was falsified. TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim slot was appended to the WAL FIRST, then validated, then answered 500 — so an already-durable, unapplicable record shipped to both followers, halted both receivers, and put shard 1 into a quorum-write outage. Validation now runs before the append and returns 400 via invalid_input; nothing enters the log. `storage::vector::validate_dimensions` is now the single comparison, replacing an inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a dimension check drift, and the apply-path copy is the one that halts replication when it disagrees. The receiver's halt-vs-skip decision is now explicit instead of "halt on anything". A record whose failure is deterministic and node-independent (schema width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the frontier advances; a record that could become applicable after a binary upgrade (unknown batch kind, capability skew) still halts, because skipping那 would silently drop replicated data. Both branches are proven reachable by tests. TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups latched a marker per group but discharged on a single seqno, so two latches meant permanent 503 on a node whose every shard read lag 0 — it hit all three pods during the roll and each needed a manual delete. Gaps are now tracked per group in a ReseedGapSet and cleared on evidence about themselves; a REFUSED reseed_self_restart re-evaluates every 15s instead of waiting for a latch that never arrives. /health's cause ladder was also lying: it printed "joiner boot not yet converged" for a node whose groups had all converged, because the fallback asserted a state it never tested. It now names the outstanding gaps, gained the decommissioned-by-signal arm that is_ready checked but the ladder did not, and its terminal arm says "reason unavailable" rather than inventing one. COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical: cluster_routes still asserted the wire fabrication removed hours earlier (applied_events == 0 with a lag derived from it) and nothing caught it because nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search (distance contract) and cluster_poison_embedding (task 17's own gate) were in the same position, so those guards would have rotted identically. Every suite now has a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing before being scheduled. Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13 minutes in — it demanded an unreachable peer report worst-case lag, i.e. it required the fabrication task 04a deleted. Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on touched trees at 431340f); lib 2115 passed; all 8 fast suites green; cluster_chaos 5, cluster_sharding 5, cluster_poison_embedding 1, cluster_cross_shard_reads 2, cluster_graph_persistence 1, cluster_multiproc 5, cluster_e2e 2; doc-guard OK.	2026-08-31 00:46:00 -06:00
jordan	fe8d0c87e7	harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path ... Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.	2026-08-30 20:55:58 -06:00
jordan	4766f566de	feat(observability): HTTP metrics, structured logs, dashboard, live tidalctl ... There was no metric anywhere that could answer "how much traffic are we serving" or "what is our error rate". The engine published a rich DOMAIN surface (search latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so a cluster could serve 401s or 503s indefinitely with every existing gauge looking healthy. Logs were collected but unusable. There was no way to ask a RUNNING node anything. 1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a per-route duration histogram, recorded by one layer placed OUTSIDE the auth, timeout and rate-limit layers so it sees the status actually returned to the client. Cardinality is the whole design: the route label is axum's MatchedPath TEMPLATE, not the path, and unmatched requests collapse into one <unmatched> bucket so a 404 flood cannot mint series. A hard cap folds anything past it into an overflow bucket while established series keep counting. The engine owns the /metrics listener but must not learn what a route or a status code is, so it gained one registration hook (MetricsState::set_extra_renderer) and tidal-server publishes through it. One scrape target per node, not two. 2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which produced two real defects: ANSI escapes leaked into collected logs, and every line failed the collector's JSON parse and was stamped level=info — so `level:error` matched NOTHING and errors were invisible to the log platform while being collected. JSON_LOGS=1 emits the collector's exact wire format (ts/level/service/env/msg), span fields are lifted so request_id lands on every line of a request, and ANSI is off unconditionally in both formats. Verified against the running binary, which caught a defect no unit test would have: dependencies logging through the `log` crate arrived with target="log" and four log.* metadata fields (absolute cargo registry paths, indexed forever). The real module is now lifted into target and the bridge metadata pruned. 3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet as a grafana-database-dashboards key. Every metric name was checked against a live endpoint and all 26 PromQL expressions were executed against the live TSDB before commit, because a dashboard full of "No data" is worse than none. Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder). 4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take --url and talk to a running server, with --ca/--insecure because a cluster's client port is served with the INTERNAL cluster CA. Exit codes follow the crate contract, so `tidalctl cluster-status && deploy` gates on convergence. Its first real run immediately found a reporting defect: the aggregated /cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M lag, having derived lag against an uninitialised applied=0, while every node's own status reported lag=0, reseed=false and identical frontiers, with pod-to-pod connectivity open and nothing logged. cluster-status now names that signature "NO REPORT (aggregated view; query the node directly)" instead of repeating it as replication lag; a genuine non-zero-applied lag still reports BEHIND. The underlying gap is documented as open work in docs/ops/observability.md. Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new, including the cardinality proof and the cross-crate metrics seam), 23 tidalctl (10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the metrics and the log format exercised against a real running binary.	2026-08-23 10:31:57 -06:00
jordan	388e445a38	feat(cluster): separate operator authority from data-plane access ... Every destructive /cluster/* verb sat behind the SAME bearer as /items and /search, so any application key could remove a member, force a partition, or transfer a shard. There was no way to hand out a client credential without also handing out the ability to destroy the cluster. Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like the others). /cluster/promote, /cluster/partition, /cluster/heal, /cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas, transfer} move into their own router subtree behind an admin gate; the data bearer now gets 403 there - authenticated but not authorized, distinct from the 401 for a bad token. Three things this had to get right: * The admin key must ALSO authenticate. A request carries one Authorization header, so if the admin key did not satisfy the bearer gate, an operator presenting it would be 401'd before the admin gate ran and the verbs would be reachable by nobody. Caught while writing the test, not after. * A verified sibling node token clears the gate too. Nodes relay operator verbs to the leader/target carrying whatever credential the caller sent, and the legacy fan-out promote uses the internal marker, so requiring the admin key on that hop would partition the control plane. * The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal nudge), /cluster/join + /cluster/members (seed-join) and the /cluster/reconcile* pair are dialled node-to-node, so gating them would break replication and joining. Absent admin key = previous behavior exactly, plus a startup WARN naming the exposure, so this is safe to upgrade into. The k8s secret mount is optional:true because without that a deployment lacking the key would fail to MOUNT and never start. Also closes the /cluster/status hole this exposed: it and /cluster/status/local reported leader identity, membership, term and per-shard applied/lag/commit seqnos from the UNAUTHENTICATED probe group. They are protected now, which is what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the network layer. And fixes a latent bug found on the way: seed-join discovery, reseed discovery and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly, which yields nothing on a *_FILE-only deployment - the node would dial an authenticated peer with no credential. They use security::bearer_from_env() now, which honours both shapes. Verified: 5 new unit tests; two multi-process runbook tests on real 3-process clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status and through the gate on heal; bare /cluster/status 401, 200 with the bearer). That the authenticated cluster converges at all is the load-bearing assertion - if moving status behind auth had broken leader discovery, startup would hang. Full unit suites green (2101 + 162), reseed e2e green, clippy clean.	2026-08-22 00:57:01 -06:00
jordan	5b3cfe59d9	fix(cluster): never boot-pull against the topology leader post-election ... This is the defect that kept tidaldb-0 looping, and the per-key instrument named it exactly. Live group 1 held: keys: [[0, 13540659], [1, 13540652], [2, 13540661]] Current leader tidaldb-2 is key 2 and the group was fully converged there at 13540661. Key 1 is a STALE position left from when tidaldb-1 led the group. `node.rs`'s follower boot self-heal pulled `shard_of_region(leader)` where `leader` is the BOOT TOPOLOGY leader — dead config after any election, as the topology comment itself says. For group 1 that is tidaldb-1, i.e. key 1, so the pull went out at 13540652 + 1 = 13540653, which tidaldb-1's WAL had compacted below (earliest 13540657). Permanent `snapshot-required` → marker latch → `reseed_self_restart` → repeat. The old comment claimed "term fencing + later election traffic rescue it"; they do not, because the refusal re-latches faster than the rescue converges. The boot pull is now confined to the genuine topology era (durable term 0), where the topology leader IS authoritative. Post-election, convergence is driven by the heartbeat path (which carries the CURRENT leader's frontier and works on an idle cluster since m12p5) and by the receiver's gap detection on real ship traffic — both keyed to the leader actually shipping, never a historical one. Gates: mp_follower_reseeds_via_snapshot_after_compaction and mp_multi_group_node_converges_after_reseeding_several_groups both pass; mp_quarantined_node_reseeds_without_wipe still passes, which is the term-0 path this change deliberately leaves intact.	2026-08-21 11:35:31 -06:00
jordan	925a616cda	feat(cluster): expose every tracked replication stream key's position ... The remaining reseed defect cannot be diagnosed from the current status surface. A stream key is a per-LEADER-REGION id (`shard_of_region`), not a shard group, so a group accumulates one key per leadership it has followed — but every status field reports only the CURRENT leader's key. A position retained from a previous leadership is therefore invisible, while the receiver's gap check (`receiver.rs`: `request_catchup(key, applied + 1)`) will chase ANY key that received data this round. That is the blind spot: live tidaldb-0 pulls `from_seqno=13540653`, so some key sits at 13540652, while the group it reports on converged at 13540661 — and nothing in the status can say which key that is. Adds `ReplicationState::applied_by_key` and surfaces it as `applied_by_key` on the status response. Instrument only: no behavioural change. Same instrument-first move that turned the previous two defects into one-run diagnoses instead of speculation.	2026-08-21 11:01:46 -06:00
jordan	7450cc7ef1	fix(cluster): readiness must prove convergence, not merely lack a marker ... Closes the multi-group reseed defect. `is_ready` gated convergence behind `install_boot || seed_joiner`, so a plain restarted voter fell straight through to ready — admitted to the client VIP before it had learned the leader's frontier, let alone caught up. The doc comment called that intentional ("keeps today's behavior"). It is the same anti-pattern as the marker-discharge bug: asserting health from ABSENCE of bad news. `lag_events` could not contradict it. Lag is `leader_seqno - applied`, an unsigned subtraction against a gauge that reads 0 until the frontier is known, so a node that has learned nothing computes 0 - 0 = 0 and looks perfectly caught up. Both halves together are how a PVC-wiped tidaldb-0 entered the VIP with an EMPTY corpus and how the repro node reported all groups clean while missing items: shard 0: applied_events 24, lag_events 0 shard 1: applied_events 14, lag_events 0 shard 2: applied_events 0, lag_events 0, leader null after 5600 items were written. Now: convergence is required for EVERY boot, `note_lag_for_readiness` takes the leader frontier and refuses to latch on a zero (no information is not convergence), and it is driven on every boot rather than only joiner boots — the heartbeat carries the frontier, so this works on an idle cluster (m12p5). `reseeding` becomes `!converged` for all boots, which also makes the status field mean what it says. Only ESTABLISHED leadership self-certifies. The first cut tested `current_leader()`, which is seeded from the TOPOLOGY FILE — and in a sharded topology group `s` names node `s` as its term-0 leader, so a booting node self-certified convergence for a group it merely believed it led while holding none of its data. The election-runtime role is the honest source; the durable §1.4-1 rule is that a restart always boots a follower. The leader arm stays load-bearing for bootstrap: a fresh cluster's leader has `last_seq == 0` and would otherwise be permanently 503. Gate: mp_multi_group_node_converges_after_reseeding_several_groups now PASSES and is un-ignored. All three groups converge against real frontiers (applied 3797/3726/ 3747 == leader_seqno, terms 1/5/3) and every probed item is readable, in 2 restarts of a ceiling of 5. mp_follower_reseeds_via_snapshot_after_compaction and mp_quarantined_node_reseeds_without_wipe still pass, so bootstrap and the quarantine reseed are unaffected.	2026-08-21 10:41:38 -06:00
jordan	54d1353103	fix(cluster): make a multi-group node's status readable ... Two status defects turned this incident into a day of misreading. Both are observability, both are why the functional bug survived, and neither changes readiness or replication behaviour. 1. PER-GROUP RESEED STATE. `reseed_required` / `reseeding` existed only as flat fields on LocalStatusResponse, and `status_local` fills those from `replica_for(sel.shard_id())` — the LOWEST hosted group id when no `?shard=` is given. On a 3-group node they therefore describe one group and say nothing about the other two. tidaldb-0 answered `reseed_required: false` while a different hosted group sat behind a compacted leader, and every operator reading and every diagnosis in this incident took that as converged. ShardStatusRow now carries both per group. 2. `lag_events: 0` WAS UNREADABLE. Lag is `leader_seqno_for(key) - applied`, an unsigned subtraction against a gauge that is 0 until this node learns the leader's frontier. A freshly-booted node that knows NOTHING computes 0 - 0 = 0 and reports itself perfectly caught up. Measured in the multi-group repro at the moment the node declared itself settled: shard 0: applied_events 24, lag_events 0, leader us-east shard 1: applied_events 14, lag_events 0, leader eu-west shard 2: applied_events 0, lag_events 0, leader null 5600 items had been written. All three groups claimed zero lag. Expose `leader_seqno` (the value lag subtracts from) on both the flat response and each shard row, so `lag_events: 0` with `leader_seqno: 0` reads as NO INFORMATION rather than converged. This is additive: `lag_events` keeps its value and readiness keeps its semantics, deliberately, because changing the readiness predicate during a live incident is not a change worth bundling here. Also tightens the multi-group repro's settle predicate to require EVERY hosted group's row to be clean. The first version trusted the flat fields, so it announced "settled after 0 restarts" and then failed the content probe — fooled by exactly the under-reporting above.	2026-08-21 03:01:58 -06:00
jordan	afdda7cc0f	fix(cluster): discharge a reseed marker on served evidence, never on a frontier ... da736b8 replaced `applied >= leader_last_seq` with `applied >= marker.from_seqno` and was still wrong, for the same underlying reason: the applied frontier is a HIGH-WATER-MARK, not a contiguity proof. A term join re-bases it onto the new leader's stream (`replication_state().advance(.., baseline + 1)`), so it leaps across history the node never received. Any predicate built on it discharges markers for nodes that still have a hole. Measured, not argued. `mp_follower_reseeds_via_snapshot_after_compaction` stops a follower at frontier 9, compacts the leader so it retains only from 15722, and the follower's frontier is re-based to 16810. Both predicates discharge the marker there; the node skips its reseed and then reports `lag_events: 0` while missing 10..15721 and serving reads from a log with a hole. Production showed the identical shape: `applied 13540660` against a marker resuming at 13540653 that no live WAL could serve. The marker is now discharged only on POSITIVE EVIDENCE that the stream served the latching range: a `StreamSegments` pull that began at or below the marker's `from_seqno` and ran to completion. New `CatchupServedSink` in tidal-net fires on `PullOutcome::Complete`; `NodeCatchupServedSink` routes it to `discharge_reseed_marker_if_served`. `ReseedMarker::discharged_by_served_range` replaces `discharged_by`. The other sound discharge is unchanged: a snapshot install replaces the data dir and takes the marker with it. Both frontier-based call sites are gone, with the reasoning recorded where they were. The election-won site is deliberately NOT replaced: winning proves the log beats a quorum's under the vote restriction, which is not contiguity, so discharging there could promote a leader with a hole. The owner-test for this mechanism was RED ON BASELINE and is now green. It also gained the premise assertion it never had: it used to assert only the consequence (`reseed_required == true`), so when its fixture stopped forcing compaction it failed 40s later looking like a follower bug. `assert_history_compacted_past` now checks the leader actually dropped the follower's resume seq, and prints the retained segment floors. Its content probe ("every probed offline item is searchable on the reseeded follower") is what proves the hole is really gone. Suite state: cluster_reseed's other tests pass individually. mp_graceful_rolling_restart_under_load_no_reseed remains red on baseline (pre-existing, verified by stash). mp_quarantined_node_reseeds_without_wipe and mp_election_position_consistent_across_roles_after_failover pass alone but can fail in-suite: this fix makes the compaction test run its full 565s reseed instead of failing fast at 40s, which shifts timing for later tests on shared fixed ports. Order sensitivity is pre-existing, not introduced here.	2026-08-21 00:40:06 -06:00
jordan	da736b8eb2	fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail ... A follower that latched `reseed_required` from a genuine `snapshot-required` refusal could clear its own marker ~200ms later and so never run the boot reseed that was the only way to close the gap. With `replication.reseed_self_restart: true` it exit-looped: latch -> clear -> exit(0) -> boot with no marker -> re-latch. Production tidaldb-0 did this 196 times in 21h on 2026-08-20 while the cluster ran on 2 of 3 voters. `clear_stale_reseed_marker_if_caught_up(applied >= leader_last_seq)` compared the applied frontier against the LEADER'S TAIL and documented the invariant "a node genuinely behind a COMPACTED gap never reaches caught_up". That is false: on a quiet shard any node meets the leader's tail, including one missing committed history it can never refetch. The clear also reset `tidaldb_cluster_reseed_required`, so the gauge flapped 1->0 every 30s and `TidalDBClusterReseedPending` (`== 1 for 10m`) could never fire - the code path that broke the reseed also erased the signal that would have reported it. The discharge decision now belongs to the marker. `ReseedMarker::discharged_by` requires a stream-dischargeable reason AND an applied frontier that reached the marker's own `from_seqno` - the very entry whose absence latched it. A compacted gap can never satisfy that, so the reseed runs; a node merely behind a shippable tail satisfies it as soon as the stream serves that entry, so the m12 false-alarm self-heal still works (and now clears sooner, since it no longer waits to meet a moving leader tail). The two conditions previously shared `ReseedReason::SnapshotRequired`, so reason alone could not discriminate. The term-join arm's `frontier > baseline` case deliberately sets `from_seqno = baseline`, BELOW the node's own frontier, so a bare `applied >= from_seqno` would discharge it instantly - it holds divergent post-baseline data only a snapshot can discard. It gets its own never-lag- dischargeable reason, `DivergentPostBaseline = 3`. Adding a discriminant is the sanctioned forward-only extension; a downgrade that meets one refuses to decode it, per the existing kind-3/kind-4 precedent. The election-won call site passed a hardcoded `true`; it now passes the leader's durable flushed frontier (`applied_seqno` never advances on a leader), and a `DivergentPostBaseline` node is not campaign-suppressed so it can reach there. Tests: three deterministic predicate tests pinning the incident's exact seqnos (13540653 vs earliest-available 13540657), the false-alarm discharge, and the never-discharge of every structural reason. Pre-existing and NOT introduced here: cluster_reseed's `mp_follower_reseeds_via_snapshot_after_compaction` and `mp_graceful_rolling_restart_under_load_no_reseed` fail on baseline main (verified by stashing this change). The first is the owner-test for this exact mechanism - its leader compaction no longer forces a `snapshot-required`, so it never reached the clear path and never guarded it. Tracked separately.	2026-08-20 22:23:35 -06:00
jordan	2e1484226c	fix(cluster): reconcile could not run at production scale ... The three live voters disagree on signal aggregates for the same entity (view = 10003 / 10095 / 10144 for entity 1, stable across passes) while `/cluster/status` reports applied_events equal, lag_events 0, and no divergence quarantine. The documented remedy is `POST /cluster/reconcile`. On this corpus it fails: 503 region 'tidaldb-1' unreachable: reconcile peer returned 413 Payload Too Large Two defects, both fixed here: - The whole-shard CRDT `StateSnapshot` was capped by `BODY_LIMIT_BYTES`, the 2 MiB limit sized for one client write on the public data surface. The snapshot carries one entry per entity x signal type; on 33k documents it is several MiB, so divergence was unhealable in production. The internal, marker-pinned, operator-driven snapshot route now has its own explicit ceiling. - A 413 was reported as `RegionUnreachable`. The peer answered - it is reachable and healthy - so the error sent the operator to TLS and NetworkPolicy. It now names the measured snapshot size, the peer's cap, and the fix. The ceiling is not the design: the snapshot grows with the corpus and chunked reconcile is the durable answer. Documented as such at the constant.	2026-08-18 10:07:19 -06:00
jordan	fc1cc901a1	fix(cluster): refresh the frontier gauges on the driver tick ... Publishing the pair only from a satisfied `ack=quorum` wait left both halves at 0 for the entire life of an `ack=leader` workload: the live three-voter cluster served writes at 201/204 and reported `commit_index: 13324714` through `/cluster/status` while `/metrics` showed relay_last_seq 0, relay_durable_seq 0. A gauge nobody can populate is the same blind spot as a gauge that lies. The election driver already ticks every replica ~20x/s and m12p5 seeds the readiness lag gauge from the heartbeat for exactly this reason - the tick flows even when writes do not. Two relaxed stores per tick behind a commit-index lock the quorum waiters already share.	2026-08-18 09:57:32 -06:00
jordan	12c7edc374	fix(cluster): the frontier pair has one writer, and 0 is not a commit index ... `TidalDBClusterQuorumLag` sat CRITICAL all session against the live three-voter cluster while every region reported lag 0 and every per-peer ship queue was empty. Two independent defects fed it: - `observe_ship` bumped `relay_last_seq` on every batch ship while `relay_durable_seq` only moved when a signal write completed. The two are documented as a subtractable pair, so a shipping-but-not-committing node reported the whole relay log (13.3M events) as quorum lag. The ship path now feeds only its own per-peer queue-depth gauge; the pair has one writer. - `set_frontier_gauges` published `CommitIndex::committed()` verbatim, but that returns 0 as a SENTINEL for "no quorum information in this term yet". It now publishes both halves or neither, and every satisfied `await_quorum` -- not just signal writes -- refreshes them, so item and embedding workloads keep the pair live. Regression test asserts a busy ship loop leaves both halves at 0 (lag 0, not 13.3M) and that the single writer still moves them together. Also excludes the `tmp` emptyDir from velero fs-backup: three 0-byte PodVolumeBackups a night whose only other outcome is failing the whole fleet backup when a scratch file vanishes mid-snapshot.	2026-08-18 09:50:19 -06:00
jordan	897c6086f5	fix(cluster): size the read fan-out budget for the transport it crosses ... Restoring the three-node cluster for a first production consumer surfaced this immediately: EVERY cross-shard read came back {"items":[...],"scatter_gather":{"degraded":true, "unavailable_shards":["tidaldb-0","tidaldb-2"],"shards_queried":1, "elapsed_ms":50,"shard_deadline_ms":45}} HTTP 200, one shard of three, partial results. Replication itself was healthy - /cluster/status showed all three regions reachable, lag_events 0, 13.3M events applied each - so nothing in the quorum, election, or ship metrics moved. Measured on the live cluster: a COLD peer fetch (TCP + TLS handshake + remote 1536-D search) takes ~50ms; a warm one takes ~1ms. DEFAULT_DEADLINE_MS is 50 (spec §7.4) and NETWORK_OVERHEAD_MS is 5, leaving a 45ms per-shard budget - just under the cold cost. Proven by parameter sweep against one pod: deadline_ms=50 -> degraded, 1/3 shards, 0 items deadline_ms=250 -> healthy, 3/3 shards, elapsed 51ms deadline_ms=1000 -> healthy, 3/3 shards, elapsed 1ms (warm) The 50ms spec figure budgets a shard READ, not establishing a connection to another pod. m11p7 put TLS on that hop and the default never followed, so the first query after any rollout, idle period, or pod restart answered from a third of the corpus. Fixed with a transport-aware default: 50ms in-process, TLS_DEFAULT_DEADLINE_MS (250ms) once inter-node TLS is configured. An explicit `?deadline_ms=` still wins in both directions, and MAX_DEADLINE_MS is unchanged. The worse half was silence. A degraded fan-out is the one cluster failure that answers 200 OK: the caller gets a ranked list assembled from a subset of the corpus with `degraded: true` buried in response metadata. Nothing incremented, so no alert could exist - a feed quietly ranking over one third of its candidates looked identical to a healthy one. Added tidaldb_cluster_scatter_degraded_total and tidaldb_cluster_scatter_shard_unavailable_total, emitted from both HTTP fan-out paths, so partial answers are now a countable correctness signal. Also sizes the cluster StatefulSet for a consumer instead of the endurance gate: requests 2 cores -> 300m per voter (limit 2 cores). The 2-core reservation was the 200 rps soak envelope and needed 6,000m plus 2,000m free on each of three PV-pinned nodes; the fleet is 82-91% committed, so that contract could not be placed and the cluster stayed parked for a gate nobody is waiting on. 300m is what the tightest pinned node can reserve, with the quorum/write-pool alerts as the detector if real load outgrows it. Tests: default_read_budget_covers_a_cold_inter_node_tls_hop pins the budget against the measured cold hop and the explicit-override path; the cluster-metrics render test covers both new counters.	2026-08-17 20:28:21 -06:00
jordan	c97aaa8e5b	fleet remediation: make the workspace gate runnable, then fix what it caught ... `cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope.	2026-08-16 12:38:14 -06:00
jordan	cdbe9cb453	Merge remote-tracking branch 'origin/main' (m11/m12 cluster) into m9/m10 ... Reconciles two independently-developed lines from base 006d3d0: ours — M9/M10 community layers, retroactive purge + re-materialization, signal revocation, agent capability boundaries, P1 feedback loop, reason labels, instrumented metrics theirs — M11/M12 cluster mode (tidal-net gRPC transport, tidal-server cluster/scatter-gather, tidal-stress), multi-vector preference, ANN candidate-gen, warm-tier day buckets, keyed signal snapshots Notable semantic resolutions: * storage::keys::Tag — both sides allocated 0x0E..0x11 for different records. Kept theirs' 0x0E..0x1A (shipped on-disk format) and renumbered ours to 0x1B..0x1E (CommunityMembership/Revocation/PurgeManifest/ CommunityLeave); Tag::ALL grown to 30 so the contiguity drift guard holds. * ranking executor — took theirs' rewrite (SignalReadPlan pre-pass, keyed SignalKey snapshots, Result-returning reads, finalize()) and re-applied ours' M10 read suppression at the chokepoints it introduced: single_signal_score, score_hot/trending/controversial, CreatorEngagementRate, and the Stage-4 boost loop. * signals::warm — theirs' day-bucket/read-time-rotation rewrite, with ours' subtract_bucket and Clone extended to the new day tier; ours' test split kept (warm/tests.rs, warm/proptests.rs) carrying theirs' updated bodies. * db::signals — kept ours' contribution-logging try_cohort_attribution in signal_dispatch.rs and theirs' event-time try_update_preference_vector; dropped the superseded duplicates. * db::mod / from_parts — theirs' constructors, with ours' purge/ re-materialization/revocation/community/skip-counter fields and restart rebuilds; from_parts kept in its own file per the 600-line guideline. * schema::validation::builders — ours' module split with theirs' expanded tests; policy validation runs both sides' checks (read-signal lists + profile overrides, then the zero-duration limit guard). * feedback Unhide no longer writes a -1.0 "hide" signal: theirs' engine rejects negative weights (spec §8). Reverses index state only, matching every other undo action. * SessionState::new is now the single construction path (gains overrides_rejected/default_profile); AuditEntry gains kind on the deserialize path, inferred from the accepted flag as before. * Removed tidal/src/replication/tcp_transport.rs and its test: never declared in replication/mod.rs on either branch, so it had never compiled and nothing referenced it. Superseded by tidal-net's GrpcTransport. Verified: cargo clippy -p tidaldb (lib) clean; --all-targets compiles for tidaldb/tidal-net/tidal-server/tidal-stress; 2094/2094 lib tests and the integration suite pass except m8p3_reconcile_production's two CRDT-count assertions, which fail identically on MERGE_HEAD (pre-existing). tidalctl cannot build locally: its aws-sdk deps need rustc 1.91.1, local toolchain is 1.91.0.	2026-08-03 02:16:04 -06:00
jx12n	c22a3b65a6	docs: withdraw the pre-release "not ready for production" disclaimer ... M0-M12 are shipped and the HA cluster runs in production on k3s, so the pre-release disclaimer no longer describes the project. Removes it from the canonical doc set and corrects the readiness text that had gone stale. - README.md: replace the "Pre-release / not yet recommended for production" banner with a production-ready statement; drop "(experimental)" from the cluster status bullet; state the post-1.0 versioning posture (additive in minor releases, breaking changes get a documented migration path). - CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md / docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a guard against standing up a multi-node fabric by accident rather than a readiness warning. - CHANGELOG.md: record the stability posture under [Unreleased], superseding the historical 0.1.0 "no stability guarantees" note (left intact as history). - k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead. Also corrects text that was factually wrong since m11p3/m11p4: the multi-process cluster gate, its CLI help, and the served OpenAPI description all still claimed quorum-ack writes and automatic failure detection did not exist. They do. Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries, the kubernetes.md rc7 fix note) are left unchanged. Verified against a running binary, not just the build: the opt-in gate's refusal message, the startup WARN, /health 200, and the served /openapi.json description all carry the new text. cargo fmt clean; clippy -D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155 server lib tests pass; scripts/check-docs.sh OK. Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP	2026-07-30 19:03:34 -06:00
jx12n	6a937fc4bc	feat(m12): multi-vector user preference modeling + ANN candidate-gen ... Add multi-vector preference entity (per-signal-type preference vectors with event-time decay) feeding ANN candidate generation in the query executor. - entities: multi_preference vectors + event-time-aware preference updates - query/executor: ANN candidate-gen + personalization/pipeline integration - storage/keys, db ops, state_rebuild: persist & rebuild multi-vector prefs - ranking: profile + builtins support for multi-vector scoring - tidal-server/config: expose multi-preference knobs - tests/bench: m12_preference_event_time integration + multi_preference bench - docs: multi-vector-preference research, ROADMAP/ARCHITECTURE refresh, legal/tidaldb-patent-proposal - .codex/agents: codex agent definitions - chore: gitignore tool-regenerated .agents/ mirror (doc-guard rejects it)	2026-06-23 09:52:36 -06:00
jx12n	0919b0a4bf	fix(m12): seed-join learner auto-promotes after a snapshot install (report the caught-up frontier on the heartbeat) ... Fixes a regression the reseed-loop fix (c8ea05b "Fix 1") introduced: a node that seed-joins and converges via a SNAPSHOT INSTALL never auto-promotes Learner -> Voter. It catches up fully (applied == leader frontier, lag 0) but sits a Learner forever; `mp_seed_join_snapshot_catchup` caught it (base 580142d passes, c8ea05b on fails — bisected). Root cause: the leader's durable per-peer `learner_mark` (which the auto-promotion gate reads: `flushed - learner_mark <= learner_promote_lag`) advances ONLY from a follower frontier-report, which the receiver emits AFTER applying a streamed event. Before Fix 1 a joiner seeded its frontier from `last_wal_seq()` (0 on the empty WAL a checkpoint restore leaves), so it re-pulled the whole log from seqno 1 and THOSE stream applies emitted the reports that advanced `learner_mark`. Fix 1 seeds the frontier to `snapshot_seq` to stop the prod reseed loop, so the leader has nothing to ship, no stream applies, and the joiner never tells the leader it is caught up. Fix: in the m12p5 heartbeat idle-readiness drive (`note_leader_frontier_for_readiness`) report this node's caught-up frontier back to the leader via the existing `ReportApplied` channel. The heartbeat flows on an idle cluster and carries the current term, so the report is both recurring (survives the join/registration race) and term-correct (the term-checked `update_peer_for_term` fold accepts it — a boot-time report stamped term 0 does not). SCOPED to a Learner: a Voter's frontier already reaches the leader via ship-acks and DOES feed `compute_commit`, so folding one off the heartbeat could perturb the same-term commit gate (Raft fig-8); a learner mark never feeds `compute_commit`, so this is provably commit-safe. `notify_applied` dedups, so a steady follower never spams. Verified: mp_seed_join_snapshot_catchup PASS (joiner promotes, 4-voter quorum); safety preserved — mp_quarantined (divergent quarantine+reseed, no wipe), mp_graceful_rolling_restart_under_load_no_reseed (0-reseed), the failover oracle (no false quarantine), tidal-server lib 154/154, engine durability 4/4, clippy -D clean. The 5600-item voter-reseed e2e are inert to this Learner-scoped change (host RAM cannot run them locally; verified live).	2026-06-19 01:46:46 -06:00
jx12n	973f073558	fix(m12): abort the futile self-restart when a snapshot_required marker self-heals via catch-up ... The rc4 live deploy converged tidaldb-2 but through ~4 needless self-restarts: a caught-up shard whose persisted frontier is briefly behind the leader's ADVANCED baseline (the leader kept writing while the node was down) latches a snapshot_required marker on the first heartbeat's decide_join, which ARMS a self-restart. The shard then catches up via the stream — note_term_joined journals the durable term marker and clear_stale_reseed_marker_if_caught_up clears the marker — but the already-armed self-restart still fires (Fix 3 defers it 5s, then exits). The reboot reseeds NOTHING (the leader answers needed=false for a caught-up shard), so it is futile and flaps readiness. Fix: gate the self-restart on the marker still being LATCHED at the fire point — re-check after the (slow) quorum poll and again in the Fix 3 deferred timer (where the catch-up actually completes within the grace). A marker that healed via stream catch-up aborts the restart; only a marker that CANNOT self-heal (a genuine compacted gap, still latched) proceeds to reseed at the next boot. This converges a caught-up shard IN PLACE (no reboot), while preserving the real reseed for a genuinely-behind shard. Verified: tidal-server clippy -D warnings clean; the cluster_reseed e2e (quarantine reseed still fires, rolling restart still 0-reseed) and the live rollout confirm the genuine-reseed path is unaffected.	2026-06-18 21:34:47 -06:00
jx12n	fe56d1bf3e	fix(m12): cover the caught-up-empty-WAL reseed loop (decide_join frontier arm + durable term-marker repair on clean join) ... The live rc3 deploy on tidaldb-2 (which hosts all 3 shards) revealed a second loop the install-only term-marker synthesis (prior commit) does not reach: a shard that was reseeded in an earlier loop iteration has an EMPTY WAL (tail_term=0) but a frontier that already COVERS the leader's baseline. On the next boot the leader answers needed=false (its WAL covers the frontier), so the shard never installs, never gets a synthesized term marker, and decide_join — comparing (tail_term, frontier) with tail_term first — classifies it ReseedRequired forever. tidaldb-2 went 35 CrashLoops -> shard 2 converged via Fix 2 but shards 0/1 kept looping (restarts still climbing, ready=false). Two complementary fixes: - decide_join: in the `own < prev_log` arm, a frontier at/above the leader's baseline is CAUGHT UP (it holds every committed entry, only the term-marker record is missing) -> Clean, not a futile reseed. A frontier short of the baseline is genuinely behind -> ReseedRequired. Divergence is unaffected: a future-term marker and an un-replicated leader-acked suffix both quarantine BEFORE this arm, so the frontier-covers-baseline Clean never reaches a divergent node (verified: mp_quarantined still quarantines). - note_term_joined: when the WAL-tail term is stale on a clean join, durably append the kind-3 term marker (guarded, so once per stale term — never a WAL write per heartbeat), generalizing the install-boot synthesis to caught-up shards that never install. Falls back to the in-memory fold on append failure (decide_join's frontier arm keeps the node Clean regardless). Tests: decide_join now asserts caught-up->Clean, behind->ReseedRequired, divergent->Quarantine. Verified: cluster_reseed 4/4 (rolling restart 0-reseed x2, quarantine reseed, failover oracle — no false quarantine), tidal-server lib 154/154, engine durability 4/4.	2026-06-18 21:24:42 -06:00
jx12n	c8ea05b032	fix(m12): break the post-reseed false-ReseedRequired loop (durable term marker + readiness gating + restart coordinator) ... Root cause: after a checkpoint-based snapshot install the engine WAL is empty, so wal_term_mark() reports tail_term=0. decide_join compares (tail_term, frontier) lexicographically — tail_term FIRST — so 0 < leader_term classifies the reseeded shard ReseedRequired on EVERY boot regardless of the correctly-seeded frontier, re-latching the marker and self-restarting forever. Observed live on tidaldb-2: 30 CrashLoopBackOff restarts, leader tidaldb-1 term 5, baseline=536647, the frontier seeded correctly (from_seqno=536647) yet the loop persists because the (tail_term, frontier) compare never reaches the frontier. Fix 1 (already in tree): seed the post-open frontier from sentinel.snapshot_seq, not last_wal_seq() (which a checkpoint restore leaves at 0). Fix 2 (loop-breaker): durably synthesize the artifact's kind-3 TERM_MARKER WAL record in the post-open reseed seed, at the artifact's captured term + the reseed-leader region (threaded through an extended 18-byte install sentinel, back-compat with 10/8-byte). Makes wal_term_mark() truthful on this boot AND every reboot (blob records are NOT checkpoint-filtered on recovery), so decide_join returns Clean. Truthful, not a bypass: the artifact IS the leader's authoritative state at (term, seq); a genuinely-divergent node (no install sentinel) still surfaces tail_term > term -> Quarantine. Crash-idempotent via a monotonic-by-term guard. Fix 3: node-level reseed-restart coordinator — the single process-wide exit fires once, only after every hosted shard requests a restart or a bounded grace elapses, so one shard's self-restart never aborts a co-hosted sibling's in-flight install (S>1). No-op on the S=1 production topology. Fix 4: is_ready() returns 503 while any reseed marker (SnapshotRequired or Quarantine) is latched, closing the plain-restart serve-while-behind gap; readiness is bounded staleness, not "ready the instant the process is up". Tests: decide_join loop/fix/bounded-reseed unit; install-sentinel 18-byte round-trip + back-compat; engine durability (term marker survives a checkpoint advanced past it + crash-reopen); reseed-restart gate (5 cases); reseed_install carries the term. Verified: cluster_reseed 4/4 (zero-loss rolling restart x2, quarantine reseed, failover oracle), reseed_install 3/3, m12_reseed_term_marker 4/4, cluster_membership mp_idle/mp_dns/mp_remove x2, tidal-server lib 154/154. mp_scale_3_5_3 and mp_seed_join_snapshot_catchup OOM on this host (22GB colima VM); their /health/startup failure is process-down, not the is_ready path Fix 4 touches.	2026-06-18 21:06:18 -06:00
jx12n	580142df49	feat(m12): election-divergence-fix + soak-eval streak + release tooling ... Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno acked under `ack=leader` (journal-only, un-replicated); `decide_join` now quarantines on THIS node's own frontier rather than comparing stream numbers across stream boundaries — eliminates false-quarantine churn on rolling restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum before step-down so the next leader inherits a clean prefix. New `load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains the divergence-fix regression suite; `replication_ops.rs` threads the signal. Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the 30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh` gates the election-divergence suite before any k8s push. Release tooling: `docker/release/` multi-stage Dockerfile + DR image; `scripts/build-release.sh` single repeatable cross-compile+buildx path.	2026-06-18 13:08:53 -06:00
jx12n	a946c6128c	fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR ... Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests	2026-06-17 15:47:37 -06:00
jx12n	44ec87871e	fix(m12p6): 7th-edge — correct reseed seqno + skip suspect HNSW graph on reseed-pending close (rc11) ... Fixes the rc9 over-correction: forcing baseline for ALL nodes (including caught-up ones) caused needless reseed cascades. Now only divergent nodes (frontier > baseline) use baseline as the reseed seqno; at/below-baseline nodes use frontier+1 so the leader picks cheap catch-up vs snapshot. Also skips the HNSW graph checkpoint on SIGTERM when the shard is reseed- pending: the in-memory index reflects suspect/divergent data the next boot discards, so saving it risks a "Failed to read vectors" failure on the post-reseed open. Durable checkpoints and WAL flush still run. close_shared() gains a save_graphs bool; shutdown_inner_impl() is the shared implementation; node.rs passes !reseed_pending.	2026-06-16 23:45:46 -06:00
jx12n	727fbfcb6b	fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) ... Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9: 1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region` visits each hosted group's own shard subdir; per-group leader discovery appends `?shard=N` so a divergent shard heals from its own leader (not shard-0's WAL/term — cross-shard contamination). 2. leader self-join term (Bug 4): `become_leader_for_term` now calls `note_self_won_term` so the elected shard's `joined_term` is set and `cluster_promote` routes rebalances correctly (was: topology-era mis-read → legacy fenced promote → 500). 3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate prevents a node pulling its own stream (its stream isn't a registered peer) → eliminates the `PeerUnreachable(self)` loop. 4. scatter-merge degraded partial (Bug 1): failed shard logs + continues instead of `?`-failing the whole read; bounded read-admission semaphore (`offload.rs`) sheds as 429 instead of piling into a 36s p99. 5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4` most-recent sealed segments; online path gets the same retention clamp. Prevents brief-restart forced-reseed. 6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers` returns `needed=true` and the snapshot installs instead of looping. Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM (HNSW graph was not saved when request-scoped Arc clones were alive at shutdown); updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.	2026-06-16 22:34:21 -06:00
jx12n	a0399550d6	feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild ... Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads it when it matches the durable corpus (seconds), falling back to a full rebuild only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting node and triggered the reseed cascade. Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic close regardless — sibling keep-alive connections no longer block the drain past the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close fires even when a stuck connection task holds an Arc. Fix USearch insert to be a true upsert (remove+add): it was unconditional add, which a multi:false index rejects on a reseeding follower's post-snapshot WAL replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster. Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy + recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and m12p6 graph-persistence + SIGTERM tier-3 regression tests.	2026-06-15 13:09:20 -06:00
jx12n	4db3f1e597	fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback ... Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which crash-looped every scale-up joiner with the same opaque 'could not join within 120s'. The plaintext in-process harness is blind to all of them. - certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf); the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer. Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf. (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.) - join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the joiner's OWN region in the knob file to find its TLS material, but a seed-joiner is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology. Fall back to ANY region's block (every pod mounts the same cert files). - join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS); a cold TLS handshake under contention blew the sub-second budget. Discovery now logs each poll failure at WARN with the full error source chain (a silent loop made every bug present as the same 120s timeout). - statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes). - k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load. Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence; auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean; mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green; tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact) is documented as a follow-up — left unfixed because a naive patch broke the in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small corpus (stream catch-up) to keep that path out of scope. See docs/profiling/m12p5-idle-readiness-elasticity.md §6.	2026-06-14 22:41:59 -06:00
jx12n	8e39ee1078	fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider ... The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs.	2026-06-14 20:29:37 -06:00
jx12n	aa94fd9b1f	feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN ... Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup	2026-06-14 16:21:00 -06:00
jx12n	31ee612f27	feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) ... Scale write throughput across data-shard groups while keeping a single unified read surface: - scatter_gather.rs: pooled fan-out across shard groups (replaces per-request client construction); cross-shard query results merged on one node - cluster/node.rs: cross-shard read routing — a read on any node gathers from every shard group's leader and unions results - cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM on the real mTLS plane) - dto.rs: cross-shard query/result DTOs - k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers, topology-configmap) - k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A - tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job - tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind - docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings	2026-06-14 15:17:35 -06:00
jx12n	da5d2d4d53	fix(m12p2): invalidate SignalRanked top-K cache on CRDT reconciliation ... apply_crdt_state force-sets accumulated scores outside apply_event_local, so it bypassed the single note_write() chokepoint that keeps the trending top-K cache fresh. A node going quiescent right after a partition heal kept serving the pre-reconciliation candidate set indefinitely. Invalidate the cache in apply_crdt_state too, with a regression test that reconciles a new high-score entity and requires it in the next candidate read. Also: extract the for_you/related ANN candidate cap to a named constant (ANN_PROFILE_CANDIDATE_LIMIT), keep the recall test's ef_search in lockstep with its construction default (honored since m12p3), and clarify the vector_search region doc (always null until m12p4 cross-shard reads).	2026-06-14 13:02:10 -06:00
jx12n	bb21e69ae6	feat(m12): vector retrieval G1/G2 — recall harness, ANN in RETRIEVE, index tuning ... m12p1 (measurement truth): TidalDb::vector_search_items pure k-NN probe + POST /vector_search (standalone + region node, merge-by-distance) + tidal-stress --verify-recall (deterministic id-keyed corpus, in-RAM brute-force cosine oracle, open-loop ramp → recall@k + true p99 + read-knee + JSON/gate exit). Repaired fabricated p99 columns (mean-as-p99) in social-scale.md / scale.rs. Verified real: recall@10=0.9997 at 20k/1536-D vs brute-force. m12p2 (G1 unblock): ANN candidate-gen wired into RETRIEVE — for_you=preference vector, related=seed embedding (similar_to), graceful scan-fallback. Cached per-signal-type top-K (signals/ledger/hot_top_k.rs, decay-order-invariant) so trending serves O(K). related over HTTP (FeedQuery.similar_to). Harness gains --feed-profile / --seed-preferences. Verified: trending retrieve p99 3.5-7.7ms. m12p3 (G2): per-query ef_search now honored (RwLock epoch-guard with_expansion, shared guard for same-ef concurrency) + dimension-aware brute→HNSW crossover usearch_min_vectors(dim) + memory_usage() + examples/ann_grid_search.rs. Measured 1536-D/100k clustered: default M=16/ef_c=400/F16/ef_s=200 clears G1+G2 (recall 0.997, p99 1.4ms); F16 -0.25% vs F32; Int8 rejected (-28%). Recall corpus is now clustered (Gaussian mixture) in grid + harness.	2026-06-14 11:07:09 -06:00
jx12n	44b768b8c6	feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) ... End the "replicated XOR sharded" split: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes. - One unified write surface: /items,/embeddings,/signals hash-route to the owning shard group's leader (ShardRouter FNV-1a) AND replicate at RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are per-group; NotLeader names the group. - Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced leadership move) + /cluster/shards/{id}/replicas (add/remove replica). A ?shard= selector threads through every per-shard admin verb and is propagated on intra-group forwards (ShardReplica::admin_path). S=1 is byte-for-byte (no selector, no shard in NotLeader body). - Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over real OS processes — SIGKILL a node under ack=quorum load → only its shard-leaderships re-elect, reads never stop, zero acked loss across random kill points; plus a rebalance-verb test. Harness: MultiProcCluster::start_sharded. - tidal-stress drives the single path (WritePath::Leader|Sharded gone), spreading writes round-robin across gateways or pinning --leader-url. - Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err, ~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A. Known follow-up (tracked): per-group-aware node readiness and cross-node read fan-out under PARTIAL placement.	2026-06-13 18:23:43 -06:00
jx12n	005e292cbb	fix(m11): review remediation + tidal-stress perf sweep + perf wave 2 ... Resolve all BLOCKER/CRITICAL/WARNING findings from the m11p7/p8 review: - tidalctl restore: safe_join path-traversal/Zip-Slip guard + fsync on write - corrupt-WAL checkpoint_seq guard; PITR archive-before-delete - cluster: x-tidal-relayed audit-dedup marker; forward_failures counts 5xx - mTLS/HTTP-TLS handshake hardening; accept-loop EMFILE backoff - per-principal rate-limit + node-token marker-pinning tests - self-heal tier-3 coverage; 5 router-auth tests tidal-stress: measurement-fidelity fixes (schedule-lag p99/max, exact feed-over-SLO verdict, shed annotation) + typed Body, workload.next 184ns->68ns, RoundRobin len==1 short-circuit, HeaderValue cache; new benches/hotpath.rs + lib.rs. perf wave 2: signal_snapshot SmallVec/SignalKey carrier; one-get-per-type ranking pre-pass.	2026-06-13 12:28:04 -06:00
jx12n	d5d1e7d81a	feat(m11): observability+ops (m11p8) + perf-sweep wave 2 T2 ... m11p8 closes G-O + §1.4-3: - Cluster metrics: breaker state, forwards, self-heal on /metrics; multi-shard sibling render (shard="N") - Grafana cluster row + 8-rule Prometheus alert group - Request-id / TraceLayer on both cluster routers; id rides forward hop - Truthful status: flushed leader applied_events frontier; post-promote ShardId(0) keying fix - Self-driving heal: tick_self_heal re-arms stuck-peer backlog every ~3s - WAL PITR: wal.archive_dir, archive-before-delete gap-free - tidalctl backup/restore with BLAKE3 content-hash verification - Rolling-upgrade build_version handshake (N/N+1, never rejects) + Woodpecker release gate perf-sweep wave 2 T2: one-get-per-type pre-pass in ranking executor - signal_values.rs pre-fetches all signal kinds before scoring loop - Eliminates per-item repeated DashMap lookups: −18.8% for_you, −31% under writes - Byte-identical output verified with A/B test harness	2026-06-13 09:17:49 -06:00
jx12n	6651c14adc	feat(m11): cluster security (m11p7) + perf instrumentation floor ... m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte): - gRPC replication mTLS by default via a custom tokio-rustls acceptor + DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap, no pod restart, no inotify) - inter-node HTTP TLS sharing the same resolver (one rotation, both planes) + per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403 - admin audit log (operator-leg only) + per-principal rate limit (engine RateLimiter; sibling nodes exempt) - k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback; secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable) - exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs (HTTP foreign + zero-drop rotation under load), 7 security unit tests perf — instrument floor (sweep Wave 1): - new tidal/benches/wal.rs + tidal-server/benches/scatter.rs - p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md - add @tidal-performance agent (Martin Thompson) new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs, docs/planning/milestone-11/phase-7.md	2026-06-13 01:25:35 -06:00
jx12n	3bfde53b90	feat(m11): data-plane sharding × replication (m11p6 L0-L2) ... ClusterNode hosts a BTreeMap<ShardId, Arc<ShardReplica>>: writes hash-route to the owning shard leader, reads scatter over shard groups. In-group shard==region preserved so the engine and tidal-net are untouched; S=1 stays byte-for-byte (today's cluster is a 1-shard × RF=N group). Topology grows shard-group awareness; membership, election, forward, reseed, and join_boot thread ShardId through. Proven by an in-process 2×2 RF=2 gRPC test plus S=1 parity, incl. tier-3 real-OS-process failover. clippy/fmt clean.	2026-06-12 23:06:41 -06:00
jx12n	bf57be18e1	feat(m11): membership, snapshot install, and reseed (m11p5)	2026-06-12 19:55:54 -06:00
jx12n	95461d3cf8	feat(m11): Raft leader election over WAL stream (m11p4) ... Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers, heartbeat-only divergence detection + quarantine, and fenced promote. Elections converge in 0.6–1.0s; zero acked-write loss across all kill points. Closes G5 (leaderless recovery) from the v0.9 wave.	2026-06-11 23:30:24 -06:00
jx12n	d0a52e4530	feat(m11): catch-up timer retry + TSEG segment version header (m11p4) ... WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved) prepended to every new segment. Legacy headerless segments (m0-m11p3) read as implicit v0 — no migration. Unknown magic/version surfaces as WalError::SegmentFormatUnknown at open time; foreign files are never repaired or truncated (fixes the silent data-loss path from the p3 rollout incident where torn-tail repair zeroed a follower's unreadable segments). Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream errors that skip the shard now arm a timer retry (re-arm-on-skip is the load-bearing liveness fix — without it a skipped pull never re-fires and the follower stays permanently behind). Single retry pending per shard; CatchupRunner owns the Arc'd state shared between the retry tasks and the transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path. Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.	2026-06-11 17:05:20 -06:00
jx12n	5ed2edb211	feat(m11): quorum-acked writes — ack=leader|quorum, commit index, durable frontier reports (m11p3) ... ack=quorum gates replicated writes on a majority of the replica set durably holding them: followers push their durably-applied frontier (ReportApplied, once per apply round, decoupled from ship acks), the leader folds frontier reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex (k-th-largest durable mark), and handlers await it through an async watch-channel bridge (zero parked threads per waiter). Honest timeouts: retryable 503 naming the laggards; x-tidal-seq on every cluster write. Follower blob applies are batched under group-commit fsyncs (22x seeding). Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss. Seven-dimension review pass (all confirmed findings fixed): - WAL blob drain now ABORTS on the first write failure instead of reusing the failed seqno mid-drain (a torn record buried mid-segment would truncate every later acked record on replay) - apply_replicated_blobs waits every staged append even after a mid-batch failure, parses metadata once, and moves records into Arcs shared with the WAL writer (no deep clone per record on the follower apply path) - CommitIndex: zero-peer fast path now respects demotion (active checked under lock before the single-replica return), k-th-largest uses select_nth over a reused scratch buffer - await_quorum: re-reads the index once after the deadline fires (no false 503 for a write that committed in the race window), warns when the commit-watch bridge dies outside shutdown, zero-peer path checks active - notify_applied report failures: WARN on the first failure of a streak, INFO on recovery (a silently stalling frontier reads as unexplained quorum 503s); receiver skips re-notifying unadvanced frontiers - x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed through forwards) so durability cursors can tell dedup from no-seqno - docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade order (leader first — a pre-m11p3 leader silently downgrades quorum requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for report-loss diagnosis on the quorum-timeout alert Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).	2026-06-11 13:28:08 -06:00
jx12n	225751d34d	feat(m11): WAL-as-stream replication + perf floor (m11p1+m11p2) ... m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push, microseconds) separate from group-commit fsync; ShipQueue batches+windows outbound segments; receiver coalesces inbound chunks before applying. Adds first tidaldb_cluster_* metrics. m11p2 — leader WAL is now THE replicated log: fsynced batches feed a bounded WalShipFeed and ship byte-identical to followers; WAL seqnos survive restarts (relay-reset hazard gone). Item metadata and embeddings journal kind-1/2 blob records on the same stream as signals; the m8p10 HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via server-streaming RPC, triggered on gap detection, follower boot, and leader heal nudge. Promote carries a stream baseline so peers skip pre-stream history.	2026-06-11 09:10:06 -06:00
jx12n	6f17409f40	fix(cluster): forwarded item/embedding writes never replicated; heal backfill 401'd under auth ... Two correctness bugs in multi-process cluster mode (m8p10), both found live on a real 3-pod k3s cluster while validating the deployment, both invisible to the existing in-process / unauthenticated test suites: 1. Forwarded item/embedding writes dropped on the floor. create_item / write_embedding gated the leader's peer broadcast on `if internal { return }`. A write to a NON-leader gateway is forwarded to the leader with the internal marker set (loop-prevention), so it hit that branch and terminated WITHOUT broadcasting — the item landed only on the leader. Signals were unaffected (the WAL relay ships regardless of the marker), which masked it. Fix: gate on leadership, not the marker — a follower applying a marked broadcast/heal terminates; the leader (external OR forwarded) always fans out. Forwarded writes now return the {replicated_to,failed} report instead of a bodyless null. 2. Heal item/embedding backfill 401'd whenever TIDAL_API_KEY is set. post_marked_blocking sent the internal marker but no Authorization header. The marker is a trust signal, not an auth bypass (the bearer middleware runs first), so every backfill POST was rejected 401 — a region that missed an item while down stayed permanently inconsistent at lag 0. Unauthenticated tests never caught it. Fix: thread TIDAL_API_KEY into RegionClusterState and attach it (same key on every region) to the backfill POSTs. Verified live: forwarded writes via follower gateways converge to all 3 regions; a region scaled to 0 during an item write backfills on heal (item_failures=0). Follow-up: add multiproc regression tests with auth for both paths.	2026-06-10 20:37:46 -06:00
jx12n	8a0950260f	feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites ... Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps.	2026-06-10 14:07:33 -06:00
jx12n	1092d34c39	feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation ... - Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly	2026-06-09 17:06:34 -06:00