40 Commits

Author	SHA1	Message	Date
jordan	0f18243a64	deploy: pin m12-agesort-20260901 (age-aware Hot/New live) ... Digest 9234aacb, commit a588f01. Live on all three pods after a staged 2->1->0 roll with a POST /items quorum probe between stages (201 each). Rolled at 04:05Z, after today's velero backup completed at 03:53:49 (23.2 min, near the top of the calibrated range, so a 03:31 roll would likely have cancelled it). Live functional proof: builtin new/chronological return ids [551,188,167,321,6,2] on production data -- neither descending entity id nor ascending -- so they order by real created_at. hot differentiates 1.0 -> 0.9631. Zero non-finite scores. The roll produced 1/3/4 container restarts as each pod booted with a reseed marker (WAL compaction advanced the retained floor past its frontier during its ~90s absence) and snapshot-installed from the discovered leader. pod-0 logged the self-correcting arm. All three converged with no manual intervention: 9/9 shard frontiers lag 0, reseed_gaps empty. Playwright 34/34, semantics 5/5, verify-live 23/23.	2026-08-31 22:24:13 -06:00
jordan	a588f01f63	ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units ... Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL, and a CHANGELOG that named profiles that do not exist. All verified before fixing. BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0. So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable `created_at` outranked the genuinely newest item and `normalize` reported its score as 1.0. The codebase had already ruled on this and gone the other way. `score_shortest`, `score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and `helpers.rs` carries a regression test named for the exact anomaly -- `normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because `Shortest` hit it first: "a pre-clamped 0.0 floated above them and the last-ranked (missing) item reported the highest score 1.0". I reintroduced it on a new negated scale. Three reachable data classes, none hypothetical: a legacy row written before `created_at` was materialized (`state_rebuild.rs` documents the class), an empty map from `deserialize_metadata` on a short or corrupt row, and an entity dropped by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT STORAGE READ ERROR could promote an item to the top of the feed. Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole set ties, unchanged. Map present but this entity has no usable `created_at` -> NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because that is a real value that legitimately means "newest"; an absent one carries no recency claim at all. Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports `needs_item_metadata`, so the map is loaded on essentially every Hot query and the per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a single undated row the freshest divisor in the set: against a year-old cohort at the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with two views outranked correctly-dated items with tens of thousands. Across all four Hot builtins. Now scored at Hot's floor. BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_ newest` ranked candidates against the newest ids of the WHOLE universe, but `scan_candidates` iterates the universe bitmap ASCENDING and breaks at `(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a catalog whose ids are assigned in creation order -- which is exactly what `metadata_with_created_at`'s `Timestamp::now()` default produces -- the global newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by` I replaced kept the newest available. Strictly worse than the bug I set out to fix, and only visible when degraded. The key is now composite -- recency rank, then DESCENDING id for anything the oversample did not cover -- so where index and candidate window overlap the survivors are genuinely newest, and where they do not it degrades to the documented pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_ key`: this path runs ONLY when the load shedder has already decided the node cannot afford the work, so it must stay a linear partition, and only membership of `[0, cap)` matters because Stage 3 re-orders the survivors anyway. Mutation-proven, and the numbers show why the old test was blind: rank-only key returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the pre-existing 150-item test PASSES under that same mutation -- its fixture had the oversample covering the whole universe AND the newest items at the low ids, the one configuration where the defect cannot appear. New test uses a 3000-item time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200 so the guard skips), limit=25 does. Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING entity id, and I had made newest-first coincide with it, so any mutation that merely TIED the set still produced the asserted vector. The gate mutation proves it: reverting `needs_metadata_for_sort` to its old four-variant form yielded `left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`, which is the exact drift this work exists to prevent. Every ordering fixture is now non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and ascending id (the tie collapse). CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them carry a diversity block so they already loaded item metadata and change behaviour the instant this image rolls. Replaced with a verified table. Also cited the spec conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified `Sort::New` as "created_at DESC". created_at units are now trusted with a warning instead of silently. A seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores -496731 against boost sums in single digits -- re-entering through DATA the exact "recency annihilates every boost" defect this work removed -- and Hot buries the item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was inert; this work made it live. The repo's OWN fixtures made that mistake in three places, which is the proof it is the natural one. `metadata_with_created_at` now warns when the value is too small to be nanoseconds, logging entity id, value, age_hours and age_years so an operator can act. It does NOT rewrite the value -- guessing the unit would corrupt what the `created_at` range index already reads as nanoseconds -- and does NOT reject the write, which would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server: seconds, millis and micros each warn with age_years=56; a real nanosecond value is silent. The nanosecond contract is now documented on the public DTO and propagates to the OpenAPI schema, where it was invisible before. Stale docs corrected: two published blog posts and two claim-verification ledgers were certifying "entity recency (higher ID = newer)"; the e2e fixture contract justified an interleaving decision with reasoning that is now false (its conclusion still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:" rationale cited behaviour this work removed, so it is now marked PENDING RE-MEASUREMENT with the three specific measurements named rather than left reading as justified. Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors. Fast integration suites all green. Real-server e2e re-verified: `new` and `hot` both return newest-first, `new`'s scores now evenly spaced across evenly spaced ages.	2026-08-31 21:31:31 -06:00
jordan	6385425a92	ranking: make Hot and New age-aware; fix the same gap in three more places ... `score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added.	2026-08-31 19:58:01 -06:00
jordan	8f9aad1fe0	deploy: pin m12-poisonfix-20260831 (both incident defects fixed) ... Digest 93a2929d, commit 44820d4. Live on all three pods after a staged 2->1->0 roll with a POST /items quorum probe between stages (201 each). The roll needed ZERO manual pod restarts — the previous one needed three because two reseed markers latched per node and only one discharged. Pods came Ready in 30s / 80s / 110s. Verified against the exact probe that caused the outage: 128-dim vector into the 1536-dim content_vector slot now returns 400 with a specific message instead of 500, no receiver halted on any pod, all 9 shard frontiers at lag 0, reseed_gaps empty, writes 201. Playwright 34/34, semantics 5/5, verify-live 23/23.	2026-08-31 03:14:20 -06:00
jordan	77f68d181c	verify: flip the two log tripwires post-roll, calibrate the backup assertions ... Post-deploy half of the deploy-verification contract for m12-harden-20260831. Flipped, exactly as each assertion instructed its own successor to do: - 06-logs.spec.ts: asserted `jsonLines === 0`. JSON_LOGS is live, so it now asserts every sampled line parses as JSON. The ANSI check stays pinned at 0. - 09-operator-authority.spec.ts: asserted JSON_LOGS was absent from the StatefulSet. Now asserts JSON_LOGS=1 AND TIDAL_SERVICE_NAME=tidaldb, because the second is load-bearing: enabling structured logs makes the app's own `service` field win in the fleet's Vector normalize transform, silently renaming the log stream tidaldb -> tidal-server and blinding every query keyed on it. The fleet's _stream_fields contract pins field names but no legal values, so nothing there would have caught the flip. Calibrated, NOT loosened — the two backup assertions were unpassable by construction for ~25 minutes every day: - The schedule fires at 03:30 and measured runs take 9.1-24.8 min (n=15), so the newest object is legitimately InProgress during its own window. The "newest backup completed cleanly" test now selects the newest FINISHED backup; a namespace where nothing has ever finished still fails. - "no backup stuck in progress" asserted InProgress -> fail, full stop. It now bounds in-flight age at 60 min: ~2.4x the slowest success and a quarter of the 240.0 min timeout that the observed PartiallyFailed runs (2026-08-17/19/25) all hit. A gate that cries wolf on a schedule gets muted, and then it is not a gate. Both thresholds come from reading every backup in the namespace, not from a guess. Playwright 34/34 and hermetic semantics 5/5 against the deployed image.	2026-08-30 22:06:20 -06:00
jordan	fe8d0c87e7	harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path ... Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.	2026-08-30 20:55:58 -06:00
jordan	59d7dadc18	k8s(cluster): pin m12-vsc-20260830, now running on all three voters ... Built clean from 8aa1fbb. Staged roll partition 2 -> 1 -> 0 with a quorum-write probe between each (201 every time); sts 3/3 updated=3, and all three voters report build_hash=8aa1fbb4140ac720b360465073388fe5ad0f84a7 with no -dirty suffix. Post-deploy, measured live: a non-unit query (|q|^2 = 132.4) now returns distances of 1.7944-1.8059, inside the documented [0,4] and identical on all three replicas; the same class of query returned 591-1174 before. A zero-norm query returns 400. Per-group vector counts are now visible for all three shard groups (groups 1 and 2 had no series at all before) and read spread 0.	2026-08-30 14:09:38 -06:00
jordan	53c345e890	k8s(cluster): pin the vector-grow image now running on all three voters ... registry.threesix.ai/tidal/server:m12-vector-grow-20260830@sha256:2f2357eae5af 863804a85dba5514833c76730eeaac211e8da2cb60e797b34f92, built clean from 7c1c80d. Rolled staged partition 2 -> 1 -> 0 with a quorum-acked write between each voter: 201 every time, sts 3/3 updated=3. End-to-end functional verification against the public Service, 21 checks, 0 failures - the embedding write that returned 500 before this image now returns 204, text search and feed serve 200 with degraded=false, the auth boundary holds (401 unauthenticated, 401 wrong bearer, 403 data-bearer on an admin verb), and all three voters return lag 0, no reseed marker, and the same applied frontier. vector_search now returns the newly embedded entities, which was impossible before: they could not be inserted at all.	2026-08-30 11:58:30 -06:00
jordan	ef6e0b9636	k8s(cluster): pin the admin-gate image now running on all three voters ... registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060a342 658b734d258245b20f6233d96e26415b3a44956b1c3bceebe48c, built from c9adec0. This is the image that finally puts 388e445 into production. The previous pin (m12-boot-pull-fix-20260821, built from 5b3cfe5 on 2026-08-21 11:35) PREDATED the admin/data split that landed 2026-08-22 00:57, so the operator-authority separation existed in the repo, in the manifest and in the tests while the running binary had no such code - which is also why no admin-gate warning ever appeared in the pod logs. Rolled staged behind updateStrategy partition 2 -> 1 -> 0, one voter at a time, with a quorum-acked write probe between each: HTTP 201 every time, so no write availability was lost. PDB held disruptionsAllowed=1 / currentHealthy=3 throughout and all three voters returned lag_events=0 at term 94 afterwards. Measured before and after on the live cluster: POST /cluster/heal with the DATA bearer 415 -> 403 POST /cluster/members/remove with the DATA bearer 403 POST /cluster/heal with the ADMIN key 422 (past auth) POST /sharded/items with the DATA bearer 201 -> 201 (unaffected) POST /sharded/items with the ADMIN key 201 415 rather than 401 was the proof the gate had been OPEN: the data bearer was authenticated and authorized for a destructive verb and only the content type was wrong. 403 is the proof it is now shut.	2026-08-22 23:54:22 -06:00
jordan	c9adec040d	k8s(cluster): declare the limits production actually runs ... The manifest said cpu 2 / memory 6Gi while the live StatefulSet ran cpu 3 / memory 7Gi, so `kubectl apply -k k8s/cluster/` DOWNGRADED production every time it was run - a silent capacity regression on the write path, delivered by the very command used to deploy a fix. Verified before and after with kubectl diff: the cpu/memory hunks are now absent, so an apply is a no-op on resources. The live values are the intended ones; this file is now the source of truth for them. The comment keeps the reasoning that matters - the limit exists to keep the measured query/apply burst reachable without reserving it, and when neighbours burst too the answer is more REQUEST, not more limit, because CFS throttling is how CockroachDB was pushed into multi-second Raft stalls on this fleet with nodes 70% idle.	2026-08-22 19:49:37 -06:00
jordan	388e445a38	feat(cluster): separate operator authority from data-plane access ... Every destructive /cluster/* verb sat behind the SAME bearer as /items and /search, so any application key could remove a member, force a partition, or transfer a shard. There was no way to hand out a client credential without also handing out the ability to destroy the cluster. Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like the others). /cluster/promote, /cluster/partition, /cluster/heal, /cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas, transfer} move into their own router subtree behind an admin gate; the data bearer now gets 403 there - authenticated but not authorized, distinct from the 401 for a bad token. Three things this had to get right: * The admin key must ALSO authenticate. A request carries one Authorization header, so if the admin key did not satisfy the bearer gate, an operator presenting it would be 401'd before the admin gate ran and the verbs would be reachable by nobody. Caught while writing the test, not after. * A verified sibling node token clears the gate too. Nodes relay operator verbs to the leader/target carrying whatever credential the caller sent, and the legacy fan-out promote uses the internal marker, so requiring the admin key on that hop would partition the control plane. * The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal nudge), /cluster/join + /cluster/members (seed-join) and the /cluster/reconcile* pair are dialled node-to-node, so gating them would break replication and joining. Absent admin key = previous behavior exactly, plus a startup WARN naming the exposure, so this is safe to upgrade into. The k8s secret mount is optional:true because without that a deployment lacking the key would fail to MOUNT and never start. Also closes the /cluster/status hole this exposed: it and /cluster/status/local reported leader identity, membership, term and per-shard applied/lag/commit seqnos from the UNAUTHENTICATED probe group. They are protected now, which is what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the network layer. And fixes a latent bug found on the way: seed-join discovery, reseed discovery and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly, which yields nothing on a *_FILE-only deployment - the node would dial an authenticated peer with no credential. They use security::bearer_from_env() now, which honours both shapes. Verified: 5 new unit tests; two multi-process runbook tests on real 3-process clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status and through the gate on heal; bare /cluster/status 401, 200 with the bearer). That the authenticated cluster converges at all is the load-bearing assertion - if moving status behind auth had broken leader discovery, startup would hang. Full unit suites green (2101 + 162), reseed e2e green, clippy clean.	2026-08-22 00:57:01 -06:00
jordan	0a861d9144	k8s(cluster): close the unauthenticated metrics and peer plane to foreign pods ... Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics - corpus size, seqnos, leader identity, all unauthenticated - and reach the peer gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in namespace threesix returned metrics, and returns "connection refused" after this. NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed. Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and peer dial-out surfaces are not fully enumerated, and a wrong egress rule partitions the cluster instead of merely blocking a scrape. :9500 stays open, and the file records why. All three probes (startup, readiness, liveness) target it, probes come from the NODE rather than a pod, and node-to-pod handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts every pod. The exposure that would have closed is /cluster/status, which is unauthenticated by design. That is an engine defect and gets fixed in tidal-server, not worked around at the network layer. Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked write through the public ingress returns 201, and all three regions report applied=13322229 lag=0 reachable partitioned=false with zero new restarts.	2026-08-22 00:23:55 -06:00
jordan	087b83154a	k8s(cluster): publish the client surface over public TLS, data routes only ... tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt cert, verified end-to-end from the internet: 401 without a bearer, 401 with a wrong one, 200 with the real key, and a quorum-acked write returning 201 on all three node IPs. Three things this had to get right, each of which failed first: * The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers 500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so serverName pinning validates it without insecureSkipVerify. * `service.*` annotations are read from the Service, not the Ingress. Putting serversscheme/serverstransport on the Ingress is silently ignored and presents exactly as a broken backend. * http01 cannot be used behind any gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver. Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME single bearer as the data routes, so a client key could remove members or transfer shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos), /openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only on the headless peers Service, unreachable here). Documents two controls that are NOT available and why: an IP allowlist cannot work while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs the client address), and Traefik basicAuth cannot stack in front of the bearer because both occupy the Authorization header.	2026-08-21 23:14:55 -06:00
jordan	cc7066e0d8	k8s(cluster): pin the boot-pull-fix image now running on all three voters ... registry.threesix.ai/tidal/server:m12-boot-pull-fix-20260821 @sha256:4150af1044b8084b84abeafedbbaedb1f084a2c84fd0600b56f5b04ed0697eba Rolled tidaldb-0/-1/-2 onto it via OnDelete with a quorum write probe between each step: every probe returned 201, so the roll cost zero write availability. Cluster is 3/3 with all nine group-replicas converged on identical frontiers (13322227/13540667/13072512) and an identical 33,331-vector corpus. PDB disruptionsAllowed is back to 1 after being 0 for the whole incident. updateStrategy restored to RollingUpdate/partition 0.	2026-08-21 12:01:34 -06:00
jordan	da736b8eb2	fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail ... A follower that latched `reseed_required` from a genuine `snapshot-required` refusal could clear its own marker ~200ms later and so never run the boot reseed that was the only way to close the gap. With `replication.reseed_self_restart: true` it exit-looped: latch -> clear -> exit(0) -> boot with no marker -> re-latch. Production tidaldb-0 did this 196 times in 21h on 2026-08-20 while the cluster ran on 2 of 3 voters. `clear_stale_reseed_marker_if_caught_up(applied >= leader_last_seq)` compared the applied frontier against the LEADER'S TAIL and documented the invariant "a node genuinely behind a COMPACTED gap never reaches caught_up". That is false: on a quiet shard any node meets the leader's tail, including one missing committed history it can never refetch. The clear also reset `tidaldb_cluster_reseed_required`, so the gauge flapped 1->0 every 30s and `TidalDBClusterReseedPending` (`== 1 for 10m`) could never fire - the code path that broke the reseed also erased the signal that would have reported it. The discharge decision now belongs to the marker. `ReseedMarker::discharged_by` requires a stream-dischargeable reason AND an applied frontier that reached the marker's own `from_seqno` - the very entry whose absence latched it. A compacted gap can never satisfy that, so the reseed runs; a node merely behind a shippable tail satisfies it as soon as the stream serves that entry, so the m12 false-alarm self-heal still works (and now clears sooner, since it no longer waits to meet a moving leader tail). The two conditions previously shared `ReseedReason::SnapshotRequired`, so reason alone could not discriminate. The term-join arm's `frontier > baseline` case deliberately sets `from_seqno = baseline`, BELOW the node's own frontier, so a bare `applied >= from_seqno` would discharge it instantly - it holds divergent post-baseline data only a snapshot can discard. It gets its own never-lag- dischargeable reason, `DivergentPostBaseline = 3`. Adding a discriminant is the sanctioned forward-only extension; a downgrade that meets one refuses to decode it, per the existing kind-3/kind-4 precedent. The election-won call site passed a hardcoded `true`; it now passes the leader's durable flushed frontier (`applied_seqno` never advances on a leader), and a `DivergentPostBaseline` node is not campaign-suppressed so it can reach there. Tests: three deterministic predicate tests pinning the incident's exact seqnos (13540653 vs earliest-available 13540657), the false-alarm discharge, and the never-discharge of every structural reason. Pre-existing and NOT introduced here: cluster_reseed's `mp_follower_reseeds_via_snapshot_after_compaction` and `mp_graceful_rolling_restart_under_load_no_reseed` fail on baseline main (verified by stashing this change). The first is the owner-test for this exact mechanism - its leader compaction no longer forces a `snapshot-required`, so it never reached the clear path and never guarded it. Tracked separately.	2026-08-20 22:23:35 -06:00
jordan	261d78d1f1	k8s(cluster): pin the reconcile-limit image	2026-08-18 10:14:07 -06:00
jordan	bb8425b0e8	k8s(cluster): pin the frontier-tick image	2026-08-18 10:07:51 -06:00
jordan	3eaf28bf8a	k8s(cluster): roll to the frontier-fix image ... Also carries the `tmp` fs-backup exclusion committed with the engine fix.	2026-08-18 09:58:34 -06:00
jordan	12c7edc374	fix(cluster): the frontier pair has one writer, and 0 is not a commit index ... `TidalDBClusterQuorumLag` sat CRITICAL all session against the live three-voter cluster while every region reported lag 0 and every per-peer ship queue was empty. Two independent defects fed it: - `observe_ship` bumped `relay_last_seq` on every batch ship while `relay_durable_seq` only moved when a signal write completed. The two are documented as a subtractable pair, so a shipping-but-not-committing node reported the whole relay log (13.3M events) as quorum lag. The ship path now feeds only its own per-peer queue-depth gauge; the pair has one writer. - `set_frontier_gauges` published `CommitIndex::committed()` verbatim, but that returns 0 as a SENTINEL for "no quorum information in this term yet". It now publishes both halves or neither, and every satisfied `await_quorum` -- not just signal writes -- refreshes them, so item and embedding workloads keep the pair live. Regression test asserts a busy ship loop leaves both halves at 0 (lag 0, not 13.3M) and that the single writer still moves them together. Also excludes the `tmp` emptyDir from velero fs-backup: three 0-byte PodVolumeBackups a night whose only other outcome is failing the whole fleet backup when a scratch file vanishes mid-snapshot.	2026-08-18 09:50:19 -06:00
jordan	d923b036af	k8s(cluster): three voters is the desired state, on the read-path fix image ... Two outages in one session came from this file claiming `replicas: 0` while the cluster served traffic: a plain `kubectl apply` scaled it to zero, twice. Source now states production intent. Parking stays an explicit `kubectl scale` divergence recorded in k3s-fleet/cluster-state.yaml, and scripts/restore-fleet.sh remains the guarded path back - its storage, image, and per-node capacity preflights are exactly what a bare apply does not do. Pins server:m12-consumer-readpath-20260818@sha256:9191233d..., the build with the transport-aware read budget and the scatter-degraded counters. Verified after the rolling update: 3/3 Ready, all regions lag 0, every replica answering 3/3 shards with degraded=false on the default budget.	2026-08-17 20:52:32 -06:00
jordan	897c6086f5	fix(cluster): size the read fan-out budget for the transport it crosses ... Restoring the three-node cluster for a first production consumer surfaced this immediately: EVERY cross-shard read came back {"items":[...],"scatter_gather":{"degraded":true, "unavailable_shards":["tidaldb-0","tidaldb-2"],"shards_queried":1, "elapsed_ms":50,"shard_deadline_ms":45}} HTTP 200, one shard of three, partial results. Replication itself was healthy - /cluster/status showed all three regions reachable, lag_events 0, 13.3M events applied each - so nothing in the quorum, election, or ship metrics moved. Measured on the live cluster: a COLD peer fetch (TCP + TLS handshake + remote 1536-D search) takes ~50ms; a warm one takes ~1ms. DEFAULT_DEADLINE_MS is 50 (spec §7.4) and NETWORK_OVERHEAD_MS is 5, leaving a 45ms per-shard budget - just under the cold cost. Proven by parameter sweep against one pod: deadline_ms=50 -> degraded, 1/3 shards, 0 items deadline_ms=250 -> healthy, 3/3 shards, elapsed 51ms deadline_ms=1000 -> healthy, 3/3 shards, elapsed 1ms (warm) The 50ms spec figure budgets a shard READ, not establishing a connection to another pod. m11p7 put TLS on that hop and the default never followed, so the first query after any rollout, idle period, or pod restart answered from a third of the corpus. Fixed with a transport-aware default: 50ms in-process, TLS_DEFAULT_DEADLINE_MS (250ms) once inter-node TLS is configured. An explicit `?deadline_ms=` still wins in both directions, and MAX_DEADLINE_MS is unchanged. The worse half was silence. A degraded fan-out is the one cluster failure that answers 200 OK: the caller gets a ranked list assembled from a subset of the corpus with `degraded: true` buried in response metadata. Nothing incremented, so no alert could exist - a feed quietly ranking over one third of its candidates looked identical to a healthy one. Added tidaldb_cluster_scatter_degraded_total and tidaldb_cluster_scatter_shard_unavailable_total, emitted from both HTTP fan-out paths, so partial answers are now a countable correctness signal. Also sizes the cluster StatefulSet for a consumer instead of the endurance gate: requests 2 cores -> 300m per voter (limit 2 cores). The 2-core reservation was the 200 rps soak envelope and needed 6,000m plus 2,000m free on each of three PV-pinned nodes; the fleet is 82-91% committed, so that contract could not be placed and the cluster stayed parked for a gate nobody is waiting on. 300m is what the tightest pinned node can reserve, with the quorum/write-pool alerts as the detector if real load outgrows it. Tests: default_read_budget_covers_a_cold_inter_node_tls_hop pins the budget against the measured cold hop and the explicit-override path; the cluster-metrics render test covers both new counters.	2026-08-17 20:28:21 -06:00
jordan	c97aaa8e5b	fleet remediation: make the workspace gate runnable, then fix what it caught ... `cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope.	2026-08-16 12:38:14 -06:00
jx12n	c22a3b65a6	docs: withdraw the pre-release "not ready for production" disclaimer ... M0-M12 are shipped and the HA cluster runs in production on k3s, so the pre-release disclaimer no longer describes the project. Removes it from the canonical doc set and corrects the readiness text that had gone stale. - README.md: replace the "Pre-release / not yet recommended for production" banner with a production-ready statement; drop "(experimental)" from the cluster status bullet; state the post-1.0 versioning posture (additive in minor releases, breaking changes get a documented migration path). - CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md / docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a guard against standing up a multi-node fabric by accident rather than a readiness warning. - CHANGELOG.md: record the stability posture under [Unreleased], superseding the historical 0.1.0 "no stability guarantees" note (left intact as history). - k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead. Also corrects text that was factually wrong since m11p3/m11p4: the multi-process cluster gate, its CLI help, and the served OpenAPI description all still claimed quorum-ack writes and automatic failure detection did not exist. They do. Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries, the kubernetes.md rc7 fix note) are left unchanged. Verified against a running binary, not just the build: the opt-in gate's refusal message, the startup WARN, /health 200, and the served /openapi.json description all carry the new text. cargo fmt clean; clippy -D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155 server lib tests pass; scripts/check-docs.sh OK. Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP	2026-07-30 19:03:34 -06:00
jx12n	0847c3d36f	chore(k8s): roll cluster statefulset to m12-writeburst-rc7 (write-burst fix live) ... Pin the live cluster to the rc7 amd64 image (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1, 6-layer manifest, imagetools-verified) carrying the write-burst false-partition fix (tidal-net record_timeout). Deployed via `kubectl set image`; rolling update completed 3/3 Ready. Reseeds across the rollout were the safe snapshot-install fallback (rejoining behind WAL retention), all converged lag=0, no loop, no loss.	2026-06-19 16:48:49 -06:00
jx12n	1b5bcbacd7	fix(net): classify ship deadline as timeout, not partition (write-burst false-partition) ... A client-side ship DEADLINE means the RPC did not round-trip within request_timeout — which a slow-but-ALIVE follower produces under a sustained 1536-D ack=quorum apply burst (transport runtime momentarily starved by the CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a genuinely blackholed peer does. Counting that as record_failure was the write-burst false-partition: 5 such opened both followers' breakers, the commit index stalled, ack=quorum 503'd, and retries re-burst the same starved peers with no self-heal. - CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof of life (no round-tripped success/backpressure within reset_duration); neutral no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes the liveness stamp (no reply arrived). - PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout; genuine severance still surfaces as connect-level Unavailable/transport reset -> record_failure and still opens the breaker. - ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server (handler succeeds once then hangs past the client deadline) + 6 unit tests. Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt soak_eval / soak-eval / s3 / tidalctl.	2026-06-19 16:25:50 -06:00
jx12n	a946c6128c	fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR ... Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests	2026-06-17 15:47:37 -06:00
jx12n	44ec87871e	fix(m12p6): 7th-edge — correct reseed seqno + skip suspect HNSW graph on reseed-pending close (rc11) ... Fixes the rc9 over-correction: forcing baseline for ALL nodes (including caught-up ones) caused needless reseed cascades. Now only divergent nodes (frontier > baseline) use baseline as the reseed seqno; at/below-baseline nodes use frontier+1 so the leader picks cheap catch-up vs snapshot. Also skips the HNSW graph checkpoint on SIGTERM when the shard is reseed- pending: the in-memory index reflects suspect/divergent data the next boot discards, so saving it risks a "Failed to read vectors" failure on the post-reseed open. Durable checkpoints and WAL flush still run. close_shared() gains a save_graphs bool; shutdown_inner_impl() is the shared implementation; node.rs passes !reseed_pending.	2026-06-16 23:45:46 -06:00
jx12n	727fbfcb6b	fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9) ... Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9: 1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region` visits each hosted group's own shard subdir; per-group leader discovery appends `?shard=N` so a divergent shard heals from its own leader (not shard-0's WAL/term — cross-shard contamination). 2. leader self-join term (Bug 4): `become_leader_for_term` now calls `note_self_won_term` so the elected shard's `joined_term` is set and `cluster_promote` routes rebalances correctly (was: topology-era mis-read → legacy fenced promote → 500). 3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate prevents a node pulling its own stream (its stream isn't a registered peer) → eliminates the `PeerUnreachable(self)` loop. 4. scatter-merge degraded partial (Bug 1): failed shard logs + continues instead of `?`-failing the whole read; bounded read-admission semaphore (`offload.rs`) sheds as 429 instead of piling into a 36s p99. 5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4` most-recent sealed segments; online path gets the same retention clamp. Prevents brief-restart forced-reseed. 6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers` returns `needed=true` and the snapshot installs instead of looping. Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM (HNSW graph was not saved when request-scoped Arc clones were alive at shutdown); updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.	2026-06-16 22:34:21 -06:00
jx12n	a0399550d6	feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild ... Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads it when it matches the durable corpus (seconds), falling back to a full rebuild only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting node and triggered the reseed cascade. Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic close regardless — sibling keep-alive connections no longer block the drain past the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close fires even when a stuck connection task holds an Arc. Fix USearch insert to be a true upsert (remove+add): it was unconditional add, which a multi:false index rejects on a reseeding follower's post-snapshot WAL replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster. Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy + recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and m12p6 graph-persistence + SIGTERM tier-3 regression tests.	2026-06-15 13:09:20 -06:00
jx12n	4db3f1e597	fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback ... Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which crash-looped every scale-up joiner with the same opaque 'could not join within 120s'. The plaintext in-process harness is blind to all of them. - certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf); the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer. Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf. (scripts/gen-cluster-certs.sh already did this; the two were inconsistent.) - join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the joiner's OWN region in the knob file to find its TLS material, but a seed-joiner is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology. Fall back to ANY region's block (every pod mounts the same cert files). - join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS); a cold TLS handshake under contention blew the sub-second budget. Discovery now logs each poll failure at WARN with the full error source chain (a silent loop made every bug present as the same 120s timeout). - statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes). - k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load. Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence; auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean; mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green; tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact) is documented as a follow-up — left unfixed because a naive patch broke the in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small corpus (stream catch-up) to keep that path out of scope. See docs/profiling/m12p5-idle-readiness-elasticity.md §6.	2026-06-14 22:41:59 -06:00
jx12n	8e39ee1078	fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider ... The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs.	2026-06-14 20:29:37 -06:00
jx12n	aa94fd9b1f	feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN ... Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup	2026-06-14 16:21:00 -06:00
jx12n	31ee612f27	feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4) ... Scale write throughput across data-shard groups while keeping a single unified read surface: - scatter_gather.rs: pooled fan-out across shard groups (replaces per-request client construction); cross-shard query results merged on one node - cluster/node.rs: cross-shard read routing — a read on any node gathers from every shard group's leader and unions results - cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM on the real mTLS plane) - dto.rs: cross-shard query/result DTOs - k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers, topology-configmap) - k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A - tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job - tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind - docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings	2026-06-14 15:17:35 -06:00
jx12n	81093a6779	bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling ... Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.	2026-06-13 21:40:58 -06:00
jx12n	df0e1b98de	feat(deploy+stress): m11p6/mTLS deploy fixes — HTTPS probes + TLS-aware generator ... Deploying the m11-44b768b image (p6 sharding + p7 mTLS + p8 ops + p9 correctness) surfaced two blockers; both fixed here. 1. statefulset.yaml: the m11p7 change made the :9500 HTTP plane serve TLS, but the startup/liveness/readiness probes still used scheme HTTP — kubelet got a TLS handshake back ("malformed HTTP response \x15\x03\x03") and pods never went Ready. Set scheme: HTTPS on all three probes (kubelet skips cert verification for httpGet probes, so the cert's DNS-only SANs are fine). Image pinned to the m11-44b768b amd64 digest. 2. tidal-stress: the generator's reqwest client did default cert verification and had no way to trust the cluster's private CA, so https:// targets failed. Added --ca-cert <pem> (verified TLS against the mounted tidaldb-cluster-tls ca.crt) and --insecure (skip verification, escape hatch). New StressError::CaCert for the PEM read fault. stress-job-m11p6-baseline.yaml: T2-A-equivalent quorum-write throughput run on the new stack — https:// targets, ca.crt mounted from the tidaldb-cluster-tls Secret, --ca-cert verified TLS. Drops the removed --write-path flag (m11p6 unified the write path to hash-routing).	2026-06-13 18:39:30 -06:00
jx12n	44b768b8c6	feat(m11): sharding × replication + rebalancing (m11p6 L3-L5) ... End the "replicated XOR sharded" split: S shard groups, each a replication group at RF with its own elected leader, leaders balanced across nodes; any gateway hash-routes. - One unified write surface: /items,/embeddings,/signals hash-route to the owning shard group's leader (ShardRouter FNV-1a) AND replicate at RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are per-group; NotLeader names the group. - Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced leadership move) + /cluster/shards/{id}/replicas (add/remove replica). A ?shard= selector threads through every per-shard admin verb and is propagated on intra-group forwards (ShardReplica::admin_path). S=1 is byte-for-byte (no selector, no shard in NotLeader body). - Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over real OS processes — SIGKILL a node under ack=quorum load → only its shard-leaderships re-elect, reads never stop, zero acked loss across random kill points; plus a rebalance-verb test. Harness: MultiProcCluster::start_sharded. - tidal-stress drives the single path (WritePath::Leader|Sharded gone), spreading writes round-robin across gateways or pinning --leader-url. - Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err, ~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A. Known follow-up (tracked): per-group-aware node readiness and cross-node read fan-out under PARTIAL placement.	2026-06-13 18:23:43 -06:00
jx12n	6651c14adc	feat(m11): cluster security (m11p7) + perf instrumentation floor ... m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte): - gRPC replication mTLS by default via a custom tokio-rustls acceptor + DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap, no pod restart, no inotify) - inter-node HTTP TLS sharing the same resolver (one rotation, both planes) + per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403 - admin audit log (operator-leg only) + per-principal rate limit (engine RateLimiter; sibling nodes exempt) - k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback; secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable) - exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs (HTTP foreign + zero-drop rotation under load), 7 security unit tests perf — instrument floor (sweep Wave 1): - new tidal/benches/wal.rs + tidal-server/benches/scatter.rs - p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md - add @tidal-performance agent (Martin Thompson) new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs, docs/planning/milestone-11/phase-7.md	2026-06-13 01:25:35 -06:00
jx12n	25ec7630a1	feat(k8s): m11p5 cluster manifest — local-path PVCs, initContainer, T3 tooling ... statefulset.yaml: - Pin image to m11p5 digest (173e803...) - Add initContainer init-datadir (busybox, uid 10001) to create /data/db before main - Set storageClassName: local-path (5Gi) — Longhorn networked-fsync caused 74-85% error rate T3 tooling: - tidal-stress/k8s/stress-job-t3.yaml: 1500 rps quorum-write job for election gate - tidal-stress/scripts/t3-kill-loop-v3.sh: HTTP-polling kill loop (port-forward + curl, 200ms poll, ns-precision timing, no exec into tidaldb pods) T3 result: 10/10 kills PASS, max 6157ms (gate < 10 000ms), zero acked loss.	2026-06-12 22:01:37 -06:00
jx12n	bf57be18e1	feat(m11): membership, snapshot install, and reseed (m11p5)	2026-06-12 19:55:54 -06:00
jx12n	1092d34c39	feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation ... - Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly	2026-06-09 17:06:34 -06:00