main
45 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
e117333fec |
ops(k8s): pin the fixed image, then park the GKE cluster at 0 with both blockers recorded
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Pipeline #19 finally produced an image once the 60-min repo timeout was raised to 180 (fast-suites passed at 60m26s — it would have died at the old wall for a third time). Pinned by digest in the OVERLAY, never the base, because the k3s fleet applies k8s/cluster/ directly and editing the base pin would silently repoint the fleet's production cluster as a side effect of a GKE change. Applied, and it does not reach Ready. Parked at replicas: 0 rather than deleted, because deleting the namespace destroys the self-signed root and stales every copy of it — the coupling documented one section above. The credentials, both certs and all three PVCs are retained, so resuming is one kubectl scale. Two blockers recorded in dependency order, because fixing them out of order wastes a 75-minute build: 1. The image came from the wrong build path. The runbook is explicit that scripts/build-release.sh must be used and the runtime must be trixie, because the binary needs libmvec.so.1 which is ABSENT on bookworm — and the CI Dockerfile's runtime stage is FROM debian:bookworm-slim. The pods boot on it anyway (WAL recovery, elections, HTTPS listener all fine), so nothing fails loudly; libmvec is vectorized math reached on the vector paths, not at startup. That is exactly why it is written down instead of trusting that it started. 2. A fresh cluster did not converge: all three pods Running with /health/live and /health/startup at 200, but /health 503 for 15 minutes on "shard group N has not yet first-converged against a KNOWN leader frontier". tidaldb-0 saw its own groups at leader_seqno 1 but group 1 at leader_seqno 0 and reseeding true, while both siblings had applied that term marker as followers. Readiness needs the converged flag, which note_lag_for_readiness will not set while leader_seqno == 0 ("absence of lag is not convergence", cluster/node.rs:3840). Blocker 2 is NOT called a bug here and must be re-tested on a correctly built image first — diagnosing a convergence stall against a runtime the runbook rejects is how a wrong root cause gets written down as fact. |
||
|
jordan
|
2c25b6d375 |
docs(k8s): record the CA-copy coupling for the GKE overlay
The consumer needs the cluster's CA and Secrets do not cross namespaces, so ca.crt has to be copied into peach's namespace. A copied certificate deserves suspicion — k3s-fleet's own history records an incident where the expired central SDLC wildcard was copied manually and never renewed into its consumer namespaces — so the numbers are written down rather than assumed: tidaldb-cluster-ca 87600h (10y), renewBefore 8760h (1y), isCA tidaldb-cluster-tls 2160h (90d), renewBefore 720h (30d) The 90-day LEAF is what rotates, roughly every 60 days, and it rotates under this same root, so the copied CA keeps verifying it with no attention. The fleet incident was an expiring leaf; this is a decade-lived root. The coupling that actually bites is recreation, not expiry: the root is self-signed and minted fresh on first apply, so destroying and re-applying the namespace invalidates every copy at once and every consumer request then fails TLS with nothing visibly wrong on either side. Recorded as a mandatory step of any recreate. |
||
|
jordan
|
19316227c5 |
feat(k8s): GKE overlay for the RF3 cluster, internal-only
thepeach's discover corpus should run the CLUSTER, not the standalone, and it
should run on GKE. Both halves are now evidenced rather than assumed.
Cluster, because the reason for the standalone is gone. The standalone was
chosen to work around cluster-mode `POST /signals` silently discarding
user_id/creator_id behind a 204 — fixed at the source in
|
||
|
jx12n
|
34cc5f08ca | fix(k8s): reconcile tidaldb StatefulSet resources with live cluster (300m/4Gi requests, 3/7Gi limits) | ||
|
jordan
|
2dc00538e8 |
discover: repin to m12-agesort-20260901
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
The discover profile is the reason the age-aware ranking work exists, but its kustomization still pinned m12-poisonfix-20260831 (sha256:93a2929d) -- the image on which score_hot reads no created_at at all and Sort::New scores by raw entity id. Deploying it against that digest would reproduce exactly the limitation its own NO sort: block documents. Necessary but not sufficient: cold-start still ties because hot's log10(max(views,1)) numerator is exactly 0.0 at 0 or 1 views, so the NO sort: decision stands and its STATUS: PENDING RE-MEASUREMENT is unchanged. No deployment performed. The StatefulSet stays replicas: 0 and targets ns thepeach-staging, which does not exist on orchard9-k3sf (zero *peach* namespaces); its base image comment says tidaldb:deploy is kind-loaded, so this is a local target, not a fleet one. |
||
|
jordan
|
0f18243a64 |
deploy: pin m12-agesort-20260901 (age-aware Hot/New live)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Digest 9234aacb, commit
|
||
|
jordan
|
a588f01f63 |
ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Three parallel reviews of
|
||
|
jordan
|
6385425a92 |
ranking: make Hot and New age-aware; fix the same gap in three more places
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
`score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added. |
||
|
jordan
|
8f9aad1fe0 |
deploy: pin m12-poisonfix-20260831 (both incident defects fixed)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Digest 93a2929d, commit
|
||
|
jordan
|
77f68d181c |
verify: flip the two log tripwires post-roll, calibrate the backup assertions
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Post-deploy half of the deploy-verification contract for m12-harden-20260831. Flipped, exactly as each assertion instructed its own successor to do: - 06-logs.spec.ts: asserted `jsonLines === 0`. JSON_LOGS is live, so it now asserts every sampled line parses as JSON. The ANSI check stays pinned at 0. - 09-operator-authority.spec.ts: asserted JSON_LOGS was absent from the StatefulSet. Now asserts JSON_LOGS=1 AND TIDAL_SERVICE_NAME=tidaldb, because the second is load-bearing: enabling structured logs makes the app's own `service` field win in the fleet's Vector normalize transform, silently renaming the log stream tidaldb -> tidal-server and blinding every query keyed on it. The fleet's _stream_fields contract pins field names but no legal values, so nothing there would have caught the flip. Calibrated, NOT loosened — the two backup assertions were unpassable by construction for ~25 minutes every day: - The schedule fires at 03:30 and measured runs take 9.1-24.8 min (n=15), so the newest object is legitimately InProgress during its own window. The "newest backup completed cleanly" test now selects the newest FINISHED backup; a namespace where nothing has ever finished still fails. - "no backup stuck in progress" asserted InProgress -> fail, full stop. It now bounds in-flight age at 60 min: ~2.4x the slowest success and a quarter of the 240.0 min timeout that the observed PartiallyFailed runs (2026-08-17/19/25) all hit. A gate that cries wolf on a schedule gets muted, and then it is not a gate. Both thresholds come from reading every backup in the namespace, not from a guess. Playwright 34/34 and hermetic semantics 5/5 against the deployed image. |
||
|
jordan
|
fe8d0c87e7 |
harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|
jordan
|
59d7dadc18 |
k8s(cluster): pin m12-vsc-20260830, now running on all three voters
Built clean from
|
||
|
jordan
|
53c345e890 |
k8s(cluster): pin the vector-grow image now running on all three voters
registry.threesix.ai/tidal/server:m12-vector-grow-20260830@sha256:2f2357eae5af
863804a85dba5514833c76730eeaac211e8da2cb60e797b34f92, built clean from
|
||
|
jordan
|
ef6e0b9636 |
k8s(cluster): pin the admin-gate image now running on all three voters
registry.threesix.ai/tidal/server:m12-admin-gate-20260823@sha256:6e220060a342 658b734d258245b20f6233d96e26415b3a44956b1c3bceebe48c, built from |
||
|
jordan
|
c9adec040d |
k8s(cluster): declare the limits production actually runs
The manifest said cpu 2 / memory 6Gi while the live StatefulSet ran cpu 3 / memory 7Gi, so `kubectl apply -k k8s/cluster/` DOWNGRADED production every time it was run - a silent capacity regression on the write path, delivered by the very command used to deploy a fix. Verified before and after with kubectl diff: the cpu/memory hunks are now absent, so an apply is a no-op on resources. The live values are the intended ones; this file is now the source of truth for them. The comment keeps the reasoning that matters - the limit exists to keep the measured query/apply burst reachable without reserving it, and when neighbours burst too the answer is more REQUEST, not more limit, because CFS throttling is how CockroachDB was pushed into multi-second Raft stalls on this fleet with nodes 70% idle. |
||
|
jordan
|
388e445a38 |
feat(cluster): separate operator authority from data-plane access
Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
|
||
|
jordan
|
0a861d9144 |
k8s(cluster): close the unauthenticated metrics and peer plane to foreign pods
Before this, ANY pod in the k3s cluster could read tidalDB's :9091 metrics - corpus size, seqnos, leader identity, all unauthenticated - and reach the peer gRPC plane. Measured, not assumed: scraping tidaldb-0:9091 from gitea-0 in namespace threesix returned metrics, and returns "connection refused" after this. NetworkPolicy enforcement on this k3s is therefore confirmed, not presumed. Ingress only. Egress is left unrestricted deliberately - the WAL-archival/S3 and peer dial-out surfaces are not fully enumerated, and a wrong egress rule partitions the cluster instead of merely blocking a scrape. :9500 stays open, and the file records why. All three probes (startup, readiness, liveness) target it, probes come from the NODE rather than a pod, and node-to-pod handling is CNI-specific - a wrong rule there fails liveness at 6x10s and restarts every pod. The exposure that would have closed is /cluster/status, which is unauthenticated by design. That is an engine defect and gets fixed in tidal-server, not worked around at the network layer. Verified after apply: scraper still collects 353 tidaldb_ series, a quorum-acked write through the public ingress returns 201, and all three regions report applied=13322229 lag=0 reachable partitioned=false with zero new restarts. |
||
|
jordan
|
087b83154a |
k8s(cluster): publish the client surface over public TLS, data routes only
tidaldb.threesix.ai now serves the cluster's data surface over a Let's Encrypt cert, verified end-to-end from the internet: 401 without a bearer, 401 with a wrong one, 200 with the real key, and a quorum-acked write returning 201 on all three node IPs. Three things this had to get right, each of which failed first: * The backend is HTTPS, not HTTP. Pods serve :9500 over TLS with the internal cluster CA whenever grpc_tls is configured, so a plaintext backend dial answers 500. Added a ServersTransport that VERIFIES that hop - every pod mounts the same tidaldb-cluster-tls leaf and its SANs include the client-Service DNS name, so serverName pinning validates it without insecureSkipVerify. * `service.*` annotations are read from the Service, not the Ingress. Putting serversscheme/serverstransport on the Ingress is silently ignored and presents exactly as a broken backend. * http01 cannot be used behind any gateway gate that rejects unknown callers, because it rejects the ACME challenge too. Uses the Cloudflare dns01 solver. Deliberately unpublished: /cluster/* (every mutating admin verb shares the SAME single bearer as the data routes, so a client key could remove members or transfer shards), /cluster/status (unauthenticated - leaks leader, membership, seqnos), /openapi.json (unauthenticated, enumerates the admin routes), and /metrics (only on the headless peers Service, unreachable here). Documents two controls that are NOT available and why: an IP allowlist cannot work while the shared Traefik Service runs externalTrafficPolicy=Cluster (svclb SNATs the client address), and Traefik basicAuth cannot stack in front of the bearer because both occupy the Authorization header. |
||
|
jordan
|
cc7066e0d8 |
k8s(cluster): pin the boot-pull-fix image now running on all three voters
registry.threesix.ai/tidal/server:m12-boot-pull-fix-20260821 @sha256:4150af1044b8084b84abeafedbbaedb1f084a2c84fd0600b56f5b04ed0697eba Rolled tidaldb-0/-1/-2 onto it via OnDelete with a quorum write probe between each step: every probe returned 201, so the roll cost zero write availability. Cluster is 3/3 with all nine group-replicas converged on identical frontiers (13322227/13540667/13072512) and an identical 33,331-vector corpus. PDB disruptionsAllowed is back to 1 after being 0 for the whole incident. updateStrategy restored to RollingUpdate/partition 0. |
||
|
jordan
|
da736b8eb2 |
fix(cluster): anchor the reseed-marker clear to the marker, not the leader tail
A follower that latched `reseed_required` from a genuine `snapshot-required` refusal could clear its own marker ~200ms later and so never run the boot reseed that was the only way to close the gap. With `replication.reseed_self_restart: true` it exit-looped: latch -> clear -> exit(0) -> boot with no marker -> re-latch. Production tidaldb-0 did this 196 times in 21h on 2026-08-20 while the cluster ran on 2 of 3 voters. `clear_stale_reseed_marker_if_caught_up(applied >= leader_last_seq)` compared the applied frontier against the LEADER'S TAIL and documented the invariant "a node genuinely behind a COMPACTED gap never reaches caught_up". That is false: on a quiet shard any node meets the leader's tail, including one missing committed history it can never refetch. The clear also reset `tidaldb_cluster_reseed_required`, so the gauge flapped 1->0 every 30s and `TidalDBClusterReseedPending` (`== 1 for 10m`) could never fire - the code path that broke the reseed also erased the signal that would have reported it. The discharge decision now belongs to the marker. `ReseedMarker::discharged_by` requires a stream-dischargeable reason AND an applied frontier that reached the marker's own `from_seqno` - the very entry whose absence latched it. A compacted gap can never satisfy that, so the reseed runs; a node merely behind a shippable tail satisfies it as soon as the stream serves that entry, so the m12 false-alarm self-heal still works (and now clears sooner, since it no longer waits to meet a moving leader tail). The two conditions previously shared `ReseedReason::SnapshotRequired`, so reason alone could not discriminate. The term-join arm's `frontier > baseline` case deliberately sets `from_seqno = baseline`, BELOW the node's own frontier, so a bare `applied >= from_seqno` would discharge it instantly - it holds divergent post-baseline data only a snapshot can discard. It gets its own never-lag- dischargeable reason, `DivergentPostBaseline = 3`. Adding a discriminant is the sanctioned forward-only extension; a downgrade that meets one refuses to decode it, per the existing kind-3/kind-4 precedent. The election-won call site passed a hardcoded `true`; it now passes the leader's durable flushed frontier (`applied_seqno` never advances on a leader), and a `DivergentPostBaseline` node is not campaign-suppressed so it can reach there. Tests: three deterministic predicate tests pinning the incident's exact seqnos (13540653 vs earliest-available 13540657), the false-alarm discharge, and the never-discharge of every structural reason. Pre-existing and NOT introduced here: cluster_reseed's `mp_follower_reseeds_via_snapshot_after_compaction` and `mp_graceful_rolling_restart_under_load_no_reseed` fail on baseline main (verified by stashing this change). The first is the owner-test for this exact mechanism - its leader compaction no longer forces a `snapshot-required`, so it never reached the clear path and never guarded it. Tracked separately. |
||
|
jordan
|
261d78d1f1 | k8s(cluster): pin the reconcile-limit image | ||
|
jordan
|
bb8425b0e8 | k8s(cluster): pin the frontier-tick image | ||
|
jordan
|
3eaf28bf8a |
k8s(cluster): roll to the frontier-fix image
Also carries the `tmp` fs-backup exclusion committed with the engine fix. |
||
|
jordan
|
12c7edc374 |
fix(cluster): the frontier pair has one writer, and 0 is not a commit index
`TidalDBClusterQuorumLag` sat CRITICAL all session against the live three-voter cluster while every region reported lag 0 and every per-peer ship queue was empty. Two independent defects fed it: - `observe_ship` bumped `relay_last_seq` on every batch ship while `relay_durable_seq` only moved when a signal write completed. The two are documented as a subtractable pair, so a shipping-but-not-committing node reported the whole relay log (13.3M events) as quorum lag. The ship path now feeds only its own per-peer queue-depth gauge; the pair has one writer. - `set_frontier_gauges` published `CommitIndex::committed()` verbatim, but that returns 0 as a SENTINEL for "no quorum information in this term yet". It now publishes both halves or neither, and every satisfied `await_quorum` -- not just signal writes -- refreshes them, so item and embedding workloads keep the pair live. Regression test asserts a busy ship loop leaves both halves at 0 (lag 0, not 13.3M) and that the single writer still moves them together. Also excludes the `tmp` emptyDir from velero fs-backup: three 0-byte PodVolumeBackups a night whose only other outcome is failing the whole fleet backup when a scratch file vanishes mid-snapshot. |
||
|
jordan
|
d923b036af |
k8s(cluster): three voters is the desired state, on the read-path fix image
Two outages in one session came from this file claiming `replicas: 0` while the cluster served traffic: a plain `kubectl apply` scaled it to zero, twice. Source now states production intent. Parking stays an explicit `kubectl scale` divergence recorded in k3s-fleet/cluster-state.yaml, and scripts/restore-fleet.sh remains the guarded path back - its storage, image, and per-node capacity preflights are exactly what a bare apply does not do. Pins server:m12-consumer-readpath-20260818@sha256:9191233d..., the build with the transport-aware read budget and the scatter-degraded counters. Verified after the rolling update: 3/3 Ready, all regions lag 0, every replica answering 3/3 shards with degraded=false on the default budget. |
||
|
jordan
|
897c6086f5 |
fix(cluster): size the read fan-out budget for the transport it crosses
Restoring the three-node cluster for a first production consumer surfaced this
immediately: EVERY cross-shard read came back
{"items":[...],"scatter_gather":{"degraded":true,
"unavailable_shards":["tidaldb-0","tidaldb-2"],"shards_queried":1,
"elapsed_ms":50,"shard_deadline_ms":45}}
HTTP 200, one shard of three, partial results. Replication itself was healthy -
/cluster/status showed all three regions reachable, lag_events 0, 13.3M events
applied each - so nothing in the quorum, election, or ship metrics moved.
Measured on the live cluster: a COLD peer fetch (TCP + TLS handshake + remote
1536-D search) takes ~50ms; a warm one takes ~1ms. DEFAULT_DEADLINE_MS is 50
(spec §7.4) and NETWORK_OVERHEAD_MS is 5, leaving a 45ms per-shard budget -
just under the cold cost. Proven by parameter sweep against one pod:
deadline_ms=50 -> degraded, 1/3 shards, 0 items
deadline_ms=250 -> healthy, 3/3 shards, elapsed 51ms
deadline_ms=1000 -> healthy, 3/3 shards, elapsed 1ms (warm)
The 50ms spec figure budgets a shard READ, not establishing a connection to
another pod. m11p7 put TLS on that hop and the default never followed, so the
first query after any rollout, idle period, or pod restart answered from a third
of the corpus. Fixed with a transport-aware default: 50ms in-process,
TLS_DEFAULT_DEADLINE_MS (250ms) once inter-node TLS is configured. An explicit
`?deadline_ms=` still wins in both directions, and MAX_DEADLINE_MS is unchanged.
The worse half was silence. A degraded fan-out is the one cluster failure that
answers 200 OK: the caller gets a ranked list assembled from a subset of the
corpus with `degraded: true` buried in response metadata. Nothing incremented,
so no alert could exist - a feed quietly ranking over one third of its
candidates looked identical to a healthy one. Added
tidaldb_cluster_scatter_degraded_total and
tidaldb_cluster_scatter_shard_unavailable_total, emitted from both HTTP fan-out
paths, so partial answers are now a countable correctness signal.
Also sizes the cluster StatefulSet for a consumer instead of the endurance gate:
requests 2 cores -> 300m per voter (limit 2 cores). The 2-core reservation was
the 200 rps soak envelope and needed 6,000m plus 2,000m free on each of three
PV-pinned nodes; the fleet is 82-91% committed, so that contract could not be
placed and the cluster stayed parked for a gate nobody is waiting on. 300m is
what the tightest pinned node can reserve, with the quorum/write-pool alerts as
the detector if real load outgrows it.
Tests: default_read_budget_covers_a_cold_inter_node_tls_hop pins the budget
against the measured cold hop and the explicit-override path; the cluster-metrics
render test covers both new counters.
|
||
|
jordan
|
c97aaa8e5b |
fleet remediation: make the workspace gate runnable, then fix what it caught
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|
jx12n
|
c22a3b65a6 |
docs: withdraw the pre-release "not ready for production" disclaimer
M0-M12 are shipped and the HA cluster runs in production on k3s, so the pre-release disclaimer no longer describes the project. Removes it from the canonical doc set and corrects the readiness text that had gone stale. - README.md: replace the "Pre-release / not yet recommended for production" banner with a production-ready statement; drop "(experimental)" from the cluster status bullet; state the post-1.0 versioning posture (additive in minor releases, breaking changes get a documented migration path). - CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md / docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a guard against standing up a multi-node fabric by accident rather than a readiness warning. - CHANGELOG.md: record the stability posture under [Unreleased], superseding the historical 0.1.0 "no stability guarantees" note (left intact as history). - k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead. Also corrects text that was factually wrong since m11p3/m11p4: the multi-process cluster gate, its CLI help, and the served OpenAPI description all still claimed quorum-ack writes and automatic failure detection did not exist. They do. Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries, the kubernetes.md rc7 fix note) are left unchanged. Verified against a running binary, not just the build: the opt-in gate's refusal message, the startup WARN, /health 200, and the served /openapi.json description all carry the new text. cargo fmt clean; clippy -D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155 server lib tests pass; scripts/check-docs.sh OK. Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP |
||
|
jx12n
|
0847c3d36f |
chore(k8s): roll cluster statefulset to m12-writeburst-rc7 (write-burst fix live)
Pin the live cluster to the rc7 amd64 image (@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1, 6-layer manifest, imagetools-verified) carrying the write-burst false-partition fix (tidal-net record_timeout). Deployed via `kubectl set image`; rolling update completed 3/3 Ready. Reseeds across the rollout were the safe snapshot-install fallback (rejoining behind WAL retention), all converged lag=0, no loop, no loss. |
||
|
jx12n
|
1b5bcbacd7 |
fix(net): classify ship deadline as timeout, not partition (write-burst false-partition)
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit
|
||
|
jx12n
|
a946c6128c |
fix(m12-rc13): read-SLA collapse + WAL_RETENTION_SEGMENTS 16 + tidalctl S3 DR
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning elections): - offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429) so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge fan-out (join_all) replaces the serial blocking offload_region_read loop - node.rs: scatter_merge → async; per-shard futures run via offload_search (each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the permit is held for the search's full CPU lifetime) - main.rs: explicit tokio runtime with worker_threads floored at 4, independent of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its own workers even when quota < 4 - k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async workers; search burst starved the reactor) - tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard catch-up window; a briefly-down follower across a rolling restart streams up instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming) - cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment retention window (19 segs > 17); fix sequential quarantine/reseed race via await_status_bool tidalctl S3/R2 backup DR: - tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile - commands/s3.rs: S3Target + export_dir (upload every file, manifest last as atomicity marker) + import_to_dir (download prefix into temp staging dir) - commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export is additive after local fsync barrier; S3 import stages into TempDir then runs the unchanged verified restore on it - main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing endpoint+bucket validation; usage updated tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor, soak-results-pvc, t5-readtput-job manifests |
||
|
jx12n
|
44ec87871e |
fix(m12p6): 7th-edge — correct reseed seqno + skip suspect HNSW graph on reseed-pending close (rc11)
Fixes the rc9 over-correction: forcing baseline for ALL nodes (including caught-up ones) caused needless reseed cascades. Now only divergent nodes (frontier > baseline) use baseline as the reseed seqno; at/below-baseline nodes use frontier+1 so the leader picks cheap catch-up vs snapshot. Also skips the HNSW graph checkpoint on SIGTERM when the shard is reseed- pending: the in-memory index reflects suspect/divergent data the next boot discards, so saving it risks a "Failed to read vectors" failure on the post-reseed open. Durable checkpoints and WAL flush still run. close_shared() gains a save_graphs bool; shutdown_inner_impl() is the shared implementation; node.rs passes !reseed_pending. |
||
|
jx12n
|
727fbfcb6b |
fix(m12p6): 6-bug k3s 3-shard cluster repair (rc8+rc9)
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9: 1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region` visits each hosted group's own shard subdir; per-group leader discovery appends `?shard=N` so a divergent shard heals from its own leader (not shard-0's WAL/term — cross-shard contamination). 2. leader self-join term (Bug 4): `become_leader_for_term` now calls `note_self_won_term` so the elected shard's `joined_term` is set and `cluster_promote` routes rebalances correctly (was: topology-era mis-read → legacy fenced promote → 500). 3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate prevents a node pulling its own stream (its stream isn't a registered peer) → eliminates the `PeerUnreachable(self)` loop. 4. scatter-merge degraded partial (Bug 1): failed shard logs + continues instead of `?`-failing the whole read; bounded read-admission semaphore (`offload.rs`) sheds as 429 instead of piling into a 36s p99. 5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4` most-recent sealed segments; online path gets the same retention clamp. Prevents brief-restart forced-reseed. 6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches `from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers` returns `needed=true` and the snapshot installs instead of looping. Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM (HNSW graph was not saved when request-scoped Arc clones were alive at shutdown); updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs. |
||
|
jx12n
|
a0399550d6 |
feat(m12p6): persist HNSW graph + bounded SIGTERM drain — boot loads, no rebuild
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
|
||
|
jx12n
|
4db3f1e597 |
fix(m12p6): complete T4 TLS scale-up — two-tier PKI + join_boot grpc_tls fallback
Completes the seed-join-over-TLS enablement begun in
|
||
|
jx12n
|
8e39ee1078 |
fix(m12p6): T4 TLS scale-up enablement — https seed-join via ready-only Service + up-front rustls provider
The m12p5 idle-readiness work converged on an idle cluster, but the real T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real blockers, all invisible to the plaintext in-process tests: - CryptoProvider crash-loop: the seed-join/reseed boot path builds a blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE GrpcTransport::new installs the process-wide provider, so every TLS joiner panicked. Install it at the top of main(); ensure_crypto_provider() is now pub, idempotent, harmless on the plaintext standalone path. - Wrong seed scheme + target: peer_url honors an explicit URL scheme verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now https:// AND points at the ready-only client Service (ClusterIP VIP), not the headless peers Service — so a joiner never round-robins onto a not-ready pod (incl. itself) and burns the 120s discovery window. - Too-tight poll budget: a cold status poll pays a full rustls handshake on top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so the joiner timed out every poll for the whole window despite the peer being reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS) with a separate 2s connect timeout (dead seeds still fail fast) and debug-level logging on every discovery failure mode. Refactors riding along: - on_heartbeat takes a HeartbeatContext struct (additive fields, no silent u64 transposition) across tidal-net, election_driver, and both test hooks. - ShardReplica::applied_for_leader_shard centralizes per-source-shard keying (BUG 1) shared by the readiness drive and local_status. - idle-readiness test now asserts convergence within ½ budget — a slow-path regression (periodic self-heal / status-poll dependency) the binary budget check would otherwise wave through. New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology patch; tidal-stress t4 seed/load Jobs. |
||
|
jx12n
|
aa94fd9b1f |
feat(m12p5): idle-readiness convergence via heartbeat live frontier + wildcard cert SAN
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq, proto field 14) so a snapshot-installed joiner converges its sticky readiness latch from the heartbeat — which flows even on a fully idle cluster — instead of only from observed ship traffic or an external status poll. Fixes the idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a caught-up joiner never joined the Service VIP). - proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall back to the status-poll readiness path - ElectionHooks::on_heartbeat threads leader_last_seq through net + driver - ShardReplica::note_leader_frontier_for_readiness folds the frontier into the lag gauge (monotonic per shard) and drives the readiness latch using a REAL leader frontier (never the uninitialized-0 gauge, which would false-converge a still-behind joiner); a joiner that WINS leadership converges trivially - tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic — snapshot joiner flips /health ready on an idle cluster with zero writes and no status poll, then proves content parity (honest convergence) - certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod names kept as belt-and-suspenders - docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup |
||
|
jx12n
|
31ee612f27 |
feat(m12p4): sharded ingestion — scatter-gather pool + cross-shard unified reads (L4)
Scale write throughput across data-shard groups while keeping a single unified read surface: - scatter_gather.rs: pooled fan-out across shard groups (replaces per-request client construction); cross-shard query results merged on one node - cluster/node.rs: cross-shard read routing — a read on any node gathers from every shard group's leader and unions results - cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM on the real mTLS plane) - dto.rs: cross-shard query/result DTOs - k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers, topology-configmap) - k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A - tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job - tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind - docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings |
||
|
jx12n
|
81093a6779 |
bench(1536): production-shape capacity — read path is cheap, quorum write is the ceiling
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production width) and ran the realistic peach mix (feed-profile reads + signal writes) on the m11p6 mTLS cluster. Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps (128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader pool, not vector size. The vector READ path (feed-profile retrieve — the db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps, never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput. Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA. Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md. |
||
|
jx12n
|
df0e1b98de |
feat(deploy+stress): m11p6/mTLS deploy fixes — HTTPS probes + TLS-aware generator
Deploying the m11-44b768b image (p6 sharding + p7 mTLS + p8 ops + p9 correctness)
surfaced two blockers; both fixed here.
1. statefulset.yaml: the m11p7 change made the :9500 HTTP plane serve TLS, but the
startup/liveness/readiness probes still used scheme HTTP — kubelet got a TLS
handshake back ("malformed HTTP response \x15\x03\x03") and pods never went
Ready. Set scheme: HTTPS on all three probes (kubelet skips cert verification
for httpGet probes, so the cert's DNS-only SANs are fine). Image pinned to the
m11-44b768b amd64 digest.
2. tidal-stress: the generator's reqwest client did default cert verification and
had no way to trust the cluster's private CA, so https:// targets failed. Added
--ca-cert <pem> (verified TLS against the mounted tidaldb-cluster-tls ca.crt)
and --insecure (skip verification, escape hatch). New StressError::CaCert for
the PEM read fault.
stress-job-m11p6-baseline.yaml: T2-A-equivalent quorum-write throughput run on the
new stack — https:// targets, ca.crt mounted from the tidaldb-cluster-tls Secret,
--ca-cert verified TLS. Drops the removed --write-path flag (m11p6 unified the
write path to hash-routing).
|
||
|
jx12n
|
44b768b8c6 |
feat(m11): sharding × replication + rebalancing (m11p6 L3-L5)
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
|
||
|
jx12n
|
6651c14adc |
feat(m11): cluster security (m11p7) + perf instrumentation floor
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
(HTTP foreign + zero-drop rotation under load), 7 security unit tests
perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)
new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
|
||
|
jx12n
|
25ec7630a1 |
feat(k8s): m11p5 cluster manifest — local-path PVCs, initContainer, T3 tooling
statefulset.yaml: - Pin image to m11p5 digest (173e803...) - Add initContainer init-datadir (busybox, uid 10001) to create /data/db before main - Set storageClassName: local-path (5Gi) — Longhorn networked-fsync caused 74-85% error rate T3 tooling: - tidal-stress/k8s/stress-job-t3.yaml: 1500 rps quorum-write job for election gate - tidal-stress/scripts/t3-kill-loop-v3.sh: HTTP-polling kill loop (port-forward + curl, 200ms poll, ns-precision timing, no exec into tidaldb pods) T3 result: 10/10 kills PASS, max 6157ms (gate < 10 000ms), zero acked loss. |
||
|
jx12n
|
bf57be18e1 | feat(m11): membership, snapshot install, and reseed (m11p5) | ||
|
jx12n
|
1092d34c39 |
feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md - Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router - Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example - Consolidate tidal/docker/ into root docker/ (single canonical home) - Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly |