main
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
67a175e19a |
fix(cluster): /cluster/status reported a total partition on a healthy fleet
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
The status aggregator probed every peer's /cluster/status/local with NO credential. That route is token-gated, so on any cluster with TIDAL_API_KEY set each peer answered 401, and every peer row collapsed to the honest-unknown placeholder: reachable false, partitioned true, applied_events null, lag_events null, version "". Only the OWN region survived, because it is served in-process with no HTTP hop. The result: the one surface an operator reads to clear the N/N+1 version skew before a rolling upgrade - and the deploy runbook's own step 4 - reported the whole cluster partitioned while it was perfectly healthy, with every peer's version blank so the skew check was blind. Observed on the GKE cluster: all three pods answered /health 200 under leader tidaldb-1, replication applied, and a curl between the exact same pod FQDNs returned 200, while /cluster/status insisted both siblings were unreachable. security::bearer_from_env documents this precise trap - a node that "dials an authenticated peer with NO credential" - and count_alive_other_voters already attaches the bearer. This forwards the CALLER's Authorization header instead of reaching for creds.bearer(), matching the relayed-operator-hop convention that /cluster/promote already uses, so a weakly-authenticated caller cannot borrow the node's own credential to read peers it could not read directly. Why it escaped: every other multi-process test runs with no TIDAL_API_KEY, where a credential-less probe succeeds - including cluster_multiproc's all-reachable assertion. The new test carries the key. Verified differential: it fails on the reverted code with exactly the observed shape (us-east reachable, both peers null/false/empty) and passes with the fix. cluster_multiproc still 5/5. |
||
|
jordan
|
a6f663f002 |
harden: validate embeddings before the WAL, fix the reseed-latch leak, run every test suite
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Fixes the two defects a malformed probe exposed on the live cluster, plus the
coverage gap that let a stale assertion survive the same day it was falsified.
TASK 17 — validate before the WAL append. A 128-dim vector against a 1536-dim
slot was appended to the WAL FIRST, then validated, then answered 500 — so an
already-durable, unapplicable record shipped to both followers, halted both
receivers, and put shard 1 into a quorum-write outage. Validation now runs before
the append and returns 400 via invalid_input; nothing enters the log.
`storage::vector::validate_dimensions` is now the single comparison, replacing an
inline duplicate of the same rule in lifecycle/ops.rs:57-62 — two copies of a
dimension check drift, and the apply-path copy is the one that halts replication
when it disagrees.
The receiver's halt-vs-skip decision is now explicit instead of "halt on
anything". A record whose failure is deterministic and node-independent (schema
width) is skipped, counted on blobs_apply_failed_total and ERROR-logged, so the
frontier advances; a record that could become applicable after a binary upgrade
(unknown batch kind, capability skew) still halts, because skipping那 would
silently drop replicated data. Both branches are proven reachable by tests.
TASK 18 — the reseed latch outlived its discharge. A node hosting 3 shard groups
latched a marker per group but discharged on a single seqno, so two latches meant
permanent 503 on a node whose every shard read lag 0 — it hit all three pods
during the roll and each needed a manual delete. Gaps are now tracked per group
in a ReseedGapSet and cleared on evidence about themselves; a REFUSED
reseed_self_restart re-evaluates every 15s instead of waiting for a latch that
never arrives. /health's cause ladder was also lying: it printed "joiner boot not
yet converged" for a node whose groups had all converged, because the fallback
asserted a state it never tested. It now names the outstanding gaps, gained the
decommissioned-by-signal arm that is_ready checked but the ladder did not, and
its terminal arm says "reason unavailable" rather than inventing one.
COVERAGE — 14 of 23 integration suites were run by NO pipeline. Not theoretical:
cluster_routes still asserted the wire fabrication removed hours earlier
(applied_events == 0 with a lag derived from it) and nothing caught it because
nothing ran it. cluster_sharding (dense-rank, /sharded/* opt-in), vector_search
(distance contract) and cluster_poison_embedding (task 17's own gate) were in the
same position, so those guards would have rotted identically. Every suite now has
a runner: 8 in-process ones in a new `fast-suites` push step (measured 71s, runs
FIRST so a cheap failure precedes the 6.5-min gate), 6 multiproc ones in the
nightly. All 23 scheduled; all 4 never-before-run heavy suites verified passing
before being scheduled.
Also fixes cluster_chaos.rs:329, which the nightly's FIRST EVER run caught 13
minutes in — it demanded an unreachable peer report worst-case lag, i.e. it
required the fabrication task 04a deleted.
Verified: fmt clean; clippy 72 vs 73 baseline (one FEWER, zero added, measured on
touched trees at
|
||
|
jordan
|
fe8d0c87e7 |
harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|
jordan
|
388e445a38 |
feat(cluster): separate operator authority from data-plane access
Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
|
||
|
jx12n
|
8a0950260f |
feat(m8p10): multi-process cluster mode — scatter-gather, reconcile relay, chaos/UAT suites
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites (chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP updated with G4/G5/G6 known gaps. |