67a175e19a
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
fe8d0c87e7 |
harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|
jordan
|
8aa1fbb414 |
vector search: normalize the query, instrument the blob path, expose per-group vector counts
Three real defects, plus a retracted fourth that was a probe artifact. P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized every stored vector; the read path passed the caller's raw query straight to the index, so the two sides lived in different spaces. With unit v, d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by |q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant across candidates - which is why it went unnoticed; what broke was every absolute use of the number. WIRE-VISIBLE, recorded in CHANGELOG. P2 (fixed) - the blob path had zero instrumentation. Added per-kind tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind is now the ONE exhaustive match over the variants (kind() derives from it), so a new variant is a compile error in one place instead of a silent zero in three. Only the live apply path is counted - boot replay would inflate applied past originated on every restart. Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics owner's shard group, so on a 3-group node two thirds of the corpus had no vector-count series at all. Co-located groups now render shard="N"; the owner stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets each replica set separately without double-counting. P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe wrote through the /sharded/ surface, which hash-partitions and applies to the owning region's local store with no WAL append, and therefore does not replicate BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus /embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six reach exactly one node. Both are now pinned by tests. See tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml entry RETRACTED_blob_replication_rf1_2026_08_30. Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling (the project's existing path-attribute convention), and the three hand-rolled l2_normalize copies collapsed to one. The two entity copies used a zero threshold about 2900x looser than the canonical one; normalize_centroid now names the centroid zero-tolerance policy once, and a test pins the tightened behavior. Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in cluster_sharding are now serialized - four concurrent 3-node clusters made the pre-existing failover test miss its 10s budget. |
||
|
jx12n
|
44b768b8c6 |
feat(m11): sharding × replication + rebalancing (m11p6 L3-L5)
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
|