Three real defects, plus a retracted fourth that was a probe artifact.
P3 (fixed) - query/stored normalization asymmetry. The write path L2-normalized
every stored vector; the read path passed the caller's raw query straight to the
index, so the two sides lived in different spaces. With unit v,
d = |q|^2 - 2q.v + 1, so a non-unit query shifted and scaled every distance by
|q|^2. Measured live: 591-1174 against a documented [0,4], and an exact match
scoring |q|^2 - 1 instead of ~0. vector_search_items now normalizes with the
canonical l2_normalize; a zero-norm query (no direction, so nearest-by-cosine is
undefined) is rejected with 400. Ranking is unchanged - |q|^2 and 1 are constant
across candidates - which is why it went unnoticed; what broke was every absolute
use of the number. WIRE-VISIBLE, recorded in CHANGELOG.
P2 (fixed) - the blob path had zero instrumentation. Added per-kind
tidaldb_cluster_blobs_originated/applied/apply_failed totals. Label cardinality
is fixed at 4 by construction via a new BlobKind enum, and BlobRecord::blob_kind
is now the ONE exhaustive match over the variants (kind() derives from it), so a
new variant is a compile error in one place instead of a silent zero in three.
Only the live apply path is counted - boot replay would inflate applied past
originated on every restart.
Coverage gap (fixed) - tidaldb_usearch_vector_count rendered only the metrics
owner's shard group, so on a 3-group node two thirds of the corpus had no
vector-count series at all. Co-located groups now render shard="N"; the owner
stays unlabeled for wire compatibility, so an alert grouped by (shard) buckets
each replica set separately without double-counting.
P1 (RETRACTED) - the "replica-divergent vector index" does not exist. Every probe
wrote through the /sharded/ surface, which hash-partitions and applies to the
owning region's local store with no WAL append, and therefore does not replicate
BY DESIGN (cluster/node.rs:8828-8829). A controlled A/B settled it: on /items plus
/embeddings all 6 entities reach all 3 replicas; on the sharded surface four of six
reach exactly one node. Both are now pinned by tests. See
tmp/vector-search-correctness/diagnosis.md and the k3s-fleet cluster-state.yaml
entry RETRACTED_blob_replication_rf1_2026_08_30.
Pre-work: usearch_index.rs 872 to 503 lines by extracting its tests to a sibling
(the project's existing path-attribute convention), and the three hand-rolled
l2_normalize copies collapsed to one. The two entity copies used a zero threshold
about 2900x looser than the canonical one; normalize_centroid now names the
centroid zero-tolerance policy once, and a test pins the tightened behavior.
Tests: 2107 lib (+5), 8 vector_search e2e (+4, three of which fail without the
P3 fix), 4 cluster_sharding e2e (+2). The heavy multiproc tests in
cluster_sharding are now serialized - four concurrent 3-node clusters made the
pre-existing failover test miss its 10s budget.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
