Found by running the standalone server end to end: after a clean shutdown that
logged "persisted HNSW graphs to disk (next boot loads instead of rebuilding)",
the next boot logged
WARN persisted HNSW graph failed to load; falling back to rebuild
slot="content_vector" error=USearch load failed: Failed to read vectors
and rebuilt the index. Every boot. Correct results, no data loss, a WARN nobody
reads - and the optimization was dead.
Cause: `checkpoint_graphs` saves whichever index the slot holds, and
`build_slot_index` holds a BRUTE-FORCE index below the dimension-aware
crossover. Both write to the same `<kind>__<slot>.usearch` path, so a small slot
persisted a `BFVI` file that `load_persisted_slot` then handed to the USearch
reader. Observed on a 6-item, 128-D store; at production scale the slot is
USearch-backed, which is why the cluster never surfaced it - but every dev,
staging, and small-tenant instance pays a full index rebuild on every start, and
that rebuild is what the 20-minute startup probe budget exists for.
Fix: sniff the file's magic (brute-force `MAGIC` is now `pub(crate)` so the
reader and writer cannot drift) and dispatch to the matching loader. A
brute-force graph is still rejected when `expected_count` has grown past the
crossover, so the rebuild upgrades the backend to HNSW rather than pinning a
linear scan forever. Log lines now say "vector graph" and name the backend
instead of claiming HNSW for both.
Tests: `brute_force_slot_graph_round_trips_through_registry_persistence` covers
the case that was broken (the existing round-trip test forces USearch, which is
why it passed throughout), and
`grown_corpus_rejects_a_brute_force_graph_so_the_rebuild_upgrades_it` pins the
upgrade path. Verified live: the same data directory that produced the WARN now
logs `loaded persisted vector graph from disk (skipped full rebuild)
backend="brute-force" count=6`, with feed, text search, and vector search all
returning the pre-restart results.
Reconciles two independently-developed lines from base 006d3d0:
ours — M9/M10 community layers, retroactive purge + re-materialization,
signal revocation, agent capability boundaries, P1 feedback loop,
reason labels, instrumented metrics
theirs — M11/M12 cluster mode (tidal-net gRPC transport, tidal-server
cluster/scatter-gather, tidal-stress), multi-vector preference,
ANN candidate-gen, warm-tier day buckets, keyed signal snapshots
Notable semantic resolutions:
* storage::keys::Tag — both sides allocated 0x0E..0x11 for different
records. Kept theirs' 0x0E..0x1A (shipped on-disk format) and renumbered
ours to 0x1B..0x1E (CommunityMembership/Revocation/PurgeManifest/
CommunityLeave); Tag::ALL grown to 30 so the contiguity drift guard holds.
* ranking executor — took theirs' rewrite (SignalReadPlan pre-pass, keyed
SignalKey snapshots, Result-returning reads, finalize()) and re-applied
ours' M10 read suppression at the chokepoints it introduced:
single_signal_score, score_hot/trending/controversial,
CreatorEngagementRate, and the Stage-4 boost loop.
* signals::warm — theirs' day-bucket/read-time-rotation rewrite, with ours'
subtract_bucket and Clone extended to the new day tier; ours' test split
kept (warm/tests.rs, warm/proptests.rs) carrying theirs' updated bodies.
* db::signals — kept ours' contribution-logging try_cohort_attribution in
signal_dispatch.rs and theirs' event-time try_update_preference_vector;
dropped the superseded duplicates.
* db::mod / from_parts — theirs' constructors, with ours' purge/
re-materialization/revocation/community/skip-counter fields and restart
rebuilds; from_parts kept in its own file per the 600-line guideline.
* schema::validation::builders — ours' module split with theirs' expanded
tests; policy validation runs both sides' checks (read-signal lists +
profile overrides, then the zero-duration limit guard).
* feedback Unhide no longer writes a -1.0 "hide" signal: theirs' engine
rejects negative weights (spec §8). Reverses index state only, matching
every other undo action.
* SessionState::new is now the single construction path (gains
overrides_rejected/default_profile); AuditEntry gains kind on the
deserialize path, inferred from the accepted flag as before.
* Removed tidal/src/replication/tcp_transport.rs and its test: never
declared in replication/mod.rs on either branch, so it had never
compiled and nothing referenced it. Superseded by tidal-net's
GrpcTransport.
Verified: cargo clippy -p tidaldb (lib) clean; --all-targets compiles for
tidaldb/tidal-net/tidal-server/tidal-stress; 2094/2094 lib tests and the
integration suite pass except m8p3_reconcile_production's two CRDT-count
assertions, which fail identically on MERGE_HEAD (pre-existing).
tidalctl cannot build locally: its aws-sdk deps need rustc 1.91.1, local
toolchain is 1.91.0.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
- m0p3: CONTRIBUTING.md with run-samples checklist, all 4 examples
(quickstart, cli_embedding, axum_embedding, actix_embedding), doc-test
coverage for every public API surface
- m1p5: TidalDb public API — write_item, signal, read_decay_score,
read_windowed_count, read_velocity; StorageBox enum routing memory vs
fjall; WalSender/WalHandleWriter bridge; WAL replay on open
- Periodic checkpoint: 30s background thread for persistent+schema mode;
FjallBackend::Clone (O(1), fjall::Keyspace is ref-counted); graceful
shutdown via Arc<AtomicBool> + join before final checkpoint
- ROADMAP.md: M0 and M1 fully marked COMPLETE (341 tests passing)
- Milestone 2 planning scaffolding added under docs/planning/milestone-2/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
