`POST /signals` on a cluster node applied (signal, entity, weight) and
dropped `user_id`/`creator_id` while still answering 204. Everything
user-scoped was silently lost: hard negatives, seen tracking, (user,
creator) interaction weight, preference vector, per-user trending index,
cohort attribution, community forwarding. A clustered deployment looked
healthy, accepted every behavioural signal, and learned nothing — with no
wire evidence of the loss.
This was known and worked around rather than fixed: `dto.rs` documented the
fields as "(standalone path only)", and thepeach's staging tofu pinned its
discover corpus to a STANDALONE instance citing this exact defect
(infra/tofu/envs/staging/svc_api.tf). thepeach's client does populate both
fields (crates/tidaldb-client/src/lib.rs), so the loss was live, not
theoretical.
Cause: `stage_signal_local` called `signal_staged`, which takes no context.
`signal_with_context` (standalone) takes both and runs the side effects
after the base write.
Fix, engine side. Extract the two halves of `signal_with_context` that were
not reusable: `validate_context_entity` (the u32 item-slot guard, which must
reject BEFORE anything is written — a truncated id in a durable Tag::HardNeg
row is a permanent cross-item collision that survives restart) and
`apply_signal_context` (every side effect). `StagedSignal` now optionally
carries the context and applies it in `wait()` AFTER durability, so the
ordering matches the synchronous path and a crash cannot leave a side effect
whose base signal was never logged. `signal_with_context_staged` is the new
entry point; both staged constructors share one admission sequence.
Not routed through the synchronous `signal_with_context` on purpose: that
would serialise every context-carrying write on its own fsync, and since
essentially every real signal carries a user, it would have cost the whole
m11p1 group-commit win on the hot path.
Fix, server side. `stage_signal_local` takes the two ids and ALWAYS uses the
context-carrying stage — it degrades to the plain staged write when both are
None, so there is no second path to keep in step. Both production handlers
(`write_signal`, `sharded_write_signal` in cluster/node.rs, served by
`build_region_router`, which is what `run_seed_join_cluster` and
`run_region_cluster` boot — the deployed RF3 topology) now pass them.
The experimental single-process router (cluster/routes.rs, `SimulatedCluster`
+ scatter_gather) genuinely cannot honour context: its relay applies
(signal, entity, weight). It now REFUSES such a request with 400 naming the
supported route, rather than accepting and discarding. A 204 over a dropped
user_id is the failure mode that caused this.
Verified, not assumed: reverted the fix to context-always-None and the new
differential tests fail exactly as the bug did — observed (false, false,
0.0) against the synchronous path's (false, true, 1.0), i.e. no seen bit and
no interaction weight. Tests are differential (staged vs synchronous end
state, with positive controls so two empty states cannot agree) precisely
because that is the check the original bug got past. Existing suites: 2144
engine + 197 server, zero failures.
`cargo test --workspace` could not run at all: dependency resolution failed with
"aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the
gate the project documents was dead. Making it run exposed a compile break and
two wrong tests that had been invisible for months. Now green end to end:
143 suites, 3155 tests, exit 0.
Toolchain
- rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays
`rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK
chain needs the patch release, and it now declares that itself.
Consumer crates migrated to the current engine API (clean cutover)
- iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields;
the literal now spreads `..AgentPolicy::default()` as the engine's own doc
example does, so future fields do not break it again.
- forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own
candidate pool, so it now tags what it knows: PreferenceMatch for the
preference-vector blend, SemanticMatch (with the seed item) for
similar-to-saved, ExplorationBudget for pinned discoveries.
- forage-engine: `url_to_item_id` folded into the u32 item universe. The engine
narrows item IDs to a u32 slot in durable per-user state and rejects anything
above u32::MAX rather than alias two items forever, so every add_item with a
64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone.
- forage-engine: bridge items read the top-2 preference CLUSTERS via
`query_vectors`, not the single centroid from `preference_vectors().get()`.
Since m12 that accessor returns only the strongest cluster, so a tech+jazz user
whose interests split into two clusters looked single-interest and never
bridged. Falls back to top-2 dimensions when a user has one cluster.
Reconcile tests corrected to the shipped contract
- tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed
count after heal. `take_crdt_snapshot` deliberately keys signal contributions
to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from
a single writer, so per-node attribution double-counted every replicated event
on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus
PN-counter per-node max: nodes converge on the more complete accumulator. The
old expectation was asserting the bug that fix removed.
- Rewrote to assert convergence, count survival (not 0), and no inflation, and
added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression
guard for the creep itself, which nothing covered.
Pre-commit hook unified
- hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the
source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the
flag promoted ~58 deliberate pedantic warnings in integration tests to errors,
making every Rust commit impossible.
- It now lints all five tidal crates instead of path-matching `tidal/`, which
silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and
applications/ - the rot above lived in exactly those crates. Ported the
CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the
divergent untracked copy that this replaces.
- CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split.
Fleet recovery and soak
- scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an
ignored tmp/ directory into the repository. Preflights retained storage,
digest-pinned images, parked state, and aggregate plus per-PV-node scheduler
headroom before the first scale; writes a durable transcript under
tmp/restore-logs/ with structured start/error/rollback/complete events.
- k8s manifests park the standalone store, the RF3 cluster, and the soak monitor
at zero replicas with restore-fleet.sh as the only supported scale-up path.
- soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing
restart evidence instead of silently skipping the restart-aware half of the gate.
- docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier
formula and separates analytic totals from the measured process envelope.
Reconciles two independently-developed lines from base 006d3d0:
ours — M9/M10 community layers, retroactive purge + re-materialization,
signal revocation, agent capability boundaries, P1 feedback loop,
reason labels, instrumented metrics
theirs — M11/M12 cluster mode (tidal-net gRPC transport, tidal-server
cluster/scatter-gather, tidal-stress), multi-vector preference,
ANN candidate-gen, warm-tier day buckets, keyed signal snapshots
Notable semantic resolutions:
* storage::keys::Tag — both sides allocated 0x0E..0x11 for different
records. Kept theirs' 0x0E..0x1A (shipped on-disk format) and renumbered
ours to 0x1B..0x1E (CommunityMembership/Revocation/PurgeManifest/
CommunityLeave); Tag::ALL grown to 30 so the contiguity drift guard holds.
* ranking executor — took theirs' rewrite (SignalReadPlan pre-pass, keyed
SignalKey snapshots, Result-returning reads, finalize()) and re-applied
ours' M10 read suppression at the chokepoints it introduced:
single_signal_score, score_hot/trending/controversial,
CreatorEngagementRate, and the Stage-4 boost loop.
* signals::warm — theirs' day-bucket/read-time-rotation rewrite, with ours'
subtract_bucket and Clone extended to the new day tier; ours' test split
kept (warm/tests.rs, warm/proptests.rs) carrying theirs' updated bodies.
* db::signals — kept ours' contribution-logging try_cohort_attribution in
signal_dispatch.rs and theirs' event-time try_update_preference_vector;
dropped the superseded duplicates.
* db::mod / from_parts — theirs' constructors, with ours' purge/
re-materialization/revocation/community/skip-counter fields and restart
rebuilds; from_parts kept in its own file per the 600-line guideline.
* schema::validation::builders — ours' module split with theirs' expanded
tests; policy validation runs both sides' checks (read-signal lists +
profile overrides, then the zero-duration limit guard).
* feedback Unhide no longer writes a -1.0 "hide" signal: theirs' engine
rejects negative weights (spec §8). Reverses index state only, matching
every other undo action.
* SessionState::new is now the single construction path (gains
overrides_rejected/default_profile); AuditEntry gains kind on the
deserialize path, inferred from the accepted flag as before.
* Removed tidal/src/replication/tcp_transport.rs and its test: never
declared in replication/mod.rs on either branch, so it had never
compiled and nothing referenced it. Superseded by tidal-net's
GrpcTransport.
Verified: cargo clippy -p tidaldb (lib) clean; --all-targets compiles for
tidaldb/tidal-net/tidal-server/tidal-stress; 2094/2094 lib tests and the
integration suite pass except m8p3_reconcile_production's two CRDT-count
assertions, which fail identically on MERGE_HEAD (pre-existing).
tidalctl cannot build locally: its aws-sdk deps need rustc 1.91.1, local
toolchain is 1.91.0.
Root cause: after a checkpoint-based snapshot install the engine WAL is empty,
so wal_term_mark() reports tail_term=0. decide_join compares (tail_term, frontier)
lexicographically — tail_term FIRST — so 0 < leader_term classifies the reseeded
shard ReseedRequired on EVERY boot regardless of the correctly-seeded frontier,
re-latching the marker and self-restarting forever. Observed live on tidaldb-2:
30 CrashLoopBackOff restarts, leader tidaldb-1 term 5, baseline=536647, the
frontier seeded correctly (from_seqno=536647) yet the loop persists because the
(tail_term, frontier) compare never reaches the frontier.
Fix 1 (already in tree): seed the post-open frontier from sentinel.snapshot_seq,
not last_wal_seq() (which a checkpoint restore leaves at 0).
Fix 2 (loop-breaker): durably synthesize the artifact's kind-3 TERM_MARKER WAL
record in the post-open reseed seed, at the artifact's captured term + the
reseed-leader region (threaded through an extended 18-byte install sentinel,
back-compat with 10/8-byte). Makes wal_term_mark() truthful on this boot AND
every reboot (blob records are NOT checkpoint-filtered on recovery), so
decide_join returns Clean. Truthful, not a bypass: the artifact IS the leader's
authoritative state at (term, seq); a genuinely-divergent node (no install
sentinel) still surfaces tail_term > term -> Quarantine. Crash-idempotent via a
monotonic-by-term guard.
Fix 3: node-level reseed-restart coordinator — the single process-wide exit fires
once, only after every hosted shard requests a restart or a bounded grace
elapses, so one shard's self-restart never aborts a co-hosted sibling's
in-flight install (S>1). No-op on the S=1 production topology.
Fix 4: is_ready() returns 503 while any reseed marker (SnapshotRequired or
Quarantine) is latched, closing the plain-restart serve-while-behind gap;
readiness is bounded staleness, not "ready the instant the process is up".
Tests: decide_join loop/fix/bounded-reseed unit; install-sentinel 18-byte
round-trip + back-compat; engine durability (term marker survives a checkpoint
advanced past it + crash-reopen); reseed-restart gate (5 cases); reseed_install
carries the term. Verified: cluster_reseed 4/4 (zero-loss rolling restart x2,
quarantine reseed, failover oracle), reseed_install 3/3, m12_reseed_term_marker
4/4, cluster_membership mp_idle/mp_dns/mp_remove x2, tidal-server lib 154/154.
mp_scale_3_5_3 and mp_seed_join_snapshot_catchup OOM on this host (22GB colima
VM); their /health/startup failure is process-down, not the is_ready path Fix 4
touches.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
apply_crdt_state force-sets accumulated scores outside apply_event_local,
so it bypassed the single note_write() chokepoint that keeps the trending
top-K cache fresh. A node going quiescent right after a partition heal kept
serving the pre-reconciliation candidate set indefinitely. Invalidate the
cache in apply_crdt_state too, with a regression test that reconciles a new
high-score entity and requires it in the next candidate read.
Also: extract the for_you/related ANN candidate cap to a named constant
(ANN_PROFILE_CANDIDATE_LIMIT), keep the recall test's ef_search in lockstep
with its construction default (honored since m12p3), and clarify the
vector_search region doc (always null until m12p4 cross-shard reads).
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
Three changes closing M8 gaps identified during verification:
1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE).
Added Known Gaps table (G1: in-process transport, G2: tier-3
tests, G3: hash inconsistency).
2. SimulatedCluster transport now pluggable via ClusterConfig.transports.
Default (None) uses new ChannelTransport (crossbeam, same behavior).
When Some, accepts external transports (e.g., GrpcTransport from
tidal-net). Updated redeliver_missed to use &dyn Transport.
Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged.
3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs).
ClusterHarness spawns real tidal-server cluster OS processes,
allocates dynamic ports, generates topology YAML, polls health,
and cleans up via SIGTERM. Two tests: smoke (write + converge +
verify follower reads) and promote (leader change + continued writes).
Feature-gated behind cluster-e2e.
After a pod restart, in-memory indexes (universe bitmap, category,
format, creator, tags, duration, created_at) were empty because they
were only populated by write_item_with_metadata() calls. This caused
/health to report items: 0 and queries to return no results despite
data persisting on disk in fjall storage.
Add rebuild_item_indexes() which scans the items keyspace for Tag::Meta
entries on startup and repopulates all indexes from stored metadata.
Update m2_uat crash recovery test to assert indexes survive restart
without rewriting items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deserialize_start_record now returns the full (session_id, user_id,
started_at_ns, metadata) tuple — the metadata bytes were already written
by serialize_start_record but silently discarded on read.
restore_session_wal_events looks up the persisted start record in storage
for each open session and uses the deserialized metadata instead of
HashMap::new(), so fields like {"tool":"planner"} survive a crash.
The signal replay loop no longer discards _annotation — annotations are
now pushed into state.annotations during WAL replay, restoring preference
hints like "more jazz today" so FOR SESSION ranking works post-restart.
Two new integration tests in session_durability.rs verify both fixes
against a real persistent store with simulated crash (drop without
close_session). session/serde.rs split into serde/mod.rs + serde/start_record.rs
to satisfy the 600-line limit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- m0p3: CONTRIBUTING.md with run-samples checklist, all 4 examples
(quickstart, cli_embedding, axum_embedding, actix_embedding), doc-test
coverage for every public API surface
- m1p5: TidalDb public API — write_item, signal, read_decay_score,
read_windowed_count, read_velocity; StorageBox enum routing memory vs
fjall; WalSender/WalHandleWriter bridge; WAL replay on open
- Periodic checkpoint: 30s background thread for persistent+schema mode;
FjallBackend::Clone (O(1), fjall::Keyspace is ref-counted); graceful
shutdown via Arc<AtomicBool> + join before final checkpoint
- ROADMAP.md: M0 and M1 fully marked COMPLETE (341 tests passing)
- Milestone 2 planning scaffolding added under docs/planning/milestone-2/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
