Root cause: after a checkpoint-based snapshot install the engine WAL is empty,
so wal_term_mark() reports tail_term=0. decide_join compares (tail_term, frontier)
lexicographically — tail_term FIRST — so 0 < leader_term classifies the reseeded
shard ReseedRequired on EVERY boot regardless of the correctly-seeded frontier,
re-latching the marker and self-restarting forever. Observed live on tidaldb-2:
30 CrashLoopBackOff restarts, leader tidaldb-1 term 5, baseline=536647, the
frontier seeded correctly (from_seqno=536647) yet the loop persists because the
(tail_term, frontier) compare never reaches the frontier.
Fix 1 (already in tree): seed the post-open frontier from sentinel.snapshot_seq,
not last_wal_seq() (which a checkpoint restore leaves at 0).
Fix 2 (loop-breaker): durably synthesize the artifact's kind-3 TERM_MARKER WAL
record in the post-open reseed seed, at the artifact's captured term + the
reseed-leader region (threaded through an extended 18-byte install sentinel,
back-compat with 10/8-byte). Makes wal_term_mark() truthful on this boot AND
every reboot (blob records are NOT checkpoint-filtered on recovery), so
decide_join returns Clean. Truthful, not a bypass: the artifact IS the leader's
authoritative state at (term, seq); a genuinely-divergent node (no install
sentinel) still surfaces tail_term > term -> Quarantine. Crash-idempotent via a
monotonic-by-term guard.
Fix 3: node-level reseed-restart coordinator — the single process-wide exit fires
once, only after every hosted shard requests a restart or a bounded grace
elapses, so one shard's self-restart never aborts a co-hosted sibling's
in-flight install (S>1). No-op on the S=1 production topology.
Fix 4: is_ready() returns 503 while any reseed marker (SnapshotRequired or
Quarantine) is latched, closing the plain-restart serve-while-behind gap;
readiness is bounded staleness, not "ready the instant the process is up".
Tests: decide_join loop/fix/bounded-reseed unit; install-sentinel 18-byte
round-trip + back-compat; engine durability (term marker survives a checkpoint
advanced past it + crash-reopen); reseed-restart gate (5 cases); reseed_install
carries the term. Verified: cluster_reseed 4/4 (zero-loss rolling restart x2,
quarantine reseed, failover oracle), reseed_install 3/3, m12_reseed_term_marker
4/4, cluster_membership mp_idle/mp_dns/mp_remove x2, tidal-server lib 154/154.
mp_scale_3_5_3 and mp_seed_join_snapshot_catchup OOM on this host (22GB colima
VM); their /health/startup failure is process-down, not the is_ready path Fix 4
touches.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
apply_crdt_state force-sets accumulated scores outside apply_event_local,
so it bypassed the single note_write() chokepoint that keeps the trending
top-K cache fresh. A node going quiescent right after a partition heal kept
serving the pre-reconciliation candidate set indefinitely. Invalidate the
cache in apply_crdt_state too, with a regression test that reconciles a new
high-score entity and requires it in the next candidate read.
Also: extract the for_you/related ANN candidate cap to a named constant
(ANN_PROFILE_CANDIDATE_LIMIT), keep the recall test's ef_search in lockstep
with its construction default (honored since m12p3), and clarify the
vector_search region doc (always null until m12p4 cross-shard reads).
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
Three changes closing M8 gaps identified during verification:
1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE).
Added Known Gaps table (G1: in-process transport, G2: tier-3
tests, G3: hash inconsistency).
2. SimulatedCluster transport now pluggable via ClusterConfig.transports.
Default (None) uses new ChannelTransport (crossbeam, same behavior).
When Some, accepts external transports (e.g., GrpcTransport from
tidal-net). Updated redeliver_missed to use &dyn Transport.
Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged.
3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs).
ClusterHarness spawns real tidal-server cluster OS processes,
allocates dynamic ports, generates topology YAML, polls health,
and cleans up via SIGTERM. Two tests: smoke (write + converge +
verify follower reads) and promote (leader change + continued writes).
Feature-gated behind cluster-e2e.
After a pod restart, in-memory indexes (universe bitmap, category,
format, creator, tags, duration, created_at) were empty because they
were only populated by write_item_with_metadata() calls. This caused
/health to report items: 0 and queries to return no results despite
data persisting on disk in fjall storage.
Add rebuild_item_indexes() which scans the items keyspace for Tag::Meta
entries on startup and repopulates all indexes from stored metadata.
Update m2_uat crash recovery test to assert indexes survive restart
without rewriting items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deserialize_start_record now returns the full (session_id, user_id,
started_at_ns, metadata) tuple — the metadata bytes were already written
by serialize_start_record but silently discarded on read.
restore_session_wal_events looks up the persisted start record in storage
for each open session and uses the deserialized metadata instead of
HashMap::new(), so fields like {"tool":"planner"} survive a crash.
The signal replay loop no longer discards _annotation — annotations are
now pushed into state.annotations during WAL replay, restoring preference
hints like "more jazz today" so FOR SESSION ranking works post-restart.
Two new integration tests in session_durability.rs verify both fixes
against a real persistent store with simulated crash (drop without
close_session). session/serde.rs split into serde/mod.rs + serde/start_record.rs
to satisfy the 600-line limit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- m0p3: CONTRIBUTING.md with run-samples checklist, all 4 examples
(quickstart, cli_embedding, axum_embedding, actix_embedding), doc-test
coverage for every public API surface
- m1p5: TidalDb public API — write_item, signal, read_decay_score,
read_windowed_count, read_velocity; StorageBox enum routing memory vs
fjall; WalSender/WalHandleWriter bridge; WAL replay on open
- Periodic checkpoint: 30s background thread for persistent+schema mode;
FjallBackend::Clone (O(1), fjall::Keyspace is ref-counted); graceful
shutdown via Arc<AtomicBool> + join before final checkpoint
- ROADMAP.md: M0 and M1 fully marked COMPLETE (341 tests passing)
- Milestone 2 planning scaffolding added under docs/planning/milestone-2/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
