m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Partition recovery (heal_region) and await_convergence never worked over real
GrpcTransports: redeliver_missed sent to entry.source_shard (the LEADER's
shard), but a follower's transport only knows its own shard as a peer
(self-loop wiring in tidal-server::cluster), so every recovery ship failed
the peer lookup — and the error was swallowed by 'let _ ='. ChannelTransport
ignores the destination, which is why the in-process suite never caught it.
Found live on k3s: POST /cluster/partition + writes + /cluster/heal left the
region at lag=3 forever, no log line.
Fix: ship to transport.local_shard() (matching the eager path's
send_segment(ShardId(region.0), ...)); keep entry.source_shard in the payload
(receiver advances applied_seqno per SOURCE shard). WARN on ship failure
instead of dropping it. Adds a RecordingTransport regression test that locks
dest == local_shard + payload source == leader — the exact case channels
cannot catch. cargo test: 263 replication + 13 testing + tidal-server all
pass.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
Three changes closing M8 gaps identified during verification:
1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE).
Added Known Gaps table (G1: in-process transport, G2: tier-3
tests, G3: hash inconsistency).
2. SimulatedCluster transport now pluggable via ClusterConfig.transports.
Default (None) uses new ChannelTransport (crossbeam, same behavior).
When Some, accepts external transports (e.g., GrpcTransport from
tidal-net). Updated redeliver_missed to use &dyn Transport.
Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged.
3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs).
ClusterHarness spawns real tidal-server cluster OS processes,
allocates dynamic ports, generates topology YAML, polls health,
and cleans up via SIGTERM. Two tests: smoke (write + converge +
verify follower reads) and promote (leader change + continued writes).
Feature-gated behind cluster-e2e.
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix 9 compilation errors across tidal-server and testing/cluster.rs so
that `cargo run -p tidal-server -- standalone` works end-to-end.
Bugs fixed:
- cluster.rs: wrong return types `RetrieveResult`→`Results` and
`SearchResult`→`SearchResults` on retrieve/search helpers
- state.rs: `RegionId` imported from private path; now uses
`tidaldb::replication::RegionId`
- state.rs: missing `Ok()` wrapper on `ServerState::cluster()` return
- state.rs: cluster match arms returned `TidalError` where `ServerError`
required; added `.map_err(ServerError::from)` on write_item,
write_embedding, retrieve, search
- error.rs: `Result<T>` alias lacked default E param; callers in router
used two-arg form `Result<T, AppError>` — changed to
`Result<T, E = ServerError>`
- router.rs: `with_state()` called before cluster routes were added,
making `app` `Router<()>`; restructured to call `with_state` once at end
- router.rs: `TidalErrorWrapper(TidalError)` used to map `QueryError`;
fixed with `|e| TidalErrorWrapper(e.into())`
- router.rs: `Search::limit()` takes `u32` but code cast to `usize`
- router.rs: `bm25_score`/`semantic_score` are `f32` in SearchResultItem
but `f64` in response struct; added `.map(f64::from)` conversion
Also split cluster.rs into cluster.rs + cluster_transport.rs to stay
under the 600-line limit required by CODING_GUIDELINES §9.
Verified all README curl examples work:
POST /items, POST /embeddings, POST /signals, GET /feed, GET /search,
GET /health all return correct HTTP status codes and JSON responses.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
