The three live voters disagree on signal aggregates for the same entity
(view = 10003 / 10095 / 10144 for entity 1, stable across passes) while
`/cluster/status` reports applied_events equal, lag_events 0, and no divergence
quarantine. The documented remedy is `POST /cluster/reconcile`. On this corpus
it fails:
503 region 'tidaldb-1' unreachable:
reconcile peer returned 413 Payload Too Large
Two defects, both fixed here:
- The whole-shard CRDT `StateSnapshot` was capped by `BODY_LIMIT_BYTES`, the
2 MiB limit sized for one client write on the public data surface. The
snapshot carries one entry per entity x signal type; on 33k documents it is
several MiB, so divergence was unhealable in production. The internal,
marker-pinned, operator-driven snapshot route now has its own explicit
ceiling.
- A 413 was reported as `RegionUnreachable`. The peer answered - it is
reachable and healthy - so the error sent the operator to TLS and
NetworkPolicy. It now names the measured snapshot size, the peer's cap, and
the fix.
The ceiling is not the design: the snapshot grows with the corpus and chunked
reconcile is the durable answer. Documented as such at the constant.
Scale write throughput across data-shard groups while keeping a single unified
read surface:
- scatter_gather.rs: pooled fan-out across shard groups (replaces per-request
client construction); cross-shard query results merged on one node
- cluster/node.rs: cross-shard read routing — a read on any node gathers from
every shard group's leader and unions results
- cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body
for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM
on the real mTLS plane)
- dto.rs: cross-shard query/result DTOs
- k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers,
topology-configmap)
- k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A
- tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job
- tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind
- docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
Matches the response shape used by all other services in the
infrastructure (ok, service, cause on failure).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add /health/startup and /health/live probes, flip readiness to 503 on
SIGTERM so k8s stops routing before drain. Update standalone Dockerfile
for internal deployment: port 9500, schema mounted at runtime (not baked
in), persistent data dir, non-root user with fixed UID 10001.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix 9 compilation errors across tidal-server and testing/cluster.rs so
that `cargo run -p tidal-server -- standalone` works end-to-end.
Bugs fixed:
- cluster.rs: wrong return types `RetrieveResult`→`Results` and
`SearchResult`→`SearchResults` on retrieve/search helpers
- state.rs: `RegionId` imported from private path; now uses
`tidaldb::replication::RegionId`
- state.rs: missing `Ok()` wrapper on `ServerState::cluster()` return
- state.rs: cluster match arms returned `TidalError` where `ServerError`
required; added `.map_err(ServerError::from)` on write_item,
write_embedding, retrieve, search
- error.rs: `Result<T>` alias lacked default E param; callers in router
used two-arg form `Result<T, AppError>` — changed to
`Result<T, E = ServerError>`
- router.rs: `with_state()` called before cluster routes were added,
making `app` `Router<()>`; restructured to call `with_state` once at end
- router.rs: `TidalErrorWrapper(TidalError)` used to map `QueryError`;
fixed with `|e| TidalErrorWrapper(e.into())`
- router.rs: `Search::limit()` takes `u32` but code cast to `usize`
- router.rs: `bm25_score`/`semantic_score` are `f32` in SearchResultItem
but `f64` in response struct; added `.map(f64::from)` conversion
Also split cluster.rs into cluster.rs + cluster_transport.rs to stay
under the 600-line limit required by CODING_GUIDELINES §9.
Verified all README curl examples work:
POST /items, POST /embeddings, POST /signals, GET /feed, GET /search,
GET /health all return correct HTTP status codes and JSON responses.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
