A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:
- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
GrpcTransport::new installs the process-wide provider, so every TLS joiner
panicked. Install it at the top of main(); ensure_crypto_provider() is now
pub, idempotent, harmless on the plaintext standalone path.
- Wrong seed scheme + target: peer_url honors an explicit URL scheme
verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
https:// AND points at the ready-only client Service (ClusterIP VIP), not
the headless peers Service — so a joiner never round-robins onto a
not-ready pod (incl. itself) and burns the 120s discovery window.
- Too-tight poll budget: a cold status poll pays a full rustls handshake on
top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
the joiner timed out every poll for the whole window despite the peer being
reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
with a separate 2s connect timeout (dead seeds still fail fast) and
debug-level logging on every discovery failure mode.
Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
(BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
regression (periodic self-heal / status-poll dependency) the binary budget
check would otherwise wave through.
New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).
- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
lag gauge (monotonic per shard) and drives the readiness latch using a REAL
leader frontier (never the uninitialized-0 gauge, which would false-converge a
still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
— snapshot joiner flips /health ready on an idle cluster with zero writes and
no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
