da736b8 replaced `applied >= leader_last_seq` with `applied >= marker.from_seqno`
and was still wrong, for the same underlying reason: the applied frontier is a
HIGH-WATER-MARK, not a contiguity proof. A term join re-bases it onto the new
leader's stream (`replication_state().advance(.., baseline + 1)`), so it leaps
across history the node never received. Any predicate built on it discharges
markers for nodes that still have a hole.
Measured, not argued. `mp_follower_reseeds_via_snapshot_after_compaction` stops a
follower at frontier 9, compacts the leader so it retains only from 15722, and
the follower's frontier is re-based to 16810. Both predicates discharge the
marker there; the node skips its reseed and then reports `lag_events: 0` while
missing 10..15721 and serving reads from a log with a hole. Production showed the
identical shape: `applied 13540660` against a marker resuming at 13540653 that no
live WAL could serve.
The marker is now discharged only on POSITIVE EVIDENCE that the stream served the
latching range: a `StreamSegments` pull that began at or below the marker's
`from_seqno` and ran to completion. New `CatchupServedSink` in tidal-net fires on
`PullOutcome::Complete`; `NodeCatchupServedSink` routes it to
`discharge_reseed_marker_if_served`. `ReseedMarker::discharged_by_served_range`
replaces `discharged_by`. The other sound discharge is unchanged: a snapshot
install replaces the data dir and takes the marker with it.
Both frontier-based call sites are gone, with the reasoning recorded where they
were. The election-won site is deliberately NOT replaced: winning proves the log
beats a quorum's under the vote restriction, which is not contiguity, so
discharging there could promote a leader with a hole.
The owner-test for this mechanism was RED ON BASELINE and is now green. It also
gained the premise assertion it never had: it used to assert only the consequence
(`reseed_required == true`), so when its fixture stopped forcing compaction it
failed 40s later looking like a follower bug. `assert_history_compacted_past` now
checks the leader actually dropped the follower's resume seq, and prints the
retained segment floors. Its content probe ("every probed offline item is
searchable on the reseeded follower") is what proves the hole is really gone.
Suite state: cluster_reseed's other tests pass individually.
mp_graceful_rolling_restart_under_load_no_reseed remains red on baseline
(pre-existing, verified by stash). mp_quarantined_node_reseeds_without_wipe and
mp_election_position_consistent_across_roles_after_failover pass alone but can
fail in-suite: this fix makes the compaction test run its full 565s reseed
instead of failing fast at 40s, which shifts timing for later tests on shared
fixed ports. Order sensitivity is pre-existing, not introduced here.
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:
- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
GrpcTransport::new installs the process-wide provider, so every TLS joiner
panicked. Install it at the top of main(); ensure_crypto_provider() is now
pub, idempotent, harmless on the plaintext standalone path.
- Wrong seed scheme + target: peer_url honors an explicit URL scheme
verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
https:// AND points at the ready-only client Service (ClusterIP VIP), not
the headless peers Service — so a joiner never round-robins onto a
not-ready pod (incl. itself) and burns the 120s discovery window.
- Too-tight poll budget: a cold status poll pays a full rustls handshake on
top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
the joiner timed out every poll for the whole window despite the peer being
reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
with a separate 2s connect timeout (dead seeds still fail fast) and
debug-level logging on every discovery failure mode.
Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
(BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
regression (periodic self-heal / status-poll dependency) the binary budget
check would otherwise wave through.
New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).
- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
lag gauge (monotonic per shard) and drives the readiness latch using a REAL
leader frontier (never the uninitialized-0 gauge, which would false-converge a
still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
— snapshot joiner flips /health ready on an idle cluster with zero writes and
no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
