The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:
- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
GrpcTransport::new installs the process-wide provider, so every TLS joiner
panicked. Install it at the top of main(); ensure_crypto_provider() is now
pub, idempotent, harmless on the plaintext standalone path.
- Wrong seed scheme + target: peer_url honors an explicit URL scheme
verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
https:// AND points at the ready-only client Service (ClusterIP VIP), not
the headless peers Service — so a joiner never round-robins onto a
not-ready pod (incl. itself) and burns the 120s discovery window.
- Too-tight poll budget: a cold status poll pays a full rustls handshake on
top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
the joiner timed out every poll for the whole window despite the peer being
reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
with a separate 2s connect timeout (dead seeds still fail fast) and
debug-level logging on every discovery failure mode.
Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
(BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
regression (periodic self-heal / status-poll dependency) the binary budget
check would otherwise wave through.
New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).
- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
lag gauge (monotonic per shard) and drives the readiness latch using a REAL
leader frontier (never the uninitialized-0 gauge, which would false-converge a
still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
— snapshot joiner flips /health ready on an idle cluster with zero writes and
no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
Scale write throughput across data-shard groups while keeping a single unified
read surface:
- scatter_gather.rs: pooled fan-out across shard groups (replaces per-request
client construction); cross-shard query results merged on one node
- cluster/node.rs: cross-shard read routing — a read on any node gathers from
every shard group's leader and unions results
- cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body
for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM
on the real mTLS plane)
- dto.rs: cross-shard query/result DTOs
- k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers,
topology-configmap)
- k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A
- tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job
- tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind
- docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
ClusterNode hosts a BTreeMap<ShardId, Arc<ShardReplica>>: writes hash-route
to the owning shard leader, reads scatter over shard groups. In-group
shard==region preserved so the engine and tidal-net are untouched; S=1 stays
byte-for-byte (today's cluster is a 1-shard × RF=N group). Topology grows
shard-group awareness; membership, election, forward, reseed, and join_boot
thread ShardId through.
Proven by an in-process 2×2 RF=2 gRPC test plus S=1 parity, incl. tier-3
real-OS-process failover. clippy/fmt clean.
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Two correctness bugs in multi-process cluster mode (m8p10), both found live on a
real 3-pod k3s cluster while validating the deployment, both invisible to the
existing in-process / unauthenticated test suites:
1. Forwarded item/embedding writes dropped on the floor. create_item /
write_embedding gated the leader's peer broadcast on `if internal { return }`.
A write to a NON-leader gateway is forwarded to the leader with the internal
marker set (loop-prevention), so it hit that branch and terminated WITHOUT
broadcasting — the item landed only on the leader. Signals were unaffected
(the WAL relay ships regardless of the marker), which masked it. Fix: gate on
leadership, not the marker — a follower applying a marked broadcast/heal
terminates; the leader (external OR forwarded) always fans out. Forwarded
writes now return the {replicated_to,failed} report instead of a bodyless null.
2. Heal item/embedding backfill 401'd whenever TIDAL_API_KEY is set.
post_marked_blocking sent the internal marker but no Authorization header. The
marker is a trust signal, not an auth bypass (the bearer middleware runs
first), so every backfill POST was rejected 401 — a region that missed an item
while down stayed permanently inconsistent at lag 0. Unauthenticated tests
never caught it. Fix: thread TIDAL_API_KEY into RegionClusterState and attach
it (same key on every region) to the backfill POSTs.
Verified live: forwarded writes via follower gateways converge to all 3 regions;
a region scaled to 0 during an item write backfills on heal (item_failures=0).
Follow-up: add multiproc regression tests with auth for both paths.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:
- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
Three changes closing M8 gaps identified during verification:
1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE).
Added Known Gaps table (G1: in-process transport, G2: tier-3
tests, G3: hash inconsistency).
2. SimulatedCluster transport now pluggable via ClusterConfig.transports.
Default (None) uses new ChannelTransport (crossbeam, same behavior).
When Some, accepts external transports (e.g., GrpcTransport from
tidal-net). Updated redeliver_missed to use &dyn Transport.
Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged.
3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs).
ClusterHarness spawns real tidal-server cluster OS processes,
allocates dynamic ports, generates topology YAML, polls health,
and cleans up via SIGTERM. Two tests: smoke (write + converge +
verify follower reads) and promote (leader change + continued writes).
Feature-gated behind cluster-e2e.
Add support for defining ranking profiles in schema YAML, allowing
deployments to override builtin profiles with deployment-specific
signal names and tuning parameters.
- Add override_register() to ProfileRegistry for clean builtin replacement
- Add with_profiles() to TidalDbBuilder to thread schema profiles
- Parse profiles section in config.rs with full sort/strategy/agg support
- Validate that profile signal references exist in schema at startup
- Change load_schema() to return (Schema, Vec<RankingProfile>)
This closes the gap where the builtin for_you profile referenced
signals (view, like, share) that don't exist in deployment schemas,
causing all feed scores to normalize to 1.0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
