Every destructive /cluster/* verb sat behind the SAME bearer as /items and
/search, so any application key could remove a member, force a partition, or
transfer a shard. There was no way to hand out a client credential without also
handing out the ability to destroy the cluster.
Adds TIDAL_ADMIN_KEY (and TIDAL_ADMIN_KEY_FILE, rotatable without restart like
the others). /cluster/promote, /cluster/partition, /cluster/heal,
/cluster/members/remove, /cluster/reseed and /cluster/shards/{id}/{replicas,
transfer} move into their own router subtree behind an admin gate; the data
bearer now gets 403 there - authenticated but not authorized, distinct from the
401 for a bad token.
Three things this had to get right:
* The admin key must ALSO authenticate. A request carries one Authorization
header, so if the admin key did not satisfy the bearer gate, an operator
presenting it would be 401'd before the admin gate ran and the verbs would be
reachable by nobody. Caught while writing the test, not after.
* A verified sibling node token clears the gate too. Nodes relay operator verbs
to the leader/target carrying whatever credential the caller sent, and the
legacy fan-out promote uses the internal marker, so requiring the admin key on
that hop would partition the control plane.
* The peer-callable verbs stay on the plain bearer. /cluster/catchup (self-heal
nudge), /cluster/join + /cluster/members (seed-join) and the
/cluster/reconcile* pair are dialled node-to-node, so gating them would break
replication and joining.
Absent admin key = previous behavior exactly, plus a startup WARN naming the
exposure, so this is safe to upgrade into. The k8s secret mount is optional:true
because without that a deployment lacking the key would fail to MOUNT and never
start.
Also closes the /cluster/status hole this exposed: it and /cluster/status/local
reported leader identity, membership, term and per-shard applied/lag/commit
seqnos from the UNAUTHENTICATED probe group. They are protected now, which is
what k8s/cluster/networkpolicy.yaml deferred to rather than working around at the
network layer.
And fixes a latent bug found on the way: seed-join discovery, reseed discovery
and the self-heal catch-up nudge read std::env::var("TIDAL_API_KEY") directly,
which yields nothing on a *_FILE-only deployment - the node would dial an
authenticated peer with no credential. They use security::bearer_from_env() now,
which honours both shapes.
Verified: 5 new unit tests; two multi-process runbook tests on real 3-process
clusters (data bearer 403 on promote / 204 on signals, admin key 200 on status
and through the gate on heal; bare /cluster/status 401, 200 with the bearer).
That the authenticated cluster converges at all is the load-bearing assertion -
if moving status behind auth had broken leader discovery, startup would hang.
Full unit suites green (2101 + 162), reseed e2e green, clippy clean.
The 2026-08-20 livelock ran 21h with no page. TidalDBClusterReseedPending is
`tidaldb_cluster_reseed_required == 1 for 10m`, written for exactly this, but the
defect cleared the marker ~200ms after each latch, so the gauge flapped 1->0
every ~30s and never held 1 for 10m. Add TidalDBClusterReseedFlapping, which
keys off `changes(...[15m]) > 2` instead of a hold duration, so a latch/clear
loop pages even when the gauge reads 0 at both ends of the window and even if a
future clear path reintroduces the spurious clear.
Also add an on-call banner for the operator intervention now in force: the
client Service selector is narrowed to keep tidaldb-0 out of the read path,
because its shard-1 frontier was cross-seeded from shard 2's snapshot artifact
and its reads are untrustworthy even at lag_events: 0. The banner carries the
footgun (the label is on the pods, not the template, so it does not survive pod
recreation) plus repair and revert commands.
`cargo test --workspace` could not run at all: dependency resolution failed with
"aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the
gate the project documents was dead. Making it run exposed a compile break and
two wrong tests that had been invisible for months. Now green end to end:
143 suites, 3155 tests, exit 0.
Toolchain
- rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays
`rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK
chain needs the patch release, and it now declares that itself.
Consumer crates migrated to the current engine API (clean cutover)
- iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields;
the literal now spreads `..AgentPolicy::default()` as the engine's own doc
example does, so future fields do not break it again.
- forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own
candidate pool, so it now tags what it knows: PreferenceMatch for the
preference-vector blend, SemanticMatch (with the seed item) for
similar-to-saved, ExplorationBudget for pinned discoveries.
- forage-engine: `url_to_item_id` folded into the u32 item universe. The engine
narrows item IDs to a u32 slot in durable per-user state and rejects anything
above u32::MAX rather than alias two items forever, so every add_item with a
64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone.
- forage-engine: bridge items read the top-2 preference CLUSTERS via
`query_vectors`, not the single centroid from `preference_vectors().get()`.
Since m12 that accessor returns only the strongest cluster, so a tech+jazz user
whose interests split into two clusters looked single-interest and never
bridged. Falls back to top-2 dimensions when a user has one cluster.
Reconcile tests corrected to the shipped contract
- tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed
count after heal. `take_crdt_snapshot` deliberately keys signal contributions
to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from
a single writer, so per-node attribution double-counted every replicated event
on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus
PN-counter per-node max: nodes converge on the more complete accumulator. The
old expectation was asserting the bug that fix removed.
- Rewrote to assert convergence, count survival (not 0), and no inflation, and
added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression
guard for the creep itself, which nothing covered.
Pre-commit hook unified
- hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the
source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the
flag promoted ~58 deliberate pedantic warnings in integration tests to errors,
making every Rust commit impossible.
- It now lints all five tidal crates instead of path-matching `tidal/`, which
silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and
applications/ - the rot above lived in exactly those crates. Ported the
CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the
divergent untracked copy that this replaces.
- CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split.
Fleet recovery and soak
- scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an
ignored tmp/ directory into the repository. Preflights retained storage,
digest-pinned images, parked state, and aggregate plus per-PV-node scheduler
headroom before the first scale; writes a durable transcript under
tmp/restore-logs/ with structured start/error/rollback/complete events.
- k8s manifests park the standalone store, the RF3 cluster, and the soak monitor
at zero replicas with restore-fleet.sh as the only supported scale-up path.
- soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing
restart evidence instead of silently skipping the restart-aware half of the gate.
- docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier
formula and separates analytic totals from the measured process envelope.
M0-M12 are shipped and the HA cluster runs in production on k3s, so the
pre-release disclaimer no longer describes the project. Removes it from the
canonical doc set and corrects the readiness text that had gone stale.
- README.md: replace the "Pre-release / not yet recommended for production"
banner with a production-ready statement; drop "(experimental)" from the
cluster status bullet; state the post-1.0 versioning posture (additive in
minor releases, breaking changes get a documented migration path).
- CLAUDE.md / QUICKSTART.md / docs/guides/server-deployment.md /
docs/runbooks/cluster.md: same withdrawal; reframe the cluster opt-in as a
guard against standing up a multi-node fabric by accident rather than a
readiness warning.
- CHANGELOG.md: record the stability posture under [Unreleased], superseding
the historical 0.1.0 "no stability guarantees" note (left intact as history).
- k8s/statefulset.yaml: the "NOT production HA, tracked as m8p10" comment was
stale (m8p10 shipped); point at k8s/cluster/ for the HA deployment instead.
Also corrects text that was factually wrong since m11p3/m11p4: the
multi-process cluster gate, its CLI help, and the served OpenAPI description
all still claimed quorum-ack writes and automatic failure detection did not
exist. They do.
Historical records (docs/reviews/, docs/profiling/, past CHANGELOG entries,
the kubernetes.md rc7 fix note) are left unchanged.
Verified against a running binary, not just the build: the opt-in gate's
refusal message, the startup WARN, /health 200, and the served
/openapi.json description all carry the new text. cargo fmt clean; clippy
-D warnings clean on tidaldb and the tidal-server lib; 1943 engine + 155
server lib tests pass; scripts/check-docs.sh OK.
Claude-Session: https://claude.ai/code/session_01QdqSDw1tUhK1JT9Pb1vryP
The runbooks had drifted to the retired m8/m11p5 design while all m12 production
reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens.
This promotes that reality into the runbooks and fixes the contradictions.
Contradictions fixed:
- runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE
(quorum-ack + automatic election have been live since m11p3/p4). Rewritten to
state the deployed reality (single-StatefulSet full-placement RF3, rc7).
- README.md: the cluster section called the HA cluster a "built-in simulated
cluster / multi-region fabric" demo and showed promote-by-region as failover.
Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance
verb. Kept the honest caveats (experimental gate, global-signals-only).
Reality promoted into the runbooks:
- Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3,
gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real
build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest,
not the index/attestation digest) in cluster.md + kubernetes.md.
- Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the
pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added.
- ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section
(read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources).
- ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and-
copy note to standalone (the cluster uses tidalctl + the DR runbook).
New docs:
- runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify
-> query-proof procedure, full-cluster rebuild, PITR posture (previously
undocumented despite being proven against real S3).
- runbooks/on-call.md: incident response — symptom -> golden signal -> runbook,
severity, escalation, and the open alert-wiring step.
- runbooks/README.md: the runbook index + current production facts.
Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but
design-reference; promoting it to a live PrometheusRule is the one unwired step.
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno
acked under `ack=leader` (journal-only, un-replicated); `decide_join` now
quarantines on THIS node's own frontier rather than comparing stream numbers
across stream boundaries — eliminates false-quarantine churn on rolling
restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum
before step-down so the next leader inherits a clean prefix. New
`load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains
the divergence-fix regression suite; `replication_ops.rs` threads the signal.
Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the
30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly
CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak
definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh`
gates the election-divergence suite before any k8s push.
Release tooling: `docker/release/` multi-stage Dockerfile + DR image;
`scripts/build-release.sh` single repeatable cross-compile+buildx path.
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:
1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
visits each hosted group's own shard subdir; per-group leader discovery
appends `?shard=N` so a divergent shard heals from its own leader (not
shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
`note_self_won_term` so the elected shard's `joined_term` is set and
`cluster_promote` routes rebalances correctly (was: topology-era mis-read
→ legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
prevents a node pulling its own stream (its stream isn't a registered peer)
→ eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
instead of `?`-failing the whole read; bounded read-admission semaphore
(`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
most-recent sealed segments; online path gets the same retention clamp.
Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
`from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
returns `needed=true` and the snapshot installs instead of looping.
Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.
- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
(scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
a cold TLS handshake under contention blew the sub-second budget. Discovery now
logs each poll failure at WARN with the full error source chain (a silent loop
made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.
Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:
- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
GrpcTransport::new installs the process-wide provider, so every TLS joiner
panicked. Install it at the top of main(); ensure_crypto_provider() is now
pub, idempotent, harmless on the plaintext standalone path.
- Wrong seed scheme + target: peer_url honors an explicit URL scheme
verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
https:// AND points at the ready-only client Service (ClusterIP VIP), not
the headless peers Service — so a joiner never round-robins onto a
not-ready pod (incl. itself) and burns the 120s discovery window.
- Too-tight poll budget: a cold status poll pays a full rustls handshake on
top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
the joiner timed out every poll for the whole window despite the peer being
reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
with a separate 2s connect timeout (dead seeds still fail fast) and
debug-level logging on every discovery failure mode.
Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
(BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
regression (periodic self-heal / status-poll dependency) the binary budget
check would otherwise wave through.
New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).
- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
lag gauge (monotonic per shard) and drives the readiness latch using a REAL
leader frontier (never the uninitialized-0 gauge, which would false-converge a
still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
— snapshot joiner flips /health ready on an idle cluster with zero writes and
no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
Scale write throughput across data-shard groups while keeping a single unified
read surface:
- scatter_gather.rs: pooled fan-out across shard groups (replaces per-request
client construction); cross-shard query results merged on one node
- cluster/node.rs: cross-shard read routing — a read on any node gathers from
every shard group's leader and unions results
- cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body
for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM
on the real mTLS plane)
- dto.rs: cross-shard query/result DTOs
- k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers,
topology-configmap)
- k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A
- tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job
- tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind
- docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production
width) and ran the realistic peach mix (feed-profile reads + signal writes) on the
m11p6 mTLS cluster.
Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps
(128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader
pool, not vector size. The vector READ path (feed-profile retrieve — the
db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps,
never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod
at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput.
Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of
knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA.
Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects
the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
Replace ScoredCandidate.signal_snapshot Vec<(String,f64)> with
SmallVec<[(SignalKey,f64); 4]> where SignalKey is Static(&'static str)
| Owned(Arc<str>):
- Compile-time-constant labels (sort bases, relevance, co_engagement,
preference_affinity) -> Static: zero allocation, pointer-copy clone.
- Dynamic {signal}_boost/_penalty/_decay labels built once per query in a
RuleLabels hoist (was format! per-candidate-per-rule), shared by Arc.
Cohort rescore hoisted the same way.
- scored accumulator pre-sized to candidates.len().
- Owned Strings rebuilt only at the two response-assembly sites (<= limit).
Byte-identical output: full 1896-test lib suite green. Measured win
(cargo bench --bench ranking, committed-base vs working-tree):
score_200_hot 23.89->21.04us (-11.9%), score_200_trending
27.66->25.85us (-6.5%), score_200_full_pipeline 27.88->26.97us (-3.3%).
smallvec promoted from the lock to a direct dep (no new dependency
surface). perf-sweep doc updated; T2 (per-term DashMap collapse) next.
ClusterNode hosts a BTreeMap<ShardId, Arc<ShardReplica>>: writes hash-route
to the owning shard leader, reads scatter over shard groups. In-group
shard==region preserved so the engine and tidal-net are untouched; S=1 stays
byte-for-byte (today's cluster is a 1-shard × RF=N group). Topology grows
shard-group awareness; membership, election, forward, reseed, and join_boot
thread ShardId through.
Proven by an in-process 2×2 RF=2 gRPC test plus S=1 parity, incl. tier-3
real-OS-process failover. clippy/fmt clean.
Kind-3 term markers in the WAL stream, STREAM-relative vote frontiers,
heartbeat-only divergence detection + quarantine, and fenced promote.
Elections converge in 0.6–1.0s; zero acked-write loss across all kill points.
Closes G5 (leaderless recovery) from the v0.9 wave.
WAL segment format: 8-byte TSEG header (magic + version byte + 3 reserved)
prepended to every new segment. Legacy headerless segments (m0-m11p3) read
as implicit v0 — no migration. Unknown magic/version surfaces as
WalError::SegmentFormatUnknown at open time; foreign files are never
repaired or truncated (fixes the silent data-loss path from the p3 rollout
incident where torn-tail repair zeroed a follower's unreadable segments).
Catch-up transport: FAILED_PRECONDITION ("snapshot required") and stream
errors that skip the shard now arm a timer retry (re-arm-on-skip is the
load-bearing liveness fix — without it a skipped pull never re-fires and
the follower stays permanently behind). Single retry pending per shard;
CatchupRunner owns the Arc'd state shared between the retry tasks and the
transport. Test: tidal-net/tests/catchup_retry.rs covers the retry path.
Stress: k8s stress-job-t2a/t2b yaml + ops/stress-test-p3-t2 runbook.
ack=quorum gates replicated writes on a majority of the replica set durably
holding them: followers push their durably-applied frontier (ReportApplied,
once per apply round, decoupled from ship acks), the leader folds frontier
reports + ship-ack hints + heal resumes into a leadership-scoped CommitIndex
(k-th-largest durable mark), and handlers await it through an async
watch-channel bridge (zero parked threads per waiter). Honest timeouts:
retryable 503 naming the laggards; x-tidal-seq on every cluster write.
Follower blob applies are batched under group-commit fsyncs (22x seeding).
Exit gate: 167/167 leader-SIGKILL kill points, zero acked-write loss.
Seven-dimension review pass (all confirmed findings fixed):
- WAL blob drain now ABORTS on the first write failure instead of reusing
the failed seqno mid-drain (a torn record buried mid-segment would
truncate every later acked record on replay)
- apply_replicated_blobs waits every staged append even after a mid-batch
failure, parses metadata once, and moves records into Arcs shared with
the WAL writer (no deep clone per record on the follower apply path)
- CommitIndex: zero-peer fast path now respects demotion (active checked
under lock before the single-replica return), k-th-largest uses
select_nth over a reused scratch buffer
- await_quorum: re-reads the index once after the deadline fires (no false
503 for a write that committed in the race window), warns when the
commit-watch bridge dies outside shutdown, zero-peer path checks active
- notify_applied report failures: WARN on the first failure of a streak,
INFO on recovery (a silently stalling frontier reads as unexplained
quorum 503s); receiver skips re-notifying unadvanced frontiers
- x-tidal-deduplicated: 1 marks dedup-suppressed signal writes (relayed
through forwards) so durability cursors can tell dedup from no-seqno
- docs: 167/167 kill-point record corrected in CHANGELOG; rolling-upgrade
order (leader first — a pre-m11p3 leader silently downgrades quorum
requests to leader-ack) in CHANGELOG + runbook §8; monitoring note for
report-loss diagnosis on the quorum-timeout alert
Verified: workspace clippy -D warnings (incl. cluster-e2e targets), full
tidaldb/tidal-net/tidal-server/tidalctl suites green, tier-3 multi-process
quorum suite green (8/8 kill points, zero acked loss, partition gate/recover).
m11p1 — decoupled ack/ship path: staged writes (seqno+WAL+relay-push,
microseconds) separate from group-commit fsync; ShipQueue batches+windows
outbound segments; receiver coalesces inbound chunks before applying.
Adds first tidaldb_cluster_* metrics.
m11p2 — leader WAL is now THE replicated log: fsynced batches feed a
bounded WalShipFeed and ship byte-identical to followers; WAL seqnos
survive restarts (relay-reset hazard gone). Item metadata and embeddings
journal kind-1/2 blob records on the same stream as signals; the m8p10
HTTP broadcast is deleted. StreamSegments catch-up is follower-pulled via
server-streaming RPC, triggered on gap detection, follower boot, and
leader heal nudge. Promote carries a stream baseline so peers skip
pre-stream history.
Gap analysis + 9-phase plan (m11p1-p9) from the experimental m8p10 cluster to an
enterprise-grade one, grounded in the 2026-06-10 live stress-test baselines:
replicated /signals ~90/s vs sharded 3,669/s vs embedded ~82ns, plus four
live-observed incidents (restart leadership amnesia, side-channel broadcast bugs,
breaker-eaten heal, the 178ms ship) promoted to requirements. Phases: perf floor,
one replicated log, quorum acks (G4), election+fencing (G5), membership/discovery,
sharding x RF, security, observability/ops, chaos CI — each with a testable
Ref-A exit gate; release waves v0.9 credible / v1.0 scalable / v1.1 enterprise.
Indexed in docs/README.md; pointer from ROADMAP's Post-M8 Follow-ups.
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load
generator + capacity ramp for the standalone and multi-process cluster surfaces,
modeling a thepeach feed session (feed reads + view/like/skip signals + search,
signal-dominated per their user-graph spec). Throttleable target rate, ramp
presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom
mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a
backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict
translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/).
Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures
latency from each request's intended send time, so a server stall inflates the
percentiles a closed-loop test hides; it shed-and-counts rather than blocking when
the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps.
Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md):
reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at
~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship);
the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the
100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod
restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns).
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
Resolves every finding in docs/reviews/M0-M10-code-review-2026-06-08-pass2.md
across the engine, network, server, and CLI crates: session restore,
replication/CRDT, WAL format and recovery, storage indexes, query/ranking
executors, cohort/community governance, and scatter-gather routing.
Adds regression tests:
- review_pass2_creator_search_filter
- review_pass2_d_replication
- review_pass2_query_for_session
- review_pass2_storage_indexes_bitmap_cache
- review_pass2_zone_a_sessions
Verified: cargo clippy -D warnings and full test suite green across all crates.
Document the six issues a follow-up adversarial sweep surfaced in the
M0-M10 remediation diff and their fixes (code already landed):
- BLOCKER warm-tier double-count was a half-fix (read side only); now
bounded on read + write (hour_agg anchored at last_min) + day tier.
- CRITICAL W22 stale-index scrub now wired into the item overwrite path.
- CRITICAL checkpoint read() made side-effect-free; temp-sweep moved to
the writable recover() path.
- WARNING W13 partition_id wired at construction; W22-class social-graph
/collection u32 truncation routed through checked entity_as_u32.
Updates the green-state line (1672 lib tests, clippy -D on all crates).
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Three changes closing M8 gaps identified during verification:
1. ROADMAP.md: Mark m8p8 and m8p10 as PARTIAL (not COMPLETE).
Added Known Gaps table (G1: in-process transport, G2: tier-3
tests, G3: hash inconsistency).
2. SimulatedCluster transport now pluggable via ClusterConfig.transports.
Default (None) uses new ChannelTransport (crossbeam, same behavior).
When Some, accepts external transports (e.g., GrpcTransport from
tidal-net). Updated redeliver_missed to use &dyn Transport.
Zero regressions: all 1209 lib + 8 m8_uat tests pass unchanged.
3. Multi-process E2E test harness (tidal-server/tests/cluster_e2e.rs).
ClusterHarness spawns real tidal-server cluster OS processes,
allocates dynamic ports, generates topology YAML, polls health,
and cleans up via SIGTERM. Two tests: smoke (write + converge +
verify follower reads) and promote (leader change + continued writes).
Feature-gated behind cluster-e2e.
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add milestone-level COMPLETE summary bullets for M0–M8 (only M8 had one)
- Fix m8p6 lib test count (1199 → 1206 after latest additions)
- Update iknowyou/Aeries date to 2026-02-24
- Each summary captures the key capabilities proved by that milestone
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- m0p3: CONTRIBUTING.md with run-samples checklist, all 4 examples
(quickstart, cli_embedding, axum_embedding, actix_embedding), doc-test
coverage for every public API surface
- m1p5: TidalDb public API — write_item, signal, read_decay_score,
read_windowed_count, read_velocity; StorageBox enum routing memory vs
fjall; WalSender/WalHandleWriter bridge; WAL replay on open
- Periodic checkpoint: 30s background thread for persistent+schema mode;
FjallBackend::Clone (O(1), fjall::Keyspace is ref-counted); graceful
shutdown via Arc<AtomicBool> + join before final checkpoint
- ROADMAP.md: M0 and M1 fully marked COMPLETE (341 tests passing)
- Milestone 2 planning scaffolding added under docs/planning/milestone-2/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
