`cargo test --workspace` could not run at all: dependency resolution failed with
"aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the
gate the project documents was dead. Making it run exposed a compile break and
two wrong tests that had been invisible for months. Now green end to end:
143 suites, 3155 tests, exit 0.
Toolchain
- rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays
`rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK
chain needs the patch release, and it now declares that itself.
Consumer crates migrated to the current engine API (clean cutover)
- iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields;
the literal now spreads `..AgentPolicy::default()` as the engine's own doc
example does, so future fields do not break it again.
- forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own
candidate pool, so it now tags what it knows: PreferenceMatch for the
preference-vector blend, SemanticMatch (with the seed item) for
similar-to-saved, ExplorationBudget for pinned discoveries.
- forage-engine: `url_to_item_id` folded into the u32 item universe. The engine
narrows item IDs to a u32 slot in durable per-user state and rejects anything
above u32::MAX rather than alias two items forever, so every add_item with a
64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone.
- forage-engine: bridge items read the top-2 preference CLUSTERS via
`query_vectors`, not the single centroid from `preference_vectors().get()`.
Since m12 that accessor returns only the strongest cluster, so a tech+jazz user
whose interests split into two clusters looked single-interest and never
bridged. Falls back to top-2 dimensions when a user has one cluster.
Reconcile tests corrected to the shipped contract
- tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed
count after heal. `take_crdt_snapshot` deliberately keys signal contributions
to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from
a single writer, so per-node attribution double-counted every replicated event
on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus
PN-counter per-node max: nodes converge on the more complete accumulator. The
old expectation was asserting the bug that fix removed.
- Rewrote to assert convergence, count survival (not 0), and no inflation, and
added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression
guard for the creep itself, which nothing covered.
Pre-commit hook unified
- hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the
source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the
flag promoted ~58 deliberate pedantic warnings in integration tests to errors,
making every Rust commit impossible.
- It now lints all five tidal crates instead of path-matching `tidal/`, which
silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and
applications/ - the rot above lived in exactly those crates. Ported the
CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the
divergent untracked copy that this replaces.
- CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split.
Fleet recovery and soak
- scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an
ignored tmp/ directory into the repository. Preflights retained storage,
digest-pinned images, parked state, and aggregate plus per-PV-node scheduler
headroom before the first scale; writes a durable transcript under
tmp/restore-logs/ with structured start/error/rollback/complete events.
- k8s manifests park the standalone store, the RF3 cluster, and the soak monitor
at zero replicas with restore-fleet.sh as the only supported scale-up path.
- soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing
restart evidence instead of silently skipping the restart-aware half of the gate.
- docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier
formula and separates analytic totals from the measured process envelope.
Pin the live cluster to the rc7 amd64 image
(@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1,
6-layer manifest, imagetools-verified) carrying the write-burst false-partition
fix (tidal-net record_timeout). Deployed via `kubectl set image`; rolling update
completed 3/3 Ready. Reseeds across the rollout were the safe snapshot-install
fallback (rejoining behind WAL retention), all converged lag=0, no loop, no loss.
A client-side ship DEADLINE means the RPC did not round-trip within
request_timeout — which a slow-but-ALIVE follower produces under a sustained
1536-D ack=quorum apply burst (transport runtime momentarily starved by the
CPU-heavy HNSW apply on its single segment-receiver thread) exactly as a
genuinely blackholed peer does. Counting that as record_failure was the
write-burst false-partition: 5 such opened both followers' breakers, the commit
index stalled, ack=quorum 503'd, and retries re-burst the same starved peers
with no self-heal.
- CircuitBreaker::record_timeout: opens ONLY when the peer shows no recent proof
of life (no round-tripped success/backpressure within reset_duration); neutral
no-op when liveness is fresh; re-opens (never wedges) HalfOpen; never refreshes
the liveness stamp (no reply arrived).
- PeerPool::send_to routes tonic DeadlineExceeded/Cancelled -> record_timeout;
genuine severance still surfaces as connect-level Unavailable/transport reset
-> record_failure and still opens the breaker.
- ship_timeout_breaker.rs: end-to-end proof over a REAL tonic WalShipping server
(handler succeeds once then hangs past the client deadline) + 6 unit tests.
Also: re-scope G-S Scalability guarantee to read-throughput with the Ref-A
tidal-t5-readtput owner-test (write 2.5x is structurally impossible on 3-node
full-placement RF3); bump k8s image to m12-rc6 (live, commit 0919b0a); rustfmt
soak_eval / soak-eval / s3 / tidalctl.
Read-SLA fix (rc12→rc13 — cpu-cgroup starvation → multi-second p99 + churning
elections):
- offload.rs: add SEARCH_GATE semaphore (core_count+1 permits, 50ms shed to 429)
so per-shard searches gate on CPU, not reactor threads; concurrent scatter_merge
fan-out (join_all) replaces the serial blocking offload_region_read loop
- node.rs: scatter_merge → async; per-shard futures run via offload_search
(each acquires one SEARCH_GATE permit, moves it into spawn_blocking so the
permit is held for the search's full CPU lifetime)
- main.rs: explicit tokio runtime with worker_threads floored at 4, independent
of the cgroup quota — keeps the control plane (heartbeat/election/apply) on its
own workers even when quota < 4
- k8s statefulset: CPU limit 2→3 (was: available_parallelism()=2 → only 2 async
workers; search burst starved the reactor)
- tidal/wal/compaction.rs: WAL_RETENTION_SEGMENTS 4→16 (64 MiB→256 MiB per-shard
catch-up window; a briefly-down follower across a rolling restart streams up
instead of forcing snapshot reseed; disk floor 768 MiB/pod, self-trimming)
- cluster_reseed.rs: OFFLINE_ITEMS 1800→5600 to exceed the new 16-segment
retention window (19 segs > 17); fix sequential quarantine/reseed race via
await_status_bool
tidalctl S3/R2 backup DR:
- tidalctl/Cargo.toml: aws-config, aws-sdk-s3, aws-credential-types, tokio, tempfile
- commands/s3.rs: S3Target + export_dir (upload every file, manifest last as
atomicity marker) + import_to_dir (download prefix into temp staging dir)
- commands/backup.rs: run_backup/run_restore accept Option<&S3Target>; S3 export
is additive after local fsync barrier; S3 import stages into TempDir then runs
the unchanged verified restore on it
- main.rs: --s3-endpoint / --s3-bucket / --s3-prefix flags; all-or-nothing
endpoint+bucket validation; usage updated
tidal-stress/k8s: recall-rc12-spread-job, soak-nightly-cronjob, soak-monitor,
soak-results-pvc, t5-readtput-job manifests
Fixes the rc9 over-correction: forcing baseline for ALL nodes (including
caught-up ones) caused needless reseed cascades. Now only divergent nodes
(frontier > baseline) use baseline as the reseed seqno; at/below-baseline
nodes use frontier+1 so the leader picks cheap catch-up vs snapshot.
Also skips the HNSW graph checkpoint on SIGTERM when the shard is reseed-
pending: the in-memory index reflects suspect/divergent data the next boot
discards, so saving it risks a "Failed to read vectors" failure on the
post-reseed open. Durable checkpoints and WAL flush still run.
close_shared() gains a save_graphs bool; shutdown_inner_impl() is the
shared implementation; node.rs passes !reseed_pending.
Root-caused and fixed five sharding bugs exposed on the real k3s 3-shard
cluster (rc5→rc7), plus a divergent-rejoin reseed loop found in rc9:
1. reseed shard-awareness (Bug 3, keystone): `run_boot_install_for_region`
visits each hosted group's own shard subdir; per-group leader discovery
appends `?shard=N` so a divergent shard heals from its own leader (not
shard-0's WAL/term — cross-shard contamination).
2. leader self-join term (Bug 4): `become_leader_for_term` now calls
`note_self_won_term` so the elected shard's `joined_term` is set and
`cluster_promote` routes rebalances correctly (was: topology-era mis-read
→ legacy fenced promote → 500).
3. boot self-heal self-pull guard (Bug 2): `leader_shard != my_shard` gate
prevents a node pulling its own stream (its stream isn't a registered peer)
→ eliminates the `PeerUnreachable(self)` loop.
4. scatter-merge degraded partial (Bug 1): failed shard logs + continues
instead of `?`-failing the whole read; bounded read-admission semaphore
(`offload.rs`) sheds as 429 instead of piling into a 36s p99.
5. WAL retention (Bug 5): `compact_wal_retained` keeps `WAL_RETENTION_SEGMENTS=4`
most-recent sealed segments; online path gets the same retention clamp.
Prevents brief-restart forced-reseed.
6. divergent-rejoin reseed loop (Bug 6, rc9): `note_quarantined` latches
`from_seqno = stream_baseline` (not `frontier + 1`) so `wal_covers`
returns `needed=true` and the snapshot installs instead of looping.
Also: `TidalDb::close_shared` for deterministic HNSW save on cluster SIGTERM
(HNSW graph was not saved when request-scoped Arc clones were alive at shutdown);
updated profiling doc with full rc8/rc9 fix narrative; k8s recall job YAMLs.
Boot now LOADS the per-slot HNSW graph instead of rebuilding it. Clean
shutdown writes {data_dir}/vector/<kind>__<slot>.usearch; the next open loads
it when it matches the durable corpus (seconds), falling back to a full rebuild
only when the graph is missing/stale/corrupt. Eliminates the multi-minute boot
rebuild (~50-70 min at 1M/1536-D) that let the WAL compact past a restarting
node and triggered the reseed cascade.
Graceful SIGTERM now actually runs the close: bounded_drain caps the post-signal
HTTP drain (TIDAL_SHUTDOWN_DRAIN_MS, default 15s) then runs the deterministic
close regardless — sibling keep-alive connections no longer block the drain past
the k8s 60s grace into a SIGKILL (which cannot run Drop). ClusterNode and
ShardReplica::shutdown are now &self (db handle is an ArcSwapOption) so the close
fires even when a stuck connection task holds an Arc.
Fix USearch insert to be a true upsert (remove+add): it was unconditional add,
which a multi:false index rejects on a reseeding follower's post-snapshot WAL
replay -> applied_events stalls -> catch-up deadlock -> unrecoverable cluster.
Also: circuit-breaker peer last-contact tracking; real k3s 1536-dim deploy +
recall findings (recall@10 0.9869, read p99 8.71ms @ 200rps @ 100k) in
docs/profiling/m12-cluster-deploy-findings.md; new tidal-stress k8s jobs and
m12p6 graph-persistence + SIGTERM tier-3 regression tests.
Completes the seed-join-over-TLS enablement begun in 8e39ee1. A real
kubectl scale 3->5 on a real mTLS k8s cluster (kind) exercised the seed-join
path over TLS for the first time and surfaced two more blockers beyond 8e39ee1's
https-seed / ready-only-Service / up-front-rustls-provider fixes — both of which
crash-looped every scale-up joiner with the same opaque 'could not join within
120s'. The plaintext in-process harness is blind to all of them.
- certs.yaml: a real TWO-TIER PKI. The leaf was issued DIRECTLY from a selfSigned
Issuer (a self-signed CA:FALSE end-entity whose ca.crt is a copy of the leaf);
the joiner's strict webpki verifier rejected the peer cert as UnknownIssuer.
Now: selfSigned Issuer -> CA cert (CA:TRUE) -> ca: Issuer signs the leaf.
(scripts/gen-cluster-certs.sh already did this; the two were inconsistent.)
- join_boot.rs: grpc_tls_for() fallback. own_grpc_tls/self_tls_spec looked up the
joiner's OWN region in the knob file to find its TLS material, but a seed-joiner
is NEVER in the shared-ConfigMap regions: list -> None -> the seed client built
with NO CA (the real UnknownIssuer cause) and a plaintext synthesized topology.
Fall back to ANY region's block (every pod mounts the same cert files).
- join_boot.rs: STATUS_POLL_TIMEOUT 500ms -> 5s (env TIDAL_SEED_STATUS_TIMEOUT_MS);
a cold TLS handshake under contention blew the sub-second budget. Discovery now
logs each poll failure at WARN with the full error source chain (a silent loop
made every bug present as the same 120s timeout).
- statefulset.yaml: pin the m12-8e39ee1 server image (carries these fixes).
- k8s/cluster-t4-kind + tidal-stress/k8s/t4-*: local-kind T4 overlay + seed/load.
Verified GREEN on kind: idle scale 3->5, both joiners seed-join over mTLS, catch
up, and flip /health Ready in 13s via the idle-readiness heartbeat convergence;
auto-promote to Voter; full content parity; all 5 regions lag=0. clippy clean;
mp_seed_join_snapshot_catchup + mp_idle_cluster_..._without_traffic green;
tidal-server/tidal-net lib green. A separate, root-caused snapshot-frontier bug
on a DEEPLY-compacted WAL (node.rs:734 last_wal_seq=0 for a state-only artifact)
is documented as a follow-up — left unfixed because a naive patch broke the
in-process snapshot test (own-WAL<->stream numbering); the GREEN run uses a small
corpus (stream catch-up) to keep that path out of scope. See
docs/profiling/m12p5-idle-readiness-elasticity.md §6.
The m12p5 idle-readiness work converged on an idle cluster, but the real
T4 1M/1536 scale-up over mTLS still failed to admit new pods. Three real
blockers, all invisible to the plaintext in-process tests:
- CryptoProvider crash-loop: the seed-join/reseed boot path builds a
blocking reqwest (rustls) HTTPS client on a dedicated boot thread BEFORE
GrpcTransport::new installs the process-wide provider, so every TLS joiner
panicked. Install it at the top of main(); ensure_crypto_provider() is now
pub, idempotent, harmless on the plaintext standalone path.
- Wrong seed scheme + target: peer_url honors an explicit URL scheme
verbatim, so http:// dialed plaintext at the TLS :9500 port. Seed is now
https:// AND points at the ready-only client Service (ClusterIP VIP), not
the headless peers Service — so a joiner never round-robins onto a
not-ready pod (incl. itself) and burns the 120s discovery window.
- Too-tight poll budget: a cold status poll pays a full rustls handshake on
top of DNS+TCP; under CPU contention that alone blew the 500ms budget, so
the joiner timed out every poll for the whole window despite the peer being
reachable. Status-poll timeout is now 5s (env: TIDAL_SEED_STATUS_TIMEOUT_MS)
with a separate 2s connect timeout (dead seeds still fail fast) and
debug-level logging on every discovery failure mode.
Refactors riding along:
- on_heartbeat takes a HeartbeatContext struct (additive fields, no silent
u64 transposition) across tidal-net, election_driver, and both test hooks.
- ShardReplica::applied_for_leader_shard centralizes per-source-shard keying
(BUG 1) shared by the readiness drive and local_status.
- idle-readiness test now asserts convergence within ½ budget — a slow-path
regression (periodic self-heal / status-poll dependency) the binary budget
check would otherwise wave through.
New k8s T4 manifests: cluster-t4-kind kustomization + single-group topology
patch; tidal-stress t4 seed/load Jobs.
Leader heartbeat now carries its live flushed WAL frontier (leader_last_seq,
proto field 14) so a snapshot-installed joiner converges its sticky readiness
latch from the heartbeat — which flows even on a fully idle cluster — instead of
only from observed ship traffic or an external status poll. Fixes the
idle-readiness stall (WORKLOG 2026-06-13: an 11.5h /health 503 hang where a
caught-up joiner never joined the Service VIP).
- proto: HeartbeatRequest.leader_last_seq (field 14); 0 = pre-m12p5 leader → fall
back to the status-poll readiness path
- ElectionHooks::on_heartbeat threads leader_last_seq through net + driver
- ShardReplica::note_leader_frontier_for_readiness folds the frontier into the
lag gauge (monotonic per shard) and drives the readiness latch using a REAL
leader frontier (never the uninitialized-0 gauge, which would false-converge a
still-behind joiner); a joiner that WINS leadership converges trivially
- tier-3 regression: mp_idle_cluster_snapshot_joiner_flips_ready_without_traffic
— snapshot joiner flips /health ready on an idle cluster with zero writes and
no status poll, then proves content parity (honest convergence)
- certs: wildcard pod SAN (*.tidaldb-peers...) in k8s/cluster/certs.yaml and
scripts/gen-cluster-certs.sh so StatefulSet scale-up/down with --seed needs no
cert re-issue (T4 scale-to-5 broke mTLS on tidaldb-3/4); explicit per-pod
names kept as belt-and-suspenders
- docs/profiling/m12p5-idle-readiness-elasticity.md: root-cause + fix writeup
Scale write throughput across data-shard groups while keeping a single unified
read surface:
- scatter_gather.rs: pooled fan-out across shard groups (replaces per-request
client construction); cross-shard query results merged on one node
- cluster/node.rs: cross-shard read routing — a read on any node gathers from
every shard group's leader and unions results
- cluster/forward.rs: fix h2 204 forward-relay bug (relay_forwarded skips body
for 1xx/204/304 — synthesized JSON body on a 204 triggered HTTP/2 RST_STREAM
on the real mTLS plane)
- dto.rs: cross-shard query/result DTOs
- k8s/cluster/: enable 3-group `shards:` topology (statefulset, service-peers,
topology-configmap)
- k8s/cluster-local-kind/: local-kind overlay to run the T5 gate without Ref-A
- tidal-stress/k8s/stress-job-t5.yaml: 2-generator sharded throughput job
- tests: cluster_cross_shard_reads.rs + multiproc support; ran real on kind
- docs/profiling/m12p4-t5-sharded-throughput.md: T5 throughput findings
Switched content_vector to 1536-dim (text-embedding-3-small, thepeach production
width) and ran the realistic peach mix (feed-profile reads + signal writes) on the
m11p6 mTLS cluster.
Result: 1536-dim costs ~nothing on throughput vs 128-dim — knee still ~2,976 rps
(128-dim was 2,981). The write bottleneck is quorum-commit on the 2-worker leader
pool, not vector size. The vector READ path (feed-profile retrieve — the
db.retrieve(profile) path thepeach E2/R8 calls) stays p99 3-11ms through 1500 rps,
never the bottleneck. Memory is the only dim-sensitive resource (567-751 MiB/pod
at 20k items, ~12x 128-dim) — capacity-plan RAM, not throughput.
Recommended sustained target: <=1,000 signal-ingest rps (~1,200 full mix) — 40% of
knee, 2.5x headroom, survives single-node failover, write p99 ~45ms within SLA.
Also: fixed the stale "deployed schema is 128" note in tidal-stress (now reflects
the configurable width). Full writeup: docs/ops/benchmark-1536-peach.md.
Deploying the m11-44b768b image (p6 sharding + p7 mTLS + p8 ops + p9 correctness)
surfaced two blockers; both fixed here.
1. statefulset.yaml: the m11p7 change made the :9500 HTTP plane serve TLS, but the
startup/liveness/readiness probes still used scheme HTTP — kubelet got a TLS
handshake back ("malformed HTTP response \x15\x03\x03") and pods never went
Ready. Set scheme: HTTPS on all three probes (kubelet skips cert verification
for httpGet probes, so the cert's DNS-only SANs are fine). Image pinned to the
m11-44b768b amd64 digest.
2. tidal-stress: the generator's reqwest client did default cert verification and
had no way to trust the cluster's private CA, so https:// targets failed. Added
--ca-cert <pem> (verified TLS against the mounted tidaldb-cluster-tls ca.crt)
and --insecure (skip verification, escape hatch). New StressError::CaCert for
the PEM read fault.
stress-job-m11p6-baseline.yaml: T2-A-equivalent quorum-write throughput run on the
new stack — https:// targets, ca.crt mounted from the tidaldb-cluster-tls Secret,
--ca-cert verified TLS. Drops the removed --write-path flag (m11p6 unified the
write path to hash-routing).
End the "replicated XOR sharded" split: S shard groups, each a
replication group at RF with its own elected leader, leaders balanced
across nodes; any gateway hash-routes.
- One unified write surface: /items,/embeddings,/signals hash-route to
the owning shard group's leader (ShardRouter FNV-1a) AND replicate at
RF. x-tidal-ack/x-tidal-seq, quorum await, NotLeader/QuorumTimeout are
per-group; NotLeader names the group.
- Rebalance verbs (L3): POST /cluster/shards/{id}/transfer (fenced
leadership move) + /cluster/shards/{id}/replicas (add/remove replica).
A ?shard= selector threads through every per-shard admin verb and is
propagated on intra-group forwards (ShardReplica::admin_path). S=1 is
byte-for-byte (no selector, no shard in NotLeader body).
- Tier-3 exit gate (cluster_sharding.rs): 3 nodes × 3 shards × RF=3 over
real OS processes — SIGKILL a node under ack=quorum load → only its
shard-leaderships re-elect, reads never stop, zero acked loss across
random kill points; plus a rebalance-verb test. Harness:
MultiProcCluster::start_sharded.
- tidal-stress drives the single path (WritePath::Leader|Sharded gone),
spreading writes round-robin across gateways or pinning --leader-url.
- Throughput: local 3×3 sustains 3,000 quorum signal-writes/s @ 0% err,
~30% CPU, lag ~0 (generator-bound). ≥5,000/s + ≥2.5× scaling is Ref-A.
Known follow-up (tracked): per-group-aware node readiness and cross-node
read fan-out under PARTIAL placement.
