`cargo test --workspace` could not run at all: dependency resolution failed with
"aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the
gate the project documents was dead. Making it run exposed a compile break and
two wrong tests that had been invisible for months. Now green end to end:
143 suites, 3155 tests, exit 0.
Toolchain
- rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays
`rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK
chain needs the patch release, and it now declares that itself.
Consumer crates migrated to the current engine API (clean cutover)
- iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields;
the literal now spreads `..AgentPolicy::default()` as the engine's own doc
example does, so future fields do not break it again.
- forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own
candidate pool, so it now tags what it knows: PreferenceMatch for the
preference-vector blend, SemanticMatch (with the seed item) for
similar-to-saved, ExplorationBudget for pinned discoveries.
- forage-engine: `url_to_item_id` folded into the u32 item universe. The engine
narrows item IDs to a u32 slot in durable per-user state and rejects anything
above u32::MAX rather than alias two items forever, so every add_item with a
64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone.
- forage-engine: bridge items read the top-2 preference CLUSTERS via
`query_vectors`, not the single centroid from `preference_vectors().get()`.
Since m12 that accessor returns only the strongest cluster, so a tech+jazz user
whose interests split into two clusters looked single-interest and never
bridged. Falls back to top-2 dimensions when a user has one cluster.
Reconcile tests corrected to the shipped contract
- tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed
count after heal. `take_crdt_snapshot` deliberately keys signal contributions
to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from
a single writer, so per-node attribution double-counted every replicated event
on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus
PN-counter per-node max: nodes converge on the more complete accumulator. The
old expectation was asserting the bug that fix removed.
- Rewrote to assert convergence, count survival (not 0), and no inflation, and
added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression
guard for the creep itself, which nothing covered.
Pre-commit hook unified
- hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the
source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the
flag promoted ~58 deliberate pedantic warnings in integration tests to errors,
making every Rust commit impossible.
- It now lints all five tidal crates instead of path-matching `tidal/`, which
silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and
applications/ - the rot above lived in exactly those crates. Ported the
CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the
divergent untracked copy that this replaces.
- CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split.
Fleet recovery and soak
- scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an
ignored tmp/ directory into the repository. Preflights retained storage,
digest-pinned images, parked state, and aggregate plus per-PV-node scheduler
headroom before the first scale; writes a durable transcript under
tmp/restore-logs/ with structured start/error/rollback/complete events.
- k8s manifests park the standalone store, the RF3 cluster, and the soak monitor
at zero replicas with restore-fleet.sh as the only supported scale-up path.
- soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing
restart evidence instead of silently skipping the restart-aware half of the gate.
- docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier
formula and separates analytic totals from the measured process envelope.
Reconciles two independently-developed lines from base 006d3d0:
ours — M9/M10 community layers, retroactive purge + re-materialization,
signal revocation, agent capability boundaries, P1 feedback loop,
reason labels, instrumented metrics
theirs — M11/M12 cluster mode (tidal-net gRPC transport, tidal-server
cluster/scatter-gather, tidal-stress), multi-vector preference,
ANN candidate-gen, warm-tier day buckets, keyed signal snapshots
Notable semantic resolutions:
* storage::keys::Tag — both sides allocated 0x0E..0x11 for different
records. Kept theirs' 0x0E..0x1A (shipped on-disk format) and renumbered
ours to 0x1B..0x1E (CommunityMembership/Revocation/PurgeManifest/
CommunityLeave); Tag::ALL grown to 30 so the contiguity drift guard holds.
* ranking executor — took theirs' rewrite (SignalReadPlan pre-pass, keyed
SignalKey snapshots, Result-returning reads, finalize()) and re-applied
ours' M10 read suppression at the chokepoints it introduced:
single_signal_score, score_hot/trending/controversial,
CreatorEngagementRate, and the Stage-4 boost loop.
* signals::warm — theirs' day-bucket/read-time-rotation rewrite, with ours'
subtract_bucket and Clone extended to the new day tier; ours' test split
kept (warm/tests.rs, warm/proptests.rs) carrying theirs' updated bodies.
* db::signals — kept ours' contribution-logging try_cohort_attribution in
signal_dispatch.rs and theirs' event-time try_update_preference_vector;
dropped the superseded duplicates.
* db::mod / from_parts — theirs' constructors, with ours' purge/
re-materialization/revocation/community/skip-counter fields and restart
rebuilds; from_parts kept in its own file per the 600-line guideline.
* schema::validation::builders — ours' module split with theirs' expanded
tests; policy validation runs both sides' checks (read-signal lists +
profile overrides, then the zero-duration limit guard).
* feedback Unhide no longer writes a -1.0 "hide" signal: theirs' engine
rejects negative weights (spec §8). Reverses index state only, matching
every other undo action.
* SessionState::new is now the single construction path (gains
overrides_rejected/default_profile); AuditEntry gains kind on the
deserialize path, inferred from the accepted flag as before.
* Removed tidal/src/replication/tcp_transport.rs and its test: never
declared in replication/mod.rs on either branch, so it had never
compiled and nothing referenced it. Superseded by tidal-net's
GrpcTransport.
Verified: cargo clippy -p tidaldb (lib) clean; --all-targets compiles for
tidaldb/tidal-net/tidal-server/tidal-stress; 2094/2094 lib tests and the
integration suite pass except m8p3_reconcile_production's two CRDT-count
assertions, which fail identically on MERGE_HEAD (pre-existing).
tidalctl cannot build locally: its aws-sdk deps need rustc 1.91.1, local
toolchain is 1.91.0.
Consumers (e.g. thepeach) need a PULLABLE tidaldb image — not every dev has the
source for a compose `build:` context. Publishes registry.threesix.ai/tidal/standalone
as a manifest list: linux/amd64 (Linux/CI) + linux/arm64 (Apple-silicon Macs),
defaulting to STANDALONE mode (:9400, --data-dir /data) so it drops into a consumer's
docker-compose like postgres.
Hybrid build because `rustc` SIGSEGVs under QEMU (an in-container amd64 cross-build on
an arm64 host fails):
- amd64: docker/standalone/amd64.Dockerfile packages the HOST cross-compiled x86_64
binary (the proven build-release path; trixie-slim for glibc 2.41/libmvec) — apt+COPY only.
- arm64: native in-container build from docker/standalone/Dockerfile.
- scripts/build-standalone-image.sh stitches both into one manifest list (:m12 + :latest).
Verified: pulled :m12 (arm64), booted standalone, GET /health -> {ok:true,mode:standalone}.
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno
acked under `ack=leader` (journal-only, un-replicated); `decide_join` now
quarantines on THIS node's own frontier rather than comparing stream numbers
across stream boundaries — eliminates false-quarantine churn on rolling
restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum
before step-down so the next leader inherits a clean prefix. New
`load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains
the divergence-fix regression suite; `replication_ops.rs` threads the signal.
Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the
30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly
CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak
definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh`
gates the election-divergence suite before any k8s push.
Release tooling: `docker/release/` multi-stage Dockerfile + DR image;
`scripts/build-release.sh` single repeatable cross-compile+buildx path.
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load
generator + capacity ramp for the standalone and multi-process cluster surfaces,
modeling a thepeach feed session (feed reads + view/like/skip signals + search,
signal-dominated per their user-graph spec). Throttleable target rate, ramp
presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom
mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a
backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict
translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/).
Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures
latency from each request's intended send time, so a server stall inflates the
percentiles a closed-loop test hides; it shed-and-counts rather than blocking when
the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps.
Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md):
reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at
~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship);
the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the
100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod
restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns).
rust:1.91-slim doesn't include a C++ compiler, which cc-rs needs.
The standalone/cluster Dockerfiles use the full rust:1.91 image so
they weren't affected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clap now reads PORT from the environment, accepting either a bare port
number (e.g. 8080 -> 0.0.0.0:8080) or a full host:port. CLI --listen
flag still takes precedence. Deploy Dockerfile defaults PORT=9500 and
removes the hardcoded --listen argument.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docker/standalone/Dockerfile stays generic (port 9400, baked-in config,
auth healthcheck) for anyone running tidalDB independently.
docker/deploy/Dockerfile is purpose-built for k8s: port 9500, schema
mounted at runtime, persistent data dir, fixed UID for PVC ownership.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add /health/startup and /health/live probes, flip readiness to 503 on
SIGTERM so k8s stops routing before drain. Update standalone Dockerfile
for internal deployment: port 9500, schema mounted at runtime (not baked
in), persistent data dir, non-root user with fixed UID 10001.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract redeliver_missed(tx, db, log) helper into cluster_transport.rs
- heal_region now removes partition then immediately ships any missed
batch-log entries to the healed follower's channel
- await_convergence refactored to call the same helper (no logic change)
- tidal-server: reload_text_index before search in cluster mode
- tidal-server: write_signal returns Result instead of panicking on unknown signal
- tidal-server: leader shows lag_events=0 (writes directly, no receiver thread)
- tidal-server: fix cluster mode error propagation (ServerError::from)
- docs/runbooks/cluster.md: add full cluster operations runbook
- docker/: add Dockerfile for containerised cluster deployment
- README.md: add tidal-server HTTP API getting-started section
- Split oversized source files per CODING_GUIDELINES §9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
