0634d2f4dc
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jordan
|
c97aaa8e5b |
fleet remediation: make the workspace gate runnable, then fix what it caught
`cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. |
||
|
jx12n
|
580142df49 |
feat(m12): election-divergence-fix + soak-eval streak + release tooling
Durable `leader_acked` frontier in `ShardReplica` tracks the highest seqno acked under `ack=leader` (journal-only, un-replicated); `decide_join` now quarantines on THIS node's own frontier rather than comparing stream numbers across stream boundaries — eliminates false-quarantine churn on rolling restarts. `SHUTDOWN_HANDOFF_WAIT` (3s) drains the leader's tail to quorum before step-down so the next leader inherits a clean prefix. New `load_leader_acked`/`persist_leader_acked` helpers; `cluster_reseed.rs` gains the divergence-fix regression suite; `replication_ops.rs` threads the signal. Soak-eval: `tidal_stress::soak_eval` + `soak-eval` binary implement the 30-night streak (ledger.tsv × restarts.tsv → streak.tsv); monitor and nightly CronJob k8s YAMLs updated; phase-9 doc clarifies the dual-stream streak definition (ledger PASS AND zero pod restarts in window). `run-reliability.sh` gates the election-divergence suite before any k8s push. Release tooling: `docker/release/` multi-stage Dockerfile + DR image; `scripts/build-release.sh` single repeatable cross-compile+buildx path. |
||
|
jx12n
|
f640764d89 |
feat(tidal-stress): open-loop capacity load generator (thepeach feed workload)
New workspace crate: an open-loop, coordinated-omission-corrected HTTP load generator + capacity ramp for the standalone and multi-process cluster surfaces, modeling a thepeach feed session (feed reads + view/like/skip signals + search, signal-dominated per their user-graph spec). Throttleable target rate, ramp presets (smoke/quick/peach-100k/max) or rps:secs specs, peach/reads/writes/custom mixes, leader vs sharded write paths, per-op p50/p90/p99/p999/max latency, a backpressure-aware status breakdown (429/408/503/4xx/5xx/transport), and a verdict translated to supported DAU. Runs in-cluster as a k8s Job (tidal-stress/k8s/). Open-loop scheduler (scheduler.rs) fires at a fixed arrival rate and measures latency from each request's intended send time, so a server stall inflates the percentiles a closed-loop test hides; it shed-and-counts rather than blocking when the in-flight cap is reached. Pure-Rust (tokio + reqwest/rustls), no engine deps. Findings on the live 3-region k3s cluster (docs/ops/stress-test-thepeach.md): reads scale to thousands/s at <15ms p99; the replicated /signals path saturates at ~90 signals/s (single-leader funnel + 2-worker write pool + synchronous gRPC ship); the sharded path sustains 3,669 signals/s at 0 errors and ~27% cluster CPU (≈ the 100k-DAU peak, knee not reached). Overload degrades gracefully (429; 0 pod restarts). thepeach's planned in-process embedding sidesteps all of it (write ≈82ns). |