From c97aaa8e5bd30121280922a766cb3e340043c4aa Mon Sep 17 00:00:00 2001 From: jordan Date: Sun, 16 Aug 2026 12:38:14 -0600 Subject: [PATCH] fleet remediation: make the workspace gate runnable, then fix what it caught `cargo test --workspace` could not run at all: dependency resolution failed with "aws-types@1.3.16 requires rustc 1.91.1" on the 1.91.0 default toolchain, so the gate the project documents was dead. Making it run exposed a compile break and two wrong tests that had been invisible for months. Now green end to end: 143 suites, 3155 tests, exit 0. Toolchain - rust-toolchain.toml pins the DEV toolchain to 1.91.1. The published MSRV stays `rust-version = "1.91"` (the engine builds on 1.91.0); only tidalctl's AWS SDK chain needs the patch release, and it now declares that itself. Consumer crates migrated to the current engine API (clean cutover) - iknowyou-engine: `AgentPolicy` gained five m10 read/profile-override fields; the literal now spreads `..AgentPolicy::default()` as the engine's own doc example does, so future fields do not break it again. - forage-engine: `RetrieveResult` gained p1 `reasons`. The app builds its own candidate pool, so it now tags what it knows: PreferenceMatch for the preference-vector blend, SemanticMatch (with the seed item) for similar-to-saved, ExplorationBudget for pinned discoveries. - forage-engine: `url_to_item_id` folded into the u32 item universe. The engine narrows item IDs to a u32 slot in durable per-user state and rejects anything above u32::MAX rather than alias two items forever, so every add_item with a 64-bit FNV hash failed. 9 of 28 smoke tests were failing on this alone. - forage-engine: bridge items read the top-2 preference CLUSTERS via `query_vectors`, not the single centroid from `preference_vectors().get()`. Since m12 that accessor returns only the strongest cluster, so a tech+jazz user whose interests split into two clusters looked single-interest and never bridged. Falls back to top-2 dimensions when a user has one cluster. Reconcile tests corrected to the shipped contract - tidal/tests/m8p3_reconcile_production.rs asserted `3 + 5 == 8` for a windowed count after heal. `take_crdt_snapshot` deliberately keys signal contributions to ONE canonical contributor (ShardId::SINGLE) because signals are relayed from a single writer, so per-node attribution double-counted every replicated event on every reconcile. Merge is therefore LWW on (last_update_ns, score) plus PN-counter per-node max: nodes converge on the more complete accumulator. The old expectation was asserting the bug that fix removed. - Rewrote to assert convergence, count survival (not 0), and no inflation, and added `repeated_reconcile_of_converged_nodes_does_not_creep` - the regression guard for the creep itself, which nothing covered. Pre-commit hook unified - hooks/pre-commit dropped `-D warnings`: each crate's `[lints]` table is the source of truth (`clippy::all`/`unwrap_used` deny, `pedantic` warn), and the flag promoted ~58 deliberate pedantic warnings in integration tests to errors, making every Rust commit impossible. - It now lints all five tidal crates instead of path-matching `tidal/`, which silently skipped tidal-server, tidal-net, tidal-stress, tidalctl and applications/ - the rot above lived in exactly those crates. Ported the CODING_GUIDELINES file-length, println, and unsafe-SAFETY checks from the divergent untracked copy that this replaces. - CONTRIBUTING.md now documents the real commands and the toolchain/MSRV split. Fleet recovery and soak - scripts/restore-fleet.sh: the fail-closed selective restore, promoted out of an ignored tmp/ directory into the repository. Preflights retained storage, digest-pinned images, parked state, and aggregate plus per-PV-node scheduler headroom before the first scale; writes a durable transcript under tmp/restore-logs/ with structured start/error/rollback/complete events. - k8s manifests park the standalone store, the RF3 cluster, and the soak monitor at zero replicas with restore-fleet.sh as the only supported scale-up path. - soak-eval/soak-watch and the nightly CronJob fail closed on stale or missing restart evidence instead of silently skipping the restart-aware half of the gate. - docs/ops/capacity-planning.md corrects the RAM envelope to the real hot-tier formula and separates analytic totals from the measured process envelope. --- CONTRIBUTING.md | 36 +- Cargo.lock | 1 + Cargo.toml | 3 + applications/forage/engine/src/lib.rs | 79 ++- applications/forage/engine/src/seed.rs | 23 +- applications/iknowyou/engine/src/lib.rs | 3 + docker/stress/Dockerfile | 5 + docs/ops/capacity-planning.md | 77 ++- docs/ops/prometheus-alerts.yaml | 44 +- docs/planning/ROADMAP.md | 2 +- docs/planning/milestone-11/phase-9.md | 30 +- docs/runbooks/cluster.md | 38 +- hooks/pre-commit | 74 ++- k8s/cluster/statefulset.yaml | 49 +- k8s/statefulset.yaml | 12 +- rust-toolchain.toml | 14 + scripts/build-release.sh | 71 +- scripts/restore-fleet.sh | 523 +++++++++++++++ tidal-server/src/cluster/node.rs | 79 ++- .../tests/cluster_graph_persistence.rs | 4 +- tidal-stress/Cargo.toml | 1 + tidal-stress/k8s/soak-monitor.yaml | 76 +-- tidal-stress/k8s/soak-nightly-cronjob.yaml | 163 +++-- tidal-stress/k8s/soak-results-pvc.yaml | 2 + tidal-stress/src/bin/soak-eval.rs | 114 +++- tidal-stress/src/bin/soak-watch.rs | 304 +++++++++ tidal-stress/src/soak_eval.rs | 613 +++++++++++++----- tidal/tests/m8p3_reconcile_production.rs | 169 +++-- tidalctl/Cargo.toml | 6 +- 29 files changed, 2101 insertions(+), 514 deletions(-) create mode 100644 rust-toolchain.toml create mode 100755 scripts/restore-fleet.sh create mode 100644 tidal-stress/src/bin/soak-watch.rs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1ef8f94..388c881 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,6 +14,15 @@ cargo test --doc --manifest-path tidal/Cargo.toml cargo test --examples --manifest-path tidal/Cargo.toml ``` +### Toolchain + +`rust-toolchain.toml` pins the development toolchain to **1.91.1**; rustup honors +it automatically. That pin is not the library's MSRV — `tidaldb` and its siblings +publish `rust-version = "1.91"` and build on 1.91.0. The extra patch release is +required by `tidalctl`'s AWS SDK chain (`aws-types` declares 1.91.1), and without +it every workspace-wide command fails during dependency resolution instead of +compiling, which silently disables the gates below. + ## Run Samples Checklist Before opening a PR that touches public API or examples, verify all samples still work: @@ -43,21 +52,28 @@ tidalDB opened, verified, and closed. M0 complete. ## Full Quality Gate -The pre-commit hook enforces these automatically on staged Rust files: +The pre-commit hook (`hooks/pre-commit`, activated by `scripts/install-hooks.sh`) +enforces these on staged Rust files: ```bash -cargo fmt --manifest-path tidal/Cargo.toml -- --check -cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings -cargo test --manifest-path tidal/Cargo.toml --lib +cargo fmt +cargo clippy -p tidaldb -p tidal-net -p tidal-server -p tidal-stress -p tidalctl --all-targets +cargo test -p tidaldb --lib ``` -Run the complete gate manually: +Clippy runs **without** `-D warnings` on purpose. Each crate's `[lints]` table is +the single source of truth: `clippy::all` and `unwrap_used` are `deny` (so they +fail the build), while `pedantic` and `nursery` are `warn` (advisory). Passing +`-D warnings` on the command line overrides that and turns ~58 deliberate +pedantic warnings in the integration tests into hard errors. + +Run the complete gate manually, across the whole workspace: ```bash -cargo fmt --manifest-path tidal/Cargo.toml -cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings -cargo test --manifest-path tidal/Cargo.toml -cargo bench --manifest-path tidal/Cargo.toml --no-run # ensure benches compile +cargo fmt --check +cargo clippy --workspace --all-targets +cargo test --workspace +cargo bench --manifest-path tidal/Cargo.toml --no-run # ensure benches compile ``` ## Project Layout @@ -75,4 +91,4 @@ Key rules: - `Result` everywhere — no panics on recoverable failures - `#![forbid(unsafe_code)]` — relaxed only at explicit FFI boundaries with `// SAFETY:` comment - Property tests for invariants, criterion benchmarks for performance claims -- `cargo clippy -D warnings` must pass with zero warnings +- Deny-level clippy (`clippy::all`, `unwrap_used`) must pass; pedantic/nursery are advisory diff --git a/Cargo.lock b/Cargo.lock index 879ed17..85db1a7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4282,6 +4282,7 @@ dependencies = [ "serde", "serde_json", "thiserror 2.0.18", + "time", "tokio", "tracing", "tracing-subscriber", diff --git a/Cargo.toml b/Cargo.toml index ff8c71d..52b8d46 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,5 +14,8 @@ resolver = "2" [workspace.package] edition = "2024" +# Published MSRV for the engine and its siblings; they compile on 1.91.0. The +# dev toolchain is pinned separately in rust-toolchain.toml because `tidalctl` +# needs a patch release above this floor - see that file and tidalctl/Cargo.toml. rust-version = "1.91" license = "MIT" diff --git a/applications/forage/engine/src/lib.rs b/applications/forage/engine/src/lib.rs index 7fdbcca..506baf5 100644 --- a/applications/forage/engine/src/lib.rs +++ b/applications/forage/engine/src/lib.rs @@ -32,6 +32,7 @@ const BRIDGE_MIN_CATEGORY_SCORE: f32 = 0.05; const EMBEDDER_TIMEOUT_SECS: u64 = 10; use tidaldb::TidalDb; +use tidaldb::ranking::reason::{ReasonCode, ReasonLabel}; use tidaldb::schema::{EntityId, Timestamp}; pub use labels::{ForageItem, ItemLabel}; @@ -797,11 +798,23 @@ impl ForageEngine { 1.0 }; let score = boost * semantic_sim + (1.0 - boost) * rank_score; + // The pool is app-built, so the engine cannot tag it: + // record why each item is here. `weight` is the share of + // the blended score that semantic similarity supplied. + let semantic_share = if score > 0.0 { + (boost * semantic_sim / score).clamp(0.0, 1.0) + } else { + f64::from(self.mab.semantic_boost) + }; RetrieveResult { entity_id: r.entity_id, score, rank: r.rank, signals: vec![], + reasons: vec![ReasonLabel::new( + ReasonCode::PreferenceMatch, + semantic_share, + )], } }) .collect(); @@ -874,6 +887,12 @@ impl ForageEngine { score, rank: r.rank, signals: vec![], + reasons: vec![ReasonLabel::with_context( + ReasonCode::SemanticMatch, + score, + "similar_to", + &saved_id.to_string(), + )], }); } } @@ -902,6 +921,8 @@ impl ForageEngine { score: 0.5, rank: 0, signals: vec![], + // Pinned by the discovery mechanism, not by ranking. + reasons: vec![ReasonLabel::new(ReasonCode::ExplorationBudget, 1.0)], }); } } @@ -1137,21 +1158,59 @@ impl ForageEngine { "health", ]; - let pref = self.db.preference_vectors().get(user_id)?; + // Since m12 the engine may model one user as several preference clusters, + // and `preference_vectors().get()` returns only the strongest cluster's + // centroid. Through that accessor a tech+jazz user whose interests split + // into two clusters looks single-interest and never bridges. Ask for the + // top-2 cluster vectors instead; the accessor's uniform fallback returns + // one vector for cold-start users. + let vectors = + self.db + .preference_vectors() + .query_vectors(user_id, Timestamp::now().as_nanos(), 2); + let pref = vectors.first()?.clone(); // Build the midpoint vector and extract category names. // Two cases: category-axis 8-dim embeddings (P0–P3) vs real embedder. let (cat_a, cat_b, midpoint) = if pref.len() == CATS.len() { - // 8-dim: find the top-2 dimensions by preference score. - let mut indexed: Vec<(usize, f32)> = pref.iter().copied().enumerate().collect(); - indexed.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); - let (dim_a, score_a) = indexed[0]; - let (dim_b, score_b) = indexed[1]; + let dominant_dim = |v: &[f32]| -> Option<(usize, f32)> { + v.iter() + .copied() + .enumerate() + .max_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal)) + }; - // Require both categories to have meaningful signal. - if score_a < BRIDGE_MIN_CATEGORY_SCORE || score_b < BRIDGE_MIN_CATEGORY_SCORE { - return None; - } + // Two distinct clusters are two interests the user actually holds - + // the honest pair to bridge. Fall back to the two strongest + // dimensions inside one cluster when there is only one. + let across_clusters = match vectors.get(1) { + Some(second) => { + let (dim_a, score_a) = dominant_dim(&pref)?; + let (dim_b, score_b) = dominant_dim(second)?; + (dim_a != dim_b + && score_a >= BRIDGE_MIN_CATEGORY_SCORE + && score_b >= BRIDGE_MIN_CATEGORY_SCORE) + .then_some((dim_a, dim_b)) + } + None => None, + }; + + let (dim_a, dim_b) = match across_clusters { + Some(pair) => pair, + None => { + let mut indexed: Vec<(usize, f32)> = pref.iter().copied().enumerate().collect(); + indexed + .sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); + let (dim_a, score_a) = indexed[0]; + let (dim_b, score_b) = indexed[1]; + + // Require both categories to have meaningful signal. + if score_a < BRIDGE_MIN_CATEGORY_SCORE || score_b < BRIDGE_MIN_CATEGORY_SCORE { + return None; + } + (dim_a, dim_b) + } + }; let cat_a = CATS.get(dim_a).copied()?.to_string(); let cat_b = CATS.get(dim_b).copied()?.to_string(); diff --git a/applications/forage/engine/src/seed.rs b/applications/forage/engine/src/seed.rs index 6174723..36743ed 100644 --- a/applications/forage/engine/src/seed.rs +++ b/applications/forage/engine/src/seed.rs @@ -14,22 +14,27 @@ pub struct SeedItem { pub url: String, } -/// Map a URL to a stable item ID using FNV-1a. +/// Map a URL to a stable item ID using FNV-1a, folded into the engine's item +/// universe. /// -/// Produces a deterministic, collision-resistant u64 that is always > 100_000, -/// keeping it well clear of the seed corpus range (1–100). +/// Produces a deterministic ID that is always `> 100_000` (clear of the seed +/// corpus range 1-100) and always `<= u32::MAX`. The upper bound is a hard +/// engine contract, not a style choice: durable per-user state (seen / saved / +/// liked / hard-negative rows) narrows an item ID to a `u32` slot, so +/// `write_item_with_metadata` and every `for_user` signal reject an ID above +/// `u32::MAX` rather than alias two items forever. The fold xors both halves of +/// the hash so entropy from the high word is not discarded. pub fn url_to_item_id(url: &str) -> u64 { + /// Highest ID reserved for the seed corpus and its headroom. + const SEED_CEILING: u32 = 100_000; + let mut hash: u64 = 14_695_981_039_346_656_037; for byte in url.bytes() { hash ^= u64::from(byte); hash = hash.wrapping_mul(1_099_511_628_211); } - // Map tiny hashes out of the seed range without clustering. - if hash <= 100_000 { - hash.wrapping_add(100_001) - } else { - hash - } + let folded = ((hash >> 32) ^ (hash & 0xffff_ffff)) as u32; + u64::from(SEED_CEILING + 1 + folded % (u32::MAX - SEED_CEILING)) } const CATEGORIES: &[&str] = &[ diff --git a/applications/iknowyou/engine/src/lib.rs b/applications/iknowyou/engine/src/lib.rs index c2c54c3..855809e 100644 --- a/applications/iknowyou/engine/src/lib.rs +++ b/applications/iknowyou/engine/src/lib.rs @@ -752,6 +752,9 @@ fn build_schema() -> std::result::Result { denied_signals: vec![], max_session_duration: Duration::from_secs(60 * 60), max_signals_per_session: 1_000, + // Read-path and profile-override controls (m10) stay at their + // defaults: this app writes signals and never overrides profiles. + ..AgentPolicy::default() }, ); diff --git a/docker/stress/Dockerfile b/docker/stress/Dockerfile index c1ae0fa..33157e7 100644 --- a/docker/stress/Dockerfile +++ b/docker/stress/Dockerfile @@ -21,6 +21,8 @@ RUN cargo build -p tidal-stress --release --locked FROM debian:trixie-slim ARG DEBIAN_FRONTEND=noninteractive +ARG GIT_HASH=unknown +LABEL org.opencontainers.image.revision="${GIT_HASH}" WORKDIR /srv # ca-certificates only matters if a target is https:// (in-cluster targets are @@ -34,6 +36,9 @@ COPY --from=builder /app/target/release/tidal-stress /usr/local/bin/tidal-stress # tidal-stress/src/bin/soak-eval.rs). The soak monitor's evaluator container runs # it to join ledger.tsv + restarts.tsv into the honest streak verdict. COPY --from=builder /app/target/release/soak-eval /usr/local/bin/soak-eval +# soak-watch: direct in-cluster Kubernetes restart evidence recorder. Keeping +# it in this image removes the mutable shell+kubectl sidecar dependency. +COPY --from=builder /app/target/release/soak-watch /usr/local/bin/soak-watch USER stress diff --git a/docs/ops/capacity-planning.md b/docs/ops/capacity-planning.md index 57185a7..4556e7e 100644 --- a/docs/ops/capacity-planning.md +++ b/docs/ops/capacity-planning.md @@ -10,17 +10,17 @@ All estimates assume a single-node deployment with default configuration (30-sec tidalDB is an in-memory-first database. USearch HNSW indexes, the signal ledger hot tier, and Tantivy reader segments all reside in RAM during operation. There is no swap tolerance for USearch -- if the process is swapped, ANN query latency degrades from microseconds to seconds. -| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 signals) | Tantivy RAM | Total Estimate | -|------:|---------------:|------------:|-------------------------------:|------------:|---------------:| -| 100K | 128D | ~26 MB | ~110 MB | ~50 MB | ~200 MB | -| 100K | 768D | ~154 MB | ~110 MB | ~50 MB | ~320 MB | -| 100K | 1536D | ~307 MB | ~110 MB | ~50 MB | ~470 MB | -| 1M | 128D | ~256 MB | ~1.1 GB | ~200 MB | ~1.6 GB | -| 1M | 768D | ~1.5 GB | ~1.1 GB | ~200 MB | ~2.8 GB | -| 1M | 1536D | ~3.1 GB | ~1.1 GB | ~200 MB | ~4.4 GB | -| 10M | 128D | ~2.6 GB | ~11 GB | ~500 MB | ~14 GB | -| 10M | 768D | ~15 GB | ~11 GB | ~500 MB | ~27 GB | -| 10M | 1536D | ~31 GB | ~11 GB | ~500 MB | ~43 GB | +| Items | Embedding Dims | USearch RAM | Signal Ledger RAM (10 active signals) | Tantivy RAM | Analytic Total | +|------:|---------------:|------------:|--------------------------------------:|------------:|---------------:| +| 100K | 128D | ~31 MB | ~1.09 GB | ~50 MB | ~1.17 GB | +| 100K | 768D | ~184 MB | ~1.09 GB | ~50 MB | ~1.32 GB | +| 100K | 1536D | ~369 MB | ~1.09 GB | ~50 MB | ~1.51 GB | +| 1M | 128D | ~307 MB | ~5.44 GB (hot-tier cap) | ~200 MB | ~5.95 GB | +| 1M | 768D | ~1.84 GB | ~5.44 GB (hot-tier cap) | ~200 MB | ~7.48 GB | +| 1M | 1536D | ~3.69 GB | ~5.44 GB (hot-tier cap) | ~200 MB | ~9.33 GB | +| 10M | 128D | ~3.07 GB | ~5.44 GB (hot-tier cap) | ~500 MB | ~9.01 GB | +| 10M | 768D | ~18.43 GB | ~5.44 GB (hot-tier cap) | ~500 MB | ~24.37 GB | +| 10M | 1536D | ~36.86 GB | ~5.44 GB (hot-tier cap) | ~500 MB | ~42.80 GB | ### Formulas @@ -35,12 +35,22 @@ The 20% graph overhead accounts for HNSW neighbor lists (M=16 default, two layer **Signal ledger hot tier:** ``` -items * signal_count * ~1,088 bytes/entry +min(items * active_signal_types_per_item, DEFAULT_MAX_SIGNAL_ENTRIES) + * ~1,088 bytes/entry ``` -Each `(entity_id, signal_type_id)` entry in the DashMap holds the running decay score, windowed counters (BucketedCounter with minute and hour buckets), velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure was measured in the m7p3 scale benchmarks. +Each `(entity_id, signal_type_id)` entry in the DashMap holds the running decay +score, windowed counters (BucketedCounter with minute and hour buckets), +velocity state, and the DashMap per-shard overhead. The 1,088 bytes/entry figure +was measured in the m7p3 scale benchmarks. The table uses ten active signal +types per item and the default 5M-entry hot-tier ceiling. Without trimming, the +signal column would be ~10.88 GB at 1M items and ~108.8 GB at 10M items. -The signal ledger has a memory budget of 5M entries (`DEFAULT_MAX_SIGNAL_ENTRIES`). When exceeded, the checkpoint thread evicts cold entries (oldest `last_update` timestamp). If your workload has more than 5M active `(entity, signal_type)` pairs, cold entries will be served from fjall checkpoints (slower, but correct). +The signal ledger has a memory budget of 5M entries +(`DEFAULT_MAX_SIGNAL_ENTRIES`, ~5.44 GB). When exceeded, the checkpoint thread +evicts cold entries (oldest `last_update` timestamp). If your workload has more +than 5M active `(entity, signal_type)` pairs, cold entries will be served from +fjall checkpoints (slower, but correct). **Tantivy text index:** @@ -49,8 +59,14 @@ Tantivy's RAM usage depends on the number of indexed documents, average document ### Notes - Signal ledger RAM is for the in-memory hot tier only. The WAL and fjall checkpoints add disk usage, not RAM. -- The "10 signals" column assumes 10 distinct signal types per entity. Scale linearly for more signal types. -- USearch RAM is the dominant cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), plan for USearch to consume 70%+ of total RAM at 10M items. +- The table assumes ten active signal types per item. Below the 5M-entry hot-tier + cap it scales linearly; beyond the cap, additional cold entries trade RAM for + checkpoint-backed lookup cost. +- USearch RAM is the dominant index cost at high dimensionality. If you use 1536D embeddings (e.g., OpenAI text-embedding-3-large), budget from the complete analytic total and measured process envelope, not the vector column alone. +- The analytic table excludes process/runtime overhead, replication and catch-up + buffers, allocator fragmentation, and concurrent query/write working memory. + It is a sizing input, not a container limit. Use the measured envelope below + for the multi-process cluster. --- @@ -175,8 +191,8 @@ The tables above are single-node, analytic estimates. This section is the **meas | metric | value | note | |--------|-------|------| -| write knee | **~250 rps** | the peach mix is WRITE-heavy (view/like/skip ≈ 90% of ops) | -| 30-night soak | **200 rps** | sustained with margin | +| write knee | **~250 rps in the clean sweep** | one benchmark, not a sustained-safe rate | +| nightly 200 rps soak | **23 PASS / 32 FAIL (42% green)** | not sustained with margin; gate parked | | write scaling vs node count | **~1.0×** | does NOT scale at full-placement RF3 | - Write tput does **NOT** scale with node count at full-placement RF3: every per-shard quorum spans all 3 nodes, so **every follower applies every 1536-D write**. Adding nodes adds replication work, not write capacity (~1.0×, not 2.5×). @@ -184,13 +200,20 @@ The tables above are single-node, analytic estimates. This section is the **meas ### Per-pod memory at 1536-D -| corpus | per-pod RSS | pod mem limit | verdict | -|--------|-------------|---------------|---------| -| 100k × 1536-D | **~1.9 GiB** (HNSW load peak) | 4 GiB | fits | -| 1M × 1536-D | **~7–8 GB** | 4 GiB | **OOMs the 16 GiB nodes** | +| corpus / workload | observed per-pod working set | configured limit at observation | verdict | +|-------------------|------------------------------|---------------------------------|---------| +| 100k idle baseline | **~3.58 GiB** | 4 GiB | only ~420 MiB load/recovery headroom | +| 100k / 200 rps soak | **3.97-4.00 GiB before OOMKill** | 4 GiB | limit is insufficient | +| 1M × 1536-D estimate | **~7-8 GiB before runtime overhead** | — | requires Ref-B measurement | -- Full placement → **each pod holds the WHOLE corpus** (no per-shard sharding of memory). -- 1M × 1536-D ≈ 7–8 GB/pod RSS overruns the ~13 GiB allocatable (already ~7 GiB of co-tenants) → the **1M production-read gate needs nodes >16 GiB** (a Ref-B requirement). +- Full placement means each pod holds the whole corpus. +- Four independent historical windows show `OOMKilled` at 3.97-4.00 GiB, followed + by snapshot-required reseed self-restarts. The exact internal growth source is + not yet isolated; signal hot-entry count stayed flat, so do not label it a leak + without allocation/heap evidence. +- The corrected canary envelope is a 4 GiB request and 6 GiB limit (measured peak + plus 50% recovery/profiling headroom). Profile the exact 100k/1536-D/200-rps + trajectory before treating 6 GiB as a production ceiling. ### Startup / boot @@ -198,12 +221,12 @@ The tables above are single-node, analytic estimates. This section is the **meas - `startupProbe` budget is **~20 min** (`failureThreshold` 240 × 5 s). - Graceful shutdown **saves the graphs** (grace **600 s**), so a clean restart **skips the rebuild** (load, not rebuild). A SIGKILL/crash skips the save → next boot rebuilds. -### Pod resources (live) +### Pod resources (corrected canary manifest) | resource | request | limit | |----------|---------|-------| -| CPU | 500m | 3 | -| memory | 1 GiB | 4 GiB | +| CPU | 2 | 3 | +| memory | 4 GiB | 6 GiB | | PVC | — | 5 GiB/pod, `local-path` | - PVC is **local NVMe** (`local-path`) — longhorn's fsync overhead was unacceptable for the WAL path. diff --git a/docs/ops/prometheus-alerts.yaml b/docs/ops/prometheus-alerts.yaml index f2c8b60..faacee1 100644 --- a/docs/ops/prometheus-alerts.yaml +++ b/docs/ops/prometheus-alerts.yaml @@ -1,39 +1,19 @@ # ============================================================================= -# DESIGN-REFERENCE RULE SET — NOT LOADED BY ANY ALERTMANAGER TODAY. +# TIDALDB ALERT CONTRACT # ============================================================================= -# This file is reference/design config. It is NOT wired into any running -# alerting pipeline: -# - vmalert (pilot/dev) loads ONLY ops/vmalert/rules/*.yaml -# - prod CRDs live ONLY in infra/k8s/prom/prometheus-rules/*.yaml -# Nothing mounts or imports docs/ops/prometheus-alerts.yaml. -# Do not read these as live pager rules. +# The orchard9 fleet runs these expressions through vmalert from: +# ../k3s-fleet/deployments/k8s/base/observability/alerting-rules.yaml # -# Provenance: every metric referenced below (tidaldb_health_ok, -# tidaldb_checkpoint_age_seconds, tidaldb_checkpoint_failures_total, -# tidaldb_wal_lag_bytes, tidaldb_signal_hot_entries, tidaldb_degradation_level, -# tidaldb_active_sessions, tidaldb_rate_limited_total, -# tidaldb_tantivy_segment_count, tidaldb_retrieve_latency_us_bucket, -# tidaldb_search_latency_us_bucket) IS really emitted today by -# tidal/src/db/metrics/mod.rs. So these rules are promotable — they -# are not orphaned against phantom metrics. +# This product-side copy documents the emitted metrics and alert thresholds. +# Keep every expression and threshold aligned with the fleet source when the +# contract changes. The fleet source adds routing labels (`capability`, +# `surface`) and runbook URLs required by the shared Alertmanager pipeline. # -# Promotion path (the right long-term fix — do it deliberately, do NOT -# hand-copy these as live pager rules without the steps below): -# 1. Add a vmalert rule file: ops/vmalert/rules/tidaldb.yaml, translating -# each rule and stamping the canonical labels every routed rule carries — -# severity + capability + surface (+ optional component). Map per the -# canonical severity taxonomy in ops/alerting.md: -# labels {severity: critical} = pager class (TidalDBDown only here), -# {severity: warning} = non-paging / business-hours, -# {severity: info} = pure-FYI. -# Add capability: tidaldb and a surface label per rule, and a -# runbook_url annotation: -# https://github.com/orchard9/tidaldb/blob/main/docs/runbooks/.md -# 2. Add the prod CRD twin: infra/k8s/prom/prometheus-rules/tidaldb.yaml, -# kept byte-aligned in expr/threshold with the vmalert file. -# 3. Verify the alerts fire against real metrics (scrape tidaldb, force a -# degraded state) before declaring them on-call-ready. -# Until steps 1–3 land, this file stays a design reference only. +# Every metric referenced below is emitted by tidalDB today. Standalone rules +# stay inactive when no standalone instance is running; cluster-only series +# activate only in cluster mode. The shared fleet rules additionally cover +# target loss, unavailable replicas, image-pull failures, restarts, PVCs, and +# Kubernetes Jobs. # ============================================================================= groups: - name: tidaldb diff --git a/docs/planning/ROADMAP.md b/docs/planning/ROADMAP.md index 67b4524..8b832f9 100644 --- a/docs/planning/ROADMAP.md +++ b/docs/planning/ROADMAP.md @@ -156,7 +156,7 @@ The roadmap now has two tracks: **Engine status:** M0–M12 **COMPLETE**. M9/M10 shipped 2026-06-06; M11 (Enterprise-Grade Cluster, all nine phases) closed 2026-06-13; M12 (Vector Retrieval at production shape) closed 2026-06-14 / 2026-06-23 — see the M11/M12 milestone rows above and *Implementation Status* below. -**Next (engine):** close the v1.0 bar — the 30-day-green nightly chaos+soak calendar (m11p9) and the standing Ref-A/k3s throughput re-runs (≥5,000/s + ≥2.5× single-shard scaling). Deferred follow-ups: the `writer_agent` u16 interning on the WAL v3 envelope and the offline medoid-recluster tier for multi-vector preference. +**Next (engine):** isolate the Ref-A 100k/1536-D memory growth under the corrected 4 GiB request / 6 GiB canary limit. The 30-day nightly gate remains parked: its 200 rps history is 23 PASS / 32 FAIL with repeated 4 GiB OOMKills, so it is not a production-readiness signal yet. Resume only after the controlled profile passes and restart evidence is fail-closed. The old ≥2.5× write-scaling goal belongs to Ref-B (≥5 nodes **with partitioned placement**); adding nodes to full-placement RF3 does not scale writes. Deferred follow-ups: the `writer_agent` u16 interning on the WAL v3 envelope and the offline medoid-recluster tier for multi-vector preference. **Next (product):** iknowyou M5 acceptance pass, then M6 Closed Loop (session lifecycle + preference drift validation). --- diff --git a/docs/planning/milestone-11/phase-9.md b/docs/planning/milestone-11/phase-9.md index 73d7695..716309f 100644 --- a/docs/planning/milestone-11/phase-9.md +++ b/docs/planning/milestone-11/phase-9.md @@ -140,22 +140,24 @@ The bar is NOT "30 green soak Job exits". It is **30 consecutive nights where th soak passed its SLO gates AND no cluster pod restarted under load**. The two halves come from two streams on the shared result PVC: -- `ledger.tsv` — one row per night from the nightly CronJob: `PASS`/`FAIL` on the - armed gates (`--fail-on-knee`, `--max-p99-ms 150`, `--max-error-pct 1`). -- `restarts.tsv` — the monitor's 5-minute snapshot of each `tidaldb-{0,1,2}` pod's - cumulative `restartCount`. +- `ledger.tsv` — one row per UTC date from the nightly CronJob: `PASS`/`FAIL` + on the armed gates (`--fail-on-knee`, `--max-p99-ms 150`, + `--max-error-pct 1`), plus exact `start_utc` / `end_utc` load bounds. +- `restarts.tsv` — the monitor's one-minute snapshot of each + `tidaldb-{0,1,2}` pod UID and cumulative `restartCount`. -`soak-eval` (the proven core in `tidal_stress::soak_eval`, table-driven-tested; -the binary runs in the monitor's evaluator container and once per night in the -CronJob) joins them into `streak.tsv`: +`soak-eval` (the proven core in `tidal_stress::soak_eval`, +table-driven-tested; the binary runs in the monitor's evaluator container and +once per night in the CronJob) joins them into `streak.tsv`: -- a night is **GREEN** iff its ledger row is `PASS` **and** no pod's cumulative - restart count rose within that night's window; -- the **streak** is the trailing run of consecutive green nights; -- **any** non-green night resets the streak to **0**. A night resets it when - EITHER the soak breached an SLO gate (Job `FAIL`) **OR** a pod restarted under - load (the zero-unrecovered-restart half — a green ledger row with a restart - in-window is a FAIL, not a PASS: that was the false-confidence hole). +- a night is **GREEN** iff its ledger row is `PASS`, every expected pod has a + fresh sample immediately before and after the measured load window, and + neither its pod UID nor restart count changes between those samples; +- the **streak** is the trailing run of unique, consecutive UTC calendar dates; +- a gate breach, restart, pod replacement, missing/stale boundary sample, + duplicate date, or date gap cannot advance the streak. This closes both + fail-open paths: a green load exit with a recovery event and multiple rows + masquerading as multiple nights. The evaluator emits exactly one alert per non-green transition (a durable `ALERT-.txt` the http surface serves, plus an optional `NOTIFY_WEBHOOK` diff --git a/docs/runbooks/cluster.md b/docs/runbooks/cluster.md index 0396dac..a55b560 100644 --- a/docs/runbooks/cluster.md +++ b/docs/runbooks/cluster.md @@ -1512,29 +1512,35 @@ writes with `ENOSPC` after `n` cumulative bytes this process lifetime (arm it on a node at restart to fail after `n` bytes of post-restart writes). NEVER set these on a production node. -**Soak with regression gates** (`tidal-stress`): +**Fleet soak with regression gates** (`tidal-stress`): ``` -tidal-stress --target https:// --ramp "200:3600" --mix peach \ - --json-summary soak.json --max-error-pct 1 --max-p99-ms 250 --fail-on-knee +tidal-stress --target https://:9500 \ + --ca-cert /etc/tidaldb/tls/ca.crt --skip-seed \ + --corpus 100000 --embedding-dim 1536 \ + --ramp "200:3600" --mix peach --json-summary soak.json \ + --max-error-pct 1 --max-p99-ms 150 --fail-on-knee ``` -> **Soak runs at 200 rps (re-scoped 2026-06-19).** The peach mix is write-heavy -> and the write knee on this 3-node fleet is **~250 rps** (1536-D `ack=quorum` -> ingest), so the soak ramp is a single **measured-sustainable 200-rps** stage -> (`--ramp "200:3600"`) rather than the retired `3900:3600` constant (a pre-m12 -> signal-write figure that does not hold at the 1536-D production shape). Point -> `--target` at an `https://` gateway — the `:9500` plane serves TLS. +> **The Ref-A gate is 200 rps (re-scoped 2026-06-19).** The peach mix is +> write-heavy and the measured write knee on this full-placement three-node +> fleet is about 250 rps. The one-hour gate therefore uses the measured 200-rps +> stage, not the retired pre-m12 `3900:3600` figure. The `:9500` plane is TLS. -`--fail-on-knee` (built-in SLO), `--max-p99-ms`, and `--max-error-pct` make the -run exit non-zero on a regression; `--json-summary` writes a machine-readable -per-stage roll-up for trend lines. A bounded version runs nightly; the GA-bar -100k-DAU soak points `--target` at the live Ref-A cluster at 200 rps. +`--fail-on-knee`, `--max-p99-ms`, and `--max-error-pct` make the load process +exit non-zero on regression. The in-cluster `tidal-soak-nightly` CronJob adds +the recovery half of the gate: one ledger row per UTC date, one-minute pod UID +and restart-count evidence immediately before and after the exact load window, +and an atomic `streak.tsv`. A date advances the streak only when the load passes, +all three pods have fresh boundary samples, and no UID or counter changes. +Duplicate dates, gaps, stale evidence, pod replacement, and load failure cannot +inflate the 30-night streak. **Nightly CI** (`.woodpecker.yaml`, cron `nightly` — Woodpecker, never GitHub -Actions): the chaos suites with elevated kill-points (`TIDAL_QUORUM_KILLPOINTS`, -`TIDAL_ELECTION_KILLPOINTS`) then the gated soak. A nightly failure flags the -day's correctness or performance regression. The guarantee→test map is +Actions) remains the bounded code-regression signal: tier-3 chaos with elevated +kill-points, then a local gated soak. It is not the Ref-A calendar gate unless +`TIDAL_SOAK_TARGET` and its duration/rate are explicitly pointed at that fleet. +The guarantee→test map is [docs/planning/milestone-11/guarantee-traceability.md](../planning/milestone-11/guarantee-traceability.md). ## Cross-references diff --git a/hooks/pre-commit b/hooks/pre-commit index 9b56c8f..e678fd7 100755 --- a/hooks/pre-commit +++ b/hooks/pre-commit @@ -4,8 +4,22 @@ # git config core.hooksPath hooks # or run scripts/install-hooks.sh once. # -# Gates: Rust fmt/clippy/test (only when Rust is staged), the documentation -# consolidation guard (always), and site eslint (when node_modules exist). +# Gates: Rust fmt/clippy/test (only when Rust is staged), the CODING_GUIDELINES +# source checks, the documentation consolidation guard (always), and site eslint +# (when node_modules exist). +# +# Two rules this file learned the hard way, 2026-08-16: +# +# * Clippy runs WITHOUT `-D warnings`. Every tidal crate declares its own +# posture in `[lints]` (`clippy::all = deny`, `unwrap_used = deny`, +# `pedantic = warn`), and that table is the single source of truth. Adding +# `-D warnings` here promoted ~58 deliberate pedantic warnings in the +# integration tests to errors and made every Rust commit impossible. +# * The Rust gates key off ALL crates, not a `tidal/` path match. An earlier +# untracked copy of this hook tested `grep -q 'tidal/'`, which silently +# skipped `tidal-server/`, `tidal-net/`, `tidal-stress/`, `tidalctl/`, and +# `applications/` - the workspace was left to rot until a workspace-wide +# `cargo test` was attempted months later. set -uo pipefail ROOT="$(git rev-parse --show-toplevel)" @@ -15,16 +29,64 @@ staged() { git diff --cached --name-only --diff-filter=ACM; } rust_staged=$(staged | grep -E '\.rs$' || true) site_staged=$(staged | grep -E '^site/.*\.(ts|tsx|js|jsx|mjs)$' || true) -# --- Rust (engine crate) ---------------------------------------------------- +# --- Rust ------------------------------------------------------------------- if [ -n "$rust_staged" ]; then echo "pre-commit: cargo fmt" - cargo fmt -p tidaldb || { echo "cargo fmt failed" >&2; exit 1; } + cargo fmt || { echo "cargo fmt failed" >&2; exit 1; } # re-stage any files fmt rewrote echo "$rust_staged" | while IFS= read -r f; do [ -f "$f" ] && git add "$f"; done - echo "pre-commit: cargo clippy -p tidaldb -D warnings" - cargo clippy -p tidaldb --all-targets -- -D warnings || exit 1 + + echo "pre-commit: cargo clippy -p tidaldb -p tidal-net -p tidal-server -p tidal-stress -p tidalctl" + cargo clippy -p tidaldb -p tidal-net -p tidal-server -p tidal-stress -p tidalctl \ + --all-targets || exit 1 + echo "pre-commit: cargo test -p tidaldb --lib" cargo test -p tidaldb --lib || exit 1 + + # ── CODING_GUIDELINES source checks (engine sources only) ───────────────── + engine_src=$(echo "$rust_staged" | grep -E '^tidal/src/' || true) + if [ -n "$engine_src" ]; then + fail=0 + + # §9 file length: 600 lines max. + while IFS= read -r f; do + [ -z "$f" ] && continue + lines=$(wc -l < "$f" 2>/dev/null || echo 0) + if [ "$lines" -gt 600 ]; then + echo " FAIL: $f is $lines lines (max 600) - split it (CODING_GUIDELINES §9)" >&2 + fail=1 + fi + done <<< "$engine_src" + + # §11 no println!/eprintln! in engine sources - use tracing::. + while IFS= read -r f; do + [ -z "$f" ] && continue + case "$f" in */benches/*) continue ;; esac + hits=$(grep -n 'println!\|eprintln!' "$f" 2>/dev/null | grep -vE '^[0-9]+:[[:space:]]*//' || true) + if [ -n "$hits" ]; then + echo " FAIL: println!/eprintln! in $f - use tracing:: (CODING_GUIDELINES §11)" >&2 + awk '{print " " $0}' <<< "$hits" >&2 + fail=1 + fi + done <<< "$engine_src" + + # §10 every unsafe block carries a // SAFETY: comment. + while IFS= read -r f; do + [ -z "$f" ] && continue + hits=$(awk ' + /\/\/ SAFETY:/ { had_safety=1; next } + /unsafe \{/ { if (!had_safety) print NR": "$0; had_safety=0; next } + { had_safety=0 } + ' "$f" 2>/dev/null || true) + if [ -n "$hits" ]; then + echo " FAIL: unsafe block without // SAFETY: in $f (CODING_GUIDELINES §10)" >&2 + awk '{print " " $0}' <<< "$hits" >&2 + fail=1 + fi + done <<< "$engine_src" + + [ "$fail" -eq 0 ] || exit 1 + fi fi # --- Documentation consolidation guard (always) ----------------------------- diff --git a/k8s/cluster/statefulset.yaml b/k8s/cluster/statefulset.yaml index 9bc19f8..582b817 100644 --- a/k8s/cluster/statefulset.yaml +++ b/k8s/cluster/statefulset.yaml @@ -1,10 +1,9 @@ # The tidalDB CLUSTER: ONE StatefulSet, every pod a region (m11p5 §4). # -# MUTUALLY EXCLUSIVE with the standalone set in k8s/ (namespace `tidaldb`, -# replicas: 1, `standalone` subcommand). This is namespace `tidaldb-cluster`, -# replicas: 3, the `cluster --region` subcommand, real quorum-ack writes, -# automatic election, and elastic membership. Deploy ONE or the OTHER per -# namespace — never both. +# MUTUALLY EXCLUSIVE with the standalone set in k8s/. Both source workloads are +# parked at 0. `scripts/restore-fleet.sh` restores one selected data plane: +# cardinality 1 in namespace `tidaldb`, or three `cluster --region` processes in +# namespace `tidaldb-cluster` with real quorum-ack writes. Never run both. # # WHY ONE StatefulSet (not one-per-region): the m11p5 bind/advertise split lets # every pod mount the SAME topology ConfigMap (peers are advertised by per-pod @@ -22,7 +21,7 @@ metadata: app.kubernetes.io/component: cluster-node spec: serviceName: tidaldb-peers # the headless peer Service — stable per-pod DNS - replicas: 3 # the initial voter set; scale up/down per the runbook + replicas: 0 # parked; scripts/restore-fleet.sh restores three voters explicitly # Parallel: bring all pods up at once. There is no ordered-bootstrap # dependency — siblings boot in any order (an unreachable-at-startup peer is # normal; the election + catch-up timer converge them). Ordered start would @@ -72,7 +71,7 @@ spec: type: RuntimeDefault initContainers: - name: init-datadir - image: busybox:1.36 + image: busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662 imagePullPolicy: IfNotPresent command: ["sh", "-c", "mkdir -p /data/db && chown -R 10001:10001 /data/db"] securityContext: @@ -84,7 +83,7 @@ spec: mountPath: /data containers: - name: tidaldb - image: registry.threesix.ai/tidal/server@sha256:171505745b801dcf231b531de6167dbc309a7182957811cbc2228f0a302572b1 # m12-writeburst-rc7 (LIVE; tidaldb commit 1b5bcba). Adds the write-burst false-partition fix on top of rc6 (bd211e7338): a follower whose 1536-D HNSW apply momentarily starves its transport runtime makes a leader ship RPC miss the 10s deadline -> tonic DeadlineExceeded was counted as a transport failure (record_failure) -> both followers' breakers latched Open -> commit stalled -> ack=quorum 503-stormed with no self-heal. Fix (tidal-net, heuristic set only; commit.rs/election.rs/vote path UNTOUCHED): record_timeout opens the breaker ONLY when there's no recent proof of life (last_contact stale -> genuine blackhole still detected); DeadlineExceeded|Cancelled route to it, genuine Unavailable still opens. Verified: 6 unit + 2 real-gRPC integration + slow-fsync multi-process regression. Inherits rc6's seed-join + reseed-loop + election-divergence fixes and rc13's WAL_RETENTION_SEGMENTS=16. amd64 manifest (6 layers, imagetools-verified). + image: registry.threesix.ai/tidal/server:m12-fleet-remediation-20260813@sha256:2e4baaf974ad2cf650609b1843689122bf4fd2751bcb8749906df8089223f4a3 imagePullPolicy: IfNotPresent # The image ENTRYPOINT is the bare binary. We override the command with # a tiny /bin/sh wrapper (the bookworm-slim runtime HAS a shell) so we @@ -220,9 +219,13 @@ spec: path: /health/live port: http scheme: HTTPS + # A transiently saturated query runtime must shed readiness before + # kubelet turns load into a cascading restart. Six 10s failures give + # the process roughly one minute to recover while still detecting a + # genuinely wedged runtime. periodSeconds: 10 - timeoutSeconds: 3 - failureThreshold: 3 + timeoutSeconds: 5 + failureThreshold: 6 readinessProbe: httpGet: path: /health # cluster-aware: 503 joiner/quarantined/draining @@ -233,18 +236,22 @@ spec: failureThreshold: 3 resources: requests: - cpu: "500m" - memory: 1Gi + # Measured Ref-A soak envelope: the busiest leader sustained + # ~1.7 cores and reached ~2.5, so 2 cores is the honest scheduler + # reservation. A fleet that cannot place this request cannot run + # the 200 rps gate without request-level contention. + cpu: "2" + # Baseline working set was ~3.6 GiB before load. Reserving 4 GiB + # prevents the scheduler from hiding that resident footprint. + memory: 4Gi limits: - # m12 read-SLA fix: 2->3. The cgroup cpu quota is what the engine's - # available_parallelism() reads (SEARCH_GATE / worker_threads sizing). - # At "2" a cross-shard search burst starved the async reactor + the - # election/heartbeat/apply loops (reads hung to the 30s route timeout; - # the starved control plane churned elections -> reseed self-exit). - # "3" leaves ~1 core for kubelet/system on the 4-core nodes; requests - # stay at 500m so the pod still schedules (server nodes alloc=3). + # Three cores preserves one core for kubelet/system on the Ref-A + # four-core node while allowing the measured query burst. cpu: "3" - memory: 4Gi # m12/1536: 100k×1536 HNSW load peaks ~1.9Gi; 1M needs more headroom (nodes have 13Gi allocatable) + # Four independent OOMKills occurred at 3.97-4.00 GiB. Six GiB is + # measured peak plus 50% recovery/profiling headroom; the exact + # internal growth source still requires heap/allocation profiling. + memory: 6Gi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true # writes only /data (PVC) and /tmp (emptyDir) @@ -297,6 +304,8 @@ spec: name: data labels: app.kubernetes.io/name: tidaldb + backup.orchard9.ai/class: expendable + backup.orchard9.ai/method: tidal-stress-reseed spec: accessModes: ["ReadWriteOnce"] storageClassName: local-path diff --git a/k8s/statefulset.yaml b/k8s/statefulset.yaml index 39ce94c..276c273 100644 --- a/k8s/statefulset.yaml +++ b/k8s/statefulset.yaml @@ -3,11 +3,11 @@ # WHY A STATEFULSET (not a Deployment): tidalDB is single-node-first and # embeddable — the server wraps ONE engine instance whose state (WAL + # checkpoints + indexes) lives on a durable data dir. It scales VERTICALLY -# (bigger node), not by adding replicas. `replicas: 1` is intentional and load- -# bearing: there is no shared-storage multi-writer mode. If you need multi-node -# HA, that is a DIFFERENT deployment: the multi-process `cluster` subcommand -# (one process per region, quorum-acked writes, automatic failover) ships as its -# own StatefulSet under k8s/cluster/ — it does not change this manifest. +# (bigger node), not by adding active replicas. The parked source state keeps +# `replicas: 0`; `scripts/restore-fleet.sh --standalone` restores cardinality 1. +# For HA, use the DIFFERENT multi-process `cluster` deployment: one process per +# region, quorum-acked writes, and automatic failover. It ships as its own +# StatefulSet under k8s/cluster/ and does not change this manifest. # See docs/runbooks/kubernetes.md and docs/runbooks/cluster.md. apiVersion: apps/v1 kind: StatefulSet @@ -19,7 +19,7 @@ metadata: app.kubernetes.io/component: server spec: serviceName: tidaldb # the headless Service in service.yaml — stable network id - replicas: 1 # single-node-first: scale up, not out (see header) + replicas: 0 # parked; scripts/restore-fleet.sh is the only supported scale-up path selector: matchLabels: app.kubernetes.io/name: tidaldb diff --git a/rust-toolchain.toml b/rust-toolchain.toml new file mode 100644 index 0000000..ef63ce8 --- /dev/null +++ b/rust-toolchain.toml @@ -0,0 +1,14 @@ +# Development toolchain for the whole workspace. +# +# This is NOT the library's MSRV. `tidaldb` and its siblings publish +# `rust-version = "1.91"` (see Cargo.toml) and build on 1.91.0. The pin exists +# because `tidalctl`'s AWS SDK dependency chain requires 1.91.1: on 1.91.0 every +# workspace-wide command (`cargo test --workspace`, `cargo clippy --workspace`) +# fails during resolution with "aws-types@1.3.16 requires rustc 1.91.1" and no +# gate can run at all. Pinning the dev toolchain keeps the published MSRV honest +# while making the workspace gates executable from a fresh clone. +# +# Raise this only alongside the `rust-version` fields it is meant to exceed. +[toolchain] +channel = "1.91.1" +components = ["rustfmt", "clippy"] diff --git a/scripts/build-release.sh b/scripts/build-release.sh index b3f4046..33903e2 100755 --- a/scripts/build-release.sh +++ b/scripts/build-release.sh @@ -26,26 +26,61 @@ BUILDER="${TIDAL_BUILDX_BUILDER:-amd64builder}" die() { echo "build-release: $*" >&2; exit 1; } [ -n "$TAG" ] || die "usage: build-release.sh [server|dr|stress|all]" +[ "$TAG" != "latest" ] || die "refusing mutable release tag 'latest'" +[[ "$TAG" =~ ^[a-z0-9][a-z0-9._-]{0,127}$ ]] \ + || die "tag must match ^[a-z0-9][a-z0-9._-]{0,127}$" case "$COMPONENT" in server|dr|stress|all) ;; *) die "component must be server|dr|stress|all" ;; esac # ── Toolchain preflight (fail fast, no half-built image) ───────────────────── -command -v cargo >/dev/null || die "cargo not on PATH" -command -v docker >/dev/null || die "docker not on PATH" -command -v "${TARGET}-gcc" >/dev/null || die "missing ${TARGET}-gcc (brew install ${TARGET})" -PROTOC_BIN="${PROTOC:-$(command -v protoc || true)}" -[ -n "$PROTOC_BIN" ] || die "protoc not found (brew install protobuf), or set PROTOC" -rustup target list --installed 2>/dev/null | grep -qx "$TARGET" \ - || die "rust target $TARGET not installed (rustup target add $TARGET)" +command -v docker >/dev/null || die "docker not on PATH" +command -v git >/dev/null || die "git not on PATH" +command -v jq >/dev/null || die "jq not on PATH" +if [ "$COMPONENT" != "stress" ]; then + command -v cargo >/dev/null || die "cargo not on PATH" + command -v "${TARGET}-gcc" >/dev/null \ + || die "missing ${TARGET}-gcc (brew install ${TARGET})" + PROTOC_BIN="${PROTOC:-$(command -v protoc || true)}" + [ -n "$PROTOC_BIN" ] || die "protoc not found (brew install protobuf), or set PROTOC" + rustup target list --installed 2>/dev/null | grep -qx "$TARGET" \ + || die "rust target $TARGET not installed (rustup target add $TARGET)" +fi REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" cd "$REPO_ROOT" +# Every image must have one reproducible source identity. Clean releases use +# HEAD. An explicitly allowed canary hashes HEAD, both tracked diffs, and every +# untracked non-ignored file (path + Git blob hash), so two distinct source +# trees cannot silently share the same `-dirty` identity. This check also covers +# stress: its Docker build consumes the repository source directly. +UNTRACKED="$(git ls-files --others --exclude-standard)" +if git diff --quiet && git diff --cached --quiet && [ -z "$UNTRACKED" ]; then + export GIT_HASH="${TIDAL_BUILD_HASH:-$(git rev-parse --verify HEAD)}" +else + [ "${TIDAL_ALLOW_DIRTY:-0}" = "1" ] \ + || die "source changes present; commit them or set TIDAL_ALLOW_DIRTY=1 for an explicit canary" + DIRTY_HASH="$( + { + git rev-parse --verify HEAD + git diff --binary + git diff --cached --binary + git ls-files --others --exclude-standard -z | + while IFS= read -r -d '' file; do + printf 'untracked\0%s\0' "$file" + git hash-object -- "$file" + done + } | shasum -a 256 | cut -c1-16 + )" + export GIT_HASH="${TIDAL_BUILD_HASH:-${DIRTY_HASH}-dirty}" +fi +echo "==> source identity $GIT_HASH" + # ── Pinned cross-compile environment (the documented recipe) ───────────────── export CC_x86_64_unknown_linux_gnu="${TARGET}-gcc" export CXX_x86_64_unknown_linux_gnu="${TARGET}-g++" export AR_x86_64_unknown_linux_gnu="${TARGET}-ar" export CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER="${TARGET}-gcc" -export PROTOC="$PROTOC_BIN" +export PROTOC="${PROTOC_BIN:-}" # Which crates to cross-compile for the selected component(s). PKGS=() @@ -78,8 +113,9 @@ build_image() { # $1=image_name $2=dockerfile $3=stage_subdir docker buildx build --builder "$BUILDER" --platform linux/amd64 \ -f "$dockerfile" -t "$image" --push "$ctx" local digest - digest="$(docker buildx imagetools inspect "$image" --format '{{.Manifest.Digest}}' 2>/dev/null || true)" - echo "RELEASE $1: $REGISTRY/$1@${digest:-} (tag $TAG)" + digest="$(docker buildx imagetools inspect "$image" --format '{{json .Manifest}}' | jq -er '.digest')" + [ -n "$digest" ] || die "registry returned no digest for $image" + echo "RELEASE $1: $image@$digest" } if [ "$COMPONENT" = server ] || [ "$COMPONENT" = all ]; then @@ -98,13 +134,16 @@ if [ "$COMPONENT" = dr ] || [ "$COMPONENT" = all ]; then fi if [ "$COMPONENT" = stress ] || [ "$COMPONENT" = all ]; then - # Pure-Rust, in-container build (includes the soak-eval binary). Context = repo - # root so the workspace manifests resolve; the .dockerignore prunes it. - echo "==> building $REGISTRY/stress:$TAG (in-container, includes soak-eval)" + # Pure-Rust, in-container build (includes soak-eval + soak-watch). Context = + # repo root so workspace manifests resolve; the .dockerignore prunes it. The + # OCI revision label makes the source identity inspectable after the build. + echo "==> building $REGISTRY/stress:$TAG (in-container, includes soak-eval + soak-watch)" docker buildx build --builder "$BUILDER" --platform linux/amd64 \ + --build-arg "GIT_HASH=$GIT_HASH" \ -f docker/stress/Dockerfile -t "$REGISTRY/stress:$TAG" --push . - d="$(docker buildx imagetools inspect "$REGISTRY/stress:$TAG" --format '{{.Manifest.Digest}}' 2>/dev/null || true)" - echo "RELEASE stress: $REGISTRY/stress@${d:-} (tag $TAG)" + d="$(docker buildx imagetools inspect "$REGISTRY/stress:$TAG" --format '{{json .Manifest}}' | jq -er '.digest')" + [ -n "$d" ] || die "registry returned no digest for $REGISTRY/stress:$TAG" + echo "RELEASE stress: $REGISTRY/stress:$TAG@$d" fi -echo "==> done. Pin the printed @sha256 digest(s) in k8s/cluster/statefulset.yaml and the DR/soak manifests." +echo "==> done. Deploy only the printed tag@sha256 references." diff --git a/scripts/restore-fleet.sh b/scripts/restore-fleet.sh new file mode 100755 index 0000000..80ca265 --- /dev/null +++ b/scripts/restore-fleet.sh @@ -0,0 +1,523 @@ +#!/usr/bin/env bash +# Restore selected tidalDB workloads parked after the 2026-08-12 fleet review. +# +# This script never recreates the retired public Ingresses or DNS records. It +# fails before mutation unless retained storage, immutable images, and scheduler +# headroom satisfy the selected workload's current resource contract. + +set -euo pipefail +umask 077 +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="${SCRIPT_DIR%/scripts}" +RESTORE_LOG_DIR="${TIDAL_RESTORE_LOG_DIR:-$REPO_ROOT/tmp/restore-logs}" +mkdir -p "$RESTORE_LOG_DIR" +RESTORE_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +RESTORE_LOG_FILE="$RESTORE_LOG_DIR/restore-${RESTORE_STARTED_AT//:/}-$$.log" +exec > >(tee -a "$RESTORE_LOG_FILE") 2>&1 +printf 'event=restore_start ts=%s pid=%s argv=' "$RESTORE_STARTED_AT" "$$" +printf '%q ' "$@" +printf 'log_file=%q\n' "${RESTORE_LOG_FILE##*/}" + +export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/orchard9-k3sf.yaml}" + +RESTORE_CLUSTER=0 +RESTORE_STANDALONE=0 +RESTORE_SOAK=0 +ACCEPT_FAILED_GATE=0 +CHECK_ONLY=0 +STANDALONE_DATA_NODE="" +CLUSTER_DATA_NODE_0="" +CLUSTER_DATA_NODE_1="" +CLUSTER_DATA_NODE_2="" +PREFLIGHT_POD="" + +usage() { + cat <<'USAGE' +Usage: + scripts/restore-fleet.sh --cluster [--check] + scripts/restore-fleet.sh --standalone [--check] + scripts/restore-fleet.sh --cluster --soak --accept-failed-gate [--check] + +Options: + --cluster Restore the three-node RF3 cluster and soak monitor. + --standalone Restore the internal standalone store. Public routing + remains retired and must be provisioned separately. + --soak Unsuspend the nightly endurance job after restoring the + cluster. Requires --accept-failed-gate. + --accept-failed-gate Acknowledge that the unchanged 200 rps gate passed only + 23/55 measured nights and is not a release signal. + --check Run every preflight and print capacity; mutate nothing. + -h, --help Show this help. + +Exactly one of --cluster or --standalone is required. + +No-argument execution is intentionally a no-op. The old script could restore a +6-core cluster into a fleet with only ~2 cores free and then claim success via +public routes that no longer exist. +USAGE +} + +die() { + printf 'event=restore_error ts=%s message=%q\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2 + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} + +for arg in "$@"; do + case "$arg" in + --cluster) RESTORE_CLUSTER=1 ;; + --standalone) RESTORE_STANDALONE=1 ;; + --soak) RESTORE_SOAK=1 ;; + --accept-failed-gate) ACCEPT_FAILED_GATE=1 ;; + --check) CHECK_ONLY=1 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown option: $arg" ;; + esac +done + +if [ "$RESTORE_CLUSTER" -eq 0 ] && [ "$RESTORE_STANDALONE" -eq 0 ]; then + usage >&2 + exit 2 +fi +if [ "$RESTORE_CLUSTER" -eq 1 ] && [ "$RESTORE_STANDALONE" -eq 1 ]; then + die "choose exactly one data plane: --cluster or --standalone" +fi +if [ "$RESTORE_SOAK" -eq 1 ] && [ "$RESTORE_CLUSTER" -eq 0 ]; then + die "--soak requires --cluster" +fi +if [ "$RESTORE_SOAK" -eq 1 ] && [ "$ACCEPT_FAILED_GATE" -eq 0 ]; then + die "--soak requires --accept-failed-gate; the unchanged gate passed only 23/55 measured nights" +fi + +for command in kubectl jq grep; do + command -v "$command" >/dev/null 2>&1 || die "$command is required" +done +kubectl version --client >/dev/null +kubectl get namespace tidaldb tidaldb-cluster >/dev/null + +KUBECONFIG_DISPLAY="$KUBECONFIG" +if [[ "$KUBECONFIG_DISPLAY" == "$HOME/"* ]]; then + # Display a literal tilde so the runtime log never records workstation identity. + # shellcheck disable=SC2088 + printf -v KUBECONFIG_DISPLAY '~/%s' "${KUBECONFIG_DISPLAY#"$HOME"/}" +fi +printf '==> kubeconfig: %s\n' "$KUBECONFIG_DISPLAY" +printf '==> context: %s\n' "$(kubectl config current-context)" +printf '==> selected: cluster=%s standalone=%s soak=%s check_only=%s\n' \ + "$RESTORE_CLUSTER" "$RESTORE_STANDALONE" "$RESTORE_SOAK" "$CHECK_ONLY" + +check_claim() { + local namespace="$1" claim="$2" phase volume policy node + IFS=$'\t' read -r phase volume <<<"$( + kubectl -n "$namespace" get "pvc/$claim" \ + -o jsonpath='{.status.phase}{"\t"}{.spec.volumeName}' + )" + [ "$phase" = "Bound" ] || die "$namespace/$claim is $phase, expected Bound" + [ -n "$volume" ] || die "$namespace/$claim has no backing PV" + policy="$(kubectl get "pv/$volume" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')" + [ "$policy" = "Retain" ] || die "$namespace/$claim backing PV $volume uses $policy, expected Retain" + node="$( + kubectl get "pv/$volume" -o json | + jq -r '[ + try ( + .spec.nodeAffinity.required.nodeSelectorTerms[].matchExpressions[] + | select(.key == "kubernetes.io/hostname" and .operator == "In") + | .values[] + ) catch empty + ] | unique | if length <= 1 then (.[]? // "") else error("PV has ambiguous hostname affinity") end' + )" + case "$namespace/$claim" in + tidaldb/tidaldb-data) STANDALONE_DATA_NODE="$node" ;; + tidaldb-cluster/data-tidaldb-0) CLUSTER_DATA_NODE_0="$node" ;; + tidaldb-cluster/data-tidaldb-1) CLUSTER_DATA_NODE_1="$node" ;; + tidaldb-cluster/data-tidaldb-2) CLUSTER_DATA_NODE_2="$node" ;; + esac + printf ' %-18s %-28s Bound / Retain (node=%s)\n' \ + "$namespace" "$claim" "${node:-network-backed}" +} + +printf '==> retained storage\n' +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + check_claim tidaldb tidaldb-data +fi +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + check_claim tidaldb-cluster data-tidaldb-0 + check_claim tidaldb-cluster data-tidaldb-1 + check_claim tidaldb-cluster data-tidaldb-2 + check_claim tidaldb-cluster tidal-soak-results +fi + +check_pod_template_images() { + local namespace="$1" resource="$2" image images + images="$( + kubectl -n "$namespace" get "$resource" -o jsonpath='{range .spec.template.spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.template.spec.containers[*]}{.image}{"\n"}{end}' + )" + [ -n "$images" ] || die "$namespace/$resource has no images" + for image in $images; do + case "$image" in + *@sha256:*) ;; + *) die "$namespace/$resource uses mutable image $image" ;; + esac + done +} + +check_cronjob_images() { + local namespace="$1" name="$2" image images + images="$( + kubectl -n "$namespace" get "cronjob/$name" -o jsonpath='{range .spec.jobTemplate.spec.template.spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.jobTemplate.spec.template.spec.containers[*]}{.image}{"\n"}{end}' + )" + [ -n "$images" ] || die "$namespace/cronjob/$name has no images" + for image in $images; do + case "$image" in + *@sha256:*) ;; + *) die "$namespace/cronjob/$name uses mutable image $image" ;; + esac + done +} + +printf '==> immutable workload images\n' +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + check_pod_template_images tidaldb deployment/tidaldb +fi +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + check_pod_template_images tidaldb-cluster statefulset/tidaldb + check_pod_template_images tidaldb-cluster deployment/tidal-soak-monitor +fi +if [ "$RESTORE_SOAK" -eq 1 ]; then + check_cronjob_images tidaldb-cluster tidal-soak-nightly +fi +printf ' all selected restore dependencies are digest-pinned\n' + +require_parked() { + local namespace="$1" resource="$2" replicas + replicas="$(kubectl -n "$namespace" get "$resource" -o jsonpath='{.spec.replicas}')" + [ "${replicas:-0}" -eq 0 ] || \ + die "$namespace/$resource already requests $replicas replicas; this restore only accepts the recorded parked state" +} + +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + require_parked tidaldb deployment/tidaldb +fi +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + require_parked tidaldb-cluster statefulset/tidaldb + require_parked tidaldb-cluster deployment/tidal-soak-monitor + SOAK_SUSPENDED="$(kubectl -n tidaldb-cluster get cronjob/tidal-soak-nightly -o jsonpath='{.spec.suspend}')" + [ "$SOAK_SUSPENDED" = "true" ] || \ + die "tidaldb-cluster/cronjob/tidal-soak-nightly must be suspended before restore" +fi + +# One quantity implementation serves both live pod accounting and workload +# templates. The restore requirement therefore follows the manifest instead of +# duplicating CPU/memory constants that can silently drift. +# The `$q` names below are jq variables and must remain literal shell input. +# shellcheck disable=SC2016 +RESOURCE_FILTERS=' + def cpu_m: + tostring as $q | + if $q == "" or $q == "null" then 0 + elif ($q | endswith("m")) then ($q | rtrimstr("m") | tonumber) + elif ($q | endswith("u")) then (($q | rtrimstr("u") | tonumber) / 1000) + elif ($q | endswith("n")) then (($q | rtrimstr("n") | tonumber) / 1000000) + else (($q | tonumber) * 1000) + end; + def memory_bytes: + tostring as $q | + ($q | capture("^(?[0-9]+(?:\\.[0-9]+)?)(?[A-Za-z]*)$")) as $p | + ($p.n | tonumber) as $n | + if $p.u == "Ki" then $n * 1024 + elif $p.u == "Mi" then $n * 1048576 + elif $p.u == "Gi" then $n * 1073741824 + elif $p.u == "Ti" then $n * 1099511627776 + elif $p.u == "K" then $n * 1000 + elif $p.u == "M" then $n * 1000000 + elif $p.u == "G" then $n * 1000000000 + elif $p.u == "T" then $n * 1000000000000 + elif $p.u == "" then $n + else error("unsupported memory quantity: " + $q) + end; + def pod_cpu: + ([.containers[]? | (.resources.requests.cpu // "0") | cpu_m] | add // 0) as $regular | + ([.initContainers[]? | (.resources.requests.cpu // "0") | cpu_m] | max // 0) as $init | + ([ $regular, $init ] | max) + ((.overhead.cpu // "0") | cpu_m); + def pod_memory: + ([.containers[]? | (.resources.requests.memory // "0") | memory_bytes] | add // 0) as $regular | + ([.initContainers[]? | (.resources.requests.memory // "0") | memory_bytes] | max // 0) as $init | + ([ $regular, $init ] | max) + ((.overhead.memory // "0") | memory_bytes); +' + +workload_request() { + local namespace="$1" resource="$2" template_kind="$3" + kubectl -n "$namespace" get "$resource" -o json | + jq -c --arg template_kind "$template_kind" "$RESOURCE_FILTERS"' + ( + if $template_kind == "cronjob" + then .spec.jobTemplate.spec.template.spec + else .spec.template.spec + end + ) + | {cpu_m: (pod_cpu | ceil), memory_bytes: (pod_memory | ceil)} + ' +} + +capacity_snapshot() { + jq -n \ + --slurpfile nodes <(kubectl get nodes -o json) \ + --slurpfile pods <(kubectl get pods -A -o json) "$RESOURCE_FILTERS"' + ($pods[0].items + | map(select(.spec.nodeName != null and .status.phase != "Succeeded" and .status.phase != "Failed")) + | group_by(.spec.nodeName) + | map({ + key: .[0].spec.nodeName, + value: { + cpu_m: (map(.spec | pod_cpu) | add // 0), + memory_bytes: (map(.spec | pod_memory) | add // 0) + } + }) + | from_entries) as $used | + $nodes[0].items + | map({ + node: .metadata.name, + ready: any(.status.conditions[]?; .type == "Ready" and .status == "True"), + schedulable: ((.spec.unschedulable // false) | not), + alloc_cpu_m: (.status.allocatable.cpu | cpu_m), + alloc_memory_bytes: (.status.allocatable.memory | memory_bytes), + used_cpu_m: ($used[.metadata.name].cpu_m // 0), + used_memory_bytes: ($used[.metadata.name].memory_bytes // 0) + } + | . + { + free_cpu_m: (.alloc_cpu_m - .used_cpu_m), + free_memory_bytes: (.alloc_memory_bytes - .used_memory_bytes) + }) + ' +} + +REQUIRED_CPU_M=0 +REQUIRED_MEMORY_BYTES=0 +DB_CPU_M=0 +DB_MEMORY_BYTES=0 + +add_requirement() { + local request="$1" replicas="$2" cpu memory + cpu="$(printf '%s\n' "$request" | jq -r '.cpu_m')" + memory="$(printf '%s\n' "$request" | jq -r '.memory_bytes')" + REQUIRED_CPU_M=$((REQUIRED_CPU_M + cpu * replicas)) + REQUIRED_MEMORY_BYTES=$((REQUIRED_MEMORY_BYTES + memory * replicas)) +} + +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + DB_REQUEST="$(workload_request tidaldb-cluster statefulset/tidaldb pod)" + MONITOR_REQUEST="$(workload_request tidaldb-cluster deployment/tidal-soak-monitor pod)" + DB_CPU_M="$(printf '%s\n' "$DB_REQUEST" | jq -r '.cpu_m')" + DB_MEMORY_BYTES="$(printf '%s\n' "$DB_REQUEST" | jq -r '.memory_bytes')" + add_requirement "$DB_REQUEST" 3 + add_requirement "$MONITOR_REQUEST" 1 +fi +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + DB_REQUEST="$(workload_request tidaldb deployment/tidaldb pod)" + DB_CPU_M="$(printf '%s\n' "$DB_REQUEST" | jq -r '.cpu_m')" + DB_MEMORY_BYTES="$(printf '%s\n' "$DB_REQUEST" | jq -r '.memory_bytes')" + add_requirement "$DB_REQUEST" 1 +fi +if [ "$RESTORE_SOAK" -eq 1 ]; then + SOAK_REQUEST="$(workload_request tidaldb-cluster cronjob/tidal-soak-nightly cronjob)" + add_requirement "$SOAK_REQUEST" 1 +fi + +CAPACITY="$(capacity_snapshot)" +printf '==> scheduler request headroom\n' +printf '%s\n' "$CAPACITY" | jq -r '.[] | " \(.node): cpu=\(.free_cpu_m | floor)m free, memory=\((.free_memory_bytes / 1048576) | floor)Mi free, ready=\(.ready), schedulable=\(.schedulable)"' + +FREE_CPU_M="$(printf '%s\n' "$CAPACITY" | jq '[.[] | select(.ready and .schedulable) | .free_cpu_m] | add // 0 | floor')" +FREE_MEMORY_BYTES="$(printf '%s\n' "$CAPACITY" | jq '[.[] | select(.ready and .schedulable) | .free_memory_bytes] | add // 0 | floor')" + +[ "$FREE_CPU_M" -ge "$REQUIRED_CPU_M" ] || \ + die "selected restore needs ${REQUIRED_CPU_M}m CPU requests; only ${FREE_CPU_M}m is free" +[ "$FREE_MEMORY_BYTES" -ge "$REQUIRED_MEMORY_BYTES" ] || \ + die "selected restore needs $((REQUIRED_MEMORY_BYTES / 1048576))Mi memory requests; only $((FREE_MEMORY_BYTES / 1048576))Mi is free" + +node_has_capacity() { + local node="$1" + printf '%s\n' "$CAPACITY" | + jq -e --arg node "$node" --argjson cpu "$DB_CPU_M" --argjson memory "$DB_MEMORY_BYTES" \ + 'any(.[]; .node == $node and .ready and .schedulable and .free_cpu_m >= $cpu and .free_memory_bytes >= $memory)' \ + >/dev/null +} + +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + if [ -n "$CLUSTER_DATA_NODE_0" ] && [ -n "$CLUSTER_DATA_NODE_1" ] && [ -n "$CLUSTER_DATA_NODE_2" ]; then + UNIQUE_DATA_NODES="$( + jq -n --arg n0 "$CLUSTER_DATA_NODE_0" --arg n1 "$CLUSTER_DATA_NODE_1" --arg n2 "$CLUSTER_DATA_NODE_2" \ + '[$n0, $n1, $n2] | unique | length' + )" + [ "$UNIQUE_DATA_NODES" -eq 3 ] || die "RF3 data volumes are not pinned to three distinct nodes" + for node in "$CLUSTER_DATA_NODE_0" "$CLUSTER_DATA_NODE_1" "$CLUSTER_DATA_NODE_2"; do + node_has_capacity "$node" || \ + die "RF3 data volume node $node lacks ${DB_CPU_M}m CPU or $((DB_MEMORY_BYTES / 1048576))Mi memory" + done + elif [ -z "$CLUSTER_DATA_NODE_0$CLUSTER_DATA_NODE_1$CLUSTER_DATA_NODE_2" ]; then + CLUSTER_NODES="$( + printf '%s\n' "$CAPACITY" | + jq --argjson cpu "$DB_CPU_M" --argjson memory "$DB_MEMORY_BYTES" \ + '[.[] | select(.ready and .schedulable and .free_cpu_m >= $cpu and .free_memory_bytes >= $memory)] | length' + )" + [ "$CLUSTER_NODES" -ge 3 ] || \ + die "RF3 topology needs three nodes with ${DB_CPU_M}m CPU and $((DB_MEMORY_BYTES / 1048576))Mi memory free each; only $CLUSTER_NODES qualify" + else + die "RF3 data volumes have mixed local and network-backed placement; manual scheduling proof required" + fi +fi + +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + if [ -n "$STANDALONE_DATA_NODE" ]; then + node_has_capacity "$STANDALONE_DATA_NODE" || \ + die "standalone data volume node $STANDALONE_DATA_NODE lacks ${DB_CPU_M}m CPU or $((DB_MEMORY_BYTES / 1048576))Mi memory" + else + STANDALONE_TARGET_NODE="$( + printf '%s\n' "$CAPACITY" | + jq -r --argjson cpu "$DB_CPU_M" --argjson memory "$DB_MEMORY_BYTES" \ + 'first(.[] | select(.ready and .schedulable and .free_cpu_m >= $cpu and .free_memory_bytes >= $memory) | .node) // ""' + )" + [ -n "$STANDALONE_TARGET_NODE" ] || \ + die "no node can place standalone request ${DB_CPU_M}m/$((DB_MEMORY_BYTES / 1048576))Mi" + node_has_capacity "$STANDALONE_TARGET_NODE" || \ + die "selected standalone node $STANDALONE_TARGET_NODE lost capacity during preflight" + fi +fi + +printf '==> preflight passed: manifest-derived request %sm CPU and %sMi memory\n' \ + "$REQUIRED_CPU_M" "$((REQUIRED_MEMORY_BYTES / 1048576))" + +if [ "$CHECK_ONLY" -eq 1 ]; then + printf 'VERDICT: RESTORE PREFLIGHT PASSED (no resources changed)\n' + printf 'event=restore_complete ts=%s mode=check verdict=passed\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + exit 0 +fi +MUTATION_STARTED=1 +rollback_on_exit() { + local rc=$? + trap - EXIT + set +e + + if [ -n "$PREFLIGHT_POD" ]; then + kubectl -n tidaldb-cluster delete "pod/$PREFLIGHT_POD" \ + --ignore-not-found --wait=true >/dev/null 2>&1 || + printf 'WARNING: could not delete evaluator preflight pod %s\n' "$PREFLIGHT_POD" >&2 + fi + if [ "$rc" -eq 0 ] || [ "$MUTATION_STARTED" -eq 0 ]; then + exit "$rc" + fi + + printf 'ERROR: restore failed rc=%s; returning selected workloads to the parked state\n' "$rc" >&2 + printf 'event=restore_rollback ts=%s rc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$rc" >&2 + if [ "$RESTORE_SOAK" -eq 1 ]; then + kubectl -n tidaldb-cluster patch cronjob/tidal-soak-nightly \ + --type=merge -p '{"spec":{"suspend":true}}' >/dev/null || + printf 'ERROR: rollback could not suspend tidal-soak-nightly\n' >&2 + fi + if [ "$RESTORE_CLUSTER" -eq 1 ]; then + kubectl -n tidaldb-cluster scale deployment/tidal-soak-monitor --replicas=0 >/dev/null || + printf 'ERROR: rollback could not park tidal-soak-monitor\n' >&2 + kubectl -n tidaldb-cluster scale statefulset/tidaldb --replicas=0 >/dev/null || + printf 'ERROR: rollback could not park tidaldb StatefulSet\n' >&2 + kubectl -n tidaldb-cluster wait --for=delete pod \ + -l app.kubernetes.io/name=tidaldb --timeout=300s >/dev/null 2>&1 || + printf 'WARNING: rollback did not observe all cluster pods deleted within 300s\n' >&2 + kubectl -n tidaldb-cluster wait --for=delete pod \ + -l app.kubernetes.io/name=tidal-soak-monitor --timeout=300s >/dev/null 2>&1 || + printf 'WARNING: rollback did not observe soak monitor deletion within 300s\n' >&2 + fi + if [ "$RESTORE_STANDALONE" -eq 1 ]; then + kubectl -n tidaldb scale deployment/tidaldb --replicas=0 >/dev/null || + printf 'ERROR: rollback could not park standalone tidaldb\n' >&2 + kubectl -n tidaldb wait --for=delete pod \ + -l app.kubernetes.io/name=tidaldb --timeout=300s >/dev/null 2>&1 || + printf 'WARNING: rollback did not observe standalone pod deletion within 300s\n' >&2 + fi + printf '==> rollback requested; inspect workload state before retrying\n' >&2 + exit "$rc" +} +trap rollback_on_exit EXIT + +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + printf '==> restoring RF3 cluster\n' + kubectl -n tidaldb-cluster scale statefulset/tidaldb --replicas=3 + kubectl -n tidaldb-cluster rollout status statefulset/tidaldb --timeout=900s + kubectl -n tidaldb-cluster wait --for=condition=Ready \ + pod -l app.kubernetes.io/name=tidaldb --timeout=300s + + printf '==> restoring soak monitor (nightly job stays suspended)\n' + kubectl -n tidaldb-cluster scale deployment/tidal-soak-monitor --replicas=1 + kubectl -n tidaldb-cluster rollout status deployment/tidal-soak-monitor --timeout=300s +fi + +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + printf '==> restoring internal standalone store\n' + kubectl -n tidaldb scale deployment/tidaldb --replicas=1 + kubectl -n tidaldb rollout status deployment/tidaldb --timeout=600s + printf ' public Ingress and DNS remain retired by design\n' +fi + +if [ "$RESTORE_SOAK" -eq 1 ]; then + SOAK_IMAGE="$( + kubectl -n tidaldb-cluster get cronjob/tidal-soak-nightly \ + -o jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}' + )" + PREFLIGHT_POD=tidal-soak-eval-preflight + kubectl -n tidaldb-cluster delete "pod/$PREFLIGHT_POD" \ + --ignore-not-found --wait=true >/dev/null + PREFLIGHT_OVERRIDES="$( + jq -nc --arg image "$SOAK_IMAGE" '{ + spec: { + automountServiceAccountToken: false, + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + seccompProfile: {type: "RuntimeDefault"} + }, + containers: [{ + name: "preflight", + image: $image, + imagePullPolicy: "IfNotPresent", + command: ["soak-eval", "--help"], + resources: { + requests: {cpu: "10m", memory: "16Mi"}, + limits: {cpu: "100m", memory: "64Mi"} + }, + securityContext: { + allowPrivilegeEscalation: false, + readOnlyRootFilesystem: true, + capabilities: {drop: ["ALL"]} + } + }] + } + }' + )" + kubectl -n tidaldb-cluster run "$PREFLIGHT_POD" \ + --restart=Never --image="$SOAK_IMAGE" --overrides="$PREFLIGHT_OVERRIDES" + kubectl -n tidaldb-cluster wait --for=jsonpath='{.status.phase}'=Succeeded \ + "pod/$PREFLIGHT_POD" --timeout=180s + PREFLIGHT_HELP="$(kubectl -n tidaldb-cluster logs "$PREFLIGHT_POD")" + printf '%s\n' "$PREFLIGHT_HELP" + grep -q -- '--ledger-file' <<<"$PREFLIGHT_HELP" || + die "stress image soak-eval lacks --ledger-file; refusing the non-atomic nightly contract" + grep -q -- '--no-write-streak' <<<"$PREFLIGHT_HELP" || + die "stress image soak-eval lacks --no-write-streak; refusing the non-atomic nightly contract" + kubectl -n tidaldb-cluster delete "pod/$PREFLIGHT_POD" --wait=true >/dev/null + PREFLIGHT_POD="" + + printf '==> unsuspending nightly soak after evaluator proof\n' + kubectl -n tidaldb-cluster patch cronjob/tidal-soak-nightly \ + --type=merge -p '{"spec":{"suspend":false}}' +fi + +printf '==> final state\n' +if [ "$RESTORE_CLUSTER" -eq 1 ]; then + kubectl -n tidaldb-cluster get \ + statefulset/tidaldb deployment/tidal-soak-monitor cronjob/tidal-soak-nightly +fi +if [ "$RESTORE_STANDALONE" -eq 1 ]; then + kubectl -n tidaldb get deployment/tidaldb +fi +printf 'event=restore_complete ts=%s mode=restore verdict=passed\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" +printf 'VERDICT: SELECTED TIDALDB WORKLOADS RESTORED\n' diff --git a/tidal-server/src/cluster/node.rs b/tidal-server/src/cluster/node.rs index 9c5d9d6..eefb33e 100644 --- a/tidal-server/src/cluster/node.rs +++ b/tidal-server/src/cluster/node.rs @@ -497,6 +497,48 @@ struct DeferredRetire { deadline: Instant, } +const RESEED_TERMINATION_LOG: &str = "/dev/termination-log"; + +fn write_reseed_termination_message( + path: &std::path::Path, + region: &str, + shard: ShardId, +) -> std::io::Result<()> { + let message = serde_json::json!({ + "reason": "reseed_self_restart", + "region": region, + "shard": shard.0, + }); + std::fs::write(path, message.to_string()) +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod termination_message_tests { + use super::*; + + #[test] + fn reseed_self_restart_writes_machine_readable_termination_reason() { + let suffix = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let path = std::env::temp_dir().join(format!( + "tidaldb-termination-message-{}-{suffix}", + std::process::id() + )); + + write_reseed_termination_message(&path, "us-east", ShardId(7)).unwrap(); + let message: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + std::fs::remove_file(path).unwrap(); + + assert_eq!(message["reason"], "reseed_self_restart"); + assert_eq!(message["region"], "us-east"); + assert_eq!(message["shard"], 7); + } +} + impl ShardReplica { /// Build the single region named `region_name` from `topology`. /// @@ -3259,6 +3301,19 @@ impl ShardReplica { /// use the existing shutdown-signal path (SIGTERM-equivalent) so /// `serve_state`'s graceful shutdown fires — never `process::abort`. fn fire_graceful_self_restart(&self) { + if let Err(error) = write_reseed_termination_message( + std::path::Path::new(RESEED_TERMINATION_LOG), + &self.region_name, + self.group_shard, + ) { + tracing::warn!( + path = RESEED_TERMINATION_LOG, + region = %self.region_name, + shard = self.group_shard.0, + error = %error, + "failed to write Kubernetes termination message for reseed self-restart" + ); + } self.set_shutting_down(); if let Err(e) = std::thread::Builder::new() .name("tidal-reseed-restart".into()) @@ -8617,10 +8672,6 @@ where offload_read(f).await.map_err(ClusterAppError) } -/// m11p7 — the cluster auth middleware exercised through the real layer -/// composition (`middleware::from_fn`), not just the pure `ClusterCreds` -/// helpers. A layer-order or marker-set regression in [`cluster_auth_middleware`] -/// is caught HERE (the helper unit tests in `security.rs` would still pass). #[cfg(test)] #[allow(clippy::unwrap_used)] mod aggregate_region_row_tests { @@ -8983,22 +9034,19 @@ mod forward_retry_tests { }; use axum::http::StatusCode; - fn ok( - status: StatusCode, - body: serde_json::Value, - ) -> Result { - Ok(forward::ForwardedResponse { + fn forwarded(status: StatusCode, body: serde_json::Value) -> forward::ForwardedResponse { + forward::ForwardedResponse { seq: None, deduplicated: false, status, body, - }) + } } #[test] fn success_relays_immediately() { // A 2xx leader verdict is handed straight back — never retried. - let outcome = ok(StatusCode::CREATED, serde_json::Value::Null); + let outcome = Ok::<_, String>(forwarded(StatusCode::CREATED, serde_json::Value::Null)); assert_eq!( classify_forward_attempt(&outcome, 1, Duration::ZERO), ForwardStep::Relay @@ -9010,10 +9058,10 @@ mod forward_retry_tests { // The leader's bounded write pool shed this write (429 + retry_after_ms): // it created no log entry, so re-forwarding is safe. We retry, honoring // the leader's hint. - let outcome = ok( + let outcome = Ok::<_, String>(forwarded( StatusCode::TOO_MANY_REQUESTS, serde_json::json!({ "retry_after_ms": 80 }), - ); + )); assert_eq!( classify_forward_attempt(&outcome, 1, Duration::ZERO), ForwardStep::Retry { backoff_ms: 80 } @@ -9025,7 +9073,10 @@ mod forward_retry_tests { // On the LAST attempt a 429 is relayed verbatim: the targeted-follower // client sees a retryable 429 (honest backpressure), NOT a 503 that would // wrongly say the leader is unreachable. - let outcome = ok(StatusCode::TOO_MANY_REQUESTS, serde_json::Value::Null); + let outcome = Ok::<_, String>(forwarded( + StatusCode::TOO_MANY_REQUESTS, + serde_json::Value::Null, + )); assert_eq!( classify_forward_attempt(&outcome, FORWARD_MAX_ATTEMPTS, Duration::ZERO), ForwardStep::Relay diff --git a/tidal-server/tests/cluster_graph_persistence.rs b/tidal-server/tests/cluster_graph_persistence.rs index 1b76b2e..bfec161 100644 --- a/tidal-server/tests/cluster_graph_persistence.rs +++ b/tidal-server/tests/cluster_graph_persistence.rs @@ -9,8 +9,8 @@ //! (`axum::serve(...).with_graceful_shutdown(...)`) blocks until every in-flight //! connection closes. Sibling region nodes hold long-lived keep-alive HTTP //! connections that do not close promptly on our SIGTERM, so the drain — and thus -//! `axum::serve(...).await` — never returned inside the 60s grace; k8s then -//! SIGKILLed the process, and SIGKILL cannot run `Drop`/`shutdown_inner` (where +//! `axum::serve(...).await` — never returned inside the 60s grace; k8s then sent +//! `SIGKILL`. That signal cannot run `Drop`/`shutdown_inner` (where //! the WAL checkpoint marker AND the HNSW-graph checkpoint are written). //! //! The fix (in `main.rs` + `cluster/node.rs`): diff --git a/tidal-stress/Cargo.toml b/tidal-stress/Cargo.toml index e8db283..7618077 100644 --- a/tidal-stress/Cargo.toml +++ b/tidal-stress/Cargo.toml @@ -54,6 +54,7 @@ rand = "0.9" # tidal-net — drops the OpenSSL/native-tls system dependency for a clean static # build. `json` for the typed request/response bodies. reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +time = { version = "0.3", features = ["formatting", "parsing"] } [dev-dependencies] # Honest before/after for the generator's OWN per-request cost (no server, no diff --git a/tidal-stress/k8s/soak-monitor.yaml b/tidal-stress/k8s/soak-monitor.yaml index 9846c45..71d9fcb 100644 --- a/tidal-stress/k8s/soak-monitor.yaml +++ b/tidal-stress/k8s/soak-monitor.yaml @@ -4,7 +4,7 @@ # writes its JSON summaries to, so EVERYTHING the operator needs to judge the # 30-night streak lives in one place and survives the laptop session ending: # -# 1. Restart watch: every 5 min, snapshot the tidaldb-{0,1,2} pod restart +# 1. Restart watch: every minute, snapshot the tidaldb-{0,1,2} pod identity, # counts + phase into /results/restarts.tsv. The GA bar is not just # "30 green soak verdicts" — it is ZERO unrecovered failures over the # window. An under-load pod restart during a soak night must be visible @@ -15,9 +15,9 @@ # and read the ledger / nightly summaries / restart log from a browser at # any time, from any machine, without exec'ing into a pod. # -# The monitor does NOT generate load and does NOT gate anything — it is a passive -# recorder. The pass/fail signal is the CronJob's Job exit codes; this just makes -# the 30-night picture observable and durable. +# The monitor does not generate load. The nightly Job is the authoritative gate; +# this deployment maintains the evidence stream and continuously recomputes the +# durable streak/status surface. # # Apply: kubectl apply -f tidal-stress/k8s/soak-monitor.yaml # Read: kubectl port-forward deploy/tidal-soak-monitor 8080:8080 -n tidaldb-cluster @@ -76,7 +76,7 @@ metadata: app.kubernetes.io/name: tidal-soak app.kubernetes.io/part-of: tidaldb spec: - replicas: 1 + replicas: 0 # parked; scripts/restore-fleet.sh starts it only after capacity checks selector: matchLabels: app.kubernetes.io/name: tidal-soak-monitor @@ -95,28 +95,22 @@ spec: seccompProfile: type: RuntimeDefault containers: - # ── Restart-watch sidecar: kubectl snapshot loop ────────────────────── + # ── Restart watcher: direct Kubernetes API sampler ─────────────────── - name: restart-watch - image: bitnami/kubectl:latest + image: registry.threesix.ai/tidal/stress:m12-fleet-remediation-20260815-r2@sha256:86bfa95f63ba01ed4afa120e41684cd8694743ec39aca72ef0bd68e5d9ef7f8b imagePullPolicy: IfNotPresent - command: ["/bin/sh", "-c"] + command: ["soak-watch"] args: - - | - echo "restart-watch up $(date -u +%FT%TZ)" - # Header once (only if the file is new/empty). - if [ ! -s /results/restarts.tsv ]; then - printf 'ts_utc\tpod\trestarts\tphase\tready\n' >> /results/restarts.tsv - fi - while true; do - TS="$(date -u +%FT%TZ)" - kubectl get pods -n tidaldb-cluster \ - -l app.kubernetes.io/name=tidaldb \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[0].ready}{"\n"}{end}' 2>/dev/null \ - | while IFS="$(printf '\t')" read -r POD RC PH RD; do - [ -n "$POD" ] && printf '%s\t%s\t%s\t%s\t%s\n' "$TS" "$POD" "$RC" "$PH" "$RD" >> /results/restarts.tsv - done - sleep 300 - done + - --namespace + - tidaldb-cluster + - --label-selector + - app.kubernetes.io/name=tidaldb + - --container + - tidaldb + - --results-file + - /results/restarts.tsv + - --interval-seconds + - "60" resources: requests: { cpu: 10m, memory: 32Mi } limits: { cpu: 100m, memory: 128Mi } @@ -127,18 +121,12 @@ spec: volumeMounts: - name: results mountPath: /results - # ── Streak evaluator: the GA-bar verdict + alerting ─────────────────── - # Joins ledger.tsv (the nightly Job verdicts) with restarts.tsv (the - # cumulative per-pod restart counts the sidecar above records) into the - # HONEST streak: a night is green iff the soak passed AND no pod restarted - # under load. Runs the proven `soak-eval` core (table-driven-tested in - # tidal_stress::soak_eval), writes streak.tsv, and emits EXACTLY ONE alert - # per non-green night transition (a durable ALERT-.txt the http - # surface serves, plus an optional webhook POST when NOTIFY_WEBHOOK is set - # — reuse the cluster's notify service by pointing that at it). It does NOT - # gate load; it makes the 30-night picture trustworthy and observable. + # ── Streak evaluator: durable GA-bar status ──────────────────────────── + # Joins the nightly verdict ledger with the restart evidence above. + # The nightly Job fails closed on this same evaluator; this sidecar keeps + # streak.tsv and one durable ALERT-.txt current for operators. - name: evaluator - image: registry.threesix.ai/tidal/stress:m12-soak-eval + image: registry.threesix.ai/tidal/stress:m12-fleet-remediation-20260815-r2@sha256:86bfa95f63ba01ed4afa120e41684cd8694743ec39aca72ef0bd68e5d9ef7f8b imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] args: @@ -159,8 +147,6 @@ spec: MSG="tidalDB soak: night $LAST NON-GREEN — $REASON (streak reset)" echo "ALERT: $MSG" printf '%s\t%s\n' "$LAST" "$REASON" > "/results/ALERT-$LAST.txt" - [ -n "${NOTIFY_WEBHOOK:-}" ] && \ - wget -q -O- --post-data="{\"text\":\"$MSG\"}" --header='Content-Type: application/json' "$NOTIFY_WEBHOOK" >/dev/null 2>&1 || true echo "$LAST" > /results/.last-alert fi fi @@ -171,11 +157,6 @@ spec: find /results -maxdepth 1 -name 'soak-*.json' -mtime +31 -delete 2>/dev/null || true sleep 300 done - env: - # Point at the cluster's notify service to deliver alerts off-cluster; - # unset = log + durable ALERT-.txt only (still status-surfaced). - - name: NOTIFY_WEBHOOK - value: "" resources: requests: { cpu: 10m, memory: 32Mi } limits: { cpu: 200m, memory: 128Mi } @@ -188,7 +169,7 @@ spec: mountPath: /results # ── HTTP read surface: serve the durable result dir ─────────────────── - name: http - image: busybox:1.36 + image: busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662 imagePullPolicy: IfNotPresent # busybox httpd: one-shot static file server rooted at /results. command: ["/bin/sh", "-c"] @@ -199,12 +180,11 @@ spec: ports: - name: http containerPort: 8080 - # Readiness gates the Service endpoint on the http surface actually - # serving — a monitor whose node is rebooting must drop out of endpoints - # rather than silently serve nothing while a restart it should be - # recording slips by unobserved. + # Readiness gates the Service endpoint on the HTTP listener. The + # result directory intentionally has no index file, so GET / returns + # 404 even while the static server is healthy. readinessProbe: - httpGet: { path: /, port: 8080 } + tcpSocket: { port: http } initialDelaySeconds: 5 periodSeconds: 10 failureThreshold: 3 diff --git a/tidal-stress/k8s/soak-nightly-cronjob.yaml b/tidal-stress/k8s/soak-nightly-cronjob.yaml index 5bc0fa0..48c86c8 100644 --- a/tidal-stress/k8s/soak-nightly-cronjob.yaml +++ b/tidal-stress/k8s/soak-nightly-cronjob.yaml @@ -22,17 +22,14 @@ # The streak passes when 30 CONSECUTIVE nightly Jobs have exited 0 with zero # under-load pod restarts (the monitor records restarts; see soak-monitor.yaml). # -# SUSTAINED RATE: 200 rps mixed (peach: feed/search/view/like/skip/item/embed). -# RE-SCOPED 2026-06-19 from 500 rps. The original 500 rps cited the rc12 READ-SLA -# gate, but the peach mix is WRITE-heavy (view/like/skip ≈ 90% of ops) and the -# write path is structurally capped on this 3-node RF3 full-placement cluster -# (every follower applies every 1536-D write; T5). At 500 rps the soak FAILED -# every night (2026-06-17/18/19: p99 198-305ms, error 12-33%). A 2026-06-19 -# capacity sweep on a settled cluster measured the peach mix CLEAN (0.00% error, -# p99 schedule-lag <5ms, 0 follower restarts) at 100/150/200/250 rps; the knee is -# between 250 and 500. 200 rps is the measured sustainable rate with margin for a -# 1-hour × 30-consecutive-night endurance run. Raise only with more write capacity -# (≥5 nodes / partitioned placement) — see orchard9-k3sf cluster-state.yaml. +# HISTORICAL RATE: 200 rps mixed (peach: feed/search/view/like/skip/item/embed). +# The 2026-06-19 one-shot sweep found 200 rps clean, but the subsequent nightly +# history was only 23 PASS / 32 FAIL and included repeated 4 GiB OOMKills. This +# CronJob is therefore parked (`suspend: true`). Resume it only after a controlled +# 100k/1536-D/200-rps memory profile passes under the corrected resource envelope, +# restart evidence is fail-closed, and the exact immutable images below exist. +# Adding nodes to RF3 full placement does not increase write capacity; the old +# ≥2.5x scaling goal requires Ref-B partitioned placement. # # WHY A CRONJOB, NOT ONE LONG JOB: the GA bar is literally "30 CONSECUTIVE DAYS" # of an independent nightly verdict — a CronJob produces exactly that audit trail @@ -57,6 +54,7 @@ spec: # morning activity. concurrencyPolicy Forbid: never overlap two soaks (they # would contend for the same 3 nodes and mutually depress p99 → false FAIL). schedule: "0 2 * * *" + suspend: true concurrencyPolicy: Forbid startingDeadlineSeconds: 3600 successfulJobsHistoryLimit: 30 # keep all 30 nights of PASS verdicts @@ -87,7 +85,7 @@ spec: type: RuntimeDefault containers: - name: soak - image: registry.threesix.ai/tidal/stress:m12-rc7-seedretry + image: registry.threesix.ai/tidal/stress:m12-fleet-remediation-20260815-r2@sha256:86bfa95f63ba01ed4afa120e41684cd8694743ec39aca72ef0bd68e5d9ef7f8b imagePullPolicy: IfNotPresent # entrypoint is tidal-stress; wrap it so we can stamp the result file # name with the date and append a ledger line regardless of verdict. @@ -96,8 +94,16 @@ spec: - | set -u DATE="$(date -u +%Y-%m-%d)" + START_TS="$(date -u +%FT%TZ)" OUT="/results/soak-$DATE.json" - echo "soak $DATE start $(date -u +%H:%M:%SZ) target=cluster image=m12-rc7-seedretry-200rps" + OUT_TMP="/results/.soak-$DATE-$$.json" + TAB="$(printf '\t')" + if [ -f /results/ledger.tsv ] && grep -q "^${DATE}${TAB}" /results/ledger.tsv; then + echo "FATAL: ledger already contains $DATE; one calendar night cannot count twice" >&2 + exit 124 + fi + rm -f "$OUT_TMP" + echo "soak $DATE start $START_TS target=cluster image=m12-fleet-remediation-200rps" tidal-stress \ --target https://tidaldb-0.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \ --target https://tidaldb-1.tidaldb-peers.tidaldb-cluster.svc.cluster.local:9500 \ @@ -109,36 +115,115 @@ spec: --mix peach \ --users 50000 \ --ramp "200:3600" \ - --json-summary "$OUT" \ + --json-summary "$OUT_TMP" \ --max-p99-ms 150 \ --max-error-pct 1 \ --fail-on-knee - RC=$? - # Extract the verdict + headline p99/error from the JSON summary - # for a one-line ledger entry (grep, no jq dependency in the image). - # Fields are the real summary.rs keys: top-level "passed" (bool), - # per-stage "overall_p99_ms" (ms) and "error_rate" (FRACTION 0-1). - # Worst stage = max p99 over the run; we report the LAST stage's - # numbers (the sustained-rate stage, since the soak ramp is single- - # stage 500:3600 anyway) and trust "passed" for the verdict. - VERD="PASS"; [ "$RC" -ne 0 ] && VERD="FAIL" - PASSED="$(grep -o '"passed"[: ]*[a-z]*' "$OUT" 2>/dev/null | head -1 | grep -o '[a-z]*$')" - P99="$(grep -o '"overall_p99_ms"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" - ERRF="$(grep -o '"error_rate"[: ]*[0-9.]*' "$OUT" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" - printf '%s\t%s\trc=%s\tpassed=%s\tp99_ms=%s\terr_frac=%s\timage=m12-rc7-seedretry-200rps\n' \ - "$DATE" "$VERD" "$RC" "${PASSED:-?}" "${P99:-?}" "${ERRF:-?}" >> /results/ledger.tsv - echo "soak $DATE end $(date -u +%H:%M:%SZ) verdict=$VERD rc=$RC p99=${P99:-?} err=${ERRF:-?}" - # Refresh the streak immediately, CONSULTING restarts.tsv: a night - # whose Job passed but whose window carried a pod restart is recorded - # NON-GREEN by soak-eval (the zero-under-load-restart half of the GA - # bar). This makes the per-night verdict restart-aware without giving - # the soak Job pod-read RBAC — the monitor records restarts; soak-eval - # only reads the two files. The Job's OWN exit stays the SLO verdict. - if command -v soak-eval >/dev/null 2>&1; then - soak-eval --results-dir /results --target 30 || \ - echo "soak $DATE: streak NON-GREEN per soak-eval (see /results/streak.tsv)" + LOAD_RC=$? + END_TS="$(date -u +%FT%TZ)" + + # Extract the sustained stage's headline values for the ledger. + # The process exit remains authoritative, but missing/malformed + # evidence is independently fatal: an empty summary can never + # turn a completed process into a green night. + PASSED="$(grep -o '"passed"[: ]*[a-z]*' "$OUT_TMP" 2>/dev/null | head -1 | grep -o '[a-z]*$')" + P99="$(grep -o '"overall_p99_ms"[: ]*[0-9.]*' "$OUT_TMP" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" + ERRF="$(grep -o '"error_rate"[: ]*[0-9.]*' "$OUT_TMP" 2>/dev/null | tail -1 | grep -o '[0-9.]*$')" + SUMMARY_RC=0 + case "$PASSED" in true|false) ;; *) SUMMARY_RC=126 ;; esac + printf '%s\n' "$P99" | grep -Eq '^[0-9]+([.][0-9]+)?$' || SUMMARY_RC=126 + printf '%s\n' "$ERRF" | grep -Eq '^[0-9]+([.][0-9]+)?$' || SUMMARY_RC=126 + if [ "$LOAD_RC" -eq 0 ] && [ "$PASSED" != "true" ]; then + SUMMARY_RC=126 fi - exit "$RC" + if [ "$SUMMARY_RC" -eq 0 ]; then + if ! mv "$OUT_TMP" "$OUT"; then + echo "FATAL: failed to publish $OUT atomically" >&2 + SUMMARY_RC=125 + fi + else + echo "FATAL: malformed or inconsistent JSON summary (passed=${PASSED:-?} p99=${P99:-?} err=${ERRF:-?})" >&2 + mv "$OUT_TMP" "/results/soak-$DATE-invalid-$$.json" 2>/dev/null || true + fi + + RC="$LOAD_RC" + if [ "$RC" -eq 0 ] && [ "$SUMMARY_RC" -ne 0 ]; then + RC="$SUMMARY_RC" + fi + SOAK_VERD="PASS"; [ "$RC" -ne 0 ] && SOAK_VERD="FAIL" + # Stage the candidate ledger separately. Nothing reads this + # path except the one-shot evaluator, so a killed Job cannot + # leave a provisional PASS in the durable public ledger. + PENDING_LEDGER_NAME=".ledger-$DATE-pending.tsv" + PENDING_LEDGER="/results/$PENDING_LEDGER_NAME" + if [ -f /results/ledger.tsv ]; then + cp /results/ledger.tsv "$PENDING_LEDGER" || { + echo "FATAL: failed to stage existing /results/ledger.tsv" >&2 + exit 125 + } + : > "$PENDING_LEDGER" || { + echo "FATAL: failed to create candidate ledger" >&2 + exit 125 + } + fi + if ! printf '%s\t%s\tsoak_rc=%s\tsummary_rc=%s\tgate_rc=pending\tpassed=%s\tp99_ms=%s\terr_frac=%s\timage=m12-fleet-remediation-200rps\tstart_utc=%s\tend_utc=%s\n' \ + "$DATE" "$SOAK_VERD" "$RC" "$SUMMARY_RC" "${PASSED:-?}" "${P99:-?}" "${ERRF:-?}" "$START_TS" "$END_TS" \ + >> "$PENDING_LEDGER"; then + echo "FATAL: failed to stage candidate ledger" >&2 + rm -f "$PENDING_LEDGER" + exit 125 + fi + echo "soak $DATE load end $END_TS verdict=$SOAK_VERD load_rc=$LOAD_RC summary_rc=$SUMMARY_RC p99=${P99:-?} err=${ERRF:-?}" + + # Fail closed on the other half of the GA contract. The + # monitor samples once per minute; wait one full interval, + # then require fresh evidence bracketing both load boundaries + # for every expected cluster pod. + FINAL_RC="$RC" + if ! command -v soak-eval >/dev/null 2>&1; then + echo "FATAL: soak-eval missing from stress image" >&2 + EVAL_RC=127 + else + sleep 70 + soak-eval \ + --ledger-file "$PENDING_LEDGER_NAME" \ + --no-write-streak \ + --results-dir /results \ + --target 30 \ + --require-date "$DATE" \ + --require-samples-through "$END_TS" \ + --expected-pods 3 + EVAL_RC=$? + fi + FINAL_VERD="$SOAK_VERD" + if [ "$EVAL_RC" -ne 0 ]; then + echo "soak $DATE: recovery-evidence gate failed rc=$EVAL_RC (see this Job log and /results/restarts.tsv)" >&2 + FINAL_RC="$EVAL_RC" + FINAL_VERD="FAIL" + fi + + # Convert the private candidate's last row into the combined + # load + recovery verdict. CronJob concurrency is Forbid, so + # this is the only writer; rename publishes one atomic ledger. + LEDGER_TMP="/results/.ledger-$DATE.tmp" + if ! { + sed '$d' "$PENDING_LEDGER" + printf '%s\t%s\tsoak_rc=%s\tsummary_rc=%s\tgate_rc=%s\tpassed=%s\tp99_ms=%s\terr_frac=%s\timage=m12-fleet-remediation-200rps\tstart_utc=%s\tend_utc=%s\n' \ + "$DATE" "$FINAL_VERD" "$RC" "$SUMMARY_RC" "$EVAL_RC" "${PASSED:-?}" "${P99:-?}" "${ERRF:-?}" "$START_TS" "$END_TS" + } > "$LEDGER_TMP" || ! mv "$LEDGER_TMP" /results/ledger.tsv; then + echo "FATAL: failed to finalize /results/ledger.tsv" >&2 + rm -f "$LEDGER_TMP" "$PENDING_LEDGER" + exit 125 + fi + rm -f "$PENDING_LEDGER" + + # Recompute streak.tsv from the finalized ledger. A non-zero + # result is expected on a failed night and must not mask FINAL_RC. + if command -v soak-eval >/dev/null 2>&1; then + soak-eval --results-dir /results --target 30 >/dev/null 2>&1 || true + fi + echo "soak $DATE final verdict=$FINAL_VERD load_rc=$LOAD_RC summary_rc=$SUMMARY_RC gate_rc=$EVAL_RC" + exit "$FINAL_RC" env: - name: TIDAL_API_KEY valueFrom: diff --git a/tidal-stress/k8s/soak-results-pvc.yaml b/tidal-stress/k8s/soak-results-pvc.yaml index a921944..8920a68 100644 --- a/tidal-stress/k8s/soak-results-pvc.yaml +++ b/tidal-stress/k8s/soak-results-pvc.yaml @@ -23,6 +23,8 @@ metadata: labels: app.kubernetes.io/name: tidal-soak app.kubernetes.io/part-of: tidaldb + backup.orchard9.ai/class: protected + backup.orchard9.ai/method: velero-kopia spec: accessModes: - ReadWriteMany diff --git a/tidal-stress/src/bin/soak-eval.rs b/tidal-stress/src/bin/soak-eval.rs index c4b3175..7160787 100644 --- a/tidal-stress/src/bin/soak-eval.rs +++ b/tidal-stress/src/bin/soak-eval.rs @@ -27,30 +27,117 @@ struct Cli { #[arg(long, default_value = "/results")] results_dir: PathBuf, + /// Ledger filename within the result dir. The nightly Job points this at an + /// unpublished candidate so a killed Job cannot expose a provisional PASS. + #[arg(long, default_value = "ledger.tsv")] + ledger_file: PathBuf, + + /// Evaluate and return the verdict without replacing `streak.tsv`. + #[arg(long)] + no_write_streak: bool, + /// Consecutive green nights required for GA (the exit gate). #[arg(long, default_value_t = 30)] target: usize, + + /// Require the newest ledger row to be this date. The nightly Job passes + /// its own UTC date so missing/stale inputs are fatal rather than streak 0. + #[arg(long)] + require_date: Option, + + /// Require the newest ledger row to end at this ISO-8601 timestamp. Restart + /// evaluation then requires fresh per-pod samples immediately before the + /// row's start and after this end bound. + #[arg(long, requires = "require_date")] + require_samples_through: Option, + + /// Number of cluster pods whose restart evidence must be present. + #[arg(long, default_value_t = 3)] + expected_pods: usize, } fn main() -> ExitCode { let cli = Cli::parse(); - let ledger_path = cli.results_dir.join("ledger.tsv"); + if cli.target == 0 || cli.expected_pods == 0 { + eprintln!("soak-eval: --target and --expected-pods must both be greater than zero"); + return ExitCode::from(2); + } + let ledger_path = cli.results_dir.join(&cli.ledger_file); let restarts_path = cli.results_dir.join("restarts.tsv"); - // A missing ledger means no night has run yet — streak 0, not an error. - let ledger = std::fs::read_to_string(&ledger_path).unwrap_or_default(); - let restarts = std::fs::read_to_string(&restarts_path).unwrap_or_default(); + let read_input = |path: &std::path::Path| match std::fs::read_to_string(path) { + Ok(input) => Ok(input), + Err(error) + if error.kind() == std::io::ErrorKind::NotFound && cli.require_date.is_none() => + { + Ok(String::new()) + } + Err(error) => Err(error), + }; + let ledger = match read_input(&ledger_path) { + Ok(input) => input, + Err(error) => { + eprintln!( + "soak-eval: failed to read {}: {error}", + ledger_path.display() + ); + return ExitCode::from(2); + } + }; + let restarts = match read_input(&restarts_path) { + Ok(input) => input, + Err(error) => { + eprintln!( + "soak-eval: failed to read {}: {error}", + restarts_path.display() + ); + return ExitCode::from(2); + } + }; let nights = parse_ledger(&ledger); + if let Some(required) = cli.require_date.as_deref() + && nights.last().map(|night| night.date.as_str()) != Some(required) + { + eprintln!( + "soak-eval: newest ledger date is {:?}, expected {required}", + nights.last().map(|night| night.date.as_str()) + ); + return ExitCode::from(2); + } let samples = parse_restarts(&restarts); - let verdicts = evaluate(&nights, &samples); + if let (Some(required_date), Some(required_timestamp)) = ( + cli.require_date.as_deref(), + cli.require_samples_through.as_deref(), + ) && nights + .last() + .and_then(|night| night.end_timestamp.as_deref()) + != Some(required_timestamp) + { + eprintln!( + "soak-eval: ledger end timestamp for {required_date} is {:?}, expected {required_timestamp}", + nights + .last() + .and_then(|night| night.end_timestamp.as_deref()) + ); + return ExitCode::from(2); + } + let verdicts = evaluate(&nights, &samples, cli.expected_pods); let streak = current_streak(&verdicts); - let tsv = render_streak_tsv(&verdicts, cli.target); - let out_path = cli.results_dir.join("streak.tsv"); - if let Err(e) = std::fs::write(&out_path, &tsv) { - eprintln!("soak-eval: failed to write {}: {e}", out_path.display()); - return ExitCode::from(2); + if !cli.no_write_streak { + let tsv = render_streak_tsv(&verdicts, cli.target); + let out_path = cli.results_dir.join("streak.tsv"); + let tmp_path = cli + .results_dir + .join(format!(".streak-{}.tmp", std::process::id())); + if let Err(error) = + std::fs::write(&tmp_path, &tsv).and_then(|()| std::fs::rename(&tmp_path, &out_path)) + { + let _ = std::fs::remove_file(&tmp_path); + eprintln!("soak-eval: failed to write {}: {error}", out_path.display()); + return ExitCode::from(2); + } } println!( @@ -59,15 +146,14 @@ fn main() -> ExitCode { verdicts.len() ); if streak >= cli.target { - println!("soak-eval: GA STREAK MET ({streak} ≥ {})", cli.target); + println!("soak-eval: GA STREAK MET ({streak} >= {})", cli.target); } - // Exit non-zero iff the MOST RECENT night is non-green — the alert trigger. match verdicts.last() { - Some(v) if !v.green => { + Some(verdict) if !verdict.green => { eprintln!( "soak-eval: ALERT — last night {} is NON-GREEN: {}", - v.date, v.reason + verdict.date, verdict.reason ); ExitCode::from(1) } diff --git a/tidal-stress/src/bin/soak-watch.rs b/tidal-stress/src/bin/soak-watch.rs new file mode 100644 index 0000000..06a61ea --- /dev/null +++ b/tidal-stress/src/bin/soak-watch.rs @@ -0,0 +1,304 @@ +//! `soak-watch` records Kubernetes pod generations and restart evidence. +//! +//! It talks directly to the in-cluster Kubernetes API with the pod's service +//! account. This keeps the durable soak evidence independent of a mutable +//! `kubectl` utility image. + +use std::{ + error::Error, + fs::OpenOptions, + io::{self, Write}, + path::{Path, PathBuf}, + process::ExitCode, + time::Duration, +}; + +use clap::Parser; +use serde::Deserialize; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; + +#[derive(Parser)] +#[command(version, about = "Record Kubernetes pod restart evidence as TSV")] +struct Cli { + #[arg(long, default_value = "tidaldb-cluster")] + namespace: String, + + #[arg(long, default_value = "app.kubernetes.io/name=tidaldb")] + label_selector: String, + + #[arg(long, default_value = "tidaldb")] + container: String, + + #[arg(long, default_value = "/results/restarts.tsv")] + results_file: PathBuf, + + #[arg(long, default_value_t = 60)] + interval_seconds: u64, +} + +#[derive(Deserialize)] +struct PodList { + items: Vec, +} + +#[derive(Deserialize)] +struct Pod { + metadata: Metadata, + #[serde(default)] + status: PodStatus, +} + +#[derive(Deserialize)] +struct Metadata { + name: String, + uid: String, +} + +#[derive(Default, Deserialize)] +#[serde(rename_all = "camelCase")] +struct PodStatus { + #[serde(default)] + phase: String, + #[serde(default)] + container_statuses: Vec, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ContainerStatus { + name: String, + restart_count: u64, + ready: bool, + #[serde(default)] + last_state: ContainerState, +} + +#[derive(Default, Deserialize)] +struct ContainerState { + terminated: Option, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct TerminatedState { + #[serde(default)] + reason: String, + exit_code: i32, + #[serde(default)] + message: String, +} + +struct KubernetesClient { + client: reqwest::Client, + pods_url: String, + token: String, +} + +impl KubernetesClient { + fn in_cluster(namespace: &str) -> Result> { + let host = std::env::var("KUBERNETES_SERVICE_HOST").map_err(|error| { + io::Error::new( + io::ErrorKind::NotFound, + format!("KUBERNETES_SERVICE_HOST is unavailable: {error}"), + ) + })?; + let port = + std::env::var("KUBERNETES_SERVICE_PORT_HTTPS").unwrap_or_else(|_| String::from("443")); + let service_account = Path::new("/var/run/secrets/kubernetes.io/serviceaccount"); + let token = std::fs::read_to_string(service_account.join("token"))? + .trim() + .to_owned(); + let ca = std::fs::read(service_account.join("ca.crt"))?; + let certificate = reqwest::Certificate::from_pem(&ca)?; + let client = reqwest::Client::builder() + .add_root_certificate(certificate) + .https_only(true) + .timeout(Duration::from_secs(15)) + .build()?; + + Ok(Self { + client, + pods_url: format!("https://{host}:{port}/api/v1/namespaces/{namespace}/pods"), + token, + }) + } + + async fn pods(&self, label_selector: &str) -> Result { + self.client + .get(&self.pods_url) + .bearer_auth(&self.token) + .query(&[("labelSelector", label_selector)]) + .send() + .await? + .error_for_status()? + .json() + .await + } +} + +fn timestamp() -> Result { + OffsetDateTime::now_utc().format(&Rfc3339) +} + +fn sanitize_tsv(value: &str) -> String { + value + .chars() + .map(|character| match character { + '\t' | '\n' | '\r' => ' ', + other => other, + }) + .collect() +} + +fn append_sample( + path: &Path, + sampled_at: &str, + pods: &PodList, + container_name: &str, +) -> io::Result { + let write_header = std::fs::metadata(path).map_or(true, |metadata| metadata.len() == 0); + let mut output = OpenOptions::new().create(true).append(true).open(path)?; + if write_header { + writeln!( + output, + "ts_utc\tpod\tuid\trestarts\tphase\tready\tlast_reason\tlast_exit\tlast_message" + )?; + } + + let mut rows = 0; + for pod in &pods.items { + let Some(container) = pod + .status + .container_statuses + .iter() + .find(|status| status.name == container_name) + else { + continue; + }; + let (reason, exit_code, message) = container.last_state.terminated.as_ref().map_or( + ("", String::new(), ""), + |terminated| { + ( + terminated.reason.as_str(), + terminated.exit_code.to_string(), + terminated.message.as_str(), + ) + }, + ); + writeln!( + output, + "{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}", + sampled_at, + sanitize_tsv(&pod.metadata.name), + sanitize_tsv(&pod.metadata.uid), + container.restart_count, + sanitize_tsv(&pod.status.phase), + container.ready, + sanitize_tsv(reason), + exit_code, + sanitize_tsv(message), + )?; + rows += 1; + } + output.flush()?; + Ok(rows) +} + +async fn run() -> Result<(), Box> { + let cli = Cli::parse(); + if cli.interval_seconds == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "--interval-seconds must be greater than zero", + ) + .into()); + } + let kubernetes = KubernetesClient::in_cluster(&cli.namespace)?; + println!( + "soak-watch started: namespace={} selector={} container={} interval={}s output={}", + cli.namespace, + cli.label_selector, + cli.container, + cli.interval_seconds, + cli.results_file.display() + ); + + loop { + let sampled_at = timestamp()?; + match kubernetes.pods(&cli.label_selector).await { + Ok(pods) => { + match append_sample(&cli.results_file, &sampled_at, &pods, &cli.container) { + Ok(rows) => println!("soak-watch sample: ts={sampled_at} rows={rows}"), + Err(error) => eprintln!( + "soak-watch write failed: ts={sampled_at} path={} error={error}", + cli.results_file.display() + ), + } + } + Err(error) => { + eprintln!("soak-watch Kubernetes sample failed: ts={sampled_at} error={error}"); + } + } + tokio::time::sleep(Duration::from_secs(cli.interval_seconds)).await; + } +} + +#[tokio::main] +async fn main() -> ExitCode { + match run().await { + Ok(()) => ExitCode::SUCCESS, + Err(error) => { + eprintln!("soak-watch fatal: {error}"); + ExitCode::from(1) + } + } +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod tests { + use super::*; + + #[test] + fn append_sample_records_uid_and_termination_message_without_multiline_tsv() { + let path = std::env::temp_dir().join(format!( + "soak-watch-{}-{}.tsv", + std::process::id(), + OffsetDateTime::now_utc().unix_timestamp_nanos() + )); + let pods = PodList { + items: vec![Pod { + metadata: Metadata { + name: String::from("tidaldb-0"), + uid: String::from("uid-1"), + }, + status: PodStatus { + phase: String::from("Running"), + container_statuses: vec![ContainerStatus { + name: String::from("tidaldb"), + restart_count: 4, + ready: true, + last_state: ContainerState { + terminated: Some(TerminatedState { + reason: String::from("Completed"), + exit_code: 0, + message: String::from("{\"reason\":\"reseed\"}\nnext"), + }), + }, + }], + }, + }], + }; + + assert_eq!( + append_sample(&path, "2026-08-13T02:00:00Z", &pods, "tidaldb").unwrap(), + 1 + ); + let output = std::fs::read_to_string(&path).unwrap(); + std::fs::remove_file(path).unwrap(); + + assert_eq!(output.lines().count(), 2); + assert!(output.contains("tidaldb-0\tuid-1\t4\tRunning\ttrue\tCompleted\t0")); + assert!(output.contains("{\"reason\":\"reseed\"} next")); + } +} diff --git a/tidal-stress/src/soak_eval.rs b/tidal-stress/src/soak_eval.rs index 13374f4..36340c9 100644 --- a/tidal-stress/src/soak_eval.rs +++ b/tidal-stress/src/soak_eval.rs @@ -18,15 +18,20 @@ //! The logic is pure and table-driven-tested so the k8s evaluator container is a //! thin shell around a proven core, not fragile inline `awk`. -use std::collections::BTreeMap; use std::collections::BTreeSet; -/// One night's ledger row: the date (`YYYY-MM-DD`) and whether the soak Job -/// passed its armed SLO gates. +use time::{Date, Duration, Month, OffsetDateTime, format_description::well_known::Rfc3339}; + +/// One night's ledger row: date, load window, and armed-gate result. +/// +/// A trustworthy recovery verdict requires monitor samples immediately +/// outside both window bounds. #[derive(Debug, Clone, PartialEq, Eq)] pub struct NightResult { pub date: String, pub passed: bool, + pub start_timestamp: Option, + pub end_timestamp: Option, } /// One restart-watch sample: the calendar date of the sample, the pod, and its @@ -34,8 +39,15 @@ pub struct NightResult { /// rises until the pod object is recreated). #[derive(Debug, Clone, PartialEq, Eq)] pub struct RestartSample { + /// Full ISO-8601 sample timestamp. `restarts.tsv` is append-only, so this + /// also documents the ordering contract used by restart detection. + pub timestamp: String, pub date: String, pub pod: String, + /// Kubernetes pod UID when the monitor emits the current schema. Legacy + /// rows omit it; those rows still detect counter increases but cannot prove + /// a restart that happened before the first sample of a recreated pod. + pub pod_uid: Option, pub restarts: u64, } @@ -50,10 +62,10 @@ pub struct NightVerdict { /// Parse `ledger.tsv` (the nightly CronJob's append-only verdict log). /// -/// Columns (tab-separated, as the cronjob writes them): -/// `DATE VERD rc=N passed=X p99_ms=X err_frac=X image=...`. The verdict is -/// taken from the `VERD` column (`PASS`/`FAIL`) — the authoritative Job exit -/// translation. Blank lines and a leading header (if any) are skipped. +/// Columns (tab-separated, as the CronJob writes them): +/// `DATE VERD ... start_utc=TS end_utc=TS`. The verdict is taken from the +/// `VERD` column (`PASS`/`FAIL`) and the timestamps delimit the load window. +/// Blank lines and a leading header (if any) are skipped. #[must_use] pub fn parse_ledger(tsv: &str) -> Vec { let mut out = Vec::new(); @@ -67,24 +79,31 @@ pub fn parse_ledger(tsv: &str) -> Vec { continue; } let date = cols[0].trim(); - // A header row or anything whose first cell is not a date is skipped. + // A header row or anything whose first cell is not a real date is skipped. if !is_date(date) { continue; } - let verd = cols[1].trim(); + let field = |name: &str| { + cols.iter() + .skip(2) + .find_map(|column| column.trim().strip_prefix(name)) + .map(str::to_string) + }; out.push(NightResult { date: date.to_string(), - passed: verd.eq_ignore_ascii_case("PASS"), + passed: cols[1].trim().eq_ignore_ascii_case("PASS"), + start_timestamp: field("start_utc="), + end_timestamp: field("end_utc="), }); } out } -/// Parse `restarts.tsv` (the monitor's 5-minute restart-count snapshots). +/// Parse `restarts.tsv` (the monitor's restart-count snapshots). /// -/// Columns: `ts_utc pod restarts phase ready` with an ISO `ts_utc` whose -/// leading `YYYY-MM-DD` is the night key. The header row and malformed rows are -/// skipped. +/// Legacy columns are `ts_utc pod restarts phase ready`; current rows insert +/// the Kubernetes pod UID before `restarts`. Invalid timestamps and malformed +/// rows are skipped. #[must_use] pub fn parse_restarts(tsv: &str) -> Vec { let mut out = Vec::new(); @@ -97,263 +116,511 @@ pub fn parse_restarts(tsv: &str) -> Vec { if cols.len() < 3 { continue; } - let date = cols[0].get(0..10).unwrap_or(""); - if !is_date(date) { + let timestamp = cols[0].trim(); + let date = timestamp.get(0..10).unwrap_or(""); + if !is_date(date) || parse_utc_timestamp(timestamp).is_none() { continue; // header ("ts_utc") or junk } let pod = cols[1].trim(); - let Ok(restarts) = cols[2].trim().parse::() else { - continue; + // A recreated pod starts a fresh counter generation. Legacy rows lack + // the UID but remain useful because any bounded-window counter change + // is still a recovery event. + let (pod_uid, restarts) = if let Ok(restarts) = cols[2].trim().parse::() { + (None, restarts) + } else { + let Some(restarts) = cols.get(3).and_then(|v| v.trim().parse::().ok()) else { + continue; + }; + let uid = cols[2].trim(); + let pod_uid = if uid.is_empty() || uid == "" { + None + } else { + Some(uid.to_string()) + }; + (pod_uid, restarts) }; out.push(RestartSample { + timestamp: timestamp.to_string(), date: date.to_string(), pod: pod.to_string(), + pod_uid, restarts, }); } out } -/// `YYYY-MM-DD` shape check (cheap, no chrono dependency). -fn is_date(s: &str) -> bool { - let b = s.as_bytes(); - b.len() == 10 - && b[4] == b'-' - && b[7] == b'-' - && b[0].is_ascii_digit() - && b[1].is_ascii_digit() - && b[2].is_ascii_digit() - && b[3].is_ascii_digit() - && b[5].is_ascii_digit() - && b[6].is_ascii_digit() - && b[8].is_ascii_digit() - && b[9].is_ascii_digit() +fn parse_date(value: &str) -> Option { + let bytes = value.as_bytes(); + if bytes.len() != 10 || bytes[4] != b'-' || bytes[7] != b'-' { + return None; + } + let year = value.get(0..4)?.parse::().ok()?; + let month = Month::try_from(value.get(5..7)?.parse::().ok()?).ok()?; + let day = value.get(8..10)?.parse::().ok()?; + Date::from_calendar_date(year, month, day).ok() } -/// Whether any pod restarted within night `date`, given the per-pod cumulative -/// restart counts observed up to and including each night. +fn is_date(value: &str) -> bool { + parse_date(value).is_some() +} + +fn parse_utc_timestamp(value: &str) -> Option { + value + .ends_with('Z') + .then(|| OffsetDateTime::parse(value, &Rfc3339).ok()) + .flatten() +} + +fn window_timestamp(value: Option<&str>, date: &str) -> Option { + let timestamp = parse_utc_timestamp(value?)?; + (timestamp.date() == parse_date(date)?).then_some(timestamp) +} + +#[derive(Debug, Default)] +struct RestartEvidence { + dirty: bool, + complete: bool, +} + +/// Maximum tolerated distance between a load boundary and its monitor sample. +/// The watcher runs every minute and the Job waits 70 seconds after load, so +/// two minutes admits ordinary scheduling jitter without accepting a stale +/// watcher as evidence. +const MAX_BOUNDARY_SKEW: Duration = Duration::seconds(120); + +/// Classify restart evidence for exactly one measured load window. /// -/// A restart shows as the cumulative count RISING. For night D and pod P: -/// `restarted = max_count(P, D) > floor`, where `floor` is P's highest count on -/// any night strictly before D (its carried-forward baseline), or — for the -/// first night P is seen — the MINIMUM count that night (so a restart *within* -/// the very first observed night, e.g. 0 → 1, still registers). -fn nights_with_restart(samples: &[RestartSample]) -> BTreeSet { - // pod -> (date -> (min, max)) of that date's samples. - let mut by_pod: BTreeMap<&str, BTreeMap<&str, (u64, u64)>> = BTreeMap::new(); - for s in samples { - let e = by_pod - .entry(&s.pod) - .or_default() - .entry(&s.date) - .or_insert((s.restarts, s.restarts)); - e.0 = e.0.min(s.restarts); - e.1 = e.1.max(s.restarts); +/// Every expected pod needs a sample no more than two minutes before the load +/// starts and another no more than two minutes after it ends. Counter changes +/// or pod-UID changes between those bracketing samples make the night dirty. +/// This avoids both prior false positives from unrelated daytime restarts and +/// false greens from two arbitrary samples somewhere on the same calendar day. +fn restart_evidence( + night: &NightResult, + samples: &[RestartSample], + expected_pods: usize, +) -> RestartEvidence { + let (Some(start), Some(end)) = ( + window_timestamp(night.start_timestamp.as_deref(), &night.date), + window_timestamp(night.end_timestamp.as_deref(), &night.date), + ) else { + return RestartEvidence::default(); + }; + if end <= start { + return RestartEvidence::default(); } - let mut dirty = BTreeSet::new(); - for dates in by_pod.values() { - let mut prev_max: Option = None; - for (date, &(min, max)) in dates { - let floor = prev_max.unwrap_or(min); - if max > floor { - dirty.insert((*date).to_string()); + + let mut dated_samples: Vec<_> = samples + .iter() + .filter(|sample| sample.date == night.date) + .filter_map(|sample| parse_utc_timestamp(&sample.timestamp).map(|at| (sample, at))) + .collect(); + dated_samples.sort_by_key(|(_, sampled_at)| *sampled_at); + + let pods: BTreeSet<&str> = dated_samples + .iter() + .map(|(sample, _)| sample.pod.as_str()) + .collect(); + let mut evidence = RestartEvidence { + dirty: false, + complete: expected_pods > 0 && pods.len() == expected_pods, + }; + + for pod in pods { + let pod_samples: Vec<_> = dated_samples + .iter() + .copied() + .filter(|(sample, _)| sample.pod == pod) + .collect(); + let before = pod_samples + .iter() + .rposition(|(_, sampled_at)| *sampled_at <= start); + let after = pod_samples + .iter() + .position(|(_, sampled_at)| *sampled_at >= end); + let (Some(before), Some(after)) = (before, after) else { + evidence.complete = false; + continue; + }; + let before_at = pod_samples[before].1; + let after_at = pod_samples[after].1; + if start - before_at > MAX_BOUNDARY_SKEW || after_at - end > MAX_BOUNDARY_SKEW { + evidence.complete = false; + continue; + } + + for pair in pod_samples[before..=after].windows(2) { + let previous = pair[0].0; + let current = pair[1].0; + let uid_changed = matches!( + (previous.pod_uid.as_deref(), current.pod_uid.as_deref()), + (Some(previous_uid), Some(current_uid)) if previous_uid != current_uid + ); + if uid_changed || previous.restarts != current.restarts { + evidence.dirty = true; } - prev_max = Some(prev_max.map_or(max, |p| p.max(max))); } } - dirty + + evidence } -/// Join the ledger and the restart samples into a per-night verdict list, in -/// ledger order. A night is green iff it passed AND no pod restarted in its -/// window. +/// Join the ledger and restart samples into per-night verdicts, in ledger order. +/// +/// A night is green only when the load passed and complete bracketing evidence +/// proves that every expected pod remained in the same generation and restart +/// count throughout the measured window. #[must_use] -pub fn evaluate(nights: &[NightResult], samples: &[RestartSample]) -> Vec { - let dirty = nights_with_restart(samples); +pub fn evaluate( + nights: &[NightResult], + samples: &[RestartSample], + expected_pods: usize, +) -> Vec { nights .iter() - .map(|n| { - let restarted = dirty.contains(&n.date); - let (green, reason) = match (n.passed, restarted) { - (true, false) => (true, String::from("pass; no under-load restart")), - (false, false) => (false, String::from("soak SLO gate breach (Job FAIL)")), - (true, true) => ( - false, - String::from( - "under-load pod restart in window (Job passed but streak-breaking)", - ), - ), - (false, true) => (false, String::from("soak FAIL and under-load pod restart")), - }; + .map(|night| { + let evidence = restart_evidence(night, samples, expected_pods); + let mut reasons = Vec::with_capacity(3); + if !night.passed { + reasons.push("soak SLO gate breach (Job FAIL)"); + } + if evidence.dirty { + reasons.push("under-load pod restart in window"); + } + if !evidence.complete { + reasons.push( + "restart evidence missing or stale (need every expected pod at both load boundaries)", + ); + } + + let green = reasons.is_empty(); NightVerdict { - date: n.date.clone(), + date: night.date.clone(), green, - reason, + reason: if green { + String::from("pass; complete restart evidence; no under-load restart") + } else { + reasons.join("; ") + }, } }) .collect() } -/// The trailing run of consecutive green nights (the GA streak). Any non-green -/// night resets it; only the most recent unbroken tail counts. +fn dates_are_consecutive(previous: &str, current: &str) -> bool { + parse_date(previous).and_then(Date::next_day) == parse_date(current) +} + +fn advance_streak(previous_date: Option<&str>, current: &NightVerdict, run: usize) -> usize { + if !current.green { + return 0; + } + if previous_date.is_some_and(|previous| dates_are_consecutive(previous, ¤t.date)) { + run + 1 + } else { + 1 + } +} + +/// The trailing run of unique, consecutive calendar dates with green verdicts. +/// A failure, duplicate date, out-of-order row, or date gap starts a new run. #[must_use] pub fn current_streak(verdicts: &[NightVerdict]) -> usize { - verdicts.iter().rev().take_while(|v| v.green).count() + let mut run = 0; + let mut previous_date = None; + for verdict in verdicts { + run = advance_streak(previous_date, verdict, run); + previous_date = Some(verdict.date.as_str()); + } + run } /// Render `streak.tsv`: one row per night (`date green streak_after reason`) /// plus a trailing `# streak=/` summary line for an at-a-glance read. #[must_use] pub fn render_streak_tsv(verdicts: &[NightVerdict], target: usize) -> String { - let mut s = String::from("date\tgreen\tstreak\treason\n"); + let mut output = String::from("date\tgreen\tstreak\treason\n"); let mut run = 0usize; - for v in verdicts { - run = if v.green { run + 1 } else { 0 }; - s.push_str(&format!("{}\t{}\t{}\t{}\n", v.date, v.green, run, v.reason)); + let mut previous_date = None; + for verdict in verdicts { + run = advance_streak(previous_date, verdict, run); + output.push_str(&format!( + "{}\t{}\t{}\t{}\n", + verdict.date, verdict.green, run, verdict.reason + )); + previous_date = Some(verdict.date.as_str()); } - let streak = current_streak(verdicts); - s.push_str(&format!("# streak={streak}/{target}\n")); - s + output.push_str(&format!("# streak={run}/{target}\n")); + output } #[cfg(test)] mod tests { use super::*; + const EXPECTED_PODS: usize = 3; + fn pass_nights(n: usize) -> Vec { (1..=n) - .map(|i| NightResult { - date: format!("2026-06-{i:02}"), - passed: true, + .map(|i| { + let date = format!("2026-06-{i:02}"); + NightResult { + start_timestamp: Some(format!("{date}T02:00:00Z")), + end_timestamp: Some(format!("{date}T03:00:00Z")), + date, + passed: true, + } }) .collect() } - /// A constant-restart-count sample stream (no restarts) for every night — - /// pods sitting at a steady `restartCount` (e.g. 2) all window. + fn sample( + date: &str, + time: &str, + pod: &str, + pod_uid: Option<&str>, + restarts: u64, + ) -> RestartSample { + RestartSample { + timestamp: format!("{date}T{time}"), + date: date.to_string(), + pod: pod.to_string(), + pod_uid: pod_uid.map(str::to_string), + restarts, + } + } + + /// A constant-restart-count stream that tightly brackets every load window. fn steady_samples(nights: usize, count: u64) -> Vec { - let mut v = Vec::new(); + let mut samples = Vec::new(); for i in 1..=nights { + let date = format!("2026-06-{i:02}"); for pod in ["tidaldb-0", "tidaldb-1", "tidaldb-2"] { - v.push(RestartSample { - date: format!("2026-06-{i:02}"), - pod: pod.to_string(), - restarts: count, - }); + samples.push(sample(&date, "01:59:00Z", pod, None, count)); + samples.push(sample(&date, "03:01:00Z", pod, None, count)); } } - v + samples } #[test] - fn thirty_clean_nights_make_a_full_streak() { + fn thirty_clean_consecutive_nights_make_a_full_streak() { let nights = pass_nights(30); let samples = steady_samples(30, 2); - let verdicts = evaluate(&nights, &samples); - assert!(verdicts.iter().all(|v| v.green), "all 30 must be green"); - assert_eq!(current_streak(&verdicts), 30, "30 clean nights → streak 30"); + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); + assert!(verdicts.iter().all(|verdict| verdict.green)); + assert_eq!(current_streak(&verdicts), 30); } #[test] fn an_under_load_restart_breaks_the_streak_even_when_the_job_passed() { let nights = pass_nights(30); let mut samples = steady_samples(30, 2); - // Night 15: tidaldb-1's cumulative restartCount rises 2 → 3 within the - // window (an under-load restart) while the soak Job still PASSED. - samples.push(RestartSample { - date: "2026-06-15".into(), - pod: "tidaldb-1".into(), - restarts: 3, - }); - let verdicts = evaluate(&nights, &samples); - let n15 = verdicts.iter().find(|v| v.date == "2026-06-15").unwrap(); - assert!( - !n15.green, - "a night with an under-load restart is NOT green" - ); - assert!(n15.reason.contains("restart"), "reason names the restart"); - // The streak is only the clean tail after night 15: nights 16..=30 = 15. - assert_eq!( - current_streak(&verdicts), - 15, - "streak resets at the restart night" - ); + samples.push(sample("2026-06-15", "02:30:00Z", "tidaldb-1", None, 3)); + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); + let n15 = verdicts + .iter() + .find(|verdict| verdict.date == "2026-06-15") + .expect("night 15 exists"); + assert!(!n15.green); + assert!(n15.reason.contains("restart")); + assert_eq!(current_streak(&verdicts), 15); } #[test] fn a_gate_breach_night_resets_the_streak() { let mut nights = pass_nights(30); - nights[19].passed = false; // night 20 breached an SLO gate (Job FAIL) + nights[19].passed = false; let samples = steady_samples(30, 2); - let verdicts = evaluate(&nights, &samples); - let n20 = verdicts.iter().find(|v| v.date == "2026-06-20").unwrap(); - assert!(!n20.green, "a gate-breach night is not green"); - assert!(n20.reason.contains("gate"), "reason names the gate breach"); - assert_eq!( - current_streak(&verdicts), - 10, - "tail after night 20 = nights 21..=30" - ); + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); + let n20 = &verdicts[19]; + assert!(!n20.green); + assert!(n20.reason.contains("gate")); + assert_eq!(current_streak(&verdicts), 10); } #[test] fn a_restart_within_the_first_observed_night_registers() { - // No carried-forward baseline: a 0 → 1 rise on the first night must still - // count as a restart (min < max that night). let nights = pass_nights(1); let samples = vec![ - RestartSample { - date: "2026-06-01".into(), - pod: "tidaldb-0".into(), - restarts: 0, - }, - RestartSample { - date: "2026-06-01".into(), - pod: "tidaldb-0".into(), - restarts: 1, - }, + sample("2026-06-01", "01:59:00Z", "tidaldb-0", None, 0), + sample("2026-06-01", "03:01:00Z", "tidaldb-0", None, 1), ]; - let verdicts = evaluate(&nights, &samples); - assert!( - !verdicts[0].green, - "0→1 within the first night is a restart" + let verdicts = evaluate(&nights, &samples, 1); + assert!(!verdicts[0].green); + } + + #[test] + fn pod_recreated_during_window_already_at_one_restart_registers() { + let nights = pass_nights(1); + let samples = vec![ + sample( + "2026-06-01", + "01:59:00Z", + "tidaldb-0", + Some("pod-before"), + 0, + ), + sample("2026-06-01", "03:01:00Z", "tidaldb-0", Some("pod-after"), 1), + ]; + let verdicts = evaluate(&nights, &samples, 1); + assert!(!verdicts[0].green); + } + + #[test] + fn pod_recreation_with_zero_container_restarts_breaks_the_night() { + let nights = pass_nights(1); + let samples = vec![ + sample( + "2026-06-01", + "01:59:00Z", + "tidaldb-0", + Some("pod-before"), + 0, + ), + sample("2026-06-01", "03:01:00Z", "tidaldb-0", Some("pod-after"), 0), + ]; + let verdicts = evaluate(&nights, &samples, 1); + assert!(!verdicts[0].green); + } + + #[test] + fn missing_restart_evidence_fails_closed() { + let nights = pass_nights(1); + let samples = vec![sample( + "2026-06-01", + "03:01:00Z", + "tidaldb-0", + Some("pod-uid"), + 0, + )]; + let verdicts = evaluate(&nights, &samples, 1); + assert!(!verdicts[0].green); + assert!(verdicts[0].reason.contains("evidence")); + } + + #[test] + fn stale_boundary_samples_fail_closed() { + let nights = pass_nights(1); + let samples = vec![ + sample("2026-06-01", "01:55:00Z", "tidaldb-0", None, 0), + sample("2026-06-01", "03:05:00Z", "tidaldb-0", None, 0), + ]; + let verdicts = evaluate(&nights, &samples, 1); + assert!(!verdicts[0].green); + assert!(verdicts[0].reason.contains("stale")); + } + + #[test] + fn a_legacy_counter_reset_inside_the_window_is_a_recovery_event() { + let nights = pass_nights(2); + let mut samples = steady_samples(1, 20); + samples.extend([ + sample("2026-06-02", "01:59:00Z", "tidaldb-0", None, 20), + sample("2026-06-02", "02:30:00Z", "tidaldb-0", None, 0), + sample("2026-06-02", "03:01:00Z", "tidaldb-0", None, 1), + ]); + for pod in ["tidaldb-1", "tidaldb-2"] { + samples.push(sample("2026-06-02", "01:59:00Z", pod, None, 20)); + samples.push(sample("2026-06-02", "03:01:00Z", pod, None, 20)); + } + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); + assert!(!verdicts[1].green); + } + + #[test] + fn a_restart_before_the_load_window_does_not_dirty_the_night() { + let nights = pass_nights(1); + let samples = vec![ + sample("2026-06-01", "00:01:00Z", "tidaldb-0", None, 0), + sample("2026-06-01", "00:02:00Z", "tidaldb-0", None, 1), + sample("2026-06-01", "01:59:00Z", "tidaldb-0", None, 1), + sample("2026-06-01", "03:01:00Z", "tidaldb-0", None, 1), + ]; + let verdicts = evaluate(&nights, &samples, 1); + assert!(verdicts[0].green); + } + + #[test] + fn a_future_extra_pod_does_not_contaminate_prior_night_coverage() { + let nights = pass_nights(2); + let mut samples = steady_samples(2, 0); + samples.push(sample("2026-06-02", "01:59:00Z", "tidaldb-3", None, 0)); + samples.push(sample("2026-06-02", "03:01:00Z", "tidaldb-3", None, 0)); + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); + assert!(verdicts[0].green); + assert!(!verdicts[1].green); + } + + #[test] + fn missing_load_window_timestamps_fail_closed() { + let mut nights = pass_nights(1); + nights[0].start_timestamp = None; + let verdicts = evaluate(&nights, &steady_samples(1, 0), EXPECTED_PODS); + assert!(!verdicts[0].green); + } + + #[test] + fn date_gaps_and_duplicates_cannot_inflate_the_streak() { + let green = |date: &str| NightVerdict { + date: date.to_string(), + green: true, + reason: String::from("pass"), + }; + let gap = vec![ + green("2026-06-01"), + green("2026-06-02"), + green("2026-06-04"), + ]; + let duplicate = vec![ + green("2026-06-01"), + green("2026-06-02"), + green("2026-06-02"), + ]; + assert_eq!(current_streak(&gap), 1); + assert_eq!(current_streak(&duplicate), 1); + } + + #[test] + fn parse_ledger_reads_window_fields_and_rejects_impossible_dates() { + let tsv = "\ +2026-06-01\tPASS\tsoak_rc=0\tstart_utc=2026-06-01T02:00:00Z\tend_utc=2026-06-01T03:00:00Z\n\ +2026-06-02\tFAIL\tsoak_rc=1\tstart_utc=2026-06-02T02:00:00Z\tend_utc=2026-06-02T03:00:00Z\n\ +2026-02-30\tPASS\tjunk\n"; + let rows = parse_ledger(tsv); + assert_eq!(rows.len(), 2); + assert!(rows[0].passed); + assert!(!rows[1].passed); + assert_eq!( + rows[0].start_timestamp.as_deref(), + Some("2026-06-01T02:00:00Z") ); } #[test] - fn parse_ledger_skips_headers_and_reads_the_verdict() { + fn parse_restarts_accepts_legacy_and_uid_rows() { let tsv = "\ -2026-06-01\tPASS\trc=0\tpassed=true\tp99_ms=9.3\terr_frac=0.0001\timage=m12\n\ -2026-06-02\tFAIL\trc=1\tpassed=false\tp99_ms=210\terr_frac=0.04\timage=m12\n\ -not-a-date\tPASS\tjunk\n"; - let rows = parse_ledger(tsv); - assert_eq!(rows.len(), 2, "the junk/header row is skipped"); - assert!(rows[0].passed); - assert!(!rows[1].passed); - } - - #[test] - fn parse_restarts_skips_header_and_keys_by_date() { - let tsv = "\ -ts_utc\tpod\trestarts\tphase\tready\n\ +ts_utc\tpod\tuid\trestarts\tphase\tready\n\ 2026-06-01T02:05:00Z\ttidaldb-0\t2\tRunning\ttrue\n\ -2026-06-01T02:10:00Z\ttidaldb-1\t2\tRunning\ttrue\n"; +2026-06-01T02:10:00Z\ttidaldb-1\tuid-1\t3\tRunning\ttrue\n\ +not-a-timestamp\ttidaldb-2\t4\tRunning\ttrue\n"; let rows = parse_restarts(tsv); - assert_eq!(rows.len(), 2, "header skipped, two data rows parsed"); + assert_eq!(rows.len(), 2); assert_eq!(rows[0].date, "2026-06-01"); assert_eq!(rows[0].restarts, 2); + assert_eq!(rows[0].pod_uid, None); + assert_eq!(rows[1].restarts, 3); + assert_eq!(rows[1].pod_uid.as_deref(), Some("uid-1")); } #[test] fn render_streak_tsv_carries_running_count_and_summary() { let nights = pass_nights(3); let samples = steady_samples(3, 0); - let verdicts = evaluate(&nights, &samples); + let verdicts = evaluate(&nights, &samples, EXPECTED_PODS); let tsv = render_streak_tsv(&verdicts, 30); - assert!(tsv.contains("date\tgreen\tstreak\treason"), "has a header"); - assert!( - tsv.trim_end().ends_with("# streak=3/30"), - "summary line present: {tsv}" - ); + assert!(tsv.contains("date\tgreen\tstreak\treason")); + assert!(tsv.trim_end().ends_with("# streak=3/30")); } } diff --git a/tidal/tests/m8p3_reconcile_production.rs b/tidal/tests/m8p3_reconcile_production.rs index 760eece..f17db95 100644 --- a/tidal/tests/m8p3_reconcile_production.rs +++ b/tidal/tests/m8p3_reconcile_production.rs @@ -3,13 +3,25 @@ //! //! These tests close the "CRDT engine is never invoked in production" gap: they //! drive the LIVE production entry points on real `TidalDb` instances (not the -//! `ReconciliationEngine` directly), partition two nodes, write divergently on -//! each side, exchange snapshots, reconcile, and assert the merged signal decay -//! score AND the windowed (`AllTime`) bucket count both survive end to end. +//! `ReconciliationEngine` directly), diverge two nodes, exchange snapshots, +//! reconcile, and assert what the heal actually guarantees. //! -//! Each `TidalDb` runs as a distinct shard, so `take_crdt_snapshot` attributes -//! its contributions to a distinct node in the CRDT — exactly the multi-node -//! divergence the reconciliation engine exists to merge. +//! # The contract these tests hold (read before changing an expectation) +//! +//! `take_crdt_snapshot` keys every signal contribution to ONE canonical +//! contributor ([`ShardId::SINGLE`]), not to the local shard. Signals are +//! replicated from a single writer through the WAL relay, so each node's hot +//! accumulator ALREADY contains the other nodes' relayed events. Attributing it +//! per-node would fabricate N disjoint contributions for one logical stream, +//! which `CrdtSignalState::merge` then sums - double-counting every replicated +//! event on every reconcile (the creep a UAT caught). See the rationale on +//! `TidalDb::take_crdt_snapshot`. +//! +//! With one contributor the merge is therefore deterministic convergence, not +//! addition: the decay score is last-writer-wins on `(last_update_ns, score)`, +//! and the windowed bucket count is the PN-counter per-node max. Both nodes +//! converge on the MORE COMPLETE accumulator and stay there under repeated +//! exchange. A test that asserts `3 + 5 == 8` here is asserting the bug. #![allow(clippy::unwrap_used, clippy::float_cmp)] use std::time::Duration; @@ -66,19 +78,18 @@ fn node(shard: ShardId) -> TidalDb { .expect("ephemeral node opens") } -/// Two partitioned nodes write divergent signals to the SAME entity, exchange -/// snapshots, and reconcile. After healing, BOTH nodes report the merged decay -/// score (sum of both contributions) AND the merged windowed count (sum of both -/// event counts) — the per-node bucket count round-trips, not dropping to 0. +/// Two diverged nodes exchange snapshots and reconcile. Both converge on the +/// more complete accumulator, the windowed count survives the round trip (it +/// does not drop to 0 - the original finding), and neither side inflates past +/// the true event total. #[test] -fn two_node_partition_heals_signals_and_windowed_counts() { +fn two_node_divergence_converges_on_the_more_complete_accumulator() { let node_a = node(ShardId(0)); let node_b = node(ShardId(1)); let item = EntityId::new(42); - // ── Partition: each node accumulates independent events for `item`. ── - // Node A: 3 views. Node B: 5 views. Same fixed recent timestamp so decay is - // negligible and the arithmetic is exact-ish. + // Node A saw 3 views, node B saw 5 of the same logical stream (B is the + // more complete replica). Same fixed timestamp so decay is negligible. let ts = Timestamp::now(); for _ in 0..3 { node_a.signal("view", item, 1.0, ts).unwrap(); @@ -87,7 +98,6 @@ fn two_node_partition_heals_signals_and_windowed_counts() { node_b.signal("view", item, 1.0, ts).unwrap(); } - // Pre-reconcile each node sees ONLY its own events. let a_count_before = node_a .read_windowed_count(item, "view", Window::AllTime) .unwrap(); @@ -97,10 +107,6 @@ fn two_node_partition_heals_signals_and_windowed_counts() { assert_eq!(a_count_before, 3, "node A sees only its 3 events pre-heal"); assert_eq!(b_count_before, 5, "node B sees only its 5 events pre-heal"); - let a_score_before = node_a - .read_decay_score(item, "view", 0) - .unwrap() - .unwrap_or(0.0); let b_score_before = node_b .read_decay_score(item, "view", 0) .unwrap() @@ -115,26 +121,28 @@ fn two_node_partition_heals_signals_and_windowed_counts() { assert!(ops_a >= 1, "reconcile must apply at least the signal merge"); assert!(ops_b >= 1); - // ── Post-heal: both nodes converge to the merged truth. ── - // Windowed count: 3 + 5 = 8 on BOTH nodes (finding 6 — the bucket count is - // carried through from_node_contribution and survives merge, not 0). let a_count_after = node_a .read_windowed_count(item, "view", Window::AllTime) .unwrap(); let b_count_after = node_b .read_windowed_count(item, "view", Window::AllTime) .unwrap(); + + // Convergence is the property that matters: both sides agree. assert_eq!( - a_count_after, 8, - "node A windowed count must be merged total (3+5), not its local 3 or 0" + a_count_after, b_count_after, + "both nodes must converge on one windowed count after the exchange" ); + // ... on the more complete accumulator (PN-counter per-node max of 3 and 5), + // never 0 (the count survives the snapshot round trip) and never 8 (summing + // one logical stream twice is the double-count bug). assert_eq!( - b_count_after, 8, - "node B windowed count must be merged total (3+5), not its local 5 or 0" + a_count_after, 5, + "converged count is the more complete accumulator, not a sum" ); - // Decay score: the merged score is the sum of both nodes' contributions. - let expected = a_score_before + b_score_before; + // Decay score: LWW on (last_update_ns, score) with a single contributor, so + // both sides hold node B's larger accumulator. let a_score_after = node_a .read_decay_score(item, "view", 0) .unwrap() @@ -145,27 +153,78 @@ fn two_node_partition_heals_signals_and_windowed_counts() { .unwrap_or(0.0); // Tolerance: decay over the few-ms reconcile window is negligible but nonzero. assert!( - (a_score_after - expected).abs() < 1e-3, - "node A merged score {a_score_after} should be ~sum {expected}" + (a_score_after - b_score_after).abs() < 1e-3, + "both nodes must converge on one score: A {a_score_after} vs B {b_score_after}" ); assert!( - (b_score_after - expected).abs() < 1e-3, - "node B merged score {b_score_after} should be ~sum {expected}" + (a_score_after - b_score_before).abs() < 1e-3, + "converged score {a_score_after} should be node B's {b_score_before}" ); } -/// Reconciliation against a remote snapshot that this node has already fully -/// absorbed (its events are a subset of what the local side counted) is a no-op -/// for the windowed count — it never shrinks a locally-durable count nor -/// double-counts an already-merged remote contribution. +/// Repeated exchange between already-converged nodes changes nothing. /// -/// This is the idempotency that the heal path actually guarantees: re-merging a -/// remote snapshot whose every contribution is `<=` what the local node already -/// holds leaves the count unchanged. (Re-*snapshotting* after a merge and -/// re-merging is NOT idempotent — `apply_crdt_state` force-sets a single warm -/// counter and loses per-node provenance — so the heal is a one-shot exchange, -/// not a repeatedly-applied stream; we assert the property that genuinely -/// holds.) +/// This is the regression guard for the creep that per-node attribution caused: +/// with the whole accumulator attributed per shard, every reconcile re-summed +/// the same relayed events and the count grew without any new signal being +/// written. Under the canonical-contributor keying the second and third rounds +/// are exact no-ops. +#[test] +fn repeated_reconcile_of_converged_nodes_does_not_creep() { + let node_a = node(ShardId(0)); + let node_b = node(ShardId(1)); + let item = EntityId::new(99); + + let ts = Timestamp::now(); + for _ in 0..4 { + node_a.signal("view", item, 1.0, ts).unwrap(); + } + for _ in 0..4 { + node_b.signal("view", item, 1.0, ts).unwrap(); + } + + // First exchange converges the pair. + let snap_b = node_b.take_crdt_snapshot().unwrap(); + node_a.reconcile_with(&snap_b).unwrap(); + let converged = node_a + .read_windowed_count(item, "view", Window::AllTime) + .unwrap(); + assert_eq!(converged, 4, "converged on the shared 4-event accumulator"); + + // Two more rounds with FRESH snapshots taken after the merge - the shape + // anti-entropy would actually run - must not move the count or the score. + let score_after_first = node_a + .read_decay_score(item, "view", 0) + .unwrap() + .unwrap_or(0.0); + for round in 1..=2 { + let fresh_a = node_a.take_crdt_snapshot().unwrap(); + let fresh_b = node_b.take_crdt_snapshot().unwrap(); + node_b.reconcile_with(&fresh_a).unwrap(); + node_a.reconcile_with(&fresh_b).unwrap(); + + let count = node_a + .read_windowed_count(item, "view", Window::AllTime) + .unwrap(); + assert_eq!( + count, converged, + "round {round}: repeated reconcile must not inflate the count" + ); + let score = node_a + .read_decay_score(item, "view", 0) + .unwrap() + .unwrap_or(0.0); + assert!( + (score - score_after_first).abs() < 1e-3, + "round {round}: repeated reconcile must not inflate the score \ + ({score} vs {score_after_first})" + ); + } +} + +/// Reconciliation against a remote snapshot the local node already covers is a +/// no-op for the windowed count: it never shrinks a locally-durable count, and +/// re-applying the same snapshot never grows one. #[test] fn reconcile_with_already_absorbed_remote_is_noop_for_count() { let node_a = node(ShardId(0)); @@ -180,30 +239,28 @@ fn reconcile_with_already_absorbed_remote_is_noop_for_count() { node_b.signal("view", item, 1.0, ts).unwrap(); } - // Snapshot B (node 1, 4 events) BEFORE any merge. + // Snapshot B (4 events) BEFORE any merge. A is the more complete side. let snap_b = node_b.take_crdt_snapshot().unwrap(); - // First reconcile: 6 (node 0) + 4 (node 1) = 10. + // Merging a strictly smaller remote accumulator must not shrink A. node_a.reconcile_with(&snap_b).unwrap(); let after_first = node_a .read_windowed_count(item, "view", Window::AllTime) .unwrap(); - assert_eq!(after_first, 10, "first reconcile merges to 10"); + assert_eq!( + after_first, 6, + "merging a smaller remote accumulator keeps the local 6" + ); - // Re-merge the SAME pre-merge snapshot B. The merged per-node total - // (node 0 = 10 absorbed, node 1 = 4) is 14 > 10, so the warm tier would - // grow — confirming the heal is intentionally one-shot. We instead assert - // the never-shrink guarantee by re-merging a snapshot whose node-1 - // contribution (4) the local count (10) already covers: applying snap_b's - // node-1 bucket of 4 against a local node-1 bucket that is now part of the - // 10 must not drop below 10. + // Re-merging the same snapshot is idempotent in both directions: no shrink + // below 6 and no growth toward 6 + 4. node_a.reconcile_with(&snap_b).unwrap(); let after_second = node_a .read_windowed_count(item, "view", Window::AllTime) .unwrap(); - assert!( - after_second >= 10, - "re-merge must never shrink the locally-durable count below 10 (got {after_second})" + assert_eq!( + after_second, 6, + "re-merge must neither shrink nor inflate the locally-durable count" ); } diff --git a/tidalctl/Cargo.toml b/tidalctl/Cargo.toml index 6eb7b35..67d890f 100644 --- a/tidalctl/Cargo.toml +++ b/tidalctl/Cargo.toml @@ -2,7 +2,11 @@ name = "tidalctl" version = "0.1.0" edition.workspace = true -rust-version.workspace = true +# Higher than the workspace floor: aws-config -> aws-types 1.3.16 declares +# rustc 1.91.1, and resolution fails workspace-wide below it. Declared here so +# the requirement is visible where it originates rather than only in a lockfile +# error. +rust-version = "1.91.1" description = "Command-line inspector for embedded tidalDB instances" license.workspace = true