fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|---|---|---|
| .. | ||
| phase-1.md | ||
| phase-2.md | ||
| phase-3.md | ||
| phase-4.md | ||
| phase-5.md | ||
| phase-6.md | ||
| README.md | ||
Milestone 6 · Full Surface Coverage (✅ COMPLETE 2026-02-23)
Milestone spec, thesis, UAT scenario, and per-phase acceptance criteria: ROADMAP · Milestone 6.
These are backfilled records, not the original plans. M6 shipped in the squashed commit
213b8ef(2026-02-23, "complete M6-M7 + Enterprise Readiness milestones") before this project kept per-milestone planning directories. The six phase docs here were written after the fact (2026-08-30) from the ROADMAP acceptance criteria, the shipped source tree, and the M6 test suite. Every claim names the artifact that proves it, and where the shipped surface diverges from the ROADMAP text the phase doc says so instead of restating the plan.
What the milestone proves
Every use case in USE_CASES.md, every sort mode, every filter, and every feedback loop resolves inside one query engine — cohort-scoped trending, social-graph scoping and collaborative filtering, the full sort surface, collections and saved searches, scoped SEARCH plus autocomplete, and notification capping — with no application-side ranking logic left over.
Phases
| Phase | Name | Record | Primary test |
|---|---|---|---|
| m6p1 | Cohort Engine + Cohort-Scoped Trending | phase-1.md | tidal/tests/m6_cohort.rs (12) |
| m6p2 | Social Graph + Collaborative Filtering | phase-2.md | tidal/tests/m6_social.rs (8) |
| m6p3 | Full Sort Modes + Live Content + Engagement Filters | phase-3.md | tidal/tests/m6p3_sorts.rs (9), m6p3_filters.rs (3), m6p3_edge_cases.rs (3) |
| m6p4 | Collections + Watch History + Saved Searches | phase-4.md | tidal/tests/m6p4_collections.rs (10) |
| m6p5 | Query Composition + SUGGEST Autocomplete | phase-5.md | tidal/tests/m6p5_scope.rs (13) |
| m6p6 | Notification Capping + Adaptive Preferences + M6 UAT | phase-6.md | tidal/tests/m6_uat.rs (9), m6p6_creator_profile.rs (6) |
Test counts are #[test] functions in the working tree today; the ROADMAP status
rows record the smaller counts these files had at close (e.g. m6p1 closed at
"9 m6_cohort"). The suites grew with later hardening — most visibly
tidal/tests/m6_crash_surfaces.rs (4) and tidal/tests/m7_crash_m6.rs, added by
M7 to crash-fence the state M6 introduced.
Shipped surface (verified against the working tree)
| Concern | Artifact |
|---|---|
| Cohort definitions, predicates | tidal/src/cohort/types.rs |
| Cohort membership resolution | tidal/src/cohort/resolver.rs |
| Per-cohort signal aggregation | tidal/src/cohort/ledger.rs |
| Cohort checkpoint/restore | tidal/src/cohort/checkpoint.rs |
| Cohort db API | tidal/src/db/cohorts.rs |
| Social-graph filter | tidal/src/query/executor/social_filter.rs |
| Co-engagement / collaborative filtering | tidal/src/entities/co_engagement.rs |
| Collections | tidal/src/entities/collection.rs, tidal/src/db/collections.rs |
| Saved searches | tidal/src/session/saved_search.rs |
| SUGGEST autocomplete | tidal/src/query/suggest.rs |
| Notification capping | tidal/src/db/notification_tracker.rs |
| Adaptive preference learning rate | tidal/src/entities/preference.rs |
| Sort surface | tidal/src/ranking/profile.rs (Sort), tidal/src/ranking/executor/tests/sort_tests.rs |
| Benchmarks added by M6 | tidal/benches/social.rs, tidal/benches/sort.rs |
All 62 M6 acceptance criteria in ROADMAP · Milestone 6 are
marked [x]. Recorded at close: 1,082 total (835 lib + 247 integration), 9
m6_uat passing; re-verified 2026-02-24 at 1,206 lib + 70 M6 integration tests.
Divergences found while writing these records
Three ROADMAP criteria describe behaviour the shipped code deliberately does not have. Each is documented in the source itself, and each is recorded in the relevant phase doc rather than smoothed over here:
- Co-engagement eviction is minimum-weight, not LRU (phase-2).
- The
relatedblend is additive with a 0.3 co-engagement term, not the convex combination0.6/0.3/0.1(phase-2). - Per-user preference
update_countswere documented as unpersisted and "deferred to M7"; they are in fact persisted now, and by M12, not M7 (phase-6).