fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
73 lines
4.2 KiB
Markdown
73 lines
4.2 KiB
Markdown
# Milestone 6 · Full Surface Coverage (✅ COMPLETE 2026-02-23)
|
|
|
|
Milestone spec, thesis, UAT scenario, and per-phase acceptance criteria:
|
|
[ROADMAP · Milestone 6](../ROADMAP.md).
|
|
|
|
> **These are backfilled records, not the original plans.** M6 shipped in the
|
|
> squashed commit `213b8ef` (2026-02-23, "complete M6-M7 + Enterprise Readiness
|
|
> milestones") before this project kept per-milestone planning directories. The
|
|
> six phase docs here were written after the fact (2026-08-30) from the ROADMAP
|
|
> acceptance criteria, the shipped source tree, and the M6 test suite. Every
|
|
> claim names the artifact that proves it, and where the shipped surface diverges
|
|
> from the ROADMAP text the phase doc says so instead of restating the plan.
|
|
|
|
## What the milestone proves
|
|
|
|
Every use case in [USE_CASES.md](../../../USE_CASES.md), every sort mode, every
|
|
filter, and every feedback loop resolves inside one query engine — cohort-scoped
|
|
trending, social-graph scoping and collaborative filtering, the full sort
|
|
surface, collections and saved searches, scoped SEARCH plus autocomplete, and
|
|
notification capping — with no application-side ranking logic left over.
|
|
|
|
## Phases
|
|
|
|
| Phase | Name | Record | Primary test |
|
|
|-------|------|--------|--------------|
|
|
| m6p1 | Cohort Engine + Cohort-Scoped Trending | [phase-1.md](phase-1.md) | `tidal/tests/m6_cohort.rs` (12) |
|
|
| m6p2 | Social Graph + Collaborative Filtering | [phase-2.md](phase-2.md) | `tidal/tests/m6_social.rs` (8) |
|
|
| m6p3 | Full Sort Modes + Live Content + Engagement Filters | [phase-3.md](phase-3.md) | `tidal/tests/m6p3_sorts.rs` (9), `m6p3_filters.rs` (3), `m6p3_edge_cases.rs` (3) |
|
|
| m6p4 | Collections + Watch History + Saved Searches | [phase-4.md](phase-4.md) | `tidal/tests/m6p4_collections.rs` (10) |
|
|
| m6p5 | Query Composition + SUGGEST Autocomplete | [phase-5.md](phase-5.md) | `tidal/tests/m6p5_scope.rs` (13) |
|
|
| m6p6 | Notification Capping + Adaptive Preferences + M6 UAT | [phase-6.md](phase-6.md) | `tidal/tests/m6_uat.rs` (9), `m6p6_creator_profile.rs` (6) |
|
|
|
|
Test counts are `#[test]` functions in the working tree today; the ROADMAP status
|
|
rows record the smaller counts these files had at close (e.g. m6p1 closed at
|
|
"9 m6_cohort"). The suites grew with later hardening — most visibly
|
|
`tidal/tests/m6_crash_surfaces.rs` (4) and `tidal/tests/m7_crash_m6.rs`, added by
|
|
M7 to crash-fence the state M6 introduced.
|
|
|
|
## Shipped surface (verified against the working tree)
|
|
|
|
| Concern | Artifact |
|
|
|---------|----------|
|
|
| Cohort definitions, predicates | `tidal/src/cohort/types.rs` |
|
|
| Cohort membership resolution | `tidal/src/cohort/resolver.rs` |
|
|
| Per-cohort signal aggregation | `tidal/src/cohort/ledger.rs` |
|
|
| Cohort checkpoint/restore | `tidal/src/cohort/checkpoint.rs` |
|
|
| Cohort db API | `tidal/src/db/cohorts.rs` |
|
|
| Social-graph filter | `tidal/src/query/executor/social_filter.rs` |
|
|
| Co-engagement / collaborative filtering | `tidal/src/entities/co_engagement.rs` |
|
|
| Collections | `tidal/src/entities/collection.rs`, `tidal/src/db/collections.rs` |
|
|
| Saved searches | `tidal/src/session/saved_search.rs` |
|
|
| SUGGEST autocomplete | `tidal/src/query/suggest.rs` |
|
|
| Notification capping | `tidal/src/db/notification_tracker.rs` |
|
|
| Adaptive preference learning rate | `tidal/src/entities/preference.rs` |
|
|
| Sort surface | `tidal/src/ranking/profile.rs` (`Sort`), `tidal/src/ranking/executor/tests/sort_tests.rs` |
|
|
| Benchmarks added by M6 | `tidal/benches/social.rs`, `tidal/benches/sort.rs` |
|
|
|
|
All 62 M6 acceptance criteria in [ROADMAP · Milestone 6](../ROADMAP.md) are
|
|
marked `[x]`. Recorded at close: 1,082 total (835 lib + 247 integration), 9
|
|
`m6_uat` passing; re-verified 2026-02-24 at 1,206 lib + 70 M6 integration tests.
|
|
|
|
## Divergences found while writing these records
|
|
|
|
Three ROADMAP criteria describe behaviour the shipped code deliberately does not
|
|
have. Each is documented in the source itself, and each is recorded in the
|
|
relevant phase doc rather than smoothed over here:
|
|
|
|
1. Co-engagement eviction is **minimum-weight**, not LRU (phase-2).
|
|
2. The `related` blend is **additive with a 0.3 co-engagement term**, not the
|
|
convex combination `0.6/0.3/0.1` (phase-2).
|
|
3. Per-user preference `update_counts` were documented as unpersisted and
|
|
"deferred to M7"; they are in fact persisted now, and by M12, not M7 (phase-6).
|