Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
4.4 KiB
m4p3 — Policy Enforcement and Audit (✅ COMPLETE 2026-02-21)
Phase spec and acceptance criteria: ROADMAP · Milestone 4 · Phase 3. Milestone index: README.md. Backfilled record.
What shipped
-
Policy evaluation on the write path (
tidal/src/session/policy.rs). Everysession_signalcall is checked before any state mutation. The evaluator returns a typedPolicyViolation { kind, .. }rather than a stringly-typed reason. The four kinds M4 shipped:PolicyViolationKindTrigger Deniedsignal type appears in denied_signalsNotAllowedallowed_signalsis non-empty and the type is absent from itCountCapsignals_writtenreachedmax_signals_per_sessionExpiredwall clock passed max_session_duration -
Error surface (
tidal/src/db/sessions.rs:405-417).Expiredmaps toTidalError::SessionExpired; every other kind maps toTidalError::PolicyViolation(tidal/src/schema/error.rs:123). A rejected write mutates no session signal state — only the rejection counter and the audit log advance. -
Bounded audit log (
tidal/src/session/audit.rs).AuditEntrycarriestimestamp_ns, the signal type, anacceptedflag, anAuditKind, and an optionalreason. It is aVecDequecapped atMAX_AUDIT_ENTRIES = 10_000; past the cap the oldest entries are evicted and the snapshot'saudit_truncatedflag is set, so a reader can tell a complete log from a truncated one instead of silently believing a partial record. -
db.session_audit(session_id)reads the live log for an active session; for a closed session the log ships inside the archivedSessionSnapshot(audit_logfield), so it is still readable after close and after restart.
Evidence
| Criterion | Proof |
|---|---|
| A denied signal is rejected, counted, and audited with a reason | m4_uat.rs::step4_policy_rejects_denied_signal — asserts signals_rejected == 1, signals_written == 0, audit.len() == 1, !audit[0].accepted, audit[0].reason.is_some() |
| A signal absent from a non-empty allow list is rejected | m4_uat.rs::step5_policy_rejects_non_allowed_signal |
| Accepted writes are audited as accepted | m4_uat.rs::step3_session_signal_and_audit |
| Truncation is observable, not silent | session_durability.rs::audit_truncation_marker_set_when_cap_exceeded |
| Audit survives close | m4_uat.rs::step7_closed_session_snapshot, session_durability.rs::archived_session_readable_after_close_and_reopen |
Divergence from the plan
- Typed kinds, not free-text reasons. The ROADMAP specified
AuditEntry = { timestamp, signal_type, outcome: Accepted | Rejected(reason) }and areason: Stringon the error. Shipped design keeps the human-readable reason but addsPolicyViolationKindandAuditKindenums, so callers branch on a variant instead of matching on prose. This is strictly better and is what M9/M10 later extended. - The
< 1 µspolicy-evaluation criterion has no recorded measurement. The ROADMAP marks it[x]"(benchmarked)". It is aHashMaplookup on the write path, which makes the claim plausible, but no benchmark isolates it and no recorded run exists.tidal/benches/session.rs::session_signalmeasures the whole write including WAL, not policy evaluation alone. Treat the figure as unevidenced. - The property test in the ROADMAP criteria is not present as a property
test. "For any sequence of allowed and denied signal writes, the audit log
exactly matches the write outcomes and no denied signal modifies session
state" is proven by the concrete
m4_uat.rssteps 3–5 and bysession_durability.rs, not by aproptest. The invariant is covered; the stated form (randomized property test) was not built.
Extended later
M9/M10 governance (d8e4083) added seven further PolicyViolationKind variants
— CommunityWriteDenied, CommunityWriteNotAllowed, ReadDenied,
ReadNotAllowed, AttributeReadDenied, AttributeReadNotAllowed,
ProfileOverrideNotAllowed — and the matching AuditKind variants
(ReadDenied, AttributeReadDenied, ProfileOverrideRejected) plus the
overrides_rejected counter on SessionSnapshot. The M4 mechanism is the one
they extended: read-path and community policy reuse this evaluator and this
audit log rather than adding a parallel one.