fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
76 lines
4.4 KiB
Markdown
76 lines
4.4 KiB
Markdown
# m4p3 — Policy Enforcement and Audit (✅ COMPLETE 2026-02-21)
|
||
|
||
Phase spec and acceptance criteria: [ROADMAP · Milestone 4 · Phase 3](../ROADMAP.md).
|
||
Milestone index: [README.md](README.md). Backfilled record.
|
||
|
||
## What shipped
|
||
|
||
1. **Policy evaluation on the write path** (`tidal/src/session/policy.rs`).
|
||
Every `session_signal` call is checked before any state mutation. The
|
||
evaluator returns a typed `PolicyViolation { kind, .. }` rather than a
|
||
stringly-typed reason. The four kinds M4 shipped:
|
||
|
||
| `PolicyViolationKind` | Trigger |
|
||
|-----------------------|---------|
|
||
| `Denied` | signal type appears in `denied_signals` |
|
||
| `NotAllowed` | `allowed_signals` is non-empty and the type is absent from it |
|
||
| `CountCap` | `signals_written` reached `max_signals_per_session` |
|
||
| `Expired` | wall clock passed `max_session_duration` |
|
||
|
||
2. **Error surface** (`tidal/src/db/sessions.rs:405-417`). `Expired` maps to
|
||
`TidalError::SessionExpired`; every other kind maps to
|
||
`TidalError::PolicyViolation` (`tidal/src/schema/error.rs:123`). A rejected
|
||
write mutates no session signal state — only the rejection counter and the
|
||
audit log advance.
|
||
3. **Bounded audit log** (`tidal/src/session/audit.rs`). `AuditEntry` carries
|
||
`timestamp_ns`, the signal type, an `accepted` flag, an `AuditKind`, and an
|
||
optional `reason`. It is a `VecDeque` capped at `MAX_AUDIT_ENTRIES = 10_000`;
|
||
past the cap the oldest entries are evicted and the snapshot's
|
||
`audit_truncated` flag is set, so a reader can tell a complete log from a
|
||
truncated one instead of silently believing a partial record.
|
||
4. **`db.session_audit(session_id)`** reads the live log for an active session;
|
||
for a closed session the log ships inside the archived `SessionSnapshot`
|
||
(`audit_log` field), so it is still readable after close and after restart.
|
||
|
||
## Evidence
|
||
|
||
| Criterion | Proof |
|
||
|-----------|-------|
|
||
| A denied signal is rejected, counted, and audited with a reason | `m4_uat.rs::step4_policy_rejects_denied_signal` — asserts `signals_rejected == 1`, `signals_written == 0`, `audit.len() == 1`, `!audit[0].accepted`, `audit[0].reason.is_some()` |
|
||
| A signal absent from a non-empty allow list is rejected | `m4_uat.rs::step5_policy_rejects_non_allowed_signal` |
|
||
| Accepted writes are audited as accepted | `m4_uat.rs::step3_session_signal_and_audit` |
|
||
| Truncation is observable, not silent | `session_durability.rs::audit_truncation_marker_set_when_cap_exceeded` |
|
||
| Audit survives close | `m4_uat.rs::step7_closed_session_snapshot`, `session_durability.rs::archived_session_readable_after_close_and_reopen` |
|
||
|
||
## Divergence from the plan
|
||
|
||
- **Typed kinds, not free-text reasons.** The ROADMAP specified
|
||
`AuditEntry = { timestamp, signal_type, outcome: Accepted | Rejected(reason) }`
|
||
and a `reason: String` on the error. Shipped design keeps the human-readable
|
||
reason but adds `PolicyViolationKind` and `AuditKind` enums, so callers branch
|
||
on a variant instead of matching on prose. This is strictly better and is what
|
||
M9/M10 later extended.
|
||
- **The `< 1 µs` policy-evaluation criterion has no recorded measurement.** The
|
||
ROADMAP marks it `[x]` "(benchmarked)". It is a `HashMap` lookup on the write
|
||
path, which makes the claim plausible, but no benchmark isolates it and no
|
||
recorded run exists. `tidal/benches/session.rs::session_signal` measures the
|
||
whole write including WAL, not policy evaluation alone. Treat the figure as
|
||
unevidenced.
|
||
- **The property test in the ROADMAP criteria is not present as a property
|
||
test.** "For any sequence of allowed and denied signal writes, the audit log
|
||
exactly matches the write outcomes and no denied signal modifies session
|
||
state" is proven by the concrete `m4_uat.rs` steps 3–5 and by
|
||
`session_durability.rs`, not by a `proptest`. The invariant is covered; the
|
||
stated *form* (randomized property test) was not built.
|
||
|
||
## Extended later
|
||
|
||
M9/M10 governance (`d8e4083`) added seven further `PolicyViolationKind` variants
|
||
— `CommunityWriteDenied`, `CommunityWriteNotAllowed`, `ReadDenied`,
|
||
`ReadNotAllowed`, `AttributeReadDenied`, `AttributeReadNotAllowed`,
|
||
`ProfileOverrideNotAllowed` — and the matching `AuditKind` variants
|
||
(`ReadDenied`, `AttributeReadDenied`, `ProfileOverrideRejected`) plus the
|
||
`overrides_rejected` counter on `SessionSnapshot`. The M4 mechanism is the one
|
||
they extended: read-path and community policy reuse this evaluator and this
|
||
audit log rather than adding a parallel one.
|