tidaldb/docs/planning/milestone-4/phase-3.md
jordan fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.
2026-08-30 20:55:58 -06:00

4.4 KiB
Raw Blame History

m4p3 — Policy Enforcement and Audit ( COMPLETE 2026-02-21)

Phase spec and acceptance criteria: ROADMAP · Milestone 4 · Phase 3. Milestone index: README.md. Backfilled record.

What shipped

  1. Policy evaluation on the write path (tidal/src/session/policy.rs). Every session_signal call is checked before any state mutation. The evaluator returns a typed PolicyViolation { kind, .. } rather than a stringly-typed reason. The four kinds M4 shipped:

    PolicyViolationKind Trigger
    Denied signal type appears in denied_signals
    NotAllowed allowed_signals is non-empty and the type is absent from it
    CountCap signals_written reached max_signals_per_session
    Expired wall clock passed max_session_duration
  2. Error surface (tidal/src/db/sessions.rs:405-417). Expired maps to TidalError::SessionExpired; every other kind maps to TidalError::PolicyViolation (tidal/src/schema/error.rs:123). A rejected write mutates no session signal state — only the rejection counter and the audit log advance.

  3. Bounded audit log (tidal/src/session/audit.rs). AuditEntry carries timestamp_ns, the signal type, an accepted flag, an AuditKind, and an optional reason. It is a VecDeque capped at MAX_AUDIT_ENTRIES = 10_000; past the cap the oldest entries are evicted and the snapshot's audit_truncated flag is set, so a reader can tell a complete log from a truncated one instead of silently believing a partial record.

  4. db.session_audit(session_id) reads the live log for an active session; for a closed session the log ships inside the archived SessionSnapshot (audit_log field), so it is still readable after close and after restart.

Evidence

Criterion Proof
A denied signal is rejected, counted, and audited with a reason m4_uat.rs::step4_policy_rejects_denied_signal — asserts signals_rejected == 1, signals_written == 0, audit.len() == 1, !audit[0].accepted, audit[0].reason.is_some()
A signal absent from a non-empty allow list is rejected m4_uat.rs::step5_policy_rejects_non_allowed_signal
Accepted writes are audited as accepted m4_uat.rs::step3_session_signal_and_audit
Truncation is observable, not silent session_durability.rs::audit_truncation_marker_set_when_cap_exceeded
Audit survives close m4_uat.rs::step7_closed_session_snapshot, session_durability.rs::archived_session_readable_after_close_and_reopen

Divergence from the plan

  • Typed kinds, not free-text reasons. The ROADMAP specified AuditEntry = { timestamp, signal_type, outcome: Accepted | Rejected(reason) } and a reason: String on the error. Shipped design keeps the human-readable reason but adds PolicyViolationKind and AuditKind enums, so callers branch on a variant instead of matching on prose. This is strictly better and is what M9/M10 later extended.
  • The < 1 µs policy-evaluation criterion has no recorded measurement. The ROADMAP marks it [x] "(benchmarked)". It is a HashMap lookup on the write path, which makes the claim plausible, but no benchmark isolates it and no recorded run exists. tidal/benches/session.rs::session_signal measures the whole write including WAL, not policy evaluation alone. Treat the figure as unevidenced.
  • The property test in the ROADMAP criteria is not present as a property test. "For any sequence of allowed and denied signal writes, the audit log exactly matches the write outcomes and no denied signal modifies session state" is proven by the concrete m4_uat.rs steps 35 and by session_durability.rs, not by a proptest. The invariant is covered; the stated form (randomized property test) was not built.

Extended later

M9/M10 governance (d8e4083) added seven further PolicyViolationKind variants — CommunityWriteDenied, CommunityWriteNotAllowed, ReadDenied, ReadNotAllowed, AttributeReadDenied, AttributeReadNotAllowed, ProfileOverrideNotAllowed — and the matching AuditKind variants (ReadDenied, AttributeReadDenied, ProfileOverrideRejected) plus the overrides_rejected counter on SessionSnapshot. The M4 mechanism is the one they extended: read-path and community policy reuse this evaluator and this audit log rather than adding a parallel one.