jordan fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path

Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.

2026-08-30 20:55:58 -06:00

5.3 KiB

Raw Permalink Blame History

m4p2 — Session Signal Engine (✅ COMPLETE 2026-02-21)

Phase spec and acceptance criteria: ROADMAP · Milestone 4 · Phase 2. Milestone index: README.md. Backfilled record.

What shipped

session_signal() write path (tidal/src/db/sessions.rs). db.session_signal(&handle, signal_type, entity_id, weight, timestamp, Option<annotation>) validates the session is open, evaluates policy (m4p3), folds the weight into the per-signal-type accumulator, bumps the session counters, and journals WalCommand::SessionSignal (tidal/src/wal/writer.rs:109).
SessionHotState — reuse, not a second decay implementation (tidal/src/session/signal_state.rs). Session scores delegate the running arithmetic to the canonical forward_decay_step kernel, the same kernel HotSignalState uses, so the two tiers cannot drift (CODING_GUIDELINES.md §3). Session decay is deliberately aggressive: DEFAULT_SESSION_LAMBDA = ln(2)/300s — a five-minute half-life, matching session timescales rather than the multi-day content half-lives. Score/timestamp/count are three atomics updated by a CAS loop; the struct is intentionally not cache-line padded, because sessions are not on the 200-entity hot ranking path.
Windowed counters per session signal type. Each SessionSignalState owns a BucketedCounter::with_start_time(now_ns), so a snapshot reports a 1-hour window count alongside the decayed score.
Snapshot read model (tidal/src/session/snapshot.rs). db.session_snapshot(session_id) -> SessionSnapshot carries signals_written, signals_rejected, overrides_rejected, duration_ms, metadata, timestamped annotations, reward_velocity, signaled_entities, the audit_log + audit_truncated flag, per-signal-type signals: HashMap<String, SignalSnapEntry>, and started_at_ns / closed_at_ns. Active sessions decay lazily to wall-clock read time; archived sessions are frozen at close_session.
Isolation by construction. Session signals never touch the global item ledger, the user preference vector, or the interaction-weight ledger. A session's influence is read-time only, via the m4p4 FOR SESSION path.
Bounded memory, stated in constants (tidal/src/session/audit.rs): MAX_ANNOTATIONS = 100, MAX_AUDIT_ENTRIES = 10_000, MAX_CLOSED_SESSIONS = 10_000 with EVICT_BATCH_SIZE = 1_000, plus a MAX_SIGNALED_ENTITIES structural cap on the distinct-entity set. The entity cap is independent of max_signals_per_session (which may be 0 = unlimited) so an adversarial session cannot grow the boost set without bound; past the cap, new distinct entities are dropped and the boost degrades gracefully instead of the process growing.

Evidence

Criterion	Proof
Accepted writes update counts and audit	`m4_uat.rs::step3_session_signal_and_audit`
Annotations captured, entity recorded in `signaled_entities`	`m4_uat.rs::step6_session_annotations_and_snapshot`
Archived snapshot readable after close	`m4_uat.rs::step7_closed_session_snapshot`
Archived snapshot readable after close and reopen	`session_durability.rs::archived_session_readable_after_close_and_reopen`
Per-signal windowed counts surface in the snapshot	`session_durability.rs::per_signal_snapshot_shows_windowed_counts`
Annotation timestamps preserved; annotations survive crash	`session_durability.rs::annotation_timestamps_preserved`, `annotations_survive_crash`
WAL replay reproduces the accumulators exactly	`session_durability.rs::wal_replay_restores_signal_counts_exactly`, `wal_replay_reproduces_identical_window_1h`
Two sessions never see each other's entities	`m4_uat.rs::step9_session_isolation`

Divergence from the plan

Where the decay constant lives. The ROADMAP framed session decay as per-signal-type schema decay reused verbatim. Shipped behaviour is a single session-tier lambda (DEFAULT_SESSION_LAMBDA, 5-minute half-life) captured at SessionSignalState construction — one knob for the whole session tier rather than per-type specs.
The performance criteria have a harness but no recorded numbers. The ROADMAP asserts session_signal < 200 µs, session_snapshot < 50 µs, and 50,000 session signals/s "(benchmarked)". The harness is real — tidal/benches/session.rs (added one commit later by 192c473) benches exactly session_signal, session_snapshot_100_signals, and retrieve_1k_items/{without_session,with_session} — but no run of it is recorded anywhere in docs/, unlike the M11/M12 numbers in docs/profiling/. So the functional criteria are proven by the tests above; the three µs/throughput figures are asserted, measurable on demand (cargo bench -p tidaldb --bench session), and not currently evidenced.

Fixed later, worth recording

The out-of-order arm of the session decay update originally clamped dt to zero and folded a late weight in at full value, silently over-crediting stale activity. That is why SessionHotState::on_signal now delegates to forward_decay_step: the kernel folds the pre-decayed weight (weight × exp(-λ·age)) and refuses to regress last_update_ns. The divergence, and the reason the kernel exists, are documented in the doc comment at tidal/src/session/signal_state.rs.

5.3 KiB Raw Permalink Blame History Unescape Escape