fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
87 lines
5.3 KiB
Markdown
87 lines
5.3 KiB
Markdown
# m4p2 — Session Signal Engine (✅ COMPLETE 2026-02-21)
|
||
|
||
Phase spec and acceptance criteria: [ROADMAP · Milestone 4 · Phase 2](../ROADMAP.md).
|
||
Milestone index: [README.md](README.md). Backfilled record.
|
||
|
||
## What shipped
|
||
|
||
1. **`session_signal()` write path** (`tidal/src/db/sessions.rs`).
|
||
`db.session_signal(&handle, signal_type, entity_id, weight, timestamp,
|
||
Option<annotation>)` validates the session is open, evaluates policy
|
||
(m4p3), folds the weight into the per-signal-type accumulator, bumps the
|
||
session counters, and journals `WalCommand::SessionSignal`
|
||
(`tidal/src/wal/writer.rs:109`).
|
||
2. **`SessionHotState` — reuse, not a second decay implementation**
|
||
(`tidal/src/session/signal_state.rs`). Session scores delegate the running
|
||
arithmetic to the canonical `forward_decay_step` kernel, the same kernel
|
||
`HotSignalState` uses, so the two tiers cannot drift
|
||
(`CODING_GUIDELINES.md` §3). Session decay is deliberately aggressive:
|
||
`DEFAULT_SESSION_LAMBDA = ln(2)/300s` — a five-minute half-life, matching
|
||
session timescales rather than the multi-day content half-lives.
|
||
Score/timestamp/count are three atomics updated by a CAS loop; the struct is
|
||
intentionally *not* cache-line padded, because sessions are not on the
|
||
200-entity hot ranking path.
|
||
3. **Windowed counters per session signal type.** Each
|
||
`SessionSignalState` owns a `BucketedCounter::with_start_time(now_ns)`, so a
|
||
snapshot reports a 1-hour window count alongside the decayed score.
|
||
4. **Snapshot read model** (`tidal/src/session/snapshot.rs`).
|
||
`db.session_snapshot(session_id) -> SessionSnapshot` carries
|
||
`signals_written`, `signals_rejected`, `overrides_rejected`, `duration_ms`,
|
||
`metadata`, timestamped `annotations`, `reward_velocity`,
|
||
`signaled_entities`, the `audit_log` + `audit_truncated` flag, per-signal-type
|
||
`signals: HashMap<String, SignalSnapEntry>`, and `started_at_ns` /
|
||
`closed_at_ns`. Active sessions decay lazily to wall-clock read time;
|
||
archived sessions are frozen at `close_session`.
|
||
5. **Isolation by construction.** Session signals never touch the global item
|
||
ledger, the user preference vector, or the interaction-weight ledger. A
|
||
session's influence is read-time only, via the m4p4 `FOR SESSION` path.
|
||
6. **Bounded memory, stated in constants** (`tidal/src/session/audit.rs`):
|
||
`MAX_ANNOTATIONS = 100`, `MAX_AUDIT_ENTRIES = 10_000`,
|
||
`MAX_CLOSED_SESSIONS = 10_000` with `EVICT_BATCH_SIZE = 1_000`, plus a
|
||
`MAX_SIGNALED_ENTITIES` structural cap on the distinct-entity set. The
|
||
entity cap is independent of `max_signals_per_session` (which may be `0` =
|
||
unlimited) so an adversarial session cannot grow the boost set without bound;
|
||
past the cap, new distinct entities are dropped and the boost degrades
|
||
gracefully instead of the process growing.
|
||
|
||
## Evidence
|
||
|
||
| Criterion | Proof |
|
||
|-----------|-------|
|
||
| Accepted writes update counts and audit | `m4_uat.rs::step3_session_signal_and_audit` |
|
||
| Annotations captured, entity recorded in `signaled_entities` | `m4_uat.rs::step6_session_annotations_and_snapshot` |
|
||
| Archived snapshot readable after close | `m4_uat.rs::step7_closed_session_snapshot` |
|
||
| Archived snapshot readable after close **and reopen** | `session_durability.rs::archived_session_readable_after_close_and_reopen` |
|
||
| Per-signal windowed counts surface in the snapshot | `session_durability.rs::per_signal_snapshot_shows_windowed_counts` |
|
||
| Annotation timestamps preserved; annotations survive crash | `session_durability.rs::annotation_timestamps_preserved`, `annotations_survive_crash` |
|
||
| WAL replay reproduces the accumulators exactly | `session_durability.rs::wal_replay_restores_signal_counts_exactly`, `wal_replay_reproduces_identical_window_1h` |
|
||
| Two sessions never see each other's entities | `m4_uat.rs::step9_session_isolation` |
|
||
|
||
## Divergence from the plan
|
||
|
||
- **Where the decay constant lives.** The ROADMAP framed session decay as
|
||
per-signal-type schema decay reused verbatim. Shipped behaviour is a single
|
||
session-tier lambda (`DEFAULT_SESSION_LAMBDA`, 5-minute half-life) captured at
|
||
`SessionSignalState` construction — one knob for the whole session tier rather
|
||
than per-type specs.
|
||
- **The performance criteria have a harness but no recorded numbers.** The
|
||
ROADMAP asserts `session_signal` < 200 µs, `session_snapshot` < 50 µs, and
|
||
50,000 session signals/s "(benchmarked)". The harness is real —
|
||
`tidal/benches/session.rs` (added one commit later by `192c473`) benches
|
||
exactly `session_signal`, `session_snapshot_100_signals`, and
|
||
`retrieve_1k_items/{without_session,with_session}` — but no run of it is
|
||
recorded anywhere in `docs/`, unlike the M11/M12 numbers in
|
||
`docs/profiling/`. So the functional criteria are proven by the tests above;
|
||
the three µs/throughput figures are asserted, measurable on demand
|
||
(`cargo bench -p tidaldb --bench session`), and **not currently evidenced**.
|
||
|
||
## Fixed later, worth recording
|
||
|
||
The out-of-order arm of the session decay update originally clamped `dt` to zero
|
||
and folded a late weight in at full value, silently over-crediting stale
|
||
activity. That is why `SessionHotState::on_signal` now delegates to
|
||
`forward_decay_step`: the kernel folds the *pre-decayed* weight
|
||
(`weight × exp(-λ·age)`) and refuses to regress `last_update_ns`. The
|
||
divergence, and the reason the kernel exists, are documented in the doc comment
|
||
at `tidal/src/session/signal_state.rs`.
|