fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
86 lines
5.0 KiB
Markdown
86 lines
5.0 KiB
Markdown
# Milestone 4 · Agent Memory (✅ COMPLETE 2026-02-21)
|
||
|
||
Milestone spec, thesis, UAT scenario, and per-phase acceptance criteria:
|
||
[ROADMAP · Milestone 4](../ROADMAP.md). Status row: `m4: Agent Session Layer`.
|
||
|
||
> **These are backfilled records, not the original plans.** M4 shipped in the
|
||
> squashed commit `39ada28` (2026-02-21) before this project kept per-milestone
|
||
> planning directories, so the four phase docs here were written after the fact
|
||
> (2026-08-30) from the ROADMAP acceptance criteria, the shipped source tree, and
|
||
> the M4 test suite. Every claim below names the artifact that proves it. Where
|
||
> the shipped surface diverges from the ROADMAP text, the phase doc says so
|
||
> rather than restating the plan.
|
||
|
||
## What the milestone proves
|
||
|
||
An agent — not a human clicking a UI — can own a scoped memory lane inside the
|
||
same embedded process: open a session bound to a `(user, agent, policy)` triple,
|
||
write short-lived signals that the schema-declared policy accepts or rejects,
|
||
read a live snapshot of that session's decayed state, blend the session into a
|
||
`RETRIEVE` ranking, then close it and read the frozen archive. No Redis, no
|
||
feature store, no policy middleware.
|
||
|
||
## Phases
|
||
|
||
| Phase | Name | Record |
|
||
|-------|------|--------|
|
||
| m4p1 | Session Schema and Lifecycle | [phase-1.md](phase-1.md) |
|
||
| m4p2 | Session Signal Engine | [phase-2.md](phase-2.md) |
|
||
| m4p3 | Policy Enforcement and Audit | [phase-3.md](phase-3.md) |
|
||
| m4p4 | Session-Aware Ranking and M4 UAT | [phase-4.md](phase-4.md) |
|
||
|
||
The four phases were strictly sequential (each needed the previous phase's write
|
||
path), and all four landed in one commit; the phase split below is therefore a
|
||
split of the *delivered surface*, traced from the ROADMAP's phase definitions to
|
||
the modules and tests that implement each one.
|
||
|
||
## Shipped surface (verified against the working tree)
|
||
|
||
| Concern | Artifact |
|
||
|---------|----------|
|
||
| Session types | `tidal/src/session/types.rs` — `SessionId`, `AgentId`, `SessionHandle` |
|
||
| Live session state | `tidal/src/session/state.rs` — per-session accumulators |
|
||
| Session signal decay | `tidal/src/session/signal_state.rs` |
|
||
| Snapshot / archive read model | `tidal/src/session/snapshot.rs` |
|
||
| Policy evaluation | `tidal/src/session/policy.rs` |
|
||
| Audit log | `tidal/src/session/audit.rs` |
|
||
| Archive serialization | `tidal/src/session/serde/mod.rs`, `serde/start_record.rs` |
|
||
| Policy declaration in schema | `tidal/src/schema/validation/policies.rs` — `AgentPolicy` |
|
||
| `db.*` session API | `tidal/src/db/sessions.rs` |
|
||
| WAL durability | `WalCommand::SessionStart` / `SessionSignal` / `SessionClose` in `tidal/src/wal/writer.rs` |
|
||
|
||
## Evidence
|
||
|
||
| Claim | Proof |
|
||
|-------|-------|
|
||
| Full agent workflow works end to end | `tidal/tests/m4_uat.rs` — 12 `#[test]` steps, "M4 User Acceptance Test: Agent Session Layer". Added by `39ada28`, the M4 commit. |
|
||
| Sessions survive a crash | `tidal/tests/session_durability.rs` — 10 tests incl. `active_session_state_restored_after_crash`, `metadata_survives_crash`, `wal_replay_restores_signal_counts_exactly`. Added one commit later by `192c473` (M5), so read it as M4-surface hardening rather than M4 itself. |
|
||
| Session semantics still hold after later reworks | `tidal/tests/review_pass2_zone_a_sessions.rs`, `tidal/tests/review_pass2_query_for_session.rs` — added by `9728194` (M0–M10 review pass 2, 2026-06-09) |
|
||
| Sessions survive M7 crash hardening | `tidal/tests/m7_crash_m6.rs`, `tidal/tests/m7_crash_invariant.rs` (session-touching cases) |
|
||
| Recorded at close | ROADMAP status row: 607 lib + 12 m4_uat + prior UATs passing |
|
||
|
||
All 49 M4 acceptance criteria in [ROADMAP · Milestone 4](../ROADMAP.md) are
|
||
marked `[x]`.
|
||
|
||
## Naming drift worth knowing
|
||
|
||
The ROADMAP's M4 text names errors `LumenError::PolicyViolation` /
|
||
`LumenError::SessionExpired`. The shipped enum is `TidalError` — the `Lumen`
|
||
prefix is a pre-rename artifact left in the planning prose. Real variants:
|
||
`TidalError::PolicyViolation` and `TidalError::SessionExpired`
|
||
(`tidal/src/schema/error.rs:123,130`), plus the session-internal
|
||
`PolicyViolation { kind: PolicyViolationKind }` re-exported from
|
||
`tidal/src/lib.rs`.
|
||
|
||
## What M4 deliberately did not do
|
||
|
||
The ROADMAP's "Deferred to Later Milestones" list for M4 held nine items. Their
|
||
real fates:
|
||
|
||
- Cross-session aggregation → shipped in m6p4 (see [milestone-6/phase-4.md](../milestone-6/phase-4.md)).
|
||
- Session signal influence on the global preference vector → shipped in m6p4.
|
||
- Semantic hint matching → M4 shipped keyword matching; embeddings arrived with M5.
|
||
- RLHF training-data export, per-agent QPS rate limiting, session TTL sweeper → shipped in M7 (m7p2/m7p4).
|
||
- User revocation of agent-contributed signals, multi-agent sessions → shipped in M9/M10; the read-path and profile-override fields on `AgentPolicy` (`allowed_read_signals`, `denied_read_signals`, `allowed_user_attributes`, `denied_user_attributes`, `allowed_profile_overrides`) were added by commit `d8e4083` for that work, not by M4.
|
||
- Session forking and merging → still not built. No commit, no test, no issue; it remains an open idea, not a shipped feature.
|