fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof. |
||
|---|---|---|
| .. | ||
| phase-1.md | ||
| phase-2.md | ||
| phase-3.md | ||
| phase-4.md | ||
| README.md | ||
Milestone 4 · Agent Memory (✅ COMPLETE 2026-02-21)
Milestone spec, thesis, UAT scenario, and per-phase acceptance criteria:
ROADMAP · Milestone 4. Status row: m4: Agent Session Layer.
These are backfilled records, not the original plans. M4 shipped in the squashed commit
39ada28(2026-02-21) before this project kept per-milestone planning directories, so the four phase docs here were written after the fact (2026-08-30) from the ROADMAP acceptance criteria, the shipped source tree, and the M4 test suite. Every claim below names the artifact that proves it. Where the shipped surface diverges from the ROADMAP text, the phase doc says so rather than restating the plan.
What the milestone proves
An agent — not a human clicking a UI — can own a scoped memory lane inside the
same embedded process: open a session bound to a (user, agent, policy) triple,
write short-lived signals that the schema-declared policy accepts or rejects,
read a live snapshot of that session's decayed state, blend the session into a
RETRIEVE ranking, then close it and read the frozen archive. No Redis, no
feature store, no policy middleware.
Phases
| Phase | Name | Record |
|---|---|---|
| m4p1 | Session Schema and Lifecycle | phase-1.md |
| m4p2 | Session Signal Engine | phase-2.md |
| m4p3 | Policy Enforcement and Audit | phase-3.md |
| m4p4 | Session-Aware Ranking and M4 UAT | phase-4.md |
The four phases were strictly sequential (each needed the previous phase's write path), and all four landed in one commit; the phase split below is therefore a split of the delivered surface, traced from the ROADMAP's phase definitions to the modules and tests that implement each one.
Shipped surface (verified against the working tree)
| Concern | Artifact |
|---|---|
| Session types | tidal/src/session/types.rs — SessionId, AgentId, SessionHandle |
| Live session state | tidal/src/session/state.rs — per-session accumulators |
| Session signal decay | tidal/src/session/signal_state.rs |
| Snapshot / archive read model | tidal/src/session/snapshot.rs |
| Policy evaluation | tidal/src/session/policy.rs |
| Audit log | tidal/src/session/audit.rs |
| Archive serialization | tidal/src/session/serde/mod.rs, serde/start_record.rs |
| Policy declaration in schema | tidal/src/schema/validation/policies.rs — AgentPolicy |
db.* session API |
tidal/src/db/sessions.rs |
| WAL durability | WalCommand::SessionStart / SessionSignal / SessionClose in tidal/src/wal/writer.rs |
Evidence
| Claim | Proof |
|---|---|
| Full agent workflow works end to end | tidal/tests/m4_uat.rs — 12 #[test] steps, "M4 User Acceptance Test: Agent Session Layer". Added by 39ada28, the M4 commit. |
| Sessions survive a crash | tidal/tests/session_durability.rs — 10 tests incl. active_session_state_restored_after_crash, metadata_survives_crash, wal_replay_restores_signal_counts_exactly. Added one commit later by 192c473 (M5), so read it as M4-surface hardening rather than M4 itself. |
| Session semantics still hold after later reworks | tidal/tests/review_pass2_zone_a_sessions.rs, tidal/tests/review_pass2_query_for_session.rs — added by 9728194 (M0–M10 review pass 2, 2026-06-09) |
| Sessions survive M7 crash hardening | tidal/tests/m7_crash_m6.rs, tidal/tests/m7_crash_invariant.rs (session-touching cases) |
| Recorded at close | ROADMAP status row: 607 lib + 12 m4_uat + prior UATs passing |
All 49 M4 acceptance criteria in ROADMAP · Milestone 4 are
marked [x].
Naming drift worth knowing
The ROADMAP's M4 text names errors LumenError::PolicyViolation /
LumenError::SessionExpired. The shipped enum is TidalError — the Lumen
prefix is a pre-rename artifact left in the planning prose. Real variants:
TidalError::PolicyViolation and TidalError::SessionExpired
(tidal/src/schema/error.rs:123,130), plus the session-internal
PolicyViolation { kind: PolicyViolationKind } re-exported from
tidal/src/lib.rs.
What M4 deliberately did not do
The ROADMAP's "Deferred to Later Milestones" list for M4 held nine items. Their real fates:
- Cross-session aggregation → shipped in m6p4 (see milestone-6/phase-4.md).
- Session signal influence on the global preference vector → shipped in m6p4.
- Semantic hint matching → M4 shipped keyword matching; embeddings arrived with M5.
- RLHF training-data export, per-agent QPS rate limiting, session TTL sweeper → shipped in M7 (m7p2/m7p4).
- User revocation of agent-contributed signals, multi-agent sessions → shipped in M9/M10; the read-path and profile-override fields on
AgentPolicy(allowed_read_signals,denied_read_signals,allowed_user_attributes,denied_user_attributes,allowed_profile_overrides) were added by commitd8e4083for that work, not by M4. - Session forking and merging → still not built. No commit, no test, no issue; it remains an open idea, not a shipped feature.