fe8d0c87e7
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.
65 lines
3.9 KiB
Markdown
65 lines
3.9 KiB
Markdown
# m4p1 — Session Schema and Lifecycle (✅ COMPLETE 2026-02-21)
|
||
|
||
Phase spec and acceptance criteria: [ROADMAP · Milestone 4 · Phase 1](../ROADMAP.md).
|
||
Milestone index: [README.md](README.md). Backfilled record — see the README's note
|
||
on provenance.
|
||
|
||
## What shipped
|
||
|
||
1. **Session identity types** (`tidal/src/session/types.rs`). `SessionId` is a
|
||
`u64` newtype (`Copy`, `Hash`, `Ord`, `Display` as `session:<n>`) handed out
|
||
monotonically by `start_session`; `from_raw` exists for deserialization.
|
||
`AgentId` is a validated `String` newtype: 1–64 bytes, `[a-z0-9_-]` only —
|
||
uppercase, spaces, empty, and 65-byte inputs are all rejected at construction.
|
||
2. **`SessionHandle` as the capability** (`tidal/src/session/state.rs:151`).
|
||
Carries `id`, `user_id`, `agent_id`, `policy_name`, `started_at: Instant`, and
|
||
a `closed: Arc<AtomicBool>` shared with the live `SessionState`.
|
||
`close_session` takes the handle **by value**, so use-after-close is a compile
|
||
error in the common case; the shared `closed` flag is the runtime
|
||
defence-in-depth for handles cloned into another thread.
|
||
3. **Policy declaration in schema** (`tidal/src/schema/validation/policies.rs`).
|
||
`AgentPolicy` with `allowed_signals`, `denied_signals`, `max_session_duration`,
|
||
`max_signals_per_session` (`0` = unlimited), registered through
|
||
`SchemaBuilder::session_policy(name, policy)` and read back via
|
||
`Schema::session_policy(name)`. Validated at schema build time.
|
||
4. **Lifecycle API** (`tidal/src/db/sessions.rs`). `start_session(user_id,
|
||
agent_id, policy_name, metadata) -> Result<SessionHandle>`,
|
||
`close_session(handle) -> Result<SessionSummary>`, and
|
||
`active_sessions() -> Vec<…>`. An undeclared policy name is refused at
|
||
`start_session`, not at first write.
|
||
5. **WAL durability for the lifecycle**. `WalCommand::SessionStart` /
|
||
`SessionClose` (`tidal/src/wal/writer.rs:101,124`) journal session boundaries
|
||
on the same stream as signals; replay restores start-without-close as an
|
||
active session and start-with-close as an archive.
|
||
6. **Archive keyspace.** `Tag::Session = 0x07` (`tidal/src/storage/keys.rs:29`)
|
||
holds session snapshots and audit logs, so a closed session is readable after
|
||
process restart, not just after close.
|
||
|
||
## Evidence
|
||
|
||
| Criterion | Proof |
|
||
|-----------|-------|
|
||
| Schema accepts and returns a declared policy | `m4_uat.rs::step1_schema_with_session_policy_builds` |
|
||
| Start → active_sessions → close → not active | `m4_uat.rs::step2_session_start_and_close` |
|
||
| Multiple concurrent sessions tracked independently | `m4_uat.rs::step12_active_sessions_tracking` |
|
||
| Undeclared policy name rejected | `m4_uat.rs::step10_invalid_policy_name_rejected` |
|
||
| `AgentId` format enforcement | `m4_uat.rs::step11_agent_id_validation` |
|
||
| Active session restored after crash | `session_durability.rs::active_session_state_restored_after_crash` |
|
||
| Session metadata survives crash | `session_durability.rs::metadata_survives_crash` |
|
||
| Closed session not resurrected as active | `review_pass2_zone_a_sessions.rs::closed_session_is_not_restored_as_active` |
|
||
|
||
## Divergence from the plan
|
||
|
||
- **Error naming.** The ROADMAP says `LumenError::SessionExpired`; the shipped
|
||
enum is `TidalError::SessionExpired` (`tidal/src/schema/error.rs:130`).
|
||
- **Session-ID reuse across restart.** The doc comment on `SessionId` still says
|
||
uniqueness is "not guaranteed across restarts". That is now stale: the review
|
||
pass-2 remediation (`9728194`) added
|
||
`review_pass2_zone_a_sessions.rs::reopen_does_not_reissue_archived_session_id`,
|
||
which proves a reopened database does not hand an archived id to a new session.
|
||
The guarantee is real; only the comment lags.
|
||
- **`AgentPolicy` grew after M4.** The five read-path and profile-override fields
|
||
(`allowed_read_signals`, `denied_read_signals`, `allowed_user_attributes`,
|
||
`denied_user_attributes`, `allowed_profile_overrides`) were added by `d8e4083`
|
||
for M9/M10 governance. M4 shipped only the four write-path fields.
|