jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
e117333fec
tidaldb/docs/planning/milestone-4/phase-1.md
jordan fe8d0c87e7 harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path 
Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.
2026-08-30 20:55:58 -06:00

3.9 KiB
Raw Blame History

m4p1 — Session Schema and Lifecycle (✅ COMPLETE 2026-02-21)

Phase spec and acceptance criteria: ROADMAP · Milestone 4 · Phase 1. Milestone index: README.md. Backfilled record — see the README's note on provenance.

What shipped

  1. Session identity types (tidal/src/session/types.rs). SessionId is a u64 newtype (Copy, Hash, Ord, Display as session:<n>) handed out monotonically by start_session; from_raw exists for deserialization. AgentId is a validated String newtype: 1–64 bytes, [a-z0-9_-] only — uppercase, spaces, empty, and 65-byte inputs are all rejected at construction.
  2. SessionHandle as the capability (tidal/src/session/state.rs:151). Carries id, user_id, agent_id, policy_name, started_at: Instant, and a closed: Arc<AtomicBool> shared with the live SessionState. close_session takes the handle by value, so use-after-close is a compile error in the common case; the shared closed flag is the runtime defence-in-depth for handles cloned into another thread.
  3. Policy declaration in schema (tidal/src/schema/validation/policies.rs). AgentPolicy with allowed_signals, denied_signals, max_session_duration, max_signals_per_session (0 = unlimited), registered through SchemaBuilder::session_policy(name, policy) and read back via Schema::session_policy(name). Validated at schema build time.
  4. Lifecycle API (tidal/src/db/sessions.rs). start_session(user_id, agent_id, policy_name, metadata) -> Result<SessionHandle>, close_session(handle) -> Result<SessionSummary>, and active_sessions() -> Vec<…>. An undeclared policy name is refused at start_session, not at first write.
  5. WAL durability for the lifecycle. WalCommand::SessionStart / SessionClose (tidal/src/wal/writer.rs:101,124) journal session boundaries on the same stream as signals; replay restores start-without-close as an active session and start-with-close as an archive.
  6. Archive keyspace. Tag::Session = 0x07 (tidal/src/storage/keys.rs:29) holds session snapshots and audit logs, so a closed session is readable after process restart, not just after close.

Evidence

Criterion Proof
Schema accepts and returns a declared policy m4_uat.rs::step1_schema_with_session_policy_builds
Start → active_sessions → close → not active m4_uat.rs::step2_session_start_and_close
Multiple concurrent sessions tracked independently m4_uat.rs::step12_active_sessions_tracking
Undeclared policy name rejected m4_uat.rs::step10_invalid_policy_name_rejected
AgentId format enforcement m4_uat.rs::step11_agent_id_validation
Active session restored after crash session_durability.rs::active_session_state_restored_after_crash
Session metadata survives crash session_durability.rs::metadata_survives_crash
Closed session not resurrected as active review_pass2_zone_a_sessions.rs::closed_session_is_not_restored_as_active

Divergence from the plan

  • Error naming. The ROADMAP says LumenError::SessionExpired; the shipped enum is TidalError::SessionExpired (tidal/src/schema/error.rs:130).
  • Session-ID reuse across restart. The doc comment on SessionId still says uniqueness is "not guaranteed across restarts". That is now stale: the review pass-2 remediation (9728194) added review_pass2_zone_a_sessions.rs::reopen_does_not_reissue_archived_session_id, which proves a reopened database does not hand an archived id to a new session. The guarantee is real; only the comment lags.
  • AgentPolicy grew after M4. The five read-path and profile-override fields (allowed_read_signals, denied_read_signals, allowed_user_attributes, denied_user_attributes, allowed_profile_overrides) were added by d8e4083 for M9/M10 governance. M4 shipped only the four write-path fields.