tidaldb

jordan/tidaldb

Fork 0

Commit Graph

Author	SHA1	Message	Date
jordan	fe8d0c87e7	harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement). Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release gate past its budget. Both push-path steps now declare backend_options.kubernetes.resources as two YAML anchors declared once on their first consuming step. The values are CALIBRATED against measured free node capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's original figure) fits on NO node and would sit Pending forever, because `ci-build-bounds` grants permission and the nodes supply capacity, and those are not the same thing. The `nightly` cron described in this file for 216 days was never created, so tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero signal while reading like standing coverage. nightly-chaos and nightly-security-ops now alias the anchors and have budgets matching the gate (their 120/90 were TIGHTER on the same runner, so they would have failed nightly for a budget reason, not a correctness one). nightly-soak is REMOVED, not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best node has 1700m free CPU, so it would fail on starvation rather than regression — manufacturing a nightly false alarm. Its commands move verbatim to docs/runbooks/nightly-soak.md. Ring 1 — four fabrications removed from the wire. - scatter_merge sorted and truncated without re-stamping rank, so /feed and /search returned 1,1,2 under full placement. Reuses merge_cross_shard's existing stamp; asserted on BOTH the multi-group merge path and the single-group [only] fast path that bypasses it. - aggregate_region_row's None arm invented `applied_events: 0` plus a deficit derived from it. applied_events/lag_events are now Option<u64>, null on the wire. leader_last_seq was also unwrap_or(0), so a node that could not reach the LEADER computed 0 - applied = 0 for every region and reported a converged cluster it had never measured — a fabrication pointing the dangerous way. - tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was actively hiding the PVC-wipe shape: a measured zero with a real deficit rendered as "no report" instead of BEHIND. Now read off the wire; converged exits 0, partitioned still exits nonzero. - /sharded/* answered 201/204 for single-copy writes with nothing anywhere saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the existing invalid_input path. Six call sites migrated, not the two this roadmap predicted — including docs/runbooks/cluster.md §16.3, which told operators to run a quorum-write probe via POST /sharded/items. That probe cannot verify quorum: the surface applies locally with no WAL append. It was used as the safety check between every step of a staged deploy earlier today. Ring 2 — observability. JSON_LOGS was already implemented and the deployment simply never asked for it; the StatefulSet now sets it, plus TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the VictoriaLogs `service` stream field and would have blinded every query keyed on it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only the origin would mean each vector lands on exactly one node, replicas never agree, and the alert built on it pages forever. Found by measurement, not planned: the 401 path discarded every fact about every rejection. Traefik has served 101,858 rejected requests to the public ingress — 87.6% of all its traffic — with no record of who or why anywhere. unauthorized_response now emits reason (missing_token vs invalid_token, the distinction that separates a scanner from a rotation that missed a consumer) and the forwarded client. The token is never logged. Also: scripts/restore-fleet.sh --cluster started the soak monitor while deliberately leaving its gate suspended, orphaning a watcher that has reported "0/30 green nights" for 13 days. The pair now moves together. Doc-guard's three-warning backlog is cleared with real backfill for M4/M6/M12. Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline, counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5; cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with the two remaining failures asserting the rank fix against the not-yet-rolled image — they are the post-deploy proof.	2026-08-30 20:55:58 -06:00
jordan	d21a202a56	docs(runbooks): add an executable deploy verification checklist Every command in it was run against the live orchard9-k3sf deployment and its output recorded before commit. Nothing is aspirational, and the three defects found while dogfooding it are fixed rather than left for the reader: * the backup check sorted ALL backups by timestamp and selected a restore-canary run (20 items, 1 volume) — it would have "passed" while telling you nothing about the fleet. Now filters on the schedule label. * the certificate check dialled the hostname, which fails on a workstation behind a split-DNS resolver. Now connects by IP with SNI. * a prose line was sitting inside a bash fence. Sections 1-8 verify what is deployed today. Section 9 is deliberately separate: HTTP metrics, the operator/data credential split, and structured logs are committed and tested but INERT until an image roll, so their absence is not mistaken for a regression. Five dashboard panels are legitimately empty for the same reason and the doc says which. Carries the two measurement traps this deploy actually produced, because both generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind and reads like a dead node), and pod-to-pod reachability must not be probed with /dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that briefly looked like a cluster partition). Also records the known-red reseed tests as environmental rather than regressions: bisect against the preceding commit shows all three fail identically there, on the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship channels exist, against the harness's own 3s client timeout.	2026-08-23 11:17:01 -06:00
jx12n	25296bcc5b	docs: refresh ops runbooks to the live rc7 / full-placement reality The runbooks had drifted to the retired m8/m11p5 design while all m12 production reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens. This promotes that reality into the runbooks and fixes the contradictions. Contradictions fixed: - runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE (quorum-ack + automatic election have been live since m11p3/p4). Rewritten to state the deployed reality (single-StatefulSet full-placement RF3, rc7). - README.md: the cluster section called the HA cluster a "built-in simulated cluster / multi-region fabric" demo and showed promote-by-region as failover. Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance verb. Kept the honest caveats (experimental gate, global-signals-only). Reality promoted into the runbooks: - Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3, gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest, not the index/attestation digest) in cluster.md + kubernetes.md. - Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added. - ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section (read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources). - ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and- copy note to standalone (the cluster uses tidalctl + the DR runbook). New docs: - runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify -> query-proof procedure, full-cluster rebuild, PITR posture (previously undocumented despite being proven against real S3). - runbooks/on-call.md: incident response — symptom -> golden signal -> runbook, severity, escalation, and the open alert-wiring step. - runbooks/README.md: the runbook index + current production facts. Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but design-reference; promoting it to a live PrometheusRule is the one unwired step.	2026-06-19 19:53:29 -06:00

Author

SHA1

Message

Date

jordan

fe8d0c87e7

harden: restore CI verification, remove four wire-level fabrications, instrument the 401 path

Implements tmp/tidaldb-fleet-hardening (20 planned tasks + 2 found by measurement).

Ring 0 — restore verification. .woodpecker.yaml step pods ran at the namespace
default of 1500m/2Gi, which OOMKilled a prior pipeline and starved the release
gate past its budget. Both push-path steps now declare
backend_options.kubernetes.resources as two YAML anchors declared once on their
first consuming step. The values are CALIBRATED against measured free node
capacity, not against the LimitRange max: `requests: cpu 2` (this roadmap's
original figure) fits on NO node and would sit Pending forever, because
`ci-build-bounds` grants permission and the nodes supply capacity, and those are
not the same thing.

The `nightly` cron described in this file for 216 days was never created, so
tier-3 chaos, the fault classes, mTLS and the PITR test produced exactly zero
signal while reading like standing coverage. nightly-chaos and
nightly-security-ops now alias the anchors and have budgets matching the gate
(their 120/90 were TIGHTER on the same runner, so they would have failed
nightly for a budget reason, not a correctness one). nightly-soak is REMOVED,
not scheduled: it drives 1000 rps for 600s gating on p99 <= 250ms, and the best
node has 1700m free CPU, so it would fail on starvation rather than regression —
manufacturing a nightly false alarm. Its commands move verbatim to
docs/runbooks/nightly-soak.md.

Ring 1 — four fabrications removed from the wire.
- scatter_merge sorted and truncated without re-stamping rank, so /feed and
  /search returned 1,1,2 under full placement. Reuses merge_cross_shard's
  existing stamp; asserted on BOTH the multi-group merge path and the
  single-group [only] fast path that bypasses it.
- aggregate_region_row's None arm invented `applied_events: 0` plus a deficit
  derived from it. applied_events/lag_events are now Option<u64>, null on the
  wire. leader_last_seq was also unwrap_or(0), so a node that could not reach
  the LEADER computed 0 - applied = 0 for every region and reported a converged
  cluster it had never measured — a fabrication pointing the dangerous way.
- tidalctl inferred NO REPORT from `applied == 0 && lag > 0`. That heuristic was
  actively hiding the PVC-wipe shape: a measured zero with a real deficit
  rendered as "no report" instead of BEHIND. Now read off the wire; converged
  exits 0, partitioned still exits nonzero.
- /sharded/* answered 201/204 for single-copy writes with nothing anywhere
  saying so. Now requires `x-tidal-ack: local`, rejecting with 400 via the
  existing invalid_input path. Six call sites migrated, not the two this
  roadmap predicted — including docs/runbooks/cluster.md §16.3, which told
  operators to run a quorum-write probe via POST /sharded/items. That probe
  cannot verify quorum: the surface applies locally with no WAL append. It was
  used as the safety check between every step of a staged deploy earlier today.

Ring 2 — observability. JSON_LOGS was already implemented and the deployment
simply never asked for it; the StatefulSet now sets it, plus
TIDAL_SERVICE_NAME=tidaldb because enabling it silently renames the
VictoriaLogs `service` stream field and would have blinded every query keyed on
it. Adds tidaldb_usearch_replicated_vectors_total, incremented on BOTH the
origin (wal_blob_first -> Ok(Some)) and the follower apply path — counting only
the origin would mean each vector lands on exactly one node, replicas never
agree, and the alert built on it pages forever.

Found by measurement, not planned: the 401 path discarded every fact about
every rejection. Traefik has served 101,858 rejected requests to the public
ingress — 87.6% of all its traffic — with no record of who or why anywhere.
unauthorized_response now emits reason (missing_token vs invalid_token, the
distinction that separates a scanner from a rotation that missed a consumer)
and the forwarded client. The token is never logged.

Also: scripts/restore-fleet.sh --cluster started the soak monitor while
deliberately leaving its gate suspended, orphaning a watcher that has reported
"0/30 green nights" for 13 days. The pair now moves together. Doc-guard's
three-warning backlog is cleared with real backfill for M4/M6/M12.

Verified: fmt clean; clippy 5 crates 0 new warnings (74 vs 74 baseline,
counted in a detached worktree at HEAD); lib 2110 passed; cluster_sharding 5;
cluster_runbook 10; tidalctl 38; doc-guard 0 warnings. Playwright 32/34 with
the two remaining failures asserting the rank fix against the not-yet-rolled
image — they are the post-deploy proof.

2026-08-30 20:55:58 -06:00

jordan

d21a202a56

docs(runbooks): add an executable deploy verification checklist

Every command in it was run against the live orchard9-k3sf deployment and its
output recorded before commit. Nothing is aspirational, and the three defects
found while dogfooding it are fixed rather than left for the reader:

  * the backup check sorted ALL backups by timestamp and selected a
    restore-canary run (20 items, 1 volume) — it would have "passed" while
    telling you nothing about the fleet. Now filters on the schedule label.
  * the certificate check dialled the hostname, which fails on a workstation
    behind a split-DNS resolver. Now connects by IP with SNI.
  * a prose line was sitting inside a bash fence.

Sections 1-8 verify what is deployed today. Section 9 is deliberately separate:
HTTP metrics, the operator/data credential split, and structured logs are
committed and tested but INERT until an image roll, so their absence is not
mistaken for a regression. Five dashboard panels are legitimately empty for the
same reason and the doc says which.

Carries the two measurement traps this deploy actually produced, because both
generated false alarms: port-forward needs sleep 8 (a shorter wait races the bind
and reads like a dead node), and pod-to-pod reachability must not be probed with
/dev/tcp under sh (dash has no /dev/tcp, so an OPEN port reports refused - that
briefly looked like a cluster partition).

Also records the known-red reseed tests as environmental rather than regressions:
bisect against the preceding commit shows all three fail identically there, on
the first ack=quorum write ~1s after the gRPC listeners bind and before peer ship
channels exist, against the harness's own 3s client timeout.

2026-08-23 11:17:01 -06:00

jx12n

25296bcc5b

docs: refresh ops runbooks to the live rc7 / full-placement reality

The runbooks had drifted to the retired m8/m11p5 design while all m12 production
reality (topology, perf, fixes, DR) sat only in a profiling doc no operator opens.
This promotes that reality into the runbooks and fixes the contradictions.

Contradictions fixed:
- runbooks/cluster.md: the "NEITHER IS QUORUM-ACKED HA YET" status banner was FALSE
  (quorum-ack + automatic election have been live since m11p3/p4). Rewritten to
  state the deployed reality (single-StatefulSet full-placement RF3, rc7).
- README.md: the cluster section called the HA cluster a "built-in simulated
  cluster / multi-region fabric" demo and showed promote-by-region as failover.
  Rewritten — real quorum HA, automatic failover, /cluster/promote is a maintenance
  verb. Kept the honest caveats (experimental gate, global-signals-only).

Reality promoted into the runbooks:
- Live topology (single STS, ns tidaldb-cluster, 3 voters, full-placement RF3,
  gRPC 9601/9602/9603, HTTPS+mTLS :9500), the five shipped fixes, and the real
  build+digest-pin procedure (cross-compile -> trixie -> amd64 PLATFORM manifest,
  not the index/attestation digest) in cluster.md + kubernetes.md.
- Stale constants: soak ramp 3900 -> 200 rps; cluster grace 60 -> 600s; the
  pre-m12 4.5k/s signal-write perf table annotated + the 1536-D read reality added.
- ops/capacity-planning.md: new "Ref-A 3-node fleet — measured capacity" section
  (read p99/ceiling, ~250 rps write knee, 1M needs >16GiB nodes, pod resources).
- ops/recovery.md: new cluster-recovery routing section + scoped the quiesce-and-
  copy note to standalone (the cluster uses tidalctl + the DR runbook).

New docs:
- runbooks/disaster-recovery.md: the proven S3/R2 backup -> restore -> byte-verify
  -> query-proof procedure, full-cluster rebuild, PITR posture (previously
  undocumented despite being proven against real S3).
- runbooks/on-call.md: incident response — symptom -> golden signal -> runbook,
  severity, escalation, and the open alert-wiring step.
- runbooks/README.md: the runbook index + current production facts.

Open follow-up (infra, not docs): ops/prometheus-alerts.yaml is accurate but
design-reference; promoting it to a live PrometheusRule is the one unwired step.

2026-06-19 19:53:29 -06:00

3 Commits