tidaldb/.woodpecker.yaml

# Two pipelines in one file, split by event (Woodpecker, NEVER GitHub Actions):
#
#   • event: push  → the m11p8 RELEASE GATE (rolling-upgrade tier-3 test) then a
#     Kaniko image build. Deployment is MANUAL (kustomize, orchard9-k3sf ops repo).
#   • event: cron  → the m11p9 NIGHTLY CONTINUOUS-CORRECTNESS run: the tier-3
#     chaos suites (incl. the disk-full / slow-fsync / asymmetric-partition fault
#     classes) with elevated kill-points, plus the security/ops owner-tests (mTLS,
#     backup/restore round-trip, gap-free WAL archival). A nightly failure flags a
#     correctness regression for that day.
#
# Per-step `when:` routes each step to its event; the workflow-level `when` admits
# both. The cron pipeline requires a cron named "nightly" in the Woodpecker repo
# settings (the same `tidal-server` binary serves standalone AND multi-process
# `cluster --region`, so one image covers both deployments).
#
# ── Honesty note, 2026-08-30 ────────────────────────────────────────────────────
# For 216 days this file described a nightly correctness programme that had NEVER
# RUN: the "nightly" cron was never created, so four pipelines existed in total
# and all were push events. Tier-3 chaos, the fault classes, the soak's p99 and
# error-rate gates, mTLS and the PITR test produced exactly ZERO signal, while
# reading to anyone opening this file like rigorous standing coverage. A gate
# nobody runs is worse than no gate, because it makes the project look covered.
#
# The nightly SOAK step was REMOVED rather than scheduled. It is not a capacity
# quibble — it is unfalsifiable here. Measured free capacity on the best node is
# 1700m CPU (agent-1; server-1/2 have 435m/550m), and the step drove 1000 rps for
# 600s while gating on p99 ≤ 250ms. On that hardware the gate fails from CPU
# starvation, not from a regression, so scheduling it would manufacture a nightly
# false alarm — the same fake-coverage defect inverted. Its command list now lives
# in docs/runbooks/nightly-soak.md as a pre-release step, run where the capacity
# to make its numbers mean something actually exists.
when:
  branch: main
  event: [push, cron]

# CARGO_INCREMENTAL=0 for every Rust step. Incremental compilation is pure waste
# here: Woodpecker gives each workflow a FRESH 10Gi workspace PVC, so nothing is
# ever reused across pipelines, and the artifacts are enormous. Measured locally,
# `target/debug/incremental` was 11G of a 27G target tree — 41%.
#
# ── Why the release gate is no longer on the push path (2026-08-31) ────────────
# `mp_rolling_upgrade_no_loss_no_stall` gated the image build from m11p8 until
# today. It is MARGINAL on this hardware and has now been measured as such:
# pipelines #5 and #6 passed, #9 and #10 failed, all four on the same 3 CPU / 6Gi
# step and all four failing identically at
#
#   cluster_lifecycle.rs:362  timed out: WAL relay alone must reconverge all
#                             three nodes to 1e-6 after the rolling upgrade
#
# The same test on the same commit passes locally in 19.61s. It spawns three real
# tidal-server processes, each with its own WAL, HNSW index, gRPC transport and
# tokio runtime, and asks them to reconverge to 1e-6 inside 180s — on a node with
# ~1700m of free CPU shared with the production tidalDB cluster. Two passes and
# two failures is a coin flip, and a coin flip that blocks image builds is worse
# than no gate: it teaches everyone to re-run until it goes green, which is how a
# gate stops being one.
#
# Raising the budget again would be loosening a measured threshold to hide the
# hardware, so it was not done. This is the escalation the roadmap's own task 01
# prescribed: a tier-3 three-process test does not belong on a 4-CPU shared node,
# so it becomes a pre-release step run where it demonstrably passes
# (docs/runbooks/deploy-verification.md), and it still runs nightly inside
# `cluster_lifecycle` with the same budgets, where a flake costs a re-read of the
# morning report instead of a blocked release.
#
# What gates the image now is `fast-suites`: 51 tests across nine deterministic
# in-process suites, no spawned processes, no timing budget to starve. It catches
# the class of regression that actually reached main today — `cluster_routes`
# asserting a wire fabrication that had been deleted hours earlier.

steps:
  # ── Push gate: deterministic in-process suites ──────────────────────────────
  # These nine suites hold the guards that matter for a release and none of them
  # spawn processes or depend on a convergence budget, so their result means the
  # same thing on a loaded shared node as on a workstation. Measured 872s in CI
  # (mostly cold-cache rustc) and 71s locally.
  #
  # Fourteen of the 23 integration suites were run by NO pipeline before today.
  # That is how `cluster_routes` kept asserting `applied_events == 0` with a lag
  # derived from it — a fabrication deleted earlier the same day — and how the
  # dense-rank (`cluster_sharding`) and vector distance-contract (`vector_search`)
  # guards came to sit unexecuted. A failure here BLOCKS the image build below.
  fast-suites:
    image: rust:1-bookworm
    when:
      event: push
    backend_options: &resources-light
      kubernetes:
        resources:
          requests: { cpu: "500m", memory: 1Gi }
          limits: { cpu: "2", memory: 4Gi }
    environment: &cargo_env
      CARGO_INCREMENTAL: "0"
      CARGO_TERM_COLOR: never
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      - cargo test -p tidaldb --lib
      - cargo test -p tidal-server --test middleware
      - cargo test -p tidal-server --test standalone
      - cargo test -p tidal-server --test standalone_offload
      - cargo test -p tidal-server --test vector_search
      - cargo test -p tidal-server --test cluster_routes
      - cargo test -p tidal-server --test cluster_region
      - cargo test -p tidal-server --test cluster_grpc
      - cargo test -p tidal-server --test reseed_install

  build:
    image: woodpeckerci/plugin-kaniko
    when:
      event: push
    backend_options: *resources-light
    settings:
      repo: tidal/server
      dockerfile: docker/standalone/Dockerfile
      context: .
      # Tag by immutable identity only: `latest` plus the commit SHA (the durable
      # handle). A per-milestone literal tag drifts every milestone and lies about
      # the image's vintage (a hand-edited `m8p10` once tagged m11 code) — the SHA
      # never goes stale and the version-skew machinery keys off the binary's
      # CARGO_PKG_VERSION/BUILD_HASH, not the image tag.
      tags:
        - latest
        - ${CI_COMMIT_SHA}
      registry: registry.threesix.ai
      build_args:
        - TARGETPLATFORM=linux/amd64
      extra_args: --customPlatform=linux/amd64

  # ── Nightly chaos (cron) ────────────────────────────────────────────────────
  # The tier-3 chaos/correctness suites over REAL OS processes, run serially
  # (fixed ports + spawned processes must not overlap). Kill-points are elevated
  # above the per-PR defaults (8/5) but kept below the 100-point exit-gate run so
  # the nightly stays bounded; widen TIDAL_*_KILLPOINTS for a deeper sweep. The
  # `fault-injection` feature compiles in the WAL slow-fsync / disk-full hooks for
  # cluster_faults (inert in every other suite — see tidal/src/fault.rs). Boot /
  # convergence budgets are raised for a shared CI runner. A failure = a
  # correctness regression for the night.
  nightly-chaos:
    image: rust:1-bookworm
    when:
      event: cron
      cron: nightly
    # The HEAVY shape, written INLINE because this is now its only consumer.
    #
    # It was a `&resources-heavy` anchor defined on the push release gate. Moving
    # that gate off the push path broke this alias — the third time in one session
    # that relocating a step broke an anchor defined on it (resources-light and
    # cargo_env were the others). First-consumer anchoring avoids a schema-risky
    # top-level key but makes step order load-bearing in a way nothing warns about.
    # With one consumer an anchor buys nothing, so there is none.
    #
    # These values ARE the `ci-build-bounds` LimitRange max (cpu 3 / memory 6Gi),
    # not a request to raise it, and the requests are sized to actually schedule:
    # measured free capacity was agent-1 1700m, server-1 435m, server-2 550m, so a
    # `requests: cpu: "2"` pod fits on no node and would sit Pending forever.
    backend_options:
      kubernetes:
        resources:
          requests: { cpu: "1", memory: 2Gi }
          limits: { cpu: "3", memory: 6Gi }
    environment:
      CARGO_INCREMENTAL: "0"
      CARGO_TERM_COLOR: never
      TIDAL_QUORUM_KILLPOINTS: "25"
      TIDAL_ELECTION_KILLPOINTS: "15"
      # Raised from 120/90 to match the release gate. These budgets were TIGHTER
      # than the gate's on the same constrained runner while running the same
      # three-process suites, so the step would have failed nightly for the
      # identical budget-sensitivity reason measured on 2026-08-30 — not for a
      # correctness regression, which is the only thing this step should ever
      # report.
      TIDAL_TEST_BOOT_BUDGET_SECS: "300"
      TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      # New m11p9 fault classes first (disk-full, slow-fsync, asymmetric partition).
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_faults -- --nocapture --test-threads 1
      # The standing chaos / durability / availability / elasticity gates.
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_chaos -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_quorum -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_election -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_membership -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_reseed -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_lifecycle -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_runbook -- --nocapture --test-threads 1
      # ADDED 2026-08-31. These six spawn real 3-process clusters and were run by
      # NO pipeline before today — including `cluster_sharding`, which holds the
      # dense-rank and /sharded/* opt-in gates, and `cluster_poison_embedding`,
      # the regression gate for the malformed-embedding outage. Guards nothing
      # runs are guards that rot: `cluster_routes` proved it by still asserting a
      # wire fabrication that had been removed hours earlier.
      #
      # They live in the nightly rather than the push path because each boots
      # three OS processes; the eight in-process suites moved to `fast-suites`
      # above, which costs 71s and runs on every push.
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_sharding -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_poison_embedding -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_cross_shard_reads -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_graph_persistence -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_multiproc -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_e2e -- --nocapture --test-threads 1

  # ── Nightly security + ops correctness (cron) ───────────────────────────────
  # The G-Sec and G-Op owner-tests run nightly too (not just on-demand): mTLS /
  # foreign-pod rejection / cert rotation (tidal-net mtls + cluster_security), the
  # tidalctl backup/restore round-trip, and the gap-free WAL-archival (PITR) unit
  # test (in the engine lib). In-process + fast; serial for the TLS port binds.
  nightly-security-ops:
    image: rust:1-bookworm
    when:
      event: cron
      cron: nightly
    # In-process and fast; the light shape is sufficient.
    backend_options: *resources-light
    environment: *cargo_env
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      - cargo test -p tidal-net --test mtls -- --test-threads 1
      - cargo test -p tidal-server --test cluster_security -- --test-threads 1
      - cargo test -p tidal-server --lib
      - cargo test -p tidalctl
      - cargo test -p tidaldb --lib wal::compaction
      # ADDED 2026-08-31. These eight were run by NO pipeline before today, which
      # is how `cluster_routes` kept asserting a wire fabrication removed hours
      # earlier, and how the dense-rank (`cluster_sharding`) and vector
      # distance-contract (`vector_search`) guards came to sit unexecuted.
      #
      # They were briefly on the PUSH path and are deliberately here instead.
      # Pipelines #8 and #9 proved why: running them first burns ~872s of
      # CPU-saturating rustc, and the release gate immediately after then failed on
      # "WAL relay alone must reconverge all three nodes to 1e-6" — the exact
      # budget sensitivity this file already documents at the gate. The gate's
      # 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen
      # minutes of parallel compilation.
      #
      # Raising the gate's budget to accommodate them would be loosening a measured
      # threshold to hide load I introduced. Blocking the image build is the gate's
      # job and it matters more than fifteen-minutes-faster feedback on in-process
      # suites, so the suites moved and the gate kept its calibration. Coverage
      # still went from never to nightly.
      - cargo test -p tidal-server --test middleware
      - cargo test -p tidal-server --test standalone
      - cargo test -p tidal-server --test standalone_offload
      - cargo test -p tidal-server --test vector_search
      - cargo test -p tidal-server --test cluster_routes
      - cargo test -p tidal-server --test cluster_region
      - cargo test -p tidal-server --test cluster_grpc
      - cargo test -p tidal-server --test reseed_install