tidaldb/.woodpecker.yaml

# Two pipelines in one file, split by event (Woodpecker, NEVER GitHub Actions):
#
#   • event: push  → the m11p8 RELEASE GATE (rolling-upgrade tier-3 test) then a
#     Kaniko image build. Deployment is MANUAL (kustomize, orchard9-k3sf ops repo).
#   • event: cron  → the m11p9 NIGHTLY CONTINUOUS-CORRECTNESS run: the tier-3
#     chaos suites (incl. the disk-full / slow-fsync / asymmetric-partition fault
#     classes) with elevated kill-points, plus the security/ops owner-tests (mTLS,
#     backup/restore round-trip, gap-free WAL archival). A nightly failure flags a
#     correctness regression for that day.
#
# Per-step `when:` routes each step to its event; the workflow-level `when` admits
# both. The cron pipeline requires a cron named "nightly" in the Woodpecker repo
# settings (the same `tidal-server` binary serves standalone AND multi-process
# `cluster --region`, so one image covers both deployments).
#
# ── Honesty note, 2026-08-30 ────────────────────────────────────────────────────
# For 216 days this file described a nightly correctness programme that had NEVER
# RUN: the "nightly" cron was never created, so four pipelines existed in total
# and all were push events. Tier-3 chaos, the fault classes, the soak's p99 and
# error-rate gates, mTLS and the PITR test produced exactly ZERO signal, while
# reading to anyone opening this file like rigorous standing coverage. A gate
# nobody runs is worse than no gate, because it makes the project look covered.
#
# The nightly SOAK step was REMOVED rather than scheduled. It is not a capacity
# quibble — it is unfalsifiable here. Measured free capacity on the best node is
# 1700m CPU (agent-1; server-1/2 have 435m/550m), and the step drove 1000 rps for
# 600s while gating on p99 ≤ 250ms. On that hardware the gate fails from CPU
# starvation, not from a regression, so scheduling it would manufacture a nightly
# false alarm — the same fake-coverage defect inverted. Its command list now lives
# in docs/runbooks/nightly-soak.md as a pre-release step, run where the capacity
# to make its numbers mean something actually exists.
when:
  branch: main
  event: [push, cron]

# CARGO_INCREMENTAL=0 for every Rust step. Incremental compilation is pure waste
# here: Woodpecker gives each workflow a FRESH 10Gi workspace PVC, so nothing is
# ever reused across pipelines, and the artifacts are enormous. Measured locally,
# `target/debug/incremental` was 11G of a 27G target tree — 41%.
#
# It is also what broke pipeline #8. The 10Gi workspace held the release gate
# alone (pipelines #5 and #6 both passed), but once `fast-suites` built eight test
# binaries ahead of it, the gate's nested `cargo build --features fault-injection`
# ran out of room and failed with its stderr discarded, which is why the failure
# said nothing (now fixed in tests/support/multiproc.rs).

steps:
  # ── Release gate (push) ─────────────────────────────────────────────────────
  # m11p8 release gate: prove a rolling upgrade under load loses no acknowledged
  # write and never stalls (mp_rolling_upgrade_no_loss_no_stall — a tier-3 test
  # spawning three real OS processes with a graceful SIGTERM → version-tagged
  # restart → heal-until-converged → fixpoint cycle). Serial (--test-threads 1):
  # the harness binds fixed ports and spawns real processes, so suites must not
  # overlap. A failure here BLOCKS the image build below — the gate, not the start.
  rolling-upgrade-gate:
    image: rust:1-bookworm
    when:
      event: push
    # ── Resource shape: HEAVY (declared once here, aliased by later steps) ─────
    # This step spawns three real tidal-server processes, each with its own WAL,
    # HNSW index, gRPC transport and tokio runtime. Under the namespace default
    # (`ci-build-bounds`: limits 1500m/2Gi, requests 50m/128Mi) a prior pipeline
    # pod was OOMKilled outright at 2Gi, and convergence starved past 12 min on
    # 1.5 CPU against 21s locally.
    #
    # CALIBRATED 2026-08-30 against real node free capacity, NOT against the
    # LimitRange max. `ci-build-bounds` permits up to cpu 3 / memory 6Gi, but
    # permission is not capacity — measured free requests were agent-1 1700m/4193Mi,
    # server-1 435m/2309Mi, server-2 550m/2257Mi. A `requests: cpu: "2"` step pod
    # (the value this file's roadmap originally specified) fits on NO node and
    # would sit Pending forever, which is a worse failure than being slow.
    #
    # So: requests are sized to actually schedule (fits agent-1 with ~700m CPU and
    # ~2.1Gi memory of slack), while limits take the full LimitRange max. Limits do
    # not affect scheduling, and the 6Gi limit is what eliminates the OOMKill; the
    # 3 CPU limit doubles the old burst ceiling. Raising the LimitRange itself is
    # explicitly NOT the fix here.
    backend_options: &resources-heavy
      kubernetes:
        resources:
          requests: { cpu: "1", memory: 2Gi }
          limits: { cpu: "3", memory: 6Gi }
    # Budget headroom, matching the nightly steps below. The defaults are 60s boot
    # / 30s convergence (support/multiproc.rs:54,62), tuned for a developer
    # machine; this step spawns three real OS processes, drives a graceful
    # SIGTERM → version-tagged restart → heal cycle, and then waits for three-way
    # feed parity to 1e-6 over loopback gRPC.
    #
    # Measured 2026-08-30: at the default budget this test times out on
    # "WAL relay alone must reconverge all three nodes to 1e-6" on a loaded
    # machine, and passes in 21s with the raised budget — the convergence itself
    # is fast, the default just leaves no slack. It reproduced identically on a
    # pre-change baseline, so it is budget sensitivity and not a regression.
    #
    # This was the ONLY push-path step without headroom, while every nightly step
    # already sets it "for a shared CI runner" — and this is the step that BLOCKS
    # the image build, so its flake cost is the highest of any step in the file.
    environment: &cargo_env
      # Declared ONCE here (first consumer) and aliased by every later Rust step.
      # Incremental artifacts are pure waste in CI — each workflow gets a fresh
      # 10Gi workspace PVC so nothing is ever reused, and they measured 11G of a
      # 27G target tree locally.
      CARGO_INCREMENTAL: "0"
      CARGO_TERM_COLOR: never
      TIDAL_TEST_BOOT_BUDGET_SECS: "300"
      TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      - cargo test -p tidal-server --features cluster-e2e --test cluster_lifecycle
        mp_rolling_upgrade_no_loss_no_stall -- --nocapture --test-threads 1

  build:
    image: woodpeckerci/plugin-kaniko
    when:
      event: push
    # ── Resource shape: LIGHT (declared once here, aliased by later steps) ─────
    # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
    # A build does not need the heavy shape's CPU floor, so it is sized separately
    # to leave headroom for anything co-scheduled on the same node.
    # ── Resource shape: LIGHT (declared once here, aliased by later steps) ─────
    # Kaniko build: single process, but it OOMKilled at the 2Gi namespace default.
    #
    # NOTE on anchoring: both shapes are declared on their FIRST CONSUMING STEP
    # rather than a top-level `variables:` key, because the file is schema-validated
    # and `when: branch: main` means a rejected key could not be caught on a
    # throwaway branch — it would break every push. The cost of that choice is that
    # deleting a defining step breaks its alias, which happened once on 2026-08-31
    # when the step holding this anchor moved to the nightly. If you remove a step,
    # check whether it defines an anchor.
    backend_options: &resources-light
      kubernetes:
        resources:
          requests: { cpu: "500m", memory: 1Gi }
          limits: { cpu: "2", memory: 4Gi }
    settings:
      repo: tidal/server
      dockerfile: docker/standalone/Dockerfile
      context: .
      # Tag by immutable identity only: `latest` plus the commit SHA (the durable
      # handle). A per-milestone literal tag drifts every milestone and lies about
      # the image's vintage (a hand-edited `m8p10` once tagged m11 code) — the SHA
      # never goes stale and the version-skew machinery keys off the binary's
      # CARGO_PKG_VERSION/BUILD_HASH, not the image tag.
      tags:
        - latest
        - ${CI_COMMIT_SHA}
      registry: registry.threesix.ai
      build_args:
        - TARGETPLATFORM=linux/amd64
      extra_args: --customPlatform=linux/amd64

  # ── Nightly chaos (cron) ────────────────────────────────────────────────────
  # The tier-3 chaos/correctness suites over REAL OS processes, run serially
  # (fixed ports + spawned processes must not overlap). Kill-points are elevated
  # above the per-PR defaults (8/5) but kept below the 100-point exit-gate run so
  # the nightly stays bounded; widen TIDAL_*_KILLPOINTS for a deeper sweep. The
  # `fault-injection` feature compiles in the WAL slow-fsync / disk-full hooks for
  # cluster_faults (inert in every other suite — see tidal/src/fault.rs). Boot /
  # convergence budgets are raised for a shared CI runner. A failure = a
  # correctness regression for the night.
  nightly-chaos:
    image: rust:1-bookworm
    when:
      event: cron
      cron: nightly
    # Same three-process shape as the release gate, so the same resource shape.
    backend_options: *resources-heavy
    environment:
      CARGO_INCREMENTAL: "0"
      CARGO_TERM_COLOR: never
      TIDAL_QUORUM_KILLPOINTS: "25"
      TIDAL_ELECTION_KILLPOINTS: "15"
      # Raised from 120/90 to match the release gate. These budgets were TIGHTER
      # than the gate's on the same constrained runner while running the same
      # three-process suites, so the step would have failed nightly for the
      # identical budget-sensitivity reason measured on 2026-08-30 — not for a
      # correctness regression, which is the only thing this step should ever
      # report.
      TIDAL_TEST_BOOT_BUDGET_SECS: "300"
      TIDAL_TEST_CONVERGENCE_BUDGET_SECS: "180"
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      # New m11p9 fault classes first (disk-full, slow-fsync, asymmetric partition).
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_faults -- --nocapture --test-threads 1
      # The standing chaos / durability / availability / elasticity gates.
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_chaos -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_quorum -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_election -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_membership -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_reseed -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_lifecycle -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_runbook -- --nocapture --test-threads 1
      # ADDED 2026-08-31. These six spawn real 3-process clusters and were run by
      # NO pipeline before today — including `cluster_sharding`, which holds the
      # dense-rank and /sharded/* opt-in gates, and `cluster_poison_embedding`,
      # the regression gate for the malformed-embedding outage. Guards nothing
      # runs are guards that rot: `cluster_routes` proved it by still asserting a
      # wire fabrication that had been removed hours earlier.
      #
      # They live in the nightly rather than the push path because each boots
      # three OS processes; the eight in-process suites moved to `fast-suites`
      # above, which costs 71s and runs on every push.
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_sharding -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_poison_embedding -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_cross_shard_reads -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_graph_persistence -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_multiproc -- --nocapture --test-threads 1
      - cargo test -p tidal-server --features "cluster-e2e fault-injection" --test cluster_e2e -- --nocapture --test-threads 1

  # ── Nightly security + ops correctness (cron) ───────────────────────────────
  # The G-Sec and G-Op owner-tests run nightly too (not just on-demand): mTLS /
  # foreign-pod rejection / cert rotation (tidal-net mtls + cluster_security), the
  # tidalctl backup/restore round-trip, and the gap-free WAL-archival (PITR) unit
  # test (in the engine lib). In-process + fast; serial for the TLS port binds.
  nightly-security-ops:
    image: rust:1-bookworm
    when:
      event: cron
      cron: nightly
    # In-process and fast; the light shape is sufficient.
    backend_options: *resources-light
    environment: *cargo_env
    commands:
      - apt-get update && apt-get install -y --no-install-recommends protobuf-compiler cmake clang
      - cargo test -p tidal-net --test mtls -- --test-threads 1
      - cargo test -p tidal-server --test cluster_security -- --test-threads 1
      - cargo test -p tidal-server --lib
      - cargo test -p tidalctl
      - cargo test -p tidaldb --lib wal::compaction
      # ADDED 2026-08-31. These eight were run by NO pipeline before today, which
      # is how `cluster_routes` kept asserting a wire fabrication removed hours
      # earlier, and how the dense-rank (`cluster_sharding`) and vector
      # distance-contract (`vector_search`) guards came to sit unexecuted.
      #
      # They were briefly on the PUSH path and are deliberately here instead.
      # Pipelines #8 and #9 proved why: running them first burns ~872s of
      # CPU-saturating rustc, and the release gate immediately after then failed on
      # "WAL relay alone must reconverge all three nodes to 1e-6" — the exact
      # budget sensitivity this file already documents at the gate. The gate's
      # 300s/180s budgets are calibrated for a node that is NOT fresh off fifteen
      # minutes of parallel compilation.
      #
      # Raising the gate's budget to accommodate them would be loosening a measured
      # threshold to hide load I introduced. Blocking the image build is the gate's
      # job and it matters more than fifteen-minutes-faster feedback on in-process
      # suites, so the suites moved and the gate kept its calibration. Coverage
      # still went from never to nightly.
      - cargo test -p tidal-server --test middleware
      - cargo test -p tidal-server --test standalone
      - cargo test -p tidal-server --test standalone_offload
      - cargo test -p tidal-server --test vector_search
      - cargo test -p tidal-server --test cluster_routes
      - cargo test -p tidal-server --test cluster_region
      - cargo test -p tidal-server --test cluster_grpc
      - cargo test -p tidal-server --test reseed_install