diff --git a/docs/runbooks/deploy-verification.md b/docs/runbooks/deploy-verification.md index fc98b95..c61b114 100644 --- a/docs/runbooks/deploy-verification.md +++ b/docs/runbooks/deploy-verification.md @@ -1,5 +1,22 @@ # Deploy verification +> ## Do NOT deploy between 03:30 and 04:00 UTC +> +> `velero-fleet-daily` fires at **03:30 UTC** and takes **9–25 min** (measured over 15 successful runs). +> A rolling restart inside that window **cancels the in-flight volume backup for every pod it restarts**, and +> the parent Velero `Backup` then stalls indefinitely — it does not fail, it sits `InProgress` and blocks the +> next scheduled run. +> +> Observed 2026-08-31: a roll at 03:50/03:55/03:56 cancelled +> `podvolumebackup …-wgtkt` (`pod=tidaldb-0 volume=data`) at 2.7 GB of 5.4 GB. The Backup froze at +> **3529/3907 items** and was still frozen 67 minutes later, heading for the 240-minute timeout that produces +> `PartiallyFailed`. The same fingerprint — Canceled/Failed PVBs clustered on one date — explains the +> `PartiallyFailed` runs on 2026-08-17, 08-18, 08-19 and 08-25. +> +> If you have already done it: confirm the previous day's backup is `Completed` with full item coverage, then +> delete the stalled Backup so the schedule unblocks. Do not wait for the timeout. + + Walk this top to bottom. Every command here was executed against the live `orchard9-k3sf` deployment and its output recorded — nothing is aspirational.