jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: never deploy inside the Velero backup window
Some checks failed
ci/woodpecker/push/woodpecker Pipeline was successful
Details
ci/woodpecker/cron/woodpecker Pipeline failed
Details

Browse Source 
Found by violating it. The staged roll landed at 03:50/03:55/03:56 UTC;
velero-fleet-daily fires at 03:30 and takes 9-25 min. Restarting tidaldb-0
cancelled its own in-flight volume backup (podvolumebackup ...-wgtkt,
pod=tidaldb-0 volume=data) at 2.7 GB of 5.4 GB, and the parent Backup froze at
3529/3907 items. It did not fail - it sat InProgress for 67+ minutes heading for
the 240-minute timeout that produces PartiallyFailed, and a stalled Backup blocks
the next scheduled run.

That is almost certainly the explanation for the PartiallyFailed runs on
2026-08-17/18/19 and 08-25: the namespace holds 4 Canceled and 4 Failed PVBs, all
clustered on exactly those dates.

The hazard is invisible from both sides - nothing in the deploy path mentions
Velero and nothing in the Velero config mentions deploys - so the warning goes at
the TOP of the deploy runbook rather than in a section nobody reaches.

Note what caught it: the recalibrated 60-minute in-flight bound from the previous
commit. The old binary "any InProgress fails" assertion would have been red every
day during the normal window, so a real stall would have looked like the usual
noise. Resolved by confirming 20260830033034 was Completed at 3707/3707 and all
four PVCs Bound, then deleting the stalled Backup. Suite 34/34.
This commit is contained in:
jordan 2026-08-30 22:41:02 -06:00
parent 77f68d181c
commit 431340fc34
1 changed files with 17 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
docs/runbooks/deploy-verification.md
View File

@ -1,5 +1,22 @@
# Deploy verification # Deploy verification
> ## Do NOT deploy between 03:30 and 04:00 UTC
>
> `velero-fleet-daily` fires at **03:30 UTC** and takes **9–25 min** (measured over 15 successful runs).
> A rolling restart inside that window **cancels the in-flight volume backup for every pod it restarts**, and
> the parent Velero `Backup` then stalls indefinitely — it does not fail, it sits `InProgress` and blocks the
> next scheduled run.
>
> Observed 2026-08-31: a roll at 03:50/03:55/03:56 cancelled
> `podvolumebackup …-wgtkt` (`pod=tidaldb-0 volume=data`) at 2.7 GB of 5.4 GB. The Backup froze at
> **3529/3907 items** and was still frozen 67 minutes later, heading for the 240-minute timeout that produces
> `PartiallyFailed`. The same fingerprint — Canceled/Failed PVBs clustered on one date — explains the
> `PartiallyFailed` runs on 2026-08-17, 08-18, 08-19 and 08-25.
>
> If you have already done it: confirm the previous day's backup is `Completed` with full item coverage, then
> delete the stalled Backup so the schedule unblocks. Do not wait for the timeout.
Walk this top to bottom. Every command here was executed against the live Walk this top to bottom. Every command here was executed against the live
`orchard9-k3sf` deployment and its output recorded — nothing is aspirational. `orchard9-k3sf` deployment and its output recorded — nothing is aspirational.