My backup script was fine for a year. Then it quietly rsynced a live disk.

A backup that verified clean but restored to nothing. How a hardcoded path, a hung shutdown, and a too-eager verify step combined into a 45-minute failure and the staging/promote pattern that fixed it.

For about a year, my KVM backup script did its job. Every Wednesday and Sunday at 01:00 it shut the guests down, copied the disk images, verified them, and moved on. I slept fine.

Then one Sunday it produced nothing, told me nothing useful, and burned 45 minutes doing it. The worst part wasn't the failure. It was that the script thought it succeeded.

Here's what happened, why the verification I trusted was lying, and the pattern I now use instead.

The Setup:

A small KVM host running three guests: two Windows Server boxes and a Linux mail server. Total disk footprint a few hundred GB. Nothing exotic. The backup logic was a bash script fired by cron:

Loop over a hardcoded list of VMs
`virsh shutdown` each one
Wait for it to stop
`rsync` its qcow2 to a dated directory
Verify the copy
Start the VMs back up

The verification step was the one I was proud of. After each copy it checked three things: the destination file existed, its size was greater than zero, and its size matched the source. If all three passed, the backup was good.

That's the sentence I want you to re-read. If all three passed, the backup was good.

The night it fell apart

The shutdown step hung. One guest a Windows Server 2003 box, old enough that it has no working guest agent ignored the ACPI shutdown request the way it always had. I'd seen this before and it usually resolved within a couple of minutes, so nobody had bothered to add a timeout.

Except this time it didn't resolve. The script sat there. Eventually the cron wrapper moved on to the next phase, the shutdown wait expired in a way I hadn't designed for, and the copy phase started against a guest that was still running.

A live qcow2 is not a file you can meaningfully `rsync`. It's being written to while you copy it. The bytes shift under you. You get a file that is the right size, from a source that was the right size, and both of those facts are true and meaningless.

The verify step passed. Of course it passed. The file existed, the size was greater than zero, and the size matched the source. It was also garbage.

I found out weeks later when I tried to use the backup. That's the part that keeps me up at night not the failed run, but how long the failure was invisible.

Why the verify step lied

My verification checked the properties of the copy, not its validity. A 298 GB file that is the wrong 298 GB passes every size check ever written.

This is the trap with backups, and it's easy to fall into because the honest answer mount it and check it is expensive. Full restore tests are the only real verification, and nobody runs them every night. So we invent cheaper proxies, and those proxies have a failure mode: they confirm that a copy exists, not that a copy is usable.

The size check has one more problem in this specific case. `rsync` against a live qcow2 doesn't produce a smaller file. It produces a same-sized file full of inconsistent data. So the check that was supposed to catch corruption was structurally incapable of catching it.

The fix, and the thinking behind it

I rewrote the script around a few rules. None of them are clever. All of them are the result of getting this wrong once.

1. Shutdown is mandatory, and it has a timeout.

Every guest gets a `virsh shutdown`, then a wait with a hard limit. If the timeout is hit, the run aborts it does not continue to the copy phase. The abort writes a marker file and logs a line tagged for `journalctl`, so the failure is visible even if the notification email never arrives.

SHUTDOWN_TIMEOUT=300
if ! wait_for_stop "$vm" "$SHUTDOWN_TIMEOUT"; then
    logger -t vm-backup "ABORT: $vm did not stop within ${SHUTDOWN_TIMEOUT}s"
    die "shutdown timeout for $vm"
fi

The die function does one more thing that took a live failure to prove out: it removes the staging directory it had been building, so a failed run leaves no half-written litter behind.

die() {
    logger -t vm-backup "ABORT: $*"
    touch /var/log/vm-backup-ABORT
    rmdir "$STAGE" 2>/dev/null || true
    exit 1
}

2. Write to staging, promote on success.

The script no longer copies straight into the backup directory. It builds the whole backup in a staging directory, and only when every disk of every VM has copied and verified does it move the staged directory into place and flip a single current symlink.

STAGE="/kvmdata/backup/.staging-$(date +%Y%m%d_%H%M%S)"
FINAL="/kvmdata/backup/backup-$(date +%Y%m%d_%H%M%S)"

# ... copy into $STAGE, verify each disk ...

mv "$STAGE" "$FINAL"
ln -sfn "$FINAL" /kvmdata/backup/current

This one change is worth more than the rest combined. A backup is a symlink target. A failed run never touches the symlink, so current always points at the last fully-successful backup, no matter what happened at 01:00. The host reboots for kernel updates every few days; a reboot mid-backup now leaves the last good backup completely untouched.

3. Verify per disk, and fail the whole run on any failure.

The verify loop now checks each disk individually: the basename is present in the destination, the size is greater than zero, and the source and destination sizes match. Any single failure aborts the run. It's still a size check I'm not going to pretend otherwise but it runs against a stopped guest, which is the difference between a meaningful check and a formality.

4. Discover disks, don't hardcode them.

The original script hardcoded $vm.qcow2. That's how a disk gets silently skipped when someone adds a second volume. I replaced it with a small helper that asks libvirt for the guest's actual disks, resolves pool volumes, and excludes cdrom/ISO devices:

/usr/local/bin/get-vm-disks "$vm" | while read -r disk; do
    rsync -a --sparse --numeric-ids --info=stats2 "$disk" "$STAGE/"
    verify_disk "$disk" "$STAGE"
done

5. Retention only ever touches promoted backups.

Cleanup lists backup-* directories, sorts them, and removes everything past the keep count. Manually created copies (for example, a hand-made cold copy taken before a risky change) are named differently, so retention can't touch them. Keep count is one higher than the number of runs you want on disk six kept means seven directories, because head -n -$((KEEP+1)) is the actual math, not -KEEP.

What I'd do differently

Three things, in order of how much they'd have helped:

I'd have treated "the guest didn't shut down" as a hard error from day one. The most expensive decision in this whole story was letting the script continue past a hung shutdown. A backup of a running VM is not a backup; it's a busy file. The script should have refused, loudly.
I'd have understood that my verify step was a liveness check, not a correctness check. It's fine to have one. It's not fine to believe it. The honest mental model is: this step proves the copy happened, and nothing more. Real proof is a restore test, and I schedule those separately now.
I'd have made failure visible outside the notification path. Email is the easiest thing to lose. Logging to the journal with its own tag, plus an abort marker file, means the failure survives even if the mail never leaves the box.

None of this is novel. The staging/promote pattern is older than I am. But there's a difference between knowing the pattern and having had a backup you trusted turn out to be a same-sized lie. If your script copies qcow2 files, check one thing right now: does it abort if the guest didn't actually stop? If the answer is I think so, go look. It's the cheapest insurance you'll ever buy.