One guest on a mixed-age KVM host never responds to ACPI shutdown. No guest agent is installable. Overlay snapshots were ruled out after a data-loss experiment. Here's the destroy fallback I ended up accepting with the guardrails that make it safe to run unattended.
Every KVM host I've run has one guest like this. It's old, it works, nobody wants to touch it, and it does not respond to a polite shutdown request. On a backup schedule, that's not a quirk. That's the difference between a backup that completes at 01:00 and one that doesn't complete at all.
This is the story of a Windows Server 2003 guest on a host with two modern Windows guests and a Linux mail server, why every "proper" fix was off the table, and the destroy-based fallback I ended up accepting plus the guardrails that keep it from becoming the catastrophic version of itself.
The problem, concretely
The backup script needs guests stopped before it copies their disks. That's non-negotiable; I learned the hard way that copying a live qcow2 gives you a same-sized file that's a lie (that's a different post). So the script sends a shutdown to each guest and waits with a timeout.
Three guests stop. One doesn't.
The guest is a Windows Server 2003 box 32-bit, older than most of my tooling. When the script sends `virsh shutdown`, which is an ACPI shutdown request, the guest ignores it. I've waited the full five-minute timeout more than once. It never comes down.
Modern guests on the same host stop cleanly in seconds. This one never does. Not a "sometimes" every single time, for as long as I've been tracking it.
What I tried, and why each failed
I worked through the obvious options before accepting the ugly one, and it's worth writing down why each failed so the next person doesn't repeat the walk.
Guest agent. The clean answer is a qemu-guest-agent that lets the host ask the guest to shut down through a channel that doesn't depend on ACPI. The agent channel is present in the guest's libvirt XML I can see it in the inactive config. The problem is the agent itself can't be installed. This is Windows Server 2003 32-bit, and none of the virtio driver ISOs I have contain the 2003/x86 driver set. The modern agent packages don't support an OS this old. I went looking, and the driver simply isn't there to install. Dead end, and a confirmed one not a "didn't try hard enough."
Overlay snapshots. This is the answer a lot of people would reach for: take a disk snapshot, back up the base image without stopping the guest, done. In theory it avoids the shutdown entirely. In practice, on this guest, an earlier experiment with this approach lost several days of user data. I'm not going to detail the mechanism here because the lesson isn't the mechanism the lesson is that this specific guest, on this specific host, has already demonstrated that the clever approach costs real data. Overlay snapshots are ruled out for this guest. Not "risky," not "needs more testing" ruled out, permanently. If you're tempted to suggest it, don't; it's already been tried and it already cost days of someone's work. So the clean options are gone. What's left is forcing the guest down.
The fallback: `virsh destroy`, with a fence around it
`virsh destroy` is the libvirt way of saying "stop this VM now" equivalent to pulling the power. For a running guest, that's a hard reset, and you should treat it with the respect that implies. On a modern guest with a journaling filesystem, it's usually survivable. On an old Windows server, it's a last resort. I accepted it for exactly one guest. Not three. The script has an explicit allowlist of guests that *may* be forced down:VMS=(legacy-app mailvm app2016)
DESTROY_VMS=(legacy-app)
Only legacy-app the one that can't be shut down any other way is eligible. The other two follow the normal path, if they don't stop within the timeout, the run aborts. Forcing down a guest that could have stopped cleanly would be inexcusable; the script makes that impossible.
When a guest in DESTROY_VMS hits the timeout, the script does this, in order:
- Logs four explicit WARNING lines — not one, not a debug line buried in a normal message. Four separate
loggercalls so it's impossible to miss in the journal and impossible to mistake for a clean shutdown:
logger -t vm-backup "WARNING: $vm did not stop within ${SHUTDOWN_TIMEOUT}s"
logger -t vm-backup "WARNING: $vm is in DESTROY_VMS — falling back to virsh destroy"
logger -t vm-backup "WARNING: this is a hard stop; guest may need filesystem check"
logger -t vm-backup "WARNING: proceeding with destroy for $vm"
virsh destroy "$vm".sleep 5give libvirt a moment to actually mark it stopped.- Re-check state and proceed only if it's genuinely stopped. If it's still running after that, abort. The script never copies a disk it isn't certain is quiescent:
virsh destroy "$vm" || die "virsh destroy failed for $vm"
sleep 5
state=$(virsh domstate "$vm" 2>/dev/null || echo unknown)
if [ "$state" != "shut off" ]; then
die "$vm still '$state' after destroy — aborting, not copying a live disk"
fi
That last check is the whole point. The fallback is acceptable because the script refuses to move on until the guest is verifiably off. If the force stop fails, we fail the run. We never fall through to copying.
Why 01:00 is part of the safety argument
I want to be honest about the risk calculus here, because "just force the VM off" sounds reckless and I want to explain why it isn't (as much) in this specific case.
The backup runs in the early morning. On this host, that's a window with essentially no users on that guest. The guest is not a database with continuous writes; it's a legacy application server with a quiet overnight period. That's not an accident — it's the reason a hard stop is tolerable for this guest during this window. It would not be tolerable for the mail server, which is why the mail server isn't in DESTROY_VMS.
The point is: virsh destroy at 01:00 on a quiet legacy guest is a calculated, narrow, documented risk. virsh destroy at any time on a busy database is a different thing entirely and nobody should conflate them. The allowlist keeps the two separated.
The restore baseline is still a cold copy
Here's the part I want to be clearest about. A destroy-then-rsync backup is a best-effort backup. It's much better than no backup, and far better than copying a live disk, but it isn't the same as a backup taken from a guest that shut down cleanly. On a hard stop, the guest's filesystem was frozen mid-state; the copy is consistent as a disk image but the guest inside it may come up needing a filesystem check.
So the trusted restore point for this guest is not the scheduled backup. It's a manual cold copy I take during a planned outage, when the guest is stopped deliberately and I can afford to verify it came down. That's the copy I actually trust. The scheduled backups are useful and automated and I keep them, but they are explicitly best-effort and I treat them that way.
If you run this pattern, keep that distinction. The scheduled backup is the safety net. The cold copy is the known-good baseline. Don't let the existence of the first make you skip the second.
What I'd do differently
- I'd have put the destroy fallback behind an allowlist from the start, rather than as a global "if it won't stop, force it" option. The difference is the difference between a targeted last resort and a habit. Only the guest that needs it should be eligible.
- I'd have made the WARNING loud enough that I couldn't ignore it for months. The four-line explicit warning exists because a single log line gets skimmed. If a guest has been force-stopped every backup for a year, that should be screaming at you, not whispering.
- I'd have re-checked state after the destroy before copying. This is the one that would have caught the worst failure.
virsh destroyreturning success is not the same as the guest being off. Trust the state check, not the exit code. Never copy a disk you haven't confirmed is quiescent. - I'd have accepted the cold copy as the real baseline earlier. I spent time wishing the scheduled backup were the trusted one. It isn't. Once I accepted that and took a proper cold copy during an outage, the scheduled backups became a comfortable second layer instead of a source of anxiety.
The right answer for this guest, it turns out, isn't a clever one. It's: stop it if you can, force it if you must, verify it's actually stopped, and keep a clean cold copy you trust for the day the force stop bites back. The clever answers were all ruled out one by a missing driver that doesn't exist, one by a data-loss experiment I'm not repeating.