Backup of vm disk images

Backup of vm disk images

[Anders Östling]

I am fighting to understand the difference between backing up a VM by
using a regular copy vs using the virsh blockcopy command.
What I want to do is to suspend the vm, copy the XML and .QCOW2 files
and then resume the vm again. What are your thoughts? What are the
drawbacks compared to other methods?
Thanks

-- 
-----------------------------------------------------------------------------------------------------------------------
This signature contains 100% recyclable electrons as prescribed by Mother Nature

Anders Östling
+46 768 716 165 (Mobil)
+46 431 45 56 01  (Hem)

Re: Backup of vm disk images

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

On Wed, Apr 22, 2020 at 07:51:09AM +0200, Anders Östling wrote:
> I am fighting to understand the difference between backing up a VM by
> using a regular copy vs using the virsh blockcopy command.
> What I want to do is to suspend the vm, copy the XML and .QCOW2 files
> and then resume the vm again. What are your thoughts? What are the
> drawbacks compared to other methods?

Hi Anders,
The kvm@vger.kernel.org mailing list is mostly for the discussion and
development of the KVM kernel module so you may not get replies.  I have
CCed libvir-list and developers who have been involved in libvirt backup
features.

A naive cp(1) command will be very slow because the entire disk image is
copied to a new file.  The fastest solution with cp(1) is the --reflink
flag which basically takes a snapshot of the file and shares the disk
blocks (only available when the host file system supports it and not
available across mounts).

Libvirt's backup commands are more powerful.  They can do things like
copy out a point-in-time snapshot of the disk while the guest is
running.  They also support incremental backup so you don't need to
store a full copy of the disk image each time you take a backup.

I hope others will join the discussion and give examples of some of the
available features.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Backup of vm disk images

[Peter Krempa]

On Fri, May 01, 2020 at 16:05:47 +0100, Stefan Hajnoczi wrote:
> On Wed, Apr 22, 2020 at 07:51:09AM +0200, Anders Östling wrote:
> > I am fighting to understand the difference between backing up a VM by
> > using a regular copy vs using the virsh blockcopy command.
> > What I want to do is to suspend the vm, copy the XML and .QCOW2 files
> > and then resume the vm again. What are your thoughts? What are the
> > drawbacks compared to other methods?

The approaches have diffrerent kind of data integrity they provide and
downtime they require.

Assuming from the above that you don't want to shutdown the OS in the VM
you've got following options:

1) pause VM and copy images as described above
  I definitely don't recommend this approach at all. The drawback is
  that the QCOW2 file on the disk may have inconsistent metadata. Even
  if you ensure that the gues OS state is consistend and written to disk
  it's not guaranteed that qemu's buffers were flushed.

  Also the VM needs to be suspended during the whole copy, unless you
  have the image on a filesystem which has --reflink support as  pointed
  out by Stefan.

2) 'virsh blockcopy'
 The original idea of blockcopy is to move storage of an active VM to a
 different location. It can be used though to "copy" the active disk and
 ensure that the metadata is correct when combined with 'virsh blockjob
 --abort' to finish it. This still requires the guest OS in the VM to
 ensure that the filesystems on the backed-up disk are consistent.

 Also the API has one very severe limitation if your VM has multiple
 disks: There is no way to ensure that you cancel all the copies at the
 same time, so the 'backup' done this way is not taken at a single point
 in time. It's also worth noting that the point in time the backup is
 taken is when the job is --abort-ed.

3) virsh backup
 This is the newest set of APIs specifically designed to do disk backups
 of the VM, offers consistency of the image metadata, and taking of the
 backups of multiple disks at the same point in time. Also the point in
 time is when the API is started, regardless of how long the actual data
 handling takes.

 Your gues OS still needs to ensure filesystem consistency though.

 Additionally as mentioned by Stefan below you can also do incremental
 backups as well.

 One thing to note though is that the backup integration is not entirely
 finished in libvirt and thus in a 'tech-preview' state. Some
 interactions corrupt the state for incremental backups.

 If you are interested, I can give you specific info how to enable
 support for backups as well as the specifics of the current state of
 implementation.

4) snapshots
 Libvirt's snapshot implementation supports taking full VM snapshots
 including memory and disk image state. This sidesteps the problem of
 inconsistent filesystem state as the memory state contains also all the
 buffers.

 When an external snapshot is created, we add a new set of overlay files
 on top of the original disk images. This means that they become
 effectively read-only. You can then copy them aside if you want so. The
 memory image taken along can be then used to fully restore the state of
 the VM.

 There are a few caveats here as well. If the image chain created this
 way becomes too long it may negatively impact performance. Also
 reverting the memory image is a partially manual operation for now. I
 can give specifics if you want.

> 
> Hi Anders,
> The kvm@vger.kernel.org mailing list is mostly for the discussion and
> development of the KVM kernel module so you may not get replies.  I have
> CCed libvir-list and developers who have been involved in libvirt backup
> features.
> 
> A naive cp(1) command will be very slow because the entire disk image is
> copied to a new file.  The fastest solution with cp(1) is the --reflink
> flag which basically takes a snapshot of the file and shares the disk
> blocks (only available when the host file system supports it and not
> available across mounts).
> 
> Libvirt's backup commands are more powerful.  They can do things like
> copy out a point-in-time snapshot of the disk while the guest is
> running.  They also support incremental backup so you don't need to
> store a full copy of the disk image each time you take a backup.
> 
> I hope others will join the discussion and give examples of some of the
> available features.
> 
> Stefan

Re: Backup of vm disk images

[Kashyap Chamarthy]

On Fri, May 01, 2020 at 04:05:47PM +0100, Stefan Hajnoczi wrote:
> On Wed, Apr 22, 2020 at 07:51:09AM +0200, Anders Östling wrote:

Hi Anders,

> > I am fighting to understand the difference between backing up a VM by
> > using a regular copy vs using the virsh blockcopy command.

tl;dr: While 'blockcopy' is one way to do a full backup; there's a
       better way: 'blockcommit'; see below for a URL to an example.

To add to what Stefan says below, here's the difference: a regular 'cp'
is just that — an offline copy when your guest is shutdown or suspended.
However, libvirt's 'blockcopy' lets you create a "point-in-time snapshot
[or copy]" of your VM's disk _while_ the VM is running, and optionally,
lets your VM to "live-pivot" its storage to the just-created copy.

The use-case is "live storage migration".  

Say, your original storage of your VM is on NFS-A, but you want to do
some maintenance on NFS-A.  Here, 'blockcopy' lets you live-copy the
VM's storage from NFS-A to NFS-B, _and_ make the VM use the copy — all
this without causing any downtime to the users of the VM.  Now you can
freely do the maintenance on NFS-A.

A 'blockcopy' operation has two phases:

(1) It copies a VM's disk image, e.g. from 'orig.raw' to 'copy.qcow2',
    while the VM is running.  And it will _keep_ copying as long as your
    VM keeps writing new data.  This is called the "mirroring" phase.

(2) Once the copy.qcow2 has the same content as orig.raw, then you can
    do two things, either (a) end the copying/mirroring, which results
    in a point-in-time 'snapshot' of the orig.raw; or (b) you can
    "live-pivot" the VM's disk image to just-created copy.qcow2.

Management tools like OpenStack (and possibly others) use libvirt's
'blockcopy' API under the hood to allow live storage migration.

There are other useful details here, but I'll skip them for brevity.
Read the "blockcopy" section in `man virsh`.  I admit it can be a little
hard to parse when you normally don't dwell on these matters, but taking
time to experiement gives a robust understanding.

> > What I want to do is to suspend the vm, copy the XML and .QCOW2 files
> > and then resume the vm again. What are your thoughts? What are the
> > drawbacks compared to other methods?

If your main goal is to take a full backup of your disk, *without* any
downtime to your VM, libvirt does provide some neat ways.

One of the most efficient methods to is the so-called "active
block-commit".  It uses a combination of libvirt commands: `virsh
snapshot-create-as`, `virsh blockcommit`, and `rsync`:
   
I've written up a full example here:

    https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

You might also want to refer to this page.

    https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

            - - -

For libvirt-based incremental backup examples, I'll defer that to Eric
Blake (in Cc).

[...]

> A naive cp(1) command will be very slow because the entire disk image is
> copied to a new file.  The fastest solution with cp(1) is the --reflink
> flag which basically takes a snapshot of the file and shares the disk
> blocks (only available when the host file system supports it and not
> available across mounts).
> 
> Libvirt's backup commands are more powerful.  They can do things like
> copy out a point-in-time snapshot of the disk while the guest is
> running.  They also support incremental backup so you don't need to
> store a full copy of the disk image each time you take a backup.
> 
> I hope others will join the discussion and give examples of some of the
> available features.


-- 
/kashyap

Re: Backup of vm disk images

[Anders Östling]

Thanks Peter and Stefan for enlightening me!

On Mon, May 4, 2020 at 9:58 AM Peter Krempa <pkrempa@redhat.com> wrote:
>
>  One thing to note though is that the backup integration is not entirely
>  finished in libvirt and thus in a 'tech-preview' state. Some
>  interactions corrupt the state for incremental backups.
>
>  If you are interested, I can give you specific info how to enable
>  support for backups as well as the specifics of the current state of
>  implementation.
>

I would very much appreciate if you can tell me more on this! It's for
a client, and I want to be as sure as possible that the solution is
robust.

Also, the wiki page referred by Kashyap is also something that I will
experiment with!

Thanks again folks!

Anders

-- 
-----------------------------------------------------------------------------------------------------------------------
This signature contains 100% recyclable electrons as prescribed by Mother Nature

Anders Östling
+46 768 716 165 (Mobil)
+46 431 45 56 01  (Hem)