Re: [Qemu-devel] [RFC] live snapshot, live merge, live block migration

[Jagane Sundar <jagane@sundar.org>]

Hello Dor,

I'm glad I could convince you of the value of Livebackup. I
think Livesnapshot/Livemerge, Livebackup and Block
Migration all have very interesting use cases. For example:

- Livesnapshot/Livemerge is very useful in development/QA
   environments where one might want to create a snapshot
   before trying out some new software and then committing.
- Livebackup is useful in cloud environments where the
   Cloud Service Provider may want to offer regularly scheduled
   backed up VMs with no effort on the part of the customer
- Block Migration with COR is useful in Cloud Service provider
   environments where an arbitrary VM may need to be
   migrated over to another VM server, even though the VM
   is on direct attached storage.

The above is by no means an exhaustive list of use cases. I
am sure qemu/qemu-kvm users can come up with more.

Although there are some common concepts in these three
technologies, I think we should support all three in base
qemu. This would make qemu/qemu-kvm more feature rich
than vmware, xen and hyper-v.

Thanks,
Jagane

On 5/17/2011 3:53 PM, Dor Laor wrote:
> On 05/16/2011 11:23 AM, Jagane Sundar wrote:
>> Hello Dor,
>>
>> Let me see if I understand live snapshot correctly:
>> If I want to configure a VM for daily backup, then I would do
>> the following:
>> - Create a snapshot s1. s0 is marked read-only.
>> - Do a full backup of s0 on day 0.
>> - On day 1, I would create a new snapshot s2, then
>> copy over the snapshot s1, which is the incremental
>> backup image from s0 to s1.
>> - After copying s1 over, I do not need that snapshot, so
>> I would live merge s1 with s0, to create a new merged
>> read-only image s1'.
>> - On day 2, I would create a new snapshot s3, then
>> copy over s2, which is the incremental backup from
>> s1' to s2
>> - And so on...
>>
>> With this sequence of operations, I would need to keep a
>> snapshot active at all times, in order to enable the
>> incremental backup capability, right?
> No and yes ;-)
>
> For regular non incremental backup you can have no snapshot active most
> times:
>
>    - Create a snapshot s1. s0 is marked read-only.
>    - Do a full backup of s0 on day 0.
>    - Once backup is finished, live merge s1 into s0 and make s0 writeable
>      again.
>
> So this way there are no performance penalty here.
> Here we need an option to track dirty block bits (either as internal
> format or external file). This will be both efficient and get the job done.
>
> But in order to be efficient in storage we'll need to ask the snapshot
> creation to only refer to these dirt blocks.
> Well, thinking out load, it turned out to your solution :)
>
> Ok, I do see the value there is with incremental backups.
>
> I'm aware that there were requirements that the backup software itself
> will be done from the guest filesystem level, there incremental backup
> would be done on the FS layer.
>
> Still I do see the value in your solution.
>
> Another option for us would be to keep the latest snapshots around and
> and let the guest IO go through them all the time. There is some
> performance cost but as the newer image format develop, this cost is
> relatively very low.
>
>> If the base image is s0 and there is a single snapshot s1, then a
>> read operation from the VM will first look in s1. if the block is
>> not present in s1, then it will read the block from s0, right?
>> So most reads from the VM will effectively translate into two
>> reads, right?
>>
>> Isn't this a continuous performance penalty for the VM,
>> amounting to almost doubling the read I/O from the VM?
>>
>> Please read below for more comments:
>>>> 2. Robustness of this solution in the face of
>>>> errors in the disk, etc. If any one of the snapshot
>>>> files were to get corrupted, the whole VM is
>>>> adversely impacted.
>>> Since the base images and any snapshot which is not a leaf is marked as
>>> read only there is no such risk.
>>>
>> What happens when a VM host reboots while a live merge of s0
>> and s1 is being done?
> Live merge is using live copy that does duplicates each write IO.
> On a host crash, the merge will continue from the same point where it
> stopped.
>
> I think I answered the your other good comments above.
> Thanks,
> Dor
>
>>>> The primary goal of Livebackup architecture was to have zero
>>>> performance impact on the running VM.
>>>>
>>>> Livebackup impacts performance of the VM only when the
>>>> backup client connects to qemu to transfer the modified
>>>> blocks over, which should be, say 15 minutes a day, for a
>>>> daily backup schedule VM.
>>> In case there were lots of changing for example additional 50GB changes
>>> it will take more time and there will be a performance hit.
>>>
>> Of course, the performance hit is proportional to the amount of data
>> being copied over. However, the performance penalty is paid during
>> the backup operation, and not during normal VM operation.
>>
>>>> One useful thing to do is to evaluate the important use cases
>>>> for this technology, and then decide which approach makes
>>>> most sense. As an example, let me state this use case:
>>>> - A IaaS cloud, where VMs are always on, running off of a local
>>>> disk, and need to be backed up once a day or so.
>>>>
>>>> Can you list some of the other use cases that live snapshot and
>>>> live merge were designed to solve. Perhaps we can put up a
>>>> single wiki page that describes all of these proposals.
>>> Both solutions can serve for the same scenario:
>>> With live snapshot the backup is done the following:
>>>
>>> 1. Take a live snapshot (s1) of image s0.
>>> 2. Newer writes goes to the snapshot s1 while s0 is read only.
>>> 3. Backup software processes s0 image.
>>> There are multiple ways for doing that -
>>> 1. Use qemu-img and get the dirty blocks from former backup.
>>> - Currently qemu-img does not support it.
>>> - Nevertheless, such mechanism will work for lvm, btrfs, NetApp
>>> 2. Mount the s0 image to another guest that runs traditional backup
>>> software at the file system level and let it do the backup.
>>> 4. Live merge s1->s0
>>> We'll use live copy for that so each write is duplicated (like your
>>> live backup solution).
>>> 5. Delete s1
>>>
>>> As you can see, both approaches are very similar, while live snapshot is
>>> more general and not tied to backup specifically.
>>>
>> As I explained at the head of this email, I believe that live snapshot
>> results in the VM read I/O paying a high penalty during normal operation
>> of the VM, whereas Livebackup results in this penalty being paid only
>> during the backup dirty block transfer operation.
>>
>> Finally, I would like to bring up considerations of disk space. To
>> expand on
>> my use case further, consider a Cloud Compute service with 100 VMs
>> running on a host. If live snapshot is used to create snapshot COW files,
>> then potentially each VM could grow the COW snapshot file to the size
>> of the base file, which means the VM host needs to reserve space for
>> the snapshot that equals the size of the VMs - i.e. a 8GB VM would
>> require an additional 8GB of space to be reserved for the snapshot,
>> so that the service provider could safely guarantee that the snapshot
>> will not run out of space.
>> Contrast this with livebackup, wherein the COW files are kept only when
>> the dirty block transfers are being done. This means that for a host with
>> 100 VMs, if the backup server is connecting to each of the 100 qemu's
>> one by one and doing a livebackup, the service provider would need
>> to provision spare disk for at most the COW size of one VM.
>>
>> Thanks,
>> Jagane
>>
>>