Bug Repoting Directions Request

Bug Repoting Directions Request

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

Dear all,

As indicated on http://www.linux-kvm.org/page/Bugs, I am seeking directions as to where to post the following bug (or support request):

"I am running KVM virtualization hosts both with Centos 6 and Centos 7 (all installed from packages and updated to the current version). The guests are Centos, Debian, Windows 2008 R2 and Windows 7 machines. I am using qcow2 virtual disks with virtio drivers and cache disabled throughout. My aim is to take a backups based live snapshots each night.

Since some months ago, I am facing the problem, that backups fail unpredictably. A failed backup does not generate a backup file and thereafter, snapshots can no longer be created or deleted and the guest cannot be started anymore. The resulting error message is "File too large".

The consequence is lack of stability as the guests can no longer be rebooted. If the backup did work needs to be - at least with my means and skills - checked manually. If the problem occurred, the guests needs to be recreated from the last backup with all database and version issues.

This happens both with linux and windows guests. It happens with guests in use for many months and recently created guests. It happens with large and small files (my range ca. is 5GB to 60GB per guest). It happens on Centos 6 and 7 based kvm hypervisors and regardless if the storage consists of a local Raid 1 array or an nfs share on a NAS drive.

There are no apparent disk, ram, swap or configuration issues. Logfiles are not revealing useful information as far as I can tell. Searching forums, I finds similar questions sometimes, but usually in the context of Windows vhd only. More similar forum requests did remain unanswered.

Please point me to a direction I could look to for a solution."

Regards,

Michael Schefczyk
___________________________________________________________ 
Technische Universität Dresden
Fakultät Wirtschaftswissenschaften
Lehrstuhl für Entrepreneurship und Innovation
Prof. Dr. Michael Schefczyk
D-01062 Dresden 
 
Fon: +49-3 51-4 63-3 68 81 
Fax: +49-3 51-4 63-3 68 83 
www.gruenderlehrstuhl.de





[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: Bug Repoting Directions Request

[Paolo Bonzini]

On 19/11/2014 11:36, Prof. Dr. Michael Schefczyk wrote:
> "I am running KVM virtualization hosts both with Centos 6 and Centos
> 7 (all installed from packages and updated to the current version).
> The guests are Centos, Debian, Windows 2008 R2 and Windows 7
> machines. I am using qcow2 virtual disks with virtio drivers and
> cache disabled throughout. My aim is to take a backups based live
> snapshots each night.
> 
> Since some months ago, I am facing the problem, that backups fail
> unpredictably. A failed backup does not generate a backup file and
> thereafter, snapshots can no longer be created or deleted and the
> guest cannot be started anymore. The resulting error message is "File
> too large".

Who is reporting the "File too large" error?  Can you please include the
error in full detail?

Paolo

[Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Paolo Bonzini]

On 19/11/2014 12:39, Prof. Dr. Michael Schefczyk wrote:
> Dear Paolo,
> 
> thank you very much for your prompt reply.

Hi,

please keep the replies on the mailing list.  I've added the qemu-devel
mailing list, where more people should be able to see the message and
hopefully reply with something helpful.

Also, how do you do the backups?

Paolo

> For example, I have a guest named "gatewayb72.img" where the backup failed. If I thereafter try to create or delete a snapshot, the following reply occurs on the command line:
> 
> 
> [root@linuxhost2 kvm02]# qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img
> qemu-img: Could not open '/kvm02/gatewayb72.img': Could not read snapshots: File too large
> 
> 
> If I want to reboot that machine, I get the following error:
> 
> 
> Fehler beim Starten der Domain: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large
> 
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 100, in cb_wrapper
>     callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 122, in tmpcb
>     callback(*args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/domain.py", line 1220, in startup
>     self._backend.create()
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 698, in create
>     if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
> libvirtError: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large
> 
> 
> Based on the facts I can see, the file is not too large. When reading the first error, the file size was 13.8 GB, while the limit is 14.5 GB. The same does also happen with files which are only, for example 6 GB big, while their limit is also 14.5 GB. Therefore, I think that the "file too large" really stands for something else.
> 
> Can I provide any additional information?
> 
> Regards,
> 
> Michael
> ___________________________________________________________ 
> Technische Universität Dresden
> Fakultät Wirtschaftswissenschaften
> Lehrstuhl für Entrepreneurship und Innovation
> Prof. Dr. Michael Schefczyk
> D-01062 Dresden 
>  
> Fon: +49-3 51-4 63-3 68 81 
> Fax: +49-3 51-4 63-3 68 83 
> www.gruenderlehrstuhl.de

>> Since some months ago, I am facing the problem, that backups fail 
>> unpredictably. A failed backup does not generate a backup file and 
>> thereafter, snapshots can no longer be created or deleted and the 
>> guest cannot be started anymore. The resulting error message is "File 
>> too large".
> 
> Who is reporting the "File too large" error?  Can you please include the error in full detail?
> 
> Paolo
> 

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 3897 bytes --]

Dear Paolo,

Thank you very much. To execute the backup, I run a script. For the machine in question, it looks as follows:

#!/bin/bash
dt=`date +%y%m%d`
qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img
qemu-img convert  -f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img
qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img
/bin/find /backup/* -mtime +100 -exec rm {} \;

Regards,

Michael


-----Ursprüngliche Nachricht-----
Von: Paolo Bonzini [mailto:pbonzini@redhat.com] 
Gesendet: Mittwoch, 19. November 2014 12:42
An: Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)



On 19/11/2014 12:39, Prof. Dr. Michael Schefczyk wrote:
> Dear Paolo,
> 
> thank you very much for your prompt reply.

Hi,

please keep the replies on the mailing list.  I've added the qemu-devel mailing list, where more people should be able to see the message and hopefully reply with something helpful.

Also, how do you do the backups?

Paolo

> For example, I have a guest named "gatewayb72.img" where the backup failed. If I thereafter try to create or delete a snapshot, the following reply occurs on the command line:
> 
> 
> [root@linuxhost2 kvm02]# qemu-img snapshot -d gatewayb72 
> /kvm02/gatewayb72.img
> qemu-img: Could not open '/kvm02/gatewayb72.img': Could not read 
> snapshots: File too large
> 
> 
> If I want to reboot that machine, I get the following error:
> 
> 
> Fehler beim Starten der Domain: Interner Fehler: Prozess wurde während 
> der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive 
> file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,
> cache=none: could not open disk image /kvm02/gatewayb72.img: Could not 
> read snapshots: File too large
> 
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 100, in cb_wrapper
>     callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 122, in tmpcb
>     callback(*args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/domain.py", line 1220, in startup
>     self._backend.create()
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 698, in create
>     if ret == -1: raise libvirtError ('virDomainCreate() failed', 
> dom=self)
> libvirtError: Interner Fehler: Prozess wurde während der 
> Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive 
> file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,
> cache=none: could not open disk image /kvm02/gatewayb72.img: Could not 
> read snapshots: File too large
> 
> 
> Based on the facts I can see, the file is not too large. When reading the first error, the file size was 13.8 GB, while the limit is 14.5 GB. The same does also happen with files which are only, for example 6 GB big, while their limit is also 14.5 GB. Therefore, I think that the "file too large" really stands for something else.
> 
> Can I provide any additional information?
> 
> Regards,
> 
> Michael
> ___________________________________________________________
> Technische Universität Dresden
> Fakultät Wirtschaftswissenschaften
> Lehrstuhl für Entrepreneurship und Innovation Prof. Dr. Michael 
> Schefczyk
> D-01062 Dresden
>  
> Fon: +49-3 51-4 63-3 68 81
> Fax: +49-3 51-4 63-3 68 83
> www.gruenderlehrstuhl.de

>> Since some months ago, I am facing the problem, that backups fail 
>> unpredictably. A failed backup does not generate a backup file and 
>> thereafter, snapshots can no longer be created or deleted and the 
>> guest cannot be started anymore. The resulting error message is "File 
>> too large".
> 
> Who is reporting the "File too large" error?  Can you please include the error in full detail?
> 
> Paolo
> 


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Paolo Bonzini]

On 19/11/2014 12:48, Prof. Dr. Michael Schefczyk wrote:
> 
> Thank you very much. To execute the backup, I run a script. For the machine in question, it looks as follows:
> 
> #!/bin/bash
> dt=`date +%y%m%d`
> qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img
> qemu-img convert  -f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img
> qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img
> /bin/find /backup/* -mtime +100 -exec rm {} \;

Is it done while the VM is running?

Paolo

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

Yes! My level of knowledge is that one uses the qcow2 format in order to be able to create live snapshots/backups. Otherwise one would tend to use the more efficient raw format. Is this not correct and did I apply the backup mechanism in the wrong way?


-----Ursprüngliche Nachricht-----
Von: Paolo Bonzini [mailto:pbonzini@redhat.com] 
Gesendet: Mittwoch, 19. November 2014 12:54
An: Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: Re: AW: "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)



On 19/11/2014 12:48, Prof. Dr. Michael Schefczyk wrote:
> 
> Thank you very much. To execute the backup, I run a script. For the machine in question, it looks as follows:
> 
> #!/bin/bash
> dt=`date +%y%m%d`
> qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img qemu-img convert  
> -f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img 
> qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img /bin/find 
> /backup/* -mtime +100 -exec rm {} \;

Is it done while the VM is running?

Paolo


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Paolo Bonzini]

On 19/11/2014 13:07, Prof. Dr. Michael Schefczyk wrote:
> Yes! My level of knowledge is that one uses the qcow2 format in order
> to be able to create live snapshots/backups. Otherwise one would tend
> to use the more efficient raw format. Is this not correct and did I
> apply the backup mechanism in the wrong way?

That's correct, but you still have to create live snapshots from within
QEMU.

This is done with a QMP (QEMU Management Protocol) command like

{ "execute": "blockdev-snapshot-internal-sync",
                "arguments": { "device": "ide-hd0",
                               "name": "snapshot0" }
   }

QMP is accessed through normal sockets, or via libvirt.

However, I'm not sure if running "qemu-img convert" on the resulting
snapshot is possible though, and there is no equivalent of "qemu-img
snapshot -d".

You can instead use QEMU's support for backup, which will do what you
wanted directly while the VM is running.  For example:

{ "execute": "drive-backup", "arguments": { "device": "ide-hd0",
                                     "sync": "full", "format": "qcow2",
                                     "target": "backup.img" } }

This does not even require qcow2 for the image.  The downside is that
you must not turn off the VM until the job has completed.

Paolo

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

On 11/19/2014 07:54 AM, Paolo Bonzini wrote:
> On 19/11/2014 13:07, Prof. Dr. Michael Schefczyk wrote:
>> Yes! My level of knowledge is that one uses the qcow2 format in order
>> to be able to create live snapshots/backups. Otherwise one would tend
>> to use the more efficient raw format. Is this not correct and did I
>> apply the backup mechanism in the wrong way?
> 
> That's correct, but you still have to create live snapshots from within
> QEMU.

You absolutely CANNOT modify a qcow2 file via qemu-img or any other
external means while qemu is using the file in read-write mode, or you
risk file corruption (which appears to be what happened to you).
Monitor commands to the qemu process are the only supported means for
taking a live snapshot/backup of an in-use disk.

> 
> This is done with a QMP (QEMU Management Protocol) command like
> 
> { "execute": "blockdev-snapshot-internal-sync",
>                 "arguments": { "device": "ide-hd0",
>                                "name": "snapshot0" }
>    }
> 
> QMP is accessed through normal sockets, or via libvirt.
> 
> However, I'm not sure if running "qemu-img convert" on the resulting
> snapshot is possible though, and there is no equivalent of "qemu-img
> snapshot -d".

qemu-img snapshot -d can be achieved via the QMP monitor command
'human-monitor-command' forwarding to the HMP 'delvm' command.  But you
are correct that I don't know of any way to reproduce qemu-img convert
on a read-write image.

> 
> You can instead use QEMU's support for backup, which will do what you
> wanted directly while the VM is running.  For example:
> 
> { "execute": "drive-backup", "arguments": { "device": "ide-hd0",
>                                      "sync": "full", "format": "qcow2",
>                                      "target": "backup.img" } }
> 
> This does not even require qcow2 for the image.  The downside is that
> you must not turn off the VM until the job has completed.

Another option is to create a live snapshot (so that your file under
interest is now read-only with a temporary qcow2 overlay as the
read-write file), then you can do whatever you want with the file (such
as qemu-img convert), and finally do an active commit to get rid of the
temporary qcow2 overlay and back to just the main live image.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 4160 bytes --]

Dear Eric, dear all,

Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time.

A major book on KVM in German language by Kofler & Spenneberg recommends the following approach for online backups (with and without "--disk-only"):

virsh snapshot-create-as vm XXX --disk-only
qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img
virsh snapshot-delete vm XXX

Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical.

The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization Deployment and Administration Guide on backups, but I found that too cryptic for someone like myself to draw practical consequences from it.

Regards,

Michael
___________________________________________________________ 
Technische Universität Dresden
Fakultät Wirtschaftswissenschaften
Lehrstuhl für Entrepreneurship und Innovation
Prof. Dr. Michael Schefczyk
D-01062 Dresden 
 
Fon: +49-3 51-4 63-3 68 81 
Fax: +49-3 51-4 63-3 68 83 
www.gruenderlehrstuhl.de






-----Ursprüngliche Nachricht-----
Von: Eric Blake [mailto:eblake@redhat.com] 
Gesendet: Mittwoch, 19. November 2014 16:44
An: Paolo Bonzini; Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

On 11/19/2014 07:54 AM, Paolo Bonzini wrote:
> On 19/11/2014 13:07, Prof. Dr. Michael Schefczyk wrote:
>> Yes! My level of knowledge is that one uses the qcow2 format in order 
>> to be able to create live snapshots/backups. Otherwise one would tend 
>> to use the more efficient raw format. Is this not correct and did I 
>> apply the backup mechanism in the wrong way?
> 
> That's correct, but you still have to create live snapshots from 
> within QEMU.

You absolutely CANNOT modify a qcow2 file via qemu-img or any other external means while qemu is using the file in read-write mode, or you risk file corruption (which appears to be what happened to you).
Monitor commands to the qemu process are the only supported means for taking a live snapshot/backup of an in-use disk.

> 
> This is done with a QMP (QEMU Management Protocol) command like
> 
> { "execute": "blockdev-snapshot-internal-sync",
>                 "arguments": { "device": "ide-hd0",
>                                "name": "snapshot0" }
>    }
> 
> QMP is accessed through normal sockets, or via libvirt.
> 
> However, I'm not sure if running "qemu-img convert" on the resulting 
> snapshot is possible though, and there is no equivalent of "qemu-img 
> snapshot -d".

qemu-img snapshot -d can be achieved via the QMP monitor command 'human-monitor-command' forwarding to the HMP 'delvm' command.  But you are correct that I don't know of any way to reproduce qemu-img convert on a read-write image.

> 
> You can instead use QEMU's support for backup, which will do what you 
> wanted directly while the VM is running.  For example:
> 
> { "execute": "drive-backup", "arguments": { "device": "ide-hd0",
>                                      "sync": "full", "format": "qcow2",
>                                      "target": "backup.img" } }
> 
> This does not even require qcow2 for the image.  The downside is that 
> you must not turn off the VM until the job has completed.

Another option is to create a live snapshot (so that your file under interest is now read-only with a temporary qcow2 overlay as the read-write file), then you can do whatever you want with the file (such as qemu-img convert), and finally do an active commit to get rid of the temporary qcow2 overlay and back to just the main live image.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4704 bytes --]

On 11/19/2014 10:32 AM, Prof. Dr. Michael Schefczyk wrote:
> Dear Eric, dear all,
> 
> Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time.

[can you convince your mailer to wrap long lines?  also, we tend to
frown on top-posting on technical lists]

Yeah, you were just getting lucky that two different processes weren't
both trying to allocate a cluster for different purposes at the same
time.  When the collision finally did happen, it had catastrophic
results on your image.

> 
> A major book on KVM in German language by Kofler & Spenneberg recommends the following approach for online backups (with and without "--disk-only"):
> 
> virsh snapshot-create-as vm XXX --disk-only
> qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img
> virsh snapshot-delete vm XXX

Yes, virsh is using QMP commands under the hood, so this method is safe.
 One slight issue is that this sequence is incomplete (it does not
shrink the backing file chain after the copy is complete), so if you
keep repeating it, you will eventually cause reduced performance when
you have a really long chain of multiple qcow2 overlays, or even cause
qemu to run into fd exhaustion.  Also, this command does not show that
unless you clean things up, then the second time you run this you do not
want to copy XXX.img into backup, but instead the qcow2 wrapper file
that was created by the first snapshot (and which itself wrapped by the
second snapshot).

With new enough libvirt and qemu, you can shrink the chain back down
with an active commit, as in:

virsh blockcommit vm XXX --verbose --active --pivot

Also, the use of --disk-only means that your disks have a snapshot taken
as if at a point in time when you yank the power cord; reverting to such
a backup may require fsck, and may suffer from other problems from
anything that was depending on I/O that had not yet been flushed to
disk.  If you add the --quiesce option (which implies that you set up
your guest to use qemu-guest-agent, and told libvirt to manage the
agent), then you can guarantee that your guest has flushed and frozen
all filesystems prior to the point in time where the snapshot is
created; and you can even install hooks in the guest to extend that
stability to things like databases.  Then your backups are much easier
to use.  If you omit --disk-only, and take a full snapshot, then you
have the guest memory state that is necessary to restore all pending
I/O, and don't need to worry about freezing the guest file systems; but
then you have to remember to back up the memory state in addition to
your disk state.

> 
> Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical.

No, remember what I said.  qemu-img may not be used on any image that is
opened read-write by qemu, but is perfectly safe to do read-only
operations on any image that is opened read-only by qemu.  That sequence
of commands goes from the initial:

disk.img [read-write]

then the snapshot-create command causes libvirt to issue a QMP command
to switch qemu to:

disk.img [read-only] <- overlay.qcow2 [read-write]

at which point you can do anything read-only to disk.img (whether
'qemu-img convert' or 'cp' or any other command that doesn't alter the
contents of the file)

finally, the 'virsh blockcommit' command would take you back to:

disk.img [read-write]

> 
> The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization Deployment and Administration Guide on backups, but I found that too cryptic for someone like myself to draw practical consequences from it.

If you are using libvirt to manage your guests, then yes, the qemu
documentation is cryptic, but that shouldn't matter - you should be
asking the libvirt community how to accomplish a job.  And yes, libvirt
could probably do a better job of advertising and documenting its level
of support for live backups, but that is more a question for the
libvirt-users@redhat.com mailing list (the archive of that list show
that questions like this come up very regularly; you are not the first
to ask about live backups, so perusing those archives may give you some
ideas on what works).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 5880 bytes --]

Dear All,

after some trying, my impression is that the following steps do work with plain Centos 7:

virsh snapshot-create-as VM backsnap

qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img

virsh snapshot-delete VM backsnap

Am I on the right track with these commands?


The further features seem to be reserved to RHEL (and potentially other distributions) but not included in Centos:

virsh snapshot-create-as issues "Live Disk Snapshot not supported in this version of QEMU" (retranslated from German).

virsh blockcommit issues "Online Transfer not supported ..."

Thus, with Centos 7 it should be possible to back up VMs without shutting them down. They are being paused, however, as long as the snapshot is created. The qemu-guest-agent does not help in this context, as the features which depend on it are not available in Centos.

Regards,

Michael




-----Ursprüngliche Nachricht-----
Von: Eric Blake [mailto:eblake@redhat.com] 
Gesendet: Mittwoch, 19. November 2014 19:13
An: Prof. Dr. Michael Schefczyk; Paolo Bonzini; qemu-devel
Betreff: Re: AW: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

On 11/19/2014 10:32 AM, Prof. Dr. Michael Schefczyk wrote:
> Dear Eric, dear all,
> 
> Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time.

[can you convince your mailer to wrap long lines?  also, we tend to frown on top-posting on technical lists]

Yeah, you were just getting lucky that two different processes weren't both trying to allocate a cluster for different purposes at the same time.  When the collision finally did happen, it had catastrophic results on your image.

> 
> A major book on KVM in German language by Kofler & Spenneberg recommends the following approach for online backups (with and without "--disk-only"):
> 
> virsh snapshot-create-as vm XXX --disk-only qemu-img convert -f qcow2 
> -s XXX -O qcow2 XXX.img /backup/YYY.img virsh snapshot-delete vm XXX

Yes, virsh is using QMP commands under the hood, so this method is safe.
 One slight issue is that this sequence is incomplete (it does not shrink the backing file chain after the copy is complete), so if you keep repeating it, you will eventually cause reduced performance when you have a really long chain of multiple qcow2 overlays, or even cause qemu to run into fd exhaustion.  Also, this command does not show that unless you clean things up, then the second time you run this you do not want to copy XXX.img into backup, but instead the qcow2 wrapper file that was created by the first snapshot (and which itself wrapped by the second snapshot).

With new enough libvirt and qemu, you can shrink the chain back down with an active commit, as in:

virsh blockcommit vm XXX --verbose --active --pivot

Also, the use of --disk-only means that your disks have a snapshot taken as if at a point in time when you yank the power cord; reverting to such a backup may require fsck, and may suffer from other problems from anything that was depending on I/O that had not yet been flushed to disk.  If you add the --quiesce option (which implies that you set up your guest to use qemu-guest-agent, and told libvirt to manage the agent), then you can guarantee that your guest has flushed and frozen all filesystems prior to the point in time where the snapshot is created; and you can even install hooks in the guest to extend that stability to things like databases.  Then your backups are much easier to use.  If you omit --disk-only, and take a full snapshot, then you have the guest memory state that is necessary to restore all pending I/O, and don't need to worry about freezing the guest file systems; but then you have to remember to back up the memory state in addition to your disk state.

> 
> Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical.

No, remember what I said.  qemu-img may not be used on any image that is opened read-write by qemu, but is perfectly safe to do read-only operations on any image that is opened read-only by qemu.  That sequence of commands goes from the initial:

disk.img [read-write]

then the snapshot-create command causes libvirt to issue a QMP command to switch qemu to:

disk.img [read-only] <- overlay.qcow2 [read-write]

at which point you can do anything read-only to disk.img (whether 'qemu-img convert' or 'cp' or any other command that doesn't alter the contents of the file)

finally, the 'virsh blockcommit' command would take you back to:

disk.img [read-write]

> 
> The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization Deployment and Administration Guide on backups, but I found that too cryptic for someone like myself to draw practical consequences from it.

If you are using libvirt to manage your guests, then yes, the qemu documentation is cryptic, but that shouldn't matter - you should be asking the libvirt community how to accomplish a job.  And yes, libvirt could probably do a better job of advertising and documenting its level of support for live backups, but that is more a question for the libvirt-users@redhat.com mailing list (the archive of that list show that questions like this come up very regularly; you are not the first to ask about live backups, so perusing those archives may give you some ideas on what works).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Paolo Bonzini]

On 22/11/2014 18:02, Prof. Dr. Michael Schefczyk wrote:
> The further features seem to be reserved to RHEL (and potentially
> other distributions) but not included in Centos:
> 
> virsh snapshot-create-as issues "Live Disk Snapshot not supported in
> this version of QEMU" (retranslated from German).
> 
> virsh blockcommit issues "Online Transfer not supported ..."
> 
> Thus, with Centos 7 it should be possible to back up VMs without
> shutting them down. They are being paused, however, as long as the
> snapshot is created. The qemu-guest-agent does not help in this
> context, as the features which depend on it are not available in
> Centos.

Actually, they are available in most distributions except RHEL and 
CentOS.  For various reasons that I cannot really detail here, Red Hat 
is only shipping these features as part of the "Red Hat Cloud 
Infrastructure" product.

Luckily, since you are using CentOS you do not really care about 
official Red Hat RPMs, and you can get the packages here:

http://resources.ovirt.org/pub/ovirt-3.5/rpm/el7Server/

This yum repository file will help:

    [qemu-kvm-rhev]
    name=oVirt rebuilds of qemu-kvm-rhev
    baseurl=http://resources.ovirt.org/pub/ovirt-3.5/rpm/el7Server/
    mirrorlist=http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-el7Server
    enabled=1
    skip_if_unavailable=1
    gpgcheck=0

Drop it in /etc/yum.repos.d/qemu-kvm-rhev.repo and install qemu-kvm-rhev
with yum.  All features will be available.  It would be simpler to have
a CentOS SIG build this, but it hasn't happened yet.

Thanks,

Paolo

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

Am 22.11.2014 um 18:02 hat Prof. Dr. Michael Schefczyk geschrieben:
> Dear All,
> 
> after some trying, my impression is that the following steps do work with plain Centos 7:
> 
> virsh snapshot-create-as VM backsnap
> qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img
> virsh snapshot-delete VM backsnap
> 
> Am I on the right track with these commands?

I won't tell you that you're bound to break your image with this
sequence, because chances are that you won't break it. (In practice,
this happens to work in most cases, because the essential snapshot
metadata generally isn't written to without explicit user actions on
that snapshot.)

But you're violating the rule of "one writer or many readers", so as far
as qemu is concerned, we won't be careful to avoid breaking this setup.
Don't blame us if it fails one day.

With the QMP solutions (either drive-backup or snapshot/commit) you're
using official APIs, so I'd encourage you to use these.

Kevin

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Prof. Dr. Michael Schefczyk]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Dear Kevin,

thank you very much! For the moment, I did stabilize my systems with these commands. For a next step, will explore three further options, which should all solve the issue better:

- move to ovirt altogether - probably as foolproof as needed at my skill level

- install ovirt's qemu-kvm-rhev to use the more extensive options

- move to QMP

Thanks again also to Paolo and Eric for providing substantial insights!

Regards,

Michael

-----Ursprüngliche Nachricht-----
Von: Kevin Wolf [mailto:kwolf@redhat.com] 
Gesendet: Dienstag, 25. November 2014 12:34
An: Prof. Dr. Michael Schefczyk
Cc: Eric Blake; Paolo Bonzini; qemu-devel
Betreff: Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

Am 22.11.2014 um 18:02 hat Prof. Dr. Michael Schefczyk geschrieben:
> Dear All,
> 
> after some trying, my impression is that the following steps do work with plain Centos 7:
> 
> virsh snapshot-create-as VM backsnap
> qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img virsh 
> snapshot-delete VM backsnap
> 
> Am I on the right track with these commands?

I won't tell you that you're bound to break your image with this sequence, because chances are that you won't break it. (In practice, this happens to work in most cases, because the essential snapshot metadata generally isn't written to without explicit user actions on that snapshot.)

But you're violating the rule of "one writer or many readers", so as far as qemu is concerned, we won't be careful to avoid breaking this setup.
Don't blame us if it fails one day.

With the QMP solutions (either drive-backup or snapshot/commit) you're using official APIs, so I'd encourage you to use these.

Kevin

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5980 bytes --]

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Kashyap Chamarthy]

On Wed, Nov 19, 2014 at 11:13:27AM -0700, Eric Blake wrote:
> On 11/19/2014 10:32 AM, Prof. Dr. Michael Schefczyk wrote:

[. . .]

> > virsh snapshot-create-as vm XXX --disk-only qemu-img convert -f
> > qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img virsh snapshot-delete
> > vm XXX
> 
> Yes, virsh is using QMP commands under the hood, so this method is
> safe.  One slight issue is that this sequence is incomplete (it does
> not shrink the backing file chain after the copy is complete), so if
> you keep repeating it, you will eventually cause reduced performance
> when you have a really long chain of multiple qcow2 overlays, or even
> cause qemu to run into fd exhaustion.  Also, this command does not
> show that unless you clean things up, then the second time you run
> this you do not want to copy XXX.img into backup, but instead the
> qcow2 wrapper file that was created by the first snapshot (and which
> itself wrapped by the second snapshot).
> 
> With new enough libvirt and qemu, you can shrink the chain back down
> with an active commit, as in:
> 
> virsh blockcommit vm XXX --verbose --active --pivot
> 
> Also, the use of --disk-only means that your disks have a snapshot
> taken as if at a point in time when you yank the power cord; reverting
> to such a backup may require fsck, and may suffer from other problems
> from anything that was depending on I/O that had not yet been flushed
> to disk.  If you add the --quiesce option (which implies that you set
> up your guest to use qemu-guest-agent, and told libvirt to manage the
> agent), then you can guarantee that your guest has flushed and frozen
> all filesystems prior to the point in time where the snapshot is
> created; and you can even install hooks in the guest to extend that
> stability to things like databases.  Then your backups are much easier
> to use.  If you omit --disk-only, and take a full snapshot, then you
> have the guest memory state that is necessary to restore all pending
> I/O, and don't need to worry about freezing the guest file systems;
> but then you have to remember to back up the memory state in addition
> to your disk state.
> 
> > 
> > Would this be any better than my script, because it uses virsh
> > snapshot-create-as instead of qemu-img snapshot? The second command
> > is still qemu-img convert which may be problematical.
> 
> No, remember what I said.  qemu-img may not be used on any image that
> is opened read-write by qemu, but is perfectly safe to do read-only
> operations on any image that is opened read-only by qemu.  That
> sequence of commands goes from the initial:
> 
> disk.img [read-write]
> 
> then the snapshot-create command causes libvirt to issue a QMP command
> to switch qemu to:
> 
> disk.img [read-only] <- overlay.qcow2 [read-write]
> 
> at which point you can do anything read-only to disk.img (whether
> 'qemu-img convert' or 'cp' or any other command that doesn't alter the
> contents of the file)
> 
> finally, the 'virsh blockcommit' command would take you back to:
> 
> disk.img [read-write]
> 
> > 
> > The problem I am facing is that the documentation is not easy to
> > understand for the average user/administrator who is not among the
> > kvm developers and experts. I have of course tried to read section
> > 14.2.3 of RHEL 7 Virtualization Deployment and Administration Guide
> > on backups, but I found that too cryptic for someone like myself to
> > draw practical consequences from it.
> 
> If you are using libvirt to manage your guests, then yes, the qemu
> documentation is cryptic, but that shouldn't matter - you should be
> asking the libvirt community how to accomplish a job.  And yes,
> libvirt could probably do a better job of advertising and documenting
> its level of support for live backups, but that is more a question for
> the libvirt-users@redhat.com mailing list (the archive of that list
> show that questions like this come up very regularly; you are not the
> first to ask about live backups, so perusing those archives may give
> you some ideas on what works).

As an addendum to Eric's detailed notes above, below are some examples
for live disk backups and shortening disk image chains with libvirt's
'blockcommit '.

Applicable for: QEMU 2.1, libvirt-1.2.9 or newer.

Efficient live disk backup with active blockcommit: 

    http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

Merge an entire chain (including current active image) into its base
image:

    http://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk 

-- 
/kashyap

Re: [Qemu-devel] "File too large" error from "qemu-img snapshot" (was Re: AW: Bug Repoting Directions Request)

[Max Reitz]

On 19.11.2014 12:41, Paolo Bonzini wrote:
>
> On 19/11/2014 12:39, Prof. Dr. Michael Schefczyk wrote:
>> Dear Paolo,
>>
>> thank you very much for your prompt reply.
> Hi,
>
> please keep the replies on the mailing list.  I've added the qemu-devel
> mailing list, where more people should be able to see the message and
> hopefully reply with something helpful.
>
> Also, how do you do the backups?
>
> Paolo
>
>> For example, I have a guest named "gatewayb72.img" where the backup failed. If I thereafter try to create or delete a snapshot, the following reply occurs on the command line:
>>
>>
>> [root@linuxhost2 kvm02]# qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img
>> qemu-img: Could not open '/kvm02/gatewayb72.img': Could not read snapshots: File too large
>>
>>
>> If I want to reboot that machine, I get the following error:
>>
>>
>> Fehler beim Starten der Domain: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large
>>
>>
>> Traceback (most recent call last):
>>    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 100, in cb_wrapper
>>      callback(asyncjob, *args, **kwargs)
>>    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 122, in tmpcb
>>      callback(*args, **kwargs)
>>    File "/usr/share/virt-manager/virtManager/domain.py", line 1220, in startup
>>      self._backend.create()
>>    File "/usr/lib64/python2.7/site-packages/libvirt.py", line 698, in create
>>      if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
>> libvirtError: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large
>>
>>
>> Based on the facts I can see, the file is not too large. When reading the first error, the file size was 13.8 GB, while the limit is 14.5 GB. The same does also happen with files which are only, for example 6 GB big, while their limit is also 14.5 GB. Therefore, I think that the "file too large" really stands for something else.

It most probably does, yes. Some months ago, we included a limit in 
qemu's qcow2 implementation for the (internal) snapshot table size. When 
this limit is hit, "File too large" will be printed (it's a pretty bad 
error message, I know, but we thought it never to occur for reasonable 
images, so we did not see the need to improve it). That limit is 64 MB, 
which comes from having 1 kB of metadata per snapshot in average and a 
maximum of 64k snapshots. 1 kB per snapshot is reasonable: Normally, you 
have 56 bytes plus the length of the snapshot's name plus the length of 
the snapshot's ID (plus a couple of bytes of padding).

We have indeed seen some bug reports regarding this limit; however, 
nobody could provide us with an image to look into this issue so far (as 
far as I'm aware), and we could not reproduce it ourselves.

I understand that providing a 13.8 GB image is a bit much, but maybe you 
are able to do it anyway (the TU Dresden does have a reasonable internet 
connection, as far as I'm aware) or, if possible, provide us with 
information on how to recreate a similar image ourselves.

Max