[Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Andrew Martin]

Hello,

I am running several qemu-kvm VM servers on Ubuntu 12.04 with qemu-kvm 1.4.0.
Most of the guests are also running Ubuntu 12.04. I am using qcow2 disk images
with the virtio driver in almost all cases, and am storing the disks on two
types of storage devices: 
* a shared NFS server mounted on each host 
* local ext4 filesystems on each host (on top of an md RAID1 of HDDs)

It seems like the consensus for NFS is to use cache=none in order to support
live migration between hosts.

For local ext4 filesystems, I see from this message and elsewhere that
cache=writeback is now the default disk cache mode, and is safe on ext4:
http://lists.gnu.org/archive/html/qemu-devel/2012-02/msg02689.html

All of the hosts and guests are using ext4 (with the default
data=ordered,barrier). Thus it seems probable that the kernel (3.2.x or newer)
should support flushing correctly? Is there a way to check in the guest is
utilizing WCE (or any other required components) correctly?

I recently experienced UPS failure on several hosts which caused a hard
shutdown. After restarting, 3 of the guests had corruption on their disks and
required a fairly long fsck to fix. Afterwards, data that had been written to
the disks several hours before the crash was corrupted, which makes me think
that it was never fsync()-ed to the non-volatile storage.

Is it safe in this setup to use cache=writeback? Or, should I use
cache=writethrough instead?

Thanks,

Andrew Martin

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

On Fri, Aug 15, 2014 at 03:18:15PM -0500, Andrew Martin wrote:
> Hello,
> 
> I am running several qemu-kvm VM servers on Ubuntu 12.04 with qemu-kvm 1.4.0.
> Most of the guests are also running Ubuntu 12.04. I am using qcow2 disk images
> with the virtio driver in almost all cases, and am storing the disks on two
> types of storage devices: 
> * a shared NFS server mounted on each host 
> * local ext4 filesystems on each host (on top of an md RAID1 of HDDs)
> 
> It seems like the consensus for NFS is to use cache=none in order to support
> live migration between hosts.
> 
> For local ext4 filesystems, I see from this message and elsewhere that
> cache=writeback is now the default disk cache mode, and is safe on ext4:
> http://lists.gnu.org/archive/html/qemu-devel/2012-02/msg02689.html
> 
> All of the hosts and guests are using ext4 (with the default
> data=ordered,barrier). Thus it seems probable that the kernel (3.2.x or newer)
> should support flushing correctly? Is there a way to check in the guest is
> utilizing WCE (or any other required components) correctly?

If you strace -f the QEMU process on the host, you will see fdatasync(2)
system calls when the guest flushes the disk.

You can find the file descriptor number by checking ls -l
/proc/$PID_OF_QEMU/fd and looking for the disk image file.

> I recently experienced UPS failure on several hosts which caused a hard
> shutdown. After restarting, 3 of the guests had corruption on their disks and
> required a fairly long fsck to fix. Afterwards, data that had been written to
> the disks several hours before the crash was corrupted, which makes me think
> that it was never fsync()-ed to the non-volatile storage.

What exactly was the "corruption" you encountered?  Which application,
error message, etc.

> Is it safe in this setup to use cache=writeback? Or, should I use
> cache=writethrough instead?

Ubuntu 12.04 is recent and sends write cache flushes.

Are you sure the file system and/or application workload are flushing
the disk cache?  Please check the mount options and application-specific
configuration.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Andrew Martin]

----- Original Message -----
> From: "Stefan Hajnoczi" <stefanha@gmail.com>
> To: "Andrew Martin" <amartin@xes-inc.com>
> Cc: qemu-devel@nongnu.org
> Sent: Tuesday, August 19, 2014 9:59:25 AM
> Subject: Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later
> 
> If you strace -f the QEMU process on the host, you will see fdatasync(2)
> system calls when the guest flushes the disk.
> 
> You can find the file descriptor number by checking ls -l
> /proc/$PID_OF_QEMU/fd and looking for the disk image file.

When the disk is set to cache=writethrough on one of the same VMs, I see frequent 
fdatasync(2) calls (every few seconds). However, when I change the disk over to
cache=writeback, since boot I have not yet seen a single fdatasync(2) call, even
after writing data 2x the amount of RAM:
# time strace -ft -p4113 2>&1 | grep fdatasync
^C

real    15m39.245s
user    0m7.940s
sys     0m18.280s

Note that the disk is defined as follows:
<disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='writeback'/>
        <source file='/var/lib/libvirt/images/vm.img'/>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>


> > I recently experienced UPS failure on several hosts which caused a hard
> > shutdown. After restarting, 3 of the guests had corruption on their disks
> > and
> > required a fairly long fsck to fix. Afterwards, data that had been written
> > to
> > the disks several hours before the crash was corrupted, which makes me
> > think
> > that it was never fsync()-ed to the non-volatile storage.
> 
> What exactly was the "corruption" you encountered?  Which application,
> error message, etc.

Two of the servers are web servers with apache2. In one case, a python daemon
copies JPGs onto the server - the last 100 copied onto the server were corrupted. 
In another case, some files had been uploaded several days prior to the www-root, 
but after the hard reset said files were no longer present in the filesystem. 


> > Is it safe in this setup to use cache=writeback? Or, should I use
> > cache=writethrough instead?
> 
> Ubuntu 12.04 is recent and sends write cache flushes.
> 
> Are you sure the file system and/or application workload are flushing
> the disk cache?  Please check the mount options and application-specific
> configuration.

The mount options for the ext4 filesystem in the VM in both cases are:
rw,relatime,errors=remount-ro,data=ordered

Similarly, the host's ext4 filesystem holding the images is mounted with:
rw,relatime,data=ordered

I did not see any errors in the kernel log in the guest, probably because the 
root filesystem was read-only until the fsck had completed.

Thanks,

Andrew

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Stefan Hajnoczi]

On Wed, Aug 20, 2014 at 12:20 AM, Andrew Martin <amartin@xes-inc.com> wrote:
> ----- Original Message -----
>> From: "Stefan Hajnoczi" <stefanha@gmail.com>
>> To: "Andrew Martin" <amartin@xes-inc.com>
>> Cc: qemu-devel@nongnu.org
>> Sent: Tuesday, August 19, 2014 9:59:25 AM
>> Subject: Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later
>>
>> If you strace -f the QEMU process on the host, you will see fdatasync(2)
>> system calls when the guest flushes the disk.
>>
>> You can find the file descriptor number by checking ls -l
>> /proc/$PID_OF_QEMU/fd and looking for the disk image file.
>
> When the disk is set to cache=writethrough on one of the same VMs, I see frequent
> fdatasync(2) calls (every few seconds). However, when I change the disk over to
> cache=writeback, since boot I have not yet seen a single fdatasync(2) call, even
> after writing data 2x the amount of RAM:

There is a misconception here.  Writing data to disk does not make it
persistent across power failure.  The behavior you observed is
actually expected.

Flushing the disk cache is a deliberate operation that applications
must perform to ensure data is safe on disk.

You can use sync(1) to manually flush file system buffers for testing.
On the host you should see an fdatasync(2).

>> > I recently experienced UPS failure on several hosts which caused a hard
>> > shutdown. After restarting, 3 of the guests had corruption on their disks
>> > and
>> > required a fairly long fsck to fix. Afterwards, data that had been written
>> > to
>> > the disks several hours before the crash was corrupted, which makes me
>> > think
>> > that it was never fsync()-ed to the non-volatile storage.
>>
>> What exactly was the "corruption" you encountered?  Which application,
>> error message, etc.
>
> Two of the servers are web servers with apache2. In one case, a python daemon
> copies JPGs onto the server - the last 100 copied onto the server were corrupted.
> In another case, some files had been uploaded several days prior to the www-root,
> but after the hard reset said files were no longer present in the filesystem.

Did the Python daemon fsync the files and directories it modified/created?

Did you sync(1) after copying files to www-root?

Also, you didn't explain what "corrupted" means.  Where the jpg files
missing, were they zero bytes in size, were they filled with junk,
etc?

>> > Is it safe in this setup to use cache=writeback? Or, should I use
>> > cache=writethrough instead?
>>
>> Ubuntu 12.04 is recent and sends write cache flushes.
>>
>> Are you sure the file system and/or application workload are flushing
>> the disk cache?  Please check the mount options and application-specific
>> configuration.
>
> The mount options for the ext4 filesystem in the VM in both cases are:
> rw,relatime,errors=remount-ro,data=ordered
>
> Similarly, the host's ext4 filesystem holding the images is mounted with:
> rw,relatime,data=ordered
>
> I did not see any errors in the kernel log in the guest, probably because the
> root filesystem was read-only until the fsck had completed.

Good, that is a sane setup.  I guess the behavior you saw would also
happen on a physical machine, it is necessary to fsync the file or
sync the file system to guarantee that data is safely on disk.

Stefan

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Andrew Martin]

----- Original Message -----
> From: "Stefan Hajnoczi" <stefanha@gmail.com>
> To: "Andrew Martin" <amartin@xes-inc.com>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Sent: Thursday, August 21, 2014 7:59:50 AM
> Subject: Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later
>
> > When the disk is set to cache=writethrough on one of the same VMs, I see
> > frequent
> > fdatasync(2) calls (every few seconds). However, when I change the disk
> > over to
> > cache=writeback, since boot I have not yet seen a single fdatasync(2) call,
> > even
> > after writing data 2x the amount of RAM:
> 
> There is a misconception here.  Writing data to disk does not make it
> persistent across power failure.  The behavior you observed is
> actually expected.
> 
> Flushing the disk cache is a deliberate operation that applications
> must perform to ensure data is safe on disk.
> 
> You can use sync(1) to manually flush file system buffers for testing.
> On the host you should see an fdatasync(2).
> 
I booted a VM into both writeback and none, and in both cases wrote some data to
the disk and then issued sync(1) as root while stracing the QEMU process on the
host. In both cases, I did not see any fdatasync, fsync, or sync syscalls passed.

> >> > I recently experienced UPS failure on several hosts which caused a hard
> >> > shutdown. After restarting, 3 of the guests had corruption on their
> >> > disks
> >> > and
> >> > required a fairly long fsck to fix. Afterwards, data that had been
> >> > written
> >> > to
> >> > the disks several hours before the crash was corrupted, which makes me
> >> > think
> >> > that it was never fsync()-ed to the non-volatile storage.
> >>
> >> What exactly was the "corruption" you encountered?  Which application,
> >> error message, etc.
> >
> > Two of the servers are web servers with apache2. In one case, a python
> > daemon
> > copies JPGs onto the server - the last 100 copied onto the server were
> > corrupted.
> > In another case, some files had been uploaded several days prior to the
> > www-root,
> > but after the hard reset said files were no longer present in the
> > filesystem.
> 
> Did the Python daemon fsync the files and directories it modified/created?
> 
> Did you sync(1) after copying files to www-root?
> 
> Also, you didn't explain what "corrupted" means.  Where the jpg files
> missing, were they zero bytes in size, were they filled with junk,
> etc?
> 
The jpgs appeared to be a normal size, but were filled with junk. The files
uploaded by apache2 were missing from the filesystem.

Even if the python daemon or apache2 did not fsync the modified files, isn't 
there some action that the OS takes periodically to flush dirty pages to disk? 
This seems to be implied in the SuSE documentation:
https://www.suse.com/documentation/sles11/book_kvm/data/sect1_1_chapter_book_kvm.html
"the normal page cache management will handle commitment to the storage device."


In the case of the files uploaded by apache2, they were added to the server days 
before  the power outage, so it seems like there would have been ample time for 
those changes to have been flushed.

Thanks!

Andrew

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Paolo Bonzini]

Il 25/08/2014 20:13, Andrew Martin ha scritto:
> Even if the python daemon or apache2 did not fsync the modified files, isn't 
> there some action that the OS takes periodically to flush dirty pages to disk? 
> This seems to be implied in the SuSE documentation:
> https://www.suse.com/documentation/sles11/book_kvm/data/sect1_1_chapter_book_kvm.html
> "the normal page cache management will handle commitment to the storage device."

... using fsync :)

> In the case of the files uploaded by apache2, they were added to the server days 
> before  the power outage, so it seems like there would have been ample time for 
> those changes to have been flushed.

Remember that cache=writeback has a huge cache, whose size is several
gigabytes.

At some point there was a QEMU bug that caused the guest not to do
fsyncs.  It was fixed by commit ef5bc96268.

Paolo

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Andrew Martin]

----- Original Message -----
> From: "Paolo Bonzini" <pbonzini@redhat.com>
> To: "Andrew Martin" <amartin@xes-inc.com>, "Stefan Hajnoczi" <stefanha@gmail.com>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Sent: Tuesday, August 26, 2014 2:03:18 AM
> Subject: Re: Using cache=writeback safely on qemu 1.4.0 and later
> 
> > In the case of the files uploaded by apache2, they were added to the server
> > days
> > before  the power outage, so it seems like there would have been ample time
> > for
> > those changes to have been flushed.
> 
> Remember that cache=writeback has a huge cache, whose size is several
> gigabytes.
> 
> At some point there was a QEMU bug that caused the guest not to do
> fsyncs.  It was fixed by commit ef5bc96268.

Paolo,

Thanks for the info about this bug - I am using QEMU 1.4.0 (released on 02/15/13)
and ef5bc96268 was added on 09/20/13, so the bug had not been fixed in the version
of QEMU I am using. I have had trouble reproducing it on-demand - was there specific
behavior you used to reproduce it, or just the lack of fsyncs or fdatasyncs when
stracing the QEMU process when using cache=writeback?

Thanks,

Andrew

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Paolo Bonzini]

Il 27/08/2014 16:23, Andrew Martin ha scritto:
> Thanks for the info about this bug - I am using QEMU 1.4.0 (released on 02/15/13)
> and ef5bc96268 was added on 09/20/13, so the bug had not been fixed in the version
> of QEMU I am using. I have had trouble reproducing it on-demand - was there specific
> behavior you used to reproduce it, or just the lack of fsyncs or fdatasyncs when
> stracing the QEMU process when using cache=writeback?

Just the lack of fdatasyncs.

Paolo

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Andrew Martin]

----- Original Message -----
> From: "Paolo Bonzini" <pbonzini@redhat.com>
> To: "Andrew Martin" <amartin@xes-inc.com>
> Cc: "Stefan Hajnoczi" <stefanha@gmail.com>, "qemu-devel" <qemu-devel@nongnu.org>
> Sent: Wednesday, August 27, 2014 9:34:27 AM
> Subject: Re: Using cache=writeback safely on qemu 1.4.0 and later
> 
> Il 27/08/2014 16:23, Andrew Martin ha scritto:
> > Thanks for the info about this bug - I am using QEMU 1.4.0 (released on
> > 02/15/13)
> > and ef5bc96268 was added on 09/20/13, so the bug had not been fixed in the
> > version
> > of QEMU I am using. I have had trouble reproducing it on-demand - was there
> > specific
> > behavior you used to reproduce it, or just the lack of fsyncs or fdatasyncs
> > when
> > stracing the QEMU process when using cache=writeback?
> 
> Just the lack of fdatasyncs.
> 
Paolo,

I also did not see any fdatasyncs when using cache=none - would that also be
explained by this bug?

Thanks,

Andrew

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Paolo Bonzini]

> > Just the lack of fdatasyncs.
> 
> I also did not see any fdatasyncs when using cache=none - would that also be
> explained by this bug?

Yes, cache=none and cache=writeback are the same as far as the _disk_ cache
is concerned, and you need fdatasyncs to flush the disk cache too.

Paolo

Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2626 bytes --]

On Mon, Aug 25, 2014 at 01:13:09PM -0500, Andrew Martin wrote:
> > >> > I recently experienced UPS failure on several hosts which caused a hard
> > >> > shutdown. After restarting, 3 of the guests had corruption on their
> > >> > disks
> > >> > and
> > >> > required a fairly long fsck to fix. Afterwards, data that had been
> > >> > written
> > >> > to
> > >> > the disks several hours before the crash was corrupted, which makes me
> > >> > think
> > >> > that it was never fsync()-ed to the non-volatile storage.
> > >>
> > >> What exactly was the "corruption" you encountered?  Which application,
> > >> error message, etc.
> > >
> > > Two of the servers are web servers with apache2. In one case, a python
> > > daemon
> > > copies JPGs onto the server - the last 100 copied onto the server were
> > > corrupted.
> > > In another case, some files had been uploaded several days prior to the
> > > www-root,
> > > but after the hard reset said files were no longer present in the
> > > filesystem.
> > 
> > Did the Python daemon fsync the files and directories it modified/created?
> > 
> > Did you sync(1) after copying files to www-root?
> > 
> > Also, you didn't explain what "corrupted" means.  Where the jpg files
> > missing, were they zero bytes in size, were they filled with junk,
> > etc?
> > 
> The jpgs appeared to be a normal size, but were filled with junk. The files
> uploaded by apache2 were missing from the filesystem.
> 
> Even if the python daemon or apache2 did not fsync the modified files, isn't 
> there some action that the OS takes periodically to flush dirty pages to disk? 
> This seems to be implied in the SuSE documentation:
> https://www.suse.com/documentation/sles11/book_kvm/data/sect1_1_chapter_book_kvm.html
> "the normal page cache management will handle commitment to the storage device."
> 
> 
> In the case of the files uploaded by apache2, they were added to the server days 
> before  the power outage, so it seems like there would have been ample time for 
> those changes to have been flushed.

In the general case of copying/creating some files and hoping that they
will be persistent, it usually works.  If you want to be 100% sure you
still need to flush the cache explicitly.

It doesn't work when updates are made to data on disk and the ordering
matters (e.g. wrong ordering could corrupt data or cause it to be lost).
In that case relying on the kernel to flush dirty buffers periodically
is not a feasible approach because you don't know when the will happen
and therefore have no control over ordering.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]