Re: [Qemu-devel] 'qemu-nbd' explicit flush

[Stefan Hajnoczi <stefanha@gmail.com>]

On Tue, May 28, 2013 at 06:00:08PM +0000, Mark Trumpold wrote:
> 
> >-----Original Message-----
> >From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> >Sent: Monday, May 27, 2013 05:36 AM
> >To: 'Mark Trumpold'
> >Cc: 'Paolo Bonzini', qemu-devel@nongnu.org, markt@tachyon.net
> >Subject: Re: 'qemu-nbd' explicit flush
> >
> >On Sat, May 25, 2013 at 09:42:08AM -0800, Mark Trumpold wrote:
> >> On 5/24/13 1:05 AM, "Stefan Hajnoczi" <stefanha@gmail.com> wrote:
> >> >On Thu, May 23, 2013 at 09:58:31PM +0000, Mark Trumpold wrote:
> >> >One thing to be careful of is whether these operations are asynchronous.
> >> >The signal is asynchronous, you have no way of knowing when qemu-nbd is
> >> >finished flushing to the physical disk.
> >>
> >> Right, of course.  I missed the obvious.
> >
> >I missed something too.  Paolo may have already hinted at this when he
> >posted a dd oflag=sync command-line option:
> >
> >blockdev --flushbufs is the wrong tool because ioctl(BLKFLSBUF) only
> >writes out dirty pages to the block device.  It does *not* guarantee to
> >send a flush request to the device.
> >
> >Therefore, the underlying image file may not be put into an up-to-date
> >state by qemu-nbd.
> >
> >
> >I suggest trying the following instead of blockdev --flushbufs:
> >
> >  python -c 'import os; os.fsync(open("/dev/loopX", "r+b"))'
> >
> >This should do the same as blockdev --flushbufs *plus* it sends and
> >waits for the NBD FLUSH command.
> >
> >You may have to play with this command-line a little but the main idea
> >is to open the block device and fsync it.
> >
> >Stefan
> >
> 
> Hi Stefan,
> 
> One of my early experiments was adding a command line option to 'qemu-nbd' that did an open on 'device' (similar to the -c option), and then calling 'fsync' on the 'device'.  By itself, I did not get a complete flush to disk.  Was I missing something?
> 
> Empirically, the signal solution (blockdev --flushbufs plus 'bdrv_flush_all') was keeping my disk consistent.  My unit test exercises the flush and snapshot pretty rigorously; that is, it never passed before with 'qemu-nbd --cache=writeback ...'.  However, I did not want to rely on 'sleep' for the race condition.
> 
> Is there any opportunity with the nbd client socket interface?  The advantage for me there is not modifying 'qemu-nbd' source.

I'm suggesting that you don't need to modify qemu-nbd.  If your host is
running nbd.ko with flush support, then it should be enough to open the
device and issue fsync(2).

You can verify this using tcpdump(8) and checking that the NBD FLUSH
command is really being sent by the host kernel.  If not, double check
you're using the latest nbd.ko.

Stefan