From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: Notes on block I/O data integrity Date: Thu, 27 Aug 2009 02:15:42 +0200 Message-ID: <20090827001542.GA8317@lst.de> References: <20090825181120.GA4863@lst.de> <20090825202508.GB4807@nik-comp.linuxbox.cz> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="wac7ysb48OaltWcw" Cc: Christoph Hellwig , qemu-devel@nongnu.org, kvm@vger.kernel.org, rusty@rustcorp.com.au, nikola.ciprich@linuxbox.cz, kopi@linuxbox.cz To: Nikola Ciprich Return-path: Received: from verein.lst.de ([213.95.11.210]:34987 "EHLO verein.lst.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750761AbZH0APw (ORCPT ); Wed, 26 Aug 2009 20:15:52 -0400 Content-Disposition: inline In-Reply-To: <20090825202508.GB4807@nik-comp.linuxbox.cz> Sender: kvm-owner@vger.kernel.org List-ID: --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Aug 25, 2009 at 10:25:08PM +0200, Nikola Ciprich wrote: > Hello Christopher, > > thanks a lot vor this overview, it answers a lot of my questions! > May I suggest You put it somewhere on the wiki so it doesn't get > forgotten in the maillist only? I'll rather try to get the worst issues fixed ASAP. > It also rises few new questions though. We have experienced postgresql > database corruptions lately, two times to be exact. First time, I blamed > server crash, but lately (freshly created) database got corrupted for the > second time and there were no crashes since the initialisation. The server > hardware is surely OK. I didn't have much time to look into this > yet, but Your mail just poked me to return to the subject. The situation > is a bit more complex, as there are additional two layers of storage there: > we're using SATA/SAS drives, network-mirrored by DRBD, clustered LVM on top > of those, and finally qemu-kvm using virtio on top of created logical > volumes. So there are plenty of possible culprits, but Your mention of virtio > unsafeness while using cache=writethrough (which is the default for drive > types other then qcow) leads me to suspicion that this might be the reason of > the problem. Databases are sensitive for requests reordering, so I guess > using virtio for postgres storage was quite stupid from me :( > So my question is, could You please advise me a bit on the storage > configuration? virtio performed much better then SCSI, but of course > data integrity is crucial, so would You suggest rather using SCSI? > DRBD doesn't have problem with barriers, clustered LVM SHOULD not > have problems with it, as we're using just striped volumes, but I'll > check it to be sure. So is it safe for me to keep cache=writethrough > for the database volume? I'm pretty sure one of the many laters in your setup will not pass through write barriers, so defintively make sure your write caches are disabled. Also right now virtio is not a good idea for data integrity. The guest side fix for a setup with cache=writethrough or cache=none on block device without volatile disk write cache is however a trivial one line patch I've already submitted. I've attached it below for reference: --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=virtio-blk-drain Subject: [PATCH] virtio-blk: set QUEUE_ORDERED_DRAIN by default From: Christoph Hellwig Currently virtio-blk doesn't set any QUEUE_ORDERED_ flag by default, which means it does not allow filesystems to use barriers. But the typical use case for virtio-blk is to use a backed that uses synchronous I/O, and in that case we can simply set QUEUE_ORDERED_DRAIN to make the block layer drain the request queue around barrier I/O and provide the semantics that the filesystems need. This is what the SCSI disk driver does for disks that have the write cache disabled. With this patch we incorrectly advertise barrier support if someone configure qemu with write back caching. While this displays wrong information in the guest there is nothing that guest could have done even if we rightfully told it that we do not support any barriers. Signed-off-by: Christoph Hellwig Index: linux-2.6/drivers/block/virtio_blk.c =================================================================== --- linux-2.6.orig/drivers/block/virtio_blk.c 2009-08-20 17:41:37.019718433 -0300 +++ linux-2.6/drivers/block/virtio_blk.c 2009-08-20 17:45:40.511747922 -0300 @@ -336,9 +336,16 @@ static int __devinit virtblk_probe(struc vblk->disk->driverfs_dev = &vdev->dev; index++; - /* If barriers are supported, tell block layer that queue is ordered */ + /* + * If barriers are supported, tell block layer that queue is ordered. + * + * If no barriers are supported assume the host uses synchronous + * writes and just drain the the queue before and after the barrier. + */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_BARRIER)) blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_TAG, NULL); + else + blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_DRAIN, NULL); /* If disk is read-only in the host, the guest should obey */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_RO)) --wac7ysb48OaltWcw-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MgSed-0006Ad-FN for qemu-devel@nongnu.org; Wed, 26 Aug 2009 20:15:51 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MgSeY-00064D-HJ for qemu-devel@nongnu.org; Wed, 26 Aug 2009 20:15:50 -0400 Received: from [199.232.76.173] (port=42702 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MgSeY-00063y-71 for qemu-devel@nongnu.org; Wed, 26 Aug 2009 20:15:46 -0400 Received: from verein.lst.de ([213.95.11.210]:35632) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_3DES_EDE_CBC_SHA1:24) (Exim 4.60) (envelope-from ) id 1MgSeX-0001DL-IG for qemu-devel@nongnu.org; Wed, 26 Aug 2009 20:15:45 -0400 Date: Thu, 27 Aug 2009 02:15:42 +0200 From: Christoph Hellwig Message-ID: <20090827001542.GA8317@lst.de> References: <20090825181120.GA4863@lst.de> <20090825202508.GB4807@nik-comp.linuxbox.cz> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="wac7ysb48OaltWcw" Content-Disposition: inline In-Reply-To: <20090825202508.GB4807@nik-comp.linuxbox.cz> Subject: [Qemu-devel] Re: Notes on block I/O data integrity List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Nikola Ciprich Cc: kopi@linuxbox.cz, kvm@vger.kernel.org, rusty@rustcorp.com.au, qemu-devel@nongnu.org, nikola.ciprich@linuxbox.cz, Christoph Hellwig --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Aug 25, 2009 at 10:25:08PM +0200, Nikola Ciprich wrote: > Hello Christopher, > > thanks a lot vor this overview, it answers a lot of my questions! > May I suggest You put it somewhere on the wiki so it doesn't get > forgotten in the maillist only? I'll rather try to get the worst issues fixed ASAP. > It also rises few new questions though. We have experienced postgresql > database corruptions lately, two times to be exact. First time, I blamed > server crash, but lately (freshly created) database got corrupted for the > second time and there were no crashes since the initialisation. The server > hardware is surely OK. I didn't have much time to look into this > yet, but Your mail just poked me to return to the subject. The situation > is a bit more complex, as there are additional two layers of storage there: > we're using SATA/SAS drives, network-mirrored by DRBD, clustered LVM on top > of those, and finally qemu-kvm using virtio on top of created logical > volumes. So there are plenty of possible culprits, but Your mention of virtio > unsafeness while using cache=writethrough (which is the default for drive > types other then qcow) leads me to suspicion that this might be the reason of > the problem. Databases are sensitive for requests reordering, so I guess > using virtio for postgres storage was quite stupid from me :( > So my question is, could You please advise me a bit on the storage > configuration? virtio performed much better then SCSI, but of course > data integrity is crucial, so would You suggest rather using SCSI? > DRBD doesn't have problem with barriers, clustered LVM SHOULD not > have problems with it, as we're using just striped volumes, but I'll > check it to be sure. So is it safe for me to keep cache=writethrough > for the database volume? I'm pretty sure one of the many laters in your setup will not pass through write barriers, so defintively make sure your write caches are disabled. Also right now virtio is not a good idea for data integrity. The guest side fix for a setup with cache=writethrough or cache=none on block device without volatile disk write cache is however a trivial one line patch I've already submitted. I've attached it below for reference: --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=virtio-blk-drain Subject: [PATCH] virtio-blk: set QUEUE_ORDERED_DRAIN by default From: Christoph Hellwig Currently virtio-blk doesn't set any QUEUE_ORDERED_ flag by default, which means it does not allow filesystems to use barriers. But the typical use case for virtio-blk is to use a backed that uses synchronous I/O, and in that case we can simply set QUEUE_ORDERED_DRAIN to make the block layer drain the request queue around barrier I/O and provide the semantics that the filesystems need. This is what the SCSI disk driver does for disks that have the write cache disabled. With this patch we incorrectly advertise barrier support if someone configure qemu with write back caching. While this displays wrong information in the guest there is nothing that guest could have done even if we rightfully told it that we do not support any barriers. Signed-off-by: Christoph Hellwig Index: linux-2.6/drivers/block/virtio_blk.c =================================================================== --- linux-2.6.orig/drivers/block/virtio_blk.c 2009-08-20 17:41:37.019718433 -0300 +++ linux-2.6/drivers/block/virtio_blk.c 2009-08-20 17:45:40.511747922 -0300 @@ -336,9 +336,16 @@ static int __devinit virtblk_probe(struc vblk->disk->driverfs_dev = &vdev->dev; index++; - /* If barriers are supported, tell block layer that queue is ordered */ + /* + * If barriers are supported, tell block layer that queue is ordered. + * + * If no barriers are supported assume the host uses synchronous + * writes and just drain the the queue before and after the barrier. + */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_BARRIER)) blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_TAG, NULL); + else + blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_DRAIN, NULL); /* If disk is read-only in the host, the guest should obey */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_RO)) --wac7ysb48OaltWcw--