From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39733) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPQHM-00084W-Ec for qemu-devel@nongnu.org; Tue, 19 Jul 2016 04:21:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bPQHI-0000P3-3V for qemu-devel@nongnu.org; Tue, 19 Jul 2016 04:21:23 -0400 Received: from indium.canonical.com ([91.189.90.7]:59856) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPQHH-0000Oq-R6 for qemu-devel@nongnu.org; Tue, 19 Jul 2016 04:21:20 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.76 #1 (Debian)) id 1bPQHG-00063x-SD for ; Tue, 19 Jul 2016 08:21:19 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 0BAD82E80E2 for ; Tue, 19 Jul 2016 08:21:16 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Tue, 19 Jul 2016 08:14:33 -0000 From: "T. Huth" <1368815@bugs.launchpad.net> Reply-To: Bug 1368815 <1368815@bugs.launchpad.net> Sender: bounces@canonical.com References: <20140912145823.442.8812.malonedeb@gac.canonical.com> Message-Id: <20160719081433.9334.1197.malone@soybean.canonical.com> Errors-To: bounces@canonical.com Subject: [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org According to comment #8 the fixes have been included in the upstream QEMU repository, so setting the status to "Fix released" now. ** Changed in: qemu Status: In Progress =3D> Fix Released -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1368815 Title: qemu-img convert intermittently corrupts output images Status in Cinder: Won't Fix Status in OpenStack Compute (nova): Won't Fix Status in QEMU: Fix Released Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Trusty: Fix Released Status in qemu source package in Utopic: Fix Released Status in qemu source package in Vivid: Fix Released Bug description: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D Impact: occasional image corruption (any format on local filesystem) Test case: see the qemu-img command below Regression potential: this cherrypicks a patch from upstream to a not-ins= ignificantly older qemu source tree. While the cherrypick seems sane, it's= possible that there are subtle interactions with the other delta. I'd rea= lly like for a full qa-regression-test qemu testcase to be run against this= package. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on Ubuntu 14.04 using Ext4 filesystems. The command =C2=A0=C2=A0qemu-img convert -O raw inputimage.qcow2 outputimage.raw intermittently creates corrupted output images, when the input image is not yet fully synchronized to disk. While the issue has actually been discovered in operation of of OpenStack nova, it can be reproduced "easily" on command line using =C2=A0=C2=A0cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $T= MP_PATH $DST_PATH && cksum $DST_PATH on filesystems exposing this behavior. (The difficult part of this exercise is to prepare a filesystem to reliably trigger this race. On my test machine some filesystems are affected while other aren't, and unfortunately I haven't found the relevant difference between them, yet. Possible it's timing issues completely out of userspace control ...) The root cause, however, is the same as in =C2=A0=C2=A0http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.= html and it can be solved the same way as suggested in =C2=A0=C2=A0http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.= html In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change =C2=A0=C2=A0=C2=A0=C2=A0f.fm.fm_flags =3D 0; to =C2=A0=C2=A0=C2=A0=C2=A0f.fm.fm_flags =3D FIEMAP_FLAG_SYNC; As discussed in the thread mentioned above, retrieving a page cache coherent map of file extents is possible only after fsync on that file. See also =C2=A0=C2=A0https://bugs.launchpad.net/nova/+bug/1350766 In that bug report filed against nova, fsync had been suggested to be performed by the framework invoking qemu-img. However, as the choice of fiemap -- implying this otherwise unneeded fsync of a temporary file -- is not made by the caller but by qemu-img, I agree with the nova bug reviewer's objection to put it into nova. The fsync should instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically intended for that purpose. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1368815/+subscriptions