All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [Bug 1368815] [NEW] qemu-img convert intermittently corrupts output images
@ 2014-09-12 14:58 Michael Steffens
  2014-09-12 15:08 ` [Qemu-devel] [Bug 1368815] " Michael Steffens
                   ` (57 more replies)
  0 siblings, 58 replies; 59+ messages in thread
From: Michael Steffens @ 2014-09-12 14:58 UTC (permalink / raw)
  To: qemu-devel

Public bug reported:

-- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on
Ubuntu 14.04 using Ext4 filesystems.

The command

  qemu-img convert -O raw inputimage.qcow2 outputimage.raw

intermittently creates corrupted output images, when the input image is
not yet fully synchronized to disk. While the issue has actually been
discovered in operation of of OpenStack nova, it can be reproduced
"easily" on command line using

  cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH
$DST_PATH && cksum $DST_PATH

on filesystems exposing this behavior. (The difficult part of this
exercise is to prepare a filesystem to reliably trigger this race. On my
test machine some filesystems are affected while other aren't, and
unfortunately I haven't found the relevant difference between them, yet.
Possible it's timing issues completely out of userspace control ...)

The root cause, however, is the same as in

  http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html

and it can be solved the same way as suggested in

  http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html

In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change

    f.fm.fm_flags = 0;

to

    f.fm.fm_flags = FIEMAP_FLAG_SYNC;

As discussed in the thread mentioned above, retrieving a page cache
coherent map of file extents is possible only after fsync on that file.

See also

  https://bugs.launchpad.net/nova/+bug/1350766

In that bug report filed against nova, fsync had been suggested to be
performed by the framework invoking qemu-img. However, as the choice of
fiemap -- implying this otherwise unneeded fsync of a temporary file  --
is not made by the caller but by qemu-img, I agree with the nova bug
reviewer's objection to put it into nova. The fsync should instead be
triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically
intended for that purpose.

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: fiemap nova openstack qemu-img

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1368815

Title:
  qemu-img convert intermittently corrupts output images

Status in QEMU:
  New

Bug description:
  -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on
  Ubuntu 14.04 using Ext4 filesystems.

  The command

    qemu-img convert -O raw inputimage.qcow2 outputimage.raw

  intermittently creates corrupted output images, when the input image
  is not yet fully synchronized to disk. While the issue has actually
  been discovered in operation of of OpenStack nova, it can be
  reproduced "easily" on command line using

    cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH
  $DST_PATH && cksum $DST_PATH

  on filesystems exposing this behavior. (The difficult part of this
  exercise is to prepare a filesystem to reliably trigger this race. On
  my test machine some filesystems are affected while other aren't, and
  unfortunately I haven't found the relevant difference between them,
  yet. Possible it's timing issues completely out of userspace control
  ...)

  The root cause, however, is the same as in

    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html

  and it can be solved the same way as suggested in

    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html

  In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change

      f.fm.fm_flags = 0;

  to

      f.fm.fm_flags = FIEMAP_FLAG_SYNC;

  As discussed in the thread mentioned above, retrieving a page cache
  coherent map of file extents is possible only after fsync on that
  file.

  See also

    https://bugs.launchpad.net/nova/+bug/1350766

  In that bug report filed against nova, fsync had been suggested to be
  performed by the framework invoking qemu-img. However, as the choice
  of fiemap -- implying this otherwise unneeded fsync of a temporary
  file  -- is not made by the caller but by qemu-img, I agree with the
  nova bug reviewer's objection to put it into nova. The fsync should
  instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC,
  specifically intended for that purpose.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1368815/+subscriptions

^ permalink raw reply	[flat|nested] 59+ messages in thread

end of thread, other threads:[~2016-07-19  8:21 UTC | newest]

Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-12 14:58 [Qemu-devel] [Bug 1368815] [NEW] qemu-img convert intermittently corrupts output images Michael Steffens
2014-09-12 15:08 ` [Qemu-devel] [Bug 1368815] " Michael Steffens
2014-09-12 15:13 ` Matt Riedemann
2014-09-14 21:04 ` Michael Steffens
2014-09-15 10:08 ` Michael Steffens
2014-09-15 10:30 ` Pádraig Brady
2014-09-15 12:52 ` Michael Steffens
2014-09-15 15:16 ` Matt Riedemann
2014-09-15 18:27 ` Serge Hallyn
2014-09-30  4:51 ` Tony Breeds
2014-10-22 23:21 ` Tony Breeds
2014-10-23 18:22 ` Serge Hallyn
2014-10-27  5:27 ` Launchpad Bug Tracker
2014-10-30  6:40 ` Launchpad Bug Tracker
2014-11-20 22:23 ` Serge Hallyn
2014-11-20 22:42 ` Tony Breeds
2014-11-21  4:17 ` Serge Hallyn
2014-11-21  4:47 ` Tony Breeds
2014-11-25 11:44 ` Kevin Wolf
2014-11-25 17:22 ` Chris J Arges
2014-11-25 17:24 ` [Qemu-devel] [Bug 1368815] Please test proposed package Chris J Arges
2014-12-01  8:23 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images Michael Steffens
2014-12-01  8:32 ` Michael Steffens
2014-12-03  5:14 ` Mathew Hodson
2014-12-08 18:40 ` [Qemu-devel] [Bug 1368815] Update Released Brian Murray
2014-12-08 18:46 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images Launchpad Bug Tracker
2014-12-10 16:38 ` Serge Hallyn
2014-12-10 18:06 ` Vladimir Kuklin
2014-12-10 18:32 ` Serge Hallyn
2014-12-10 20:34 ` Chris J Arges
2014-12-10 20:49 ` Chris J Arges
2014-12-10 20:50 ` Dmitry Mescheryakov
2014-12-10 21:54 ` Tony Breeds
2014-12-11  3:54 ` Chris J Arges
2014-12-11  3:57 ` Chris J Arges
2014-12-11  4:47 ` Tony Breeds
2014-12-11 17:53 ` Launchpad Bug Tracker
2014-12-11 20:11 ` Vish Ishaya
2014-12-12  1:56 ` Tony Breeds
2014-12-12  1:58 ` Tony Breeds
2014-12-12  3:28 ` OpenStack Infra
2014-12-15 16:26 ` Jay Bryant
2014-12-16 12:31 ` Alexei Sheplyakov
2014-12-16 12:33 ` Alexei Sheplyakov
2014-12-16 22:00 ` Tony Breeds
2014-12-17  6:44 ` Alexei Sheplyakov
2014-12-21 19:29 ` [Qemu-devel] [Bug 1368815] Change abandoned on cinder (master) OpenStack Infra
2014-12-23  0:44 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images OpenStack Infra
2015-01-26 15:27 ` [Qemu-devel] [Bug 1368815] Change abandoned on cinder (master) OpenStack Infra
2015-01-26 20:32 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images Mike Perez
2015-01-26 22:27 ` Tony Breeds
2015-01-27 17:23 ` Tony Breeds
2015-01-28  0:41 ` [Qemu-devel] [Bug 1368815] Change abandoned on nova (master) OpenStack Infra
2015-02-05 10:43 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images Thierry Carrez
2015-02-12 15:09 ` Davanum Srinivas (DIMS)
2015-02-24  9:25 ` Thierry Carrez
2015-05-08 22:29 ` [Qemu-devel] [Bug 1368815] Change abandoned on cinder (master) OpenStack Infra
2015-10-30  1:03 ` [Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images Eric Harney
2016-07-19  8:14 ` T. Huth

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.