[Qemu-devel] [Bug 1776920] [NEW] qemu-img convert on Mac OSX creates corrupt images

[Qemu-devel] [Bug 1776920] [NEW] qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Public bug reported:

An image created by qemu-img create, then modified by another program is
converted to bad/corrupt image when using convert sub command on Mac
OSX. The same convert works on Linux. The version of qemu-img is 2.12.

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[ Christian Ehrhardt ]

Can this be done with like a 1M example file that you could copy off in
all stages.

Provide the commands you use like
1. create
2. do ??
3. convert

Then for Mac and Linux you'd have M1/M2/M3 L1/L2/L3 files that can all
be attached here to be evaluated for what might be broken.

IMHO In the current state there is neither enough data for good
debugging, not enough steps to reproduce what exactly you faced.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

I will provide all necessary info. Unfortunately the smallest image I can
provide is around 10M.

What is M1/M2/M3 L1/L2/L3 file?

Waldek

On Thu, Jun 14, 2018 at 10:46 AM,  Christian Ehrhardt  <
1776920@bugs.launchpad.net> wrote:

> Can this be done with like a 1M example file that you could copy off in
> all stages.
>
> Provide the commands you use like
> 1. create
> 2. do ??
> 3. convert
>
> Then for Mac and Linux you'd have M1/M2/M3 L1/L2/L3 files that can all
> be attached here to be evaluated for what might be broken.
>
> IMHO In the current state there is neither enough data for good
> debugging, not enough steps to reproduce what exactly you faced.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1776920
>
> Title:
>   qemu-img convert on Mac OSX creates corrupt images
>
> Status in QEMU:
>   New
>
> Bug description:
>   An image created by qemu-img create, then modified by another program
>   is converted to bad/corrupt image when using convert sub command on
>   Mac OSX. The same convert works on Linux. The version of qemu-img is
>   2.12.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions
>

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Christian Ehrhardt]

>
> I will provide all necessary info. Unfortunately the smallest image I can
> provide is around 10M.
>
> What is M1/M2/M3 L1/L2/L3 file?
>

Just a suggestion how you could name the files
Linux-at-step-1 would be L1 and similar.

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

Are you converting to a destination on APFS? If yes, do you have the
ability to easily reproduce this by converting to a non-APFS destination
and seeing if that breaks too?

Thanks,
--js

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

I believe I have distilled entire process to few repeatable steps that
can be fully reproduced on my Mac. The binary source files - - boot.bin
and lzloader.elf - were created on my Linux VM running in VirtualBox on
same Mac but I do not think it matters as the execution completely
happens on Mac.

The steps executed on my mac:
1. dd if=boot.bin of=image.img > /dev/null 2>&1
2. dd if=lzloader.elf of=image.img conv=notrunc seek=128 > /dev/null 2>&1
3. qemu-img convert image.img -O qcow2 image.qemu
4. qemu-img convert image.qemu -O qcow2 image2.qemu

The end result:
ll image*
-rw-r--r--  1 *** *** 6684672 Jun 14 17:17 image.img
-rw-r--r--  1 *** *** 7012352 Jun 14 17:40 image.qemu
-rw-r--r--  1 *** ***  196616 Jun 14 17:40 image2.qemu

The result of regular compare:
qemu-img compare image.qemu image2.qemu
Images are identical.

The result of strict compare:
qemu-img compare -s image.qemu image2.qemu
Strict mode: Offset 0 block status mismatch!

Images are clearly different.

The same 4 steps executed on my Linux VM behave correctly - image.qemu
is TRULY identical with image2.qemu.

Qemu-img on my Mac:
qemu-img --version
qemu-img version 2.12.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

Details about Mac and OSX version:
Model Name:	MacBook Pro
  Model Identifier:	MacBookPro13,3
  Processor Name:	Intel Core i7
  Processor Speed:	2.7 GHz
  Number of Processors:	1
  Total Number of Cores:	4
  L2 Cache (per Core):	256 KB
  L3 Cache:	8 MB
  Memory:	16 GB

Mac filesystem:
Mount Point:	/
  File System:	APFS
  Writable:	Yes
  Ignore Ownership:	No
  BSD Name:	disk1s1
  Physical Drive:
  Device Name:	APPLE SSD SM0512L
  Media Name:	AppleAPFSMedia
  Medium Type:	SSD
  Protocol:	PCI-Express
  Internal:	Yes
  Partition Map Type:	Unknown

I am also attaching both source files and images for examination.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Source file 1

** Attachment added: "lzloader.elf"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152720/+files/lzloader.elf

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Source file 2

** Attachment added: "boot.bin"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152721/+files/boot.bin

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Raw image created by dd in steps 1 and 2.

** Attachment added: "image.img"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152722/+files/image.img

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Original qcow2 image converted from raw image in step 3.

** Attachment added: "image.qemu"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152723/+files/image.qemu

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

The corrupt qcow2 image created by converting image.qemu in step 4.

** Attachment added: "image2.qemu"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152724/+files/image2.qemu

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Also if I use the same image.qemu file and convert to vmdk format I get
even smaller file which for sure is wrong as well:

qemu-img convert image.qemu -O vmdk image2.vbox

ll image*
-rw-r--r--  1 *** ***  6684672 Jun 14 17:17 image.img
-rw-r--r--  1 *** ***  7012352 Jun 14 17:40 image.qemu
-rw-r--r--  1 *** ***   196616 Jun 14 17:40 image2.qemu
-rw-r--r--  1 *** ***    65536 Jun 14 18:00 image2.vbox

** Attachment added: "image2.vbox"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5152725/+files/image2.vbox

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Have I provided all necessary data and other details?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

I haven't had the time to look just yet.

If there's a developer out there with a Mac has the bandwidth to take a
look at this I'd be grateful. (I don't have access to one presently.)

I see that the target filesystem is APFS however -- I think we might
have a bug in our APFS support. Do you have the ability to try it on
your Mac but on a non-APFS destination?

It looks like the image is pretty small (~6MB?) so maybe you can try
with a non-APFS formatted thumb drive?

My hunch is that:
- APFS source to FAT32 destination might work correctly.
- FAT32 source to FAT32 destination will definitely work correctly.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

I have done more tests based on your suggestion and I think it is the
opposite. The problem happens is the source is APFS.

Following failed:
APFS -> ExFAT
APFS -> Fat32

Following worked:
ExFAT -> APFS
FAT32 -> APFS

So for now my workaround is to use USB stick formatted with FAT32 or
ExFAT, copy the source images there and then convert to somewhere or my
main drive with APFS.

My regards,
Waldek

Sent from my iPhone

> On Jun 20, 2018, at 11:54, John Snow <1776920@bugs.launchpad.net> wrote:
> 
> I haven't had the time to look just yet.
> 
> If there's a developer out there with a Mac has the bandwidth to take a
> look at this I'd be grateful. (I don't have access to one presently.)
> 
> I see that the target filesystem is APFS however -- I think we might
> have a bug in our APFS support. Do you have the ability to try it on
> your Mac but on a non-APFS destination?
> 
> It looks like the image is pretty small (~6MB?) so maybe you can try
> with a non-APFS formatted thumb drive?
> 
> My hunch is that:
> - APFS source to FAT32 destination might work correctly.
> - FAT32 source to FAT32 destination will definitely work correctly.
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1776920
> 
> Title:
>  qemu-img convert on Mac OSX creates corrupt images
> 
> Status in QEMU:
>  New
> 
> Bug description:
>  An image created by qemu-img create, then modified by another program
>  is converted to bad/corrupt image when using convert sub command on
>  Mac OSX. The same convert works on Linux. The version of qemu-img is
>  2.12.
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

Great, thanks for the additional info!

It sounds like maybe something to do with our sparse detection on APFS
is broken. Maybe lseek calls are failing? This is a bad problem because
it means that any read from an APFS source could break similarly.

I'm still a bit tied down with other work at the moment, but I will try
to address this prior to the 3.0 release if nobody else gets to it
first.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

I am not familiar with QEMU source code but I might have some cycles to
look into it. Where would I look -
https://github.com/qemu/qemu/blob/master/qemu-img.c? Or somewhere else?
Any suggestions would be appreciated.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

My hunch is that we're handling zero regions incorrectly and we might be
skipping data regions we ought to be copying.

...I'm looking at the `image2.qemu` file you've uploaded and it's empty!
we didn't copy *anything* from your source file.

Try converting with '-S 0` to disable sparse protection and see if that
might help.

  '-S' indicates the consecutive number of bytes (defaults to 4k) that must
       contain only zeros for qemu-img to create a sparse image during
       conversion. If the number of bytes is 0, the source will not be scanned for
       unallocated or zero sectors, and the destination image will always be
       fully allocated

If that helps, I'd take a look at img_convert in qemu-img.c ...

convert_iteration_sectors looks relevant, it appears to help us know
which sectors to copy when called by convert_co_do_copy.

Tracing around "convert_co_read" in the same function might also help us
to know which portions of the image we're even deciding to read; and we
could probably track backwards from there to figure out which
condition(s) are disqualifying us if we're deciding not to copy data.

My current hunch is that bdrv_block_status[_above] is returning
something wrong when backed by APFS.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Waldemar Kozaczuk]

Bingo! Adding '-S 0' makes convert work. But it is not perfect as the
end result is fully allocated image.

So with qcow2 like this:

image: mysql-example.qemu
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 50M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

and do this:

qemu-img convert mysql-example.qemu -S 0 -O vmdk mysql-example.vbox

I get 10G vmdk file:
10738794496 Jul  6 18:27 mysql-example.vbox

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

I marked https://bugs.launchpad.net/qemu/+bug/1738840 as a duplicate of
this bug, even though that bug was older, this bug has a slightly more
active thread.

I'm still in need of either an APFS capable machine that I can reproduce
on, or another mac dev willing to help a bit, though.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Yan-Jie Wang]

I have done some experiments and find out that 
the behavior of lseek with whence set to SEEK_DATA is different from the behavior of Linux's lseek.

If the supplied offset is in the middle of a data region, it returns the
start of the next data region.  There may be many data regions in a big
file even though it has no hole.

return value of lseek with whence set to SEEK_DATA:

|--(offset)--Data----|(return value)----Data----|
|--(offset)--Data----|----Hole----|(return value)----Data----|


** Patch added: "macOS-lseek.patch"
   https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5186138/+files/macOS-lseek.patch

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

Re: [Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Eric Blake]

On 09/07/2018 01:04 PM, Yan-Jie Wang wrote:
> I have done some experiments and find out that
> the behavior of lseek with whence set to SEEK_DATA is different from the behavior of Linux's lseek.
> 
> If the supplied offset is in the middle of a data region, it returns the
> start of the next data region.  There may be many data regions in a big
> file even though it has no hole.
> 
> return value of lseek with whence set to SEEK_DATA:
> 
> |--(offset)--Data----|(return value)----Data----|
> |--(offset)--Data----|----Hole----|(return value)----Data----|
> 
> 
> ** Patch added: "macOS-lseek.patch"
>     https://bugs.launchpad.net/qemu/+bug/1776920/+attachment/5186138/+files/macOS-lseek.patch

While a developer can chase a URL, our CI tools can't.  Can you please 
also send that patch directly to qemu-devel@nongnu.org, so that it gets 
the same level of review as other patches?

But I am very reluctant to take your patch. MacOS is flat-out buggy and 
in violation of the specification of lseek(SEEK_DATA), so making all 
applications work around their bug is not going to scale as well as 
having them fix their bug in the first place.

Here's the proposed POSIX specification for what lseek(SEEK_DATA) is 
supposed to do:
http://austingroupbugs.net/view.php?id=415#c862

That text has been present for 7 years now - so anyone implementing 
SEEK_DATA should really be paying attention to interoperability with 
that text.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Sven R]

Hi,

I recently ran into problems and after a long time trying to find out the cause landed here, I got in trouble using a CentOs Cloud image:
https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1905.qcow2.xz
which extracts to a .qcow2 image with sha256 of:
b376afdc0150601f15e53516327d9832216e01a300fecfc302066e36e2ac2b39

image: CentOS-7-x86_64-GenericCloud-1905.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 898M
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16

I use this command on a Mac, OS X 10.13.6 (17G7024), qemu installed via brew:
qemu-img convert -f qcow2 -O vmdk -o adapter_type=lsilogic,subformat=streamOptimized,compat6=on -p CentOS-7-x86_64-GenericCloud-1905.qcow2 -S 0 result.vmdk


941359104 21 Jul 17:11 CentOS-7-x86_64-GenericCloud-1905.qcow2 - original image

Converting this gives these results:
214551040 23 Jul 20:45 conv_mac_v3_1_mit_s_0.vmdk - doesnt work, made on Mac (APFS) with -S 0 
402303488 23 Jul 20:50 conv_mac_v3_1_auf_exfat.vmdk - works and is bootable, made on same Mac, on external drive, exFAT formatted.
149743104 23 Jul 21:21 conv_mac_v4_0.vmdk - doesnt work, made on Mac (APFS) without -S 0 
214551040 23 Jul 21:20 conv_mac_v4_0mit_S0.vmdk - doesnt work, made on Mac (APFS) with -S 0 

converting on non-Mac also works fine:
402303488 23 Jul 18:48 conv_centos7_v1-5-3.vmdk - works and is bootable, made on Centos, qemu-img version 1.5.3

So it seems that '-S 0' didn't fix it for me, or is that only in the
development branch?

Best Regards

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Qemu-devel] [Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[John Snow]

Hi, there isn't really a development branch; if '-S 0' didn't help
alleviate the problem there may be other problems at hand, or the APFS
implementation of SEEK_DATA is causing us even more problems in new
versions.

You could try Yan-Jie Wang's patch that was posted in #20, but until
it's posted to the mailing list with a Signed-off-by tag, we can't
include it ourselves.

However, it would still be nice to know if it fixes your problem.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  New

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Thomas Huth]

The QEMU project is currently considering to move its bug tracking to
another system. For this we need to know which bugs are still valid
and which could be closed already. Thus we are setting older bugs to
"Incomplete" now.

If you still think this bug report here is valid, then please switch
the state back to "New" within the next 60 days, otherwise this report
will be marked as "Expired". Or please mark it as "Fix Released" if
the problem has been solved with a newer version of QEMU already.

Thank you and sorry for the inconvenience.

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  Incomplete

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Thomas Huth]

** Bug watch removed: austingroupbugs.net/ #415
   http://austingroupbugs.net/view.php?id=415

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  Incomplete

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  Expired

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions

[Bug 1776920] Re: qemu-img convert on Mac OSX creates corrupt images

[Juan Niño]

Hey there! I tested @wkozaczuk's suggested minimal steps and THEY WORKED
FOR ME!!

The steps executed on my mac:
1. dd if=boot.bin of=image.img > /dev/null 2>&1
2. dd if=lzloader.elf of=image.img conv=notrunc seek=128 > /dev/null 2>&1
3. qemu-img convert image.img -O qcow2 image.qemu
4. qemu-img convert image.qemu -O qcow2 image2.qemu

The end result:
-rw-r--r--  1 ***  ***  6684672 Jun 22 14:19 image.img
-rw-r--r--  1 ***  ***  7012352 Jun 22 14:20 image.qemu
-rw-r--r--  1 ***  ***  7012352 Jun 22 14:20 image2.qemu
-rw-r--r--  1 ***  ***  6750208 Jun 22 14:22 image2.vbox

The result of regular compare:
qemu-img compare image.qemu image2.qemu
Images are identical.

The result of strict compare:
qemu-img compare -s image.qemu image2.qemu
Images are identical.

Qemu-img on my Mac:
qemu-img --version
qemu-img version 6.0.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

Hardware Overview:

  Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,1
  Processor Name:	8-Core Intel Core i9
  Processor Speed:	2,4 GHz
  Number of Processors:	1
  Total Number of Cores:	8
  L2 Cache (per Core):	256 KB
  L3 Cache:	16 MB
  Hyper-Threading Technology:	Enabled
  Memory:	64 GB
  Activation Lock Status:	Enabled

Storage:

  Mount Point:	/
  File System:	APFS
  Writable:	No
  Ignore Ownership:	No
  BSD Name:	disk1s1
  Volume UUID:	67798918-C522-45C3-918F-3C4155EF4D13
  Physical Drive:
  Device Name:	APPLE SSD AP1024N
  Media Name:	AppleAPFSMedia
  Medium Type:	SSD
  Protocol:	PCI-Express
  Internal:	Yes
  Partition Map Type:	Unknown
  S.M.A.R.T. Status:	Verified

System Software Overview:

  System Version:	macOS 10.15.7 (19H1217)
  Kernel Version:	Darwin 19.6.0
  Boot Volume:	Macintosh HD
  Boot Mode:	Normal
  Secure Virtual Memory:	Enabled
  System Integrity Protection:	Enabled

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1776920

Title:
  qemu-img convert on Mac OSX creates corrupt images

Status in QEMU:
  Expired

Bug description:
  An image created by qemu-img create, then modified by another program
  is converted to bad/corrupt image when using convert sub command on
  Mac OSX. The same convert works on Linux. The version of qemu-img is
  2.12.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1776920/+subscriptions