[Qemu-devel] [Bug 1662050] [NEW] qemu-img convert a delta qcow2 image into a complete image

[Qemu-devel] [Bug 1662050] [NEW] qemu-img convert a delta qcow2 image into a complete image

[wayen]

Public bug reported:

I have a base image file "base.qcow2" and a delta qcow2 image file
"delta.qcow2" whose backing file is "base.qcow2".

Now I use qemu-img to convert "delta.qcow2" and will get a new image
file "new.qcow2" which is complete and equivalent to combination of
"base.qcow2" and "delta.qcow2".

In fact, i just want to convert the delta qcow2 image file and get a new
delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
have to admit that this is not bug. Could you please take this as a new
feature and enable qemu-img to convert images in-place?

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: feature

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a delta qcow2 image into a complete image

Status in QEMU:
  New

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is complete and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact, i just want to convert the delta qcow2 image file and get a
  new delta overlay qcow2 image file. So the "new.qcow2" is not what i
  want. I have to admit that this is not bug. Could you please take this
  as a new feature and enable qemu-img to convert images in-place?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a delta qcow2 image into a complete image

[wayen]

** Summary changed:

- qemu-img convert a delta qcow2 image into a complete image
+ qemu-img convert a overlay qcow2 image into a complete image

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a complete image

Status in QEMU:
  New

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is complete and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact, i just want to convert the delta qcow2 image file and get a
  new delta overlay qcow2 image file. So the "new.qcow2" is not what i
  want. I have to admit that this is not bug. Could you please take this
  as a new feature and enable qemu-img to convert images in-place?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Summary changed:

- qemu-img convert a overlay qcow2 image into a complete image
+ qemu-img convert a overlay qcow2 image into a entire image

** Description changed:

  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".
  
  Now I use qemu-img to convert "delta.qcow2" and will get a new image
- file "new.qcow2" which is complete and equivalent to combination of
+ file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".
  
  In fact, i just want to convert the delta qcow2 image file and get a new
  delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a new
  feature and enable qemu-img to convert images in-place?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  New

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact, i just want to convert the delta qcow2 image file and get a
  new delta overlay qcow2 image file. So the "new.qcow2" is not what i
  want. I have to admit that this is not bug. Could you please take this
  as a new feature and enable qemu-img to convert images in-place?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

On 02/05/2017 09:19 PM, wayen wrote:
> 
>   I have a base image file "base.qcow2" and a delta qcow2 image file
>   "delta.qcow2" whose backing file is "base.qcow2".
>   
>   Now I use qemu-img to convert "delta.qcow2" and will get a new image
> + file "new.qcow2" which is entire and equivalent to combination of
>   "base.qcow2" and "delta.qcow2".
>   
>   In fact, i just want to convert the delta qcow2 image file and get a new
>   delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
>   have to admit that this is not bug. Could you please take this as a new
>   feature and enable qemu-img to convert images in-place?

This feature already exists.  Use:

qemu-img rebase -F $backing_fmt -b '' -f qcow2 delta.qcow2

and now delta.qcow2 will be a complete image.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

@Eric Blake, Thanks very much for your help. In your way, I have
verified that this feature already exists.

** Changed in: qemu
       Status: New => Invalid

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact, i just want to convert the delta qcow2 image file and get a
  new delta overlay qcow2 image file. So the "new.qcow2" is not what i
  want. I have to admit that this is not bug. Could you please take this
  as a new feature and enable qemu-img to convert images in-place?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

@Eric Blake. Sorry, I didn't make it clear. In fact, I don't want to get
a complete image. I just want to convert qcow2 overlay and get a new
qcow2 overlay. Maybe you think my intention is meaningless， but this is
what I want.

** Changed in: qemu
       Status: Invalid => Incomplete

** Description changed:

  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".
  
  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".
  
- In fact, i just want to convert the delta qcow2 image file and get a new
+ In fact, i just want to convert the delta qcow2 image file and get a New
  delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a new
- feature and enable qemu-img to convert images in-place?
+ feature and enable qemu-img to convert qcow2 overlay itself?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact, i just want to convert the delta qcow2 image file and get a
  New delta overlay qcow2 image file. So the "new.qcow2" is not what i
  want. I have to admit that this is not bug. Could you please take this
  as a new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Description changed:

  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".
  
  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".
  
- In fact, i just want to convert the delta qcow2 image file and get a New
- delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
- have to admit that this is not bug. Could you please take this as a new
- feature and enable qemu-img to convert qcow2 overlay itself?
+ In fact,I don't want to get a complete image.I just want to convert
+ delta qcow2 image file "A" to a New delta overlay qcow2 image "B" which
+ is equivalent to "A". So the "new.qcow2" is not what i want. I have to
+ admit that this is not bug. Could you please take this as a new feature
+ and enable qemu-img to convert qcow2 overlay itself?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Thomas Huth]

Can't you simply do a "cp delta.qcow2 new.qcow2" ?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

@Thomas Huth, when we use qemu-img to convert a image "A" and will get a
new image "B" which is equivalent to "A" . But "B" may be a lot
different from "A",such as file size. The file size of "B" is usually
less than "A" and this is very valuable to me. So I can't simply do a
"cp delta.qcow2 new.qcow2" ?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Thomas Huth]

I meant to copy B, not A ... but I likely just haven't really understood
what you're really trying to do here, so never mind.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

@Thomas Huth, I just want to reduce the file size of qcow2 overlay image
by this conversion.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

On 02/08/2017 02:16 AM, wayen wrote:
> @Thomas Huth, I just want to reduce the file size of qcow2 overlay image
> by this conversion.

Are you merely trying to sparsify the file (punch holes on the portion
of the file that is not mapped to disk), so that the apparent file size
is unchanged, but the actual disk usage is minimized? That's probably
already possible (virt-sparsify from libguestfs seems to be able to do
it; look at what steps it uses).

Or are you asking for a way to defragment the file, so that the apparent
size shrinks because all occupied clusters are now contiguous, so that
there are no longer any need for holes?  The end result is the same
amount of disk usage, but right now, the only way to defragment is to
convert from one image to a copy; we don't support in-place
defragmentation yet.  So far, no one has come up with a compelling
reason why it is needed, or a patch to implement it, but there's no
technical reason why it can't be done.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

@Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image.
As you said, the actual disk usage is minimized and the apparent file
size is unchanged.It is very valuable for me, because it means that my
disk can hold more files.

But we need to be careful when transfer the sparse qcow2 overlays.
Because some tools do not support sparse file and will convert it into a
common file,for example, scp. And I doubt what will happen when transfer
sparse files between different file systems.

So I want to use qemu-img to convert the sparse qcow2 overlay into
common file. If this was feasible, the holes in the sparse overlay will
be removed and the above problems will disappear.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

On Thu, Feb 09, 2017 at 02:05:54AM -0000, wayen wrote:
> @Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image.
> As you said, the actual disk usage is minimized and the apparent file
> size is unchanged.It is very valuable for me, because it means that my
> disk can hold more files.
> 
> But we need to be careful when transfer the sparse qcow2 overlays.
> Because some tools do not support sparse file and will convert it into a
> common file,for example, scp. And I doubt what will happen when transfer
> sparse files between different file systems.
> 
> So I want to use qemu-img to convert the sparse qcow2 overlay into
> common file. If this was feasible, the holes in the sparse overlay will
> be removed and the above problems will disappear.

You can use the drive_mirror HMP command or drive-mirror QMP command on
a running QEMU instance to copy the data to a new file and switch to the
new file.

I'm curious what exactly is being optimized by copying out a fresh qcow2
image.

Please post the output of:

  $ stat delta.qcow2
  $ qemu-img map delta.qcow2
  $ stat new.qcow2
  $ qemu-img map new.qcow2

This will allow us to see the size and data layout differences.

Maybe a new command should be added to QEMU to do the optimization
in-place.  This would avoid the disk space overhead associated with
drive-mirror, cp, qemu-img convert, etc.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Attachment added: "stat delta.qcow2 output"
   https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818561/+files/stat_delta_qcow2.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Attachment added: "qemu-img map delta.qcow2 output"
   https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818562/+files/qemu_img_map_delta_qcow2.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Attachment added: "stat new.qcow2 output"
   https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818563/+files/stat_new_qcow2.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

** Attachment added: "qemu-img map new.qcow2 output"
   https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818564/+files/qemu_img_map_new_qcow2.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4862 bytes --]

On Tue, Feb 14, 2017 at 02:17:16AM -0000, wayen wrote:
> ** Attachment added: "qemu-img map new.qcow2 output"
>    https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818564/+files/qemu_img_map_new_qcow2.txt

Thanks for posting the attachments.

I ran a script to find unallocated clusters in the delta.qcow2 host
file.  Most are actually qcow2 metadata (L1/L2 tables, refcount blocks).

This output shows that any image file size reduction you are hoping to
achieve can only come from zero clusters.

There are no holes in the files that would result in significant image
file size reduction if a new image were written out.

Just wanted to share this info in case anyone else is thinking about how
to optimize qcow2 files.  I still think rewriting images sequentially
can be useful - if internal snapshots were used and deleted then COW can
result in holes.

Hole at 0 size 5.0 clusters
Hole at 393216 size 1.0 clusters
Hole at 589824 size 1.0 clusters
Hole at 1114112 size 1.0 clusters
Hole at 1310720 size 1.0 clusters
Hole at 1507328 size 1.0 clusters
Hole at 1703936 size 1.0 clusters
Hole at 2293760 size 1.0 clusters
Hole at 2621440 size 1.0 clusters
Hole at 3080192 size 1.0 clusters
Hole at 5111808 size 1.0 clusters
Hole at 6291456 size 1.0 clusters
Hole at 30408704 size 1.0 clusters
Hole at 47906816 size 1.0 clusters
Hole at 142671872 size 1.0 clusters
Hole at 219545600 size 1.0 clusters
Hole at 667090944 size 1.0 clusters
Hole at 853868544 size 1.0 clusters
Hole at 1562640384 size 1.0 clusters
Hole at 2147483648 size 1.0 clusters
Hole at 2617180160 size 1.0 clusters
Hole at 3411148800 size 1.0 clusters
Hole at 4107075584 size 1.0 clusters
Hole at 4294967296 size 1.0 clusters
Hole at 4452646912 size 1.0 clusters
Hole at 4792057856 size 1.0 clusters
Hole at 5494865920 size 1.0 clusters
Hole at 5645271040 size 1.0 clusters
Hole at 5702483968 size 1.0 clusters
Hole at 6187188224 size 1.0 clusters
Hole at 6442450944 size 1.0 clusters
Hole at 6862995456 size 1.0 clusters
Hole at 6987317248 size 1.0 clusters
Hole at 7567245312 size 1.0 clusters
Hole at 8135245824 size 1.0 clusters
Hole at 8590589952 size 1.0 clusters
Hole at 8613462016 size 1.0 clusters
Hole at 9055436800 size 1.0 clusters
Hole at 9703522304 size 1.0 clusters
Hole at 10279321600 size 1.0 clusters
Hole at 10737418240 size 3.0 clusters
Hole at 10844372992 size 1.0 clusters
Hole at 11167858688 size 1.0 clusters
Hole at 11209605120 size 1.0 clusters
Hole at 11209801728 size 1.0 clusters
Hole at 11730944000 size 1.0 clusters
Hole at 12183207936 size 1.0 clusters
Hole at 12705464320 size 1.0 clusters
Hole at 12884901888 size 1.0 clusters
Hole at 13444120576 size 1.0 clusters
Hole at 13910016000 size 1.0 clusters
Hole at 14182711296 size 1.0 clusters
Hole at 15025635328 size 1.0 clusters
Hole at 15032385536 size 1.0 clusters

The following script draws the allocated clusters and holes in the image
file.  I took your qemu-img map output, filtered out any lines with
base.qcow2, and sorted using sort -k3 -g to sort on the "Mapped to"
field.  Then I ran ./qcow2-map-svg.py <filtered.txt >output.svg.

#!/usr/bin/python3
import sys
import io

COLOR_ALLOCATED = '#ffaaaa'
COLOR_HOLE = '#999999'

def svg_percentage(value, total):
    return '{0}%'.format(100.0 * value / total)

def svg_rect(x, width, color):
    print('<rect x="{0}" y="0" width="{1}" height="40" fill="{2}" stroke="none" />'.format(x, width, color), file=out)

def svg_text(x, y, text):
    print('<text x="{0}" y="{1}">{2}</text>'.format(x, y, text), file=out)

out = io.StringIO()

print('''<?xml version="1.0" encoding="UTF-8" ?>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1">''', file=out)

file_map = []
header = True
for line in sys.stdin:
    if header:
        header = False
        continue

    offset, length, mapped, filename = line.split()
    offset = int(offset, 16)
    length = int(length, 16)
    mapped = int(mapped, 16)

    file_map.append((offset, length, mapped, filename))

file_size = file_map[-1][2] + file_map[-1][1]
last_mapped = 0
for _, length, mapped, _ in file_map:
    if mapped > last_mapped:
#        if mapped - last_mapped:
#            print('Hole at {0} size {1} clusters'.format(last_mapped, (mapped - last_mapped) / 65536))
        svg_rect(svg_percentage(last_mapped, file_size),
                 svg_percentage(mapped - last_mapped, file_size),
                 COLOR_HOLE)

    svg_rect(svg_percentage(mapped, file_size),
             svg_percentage(length, file_size),
             COLOR_ALLOCATED)
    last_mapped = mapped + length
if last_mapped < file_size:
    svg_rect(svg_percentage(last_mapped, file_size),
             svg_percentage(file_size - last_mapped, file_size),
             COLOR_HOLE)

for i in range(10):
    svg_text(svg_percentage(i, 10), 60, svg_percentage(i, 10))

print('</svg>', file=out)

print(out.getvalue())

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

Is there any way to remove holes from qcow2 overlay images? It's very
important to me. I am looking forward to your reply.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[858585 jemmy]

Hi wayen:
    Which version are you used?
    I also find this problem on old version qemu, and i write a patch
for it. It works.
    I'm not sure the mainline version have solve this problem.
    what command are you used?

On Mon, Apr 10, 2017 at 10:14 AM, wayen <1662050@bugs.launchpad.net> wrote:
> Is there any way to remove holes from qcow2 overlay images? It's very
> important to me. I am looking forward to your reply.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1662050
>
> Title:
>   qemu-img convert a overlay qcow2 image into a entire image
>
> Status in QEMU:
>   Incomplete
>
> Bug description:
>   I have a base image file "base.qcow2" and a delta qcow2 image file
>   "delta.qcow2" whose backing file is "base.qcow2".
>
>   Now I use qemu-img to convert "delta.qcow2" and will get a new image
>   file "new.qcow2" which is entire and equivalent to combination of
>   "base.qcow2" and "delta.qcow2".
>
>   In fact,I don't want to get a complete image.I just want to convert
>   delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
>   which is equivalent to "A". So the "new.qcow2" is not what i want. I
>   have to admit that this is not bug. Could you please take this as a
>   new feature and enable qemu-img to convert qcow2 overlay itself?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions
>

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

Is there any way to remove holes from sparse qcow2 overlay? It's very
important to me. I am looking forward to your reply.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

Hi Lidong Chen:
    I used QEMU 2.0.0 on Ubuntu 14.04.
    Do you mean your patch can make qemu-img convert qcow2 overlay into a new overlay?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[858585 jemmy]

On Mon, Apr 10, 2017 at 1:07 PM, wayen <1662050@bugs.launchpad.net> wrote:
> Hi Lidong Chen:
>     I used QEMU 2.0.0 on Ubuntu 14.04.
>     Do you mean your patch can make qemu-img convert qcow2 overlay into a new overlay?

yes. but i find it already fixed in 2.0.0.
do you add the -o backing_file= option in the command?

>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1662050
>
> Title:
>   qemu-img convert a overlay qcow2 image into a entire image
>
> Status in QEMU:
>   Incomplete
>
> Bug description:
>   I have a base image file "base.qcow2" and a delta qcow2 image file
>   "delta.qcow2" whose backing file is "base.qcow2".
>
>   Now I use qemu-img to convert "delta.qcow2" and will get a new image
>   file "new.qcow2" which is entire and equivalent to combination of
>   "base.qcow2" and "delta.qcow2".
>
>   In fact,I don't want to get a complete image.I just want to convert
>   delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
>   which is equivalent to "A". So the "new.qcow2" is not what i want. I
>   have to admit that this is not bug. Could you please take this as a
>   new feature and enable qemu-img to convert qcow2 overlay itself?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions
>

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[wayen]

Hi Lidong Chen:

    I forgot to use this option.I think the way you said is effective.I
will try it.Thank you very much for your help

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Incomplete

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions

[Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1662050

Title:
  qemu-img convert a overlay qcow2 image into a entire image

Status in QEMU:
  Expired

Bug description:
  I have a base image file "base.qcow2" and a delta qcow2 image file
  "delta.qcow2" whose backing file is "base.qcow2".

  Now I use qemu-img to convert "delta.qcow2" and will get a new image
  file "new.qcow2" which is entire and equivalent to combination of
  "base.qcow2" and "delta.qcow2".

  In fact,I don't want to get a complete image.I just want to convert
  delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
  which is equivalent to "A". So the "new.qcow2" is not what i want. I
  have to admit that this is not bug. Could you please take this as a
  new feature and enable qemu-img to convert qcow2 overlay itself?

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions