[Qemu-devel] [Bug 498523] Re: Add on-line write compression support to qcow2

[Qemu-devel] [Bug 498523] Re: Add on-line write compression support to qcow2

[Paul Maunders]

+1 vote for this feature.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/498523

Title:
  Add on-line write compression support to qcow2

Status in QEMU:
  Confirmed

Bug description:
  This is a wishlist item.  Launchpad really need a way for the
  submitter to indicate this.

  It would be really cool if qemu were to support disk compression on-
  line for writes.

  I know this wouldn't be really easy.  Although most OS's use blocks,
  you can really only count on being able to compress 512-byte sectors,
  which doesn't give much room for a good compression ratio.  Moreover,
  the index indicating where in the image file each sector is located
  would be complex to manage, since the compressed blocks would be
  variable sized, and you'd be wanting to do some kind of best-fit
  allocation of space in the image file.  (If you were to make the image
  file compressed block size granularity, say, 64 bytes, you could
  probably do this best fit O(1).)  If you were to buffer enough writes,
  you could group arbitrary sequences of written sectors into blocks to
  compress (which with writeback could be sent to a helper thread on
  another CPU, so the throughput would be good).

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/498523/+subscriptions

[Bug 498523] Re: Add on-line write compression support to qcow2

[Thomas Huth]

As far as I know, QEMU v5.1 now has support for compression filters,
e.g. by creating a qcow2 image with:

 qemu-img create -f qcow2 -o compression_type=zlib image.qcow2 1G

... so I think we can finally mark this ticket here as done.

** Changed in: qemu
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/498523

Title:
  Add on-line write compression support to qcow2

Status in QEMU:
  Fix Released

Bug description:
  This is a wishlist item.  Launchpad really need a way for the
  submitter to indicate this.

  It would be really cool if qemu were to support disk compression on-
  line for writes.

  I know this wouldn't be really easy.  Although most OS's use blocks,
  you can really only count on being able to compress 512-byte sectors,
  which doesn't give much room for a good compression ratio.  Moreover,
  the index indicating where in the image file each sector is located
  would be complex to manage, since the compressed blocks would be
  variable sized, and you'd be wanting to do some kind of best-fit
  allocation of space in the image file.  (If you were to make the image
  file compressed block size granularity, say, 64 bytes, you could
  probably do this best fit O(1).)  If you were to buffer enough writes,
  you could group arbitrary sequences of written sectors into blocks to
  compress (which with writeback could be sent to a helper thread on
  another CPU, so the throughput would be good).

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/498523/+subscriptions

Re: [Bug 498523] Re: Add on-line write compression support to qcow2

[Eric Blake]

On 11/19/20 3:39 AM, Thomas Huth wrote:
> As far as I know, QEMU v5.1 now has support for compression filters,
> e.g. by creating a qcow2 image with:
> 
>  qemu-img create -f qcow2 -o compression_type=zlib image.qcow2 1G
> 
> ... so I think we can finally mark this ticket here as done.

That says what compression type to use when writing the entire disk in
one pass, but not online write compression. I think we may be a bit
premature in calling this 'fix released', although I'm not certain we
will ever try to add the feature requested.

> 
> ** Changed in: qemu
>        Status: Confirmed => Fix Released
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Bug 498523] Re: Add on-line write compression support to qcow2

[Eric Blake]

On 11/19/20 3:39 AM, Thomas Huth wrote:
> As far as I know, QEMU v5.1 now has support for compression filters,
> e.g. by creating a qcow2 image with:
> 
>  qemu-img create -f qcow2 -o compression_type=zlib image.qcow2 1G
> 
> ... so I think we can finally mark this ticket here as done.

That says what compression type to use when writing the entire disk in
one pass, but not online write compression. I think we may be a bit
premature in calling this 'fix released', although I'm not certain we
will ever try to add the feature requested.

> 
> ** Changed in: qemu
>        Status: Confirmed => Fix Released
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/498523

Title:
  Add on-line write compression support to qcow2

Status in QEMU:
  Fix Released

Bug description:
  This is a wishlist item.  Launchpad really need a way for the
  submitter to indicate this.

  It would be really cool if qemu were to support disk compression on-
  line for writes.

  I know this wouldn't be really easy.  Although most OS's use blocks,
  you can really only count on being able to compress 512-byte sectors,
  which doesn't give much room for a good compression ratio.  Moreover,
  the index indicating where in the image file each sector is located
  would be complex to manage, since the compressed blocks would be
  variable sized, and you'd be wanting to do some kind of best-fit
  allocation of space in the image file.  (If you were to make the image
  file compressed block size granularity, say, 64 bytes, you could
  probably do this best fit O(1).)  If you were to buffer enough writes,
  you could group arbitrary sequences of written sectors into blocks to
  compress (which with writeback could be sent to a helper thread on
  another CPU, so the throughput would be good).

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/498523/+subscriptions

[Bug 498523] Re: Add on-line write compression support to qcow2

[Thomas Huth]

Ok, sorry, seems like I mis-understood that new compression_type
feature. If the requested feature will likely never be implemented,
should we move this to WontFix instead?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/498523

Title:
  Add on-line write compression support to qcow2

Status in QEMU:
  Fix Released

Bug description:
  This is a wishlist item.  Launchpad really need a way for the
  submitter to indicate this.

  It would be really cool if qemu were to support disk compression on-
  line for writes.

  I know this wouldn't be really easy.  Although most OS's use blocks,
  you can really only count on being able to compress 512-byte sectors,
  which doesn't give much room for a good compression ratio.  Moreover,
  the index indicating where in the image file each sector is located
  would be complex to manage, since the compressed blocks would be
  variable sized, and you'd be wanting to do some kind of best-fit
  allocation of space in the image file.  (If you were to make the image
  file compressed block size granularity, say, 64 bytes, you could
  probably do this best fit O(1).)  If you were to buffer enough writes,
  you could group arbitrary sequences of written sectors into blocks to
  compress (which with writeback could be sent to a helper thread on
  another CPU, so the throughput would be good).

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/498523/+subscriptions

[Bug 498523] Re: Add on-line write compression support to qcow2

[Max Reitz]

The compression filter can be used e.g. with -drive
driver=compress,file.driver=qcow2,file.file.filename=foo.qcow2.
However, it shouldn’t be used lightly, as it will only do the right
thing in very specific circumstances, namely every cluster that’s
written to must not be allocated already.  So writing to the same
cluster twice will not work.  (Which is why I was hesitant to merge this
filter, but in the end I was contend with the fact that it’s at least
difficult enough to use that unsuspecting users hopefully won’t
accidentally enable it.)

(It should be noted that this is not a limitation of the compression
filter, though, but of qcow2’s implementation (VMDK isn’t any better).
So technically qemu has the feature now, but qcow2 is still missing it.)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/498523

Title:
  Add on-line write compression support to qcow2

Status in QEMU:
  Fix Released

Bug description:
  This is a wishlist item.  Launchpad really need a way for the
  submitter to indicate this.

  It would be really cool if qemu were to support disk compression on-
  line for writes.

  I know this wouldn't be really easy.  Although most OS's use blocks,
  you can really only count on being able to compress 512-byte sectors,
  which doesn't give much room for a good compression ratio.  Moreover,
  the index indicating where in the image file each sector is located
  would be complex to manage, since the compressed blocks would be
  variable sized, and you'd be wanting to do some kind of best-fit
  allocation of space in the image file.  (If you were to make the image
  file compressed block size granularity, say, 64 bytes, you could
  probably do this best fit O(1).)  If you were to buffer enough writes,
  you could group arbitrary sequences of written sectors into blocks to
  compress (which with writeback could be sent to a helper thread on
  another CPU, so the throughput would be good).

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/498523/+subscriptions