Question on Compression for Raw Image

Question on Compression for Raw Image

[Wang, Wei W]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

Hi,

Does anyone know the reason why raw-format.c doesn't have compression support (but qcow has the supported added)?
For example, raw image backup with compression, "qemu-img convert -c -O raw origin.img  dist.img", doesn't work.

Thanks,
Wei

[-- Attachment #2: Type: text/html, Size: 2539 bytes --]

Re: Question on Compression for Raw Image

[Kevin Wolf]

Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben:
> Hi,
> 
> Does anyone know the reason why raw-format.c doesn't have compression
> support (but qcow has the supported added)?  For example, raw image
> backup with compression, "qemu-img convert -c -O raw origin.img
> dist.img", doesn't work.

A raw image is by definition a file that contains the exact same
sequence of bytes as the guest sees, without any additional information
or encoding. If you compress a raw file, the guest will see compressed
data on its hard disk instead of the real data.

Anything you could do to add transparent compression to it would mean
that it's not a raw image any more, but a new image format.

Kevin

RE: Question on Compression for Raw Image

[Wang, Wei W]

On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote:
> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben:
> > Hi,
> >
> > Does anyone know the reason why raw-format.c doesn't have
> compression
> > support (but qcow has the supported added)?  For example, raw image
> > backup with compression, "qemu-img convert -c -O raw origin.img
> > dist.img", doesn't work.
> 
> A raw image is by definition a file that contains the exact same sequence of
> bytes as the guest sees, without any additional information or encoding. If
> you compress a raw file, the guest will see compressed data on its hard disk
> instead of the real data.

Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data.

> 
> Anything you could do to add transparent compression to it would mean that
> it's not a raw image any more, but a new image format.
> 
Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option?

Thanks,
Wei

Re: Question on Compression for Raw Image

[Eric Blake]

[-- Attachment #1.1.1: Type: text/plain, Size: 2059 bytes --]

On 10/20/20 9:22 AM, Wang, Wei W wrote:
> On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote:
>> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben:
>>> Hi,
>>>
>>> Does anyone know the reason why raw-format.c doesn't have
>> compression
>>> support (but qcow has the supported added)?  For example, raw image
>>> backup with compression, "qemu-img convert -c -O raw origin.img
>>> dist.img", doesn't work.
>>
>> A raw image is by definition a file that contains the exact same sequence of
>> bytes as the guest sees, without any additional information or encoding. If
>> you compress a raw file, the guest will see compressed data on its hard disk
>> instead of the real data.
> 
> Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data.
> 
>>
>> Anything you could do to add transparent compression to it would mean that
>> it's not a raw image any more, but a new image format.
>>
> Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option?

My recommendation would be implementing a new BDS filter that does
uncompression.  Then, you could do things like:

raw -> decompress -> file.xz

or even

qcow2 -> decompress -> file.qcow2.xz

By the way, the notion of filters is already possible in other ways.
For example, you can point qemu to read from an NBD server, and then use
nbdkit to do the decompression with its filters:

nbdkit --filter=xz file file.xz
raw -> nbd://localhost:10809

Also note that serving a decompressed view of a compressed image tends
to be a read-only proposition (you really CAN'T write to the image
without recompressing, but even if recompression has been blocked for
parallelism, you would end up writing far more of the file after
recompression than the amount of data written by a guest).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[-- Attachment #1.1.2: OpenPGP_0xA7A16B4A2527436A_and_old_rev.asc --]
[-- Type: application/pgp-keys, Size: 103475 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

RE: Question on Compression for Raw Image

[Alberto Garcia]

On Tue 20 Oct 2020 04:22:43 PM CEST, Wang, Wei W wrote:

> Ok, thanks. I'm thinking QEMU could do decompression of the compressed
> data in raw.img when guest reads data.

The qcow2 format already supports compression and it's already
transparent to the guest, so you can use that.

As Kevin said if it's compressed it's no longer raw so you would be
creating a new type of file format.

Berto

Re: Question on Compression for Raw Image

[Richard W.M. Jones]

On Tue, Oct 20, 2020 at 09:32:23AM -0500, Eric Blake wrote:
> On 10/20/20 9:22 AM, Wang, Wei W wrote:
> > On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote:
> >> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben:
> >>> Hi,
> >>>
> >>> Does anyone know the reason why raw-format.c doesn't have
> >> compression
> >>> support (but qcow has the supported added)?  For example, raw image
> >>> backup with compression, "qemu-img convert -c -O raw origin.img
> >>> dist.img", doesn't work.
> >>
> >> A raw image is by definition a file that contains the exact same sequence of
> >> bytes as the guest sees, without any additional information or encoding. If
> >> you compress a raw file, the guest will see compressed data on its hard disk
> >> instead of the real data.
> > 
> > Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data.
> > 
> >>
> >> Anything you could do to add transparent compression to it would mean that
> >> it's not a raw image any more, but a new image format.
> >>
> > Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option?
> 
> My recommendation would be implementing a new BDS filter that does
> uncompression.  Then, you could do things like:
> 
> raw -> decompress -> file.xz
> 
> or even
> 
> qcow2 -> decompress -> file.qcow2.xz
> 
> By the way, the notion of filters is already possible in other ways.
> For example, you can point qemu to read from an NBD server, and then use
> nbdkit to do the decompression with its filters:
> 
> nbdkit --filter=xz file file.xz
> raw -> nbd://localhost:10809

nbdkit can already act as a front-end to qemu-nbd, allowing you to use
the rich nbdkit filter ecosystem with qemu block devices.  There are
some examples in the link below, but --filter=xz could also be used
here:

https://libguestfs.org/nbdkit-nbd-plugin.1.html#Use-qemu-nbd-to-open-a-qcow2-file

> Also note that serving a decompressed view of a compressed image tends
> to be a read-only proposition (you really CAN'T write to the image
> without recompressing, but even if recompression has been blocked for
> parallelism, you would end up writing far more of the file after
> recompression than the amount of data written by a guest).

Yup!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: Question on Compression for Raw Image

[Alberto Garcia]

On Tue 20 Oct 2020 04:32:23 PM CEST, Eric Blake wrote:
> My recommendation would be implementing a new BDS filter that does
> uncompression.  Then, you could do things like:
>
> raw -> decompress -> file.xz

This would work, although read-only and you would need a compression
format that supports random access (I think that's the case with xz, at
least if you compress it using small blocks).

Berto