* Question on Compression for Raw Image @ 2020-10-20 1:31 Wang, Wei W 2020-10-20 8:00 ` Kevin Wolf 0 siblings, 1 reply; 7+ messages in thread From: Wang, Wei W @ 2020-10-20 1:31 UTC (permalink / raw) To: kwolf, mreitz, lersek; +Cc: qemu-devel, qemu-block [-- Attachment #1: Type: text/plain, Size: 252 bytes --] Hi, Does anyone know the reason why raw-format.c doesn't have compression support (but qcow has the supported added)? For example, raw image backup with compression, "qemu-img convert -c -O raw origin.img dist.img", doesn't work. Thanks, Wei [-- Attachment #2: Type: text/html, Size: 2539 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Question on Compression for Raw Image 2020-10-20 1:31 Question on Compression for Raw Image Wang, Wei W @ 2020-10-20 8:00 ` Kevin Wolf 2020-10-20 14:22 ` Wang, Wei W 0 siblings, 1 reply; 7+ messages in thread From: Kevin Wolf @ 2020-10-20 8:00 UTC (permalink / raw) To: Wang, Wei W; +Cc: lersek, qemu-devel, qemu-block, mreitz Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben: > Hi, > > Does anyone know the reason why raw-format.c doesn't have compression > support (but qcow has the supported added)? For example, raw image > backup with compression, "qemu-img convert -c -O raw origin.img > dist.img", doesn't work. A raw image is by definition a file that contains the exact same sequence of bytes as the guest sees, without any additional information or encoding. If you compress a raw file, the guest will see compressed data on its hard disk instead of the real data. Anything you could do to add transparent compression to it would mean that it's not a raw image any more, but a new image format. Kevin ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: Question on Compression for Raw Image 2020-10-20 8:00 ` Kevin Wolf @ 2020-10-20 14:22 ` Wang, Wei W 2020-10-20 14:32 ` Eric Blake 2020-10-20 14:36 ` Alberto Garcia 0 siblings, 2 replies; 7+ messages in thread From: Wang, Wei W @ 2020-10-20 14:22 UTC (permalink / raw) To: Kevin Wolf; +Cc: lersek, qemu-devel, qemu-block, mreitz On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote: > Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben: > > Hi, > > > > Does anyone know the reason why raw-format.c doesn't have > compression > > support (but qcow has the supported added)? For example, raw image > > backup with compression, "qemu-img convert -c -O raw origin.img > > dist.img", doesn't work. > > A raw image is by definition a file that contains the exact same sequence of > bytes as the guest sees, without any additional information or encoding. If > you compress a raw file, the guest will see compressed data on its hard disk > instead of the real data. Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data. > > Anything you could do to add transparent compression to it would mean that > it's not a raw image any more, but a new image format. > Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option? Thanks, Wei ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Question on Compression for Raw Image 2020-10-20 14:22 ` Wang, Wei W @ 2020-10-20 14:32 ` Eric Blake 2020-10-20 14:42 ` Richard W.M. Jones 2020-10-20 14:49 ` Alberto Garcia 2020-10-20 14:36 ` Alberto Garcia 1 sibling, 2 replies; 7+ messages in thread From: Eric Blake @ 2020-10-20 14:32 UTC (permalink / raw) To: Wang, Wei W, Kevin Wolf Cc: Richard W.M. Jones, lersek, qemu-devel, qemu-block, mreitz [-- Attachment #1.1.1: Type: text/plain, Size: 2059 bytes --] On 10/20/20 9:22 AM, Wang, Wei W wrote: > On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote: >> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben: >>> Hi, >>> >>> Does anyone know the reason why raw-format.c doesn't have >> compression >>> support (but qcow has the supported added)? For example, raw image >>> backup with compression, "qemu-img convert -c -O raw origin.img >>> dist.img", doesn't work. >> >> A raw image is by definition a file that contains the exact same sequence of >> bytes as the guest sees, without any additional information or encoding. If >> you compress a raw file, the guest will see compressed data on its hard disk >> instead of the real data. > > Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data. > >> >> Anything you could do to add transparent compression to it would mean that >> it's not a raw image any more, but a new image format. >> > Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option? My recommendation would be implementing a new BDS filter that does uncompression. Then, you could do things like: raw -> decompress -> file.xz or even qcow2 -> decompress -> file.qcow2.xz By the way, the notion of filters is already possible in other ways. For example, you can point qemu to read from an NBD server, and then use nbdkit to do the decompression with its filters: nbdkit --filter=xz file file.xz raw -> nbd://localhost:10809 Also note that serving a decompressed view of a compressed image tends to be a read-only proposition (you really CAN'T write to the image without recompressing, but even if recompression has been blocked for parallelism, you would end up writing far more of the file after recompression than the amount of data written by a guest). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org [-- Attachment #1.1.2: OpenPGP_0xA7A16B4A2527436A_and_old_rev.asc --] [-- Type: application/pgp-keys, Size: 103475 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Question on Compression for Raw Image 2020-10-20 14:32 ` Eric Blake @ 2020-10-20 14:42 ` Richard W.M. Jones 2020-10-20 14:49 ` Alberto Garcia 1 sibling, 0 replies; 7+ messages in thread From: Richard W.M. Jones @ 2020-10-20 14:42 UTC (permalink / raw) To: Eric Blake Cc: Kevin Wolf, qemu-block, qemu-devel, mreitz, Wang, Wei W, lersek On Tue, Oct 20, 2020 at 09:32:23AM -0500, Eric Blake wrote: > On 10/20/20 9:22 AM, Wang, Wei W wrote: > > On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote: > >> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben: > >>> Hi, > >>> > >>> Does anyone know the reason why raw-format.c doesn't have > >> compression > >>> support (but qcow has the supported added)? For example, raw image > >>> backup with compression, "qemu-img convert -c -O raw origin.img > >>> dist.img", doesn't work. > >> > >> A raw image is by definition a file that contains the exact same sequence of > >> bytes as the guest sees, without any additional information or encoding. If > >> you compress a raw file, the guest will see compressed data on its hard disk > >> instead of the real data. > > > > Ok, thanks. I'm thinking QEMU could do decompression of the compressed data in raw.img when guest reads data. > > > >> > >> Anything you could do to add transparent compression to it would mean that > >> it's not a raw image any more, but a new image format. > >> > > Yes, decompression makes it transparent to the guest. Would you think it's good to reuse the raw image implementation, just add the compress/decompress option? > > My recommendation would be implementing a new BDS filter that does > uncompression. Then, you could do things like: > > raw -> decompress -> file.xz > > or even > > qcow2 -> decompress -> file.qcow2.xz > > By the way, the notion of filters is already possible in other ways. > For example, you can point qemu to read from an NBD server, and then use > nbdkit to do the decompression with its filters: > > nbdkit --filter=xz file file.xz > raw -> nbd://localhost:10809 nbdkit can already act as a front-end to qemu-nbd, allowing you to use the rich nbdkit filter ecosystem with qemu block devices. There are some examples in the link below, but --filter=xz could also be used here: https://libguestfs.org/nbdkit-nbd-plugin.1.html#Use-qemu-nbd-to-open-a-qcow2-file > Also note that serving a decompressed view of a compressed image tends > to be a read-only proposition (you really CAN'T write to the image > without recompressing, but even if recompression has been blocked for > parallelism, you would end up writing far more of the file after > recompression than the amount of data written by a guest). Yup! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Question on Compression for Raw Image 2020-10-20 14:32 ` Eric Blake 2020-10-20 14:42 ` Richard W.M. Jones @ 2020-10-20 14:49 ` Alberto Garcia 1 sibling, 0 replies; 7+ messages in thread From: Alberto Garcia @ 2020-10-20 14:49 UTC (permalink / raw) To: Eric Blake, Wang, Wei W, Kevin Wolf Cc: lersek, qemu-devel, qemu-block, mreitz On Tue 20 Oct 2020 04:32:23 PM CEST, Eric Blake wrote: > My recommendation would be implementing a new BDS filter that does > uncompression. Then, you could do things like: > > raw -> decompress -> file.xz This would work, although read-only and you would need a compression format that supports random access (I think that's the case with xz, at least if you compress it using small blocks). Berto ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: Question on Compression for Raw Image 2020-10-20 14:22 ` Wang, Wei W 2020-10-20 14:32 ` Eric Blake @ 2020-10-20 14:36 ` Alberto Garcia 1 sibling, 0 replies; 7+ messages in thread From: Alberto Garcia @ 2020-10-20 14:36 UTC (permalink / raw) To: Wang, Wei W, Kevin Wolf; +Cc: lersek, qemu-devel, qemu-block, mreitz On Tue 20 Oct 2020 04:22:43 PM CEST, Wang, Wei W wrote: > Ok, thanks. I'm thinking QEMU could do decompression of the compressed > data in raw.img when guest reads data. The qcow2 format already supports compression and it's already transparent to the guest, so you can use that. As Kevin said if it's compressed it's no longer raw so you would be creating a new type of file format. Berto ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-10-20 14:53 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-10-20 1:31 Question on Compression for Raw Image Wang, Wei W 2020-10-20 8:00 ` Kevin Wolf 2020-10-20 14:22 ` Wang, Wei W 2020-10-20 14:32 ` Eric Blake 2020-10-20 14:42 ` Richard W.M. Jones 2020-10-20 14:49 ` Alberto Garcia 2020-10-20 14:36 ` Alberto Garcia
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.