From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55900) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XH9c5-0008Ip-5c for qemu-devel@nongnu.org; Tue, 12 Aug 2014 06:47:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XH9c2-0007pl-KA for qemu-devel@nongnu.org; Tue, 12 Aug 2014 06:47:33 -0400 Received: from mail-ig0-x233.google.com ([2607:f8b0:4001:c05::233]:47084) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XH9c2-0007pS-CA for qemu-devel@nongnu.org; Tue, 12 Aug 2014 06:47:30 -0400 Received: by mail-ig0-f179.google.com with SMTP id h18so6249065igc.0 for ; Tue, 12 Aug 2014 03:47:29 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20140812005214.GB6226@T430.nay.redhat.com> References: <20140812005214.GB6226@T430.nay.redhat.com> From: =?UTF-8?B?5ZC05YW05Y2a?= Date: Tue, 12 Aug 2014 06:46:49 -0400 Message-ID: Content-Type: multipart/alternative; boundary=20cf303bf576e8021505006c6504 Subject: Re: [Qemu-devel] disk image: self-organized format or raw file List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: qemu-devel@nongnu.org --20cf303bf576e8021505006c6504 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Fam, It's glad to hear you, It is said in this post that "All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files with holes while creating the files..." [ http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-fi= les ] I also heard this claim from other sources, and the only "popular" filesystems who don't support holes in real world are just the old FAT32 and other FAT*. Note that holes appear in filesystems when creating a sparse file in inode-filesystems. While "punching holes" does remove the existent contents from the file, and it was newly added to only xfs/ext4 in newer linux kernel. In qemu's disk image, a hole delivers clear message---the corresponding sectors/blocks/clusters are never written. So it's up to the guest whether to initialize the sectors to zero or just ignore them (filesystems never confuse with a uninitialized sector right?). Filesystems should ignore uninitialized data just because it's meaningless. Once written, the data would be ever meaningful to the guest. "punching holes" would add support for "DISCARD" for a image which could behave like a SSD. Otherwise the image behaves like a magnetic disk. The message in below would not be accurate: * cp has --sparse option to support read and create sparse files. * Sadly scp doesn't support sparse files. * rsync also has a -S --sparse option to properly handle sparse files. Not until recently did I realize that the hole is just widely supported in *almost* all filesystems. That's why I have come up this idea. I understand your concern about the support of hole. If this just because the "hole" is never standardized as POSIX or something else? So now I get one clear reason: hole is not guaranteed by standardized filesystems (I guess a POSIX would be enough). Is their something else? If it's the only reason of not using a sparse raw file as image, and the only impediment is no-one-should-ever-use FAT32 or say the POSIX, we may be very close to move one step forward. Cheers! =E5=90=B4=E5=85=B4=E5=8D=9A Wu, Xingbo On Mon, Aug 11, 2014 at 8:52 PM, Fam Zheng wrote: > On Mon, 08/11 19:38, =E5=90=B4=E5=85=B4=E5=8D=9A wrote: > > Hello, > > > > The introduction in the wiki page present several advantages of qcow2 > > [1]. But I'm a little confused. I really appreciate if any one can give > me > > some help on this :). > > > > (1) Currently the raw format doesn't support COW. In other words, a ra= w > > image cannot have a backing file. COW depends on the mapping table on > which > > we it knows whether each block/cluster is present (has been modified) i= n > > the current image file. Modern file-systems like xfs/ext4/etc. provide > > extent/block allocation information to user-level. Like what 'filefrag' > > does with ioctl 'FIBMAP' and 'FIEMAP'. I guess the raw file driver (may= be > > block/raw-posix.c) may obtain correct 'present information about blocks= . > > However this information may be limited to be aligned with file > allocation > > unit size. Maybe it's just because a raw file has no space to store the > > "backing file name"? I don't think this could hinder the useful feature= . > > > > (2) As most popular filesystems support delay-allocation/on-demand > > allocation/holes, whatever, a raw image is also thin provisioned as oth= er > > formats. It doesn't consume much disk space by storing useless zeros. > > However, I don't know if there is any concern on whether fragmented > extents > > would become a burden of the host filesystem. > > > > (3) For compression and encryption, I'm not an export on these topics = at > > all but I think these features may not be vital to a image format as bo= th > > guest/host's filesystem can also provide similar functionality. > > > > (4) I don't have too much understanding on how snapshot works but I > think > > theoretically it would be using the techniques no more than that used i= n > > COW and backing file. > > > > After all these thoughts, I still found no reason to not using a 'raw' > file > > image (engineering efforts in Qemu should not count as we don't ask fo= r > > more features from outside world). > > I would be very sorry if my ignorance wasted your time. > > Hi! I think what you described is theoretically possible, but I'm not so > positive about this feature. What would be the advantages, compared to > qcow2? > > My major concern is that the file system hole's transparency, meaning tha= t > the > users normally can't tell if a "hole" is really zeroes or unallocated, > would > cause data loss more easily: the user may expect scp (1) or cp (1) to wor= k > on > an image file, just as always, but these tools can legitimately fill the > whole > with actual zeroes, if the target is filesystem does not supporting hole. > That's too dangerous but totally out of control of QEMU. > > Fam > --20cf303bf576e8021505006c6504 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Fam,
=C2=A0 It's glad to hear you,
It is said in this post that=C2=A0"All files systems that sup= port inodes (ext2/3/4, xfs, btfs, etc) support files with holes while creat= ing the files..."

I also heard this c= laim from other sources, and the only "popular" filesystems who d= on't support holes in real world are just the old FAT32 and other FAT*.=
Note that holes appear in filesystems when creating a sparse file in i= node-filesystems. While "punching holes" does remove the existent= contents from the file, and it was =C2=A0newly added to only xfs/ext4 in n= ewer linux kernel.

In qemu's disk image, a hole delivers clear message= ---the corresponding sectors/blocks/clusters are never written. So it's= up to the guest whether to initialize the sectors to zero or just ignore t= hem (filesystems never confuse with a uninitialized sector right?). Filesys= tems should ignore uninitialized data just because it's meaningless. On= ce written, the data would be ever meaningful to the guest.

"punching holes" would add support for "= DISCARD" for a image which could behave like a SSD. Otherwise the imag= e behaves like a magnetic disk.

The message in bel= ow would not be accurate:
* cp has --sparse option to support read and create sparse files.=C2= =A0
* Sadly scp doesn't support sparse files.
* rsy= nc also has a -S --sparse option to properly handle sparse files.

Not until recently did I realize that the hole is just widely su= pported in *almost* all filesystems. That's why I have come up this ide= a.
I understand your concern about the support of hole. If this j= ust because the "hole" is never standardized as POSIX or somethin= g else?

So now I get one clear reason: hole is not guaranteed b= y standardized filesystems (I guess a POSIX would be enough).
Is = their something else? If it's the only reason of not using a sparse raw= file as image, and the only impediment is no-one-should-ever-use FAT32 or = say the POSIX, we may be very close to =C2=A0move one step forward.





Cheers!
=C2=A0 =C2=A0 =C2= =A0 =C2=A0=E5=90=B4=E5=85=B4=E5=8D=9A =C2=A0Wu, Xingbo <wuxb45@gmail.com>


On Mon, Aug 11, 2014 at 8:52 PM, Fam Zhe= ng <famz@redhat.com> wrote:
On Mon, 08/11 19:38, =E5=90=B4=E5= =85=B4=E5=8D=9A wrote:
> Hello,
>
> =C2=A0 The introduction in the wiki page present several advantages of= qcow2
> [1]. But I'm a little confused. I really appreciate if any one can= give me
> some help on this :).
>
> =C2=A0(1) Currently the raw format doesn't support COW. In other w= ords, a raw
> image cannot have a backing file. COW depends on the mapping table on = which
> we it knows whether each block/cluster is present (has been modified) = in
> the current image file. Modern file-systems like xfs/ext4/etc. provide=
> extent/block allocation information to user-level. Like what 'file= frag'
> does with ioctl 'FIBMAP' and 'FIEMAP'. I guess the raw= file driver (maybe
> block/raw-posix.c) may obtain correct 'present information about b= locks.
> However this information may be limited to be aligned with file alloca= tion
> unit size. Maybe it's just because a raw file has no space to stor= e the
> "backing file name"? I don't think this could hinder the= useful feature.
>
> =C2=A0(2) As most popular filesystems support delay-allocation/on-dema= nd
> allocation/holes, whatever, a raw image is also thin provisioned as ot= her
> formats. It doesn't consume much disk space by storing useless zer= os.
> However, I don't know if there is any concern on whether fragmente= d extents
> would become a burden of the host filesystem.
>
> =C2=A0(3) For compression and encryption, I'm not an export on the= se topics at
> all but I think these features may not be vital to a image format as b= oth
> guest/host's filesystem can also provide similar functionality. >
> =C2=A0(4) I don't have too much understanding on how snapshot work= s but I think
> theoretically it would be using the techniques no more than that used = in
> COW and backing file.
>
> After all these thoughts, I still found no reason to not using a '= raw' file
> image (engineering efforts in Qemu should not count as we don't as= k =C2=A0for
> more features from outside world).
> I would be very sorry if my ignorance wasted your time.

Hi! I think what you described is theoretically possible, but I= 'm not so
positive about this feature. What would be the advantages, compared to qcow= 2?

My major concern is that the file system hole's transparency, meaning t= hat the
users normally can't tell if a "hole" is really zeroes or una= llocated, would
cause data loss more easily: the user may expect scp (1) or cp (1) to work = on
an image file, just as always, but these tools can legitimately fill the wh= ole
with actual zeroes, if the target is filesystem does not supporting hole. That's too dangerous but totally out of control of QEMU.

Fam

--20cf303bf576e8021505006c6504--