[Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Stephane Chazelas]

Hello,

since qemu-2.7.0, doing synchronised I/O in a VM (tested with
Ubuntu 16.04 amd64 VM)  while the disk is backed by a qcow2
file sitting on a ZFS filesystem (zfs on Linux on Debian jessie
(PVE)), the performances are dreadful:

# time dd if=/dev/zero count=1000  of=b oflag=dsync
1000+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 21.9908 s, 23.3 kB/s
dd if=/dev/zero count=1000 of=b oflag=dsync  0.00s user 0.04s system 0% cpu 21.992 total

(22 seconds to write that half megabyte). Same with O_SYNC or
O_DIRECT, or doing fsync() or sync_file_range() after each
write().

I first noticed it for dpkg unpacking kernel headers where dpkg
does a sync_file_range() after each file is extracted.

Note that it doesn't happen when writing anything else than
zeroes (like tr '\0' x < /dev/zero | dd count=1000  of=b
oflag=dsync). In the case of the kernel headers, I suppose the
zeroes come from the non-filled parts of the ext4 blocks.

Doing strace -fc on the qemu process, 98% of the time is spent
in the lseek() system call.

That's the lseek(SEEK_DATA) followed by lseek(SEEK_HOLE) done by
find_allocation() called to find out whether sectors are within
a hole in a sparse file.

#0  lseek64 () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000561287cf4ca8 in find_allocation (bs=0x7fd898d70000, hole=<synthetic pointer>, data=<synthetic pointer>, start=<optimized out>)
    at block/raw-posix.c:1702
#2  raw_co_get_block_status (bs=0x7fd898d70000, sector_num=<optimized out>, nb_sectors=40, pnum=0x7fd80dd05aac, file=0x7fd80dd05ab0) at block/raw-posix.c:1765
#3  0x0000561287cfae92 in bdrv_co_get_block_status (bs=0x7fd898d70000, sector_num=sector_num@entry=1303680, nb_sectors=40, pnum=pnum@entry=0x7fd80dd05aac,
    file=file@entry=0x7fd80dd05ab0) at block/io.c:1709
#4  0x0000561287cfafaa in bdrv_co_get_block_status (bs=bs@entry=0x7fd898d66000, sector_num=sector_num@entry=33974144, nb_sectors=<optimized out>,
    nb_sectors@entry=40, pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1742
#5  0x0000561287cfb0bb in bdrv_co_get_block_status_above (file=0x7fd80dd05bc0, pnum=0x7fd80dd05bbc, nb_sectors=40, sector_num=33974144, base=0x0,
    bs=<optimized out>) at block/io.c:1776
#6  bdrv_get_block_status_above_co_entry (opaque=opaque@entry=0x7fd80dd05b40) at block/io.c:1792
#7  0x0000561287cfae08 in bdrv_get_block_status_above (bs=0x7fd898d66000, base=base@entry=0x0, sector_num=<optimized out>, nb_sectors=nb_sectors@entry=40,
    pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1824
#8  0x0000561287cd372d in is_zero_sectors (bs=<optimized out>, start=<optimized out>, count=40) at block/qcow2.c:2428
#9  0x0000561287cd38ed in is_zero_sectors (count=<optimized out>, start=<optimized out>, bs=<optimized out>) at block/qcow2.c:2471
#10 qcow2_co_pwrite_zeroes (bs=0x7fd898d66000, offset=33974144, count=24576, flags=2724114573) at block/qcow2.c:2452
#11 0x0000561287cfcb7f in bdrv_co_do_pwrite_zeroes (bs=bs@entry=0x7fd898d66000, offset=offset@entry=17394782208, count=count@entry=4096,
    flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block/io.c:1218
#12 0x0000561287cfd0cb in bdrv_aligned_pwritev (bs=0x7fd898d66000, req=<optimized out>, offset=17394782208, bytes=4096, align=1, qiov=0x0,
    flags=<optimized out>) at block/io.c:1320
#13 0x0000561287cfe450 in bdrv_co_do_zero_pwritev (req=<optimized out>, flags=<optimized out>, bytes=<optimized out>, offset=<optimized out>,
    bs=<optimized out>) at block/io.c:1422
#14 bdrv_co_pwritev (child=0x15, offset=17394782208, bytes=4096, qiov=0x7fd8a25eb08d <lseek64+45>, qiov@entry=0x0, flags=231758512) at block/io.c:1492
#15 0x0000561287cefdc7 in blk_co_pwritev (blk=0x7fd898cad540, offset=17394782208, bytes=4096, qiov=0x0, flags=<optimized out>) at block/block-backend.c:788
#16 0x0000561287cefeeb in blk_aio_write_entry (opaque=0x7fd812941440) at block/block-backend.c:982
#17 0x0000561287d67c7a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:78

Now, performance is really bad on ZFS for those lseek().
I believe that's https://github.com/zfsonlinux/zfs/issues/4306

Until that's fixed in ZFS, I need to find a way to avoid those
lseek()s in the first place.

One way is to downgrade to 2.6.2 where those lseek()s are not
called. The change that introduced them seems to be:

https://github.com/qemu/qemu/commit/2928abce6d1d426d37c0a9bd5f85fb95cf33f709
(and there have been further changes to improve it later).

If I understand correctly, that change was about preventing data
from being allocated when the user is writing unaligned zeroes.

I suppose the idea is that if something is trying to write
zeroes in the middle of an _allocated_ qcow2 cluster, but the
corresponding sectors in the file underneath are in a hole, we
don't want to write those zeros as that would allocate the data
at the file level.

I can see it makes sense, but in my case, the little space
efficiency it brings is largely overshadowed by the sharp
decrease in performance.

For now, I work around it by changing the "#ifdef SEEK_DATA"
to "#if 0" in find_allocation().

Note that passing detect-zeroes=off or detect-zeroes=unmap (with
discard) doesn't help (even though FALLOC_FL_PUNCH_HOLE is
supported on ZFS on Linux).

Is there any other way I could use to prevent those lseek()s
without having to rebuild qemu?

Would you consider adding an option to disable that behaviour
(skip checking allocation at file level for qcow2 image)?

Thanks,
Stephane

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Laszlo Ersek]

On 02/02/17 13:30, Stephane Chazelas wrote:
> Hello,
> 
> since qemu-2.7.0, doing synchronised I/O in a VM (tested with
> Ubuntu 16.04 amd64 VM)  while the disk is backed by a qcow2
> file sitting on a ZFS filesystem (zfs on Linux on Debian jessie
> (PVE)), the performances are dreadful:
> 
> # time dd if=/dev/zero count=1000  of=b oflag=dsync
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB, 500 KiB) copied, 21.9908 s, 23.3 kB/s
> dd if=/dev/zero count=1000 of=b oflag=dsync  0.00s user 0.04s system 0% cpu 21.992 total
> 
> (22 seconds to write that half megabyte). Same with O_SYNC or
> O_DIRECT, or doing fsync() or sync_file_range() after each
> write().
> 
> I first noticed it for dpkg unpacking kernel headers where dpkg
> does a sync_file_range() after each file is extracted.
> 
> Note that it doesn't happen when writing anything else than
> zeroes (like tr '\0' x < /dev/zero | dd count=1000  of=b
> oflag=dsync). In the case of the kernel headers, I suppose the
> zeroes come from the non-filled parts of the ext4 blocks.
> 
> Doing strace -fc on the qemu process, 98% of the time is spent
> in the lseek() system call.
> 
> That's the lseek(SEEK_DATA) followed by lseek(SEEK_HOLE) done by
> find_allocation() called to find out whether sectors are within
> a hole in a sparse file.
> 
> #0  lseek64 () at ../sysdeps/unix/syscall-template.S:81
> #1  0x0000561287cf4ca8 in find_allocation (bs=0x7fd898d70000, hole=<synthetic pointer>, data=<synthetic pointer>, start=<optimized out>)
>     at block/raw-posix.c:1702
> #2  raw_co_get_block_status (bs=0x7fd898d70000, sector_num=<optimized out>, nb_sectors=40, pnum=0x7fd80dd05aac, file=0x7fd80dd05ab0) at block/raw-posix.c:1765
> #3  0x0000561287cfae92 in bdrv_co_get_block_status (bs=0x7fd898d70000, sector_num=sector_num@entry=1303680, nb_sectors=40, pnum=pnum@entry=0x7fd80dd05aac,
>     file=file@entry=0x7fd80dd05ab0) at block/io.c:1709
> #4  0x0000561287cfafaa in bdrv_co_get_block_status (bs=bs@entry=0x7fd898d66000, sector_num=sector_num@entry=33974144, nb_sectors=<optimized out>,
>     nb_sectors@entry=40, pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1742
> #5  0x0000561287cfb0bb in bdrv_co_get_block_status_above (file=0x7fd80dd05bc0, pnum=0x7fd80dd05bbc, nb_sectors=40, sector_num=33974144, base=0x0,
>     bs=<optimized out>) at block/io.c:1776
> #6  bdrv_get_block_status_above_co_entry (opaque=opaque@entry=0x7fd80dd05b40) at block/io.c:1792
> #7  0x0000561287cfae08 in bdrv_get_block_status_above (bs=0x7fd898d66000, base=base@entry=0x0, sector_num=<optimized out>, nb_sectors=nb_sectors@entry=40,
>     pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1824
> #8  0x0000561287cd372d in is_zero_sectors (bs=<optimized out>, start=<optimized out>, count=40) at block/qcow2.c:2428
> #9  0x0000561287cd38ed in is_zero_sectors (count=<optimized out>, start=<optimized out>, bs=<optimized out>) at block/qcow2.c:2471
> #10 qcow2_co_pwrite_zeroes (bs=0x7fd898d66000, offset=33974144, count=24576, flags=2724114573) at block/qcow2.c:2452
> #11 0x0000561287cfcb7f in bdrv_co_do_pwrite_zeroes (bs=bs@entry=0x7fd898d66000, offset=offset@entry=17394782208, count=count@entry=4096,
>     flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block/io.c:1218
> #12 0x0000561287cfd0cb in bdrv_aligned_pwritev (bs=0x7fd898d66000, req=<optimized out>, offset=17394782208, bytes=4096, align=1, qiov=0x0,
>     flags=<optimized out>) at block/io.c:1320
> #13 0x0000561287cfe450 in bdrv_co_do_zero_pwritev (req=<optimized out>, flags=<optimized out>, bytes=<optimized out>, offset=<optimized out>,
>     bs=<optimized out>) at block/io.c:1422
> #14 bdrv_co_pwritev (child=0x15, offset=17394782208, bytes=4096, qiov=0x7fd8a25eb08d <lseek64+45>, qiov@entry=0x0, flags=231758512) at block/io.c:1492
> #15 0x0000561287cefdc7 in blk_co_pwritev (blk=0x7fd898cad540, offset=17394782208, bytes=4096, qiov=0x0, flags=<optimized out>) at block/block-backend.c:788
> #16 0x0000561287cefeeb in blk_aio_write_entry (opaque=0x7fd812941440) at block/block-backend.c:982
> #17 0x0000561287d67c7a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:78
> 
> Now, performance is really bad on ZFS for those lseek().
> I believe that's https://github.com/zfsonlinux/zfs/issues/4306
> 
> Until that's fixed in ZFS, I need to find a way to avoid those
> lseek()s in the first place.
> 
> One way is to downgrade to 2.6.2 where those lseek()s are not
> called. The change that introduced them seems to be:
> 
> https://github.com/qemu/qemu/commit/2928abce6d1d426d37c0a9bd5f85fb95cf33f709
> (and there have been further changes to improve it later).
> 
> If I understand correctly, that change was about preventing data
> from being allocated when the user is writing unaligned zeroes.
> 
> I suppose the idea is that if something is trying to write
> zeroes in the middle of an _allocated_ qcow2 cluster, but the
> corresponding sectors in the file underneath are in a hole, we
> don't want to write those zeros as that would allocate the data
> at the file level.
> 
> I can see it makes sense, but in my case, the little space
> efficiency it brings is largely overshadowed by the sharp
> decrease in performance.
> 
> For now, I work around it by changing the "#ifdef SEEK_DATA"
> to "#if 0" in find_allocation().
> 
> Note that passing detect-zeroes=off or detect-zeroes=unmap (with
> discard) doesn't help (even though FALLOC_FL_PUNCH_HOLE is
> supported on ZFS on Linux).
> 
> Is there any other way I could use to prevent those lseek()s
> without having to rebuild qemu?

My suggestion will likely be incredibly lame, but let's hope it at least
directs some attention to your query.

You didn't mention what qcow2 features you use -- vmstate, snapshots,
backing files (chains of them), compression?

Since commit 2928abce6d1d only modifies "block/qcow2.c", you could
switch / convert the images to "raw". "raw" still benefits from sparse
files (which ZFS-on-Linux apparently supports). Sparse files (i.e., the
disk space savings) are the most important feature to me at least.

Thanks (and sorry again about the lame idea... you likely have good
reasons for qcow2...)
Laszlo

> 
> Would you consider adding an option to disable that behaviour
> (skip checking allocation at file level for qcow2 image)?
> 
> Thanks,
> Stephane
> 
> 

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Stephane Chazelas]

2017-02-02 16:23:53 +0100, Laszlo Ersek:
[...]
> You didn't mention what qcow2 features you use -- vmstate, snapshots,
> backing files (chains of them), compression?
> 
> Since commit 2928abce6d1d only modifies "block/qcow2.c", you could
> switch / convert the images to "raw". "raw" still benefits from sparse
> files (which ZFS-on-Linux apparently supports). Sparse files (i.e., the
> disk space savings) are the most important feature to me at least.
[...]

Thanks for the feedback.

Sorry for not mentioning it in the first place, but I do need
the vmstate and snapshots (even non-linear snapshots which means
even ZFS zvol snapshots as done by Proxmox VE are not an option
either, neither is vmdk)


I hadn't tested before now, but what I observe with raw
devices and discard=on,detect-zeroes=unmap (and the virtio-scsi
interface), is that upon those "synced writes of zeroes" into
allocated data, qemu does some

[pid 10535] fallocate(14, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 136314880, 4096) = 0

into the disk image.

(and no lseek(SEEK_DATA/SEEK_HOLE))

which I don't see when using qcow2 images.

If the qcow2 interface was updated to do the same (punch holes
regardless instead of checking if the data is allocated
beforehand), that would also solve my problem (anything that
avoid those lseek()s being called).

Another thing I've not mentioned clearly is the versions of qemu
I have been testing with: 2.7, 2.7.1 (those two on Proxmox VE
4.4 (based on Debian jessie)) and 2.8.0 (the latter for
verification on a Debian unstable system, not with zfs).

-- 
Stephane

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 7227 bytes --]

Hi,

I've been thinking about the issue but I'm not sure I've come to a
resolution you'll like much.

I'm not really in favor of optimizing code for ZFS, especially if that
means worse code for every other case. I think it very much makes sense
to assume that lseek(SEEK_{DATA,HOLE}) is faster than writing data to
disk, and actually so much faster that it even pays off if you sometimes
to the lseek() only to find out that you actually have to write data still.

Therefore, the patch as it is makes sense. The fact that said lseek() is
slow on ZFS is (in my humble opinion) the ZFS driver's problem that
needs to be fixed there.

If ZFS has a good alternative for us to check whether a given area of a
file will return zeroes when read, I'm all ears and it might be a good
idea to use it. That is, if someone can write the code for it because
I'd rather not if that requires ZFS headers and a ZFS for testing.

(Determining whether a file has a hole in it and where it is has
actually plagued as for a while now. lseek() seemed to be the most
widespread way with the least amount of pitfalls to do it.)

OTOH, it may make sense to offer a way for the user to disable
lseek(SEEK_{DATA,HOLE}) in our "file" block driver. That way your issue
would be solved, too, I guess. I'll look into it.


Max



On 02.02.2017 13:30, Stephane Chazelas wrote:
> Hello,
> 
> since qemu-2.7.0, doing synchronised I/O in a VM (tested with
> Ubuntu 16.04 amd64 VM)  while the disk is backed by a qcow2
> file sitting on a ZFS filesystem (zfs on Linux on Debian jessie
> (PVE)), the performances are dreadful:
> 
> # time dd if=/dev/zero count=1000  of=b oflag=dsync
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB, 500 KiB) copied, 21.9908 s, 23.3 kB/s
> dd if=/dev/zero count=1000 of=b oflag=dsync  0.00s user 0.04s system 0% cpu 21.992 total
> 
> (22 seconds to write that half megabyte). Same with O_SYNC or
> O_DIRECT, or doing fsync() or sync_file_range() after each
> write().
> 
> I first noticed it for dpkg unpacking kernel headers where dpkg
> does a sync_file_range() after each file is extracted.
> 
> Note that it doesn't happen when writing anything else than
> zeroes (like tr '\0' x < /dev/zero | dd count=1000  of=b
> oflag=dsync). In the case of the kernel headers, I suppose the
> zeroes come from the non-filled parts of the ext4 blocks.
> 
> Doing strace -fc on the qemu process, 98% of the time is spent
> in the lseek() system call.
> 
> That's the lseek(SEEK_DATA) followed by lseek(SEEK_HOLE) done by
> find_allocation() called to find out whether sectors are within
> a hole in a sparse file.
> 
> #0  lseek64 () at ../sysdeps/unix/syscall-template.S:81
> #1  0x0000561287cf4ca8 in find_allocation (bs=0x7fd898d70000, hole=<synthetic pointer>, data=<synthetic pointer>, start=<optimized out>)
>     at block/raw-posix.c:1702
> #2  raw_co_get_block_status (bs=0x7fd898d70000, sector_num=<optimized out>, nb_sectors=40, pnum=0x7fd80dd05aac, file=0x7fd80dd05ab0) at block/raw-posix.c:1765
> #3  0x0000561287cfae92 in bdrv_co_get_block_status (bs=0x7fd898d70000, sector_num=sector_num@entry=1303680, nb_sectors=40, pnum=pnum@entry=0x7fd80dd05aac,
>     file=file@entry=0x7fd80dd05ab0) at block/io.c:1709
> #4  0x0000561287cfafaa in bdrv_co_get_block_status (bs=bs@entry=0x7fd898d66000, sector_num=sector_num@entry=33974144, nb_sectors=<optimized out>,
>     nb_sectors@entry=40, pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1742
> #5  0x0000561287cfb0bb in bdrv_co_get_block_status_above (file=0x7fd80dd05bc0, pnum=0x7fd80dd05bbc, nb_sectors=40, sector_num=33974144, base=0x0,
>     bs=<optimized out>) at block/io.c:1776
> #6  bdrv_get_block_status_above_co_entry (opaque=opaque@entry=0x7fd80dd05b40) at block/io.c:1792
> #7  0x0000561287cfae08 in bdrv_get_block_status_above (bs=0x7fd898d66000, base=base@entry=0x0, sector_num=<optimized out>, nb_sectors=nb_sectors@entry=40,
>     pnum=pnum@entry=0x7fd80dd05bbc, file=file@entry=0x7fd80dd05bc0) at block/io.c:1824
> #8  0x0000561287cd372d in is_zero_sectors (bs=<optimized out>, start=<optimized out>, count=40) at block/qcow2.c:2428
> #9  0x0000561287cd38ed in is_zero_sectors (count=<optimized out>, start=<optimized out>, bs=<optimized out>) at block/qcow2.c:2471
> #10 qcow2_co_pwrite_zeroes (bs=0x7fd898d66000, offset=33974144, count=24576, flags=2724114573) at block/qcow2.c:2452
> #11 0x0000561287cfcb7f in bdrv_co_do_pwrite_zeroes (bs=bs@entry=0x7fd898d66000, offset=offset@entry=17394782208, count=count@entry=4096,
>     flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block/io.c:1218
> #12 0x0000561287cfd0cb in bdrv_aligned_pwritev (bs=0x7fd898d66000, req=<optimized out>, offset=17394782208, bytes=4096, align=1, qiov=0x0,
>     flags=<optimized out>) at block/io.c:1320
> #13 0x0000561287cfe450 in bdrv_co_do_zero_pwritev (req=<optimized out>, flags=<optimized out>, bytes=<optimized out>, offset=<optimized out>,
>     bs=<optimized out>) at block/io.c:1422
> #14 bdrv_co_pwritev (child=0x15, offset=17394782208, bytes=4096, qiov=0x7fd8a25eb08d <lseek64+45>, qiov@entry=0x0, flags=231758512) at block/io.c:1492
> #15 0x0000561287cefdc7 in blk_co_pwritev (blk=0x7fd898cad540, offset=17394782208, bytes=4096, qiov=0x0, flags=<optimized out>) at block/block-backend.c:788
> #16 0x0000561287cefeeb in blk_aio_write_entry (opaque=0x7fd812941440) at block/block-backend.c:982
> #17 0x0000561287d67c7a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:78
> 
> Now, performance is really bad on ZFS for those lseek().
> I believe that's https://github.com/zfsonlinux/zfs/issues/4306
> 
> Until that's fixed in ZFS, I need to find a way to avoid those
> lseek()s in the first place.
> 
> One way is to downgrade to 2.6.2 where those lseek()s are not
> called. The change that introduced them seems to be:
> 
> https://github.com/qemu/qemu/commit/2928abce6d1d426d37c0a9bd5f85fb95cf33f709
> (and there have been further changes to improve it later).
> 
> If I understand correctly, that change was about preventing data
> from being allocated when the user is writing unaligned zeroes.
> 
> I suppose the idea is that if something is trying to write
> zeroes in the middle of an _allocated_ qcow2 cluster, but the
> corresponding sectors in the file underneath are in a hole, we
> don't want to write those zeros as that would allocate the data
> at the file level.
> 
> I can see it makes sense, but in my case, the little space
> efficiency it brings is largely overshadowed by the sharp
> decrease in performance.
> 
> For now, I work around it by changing the "#ifdef SEEK_DATA"
> to "#if 0" in find_allocation().
> 
> Note that passing detect-zeroes=off or detect-zeroes=unmap (with
> discard) doesn't help (even though FALLOC_FL_PUNCH_HOLE is
> supported on ZFS on Linux).
> 
> Is there any other way I could use to prevent those lseek()s
> without having to rebuild qemu?
> 
> Would you consider adding an option to disable that behaviour
> (skip checking allocation at file level for qcow2 image)?
> 
> Thanks,
> Stephane
> 
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Stephane Chazelas]

2017-02-08 00:43:18 +0100, Max Reitz:
[...]
> OTOH, it may make sense to offer a way for the user to disable
> lseek(SEEK_{DATA,HOLE}) in our "file" block driver. That way your issue
> would be solved, too, I guess. I'll look into it.
[...]

Thanks Max,

Yes, that would work for me and other users of ZFS. What I do
for now is recompile with those lseek(SEEK_{DATA,HOLE}) disabled
in the code and it's working fine.

As I already hinted, something that would also possibly work for
me and could benefit everyone (well at least Linux users on
filesystems supporting hole punching), is instead of checking
beforehand if the file is allocated, do a
fallocate(FALLOC_FL_PUNCH_HOLE), or IOW, tell the underlying
layer to deallocate the data.

That would be those two lseek() replaced by a fallocate(), and
some extra disk space being saved.

One may argue that's what one would expect would be done when
using detect-zeroes=unmap. 

I suppose that would be quite significant work as that would
imply a framework to pass those "deallocates" down and you'd
probably have to differenciate "deallocates" that zero (like
hole punching in a reguar file), and those that don't (like
BLKDISCARD on a SSD)

I also suppose that could cause fragmentation that  would be
unwanted in some contexts, so maybe it should be tunable as
well.

-- 
Stephane

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Stephane Chazelas]

2017-02-08 00:43:18 +0100, Max Reitz:
[...]
> Therefore, the patch as it is makes sense. The fact that said lseek() is
> slow on ZFS is (in my humble opinion) the ZFS driver's problem that
> needs to be fixed there.
[...]

For the record, I've mentioned the qemu performance implication at
https://github.com/zfsonlinux/zfs/issues/4306#issuecomment-277000682

Not much more I can do at that point.

That issue was raised a year ago and has not been assigned any
milestone yet.

-- 
Stephane

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 2650 bytes --]

On 08.02.2017 15:06, Stephane Chazelas wrote:
> 2017-02-08 00:43:18 +0100, Max Reitz:
> [...]
>> OTOH, it may make sense to offer a way for the user to disable
>> lseek(SEEK_{DATA,HOLE}) in our "file" block driver. That way your issue
>> would be solved, too, I guess. I'll look into it.
> [...]
> 
> Thanks Max,
> 
> Yes, that would work for me and other users of ZFS. What I do
> for now is recompile with those lseek(SEEK_{DATA,HOLE}) disabled
> in the code and it's working fine.
> 
> As I already hinted, something that would also possibly work for
> me and could benefit everyone (well at least Linux users on
> filesystems supporting hole punching), is instead of checking
> beforehand if the file is allocated, do a
> fallocate(FALLOC_FL_PUNCH_HOLE), or IOW, tell the underlying
> layer to deallocate the data.
> 
> That would be those two lseek() replaced by a fallocate(), and
> some extra disk space being saved.

When using qcow2, however, qcow2 will try to take care of that by
discarding clusters or using special zero clusters.

The lseek() thing just tries to mitigate the effect of writing less than
a cluster of zeroes. Yes, we could punch a hole, but is that faster or
has at least the same speed as lseek(SEEK_{DATA,HOLE}) on all filesystems?

Also, is it the same speed on all protocols? We not only support files
as image storage, but also network protocols etc..

But this is just generally speaking for the "write zeroes" case. With
detect-zeroes=unmap (as opposed to detect-zeroes=on), things are a bit
different. It may indeed make sense to fall through to the protocol
level and punch holes there, even if it may be slower than lseek().

> One may argue that's what one would expect would be done when
> using detect-zeroes=unmap.

A bit of a stupid question, but: How is your performance when using
detect-zeroes=off?

> I suppose that would be quite significant work as that would
> imply a framework to pass those "deallocates" down and you'd
> probably have to differenciate "deallocates" that zero (like
> hole punching in a reguar file), and those that don't (like
> BLKDISCARD on a SSD)

Well, we internally already have different functions for writing zeroes
and discarding. The thing is, though, that qcow2 is supposed to handle
those deallocations and not hand them down, because qcow2 can handle
them -- but not if they're not aligned to whole qcow2 clusters (which,
by default, are 64 kB in size).

Max

> 
> I also suppose that could cause fragmentation that  would be
> unwanted in some contexts, so maybe it should be tunable as
> well.
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [qcow2] how to avoid qemu doing lseek(SEEK_DATA/SEEK_HOLE)?

[Stephane Chazelas]

2017-02-08 15:27:11 +0100, Max Reitz:
[...]
> A bit of a stupid question, but: How is your performance when using
> detect-zeroes=off?
[...]

I did try that. See:

} Note that passing detect-zeroes=off or detect-zeroes=unmap (with
} discard) doesn't help (even though FALLOC_FL_PUNCH_HOLE is
} supported on ZFS on Linux).

In my original message. It makes no difference, I still see
those lseek()s being done.

-- 
Stephane