fio verify and short writes

fio verify and short writes

[Andreas Gruenbacher]

Hi,

I've run into a verify issue with fio in xfstest generic/300 [*]:

* The aio-dio-verifier thread in that test case writes 4k buffers and
then reads them back in.
* The filesystem has a block size if 1k.
* The test case is designed to run the filesystem out of space. When
that happens, some writes can end up short before they all fail with
-ENOSPC. (That doesn't seem to happen on ext4 or xfs filesystems with
a 1k block size --- possibly just because the test always runs on a
scratch filesystem --- but it does trigger on gfs2.)
* When the aio-dio-verifier thread reads a short 4k buffer back in,
part of the buffer will be zero-filled, so verification will fail.

This is with a current fio and kernel.

Any thoughts?

Thanks,
Andreas

[*] https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/tests/generic/300

Re: fio verify and short writes

[Sitsofe Wheeler]

On Tue, 3 Dec 2019 at 15:27, Andreas Gruenbacher <agruenba@redhat.com> wrote:
>
> Hi,
>
> I've run into a verify issue with fio in xfstest generic/300 [*]:
>
> * The aio-dio-verifier thread in that test case writes 4k buffers and
> then reads them back in.
> * The filesystem has a block size if 1k.
> * The test case is designed to run the filesystem out of space. When
> that happens, some writes can end up short before they all fail with
> -ENOSPC. (That doesn't seem to happen on ext4 or xfs filesystems with
> a 1k block size --- possibly just because the test always runs on a
> scratch filesystem --- but it does trigger on gfs2.)
> * When the aio-dio-verifier thread reads a short 4k buffer back in,
> part of the buffer will be zero-filled, so verification will fail.
>
> This is with a current fio and kernel.
>
> Any thoughts?

Hmm I don't know what fio could do here - you're explicitly ignoring
the ENOSPC error on the write phase so fio just says "OK it's not an
error" and logs the write I/O as successful. Fio verification depends
on a whole blocks worth of data being written so the best you could do
is simply to not verify the data in the final "short" block...

I suppose you could write a new feature in fio ("ignore_without_log")
that could take error codes and say "ENOSPC isn't a failure but don't
record any I/O that fails with it" so it would learn not to log that
last block. Maybe such a feature would be complicated because in many
cases it means you break the rule that you can check the rest of the
sequence of I/O that follows...

> Thanks,
> Andreas
>
> [*] https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/tests/generic/300

-- 
Sitsofe | http://sucs.org/~sits/