Question about data verification using FIO

Question about data verification using FIO

[Taylor Yang]

Hi,
I know for the option 'verify', we could specify it when we write and
read a file. But does it work with the option 'time_base' and
'runtime'?

For example, does the configuration below make sense?

[global]
direct=1
ioengine=libaio
time_based
numjobs=1
verify=crc32
runtime=50000

The verification process might not even start if the runtime ends,
right? If this is the case, what should I do to configure it in such a
way that it could do the read and write and also data verification in
some targeted period of time?

Thank you so much in advance.
Best wishes,
Taylor

Re: Question about data verification using FIO

[Sitsofe Wheeler]

Hi Taylor,

On Wed, 24 Oct 2018 at 23:29, Taylor Yang <taylor4869@gmail.com> wrote:
>
> I know for the option 'verify', we could specify it when we write and
> read a file. But does it work with the option 'time_base' and
> 'runtime'?

Maybe?

> For example, does the configuration below make sense?
>
> [global]
> direct=1
> ioengine=libaio
> time_based
> numjobs=1
> verify=crc32
> runtime=50000

This doesn't make sense to me because I can't see the rest of the
options / fio invocation you used. So first up there's no job defined
(only global options) so given the above by itself fio will complain
about "No job(s) defined". I also can't tell if you've already made
the file using some other fio invocation and you are intending this
one just to keep verifying it. fio defaults to read jobs when rw isn't
set (https://fio.readthedocs.io/en/latest/fio_man.html#cmdoption-arg-readwrite
) so this will be a pure verification job and if you haven't
previously created/filled the file with fio crc32 verification headers
you'll get an instant mismatch.

> The verification process might not even start if the runtime ends,
> right? If this is the case, what should I do to configure it in such a
> way that it could do the read and write and also data verification in
> some targeted period of time?

If you mean when you set rw=write then your deduction is right and
recent fios will even tell you this:
$ cat write_verify_timebased.fio
[global]
direct=1
ioengine=libaio
time_based
numjobs=1
verify=crc32
runtime=5s

[write_verify_timebased]
size=1M
rw=write
$ fio write_verify_timebased.fio
fio: verification read phase will never start because write phase uses
all of runtime
[...]

As for solutions, you may find doing inline verifies is good enough
for you (see http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-verify-backlog
). See https://github.com/axboe/fio/issues/322#issuecomment-283265965
for some discussion about trying to do time based verification (TLDR;
it can be tricky).



--
Sitsofe | http://sucs.org/~sits/