How to ensure split verification will generate the same configs as write phase?

How to ensure split verification will generate the same configs as write phase?

[Vaughan]

Hi all,

I want fio to do an split verification for RAID as did in
https://github.com/axboe/fio/issues/468. Once written, the verification will
be repeat several times for raid recovery for each disk.

The example job file is as below:
----- write job -----
[global]
do_verify=0
iodepth=32
filename=/dev/nvme0n6
rw=randwrite
ioengine=libaio
verify=crc32c-intel
direct=1
verify_offset=100
bs=4096
[job_0]
size=209715200
offset=0

----- read job -----
[global]
do_verify=1
verify_state_load=1
iodepth=32
filename=/dev/nvme0n6
rw=read
ioengine=libaio
verify=crc32c-intel
direct=1
verify_offset=100
bs=4096

[job_0]
size=209715200
offset=0

I don't see it use the same randseed. Can it generate the same offset as
write phase without randseed= option set? 
If I use randbs= to also use an random write blocksizes, how can I ensure
the verification will do the same as write?
Can write_iolog ensure that? However, I suppose it may slow down the write
speed and randomness of test.


Regards,
Vaughan

Re: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

Hi,

On 27 February 2018 at 10:45, Vaughan <cxt9401@163.com> wrote:
>
> I want fio to do an split verification for RAID as did in
> https://github.com/axboe/fio/issues/468. Once written, the verification will
> be repeat several times for raid recovery for each disk.
>
> The example job file is as below:
> ----- write job -----
> [global]
> do_verify=0
> iodepth=32
> filename=/dev/nvme0n6
> rw=randwrite
> ioengine=libaio
> verify=crc32c-intel
> direct=1
> verify_offset=100
> bs=4096
> [job_0]
> size=209715200
> offset=0
>
> ----- read job -----
> [global]
> do_verify=1
> verify_state_load=1
> iodepth=32
> filename=/dev/nvme0n6
> rw=read

^^^ Here's a problem - you can't randomly write and then verify
sequentially via a different run. In your case you need rw=randread.
Note this is exactly what Jens was warning about in
https://github.com/axboe/fio/issues/468#issuecomment-332821507 :

"It works for me, as long as you make sure that the rw=write is paired
with rw=read, or if rw=randwrite is paired with rw=randread. It can't
work otherwise."

> ioengine=libaio
> verify=crc32c-intel
> direct=1
> verify_offset=100
> bs=4096
>
> [job_0]
> size=209715200
> offset=0
>
> I don't see it use the same randseed. Can it generate the same offset as
> write phase without randseed= option set?

The  randseed is the same across different invocations of fio unless
you say otherwise (see randrepeat -
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
). So even if you weren't using a verification state file you will
re-generate the same pattern if the rest of the job is the same.

Next, in your case you're explicitly reloading the verification state
(notice how you carefully kept the job name the same) which contains
the random seed state and is able to restore them. So even if the
random seed were being randomly chosen your second job will reseed it
to the original value.

> If I use randbs= to also use an random write blocksizes, how can I ensure
> the verification will do the same as write?

See above. Try a very small run (e.g. where size is only 64k) with
--debug=io and bsrange (which I'm guessing is what you meant) to check
this.

> Can write_iolog ensure that? However, I suppose it may slow down the write
> speed and randomness of test.

I suppose you could do it that way. I wouldn't expect it to make
things less random but yes the speed could be impacted (and there's
the issue of how big that log file will be etc).

-- 
Sitsofe | http://sucs.org/~sits/

答复: How to ensure split verification will generate the same configs as write phase?

[Vaughan]

> ^^^ Here's a problem - you can't randomly write and then verify sequentially via a different run. In your case you need rw=randread.
Yes. It's my fault to paste the original content of the buggy job file. I understand that rw= option should be paired with or without rand- prefix.

> The  randseed is the same across different invocations of fio unless you say otherwise (see randrepeat - http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
> ). So even if you weren't using a verification state file you will re-generate the same pattern if the rest of the job is the same.
Thank you for the explanation. A deep question about the seed comes:) Though randseed is the same across different invocation of fio, is it different for different sections in one job file?

I made another job file as below:
myjob.fio:
	[global] 
	bs=4k    
	direct=1 
	ioengine=sync
	verify=crc32c  
	filename=/dev/sdf
	io_size=10k

	[write-phase]  
	rw=randwrite
	do_verify=0
	write_iolog=iolog

	[verify-phase]
	stonewall
	rw=randread
	do_verify=1

I think these two sections have the same settings except the date direction. So I expect the later one can verify the write-phase. But it failed.	
run it with --debug, and found the offset generated are not the same, so verify abort with error for the first time, as below: 
	random   24382 off rand 259043585
	io       24382 fill_io_u: io_u 0x25ad740: off=71233536/len=4096/ddir=1//dev/sdf
	random   24382 off rand 3179521932
	io       24382 fill_io_u: io_u 0x25ad740: off=874364928/len=4096/ddir=1//dev/sdf
	random   24382 off rand 3621444214
	io       24382 fill_io_u: io_u 0x25ad740: off=995893248/len=4096/ddir=1//dev/sdf
	random   24388 off rand 1094293889
	io       24388 fill_io_u: io_u 0x25ad500: off=300929024/len=4096/ddir=0//dev/sdf
You can see [24382] is write-phase and [24388] is verify-phase, and the first round of them use different seed I suppose.

I search in the source code, and found the following snippet:
	static int setup_random_seeds(struct thread_data *td)
	{
	...
			for (i = 0; i < FIO_RAND_NR_OFFS; i++) {
					td->rand_seeds[i] = seed * td->thread_number + i;
					seed *= 0x9e370001UL;
			}               

Is td->thread_number different for different sections? I thought they both equal to 1 before.

Later, I realize split verify using two sections in one file is not a formal way? Is it better to just write only one write section, and use 'fio --verify-only jobfile' to do the split verification as the updated one below?

myjob.fio updated:
	[global] 
	bs=4k    
	direct=1 
	ioengine=sync
	verify=crc32c  
	filename=/dev/sdf
	io_size=10k

	[write-phase]  
	rw=randwrite
	do_verify=0

Thank you very much again!

Regards,
Vaughan

Re: 答复: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

Hi,

On 28 February 2018 at 08:44, Vaughan <cxt9401@163.com> wrote:
>
>> The  randseed is the same across different invocations of fio unless you say otherwise (see randrepeat - http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
>> ). So even if you weren't using a verification state file you will re-generate the same pattern if the rest of the job is the same.
> Thank you for the explanation. A deep question about the seed comes:) Though randseed is the same across different invocation of fio, is it different for different sections in one job file?

> I made another job file as below:
> myjob.fio:
>         [global]
>         bs=4k
>         direct=1
>         ioengine=sync
>         verify=crc32c
>         filename=/dev/sdf
>         io_size=10k

^^^ Note that io_size may create a potential issue - we're not
necessarily going to complete a total loop (e.g. /dev/sdf is bigger
than 10k) and the second job may not be aware we stopped the first job
part way. In the issue you linked
(https://github.com/axboe/fio/issues/468#issuecomment-332828729 ) the
reporter was asking something similar (although with triggers).

>         [write-phase]
>         rw=randwrite
>         do_verify=0
>         write_iolog=iolog
>
>         [verify-phase]
>         stonewall
>         rw=randread
>         do_verify=1
>
> I think these two sections have the same settings except the date direction. So I expect the later one can verify the write-phase. But it failed.
> run it with --debug, and found the offset generated are not the same, so verify abort with error for the first time, as below:
>         random   24382 off rand 259043585
>         io       24382 fill_io_u: io_u 0x25ad740: off=71233536/len=4096/ddir=1//dev/sdf
>         random   24382 off rand 3179521932
>         io       24382 fill_io_u: io_u 0x25ad740: off=874364928/len=4096/ddir=1//dev/sdf
>         random   24382 off rand 3621444214
>         io       24382 fill_io_u: io_u 0x25ad740: off=995893248/len=4096/ddir=1//dev/sdf
>         random   24388 off rand 1094293889
>         io       24388 fill_io_u: io_u 0x25ad500: off=300929024/len=4096/ddir=0//dev/sdf
> You can see [24382] is write-phase and [24388] is verify-phase, and the first round of them use different seed I suppose.
>
> I search in the source code, and found the following snippet:
>         static int setup_random_seeds(struct thread_data *td)
>         {
>         ...
>                         for (i = 0; i < FIO_RAND_NR_OFFS; i++) {
>                                         td->rand_seeds[i] = seed * td->thread_number + i;
>                                         seed *= 0x9e370001UL;
>                         }
>
> Is td->thread_number different for different sections? I thought they both equal to 1 before.

It looks that way - it seems like thread_number is incremented for each job in
get_new_job() so this would affect the random seeds generated in
different sections of the same job file.

Note that because you're verifying by header and via a separate job I
believe fio checks the header in the current block and if it looks
intact it then uses that to regenerate the data in the rest of the
block. You can get just about away with this when you're writing and
verifying the entirety of the file with fixed block sizes but I think
what you've pointed out will make jobs like:

fio --stonewall --verify=crc32c --size=24k --bsrange=512-4k
--filename=/tmp/fio.tmp --name=write-phase --rw=randwrite
--do_verify=0 --name=verify-phase --rw=randread --do_verify=1

fail because the random ordering of block sizes now depends on the job
number too.

> Later, I realize split verify using two sections in one file is not a formal way? Is it better to just write only one write section, and use 'fio --verify-only jobfile' to do the split verification as the updated one below?

That does look like it will be more robust and will also allow you to
use a state file if you wanted too.

-- 
Sitsofe | http://sucs.org/~sits/

答复: 答复: How to ensure split verification will generate the same configs as write phase?

[Vaughan]

> It looks that way - it seems like thread_number is incremented for each job in
> get_new_job() so this would affect the random seeds generated in different sections of the same job file.
>  I think what you've pointed out will make jobs like:
> 
> fio --stonewall --verify=crc32c --size=24k --bsrange=512-4k --filename=/tmp/fio.tmp --name=write-phase --rw=randwrite --do_verify=0 --name=verify-phase --rw=randread --do_verify=1
> 
> fail because the random ordering of block sizes now depends on the job number too.
Yes, I agree with you. So the right way should be verify it with the original write jobfile and appending "--verify_only".

Can you further explain the usage of  --verify_state_save --verify_backlog --verify_backlog_batch options?

I need to split verify tera-byte size of data. According to the HOWTO of fio to avoid memory exhaustion, verify_backlog should be used. so I do the write job with "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240" in cmd line and do a split verify by appending "--verify_only" with the same write command options. But it seems it just verify the 10240 ios, not the same size as write job.
Only if I verify without "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240", it verify the whole size I write. I get the conclusion by run the write work and verify work with "write_iolog=str" and compare the iolog, and also "ios=" in "Disk stats" section of both fio output.

Another question about write process with "rw=randwrite" and "bsrange=". If run it with "ioengine=libaio iodepth=16", will this async write get any overlap?

Thanks,
Vaughan

Re: 答复: 答复: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

On 9 March 2018 at 06:11, Vaughan <cxt9401@163.com> wrote:
>
> Can you further explain the usage of  --verify_state_save --verify_backlog --verify_backlog_batch options?

(I'll just note, if you change the topic of discussion it's usually
better netiquette to start a new email thread rather than continue an
old topic)

There are explanations on
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-verify-state-save
, http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-verify-backlog
, http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-verify-backlog-batch
if you haven't seen them already. It's worth noting that
verify_state_save isn't really anything to do with verify_backlog. A
bit of background:

Normally when you run a write job with verification turned on
(remember do_verify defaults to 1 if verification has been enabled) it
will
1. Do all the writes (so called "write phase")
2. After the last write has finished, it will do reads of the all the
writes done to verify them ("verification phase").

1 and 2 happen as part of the same job (side note: to stop 2. from
occuring you would need to use do_verify=0)! However a bit of memory
is chewed up for each region written to do this style of verification
because fio actually builds verification log during 1 and then uses
that log in 2.

When you also enable verify_backlog the following happens instead:
1a. Do some writes.
1b. After verify_backlog writes are done verify the data performed by
the most recent 1a. writes.
1c. Go to 1a and repeat until all blocks have been written and verified.

Verification is still done as part of the same job but it is done
inline with the writes during the "write phase" and there's no
separate "verification phase" after all the writes finish. This means
the verification log size will be bounded to no more than
verify_backlog amount (which saves memory). For an explanation of
verify_backlog_batch I'd recommend reading the HOWTO link from earlier
and asking specific questions if that explanation is unclear.

verify_state_save saves key values into a file which can then be
reloaded in an entirely separate fio invocation (e.g. to know how far
through fio got doing writes). A side note is that separate
verification invocations generally have less information available to
them at verification time in comparison to in-job verifications.

> I need to split verify tera-byte size of data. According to the HOWTO of fio to avoid memory exhaustion, verify_backlog should be used. so I do the write job with "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240" in cmd line and do a split verify by appending "--verify_only" with the same write command options. But it seems it just verify the 10240 ios, not the same size as write job.

If you have the memory you can certainly wait for the verification
phase to kick in after all the writes are done and make use of the
in-job verify. However verify_backlog doesn't make sense if you are
not performing writes in that exact same job (as there are no writes
to verify inline) and in the above you are turning off the writes when
you use verify_only. Additionally if you are aren't doing any sort of
in-job verification fio won't bother keeping a verification log in
memory.

> Only if I verify without "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240", it verify the whole size I write. I get the conclusion by run the write work and verify work with "write_iolog=str" and compare the iolog, and also "ios=" in "Disk stats" section of both fio output.

"--verify_backlog=1 --verify_backlog_batch=10240" seems unusual
combination of values to use - can you explain what you think this
does (make sure you've read the HOWTO paragraph about it)?

> Another question about write process with "rw=randwrite" and "bsrange=". If run it with "ioengine=libaio iodepth=16", will this async write get any overlap?

Maybe (but you can check the "IO depths" output to see what was
actually achieved)? Did you see the warning for libaio in the ioengine
section of the HOWTO
(http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-ioengine
and look at the libaio heading)?

-- 
Sitsofe | http://sucs.org/~sits/

Re: 答复: 答复: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

On 9 March 2018 at 07:17, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> Another question about write process with "rw=randwrite" and "bsrange=". If run it with "ioengine=libaio iodepth=16", will this async write get any overlap?
>
> Maybe (but you can check the "IO depths" output to see what was
> actually achieved)? Did you see the warning for libaio in the ioengine
> section of the HOWTO
> (http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-ioengine
> and look at the libaio heading)?

Perhaps I misunderstood what you mean by overlap and just in case I'll
check whether you've seen
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-norandommap
? When fio picks a random region to write by default it will ensure
that each "spot" is only picked once per loop pass and you would have
to set norandommap to turn this behaviour off. If you aren't randomly
writing multiple loop passes in the same job and you're not writing
without a randommap this is normally enough to stop the same region
being written by the concurrently submitted I/Os.

-- 
Sitsofe | http://sucs.org/~sits/