How to ensure split verification will generate the same configs as write phase?

How to ensure split verification will generate the same configs as write phase?

[Vaughan]

Hi all,

I want fio to do an split verification for RAID as did in
https://github.com/axboe/fio/issues/468. Once written, the verification will
be repeat several times for raid recovery for each disk.

The example job file is as below:
----- write job -----
[global]
do_verify=0
iodepth=32
filename=/dev/nvme0n6
rw=randwrite
ioengine=libaio
verify=crc32c-intel
direct=1
verify_offset=100
bs=4096
[job_0]
size=209715200
offset=0

----- read job -----
[global]
do_verify=1
verify_state_load=1
iodepth=32
filename=/dev/nvme0n6
rw=read
ioengine=libaio
verify=crc32c-intel
direct=1
verify_offset=100
bs=4096

[job_0]
size=209715200
offset=0

I don't see it use the same randseed. Can it generate the same offset as
write phase without randseed= option set? 
If I use randbs= to also use an random write blocksizes, how can I ensure
the verification will do the same as write?
Can write_iolog ensure that? However, I suppose it may slow down the write
speed and randomness of test.


Regards,
Vaughan

Re: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

Hi,

On 27 February 2018 at 10:45, Vaughan <cxt9401@163.com> wrote:
>
> I want fio to do an split verification for RAID as did in
> https://github.com/axboe/fio/issues/468. Once written, the verification will
> be repeat several times for raid recovery for each disk.
>
> The example job file is as below:
> ----- write job -----
> [global]
> do_verify=0
> iodepth=32
> filename=/dev/nvme0n6
> rw=randwrite
> ioengine=libaio
> verify=crc32c-intel
> direct=1
> verify_offset=100
> bs=4096
> [job_0]
> size=209715200
> offset=0
>
> ----- read job -----
> [global]
> do_verify=1
> verify_state_load=1
> iodepth=32
> filename=/dev/nvme0n6
> rw=read

^^^ Here's a problem - you can't randomly write and then verify
sequentially via a different run. In your case you need rw=randread.
Note this is exactly what Jens was warning about in
https://github.com/axboe/fio/issues/468#issuecomment-332821507 :

"It works for me, as long as you make sure that the rw=write is paired
with rw=read, or if rw=randwrite is paired with rw=randread. It can't
work otherwise."

> ioengine=libaio
> verify=crc32c-intel
> direct=1
> verify_offset=100
> bs=4096
>
> [job_0]
> size=209715200
> offset=0
>
> I don't see it use the same randseed. Can it generate the same offset as
> write phase without randseed= option set?

The  randseed is the same across different invocations of fio unless
you say otherwise (see randrepeat -
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
). So even if you weren't using a verification state file you will
re-generate the same pattern if the rest of the job is the same.

Next, in your case you're explicitly reloading the verification state
(notice how you carefully kept the job name the same) which contains
the random seed state and is able to restore them. So even if the
random seed were being randomly chosen your second job will reseed it
to the original value.

> If I use randbs= to also use an random write blocksizes, how can I ensure
> the verification will do the same as write?

See above. Try a very small run (e.g. where size is only 64k) with
--debug=io and bsrange (which I'm guessing is what you meant) to check
this.

> Can write_iolog ensure that? However, I suppose it may slow down the write
> speed and randomness of test.

I suppose you could do it that way. I wouldn't expect it to make
things less random but yes the speed could be impacted (and there's
the issue of how big that log file will be etc).

-- 
Sitsofe | http://sucs.org/~sits/

答复: How to ensure split verification will generate the same configs as write phase?

[Vaughan]

> ^^^ Here's a problem - you can't randomly write and then verify sequentially via a different run. In your case you need rw=randread.
Yes. It's my fault to paste the original content of the buggy job file. I understand that rw= option should be paired with or without rand- prefix.

> The  randseed is the same across different invocations of fio unless you say otherwise (see randrepeat - http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
> ). So even if you weren't using a verification state file you will re-generate the same pattern if the rest of the job is the same.
Thank you for the explanation. A deep question about the seed comes:) Though randseed is the same across different invocation of fio, is it different for different sections in one job file?

I made another job file as below:
myjob.fio:
	[global] 
	bs=4k    
	direct=1 
	ioengine=sync
	verify=crc32c  
	filename=/dev/sdf
	io_size=10k

	[write-phase]  
	rw=randwrite
	do_verify=0
	write_iolog=iolog

	[verify-phase]
	stonewall
	rw=randread
	do_verify=1

I think these two sections have the same settings except the date direction. So I expect the later one can verify the write-phase. But it failed.	
run it with --debug, and found the offset generated are not the same, so verify abort with error for the first time, as below: 
	random   24382 off rand 259043585
	io       24382 fill_io_u: io_u 0x25ad740: off=71233536/len=4096/ddir=1//dev/sdf
	random   24382 off rand 3179521932
	io       24382 fill_io_u: io_u 0x25ad740: off=874364928/len=4096/ddir=1//dev/sdf
	random   24382 off rand 3621444214
	io       24382 fill_io_u: io_u 0x25ad740: off=995893248/len=4096/ddir=1//dev/sdf
	random   24388 off rand 1094293889
	io       24388 fill_io_u: io_u 0x25ad500: off=300929024/len=4096/ddir=0//dev/sdf
You can see [24382] is write-phase and [24388] is verify-phase, and the first round of them use different seed I suppose.

I search in the source code, and found the following snippet:
	static int setup_random_seeds(struct thread_data *td)
	{
	...
			for (i = 0; i < FIO_RAND_NR_OFFS; i++) {
					td->rand_seeds[i] = seed * td->thread_number + i;
					seed *= 0x9e370001UL;
			}               

Is td->thread_number different for different sections? I thought they both equal to 1 before.

Later, I realize split verify using two sections in one file is not a formal way? Is it better to just write only one write section, and use 'fio --verify-only jobfile' to do the split verification as the updated one below?

myjob.fio updated:
	[global] 
	bs=4k    
	direct=1 
	ioengine=sync
	verify=crc32c  
	filename=/dev/sdf
	io_size=10k

	[write-phase]  
	rw=randwrite
	do_verify=0

Thank you very much again!

Regards,
Vaughan

Re: 答复: How to ensure split verification will generate the same configs as write phase?

[Sitsofe Wheeler]

Hi,

On 28 February 2018 at 08:44, Vaughan <cxt9401@163.com> wrote:
>
>> The  randseed is the same across different invocations of fio unless you say otherwise (see randrepeat - http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-randrepeat
>> ). So even if you weren't using a verification state file you will re-generate the same pattern if the rest of the job is the same.
> Thank you for the explanation. A deep question about the seed comes:) Though randseed is the same across different invocation of fio, is it different for different sections in one job file?

> I made another job file as below:
> myjob.fio:
>         [global]
>         bs=4k
>         direct=1
>         ioengine=sync
>         verify=crc32c
>         filename=/dev/sdf
>         io_size=10k

^^^ Note that io_size may create a potential issue - we're not
necessarily going to complete a total loop (e.g. /dev/sdf is bigger
than 10k) and the second job may not be aware we stopped the first job
part way. In the issue you linked
(https://github.com/axboe/fio/issues/468#issuecomment-332828729 ) the
reporter was asking something similar (although with triggers).

>         [write-phase]
>         rw=randwrite
>         do_verify=0
>         write_iolog=iolog
>
>         [verify-phase]
>         stonewall
>         rw=randread
>         do_verify=1
>
> I think these two sections have the same settings except the date direction. So I expect the later one can verify the write-phase. But it failed.
> run it with --debug, and found the offset generated are not the same, so verify abort with error for the first time, as below:
>         random   24382 off rand 259043585
>         io       24382 fill_io_u: io_u 0x25ad740: off=71233536/len=4096/ddir=1//dev/sdf
>         random   24382 off rand 3179521932
>         io       24382 fill_io_u: io_u 0x25ad740: off=874364928/len=4096/ddir=1//dev/sdf
>         random   24382 off rand 3621444214
>         io       24382 fill_io_u: io_u 0x25ad740: off=995893248/len=4096/ddir=1//dev/sdf
>         random   24388 off rand 1094293889
>         io       24388 fill_io_u: io_u 0x25ad500: off=300929024/len=4096/ddir=0//dev/sdf
> You can see [24382] is write-phase and [24388] is verify-phase, and the first round of them use different seed I suppose.
>
> I search in the source code, and found the following snippet:
>         static int setup_random_seeds(struct thread_data *td)
>         {
>         ...
>                         for (i = 0; i < FIO_RAND_NR_OFFS; i++) {
>                                         td->rand_seeds[i] = seed * td->thread_number + i;
>                                         seed *= 0x9e370001UL;
>                         }
>
> Is td->thread_number different for different sections? I thought they both equal to 1 before.

It looks that way - it seems like thread_number is incremented for each job in
get_new_job() so this would affect the random seeds generated in
different sections of the same job file.

Note that because you're verifying by header and via a separate job I
believe fio checks the header in the current block and if it looks
intact it then uses that to regenerate the data in the rest of the
block. You can get just about away with this when you're writing and
verifying the entirety of the file with fixed block sizes but I think
what you've pointed out will make jobs like:

fio --stonewall --verify=crc32c --size=24k --bsrange=512-4k
--filename=/tmp/fio.tmp --name=write-phase --rw=randwrite
--do_verify=0 --name=verify-phase --rw=randread --do_verify=1

fail because the random ordering of block sizes now depends on the job
number too.

> Later, I realize split verify using two sections in one file is not a formal way? Is it better to just write only one write section, and use 'fio --verify-only jobfile' to do the split verification as the updated one below?

That does look like it will be more robust and will also allow you to
use a state file if you wanted too.

-- 
Sitsofe | http://sucs.org/~sits/

答复: 答复: How to ensure split verification will generate the same configs as write phase?

[Vaughan]

> It looks that way - it seems like thread_number is incremented for each job in
> get_new_job() so this would affect the random seeds generated in different sections of the same job file.
>  I think what you've pointed out will make jobs like:
> 
> fio --stonewall --verify=crc32c --size=24k --bsrange=512-4k --filename=/tmp/fio.tmp --name=write-phase --rw=randwrite --do_verify=0 --name=verify-phase --rw=randread --do_verify=1
> 
> fail because the random ordering of block sizes now depends on the job number too.
Yes, I agree with you. So the right way should be verify it with the original write jobfile and appending "--verify_only".

Can you further explain the usage of  --verify_state_save --verify_backlog --verify_backlog_batch options?

I need to split verify tera-byte size of data. According to the HOWTO of fio to avoid memory exhaustion, verify_backlog should be used. so I do the write job with "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240" in cmd line and do a split verify by appending "--verify_only" with the same write command options. But it seems it just verify the 10240 ios, not the same size as write job.
Only if I verify without "--verify_state_save=1 --verify_backlog=1 --verify_backlog_batch=10240", it verify the whole size I write. I get the conclusion by run the write work and verify work with "write_iolog=str" and compare the iolog, and also "ios=" in "Disk stats" section of both fio output.

Another question about write process with "rw=randwrite" and "bsrange=". If run it with "ioengine=libaio iodepth=16", will this async write get any overlap?

Thanks,
Vaughan