during fio scenario with verify=meta some jobs fails

during fio scenario with verify=meta some jobs fails

[Samuel Shapiro]

Hi All,

I'd recently started using fio for NAS FS tests, and it's really great
tool, as for loading system,
so for finding various protocols bugs. Thank You :)
One of the scenarios, I'm using is "fsx", that is slightly modified to
run with 16 jobs:

; This job file works pretty works similarly to running fsx-linux
; with -r 4096 -w 4096 -Z -N 500000
[global]
verify=meta
verify_dump=1

[fsx-file-job1]
directory=/mnt/FIO-some-mountpoint-to-NAS-01
ioengine=libaio
iodepth=256
rw=randrw
size=256k
bs=4k
norandommap
direct=1
loops=500000

[fsx-file-job2]
directory=/mnt/FIO-some-mountpoint-to-NAS-02
ioengine=libaio
iodepth=256
rw=randrw
size=256k
bs=4k
norandommap
direct=1
loops=500000
.
.
.
[fsx-file-job16]
directory=/mnt/FIO-some-mountpoint-to-NAS-03
etc....

After several minutes if running scenario, jobs starting to fail on
"verify meta", generating
".state" files.
When I tried to run the same scenario with verify=md5 , there were no
more verify failures reported.

So I have couple of questions:
1. What is the possible reason verify=meta can fail?
2. Should I suspect my File System for data corruption or there is
some known fio issue?
3. How can I parse ".state" file so I could compare its data to
original file on storage?

Thank You,
Samuel

Re: during fio scenario with verify=meta some jobs fails

[Sitsofe Wheeler]

On 8 April 2015 at 11:37, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
> Hi All,
>
> I'd recently started using fio for NAS FS tests, and it's really great
> tool, as for loading system,
> so for finding various protocols bugs. Thank You :)
> One of the scenarios, I'm using is "fsx", that is slightly modified to
> run with 16 jobs:
>
> ; This job file works pretty works similarly to running fsx-linux
> ; with -r 4096 -w 4096 -Z -N 500000
> [global]
> verify=meta
> verify_dump=1
>
> [fsx-file-job1]
> directory=/mnt/FIO-some-mountpoint-to-NAS-01
> ioengine=libaio
> iodepth=256
> rw=randrw
> size=256k
> bs=4k
> norandommap

^^^ This allows the same block to be overwritten multiple times. Are
you sure this is what you want?

-- 
Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Samuel Shapiro]

Well i guess it could be a good test for data integrity, so why not? :)

On Fri, Apr 10, 2015 at 9:29 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 8 April 2015 at 11:37, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>> Hi All,
>>
>> I'd recently started using fio for NAS FS tests, and it's really great
>> tool, as for loading system,
>> so for finding various protocols bugs. Thank You :)
>> One of the scenarios, I'm using is "fsx", that is slightly modified to
>> run with 16 jobs:
>>
>> ; This job file works pretty works similarly to running fsx-linux
>> ; with -r 4096 -w 4096 -Z -N 500000
>> [global]
>> verify=meta
>> verify_dump=1
>>
>> [fsx-file-job1]
>> directory=/mnt/FIO-some-mountpoint-to-NAS-01
>> ioengine=libaio
>> iodepth=256
>> rw=randrw
>> size=256k
>> bs=4k
>> norandommap
>
> ^^^ This allows the same block to be overwritten multiple times. Are
> you sure this is what you want?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Sitsofe Wheeler]

On 11 April 2015 at 04:04, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
> Well i guess it could be a good test for data integrity, so why not? :)
>
> On Fri, Apr 10, 2015 at 9:29 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> On 8 April 2015 at 11:37, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>>> Hi All,
>>>
>>> [fsx-file-job1]
>>> directory=/mnt/FIO-some-mountpoint-to-NAS-01
>>> ioengine=libaio
>>> iodepth=256
>>> rw=randrw
>>> size=256k
>>> bs=4k
>>> norandommap
>>
>> ^^^ This allows the same block to be overwritten multiple times. Are
>> you sure this is what you want?

I could be wide of the mark but my concern would be that meta tries to
verify block number (I guess this doesn't need to be saved and can be
calculated by just following a sequence). If I've overwritten a block
how do I verify that the block number of the earlier write was correct
- the block will in fact contain the block number of the last write to
it...

-- 
Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Sitsofe Wheeler]

On 11 April 2015 at 07:10, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 11 April 2015 at 04:04, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>> Well i guess it could be a good test for data integrity, so why not? :)
>>
>> On Fri, Apr 10, 2015 at 9:29 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>> On 8 April 2015 at 11:37, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>> [fsx-file-job1]
>>>> directory=/mnt/FIO-some-mountpoint-to-NAS-01
>>>> ioengine=libaio
>>>> iodepth=256
>>>> rw=randrw
>>>> size=256k
>>>> bs=4k
>>>> norandommap
>>>
>>> ^^^ This allows the same block to be overwritten multiple times. Are
>>> you sure this is what you want?
>
> I could be wide of the mark but my concern would be that meta tries to
> verify block number (I guess this doesn't need to be saved and can be
> calculated by just following a sequence). If I've overwritten a block
> how do I verify that the block number of the earlier write was correct
> - the block will in fact contain the block number of the last write to
> it...

Apologies, where I said block number I should have said "io sequence
number" (because you're using verify=meta):

Assuming I only do my verification at the end of a run, if I've
overwritten a block how do I verify that the io sequence number of the
earlier writes to that block were correct? Wouldn't the block will
contain only the io sequence number of the last write to it?

-- 
Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Sitsofe Wheeler]

On 11 April 2015 at 17:48, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
> Thanks Sitsofe,
> It make sense, if previous block meta isn't really saved....
> So maybe the right question to ask,  in which scenarios should I prefer
> "meta" verification method over "md5" etc...?

Without looking at the source I would guess they will catch different
issues and have different overheads.

For example, if I use md5 I write a header to each "block" that tells
me what the data should be. When it comes to verification time I read
the header and check the body matches the checksum. Reading the body
and calculating the checksum take time and CPU. If the error is that
two blocks of the same size (including their headers) have been
switched I won't catch that because the checksum of the body is still
correct with respect to the header (this is speculation on my part)...

If I use meta my header contains things like timestamp, block number,
io sequence number. I don't need to spend time verifying the body and
unless I used verify_pattern I don't even know what the correct data
in the body would look like if I were to check/checksum it. I can
detect identically sized block swaps because the block number and io
sequence number in the header will be wrong. If a problem is detected
I've got a better chance of working out where the problem header came
from (because more meta data is available to me).

If I'm using same sized blocks and I'm worried about entire block
swaps or I want to do a minimal a check as possible I think you could
use meta. If I'm more worried about sub block sized data being wrong I
would say you are better off with one of the checksum routines
(preferably one that is hardware accelerated). If I'm overwriting my
data within the same pass and I probably can't use meta any more. meta
may give me more information about where bad data came from if the
header is intact and I know how to decode it. However, this is all
speculation on my part!

-- 
Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Samuel Shapiro]

Thanks for the great answer Sitsofe :)
Will stick to md5 verification while testing FS data integrity.

Thanks
Samuel

On Sun, Apr 12, 2015 at 1:06 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 11 April 2015 at 17:48, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>> Thanks Sitsofe,
>> It make sense, if previous block meta isn't really saved....
>> So maybe the right question to ask,  in which scenarios should I prefer
>> "meta" verification method over "md5" etc...?
>
> Without looking at the source I would guess they will catch different
> issues and have different overheads.
>
> For example, if I use md5 I write a header to each "block" that tells
> me what the data should be. When it comes to verification time I read
> the header and check the body matches the checksum. Reading the body
> and calculating the checksum take time and CPU. If the error is that
> two blocks of the same size (including their headers) have been
> switched I won't catch that because the checksum of the body is still
> correct with respect to the header (this is speculation on my part)...
>
> If I use meta my header contains things like timestamp, block number,
> io sequence number. I don't need to spend time verifying the body and
> unless I used verify_pattern I don't even know what the correct data
> in the body would look like if I were to check/checksum it. I can
> detect identically sized block swaps because the block number and io
> sequence number in the header will be wrong. If a problem is detected
> I've got a better chance of working out where the problem header came
> from (because more meta data is available to me).
>
> If I'm using same sized blocks and I'm worried about entire block
> swaps or I want to do a minimal a check as possible I think you could
> use meta. If I'm more worried about sub block sized data being wrong I
> would say you are better off with one of the checksum routines
> (preferably one that is hardware accelerated). If I'm overwriting my
> data within the same pass and I probably can't use meta any more. meta
> may give me more information about where bad data came from if the
> header is intact and I know how to decode it. However, this is all
> speculation on my part!
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Sitsofe Wheeler]

On 12 April 2015 at 11:12, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
> Thanks for the great answer Sitsofe :)
> Will stick to md5 verification while testing FS data integrity.

If fio is in any shape or form CPU bound during writing or
verification I'd probably hint to you that crc32c-intel might be
better tradeoff if fio is running on recent Intel hardware. If you
can't use that (because you aren't on that hardware) you could use
xxhash instead. Both crc32c-intel and xxhash will "detect" less
corruptions than md5 (their checksum is smaller) but they can be
dramatically faster and perhaps allow you to get more runs / more
concurrency in and will still detect common problems.

I failed to answer question 3. of your original mail:

> 3. How can I parse ".state" file so I could compare its data to
> original file on storage?

You only need to use this file if the writing pass is interrupted
before it could be fully completed (so you know when to stop
verifying). I don't think it records the contents of the data being
written but rather how starting parameters and how far along a given
data generation sequence you got (so you have enough to replay it
later)...

Generally speaking and assuming the verification header part of a
block isn't corrupt, verify_dump can be used to make fio dump the
expected and mismatched data into two files for later inspection when
a mismatch is found. If memory serves, non important parts of the
header section may not match between the expected and actual dumps
(e.g. padding between the real values might not match up).

-- 
Sitsofe | http://sucs.org/~sits/

Re: during fio scenario with verify=meta some jobs fails

[Samuel Shapiro]

Thanks Sitsofe,
I'll try crc32c then.
Hopefully Dell servers should support it :)

On Sun, Apr 12, 2015 at 1:51 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 12 April 2015 at 11:12, Samuel Shapiro <samuel.sh79@gmail.com> wrote:
>> Thanks for the great answer Sitsofe :)
>> Will stick to md5 verification while testing FS data integrity.
>
> If fio is in any shape or form CPU bound during writing or
> verification I'd probably hint to you that crc32c-intel might be
> better tradeoff if fio is running on recent Intel hardware. If you
> can't use that (because you aren't on that hardware) you could use
> xxhash instead. Both crc32c-intel and xxhash will "detect" less
> corruptions than md5 (their checksum is smaller) but they can be
> dramatically faster and perhaps allow you to get more runs / more
> concurrency in and will still detect common problems.
>
> I failed to answer question 3. of your original mail:
>
>> 3. How can I parse ".state" file so I could compare its data to
>> original file on storage?
>
> You only need to use this file if the writing pass is interrupted
> before it could be fully completed (so you know when to stop
> verifying). I don't think it records the contents of the data being
> written but rather how starting parameters and how far along a given
> data generation sequence you got (so you have enough to replay it
> later)...
>
> Generally speaking and assuming the verification header part of a
> block isn't corrupt, verify_dump can be used to make fio dump the
> expected and mismatched data into two files for later inspection when
> a mismatch is found. If memory serves, non important parts of the
> header section may not match between the expected and actual dumps
> (e.g. padding between the real values might not match up).
>
> --
> Sitsofe | http://sucs.org/~sits/