FIO -- A few basic questions on Data Integrity.

FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi FIO users,
I am a new user of FIO, and had a few basic questions.
I tried to search the existing QA in archives, and have not found an
exact answer.
Apologies, for the length of the mail and also if this is already
addressed (if so, kindly point me to that archive article).

Referred to:
https://github.com/axboe/fio/issues/163
http://www.spinics.net/lists/fio/msg04104.html
http://www.spinics.net/lists/fio/msg03424.html

We are trying to do Data Integrity checks using FIO, while performing
Sequential & Random Writes/Reads.

1. Basic Write/Read and offline comparison for Data Integrity:
   a. Is it possible to perform Random Writes to a portion of the disk
(using --offset, --size options), and read back from those locations.
   b. Is it possible to force FIO to access the same LBAs during
Writes and Reads, when it is random.
   c. Is there a way to control the "randomness" using any seeds ?
   d. Is there a need to use the "state" files ?

   The intent was to get the data read back to a file, and then
compare against expected.

2. FIO comparison using *verify* options:
    We tried to do an FIO with
     --do_verify=1
     --verify=pattern
     --verify_pattern=TEST_PATTERN
     --rw=randwrite (or write - for sequential)

    In this case, again a few follow-on Questions:
    a. Does FIO perform writes completely, ie based on --size, or --runtime
        and then do the read access to verify.
        What parameters are used (for blk-size, numjobs, qdepth etc.)
during the  Reads operation.

    b. is there a way to get the results of the verify step into an
output file ?
    c. The above questions on control of random accesses still exist.
    d. We tried a run of the above kind, and the FIO run passed, ie
there were no obvious errors reported.
    e. In order to ensure that the verification was correct - we did a
2 step process:
        [similar to one of the reference articles]
        FIO#1 - with Writes (--do_verify=0, --verify_pattern=PAT1)
        FIO#2 - for read/verify (--do_verify=1, --verify_pattern=PAT2)
       and got some errors..
       But, we are not yet sure if that has flagged ALL the locations
in error or not.
       Is there a way to ascertain this ?

    f. Are there any restrictions in the usage of --num_jobs in such a check..

Regards,
- Saju

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

On 17 December 2016 at 10:45, Saju Nair <saju.mad.nair@gmail.com> wrote:
> Hi FIO users,
> I am a new user of FIO, and had a few basic questions.
> I tried to search the existing QA in archives, and have not found an
> exact answer.
> Apologies, for the length of the mail and also if this is already
> addressed (if so, kindly point me to that archive article).
>
> Referred to:
> https://github.com/axboe/fio/issues/163

#163 covers multiple issues (the last of which seems linked to the
'%o' verification pattern). I'm not sure it's a simple starting
example...

> http://www.spinics.net/lists/fio/msg04104.html
> http://www.spinics.net/lists/fio/msg03424.html
>
> We are trying to do Data Integrity checks using FIO, while performing
> Sequential & Random Writes/Reads.
>
> 1. Basic Write/Read and offline comparison for Data Integrity:
>    a. Is it possible to perform Random Writes to a portion of the disk
> (using --offset, --size options), and read back from those locations.

Yes - a simple example is
fio --rw=randwrite --verify=crc32c --filename=examplefile --size=20M
--offset=10M --name=verifytest

Create a file that is 30Mbytes in size but only do randomly ordered
I/O (that writes to each block exactly once) to the last 20MBytes of
the file then afterwards verify that I/O that was written in that last
20Mbytes looks correct according to the header written to that block.
Also see the basic-verify example
(https://github.com/axboe/fio/blob/fio-2.15/examples/basic-verify.fio
).

>    b. Is it possible to force FIO to access the same LBAs during
> Writes and Reads, when it is random.

That will happen with the above even if the writes happened in a random order.

>    c. Is there a way to control the "randomness" using any seeds ?

Yes see randseed= in the HOWTO
(https://github.com/axboe/fio/blob/fio-2.15/HOWTO#L456 ).

>    d. Is there a need to use the "state" files ?

In general if you want to verify I/O that was written by a *different*
job you will probably need to make use of a state file. There are
cases where you can get away without using a state file (e.g. you are
writing the same size block across the entire disk and the writes
include a verification header) but not all: sometimes you need to know
things like "what was the random seed used and how far did the write
job get" in order for the verification to be performed successfully.

>    The intent was to get the data read back to a file, and then
> compare against expected.
>
> 2. FIO comparison using *verify* options:
>     We tried to do an FIO with
>      --do_verify=1
>      --verify=pattern
>      --verify_pattern=TEST_PATTERN
>      --rw=randwrite (or write - for sequential)
>
>     In this case, again a few follow-on Questions:
>     a. Does FIO perform writes completely, ie based on --size, or --runtime
>         and then do the read access to verify.

With the initial example you put above fio will finish doing the
random (or sequential) writes and then do a sequential verify (read)
pass afterwards of the data that was written. However, IF you use
--runtime and the runtime is exceeded while still in the write pass
then no verification will happen at all (because there's no time left
to do it).

>         What parameters are used (for blk-size, numjobs, qdepth etc.)
> during the  Reads operation.

numjobs makes job clones so each clone is distinct job with various
inherited parameters (see numjobs= in the HOWTO). For the other
parameters you explicitly listed (bs, iodepth) the verify component of
the job will use whatever was specified for that job as a whole.

>     b. is there a way to get the results of the verify step into an
> output file ?

I don't understand the question - could you rephrase it with examples?
Do you mean how long it took, if errors were found etc?

>     c. The above questions on control of random accesses still exist.

See the answer in 1b and 1c.

>     d. We tried a run of the above kind, and the FIO run passed, ie
> there were no obvious errors reported.
>     e. In order to ensure that the verification was correct - we did a
> 2 step process:
>         [similar to one of the reference articles]
>         FIO#1 - with Writes (--do_verify=0, --verify_pattern=PAT1)
>         FIO#2 - for read/verify (--do_verify=1, --verify_pattern=PAT2)
>        and got some errors..

This is different to the example I gave above because you have two
separate jobs - one doing the writing and another doing the
reading/verifying. It's hard to say what went wrong without seeing the
exact job/command line you used for FIO#1 and FIO#2. It would help if
you could post the cut down versions of FIO#1 and FIO#2 that still
show the problem.

>        But, we are not yet sure if that has flagged ALL the locations
> in error or not.
>        Is there a way to ascertain this ?

Probably depends on the job parameters as to whether ALL locations can
be flagged. Generally speaking "partially damaged" blocks can't/won't
be verified (see the "intact blocks" note for norandommap in the HOWTO
- https://github.com/axboe/fio/blob/fio-2.15/HOWTO#L1031 ). Unless you
use verify_fatal=1 fio will try and report all locations it can that
have a mismatch. Is that the information you're looking for?

>     f. Are there any restrictions in the usage of --num_jobs in such a check..

It's important to ensure that two (or more) jobs do NOT write
different data (either due to contents and/or block size when using a
header) to overlapping regions of the disk before the verification
happens (this is in general - not just with numjobs). Put another way:
if you have two simultaneous jobs overwriting the same regions of the
disk with different data how can job #1 know what the correct data
should be at verification time when it knows nothing of what the job
#2 did (or when it did it) relative to its own I/O?

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi Sitsofe,
Thanks for your detailed clarifications.
We followed the example FIO, and tried a few experiments.
Sharing them, our observations, and few follow-on qestions.

Our setup is a 2TB disk.
1. Tried with a config file as captured below, and wrote a 4GB file.
    --offset was the default (0 ?).
   We observe that -
   a. Writes happen based on the --size specified,  whereas Reads (and
verify) happen for the entire disk.
   b. Both the Write and Read performance (reported by IOPS) is
significantly lower than our expectations, and observations from
standalone FIO write, read (separate) runs.
   b. In the output (we captured to a log file using --eta,
--eta_newline options) , there are 3 sections that we see
       With w(1) , R(1) - ie both Writes & Reads are happening.
       With V(1), R(1) - where Reads and verify are happening.
       With _, R(1) - not clear ?


   Questions:
   a. Is the above behavior expected ? Why does FIO read the entire
disk - since we only wrote 4GB - is it possible to only read those
locations that were written to - even in a "randwrite" operation.
   b. Does the initial writes get interspersed with Reads ?
   c. What are the _, R(1) section ?
   d. Why does FIO read the entire disk - is there a way to restrict
to a start/end offset ?
   e. How do we know that there is any data miscompare - what is the
typical FIO output for miscompare ?
   f. Performance - the performance #s [BW, IOPS] - are much lower
than the values that we typically see with standalone (ie w/o verify).
Why ?


2. In order to check for the range of access, we then tried to limit
the access using
    --offset = <a value closer to the end of the disk>

    We observe that -
    a. Writes happen for the specified size, whereas Reads (and
verify) happen from the offset till end-of-disk.


Our primary intent are:
- Do Data Integrity checks to
      - write to various locations in the disk (using random writes)
      - be able to reliably read back from those same ("random")
locations, compare and ensure data integrity.
      - would "randseed" help us achieve this ?
      - do we need to use the "state" to be able to perform reads from
ONLY those random locations that we wrote to...

- Data Integrity @ performance:
      - The above data integrity, with accesses (both write & read)
happening at peak performance
      - In this case, we will need to have num_jobs>1 and qdepth>1 as well.
      - Would it help if we do a 2 step process (as asked earlier) for
           FIO writes @ peak write performance (iops, bandwidth)
           FIO read @ peak read performance,
        Compare thru other means.
        Is there a way to read from disk, to a file on the local host machine.

We wanted to understand, if FIO can help us achieve this intent or not.


Regards,
- Saju.


*****************FIO Config File ******************************
[global]
#time_based
group_reporting
direct=1
ioengine=libaio

bs=4k
iodepth=16
numjobs=1

[write-and-verify]
rw=randwrite
size=4G
verify=crc32c

refill_buffers
scramble_buffers=1

[device]
filename=/dev/XXXXX
***********************************************************************


******************* Sample Output *********************************
Jobs: 2 (f=2): [w(1),R(1)] [0.1% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 2 (f=2): [w(1),R(1)] [0.2% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 2 (f=2): [w(1),R(1)] [0.3% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
..
Jobs: 2 (f=2): [V(1),R(1)] [2.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 2 (f=2): [V(1),R(1)] [2.6% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
..
Jobs: 2 (f=2): [V(1),R(1)] [5.4% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 1 (f=1): [_(1),R(1)] [5.4% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 1 (f=1): [_(1),R(1)] [5.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
...
Jobs: 1 (f=1): [_(1),R(1)] [99.9% done] [nnnMB/mKB/0KB /s]
[xxxK/yyyK/0 iops] [eta mm:ss]
Jobs: 1 (f=1): [_(1),R(1)] [100.0% done] [nnnMB/mKB/0KB /s]
[xxxK/yyyK/0 iops] [eta mm:ss]

On Sat, Dec 17, 2016 at 9:54 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 17 December 2016 at 10:45, Saju Nair <saju.mad.nair@gmail.com> wrote:
>> Hi FIO users,
>> I am a new user of FIO, and had a few basic questions.
>> I tried to search the existing QA in archives, and have not found an
>> exact answer.
>> Apologies, for the length of the mail and also if this is already
>> addressed (if so, kindly point me to that archive article).
>>
>> Referred to:
>> https://github.com/axboe/fio/issues/163
>
> #163 covers multiple issues (the last of which seems linked to the
> '%o' verification pattern). I'm not sure it's a simple starting
> example...
>
>> http://www.spinics.net/lists/fio/msg04104.html
>> http://www.spinics.net/lists/fio/msg03424.html
>>
>> We are trying to do Data Integrity checks using FIO, while performing
>> Sequential & Random Writes/Reads.
>>
>> 1. Basic Write/Read and offline comparison for Data Integrity:
>>    a. Is it possible to perform Random Writes to a portion of the disk
>> (using --offset, --size options), and read back from those locations.
>
> Yes - a simple example is
> fio --rw=randwrite --verify=crc32c --filename=examplefile --size=20M
> --offset=10M --name=verifytest
>
> Create a file that is 30Mbytes in size but only do randomly ordered
> I/O (that writes to each block exactly once) to the last 20MBytes of
> the file then afterwards verify that I/O that was written in that last
> 20Mbytes looks correct according to the header written to that block.
> Also see the basic-verify example
> (https://github.com/axboe/fio/blob/fio-2.15/examples/basic-verify.fio
> ).
>
>>    b. Is it possible to force FIO to access the same LBAs during
>> Writes and Reads, when it is random.
>
> That will happen with the above even if the writes happened in a random order.
>
>>    c. Is there a way to control the "randomness" using any seeds ?
>
> Yes see randseed= in the HOWTO
> (https://github.com/axboe/fio/blob/fio-2.15/HOWTO#L456 ).
>
>>    d. Is there a need to use the "state" files ?
>
> In general if you want to verify I/O that was written by a *different*
> job you will probably need to make use of a state file. There are
> cases where you can get away without using a state file (e.g. you are
> writing the same size block across the entire disk and the writes
> include a verification header) but not all: sometimes you need to know
> things like "what was the random seed used and how far did the write
> job get" in order for the verification to be performed successfully.
>
>>    The intent was to get the data read back to a file, and then
>> compare against expected.
>>
>> 2. FIO comparison using *verify* options:
>>     We tried to do an FIO with
>>      --do_verify=1
>>      --verify=pattern
>>      --verify_pattern=TEST_PATTERN
>>      --rw=randwrite (or write - for sequential)
>>
>>     In this case, again a few follow-on Questions:
>>     a. Does FIO perform writes completely, ie based on --size, or --runtime
>>         and then do the read access to verify.
>
> With the initial example you put above fio will finish doing the
> random (or sequential) writes and then do a sequential verify (read)
> pass afterwards of the data that was written. However, IF you use
> --runtime and the runtime is exceeded while still in the write pass
> then no verification will happen at all (because there's no time left
> to do it).
>
>>         What parameters are used (for blk-size, numjobs, qdepth etc.)
>> during the  Reads operation.
>
> numjobs makes job clones so each clone is distinct job with various
> inherited parameters (see numjobs= in the HOWTO). For the other
> parameters you explicitly listed (bs, iodepth) the verify component of
> the job will use whatever was specified for that job as a whole.
>
>>     b. is there a way to get the results of the verify step into an
>> output file ?
>
> I don't understand the question - could you rephrase it with examples?
> Do you mean how long it took, if errors were found etc?
>
>>     c. The above questions on control of random accesses still exist.
>
> See the answer in 1b and 1c.
>
>>     d. We tried a run of the above kind, and the FIO run passed, ie
>> there were no obvious errors reported.
>>     e. In order to ensure that the verification was correct - we did a
>> 2 step process:
>>         [similar to one of the reference articles]
>>         FIO#1 - with Writes (--do_verify=0, --verify_pattern=PAT1)
>>         FIO#2 - for read/verify (--do_verify=1, --verify_pattern=PAT2)
>>        and got some errors..
>
> This is different to the example I gave above because you have two
> separate jobs - one doing the writing and another doing the
> reading/verifying. It's hard to say what went wrong without seeing the
> exact job/command line you used for FIO#1 and FIO#2. It would help if
> you could post the cut down versions of FIO#1 and FIO#2 that still
> show the problem.
>
>>        But, we are not yet sure if that has flagged ALL the locations
>> in error or not.
>>        Is there a way to ascertain this ?
>
> Probably depends on the job parameters as to whether ALL locations can
> be flagged. Generally speaking "partially damaged" blocks can't/won't
> be verified (see the "intact blocks" note for norandommap in the HOWTO
> - https://github.com/axboe/fio/blob/fio-2.15/HOWTO#L1031 ). Unless you
> use verify_fatal=1 fio will try and report all locations it can that
> have a mismatch. Is that the information you're looking for?
>
>>     f. Are there any restrictions in the usage of --num_jobs in such a check..
>
> It's important to ensure that two (or more) jobs do NOT write
> different data (either due to contents and/or block size when using a
> header) to overlapping regions of the disk before the verification
> happens (this is in general - not just with numjobs). Put another way:
> if you have two simultaneous jobs overwriting the same regions of the
> disk with different data how can job #1 know what the correct data
> should be at verification time when it knows nothing of what the job
> #2 did (or when it did it) relative to its own I/O?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

Before I forget - you using the latest HEAD build of fio? If not can
you update to it (include the version you updated to) and rerun any
tests?

On 19 December 2016 at 09:49, Saju Nair <saju.mad.nair@gmail.com> wrote:
>
> Thanks for your detailed clarifications.
> We followed the example FIO, and tried a few experiments.
> Sharing them, our observations, and few follow-on qestions.
>
> Our setup is a 2TB disk.
> 1. Tried with a config file as captured below, and wrote a 4GB file.
>     --offset was the default (0 ?).
>    We observe that -
>    a. Writes happen based on the --size specified,  whereas Reads (and
> verify) happen for the entire disk.

Your job file looks a bit suspect (perhaps you were thinking of
vdbench configuration formats when you made the jobfile):
[global]
...

[write-and-verify]
...
[device]
filename=/dev/XXXXX

Only "global" is special as a jobname (the bit in square brackets) -
in the above you have one global section and two jobs: one job named
"write-and-verify" and the other job called "device". Bear in mind
that the non-global options in the "write-and-verify" job don't affect
the "device" job and the options in the "device" job do not affect the
"write-and-verify" job. I suspect you wanted filename=/dev/XXXXX in
the global section and no [device] job at all. See "4.0 Job file
format" in the HOWTO for details.

>    b. Both the Write and Read performance (reported by IOPS) is
> significantly lower than our expectations, and observations from
> standalone FIO write, read (separate) runs.

See above. You'll have to ensure your jobfile is correct otherwise you
might find you have been making a file called "write-and-verify" in
the current working directory. Further, the "write-and-verify" and
"device" jobs will have been running simultaneously. You might also
benefit from the allow_file_create=0 option (since you appear to be
working with block devices) as that can warn you when you are trying
to access things that don't already exist (and might otherwise be
created).

>    b. In the output (we captured to a log file using --eta,
> --eta_newline options) , there are 3 sections that we see
>        With w(1) , R(1) - ie both Writes & Reads are happening.
>        With V(1), R(1) - where Reads and verify are happening.
>        With _, R(1) - not clear ?

_ appears when a job has finished running and has been reaped - see
"6.0 Interpreting the output" in the HOWTO.

I'm going to skip over some of the following for now but I do
recommend checking your job file and (re)reading the HOWTO to see if
some of them are already answered there.

>    Questions:
>    a. Is the above behavior expected ? Why does FIO read the entire
> disk - since we only wrote 4GB - is it possible to only read those
> locations that were written to - even in a "randwrite" operation.
>    b. Does the initial writes get interspersed with Reads ?
>    c. What are the _, R(1) section ?
>    d. Why does FIO read the entire disk - is there a way to restrict
> to a start/end offset ?
>    e. How do we know that there is any data miscompare - what is the
> typical FIO output for miscompare ?
>    f. Performance - the performance #s [BW, IOPS] - are much lower
> than the values that we typically see with standalone (ie w/o verify).
> Why ?
>
>
> 2. In order to check for the range of access, we then tried to limit
> the access using
>     --offset = <a value closer to the end of the disk>
>
>     We observe that -
>     a. Writes happen for the specified size, whereas Reads (and
> verify) happen from the offset till end-of-disk.

See above with respect to the strange looking job file.

> Our primary intent are:
> - Do Data Integrity checks to
>       - write to various locations in the disk (using random writes)
>       - be able to reliably read back from those same ("random")
> locations, compare and ensure data integrity.
>       - would "randseed" help us achieve this ?

randseed can help you do that but it might be overkill for your
particular example.

>       - do we need to use the "state" to be able to perform reads from
> ONLY those random locations that we wrote to...

Not necessarily, e.g. if the write and verify are in the same "fio
job" you don't need state files.

> - Data Integrity @ performance:
>       - The above data integrity, with accesses (both write & read)
> happening at peak performance
>       - In this case, we will need to have num_jobs>1 and qdepth>1 as well.

Using higher iodepths I follow but I think increasing numjobs against
the same single disk is the wrong way when doing verification for
reasons I stated in previous emails.

>       - Would it help if we do a 2 step process (as asked earlier) for
>            FIO writes @ peak write performance (iops, bandwidth)
>            FIO read @ peak read performance,
>         Compare thru other means.

Splitting the writing and verifying into two is always an option but I
doubt it will dramatically speed things up.

>         Is there a way to read from disk, to a file on the local host machine.

I don't understand this last question - read what from the disk and
put what into the file?

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi Sitsofe,
Thanks for your prompt & detailed response, and thanks for pointing us
to the mistakes in the FIO config file.

We tried with the sample [write-and-verify] in the link you specified..

[write-and-verify]
rw=randwrite
bs=4k
direct=1
ioengine=libaio
iodepth=16
size=4g                  <-- added this line
verify=crc32c
filename=/dev/XXXX

Unfortunately, we get an error from FIO (both 2.12 and 2.15- latest).
fio-2.15
Starting 1 process
Jobs: 1 (f=1)^MJobs: 1 (f=1)
Jobs: 1 (f=1): [w(1)] [30.0% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 1 (f=1): [w(1)] [45.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
Jobs: 1 (f=1): [w(1)] [54.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
iops] [eta mm:ss]
fio: pid=9067, err=84/file:io_u.c:1979, func=io_u_queued_complete,
error=Invalid or incomplete multibyte or wide character

From a search, this error has been faced by folks before, but, looks
like it got fixed with "numjobs=1".

We are already using numjobs=1.
Are there any pointers on how to get around this issue.
We hope that with the above fixed, we will be able to run regular data
integrity checks.



Now, onto the data-integrity checks at performance...
Our device under test (DUT) is an SSD disk.
Our standalone write and read performance is achieved at a num_jobs >
1, and qdepth > 1.
This is validated in standalone "randwrite" and "randread" FIO runs.

We wanted to develop a strategy to be able to perform data-integrity
checks @ performance.
Wanted to check if it is feasible to do this check using FIO.
Approach#1:
 Extend the -do_verify approach, and do a write followed by verify in
a single FIO run.
 But, as you clarified - this will not be feasible with numjobs > 1.

Approach#2:
FIO job#1 - do FIO writes, with settings for full performance
FIO job#2 - wait for job#1 and then, do FIO reads at performance.

Is there any inbuilt way to do an at-speed comparison in FIO.
If not, we wanted to see if we can use FIO to read from our DUT, to
the host's memory or any other storage disk, and then do a simple
application that compares the data.


Regards,
- Saju

On Mon, Dec 19, 2016 at 4:30 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> Before I forget - you using the latest HEAD build of fio? If not can
> you update to it (include the version you updated to) and rerun any
> tests?
>
> On 19 December 2016 at 09:49, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>
>> Thanks for your detailed clarifications.
>> We followed the example FIO, and tried a few experiments.
>> Sharing them, our observations, and few follow-on qestions.
>>
>> Our setup is a 2TB disk.
>> 1. Tried with a config file as captured below, and wrote a 4GB file.
>>     --offset was the default (0 ?).
>>    We observe that -
>>    a. Writes happen based on the --size specified,  whereas Reads (and
>> verify) happen for the entire disk.
>
> Your job file looks a bit suspect (perhaps you were thinking of
> vdbench configuration formats when you made the jobfile):
> [global]
> ...
>
> [write-and-verify]
> ...
> [device]
> filename=/dev/XXXXX
>
> Only "global" is special as a jobname (the bit in square brackets) -
> in the above you have one global section and two jobs: one job named
> "write-and-verify" and the other job called "device". Bear in mind
> that the non-global options in the "write-and-verify" job don't affect
> the "device" job and the options in the "device" job do not affect the
> "write-and-verify" job. I suspect you wanted filename=/dev/XXXXX in
> the global section and no [device] job at all. See "4.0 Job file
> format" in the HOWTO for details.
>
>>    b. Both the Write and Read performance (reported by IOPS) is
>> significantly lower than our expectations, and observations from
>> standalone FIO write, read (separate) runs.
>
> See above. You'll have to ensure your jobfile is correct otherwise you
> might find you have been making a file called "write-and-verify" in
> the current working directory. Further, the "write-and-verify" and
> "device" jobs will have been running simultaneously. You might also
> benefit from the allow_file_create=0 option (since you appear to be
> working with block devices) as that can warn you when you are trying
> to access things that don't already exist (and might otherwise be
> created).
>
>>    b. In the output (we captured to a log file using --eta,
>> --eta_newline options) , there are 3 sections that we see
>>        With w(1) , R(1) - ie both Writes & Reads are happening.
>>        With V(1), R(1) - where Reads and verify are happening.
>>        With _, R(1) - not clear ?
>
> _ appears when a job has finished running and has been reaped - see
> "6.0 Interpreting the output" in the HOWTO.
>
> I'm going to skip over some of the following for now but I do
> recommend checking your job file and (re)reading the HOWTO to see if
> some of them are already answered there.
>
>>    Questions:
>>    a. Is the above behavior expected ? Why does FIO read the entire
>> disk - since we only wrote 4GB - is it possible to only read those
>> locations that were written to - even in a "randwrite" operation.
>>    b. Does the initial writes get interspersed with Reads ?
>>    c. What are the _, R(1) section ?
>>    d. Why does FIO read the entire disk - is there a way to restrict
>> to a start/end offset ?
>>    e. How do we know that there is any data miscompare - what is the
>> typical FIO output for miscompare ?
>>    f. Performance - the performance #s [BW, IOPS] - are much lower
>> than the values that we typically see with standalone (ie w/o verify).
>> Why ?
>>
>>
>> 2. In order to check for the range of access, we then tried to limit
>> the access using
>>     --offset = <a value closer to the end of the disk>
>>
>>     We observe that -
>>     a. Writes happen for the specified size, whereas Reads (and
>> verify) happen from the offset till end-of-disk.
>
> See above with respect to the strange looking job file.
>
>> Our primary intent are:
>> - Do Data Integrity checks to
>>       - write to various locations in the disk (using random writes)
>>       - be able to reliably read back from those same ("random")
>> locations, compare and ensure data integrity.
>>       - would "randseed" help us achieve this ?
>
> randseed can help you do that but it might be overkill for your
> particular example.
>
>>       - do we need to use the "state" to be able to perform reads from
>> ONLY those random locations that we wrote to...
>
> Not necessarily, e.g. if the write and verify are in the same "fio
> job" you don't need state files.
>
>> - Data Integrity @ performance:
>>       - The above data integrity, with accesses (both write & read)
>> happening at peak performance
>>       - In this case, we will need to have num_jobs>1 and qdepth>1 as well.
>
> Using higher iodepths I follow but I think increasing numjobs against
> the same single disk is the wrong way when doing verification for
> reasons I stated in previous emails.
>
>>       - Would it help if we do a 2 step process (as asked earlier) for
>>            FIO writes @ peak write performance (iops, bandwidth)
>>            FIO read @ peak read performance,
>>         Compare thru other means.
>
> Splitting the writing and verifying into two is always an option but I
> doubt it will dramatically speed things up.
>
>>         Is there a way to read from disk, to a file on the local host machine.
>
> I don't understand this last question - read what from the disk and
> put what into the file?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

On 19 December 2016 at 12:29, Saju Nair <saju.mad.nair@gmail.com> wrote:
>
> We tried with the sample [write-and-verify] in the link you specified..
>
> [write-and-verify]
> rw=randwrite
> bs=4k
> direct=1
> ioengine=libaio
> iodepth=16
> size=4g                  <-- added this line
> verify=crc32c
> filename=/dev/XXXX
>
> Unfortunately, we get an error from FIO (both 2.12 and 2.15- latest).
> fio-2.15
> Starting 1 process
> Jobs: 1 (f=1)^MJobs: 1 (f=1)
> Jobs: 1 (f=1): [w(1)] [30.0% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
> iops] [eta mm:ss]
> Jobs: 1 (f=1): [w(1)] [45.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
> iops] [eta mm:ss]
> Jobs: 1 (f=1): [w(1)] [54.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
> iops] [eta mm:ss]
> fio: pid=9067, err=84/file:io_u.c:1979, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character
>
> From a search, this error has been faced by folks before, but, looks
> like it got fixed with "numjobs=1".
>
> We are already using numjobs=1.
> Are there any pointers on how to get around this issue.
> We hope that with the above fixed, we will be able to run regular data
> integrity checks.

Assuming the fio jobfile you posted above was complete (i.e. no global
section no other jobs etc) it looks like what you've hit is the error
message you get when a bad verification header is found during the
verify phase (i.e. there's been a mismatch between the expected and
read back data). fio normally goes on to print a message about
"verify: bad header [...]". Did you get that too (if so what did it
say) and do you get the same error on other disks that you know are
good (i.e. are you sure the disk isn't suffering a problem)?

> Now, onto the data-integrity checks at performance...
> Our device under test (DUT) is an SSD disk.
> Our standalone write and read performance is achieved at a num_jobs >
> 1, and qdepth > 1.
> This is validated in standalone "randwrite" and "randread" FIO runs.

Ah I see. I will note that highest possible performance is a bit at
odds with proving data integrity though because if I only care about
performance I can write any old junk and just throw the data I read
away (I've never known benchmark claims to be limited to verified data
runs)...

> We wanted to develop a strategy to be able to perform data-integrity
> checks @ performance.
> Wanted to check if it is feasible to do this check using FIO.
> Approach#1:
>  Extend the -do_verify approach, and do a write followed by verify in
> a single FIO run.
>  But, as you clarified - this will not be feasible with numjobs > 1.
>
> Approach#2:
> FIO job#1 - do FIO writes, with settings for full performance
> FIO job#2 - wait for job#1 and then, do FIO reads at performance.

A few ideas spring to mind:
1. Try the usual methods that speed up a "normal" single fio job - if
a single process/thread submits as much I/O as multiple ones it isn't
going to look different from the disk's perspective (assuming that it
sheer amount of simultaneous I/O triggering a problem). Things like
reducing calls that cost CPU, doing things in bigger batches to
amortize the cost etc should also help verification speed (but I'll
leave you to find those elsewhere). You can also look at the HOWTO
information related to verify_async= option to try and allow more
parallelism.
2. Split the disk into different regions and write/verify each region
separately from any other region. See offset_increment= in the HOWTO
for something that might help achieve this if you use numjobs. More
fiddly but a good exercise in learning how to create fio job files.

> Is there any inbuilt way to do an at-speed comparison in FIO.

Personally I'd start with 1. from above and after I got that going I'd
give 2. a go. If 1. can be made to get similar disk I/O numbers to
using multiple jobs then you might even stop there.

> If not, we wanted to see if we can use FIO to read from our DUT, to
> the host's memory or any other storage disk, and then do a simple
> application that compares the data.

fio isn't a copying tool so it won't "move" data for you (and doing so
would slow things down). However, if you somehow copied the contents
into a file fio could verify against the file. The problem you'll then
have to solve is finding a tool that copies the data faster than fio
does its verifying reads...

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi Sitsofe,
Thanks.
On the possible data-verify error,
1. Yes, the config file is what I used.
2. Did not get the verify : bad header info. but got a line as below.
write-and-verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1979,
func=io_u_queued_complete, error=Invalid or incomplete multibyte or
wide character): pid=9067: Mon Dec 19 03:47:40 2016
    Wish that the response was more intuitive!.
3. Below message shows

Run status group 0 (all jobs):
   READ: io=264KB, aggrb=XXXXKB/s, minb=XXXXKB/s, maxb=XXXXKB/s,
mint=tmsec, maxt=tmsec
  WRITE: io=4096.0MB, aggrb=YYYYYKB/s, minb=YYYYYKB/s, maxb=YYYYYKB/s,
mint=t2msec, maxt=t2msec

Appears to indicate that 4GB had been written to, but, reads happened
only upto 264KB, by when we possibly got  an error ?
Is there a way to get additional info - like what was expected, and
what was actually written, which sector (address) is in error ?
Can we set the
--continue_on_error=verify, to get all the errors ?

-------------------------------------

On the Data Integrity @ performance-
our thought was that for us to ensure that the max performance also is
backed up by having data integrity to pass..
Let me think through the suggestions that you have provided for the same..
Many thanks, really appreciate your valuable support & suggestions.

Regards,
- Saju



On Mon, Dec 19, 2016 at 7:32 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 19 December 2016 at 12:29, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>
>> We tried with the sample [write-and-verify] in the link you specified..
>>
>> [write-and-verify]
>> rw=randwrite
>> bs=4k
>> direct=1
>> ioengine=libaio
>> iodepth=16
>> size=4g                  <-- added this line
>> verify=crc32c
>> filename=/dev/XXXX
>>
>> Unfortunately, we get an error from FIO (both 2.12 and 2.15- latest).
>> fio-2.15
>> Starting 1 process
>> Jobs: 1 (f=1)^MJobs: 1 (f=1)
>> Jobs: 1 (f=1): [w(1)] [30.0% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
>> iops] [eta mm:ss]
>> Jobs: 1 (f=1): [w(1)] [45.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
>> iops] [eta mm:ss]
>> Jobs: 1 (f=1): [w(1)] [54.5% done] [nnnMB/mKB/0KB /s] [xxxK/yyyK/0
>> iops] [eta mm:ss]
>> fio: pid=9067, err=84/file:io_u.c:1979, func=io_u_queued_complete,
>> error=Invalid or incomplete multibyte or wide character
>>
>> From a search, this error has been faced by folks before, but, looks
>> like it got fixed with "numjobs=1".
>>
>> We are already using numjobs=1.
>> Are there any pointers on how to get around this issue.
>> We hope that with the above fixed, we will be able to run regular data
>> integrity checks.
>
> Assuming the fio jobfile you posted above was complete (i.e. no global
> section no other jobs etc) it looks like what you've hit is the error
> message you get when a bad verification header is found during the
> verify phase (i.e. there's been a mismatch between the expected and
> read back data). fio normally goes on to print a message about
> "verify: bad header [...]". Did you get that too (if so what did it
> say) and do you get the same error on other disks that you know are
> good (i.e. are you sure the disk isn't suffering a problem)?
>
>> Now, onto the data-integrity checks at performance...
>> Our device under test (DUT) is an SSD disk.
>> Our standalone write and read performance is achieved at a num_jobs >
>> 1, and qdepth > 1.
>> This is validated in standalone "randwrite" and "randread" FIO runs.
>
> Ah I see. I will note that highest possible performance is a bit at
> odds with proving data integrity though because if I only care about
> performance I can write any old junk and just throw the data I read
> away (I've never known benchmark claims to be limited to verified data
> runs)...
>
>> We wanted to develop a strategy to be able to perform data-integrity
>> checks @ performance.
>> Wanted to check if it is feasible to do this check using FIO.
>> Approach#1:
>>  Extend the -do_verify approach, and do a write followed by verify in
>> a single FIO run.
>>  But, as you clarified - this will not be feasible with numjobs > 1.
>>
>> Approach#2:
>> FIO job#1 - do FIO writes, with settings for full performance
>> FIO job#2 - wait for job#1 and then, do FIO reads at performance.
>
> A few ideas spring to mind:
> 1. Try the usual methods that speed up a "normal" single fio job - if
> a single process/thread submits as much I/O as multiple ones it isn't
> going to look different from the disk's perspective (assuming that it
> sheer amount of simultaneous I/O triggering a problem). Things like
> reducing calls that cost CPU, doing things in bigger batches to
> amortize the cost etc should also help verification speed (but I'll
> leave you to find those elsewhere). You can also look at the HOWTO
> information related to verify_async= option to try and allow more
> parallelism.
> 2. Split the disk into different regions and write/verify each region
> separately from any other region. See offset_increment= in the HOWTO
> for something that might help achieve this if you use numjobs. More
> fiddly but a good exercise in learning how to create fio job files.
>
>> Is there any inbuilt way to do an at-speed comparison in FIO.
>
> Personally I'd start with 1. from above and after I got that going I'd
> give 2. a go. If 1. can be made to get similar disk I/O numbers to
> using multiple jobs then you might even stop there.
>
>> If not, we wanted to see if we can use FIO to read from our DUT, to
>> the host's memory or any other storage disk, and then do a simple
>> application that compares the data.
>
> fio isn't a copying tool so it won't "move" data for you (and doing so
> would slow things down). However, if you somehow copied the contents
> into a file fio could verify against the file. The problem you'll then
> have to solve is finding a tool that copies the data faster than fio
> does its verifying reads...
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

On 19 December 2016 at 17:15, Saju Nair <saju.mad.nair@gmail.com> wrote:
>
> On the possible data-verify error,
> 1. Yes, the config file is what I used.
> 2. Did not get the verify : bad header info. but got a line as below.
> write-and-verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1979,
> func=io_u_queued_complete, error=Invalid or incomplete multibyte or
> wide character): pid=9067: Mon Dec 19 03:47:40 2016
>     Wish that the response was more intuitive!.

Yeah.. the error message is a bit strange (see
http://www.spinics.net/lists/fio/msg04977.html for why),

> 3. Below message shows
>
> Run status group 0 (all jobs):
>    READ: io=264KB, aggrb=XXXXKB/s, minb=XXXXKB/s, maxb=XXXXKB/s,
> mint=tmsec, maxt=tmsec
>   WRITE: io=4096.0MB, aggrb=YYYYYKB/s, minb=YYYYYKB/s, maxb=YYYYYKB/s,
> mint=t2msec, maxt=t2msec
>
> Appears to indicate that 4GB had been written to, but, reads happened
> only upto 264KB, by when we possibly got  an error ?
> Is there a way to get additional info - like what was expected, and
> what was actually written, which sector (address) is in error ?

Normally whenever fio has hit mismatches information about the problem
offset is printed to stderr. I've just run a quick check on a device
that always returns zeros and here's what I got:

write-and-verify: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
ioengine=libaio, iodepth=16
fio-2.15-25-gea8d
Starting 1 process
verify: bad magic header 0, wanted acca at file /dev/mapper/fake
offset 259043328, length 4096
verify: bad magic header 0, wanted acca at file /dev/mapper/fake
offset 3179520000, length 4096
fio: pid=15333, err=84/file:io_u.c:1983, func=io_u_queued_complete,
error=Invalid or incomplete multibyte or wide character

write-and-verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1983,
func=io_u_queued_complete, error=Invalid or incomplete multibyte or
wide character): pid=15333: Mon Dec 19 19:10:01 2016

Are you doing something like redirecting stdout to a file but not
doing anything with stderr? It would help if you include the command
line you are using to run fio in your reply.

See the HOWTO (I say that a lot right?) for information on the
verify_dump= job option which will cause the contents of failing
blocks to be recorded within files within the current working
directory. If the verification header is correct but the contents is
wrong you will also get a dump of the expected data in a separate
file. You might want to try this out before you enable the following
option because if there are a lot of distinct blocks that mismatch a
lot of files will be generated...

> Can we set the
> --continue_on_error=verify, to get all the errors ?

Yes - I was mistaken earlier about all errors being printed by default
with verify_fatal=0 so perhaps the HOWTO needs to be updated.
continue_on_error is actually a job option so it can go into the job
file if you prefer.

> -------------------------------------
>
> On the Data Integrity @ performance-
> our thought was that for us to ensure that the max performance also is
> backed up by having data integrity to pass..
> Let me think through the suggestions that you have provided for the same..
> Many thanks, really appreciate your valuable support & suggestions.

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi,
Thanks for your clarifications.
We ran with a --continue_on_error=verify,
to let the FIO complete the full compare..

We tried to do a sequential write and compare, using the FIO config
file as below, and to bring in the complexity of "random" as a 2nd
step.
[write-and-verify]
rw=write
bs=4k
direct=1
ioengine=libaio
iodepth=16
size=2m
verify=pattern
verify_pattern=0x33333333
continue_on_error=verify
verify_dump=1
filename=/dev/XXXX

FIO reports errors and we see files of the following names created:
<filename>.<num>.received
<filename>.<num>.expected

Wanted help in interpreting the result.

We wrote 2MB worth of data, with blocksize = 4K.
So, ideally is it expected to do 2MB/4KB = 512 IO operations

1) The received/expected files:
Are they for each 4K offset that failed the comparison ?
Is the <num> to be interpreted as the (num/bs)-th block that failed ?
   For ex: if the num=438272, and bs=4096 => 107th block failed ?

It would be useful to know this information - so that we can debug further,
FYI, if we try a "dd" command and check the disk, based on the above
calculation - the data is proper (as expected).

2) What were the locations that were written to..
Tried fio-verify-state <.state_file>, and get the below:
Version:        0x3
Size:           408
CRC:            0x70ca464a
Thread:         0
Name:           write-and-verify
Completions:    16
Depth:          16
Number IOs:     512
Index:          0
Completions:
        (file= 0) 2031616
        (file= 0) 2035712
        (file= 0) 2039808
        (file= 0) 2043904
        (file= 0) 2048000
        (file= 0) 2052096
        (file= 0) 2056192
        (file= 0) 2060288
        (file= 0) 2064384
        (file= 0) 2068480
        (file= 0) 2072576
        (file= 0) 2076672
        (file= 0) 2080768
        (file= 0) 2084864
        (file= 0) 2088960
        (file= 0) 2093056

How do we interpret the above content to understand the locations of Writes.

Thanks,
- Saju.








On Tue, Dec 20, 2016 at 2:04 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 19 December 2016 at 17:15, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>
>> On the possible data-verify error,
>> 1. Yes, the config file is what I used.
>> 2. Did not get the verify : bad header info. but got a line as below.
>> write-and-verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1979,
>> func=io_u_queued_complete, error=Invalid or incomplete multibyte or
>> wide character): pid=9067: Mon Dec 19 03:47:40 2016
>>     Wish that the response was more intuitive!.
>
> Yeah.. the error message is a bit strange (see
> http://www.spinics.net/lists/fio/msg04977.html for why),
>
>> 3. Below message shows
>>
>> Run status group 0 (all jobs):
>>    READ: io=264KB, aggrb=XXXXKB/s, minb=XXXXKB/s, maxb=XXXXKB/s,
>> mint=tmsec, maxt=tmsec
>>   WRITE: io=4096.0MB, aggrb=YYYYYKB/s, minb=YYYYYKB/s, maxb=YYYYYKB/s,
>> mint=t2msec, maxt=t2msec
>>
>> Appears to indicate that 4GB had been written to, but, reads happened
>> only upto 264KB, by when we possibly got  an error ?
>> Is there a way to get additional info - like what was expected, and
>> what was actually written, which sector (address) is in error ?
>
> Normally whenever fio has hit mismatches information about the problem
> offset is printed to stderr. I've just run a quick check on a device
> that always returns zeros and here's what I got:
>
> write-and-verify: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
> ioengine=libaio, iodepth=16
> fio-2.15-25-gea8d
> Starting 1 process
> verify: bad magic header 0, wanted acca at file /dev/mapper/fake
> offset 259043328, length 4096
> verify: bad magic header 0, wanted acca at file /dev/mapper/fake
> offset 3179520000, length 4096
> fio: pid=15333, err=84/file:io_u.c:1983, func=io_u_queued_complete,
> error=Invalid or incomplete multibyte or wide character
>
> write-and-verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1983,
> func=io_u_queued_complete, error=Invalid or incomplete multibyte or
> wide character): pid=15333: Mon Dec 19 19:10:01 2016
>
> Are you doing something like redirecting stdout to a file but not
> doing anything with stderr? It would help if you include the command
> line you are using to run fio in your reply.
>
> See the HOWTO (I say that a lot right?) for information on the
> verify_dump= job option which will cause the contents of failing
> blocks to be recorded within files within the current working
> directory. If the verification header is correct but the contents is
> wrong you will also get a dump of the expected data in a separate
> file. You might want to try this out before you enable the following
> option because if there are a lot of distinct blocks that mismatch a
> lot of files will be generated...
>
>> Can we set the
>> --continue_on_error=verify, to get all the errors ?
>
> Yes - I was mistaken earlier about all errors being printed by default
> with verify_fatal=0 so perhaps the HOWTO needs to be updated.
> continue_on_error is actually a job option so it can go into the job
> file if you prefer.
>
>> -------------------------------------
>>
>> On the Data Integrity @ performance-
>> our thought was that for us to ensure that the max performance also is
>> backed up by having data integrity to pass..
>> Let me think through the suggestions that you have provided for the same..
>> Many thanks, really appreciate your valuable support & suggestions.
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

On 20 December 2016 at 12:26, Saju Nair <saju.mad.nair@gmail.com> wrote:
>
> Thanks for your clarifications.
> We ran with a --continue_on_error=verify,
> to let the FIO complete the full compare..
>
> We tried to do a sequential write and compare, using the FIO config
> file as below, and to bring in the complexity of "random" as a 2nd
> step.
> [write-and-verify]
> rw=write
> bs=4k
> direct=1
> ioengine=libaio
> iodepth=16
> size=2m
> verify=pattern
> verify_pattern=0x33333333
> continue_on_error=verify
> verify_dump=1
> filename=/dev/XXXX
>
> FIO reports errors and we see files of the following names created:
> <filename>.<num>.received
> <filename>.<num>.expected
>
> Wanted help in interpreting the result.
>
> We wrote 2MB worth of data, with blocksize = 4K.
> So, ideally is it expected to do 2MB/4KB = 512 IO operations
>
> 1) The received/expected files:
> Are they for each 4K offset that failed the comparison ?

I bet you can deduce this from the size and names of the files...

> Is the <num> to be interpreted as the (num/bs)-th block that failed ?
>    For ex: if the num=438272, and bs=4096 => 107th block failed ?
>
> It would be useful to know this information - so that we can debug further,
> FYI, if we try a "dd" command and check the disk, based on the above
> calculation - the data is proper (as expected).

You never answered my question about what you are doing with stderr.
I'll repeat it here:

>> Are you doing something like redirecting stdout to a file but not
>> doing anything with stderr? It would help if you include the command
>> line you are using to run fio in your reply.

Can you answer this question and post the full command line you ran
fio with? I think it might have relevance to your current question.

> 2) What were the locations that were written to..
> Tried fio-verify-state <.state_file>, and get the below:
> Version:        0x3
> Size:           408
> CRC:            0x70ca464a
> Thread:         0
> Name:           write-and-verify
> Completions:    16
> Depth:          16
> Number IOs:     512
> Index:          0
> Completions:
>         (file= 0) 2031616
>         (file= 0) 2035712
>         (file= 0) 2039808
>         (file= 0) 2043904
>         (file= 0) 2048000
>         (file= 0) 2052096
>         (file= 0) 2056192
>         (file= 0) 2060288
>         (file= 0) 2064384
>         (file= 0) 2068480
>         (file= 0) 2072576
>         (file= 0) 2076672
>         (file= 0) 2080768
>         (file= 0) 2084864
>         (file= 0) 2088960
>         (file= 0) 2093056
>
> How do we interpret the above content to understand the locations of Writes.

Perhaps fio tracks how far through the sequence it got rather than
individual locations written (this would be necessary to handle things
like loops=)? I personally don't know the answer to this but you can
always take a look at the source code.

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi,
Thanks. Please find the stderr messages snippet.
We get the offsets where FIO observed failures, during the "verify" phase.
The nvme**.<num>.received - is reported to contain 0.

However, when we try a "dd" to access the location (offset) specified
- we do see proper data.
It might be an issue in our DUT, and we will investigate further.

Thanks for your help.

Regards,
- Saju.

*************************************************************
/root/demo/fio-2.12/bin/fio test.fio -eta=always --eta-newline=1 1 tee
fiol.log•rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
ioengine=libaio, iodepth=16
•rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
ioengine=libaio, iodepth=16
fio-2.12
starting 1 process
fio: got pattern '00', wanted '99'. Bad bits 4
fio: bad pattern block offset 0
   received data dumped as nvme0n1.12288.received
   expected data dumped as nvme0n1.12288.expected
fio: verify type mismatch (0 media, 14 given)
fio: got pattern '00', wanted '99'. Bad bits 4
fio: bad pattern block offset 0
   received data dumped as nvmeOn1.40960.received
   expected data dumped as nvme0n1.40960.expected
fio: verify type mismatch (0 media, 14 given)


On Tue, Dec 20, 2016 at 6:56 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 20 December 2016 at 12:26, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>
>> Thanks for your clarifications.
>> We ran with a --continue_on_error=verify,
>> to let the FIO complete the full compare..
>>
>> We tried to do a sequential write and compare, using the FIO config
>> file as below, and to bring in the complexity of "random" as a 2nd
>> step.
>> [write-and-verify]
>> rw=write
>> bs=4k
>> direct=1
>> ioengine=libaio
>> iodepth=16
>> size=2m
>> verify=pattern
>> verify_pattern=0x33333333
>> continue_on_error=verify
>> verify_dump=1
>> filename=/dev/XXXX
>>
>> FIO reports errors and we see files of the following names created:
>> <filename>.<num>.received
>> <filename>.<num>.expected
>>
>> Wanted help in interpreting the result.
>>
>> We wrote 2MB worth of data, with blocksize = 4K.
>> So, ideally is it expected to do 2MB/4KB = 512 IO operations
>>
>> 1) The received/expected files:
>> Are they for each 4K offset that failed the comparison ?
>
> I bet you can deduce this from the size and names of the files...
>
>> Is the <num> to be interpreted as the (num/bs)-th block that failed ?
>>    For ex: if the num=438272, and bs=4096 => 107th block failed ?
>>
>> It would be useful to know this information - so that we can debug further,
>> FYI, if we try a "dd" command and check the disk, based on the above
>> calculation - the data is proper (as expected).
>
> You never answered my question about what you are doing with stderr.
> I'll repeat it here:
>
>>> Are you doing something like redirecting stdout to a file but not
>>> doing anything with stderr? It would help if you include the command
>>> line you are using to run fio in your reply.
>
> Can you answer this question and post the full command line you ran
> fio with? I think it might have relevance to your current question.
>
>> 2) What were the locations that were written to..
>> Tried fio-verify-state <.state_file>, and get the below:
>> Version:        0x3
>> Size:           408
>> CRC:            0x70ca464a
>> Thread:         0
>> Name:           write-and-verify
>> Completions:    16
>> Depth:          16
>> Number IOs:     512
>> Index:          0
>> Completions:
>>         (file= 0) 2031616
>>         (file= 0) 2035712
>>         (file= 0) 2039808
>>         (file= 0) 2043904
>>         (file= 0) 2048000
>>         (file= 0) 2052096
>>         (file= 0) 2056192
>>         (file= 0) 2060288
>>         (file= 0) 2064384
>>         (file= 0) 2068480
>>         (file= 0) 2072576
>>         (file= 0) 2076672
>>         (file= 0) 2080768
>>         (file= 0) 2084864
>>         (file= 0) 2088960
>>         (file= 0) 2093056
>>
>> How do we interpret the above content to understand the locations of Writes.
>
> Perhaps fio tracks how far through the sequence it got rather than
> individual locations written (this would be necessary to handle things
> like loops=)? I personally don't know the answer to this but you can
> always take a look at the source code.
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

Hi,

Bear in mind I can't see your current test.fio (which must be
different from the one you previously posted because fio appears to be
searching for 0x99 whereas the previous job was searching for 0x33) so
if you have extra options in there they may invalidate any analysis I
make.

On 22 December 2016 at 04:48, Saju Nair <saju.mad.nair@gmail.com> wrote:
> Hi,
> Thanks. Please find the stderr messages snippet.
> We get the offsets where FIO observed failures, during the "verify" phase.
> The nvme**.<num>.received - is reported to contain 0.
>
> However, when we try a "dd" to access the location (offset) specified
> - we do see proper data.
> It might be an issue in our DUT, and we will investigate further.

I'm guessing you didn't use iflag=nocache or (iflag=direct + reading
in aligned block sizes) on your dd - are you sure your dd wasn't just
reading out of Linux's page cache? For example for an offset of 12288
you could use dd iflag=direct if=/dev/nvme0n1 bs=4k skip=3 count=1 |
hexdump (bear in mind it is better to do this directly after the fio .
Do you also get this same effect on disks you know to be good?

This is all very suspicious. If the change above somehow returns the
same data as fio I think you've got a problem on your hands. If the
problem goes away when you add end_fsync=1 to the fio job file then
that suggests someone has configured the NVMe devices for speed and
has thrown away safety.

> *************************************************************
> /root/demo/fio-2.12/bin/fio test.fio -eta=always --eta-newline=1 1 tee

This line seems garbled (" 1 tee"? Did you have to re-enter all output
by hand?) but assuming the real command line is correct your tee will
only save stdout to a file and not stderr. See
http://stackoverflow.com/questions/692000/how-do-i-write-stderr-to-a-file-while-using-tee-with-a-pipe
for suggestions on also saving stderr while using tee...

> fiol.log•rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
> ioengine=libaio, iodepth=16
> •rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
> ioengine=libaio, iodepth=16
> fio-2.12

^^^ Might be worth upgrading to the latest fio release (2.16) so you
don't hit any fixed issues.

> starting 1 process
> fio: got pattern '00', wanted '99'. Bad bits 4
> fio: bad pattern block offset 0

^^ Notice how this says the block offset, what it found and what it
wanted. This should make working out where to look straight forward.

>    received data dumped as nvme0n1.12288.received
>    expected data dumped as nvme0n1.12288.expected
> fio: verify type mismatch (0 media, 14 given)
> fio: got pattern '00', wanted '99'. Bad bits 4
> fio: bad pattern block offset 0
>    received data dumped as nvmeOn1.40960.received
>    expected data dumped as nvme0n1.40960.expected
> fio: verify type mismatch (0 media, 14 given)
>
>
> On Tue, Dec 20, 2016 at 6:56 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> Hi,
>>
>> On 20 December 2016 at 12:26, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>>
>>> Thanks for your clarifications.
>>> We ran with a --continue_on_error=verify,
>>> to let the FIO complete the full compare..
>>>
>>> We tried to do a sequential write and compare, using the FIO config
>>> file as below, and to bring in the complexity of "random" as a 2nd
>>> step.
>>> [write-and-verify]
>>> rw=write
>>> bs=4k
>>> direct=1
>>> ioengine=libaio
>>> iodepth=16
>>> size=2m
>>> verify=pattern
>>> verify_pattern=0x33333333
>>> continue_on_error=verify
>>> verify_dump=1
>>> filename=/dev/XXXX
>>>
>>> FIO reports errors and we see files of the following names created:
>>> <filename>.<num>.received
>>> <filename>.<num>.expected
>>>
>>> Wanted help in interpreting the result.
>>>
>>> We wrote 2MB worth of data, with blocksize = 4K.
>>> So, ideally is it expected to do 2MB/4KB = 512 IO operations
>>>
>>> 1) The received/expected files:
>>> Are they for each 4K offset that failed the comparison ?
>>
>> I bet you can deduce this from the size and names of the files...
>>
>>> Is the <num> to be interpreted as the (num/bs)-th block that failed ?
>>>    For ex: if the num=438272, and bs=4096 => 107th block failed ?
>>>
>>> It would be useful to know this information - so that we can debug further,
>>> FYI, if we try a "dd" command and check the disk, based on the above
>>> calculation - the data is proper (as expected).
>>
>> You never answered my question about what you are doing with stderr.
>> I'll repeat it here:
>>
>>>> Are you doing something like redirecting stdout to a file but not
>>>> doing anything with stderr? It would help if you include the command
>>>> line you are using to run fio in your reply.
>>
>> Can you answer this question and post the full command line you ran
>> fio with? I think it might have relevance to your current question.
>>
>>> 2) What were the locations that were written to..
>>> Tried fio-verify-state <.state_file>, and get the below:
>>> Version:        0x3
>>> Size:           408
>>> CRC:            0x70ca464a
>>> Thread:         0
>>> Name:           write-and-verify
>>> Completions:    16
>>> Depth:          16
>>> Number IOs:     512
>>> Index:          0
>>> Completions:
>>>         (file= 0) 2031616
>>>         (file= 0) 2035712
>>>         (file= 0) 2039808
>>>         (file= 0) 2043904
>>>         (file= 0) 2048000
>>>         (file= 0) 2052096
>>>         (file= 0) 2056192
>>>         (file= 0) 2060288
>>>         (file= 0) 2064384
>>>         (file= 0) 2068480
>>>         (file= 0) 2072576
>>>         (file= 0) 2076672
>>>         (file= 0) 2080768
>>>         (file= 0) 2084864
>>>         (file= 0) 2088960
>>>         (file= 0) 2093056
>>>
>>> How do we interpret the above content to understand the locations of Writes.
>>
>> Perhaps fio tracks how far through the sequence it got rather than
>> individual locations written (this would be necessary to handle things
>> like loops=)? I personally don't know the answer to this but you can
>> always take a look at the source code.
>>
>> --
>> Sitsofe | http://sucs.org/~sits/

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Thanks.
Apologies for the delay - Based on  the FIO debug messages, we figured
out that there was an underlying issue in the drive HW, and eventually
figured out the problem and fixed it.
FIO based data integrity works fine for us now, although at lower performance.
The read-verify step runs at about 1/10-th of the normal "read" performance.

Note that we keep "numjobs=1" - in order to not create any
complications due to this, in the verify stage.

I am not sure if this is possible, but, can FIO store the data read
into the RAM of the host machine ?
If so, one solution we are exploring is to break our existing
read-verify step to :

break into N smaller # FIO accesses, and foreach of N
   FIO reads - to RAM of host machine
   special program to mem-compare against expected data.



Regards,
- Saju.

On Thu, Dec 22, 2016 at 12:35 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> Bear in mind I can't see your current test.fio (which must be
> different from the one you previously posted because fio appears to be
> searching for 0x99 whereas the previous job was searching for 0x33) so
> if you have extra options in there they may invalidate any analysis I
> make.
>
> On 22 December 2016 at 04:48, Saju Nair <saju.mad.nair@gmail.com> wrote:
>> Hi,
>> Thanks. Please find the stderr messages snippet.
>> We get the offsets where FIO observed failures, during the "verify" phase.
>> The nvme**.<num>.received - is reported to contain 0.
>>
>> However, when we try a "dd" to access the location (offset) specified
>> - we do see proper data.
>> It might be an issue in our DUT, and we will investigate further.
>
> I'm guessing you didn't use iflag=nocache or (iflag=direct + reading
> in aligned block sizes) on your dd - are you sure your dd wasn't just
> reading out of Linux's page cache? For example for an offset of 12288
> you could use dd iflag=direct if=/dev/nvme0n1 bs=4k skip=3 count=1 |
> hexdump (bear in mind it is better to do this directly after the fio .
> Do you also get this same effect on disks you know to be good?
>
> This is all very suspicious. If the change above somehow returns the
> same data as fio I think you've got a problem on your hands. If the
> problem goes away when you add end_fsync=1 to the fio job file then
> that suggests someone has configured the NVMe devices for speed and
> has thrown away safety.
>
>> *************************************************************
>> /root/demo/fio-2.12/bin/fio test.fio -eta=always --eta-newline=1 1 tee
>
> This line seems garbled (" 1 tee"? Did you have to re-enter all output
> by hand?) but assuming the real command line is correct your tee will
> only save stdout to a file and not stderr. See
> http://stackoverflow.com/questions/692000/how-do-i-write-stderr-to-a-file-while-using-tee-with-a-pipe
> for suggestions on also saving stderr while using tee...
>
>> fiol.log•rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
>> ioengine=libaio, iodepth=16
>> •rite-and-verify: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K,
>> ioengine=libaio, iodepth=16
>> fio-2.12
>
> ^^^ Might be worth upgrading to the latest fio release (2.16) so you
> don't hit any fixed issues.
>
>> starting 1 process
>> fio: got pattern '00', wanted '99'. Bad bits 4
>> fio: bad pattern block offset 0
>
> ^^ Notice how this says the block offset, what it found and what it
> wanted. This should make working out where to look straight forward.
>
>>    received data dumped as nvme0n1.12288.received
>>    expected data dumped as nvme0n1.12288.expected
>> fio: verify type mismatch (0 media, 14 given)
>> fio: got pattern '00', wanted '99'. Bad bits 4
>> fio: bad pattern block offset 0
>>    received data dumped as nvmeOn1.40960.received
>>    expected data dumped as nvme0n1.40960.expected
>> fio: verify type mismatch (0 media, 14 given)
>>
>>
>> On Tue, Dec 20, 2016 at 6:56 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>> Hi,
>>>
>>> On 20 December 2016 at 12:26, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>>>
>>>> Thanks for your clarifications.
>>>> We ran with a --continue_on_error=verify,
>>>> to let the FIO complete the full compare..
>>>>
>>>> We tried to do a sequential write and compare, using the FIO config
>>>> file as below, and to bring in the complexity of "random" as a 2nd
>>>> step.
>>>> [write-and-verify]
>>>> rw=write
>>>> bs=4k
>>>> direct=1
>>>> ioengine=libaio
>>>> iodepth=16
>>>> size=2m
>>>> verify=pattern
>>>> verify_pattern=0x33333333
>>>> continue_on_error=verify
>>>> verify_dump=1
>>>> filename=/dev/XXXX
>>>>
>>>> FIO reports errors and we see files of the following names created:
>>>> <filename>.<num>.received
>>>> <filename>.<num>.expected
>>>>
>>>> Wanted help in interpreting the result.
>>>>
>>>> We wrote 2MB worth of data, with blocksize = 4K.
>>>> So, ideally is it expected to do 2MB/4KB = 512 IO operations
>>>>
>>>> 1) The received/expected files:
>>>> Are they for each 4K offset that failed the comparison ?
>>>
>>> I bet you can deduce this from the size and names of the files...
>>>
>>>> Is the <num> to be interpreted as the (num/bs)-th block that failed ?
>>>>    For ex: if the num=438272, and bs=4096 => 107th block failed ?
>>>>
>>>> It would be useful to know this information - so that we can debug further,
>>>> FYI, if we try a "dd" command and check the disk, based on the above
>>>> calculation - the data is proper (as expected).
>>>
>>> You never answered my question about what you are doing with stderr.
>>> I'll repeat it here:
>>>
>>>>> Are you doing something like redirecting stdout to a file but not
>>>>> doing anything with stderr? It would help if you include the command
>>>>> line you are using to run fio in your reply.
>>>
>>> Can you answer this question and post the full command line you ran
>>> fio with? I think it might have relevance to your current question.
>>>
>>>> 2) What were the locations that were written to..
>>>> Tried fio-verify-state <.state_file>, and get the below:
>>>> Version:        0x3
>>>> Size:           408
>>>> CRC:            0x70ca464a
>>>> Thread:         0
>>>> Name:           write-and-verify
>>>> Completions:    16
>>>> Depth:          16
>>>> Number IOs:     512
>>>> Index:          0
>>>> Completions:
>>>>         (file= 0) 2031616
>>>>         (file= 0) 2035712
>>>>         (file= 0) 2039808
>>>>         (file= 0) 2043904
>>>>         (file= 0) 2048000
>>>>         (file= 0) 2052096
>>>>         (file= 0) 2056192
>>>>         (file= 0) 2060288
>>>>         (file= 0) 2064384
>>>>         (file= 0) 2068480
>>>>         (file= 0) 2072576
>>>>         (file= 0) 2076672
>>>>         (file= 0) 2080768
>>>>         (file= 0) 2084864
>>>>         (file= 0) 2088960
>>>>         (file= 0) 2093056
>>>>
>>>> How do we interpret the above content to understand the locations of Writes.
>>>
>>> Perhaps fio tracks how far through the sequence it got rather than
>>> individual locations written (this would be necessary to handle things
>>> like loops=)? I personally don't know the answer to this but you can
>>> always take a look at the source code.
>>>
>>> --
>>> Sitsofe | http://sucs.org/~sits/
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Sitsofe Wheeler]

(Re-sending take 2 because Google's mobile web client forced HTML mail)

Hi,

I'm glad to hear you got to the bottom of things - were you able to
get dd to return the same data as fio in the end and if so how (it
might help others)? What was the change that solved your HW issue
(again it might help someone else in the future)?

Re verification speed: When you say the speed is one tenth that of
regular reads are the "regular" reads also using numjobs=1? If not the
comparison isn't fair and you need to rerun it with numjobs=1
everywhere and tell us what the difference was for those runs.

Re store data to RAM: as stated in previous emails fio isn't a bulk
data copying/moving tool so you would have to write new code to make
it act as such.

On 26 December 2016 at 05:30, Saju Nair <saju.mad.nair@gmail.com> wrote:
> Thanks.
> Apologies for the delay - Based on  the FIO debug messages, we figured
> out that there was an underlying issue in the drive HW, and eventually
> figured out the problem and fixed it.
> FIO based data integrity works fine for us now, although at lower performance.
> The read-verify step runs at about 1/10-th of the normal "read" performance.
>
> Note that we keep "numjobs=1" - in order to not create any
> complications due to this, in the verify stage.
>
> I am not sure if this is possible, but, can FIO store the data read
> into the RAM of the host machine ?
> If so, one solution we are exploring is to break our existing
> read-verify step to :
>
> break into N smaller # FIO accesses, and foreach of N
>    FIO reads - to RAM of host machine
>    special program to mem-compare against expected data.

-- 
Sitsofe | http://sucs.org/~sits/

Re: FIO -- A few basic questions on Data Integrity.

[Saju Nair]

Hi ,Hi,

I'm glad to hear you got to the bottom of things - were you able to
get dd to return the same data as fio in the end and if so how (it
might help others)? What was the change that solved your HW issue
(again it might help someone else in the future)?
>> The problem was in the LBA -> physical address mapping in our Hardware DUT, - it was a functional bug in the specific controller software.. On the "dd" correlation - it was not 100%, because this bug was not consistent in the mapping. Very specific to the DUT.

Re verification speed: When you say the speed is one tenth that of
regular reads are the "regular" reads also using numjobs=1? If not the
comparison isn't fair and you need to rerun it with numjobs=1
everywhere and tell us what the difference was for those runs.
>>Yes, it was with num_jobs = 1 in both cases of "regular read" and "read-verify". I think it is understandable that there is a performance drop, since the compare/verify is done on-the-fly.. Where does FIO store the data read, before the verify step is executed.

Re store data to RAM: as stated in previous emails fio isn't a bulk
data copying/moving tool so you would have to write new code to make
it act as such.
>>Thanks, understood.

On Mon, Dec 26, 2016 at 7:55 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> (Resenting because Google's mobile web client forces HTML mail)
>
> Hi,
>
> I'm glad to hear you got to the bottom of things - were you able to get dd
> to return the same data as fio in the end and if so how (it might help
> others)? What was the change that solved your HW issue (again it might help
> someone else in the future)?
>
> Re verification speed: When you say the speed is one tenth that of regular
> reads are the "regular" reads also using numjobs=1? If not the comparison
> isn't fair and you need to rerun it with numjobs=1 everywhere and tell us
> what the difference was for those runs.
>
> Re store data to RAM: as stated in previous emails fio isn't a bulk data
> copying/moving tool so you would have to write new code to make it act as
> such.
>
> On 26 December 2016 at 05:30, Saju Nair <saju.mad.nair@gmail.com> wrote:
>> Thanks.
>> Apologies for the delay - Based on  the FIO debug messages, we figured
>> out that there was an underlying issue in the drive HW, and eventually
>> figured out the problem and fixed it.
>> FIO based data integrity works fine for us now, although at lower
>> performance.
>> The read-verify step runs at about 1/10-th of the normal "read"
>> performance.
>>
>> Note that we keep "numjobs=1" - in order to not create any
>> complications due to this, in the verify stage.
>>
>> I am not sure if this is possible, but, can FIO store the data read
>> into the RAM of the host machine ?
>> If so, one solution we are exploring is to break our existing
>> read-verify step to :
>>
>> break into N smaller # FIO accesses, and foreach of N
>>    FIO reads - to RAM of host machine
>>    special program to mem-compare against expected data.
>>
>>
>>
>> Regards,
>> - Saju.
>
> --
> Sitsofe | http://sucs.org/~sits/