Re: Faster fio verification

Re: Faster fio verification

[Sitsofe Wheeler]

Hi,

On 27 December 2016 at 04:33, Saju Nair <saju.mad.nair@gmail.com> wrote:
>
> > Re verification speed: When you say the speed is one tenth that of
> > regular reads are the "regular" reads also using numjobs=1? If not the
> > comparison isn't fair and you need to rerun it with numjobs=1
> > everywhere and tell us what the difference was for those runs.
>
> Yes, it was with num_jobs = 1 in both cases of "regular read" and "read-verify". I think it is understandable that there is a performance drop, since the compare/verify is done on-the-fly.. Where does FIO store the data read, before the verify step is executed.

During a verifying read the data of the block read is kept in RAM and
if needed (depends on the verification type) the data to compare
against is regenerated. Because of this very little or no data needs
to be stored during the write phase.

One option to try and speed verification up (assuming you have the
spare CPUs) could be to use verify_async (see the HOWTO). Another
option could be to use faster verification comparisons (see verify=str
of the HOWTO). For example the verify=crc32c checksum calculation will
be hardware accelerated on recent x86 machines. Do either of these
make a difference?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Faster fio verification

[Saju Nair]

Hi,
Thanks..
We will certainly try with the other verify=<options> of crc32c etc.
You mention ..
>>"During a verifying read the data of the block read is kept in RAM and
if needed (depends on the verification type) the data to compare
against is regenerated."

This is what we wanted to try and exploit..
If we have a host RAM of say 64GB, then, we wanted to understand if we
could break into "N" sets of say 16GB reads and verify. ie:
Assuming that we need to verify say 160GB of data, we could do :

fio --rw=write
for i = 0 to 10 {
  fio --rw=read --size=16g --verify=pattern|crc32c
--verify_pattern=<expected data>
}

Would that result in each of the FIO reads any faster ?

Regards,
- Saju.


On Tue, Dec 27, 2016 at 5:17 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 27 December 2016 at 04:33, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>
>> > Re verification speed: When you say the speed is one tenth that of
>> > regular reads are the "regular" reads also using numjobs=1? If not the
>> > comparison isn't fair and you need to rerun it with numjobs=1
>> > everywhere and tell us what the difference was for those runs.
>>
>> Yes, it was with num_jobs = 1 in both cases of "regular read" and "read-verify". I think it is understandable that there is a performance drop, since the compare/verify is done on-the-fly.. Where does FIO store the data read, before the verify step is executed.
>
> During a verifying read the data of the block read is kept in RAM and
> if needed (depends on the verification type) the data to compare
> against is regenerated. Because of this very little or no data needs
> to be stored during the write phase.
>
> One option to try and speed verification up (assuming you have the
> spare CPUs) could be to use verify_async (see the HOWTO). Another
> option could be to use faster verification comparisons (see verify=str
> of the HOWTO). For example the verify=crc32c checksum calculation will
> be hardware accelerated on recent x86 machines. Do either of these
> make a difference?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: Faster fio verification

[Sitsofe Wheeler]

On 27 December 2016 at 15:15, Saju Nair <saju.mad.nair@gmail.com> wrote:
> Hi,
> Thanks..
> We will certainly try with the other verify=<options> of crc32c etc.
> You mention ..
>>>"During a verifying read the data of the block read is kept in RAM and
> if needed (depends on the verification type) the data to compare
> against is regenerated."
>
> This is what we wanted to try and exploit..
> If we have a host RAM of say 64GB, then, we wanted to understand if we
> could break into "N" sets of say 16GB reads and verify. ie:
> Assuming that we need to verify say 160GB of data, we could do :
>
> fio --rw=write
> for i = 0 to 10 {
>   fio --rw=read --size=16g --verify=pattern|crc32c
> --verify_pattern=<expected data>
> }
>
> Would that result in each of the FIO reads any faster ?
>
> Regards,
> - Saju.
>
>
> On Tue, Dec 27, 2016 at 5:17 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> Hi,
>>
>> On 27 December 2016 at 04:33, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>>
>>> > Re verification speed: When you say the speed is one tenth that of
>>> > regular reads are the "regular" reads also using numjobs=1? If not the
>>> > comparison isn't fair and you need to rerun it with numjobs=1
>>> > everywhere and tell us what the difference was for those runs.
>>>
>>> Yes, it was with num_jobs = 1 in both cases of "regular read" and "read-verify". I think it is understandable that there is a performance drop, since the compare/verify is done on-the-fly.. Where does FIO store the data read, before the verify step is executed.
>>
>> During a verifying read the data of the block read is kept in RAM and
>> if needed (depends on the verification type) the data to compare
>> against is regenerated. Because of this very little or no data needs
>> to be stored during the write phase.
>>
>> One option to try and speed verification up (assuming you have the
>> spare CPUs) could be to use verify_async (see the HOWTO). Another
>> option could be to use faster verification comparisons (see verify=str
>> of the HOWTO). For example the verify=crc32c checksum calculation will
>> be hardware accelerated on recent x86 machines. Do either of these
>> make a difference?
>>
>> --
>> Sitsofe | http://sucs.org/~sits/



-- 
Sitsofe | http://sucs.org/~sits/

Re: Faster fio verification

[Sitsofe Wheeler]

Hi,

On 27 December 2016 at 15:15, Saju Nair <saju.mad.nair@gmail.com> wrote:
> Hi,
> Thanks..
> We will certainly try with the other verify=<options> of crc32c etc.
> You mention ..
>>>"During a verifying read the data of the block read is kept in RAM and
> if needed (depends on the verification type) the data to compare
> against is regenerated."
>
> This is what we wanted to try and exploit..
> If we have a host RAM of say 64GB, then, we wanted to understand if we
> could break into "N" sets of say 16GB reads and verify. ie:
> Assuming that we need to verify say 160GB of data, we could do :
>
> fio --rw=write
> for i = 0 to 10 {
>   fio --rw=read --size=16g --verify=pattern|crc32c
> --verify_pattern=<expected data>
> }
>
> Would that result in each of the FIO reads any faster ?

Not by default because the verifying happens inline after the read
finishes and before that "slot" is reused so it doesn't matter how big
the disk is - you only need memory of the order of iodepth's worth of
maximum sized I/Os. Did I mention the verify_async option changes
this? Check the HOWTO for more information.

> On Tue, Dec 27, 2016 at 5:17 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> Hi,
>>
>> On 27 December 2016 at 04:33, Saju Nair <saju.mad.nair@gmail.com> wrote:
>>>
>>> > Re verification speed: When you say the speed is one tenth that of
>>> > regular reads are the "regular" reads also using numjobs=1? If not the
>>> > comparison isn't fair and you need to rerun it with numjobs=1
>>> > everywhere and tell us what the difference was for those runs.
>>>
>>> Yes, it was with num_jobs = 1 in both cases of "regular read" and "read-verify". I think it is understandable that there is a performance drop, since the compare/verify is done on-the-fly.. Where does FIO store the data read, before the verify step is executed.
>>
>> During a verifying read the data of the block read is kept in RAM and
>> if needed (depends on the verification type) the data to compare
>> against is regenerated. Because of this very little or no data needs
>> to be stored during the write phase.
>>
>> One option to try and speed verification up (assuming you have the
>> spare CPUs) could be to use verify_async (see the HOWTO). Another
>> option could be to use faster verification comparisons (see verify=str
>> of the HOWTO). For example the verify=crc32c checksum calculation will
>> be hardware accelerated on recent x86 machines. Do either of these
>> make a difference?

-- 
Sitsofe | http://sucs.org/~sits/