odd result with iodepth

odd result with iodepth

[Chuck Tuffli]

I've been testing disk performance using fio, and am seeing results I
can't explain. This may not necessarily be a problem with fio, or
perhaps I'm not using the some of the parameters correctly. The
testing uses the latest fio.git on a RHEL system running a 2.6.32
kernel.

The fio test is:

fio --name=global --ioengine=libaio --rw=read --bs=512 --size=500m
--iodepth=32 --direct=1 --filename=/dev/sdb --name=job1

The IOPs reported in this case appears to correlate with an external analyzer.

If I change the iodepth to 64, fio reports double the IOPs reported in
the previous case, but the performance measured by the analyzer is the
same as in the iodepth=32 case (i.e. half what fio reports). I tried
the iodepth=64 case with another disk performance tool, and it is
reporting the same results as the external analyzer.

Interestingly, using iodepth=32 and adding a second job (--name=job2)
seems to have the same effect as iodepth=64 (reported performance is
2x compared to the external analyzer).

Am I doing something obviously wrong? What else can/should I check? TIA

---chuck

Re: odd result with iodepth

[Jeff Moyer]

Chuck Tuffli <ctuffli@gmail.com> writes:

> I've been testing disk performance using fio, and am seeing results I
> can't explain. This may not necessarily be a problem with fio, or
> perhaps I'm not using the some of the parameters correctly. The
> testing uses the latest fio.git on a RHEL system running a 2.6.32
> kernel.
>
> The fio test is:
>
> fio --name=global --ioengine=libaio --rw=read --bs=512 --size=500m
> --iodepth=32 --direct=1 --filename=/dev/sdb --name=job1
>
> The IOPs reported in this case appears to correlate with an external analyzer.
>
> If I change the iodepth to 64, fio reports double the IOPs reported in
> the previous case, but the performance measured by the analyzer is the
> same as in the iodepth=32 case (i.e. half what fio reports). I tried
> the iodepth=64 case with another disk performance tool, and it is
> reporting the same results as the external analyzer.
>
> Interestingly, using iodepth=32 and adding a second job (--name=job2)
> seems to have the same effect as iodepth=64 (reported performance is
> 2x compared to the external analyzer).
>
> Am I doing something obviously wrong? What else can/should I check? TIA

What's the maximum queue depth for the sd device?  I'll bet it's 32...

Cheers,
Jeff

Re: odd result with iodepth

[Chuck Tuffli]

On Thu, Mar 29, 2012 at 10:14 AM, Chris Worley <worleys@gmail.com> wrote:
> Sequential merging? �Try "randread" instead.

Great suggestion. Doing some additional analysis, it does appear that
IO's are probably getting merged as I see individual IO requests on
the wire of 2-4KB.

Googling around this area seems to suggest this kernel behavior isn't
something I can disable. So if I honest to goodness *only* want 512B
IO, is ioengine=sg the best route?

---chuck

Re: odd result with iodepth

[Jeff Moyer]

Chuck Tuffli <ctuffli@gmail.com> writes:

> On Thu, Mar 29, 2012 at 10:14 AM, Chris Worley <worleys@gmail.com> wrote:
>> Sequential merging?  Try "randread" instead.
>
> Great suggestion. Doing some additional analysis, it does appear that
> IO's are probably getting merged as I see individual IO requests on
> the wire of 2-4KB.
>
> Googling around this area seems to suggest this kernel behavior isn't
> something I can disable. So if I honest to goodness *only* want 512B
> IO, is ioengine=sg the best route?

Try setting /sys/block/sdX/queue/nomerges to 2.  That should completely
disable merging.

Cheers,
Jeff

Re: odd result with iodepth

[Jeff Moyer]

Jeff Moyer <jmoyer@redhat.com> writes:

> Chuck Tuffli <ctuffli@gmail.com> writes:
>
>> On Thu, Mar 29, 2012 at 10:14 AM, Chris Worley <worleys@gmail.com> wrote:
>>> Sequential merging?  Try "randread" instead.
>>
>> Great suggestion. Doing some additional analysis, it does appear that
>> IO's are probably getting merged as I see individual IO requests on
>> the wire of 2-4KB.
>>
>> Googling around this area seems to suggest this kernel behavior isn't
>> something I can disable. So if I honest to goodness *only* want 512B
>> IO, is ioengine=sg the best route?
>
> Try setting /sys/block/sdX/queue/nomerges to 2.  That should completely
> disable merging.

Oh wait, you said you're using a RHEL 2.6.32 based kernel, right?  That
isn't implemented there.  You'll have to backport this patch:

commit 488991e28e55b4fbca8067edf0259f69d1a6f92c
Author: Alan D. Brunelle <Alan.Brunelle@hp.com>
Date:   Fri Jan 29 09:04:08 2010 +0100

    block: Added in stricter no merge semantics for block I/O

Cheers,
Jeff

Re: odd result with iodepth

[Zhu Yanhai]

2012/3/30 Chuck Tuffli <ctuffli@gmail.com>:
> I've been testing disk performance using fio, and am seeing results I
> can't explain. This may not necessarily be a problem with fio, or
> perhaps I'm not using the some of the parameters correctly. The
> testing uses the latest fio.git on a RHEL system running a 2.6.32
> kernel.
>
> The fio test is:
>
> fio --name=global --ioengine=libaio --rw=read --bs=512 --size=500m
> --iodepth=32 --direct=1 --filename=/dev/sdb --name=job1
>
> The IOPs reported in this case appears to correlate with an external analyzer.
>
> If I change the iodepth to 64, fio reports double the IOPs reported in
> the previous case, but the performance measured by the analyzer is the
> same as in the iodepth=32 case (i.e. half what fio reports). I tried
> the iodepth=64 case with another disk performance tool, and it is
> reporting the same results as the external analyzer.
>
> Interestingly, using iodepth=32 and adding a second job (--name=job2)
> seems to have the same effect as iodepth=64 (reported performance is
> 2x compared to the external analyzer).
>
> Am I doing something obviously wrong? What else can/should I check? TIA
>
> ---chuck
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Can you see the same result if using  --rw=randread?

--
Regards,
Zhu Yanhai

Re: odd result with iodepth

[Chuck Tuffli]

On Thu, Mar 29, 2012 at 11:30 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> Chuck Tuffli <ctuffli@gmail.com> writes:
>
>> On Thu, Mar 29, 2012 at 10:14 AM, Chris Worley <worleys@gmail.com> wrote:
>>> Sequential merging? �Try "randread" instead.
>>
>> Great suggestion. Doing some additional analysis, it does appear that
>> IO's are probably getting merged as I see individual IO requests on
>> the wire of 2-4KB.
>>
>> Googling around this area seems to suggest this kernel behavior isn't
>> something I can disable. So if I honest to goodness *only* want 512B
>> IO, is ioengine=sg the best route?
>
> Try setting /sys/block/sdX/queue/nomerges to 2. �That should completely
> disable merging.

Jeff -

I upgrade to a 3.0 series kernel and set nomerges to 2 and now all the
tools agree on the IOPs number. Thanks!

---chuck