Re: best base / worst case RAID 5,6 write speeds

[Phil Turmel <philip@turmel.org>]

On 12/11/2015 09:55 PM, Dallas Clement wrote:

> Right.  I understand the fio iodepth is different than the hardware
> queue depth.  But the fio man page seems to only mention limitation on
> synchronous operations which mine are not. I'm using direct=1 and
> sync=0.

You are confusing sequential and synchronous.  The man page says it is
ineffective for *sequential* operations, especially when direct=1.

> I guess what I would really like to know is how I can achieve at or
> near 100% utilization on the raid device and its member disks with
> fio.  Do I need to increase /sys/block/sd*/device/queue_depth and
> /sys/block/sd*/queue/nr_requests to get more utilization?

I don't know specifically.  It seems to me that increasing queue depth
adds resiliency in the face of data transfer timing jitter, but at the
cost of more CPU overhead.

I'm not convinced fio is the right workload, either.  It's options are
much more flexible for random I/O workloads.  dd isn't perfect either,
especially when writing zeroes -- it actually reads zeros over and over
from the special device.  For sequential operations I like dc3dd with
its pat= wipe= mode.  That'll only generate writes.

>> That's why I suggested blktrace.  Collect a trace while a single dd is
>> writing to your raw array device.  Compare the large writes submitted to
>> the md device against the broken down writes submitted to the member
>> devices.
> 
> Sounds good.  Will do.  What signs of trouble should I be looking for?

Look for strictly increasing logical block addresses in requests to the
member devices.  Any disruption in that will break optimum positioning
for streaming throughput. Per device. Requests to the device have to be
large enough and paced quickly enough to avoid starving the write head.

Of course, any reads mixed in mean RMW cycles you didn't avoid.  You
shouldn't have any of those for sequential writes in chunk * (n-2)
multiples.

I know it's a bit hand-wavy, but you have more hardware to play with
than I do :-)

Phil