Re: best base / worst case RAID 5,6 write speeds

[Phil Turmel <philip@turmel.org>]

On 12/11/2015 07:00 PM, Dallas Clement wrote:

>> So is my workload of 12 fio jobs writing sequential 2 MB blocks with
>> direct I/O just too abusive?  Seems so with high queue depth.

I don't think you are adjusting any hardware queue depth here.  The fio
man page is quite explicit that iodepth=N is ineffective for sequential
operations.  But you are using the libaio engine, so you are piling up
many *software* queued operations for the kernel to execute, not
operations in flight to the disks.  From the histograms in your results,
the vast majority of ops are completing at depth=4.  Further queuing is
just adding kernel overhead.

The queuing differences from one kernel to another is a driver and
hardware property, not an application property.

>> I started this discussion because my RAID 5 and RAID 6 write
>> performance is really bad.  If my system is able to write to all 12
>> disks at 170 MB/s in JBOD mode, I am expecting that one fio job should
>> be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870
>> MB/s.  However, I am getting < 700 MB/s for queue depth = 32 and < 600
>> MB/s for queue depth = 256.  I get similarly disappointing results for
>> RAID 6 writes.

That's why I suggested blktrace.  Collect a trace while a single dd is
writing to your raw array device.  Compare the large writes submitted to
the md device against the broken down writes submitted to the member
devices.

Compare the patterns and sizes from older kernels against newer kernels,
possibly varying which controllers and data paths are involved.

Phil