Re: SATA RAID5 speed drop of 100 MB/s

[Carlo Wood <carlo@alinoe.com>]

On Sun, Jun 24, 2007 at 12:59:10PM -0400, Justin Piszcz wrote:
> Concerning NCQ/no NCQ, without NCQ I get an additional 15-50MB/s in speed 
> per various bonnie++ tests.

There is more going on than a bad NCQ implementation of the drive imho.
I did a long test over night (and still only got two schedulers done,
will do the other two tomorrow), and the difference between a queue depth
of 1 and 2 is DRAMATIC.

See http://www.xs4all.nl/~carlo17/noop_queue_depth.png
and http://www.xs4all.nl/~carlo17/anticipatory_queue_depth.png

The bonnie++ tests are done in a directory on the /dev/md7 and
/dev/ssd2 partitions respectively. Each bonnie test is performed
four times.

The hdparm -t tests (that show no difference with a -tT test) are
each done five times, for /dev/sdd, /dev/md7 and /dev/sda (that is
one of the RAID5 drives used for /dev/md7).

Thus in total there are 2 * 4 + 3 * 5 = 23 data points per
queue depth value in each graph.

The following can be observed:

1) There is hardly any difference between the two schedulers (noop
   is a little faster for the bonny test).
2) An NCQ depth of 1 is WAY faster on RAID5 (bonnie; around 125 MB/s),
   the NCQ depth of 2 is by far the slowest for the RAID5 (bonnie;
   around 40 MB/s). NCQ depths of 3 and higher show no difference,
   but are also slow (bonnie; around 75 MB/s).
3) There is no significant influence of the NCQ depth for non-RAID,
   either the /dev/sda (hdparm -t) or /dev/sdd disk (hdparm -t and
   bonnie).
4) With a NCQ depth > 1, the hdparm -t measurement of /dev/md7 is
   VERY unstable. Sometimes it gives the maximum (around 150 MB/s),
   and sometimes as low as 30 MB/s, seemingly independent of the
   NCQ depth. Note that those measurement were done on an otherwise
   unloaded machine in single user mode; and the measurements were
   all done one after an other. The strong fluctuation of the hdparm
   results for the RAID device (while the underlaying devices do not
   show this behaviour) are unexplainable.

>From the above I conclude that something must be wrong with the
software RAID implementation - and not just with the harddisks, imho.
At least, that's what it looks like to me. I am not an expert though ;)

-- 
Carlo Wood <carlo@alinoe.com>

PS RAID5 (md7 = sda7 + sdb7 + sdc7): Three times a Western Digital
   Raptor 10k rpm (WDC WD740ADFD-00NLR1).
   non-RAID (sdd2): Seagate barracuda 7200 rpm (ST3320620AS).

   The reason that now I measure around 145 MB/s instead of 165 MB/s
   as reported in previous post (with hdparm -t /dev/md7) is because
   before I use hdparm -t /dev/md2, which is closer to the outside
   of the disk and therefore faster. /dev/md2 still is around 165 MB/s.