Re: best base / worst case RAID 5,6 write speeds

From: Dallas Clement <dallas.a.clement@gmail.com>
To: John Stoffel <john@stoffel.org>
Cc: Mark Knecht <markknecht@gmail.com>,
	Phil Turmel <philip@turmel.org>,
	Linux-RAID <linux-raid@vger.kernel.org>
Subject: Re: best base / worst case RAID 5,6 write speeds
Date: Fri, 11 Dec 2015 10:47:32 -0600	[thread overview]
Message-ID: <CAE9DZUQ=QynBKYJvq2JSnMaACKNpm+5yrhz+5x9Tx6_TK78mCg@mail.gmail.com> (raw)
In-Reply-To: <22122.64143.522908.45940@quad.stoffel.home>

On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>
> Dallas> Hi Mark.  I have three different controllers on this
> Dallas> motherboard.  A Marvell 9485 controls 8 of the disks.  And an
> Dallas> Intel Cougar Point controls the 4 remaining disks.
>
> What type of PCIe slots are the controllers in?  And how fast are the
> controllers/drives?  Are they SATA1/2/3 drives?
>
>>> If you're spinning in IO loops then it could be a driver issue.
>
> Dallas> It sure is looking like that.  I will try to profile the
> Dallas> kernel threads today and maybe use blktrace as Phil
> Dallas> recommended to see what is going on there.
>
> what kernel aer you running?
>
> Dallas> This is pretty sad that 12 single threaded fio jobs can bring
> Dallas> this system to its knees.
>
> I think it might be better to lower the queue depth, you might be just
> blowing out the controller caches...  hard to know.

Hi John.

> What type of PCIe slots are the controllers in?  And how fast are the
> controllers/drives?  Are they SATA1/2/3 drives?

The  MV 9485 controller is attached to an Intel Sandy Bridge via PCIe
GEN2 x 8.  This one controls 8 of the disks.
The Intel Cougar Point is connected to the Intel Sandy Bridge via DMI bus.

All of the drives are SATA III, however I do have two of the drives
connected to SATA II ports on the Cougar Point.  These two drives used
to be connected to SATA III ports on a MV 9125/9120 controller.  But
it had truly horrible write performance.  Moving to the SATA II ports
on the Cougar Point boosted the performance close to the same as the
other drives.  The remaining 10 drives are all connected to SATA III
ports.

> what kernel aer you running?

Right now, I'm using 3.10.69.  But I have tried the 4.2 kernel in
Fedora 23 with similar results.

> I think it might be better to lower the queue depth, you might be just
> blowing out the controller caches...  hard to know.

Good idea.  I'll trying lowering to see what effect.