Linux Raid performance

Linux Raid performance

[Learner Study]

Hi Linux Raid Experts:

I was looking at following wiki on raid perf on linux:

https://raid.wiki.kernel.org/index.php/Performance

and notice that the performance numbers are with 2.6.12 kernel.

Do we perf numbers for:
- latest kernel (something like 2.6.27 / 2.6.31)
- raid 5 and 6

Can someone please point me to appropriate link?

Thanks!

Re: Linux Raid performance

[Keld Simonsen]

On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
> Hi Linux Raid Experts:
> 
> I was looking at following wiki on raid perf on linux:
> 
> https://raid.wiki.kernel.org/index.php/Performance
> 
> and notice that the performance numbers are with 2.6.12 kernel.
> 
> Do we perf numbers for:
> - latest kernel (something like 2.6.27 / 2.6.31)
> - raid 5 and 6
> 
> Can someone please point me to appropriate link?

The link mentioned above has a number of other performance reports, for other levels of the kernel.
Anyway you should be able to get comparable results for newer kernels, the kernel has not become 
slower since 2.6.12 on RAID.

best regards
Keld

Re: Linux Raid performance

[Learner Study]

Hi Keld:

Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
single core CPUs. How does that scale with multiple cores? Something
like intel's jasper forest???

If available, can u pls point me to numbers with multi-core CPU?

Thanks!

On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
> On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>> Hi Linux Raid Experts:
>>
>> I was looking at following wiki on raid perf on linux:
>>
>> https://raid.wiki.kernel.org/index.php/Performance
>>
>> and notice that the performance numbers are with 2.6.12 kernel.
>>
>> Do we perf numbers for:
>> - latest kernel (something like 2.6.27 / 2.6.31)
>> - raid 5 and 6
>>
>> Can someone please point me to appropriate link?
>
> The link mentioned above has a number of other performance reports, for other levels of the kernel.
> Anyway you should be able to get comparable results for newer kernels, the kernel has not become
> slower since 2.6.12 on RAID.
>
> best regards
> Keld
>

Re: Linux Raid performance

[Nicolae Mihalache]

I see some benchmarks performed at boot time on my Xeon E5410 2.33GHz
that shows
...
[   37.935702] raid6: sse2x1    3562 MB/s
[   38.003702] raid6: sse2x2    6422 MB/s
[   38.003702] raid6: using algorithm sse2x2 (6422 MB/s)

This  speed is higher that the DDR2 667 theoretical speed of
5333MBs/sec. So I expect the limiting factor will never be the CPU, so
it would not make sense to use multi-core.

Am I completely off?


nicolae


Learner Study wrote:
> Hi Keld:
>
> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
> single core CPUs. How does that scale with multiple cores? Something
> like intel's jasper forest???
>
> If available, can u pls point me to numbers with multi-core CPU?
>
> Thanks!
>
> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
>   
>> On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>>     
>>> Hi Linux Raid Experts:
>>>
>>> I was looking at following wiki on raid perf on linux:
>>>
>>> https://raid.wiki.kernel.org/index.php/Performance
>>>
>>> and notice that the performance numbers are with 2.6.12 kernel.
>>>
>>> Do we perf numbers for:
>>> - latest kernel (something like 2.6.27 / 2.6.31)
>>> - raid 5 and 6
>>>
>>> Can someone please point me to appropriate link?
>>>       
>> The link mentioned above has a number of other performance reports, for other levels of the kernel.
>> Anyway you should be able to get comparable results for newer kernels, the kernel has not become
>> slower since 2.6.12 on RAID.
>>
>> best regards
>> Keld
>>
>>     
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: Linux Raid performance

[Keld Simonsen]

On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
> Hi Keld:
> 
> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
> single core CPUs. How does that scale with multiple cores? Something
> like intel's jasper forest???

I have not checked if the benchmarks were on multi core machines. 
It should not matter much if there were more than one CPU, but
of cause it helps a little. bonnie++ test reports cpu usage, and this
is not insignificant, say in the 20 -60 % range for some tests,
but nowhere near a bottleneck. There was one with a raid5 performance
seq read of about 500 MB/s with 36 % cpu utilization, so it is
definitely possible to come beyound 150 MB/s. The speed is largely
dependent on number of disk drives you employ.


> If available, can u pls point me to numbers with multi-core CPU?

I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
so please feel free to submit your findings.

Best regards
keld

> Thanks!
> 
> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
> >> Hi Linux Raid Experts:
> >>
> >> I was looking at following wiki on raid perf on linux:
> >>
> >> https://raid.wiki.kernel.org/index.php/Performance
> >>
> >> and notice that the performance numbers are with 2.6.12 kernel.
> >>
> >> Do we perf numbers for:
> >> - latest kernel (something like 2.6.27 / 2.6.31)
> >> - raid 5 and 6
> >>
> >> Can someone please point me to appropriate link?
> >
> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
> > slower since 2.6.12 on RAID.
> >
> > best regards
> > Keld
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Keld Simonsen]

Hi!

Furthermore, I am not sure how much multiple CPU helps you.
It seems like each array are handled by a separate process.
This process possibly has internal data for management of the array, and it may 
be bound to run on a single processor, with no multithreading. 

Neil, could you explain if and how Linux MD takes advantage of
multiple processors available, for a single array?

Best regards
keld


On Fri, Apr 02, 2010 at 01:05:06PM +0200, Keld Simonsen wrote:
> On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
> > Hi Keld:
> > 
> > Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
> > the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
> > single core CPUs. How does that scale with multiple cores? Something
> > like intel's jasper forest???
> 
> I have not checked if the benchmarks were on multi core machines. 
> It should not matter much if there were more than one CPU, but
> of cause it helps a little. bonnie++ test reports cpu usage, and this
> is not insignificant, say in the 20 -60 % range for some tests,
> but nowhere near a bottleneck. There was one with a raid5 performance
> seq read of about 500 MB/s with 36 % cpu utilization, so it is
> definitely possible to come beyound 150 MB/s. The speed is largely
> dependent on number of disk drives you employ.
> 
> 
> > If available, can u pls point me to numbers with multi-core CPU?
> 
> I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
> so please feel free to submit your findings.
> 
> Best regards
> keld
> 
> > Thanks!
> > 
> > On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
> > > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
> > >> Hi Linux Raid Experts:
> > >>
> > >> I was looking at following wiki on raid perf on linux:
> > >>
> > >> https://raid.wiki.kernel.org/index.php/Performance
> > >>
> > >> and notice that the performance numbers are with 2.6.12 kernel.
> > >>
> > >> Do we perf numbers for:
> > >> - latest kernel (something like 2.6.27 / 2.6.31)
> > >> - raid 5 and 6
> > >>
> > >> Can someone please point me to appropriate link?
> > >
> > > The link mentioned above has a number of other performance reports, for other levels of the kernel.
> > > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
> > > slower since 2.6.12 on RAID.
> > >
> > > best regards
> > > Keld
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Learner Study]

Hi Keld:

Thanks for your email...

1. Can you pls point me to this benchmark (which shows 500MB/s)? I
would like to know which CPU, HDDs and kernel version used to achieve
this...

2. Secondly, I would like to understand how raid stack (md driver)
scales as we add more cores...if single core gives ~500MB/s, can two
core give ~1000MB/s? can four cores give ~2000MB/s? etc....

Thanks for your time.

On Fri, Apr 2, 2010 at 4:05 AM, Keld Simonsen <keld@keldix.com> wrote:
> On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
>> Hi Keld:
>>
>> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
>> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
>> single core CPUs. How does that scale with multiple cores? Something
>> like intel's jasper forest???
>
> I have not checked if the benchmarks were on multi core machines.
> It should not matter much if there were more than one CPU, but
> of cause it helps a little. bonnie++ test reports cpu usage, and this
> is not insignificant, say in the 20 -60 % range for some tests,
> but nowhere near a bottleneck. There was one with a raid5 performance
> seq read of about 500 MB/s with 36 % cpu utilization, so it is
> definitely possible to come beyound 150 MB/s. The speed is largely
> dependent on number of disk drives you employ.
>
>
>> If available, can u pls point me to numbers with multi-core CPU?
>
> I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
> so please feel free to submit your findings.
>
> Best regards
> keld
>
>> Thanks!
>>
>> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
>> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>> >> Hi Linux Raid Experts:
>> >>
>> >> I was looking at following wiki on raid perf on linux:
>> >>
>> >> https://raid.wiki.kernel.org/index.php/Performance
>> >>
>> >> and notice that the performance numbers are with 2.6.12 kernel.
>> >>
>> >> Do we perf numbers for:
>> >> - latest kernel (something like 2.6.27 / 2.6.31)
>> >> - raid 5 and 6
>> >>
>> >> Can someone please point me to appropriate link?
>> >
>> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
>> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
>> > slower since 2.6.12 on RAID.
>> >
>> > best regards
>> > Keld
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Learner Study]

Hi Nicolae:

Can you please let me know:
- what HDDs (rpm etc.) of your system?
- which linux kernel are you using?

Thanks!

On Fri, Apr 2, 2010 at 2:58 AM, Nicolae Mihalache <mache@abcpages.com> wrote:
> I see some benchmarks performed at boot time on my Xeon E5410 2.33GHz
> that shows
> ...
> [   37.935702] raid6: sse2x1    3562 MB/s
> [   38.003702] raid6: sse2x2    6422 MB/s
> [   38.003702] raid6: using algorithm sse2x2 (6422 MB/s)
>
> This  speed is higher that the DDR2 667 theoretical speed of
> 5333MBs/sec. So I expect the limiting factor will never be the CPU, so
> it would not make sense to use multi-core.
>
> Am I completely off?
>
>
> nicolae
>
>
> Learner Study wrote:
>> Hi Keld:
>>
>> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
>> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
>> single core CPUs. How does that scale with multiple cores? Something
>> like intel's jasper forest???
>>
>> If available, can u pls point me to numbers with multi-core CPU?
>>
>> Thanks!
>>
>> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
>>
>>> On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>>>
>>>> Hi Linux Raid Experts:
>>>>
>>>> I was looking at following wiki on raid perf on linux:
>>>>
>>>> https://raid.wiki.kernel.org/index.php/Performance
>>>>
>>>> and notice that the performance numbers are with 2.6.12 kernel.
>>>>
>>>> Do we perf numbers for:
>>>> - latest kernel (something like 2.6.27 / 2.6.31)
>>>> - raid 5 and 6
>>>>
>>>> Can someone please point me to appropriate link?
>>>>
>>> The link mentioned above has a number of other performance reports, for other levels of the kernel.
>>> Anyway you should be able to get comparable results for newer kernels, the kernel has not become
>>> slower since 2.6.12 on RAID.
>>>
>>> best regards
>>> Keld
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Keld Simonsen]

On Fri, Apr 02, 2010 at 10:55:53AM -0700, Learner Study wrote:
> Hi Keld:
> 
> Thanks for your email...
> 
> 1. Can you pls point me to this benchmark (which shows 500MB/s)? I
> would like to know which CPU, HDDs and kernel version used to achieve
> this...

http://home.comcast.net/~jpiszcz/20080329-raid/
496843	 KB/s for sequential input with 10 raptor drives
There probably is an email in the archives with more info on the
test. 

> 2. Secondly, I would like to understand how raid stack (md driver)
> scales as we add more cores...if single core gives ~500MB/s, can two
> core give ~1000MB/s? can four cores give ~2000MB/s? etc....

No, the performance is normally limited by the number of drives.
I would not wsay that more cores woould do a little 
but it would be in the order of 1-2 % I think.
This is also dependent on wheteher the code actually runs threaded.
I doubt it....

best regard
keld

> 
> Thanks for your time.
> 
> On Fri, Apr 2, 2010 at 4:05 AM, Keld Simonsen <keld@keldix.com> wrote:
> > On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
> >> Hi Keld:
> >>
> >> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
> >> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
> >> single core CPUs. How does that scale with multiple cores? Something
> >> like intel's jasper forest???
> >
> > I have not checked if the benchmarks were on multi core machines.
> > It should not matter much if there were more than one CPU, but
> > of cause it helps a little. bonnie++ test reports cpu usage, and this
> > is not insignificant, say in the 20 -60 % range for some tests,
> > but nowhere near a bottleneck. There was one with a raid5 performance
> > seq read of about 500 MB/s with 36 % cpu utilization, so it is
> > definitely possible to come beyound 150 MB/s. The speed is largely
> > dependent on number of disk drives you employ.
> >
> >
> >> If available, can u pls point me to numbers with multi-core CPU?
> >
> > I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
> > so please feel free to submit your findings.
> >
> > Best regards
> > keld
> >
> >> Thanks!
> >>
> >> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
> >> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
> >> >> Hi Linux Raid Experts:
> >> >>
> >> >> I was looking at following wiki on raid perf on linux:
> >> >>
> >> >> https://raid.wiki.kernel.org/index.php/Performance
> >> >>
> >> >> and notice that the performance numbers are with 2.6.12 kernel.
> >> >>
> >> >> Do we perf numbers for:
> >> >> - latest kernel (something like 2.6.27 / 2.6.31)
> >> >> - raid 5 and 6
> >> >>
> >> >> Can someone please point me to appropriate link?
> >> >
> >> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
> >> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
> >> > slower since 2.6.12 on RAID.
> >> >
> >> > best regards
> >> > Keld
> >> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Learner Study]

I have seen ~180MB/s RAID5 performance with 4 disks...are you saying
that I could achieve even higher if I have more number of disks (so
instead of 3+1, try 6+1 or 9+1)?
Logically, this sounds right but wanted to verify my thought process
with you....

Thanks!

On Fri, Apr 2, 2010 at 2:14 PM, Keld Simonsen <keld@keldix.com> wrote:
> On Fri, Apr 02, 2010 at 10:55:53AM -0700, Learner Study wrote:
>> Hi Keld:
>>
>> Thanks for your email...
>>
>> 1. Can you pls point me to this benchmark (which shows 500MB/s)? I
>> would like to know which CPU, HDDs and kernel version used to achieve
>> this...
>
> http://home.comcast.net/~jpiszcz/20080329-raid/
> 496843   KB/s for sequential input with 10 raptor drives
> There probably is an email in the archives with more info on the
> test.
>
>> 2. Secondly, I would like to understand how raid stack (md driver)
>> scales as we add more cores...if single core gives ~500MB/s, can two
>> core give ~1000MB/s? can four cores give ~2000MB/s? etc....
>
> No, the performance is normally limited by the number of drives.
> I would not wsay that more cores woould do a little
> but it would be in the order of 1-2 % I think.
> This is also dependent on wheteher the code actually runs threaded.
> I doubt it....
>
> best regard
> keld
>
>>
>> Thanks for your time.
>>
>> On Fri, Apr 2, 2010 at 4:05 AM, Keld Simonsen <keld@keldix.com> wrote:
>> > On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
>> >> Hi Keld:
>> >>
>> >> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
>> >> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
>> >> single core CPUs. How does that scale with multiple cores? Something
>> >> like intel's jasper forest???
>> >
>> > I have not checked if the benchmarks were on multi core machines.
>> > It should not matter much if there were more than one CPU, but
>> > of cause it helps a little. bonnie++ test reports cpu usage, and this
>> > is not insignificant, say in the 20 -60 % range for some tests,
>> > but nowhere near a bottleneck. There was one with a raid5 performance
>> > seq read of about 500 MB/s with 36 % cpu utilization, so it is
>> > definitely possible to come beyound 150 MB/s. The speed is largely
>> > dependent on number of disk drives you employ.
>> >
>> >
>> >> If available, can u pls point me to numbers with multi-core CPU?
>> >
>> > I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
>> > so please feel free to submit your findings.
>> >
>> > Best regards
>> > keld
>> >
>> >> Thanks!
>> >>
>> >> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
>> >> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>> >> >> Hi Linux Raid Experts:
>> >> >>
>> >> >> I was looking at following wiki on raid perf on linux:
>> >> >>
>> >> >> https://raid.wiki.kernel.org/index.php/Performance
>> >> >>
>> >> >> and notice that the performance numbers are with 2.6.12 kernel.
>> >> >>
>> >> >> Do we perf numbers for:
>> >> >> - latest kernel (something like 2.6.27 / 2.6.31)
>> >> >> - raid 5 and 6
>> >> >>
>> >> >> Can someone please point me to appropriate link?
>> >> >
>> >> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
>> >> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
>> >> > slower since 2.6.12 on RAID.
>> >> >
>> >> > best regards
>> >> > Keld
>> >> >
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Learner Study]

Hi:

I can't seem to find email in the archive on what test configuration
(what kernel, which CPU, how many disks, disk RPM etc.) was used to
achieve 500MB/s..

Can someone please point me to the appropriate discussion...

Thanks!

On Fri, Apr 2, 2010 at 2:14 PM, Keld Simonsen <keld@keldix.com> wrote:
> On Fri, Apr 02, 2010 at 10:55:53AM -0700, Learner Study wrote:
>> Hi Keld:
>>
>> Thanks for your email...
>>
>> 1. Can you pls point me to this benchmark (which shows 500MB/s)? I
>> would like to know which CPU, HDDs and kernel version used to achieve
>> this...
>
> http://home.comcast.net/~jpiszcz/20080329-raid/
> 496843   KB/s for sequential input with 10 raptor drives
> There probably is an email in the archives with more info on the
> test.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Mark Knecht]

On Fri, Apr 2, 2010 at 10:55 AM, Learner Study <learner.study@gmail.com> wrote:
<SNIP>
>
> 2. Secondly, I would like to understand how raid stack (md driver)
> scales as we add more cores...if single core gives ~500MB/s, can two
> core give ~1000MB/s? can four cores give ~2000MB/s? etc....
<SNIP>

More cores by themselves certainly won't do it for you.

1) More disks in parallel. (striped data)

2) More ports to attach those drives.

3) More bandwidth on those ports. SATA3 is better than SATA2 is better
than SATA is better than PATA, etc. (Obviously disks must match ports,
right? SATA1 disks on SATA3 ports isn't the right thing...)

4) More bus bandwidth getting to those ports. PCI-Express16 is better
than PCI-Express1 is better than PCI, etc.

5) Faster RAID architectures for the number of disks chosen.

Once all of that is in place then possibly more cores will help, but I
suspect even then it probably hard to use 4 billion CPU cycles/second
doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
overhead is relatively *very* low.

HTH,
Mark

Re: Linux Raid performance

[John Robinson]

On 03/04/2010 01:39, Mark Knecht wrote:
> On Fri, Apr 2, 2010 at 10:55 AM, Learner Study <learner.study@gmail.com> wrote:
> <SNIP>
>> 2. Secondly, I would like to understand how raid stack (md driver)
>> scales as we add more cores...if single core gives ~500MB/s, can two
>> core give ~1000MB/s? can four cores give ~2000MB/s? etc....
> <SNIP>
> 
> More cores by themselves certainly won't do it for you.
> 
> 1) More disks in parallel. (striped data)
> 
> 2) More ports to attach those drives.
> 
> 3) More bandwidth on those ports. SATA3 is better than SATA2 is better
> than SATA is better than PATA, etc. (Obviously disks must match ports,
> right? SATA1 disks on SATA3 ports isn't the right thing...)
> 
> 4) More bus bandwidth getting to those ports. PCI-Express16 is better
> than PCI-Express1 is better than PCI, etc.
> 
> 5) Faster RAID architectures for the number of disks chosen.
> 
> Once all of that is in place then possibly more cores will help, but I
> suspect even then it probably hard to use 4 billion CPU cycles/second
> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
> overhead is relatively *very* low.

Right. As has recently been demonstrated on this list, one core on a 
slow Xeon can do about 8TB/s of RAID-6 calculations, whereas the 
theoretical limit on memory bandwidth for the platform is about 6TB/s, 
so one CPU thread is already faster than the whole system's memory 
bandwidth. After that, current discs manage about 150MB/s at their peak 
so you'd need 40+ discs in one array to reach the memory bandwidth 
limit. The upshot appears to me to be that with current architectures 
and discs, there's no need for multi-core/multi-threading. Having said 
that, individual arrays currently run single-threaded, but multiple 
arrays can run on separate CPU cores if necessary, with traditional 
process scheduling.

There is experimental support for multi-threading in the kernel right 
now, which was awful because the threading model didn't work, and which 
has even more recently been replaced with another experimental 
multi-threading patch using btrfs thread pooling, which is as yet 
unproved. So, multi-core / multi-threading support is on the way, but at 
the moment is not required.

I haven't included references because a quick search of the last month's 
archives of this list will reveal all of them.

Overall, the bottleneck right now is the discs, as has been the case 
since ooh forever.

Cheers,

John.

Re: Linux Raid performance

[Richard Scobie]

Mark Knecht wrote:

> Once all of that is in place then possibly more cores will help, but I
> suspect even then it probably hard to use 4 billion CPU cycles/second
> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
> overhead is relatively *very* low.

There is the RAID5/6 parity calculations to be considered on writes and 
this appears to be single threaded. There is an experimental multicore 
kernel option I believe, but recent discussion indicates there may be 
some problems with it.

A very quick test on a box here on a Xeon E5440 (4 x 2.8GHz) and a SAS 
attached 16 x 750GB SATA md RAID6. The array is 72% full and probably 
quite fragmented and currently the system is idle.

dd if=/dev/zero of=/mnt/storage/dump bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 87.2374 s, 240 MB/s

Looking at the outputs of vmstat 5 and mpstat -P ALL 5 during this, one 
core (probably doing parity generation) was around 7.56% idle and the 
other 3 were around 88.5, 67.5 and 51.8% idle.

The same test run when the system was commissioned and the array was 
empty, acheived 565MB/s writes.

Regards,

Richard

Re: Linux Raid performance

[Mark Knecht]

On Fri, Apr 2, 2010 at 6:14 PM, Richard Scobie <richard@sauce.co.nz> wrote:
> Mark Knecht wrote:
>
>> Once all of that is in place then possibly more cores will help, but I
>> suspect even then it probably hard to use 4 billion CPU cycles/second
>> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
>> overhead is relatively *very* low.
>
> There is the RAID5/6 parity calculations to be considered on writes and this
> appears to be single threaded. There is an experimental multicore kernel
> option I believe, but recent discussion indicates there may be some problems
> with it.
>
> A very quick test on a box here on a Xeon E5440 (4 x 2.8GHz) and a SAS
> attached 16 x 750GB SATA md RAID6. The array is 72% full and probably quite
> fragmented and currently the system is idle.
>
> dd if=/dev/zero of=/mnt/storage/dump bs=1M count=20000
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 87.2374 s, 240 MB/s
>
> Looking at the outputs of vmstat 5 and mpstat -P ALL 5 during this, one core
> (probably doing parity generation) was around 7.56% idle and the other 3
> were around 88.5, 67.5 and 51.8% idle.
>
> The same test run when the system was commissioned and the array was empty,
> acheived 565MB/s writes.
>
> Regards,
>
> Richard

Richard,
   Good point. I was limited in my thinking to the sorts of arrays I
might use at home being no wider than 3, 4 or 5 disks. However for our
N-wide array as N approaches infinity so do the cycles required to run
it. I don think that applies to the OP but I don't know that.

   Thanks for making the point.

Cheers,
Mark

Re: Linux Raid performance

[Richard Scobie]

Mark Knecht wrote:

> Richard,
>     Good point. I was limited in my thinking to the sorts of arrays I
> might use at home being no wider than 3, 4 or 5 disks. However for our
> N-wide array as N approaches infinity so do the cycles required to run
> it. I don think that applies to the OP but I don't know that.
>

I said I thought the busiest CPU was the parity generation one, but in 
hindsight this cannot be correct, as it was almost maxed out at half the 
write speed the array achieved when it was empty.

Regards,

Richard

Re: Linux Raid performance

[Learner Study]

Hi Richard:

Thanks for sharing your results. So, 565MB/s with 16 disks in an
array. It would be nice to see how it scales as we add more disks.

What kernel version did you use for this test?

Thanks!

On Fri, Apr 2, 2010 at 6:14 PM, Richard Scobie <richard@sauce.co.nz> wrote:
> Mark Knecht wrote:
>
>> Once all of that is in place then possibly more cores will help, but I
>> suspect even then it probably hard to use 4 billion CPU cycles/second
>> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
>> overhead is relatively *very* low.
>
> There is the RAID5/6 parity calculations to be considered on writes and this
> appears to be single threaded. There is an experimental multicore kernel
> option I believe, but recent discussion indicates there may be some problems
> with it.
>
> A very quick test on a box here on a Xeon E5440 (4 x 2.8GHz) and a SAS
> attached 16 x 750GB SATA md RAID6. The array is 72% full and probably quite
> fragmented and currently the system is idle.
>
> dd if=/dev/zero of=/mnt/storage/dump bs=1M count=20000
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 87.2374 s, 240 MB/s
>
> Looking at the outputs of vmstat 5 and mpstat -P ALL 5 during this, one core
> (probably doing parity generation) was around 7.56% idle and the other 3
> were around 88.5, 67.5 and 51.8% idle.
>
> The same test run when the system was commissioned and the array was empty,
> acheived 565MB/s writes.
>
> Regards,
>
> Richard
>
>

Re: Linux Raid performance

[Learner Study]

Thanks to everyone for sharing their experiences...this was helpful.

I don't have luxury to have 16 SAS/SATA drives...I guess it would be
great if someone can share results with even higher number of
disks...I would like to know what is the max performance that can be
reached...I understand the theoretical is more like a couple of
Terabytes...but won't we get into some sort of linux file system
related or other kernel bottlenecks as we increase the # of disks?

On Fri, Apr 2, 2010 at 6:37 PM, Richard Scobie <richard@sauce.co.nz> wrote:
> Mark Knecht wrote:
>
>> Richard,
>>    Good point. I was limited in my thinking to the sorts of arrays I
>> might use at home being no wider than 3, 4 or 5 disks. However for our
>> N-wide array as N approaches infinity so do the cycles required to run
>> it. I don think that applies to the OP but I don't know that.
>>
>
> I said I thought the busiest CPU was the parity generation one, but in
> hindsight this cannot be correct, as it was almost maxed out at half the
> write speed the array achieved when it was empty.
>
> Regards,
>
> Richard
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Keld Simonsen]

On Fri, Apr 02, 2010 at 02:37:40PM -0700, Learner Study wrote:
> I have seen ~180MB/s RAID5 performance with 4 disks...are you saying
> that I could achieve even higher if I have more number of disks (so
> instead of 3+1, try 6+1 or 9+1)?
> Logically, this sounds right but wanted to verify my thought process
> with you....

Yes, with more spindles you can generally expect more performance.
Beware of bottlenecks, tho.

Best regards
keld

> Thanks!
> 
> On Fri, Apr 2, 2010 at 2:14 PM, Keld Simonsen <keld@keldix.com> wrote:
> > On Fri, Apr 02, 2010 at 10:55:53AM -0700, Learner Study wrote:
> >> Hi Keld:
> >>
> >> Thanks for your email...
> >>
> >> 1. Can you pls point me to this benchmark (which shows 500MB/s)? I
> >> would like to know which CPU, HDDs and kernel version used to achieve
> >> this...
> >
> > http://home.comcast.net/~jpiszcz/20080329-raid/
> > 496843   KB/s for sequential input with 10 raptor drives
> > There probably is an email in the archives with more info on the
> > test.
> >
> >> 2. Secondly, I would like to understand how raid stack (md driver)
> >> scales as we add more cores...if single core gives ~500MB/s, can two
> >> core give ~1000MB/s? can four cores give ~2000MB/s? etc....
> >
> > No, the performance is normally limited by the number of drives.
> > I would not wsay that more cores woould do a little
> > but it would be in the order of 1-2 % I think.
> > This is also dependent on wheteher the code actually runs threaded.
> > I doubt it....
> >
> > best regard
> > keld
> >
> >>
> >> Thanks for your time.
> >>
> >> On Fri, Apr 2, 2010 at 4:05 AM, Keld Simonsen <keld@keldix.com> wrote:
> >> > On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
> >> >> Hi Keld:
> >> >>
> >> >> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
> >> >> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
> >> >> single core CPUs. How does that scale with multiple cores? Something
> >> >> like intel's jasper forest???
> >> >
> >> > I have not checked if the benchmarks were on multi core machines.
> >> > It should not matter much if there were more than one CPU, but
> >> > of cause it helps a little. bonnie++ test reports cpu usage, and this
> >> > is not insignificant, say in the 20 -60 % range for some tests,
> >> > but nowhere near a bottleneck. There was one with a raid5 performance
> >> > seq read of about 500 MB/s with 36 % cpu utilization, so it is
> >> > definitely possible to come beyound 150 MB/s. The speed is largely
> >> > dependent on number of disk drives you employ.
> >> >
> >> >
> >> >> If available, can u pls point me to numbers with multi-core CPU?
> >> >
> >> > I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
> >> > so please feel free to submit your findings.
> >> >
> >> > Best regards
> >> > keld
> >> >
> >> >> Thanks!
> >> >>
> >> >> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
> >> >> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
> >> >> >> Hi Linux Raid Experts:
> >> >> >>
> >> >> >> I was looking at following wiki on raid perf on linux:
> >> >> >>
> >> >> >> https://raid.wiki.kernel.org/index.php/Performance
> >> >> >>
> >> >> >> and notice that the performance numbers are with 2.6.12 kernel.
> >> >> >>
> >> >> >> Do we perf numbers for:
> >> >> >> - latest kernel (something like 2.6.27 / 2.6.31)
> >> >> >> - raid 5 and 6
> >> >> >>
> >> >> >> Can someone please point me to appropriate link?
> >> >> >
> >> >> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
> >> >> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
> >> >> > slower since 2.6.12 on RAID.
> >> >> >
> >> >> > best regards
> >> >> > Keld
> >> >> >
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Learner Study]

Can you please throw light on what kind of bottlenecks that may impact perf....

Thanks!

On Sat, Apr 3, 2010 at 4:20 AM, Keld Simonsen <keld@keldix.com> wrote:
> On Fri, Apr 02, 2010 at 02:37:40PM -0700, Learner Study wrote:
>> I have seen ~180MB/s RAID5 performance with 4 disks...are you saying
>> that I could achieve even higher if I have more number of disks (so
>> instead of 3+1, try 6+1 or 9+1)?
>> Logically, this sounds right but wanted to verify my thought process
>> with you....
>
> Yes, with more spindles you can generally expect more performance.
> Beware of bottlenecks, tho.
>
> Best regards
> keld
>
>> Thanks!
>>
>> On Fri, Apr 2, 2010 at 2:14 PM, Keld Simonsen <keld@keldix.com> wrote:
>> > On Fri, Apr 02, 2010 at 10:55:53AM -0700, Learner Study wrote:
>> >> Hi Keld:
>> >>
>> >> Thanks for your email...
>> >>
>> >> 1. Can you pls point me to this benchmark (which shows 500MB/s)? I
>> >> would like to know which CPU, HDDs and kernel version used to achieve
>> >> this...
>> >
>> > http://home.comcast.net/~jpiszcz/20080329-raid/
>> > 496843   KB/s for sequential input with 10 raptor drives
>> > There probably is an email in the archives with more info on the
>> > test.
>> >
>> >> 2. Secondly, I would like to understand how raid stack (md driver)
>> >> scales as we add more cores...if single core gives ~500MB/s, can two
>> >> core give ~1000MB/s? can four cores give ~2000MB/s? etc....
>> >
>> > No, the performance is normally limited by the number of drives.
>> > I would not wsay that more cores woould do a little
>> > but it would be in the order of 1-2 % I think.
>> > This is also dependent on wheteher the code actually runs threaded.
>> > I doubt it....
>> >
>> > best regard
>> > keld
>> >
>> >>
>> >> Thanks for your time.
>> >>
>> >> On Fri, Apr 2, 2010 at 4:05 AM, Keld Simonsen <keld@keldix.com> wrote:
>> >> > On Thu, Apr 01, 2010 at 08:07:25PM -0700, Learner Study wrote:
>> >> >> Hi Keld:
>> >> >>
>> >> >> Do we have raid5/6 numbers for linux on any multi-core CPU? Most of
>> >> >> the benchmarks I have seen on wiki show raid5 perf to be ~150MB/s with
>> >> >> single core CPUs. How does that scale with multiple cores? Something
>> >> >> like intel's jasper forest???
>> >> >
>> >> > I have not checked if the benchmarks were on multi core machines.
>> >> > It should not matter much if there were more than one CPU, but
>> >> > of cause it helps a little. bonnie++ test reports cpu usage, and this
>> >> > is not insignificant, say in the 20 -60 % range for some tests,
>> >> > but nowhere near a bottleneck. There was one with a raid5 performance
>> >> > seq read of about 500 MB/s with 36 % cpu utilization, so it is
>> >> > definitely possible to come beyound 150 MB/s. The speed is largely
>> >> > dependent on number of disk drives you employ.
>> >> >
>> >> >
>> >> >> If available, can u pls point me to numbers with multi-core CPU?
>> >> >
>> >> > I dont have such benchmarks AFAIK. But new benchmarks are always welcome,
>> >> > so please feel free to submit your findings.
>> >> >
>> >> > Best regards
>> >> > keld
>> >> >
>> >> >> Thanks!
>> >> >>
>> >> >> On Wed, Mar 31, 2010 at 1:15 PM, Keld Simonsen <keld@keldix.com> wrote:
>> >> >> > On Wed, Mar 31, 2010 at 12:42:57PM -0700, Learner Study wrote:
>> >> >> >> Hi Linux Raid Experts:
>> >> >> >>
>> >> >> >> I was looking at following wiki on raid perf on linux:
>> >> >> >>
>> >> >> >> https://raid.wiki.kernel.org/index.php/Performance
>> >> >> >>
>> >> >> >> and notice that the performance numbers are with 2.6.12 kernel.
>> >> >> >>
>> >> >> >> Do we perf numbers for:
>> >> >> >> - latest kernel (something like 2.6.27 / 2.6.31)
>> >> >> >> - raid 5 and 6
>> >> >> >>
>> >> >> >> Can someone please point me to appropriate link?
>> >> >> >
>> >> >> > The link mentioned above has a number of other performance reports, for other levels of the kernel.
>> >> >> > Anyway you should be able to get comparable results for newer kernels, the kernel has not become
>> >> >> > slower since 2.6.12 on RAID.
>> >> >> >
>> >> >> > best regards
>> >> >> > Keld
>> >> >> >
>> >> >> --
>> >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> >
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[MRK]

Richard Scobie wrote:
> Mark Knecht wrote:
>
>> Once all of that is in place then possibly more cores will help, but I
>> suspect even then it probably hard to use 4 billion CPU cycles/second
>> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
>> overhead is relatively *very* low.
>
> There is the RAID5/6 parity calculations to be considered on writes 
> and this appears to be single threaded. There is an experimental 
> multicore kernel option I believe, but recent discussion indicates 
> there may be some problems with it.
>
> A very quick test on a box here on a Xeon E5440 (4 x 2.8GHz) and a SAS 
> attached 16 x 750GB SATA md RAID6. The array is 72% full and probably 
> quite fragmented and currently the system is idle.
>
> dd if=/dev/zero of=/mnt/storage/dump bs=1M count=20000
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 87.2374 s, 240 MB/s
>
> Looking at the outputs of vmstat 5 and mpstat -P ALL 5 during this, 
> one core (probably doing parity generation) was around 7.56% idle and 
> the other 3 were around 88.5, 67.5 and 51.8% idle.
>
> The same test run when the system was commissioned and the array was 
> empty, acheived 565MB/s writes.

I was able to achieve about 430MB/sec on a 24 disks raid-6 with dd on an 
XFS filesystem which was 70% full. I don't think it made great 
difference even if it was empty. It was a 54xx Xeon CPU.
I spent some time trying to optimize it but that was the best I could 
get. Anyway both my benchmark and Richard's one imply a very significant 
bottleneck somehwere.
16 SATA disks have aggregated I/O streaming performance of about 
1.4GB/sec so getting 500MB/sec it's 3 times slower.
Raid-0 does not have this problem: there is an old post of Mark Delfman 
on this ML in which he was able to obtain about 1.7GB/sec with 10 SAS 
disks (15Krpm) in RAID-0, which is much higher than 500MB/s and it's 
about the bare disk speed.
I always thought the reason of the slower raid 5/6 was the parity 
computation but now that Nicolae has pointed out that the parity 
computation speed is so high, the reason must be elsewhere.
Could that be RAM I/O? Raid 5/6 copies data, then probably reads it 
again for the parity computation and then writes the parity out... the 
CPU cache is too small to hold a stripe for large arrays so it's at 
least 3 RAM accesses but yet it should be way faster than this imho.

MRK

Re: Linux Raid performance

[Richard Scobie]

Learner Study wrote:

> What kernel version did you use for this test?

2.6.27.19-78.2.30.fc9.x86_64

Regards,

Richard

Re: Linux Raid performance

[Richard Scobie]

MRK wrote:

> I spent some time trying to optimize it but that was the best I could 
> get. Anyway both my benchmark and Richard's one imply a very significant 
> bottleneck somehwere.

This bottleneck is the SAS controller, at least in my case. I did the 
same math regarding streaming performance of one drive times number of 
drive and wondered where the shortfall was, after tests showed I could 
only streaming read at 850MB/s on the same array.

A query to an LSI engineer got the following response, which basically 
boils down to "you get what you pay for" - SAS vs SATA drives.

"Yes, you're at the "practical" limit.

With that setup and SAS disks, you will exceed 1200 MB/s.  Could go
higher than 1,400 MB/s given the right server chipset.

However with SATA disks, and the way they break up data transfers, 815
to 850 MB/s is the best you can do.

Under SATA, there are multiple connections per I/O request.
   * Command Initiator -> HDD
   * DMA Setup  Initiator -> HDD
   * DMA Activate  HDD -> Initiator
   * Data   HDD -> Initiator
   * Status    HDD -> Initiator
And there is little ability with typical SATA disks to combine traffic
from different I/Os on the same connection.  So you get lots of
individual connections being made, used, & broken.

Contrast that with SAS which has typically 2 connections per I/O, and
will combine traffic from more than 1 I/O per connection.  It uses the
SAS links much more efficiently."


Regards,

Richard

Re: Linux Raid performance

[Keld Simonsen]

On Sat, Apr 03, 2010 at 08:56:59AM -0700, Learner Study wrote:
> Can you please throw light on what kind of bottlenecks that may impact perf....

See for example https://raid.wiki.kernel.org/index.php/Performance#Bottlenecks

best regards
keld

Re: Linux Raid performance

[MRK]

Richard Scobie wrote:
> MRK wrote:
>
>> I spent some time trying to optimize it but that was the best I could 
>> get. Anyway both my benchmark and Richard's one imply a very 
>> significant bottleneck somehwere.
>
> This bottleneck is the SAS controller, at least in my case. I did the 
> same math regarding streaming performance of one drive times number of 
> drive and wondered where the shortfall was, after tests showed I could 
> only streaming read at 850MB/s on the same array.
>
> A query to an LSI engineer got the following response, which basically 
> boils down to "you get what you pay for" - SAS vs SATA drives.
>
> "Yes, you're at the "practical" limit.
>
> With that setup and SAS disks, you will exceed 1200 MB/s.  Could go
> higher than 1,400 MB/s given the right server chipset.
>
> However with SATA disks, and the way they break up data transfers, 815
> to 850 MB/s is the best you can do.
>
> Under SATA, there are multiple connections per I/O request.
>   * Command Initiator -> HDD
>   * DMA Setup  Initiator -> HDD
>   * DMA Activate  HDD -> Initiator
>   * Data   HDD -> Initiator
>   * Status    HDD -> Initiator
> And there is little ability with typical SATA disks to combine traffic
> from different I/Os on the same connection.  So you get lots of
> individual connections being made, used, & broken.
>
> Contrast that with SAS which has typically 2 connections per I/O, and
> will combine traffic from more than 1 I/O per connection.  It uses the
> SAS links much more efficiently."

Firstly: Happy Easter!  :-)

Secondly:

If this is true then one won't achieve higher speeds even on RAID-0. If 
anybody can test this... I cannot right now

I am a bit surprised though. The SATA "link" is one per drive, so if 1 
drive is able to do 90MB/sec, N drives on N cables should do Nx90MB/sec.
If this is not so, then the chipset of the controller must be the 
bottleneck.
If this is so, the newer LSI controllers at 6.0gbit/sec could be able to 
do better (they supposedly have a faster chip). Also maybe one could buy 
more controller cards and divide drives among those. These two 
workarounds would still be cheaper than SAS drives.

Re: Linux Raid performance

[Learner Study]

Happy Easter!!!

So, 550-600MB/s is the best we have seen with Linux raid using 16-24 SAS drives.

Not sure if its appropriate to ask on this list - has someone seen
better numbers with non-linux raid stack? Perhaps freebsd/lustre..

Thanks for your time!

On Sun, Apr 4, 2010 at 8:00 AM, MRK <mrk@shiftmail.org> wrote:
> Richard Scobie wrote:
>>
>> MRK wrote:
>>
>>> I spent some time trying to optimize it but that was the best I could
>>> get. Anyway both my benchmark and Richard's one imply a very significant
>>> bottleneck somehwere.
>>
>> This bottleneck is the SAS controller, at least in my case. I did the same
>> math regarding streaming performance of one drive times number of drive and
>> wondered where the shortfall was, after tests showed I could only streaming
>> read at 850MB/s on the same array.
>>
>> A query to an LSI engineer got the following response, which basically
>> boils down to "you get what you pay for" - SAS vs SATA drives.
>>
>> "Yes, you're at the "practical" limit.
>>
>> With that setup and SAS disks, you will exceed 1200 MB/s.  Could go
>> higher than 1,400 MB/s given the right server chipset.
>>
>> However with SATA disks, and the way they break up data transfers, 815
>> to 850 MB/s is the best you can do.
>>
>> Under SATA, there are multiple connections per I/O request.
>>  * Command Initiator -> HDD
>>  * DMA Setup  Initiator -> HDD
>>  * DMA Activate  HDD -> Initiator
>>  * Data   HDD -> Initiator
>>  * Status    HDD -> Initiator
>> And there is little ability with typical SATA disks to combine traffic
>> from different I/Os on the same connection.  So you get lots of
>> individual connections being made, used, & broken.
>>
>> Contrast that with SAS which has typically 2 connections per I/O, and
>> will combine traffic from more than 1 I/O per connection.  It uses the
>> SAS links much more efficiently."
>
> Firstly: Happy Easter!  :-)
>
> Secondly:
>
> If this is true then one won't achieve higher speeds even on RAID-0. If
> anybody can test this... I cannot right now
>
> I am a bit surprised though. The SATA "link" is one per drive, so if 1 drive
> is able to do 90MB/sec, N drives on N cables should do Nx90MB/sec.
> If this is not so, then the chipset of the controller must be the
> bottleneck.
> If this is so, the newer LSI controllers at 6.0gbit/sec could be able to do
> better (they supposedly have a faster chip). Also maybe one could buy more
> controller cards and divide drives among those. These two workarounds would
> still be cheaper than SAS drives.
>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[Mark Knecht]

On Sun, Apr 4, 2010 at 11:26 AM, Learner Study <learner.study@gmail.com> wrote:
> Happy Easter!!!
>
> So, 550-600MB/s is the best we have seen with Linux raid using 16-24 SAS drives.
>
> Not sure if its appropriate to ask on this list - has someone seen
> better numbers with non-linux raid stack? Perhaps freebsd/lustre..
>
> Thanks for your time!
>

Are you just trolling or something? Why are you asking about non-linux
RAID on a Linux software RAID list?

I cannot believe the name Learner Study is real, is it? Possibly it
would make sense to first study, and from that hopefully learn, and
then come ask some questions with some depth to them?

- Mark

Re: Linux Raid performance

[Jools Wills]

On Sun, 2010-04-04 at 11:46 -0700, Mark Knecht wrote:

> Are you just trolling or something? Why are you asking about non-linux
> RAID on a Linux software RAID list?

Don't want to start any flame war, but I had no problem with the mail,
and it seemed a relevant question. After all, we need to compare against
other systems to remain on top right? to strive to be the best?

If platform A and B is doing XYZ in terms of raid, then it is surely up
for discussion. I think such attitudes like this make people think we
are fanatical, rather than educated developers trying to improve things.

Best Regards

Jools

Jools Wills
-- 
IT Consultant
Oxford Inspire - http://www.oxfordinspire.co.uk - be inspired
t: 01235 519446 m: 07966 577498
jools@oxfordinspire.co.uk

RE: Linux Raid performance

[Guy Watkins]

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Mark Knecht
} Sent: Sunday, April 04, 2010 2:47 PM
} To: Learner Study
} Cc: linux-raid@vger.kernel.org
} Subject: Re: Linux Raid performance
} 
} On Sun, Apr 4, 2010 at 11:26 AM, Learner Study <learner.study@gmail.com>
} wrote:
} > Happy Easter!!!
} >
} > So, 550-600MB/s is the best we have seen with Linux raid using 16-24 SAS
} drives.
} >
} > Not sure if its appropriate to ask on this list - has someone seen
} > better numbers with non-linux raid stack? Perhaps freebsd/lustre..
} >
} > Thanks for your time!
} >
} 
} Are you just trolling or something? Why are you asking about non-linux
} RAID on a Linux software RAID list?
} 
} I cannot believe the name Learner Study is real, is it? Possibly it
} would make sense to first study, and from that hopefully learn, and
} then come ask some questions with some depth to them?
} 
} - Mark

This seemed like a fair question to me.  I just wonder why Winblows was left
out.  After all, if any OS, including Winblows can outperform Linux, we
should know about it.

Maybe you are afraid the other operating systems out perform Linux?

Re: Linux Raid performance

[Mark Knecht]

On Sun, Apr 4, 2010 at 2:28 PM, Jools Wills <jools@oxfordinspire.co.uk> wrote:
> On Sun, 2010-04-04 at 11:46 -0700, Mark Knecht wrote:
>
>> Are you just trolling or something? Why are you asking about non-linux
>> RAID on a Linux software RAID list?
>
> Don't want to start any flame war, but I had no problem with the mail,
> and it seemed a relevant question. After all, we need to compare against
> other systems to remain on top right? to strive to be the best?
>
> If platform A and B is doing XYZ in terms of raid, then it is surely up
> for discussion. I think such attitudes like this make people think we
> are fanatical, rather than educated developers trying to improve things.
>
> Best Regards
>
> Jools
>
> Jools Wills
> --
> IT Consultant
> Oxford Inspire - http://www.oxfordinspire.co.uk - be inspired
> t: 01235 519446 m: 07966 577498
> jools@oxfordinspire.co.uk
>
>

You are right. There wasn't anything wrong with the questionas asked
and it was not my intent to start any wars. My apologies if it sounded
that way.

My reading of this thread in total has been pretty much that the OP
doesn't know much about RAID (fair enough), or PC architecture (fair
enough) and wants to ask little questions over and over to get
information without doing much work on his own. (possibly not so
fair.) I should just shut my mouth and let the messages go by which I
will do with this individual from here on out.

My apologies,
Mark

Re: Linux Raid performance

[Richard Scobie]

MRK wrote:

> If this is so, the newer LSI controllers at 6.0gbit/sec could be able to
> do better (they supposedly have a faster chip). Also maybe one could buy
> more controller cards and divide drives among those. These two

Yes, both of these would work.

Someone posted previously on this list and was writing at 1.7GB/s using 
10 x 15K SAS drives md RAID0. He did mention the troughput was higher 
with the LSI SAS2 cards, even with SAS1 port expanders connected.

Regards,

Richard

Re: Linux Raid performance

[Learner Study]

Hi Mark, all:

My apologies if I sounded like a pest by asking too many
questions...Since I don't have access to array of disks, I sought to
the list to see if I could use the info already out there instead of
reinventing it...how well suited is linux raid for 10G kind of
traffic..I don't have resources to check it myself and thought of
asking for help/guidance!

Thanks!

On Sun, Apr 4, 2010 at 3:38 PM, Mark Knecht <markknecht@gmail.com> wrote:
> On Sun, Apr 4, 2010 at 2:28 PM, Jools Wills <jools@oxfordinspire.co.uk> wrote:
>> On Sun, 2010-04-04 at 11:46 -0700, Mark Knecht wrote:
>>
>>> Are you just trolling or something? Why are you asking about non-linux
>>> RAID on a Linux software RAID list?
>>
>> Don't want to start any flame war, but I had no problem with the mail,
>> and it seemed a relevant question. After all, we need to compare against
>> other systems to remain on top right? to strive to be the best?
>>
>> If platform A and B is doing XYZ in terms of raid, then it is surely up
>> for discussion. I think such attitudes like this make people think we
>> are fanatical, rather than educated developers trying to improve things.
>>
>> Best Regards
>>
>> Jools
>>
>> Jools Wills
>> --
>> IT Consultant
>> Oxford Inspire - http://www.oxfordinspire.co.uk - be inspired
>> t: 01235 519446 m: 07966 577498
>> jools@oxfordinspire.co.uk
>>
>>
>
> You are right. There wasn't anything wrong with the questionas asked
> and it was not my intent to start any wars. My apologies if it sounded
> that way.
>
> My reading of this thread in total has been pretty much that the OP
> doesn't know much about RAID (fair enough), or PC architecture (fair
> enough) and wants to ask little questions over and over to get
> information without doing much work on his own. (possibly not so
> fair.) I should just shut my mouth and let the messages go by which I
> will do with this individual from here on out.
>
> My apologies,
> Mark
>

Re: Linux Raid performance

[MRK]

Richard Scobie wrote:
> MRK wrote:
>
>> If this is so, the newer LSI controllers at 6.0gbit/sec could be able to
>> do better (they supposedly have a faster chip). Also maybe one could buy
>> more controller cards and divide drives among those. These two
>
> Yes, both of these would work.
>
> Someone posted previously on this list and was writing at 1.7GB/s 
> using 10 x 15K SAS drives md RAID0. He did mention the troughput was 
> higher with the LSI SAS2 cards, even with SAS1 port expanders connected.

Not so fast... actually I see a problem with previous deduction of what 
is the bottleneck.

The answer from the LSI engineer leads to think that the bottleneck with 
SATA is the number of IOPS, it's because there are 5 connections 
established and then broken for each I/O. And this is independent from 
the size transferred by each I/O operation via DMA (the overhead of the 
data transfer is the same in SAS and SATA case, it's always the same DMA 
chip doing the transfer).

However, if really the total number of IOPS is the bottleneck in SATA 
with the 3.0gbit/sec LSI cards, why they don't slow down a single SSD 
doing 4k random I/O?

Look at this
http://www.anandtech.com/show/2954/5
OCZ vertex LE doing 162 MB/sec at 4K aligned random writes, that means 
41472 IOPS, independent and unmergeable requests. And that is SATA not SAS.
This is on Windows. Unfortunately we don't know the controller which was 
used for this benchmark.

During MD-RAID sequential dd write I have seen linux (via iostat -x 1) 
merging requests by a factor at least 400 (sometimes much higher), so I 
suppose requests issued to the controller would be at least 1.6 MB long 
(original requests are certainly not shorter than 4K, and 4K x 400=1.6MB).

If the system tops out at about 600MB/sec and writes issued are 1.6MB 
long or more, it means that the controller tops out at 375 IOPS or less.

So how come the controller of the anandtech test above is capable of 
doing 41472 IOPS?


This is also interesting:

Richard Scobie wrote:
> This bottleneck is the SAS controller, at least in my case. I did the 
> same math regarding streaming performance of one drive times number of 
> drive and wondered where the shortfall was, after tests showed I could 
> only streaming read at 850MB/s on the same array. 
I think if you use dd to read from the 16 underlying devices 
simultaneously, independently, and not using MD, (output to /dev/null) 
you should obtain the full disk speed of 1.4 GB/sec or so (aggregated). 
I think I did this test in the past and I noticed this. Can you try? I 
don't have our big disk array in my hands any more :-(

Re: Linux Raid performance

[Drew]

> This seemed like a fair question to me.  I just wonder why Winblows was left
> out.  After all, if any OS, including Winblows can outperform Linux, we
> should know about it.

Because M$ Windows implementation of software raid sucks?

As part of my job I support windows servers and almost every 3rd party
manual I've ever read says something along the lines of "If your data
is mission critical, do not depend on windows built-in software raid.
Get a real hardware raid card." Some then go on to explain all the
pitfalls if a user is still inclined to use it, and some of the issues
are scary.

Contrast that to the reports I see on here of people using 6+, 10+,
12+ member raid arrays on mdadm (software) raid and having no issues
other then perhaps some performance tweaking.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux Raid performance

[John Robinson]

On 05/04/2010 11:07, Learner Study wrote:
> Hi Mark, all:
> 
> My apologies if I sounded like a pest by asking too many
> questions...

Don't worry about it, but do take care - I'm afraid I was similarly 
sceptical about the email alias you've chosen, and got the impression 
from some of your questions that you might be a kid asking people here 
to do your homework for you, which does happen from time to time.

> Since I don't have access to array of disks, I sought to
> the list to see if I could use the info already out there instead of
> reinventing it...how well suited is linux raid for 10G kind of
> traffic..

If you mean 10G bits per second, or at least enough for a 10GigE 
network, then yes I think it is, as evidenced by people here saying 
they've got up to 1400MB/s, as long as you've got enough really fast SAS 
drives hooked up via high-class SAS interfaces over wide PCI Express 
buses to Xeon CPUs with fast RAM, though none of this is exactly 
commodity-class hardware.

As far as I know, 10G bytes per second is beyond PC hardware at the 
moment, and I have no knowledge or experience of the kinds of hardware 
that could reach those speeds.

You got me thinking about the bottlenecks, though. Just for fun (so you 
can see what an odd idea of fun I have), I just restarted my big storage 
server box (Core 2 Quad 3.2GHz on 1600MHz FSB, dual-channel DDR2-800 
memory, Intel P45/ICH10R motherboard) into memtest86+, and it's telling 
me that my memory bandwidth is 4.5GB/s. That's probably about the limit 
of this architecture, though the newer Core i5/7 chips with dual- and 
triple-channel memory controllers integrated can probably manage more.

Now, if I'm running RAID-5 or 6, and Linux md has to copy every page I 
want to write to disc (therefore read it from RAM and write it back 
again), and then read it again to calculate the parity block(s), then 
the SATA/SAS controllers have to read it, that's at least 4 memory 
operations for every write to disc, which is going to mean there's a 
memory subsystem bottleneck/throttle on this machine of about 
1.125GB/sec. I skipped over the bit where I have to write the parity to 
RAM and also have the SATA/SAS controllers read that to write it to the 
disc, but that's going to make a fairly modest difference, maybe down to 
1GB/sec. Since my CPU can calculate RAID-5 or 6 parity on a single core 
at 8GB/sec, that's not the bottleneck.

And I do understand that I'd be rather optimistic to think my RAM 
bandwidth was as much as 4.5GB/s for much of the time.

Reads should be quicker though :-)

Cheers,

John.

Re: Linux Raid performance

[Richard Scobie]

MRK wrote:

> However, if really the total number of IOPS is the bottleneck in SATA
> with the 3.0gbit/sec LSI cards, why they don't slow down a single SSD
> doing 4k random I/O?

We don't know, as we have no information for one of these SSDs attached 
to an LSI SAS controller.

I'm not sure this is an apples to apples comparison. The SSD is one 
device probably connected  to a motherboard SATA controller channel.

The RAID array is 16 devices attached to a port expander in turn 
attached to a SAS controller. At a most simplistic level, surely the SAS 
controller has overhead attached to which drive is being addressed.


> I think if you use dd to read from the 16 underlying devices
> simultaneously, independently, and not using MD, (output to /dev/null)
> you should obtain the full disk speed of 1.4 GB/sec or so (aggregated).
> I think I did this test in the past and I noticed this. Can you try? I
> don't have our big disk array in my hands any more :-(

I'll bear it in mind next time I am in a position to try it.

Regards,

Richard

Re: Linux Raid performance

[Drew]

> The RAID array is 16 devices attached to a port expander in turn attached to
> a SAS controller. At a most simplistic level, surely the SAS controller has
> overhead attached to which drive is being addressed.

Don't forget that with a port expander is still limited to the bus
speed of the link between it and the HBA. It doesn't matter how many
drives you hang off an expander, you will still never exceed the rated
speed (1.5/3/6Gb/s) for that one port on the HBA.

Say you have four drives behind an expander on a 6Gb/s link. Each
drive in the array could still bonnie++ at the full 6Gb/s but once you
try to write to all four drives simultaneously (RAID-5/6), the best
you can get out of each is around 1.5Gb/s.

That's why I don't use expanders except for archival SATA drives,
which AFAIK only go one expander deep. The performance penalty isn't
worth the cost savings in my books.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: Linux Raid performance

[Richard Scobie]

Drew wrote:
>> The RAID array is 16 devices attached to a port expander in turn attached to
>> a SAS controller. At a most simplistic level, surely the SAS controller has
>> overhead attached to which drive is being addressed.
>
> Don't forget that with a port expander is still limited to the bus
> speed of the link between it and the HBA. It doesn't matter how many
> drives you hang off an expander, you will still never exceed the rated
> speed (1.5/3/6Gb/s) for that one port on the HBA.

I'm not sure if you are thinking of a port multiplier instead of a port 
expander.

In any case, in my setup, there are 4 x 3Gb/s lanes connecting the HBA 
to the port expander and each drive is connected to it's own port on the 
expander at 3Gb/s ( obviously each drive is not streaming at 3Gb/s).

So there is plenty of bandwidth there.

> Say you have four drives behind an expander on a 6Gb/s link. Each
> drive in the array could still bonnie++ at the full 6Gb/s but once you
> try to write to all four drives simultaneously (RAID-5/6), the best
> you can get out of each is around 1.5Gb/s.

4 x 100MB/s (average outer edge speed of SATA drive) = 3.2Gb/s which is 
less than 6Gb and works out to 0.8Gb/s to each drive, so I'm not sure 
what you mean here.

Regards,

Richard

Re: Linux Raid performance

[Roger Heflin]

Drew wrote:
>> The RAID array is 16 devices attached to a port expander in turn attached to
>> a SAS controller. At a most simplistic level, surely the SAS controller has
>> overhead attached to which drive is being addressed.
> 
> Don't forget that with a port expander is still limited to the bus
> speed of the link between it and the HBA. It doesn't matter how many
> drives you hang off an expander, you will still never exceed the rated
> speed (1.5/3/6Gb/s) for that one port on the HBA.

If it is a SAS connect to the RAID array, they are often quad channel 
cables (12Gbits/second-ie 4x3Gbps), this is what is on the external 
connection of the card, not a single channel sas/sata like the lower 
end stuff, and most of the more expensive expanders and raid cabinents 
use that.

Still the entire 16disk setup will be limited to be less that whatever 
the interconnect is, and if you start piling more than 16 disks onto 
it things get pretty messy pretty fast.

> 
> Say you have four drives behind an expander on a 6Gb/s link. Each
> drive in the array could still bonnie++ at the full 6Gb/s but once you
> try to write to all four drives simultaneously (RAID-5/6), the best
> you can get out of each is around 1.5Gb/s.
> 
> That's why I don't use expanders except for archival SATA drives,
> which AFAIK only go one expander deep. The performance penalty isn't
> worth the cost savings in my books.
> 
> 

Re: Linux Raid performance

[Bill Davidsen]

Richard Scobie wrote:
> Mark Knecht wrote:
>
>> Once all of that is in place then possibly more cores will help, but I
>> suspect even then it probably hard to use 4 billion CPU cycles/second
>> doing nothing but disk I/O. SATA controllers are all doing DMA so CPU
>> overhead is relatively *very* low.
>
> There is the RAID5/6 parity calculations to be considered on writes 
> and this appears to be single threaded. There is an experimental 
> multicore kernel option I believe, but recent discussion indicates 
> there may be some problems with it.

That is being polite. With that option set just doing a 'check' on a 
raid-5 will generate 100s of threads and max out all cores. I was trying 
to run the experimental FC13 64 bit kernel, and all of a sudden the 
machine came to a crawl and the cpu use went to 95%+ on all cores. Also 
drove the CPU temp way up, so I have to regard this as unsuitable for 
anything but light testing.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein