Known-good LSI PCIe SAS HBAs?

Known-good LSI PCIe SAS HBAs?

[Kristleifur Daðason]

Hi,

I'm looking to replace a problematic Marvell PCIe SAS JBOD controller
from Supermicro, type AOC-SASLP-MV8. (mdadm freaks out on it and the
controller drops the drives - there has been discussion of the problem
on the list. Seems to be a mvsas driver problem but seems to surface
only under mdadm use.)

It's a nice controller, but we have to get an array going, so I'm
shelving the Marvell card and getting a new one. I'll be using mdadm
raid5 / raid6 over the controller.

This one, LSI SAS 3081E-R, looks good with eight SAS ports for around $200:
http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/internal/sas3081e-r/index.html

I can't find anything concrete regarding Linux, but I did find a
known-good confirmation for Solaris/ZFS here:
http://jmlittle.blogspot.com/2008/06/recommended-disk-controllers-for-zfs.html

Can anyone confirm/deny that the PCIe JBOD controllers from LSI work
on Linux out of the box? Or give other recommendations for 8 port
controllers in the $150-$200 range? (Or thereabouts. It'd be great to
know about any known-good 4-port controllers for even up to $250.)

Thanks!

-- Kristleifur

Re: Known-good LSI PCIe SAS HBAs?

[Richard Scobie]

Kristleifur Daðason wrote:

> Can anyone confirm/deny that the PCIe JBOD controllers from LSI work
> on Linux out of the box? Or give other recommendations for 8 port
> controllers in the $150-$200 range? (Or thereabouts. It'd be great to
> know about any known-good 4-port controllers for even up to $250.)

You should have no problem at all. I have used a number of the:

http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/combo/sas3442e-r/index.html

controllers.

The only thing to watch out for, as has been discussed in the last few 
days, is that use of smartmontools to monitor disk health on these 1068 
based cards, wll cause drives to be offlined, when used with recent kernels.

Regards,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Known-good LSI PCIe SAS HBAs?

[Thomas Fjellstrom]

On Tue November 10 2009, you wrote:
> Hi,
> 
> I'm looking to replace a problematic Marvell PCIe SAS JBOD controller
> from Supermicro, type AOC-SASLP-MV8. (mdadm freaks out on it and the
> controller drops the drives - there has been discussion of the problem
> on the list. Seems to be a mvsas driver problem but seems to surface
> only under mdadm use.)
> 
> It's a nice controller, but we have to get an array going, so I'm
> shelving the Marvell card and getting a new one. I'll be using mdadm
> raid5 / raid6 over the controller.
> 
> This one, LSI SAS 3081E-R, looks good with eight SAS ports for around
>  $200:
>  http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas
> /internal/sas3081e-r/index.html
> 
> I can't find anything concrete regarding Linux, but I did find a
> known-good confirmation for Solaris/ZFS here:
> http://jmlittle.blogspot.com/2008/06/recommended-disk-controllers-for-zfs
> .html
> 
> Can anyone confirm/deny that the PCIe JBOD controllers from LSI work
> on Linux out of the box? Or give other recommendations for 8 port
> controllers in the $150-$200 range? (Or thereabouts. It'd be great to
> know about any known-good 4-port controllers for even up to $250.)
> 
> Thanks!

Not really an answer to your question, I got a snapshot of the in 
development driver that seems to work just fine. I haven't run smartctl on 
the controller much mind you, as it seems to freak out my onboard controller 
too (AMD SB750). I could attempt it, but I've recently just switched over to 
that hardware for my main server, and I'm reluctant to try running smartctl 
on the array, as it'd probably require a reboot to get which ever disk back 
that might get dropped.

I might try it later, but I don't even have smartmontools installed.

> -- Kristleifur
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Known-good LSI PCIe SAS HBAs?

[Thomas Fjellstrom]

On Tue November 10 2009, Richard Scobie wrote:
> Kristleifur Daðason wrote:
> > Can anyone confirm/deny that the PCIe JBOD controllers from LSI work
> > on Linux out of the box? Or give other recommendations for 8 port
> > controllers in the $150-$200 range? (Or thereabouts. It'd be great to
> > know about any known-good 4-port controllers for even up to $250.)
> 
> You should have no problem at all. I have used a number of the:
> 
> http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/
> combo/sas3442e-r/index.html
> 
> controllers.
> 
> The only thing to watch out for, as has been discussed in the last few
> days, is that use of smartmontools to monitor disk health on these 1068
> based cards, wll cause drives to be offlined, when used with recent
>  kernels.
> 
> Regards,

Thats the problem he was trying to get away from. the MV8 controller likes 
to offline disks when a few smart commands arrive. at least with the main 
kernel drivers. But then the main kernel mvsas drivers decide to give up if 
you even attempt to start a raid array on them. The controller locks up and 
you have to reboot. Most of the time a reboot even fails when it tries to 
sync the block devices, so a hard reset is in order.

But as I mentioned in a previous message, the driver snapshot I got from a 
nice fellow seems to work fine now. I hope it gets into the kernel sometime 
soon.

> Richard
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Known-good LSI PCIe SAS HBAs?

[Kristleifur Daðason]

On Wed, Nov 11, 2009 at 7:01 AM, Thomas Fjellstrom <tfjellstrom@shaw.ca> wrote:
> On Tue November 10 2009, you wrote:
>> Hi,
>>
>> I'm looking to replace a problematic Marvell PCIe SAS JBOD controller
>> from Supermicro, type AOC-SASLP-MV8. (mdadm freaks out on it and the
>> controller drops the drives - there has been discussion of the problem
>> on the list. Seems to be a mvsas driver problem but seems to surface
>> only under mdadm use.)
>>
>>...
>
> Not really an answer to your question, I got a snapshot of the in
> development driver that seems to work just fine. I haven't run smartctl on
> the controller much mind you, as it seems to freak out my onboard controller
> too (AMD SB750). I could attempt it, but I've recently just switched over to
> that hardware for my main server, and I'm reluctant to try running smartctl
> on the array, as it'd probably require a reboot to get which ever disk back
> that might get dropped.
>

That's quite interesting.
Did you discuss this on a mailing list somewhere and have a link to
that discussion? Or would you be willing to forward a message to
honorable mr. driver guy? I'd love to try the patch before buying a
new controller.

Thanks!

Re: Known-good LSI PCIe SAS HBAs?

[Thomas Fjellstrom]

On Wed November 11 2009, Kristleifur Daðason wrote:
> On Wed, Nov 11, 2009 at 7:01 AM, Thomas Fjellstrom <tfjellstrom@shaw.ca> 
wrote:
> > On Tue November 10 2009, you wrote:
> >> Hi,
> >>
> >> I'm looking to replace a problematic Marvell PCIe SAS JBOD controller
> >> from Supermicro, type AOC-SASLP-MV8. (mdadm freaks out on it and the
> >> controller drops the drives - there has been discussion of the problem
> >> on the list. Seems to be a mvsas driver problem but seems to surface
> >> only under mdadm use.)
> >>
> >>...
> >
> > Not really an answer to your question, I got a snapshot of the in
> > development driver that seems to work just fine. I haven't run smartctl
> > on the controller much mind you, as it seems to freak out my onboard
> > controller too (AMD SB750). I could attempt it, but I've recently just
> > switched over to that hardware for my main server, and I'm reluctant to
> > try running smartctl on the array, as it'd probably require a reboot to
> > get which ever disk back that might get dropped.
> 
> That's quite interesting.
> Did you discuss this on a mailing list somewhere and have a link to
> that discussion? Or would you be willing to forward a message to
> honorable mr. driver guy? I'd love to try the patch before buying a
> new controller.

There seems to be something seriously wrong with the kernel.org mailing 
lists. They seem to miss a bunch of messages that get sent out. Its happened 
on several occasions now with not only my messages, but others as well, on 
several different mailing lists, including lkml, linux-raid, linux-ide, and 
linux-scsi.

The best I can do is a link to my reply to Andy Yan's message, since Andy 
Yan's message is mysteriously not in any of the archives. And I'm pretty 
sure I didn't get two copies so the one sent to the list's just disappeared.

http://lkml.org/lkml/2009/10/14/76

> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Known-good LSI PCIe SAS HBAs?

[Richard Scobie]

Thomas Fjellstrom wrote:

> Thats the problem he was trying to get away from. the MV8 controller likes 
> to offline disks when a few smart commands arrive. at least with the main 
> kernel drivers. But then the main kernel mvsas drivers decide to give up if 
> you even attempt to start a raid array on them. The controller locks up and 
> you have to reboot. Most of the time a reboot even fails when it tries to 
> sync the block devices, so a hard reset is in order.

OK, I did not know the Marvell controller also has a problem with 
smartmontools.

In his original post he mentions it having problems with mdadm:

"(mdadm freaks out on it and the controller drops the drives - there has 
been discussion of the problem on the list. Seems to be a mvsas driver 
problem but seems to surface only under mdadm use.)"

Regards,

Richard

Re: Known-good LSI PCIe SAS HBAs?

[Thomas Fjellstrom]

On Wed November 11 2009, Richard Scobie wrote:
> Thomas Fjellstrom wrote:
> > Thats the problem he was trying to get away from. the MV8 controller
> > likes to offline disks when a few smart commands arrive. at least with
> > the main kernel drivers. But then the main kernel mvsas drivers decide
> > to give up if you even attempt to start a raid array on them. The
> > controller locks up and you have to reboot. Most of the time a reboot
> > even fails when it tries to sync the block devices, so a hard reset is
> > in order.
> 
> OK, I did not know the Marvell controller also has a problem with
> smartmontools.

Given the state of the in kernel drivers, its probably a driver issue.

> In his original post he mentions it having problems with mdadm:
> 
> "(mdadm freaks out on it and the controller drops the drives - there has
> been discussion of the problem on the list. Seems to be a mvsas driver
> problem but seems to surface only under mdadm use.)"
> 
> Regards,
> 
> Richard
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Known-good LSI PCIe SAS HBAs?

[Kristleifur Daðason]

On Wed, Nov 11, 2009 at 6:51 PM, Thomas Fjellstrom <tfjellstrom@shaw.ca> wrote:
>
> On Wed November 11 2009, Richard Scobie wrote:
> > Thomas Fjellstrom wrote:
> > > Thats the problem he was trying to get away from. the MV8 controller
> > > likes to offline disks when a few smart commands arrive. at least with
> > > the main kernel drivers. But then the main kernel mvsas drivers decide
> > > to give up if you even attempt to start a raid array on them. The
> > > controller locks up and you have to reboot. Most of the time a reboot
> > > even fails when it tries to sync the block devices, so a hard reset is
> > > in order.
> >
> > OK, I did not know the Marvell controller also has a problem with
> > smartmontools.
>
> Given the state of the in kernel drivers, its probably a driver issue.
>

Thanks for the replies, all.

I'll bet it's the driver - it doesn't feel quite right that a random
external command can easily drop drives off the controller, be it
smartmontools or mdadm.

Thomas, I tried sending a private email to Andy Yan. I haven't got a
reply, which kind of isn't surprising in this world of spam. Can I ask
you for a referral - could you send him an email asking him if I can
have the patch and how to test it?

Also, to list-dwellers: I'd greatly appreciate further tips on PCIe SAS HBAs.

Thanks!

Re: Known-good LSI PCIe SAS HBAs?

[Thomas Fjellstrom]

On Fri November 13 2009, Kristleifur Daðason wrote:
> On Wed, Nov 11, 2009 at 6:51 PM, Thomas Fjellstrom <tfjellstrom@shaw.ca> 
wrote:
> > On Wed November 11 2009, Richard Scobie wrote:
> > > Thomas Fjellstrom wrote:
> > > > Thats the problem he was trying to get away from. the MV8
> > > > controller likes to offline disks when a few smart commands arrive.
> > > > at least with the main kernel drivers. But then the main kernel
> > > > mvsas drivers decide to give up if you even attempt to start a raid
> > > > array on them. The controller locks up and you have to reboot. Most
> > > > of the time a reboot even fails when it tries to sync the block
> > > > devices, so a hard reset is in order.
> > >
> > > OK, I did not know the Marvell controller also has a problem with
> > > smartmontools.
> >
> > Given the state of the in kernel drivers, its probably a driver issue.
> 
> Thanks for the replies, all.
> 
> I'll bet it's the driver - it doesn't feel quite right that a random
> external command can easily drop drives off the controller, be it
> smartmontools or mdadm.
> 
> Thomas, I tried sending a private email to Andy Yan. I haven't got a
> reply, which kind of isn't surprising in this world of spam. Can I ask
> you for a referral - could you send him an email asking him if I can
> have the patch and how to test it?

He seems to be a busy guy, but I've CCed him on this. The actual driver I 
got is a full copy of the driver, builds out of tree.

He hasn't said I can't send the package to anyone, but I'll wait a little 
while before doing so, he might pop up saying not to give it out.

All I can say is: it works well enough for me*.

For a $120 8 port sata card, it really does the trick. I was so glad when I 
didn't have to get several crappy 2-4 port jmicron cards.

> Also, to list-dwellers: I'd greatly appreciate further tips on PCIe SAS
>  HBAs.
> 
> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

* Performance (to me) is outstanding, or at least it will be once the write 
back stuff is fixed up, and the io controller stuff is also fixed up, but as 
is right now in 2.6.31, its "good enough". Might not get the absolute insane 
400-500MB/s I saw with the controller originally, but I do get a good 
100-230MB/s off my 5 drive md raid5 array (512KB chunk). I find things that 
used to take 30 seconds or more on my old array now happen in a few seconds. 
Right now writes are rather bad (60-100MB/s) but that I assume is the write 
back stuff, which should hopefully be fixed in 32.

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Known-good LSI PCIe SAS HBAs?

[Konstantinos Skarlatos]

Hello
Any news on the mvsas driver for this card? the MV8 is a ridiculously 
attractive card, and proper linux support would be a boon to many people 
who want to use it on their fileservers.


On 13/11/2009 8:59 μμ, Thomas Fjellstrom wrote:
> On Fri November 13 2009, Kristleifur Daðason wrote:
>    
>> On Wed, Nov 11, 2009 at 6:51 PM, Thomas Fjellstrom<tfjellstrom@shaw.ca>
>>      
> wrote:
>    
>>> On Wed November 11 2009, Richard Scobie wrote:
>>>        
>>>> Thomas Fjellstrom wrote:
>>>>          
>>>>> Thats the problem he was trying to get away from. the MV8
>>>>> controller likes to offline disks when a few smart commands arrive.
>>>>> at least with the main kernel drivers. But then the main kernel
>>>>> mvsas drivers decide to give up if you even attempt to start a raid
>>>>> array on them. The controller locks up and you have to reboot. Most
>>>>> of the time a reboot even fails when it tries to sync the block
>>>>> devices, so a hard reset is in order.
>>>>>            
>>>> OK, I did not know the Marvell controller also has a problem with
>>>> smartmontools.
>>>>          
>>> Given the state of the in kernel drivers, its probably a driver issue.
>>>        
>> Thanks for the replies, all.
>>
>> I'll bet it's the driver - it doesn't feel quite right that a random
>> external command can easily drop drives off the controller, be it
>> smartmontools or mdadm.
>>
>> Thomas, I tried sending a private email to Andy Yan. I haven't got a
>> reply, which kind of isn't surprising in this world of spam. Can I ask
>> you for a referral - could you send him an email asking him if I can
>> have the patch and how to test it?
>>      
> He seems to be a busy guy, but I've CCed him on this. The actual driver I
> got is a full copy of the driver, builds out of tree.
>
> He hasn't said I can't send the package to anyone, but I'll wait a little
> while before doing so, he might pop up saying not to give it out.
>
> All I can say is: it works well enough for me*.
>
> For a $120 8 port sata card, it really does the trick. I was so glad when I
> didn't have to get several crappy 2-4 port jmicron cards.
>
>    
>> Also, to list-dwellers: I'd greatly appreciate further tips on PCIe SAS
>>   HBAs.
>>
>> Thanks!
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>      
> * Performance (to me) is outstanding, or at least it will be once the write
> back stuff is fixed up, and the io controller stuff is also fixed up, but as
> is right now in 2.6.31, its "good enough". Might not get the absolute insane
> 400-500MB/s I saw with the controller originally, but I do get a good
> 100-230MB/s off my 5 drive md raid5 array (512KB chunk). I find things that
> used to take 30 seconds or more on my old array now happen in a few seconds.
> Right now writes are rather bad (60-100MB/s) but that I assume is the write
> back stuff, which should hopefully be fixed in 32.
>
>    

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html