mvsas with 3.1

mvsas with 3.1

[Thomas Fjellstrom]

Is there chance this driver will ever be stable? After more than two years I'm 
starting to get extremely frustrated. I don't really have the option at the 
moment to get a new card, otherwise I would, likely a non marvell based 
device.

It actually managed to last 10 days this time though. Which is a record. The 
interesting thing is there are no warnings or errors in dmesg coming from the 
mvsas driver or scsi code. All that's happened is processes lock up when 
trying to write. Reading seems to be fine.

Just to refresh everyone's memory, it's a AOC-SASLP-MV8 card, has a 
MV64460/64461/64462 chipset. And I have 7 (seagate 7200.12 SATA drives hooked 
up).

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: mvsas with 3.1 (mdraid+xfs locked up, single drive w/xfs not locked up)

[Thomas Fjellstrom]

On Fri Jan 13, 2012, Thomas Fjellstrom wrote:
> Is there chance this driver will ever be stable? After more than two years
> I'm starting to get extremely frustrated. I don't really have the option
> at the moment to get a new card, otherwise I would, likely a non marvell
> based device.
> 
> It actually managed to last 10 days this time though. Which is a record.
> The interesting thing is there are no warnings or errors in dmesg coming
> from the mvsas driver or scsi code. All that's happened is processes lock
> up when trying to write. Reading seems to be fine.
> 
> Just to refresh everyone's memory, it's a AOC-SASLP-MV8 card, has a
> MV64460/64461/64462 chipset. And I have 7 (seagate 7200.12 SATA drives
> hooked up).

I attached an 8th drive for kicks (for saving stuff temporarily that don't need 
to be on the big array), and if anything, the array locks up more than it did 
before. But there is one difference between the lockups that were happening and 
that are happening now. Now the entire array is still readable. It just locks 
up any process attempting to write to the array. Before both read and write 
would lock up, and a bunch of scary messages would hit dmesg. Now there are 
very few log messages about the array. In fact no messages show up from the 
scsi or other subsystems. The only evidence of "bad things" happening is the 
continual "process foo blocked for more than 120s" messages, and of course 
anything attempting to write to the array.

Here's something interesting though, I just tried writing to the one disk on 
the card that isn't part of the mdraid raid5 volume. It is fine. I can read 
from it and write to it. So something to do with mdraid, or the XFS filesystem 
(or both) is causing a bad interaction with the card itself.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: mvsas with 3.1 (mdraid+xfs locked up, single drive w/xfs not locked up)

[Brad Campbell]

On 17/01/12 11:00, Thomas Fjellstrom wrote:
>
> Here's something interesting though, I just tried writing to the one disk on
> the card that isn't part of the mdraid raid5 volume. It is fine. I can read
> from it and write to it. So something to do with mdraid, or the XFS filesystem
> (or both) is causing a bad interaction with the card itself.

Try doing IO to multiple drives simultaneously. I had a SIL card that would corrupt data when used 
in RAID, but not single drive IO tests. I could make it fail by doing multiple dd/cp/md5sum to two 
drives simultaneously.

Do yourself a favour. Whack it on fleabay and sell it to a Windows user, then get something 
supported. Leave enough scatter around that people thinking of buying a card with that Marvell 
chipset will stay clear.

Regards,
Brad

Re: mvsas with 3.1

[Simon McNair]

Hi Thomas,
I've been trying to get an answer to this same question (I have a
supermicro AOC-SASLP-MV8 too).  See my post at
https://lkml.org/lkml/2012/1/30/107  .

My only suggestion is to get a kernel that's pre 2.6.30 (my Thecus
5200 marvell based unit is on 2.6.13 and it seems okay).  If the data
is mission critical I certainly wouldn't be using the latest kernels
in production until it had 6+months in a stress test environment (of
course I'm an obsessive apt-get upgrade which makes me a hypocrite
:-P).

I can see from searching lkml.org  for mvsas that there is a lot of
code going in to this driver and unless your're a linux guru and can
fix these problems I can only suggest you keep anything important on a
=< 2.6.13 kernel and monitor the situation.

Oh, and keep back ups ;-) (again here's me being the hypocrite).

That's my plan :-(

Simon

On 13 January 2012 18:44, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
> Is there chance this driver will ever be stable? After more than two years I'm
> starting to get extremely frustrated. I don't really have the option at the
> moment to get a new card, otherwise I would, likely a non marvell based
> device.
>
> It actually managed to last 10 days this time though. Which is a record. The
> interesting thing is there are no warnings or errors in dmesg coming from the
> mvsas driver or scsi code. All that's happened is processes lock up when
> trying to write. Reading seems to be fine.
>
> Just to refresh everyone's memory, it's a AOC-SASLP-MV8 card, has a
> MV64460/64461/64462 chipset. And I have 7 (seagate 7200.12 SATA drives hooked
> up).
>
> --
> Thomas Fjellstrom
> thomas@fjellstrom.ca
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mvsas with 3.1

[Thomas Fjellstrom]

On Wed Feb 1, 2012, Simon McNair wrote:
> Hi Thomas,
> I've been trying to get an answer to this same question (I have a
> supermicro AOC-SASLP-MV8 too).  See my post at
> https://lkml.org/lkml/2012/1/30/107  .
> 
> My only suggestion is to get a kernel that's pre 2.6.30 (my Thecus
> 5200 marvell based unit is on 2.6.13 and it seems okay).  If the data
> is mission critical I certainly wouldn't be using the latest kernels
> in production until it had 6+months in a stress test environment (of
> course I'm an obsessive apt-get upgrade which makes me a hypocrite
> 
> :-P).
> 
> I can see from searching lkml.org  for mvsas that there is a lot of
> code going in to this driver and unless your're a linux guru and can
> fix these problems I can only suggest you keep anything important on a
> =< 2.6.13 kernel and monitor the situation.
> 
> Oh, and keep back ups ;-) (again here's me being the hypocrite).
> 
> That's my plan :-(

The older drivers were even worse, they'd have these really nasty pauses, or 
just outright OOPS the entire machine.

That said, I've finally had it with the lack of support. It's been over two 
years since I got the card, and enough is enough. I picked up a refurb IBM 
ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just have to 
wait and see why grub doesn't like it.

> Simon
> 
> On 13 January 2012 18:44, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
> > Is there chance this driver will ever be stable? After more than two
> > years I'm starting to get extremely frustrated. I don't really have the
> > option at the moment to get a new card, otherwise I would, likely a non
> > marvell based device.
> > 
> > It actually managed to last 10 days this time though. Which is a record.
> > The interesting thing is there are no warnings or errors in dmesg coming
> > from the mvsas driver or scsi code. All that's happened is processes
> > lock up when trying to write. Reading seems to be fine.
> > 
> > Just to refresh everyone's memory, it's a AOC-SASLP-MV8 card, has a
> > MV64460/64461/64462 chipset. And I have 7 (seagate 7200.12 SATA drives
> > hooked up).
> > 
> > --
> > Thomas Fjellstrom
> > thomas@fjellstrom.ca
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[John Robinson]

On 01/02/2012 14:03, Thomas Fjellstrom wrote:
[...]
>  I picked up a refurb IBM
> ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just have to
> wait and see why grub doesn't like it.

When you find out, please let me know - I bought one via ebay, its 
firmware was elderly and wouldn't let me put it in JBOD mode, after a 
bit of googling I flashed it with IBM's latest firmware, then GRUB 
thought my machine had no RAM in it. The card has been sitting in a box 
since.

Cheers,

John.

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[Thomas Fjellstrom]

On Wed Feb 1, 2012, John Robinson wrote:
> On 01/02/2012 14:03, Thomas Fjellstrom wrote:
> [...]
> 
> >  I picked up a refurb IBM
> > 
> > ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just have
> > to wait and see why grub doesn't like it.
> 
> When you find out, please let me know - I bought one via ebay, its
> firmware was elderly and wouldn't let me put it in JBOD mode, after a
> bit of googling I flashed it with IBM's latest firmware, then GRUB
> thought my machine had no RAM in it. The card has been sitting in a box
> since.

Yeah, GRUB immediately throws an "out of memory. Aborted" error, its annoying, 
but I reported the issue on GRUB's bug tracker after talking to a guy in #grub 
on freenode. Hopefully someone looks at it soon. It appears it happens so 
early in grub's startup that it can't even output any debug info (I was asked 
to run: `grub-install --debug-image=all /dev/sdX`, and did so, I get debug 
output when the card is not installed, but absolutely nothing when it is 
installed.

> Cheers,
> 
> John.


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[Hannes Reinecke]

On 02/01/2012 03:42 PM, Thomas Fjellstrom wrote:
> On Wed Feb 1, 2012, John Robinson wrote:
>> On 01/02/2012 14:03, Thomas Fjellstrom wrote:
>> [...]
>>
>>>  I picked up a refurb IBM
>>>
>>> ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just have
>>> to wait and see why grub doesn't like it.
>>
>> When you find out, please let me know - I bought one via ebay, its
>> firmware was elderly and wouldn't let me put it in JBOD mode, after a
>> bit of googling I flashed it with IBM's latest firmware, then GRUB
>> thought my machine had no RAM in it. The card has been sitting in a box
>> since.
> 
> Yeah, GRUB immediately throws an "out of memory. Aborted" error, its annoying, 
> but I reported the issue on GRUB's bug tracker after talking to a guy in #grub 
> on freenode. Hopefully someone looks at it soon. It appears it happens so 
> early in grub's startup that it can't even output any debug info (I was asked 
> to run: `grub-install --debug-image=all /dev/sdX`, and did so, I get debug 
> output when the card is not installed, but absolutely nothing when it is 
> installed.
> 
>From my experience the BIOS wouldn't present any disks in JBOD mode.
So you'd be needing to switch to RAID mode first.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[Thomas Fjellstrom]

On Wed Feb 1, 2012, Hannes Reinecke wrote:
> On 02/01/2012 03:42 PM, Thomas Fjellstrom wrote:
> > On Wed Feb 1, 2012, John Robinson wrote:
> >> On 01/02/2012 14:03, Thomas Fjellstrom wrote:
> >> [...]
> >> 
> >>>  I picked up a refurb IBM
> >>> 
> >>> ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just
> >>> have to wait and see why grub doesn't like it.
> >> 
> >> When you find out, please let me know - I bought one via ebay, its
> >> firmware was elderly and wouldn't let me put it in JBOD mode, after a
> >> bit of googling I flashed it with IBM's latest firmware, then GRUB
> >> thought my machine had no RAM in it. The card has been sitting in a box
> >> since.
> > 
> > Yeah, GRUB immediately throws an "out of memory. Aborted" error, its
> > annoying, but I reported the issue on GRUB's bug tracker after talking
> > to a guy in #grub on freenode. Hopefully someone looks at it soon. It
> > appears it happens so early in grub's startup that it can't even output
> > any debug info (I was asked to run: `grub-install --debug-image=all
> > /dev/sdX`, and did so, I get debug output when the card is not
> > installed, but absolutely nothing when it is installed.
> 
> From my experience the BIOS wouldn't present any disks in JBOD mode.
> So you'd be needing to switch to RAID mode first.

It does seem to notice them all in its own bios, and list them in JBOD mode. 
But the disks aren't used for booting, so at the very least it shouldn't affect 
booting at all. But I'll try that out anyhow to see if it "fixes" GRUB.

> Cheers,
> 
> Hannes


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[John Robinson]

On 01/02/2012 15:16, Hannes Reinecke wrote:
> On 02/01/2012 03:42 PM, Thomas Fjellstrom wrote:
>> On Wed Feb 1, 2012, John Robinson wrote:
>>> On 01/02/2012 14:03, Thomas Fjellstrom wrote:
>>> [...]
>>>>  I picked up a refurb IBM
>>>> ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just have
>>>> to wait and see why grub doesn't like it.
>>>
>>> When you find out, please let me know - I bought one via ebay, its
>>> firmware was elderly and wouldn't let me put it in JBOD mode, after a
>>> bit of googling I flashed it with IBM's latest firmware, then GRUB
>>> thought my machine had no RAM in it.
[...]
>
> From my experience the BIOS wouldn't present any disks in JBOD mode.
> So you'd be needing to switch to RAID mode first.

The firmware my card came with didn't have a JBOD mode, which is why I 
flashed the newer firmware. Then I could boot in JBOD mode from a rescue 
CD and see some drives, though not those that had had RAID-mode metadata 
written to them, but I couldn't boot the system with GRUB (from drives 
on another controller).

Cheers,

John.

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[Brad Campbell]

On 01/02/12 22:42, Thomas Fjellstrom wrote:
>
> Yeah, GRUB immediately throws an "out of memory. Aborted" error, its annoying,
> but I reported the issue on GRUB's bug tracker after talking to a guy in #grub
> on freenode. Hopefully someone looks at it soon. It appears it happens so
> early in grub's startup that it can't even output any debug info (I was asked
> to run: `grub-install --debug-image=all /dev/sdX`, and did so, I get debug
> output when the card is not installed, but absolutely nothing when it is
> installed.

Just another data point. I've seen this issue also. It does not appear to be GRUB's fault. It 
appears the BIOS on the card does something funky as it also prevents syslinux/isolinux/pxelinux and 
its friends from properly detecting/reporting the machines memory. Machine has 16GB but a PXE boot 
believes it has 3GB.

I "worked around" the problem by swapping out the mainboard for a later model with a different BIOS 
and the problem went away. Not pretty, but it was the only way I could get the machine to boot with 
those cards in it. I have 4 IBM cards in 2 machines and they work fine here. I still have the dodgy 
mainboard and can set it up if you need help testing/reproducing the issue.

Regards,
Brad

Re: GRUB doesn't like ServeRAID M1015/LSI 9220-8i (was Re: mvsas with 3.1)

[Thomas Fjellstrom]

On Wed Feb 1, 2012, Brad Campbell wrote:
> On 01/02/12 22:42, Thomas Fjellstrom wrote:
> > Yeah, GRUB immediately throws an "out of memory. Aborted" error, its
> > annoying, but I reported the issue on GRUB's bug tracker after talking
> > to a guy in #grub on freenode. Hopefully someone looks at it soon. It
> > appears it happens so early in grub's startup that it can't even output
> > any debug info (I was asked to run: `grub-install --debug-image=all
> > /dev/sdX`, and did so, I get debug output when the card is not
> > installed, but absolutely nothing when it is installed.
> 
> Just another data point. I've seen this issue also. It does not appear to
> be GRUB's fault. It appears the BIOS on the card does something funky as
> it also prevents syslinux/isolinux/pxelinux and its friends from properly
> detecting/reporting the machines memory. Machine has 16GB but a PXE boot
> believes it has 3GB.
> 
> I "worked around" the problem by swapping out the mainboard for a later
> model with a different BIOS and the problem went away. Not pretty, but it
> was the only way I could get the machine to boot with those cards in it. I
> have 4 IBM cards in 2 machines and they work fine here. I still have the
> dodgy mainboard and can set it up if you need help testing/reproducing the
> issue.

I meant to do some testing on it tonight, but that doesn't seem like its going 
to happen. Basically I'm going to try that grub change and see if it helps, if 
you have the time, and desire to help, that would be nice, but I'm most likely 
going to try it out tomorrow sometime regardless.

> Regards,
> Brad


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Solved (for me at least!) GRUB doesn't like ServeRAID M1015/LSI 9220-8i

[John Robinson]

On 01/02/2012 14:38, John Robinson wrote:
> On 01/02/2012 14:03, Thomas Fjellstrom wrote:
> [...]
>> I picked up a refurb IBM
>> ServeRaid M1015 (aka: LSI 9220-8i) card. It arrived yesterday. Just
>> have to
>> wait and see why grub doesn't like it.
>
> When you find out, please let me know - I bought one via ebay, its
> firmware was elderly and wouldn't let me put it in JBOD mode, after a
> bit of googling I flashed it with IBM's latest firmware, then GRUB
> thought my machine had no RAM in it. The card has been sitting in a box
> since.

RHEL/CentOS 5's GRUB is buggy, in that it doesn't always read the whole 
of the BIOS's memory map. See 
https://bugzilla.redhat.com/show_bug.cgi?id=445893 and in particular 
comment 7 where there's a link to a replacement GRUB which incorporates 
the patch that was first offered to Red Hat in July 2007. The issue 
likely affects other distros GRUBs too, since GRUB is moribund.

Cheers,

John.