stuck in megaraid_sas.c megasas_adp_reset_gen2

stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

I recently got an IBM M1015 (MegaRaid 9240-8i) card, and after getting a new 
motherboard, the system now boots, but the megaraid_sas driver seems to be 
getting stuck when trying to initialize the card.

Looking through the source, it seems to be stuck in the megasas_adp_reset_gen2 
function, in the while loop at the end. Now, according to the code it can't 
actually get stuck there permanently, but it does take quite a while for the 
loop to finish, and the udev timeout messages to stop.

I've looked around quite a bit, but haven't found any solutions thus far. If 
anyone could point me in the right direction I'd appreciate it.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[adam radford]

On Wed, Mar 21, 2012 at 4:16 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
> I recently got an IBM M1015 (MegaRaid 9240-8i) card, and after getting a new
> motherboard, the system now boots, but the megaraid_sas driver seems to be
> getting stuck when trying to initialize the card.
>
> Looking through the source, it seems to be stuck in the megasas_adp_reset_gen2
> function, in the while loop at the end. Now, according to the code it can't
> actually get stuck there permanently, but it does take quite a while for the
> loop to finish, and the udev timeout messages to stop.
>
> I've looked around quite a bit, but haven't found any solutions thus far. If
> anyone could point me in the right direction I'd appreciate it.

If you are getting controller resets during driver load, you must not
be getting interrupts or firmware is not responding to the inquiry
roll-call.  Make sure you have the latest firmware.

The code at the end of megasas_adp_reset_gen2() just looks for
DIAG_RESET_ADAPTER flag to clear on the host diag register when
issuing a controller reset... that should happen almost immediately
unless there is a hardware or firmware issue.

Are you sure your 'new' motherboard is actually good ?

Can you move your megaraid 9240-8i into a 'known working' system and re-test ?

-Adam

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

On Wed Mar 21, 2012, adam radford wrote:
> On Wed, Mar 21, 2012 at 4:16 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> 
wrote:
> > I recently got an IBM M1015 (MegaRaid 9240-8i) card, and after getting a
> > new motherboard, the system now boots, but the megaraid_sas driver seems
> > to be getting stuck when trying to initialize the card.
> > 
> > Looking through the source, it seems to be stuck in the
> > megasas_adp_reset_gen2 function, in the while loop at the end. Now,
> > according to the code it can't actually get stuck there permanently, but
> > it does take quite a while for the loop to finish, and the udev timeout
> > messages to stop.
> > 
> > I've looked around quite a bit, but haven't found any solutions thus far.
> > If anyone could point me in the right direction I'd appreciate it.
> 
> If you are getting controller resets during driver load, you must not
> be getting interrupts or firmware is not responding to the inquiry
> roll-call.  Make sure you have the latest firmware.

I updated to the latest on LSI's site today before emailing. It changes the 
behavior slightly. With the older firmware, it would not print any of the 
initial reset messages, but would once udev decides to start killing modprobe. 
With the new firmware, I get a:

ADP_RESET_GEN2: HostDiag=a0

followed by a bunch of:

RESET_GEN2: retry=%x, hostdiag=a4

Now I'm not sure the hostdiag should be different between the two. if this aN 
identifier is similar to the aN identifiers in the MegaCli tool, then it would 
mean its trying to reset a device that doesn't exist? I only have a single 
M1015 card installed.
 
> The code at the end of megasas_adp_reset_gen2() just looks for
> DIAG_RESET_ADAPTER flag to clear on the host diag register when
> issuing a controller reset... that should happen almost immediately
> unless there is a hardware or firmware issue.
> 
> Are you sure your 'new' motherboard is actually good ?

It boots and runs fine without the sas card installed. I haven't run any heavy 
load tests, but it seems ok.

> Can you move your megaraid 9240-8i into a 'known working' system and
> re-test ?

Nope. This is the furthest I've gotten it to get with this card installed. The 
old system would fail to boot into grub properly, let alone linux. These cards 
seem to be /very/ picky about what motherboard you install them in.

> -Adam


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

On Wed Mar 21, 2012, you wrote:
> On Wed Mar 21, 2012, adam radford wrote:
> > On Wed, Mar 21, 2012 at 4:16 PM, Thomas Fjellstrom <thomas@fjellstrom.ca>
> 
> wrote:
> > > I recently got an IBM M1015 (MegaRaid 9240-8i) card, and after getting
> > > a new motherboard, the system now boots, but the megaraid_sas driver
> > > seems to be getting stuck when trying to initialize the card.
> > > 
> > > Looking through the source, it seems to be stuck in the
> > > megasas_adp_reset_gen2 function, in the while loop at the end. Now,
> > > according to the code it can't actually get stuck there permanently,
> > > but it does take quite a while for the loop to finish, and the udev
> > > timeout messages to stop.
> > > 
> > > I've looked around quite a bit, but haven't found any solutions thus
> > > far. If anyone could point me in the right direction I'd appreciate
> > > it.
> > 
> > If you are getting controller resets during driver load, you must not
> > be getting interrupts or firmware is not responding to the inquiry
> > roll-call.  Make sure you have the latest firmware.
> 
> I updated to the latest on LSI's site today before emailing. It changes the
> behavior slightly. With the older firmware, it would not print any of the
> initial reset messages, but would once udev decides to start killing
> modprobe. With the new firmware, I get a:
> 
> ADP_RESET_GEN2: HostDiag=a0
> 
> followed by a bunch of:
> 
> RESET_GEN2: retry=%x, hostdiag=a4
> 
> Now I'm not sure the hostdiag should be different between the two. if this
> aN identifier is similar to the aN identifiers in the MegaCli tool, then
> it would mean its trying to reset a device that doesn't exist? I only have
> a single M1015 card installed.
> 
> > The code at the end of megasas_adp_reset_gen2() just looks for
> > DIAG_RESET_ADAPTER flag to clear on the host diag register when
> > issuing a controller reset... that should happen almost immediately
> > unless there is a hardware or firmware issue.
> > 
> > Are you sure your 'new' motherboard is actually good ?
> 
> It boots and runs fine without the sas card installed. I haven't run any
> heavy load tests, but it seems ok.

Machine has been solid as a rock (sans 9240-8i) for the past month with mild 
to half load. It runs several virtual machines, a nfs share, my firewall, a 
minecraft server, and some other miscellaneous stuff. Not a single hiccup.

> > Can you move your megaraid 9240-8i into a 'known working' system and
> > re-test ?
> 
> Nope. This is the furthest I've gotten it to get with this card installed.
> The old system would fail to boot into grub properly, let alone linux.
> These cards seem to be /very/ picky about what motherboard you install
> them in.
> 
> > -Adam

I just got a second M1015 card in today and gave it a go. Similar issues, 
different log messages. (hand typed from picture taken of screens)

Lots of:

megasas: Waiting for 1 commands to complete

for quite a while (5-10 minutes), along with udevd trying to kill modprobe. 
Then:

megasas: moving cmd[0]:hexstringherewithcolons queue as internal
megaraid_sas: FW detected to be in fault state, restarting it...
ADP_RESET_GEN2: HostDiag=a0
megaraid_sas: FW restarted successfully,initializing next stage...
megaraid_sas: HBA recovery state machine,state 1 starting...
(sits here for a while)
megasas: Waiting for FW to come to ready state
megasas: FW now in ready state
megaraid_sas: command hexstringhere, hexstringhere detected (something?) while 
HBA reset
megasas: command hexstring scsi cmd [12]detected on the internal (something?) 
again
megasas: reset successful
scsi:0:0:0:0: megasas: RESET cmd=12 retries=0
megaraid_sas: no pending cmds after reset
megasas: reset successful
scsi:0:0:0:0: megasas: RESET cmd=12 retries=0
megaraid_sas: no pending cmds after reset
megasas: reset successful
scsi:0:0:0:0: Device offlined - not ready after error recovery
(other scsi devices are detected)
(bootup hangs here)

Eventually theres some "hung task" timeout backtraces. This is where I tried 
to kill udevd, CTRL+C didn't stop it from trying to kill modprobe, and 
ALT+SYSRQ+K caused a silent oops (keyboard leds blinking, no backtrace or OOPS 
text). If its similar to last time, eventually the kernel will outright OOPS 
without any intervention.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[adam radford]

On Wed, Apr 11, 2012 at 1:17 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
>>
>> ADP_RESET_GEN2: HostDiag=a0
>>
>> followed by a bunch of:
>>
>> RESET_GEN2: retry=%x, hostdiag=a4
>>
>> Now I'm not sure the hostdiag should be different between the two. if this
>> aN identifier is similar to the aN identifiers in the MegaCli tool, then
>> it would mean its trying to reset a device that doesn't exist? I only have
>> a single M1015 card installed.

host diag register output a0 or a4 has absolutely nothing to do with
MegaCli -aN command line argument for specifying adapter number.

> I just got a second M1015 card in today and gave it a go. Similar issues,
> different log messages. (hand typed from picture taken of screens)
>
> Lots of:
>
> megasas: Waiting for 1 commands to complete

Can you try booting with kernel command line argument pcie_aspm=off

-Adam

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

On Wed Apr 11, 2012, adam radford wrote:
> On Wed, Apr 11, 2012 at 1:17 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> 
wrote:
> >> ADP_RESET_GEN2: HostDiag=a0
> >> 
> >> followed by a bunch of:
> >> 
> >> RESET_GEN2: retry=%x, hostdiag=a4
> >> 
> >> Now I'm not sure the hostdiag should be different between the two. if
> >> this aN identifier is similar to the aN identifiers in the MegaCli
> >> tool, then it would mean its trying to reset a device that doesn't
> >> exist? I only have a single M1015 card installed.
> 
> host diag register output a0 or a4 has absolutely nothing to do with
> MegaCli -aN command line argument for specifying adapter number.
> 
> > I just got a second M1015 card in today and gave it a go. Similar issues,
> > different log messages. (hand typed from picture taken of screens)
> > 
> > Lots of:
> > 
> > megasas: Waiting for 1 commands to complete
> 
> Can you try booting with kernel command line argument pcie_aspm=off

No problem.

Things are quite similar. Startup goes like:

<detected a onboard sata ports>
scsi: waiting for bus probes to complete...
Refined TSC...
Switched to clocksource tsc
<pause here>
udevd[...]: timeout: killing '/sbin/modprobe -b ...' (lots of these, so much 
that I hit scroll lock so I can see the kernel messages as they come up)
scsi 0:0:0:0: megasas: RESET cmd=12 retries=0
megasas: [ 0] waiting for 1 commands to complete
(many more waiting messages)
<hung task kworker/u:4>
Call Trace:
  [<ffffffff810641d0>] ? async_synchronize_cookie_domain+0xb2/...c
  [<ffffffff8105f583>] ? add_wait_queue+0x3c/0x3c
  ....
megasas: [55] waiting for 1 commands to complete
....
megasas: [175] waiting for 1 commands to complete
megasas: moving cmd[0]:ffff880234bcb940:0:ffff88002339beec0 the defer queue as 
internal
megaraid_sas: FW detected to be in faultstate, restarting it...
ADP_RESET_GEN2: HostDiag=a0
(10s wait)
megaraid_sas: FW restarted successfully,initializing next stage...
megaraid_sas: HBA recovery state machine,state 2 starting...
(30s wait)
megasas: Waiting for FW to come to ready state
megasas: FW now in ready state
megaraid_sas: command ffff880234bcb940, ffff8802339beec0:0detected to be pending 
while HBA reset
megasas: ffff880234bcb940 scsi cmd [12]detected on the internal queue, issue 
again.
megasas: reset successful
scsi: 0:0:0:0: megasas: RESET cmd=12 retries 0
megaraid_sas: no pending cmds after reset
megasas: reset successful
(20s wait)
(device offlined message here, missed it this time)
(detected all sata devices)

And it stalled there.

> -Adam


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[adam radford]

On Wed, Apr 11, 2012 at 2:44 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
> megasas: [55] waiting for 1 commands to complete
> ....
> megasas: [175] waiting for 1 commands to complete

You still aren't getting interrupts from the card.  Can you re-try
with megaraid_sas module parameter msix_disable ?
If that doesn't work, contact LSI support and tell them which
motherboard, any PCIe bridges, risers, etc you have.

-Adam

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

On Thu Apr 12, 2012, adam radford wrote:
> On Wed, Apr 11, 2012 at 2:44 PM, Thomas Fjellstrom <thomas@fjellstrom.ca> 
wrote:
> > megasas: [55] waiting for 1 commands to complete
> > ....
> > megasas: [175] waiting for 1 commands to complete
> 
> You still aren't getting interrupts from the card.  Can you re-try
> with megaraid_sas module parameter msix_disable ?
> If that doesn't work, contact LSI support and tell them which
> motherboard, any PCIe bridges, risers, etc you have.

No luck darn it. with either card.

> -Adam


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: stuck in megaraid_sas.c megasas_adp_reset_gen2

[Thomas Fjellstrom]

On Thu Apr 12, 2012, Thomas Fjellstrom wrote:
> On Thu Apr 12, 2012, adam radford wrote:
> > On Wed, Apr 11, 2012 at 2:44 PM, Thomas Fjellstrom <thomas@fjellstrom.ca>
> 
> wrote:
> > > megasas: [55] waiting for 1 commands to complete
> > > ....
> > > megasas: [175] waiting for 1 commands to complete
> > 
> > You still aren't getting interrupts from the card.  Can you re-try
> > with megaraid_sas module parameter msix_disable ?
> > If that doesn't work, contact LSI support and tell them which
> > motherboard, any PCIe bridges, risers, etc you have.
> 
> No luck darn it. with either card.

I've been talking to LSI support for the past few days. So far they are 
unwilling to support my current use case, or fix their INT19 firmware bug (which 
they have acknowledged).

I just hope there aren't any more serious bugs like this in their firmware. 
It's looking pretty scary. If they let this bug live for years, imagine what 
other bugs might lurk. I may have to avoid LSI completely in the future, even 
on real server systems.

> > -Adam


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca