[Bug 85101] New: hpsa + P410 does not show connected HP SAS port expanders

[Bug 85101] New: hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

            Bug ID: 85101
           Summary: hpsa + P410 does not show connected HP SAS port
                    expanders
           Product: SCSI Drivers
           Version: 2.5
    Kernel Version: 3.17
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
          Assignee: scsi_drivers-other@kernel-bugs.osdl.org
          Reporter: linux-ide@crashplan.pro
        Regression: No

ASRock H81 + Intel Celeron G1840
 HP P410, firmware 6.60
  HP SAS Expander, firmware 2.08

In the need for upgrading the firmware on the HP SAS expanders, the HP P410 is
attached to this setup. However it does not see the any of the two connected HP
SAS expander cards, only the boot drive and P410 are shown in the lsscsi
output.

Steps to reproduce:
1. blacklist hpsa
2. reboot
3. modprobe -v hpsa
4. lsscsi

# lsscsi
[0:0:0:0]    disk    ATA      OCZ-VERTEX       1.3   /dev/sda
[4:3:0:0]    storage HP       P410             6.60  -

# modprobe -v hpsa dmesg:
[   49.567169] HP HPSA Driver (v 3.4.4-1)
[   49.567251] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   49.567389] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   49.567576] hpsa 0000:01:00.0: MSIX
[   49.567649] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   49.567705] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   49.567799] hpsa 0000:01:00.0: Logical aborts not supported
[   49.587966] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   55.125143] scsi host4: hpsa
[   55.129439] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   55.129832] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
[   55.130585] scsi 4:3:0:0: Attached scsi generic sg1 type 12

Regression:
1. changed back to kernel 3.13: no difference
2. disconnect all other SAS (to SATA) cables: no difference
3. attach only one HP SAS expander: no difference
4. switch cable between P410 and SAS expander card: no differnce
5. replace HP P410 (and cables) with Adapter 7805H or Marvell 88SE9485 SAS/SATA
6Gb/s controller [1b4b:9485]: HP SAS expanders are shown

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

Robert Elliott <elliott@hp.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |elliott@hp.com

--- Comment #1 from Robert Elliott <elliott@hp.com> ---
Smart Array P410 is a RAID controller that does not expose anything about the
SAS fabric (expanders or physical drives) to the OS; it just presents logical
drives.  You need to use HP management tools to download firmware to expanders
and drives behind the controller.

If you attach the expander to an HBA, then you can use sg_write_buffer to
download microcode.  Version 1.39 added a --bpw option to download in small
chunks - an expander might require a small chunk size like 4 KiB (see
http://sg.danny.cz/sg for the latest sg3_utils code).

Although they sound like DOS programs, .scexe files from HP are really linux
executable scripts that you can view in a text editor (at least the first
part).  You can do this to extract the files:
mkdir tmp
./CPxyzzy.scexe --unpack=tmp

One of those is the binary image that can be sent with sg_write_buffer.

BTW, use lsscsi -g to see the /dev/sgNN paths for all the devices presented by
the LLD, including expanders.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #2 from linux-ide@crashplan.pro ---
In this case the hardware seems broken.

The P410 controller is only used for upgrading the HP SAS expander firmware.
This time we started with upgrading the controller firmware:
===
./ccissflash

    This program consists of two phases: device discovery and device update.
    No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

Do you want to upgrade the device that has older ROM?
(yes/no) yes
1 devices will be updated.
Updating: P410 Slot: 2 from [5.70] to [6.60]
Updating: P410 Slot: 2 from [5.70] to [6.60]

As part of the reboot process, you must power cycle the server and any external
array storage devices.
===

After power cycling the first three reboots were ok. Then a controller lockup
0x00130000 occurred 468 seconds after loading the hpsa module. Since then only
one boot/module loading was ok and since then the "time outs" are occurring:

===
[  121.613269] HP HPSA Driver (v 3.4.0-1)
[  121.613344] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[  121.613447] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[  121.613669] hpsa 0000:01:00.0: MSIX
[  121.613770] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[  121.613843] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[  121.613909] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[  121.613981] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[  121.614046] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[  121.614117] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[  121.614182] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[  121.614251] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[  121.634560] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[  127.171906] scsi4 : hpsa
[  127.174883] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[  127.175961] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
[  127.176335] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[  424.473998] HP HPSA Driver (v 3.4.0-1)
===
[   16.127857] HP HPSA Driver (v 3.4.0-1)
[   16.127927] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   16.128020] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   16.128231] hpsa 0000:01:00.0: MSIX
[   16.128327] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[   16.128394] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[   16.128460] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[   16.128524] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[   16.128590] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[   16.128660] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[   16.128726] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[   16.128791] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[   16.149110] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[   34.767566] scsi4 : hpsa
[   34.770583] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   34.771054] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
===
[   48.389920] HP HPSA Driver (v 3.4.4-1)
[   48.389985] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   48.390059] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   48.390239] hpsa 0000:01:00.0: MSIX
[   48.390312] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   48.390362] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   48.390453] hpsa 0000:01:00.0: Logical aborts not supported
[   48.410617] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   53.947809] scsi host4: hpsa
[   53.953488] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   53.953986] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
===
[   17.076307] HP HPSA Driver (v 3.4.4-1)
[   17.076388] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   17.076483] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   17.076695] hpsa 0000:01:00.0: MSIX
[   17.076782] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   17.076837] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   17.076942] hpsa 0000:01:00.0: Logical aborts not supported
[   17.097108] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   33.659110] scsi host4: hpsa
[   33.663958] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   33.664360] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
[   33.665053] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[  485.066696] hpsa 0000:01:00.0: Controller lockup detected: 0x00130000 
===
[   49.567169] HP HPSA Driver (v 3.4.4-1)
[   49.567251] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   49.567389] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   49.567576] hpsa 0000:01:00.0: MSIX
[   49.567649] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[   49.567705] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[   49.567799] hpsa 0000:01:00.0: Logical aborts not supported
[   49.587966] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[   55.125143] scsi host4: hpsa
[   55.129439] hpsa 0000:01:00.0: RAID              device c4b3t0l0 added.
[   55.129832] scsi 4:3:0:0: RAID              HP       P410             6.60
PQ: 0 ANSI: 5
===
[   38.305788] HP HPSA Driver (v 3.4.4-1)
[   38.305854] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   38.306052] hpsa 0000:01:00.0: MSIX
[   38.306127] hpsa 0000:01:00.0: irq 29 for MSI/MSI-X
[   38.306179] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[  163.163631] hpsa 0000:01:00.0: board not ready, timed out.
===
[   23.324890] HP HPSA Driver (v 3.4.0-1)
[   23.324957] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[   23.325060] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[   23.325281] hpsa 0000:01:00.0: MSIX
[   23.325379] hpsa 0000:01:00.0: irq 45 for MSI/MSI-X
[   23.325453] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[   23.325518] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[   23.325591] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[   23.325655] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[   23.325728] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[   23.325793] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[   23.325865] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[  148.185212] hpsa 0000:01:00.0: board not ready, timed out.
===

Right now loading the hpsa module permanently errors out with "board not ready,
timed out" and the "DS1: Diagnostics Error" LED turns Amber (=One of the server
diagnostics utilities has detected a controller error.)

Seems like a hardware issue, therefore this case can be closed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #3 from Robert Elliott <elliott@hp.com> ---
Controller lockup means the firmware crashed (e.g. an ASSERT statement
triggered).  That can be due to hardware problems or firmware bugs.  You might
see if it boots with nothing attached; then with one expander; then with two;
then with drives attached to the expanders.

If you're not really using the P410, though, just use sg_write_buffer through
the Adaptec 7805H HBA.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #4 from linux-ide@crashplan.pro ---
@Robert Elliott
Thanks for all your help.
The issue turns out not to be the HP P410 controller.
Using another mainboard the same HP P410 controller is ok.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-ide@crashplan.pro changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |UNREPRODUCIBLE
           Severity|normal                      |low

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-ide@crashplan.pro changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|UNREPRODUCIBLE              |---

--- Comment #5 from linux-ide@crashplan.pro ---
After replacing the faulty hardware, the HP P410 boots normally again.

However the P410+hpsa still does not detect either of the HP SAS expander
cards, neither does ./ccissflash detect any connected HP SAS expander.

LED status of the HP SAS expander card:
CR1 = solid green
CR2 = solid green
CR3 = solid green
CR4 = off
CR5 = off
CR6 = blinks green

Note: the Adaptec pm80xx module fails loading in sas expander discovery, I will
file a separate bug report for that.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #6 from linux-ide@crashplan.pro ---
The HP Smart Array P410 still has its issues: sometimes the diagnostics LED
turns amber, other times it works ok.

As a fall back a Marvell controller detects the HP SAS expander card fine:
# lsscsi -g | grep HP
[6:0:0:0]    enclosu HP       HP SAS EXP Card  2.08  -          /dev/sg1

I am not able to figure out a correct sg_write_buffer command. Run from the
directory where the unpacked binary image is stored:
# sg_write_buffer --bpw=4096 -v --in=PUF21000.bin /dev/sg1
tried to read 8388608 bytes from PUF21000.bin, got 1509632 bytes
will write 1509632 bytes
sending write buffer, mode=0x0, mspec=0, id=0,  offset=0, len=4096
    Write buffer cmd: 3b 00 00 00 00 00 00 10 00 00
write buffer:  Fixed format, current;  Sense key: Illegal Request
 Additional sense: Invalid field in cdb
Write buffer failed: Illegal request sense key, apart from Invalid opcode

Any suggestions for the correct sg_write_buffer statement for writing to an HP
SAS expander card?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #7 from Robert Elliott <elliott@hp.com> ---
You need to specify a download microcode mode.  Using deferred activation, the
most modern approach:

1. sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=xxx.bin /dev/sgXX
2. sg_write_buffer --mode=activate_mc /dev/sgXX

Older devices might only support mode 0x7.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #8 from linux-ide@crashplan.pro ---
Confirmation of successful firmware download/update/flash to the HP SAS
expander card using the commands:
# sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=PUF21000.bin /dev/sg1
# sg_write_buffer --mode=activate_mc /dev/sg1
# reboot

# lsscsi -g | grep HP
[6:0:0:0]    enclosu HP       HP SAS EXP Card  2.10  -          /dev/sg1

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-ide@crashplan.pro changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |DOCUMENTED

--- Comment #9 from linux-ide@crashplan.pro ---
This is expected behavior and not a bug because the HP P410 device is a SAS
RAID controller and not a serial attached scsi host bus adapter (SAS HBA).

A SAS RAID controller 
quote: "does not expose anything about the SAS fabric (expanders or physical
drives) to the OS"

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 85101] hpsa + P410 does not show connected HP SAS port expanders

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

Javier Crosby (murphyde835@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |murphyde835@gmail.com

--- Comment #10 from Javier Crosby (murphyde835@gmail.com) ---
Regulator lockup implies the firmware crashed (for example a Declare
proclamation set off). That can be because of equipment issues or firmware bugs
(source: https://freegamesonline.io). You could check whether it boots with
nothing connected; then, at that point, with one expander; then, at that point,
with two; then, at that point, with drives joined to the expanders.

On the off chance that you're not actually utilizing the P410, however, simply
use sg_write_buffer through the Adaptec 7805H HBA.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.