PCIe x4 cards not detected on Z370 mainboards

PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Hello,

sorry if this is not an appropriate list for my question - any
pointers are welcome.

Short version:

Both on a GIGABYTE Z370-HD3 and on a MSI Z370 TOMAHAWK mainboard
neither of my LSI SAS controllers (SAS3444 E PCIe x4 and
SAS3442E-R PCIe x4) gets detected.  Not by the system BIOS, there is
not messages from the SAS Controller BIOS, Linux does not see it in
the PCI bus scan - nothing.  Both controllers work find in older
mainboards.

Is this a known problem?  Can it be fixed?


More details:

I tried all combinations of PCIe slots with one or with both cards -
none is working.

I asked MSI support, who told me:

	These are old controllers which don't support UEFI/GOP. this
	is the reason they don't work.  Cards which have only legacy
	support cannot be operated in modern mainboards.

Hm...  but I also have a PCIe x 1 Adaptect SCSI controller
(29320LPE, firmware version v4.31.4 of 2007) which certainly never
heard of UEFI/GOP before, and this is working fine: I see the BIOS
messages when it does the SCSI bus scan, I see the initialization
under Linux, and I can access the SCSI devices under Linux.

Later they told me to make sure Windows 10 WHQL Option is disabled,
and to try switching PCIe speed from Auto to Gen_2 or Gen_1.

None of this helped.

These are two different mainboards from two different vendors, which
otherwise work fine.  the common part is the "Z370" in the name, and
the fact that my PCIe x4 SAS controllers are not detected.

I don't know if thisis relevant, but in Linux (Fedora 27,
4.15.6-300.fc27.x86_64 kernel), I see these error messages:

[    0.138056] ACPI: Added _OSI(Module Device)
[    0.143002] ACPI: Added _OSI(Processor Device)
[    0.147001] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.152001] ACPI: Added _OSI(Processor Aggregator Device)
[    0.158907] ACPI Error: [_SB_.PCI0.RP05.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
[    0.169002] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
[    0.178001] ACPI Error: Method parse/execution failed \_SB.PCI0.RP04.PXSX, AE_NOT_FOUND (20170831/psparse-550)
[    0.188184] ACPI Error: [_SB_.PCI0.RP09.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
[    0.198001] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
[    0.207001] ACPI Error: Method parse/execution failed \_SB.PCI0.RP08.PXSX, AE_NOT_FOUND (20170831/psparse-550)
[    0.218089] ACPI: Executed 49 blocks of module-level executable AML code
...

Booting with acpi=off on the kernel command line makes these
disappear (of course), but this does not help either - the SAS
controller does not magically show up.

Has anybody any clue 1) what could be causing these problems and
2) how to fix this, or at least 3) where I should ask for help?

Thanks in advance.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Don't worry about people stealing your ideas. If your ideas are  any
good, you'll have to ram them down people's throats."  - Howard Aiken

Re: PCIe x4 cards not detected on Z370 mainboards

[Bjorn Helgaas]

On Thu, Mar 22, 2018 at 07:48:40PM +0100, Wolfgang Denk wrote:
> Hello,
> 
> sorry if this is not an appropriate list for my question - any
> pointers are welcome.

This is a perfect place to ask!

> Short version:
> 
> Both on a GIGABYTE Z370-HD3 and on a MSI Z370 TOMAHAWK mainboard
> neither of my LSI SAS controllers (SAS3444 E PCIe x4 and
> SAS3442E-R PCIe x4) gets detected.  Not by the system BIOS, there is
> not messages from the SAS Controller BIOS, Linux does not see it in
> the PCI bus scan - nothing.  Both controllers work find in older
> mainboards.
> 
> Is this a known problem?  Can it be fixed?
> 
> 
> More details:
> 
> I tried all combinations of PCIe slots with one or with both cards -
> none is working.
> 
> I asked MSI support, who told me:
> 
> 	These are old controllers which don't support UEFI/GOP. this
> 	is the reason they don't work.  Cards which have only legacy
> 	support cannot be operated in modern mainboards.

That doesn't make sense to me.  Maybe these controllers have option
ROMs that aren't compatible with UEFI (though even that sounds
unlikely to me).  But even in that case, I would expect the card to at
least be visible in PCI config space, because that's needed even to
read the option ROM from the card.

We have had issues with LSI devices in the past, e.g.,

  822155100e58 ("PCI: Mark Cavium CN8xxx to avoid bus reset")
  12d8706963f0 ("Revert "PCI: Make sure bus number resources stay within their parents bounds"")

822155100e58 is a band-aid to avoid bus resets when LSI devices are
involved.  We don't really have a root cause for that, so it's
possible you're seeing a similar issue.

12d8706963f0 has to do with a defect in an LSI device where it
apparently fails to latch its bus number correctly when the OS changes
it.  You never see the device at all, so the only bus number change
should be when the BIOS enumerates and programs bus numbers initially.
But it seems like the BIOS doesn't see the device either.

> Hm...  but I also have a PCIe x 1 Adaptect SCSI controller
> (29320LPE, firmware version v4.31.4 of 2007) which certainly never
> heard of UEFI/GOP before, and this is working fine: I see the BIOS
> messages when it does the SCSI bus scan, I see the initialization
> under Linux, and I can access the SCSI devices under Linux.

So we know the slot is functional at least for a x1 controller.  The
spec requires that a xN port can also form a x1 link (PCIe r4.0, sec
4.2.4.10.1), so I think your x4 controllers *should* be functional at
least at x1.

I wonder the port leading to that slot is disabled or not negotiating
the link for some reason.  Can you collect "lspci -vvv" output with
the working card and with one of the non-working cards?

You could try locating the port leading to the slot and using setpci
to reset its secondary bus, then using /sys/bus/pci/rescan to rescan
the bus.

Bjorn

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

In message <20180322192759.GA252023@bhelgaas-glaptop.roam.corp.google.com> you wrote:
> 
> This is a perfect place to ask!

Thanks a lot!

> That doesn't make sense to me.  Maybe these controllers have option
> ROMs that aren't compatible with UEFI (though even that sounds
> unlikely to me).  But even in that case, I would expect the card to at
> least be visible in PCI config space, because that's needed even to
> read the option ROM from the card.

I speculate that, like on some graphics cards, the card BIOS needs
to run to initialize for example on-board memory etc., before the
card becomes realy visible to the rest of the system?

> I wonder the port leading to that slot is disabled or not negotiating
> the link for some reason.  Can you collect "lspci -vvv" output with
> the working card and with one of the non-working cards?

Done.  See below.

> You could try locating the port leading to the slot and using setpci
> to reset its secondary bus, then using /sys/bus/pci/rescan to rescan
> the bus.

Sorry, I need help here.  Which options/parameters are needed to
setpci to do that?


# uname -a
Linux sam.denx.de 4.15.9-300.fc27.x86_64 #1 SMP Mon Mar 12 17:07:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux


Test 1: no PCI card populated at all

	=> lspci results in "lspci-no-card"

Test 2: Adapted 29320LPE PCIe x1 Ultra320 SCSI adapter in PCI_E1

	- Card shows BIOS start messages
	- Card is detected in Linux
	- Card is operational in Linux

	[   18.912078] scsi host6: Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 3.0
                       <Adaptec 29320LPE PCIe Ultra320 SCSI adapter>
                       aic7901: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133MHz, 512 SCBs
	[   19.450127] scsi target6:0:2: asynchronous
	[   19.459627] scsi 6:0:2:0: Sequential-Access TANDBERG TS800            0468 PQ: 0 ANSI: 3
	[   19.485092] scsi target6:0:2: Beginning Domain Validation
	[   19.505066] scsi target6:0:2: wide asynchronous
	[   19.520553] scsi target6:0:2: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 40)
	[   19.543717] scsi 6:0:2:0: Power-on or device reset occurred
	[   19.556159] scsi 6:0:2:0: Power-on or device reset occurred
	[   19.966855] scsi target6:0:2: Domain Validation skipping write tests
	[   19.973733] scsi target6:0:2: Ending Domain Validation
	[   23.067430] scsi 6:0:2:0: Attached scsi generic sg5 type 1

	=> lspci results in "lspci-adaptec-29320LPE"

Test 3: LSI SAS3444E (aka IBM 25R8060/8071 FRU 25R8071) in PCI_E1

	- Card shows _no_ BIOS start messages
	- Card is _not_ detected in Linux

	=> lspci results in "lspci-lsi-SAS3444E"

	There are tiny diferences compared to the state without card,
	though:

	without any PCI card:

	  00:14.0 USB controller: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller (prog-if 30 [XHCI])
		Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b47
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
		Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
		Latency: 0
		Interrupt: pin A routed to IRQ 121
		Region 0: Memory at df130000 (64-bit, non-prefetchable) [size=64K]
		Capabilities: [70] Power Management version 2
			Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
			Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	!               Address: 00000000fee01004  Data: 4023
		Kernel driver in use: xhci_hcd
	...
	  00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
		Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b47
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
		Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
		Latency: 0
		Interrupt: pin A routed to IRQ 131
		Region 0: Memory at df14d000 (64-bit, non-prefetchable) [size=4K]
		Capabilities: [50] Power Management version 3
			Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
			Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	!               Address: 00000000fee10004  Data: 4023
		Kernel driver in use: mei_me
		Kernel modules: mei_me
	
	with LSI:

	  00:14.0 USB controller: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller (prog-if 30 [XHCI])
		Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b47
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
		Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
		Latency: 0
		Interrupt: pin A routed to IRQ 121
		Region 0: Memory at df130000 (64-bit, non-prefetchable) [size=64K]
		Capabilities: [70] Power Management version 2
			Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
			Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	!               Address: 00000000fee08004  Data: 4021
		Kernel driver in use: xhci_hcd
	...
	00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
		Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b47
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
		Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
		Latency: 0
		Interrupt: pin A routed to IRQ 131
		Region 0: Memory at df14d000 (64-bit, non-prefetchable) [size=4K]
		Capabilities: [50] Power Management version 3
			Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
			Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	!               Address: 00000000fee08004  Data: 4023
		Kernel driver in use: mei_me
		Kernel modules: mei_me
	
	...so _something_ is happening.

The log files are available here:

	https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr

Hope this sheds some light on the problem...

Thanks!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Old programmers never die, they just branch to a new address.

Re: PCIe x4 cards not detected on Z370 mainboards

[Bjorn Helgaas]

[+cc Mika, Rafael, linux-acpi]
[I cobbled this together from our earlier conversation to try to give
context for the new additions]

On Fri, Mar 23, 2018 at 09:48:55AM +0100, Wolfgang Denk wrote:
> In message <20180322192759.GA252023@bhelgaas-glaptop.roam.corp.google.com> Bjorn wrote:
>> Wolfgang Denk originally wrote:
>>> Both on a GIGABYTE Z370-HD3 and on a MSI Z370 TOMAHAWK mainboard
>>> neither of my LSI SAS controllers (SAS3444 E PCIe x4 and
>>> SAS3442E-R PCIe x4) gets detected.  Not by the system BIOS, there
>>> is not messages from the SAS Controller BIOS, Linux does not see
>>> it in the PCI bus scan - nothing.  Both controllers work find in
>>> older mainboards.
>>>
>>> I asked MSI support, who told me:
>>>
>>>   These are old controllers which don't support UEFI/GOP. this is
>>>   the reason they don't work.  Cards which have only legacy
>>>   support cannot be operated in modern mainboards.
>>
>> That doesn't make sense to me.  Maybe these controllers have option
>> ROMs that aren't compatible with UEFI (though even that sounds
>> unlikely to me).  But even in that case, I would expect the card to
>> at least be visible in PCI config space, because that's needed even
>> to read the option ROM from the card.
> 
> I speculate that, like on some graphics cards, the card BIOS needs
> to run to initialize for example on-board memory etc., before the
> card becomes realy visible to the rest of the system?

We can't run the card BIOS unless the card responds to the config
reads we use to enumerate the device.

>>> I also have a PCIe x 1 Adaptect SCSI controller (29320LPE,
>>> firmware version v4.31.4 of 2007) which certainly never heard of
>>> UEFI/GOP before, and this is working fine: I see the BIOS messages
>>> when it does the SCSI bus scan, I see the initialization under
>>> Linux, and I can access the SCSI devices under Linux.
>>> ...
>>>
>> With the SAS3444 in the system, I wonder the port leading to that
>> slot is disabled or not negotiating the link for some reason.  Can
>> you collect "lspci -vvv" output with the working card and with one
>> of the non-working cards?
> 
> Done.  See below.

>From lspci-adaptec-29320LPE:

  00:01.0 Intel Xeon E3-1200 v5/E3-1500 Root Port
    Bus: primary=00, secondary=01, subordinate=02
  01:00.0 PEX 8114 PCI Express-to-PCI/PCI-X Bridge
    Bus: primary=01, secondary=02, subordinate=02
  02:04.0 Adaptec ASC-29320ALP U320 (rev 10)

The 00:01.0 Root Port is built into your motherboard's chipset.  The
01:00.0 Bridge and ASC-29320ALP devices are both on the plugin card.
The card should have a PLX-labeled chip and an Adaptec-labeled one.

In lspci-lsi-SAS3444E, the 00:01.0 Root Port doesn't appear at all, so
it must be disabled, which means we can't reach the card in that slot.
Linux doesn't know how to enable the Root Port directly (it requires
chipset-specific knowledge, so it's done by the BIOS).

>>> I don't know if this is relevant, but in Linux (Fedora 27,
>>> 4.15.6-300.fc27.x86_64 kernel), I see these error messages:
>>> 
>>>   ACPI Error: [_SB_.PCI0.RP05.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
>>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
>>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP04.PXSX, AE_NOT_FOUND (20170831/psparse-550)
>>>   ACPI Error: [_SB_.PCI0.RP09.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
>>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
>>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP08.PXSX, AE_NOT_FOUND (20170831/psparse-550)

This is interesting.  These methods (PXSX) are connected with PCIe
Root Ports, and Mika's recent patch [1] also mentioned PXSX, though
that was in the context of Thunderbolt hotplug.

It's conceivable that Linux is doing something wrong here that causes
BIOS to leave that 00:01.0 Root Port disabled.

Do these PXSX messages also occur when you boot with the working
29320LPE card in the slot?

> > You could try locating the port leading to the slot and using setpci
> > to reset its secondary bus, then using /sys/bus/pci/rescan to rescan
> > the bus.
> 
> Sorry, I need help here.  Which options/parameters are needed to
> setpci to do that?

This idea won't work because the port leading to the slot isn't visible at
all.

> The log files are available here:
> 
> 	https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr

[1] https://lkml.kernel.org/r/20180212105523.15984-1-mika.westerberg@linux.intel.com

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

thanks - I really appreciate your help.

In message <20180324044716.GF210003@bhelgaas-glaptop.roam.corp.google.com> you wrote:
>
> From lspci-adaptec-29320LPE:
> 
>   00:01.0 Intel Xeon E3-1200 v5/E3-1500 Root Port
>     Bus: primary=00, secondary=01, subordinate=02
>   01:00.0 PEX 8114 PCI Express-to-PCI/PCI-X Bridge
>     Bus: primary=01, secondary=02, subordinate=02
>   02:04.0 Adaptec ASC-29320ALP U320 (rev 10)
> 
> The 00:01.0 Root Port is built into your motherboard's chipset.  The
> 01:00.0 Bridge and ASC-29320ALP devices are both on the plugin card.
> The card should have a PLX-labeled chip and an Adaptec-labeled one.

Correct.  It has an AIC-7901X (Adaptec) and a PEX8114-BD13BI G (PLX).

> >>> I don't know if this is relevant, but in Linux (Fedora 27,
> >>> 4.15.6-300.fc27.x86_64 kernel), I see these error messages:
> >>> 
> >>>   ACPI Error: [_SB_.PCI0.RP05.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP04.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> >>>   ACPI Error: [_SB_.PCI0.RP09.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP08.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> 
> This is interesting.  These methods (PXSX) are connected with PCIe
> Root Ports, and Mika's recent patch [1] also mentioned PXSX, though
> that was in the context of Thunderbolt hotplug.
> 
> It's conceivable that Linux is doing something wrong here that causes
> BIOS to leave that 00:01.0 Root Port disabled.

But is not in any case the problem at least also in the mainboard
BIOS? Should we not see the card's BIOS start messages in any case?
For example, if I wanted to boot from a disk attached to this SAS
controller, it would need to be up and running for booting Linux.

> Do these PXSX messages also occur when you boot with the working
> 29320LPE card in the slot?

Yes, they appear always - without any card, with the working Adaptec
SCSI controller, and with the non-working LSI SAS controllers.

For reference, I have uploaded the complete boot logs for these
three cases to the same directory, see files "dmesg-no-card",
"lspci-adaptec-29320LPE" and "lspci-lsi-SAS3444E" at

https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Let's say the docs present a simplified view of reality...    :-)
                      - Larry Wall in  <6940@jpl-devvax.JPL.NASA.GOV>

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

maybe it helps to provide some more information.  I can reproduce the
same problem on 3 different mainboards from 2 different vendors:

- Gigabyte AB350 Gaming CF
- Gigabyte Z270X Ultra Gaming CF
- MSI Z370 Tomahawk

I now have all 3 systems ready at hands to run additional tests if
needed.

I have updated the information (lspci output, dmesg output) for all
three boards here:

https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr

Thanks in advance!

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Due to lack of disk space, this fortune database has been discontinued.

Re: PCIe x4 cards not detected on Z370 mainboards

[Mika Westerberg]

On Sun, Mar 25, 2018 at 02:28:28PM +0200, Wolfgang Denk wrote:
> Dear Bjorn,
> 
> maybe it helps to provide some more information.  I can reproduce the
> same problem on 3 different mainboards from 2 different vendors:
> 
> - Gigabyte AB350 Gaming CF
> - Gigabyte Z270X Ultra Gaming CF
> - MSI Z370 Tomahawk
>
> 
> I now have all 3 systems ready at hands to run additional tests if
> needed.
> 
> I have updated the information (lspci output, dmesg output) for all
> three boards here:
> 
> https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr
> 

Looking at the MSI Tomahawk dmesg when you have the adaptec card
connected, the BIOS adds an additional ACPI SSDT table:

[    0.000000] ACPI: SSDT 0x000000008C039120 002611 (v02 PegSsd PegSsdt 00001000 INTL 20160422)

This is missing when LSI card is connected. There might be some sort of
preboot protocol or whitelist that prevents the root port to be enabled
in BIOS.

Wolfgang, could you also add acpidump output of these systems to that
share site?

Re: PCIe x4 cards not detected on Z370 mainboards

[Bjorn Helgaas]

On Sat, Mar 24, 2018 at 09:15:34PM +0100, Wolfgang Denk wrote:
> Dear Bjorn,
> 
> thanks - I really appreciate your help.
> 
> In message <20180324044716.GF210003@bhelgaas-glaptop.roam.corp.google.com> you wrote:
> >
> > From lspci-adaptec-29320LPE:
> > 
> >   00:01.0 Intel Xeon E3-1200 v5/E3-1500 Root Port
> >     Bus: primary=00, secondary=01, subordinate=02
> >   01:00.0 PEX 8114 PCI Express-to-PCI/PCI-X Bridge
> >     Bus: primary=01, secondary=02, subordinate=02
> >   02:04.0 Adaptec ASC-29320ALP U320 (rev 10)
> > 
> > The 00:01.0 Root Port is built into your motherboard's chipset.  The
> > 01:00.0 Bridge and ASC-29320ALP devices are both on the plugin card.
> > The card should have a PLX-labeled chip and an Adaptec-labeled one.
> 
> Correct.  It has an AIC-7901X (Adaptec) and a PEX8114-BD13BI G (PLX).
> 
> > >>> I don't know if this is relevant, but in Linux (Fedora 27,
> > >>> 4.15.6-300.fc27.x86_64 kernel), I see these error messages:
> > >>> 
> > >>>   ACPI Error: [_SB_.PCI0.RP05.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> > >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> > >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP04.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> > >>>   ACPI Error: [_SB_.PCI0.RP09.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> > >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> > >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP08.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> > 
> > This is interesting.  These methods (PXSX) are connected with PCIe
> > Root Ports, and Mika's recent patch [1] also mentioned PXSX, though
> > that was in the context of Thunderbolt hotplug.
> > 
> > It's conceivable that Linux is doing something wrong here that causes
> > BIOS to leave that 00:01.0 Root Port disabled.
> 
> But is not in any case the problem at least also in the mainboard
> BIOS? Should we not see the card's BIOS start messages in any case?
> For example, if I wanted to boot from a disk attached to this SAS
> controller, it would need to be up and running for booting Linux.

Oh, you're right; I forgot about that point.

I guess you've already talked to MSI support, who told you:

  These are old controllers which don't support UEFI/GOP. this
  is the reason they don't work.  Cards which have only legacy
  support cannot be operated in modern mainboards.

That seems like sort of a lame excuse as a consumer, but if the BIOS
writers decided they don't want to support older adapters, I don't
know what we can really do about it.

I searched for other reports ("GIGABYTE Z370-HD3 not detected") and
found several that could be similar.  One suggested enabling/disabling
CSM (UEFI compatibility support module) in the BIOS setup.  If these
BIOSes in fact decided not to support these old cards, my guess is
there is no CSM in them, but it's worth checking.

Otherwise, I think I'm out of ideas.  I don't think we've seen
anything yet that indicates a Linux issue, although maybe Mika will
come up with something from the acpidumps.

Bjorn

Re: PCIe x4 cards not detected on Z370 mainboards

[Bjorn Helgaas]

On Mon, Mar 26, 2018 at 08:51:07AM -0500, Bjorn Helgaas wrote:
> On Sat, Mar 24, 2018 at 09:15:34PM +0100, Wolfgang Denk wrote:
> > Dear Bjorn,
> > 
> > thanks - I really appreciate your help.
> > 
> > In message <20180324044716.GF210003@bhelgaas-glaptop.roam.corp.google.com> you wrote:
> > >
> > > From lspci-adaptec-29320LPE:
> > > 
> > >   00:01.0 Intel Xeon E3-1200 v5/E3-1500 Root Port
> > >     Bus: primary=00, secondary=01, subordinate=02
> > >   01:00.0 PEX 8114 PCI Express-to-PCI/PCI-X Bridge
> > >     Bus: primary=01, secondary=02, subordinate=02
> > >   02:04.0 Adaptec ASC-29320ALP U320 (rev 10)
> > > 
> > > The 00:01.0 Root Port is built into your motherboard's chipset.  The
> > > 01:00.0 Bridge and ASC-29320ALP devices are both on the plugin card.
> > > The card should have a PLX-labeled chip and an Adaptec-labeled one.
> > 
> > Correct.  It has an AIC-7901X (Adaptec) and a PEX8114-BD13BI G (PLX).
> > 
> > > >>> I don't know if this is relevant, but in Linux (Fedora 27,
> > > >>> 4.15.6-300.fc27.x86_64 kernel), I see these error messages:
> > > >>> 
> > > >>>   ACPI Error: [_SB_.PCI0.RP05.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> > > >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> > > >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP04.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> > > >>>   ACPI Error: [_SB_.PCI0.RP09.PXSX] Namespace lookup failure, AE_NOT_FOUND (20170831/dswload2-191)
> > > >>>   ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20170831/psobject-252)
> > > >>>   ACPI Error: Method parse/execution failed \_SB.PCI0.RP08.PXSX, AE_NOT_FOUND (20170831/psparse-550)
> > > 
> > > This is interesting.  These methods (PXSX) are connected with PCIe
> > > Root Ports, and Mika's recent patch [1] also mentioned PXSX, though
> > > that was in the context of Thunderbolt hotplug.
> > > 
> > > It's conceivable that Linux is doing something wrong here that causes
> > > BIOS to leave that 00:01.0 Root Port disabled.
> > 
> > But is not in any case the problem at least also in the mainboard
> > BIOS? Should we not see the card's BIOS start messages in any case?
> > For example, if I wanted to boot from a disk attached to this SAS
> > controller, it would need to be up and running for booting Linux.
> 
> Oh, you're right; I forgot about that point.
> 
> I guess you've already talked to MSI support, who told you:
> 
>   These are old controllers which don't support UEFI/GOP. this
>   is the reason they don't work.  Cards which have only legacy
>   support cannot be operated in modern mainboards.
> 
> That seems like sort of a lame excuse as a consumer, but if the BIOS
> writers decided they don't want to support older adapters, I don't
> know what we can really do about it.
> 
> I searched for other reports ("GIGABYTE Z370-HD3 not detected") and
> found several that could be similar.  One suggested enabling/disabling
> CSM (UEFI compatibility support module) in the BIOS setup.  If these
> BIOSes in fact decided not to support these old cards, my guess is
> there is no CSM in them, but it's worth checking.

P.S. https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
has a little interesting background on UEFI/CSM.

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Mika,

In message <20180326112647.GC2703@lahna.fi.intel.com> you wrote:
>
> > I have updated the information (lspci output, dmesg output) for all
> > three boards here:
> > 
> > https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr
> 
> Looking at the MSI Tomahawk dmesg when you have the adaptec card
> connected, the BIOS adds an additional ACPI SSDT table:
> 
> [    0.000000] ACPI: SSDT 0x000000008C039120 002611 (v02 PegSsd PegSsdt 00001000 INTL 20160422)
> 
> This is missing when LSI card is connected. There might be some sort of
> preboot protocol or whitelist that prevents the root port to be enabled
> in BIOS.
> 
> Wolfgang, could you also add acpidump output of these systems to that
> share site?

Done.

[I hope this is what you need.  Fedora does not seem to provide the
acpidump tool ready to use in any package, but I found the sources
in the kcbench package.]

All three boards behave a bit different here, but see yourself...

[What they have in common is that none works as I would like to have
it :-( ]


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I can't understand it. I can't even understand  the  people  who  can
understand it.                    - Queen Juliana of the Netherlands.

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

In message <20180326135107.GH210003@bhelgaas-glaptop.roam.corp.google.com> you wrote:
>
> I guess you've already talked to MSI support, who told you:
> 
>   These are old controllers which don't support UEFI/GOP. this
>   is the reason they don't work.  Cards which have only legacy
>   support cannot be operated in modern mainboards.
> 
> That seems like sort of a lame excuse as a consumer, but if the BIOS
> writers decided they don't want to support older adapters, I don't
> know what we can really do about it.

What makes me wonder is why the Adaptec card behaves different here
- it is as old as the LSI controllers and certainly has no UEFI
compatible BIOS either.

> I searched for other reports ("GIGABYTE Z370-HD3 not detected") and
> found several that could be similar.  One suggested enabling/disabling

I've seen these, too. Those that looked reliable were mostly about
graphics cards, though, and recent ones as well.

> CSM (UEFI compatibility support module) in the BIOS setup.  If these
> BIOSes in fact decided not to support these old cards, my guess is
> there is no CSM in them, but it's worth checking.

At least it is not mentioned in the documentation, and I cannot find
any settings in the BIOS where I could switch off UEFI.

> Otherwise, I think I'm out of ideas.  I don't think we've seen
> anything yet that indicates a Linux issue, although maybe Mika will
> come up with something from the acpidumps.

Agreed.  Any information I could use for communication with the
motherboard vendors would be helpful.

What I cannot understand is why I am apparently one of the first
ones to run into such an issue?  OK, SAS controllers is more server
stuff, and server mainboards are apparently more conservative in
their settings.  But still...

Thanks!

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Copy from one, it's plagiarism; copy from two, it's research.

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

In message <20180326135235.GA221217@bhelgaas-glaptop.roam.corp.google.com> you wrote:
>
> P.S. https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
> has a little interesting background on UEFI/CSM.

Thanks for the pointer.

I start wondering if the mainboard / BIOS vendors stop supporting
such older hardware, if such support could be re-added in Linux?
OK, obviously not for the boot device, but...

In U-Boot we have an x86 emulator to be able to initialize the
on-board BIOS on (at least some) graphics cards so you can use these
on ARM or Power architecture systems.  Do we really have to add
similar stuff to Linux to support legacy hardware?

Would that even work?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"I didn't know it was impossible when I did it."

Re: PCIe x4 cards not detected on Z370 mainboards

[Mika Westerberg]

On Mon, Mar 26, 2018 at 05:00:50PM +0200, Wolfgang Denk wrote:
> Dear Mika,
> 
> In message <20180326112647.GC2703@lahna.fi.intel.com> you wrote:
> >
> > > I have updated the information (lspci output, dmesg output) for all
> > > three boards here:
> > > 
> > > https://owncloud.denx.de/index.php/s/wXiCMMYJodGj5Kr
> > 
> > Looking at the MSI Tomahawk dmesg when you have the adaptec card
> > connected, the BIOS adds an additional ACPI SSDT table:
> > 
> > [    0.000000] ACPI: SSDT 0x000000008C039120 002611 (v02 PegSsd PegSsdt 00001000 INTL 20160422)
> > 
> > This is missing when LSI card is connected. There might be some sort of
> > preboot protocol or whitelist that prevents the root port to be enabled
> > in BIOS.
> > 
> > Wolfgang, could you also add acpidump output of these systems to that
> > share site?
> 
> Done.
> 
> [I hope this is what you need.  Fedora does not seem to provide the
> acpidump tool ready to use in any package, but I found the sources
> in the kcbench package.]

Yes, that's exactly what I wanted. Thanks!

> All three boards behave a bit different here, but see yourself...
> 
> [What they have in common is that none works as I would like to have
> it :-( ]

I compared DSDT of the MSI Tomahawk with Adaptec connected to LSI
connected and there is pretty much no difference. So the BIOS decides to
load that PegSsd SSDT table based on something else than what is in DSDT.

The Gigabyte AB350 does not do any tricks with the ACPI tables. However,
comparing dmesgs you can see that the kernel simply does not find anything
below the bridge.

Adaptec:
[    0.064806] pci 0000:04:04.0: PCI bridge to [bus 07-08]
[    0.064912] pci 0000:04:04.0:   bridge window [io  0xd000-0xdfff]
[    0.064915] pci 0000:04:04.0:   bridge window [mem 0xfd200000-0xfd3fffff]
[    0.065004] pci 0000:08:04.0: [9005:8017] type 00 class 0x010000
[    0.065050] pci 0000:08:04.0: reg 0x10: [io  0xd100-0xd1ff]
[    0.065077] pci 0000:08:04.0: reg 0x14: [mem 0xfd280000-0xfd281fff 64bit]
[    0.065095] pci 0000:08:04.0: reg 0x1c: [io  0xd000-0xd0ff]
[    0.065145] pci 0000:08:04.0: reg 0x30: [mem 0xfd200000-0xfd27ffff pref]
[    0.065312] pci 0000:07:00.0: PCI bridge to [bus 08]
[    0.065424] pci 0000:07:00.0:   bridge window [io  0xd000-0xdfff]
[    0.065429] pci 0000:07:00.0:   bridge window [mem 0xfd200000-0xfd2fffff]

LSI:
[    0.064312] pci 0000:04:04.0: PCI bridge to [bus 07]

I guess MSI Tomahawk BIOS enumerates PCI devices as well and perhaps it
disables the PegSsd SSDT table because it cannot find anything below the
PCI bridge 04:04.

So at this point I have no idea either what else you could try except
perhaps booting to Windows and see if it detects the LSI card.

Re: PCIe x4 cards not detected on Z370 mainboards

[Bjorn Helgaas]

On Mon, Mar 26, 2018 at 05:12:52PM +0200, Wolfgang Denk wrote:
> In message <20180326135235.GA221217@bhelgaas-glaptop.roam.corp.google.com> you wrote:
> >
> > P.S. https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
> > has a little interesting background on UEFI/CSM.
> 
> Thanks for the pointer.
> 
> I start wondering if the mainboard / BIOS vendors stop supporting
> such older hardware, if such support could be re-added in Linux?
> OK, obviously not for the boot device, but...
> 
> In U-Boot we have an x86 emulator to be able to initialize the
> on-board BIOS on (at least some) graphics cards so you can use these
> on ARM or Power architecture systems.  Do we really have to add
> similar stuff to Linux to support legacy hardware?
> 
> Would that even work?

On the MSI Z370 Tomahawk and the Gigabyte Z270X Ultra Gaming systems,
it looks like you're using what the BIOS designates as "Slot #1",
which is below an Intel Root Port (00:01.0 in both cases).  When the
SAS3444E is in that slot, we don't even see the Root Port itself.  To
fix that would require Linux to have chipset-specific knowledge about
how to enable that Root Port.  I guess it's conceivable that could be
done, but I don't know if the information required is public.  I
assume this would be controlled by some MSR or other Intel-specific
knob.

On the Gigabyte AB350 Gaming system, you're using "Slot #1" (this
comes from the Slot Capabilities register and may not match any
silk-screened labels on the board).  This is below an AMD PCIe switch,
where 04:04.0 is the Downstream Port leading to that slot.  When the
SAS3444E is in the slot, we do see the 04:04.0 Downstream Port, so we
*should* still be able to see the SAS device.

This is now all in the PCIe domain so the BIOS should be pretty much
out of the picture, and I can't explain why we don't see the SAS3444E
device.

On that system, you could try something like this to reset the
SAS3444E device and try to rescan the bus:

  # setpci -s04:04.0 BRIDGE_CONTROL.w=0x40
  # sleep 1
  # setpci -s04:04.0 BRIDGE_CONTROL.w=0x00
  # sleep 10
  # echo 1 > /sys/bus/pci/rescan

Check the dmesg log afterwards to see if anything happened.

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Bjorn,

In message <20180326190938.GA224629@bhelgaas-glaptop.roam.corp.google.com> you wrote:
>
> On the MSI Z370 Tomahawk and the Gigabyte Z270X Ultra Gaming systems,
> it looks like you're using what the BIOS designates as "Slot #1",
> which is below an Intel Root Port (00:01.0 in both cases).  When the

Correct.  I always used the first available (x16) slot; on these two
boards I have Intel Core i5 processors, so I can use the on-cip
graphics and have no need for a graphics adapter, so this is really
the first PCIe x16 slot.

> On the Gigabyte AB350 Gaming system, you're using "Slot #1" (this
> comes from the Slot Capabilities register and may not match any
> silk-screened labels on the board).  This is below an AMD PCIe switch,
> where 04:04.0 is the Downstream Port leading to that slot.  When the
> SAS3444E is in the slot, we do see the 04:04.0 Downstream Port, so we
> *should* still be able to see the SAS device.

This board has a AMD Ryzen 5 1600 Six-Core Processor, so I have to
use a graphics adapter, which occupies the 1st PCIe x16 slot.  So I
used the next frree one.

> On that system, you could try something like this to reset the
> SAS3444E device and try to rescan the bus:

And now comes the next surprise:

On this board the LSI controller suddenly started working!  I get
the BIOS start message from it, and Linux detects it...

The only difference is that i have installed the kcbench package to
get at the acpidump tool:

2018-03-26T14:15:15Z INFO Installed: kcbench-data-4.4-0.1-21.fc27.noarch
2018-03-26T14:15:16Z INFO Installed: kcbench-0.3-17.fc27.noarch
2018-03-26T14:15:16Z INFO Installed: kcbench-data-0.1-21.fc27.noarch

No other changes to the hardware or software at all.  And I always
powered off the system when changing the boards, so even this makes
no difference.

Oh, and we are running at DST now, so maybe it is POM dependent :-(


I have uploaded updated lspci/dmesg/acpidump output for this new,
working case, see files

Gigabyte-AB350-Gaming-CF-lspci-lsi-sas3444e.1
Gigabyte-AB350-Gaming-CF-dmesg-lsi-sas3444e.1
Gigabyte-AB350-Gaming-CF-acpidump-lsi-sas3444e.1

On the other Gigabyte board I see tiny changes in the logs, but the
card is still not working.  New log files (sith suffix .1) also
uploaded.

Gigbyte support recommeded to "try  Other PCI devices: Legacy in
bios setup".  Good point.  Tried that on the remainin non-working
Gigabyte board.  Some difference in the acpidump, no differences in
lspci output or behaviour.  Nonworking. New log files with suffix .2

Strange...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The thing is, as you progress in the Craft,  you'll  learn  there  is
another rule... When you break rules, break 'em good and hard.
                                    - Terry Pratchett, _Wyrd Sisters_

Re: PCIe x4 cards not detected on Z370 mainboards

[Wolfgang Denk]

Dear Mika,

In message <20180326155904.GJ2703@lahna.fi.intel.com> you wrote:
>
> So at this point I have no idea either what else you could try except
> perhaps booting to Windows and see if it detects the LSI card.

Windows?  shudder....


Oh well... it does not.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Ahead warp factor 1"                                  - Captain Kirk