Kernel 5.9 IOMMU groups regression/change

Kernel 5.9 IOMMU groups regression/change

[Boris V.]

With upgrade to kernel 5.9 my VMs stopped working, because some devices 
can't be passed through.
This is caused by different IOMMU groups and devices being in the same 
group.

For ex. with kernel 5.8 this are IOMMU groups:
IOMMU Group 40:
         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 43:
         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 
Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 44:
         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A 
USB 3.0 Host Controller [1b21:1142]

Ethernet, SATA and USB controller in its own group.

And with 5.9, everything is in one group:
IOMMU Group 29:
         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
Gigabit Network Connection [8086:1539] (rev 03)
         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 
Serial ATA Controller [1b21:0612] (rev 02)
         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A 
USB 3.0 Host Controller [1b21:1142]


This seems to be caused by commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
Author: Rajat Jain <rajatja@google.com>
Date:   Tue Jul 7 15:46:02 2020 -0700

     PCI: Cache ACS capability offset in device

     Currently the ACS capability is being looked up at a number of 
places. Read
     and store it once at enumeration so that it can be used by all 
later.  No
     functional change intended.

     Link: 
https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
     Signed-off-by: Rajat Jain <rajatja@google.com>
     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>

  drivers/pci/p2pdma.c |  2 +-
  drivers/pci/pci.c    | 20 ++++++++++++++++----
  drivers/pci/pci.h    |  2 +-
  drivers/pci/probe.c  |  2 +-
  drivers/pci/quirks.c |  8 ++++----
  include/linux/pci.h  |  1 +
  6 files changed, 24 insertions(+), 11 deletions(-)


If I revert this commit, I get back old groups.

In commit log there is message 'No functional change intended'. But 
there is functional change.

This is Intel Core i7-5930K CPU and X99 chipset. But I see the same 
thing on other Intel systems (didn't test on AMD).


Below are full IOMMU groups.

Kernel 5.8 or 5.9 with commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60 
reverted:

IOMMU Group 0:
         ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
         ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
         ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
IOMMU Group 1:
         ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
         ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
         ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
         ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
         ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe4] (rev 02)
         ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe5] (rev 02)
IOMMU Group 10:
         ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
         ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
         ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
         ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
         ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
IOMMU Group 11:
         ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
         ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
IOMMU Group 12:
         00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon 
E5 v3/Core i7 DMI2 [8086:2f00] (rev 02)
IOMMU Group 13:
         00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
IOMMU Group 14:
         00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02)
IOMMU Group 15:
         00:02.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 2 [8086:2f04] (rev 02)
IOMMU Group 16:
         00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
IOMMU Group 17:
         00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02)
IOMMU Group 18:
         00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management 
[8086:2f28] (rev 02)
IOMMU Group 19:
         00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
IOMMU Group 2:
         ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
         ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
         ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffc] (rev 02)
         ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffd] (rev 02)
         ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffe] (rev 02)
IOMMU Group 20:
         00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] 
(rev 02)
IOMMU Group 21:
         00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
IOMMU Group 22:
         00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 
series chipset SPSR [8086:8d7c] (rev 05)
IOMMU Group 23:
         00:11.4 SATA controller [0106]: Intel Corporation C610/X99 
series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
IOMMU Group 24:
         00:14.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
IOMMU Group 25:
         00:16.0 Communication controller [0780]: Intel Corporation 
C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
IOMMU Group 26:
         00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet 
Connection (2) I218-V [8086:15a1] (rev 05)
IOMMU Group 27:
         00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
IOMMU Group 28:
         00:1b.0 Audio device [0403]: Intel Corporation C610/X99 series 
chipset HD Audio Controller [8086:8d20] (rev 05)
IOMMU Group 29:
         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
IOMMU Group 3:
         ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
         ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
         ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
         ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
         ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
IOMMU Group 30:
         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
IOMMU Group 31:
         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
IOMMU Group 32:
         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
IOMMU Group 33:
         00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
IOMMU Group 34:
         00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series 
chipset LPC Controller [8086:8d47] (rev 05)
         00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 
series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
         00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset 
SMBus Controller [8086:8d22] (rev 05)
IOMMU Group 35:
         05:00.0 Non-Volatile memory controller [0108]: Samsung 
Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 36:
         03:00.0 VGA compatible controller [0300]: NVIDIA Corporation 
GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
         03:00.1 Audio device [0403]: NVIDIA Corporation GM204 High 
Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU Group 37:
         02:00.0 Ethernet controller [0200]: Mellanox Technologies 
MT27500 Family [ConnectX-3] [15b3:1003]
IOMMU Group 38:
         01:00.0 VGA compatible controller [0300]: NVIDIA Corporation 
GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
         01:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High 
Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 39:
         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
IOMMU Group 4:
         ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
         ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
IOMMU Group 40:
         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 41:
         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
IOMMU Group 42:
         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
IOMMU Group 43:
         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 
Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 44:
         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A 
USB 3.0 Host Controller [1b21:1142]
IOMMU Group 5:
         ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2faa] (rev 02)
         ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fab] (rev 02)
         ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fac] (rev 02)
         ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fad] (rev 02)
         ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
         ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
IOMMU Group 6:
         ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal 
Control [8086:2fb0] (rev 02)
         ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal 
Control [8086:2fb1] (rev 02)
         ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR 
Registers [8086:2fb2] (rev 02)
         ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR 
Registers [8086:2fb3] (rev 02)
         ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
         ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
         ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
         ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
IOMMU Group 7:
         ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal 
Control [8086:2fb4] (rev 02)
         ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal 
Control [8086:2fb5] (rev 02)
         ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR 
Registers [8086:2fb6] (rev 02)
         ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR 
Registers [8086:2fb7] (rev 02)
IOMMU Group 8:
         ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
         ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
IOMMU Group 9:
         ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal 
Control [8086:2fd0] (rev 02)
         ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
         ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
         ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
         ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)


Kernel 5.9:

IOMMU Group 0:
         ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
         ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
         ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
IOMMU Group 1:
         ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
         ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
         ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
         ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
         ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe4] (rev 02)
         ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe5] (rev 02)
IOMMU Group 10:
         ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
         ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
         ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
         ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
         ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
IOMMU Group 11:
         ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
         ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
IOMMU Group 12:
         00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon 
E5 v3/Core i7 DMI2 [8086:2f00] (rev 02)
IOMMU Group 13:
         00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
IOMMU Group 14:
         00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02)
IOMMU Group 15:
         00:02.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 2 [8086:2f04] (rev 02)
IOMMU Group 16:
         00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
IOMMU Group 17:
         00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02)
IOMMU Group 18:
         00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management 
[8086:2f28] (rev 02)
IOMMU Group 19:
         00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
IOMMU Group 2:
         ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
         ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
         ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffc] (rev 02)
         ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffd] (rev 02)
         ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers 
[8086:2ffe] (rev 02)
IOMMU Group 20:
         00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] 
(rev 02)
IOMMU Group 21:
         00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 
v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
IOMMU Group 22:
         00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 
series chipset SPSR [8086:8d7c] (rev 05)
IOMMU Group 23:
         00:11.4 SATA controller [0106]: Intel Corporation C610/X99 
series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
IOMMU Group 24:
         00:14.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
IOMMU Group 25:
         00:16.0 Communication controller [0780]: Intel Corporation 
C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
IOMMU Group 26:
         00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet 
Connection (2) I218-V [8086:15a1] (rev 05)
IOMMU Group 27:
         00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
IOMMU Group 28:
         00:1b.0 Audio device [0403]: Intel Corporation C610/X99 series 
chipset HD Audio Controller [8086:8d20] (rev 05)
IOMMU Group 29:
         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series 
chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
[1b21:118f]
         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
Gigabit Network Connection [8086:1539] (rev 03)
         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 
Serial ATA Controller [1b21:0612] (rev 02)
         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A 
USB 3.0 Host Controller [1b21:1142]
IOMMU Group 3:
         ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
         ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
         ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
         ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
         ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
IOMMU Group 30:
         00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 
series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
IOMMU Group 31:
         00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series 
chipset LPC Controller [8086:8d47] (rev 05)
         00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 
series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
         00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset 
SMBus Controller [8086:8d22] (rev 05)
IOMMU Group 32:
         05:00.0 Non-Volatile memory controller [0108]: Samsung 
Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 33:
         03:00.0 VGA compatible controller [0300]: NVIDIA Corporation 
GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
         03:00.1 Audio device [0403]: NVIDIA Corporation GM204 High 
Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU Group 34:
         02:00.0 Ethernet controller [0200]: Mellanox Technologies 
MT27500 Family [ConnectX-3] [15b3:1003]
IOMMU Group 35:
         01:00.0 VGA compatible controller [0300]: NVIDIA Corporation 
GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
         01:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High 
Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 4:
         ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
         ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
IOMMU Group 5:
         ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2faa] (rev 02)
         ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fab] (rev 02)
         ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fac] (rev 02)
         ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target 
Address Decoder [8086:2fad] (rev 02)
         ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
         ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
IOMMU Group 6:
         ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal 
Control [8086:2fb0] (rev 02)
         ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal 
Control [8086:2fb1] (rev 02)
         ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR 
Registers [8086:2fb2] (rev 02)
         ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR 
Registers [8086:2fb3] (rev 02)
         ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
         ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
         ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
         ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
IOMMU Group 7:
         ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal 
Control [8086:2fb4] (rev 02)
         ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal 
Control [8086:2fb5] (rev 02)
         ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR 
Registers [8086:2fb6] (rev 02)
         ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR 
Registers [8086:2fb7] (rev 02)
IOMMU Group 8:
         ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, 
Thermal & RAS Registers [8086... (rev 02)
         ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
         ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
IOMMU Group 9:
         ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal 
Control [8086:2fd0] (rev 02)
         ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
         ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
         ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
         ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 
v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)

Re: Kernel 5.9 IOMMU groups regression/change

[Boris V.]

On 25/10/2020 20:45, Boris V. wrote:
> With upgrade to kernel 5.9 my VMs stopped working, because some 
> devices can't be passed through.
> This is caused by different IOMMU groups and devices being in the same 
> group.
>
> For ex. with kernel 5.8 this are IOMMU groups:
> IOMMU Group 40:
>         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
> [1b21:118f]
>         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
> Gigabit Network Connection [8086:1539] (rev 03)
> IOMMU Group 43:
>         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. 
> ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
> IOMMU Group 44:
>         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. 
> ASM1042A USB 3.0 Host Controller [1b21:1142]
>
> Ethernet, SATA and USB controller in its own group.
>
> And with 5.9, everything is in one group:
> IOMMU Group 29:
>         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series 
> chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
>         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series 
> chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
>         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series 
> chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
>         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series 
> chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
>         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
> [1b21:118f]
>         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
> [1b21:118f]
>         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
> [1b21:118f]
>         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device 
> [1b21:118f]
>         09:00.0 Ethernet controller [0200]: Intel Corporation I211 
> Gigabit Network Connection [8086:1539] (rev 03)
>         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. 
> ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
>         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. 
> ASM1042A USB 3.0 Host Controller [1b21:1142]
>
>
> This seems to be caused by commit 
> 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
> commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
> Author: Rajat Jain <rajatja@google.com>
> Date:   Tue Jul 7 15:46:02 2020 -0700
>
>     PCI: Cache ACS capability offset in device
>
>     Currently the ACS capability is being looked up at a number of 
> places. Read
>     and store it once at enumeration so that it can be used by all 
> later.  No
>     functional change intended.
>
>     Link: 
> https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
>     Signed-off-by: Rajat Jain <rajatja@google.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>
>  drivers/pci/p2pdma.c |  2 +-
>  drivers/pci/pci.c    | 20 ++++++++++++++++----
>  drivers/pci/pci.h    |  2 +-
>  drivers/pci/probe.c  |  2 +-
>  drivers/pci/quirks.c |  8 ++++----
>  include/linux/pci.h  |  1 +
>  6 files changed, 24 insertions(+), 11 deletions(-)
>
>
> If I revert this commit, I get back old groups.
>
> In commit log there is message 'No functional change intended'. But 
> there is functional change.
>
> This is Intel Core i7-5930K CPU and X99 chipset. But I see the same 
> thing on other Intel systems (didn't test on AMD).
>
>

Some more info.
Problem seems to be that pci_dev_specific_enable_acs() is not called 
anymore.
Before, pci_enable_acs() was called from pci_init_capabilities() and in 
pci_enable_acs(), pci_dev_specific_enable_acs() was called.
I don't know anything about PCI and this stuff, but I'm guessing that 
this function enable ACS for some Intel devices.
But after this commit, pci_acs_init() is called from 
pci_init_capabilities() and if pci_find_ext_capability(dev, 
PCI_EXT_CAP_ID_ACS) returns 0,
pci_enable_acs() and pci_dev_specific_enable_acs() is not called anymore.
If I apply for ex. this patch bellow, groups are right again and 
everything works as before.

diff -ur linux-5.9.1.orig/drivers/pci/pci.c linux-5.9.1/drivers/pci/pci.c
--- linux-5.9.1.orig/drivers/pci/pci.c  2020-10-17 08:31:22.000000000 +0200
+++ linux-5.9.1/drivers/pci/pci.c       2020-10-27 19:01:32.650010803 +0100
@@ -3502,9 +3502,7 @@
  void pci_acs_init(struct pci_dev *dev)
  {
         dev->acs_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
-
-       if (dev->acs_cap)
-               pci_enable_acs(dev);
+       pci_enable_acs(dev);
  }

  /**

Re: Kernel 5.9 IOMMU groups regression/change

[Bjorn Helgaas]

[+cc Rajat, LKML]

On Tue, Oct 27, 2020 at 08:31:09PM +0100, Boris V. wrote:
> On 25/10/2020 20:45, Boris V. wrote:
> > With upgrade to kernel 5.9 my VMs stopped working, because some devices
> > can't be passed through.
> > This is caused by different IOMMU groups and devices being in the same
> > group.
> > 
> > For ex. with kernel 5.8 this are IOMMU groups:
> > IOMMU Group 40:
> >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > [1b21:118f]
> >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > Gigabit Network Connection [8086:1539] (rev 03)
> > IOMMU Group 43:
> >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > Serial ATA Controller [1b21:0612] (rev 02)
> > IOMMU Group 44:
> >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > USB 3.0 Host Controller [1b21:1142]
> > 
> > Ethernet, SATA and USB controller in its own group.
> > 
> > And with 5.9, everything is in one group:
> > IOMMU Group 29:
> >         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series
> > chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
> >         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series
> > chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
> >         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series
> > chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
> >         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series
> > chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
> >         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > [1b21:118f]
> >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > [1b21:118f]
> >         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > [1b21:118f]
> >         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > [1b21:118f]
> >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > Gigabit Network Connection [8086:1539] (rev 03)
> >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > Serial ATA Controller [1b21:0612] (rev 02)
> >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > USB 3.0 Host Controller [1b21:1142]
> > 
> > 
> > This seems to be caused by commit
> > 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
> > commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
> > Author: Rajat Jain <rajatja@google.com>
> > Date:   Tue Jul 7 15:46:02 2020 -0700
> > 
> >     PCI: Cache ACS capability offset in device
> > 
> >     Currently the ACS capability is being looked up at a number of
> > places. Read
> >     and store it once at enumeration so that it can be used by all
> > later.  No
> >     functional change intended.
> > 
> >     Link:
> > https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
> >     Signed-off-by: Rajat Jain <rajatja@google.com>
> >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> > 
> >  drivers/pci/p2pdma.c |  2 +-
> >  drivers/pci/pci.c    | 20 ++++++++++++++++----
> >  drivers/pci/pci.h    |  2 +-
> >  drivers/pci/probe.c  |  2 +-
> >  drivers/pci/quirks.c |  8 ++++----
> >  include/linux/pci.h  |  1 +
> >  6 files changed, 24 insertions(+), 11 deletions(-)
> > 
> > 
> > If I revert this commit, I get back old groups.
> > 
> > In commit log there is message 'No functional change intended'. But
> > there is functional change.
> > 
> > This is Intel Core i7-5930K CPU and X99 chipset. But I see the same
> > thing on other Intel systems (didn't test on AMD).
> 
> Some more info.
> Problem seems to be that pci_dev_specific_enable_acs() is not called
> anymore.
> Before, pci_enable_acs() was called from pci_init_capabilities() and in
> pci_enable_acs(), pci_dev_specific_enable_acs() was called.
> I don't know anything about PCI and this stuff, but I'm guessing that this
> function enable ACS for some Intel devices.
> But after this commit, pci_acs_init() is called from pci_init_capabilities()
> and if pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) returns 0,
> pci_enable_acs() and pci_dev_specific_enable_acs() is not called anymore.
> If I apply for ex. this patch bellow, groups are right again and everything
> works as before.

Thanks very much for the report and the debugging.  Maybe we can get
this sorted and fixed for v5.10-rc2 or -rc3.

> diff -ur linux-5.9.1.orig/drivers/pci/pci.c linux-5.9.1/drivers/pci/pci.c
> --- linux-5.9.1.orig/drivers/pci/pci.c  2020-10-17 08:31:22.000000000 +0200
> +++ linux-5.9.1/drivers/pci/pci.c       2020-10-27 19:01:32.650010803 +0100
> @@ -3502,9 +3502,7 @@
>  void pci_acs_init(struct pci_dev *dev)
>  {
>         dev->acs_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
> -
> -       if (dev->acs_cap)
> -               pci_enable_acs(dev);
> +       pci_enable_acs(dev);
>  }
> 
>  /**
> 
> 

Re: Kernel 5.9 IOMMU groups regression/change

[Rajat Jain]

On Wed, Oct 28, 2020 at 2:52 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Rajat, LKML]
>

Thanks for copying me. (I don't look at mailing lists actively - so
missed this). Taking a look at this now.

Thanks,

Rajat


> On Tue, Oct 27, 2020 at 08:31:09PM +0100, Boris V. wrote:
> > On 25/10/2020 20:45, Boris V. wrote:
> > > With upgrade to kernel 5.9 my VMs stopped working, because some devices
> > > can't be passed through.
> > > This is caused by different IOMMU groups and devices being in the same
> > > group.
> > >
> > > For ex. with kernel 5.8 this are IOMMU groups:
> > > IOMMU Group 40:
> > >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > [1b21:118f]
> > >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > > Gigabit Network Connection [8086:1539] (rev 03)
> > > IOMMU Group 43:
> > >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > > Serial ATA Controller [1b21:0612] (rev 02)
> > > IOMMU Group 44:
> > >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > > USB 3.0 Host Controller [1b21:1142]
> > >
> > > Ethernet, SATA and USB controller in its own group.
> > >
> > > And with 5.9, everything is in one group:
> > > IOMMU Group 29:
> > >         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
> > >         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
> > >         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
> > >         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
> > >         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > [1b21:118f]
> > >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > [1b21:118f]
> > >         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > [1b21:118f]
> > >         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > [1b21:118f]
> > >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > > Gigabit Network Connection [8086:1539] (rev 03)
> > >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > > Serial ATA Controller [1b21:0612] (rev 02)
> > >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > > USB 3.0 Host Controller [1b21:1142]
> > >
> > >
> > > This seems to be caused by commit
> > > 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
> > > commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
> > > Author: Rajat Jain <rajatja@google.com>
> > > Date:   Tue Jul 7 15:46:02 2020 -0700
> > >
> > >     PCI: Cache ACS capability offset in device
> > >
> > >     Currently the ACS capability is being looked up at a number of
> > > places. Read
> > >     and store it once at enumeration so that it can be used by all
> > > later.  No
> > >     functional change intended.
> > >
> > >     Link:
> > > https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
> > >     Signed-off-by: Rajat Jain <rajatja@google.com>
> > >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> > >
> > >  drivers/pci/p2pdma.c |  2 +-
> > >  drivers/pci/pci.c    | 20 ++++++++++++++++----
> > >  drivers/pci/pci.h    |  2 +-
> > >  drivers/pci/probe.c  |  2 +-
> > >  drivers/pci/quirks.c |  8 ++++----
> > >  include/linux/pci.h  |  1 +
> > >  6 files changed, 24 insertions(+), 11 deletions(-)
> > >
> > >
> > > If I revert this commit, I get back old groups.
> > >
> > > In commit log there is message 'No functional change intended'. But
> > > there is functional change.
> > >
> > > This is Intel Core i7-5930K CPU and X99 chipset. But I see the same
> > > thing on other Intel systems (didn't test on AMD).
> >
> > Some more info.
> > Problem seems to be that pci_dev_specific_enable_acs() is not called
> > anymore.
> > Before, pci_enable_acs() was called from pci_init_capabilities() and in
> > pci_enable_acs(), pci_dev_specific_enable_acs() was called.
> > I don't know anything about PCI and this stuff, but I'm guessing that this
> > function enable ACS for some Intel devices.
> > But after this commit, pci_acs_init() is called from pci_init_capabilities()
> > and if pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) returns 0,
> > pci_enable_acs() and pci_dev_specific_enable_acs() is not called anymore.
> > If I apply for ex. this patch bellow, groups are right again and everything
> > works as before.
>
> Thanks very much for the report and the debugging.  Maybe we can get
> this sorted and fixed for v5.10-rc2 or -rc3.
>
> > diff -ur linux-5.9.1.orig/drivers/pci/pci.c linux-5.9.1/drivers/pci/pci.c
> > --- linux-5.9.1.orig/drivers/pci/pci.c  2020-10-17 08:31:22.000000000 +0200
> > +++ linux-5.9.1/drivers/pci/pci.c       2020-10-27 19:01:32.650010803 +0100
> > @@ -3502,9 +3502,7 @@
> >  void pci_acs_init(struct pci_dev *dev)
> >  {
> >         dev->acs_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
> > -
> > -       if (dev->acs_cap)
> > -               pci_enable_acs(dev);
> > +       pci_enable_acs(dev);
> >  }
> >
> >  /**
> >
> >

Re: Kernel 5.9 IOMMU groups regression/change

[Rajat Jain]

On Wed, Oct 28, 2020 at 3:07 PM Rajat Jain <rajatja@google.com> wrote:
>
> On Wed, Oct 28, 2020 at 2:52 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc Rajat, LKML]
> >
>
> Thanks for copying me. (I don't look at mailing lists actively - so
> missed this). Taking a look at this now.
>
> Thanks,
>
> Rajat
>
>
> > On Tue, Oct 27, 2020 at 08:31:09PM +0100, Boris V. wrote:
> > > On 25/10/2020 20:45, Boris V. wrote:
> > > > With upgrade to kernel 5.9 my VMs stopped working, because some devices
> > > > can't be passed through.
> > > > This is caused by different IOMMU groups and devices being in the same
> > > > group.
> > > >
> > > > For ex. with kernel 5.8 this are IOMMU groups:
> > > > IOMMU Group 40:
> > > >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > > [1b21:118f]
> > > >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > > > Gigabit Network Connection [8086:1539] (rev 03)
> > > > IOMMU Group 43:
> > > >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > > > Serial ATA Controller [1b21:0612] (rev 02)
> > > > IOMMU Group 44:
> > > >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > > > USB 3.0 Host Controller [1b21:1142]
> > > >
> > > > Ethernet, SATA and USB controller in its own group.
> > > >
> > > > And with 5.9, everything is in one group:
> > > > IOMMU Group 29:
> > > >         00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > > chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
> > > >         00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > > chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
> > > >         00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > > chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
> > > >         00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series
> > > > chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
> > > >         07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > > [1b21:118f]
> > > >         08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > > [1b21:118f]
> > > >         08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > > [1b21:118f]
> > > >         08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device
> > > > [1b21:118f]
> > > >         09:00.0 Ethernet controller [0200]: Intel Corporation I211
> > > > Gigabit Network Connection [8086:1539] (rev 03)
> > > >         0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
> > > > Serial ATA Controller [1b21:0612] (rev 02)
> > > >         0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
> > > > USB 3.0 Host Controller [1b21:1142]
> > > >
> > > >
> > > > This seems to be caused by commit
> > > > 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
> > > > commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
> > > > Author: Rajat Jain <rajatja@google.com>
> > > > Date:   Tue Jul 7 15:46:02 2020 -0700
> > > >
> > > >     PCI: Cache ACS capability offset in device
> > > >
> > > >     Currently the ACS capability is being looked up at a number of
> > > > places. Read
> > > >     and store it once at enumeration so that it can be used by all
> > > > later.  No
> > > >     functional change intended.
> > > >
> > > >     Link:
> > > > https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
> > > >     Signed-off-by: Rajat Jain <rajatja@google.com>
> > > >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> > > >
> > > >  drivers/pci/p2pdma.c |  2 +-
> > > >  drivers/pci/pci.c    | 20 ++++++++++++++++----
> > > >  drivers/pci/pci.h    |  2 +-
> > > >  drivers/pci/probe.c  |  2 +-
> > > >  drivers/pci/quirks.c |  8 ++++----
> > > >  include/linux/pci.h  |  1 +
> > > >  6 files changed, 24 insertions(+), 11 deletions(-)
> > > >
> > > >
> > > > If I revert this commit, I get back old groups.
> > > >
> > > > In commit log there is message 'No functional change intended'. But
> > > > there is functional change.
> > > >
> > > > This is Intel Core i7-5930K CPU and X99 chipset. But I see the same
> > > > thing on other Intel systems (didn't test on AMD).
> > >
> > > Some more info.
> > > Problem seems to be that pci_dev_specific_enable_acs() is not called
> > > anymore.
> > > Before, pci_enable_acs() was called from pci_init_capabilities() and in
> > > pci_enable_acs(), pci_dev_specific_enable_acs() was called.
> > > I don't know anything about PCI and this stuff, but I'm guessing that this
> > > function enable ACS for some Intel devices.
> > > But after this commit, pci_acs_init() is called from pci_init_capabilities()
> > > and if pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) returns 0,
> > > pci_enable_acs() and pci_dev_specific_enable_acs() is not called anymore.
> > > If I apply for ex. this patch bellow, groups are right again and everything
> > > works as before.
> >
> > Thanks very much for the report and the debugging.  Maybe we can get
> > this sorted and fixed for v5.10-rc2 or -rc3.

Thank Boris for reporting and debugging! The problem was because I
overlooked the fact that some rootports (the ones quirked with
*_intel_pch_acs_* functions in this case) may not expose a standard
ACS capability structure, but rather depend on quirks to enable ACS
for them using non standard registers. Your platform is in this
category. Can you please send lspci -vvvv and lspci -xxxx for one of
your rootports to confirm?

> >
> > > diff -ur linux-5.9.1.orig/drivers/pci/pci.c linux-5.9.1/drivers/pci/pci.c
> > > --- linux-5.9.1.orig/drivers/pci/pci.c  2020-10-17 08:31:22.000000000 +0200
> > > +++ linux-5.9.1/drivers/pci/pci.c       2020-10-27 19:01:32.650010803 +0100
> > > @@ -3502,9 +3502,7 @@
> > >  void pci_acs_init(struct pci_dev *dev)
> > >  {
> > >         dev->acs_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
> > > -
> > > -       if (dev->acs_cap)
> > > -               pci_enable_acs(dev);
> > > +       pci_enable_acs(dev);
> > >  }
> > >
> > >  /**
> > >

Ack, yes, this is what needs to be done, and I just sent a patch at
https://lkml.org/lkml/2020/10/28/693.

Thanks,

Rajat

> > >

Re: Kernel 5.9 IOMMU groups regression/change

[Boris V.]

On 29/10/2020 00:37, Rajat Jain wrote:
> On Wed, Oct 28, 2020 at 3:07 PM Rajat Jain <rajatja@google.com> wrote:
>> On Wed, Oct 28, 2020 at 2:52 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> [+cc Rajat, LKML]
>>>
>> Thanks for copying me. (I don't look at mailing lists actively - so
>> missed this). Taking a look at this now.
>>
>> Thanks,
>>
>> Rajat
>>
>>
>>> On Tue, Oct 27, 2020 at 08:31:09PM +0100, Boris V. wrote:
>>>> On 25/10/2020 20:45, Boris V. wrote:
>>>>> With upgrade to kernel 5.9 my VMs stopped working, because some devices
>>>>> can't be passed through.
>>>>> This is caused by different IOMMU groups and devices being in the same
>>>>> group.
>>>>>
>>>>> For ex. with kernel 5.8 this are IOMMU groups:
>>>>> IOMMU Group 40:
>>>>>          08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
>>>>> [1b21:118f]
>>>>>          09:00.0 Ethernet controller [0200]: Intel Corporation I211
>>>>> Gigabit Network Connection [8086:1539] (rev 03)
>>>>> IOMMU Group 43:
>>>>>          0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
>>>>> Serial ATA Controller [1b21:0612] (rev 02)
>>>>> IOMMU Group 44:
>>>>>          0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
>>>>> USB 3.0 Host Controller [1b21:1142]
>>>>>
>>>>> Ethernet, SATA and USB controller in its own group.
>>>>>
>>>>> And with 5.9, everything is in one group:
>>>>> IOMMU Group 29:
>>>>>          00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series
>>>>> chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
>>>>>          00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series
>>>>> chipset PCI Express Root Port #4 [8086:8d16] (rev d5)
>>>>>          00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series
>>>>> chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
>>>>>          00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series
>>>>> chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
>>>>>          07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device
>>>>> [1b21:118f]
>>>>>          08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device
>>>>> [1b21:118f]
>>>>>          08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device
>>>>> [1b21:118f]
>>>>>          08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device
>>>>> [1b21:118f]
>>>>>          09:00.0 Ethernet controller [0200]: Intel Corporation I211
>>>>> Gigabit Network Connection [8086:1539] (rev 03)
>>>>>          0c:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062
>>>>> Serial ATA Controller [1b21:0612] (rev 02)
>>>>>          0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A
>>>>> USB 3.0 Host Controller [1b21:1142]
>>>>>
>>>>>
>>>>> This seems to be caused by commit
>>>>> 52fbf5bdeeef415b28b8e6cdade1e48927927f60.
>>>>> commit 52fbf5bdeeef415b28b8e6cdade1e48927927f60
>>>>> Author: Rajat Jain <rajatja@google.com>
>>>>> Date:   Tue Jul 7 15:46:02 2020 -0700
>>>>>
>>>>>      PCI: Cache ACS capability offset in device
>>>>>
>>>>>      Currently the ACS capability is being looked up at a number of
>>>>> places. Read
>>>>>      and store it once at enumeration so that it can be used by all
>>>>> later.  No
>>>>>      functional change intended.
>>>>>
>>>>>      Link:
>>>>> https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.com
>>>>>      Signed-off-by: Rajat Jain <rajatja@google.com>
>>>>>      Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>>>>>
>>>>>   drivers/pci/p2pdma.c |  2 +-
>>>>>   drivers/pci/pci.c    | 20 ++++++++++++++++----
>>>>>   drivers/pci/pci.h    |  2 +-
>>>>>   drivers/pci/probe.c  |  2 +-
>>>>>   drivers/pci/quirks.c |  8 ++++----
>>>>>   include/linux/pci.h  |  1 +
>>>>>   6 files changed, 24 insertions(+), 11 deletions(-)
>>>>>
>>>>>
>>>>> If I revert this commit, I get back old groups.
>>>>>
>>>>> In commit log there is message 'No functional change intended'. But
>>>>> there is functional change.
>>>>>
>>>>> This is Intel Core i7-5930K CPU and X99 chipset. But I see the same
>>>>> thing on other Intel systems (didn't test on AMD).
>>>> Some more info.
>>>> Problem seems to be that pci_dev_specific_enable_acs() is not called
>>>> anymore.
>>>> Before, pci_enable_acs() was called from pci_init_capabilities() and in
>>>> pci_enable_acs(), pci_dev_specific_enable_acs() was called.
>>>> I don't know anything about PCI and this stuff, but I'm guessing that this
>>>> function enable ACS for some Intel devices.
>>>> But after this commit, pci_acs_init() is called from pci_init_capabilities()
>>>> and if pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) returns 0,
>>>> pci_enable_acs() and pci_dev_specific_enable_acs() is not called anymore.
>>>> If I apply for ex. this patch bellow, groups are right again and everything
>>>> works as before.
>>> Thanks very much for the report and the debugging.  Maybe we can get
>>> this sorted and fixed for v5.10-rc2 or -rc3.
> Thank Boris for reporting and debugging! The problem was because I
> overlooked the fact that some rootports (the ones quirked with
> *_intel_pch_acs_* functions in this case) may not expose a standard
> ACS capability structure, but rather depend on quirks to enable ACS
> for them using non standard registers. Your platform is in this
> category. Can you please send lspci -vvvv and lspci -xxxx for one of
> your rootports to confirm?

Below is lspci output for some root ports:

lspci -vvvv -s 00:01.0
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI 
Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 64 bytes
         Interrupt: pin A routed to IRQ 25
         NUMA node: 0
         IOMMU group: 13
         Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
         I/O behind bridge: 0000f000-00000fff [disabled]
         Memory behind bridge: fff00000-000fffff [disabled]
         Prefetchable memory behind bridge: 
00000000fff00000-00000000000fffff [disabled]
         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- <SERR- <PERR-
         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- 
FastB2B-
                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
         Capabilities: [40] Subsystem: Intel Corporation Xeon E7 v3/Xeon 
E5 v3/Core i7 PCI Express Root Port 1
         Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
                 Address: fee00238  Data: 0000
                 Masking: 00000002  Pending: 00000000
         Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
                 DevCap: MaxPayload 256 bytes, PhantFunc 0
                         ExtTag- RBE+
                 DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                         MaxPayload 128 bytes, MaxReadReq 128 bytes
                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- 
AuxPwr- TransPend-
                 LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit 
Latency L1 <16us
                         ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 2.5GT/s (downgraded), Width x0 (downgraded)
                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- 
HotPlug- Surprise-
                         Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- 
HPIrq- LinkChg-
                         Control: AttnInd Off, PwrInd Off, Power- Interlock-
                 SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- 
PresDet- Interlock-
                         Changed: MRL- PresDet+ LinkState-
                 RootCap: CRSVisible+
                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- 
PMEIntEna+ CRSVisible+
                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                 DevCap2: Completion Timeout: Range BCD, TimeoutDis+ 
NROPrPrP- LTR-
                          10BitTagComp- 10BitTagReq- OBFF Not Supported, 
ExtFmt- EETLPPrefix-
                          EmergencyPowerReduction Not Supported, 
EmergencyPowerReductionInit-
                          FRS- LN System CLS Not Supported, TPHComp+ 
ExtTPHComp- ARIFwd+
                          AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+
                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- 
LTR- OBFF Disabled, ARIFwd-
                          AtomicOpsCtl: ReqEn- EgressBlck-
                 LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- 
Retimer- 2Retimers- DRS-
                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- 
SpeedDis-
                          Transmit Margin: Normal Operating Range, 
EnterModifiedCompliance- ComplianceSOS-
                          Compliance De-emphasis: -6dB
                 LnkSta2: Current De-emphasis Level: -6dB, 
EqualizationComplete- EqualizationPhase1-
                          EqualizationPhase2- EqualizationPhase3- 
LinkEqualizationRequest-
                          Retimer- 2Retimers- CrosslinkRes: unsupported
         Capabilities: [e0] Power Management version 3
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                 Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
         Capabilities: [100 v1] Vendor Specific Information: ID=0002 
Rev=0 Len=00c <?>
         Capabilities: [110 v1] Access Control Services
                 ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ 
UpstreamFwd+ EgressCtrl- DirectTrans-
                 ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ 
UpstreamFwd+ EgressCtrl- DirectTrans-
         Capabilities: [148 v1] Advanced Error Reporting
                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr-
                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr+
                 AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- 
ECRCChkCap- ECRCChkEn-
                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                 HeaderLog: 00000000 00000000 00000000 00000000
                 RootCmd: CERptEn+ NFERptEn+ FERptEn+
                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
         Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 
Rev=1 Len=00a <?>
         Capabilities: [250 v1] Secondary PCI Express
                 LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                 LaneErrStat: 0
         Capabilities: [280 v1] Vendor Specific Information: ID=0005 
Rev=3 Len=018 <?>
         Capabilities: [300 v1] Vendor Specific Information: ID=0008 
Rev=0 Len=038 <?>
         Kernel driver in use: pcieport

lspci -xxxx -s 00:01.0
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI 
Express Root Port 1 (rev 02)
00: 86 80 02 2f 07 04 10 00 02 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 04 04 00 f0 00 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 12 00
40: 0d 60 00 00 86 80 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 05 90 03 01 38 02 e0 fe 00 00 00 00 02 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 10 e0 42 01 01 80 00 00 0f 00 00 00 43 38 7a 01
a0: 40 00 01 10 00 00 10 00 c0 03 08 00 18 00 01 00
b0: 00 00 00 00 be 13 00 00 00 00 00 00 0e 00 00 00
c0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 01 00 03 c8 0b 01 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 0b 00 01 11 02 00 c0 00 07 31 00 00 00 00 00 00
110: 0d 00 81 14 1f 00 1d 00 00 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 01 00 01 1d 00 00 00 00
150: 00 00 00 00 30 20 06 00 00 00 00 00 00 20 00 00
160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
170: 00 00 00 00 07 00 00 00 00 00 00 00 00 00 00 00
180: 91 30 18 00 00 00 00 00 40 10 10 e4 20 00 00 00
190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1d0: 0b 00 01 25 03 00 a1 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
210: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
250: 19 00 01 28 00 00 00 00 00 00 00 00 77 27 77 27
260: 77 27 77 27 77 27 77 27 77 27 77 27 00 00 00 00
270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
280: 0b 00 01 30 05 00 83 01 00 00 00 40 00 00 00 00
290: 00 00 00 00 00 00 00 00 0b 00 01 30 07 00 40 02
2a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
300: 0b 00 01 00 08 00 80 03 00 00 00 00 0f 00 00 00
310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

lspci -vvvv -s 00:01.1
00:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI 
Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 64 bytes
         Interrupt: pin A routed to IRQ 26
         NUMA node: 0
         IOMMU group: 14
         Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
         I/O behind bridge: 0000f000-00000fff [disabled]
         Memory behind bridge: fb600000-fb6fffff [size=1M]
         Prefetchable memory behind bridge: 
00000000fff00000-00000000000fffff [disabled]
         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- <SERR- <PERR-
         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- 
FastB2B-
                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
         Capabilities: [40] Subsystem: Intel Corporation Xeon E7 v3/Xeon 
E5 v3/Core i7 PCI Express Root Port 1
         Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
                 Address: fee00258  Data: 0000
                 Masking: 00000002  Pending: 00000000
         Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
                 DevCap: MaxPayload 256 bytes, PhantFunc 0
                         ExtTag- RBE+
                 DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                         MaxPayload 256 bytes, MaxReadReq 128 bytes
                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- 
AuxPwr- TransPend-
                 LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit 
Latency L1 <16us
                         ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
                 LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                         TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt+
                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- 
HotPlug- Surprise-
                         Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- 
HPIrq- LinkChg-
                         Control: AttnInd Off, PwrInd Off, Power- Interlock-
                 SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- 
PresDet+ Interlock-
                         Changed: MRL- PresDet+ LinkState+
                 RootCap: CRSVisible+
                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- 
PMEIntEna+ CRSVisible+
                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                 DevCap2: Completion Timeout: Range BCD, TimeoutDis+ 
NROPrPrP- LTR-
                          10BitTagComp- 10BitTagReq- OBFF Not Supported, 
ExtFmt- EETLPPrefix-
                          EmergencyPowerReduction Not Supported, 
EmergencyPowerReductionInit-
                          FRS- LN System CLS Not Supported, TPHComp+ 
ExtTPHComp- ARIFwd+
                          AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+
                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- 
LTR- OBFF Disabled, ARIFwd-
                          AtomicOpsCtl: ReqEn- EgressBlck-
                 LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- 
Retimer- 2Retimers- DRS-
                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- 
SpeedDis-
                          Transmit Margin: Normal Operating Range, 
EnterModifiedCompliance- ComplianceSOS-
                          Compliance De-emphasis: -6dB
                 LnkSta2: Current De-emphasis Level: -6dB, 
EqualizationComplete+ EqualizationPhase1+
                          EqualizationPhase2+ EqualizationPhase3+ 
LinkEqualizationRequest-
                          Retimer- 2Retimers- CrosslinkRes: unsupported
         Capabilities: [e0] Power Management version 3
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
         Capabilities: [100 v1] Vendor Specific Information: ID=0002 
Rev=0 Len=00c <?>
         Capabilities: [110 v1] Access Control Services
                 ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ 
UpstreamFwd+ EgressCtrl- DirectTrans-
                 ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ 
UpstreamFwd+ EgressCtrl- DirectTrans-
         Capabilities: [148 v1] Advanced Error Reporting
                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr-
                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr+
                 AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- 
ECRCChkCap- ECRCChkEn-
                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                 HeaderLog: 00000000 00000000 00000000 00000000
                 RootCmd: CERptEn+ NFERptEn+ FERptEn+
                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
         Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 
Rev=1 Len=00a <?>
         Capabilities: [250 v1] Secondary PCI Express
                 LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                 LaneErrStat: 0
         Capabilities: [280 v1] Vendor Specific Information: ID=0005 
Rev=3 Len=018 <?>
         Capabilities: [300 v1] Vendor Specific Information: ID=0008 
Rev=0 Len=038 <?>
         Kernel driver in use: pcieport

lspci -xxxx -s 00:01.1
00:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI 
Express Root Port 1 (rev 02)
00: 86 80 03 2f 07 04 10 00 02 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 05 05 00 f0 00 00 00
20: 60 fb 60 fb f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 12 00
40: 0d 60 00 00 86 80 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 05 90 03 01 58 02 e0 fe 00 00 00 00 02 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 10 e0 42 01 01 80 00 00 2f 00 00 00 43 38 7a 02
a0: 42 00 43 b0 00 00 10 00 c0 03 48 01 18 00 01 00
b0: 00 00 00 00 be 13 00 00 00 00 00 00 0e 00 00 00
c0: 03 00 1e 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 01 00 03 c8 08 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 0b 00 01 11 02 00 c0 00 07 31 00 00 00 00 00 00
110: 0d 00 81 14 1f 00 1d 00 00 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 01 00 01 1d 00 00 00 00
150: 00 00 00 00 30 20 06 00 00 00 00 00 00 20 00 00
160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
170: 00 00 00 00 07 00 00 00 00 00 00 00 00 00 00 00
180: 91 30 18 00 00 00 00 00 40 10 10 e4 20 00 00 00
190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1d0: 0b 00 01 25 03 00 a1 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
210: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
250: 19 00 01 28 00 00 00 00 00 00 00 00 77 27 77 27
260: 77 27 77 27 00 00 00 00 00 00 00 00 00 00 00 00
270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
280: 0b 00 01 30 05 00 83 01 00 00 00 40 00 00 00 00
290: 00 00 00 00 00 00 00 00 0b 00 01 30 07 00 40 02
2a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
300: 0b 00 01 00 08 00 80 03 00 00 00 00 0f 00 00 00
310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


lspci -vvvv -s 00:1c.4
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI 
Express Root Port #5 (rev d5) (prog-if 00 [Normal decode])
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 64 bytes
         Interrupt: pin A routed to IRQ 34
         NUMA node: 0
         IOMMU group: 31
         Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0
         I/O behind bridge: 0000b000-0000bfff [size=4K]
         Memory behind bridge: fb400000-fb4fffff [size=1M]
         Prefetchable memory behind bridge: 
00000000fff00000-00000000000fffff [disabled]
         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- <SERR- <PERR-
         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- 
FastB2B-
                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
         Capabilities: [40] Express (v2) Root Port (Slot-), MSI 00
                 DevCap: MaxPayload 128 bytes, PhantFunc 0
                         ExtTag- RBE+
                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                         MaxPayload 128 bytes, MaxReadReq 128 bytes
                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- 
AuxPwr+ TransPend-
                 LnkCap: Port #5, Speed 5GT/s, Width x2, ASPM L0s L1, 
Exit Latency L0s <512ns, L1 <16us
                         ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
                         TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                 RootCap: CRSVisible-
                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- 
PMEIntEna+ CRSVisible-
                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                 DevCap2: Completion Timeout: Range ABC, TimeoutDis+ 
NROPrPrP- LTR+
                          10BitTagComp- 10BitTagReq- OBFF Not Supported, 
ExtFmt- EETLPPrefix-
                          EmergencyPowerReduction Not Supported, 
EmergencyPowerReductionInit-
                          FRS- LN System CLS Not Supported, TPHComp- 
ExtTPHComp- ARIFwd-
                          AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- 
LTR- OBFF Disabled, ARIFwd-
                          AtomicOpsCtl: ReqEn- EgressBlck-
                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- 
SpeedDis-
                          Transmit Margin: Normal Operating Range, 
EnterModifiedCompliance- ComplianceSOS-
                          Compliance De-emphasis: -6dB
                 LnkSta2: Current De-emphasis Level: -6dB, 
EqualizationComplete- EqualizationPhase1-
                          EqualizationPhase2- EqualizationPhase3- 
LinkEqualizationRequest-
                          Retimer- 2Retimers- CrosslinkRes: unsupported
         Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                 Address: fee00398  Data: 0000
         Capabilities: [90] Subsystem: ASUSTeK Computer Inc. C610/X99 
series chipset PCI Express Root Port
         Capabilities: [a0] Power Management version 3
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
         Kernel driver in use: pcieport

lspci -xxxx -s 00:1c.4
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI 
Express Root Port #5 (rev d5)
00: 86 80 18 8d 07 04 10 00 d5 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 0c 0c 00 b0 b0 00 00
20: 40 fb 40 fb f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 12 00
40: 10 80 42 00 00 80 00 00 00 00 10 00 22 3c 32 05
50: 40 00 12 70 00 00 04 00 00 00 40 01 08 00 00 00
60: 00 00 00 00 17 08 00 00 00 00 00 00 00 00 00 00
70: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 05 90 01 00 98 03 e0 fe 00 00 00 00 00 00 00 00
90: 0d a0 00 00 43 10 00 86 00 00 00 00 00 00 00 00
a0: 01 00 03 c8 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 01 42 18 00 00 08 80 91 89 00 00 00 00
e0: 00 3f 00 00 00 00 00 00 01 00 00 00 00 00 00 00
f0: 50 00 00 00 40 00 00 00 b1 0f 06 08 00 68 00 05
100: 00 00 00 00 00 00 00 00 00 00 00 00 11 00 06 00
110: 00 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
310: 00 00 00 00 1b 36 3a 74 00 00 14 14 31 17 42 00
320: 5b 60 09 00 20 20 5a 0a fc 1f 38 33 fb 0f f1 0a
330: 16 00 00 28 bc b5 bc 4a 00 00 00 00 74 4c 85 00
340: 33 0a 33 00 37 0a 37 00 e7 0b 7c 00 31 0e c0 00
350: 33 0e c2 00 01 00 dd 00 41 04 09 00 00 00 00 00
360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 00 00 00 00 00 00 00 00 22 14 01 04 00 00 00 00
410: 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

>>>> diff -ur linux-5.9.1.orig/drivers/pci/pci.c linux-5.9.1/drivers/pci/pci.c
>>>> --- linux-5.9.1.orig/drivers/pci/pci.c  2020-10-17 08:31:22.000000000 +0200
>>>> +++ linux-5.9.1/drivers/pci/pci.c       2020-10-27 19:01:32.650010803 +0100
>>>> @@ -3502,9 +3502,7 @@
>>>>   void pci_acs_init(struct pci_dev *dev)
>>>>   {
>>>>          dev->acs_cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS);
>>>> -
>>>> -       if (dev->acs_cap)
>>>> -               pci_enable_acs(dev);
>>>> +       pci_enable_acs(dev);
>>>>   }
>>>>
>>>>   /**
>>>>
> Ack, yes, this is what needs to be done, and I just sent a patch at
> https://lkml.org/lkml/2020/10/28/693.
>
> Thanks,
>
> Rajat
>