Hi Jim,

Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.

Regards,
Oza.

On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Oza,
>
>
>
> I am confused – why do the NVMe devices show up in your lspci output if
> they are not attached?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 8:20 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci -vvvx
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Interrupt: pin A routed to IRQ 378
>
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
>         I/O behind bridge: 00000000-00000fff
>
>         Memory behind bridge: 00000000-000fffff
>
>         Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
>         BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0
>
>                         ExtTag+ RBE+
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
>                 RootCap: CRSVisible+
>
>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
>                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
>                          Compliance De-emphasis: -6dB
>
>                 LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
>                          EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [240 v1] L1 PM Substates
>
>                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
>                           PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
>         Capabilities: [300 v1] #19
>
> lspci: Unable to load libkmod resources: error -12
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
>         Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Interrupt: pin A routed to IRQ 378
>
>         Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
>
>         [virtual] Expansion ROM at 480000000 [disabled] [size=64K]
>
>         Capabilities: [40] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
>         Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
>                 Vector table: BAR=0 offset=00002000
>
>                 PBA: BAR=0 offset=00003000
>
>         Capabilities: [60] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
>                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
>                 LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
>                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
>                          Compliance De-emphasis: -6dB
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
>                          EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [150 v1] Virtual Channel
>
>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>
>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>
>                 Ctrl:   ArbSelect=Fixed
>
>                 Status: InProgress-
>
>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
>                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
>                         Status: NegoPending- InProgress-
>
>         Capabilities: [180 v1] Power Budgeting <?>
>
>         Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 0
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
>         Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
>
>         Capabilities: [2a0 v1] #19
>
>         Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Interrupt: pin A routed to IRQ 380
>
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
>         I/O behind bridge: 00000000-00000fff
>
>         Memory behind bridge: 00000000-000fffff
>
>         Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
>         BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0
>
>                         ExtTag+ RBE+
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
>                 RootCap: CRSVisible+
>
>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
>                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
>                          Compliance De-emphasis: -6dB
>
>                 LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
>                          EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [240 v1] L1 PM Substates
>
>                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
>                           PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
>         Capabilities: [300 v1] #19
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
>         Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Latency: 0
>
>         Interrupt: pin A routed to IRQ 380
>
>         Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
>
>         [virtual] Expansion ROM at 780000000 [disabled] [size=64K]
>
>         Capabilities: [40] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
>         Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
>                 Vector table: BAR=0 offset=00002000
>
>                 PBA: BAR=0 offset=00003000
>
>         Capabilities: [60] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
>                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
>                 LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
>                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
>                          Compliance De-emphasis: -6dB
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
>                          EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [150 v1] Virtual Channel
>
>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>
>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>
>                 Ctrl:   ArbSelect=Fixed
>
>                 Status: InProgress-
>
>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
>                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
>                         Status: NegoPending- InProgress-
>
>         Capabilities: [180 v1] Power Budgeting <?>
>
>         Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 0
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
>         Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
>
>         Capabilities: [2a0 v1] #19
>
>         Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
> (prog-if 00 [Normal decode])
>
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
>         I/O behind bridge: 00000000-00000fff
>
>         Memory behind bridge: fff00000-000fffff
>
>         Prefetchable memory behind bridge: 0000000010000000-
> 00000000104fffff
>
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
>         BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [58] #00 [0000]
>
> 00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>         Subsystem: Broadcom Corporation Device 16f0
>
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
>         Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
>         Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
>         Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>         Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
>                 Vector table: BAR=4 offset=00000000
>
>                 PBA: BAR=4 offset=00001000
>
>         Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
> SpeedDis-
>
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
>                          Compliance De-emphasis: -6dB
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
>                          EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
>
>         Capabilities: [150 v1] Power Budgeting <?>
>
>         Capabilities: [160 v1] Virtual Channel
>
>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>
>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>
>                 Ctrl:   ArbSelect=Fixed
>
>                 Status: InProgress-
>
>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
>                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
>                         Status: NegoPending- InProgress-
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [1b0 v1] Latency Tolerance Reporting
>
>                 Max snoop latency: 0ns
>
>                 Max no snoop latency: 0ns
>
>         Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 1
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
>
> 20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>         Subsystem: Broadcom Corporation Device 16f0
>
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
>         Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
>         Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
>         Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>         Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
>                 Vector table: BAR=4 offset=00000000
>
>                 PBA: BAR=4 offset=00001000
>
>         Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
>                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
>                          EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
>
>         Capabilities: [150 v1] Power Budgeting <?>
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 2
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
>
> 20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>         Subsystem: Broadcom Corporation Device 16f0
>
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
>         Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
>         Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
>         Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>         Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
>                 Vector table: BAR=4 offset=00000000
>
>                 PBA: BAR=4 offset=00001000
>
>         Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
>                          EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
>
>         Capabilities: [150 v1] Power Budgeting <?>
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 3
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
>
> 20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>         Subsystem: Broadcom Corporation Device 16f0
>
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
>         Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
>         Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
>         Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
>         Capabilities: [48] Power Management version 3
>
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
>         Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
>                 Address: 0000000000000000  Data: 0000
>
>         Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
>                 Vector table: BAR=4 offset=00000000
>
>                 PBA: BAR=4 offset=00001000
>
>         Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
>                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
>                          EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
>         Capabilities: [100 v1] Advanced Error Reporting
>
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
>         Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
>
>         Capabilities: [150 v1] Power Budgeting <?>
>
>         Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
>         Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
>                 ARICap: MFVC- ACS-, Next Function: 4
>
>                 ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
>
> 20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> *From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
> *Sent:* Wednesday, August 30, 2017 8:41 PM
> *To:* 'Storage Performance Development Kit'
> *Cc:* 'oza.pawandeep(a)gmail.com'
> *Subject:* RE: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01)
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>
>
> root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
>
> 0001:01:00.0 (8086 0953): nvme -> vfio-pci
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 1520.267436] pci 0008:00:00.0:   bridge window [mem
> 0x10000000-0x104fffff 64bit pref]
>
> [ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
> header type 01)                 >> it is looking to unbind on empty slot as
> well.
>
> [ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
> -- SError
>
> [ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 1535.924850] Modules linked in:
>
> [ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
> 4.12.0-01624-gbbd4086-dirty #97
>
> [ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
>
> [ 1535.949634] PC is at 0xffff8baadca0
>
> [ 1535.953230] LR is at 0x40aca8
>
> [ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
> 80000000
>
> [ 1535.963919] sp : 0000ffffca61a970
>
> [ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
>
> [ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
>
> [ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
>
> [ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
>
> [ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
>
> [ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
>
> [ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
>
> [ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
>
> [ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
>
> [ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
>
> [ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
>
> [ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
>
> [ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
>
> [ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
>
> [ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
>
> [ 1536.049529] Process lighttpd (pid: 2108, stack limit =
> 0xffff80a162790000)
>
>
>
> Please find attached status script.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Harris, James R
> *Sent:* Tuesday, August 29, 2017 11:30 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> So to clarify, you have your system booted with no NVMe endpoint
> connected, and then when you run the SPDK setup.sh script, you see all of
> these kernel messages from trying to bind vfio to PCIe devices and system
> eventually crashes?
>
>
>
> If so, we need to determine what PCIe devices setup.sh is trying to bind
> to vfio.  It should only be trying to bind NVMe devices but if there is no
> NVMe device connected then it shouldn’t be trying to bind anything.
>
>
>
> Can you send lspci –vvvx output from your system before running setup.sh?
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 9:45 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> In my opinion, this has nothing to do with platform. Though our platform
> is ARMv8.
>
> (but, I can not test on any other, because we don’t know how the kernel
> driver is written)
>
>
>
> If kernel driver supports hotplug, which means they are allowing
> pci_create_root_bus irrespective of whether EP is plugged or not.
>
> In other words. Following APIs are never called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
> and in that case, if PCIe slots is empty, running SPDK resulting in
> stalls. (10-15 seconds) followed by crash.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
> James R
> *Sent:* Tuesday, August 29, 2017 6:20 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Do you see this issue only on your armv8 platform or do you also see it on
> amd64?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 1:51 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Sorry If I was unclear.
>
> I am not talking about hotplug feature of SPDK.
>
>
>
>
>
> Ø  PCI hotplug feature is implemented in kernel driver and working fine.
>
> Ø  But the moment I run SPDK and try to bind vfio driver it stalls
> completely.
>
> The reason is: kernel driver will not remove the root bus (when PCIe
> endpoint is not connected, during boot-time)
>
> So SPDK tries to bind driver thinking host bridge is there.
>
> Without PCI hotplug host bridge will not be there because of following API
> call in kernel driver.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
> Ø  since now we allow host bridge creation  (API: pci_create_root_bus)
> irrespective of EP is connected or not.
>
> And then if I run SPDK (with no Endpoint connected/Empty slot) I get
> stalls.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang,
> Cunyin
> *Sent:* Tuesday, August 29, 2017 2:14 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Could you please provide some details steps to reproduce the issue?
>
> SPDK take in charge  for hotplug only after you bind the device to uio or
> vfio driver,
>
> so for the new insert deivce, it will handled by kernel driver first.
>
>
>
> -Cunyin
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Oza Oza
> *Sent:* Tuesday, August 29, 2017 4:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] PCI hotplug and SPDK
>
>
>
> Hi All,
>
>
>
> PCI hotplug support; requires creation of root bus and probe to go ahead
> with all PCIe configuration.
>
>
>
> Which means following APIs ae not called.
>
>    pci_stop_root_bus(bus);
>
>    pci_remove_root_bus(bus);
>
>
>
>
>
> And then If I run SPDK, It makes system crash with following info.
>
>
>
> Note: if the disk is connected then SPDK is fine.
>
>
>
> Otherwise it stalls the system with following crash.
>
>
>
> root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
> /usr/share/spdk/scripts/setup.sh
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [   34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [   50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
>
> [   50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [   65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
>
> [   65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [   81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
>
> [   81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [   97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
>
> [   97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
> -- SError
>
> [   97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [   97.180364] Modules linked in:
>
> [   97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
> 4.12.0-01560-gc83093d-dirty #89
>
> [   97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [   97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
>
> [   97.204790] PC is at 0xffff7cbdfba8
>
> [   97.208387] LR is at 0xffff7cb8f288
>
> [   97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
> 20000000
>
> [   97.219612] sp : 0000fffffe564040
>
> [   97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
>
> [   97.228509] x27: 0000000000000000 x26: 00000000004e2000
>
> [   97.233989] x25: 00000000004e5000 x24: 0000000000000002
>
> [   97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
>
> [   97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
>
> [   97.250427] x19: 0000000000000002 x18: 0000000000000000
>
> [   97.255906] x17: 00000000004daac8 x16: 0000000000000000
>
> [   97.261386] x15: 0000000000000096 x14: 0000000000000000
>
> [   97.266865] x13: 0000000000000000 x12: 0000000000000000
>
> [   97.272344] x11: 0000000000000020 x10: 0101010101010101
>
> [   97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
>
> [   97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
>
> [   97.288782] x5 : 0000000000015551 x4 : 0000000000000888
>
> [   97.294261] x3 : 0000000000000000 x2 : 0000000000000002
>
> [   97.299741] x1 : 000000001054db10 x0 : 0000000000000002
>
> [   97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
>
> [   97.311960] ---[ end trace a1f48abe30820241 ]---
>
>
>
> Regards,
>
> Oza.
>