From: Oza Pawandeep <oza.pawandeep at gmail.com>
To: spdk@lists.01.org
Subject: Re: [SPDK] PCI hotplug and SPDK
Date: Wed, 30 Aug 2017 23:47:49 +0530 [thread overview]
Message-ID: <CAK1A=4x1Hw3zU+G4tijmQH08Yi9KNgYWHCPjU7Vj3_0h3eu9gg@mail.gmail.com> (raw)
In-Reply-To: F7A8E5F1-CD87-4C6A-AFA9-A84B74CEAB64@intel.com
[-- Attachment #1: Type: text/plain, Size: 48252 bytes --]
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
> Hi Oza,
>
>
>
> I am confused – why do the NVMe devices show up in your lspci output if
> they are not attached?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 8:20 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci -vvvx
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> lspci: Unable to load libkmod resources: error -12
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 480000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 780000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
> (prog-if 00 [Normal decode])
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: fff00000-000fffff
>
> Prefetchable memory behind bridge: 0000000010000000-
> 00000000104fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] #00 [0000]
>
> 00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [160 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b0 v1] Latency Tolerance Reporting
>
> Max snoop latency: 0ns
>
> Max no snoop latency: 0ns
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 1
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
>
> 20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 2
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
>
> 20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 3
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
>
> 20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 4
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
>
> 20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> *From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
> *Sent:* Wednesday, August 30, 2017 8:41 PM
> *To:* 'Storage Performance Development Kit'
> *Cc:* 'oza.pawandeep(a)gmail.com'
> *Subject:* RE: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01)
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>
>
> root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
>
> 0001:01:00.0 (8086 0953): nvme -> vfio-pci
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 1520.267436] pci 0008:00:00.0: bridge window [mem
> 0x10000000-0x104fffff 64bit pref]
>
> [ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
> header type 01) >> it is looking to unbind on empty slot as
> well.
>
> [ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
> -- SError
>
> [ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 1535.924850] Modules linked in:
>
> [ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
> 4.12.0-01624-gbbd4086-dirty #97
>
> [ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
>
> [ 1535.949634] PC is at 0xffff8baadca0
>
> [ 1535.953230] LR is at 0x40aca8
>
> [ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
> 80000000
>
> [ 1535.963919] sp : 0000ffffca61a970
>
> [ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
>
> [ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
>
> [ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
>
> [ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
>
> [ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
>
> [ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
>
> [ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
>
> [ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
>
> [ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
>
> [ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
>
> [ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
>
> [ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
>
> [ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
>
> [ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
>
> [ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
>
> [ 1536.049529] Process lighttpd (pid: 2108, stack limit =
> 0xffff80a162790000)
>
>
>
> Please find attached status script.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Harris, James R
> *Sent:* Tuesday, August 29, 2017 11:30 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> So to clarify, you have your system booted with no NVMe endpoint
> connected, and then when you run the SPDK setup.sh script, you see all of
> these kernel messages from trying to bind vfio to PCIe devices and system
> eventually crashes?
>
>
>
> If so, we need to determine what PCIe devices setup.sh is trying to bind
> to vfio. It should only be trying to bind NVMe devices but if there is no
> NVMe device connected then it shouldn’t be trying to bind anything.
>
>
>
> Can you send lspci –vvvx output from your system before running setup.sh?
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 9:45 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> In my opinion, this has nothing to do with platform. Though our platform
> is ARMv8.
>
> (but, I can not test on any other, because we don’t know how the kernel
> driver is written)
>
>
>
> If kernel driver supports hotplug, which means they are allowing
> pci_create_root_bus irrespective of whether EP is plugged or not.
>
> In other words. Following APIs are never called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
> and in that case, if PCIe slots is empty, running SPDK resulting in
> stalls. (10-15 seconds) followed by crash.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
> James R
> *Sent:* Tuesday, August 29, 2017 6:20 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Do you see this issue only on your armv8 platform or do you also see it on
> amd64?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 1:51 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Sorry If I was unclear.
>
> I am not talking about hotplug feature of SPDK.
>
>
>
>
>
> Ø PCI hotplug feature is implemented in kernel driver and working fine.
>
> Ø But the moment I run SPDK and try to bind vfio driver it stalls
> completely.
>
> The reason is: kernel driver will not remove the root bus (when PCIe
> endpoint is not connected, during boot-time)
>
> So SPDK tries to bind driver thinking host bridge is there.
>
> Without PCI hotplug host bridge will not be there because of following API
> call in kernel driver.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
> Ø since now we allow host bridge creation (API: pci_create_root_bus)
> irrespective of EP is connected or not.
>
> And then if I run SPDK (with no Endpoint connected/Empty slot) I get
> stalls.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang,
> Cunyin
> *Sent:* Tuesday, August 29, 2017 2:14 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Could you please provide some details steps to reproduce the issue?
>
> SPDK take in charge for hotplug only after you bind the device to uio or
> vfio driver,
>
> so for the new insert deivce, it will handled by kernel driver first.
>
>
>
> -Cunyin
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Oza Oza
> *Sent:* Tuesday, August 29, 2017 4:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] PCI hotplug and SPDK
>
>
>
> Hi All,
>
>
>
> PCI hotplug support; requires creation of root bus and probe to go ahead
> with all PCIe configuration.
>
>
>
> Which means following APIs ae not called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
>
>
> And then If I run SPDK, It makes system crash with following info.
>
>
>
> Note: if the disk is connected then SPDK is fine.
>
>
>
> Otherwise it stalls the system with following crash.
>
>
>
> root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
> /usr/share/spdk/scripts/setup.sh
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
>
> [ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
>
> [ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
>
> [ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
>
> [ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
> -- SError
>
> [ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 97.180364] Modules linked in:
>
> [ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
> 4.12.0-01560-gc83093d-dirty #89
>
> [ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
>
> [ 97.204790] PC is at 0xffff7cbdfba8
>
> [ 97.208387] LR is at 0xffff7cb8f288
>
> [ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
> 20000000
>
> [ 97.219612] sp : 0000fffffe564040
>
> [ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
>
> [ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
>
> [ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
>
> [ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
>
> [ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
>
> [ 97.250427] x19: 0000000000000002 x18: 0000000000000000
>
> [ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
>
> [ 97.261386] x15: 0000000000000096 x14: 0000000000000000
>
> [ 97.266865] x13: 0000000000000000 x12: 0000000000000000
>
> [ 97.272344] x11: 0000000000000020 x10: 0101010101010101
>
> [ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
>
> [ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
>
> [ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
>
> [ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
>
> [ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
>
> [ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
>
> [ 97.311960] ---[ end trace a1f48abe30820241 ]---
>
>
>
> Regards,
>
> Oza.
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 114312 bytes --]
next reply other threads:[~2017-08-30 18:17 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-30 18:17 Oza Pawandeep [this message]
-- strict thread matches above, loose matches on Subject: below --
2017-08-30 19:55 [SPDK] PCI hotplug and SPDK Luse, Paul E
2017-08-30 19:43 Oza Oza
2017-08-30 19:31 Luse, Paul E
2017-08-30 19:22 Oza Oza
2017-08-30 19:07 Oza Pawandeep
2017-08-30 19:03 Harris, James R
2017-08-30 18:58 Harris, James R
2017-08-30 18:51 Oza Oza
2017-08-30 18:48 Oza Oza
2017-08-30 18:39 Harris, James R
2017-08-30 15:55 Walker, Benjamin
2017-08-30 15:30 Harris, James R
2017-08-30 15:20 Oza Oza
2017-08-30 15:10 Oza Oza
2017-08-29 18:00 Harris, James R
2017-08-29 16:45 Oza Oza
2017-08-29 12:50 Harris, James R
2017-08-29 8:51 Oza Oza
2017-08-29 8:43 Chang, Cunyin
2017-08-29 8:21 Oza Oza
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAK1A=4x1Hw3zU+G4tijmQH08Yi9KNgYWHCPjU7Vj3_0h3eu9gg@mail.gmail.com' \
--to=spdk@lists.01.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.