* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 15:30 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-30 15:30 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 44359 bytes --]
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <oza.oza(a)broadcom.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: "oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 138615 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:55 Luse, Paul E
0 siblings, 0 replies; 21+ messages in thread
From: Luse, Paul E @ 2017-08-30 19:55 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 51379 bytes --]
Yeah, just wondering if there’s an off the shelf thing that anyone can run to help you along as you make progress with SPDK. Running QEUMU emulation of ARMv8 on an IA machine for example. Not sure how applicable that would be to what you’re doing but might be a way some community members w/o readily available ARM h/w can help support your efforts. I’ll take a look at the foundation model, the user’s guide looks sorta interesting…
Thx
Paul
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Wednesday, August 30, 2017 12:44 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Oza Pawandeep <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
This is ARMv8 platform, but we don’t have any SW emulation package, RTL emulation should be there but not exactly the SW package what I call.
Although ARMv8 foundation model is available from ARM, which emulates ARMv8 and you can run VM inside it with SPDK running on that.
Is that what you meant to ask ?
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Luse, Paul E
Sent: Thursday, August 31, 2017 1:01 AM
To: Storage Performance Development Kit; Oza Pawandeep
Subject: Re: [SPDK] PCI hotplug and SPDK
Nice! Hey, I’m curious if there’s any SW emulation package available for the platform you’re developing on? It would be great to get some other architectures moving in the project…
-Paul
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Wednesday, August 30, 2017 12:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>; Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok so removing rescan works well now.
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~# vi /usr/share/spdk/scripts/setup.sh
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
Thanks a lot for your prompt support.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:34 AM
To: Storage Performance Development Kit; Oza Pawandeep
Subject: Re: [SPDK] PCI hotplug and SPDK
Have you confirmed that the issue happens when binding vfio, and it’s not the echo 1 > /sys/bus/pci/rescan?
I don’t think that rescan is needed in setup.sh, so I’m curious if setup.sh works on your platform if you remove those rescans when you have empty slots.
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 11:58 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>, Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
There have been improvements to setup.sh on master as well – I would be curious to know if those also fix the parsing issue that you saw on your system.
The SPDK script is only looking for NVMe devices in lspci output, and then trying to bind them to vfio. I suspect you would see a similar issue if you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using vfio in your setup.
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 11:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>, Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Thursday, August 31, 2017 12:18 AM
To: 'Storage Performance Development Kit'; 'Oza Pawandeep'
Subject: RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably I think vfio.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:09 AM
To: Oza Pawandeep
Cc: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see three root ports in your lspci output – two for the NVMe devices, another for the multi-function Ethernet device.
-Jim
From: Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Date: Wednesday, August 30, 2017 at 11:17 AM
To: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 165234 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:43 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 19:43 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 51479 bytes --]
This is ARMv8 platform, but we don’t have any SW emulation package, RTL
emulation should be there but not exactly the SW package what I call.
Although ARMv8 foundation model is available from ARM, which emulates ARMv8
and you can run VM inside it with SPDK running on that.
Is that what you meant to ask ?
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Luse, Paul E
*Sent:* Thursday, August 31, 2017 1:01 AM
*To:* Storage Performance Development Kit; Oza Pawandeep
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Nice! Hey, I’m curious if there’s any SW emulation package available for
the platform you’re developing on? It would be great to get some other
architectures moving in the project…
-Paul
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Wednesday, August 30, 2017 12:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>; Oza
Pawandeep <oza.pawandeep(a)gmail.com>
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok so removing rescan works well now.
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~# vi /usr/share/spdk/scripts/setup.sh
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
Thanks a lot for your prompt support.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Thursday, August 31, 2017 12:34 AM
*To:* Storage Performance Development Kit; Oza Pawandeep
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Have you confirmed that the issue happens when binding vfio, and it’s not
the echo 1 > /sys/bus/pci/rescan?
I don’t think that rescan is needed in setup.sh, so I’m curious if setup.sh
works on your platform if you remove those rescans when you have empty
slots.
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of James Harris <
james.r.harris(a)intel.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 11:58 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
Pawandeep <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
There have been improvements to setup.sh on master as well – I would be
curious to know if those also fix the parsing issue that you saw on your
system.
The SPDK script is only looking for NVMe devices in lspci output, and then
trying to bind them to vfio. I suspect you would see a similar issue if
you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using
vfio in your setup.
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 11:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
Pawandeep <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in
the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print
$1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of
parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Thursday, August 31, 2017 12:18 AM
*To:* 'Storage Performance Development Kit'; 'Oza Pawandeep'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably
I think vfio.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Thursday, August 31, 2017 12:09 AM
*To:* Oza Pawandeep
*Cc:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see
three root ports in your lspci output – two for the NVMe devices, another
for the multi-function Ethernet device.
-Jim
*From: *Oza Pawandeep <oza.pawandeep(a)gmail.com>
*Date: *Wednesday, August 30, 2017 at 11:17 AM
*To: *James Harris <james.r.harris(a)intel.com>
*Cc: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if
they are not attached?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 8:20 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if
00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Wednesday, August 30, 2017 8:41 PM
*To:* 'Storage Performance Development Kit'
*Cc:* 'oza.pawandeep(a)gmail.com'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 129590 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:31 Luse, Paul E
0 siblings, 0 replies; 21+ messages in thread
From: Luse, Paul E @ 2017-08-30 19:31 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 50011 bytes --]
Nice! Hey, I’m curious if there’s any SW emulation package available for the platform you’re developing on? It would be great to get some other architectures moving in the project…
-Paul
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Wednesday, August 30, 2017 12:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Oza Pawandeep <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok so removing rescan works well now.
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~# vi /usr/share/spdk/scripts/setup.sh
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
Thanks a lot for your prompt support.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:34 AM
To: Storage Performance Development Kit; Oza Pawandeep
Subject: Re: [SPDK] PCI hotplug and SPDK
Have you confirmed that the issue happens when binding vfio, and it’s not the echo 1 > /sys/bus/pci/rescan?
I don’t think that rescan is needed in setup.sh, so I’m curious if setup.sh works on your platform if you remove those rescans when you have empty slots.
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 11:58 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>, Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
There have been improvements to setup.sh on master as well – I would be curious to know if those also fix the parsing issue that you saw on your system.
The SPDK script is only looking for NVMe devices in lspci output, and then trying to bind them to vfio. I suspect you would see a similar issue if you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using vfio in your setup.
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 11:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>, Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Thursday, August 31, 2017 12:18 AM
To: 'Storage Performance Development Kit'; 'Oza Pawandeep'
Subject: RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably I think vfio.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:09 AM
To: Oza Pawandeep
Cc: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see three root ports in your lspci output – two for the NVMe devices, another for the multi-function Ethernet device.
-Jim
From: Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Date: Wednesday, August 30, 2017 at 11:17 AM
To: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 161319 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:22 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 19:22 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 50382 bytes --]
Hi Jim,
Ok so removing rescan works well now.
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~#
root(a)bcm958742k:~# vi /usr/share/spdk/scripts/setup.sh
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
Thanks a lot for your prompt support.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Thursday, August 31, 2017 12:34 AM
*To:* Storage Performance Development Kit; Oza Pawandeep
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Have you confirmed that the issue happens when binding vfio, and it’s not
the echo 1 > /sys/bus/pci/rescan?
I don’t think that rescan is needed in setup.sh, so I’m curious if setup.sh
works on your platform if you remove those rescans when you have empty
slots.
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of James Harris <
james.r.harris(a)intel.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 11:58 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
Pawandeep <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
There have been improvements to setup.sh on master as well – I would be
curious to know if those also fix the parsing issue that you saw on your
system.
The SPDK script is only looking for NVMe devices in lspci output, and then
trying to bind them to vfio. I suspect you would see a similar issue if
you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using
vfio in your setup.
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 11:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
Pawandeep <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in
the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print
$1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of
parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Thursday, August 31, 2017 12:18 AM
*To:* 'Storage Performance Development Kit'; 'Oza Pawandeep'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably
I think vfio.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Thursday, August 31, 2017 12:09 AM
*To:* Oza Pawandeep
*Cc:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see
three root ports in your lspci output – two for the NVMe devices, another
for the multi-function Ethernet device.
-Jim
*From: *Oza Pawandeep <oza.pawandeep(a)gmail.com>
*Date: *Wednesday, August 30, 2017 at 11:17 AM
*To: *James Harris <james.r.harris(a)intel.com>
*Cc: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if
they are not attached?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 8:20 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if
00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Wednesday, August 30, 2017 8:41 PM
*To:* 'Storage Performance Development Kit'
*Cc:* 'oza.pawandeep(a)gmail.com'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 125985 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:07 Oza Pawandeep
0 siblings, 0 replies; 21+ messages in thread
From: Oza Pawandeep @ 2017-08-30 19:07 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 52213 bytes --]
ok that seems reasonable point, let me try to see that part and get back to
you.
I have not myself digged into this issue so far.
was just thinking out loud.
will let you know in some time.
Regards,
Oza.
On Thu, Aug 31, 2017 at 12:33 AM, Harris, James R <james.r.harris(a)intel.com>
wrote:
> Have you confirmed that the issue happens when binding vfio, and it’s not
> the echo 1 > /sys/bus/pci/rescan?
>
>
>
> I don’t think that rescan is needed in setup.sh, so I’m curious if
> setup.sh works on your platform if you remove those rescans when you have
> empty slots.
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of James Harris <
> james.r.harris(a)intel.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 11:58 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
> Pawandeep <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> There have been improvements to setup.sh on master as well – I would be
> curious to know if those also fix the parsing issue that you saw on your
> system.
>
>
>
> The SPDK script is only looking for NVMe devices in lspci output, and then
> trying to bind them to vfio. I suspect you would see a similar issue if
> you tried to bind one of the Ethernet PCI functions to vfio.
>
>
>
> It sounds like there are some platform-specific issues related to using
> vfio in your setup.
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 11:51 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>, Oza
> Pawandeep <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> I had attached setup.sh script earlier.
>
> I have done minor changes, and in-fact I have fixed one thing.
>
>
>
>
>
> function linux_iter_pci {
>
> # Argument is the class code
>
> # TODO: More specifically match against only class codes
> in the grep
>
> # step.
>
> lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " "
> '{print $1}'
>
> }
>
>
>
> Somehow above parsing work for us, and original SPDK setup.sh way of
> parsing fails.
>
>
>
> But anyway, I just attached for your reference
>
>
>
> Regards,
>
> Oza.
>
>
>
> *From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
> *Sent:* Thursday, August 31, 2017 12:18 AM
> *To:* 'Storage Performance Development Kit'; 'Oza Pawandeep'
> *Subject:* RE: [SPDK] PCI hotplug and SPDK
>
>
>
> Exactly, that’s the whole point.
>
> There are no root ports appearing in lspci output.
>
>
>
> They will appear only after pci_rescan_bus (scanning the root port)
>
> And even if they don’t appear, SPDK script does something with it,
> probably I think vfio.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Harris, James R
> *Sent:* Thursday, August 31, 2017 12:09 AM
> *To:* Oza Pawandeep
> *Cc:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Where are the root ports associated with these empty slots? I only see
> three root ports in your lspci output – two for the NVMe devices, another
> for the multi-function Ethernet device.
>
>
>
> -Jim
>
>
>
>
>
> *From: *Oza Pawandeep <oza.pawandeep(a)gmail.com>
> *Date: *Wednesday, August 30, 2017 at 11:17 AM
> *To: *James Harris <james.r.harris(a)intel.com>
> *Cc: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Jim,
>
>
>
> Ok, so we have 4 PCIe slots.
>
> I have kept 2 slots with NVMe driver and 2 slots empty.
>
>
>
> Regards,
>
> Oza.
>
>
>
> On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
> wrote:
>
> Hi Oza,
>
>
>
> I am confused – why do the NVMe devices show up in your lspci output if
> they are not attached?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 8:20 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci -vvvx
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> lspci: Unable to load libkmod resources: error -12
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 480000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 780000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
> (prog-if 00 [Normal decode])
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: fff00000-000fffff
>
> Prefetchable memory behind bridge: 0000000010000000-
> 00000000104fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] #00 [0000]
>
> 00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [160 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b0 v1] Latency Tolerance Reporting
>
> Max snoop latency: 0ns
>
> Max no snoop latency: 0ns
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 1
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
>
> 20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 2
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
>
> 20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 3
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
>
> 20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 4
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
>
> 20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> *From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
> *Sent:* Wednesday, August 30, 2017 8:41 PM
> *To:* 'Storage Performance Development Kit'
> *Cc:* 'oza.pawandeep(a)gmail.com'
> *Subject:* RE: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01)
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>
>
> root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
>
> 0001:01:00.0 (8086 0953): nvme -> vfio-pci
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 1520.267436] pci 0008:00:00.0: bridge window [mem
> 0x10000000-0x104fffff 64bit pref]
>
> [ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
> header type 01) >> it is looking to unbind on empty slot as
> well.
>
> [ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
> -- SError
>
> [ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 1535.924850] Modules linked in:
>
> [ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
> 4.12.0-01624-gbbd4086-dirty #97
>
> [ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
>
> [ 1535.949634] PC is at 0xffff8baadca0
>
> [ 1535.953230] LR is at 0x40aca8
>
> [ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
> 80000000
>
> [ 1535.963919] sp : 0000ffffca61a970
>
> [ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
>
> [ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
>
> [ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
>
> [ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
>
> [ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
>
> [ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
>
> [ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
>
> [ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
>
> [ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
>
> [ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
>
> [ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
>
> [ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
>
> [ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
>
> [ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
>
> [ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
>
> [ 1536.049529] Process lighttpd (pid: 2108, stack limit =
> 0xffff80a162790000)
>
>
>
> Please find attached status script.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Harris, James R
> *Sent:* Tuesday, August 29, 2017 11:30 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> So to clarify, you have your system booted with no NVMe endpoint
> connected, and then when you run the SPDK setup.sh script, you see all of
> these kernel messages from trying to bind vfio to PCIe devices and system
> eventually crashes?
>
>
>
> If so, we need to determine what PCIe devices setup.sh is trying to bind
> to vfio. It should only be trying to bind NVMe devices but if there is no
> NVMe device connected then it shouldn’t be trying to bind anything.
>
>
>
> Can you send lspci –vvvx output from your system before running setup.sh?
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 9:45 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> In my opinion, this has nothing to do with platform. Though our platform
> is ARMv8.
>
> (but, I can not test on any other, because we don’t know how the kernel
> driver is written)
>
>
>
> If kernel driver supports hotplug, which means they are allowing
> pci_create_root_bus irrespective of whether EP is plugged or not.
>
> In other words. Following APIs are never called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
> and in that case, if PCIe slots is empty, running SPDK resulting in
> stalls. (10-15 seconds) followed by crash.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
> James R
> *Sent:* Tuesday, August 29, 2017 6:20 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Do you see this issue only on your armv8 platform or do you also see it on
> amd64?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 1:51 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Sorry If I was unclear.
>
> I am not talking about hotplug feature of SPDK.
>
>
>
>
>
> Ø PCI hotplug feature is implemented in kernel driver and working fine.
>
> Ø But the moment I run SPDK and try to bind vfio driver it stalls
> completely.
>
> The reason is: kernel driver will not remove the root bus (when PCIe
> endpoint is not connected, during boot-time)
>
> So SPDK tries to bind driver thinking host bridge is there.
>
> Without PCI hotplug host bridge will not be there because of following API
> call in kernel driver.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
> Ø since now we allow host bridge creation (API: pci_create_root_bus)
> irrespective of EP is connected or not.
>
> And then if I run SPDK (with no Endpoint connected/Empty slot) I get
> stalls.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang,
> Cunyin
> *Sent:* Tuesday, August 29, 2017 2:14 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Could you please provide some details steps to reproduce the issue?
>
> SPDK take in charge for hotplug only after you bind the device to uio or
> vfio driver,
>
> so for the new insert deivce, it will handled by kernel driver first.
>
>
>
> -Cunyin
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Oza Oza
> *Sent:* Tuesday, August 29, 2017 4:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] PCI hotplug and SPDK
>
>
>
> Hi All,
>
>
>
> PCI hotplug support; requires creation of root bus and probe to go ahead
> with all PCIe configuration.
>
>
>
> Which means following APIs ae not called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
>
>
> And then If I run SPDK, It makes system crash with following info.
>
>
>
> Note: if the disk is connected then SPDK is fine.
>
>
>
> Otherwise it stalls the system with following crash.
>
>
>
> root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
> /usr/share/spdk/scripts/setup.sh
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
>
> [ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
>
> [ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
>
> [ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
>
> [ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
> -- SError
>
> [ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 97.180364] Modules linked in:
>
> [ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
> 4.12.0-01560-gc83093d-dirty #89
>
> [ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
>
> [ 97.204790] PC is at 0xffff7cbdfba8
>
> [ 97.208387] LR is at 0xffff7cb8f288
>
> [ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
> 20000000
>
> [ 97.219612] sp : 0000fffffe564040
>
> [ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
>
> [ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
>
> [ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
>
> [ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
>
> [ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
>
> [ 97.250427] x19: 0000000000000002 x18: 0000000000000000
>
> [ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
>
> [ 97.261386] x15: 0000000000000096 x14: 0000000000000000
>
> [ 97.266865] x13: 0000000000000000 x12: 0000000000000000
>
> [ 97.272344] x11: 0000000000000020 x10: 0101010101010101
>
> [ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
>
> [ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
>
> [ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
>
> [ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
>
> [ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
>
> [ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
>
> [ 97.311960] ---[ end trace a1f48abe30820241 ]---
>
>
>
> Regards,
>
> Oza.
>
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 128703 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 19:03 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-30 19:03 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 48079 bytes --]
Have you confirmed that the issue happens when binding vfio, and it’s not the echo 1 > /sys/bus/pci/rescan?
I don’t think that rescan is needed in setup.sh, so I’m curious if setup.sh works on your platform if you remove those rescans when you have empty slots.
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of James Harris <james.r.harris(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, August 30, 2017 at 11:58 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Oza Pawandeep <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
There have been improvements to setup.sh on master as well – I would be curious to know if those also fix the parsing issue that you saw on your system.
The SPDK script is only looking for NVMe devices in lspci output, and then trying to bind them to vfio. I suspect you would see a similar issue if you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using vfio in your setup.
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <oza.oza(a)broadcom.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, August 30, 2017 at 11:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Oza Pawandeep <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Thursday, August 31, 2017 12:18 AM
To: 'Storage Performance Development Kit'; 'Oza Pawandeep'
Subject: RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably I think vfio.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:09 AM
To: Oza Pawandeep
Cc: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see three root ports in your lspci output – two for the NVMe devices, another for the multi-function Ethernet device.
-Jim
From: Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Date: Wednesday, August 30, 2017 at 11:17 AM
To: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 152855 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 18:58 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-30 18:58 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 47441 bytes --]
Hi Oza,
There have been improvements to setup.sh on master as well – I would be curious to know if those also fix the parsing issue that you saw on your system.
The SPDK script is only looking for NVMe devices in lspci output, and then trying to bind them to vfio. I suspect you would see a similar issue if you tried to bind one of the Ethernet PCI functions to vfio.
It sounds like there are some platform-specific issues related to using vfio in your setup.
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <oza.oza(a)broadcom.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, August 30, 2017 at 11:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Oza Pawandeep <oza.pawandeep(a)gmail.com>
Subject: Re: [SPDK] PCI hotplug and SPDK
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Thursday, August 31, 2017 12:18 AM
To: 'Storage Performance Development Kit'; 'Oza Pawandeep'
Subject: RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably I think vfio.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Thursday, August 31, 2017 12:09 AM
To: Oza Pawandeep
Cc: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see three root ports in your lspci output – two for the NVMe devices, another for the multi-function Ethernet device.
-Jim
From: Oza Pawandeep <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Date: Wednesday, August 30, 2017 at 11:17 AM
To: James Harris <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 150908 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 18:51 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 18:51 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 47653 bytes --]
I had attached setup.sh script earlier.
I have done minor changes, and in-fact I have fixed one thing.
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in
the grep
# step.
lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print
$1}'
}
Somehow above parsing work for us, and original SPDK setup.sh way of
parsing fails.
But anyway, I just attached for your reference
Regards,
Oza.
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Thursday, August 31, 2017 12:18 AM
*To:* 'Storage Performance Development Kit'; 'Oza Pawandeep'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably
I think vfio.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Thursday, August 31, 2017 12:09 AM
*To:* Oza Pawandeep
*Cc:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see
three root ports in your lspci output – two for the NVMe devices, another
for the multi-function Ethernet device.
-Jim
*From: *Oza Pawandeep <oza.pawandeep(a)gmail.com>
*Date: *Wednesday, August 30, 2017 at 11:17 AM
*To: *James Harris <james.r.harris(a)intel.com>
*Cc: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if
they are not attached?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 8:20 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if
00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Wednesday, August 30, 2017 8:41 PM
*To:* 'Storage Performance Development Kit'
*Cc:* 'oza.pawandeep(a)gmail.com'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 115116 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 18:48 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 18:48 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 46845 bytes --]
Exactly, that’s the whole point.
There are no root ports appearing in lspci output.
They will appear only after pci_rescan_bus (scanning the root port)
And even if they don’t appear, SPDK script does something with it, probably
I think vfio.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Thursday, August 31, 2017 12:09 AM
*To:* Oza Pawandeep
*Cc:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Where are the root ports associated with these empty slots? I only see
three root ports in your lspci output – two for the NVMe devices, another
for the multi-function Ethernet device.
-Jim
*From: *Oza Pawandeep <oza.pawandeep(a)gmail.com>
*Date: *Wednesday, August 30, 2017 at 11:17 AM
*To: *James Harris <james.r.harris(a)intel.com>
*Cc: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if
they are not attached?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, August 30, 2017 at 8:20 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if
00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Wednesday, August 30, 2017 8:41 PM
*To:* 'Storage Performance Development Kit'
*Cc:* 'oza.pawandeep(a)gmail.com'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 111322 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 18:39 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-30 18:39 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 45252 bytes --]
Hi Oza,
Where are the root ports associated with these empty slots? I only see three root ports in your lspci output – two for the NVMe devices, another for the multi-function Ethernet device.
-Jim
From: Oza Pawandeep <oza.pawandeep(a)gmail.com>
Date: Wednesday, August 30, 2017 at 11:17 AM
To: James Harris <james.r.harris(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 175107 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 18:17 Oza Pawandeep
0 siblings, 0 replies; 21+ messages in thread
From: Oza Pawandeep @ 2017-08-30 18:17 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 48252 bytes --]
Hi Jim,
Ok, so we have 4 PCIe slots.
I have kept 2 slots with NVMe driver and 2 slots empty.
Regards,
Oza.
On Wed, Aug 30, 2017 at 9:00 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:
> Hi Oza,
>
>
>
> I am confused – why do the NVMe devices show up in your lspci output if
> they are not attached?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, August 30, 2017 at 8:20 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc: *"oza.pawandeep(a)gmail.com" <oza.pawandeep(a)gmail.com>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci -vvvx
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> lspci: Unable to load libkmod resources: error -12
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 378
>
> Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 480000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
>
>
>
> 0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
> [Normal decode])
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: 00000000-000fffff
>
> Prefetchable memory behind bridge: 00000000fff00000-
> 00000000000fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0
>
> ExtTag+ RBE+
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt+
>
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible+
>
> RootCap: CRSVisible+
>
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Via WAKE# ARIFwd+
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled ARIFwd+
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [240 v1] L1 PM Substates
>
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+
>
> PortCommonModeRestoreTime=8us
> PortTPowerOnTime=10us
>
> Capabilities: [300 v1] #19
>
> 00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01) (prog-if 02 [NVM Express])
>
> Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
>
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Latency: 0
>
> Interrupt: pin A routed to IRQ 380
>
> Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
>
> [virtual] Expansion ROM at 780000000 [disabled] [size=64K]
>
> Capabilities: [40] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>
> Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
>
> Vector table: BAR=0 offset=00002000
>
> PBA: BAR=0 offset=00003000
>
> Capabilities: [60] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <4us
>
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <4us, L1 <4us
>
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR-, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete+, EqualizationPhase1+
>
> EqualizationPhase2+, EqualizationPhase3+,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [150 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Power Budgeting <?>
>
> Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 0
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
>
> Capabilities: [2a0 v1] #19
>
> Kernel driver in use: nvme
>
> 00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
>
> 10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
>
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
>
>
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
> (prog-if 00 [Normal decode])
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>
> I/O behind bridge: 00000000-00000fff
>
> Memory behind bridge: fff00000-000fffff
>
> Prefetchable memory behind bridge: 0000000010000000-
> 00000000104fffff
>
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] #00 [0000]
>
> 00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
>
> 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
>
> 20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
> SpeedDis-
>
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>
> Compliance De-emphasis: -6dB
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [160 v1] Virtual Channel
>
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>
> Arb: Fixed- WRR32- WRR64- WRR128-
>
> Ctrl: ArbSelect=Fixed
>
> Status: InProgress-
>
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>
> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
> WRR256-
>
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>
> Status: NegoPending- InProgress-
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b0 v1] Latency Tolerance Reporting
>
> Max snoop latency: 0ns
>
> Max no snoop latency: 0ns
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 1
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
>
> 20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 2
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
>
> 20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 3
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
>
> 20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> Subsystem: Broadcom Corporation Device 16f0
>
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx+
>
> Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
> [size=64K]
>
> Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
> [size=1M]
>
> Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
> [size=4K]
>
> Capabilities: [48] Power Management version 3
>
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>
> Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>
> Address: 0000000000000000 Data: 0000
>
> Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
>
> Vector table: BAR=4 offset=00000000
>
> PBA: BAR=4 offset=00001000
>
> Capabilities: [ac] Express (v2) Endpoint, MSI 00
>
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> <4us, L1 <64us
>
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
>
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
> FLReset-
>
> MaxPayload 128 bytes, MaxReadReq 512 bytes
>
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
> TransPend-
>
> LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
> Latency L0s <2us, L1 <4us
>
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
> LTR+, OBFF Not Supported
>
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
> LTR-, OBFF Disabled
>
> LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
>
> Capabilities: [100 v1] Advanced Error Reporting
>
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr-
>
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> NonFatalErr+
>
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
> ChkEn-
>
> Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
>
> Capabilities: [150 v1] Power Budgeting <?>
>
> Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
> Len=028 <?>
>
> Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
>
> ARICap: MFVC- ACS-, Next Function: 4
>
> ARICtl: MFVC- ACS-, Function Group: 0
>
> 00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
>
> 10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
>
> 20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
>
> 30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
>
>
>
> *From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
> *Sent:* Wednesday, August 30, 2017 8:41 PM
> *To:* 'Storage Performance Development Kit'
> *Cc:* 'oza.pawandeep(a)gmail.com'
> *Subject:* RE: [SPDK] PCI hotplug and SPDK
>
>
>
> root(a)bcm958742k:~# lspci
>
> 0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
>
> 0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
> Center SSD (rev 01)
>
> 0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
> 0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
>
>
>
> root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
>
> 0001:01:00.0 (8086 0953): nvme -> vfio-pci
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 1520.267436] pci 0008:00:00.0: bridge window [mem
> 0x10000000-0x104fffff 64bit pref]
>
> [ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
> header type 01) >> it is looking to unbind on empty slot as
> well.
>
> [ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
> -- SError
>
> [ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 1535.924850] Modules linked in:
>
> [ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
> 4.12.0-01624-gbbd4086-dirty #97
>
> [ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
>
> [ 1535.949634] PC is at 0xffff8baadca0
>
> [ 1535.953230] LR is at 0x40aca8
>
> [ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
> 80000000
>
> [ 1535.963919] sp : 0000ffffca61a970
>
> [ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
>
> [ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
>
> [ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
>
> [ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
>
> [ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
>
> [ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
>
> [ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
>
> [ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
>
> [ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
>
> [ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
>
> [ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
>
> [ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
>
> [ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
>
> [ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
>
> [ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
>
> [ 1536.049529] Process lighttpd (pid: 2108, stack limit =
> 0xffff80a162790000)
>
>
>
> Please find attached status script.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Harris, James R
> *Sent:* Tuesday, August 29, 2017 11:30 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> So to clarify, you have your system booted with no NVMe endpoint
> connected, and then when you run the SPDK setup.sh script, you see all of
> these kernel messages from trying to bind vfio to PCIe devices and system
> eventually crashes?
>
>
>
> If so, we need to determine what PCIe devices setup.sh is trying to bind
> to vfio. It should only be trying to bind NVMe devices but if there is no
> NVMe device connected then it shouldn’t be trying to bind anything.
>
>
>
> Can you send lspci –vvvx output from your system before running setup.sh?
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 9:45 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> In my opinion, this has nothing to do with platform. Though our platform
> is ARMv8.
>
> (but, I can not test on any other, because we don’t know how the kernel
> driver is written)
>
>
>
> If kernel driver supports hotplug, which means they are allowing
> pci_create_root_bus irrespective of whether EP is plugged or not.
>
> In other words. Following APIs are never called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
> and in that case, if PCIe slots is empty, running SPDK resulting in
> stalls. (10-15 seconds) followed by crash.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
> James R
> *Sent:* Tuesday, August 29, 2017 6:20 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Do you see this issue only on your armv8 platform or do you also see it on
> amd64?
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
> oza.oza(a)broadcom.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, August 29, 2017 at 1:51 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Sorry If I was unclear.
>
> I am not talking about hotplug feature of SPDK.
>
>
>
>
>
> Ø PCI hotplug feature is implemented in kernel driver and working fine.
>
> Ø But the moment I run SPDK and try to bind vfio driver it stalls
> completely.
>
> The reason is: kernel driver will not remove the root bus (when PCIe
> endpoint is not connected, during boot-time)
>
> So SPDK tries to bind driver thinking host bridge is there.
>
> Without PCI hotplug host bridge will not be there because of following API
> call in kernel driver.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
> Ø since now we allow host bridge creation (API: pci_create_root_bus)
> irrespective of EP is connected or not.
>
> And then if I run SPDK (with no Endpoint connected/Empty slot) I get
> stalls.
>
>
>
> Regards,
>
> Oza.
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang,
> Cunyin
> *Sent:* Tuesday, August 29, 2017 2:14 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] PCI hotplug and SPDK
>
>
>
> Hi Oza,
>
>
>
> Could you please provide some details steps to reproduce the issue?
>
> SPDK take in charge for hotplug only after you bind the device to uio or
> vfio driver,
>
> so for the new insert deivce, it will handled by kernel driver first.
>
>
>
> -Cunyin
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Oza Oza
> *Sent:* Tuesday, August 29, 2017 4:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] PCI hotplug and SPDK
>
>
>
> Hi All,
>
>
>
> PCI hotplug support; requires creation of root bus and probe to go ahead
> with all PCIe configuration.
>
>
>
> Which means following APIs ae not called.
>
> pci_stop_root_bus(bus);
>
> pci_remove_root_bus(bus);
>
>
>
>
>
> And then If I run SPDK, It makes system crash with following info.
>
>
>
> Note: if the disk is connected then SPDK is fine.
>
>
>
> Otherwise it stalls the system with following crash.
>
>
>
> root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
> /usr/share/spdk/scripts/setup.sh
>
> grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
> 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
>
> file or directory
>
> [ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
>
> [ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
>
> [ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
>
> [ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
>
> [ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
>
> [ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
> -- SError
>
> [ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
>
> [ 97.180364] Modules linked in:
>
> [ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
> 4.12.0-01560-gc83093d-dirty #89
>
> [ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
> (DT)
>
> [ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
>
> [ 97.204790] PC is at 0xffff7cbdfba8
>
> [ 97.208387] LR is at 0xffff7cb8f288
>
> [ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
> 20000000
>
> [ 97.219612] sp : 0000fffffe564040
>
> [ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
>
> [ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
>
> [ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
>
> [ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
>
> [ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
>
> [ 97.250427] x19: 0000000000000002 x18: 0000000000000000
>
> [ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
>
> [ 97.261386] x15: 0000000000000096 x14: 0000000000000000
>
> [ 97.266865] x13: 0000000000000000 x12: 0000000000000000
>
> [ 97.272344] x11: 0000000000000020 x10: 0101010101010101
>
> [ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
>
> [ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
>
> [ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
>
> [ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
>
> [ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
>
> [ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
>
> [ 97.311960] ---[ end trace a1f48abe30820241 ]---
>
>
>
> Regards,
>
> Oza.
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 114312 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 15:55 Walker, Benjamin
0 siblings, 0 replies; 21+ messages in thread
From: Walker, Benjamin @ 2017-08-30 15:55 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 45335 bytes --]
Also, just to confirm – that Intel 750 NVMe SSD isn’t your boot drive, right? If you have an NVMe boot drive, I think there are going to be problems because our setup.sh script is going to try and claim all NVMe devices. You could modify setup.sh to just skip that particular bus:device:function, or manually unbind the right devices from the kernel driver. Only setup.sh, which is provided for convenience only, tries to claim all NVMe devices. The rest of SPDK just uses devices that are already bound to vfio or uio.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Wednesday, August 30, 2017 8:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: oza.pawandeep(a)gmail.com
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
I am confused – why do the NVMe devices show up in your lspci output if they are not attached?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Wednesday, August 30, 2017 at 8:20 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Cc: "oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>" <oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>>
Subject: Re: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled] [size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled] [size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled] [size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
From: Oza Oza [mailto:oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>]
Sent: Wednesday, August 30, 2017 8:41 PM
To: 'Storage Performance Development Kit'
Cc: 'oza.pawandeep(a)gmail.com<mailto:oza.pawandeep(a)gmail.com>'
Subject: RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff 64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match header type 01) >> it is looking to unbind on empty slot as well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002 -- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted 4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate: 80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit = 0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 11:30 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 140555 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 15:20 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 15:20 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 45078 bytes --]
root(a)bcm958742k:~# lspci -vvvx
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
lspci: Unable to load libkmod resources: error -12
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7a 01 00 00
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 378
Region 0: Memory at 480010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 480000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-b3-9e
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7a 01 00 00
0007:00:00.0 PCI bridge: Broadcom Corporation Device 0000 (prog-if 00
[Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Via WAKE# ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [240 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=8us
PortTPowerOnTime=10us
Capabilities: [300 v1] #19
00: e4 14 00 00 06 00 10 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 00 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 7c 01 00 00
0007:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Intel Corporation SSD 750 Series [Add-in Card]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 380
Region 0: Memory at 780010000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 780000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [150 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Power Budgeting <?>
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [270 v1] Device Serial Number 55-cd-2e-41-4d-25-bf-ba
Capabilities: [2a0 v1] #19
Kernel driver in use: nvme
00: 86 80 53 09 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 0d 37
30: 00 00 00 00 40 00 00 00 00 00 00 00 7c 01 00 00
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01) (prog-if
00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 0000000010000000-00000000104fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] #00 [0000]
00: e4 14 f0 16 00 00 10 00 01 00 00 02 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: f0 ff 00 00 01 10 41 10 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10400000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10440000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-00
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 40 10 00 00 00 00 0c 00 00 10 00 00 00 00
20: 0c 00 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10410000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10100000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10441000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-01
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 41 10 00 00 00 00 0c 00 10 10 00 00 00 00
20: 0c 10 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10420000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10200000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10442000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-02
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 42 10 00 00 00 00 0c 00 20 10 00 00 00 00
20: 0c 20 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
Subsystem: Broadcom Corporation Device 16f0
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx+
Region 0: Memory at 10430000 (64-bit, prefetchable) [disabled]
[size=64K]
Region 2: Memory at 10300000 (64-bit, prefetchable) [disabled]
[size=1M]
Region 4: Memory at 10443000 (64-bit, prefetchable) [disabled]
[size=4K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=148 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit
Latency L0s <2us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+
ChkEn-
Capabilities: [13c v1] Device Serial Number 00-10-18-ff-fe-ad-05-03
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0
Len=028 <?>
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
00: e4 14 f0 16 00 00 18 00 01 00 00 02 00 00 80 00
10: 0c 00 43 10 00 00 00 00 0c 00 30 10 00 00 00 00
20: 0c 30 44 10 00 00 00 00 00 00 00 00 e4 14 f0 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 00 00 00 00
*From:* Oza Oza [mailto:oza.oza(a)broadcom.com]
*Sent:* Wednesday, August 30, 2017 8:41 PM
*To:* 'Storage Performance Development Kit'
*Cc:* 'oza.pawandeep(a)gmail.com'
*Subject:* RE: [SPDK] PCI hotplug and SPDK
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Harris, James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 91721 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-30 15:10 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-30 15:10 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 9816 bytes --]
root(a)bcm958742k:~# lspci
0001:00:00.0 PCI bridge: Broadcom Corporation Device 0000
0001:01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data
Center SSD (rev 01)
0008:00:00.0 PCI bridge: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.0 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.1 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.2 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
0008:01:00.3 Ethernet controller: Broadcom Corporation Device 16f0 (rev 01)
root(a)bcm958742k:~# /usr/share/spdk/scripts/setup.sh
0001:01:00.0 (8086 0953): nvme -> vfio-pci
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
1520.258498] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 1520.267436] pci 0008:00:00.0: bridge window [mem 0x10000000-0x104fffff
64bit pref]
[ 1520.277225] pci 0000:00:00.0: ignoring class 0x020000 (doesn't match
header type 01) >> it is looking to unbind on empty slot as
well.
[ 1520.285324] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 1535.911738] Bad mode in Error handler detected on CPU7, code 0xbf000002
-- SError
[ 1535.919460] Internal error: Oops - bad mode: 0 [#1] SMP
[ 1535.924850] Modules linked in:
[ 1535.928001] CPU: 7 PID: 2108 Comm: lighttpd Not tainted
4.12.0-01624-gbbd4086-dirty #97
[ 1535.936257] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 1535.943527] task: ffff80a1642ce000 task.stack: ffff80a162790000
[ 1535.949634] PC is at 0xffff8baadca0
[ 1535.953230] LR is at 0x40aca8
[ 1535.956290] pc : [<0000ffff8baadca0>] lr : [<000000000040aca8>] pstate:
80000000
[ 1535.963919] sp : 0000ffffca61a970
[ 1535.967337] x29: 0000ffffca61a970 x28: 0000000000000000
[ 1535.972816] x27: 00000000283a9a00 x26: 0000000000000000
[ 1535.978296] x25: 000000000042a3a8 x24: 000000000042a3a0
[ 1535.983775] x23: 0000000000429000 x22: 000000000042a2b8
[ 1535.989254] x21: 000000000042a000 x20: 0000000058edd444
[ 1535.994734] x19: 00000000283a9010 x18: 0000000000000014
[ 1536.000213] x17: 0000ffff8baada88 x16: 0000000000442c08
[ 1536.005693] x15: 00002162cc000000 x14: 000bcd3d80000000
[ 1536.011173] x13: 00000001f4000000 x12: 0000000000000017
[ 1536.016653] x11: 0000000000061bf7 x10: 0000000058edd444
[ 1536.022131] x9 : 001dcd6500000000 x8 : 0000000000000016
[ 1536.027611] x7 : 000000000000dfda x6 : 0000ffff8bdc5000
[ 1536.033090] x5 : 0000000000000008 x4 : 0000000000000000
[ 1536.038570] x3 : 00000000000003e8 x2 : 0000000000000401
[ 1536.044049] x1 : 00000000283bd050 x0 : 0000000000000000
[ 1536.049529] Process lighttpd (pid: 2108, stack limit =
0xffff80a162790000)
Please find attached status script.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 11:30 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
So to clarify, you have your system booted with no NVMe endpoint connected,
and then when you run the SPDK setup.sh script, you see all of these kernel
messages from trying to bind vfio to PCIe devices and system eventually
crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to
vfio. It should only be trying to bind NVMe devices but if there is no
NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 9:45 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 24252 bytes --]
[-- Attachment #3: setup.sh --]
[-- Type: application/octet-stream, Size: 6259 bytes --]
#!/usr/bin/env bash
set -e
rootdir=$(readlink -f $(dirname $0))/..
function linux_iter_pci {
# Argument is the class code
# TODO: More specifically match against only class codes in the grep
# step.
#lspci -D -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
lspci -mm -n | grep $1 | tr -d '"' | awk -F " " '{print $1}'
}
function linux_bind_driver() {
bdf="$1"
driver_name="$2"
old_driver_name="no driver"
ven_dev_id=$(lspci -n -s $bdf | cut -d' ' -f3 | sed 's/:/ /')
if [ -e "/sys/bus/pci/devices/$bdf/driver" ]; then
old_driver_name=$(basename $(readlink /sys/bus/pci/devices/$bdf/driver))
if [ "$driver_name" = "$old_driver_name" ]; then
return 0
fi
echo "$ven_dev_id" > "/sys/bus/pci/devices/$bdf/driver/remove_id" 2> /dev/null || true
echo "$bdf" > "/sys/bus/pci/devices/$bdf/driver/unbind"
fi
echo "$bdf ($ven_dev_id): $old_driver_name -> $driver_name"
echo "$ven_dev_id" > "/sys/bus/pci/drivers/$driver_name/new_id" 2> /dev/null || true
echo "$bdf" > "/sys/bus/pci/drivers/$driver_name/bind" 2> /dev/null || true
iommu_group=$(basename $(readlink -f /sys/bus/pci/devices/$bdf/iommu_group))
if [ -e "/dev/vfio/$iommu_group" ]; then
if [ "$username" != "" ]; then
chown "$username" "/dev/vfio/$iommu_group"
fi
fi
}
function linux_hugetlbfs_mount() {
mount | grep '^hugetlbfs ' | awk '{ print $3 }'
}
function configure_linux {
driver_name=vfio-pci
if [ -z "$(ls /sys/kernel/iommu_groups)" ]; then
# No IOMMU. Use uio.
driver_name=uio_pci_generic
fi
# NVMe
if [ ! -d /sys/module/vfio_pci ]; then
modprobe $driver_name || true
fi
for bdf in $(linux_iter_pci 0108); do
linux_bind_driver "$bdf" "$driver_name"
done
# IOAT
TMP=`mktemp`
#collect all the device_id info of ioat devices.
grep "PCI_DEVICE_ID_INTEL_IOAT" $rootdir/include/spdk/pci_ids.h \
| awk -F"x" '{print $2}' > $TMP
for dev_id in `cat $TMP`; do
# Abuse linux_iter_pci by giving it a device ID instead of a class code
for bdf in $(linux_iter_pci $dev_id); do
linux_bind_driver "$bdf" "$driver_name"
done
done
rm $TMP
echo "1" > "/sys/bus/pci/rescan"
hugetlbfs_mount=$(linux_hugetlbfs_mount)
if [ -z "$hugetlbfs_mount" ]; then
hugetlbfs_mount=/mnt/huge
echo "Mounting hugetlbfs at $hugetlbfs_mount"
mkdir -p "$hugetlbfs_mount"
mount -t hugetlbfs nodev "$hugetlbfs_mount"
fi
echo "$NRHUGE" > /proc/sys/vm/nr_hugepages
# increase the current memlock limit
ulimit -l unlimited
if [ "$driver_name" = "vfio-pci" ]; then
if [ "$username" != "" ]; then
chown "$username" "$hugetlbfs_mount"
fi
MEMLOCK_AMNT=`ulimit -l`
if [ "$MEMLOCK_AMNT" != "unlimited" ] ; then
MEMLOCK_MB=$(( $MEMLOCK_AMNT / 1024 ))
echo ""
echo "Current user memlock limit: ${MEMLOCK_MB} MB"
echo ""
echo "This is the maximum amount of memory you will be"
echo "able to use with DPDK and VFIO if run as current user."
echo -n "To change this, please adjust limits.conf memlock "
echo "limit for current user."
if [ $MEMLOCK_AMNT -lt 65536 ] ; then
echo ""
echo "## WARNING: memlock limit is less than 64MB"
echo -n "## DPDK with VFIO may not be able to initialize "
echo "if run as current user."
fi
fi
fi
}
function reset_linux {
# NVMe
modprobe nvme || true
for bdf in $(linux_iter_pci 0108); do
linux_bind_driver "$bdf" nvme
done
# IOAT
TMP=`mktemp`
#collect all the device_id info of ioat devices.
grep "PCI_DEVICE_ID_INTEL_IOAT" $rootdir/include/spdk/pci_ids.h \
| awk -F"x" '{print $2}' > $TMP
modprobe ioatdma || true
for dev_id in `cat $TMP`; do
# Abuse linux_iter_pci by giving it a device ID instead of a class code
for bdf in $(linux_iter_pci $dev_id); do
linux_bind_driver "$bdf" ioatdma
done
done
rm $TMP
echo "1" > "/sys/bus/pci/rescan"
hugetlbfs_mount=$(linux_hugetlbfs_mount)
rm -f "$hugetlbfs_mount"/spdk*map_*
}
function status_linux {
echo "NVMe devices"
echo -e "BDF\t\tNuma Node\tDriver name\t\tDevice name"
for bdf in $(linux_iter_pci 0108); do
driver=`grep DRIVER /sys/bus/pci/devices/$bdf/uevent |awk -F"=" '{print $2}'`
node=`cat /sys/bus/pci/devices/$bdf/numa_node`;
if [ "$driver" = "nvme" ]; then
name="\t"`ls /sys/bus/pci/devices/$bdf/nvme`;
else
name="-";
fi
echo -e "$bdf\t$node\t\t$driver\t\t$name";
done
echo "I/OAT DMA"
#collect all the device_id info of ioat devices.
TMP=`grep "PCI_DEVICE_ID_INTEL_IOAT" $rootdir/include/spdk/pci_ids.h \
| awk -F"x" '{print $2}'`
echo -e "BDF\t\tNuma Node\tDriver Name"
for dev_id in $TMP; do
# Abuse linux_iter_pci by giving it a device ID instead of a class code
for bdf in $(linux_iter_pci $dev_id); do
driver=`grep DRIVER /sys/bus/pci/devices/$bdf/uevent |awk -F"=" '{print $2}'`
node=`cat /sys/bus/pci/devices/$bdf/numa_node`;
echo -e "$bdf\t$node\t\t$driver"
done
done
}
function configure_freebsd {
TMP=`mktemp`
# NVMe
GREP_STR="class=0x010802"
# IOAT
grep "PCI_DEVICE_ID_INTEL_IOAT" $rootdir/include/spdk/pci_ids.h \
| awk -F"x" '{print $2}' > $TMP
for dev_id in `cat $TMP`; do
GREP_STR="${GREP_STR}\|chip=0x${dev_id}8086"
done
AWK_PROG="{if (count > 0) printf \",\"; printf \"%s:%s:%s\",\$2,\$3,\$4; count++}"
echo $AWK_PROG > $TMP
BDFS=`pciconf -l | grep "${GREP_STR}" | awk -F: -f $TMP`
kldunload nic_uio.ko || true
kenv hw.nic_uio.bdfs=$BDFS
kldload nic_uio.ko
rm $TMP
kldunload contigmem.ko || true
kenv hw.contigmem.num_buffers=$((NRHUGE * 2 / 256))
kenv hw.contigmem.buffer_size=$((256 * 1024 * 1024))
kldload contigmem.ko
}
function reset_freebsd {
kldunload contigmem.ko || true
kldunload nic_uio.ko || true
}
: ${NRHUGE:=1024}
username=$1
mode=$2
if [ "$username" = "reset" -o "$username" = "config" -o "$username" = "status" ]; then
mode="$username"
username=""
fi
if [ "$mode" == "" ]; then
mode="config"
fi
if [ "$username" = "" ]; then
username="$SUDO_USER"
if [ "$username" = "" ]; then
username=`logname 2>/dev/null` || true
fi
fi
if [ `uname` = Linux ]; then
if [ "$mode" == "config" ]; then
configure_linux
elif [ "$mode" == "reset" ]; then
reset_linux
elif [ "$mode" == "status" ]; then
status_linux
fi
else
if [ "$mode" == "config" ]; then
configure_freebsd
elif [ "$mode" == "reset" ]; then
reset_freebsd
fi
fi
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-29 18:00 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-29 18:00 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6595 bytes --]
So to clarify, you have your system booted with no NVMe endpoint connected, and then when you run the SPDK setup.sh script, you see all of these kernel messages from trying to bind vfio to PCIe devices and system eventually crashes?
If so, we need to determine what PCIe devices setup.sh is trying to bind to vfio. It should only be trying to bind NVMe devices but if there is no NVMe device connected then it shouldn’t be trying to bind anything.
Can you send lspci –vvvx output from your system before running setup.sh?
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <oza.oza(a)broadcom.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, August 29, 2017 at 9:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] PCI hotplug and SPDK
In my opinion, this has nothing to do with platform. Though our platform is ARMv8.
(but, I can not test on any other, because we don’t know how the kernel driver is written)
If kernel driver supports hotplug, which means they are allowing pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls. (10-15 seconds) followed by crash.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Harris, James R
Sent: Tuesday, August 29, 2017 6:20 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Oza Oza <oza.oza(a)broadcom.com<mailto:oza.oza(a)broadcom.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
> PCI hotplug feature is implemented in kernel driver and working fine.
> But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
> since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21505 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-29 16:45 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-29 16:45 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5800 bytes --]
In my opinion, this has nothing to do with platform. Though our platform is
ARMv8.
(but, I can not test on any other, because we don’t know how the kernel
driver is written)
If kernel driver supports hotplug, which means they are allowing
pci_create_root_bus irrespective of whether EP is plugged or not.
In other words. Following APIs are never called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
and in that case, if PCIe slots is empty, running SPDK resulting in stalls.
(10-15 seconds) followed by crash.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Tuesday, August 29, 2017 6:20 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on
amd64?
-Jim
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <
oza.oza(a)broadcom.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, August 29, 2017 at 1:51 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API
call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 15604 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-29 12:50 Harris, James R
0 siblings, 0 replies; 21+ messages in thread
From: Harris, James R @ 2017-08-29 12:50 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4828 bytes --]
Hi Oza,
Do you see this issue only on your armv8 platform or do you also see it on amd64?
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Oza Oza <oza.oza(a)broadcom.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, August 29, 2017 at 1:51 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] PCI hotplug and SPDK
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls completely.
The reason is: kernel driver will not remove the root bus (when PCIe endpoint is not connected, during boot-time)
So SPDK tries to bind driver thinking host bridge is there.
Without PCI hotplug host bridge will not be there because of following API call in kernel driver.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus) irrespective of EP is connected or not.
And then if I run SPDK (with no Endpoint connected/Empty slot) I get stalls.
Regards,
Oza.
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Chang, Cunyin
Sent: Tuesday, August 29, 2017 2:14 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 17790 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-29 8:51 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-29 8:51 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4603 bytes --]
Sorry If I was unclear.
I am not talking about hotplug feature of SPDK.
Ø PCI hotplug feature is implemented in kernel driver and working fine.
Ø But the moment I run SPDK and try to bind vfio driver it stalls
completely.
o The reason is: kernel driver will not remove the root bus (when PCIe
endpoint is not connected, during boot-time)
o So SPDK tries to bind driver thinking host bridge is there.
§ Without PCI hotplug host bridge will not be there because of following
API call in kernel driver.
· pci_stop_root_bus(bus);
· pci_remove_root_bus(bus);
Ø since now we allow host bridge creation (API: pci_create_root_bus)
irrespective of EP is connected or not.
o And then if I run SPDK (with no Endpoint connected/Empty slot) I get
stalls.
Regards,
Oza.
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Chang, Cunyin
*Sent:* Tuesday, August 29, 2017 2:14 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] PCI hotplug and SPDK
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or
vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Oza Oza
*Sent:* Tuesday, August 29, 2017 4:22 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13462 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SPDK] PCI hotplug and SPDK
@ 2017-08-29 8:43 Chang, Cunyin
0 siblings, 0 replies; 21+ messages in thread
From: Chang, Cunyin @ 2017-08-29 8:43 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3362 bytes --]
Hi Oza,
Could you please provide some details steps to reproduce the issue?
SPDK take in charge for hotplug only after you bind the device to uio or vfio driver,
so for the new insert deivce, it will handled by kernel driver first.
-Cunyin
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Oza Oza
Sent: Tuesday, August 29, 2017 4:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] PCI hotplug and SPDK
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages; /usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[ 34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002 -- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted 4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K) (DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate: 20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8625 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* [SPDK] PCI hotplug and SPDK
@ 2017-08-29 8:21 Oza Oza
0 siblings, 0 replies; 21+ messages in thread
From: Oza Oza @ 2017-08-29 8:21 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]
Hi All,
PCI hotplug support; requires creation of root bus and probe to go ahead
with all PCIe configuration.
Which means following APIs ae not called.
pci_stop_root_bus(bus);
pci_remove_root_bus(bus);
And then If I run SPDK, It makes system crash with following info.
Note: if the disk is connected then SPDK is fine.
Otherwise it stalls the system with following crash.
root(a)bcm958742k:~# echo 2048 > /proc/sys/vm/nr_hugepages;
/usr/share/spdk/scripts/setup.sh
grep: /usr/share/spdk/scripts/../include/spdk/pci_ids.h: No such[
34.621325] pci 0008:00:00.0: PCI bridge to [bus 01]
file or directory
[ 34.640586] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 50.267056] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 50.272337] pci 0001:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 65.898762] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 65.904015] pci 0006:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 81.530437] pci 0006:00:00.0: PCI bridge to [bus 01]
[ 81.535680] pci 0007:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[ 97.162103] pci 0007:00:00.0: PCI bridge to [bus 01]
[ 97.167255] Bad mode in Error handler detected on CPU6, code 0xbf000002
-- SError
[ 97.174974] Internal error: Oops - bad mode: 0 [#1] SMP
[ 97.180364] Modules linked in:
[ 97.183515] CPU: 6 PID: 2104 Comm: bash Not tainted
4.12.0-01560-gc83093d-dirty #89
[ 97.191413] Hardware name: Stingray Combo SVK w/PCIe IOMMU (BCM958742K)
(DT)
[ 97.198683] task: ffff80a163a40000 task.stack: ffff80a1612b4000
[ 97.204790] PC is at 0xffff7cbdfba8
[ 97.208387] LR is at 0xffff7cb8f288
[ 97.211983] pc : [<0000ffff7cbdfba8>] lr : [<0000ffff7cb8f288>] pstate:
20000000
[ 97.219612] sp : 0000fffffe564040
[ 97.223029] x29: 0000fffffe564040 x28: 000000001054ce60
[ 97.228509] x27: 0000000000000000 x26: 00000000004e2000
[ 97.233989] x25: 00000000004e5000 x24: 0000000000000002
[ 97.239468] x23: 0000ffff7cc63638 x22: 0000000000000002
[ 97.244947] x21: 0000ffff7cc67480 x20: 000000001054db10
[ 97.250427] x19: 0000000000000002 x18: 0000000000000000
[ 97.255906] x17: 00000000004daac8 x16: 0000000000000000
[ 97.261386] x15: 0000000000000096 x14: 0000000000000000
[ 97.266865] x13: 0000000000000000 x12: 0000000000000000
[ 97.272344] x11: 0000000000000020 x10: 0101010101010101
[ 97.277824] x9 : ffffff80ffffffc8 x8 : 0000000000000040
[ 97.283303] x7 : 0000000000000001 x6 : 0000ffff7cc669f0
[ 97.288782] x5 : 0000000000015551 x4 : 0000000000000888
[ 97.294261] x3 : 0000000000000000 x2 : 0000000000000002
[ 97.299741] x1 : 000000001054db10 x0 : 0000000000000002
[ 97.305220] Process bash (pid: 2104, stack limit = 0xffff80a1612b4000)
[ 97.311960] ---[ end trace a1f48abe30820241 ]---
Regards,
Oza.
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5320 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2017-08-30 19:55 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-30 15:30 [SPDK] PCI hotplug and SPDK Harris, James R
-- strict thread matches above, loose matches on Subject: below --
2017-08-30 19:55 Luse, Paul E
2017-08-30 19:43 Oza Oza
2017-08-30 19:31 Luse, Paul E
2017-08-30 19:22 Oza Oza
2017-08-30 19:07 Oza Pawandeep
2017-08-30 19:03 Harris, James R
2017-08-30 18:58 Harris, James R
2017-08-30 18:51 Oza Oza
2017-08-30 18:48 Oza Oza
2017-08-30 18:39 Harris, James R
2017-08-30 18:17 Oza Pawandeep
2017-08-30 15:55 Walker, Benjamin
2017-08-30 15:20 Oza Oza
2017-08-30 15:10 Oza Oza
2017-08-29 18:00 Harris, James R
2017-08-29 16:45 Oza Oza
2017-08-29 12:50 Harris, James R
2017-08-29 8:51 Oza Oza
2017-08-29 8:43 Chang, Cunyin
2017-08-29 8:21 Oza Oza
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.