* Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux @ 2020-03-06 14:32 Luís Mendes 2020-03-06 21:47 ` Bjorn Helgaas 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-06 14:32 UTC (permalink / raw) To: Linux PCI [-- Attachment #1: Type: text/plain, Size: 496 bytes --] Hi, I'm trying to use Google/Coral TPU Edge modules for a project, on arm64 and armhf, but BAR0 doesn't get assigned during the enumeration of PCIe devices and consequently pci_enable_device(...) fails on BAR0 resource with value -22 (EINVAL) (resource has null parent) when loading gasket/apex driver. I'm also trying to adapt gasket/apex to run on armhf, but anyhow that is not the root cause for this issue. Relevant Log extracts follow in attachment. Regards, Luís Mendes [-- Attachment #2: Coral_TPU_armhf_enum.txt --] [-- Type: text/plain, Size: 11171 bytes --] [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low [ 6.993528] hub 4-1:1.0: 4 ports detected [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low [ 7.106741] sdb: sdb1 [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00 [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff] [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff] [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff] [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400 [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] [ 7.293971] PCI: bus0: Fast back to back transfers disabled [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) [ 7.383442] PCI: bus1: Fast back to back transfers disabled [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff] [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref] [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01] [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff] [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142) [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned. [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned. [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed [ 11.227825] apex 0000:01:00.0: error enabling PCI device 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode]) Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@1,0 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000f000-00000fff [empty] Memory behind bridge: e8000000-e81fffff [size=2M] Prefetchable memory behind bridge: 00000000-000fffff [size=1M] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- [virtual] Expansion ROM at e8200000 [disabled] [size=2K] BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x1 (ok) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 0.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode]) Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@2,0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 00010000-00010fff [size=4K] Memory behind bridge: d0000000-e7ffffff [size=384M] Prefetchable memory behind bridge: 00000000-000fffff [size=1M] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- [virtual] Expansion ROM at e8300000 [disabled] [size=2K] BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 0.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- 01:00.0 Non-VGA unclassified device: Device 1ac1:089a (prog-if ff) Subsystem: Device 1ac1:089a Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 48 Region 0: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=16K] Region 2: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1M] Capabilities: [80] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x1 (ok) TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [d0] MSI-X: Enable- Count=128 Masked- Vector table: BAR=2 offset=00046800 PBA: BAR=2 offset=00046068 Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [f8] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?> Capabilities: [108 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [110 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=10us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [200 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Kernel modules: apex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-06 14:32 Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux Luís Mendes @ 2020-03-06 21:47 ` Bjorn Helgaas 2020-03-07 12:11 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-03-06 21:47 UTC (permalink / raw) To: Luís Mendes Cc: linux-pci, Thomas Petazzoni, Jason Cooper, Nicholas Johnson, Benjamin Herrenschmidt [+cc Thomas, Jason, Nicholas, Ben] On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote: > Hi, > > I'm trying to use Google/Coral TPU Edge modules for a project, on > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration > of PCIe devices and consequently pci_enable_device(...) fails on BAR0 > resource with value -22 (EINVAL) (resource has null parent) when > loading gasket/apex driver. > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that > is not the root cause for this issue. > > Relevant Log extracts follow in attachment. Hi Luís, Thanks for the report, and sorry for the problem you're tripping over. I cc'd a few folks who might be interested. > [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low > [ 6.993528] hub 4-1:1.0: 4 ports detected > [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low > [ 7.106741] sdb: sdb1 > [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk > [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00 > [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff] > [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff] > [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff] > [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400 > [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] > [ 7.293971] PCI: bus0: Fast back to back transfers disabled > [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff > [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] > [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd > [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] > [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) > [ 7.383442] PCI: bus1: Fast back to back transfers disabled > [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 > [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring > [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff] > [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref] > [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01] > [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff] > [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142) > > > [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned. > [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned. > [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed > [ 11.227825] apex 0000:01:00.0: error enabling PCI device It looks like we did assign space for the bridge window leading to 01:00.0, but failed to assign space to the 01:00.0 BAR itself. I don't know offhand why that would be. Can you put the entire dmesg log somewhere we can see? That will tell us what kernel you're using and possibly other useful things. Does the same problem happen with other devices, or does it only happen with the gasket/apex combination? There shouldn't be anything device-specific in the PCI core resource assignment code. > 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode]) > Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@1,0 > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 > I/O behind bridge: 0000f000-00000fff [empty] > Memory behind bridge: e8000000-e81fffff [size=2M] > Prefetchable memory behind bridge: 00000000-000fffff [size=1M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- > [virtual] Expansion ROM at e8200000 [disabled] [size=2K] > BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0 > ExtTag- RBE+ > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited > ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 5GT/s (ok), Width x1 (ok) > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl- > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState- > RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- > RootCap: CRSVisible- > RootSta: PME ReqID 0000, PMEStatus- PMEPending- > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd- > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: ReqEn- EgressBlck- > LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- > EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- > > 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode]) > Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@2,0 > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 > I/O behind bridge: 00010000-00010fff [size=4K] > Memory behind bridge: d0000000-e7ffffff [size=384M] > Prefetchable memory behind bridge: 00000000-000fffff [size=1M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- > [virtual] Expansion ROM at e8300000 [disabled] [size=2K] > BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0 > ExtTag- RBE+ > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited > ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok) > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl- > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState- > RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- > RootCap: CRSVisible- > RootSta: PME ReqID 0000, PMEStatus- PMEPending- > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd- > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: ReqEn- EgressBlck- > LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- > EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- > > 01:00.0 Non-VGA unclassified device: Device 1ac1:089a (prog-if ff) > Subsystem: Device 1ac1:089a > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Interrupt: pin A routed to IRQ 48 > Region 0: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=16K] > Region 2: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1M] > Capabilities: [80] Express (v2) Endpoint, MSI 00 > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us > ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us > ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 5GT/s (ok), Width x1 (ok) > TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported > AtomicOpsCap: 32bit- 64bit- 128bitCAS- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled > AtomicOpsCtl: ReqEn- > LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- > EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- > Capabilities: [d0] MSI-X: Enable- Count=128 Masked- > Vector table: BAR=2 offset=00046800 > PBA: BAR=2 offset=00046068 > Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+ > Address: 0000000000000000 Data: 0000 > Capabilities: [f8] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?> > Capabilities: [108 v1] Latency Tolerance Reporting > Max snoop latency: 0ns > Max no snoop latency: 0ns > Capabilities: [110 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ > PortCommonModeRestoreTime=10us PortTPowerOnTime=10us > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > T_CommonMode=0us LTR1.2_Threshold=0ns > L1SubCtl2: T_PwrOn=10us > Capabilities: [200 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > HeaderLog: 00000000 00000000 00000000 00000000 > Kernel modules: apex > > > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-06 21:47 ` Bjorn Helgaas @ 2020-03-07 12:11 ` Luís Mendes 2020-03-07 15:26 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-07 12:11 UTC (permalink / raw) To: Bjorn Helgaas Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson, Benjamin Herrenschmidt Hi Bjorn, Thanks for your help. On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > [+cc Thomas, Jason, Nicholas, Ben] > > On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote: > > Hi, > > > > I'm trying to use Google/Coral TPU Edge modules for a project, on > > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration > > of PCIe devices and consequently pci_enable_device(...) fails on BAR0 > > resource with value -22 (EINVAL) (resource has null parent) when > > loading gasket/apex driver. > > > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that > > is not the root cause for this issue. > > > > Relevant Log extracts follow in attachment. > > Hi Luís, > > Thanks for the report, and sorry for the problem you're tripping over. > I cc'd a few folks who might be interested. > > > [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low > > [ 6.993528] hub 4-1:1.0: 4 ports detected > > [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low > > [ 7.106741] sdb: sdb1 > > [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk > > [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00 > > [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff] > > [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff] > > [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff] > > [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400 > > [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] > > [ 7.293971] PCI: bus0: Fast back to back transfers disabled > > [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > > [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff > > [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] > > [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd > > [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] > > [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) > > [ 7.383442] PCI: bus1: Fast back to back transfers disabled > > [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 > > [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring > > [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff] > > [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref] > > [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01] > > [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff] > > [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142) > > > > > > [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned. > > [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned. > > [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed > > [ 11.227825] apex 0000:01:00.0: error enabling PCI device > > It looks like we did assign space for the bridge window leading to > 01:00.0, but failed to assign space to the 01:00.0 BAR itself. > > I don't know offhand why that would be. Can you put the entire dmesg > log somewhere we can see? That will tell us what kernel you're using > and possibly other useful things. Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/ This is a custom built machine based on the Armada 388 armhf SoC, that I am using, but I've also tried an arm64 machine from Toradex, an Apalis IMX8QM with the Apalis Eval board, producing similar results with different kernels and also different ARM architectures. > > Does the same problem happen with other devices, or does it only > happen with the gasket/apex combination? There shouldn't be anything > device-specific in the PCI core resource assignment code. This issue seems to happen only with the Coral Edge TPU device, but it happens independently of whether the gasket/apex driver module is loaded or not. The BAR 0 of the Coral device is not assigned either way. Luís ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-07 12:11 ` Luís Mendes @ 2020-03-07 15:26 ` Luís Mendes 2020-03-07 21:38 ` Bjorn Helgaas 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-07 15:26 UTC (permalink / raw) To: Bjorn Helgaas Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson, Benjamin Herrenschmidt Hi, A quick look at the logs, makes me wonder if BAR0-BAR5 are only being assigned to IO space on device by Linux, and BAR6 is the first bar index that Linux is assigning on armhf/arm64 for mem space. If so, that would be wrong because registers 0x10 and and 0x18 are BAR0 and BAR2. The TP-Link Gigabit LAN card that is installed on the other PCIe slot has BAR 0 enabled but it is IO space and according to the registers for the mem space, in that device, seen in dmesg, they are regs 0x18 and 0x20, or, BAR 2 and BAR 4, but Linux is assigning them to have indices BAR 6 and BAR 8, since they are MEM space devices. That looks wrong.... maybe the TP-Link device is working just because BAR 0 does happen to exist in this case, which happens to be the minimal requirement that allows pci_enable_device(...) to work, passing in the test 'if (!r->parent)' performed by pci_enable_resources(...) at drivers/pci/setup-res.c. Since most PCIe cards have IO space, this generally works. Can it be? Luís On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Bjorn, > > Thanks for your help. > > On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > [+cc Thomas, Jason, Nicholas, Ben] > > > > On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote: > > > Hi, > > > > > > I'm trying to use Google/Coral TPU Edge modules for a project, on > > > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration > > > of PCIe devices and consequently pci_enable_device(...) fails on BAR0 > > > resource with value -22 (EINVAL) (resource has null parent) when > > > loading gasket/apex driver. > > > > > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that > > > is not the root cause for this issue. > > > > > > Relevant Log extracts follow in attachment. > > > > Hi Luís, > > > > Thanks for the report, and sorry for the problem you're tripping over. > > I cc'd a few folks who might be interested. > > > > > [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low > > > [ 6.993528] hub 4-1:1.0: 4 ports detected > > > [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low > > > [ 7.106741] sdb: sdb1 > > > [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk > > > [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00 > > > [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff] > > > [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff] > > > [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff] > > > [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400 > > > [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] > > > [ 7.293971] PCI: bus0: Fast back to back transfers disabled > > > [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > > > [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff > > > [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] > > > [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd > > > [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] > > > [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) > > > [ 7.383442] PCI: bus1: Fast back to back transfers disabled > > > [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 > > > [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring > > > [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff] > > > [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref] > > > [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01] > > > [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff] > > > [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142) > > > > > > > > > [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned. > > > [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned. > > > [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed > > > [ 11.227825] apex 0000:01:00.0: error enabling PCI device > > > > It looks like we did assign space for the bridge window leading to > > 01:00.0, but failed to assign space to the 01:00.0 BAR itself. > > > > I don't know offhand why that would be. Can you put the entire dmesg > > log somewhere we can see? That will tell us what kernel you're using > > and possibly other useful things. > > Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/ > This is a custom built machine based on the Armada 388 armhf SoC, that > I am using, but I've also tried an arm64 machine from Toradex, an > Apalis IMX8QM with the Apalis Eval board, producing similar results > with different kernels and also different ARM architectures. > > > > Does the same problem happen with other devices, or does it only > > happen with the gasket/apex combination? There shouldn't be anything > > device-specific in the PCI core resource assignment code. > > This issue seems to happen only with the Coral Edge TPU device, but it > happens independently of whether the gasket/apex driver module is > loaded or not. The BAR 0 of the Coral device is not assigned either > way. > > Luís ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-07 15:26 ` Luís Mendes @ 2020-03-07 21:38 ` Bjorn Helgaas 2020-03-08 5:51 ` Nicholas Johnson 0 siblings, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-03-07 21:38 UTC (permalink / raw) To: Luís Mendes Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson, Benjamin Herrenschmidt On Sat, Mar 07, 2020 at 03:26:49PM +0000, Luís Mendes wrote: > Hi, > > A quick look at the logs, makes me wonder if BAR0-BAR5 are only being > assigned to IO space on device by Linux, and BAR6 is the first bar > index that Linux is assigning on armhf/arm64 for mem space. If so, > that would be wrong because registers 0x10 and and 0x18 are BAR0 and > BAR2. BAR0-5 are the standard BARs and can be either mem or I/O space. 64-bit mem BARs take two slots, e.g., pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit] pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] These are BAR2 (at 0x18 and 0x1c) and BAR4 (at 0x20 and 0x24). > The TP-Link Gigabit LAN card that is installed on the other PCIe slot > has BAR 0 enabled but it is IO space and according to the registers > for the mem space, in that device, seen in dmesg, they are regs 0x18 > and 0x20, or, BAR 2 and BAR 4, but Linux is assigning them to have > indices BAR 6 and BAR 8, since they are MEM space devices. Our logging is a little screwed up: sometimes we print "reg 0x10", other times "BAR 0": pci 0000:02:00.0: reg 0x10: [io 0x0000-0x00ff] pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit] pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] pci 0000:02:00.0: BAR 4: assigned [mem 0xd0200000-0xd0203fff 64bit pref] pci 0000:02:00.0: BAR 2: assigned [mem 0xd0204000-0xd0204fff 64bit] pci 0000:02:00.0: BAR 0: assigned [io 0x10000-0x100ff] Anyway, we end up with this, which looks fine: BAR0 (reg 0x10): [io 0x10000-0x100ff] BAR2 (reg 0x18, 0x1c): [mem 0xd0204000-0xd0204fff 64bit] BAR4 (reg 0x20, 0x24): [mem 0xd0200000-0xd0203fff 64bit pref] "BAR 6" is for option ROMs. You have bridges with option ROMS (seems sort of unusual, but legal), and we assign space for them: pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] pci 0000:00:01.0: BAR 6: assigned [mem 0xd0300000-0xd03007ff pref] pci 0000:00:01.0: BAR 8: assigned [mem 0xd0000000-0xd01fffff] pci 0000:00:01.0: PCI bridge to [bus 01] pci 0000:00:01.0: bridge window [mem 0xd0000000-0xd01fffff] pci 0000:00:02.0: reg 0x38: [mem 0x00000000-0x000007ff pref] pci 0000:00:02.0: BAR 6: assigned [mem 0xd0400000-0xd04007ff pref] pci 0000:00:02.0: BAR 8: assigned [mem 0xd0200000-0xd02fffff] pci 0000:00:02.0: PCI bridge to [bus 02] pci 0000:00:02.0: bridge window [io 0x10000-0x10fff] pci 0000:00:02.0: bridge window [mem 0xd0200000-0xd02fffff] You don't have CONFIG_PCI_IOV enabled (see the enum in <linux/pci.h>), so "BAR7" is is the bridge I/O window, "BAR8" is the MMIO window, and "BAR9" is the prefetchable MMIO window. The problematic device needs 16KB + 1MB of prefetchable memory space: pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] The bridge leading to it has a 2MB non-prefetchable window: pci 0000:00:01.0: PCI bridge to [bus 01] pci 0000:00:01.0: bridge window [mem 0xd0000000-0xd01fffff] That *should* work -- it's not prefetchable, but if we don't have prefetchable space, non-prefetchable space should also work (with poor performance, of course). But maybe the fallback from prefetchable to non-prefetchable space is broken or something. > That looks wrong.... maybe the TP-Link device is working just because > BAR 0 does happen to exist in this case, which happens to be the > minimal requirement that allows pci_enable_device(...) to work, > passing in the test 'if (!r->parent)' performed by > pci_enable_resources(...) at drivers/pci/setup-res.c. Since most PCIe > cards have IO space, this generally works. > Can it be? > > Luís > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > Hi Bjorn, > > > > Thanks for your help. > > > > On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > > > [+cc Thomas, Jason, Nicholas, Ben] > > > > > > On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote: > > > > Hi, > > > > > > > > I'm trying to use Google/Coral TPU Edge modules for a project, on > > > > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration > > > > of PCIe devices and consequently pci_enable_device(...) fails on BAR0 > > > > resource with value -22 (EINVAL) (resource has null parent) when > > > > loading gasket/apex driver. > > > > > > > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that > > > > is not the root cause for this issue. > > > > > > > > Relevant Log extracts follow in attachment. > > > > > > Hi Luís, > > > > > > Thanks for the report, and sorry for the problem you're tripping over. > > > I cc'd a few folks who might be interested. > > > > > > > [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low > > > > [ 6.993528] hub 4-1:1.0: 4 ports detected > > > > [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low > > > > [ 7.106741] sdb: sdb1 > > > > [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk > > > > [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00 > > > > [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff] > > > > [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff] > > > > [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff] > > > > [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400 > > > > [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref] > > > > [ 7.293971] PCI: bus0: Fast back to back transfers disabled > > > > [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > > > > [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff > > > > [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] > > > > [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd > > > > [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] > > > > [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) > > > > [ 7.383442] PCI: bus1: Fast back to back transfers disabled > > > > [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 > > > > [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring > > > > [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff] > > > > [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref] > > > > [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01] > > > > [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff] > > > > [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142) > > > > > > > > > > > > [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned. > > > > [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned. > > > > [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed > > > > [ 11.227825] apex 0000:01:00.0: error enabling PCI device > > > > > > It looks like we did assign space for the bridge window leading to > > > 01:00.0, but failed to assign space to the 01:00.0 BAR itself. > > > > > > I don't know offhand why that would be. Can you put the entire dmesg > > > log somewhere we can see? That will tell us what kernel you're using > > > and possibly other useful things. > > > > Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/ > > This is a custom built machine based on the Armada 388 armhf SoC, that > > I am using, but I've also tried an arm64 machine from Toradex, an > > Apalis IMX8QM with the Apalis Eval board, producing similar results > > with different kernels and also different ARM architectures. > > > > > > Does the same problem happen with other devices, or does it only > > > happen with the gasket/apex combination? There shouldn't be anything > > > device-specific in the PCI core resource assignment code. > > > > This issue seems to happen only with the Coral Edge TPU device, but it > > happens independently of whether the gasket/apex driver module is > > loaded or not. The BAR 0 of the Coral device is not assigned either > > way. > > > > Luís ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-07 21:38 ` Bjorn Helgaas @ 2020-03-08 5:51 ` Nicholas Johnson 2020-03-09 11:21 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Nicholas Johnson @ 2020-03-08 5:51 UTC (permalink / raw) To: Bjorn Helgaas Cc: Luís Mendes, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi, > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > This issue seems to happen only with the Coral Edge TPU device, but it > > > happens independently of whether the gasket/apex driver module is > > > loaded or not. The BAR 0 of the Coral device is not assigned either > > > way. > > > > > > Luís So the problem only occurs with the Coral Edge TPU device, so there is a possibility that it is not a problem with the platform, or something caused by the combination of the TPU and platform. Is it possible to put the TPU into an X86 system with the same kernel version(s) to add more evidence to this theory? If it works on X86 then we can focus on the differences between X86 and ARM. Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO window assignment" or try with Linux v5.4 which does not have this commit, just to rule out the possibility of it causing issues. This patch helps me and also solved the problem of one other person using an ARM computer who came to us regarding a problem. However, it could also adversely affect unknown use cases - it is impossible to completely rule out, due to the nature of how drivers/pci/setup-bus.c is written. Kind regards, Nicholas ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-08 5:51 ` Nicholas Johnson @ 2020-03-09 11:21 ` Luís Mendes 2020-03-11 14:20 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-09 11:21 UTC (permalink / raw) To: Nicholas Johnson Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Nicholas, Thanks for your help. Replies follow below. Kind Regards, Luís On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au> wrote: > > Hi, > > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > This issue seems to happen only with the Coral Edge TPU device, but it > > > > happens independently of whether the gasket/apex driver module is > > > > loaded or not. The BAR 0 of the Coral device is not assigned either > > > > way. > > > > > > > > Luís > > So the problem only occurs with the Coral Edge TPU device, so there is a > possibility that it is not a problem with the platform, or something > caused by the combination of the TPU and platform. Is it possible to put > the TPU into an X86 system with the same kernel version(s) to add more > evidence to this theory? If it works on X86 then we can focus on the > differences between X86 and ARM. I've tested two Coral TPUs on two x86_64 platforms and the BARs seem to be assigned, however the driver fails to load, during probe and the system is unable to shutdown cleanly, but I think that is a driver issue when setting up sysfs entries. I can blacklist gasket/apex, if it helps in some way, or try an older kernel. Dmesg log for one of the x86_64 system is here: https://pastebin.ubuntu.com/p/FHhHNN6XTF/ lspci -vvv for the same x86_64 system is here: https://pastebin.ubuntu.com/p/xbSNWFQ9TS/ > > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO > window assignment" or try with Linux v5.4 which does not have this > commit, just to rule out the possibility of it causing issues. This > patch helps me and also solved the problem of one other person using an > ARM computer who came to us regarding a problem. However, it could also > adversely affect unknown use cases - it is impossible to completely rule > out, due to the nature of how drivers/pci/setup-bus.c is written. On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/ I've also tried kernel 4.20.11 and the problem also exists. I've got JTAG on this system with OpenOCD. I believe I can debug the kernel, if needed. > > Kind regards, > Nicholas ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-09 11:21 ` Luís Mendes @ 2020-03-11 14:20 ` Luís Mendes 2020-03-29 22:11 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-11 14:20 UTC (permalink / raw) To: Nicholas Johnson Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Re-sending... after 48 hours the previous message still didn't hit the mailing list... On Mon, Mar 9, 2020 at 11:21 AM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Nicholas, > > Thanks for your help. > Replies follow below. > > Kind Regards, > Luís > > On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson > <nicholas.johnson-opensource@outlook.com.au> wrote: > > > > Hi, > > > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > > This issue seems to happen only with the Coral Edge TPU device, but it > > > > > happens independently of whether the gasket/apex driver module is > > > > > loaded or not. The BAR 0 of the Coral device is not assigned either > > > > > way. > > > > > > > > > > Luís > > > > So the problem only occurs with the Coral Edge TPU device, so there is a > > possibility that it is not a problem with the platform, or something > > caused by the combination of the TPU and platform. Is it possible to put > > the TPU into an X86 system with the same kernel version(s) to add more > > evidence to this theory? If it works on X86 then we can focus on the > > differences between X86 and ARM. > > I've tested two Coral TPUs on two x86_64 platforms and the BARs seem > to be assigned, however the driver fails to load, during probe and the > system is unable to shutdown cleanly, but I think that is a driver > issue when setting up sysfs entries. I can blacklist gasket/apex, if > it helps in some way, or try an older kernel. > Dmesg log for one of the x86_64 system is here: > https://pastebin.ubuntu.com/p/FHhHNN6XTF/ > lspci -vvv for the same x86_64 system is here: > https://pastebin.ubuntu.com/p/xbSNWFQ9TS/ > > > > > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO > > window assignment" or try with Linux v5.4 which does not have this > > commit, just to rule out the possibility of it causing issues. This > > patch helps me and also solved the problem of one other person using an > > ARM computer who came to us regarding a problem. However, it could also > > adversely affect unknown use cases - it is impossible to completely rule > > out, due to the nature of how drivers/pci/setup-bus.c is written. > > On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not > assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/ > I've also tried kernel 4.20.11 and the problem also exists. > I've got JTAG on this system with OpenOCD. I believe I can debug the > kernel, if needed. > > > > > Kind regards, > > Nicholas ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-11 14:20 ` Luís Mendes @ 2020-03-29 22:11 ` Luís Mendes 2020-03-30 19:49 ` Bjorn Helgaas 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-29 22:11 UTC (permalink / raw) To: Nicholas Johnson Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Nicholas, Bjorn, I was able to make the apex driver work on a X86_64 system with the Coral Edge TPU PCIe device. So, now the PCI enumeration problem is now clearly an ARM and ARM64 platform issue. What are the recommended steps for debugging this? I hava a JTAG interface and openOCD supported configuration for it. On X86_64 the PCI device did enumerate properly, but the driver would fail to load due to a bug. That is now fixed and it did run a couple of examples, on X86_64 only, after applying a patch that I submitted to the Gasket driver maintainers: Fix incongruency in handling of sysfs entries creation. This issue could cause invalid memory accesses, by not properly detecting the end of the sysfs attributes array. Signed-off-by: Luis Mendes <luis.p.mendes@gmail.com> --- gasket_sysfs.c | 3 +-- gasket_sysfs.h | 4 ---- 2 files changed, 1 insertion(+), 6 deletions(-) Kind Regards, Luís > On Mon, Mar 9, 2020 at 11:21 AM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > Hi Nicholas, > > > > Thanks for your help. > > Replies follow below. > > > > Kind Regards, > > Luís > > > > On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson > > <nicholas.johnson-opensource@outlook.com.au> wrote: > > > > > > Hi, > > > > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > > > This issue seems to happen only with the Coral Edge TPU device, but it > > > > > > happens independently of whether the gasket/apex driver module is > > > > > > loaded or not. The BAR 0 of the Coral device is not assigned either > > > > > > way. > > > > > > > > > > > > Luís > > > > > > So the problem only occurs with the Coral Edge TPU device, so there is a > > > possibility that it is not a problem with the platform, or something > > > caused by the combination of the TPU and platform. Is it possible to put > > > the TPU into an X86 system with the same kernel version(s) to add more > > > evidence to this theory? If it works on X86 then we can focus on the > > > differences between X86 and ARM. > > > > I've tested two Coral TPUs on two x86_64 platforms and the BARs seem > > to be assigned, however the driver fails to load, during probe and the > > system is unable to shutdown cleanly, but I think that is a driver > > issue when setting up sysfs entries. I can blacklist gasket/apex, if > > it helps in some way, or try an older kernel. > > Dmesg log for one of the x86_64 system is here: > > https://pastebin.ubuntu.com/p/FHhHNN6XTF/ > > lspci -vvv for the same x86_64 system is here: > > https://pastebin.ubuntu.com/p/xbSNWFQ9TS/ > > > > > > > > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO > > > window assignment" or try with Linux v5.4 which does not have this > > > commit, just to rule out the possibility of it causing issues. This > > > patch helps me and also solved the problem of one other person using an > > > ARM computer who came to us regarding a problem. However, it could also > > > adversely affect unknown use cases - it is impossible to completely rule > > > out, due to the nature of how drivers/pci/setup-bus.c is written. > > > > On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not > > assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/ > > I've also tried kernel 4.20.11 and the problem also exists. > > I've got JTAG on this system with OpenOCD. I believe I can debug the > > kernel, if needed. > > > > > > > > Kind regards, > > > Nicholas ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-29 22:11 ` Luís Mendes @ 2020-03-30 19:49 ` Bjorn Helgaas 2020-03-31 21:28 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-03-30 19:49 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt On Sun, Mar 29, 2020 at 11:11:28PM +0100, Luís Mendes wrote: > Hi Nicholas, Bjorn, > > I was able to make the apex driver work on a X86_64 system with the > Coral Edge TPU PCIe device. > So, now the PCI enumeration problem is now clearly an ARM and ARM64 > platform issue. What are the recommended steps for debugging this? I > hava a JTAG interface and openOCD supported configuration for it. Thanks for the work of testing on X86_64. I don't have any magic ideas other than instrumenting the code and slogging through the output. Can you try the patch below and collect the dmesg? This will probably take a few iterations. diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 8e40b3e6da77..2cdb705752de 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -166,6 +166,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, resource_size_t max; type_mask |= IORESOURCE_TYPE_BITS; + pci_info(bus, "%s: %pR type_mask %#lx\n", __func__, res, type_mask); pci_bus_for_each_resource(bus, r, i) { resource_size_t min_used = min; @@ -173,6 +174,9 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, if (!r) continue; + pci_info(bus, "%s: from %pR res %#lx r %#lx\n", __func__, + r, res->flags, r->flags); + /* type_mask must match */ if ((res->flags ^ r->flags) & type_mask) continue; @@ -203,6 +207,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, if (ret == 0) return 0; } + pci_info(bus, "%s: failed\n", __func__); return -ENOMEM; } diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index f2461bf9243d..649aa90b8b29 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -280,8 +280,10 @@ static void assign_requested_resources_sorted(struct list_head *head, list_for_each_entry(dev_res, head, list) { res = dev_res->res; idx = res - &dev_res->dev->resource[0]; + pci_info(dev_res->dev, "%s BAR%d %pR\n", __func__, idx, res); if (resource_size(res) && pci_assign_resource(dev_res->dev, idx)) { + pci_info(dev_res->dev, "%s (failed)\n", __func__); if (fail_head) { /* * If the failed resource is a ROM BAR and @@ -996,6 +998,12 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, resource_size_t children_add_align = 0; resource_size_t add_align = 0; + pci_info(bus, "%s: mask %#lx type %#lx %#lx %#lx min %#llx add %#llx b_res %pR parent %pR\n", + __func__, mask, type, type2, type3, + (unsigned long long) min_size, + (unsigned long long) add_size, + b_res, b_res ? b_res->parent : NULL); + if (!b_res) return -ENOSPC; @@ -1199,6 +1207,8 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) struct resource *b_res; int ret; + pci_info(bus, "%s\n", __func__); + list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *b = dev->subordinate; if (!b) @@ -1311,6 +1321,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) void pci_bus_size_bridges(struct pci_bus *bus) { + pci_info(bus, "%s\n", __func__); __pci_bus_size_bridges(bus, NULL); } EXPORT_SYMBOL(pci_bus_size_bridges); @@ -1394,6 +1405,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, void pci_bus_assign_resources(const struct pci_bus *bus) { + pci_info(bus, "%s\n", __func__); __pci_bus_assign_resources(bus, NULL, NULL); } EXPORT_SYMBOL(pci_bus_assign_resources); @@ -1408,6 +1420,7 @@ static void pci_claim_device_resources(struct pci_dev *dev) if (!r->flags || r->parent) continue; + pci_info(dev, "%s BAR%d %pR\n", __func__, i, r); pci_claim_resource(dev, i); } } @@ -1422,6 +1435,7 @@ static void pci_claim_bridge_resources(struct pci_dev *dev) if (!r->flags || r->parent) continue; + pci_info(dev, "%s BAR%d %pR\n", __func__, i, r); pci_claim_bridge_resource(dev, i); } } @@ -1460,6 +1474,7 @@ static void pci_bus_allocate_resources(struct pci_bus *b) void pci_bus_claim_resources(struct pci_bus *b) { + pci_info(bus, "%s\n", __func__); pci_bus_allocate_resources(b); pci_bus_allocate_dev_resources(b); } ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-30 19:49 ` Bjorn Helgaas @ 2020-03-31 21:28 ` Luís Mendes 2020-04-01 18:16 ` Bjorn Helgaas 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-03-31 21:28 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, Thanks for your help. I've removed all other PCIe devices to make the analysis easier. The dmesg with the traces can be found at: https://paste.ubuntu.com/p/W3m2VQCYqg/ Didn't find anything new related to BAR0 or BAR2, in the dmesg, though. Anyway I'm no expert in this, maybe it can give you some useful information, still. Kind Regards, Luís Mendes On Mon, Mar 30, 2020 at 8:49 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Sun, Mar 29, 2020 at 11:11:28PM +0100, Luís Mendes wrote: > > Hi Nicholas, Bjorn, > > > > I was able to make the apex driver work on a X86_64 system with the > > Coral Edge TPU PCIe device. > > So, now the PCI enumeration problem is now clearly an ARM and ARM64 > > platform issue. What are the recommended steps for debugging this? I > > hava a JTAG interface and openOCD supported configuration for it. > > Thanks for the work of testing on X86_64. I don't have any magic > ideas other than instrumenting the code and slogging through the > output. Can you try the patch below and collect the dmesg? This will > probably take a few iterations. > > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-03-31 21:28 ` Luís Mendes @ 2020-04-01 18:16 ` Bjorn Helgaas 2020-04-01 21:20 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-04-01 18:16 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote: > I've removed all other PCIe devices to make the analysis easier. > The dmesg with the traces can be found at: > https://paste.ubuntu.com/p/W3m2VQCYqg/ > > Didn't find anything new related to BAR0 or BAR2, in the dmesg, > though. Anyway I'm no expert in this, maybe it can give you some > useful information, still. It looks like we assigned the right amount of space to the bridge, but for some reason didn't assign it to the device *below* the bridge. I added a few more messages in this patch. Can you remove the first one and replace it with this? This is still based on v5.6-rc1. diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 8e40b3e6da77..2cdb705752de 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -166,6 +166,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, resource_size_t max; type_mask |= IORESOURCE_TYPE_BITS; + pci_info(bus, "%s: %pR type_mask %#lx\n", __func__, res, type_mask); pci_bus_for_each_resource(bus, r, i) { resource_size_t min_used = min; @@ -173,6 +174,9 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, if (!r) continue; + pci_info(bus, "%s: from %pR res %#lx r %#lx\n", __func__, + r, res->flags, r->flags); + /* type_mask must match */ if ((res->flags ^ r->flags) & type_mask) continue; @@ -203,6 +207,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res, if (ret == 0) return 0; } + pci_info(bus, "%s: failed\n", __func__); return -ENOMEM; } diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index f2461bf9243d..b42f1bcab25f 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -177,6 +177,7 @@ static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) { u16 class = dev->class >> 8; + pci_info(dev, "%s\n", __func__); /* Don't touch classless devices or host bridges or IOAPICs */ if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) return; @@ -280,8 +281,10 @@ static void assign_requested_resources_sorted(struct list_head *head, list_for_each_entry(dev_res, head, list) { res = dev_res->res; idx = res - &dev_res->dev->resource[0]; + pci_info(dev_res->dev, "%s: BAR%d %pR\n", __func__, idx, res); if (resource_size(res) && pci_assign_resource(dev_res->dev, idx)) { + pci_info(dev_res->dev, "%s: failed\n", __func__); if (fail_head) { /* * If the failed resource is a ROM BAR and @@ -372,6 +375,7 @@ static void __assign_resources_sorted(struct list_head *head, unsigned long fail_type; resource_size_t add_align, align; + pr_info("%s\n", __func__); /* Check if optional add_size is there */ if (!realloc_head || list_empty(realloc_head)) goto requested_and_reassign; @@ -386,6 +390,7 @@ static void __assign_resources_sorted(struct list_head *head, /* Update res in head list with add_size in realloc_head list */ list_for_each_entry_safe(dev_res, tmp_res, head, list) { + pci_info(dev_res->dev, "%s: %pR\n", __func__, dev_res->res); dev_res->res->end += get_res_add_size(realloc_head, dev_res->res); @@ -436,6 +441,7 @@ static void __assign_resources_sorted(struct list_head *head, remove_from_list(realloc_head, dev_res->res); free_list(&save_head); free_list(head); + pr_info("%s: success\n", __func__); return; } @@ -483,6 +489,7 @@ static void pdev_assign_resources_sorted(struct pci_dev *dev, { LIST_HEAD(head); + pci_info(dev, "%s\n", __func__); __dev_sort_resources(dev, &head); __assign_resources_sorted(&head, add_head, fail_head); @@ -495,6 +502,7 @@ static void pbus_assign_resources_sorted(const struct pci_bus *bus, struct pci_dev *dev; LIST_HEAD(head); + pci_info(bus, "%s\n", __func__); list_for_each_entry(dev, &bus->devices, bus_list) __dev_sort_resources(dev, &head); @@ -996,6 +1004,12 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, resource_size_t children_add_align = 0; resource_size_t add_align = 0; + pci_info(bus, "%s: mask %#04lx type %#04lx %#04lx %#04lx min %#llx add %#llx b_res %pR parent %pR\n", + __func__, mask, type, type2, type3, + (unsigned long long) min_size, + (unsigned long long) add_size, + b_res, b_res ? b_res->parent : NULL); + if (!b_res) return -ENOSPC; @@ -1089,6 +1103,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, (unsigned long long) (size1 - size0), (unsigned long long) add_align); } + pci_info(bus->self, "%s: %pR\n", __func__, b_res); return 0; } @@ -1199,6 +1214,8 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) struct resource *b_res; int ret; + pci_info(bus, "%s\n", __func__); + list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *b = dev->subordinate; if (!b) @@ -1311,6 +1328,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head) void pci_bus_size_bridges(struct pci_bus *bus) { + pci_info(bus, "%s\n", __func__); __pci_bus_size_bridges(bus, NULL); } EXPORT_SYMBOL(pci_bus_size_bridges); @@ -1340,6 +1358,7 @@ static void pdev_assign_fixed_resources(struct pci_dev *dev) { int i; + pci_info(dev, "%s\n", __func__); for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct pci_bus *b; struct resource *r = &dev->resource[i]; @@ -1363,9 +1382,11 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, struct pci_bus *b; struct pci_dev *dev; + pci_info(bus, "%s\n", __func__); pbus_assign_resources_sorted(bus, realloc_head, fail_head); list_for_each_entry(dev, &bus->devices, bus_list) { + pci_info(dev, "%s\n", __func__); pdev_assign_fixed_resources(dev); b = dev->subordinate; @@ -1394,6 +1415,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus, void pci_bus_assign_resources(const struct pci_bus *bus) { + pci_info(bus, "%s\n", __func__); __pci_bus_assign_resources(bus, NULL, NULL); } EXPORT_SYMBOL(pci_bus_assign_resources); @@ -1408,6 +1430,7 @@ static void pci_claim_device_resources(struct pci_dev *dev) if (!r->flags || r->parent) continue; + pci_info(dev, "%s: BAR%d %pR\n", __func__, i, r); pci_claim_resource(dev, i); } } @@ -1422,6 +1445,7 @@ static void pci_claim_bridge_resources(struct pci_dev *dev) if (!r->flags || r->parent) continue; + pci_info(dev, "%s: BAR%d %pR\n", __func__, i, r); pci_claim_bridge_resource(dev, i); } } @@ -1432,6 +1456,7 @@ static void pci_bus_allocate_dev_resources(struct pci_bus *b) struct pci_bus *child; list_for_each_entry(dev, &b->devices, bus_list) { + pci_info(dev, "%s\n", __func__); pci_claim_device_resources(dev); child = dev->subordinate; @@ -1460,6 +1485,7 @@ static void pci_bus_allocate_resources(struct pci_bus *b) void pci_bus_claim_resources(struct pci_bus *b) { + pci_info(bus, "%s\n", __func__); pci_bus_allocate_resources(b); pci_bus_allocate_dev_resources(b); } @@ -1471,6 +1497,7 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge, { struct pci_bus *b; + pci_info(dev, "%s\n", __func__); pdev_assign_resources_sorted((struct pci_dev *)bridge, add_head, fail_head); ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-01 18:16 ` Bjorn Helgaas @ 2020-04-01 21:20 ` Luís Mendes 2020-04-01 21:55 ` Luís Mendes 2020-04-01 23:31 ` Bjorn Helgaas 0 siblings, 2 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-01 21:20 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, Here is the dmesg output with the new kernel patch: https://paste.ubuntu.com/p/7cv7ZcyrnG/ Luís On Wed, Apr 1, 2020 at 7:16 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote: > > I've removed all other PCIe devices to make the analysis easier. > > The dmesg with the traces can be found at: > > https://paste.ubuntu.com/p/W3m2VQCYqg/ > > > > Didn't find anything new related to BAR0 or BAR2, in the dmesg, > > though. Anyway I'm no expert in this, maybe it can give you some > > useful information, still. > > It looks like we assigned the right amount of space to the bridge, but > for some reason didn't assign it to the device *below* the bridge. > > I added a few more messages in this patch. Can you remove the first > one and replace it with this? This is still based on v5.6-rc1. > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c > index 8e40b3e6da77..2cdb705752de 100644 > --- a/drivers/pci/bus.c > +++ b/drivers/pci/bus.c ... ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-01 21:20 ` Luís Mendes @ 2020-04-01 21:55 ` Luís Mendes 2020-04-01 23:31 ` Bjorn Helgaas 1 sibling, 0 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-01 21:55 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjiorn I don't see any log of __pci_bridge_assign_resources. This platform makes use of pci-bridge-emul.c functions. Don't know if that can be responsible for not calling __pci_bridge_assign_resources(...). Luís On Wed, Apr 1, 2020 at 10:20 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Bjorn, > > Here is the dmesg output with the new kernel patch: > https://paste.ubuntu.com/p/7cv7ZcyrnG/ > > Luís > > On Wed, Apr 1, 2020 at 7:16 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote: > > > I've removed all other PCIe devices to make the analysis easier. > > > The dmesg with the traces can be found at: > > > https://paste.ubuntu.com/p/W3m2VQCYqg/ > > > > > > Didn't find anything new related to BAR0 or BAR2, in the dmesg, > > > though. Anyway I'm no expert in this, maybe it can give you some > > > useful information, still. > > > > It looks like we assigned the right amount of space to the bridge, but > > for some reason didn't assign it to the device *below* the bridge. > > > > I added a few more messages in this patch. Can you remove the first > > one and replace it with this? This is still based on v5.6-rc1. > > > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c > > index 8e40b3e6da77..2cdb705752de 100644 > > --- a/drivers/pci/bus.c > > +++ b/drivers/pci/bus.c > ... ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-01 21:20 ` Luís Mendes 2020-04-01 21:55 ` Luís Mendes @ 2020-04-01 23:31 ` Bjorn Helgaas 2020-04-02 14:13 ` Luís Mendes 1 sibling, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-04-01 23:31 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote: > Hi Bjorn, > > Here is the dmesg output with the new kernel patch: > https://paste.ubuntu.com/p/7cv7ZcyrnG/ Thanks. What happens if you do this: # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan (I think on v5.6 that file is "bus_rescan" instead of "rescan". There's a patch queued up to change it.) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-01 23:31 ` Bjorn Helgaas @ 2020-04-02 14:13 ` Luís Mendes 2020-04-04 1:32 ` Bjorn Helgaas 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-04-02 14:13 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, The command that worked was: # sh -c "echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/pci_bus/0000\:01/bus_rescan" The only new information compared to yesterday dmesg are the following lines: ... [ 20.633566] mvneta f1070000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off [ 20.633598] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 1048.661591] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. [ 1049.792121] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [60026.321476] pci_bus 0000:01: __pci_bus_assign_resources [60026.321481] pci_bus 0000:01: pbus_assign_resources_sorted [60026.321485] pci 0000:01:00.0: __dev_sort_resources [60026.321487] __assign_resources_sorted [60026.321490] pci 0000:01:00.0: __pci_bus_assign_resources [60026.321493] pci 0000:01:00.0: pdev_assign_fixed_resources Luís On Thu, Apr 2, 2020 at 12:31 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote: > > Hi Bjorn, > > > > Here is the dmesg output with the new kernel patch: > > https://paste.ubuntu.com/p/7cv7ZcyrnG/ > > Thanks. What happens if you do this: > > # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan > > (I think on v5.6 that file is "bus_rescan" instead of "rescan". > There's a patch queued up to change it.) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-02 14:13 ` Luís Mendes @ 2020-04-04 1:32 ` Bjorn Helgaas 2020-04-04 21:39 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-04-04 1:32 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt On Thu, Apr 02, 2020 at 03:13:36PM +0100, Luís Mendes wrote: > Hi Bjorn, > > The command that worked was: > # sh -c "echo 1 > > /sys/bus/pci/devices/0000\:00\:01.0/pci_bus/0000\:01/bus_rescan" > > The only new information compared to yesterday dmesg are the following lines: > ... > [ 20.633566] mvneta f1070000.ethernet eth0: Link is Up - 1Gbps/Full > - flow control off > [ 20.633598] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > [ 1048.661591] FAT-fs (sda1): Volume was not properly unmounted. Some > data may be corrupt. Please run fsck. > [ 1049.792121] EXT4-fs (sdb1): mounted filesystem with ordered data > mode. Opts: (null) > [60026.321476] pci_bus 0000:01: __pci_bus_assign_resources > [60026.321481] pci_bus 0000:01: pbus_assign_resources_sorted > [60026.321485] pci 0000:01:00.0: __dev_sort_resources > [60026.321487] __assign_resources_sorted > [60026.321490] pci 0000:01:00.0: __pci_bus_assign_resources > [60026.321493] pci 0000:01:00.0: pdev_assign_fixed_resources The idea I'm pushing on is that maybe Linux don't program endpoint BARs correctly. This theory is weak because I'm pretty sure there are ARM systems where firmware doesn't program the BARs, and some of them seem to work. But resource assignment is arch-specific and even mvebu-specific, so maybe this is a wrinkle. The hotplug case is much less arch-specific than the boot-time case, so I wonder if you'll see something different if you remove the device and then re-enumerate it? I don't think you have a hotplug-capable slot, but we can do something similar via sysfs. I tried to test this on my x86 system (where BARs are assigned by the BIOS at boot-time) by clearing the BAR of an idle device, removing it, and re-enumerating it. Boot-time enumeration said: pci 0000:00:1c.0: PCI bridge to [bus 02] pci 0000:02:00.0: reg 0x14: [mem 0xf1100000-0xf1100fff] Here I cleared the BAR, removed, rescanned: # setpci -s02:00.0 BASE_ADDRESS_1.L=0 # echo 1 > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/remove # echo 1 > /sys/devices/pci0000:00/0000:00:1c.0/rescan and dmesg said: pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000 pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff] pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff] pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff) So we correctly detected the device, read the cleared BAR, and allocated space for it, and tried to update the BAR. On my system the update failed, I think because of a power management issue (all config reads now return 0xffffffff). But there have been a lot of power management fixes since the v5.2 kernel I'm running, so it's possible you'd have better luck. On your system, I think you would want something like: # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan > On Thu, Apr 2, 2020 at 12:31 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote: > > > Hi Bjorn, > > > > > > Here is the dmesg output with the new kernel patch: > > > https://paste.ubuntu.com/p/7cv7ZcyrnG/ > > > > Thanks. What happens if you do this: > > > > # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan > > > > (I think on v5.6 that file is "bus_rescan" instead of "rescan". > > There's a patch queued up to change it.) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-04 1:32 ` Bjorn Helgaas @ 2020-04-04 21:39 ` Luís Mendes 2020-04-08 23:05 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-04-04 21:39 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, Thanks again for your help. Ok... I've tested your theory on this system and still no changes. The BARs 0 and 1 are still not assigned. I should note that this issue does not occur only on this particular armhf system, but also on a Toradex Apalis IMX8QM, which in this case is an arm64 device and doesn't make use of the mvebu infrastructure. So I did issue the following commands: # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/remove # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/dev_rescan" And the dmesg update after the last command is: [ 61.124696] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff [ 61.124732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] [ 61.124743] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] [ 61.161258] pci_bus 0000:01: __pci_bus_size_bridges [ 61.161270] pci_bus 0000:01: pbus_size_mem: mask 0x2200 type 0x2200 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null) [ 61.161277] pci_bus 0000:01: pbus_size_mem: mask 0x200 type 0x200 0x200 0x200 min 0x0 add 0x0 b_res [mem 0xd0000000-0xd01fffff] parent [mem 0xd0000000-0xefffffff] [ 61.161281] pci_bus 0000:02: __pci_bus_size_bridges [ 61.161286] pci_bus 0000:02: pbus_size_mem: mask 0x2200 type 0x2200 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null) [ 61.161291] pci_bus 0000:02: pbus_size_mem: mask 0x200 type 0x200 0x200 0x200 min 0x0 add 0x0 b_res [mem 0x00000000] parent (null) [ 61.161295] pci_bus 0000:00: __pci_bus_assign_resources [ 61.161298] pci_bus 0000:00: pbus_assign_resources_sorted [ 61.161302] pci 0000:00:01.0: __dev_sort_resources [ 61.161305] pci 0000:00:02.0: __dev_sort_resources [ 61.161308] __assign_resources_sorted [ 61.161311] pci 0000:00:01.0: __pci_bus_assign_resources [ 61.161314] pci 0000:00:01.0: pdev_assign_fixed_resources [ 61.161317] pci_bus 0000:01: __pci_bus_assign_resources [ 61.161319] pci_bus 0000:01: pbus_assign_resources_sorted [ 61.161323] pci 0000:01:00.0: __dev_sort_resources [ 61.161324] __assign_resources_sorted [ 61.161327] pci 0000:01:00.0: __pci_bus_assign_resources [ 61.161330] pci 0000:01:00.0: pdev_assign_fixed_resources [ 61.161333] pci 0000:00:01.0: PCI bridge to [bus 01] [ 61.161340] pci 0000:00:01.0: bridge window [mem 0xd0000000-0xd01fffff] [ 61.161344] pci 0000:00:02.0: __pci_bus_assign_resources [ 61.161347] pci 0000:00:02.0: pdev_assign_fixed_resources [ 61.161350] pci_bus 0000:02: __pci_bus_assign_resources [ 61.161353] pci_bus 0000:02: pbus_assign_resources_sorted [ 61.161354] __assign_resources_sorted [ 61.161357] pci 0000:00:02.0: PCI bridge to [bus 02] Luís On Sat, Apr 4, 2020 at 2:32 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000 > pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff] > pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff] > pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff) > > So we correctly detected the device, read the cleared BAR, and > allocated space for it, and tried to update the BAR. On my system the > update failed, I think because of a power management issue (all config > reads now return 0xffffffff). But there have been a lot of power > management fixes since the v5.2 kernel I'm running, so it's possible > you'd have better luck. > > On your system, I think you would want something like: > > # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove > # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-04 21:39 ` Luís Mendes @ 2020-04-08 23:05 ` Luís Mendes 2020-04-09 15:25 ` Luís Mendes 0 siblings, 1 reply; 25+ messages in thread From: Luís Mendes @ 2020-04-08 23:05 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, I have successfully setup a JTAG remote debug environment. And I found this: First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1, which in turn calls __pci_bus_assign_resources which visits device 1ac1:089a on that slot and calls: /* * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they are * skipped by pbus_assign_resources_sorted(). */ static void pdev_assign_fixed_resources(struct pci_dev *dev) { int i; pci_info(dev, "%s\n", __func__); for (i = 0; i < PCI_NUM_RESOURCES; i++) { struct pci_bus *b; struct resource *r = &dev->resource[i]; if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) || !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM))) continue; b = dev->bus; while (b && !r->parent) { assign_fixed_resource_on_bus(b, r); b = b->parent; } } } where dev has the following data before calling pdev_assign_fixed_resources, for some reason BAR0 and BAR2 are skipped or assign_fixed_resource_on_bus is not called : dev->vendorID 6849(0x1ac1) ->deviceID 2202(0x089a) ->resource={ {start = 0 end=16383 name="0000:01:00.0" flags=1319436(0x14220C) desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name=0x0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0, end = 1048575, name="0000:01:00.0" flags=1319436(0x14220C) desc=0, parent=0x0, sibling=0x0 child=0x0}, {start = 0 end=0 name="0000:01:00.0" flags=0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name="0000:01:00.0" flags=0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name="0000:01:00.0" flags=0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name=0x0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name=0x0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name=0x0 desc=0 parent=0x0 sibling=0x0 child=0x0}, {start = 0 end=0 name=0x0 desc=0 parent=0x0 sibling=0x0 child=0x0}} Luís On Sat, Apr 4, 2020 at 10:39 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Bjorn, > > Thanks again for your help. > > Ok... I've tested your theory on this system and still no changes. The > BARs 0 and 1 are still not assigned. I should note that this issue > does not occur only on this particular armhf system, but also on a > Toradex Apalis IMX8QM, which in this case is an arm64 device and > doesn't make use of the mvebu infrastructure. > > So I did issue the following commands: > # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/remove > # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/dev_rescan" > > And the dmesg update after the last command is: > [ 61.124696] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff > [ 61.124732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff > 64bit pref] > [ 61.124743] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff > 64bit pref] > [ 61.161258] pci_bus 0000:01: __pci_bus_size_bridges > [ 61.161270] pci_bus 0000:01: pbus_size_mem: mask 0x2200 type 0x2200 > 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null) > [ 61.161277] pci_bus 0000:01: pbus_size_mem: mask 0x200 type 0x200 > 0x200 0x200 min 0x0 add 0x0 b_res [mem 0xd0000000-0xd01fffff] parent > [mem 0xd0000000-0xefffffff] > [ 61.161281] pci_bus 0000:02: __pci_bus_size_bridges > [ 61.161286] pci_bus 0000:02: pbus_size_mem: mask 0x2200 type 0x2200 > 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null) > [ 61.161291] pci_bus 0000:02: pbus_size_mem: mask 0x200 type 0x200 > 0x200 0x200 min 0x0 add 0x0 b_res [mem 0x00000000] parent (null) > [ 61.161295] pci_bus 0000:00: __pci_bus_assign_resources > [ 61.161298] pci_bus 0000:00: pbus_assign_resources_sorted > [ 61.161302] pci 0000:00:01.0: __dev_sort_resources > [ 61.161305] pci 0000:00:02.0: __dev_sort_resources > [ 61.161308] __assign_resources_sorted > [ 61.161311] pci 0000:00:01.0: __pci_bus_assign_resources > [ 61.161314] pci 0000:00:01.0: pdev_assign_fixed_resources > [ 61.161317] pci_bus 0000:01: __pci_bus_assign_resources > [ 61.161319] pci_bus 0000:01: pbus_assign_resources_sorted > [ 61.161323] pci 0000:01:00.0: __dev_sort_resources > [ 61.161324] __assign_resources_sorted > [ 61.161327] pci 0000:01:00.0: __pci_bus_assign_resources > [ 61.161330] pci 0000:01:00.0: pdev_assign_fixed_resources > [ 61.161333] pci 0000:00:01.0: PCI bridge to [bus 01] > [ 61.161340] pci 0000:00:01.0: bridge window [mem 0xd0000000-0xd01fffff] > [ 61.161344] pci 0000:00:02.0: __pci_bus_assign_resources > [ 61.161347] pci 0000:00:02.0: pdev_assign_fixed_resources > [ 61.161350] pci_bus 0000:02: __pci_bus_assign_resources > [ 61.161353] pci_bus 0000:02: pbus_assign_resources_sorted > [ 61.161354] __assign_resources_sorted > [ 61.161357] pci 0000:00:02.0: PCI bridge to [bus 02] > > Luís > > On Sat, Apr 4, 2020 at 2:32 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000 > > pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff] > > pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff] > > pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff) > > > > So we correctly detected the device, read the cleared BAR, and > > allocated space for it, and tried to update the BAR. On my system the > > update failed, I think because of a power management issue (all config > > reads now return 0xffffffff). But there have been a lot of power > > management fixes since the v5.2 kernel I'm running, so it's possible > > you'd have better luck. > > > > On your system, I think you would want something like: > > > > # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove > > # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan > > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-08 23:05 ` Luís Mendes @ 2020-04-09 15:25 ` Luís Mendes 2020-04-09 15:29 ` Luís Mendes 2020-04-09 16:30 ` Bjorn Helgaas 0 siblings, 2 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-09 15:25 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Hi Bjorn, I've good news. I've found the culprit and it is a pretty simple issue, however the good solution is not obvious to me. Can you help in finding the best way to patch this issue? So first detailing the problem in file setup_bus.c there is this *if condition* to ignore resources from classless devices and so it is that this Google/Coral Edge TPU is a classless device with class 0xff: static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) { u16 class = dev->class >> 8; pci_info(dev, "%s\n", __func__); /* Don't touch classless devices or host bridges or IOAPICs */ if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) return; .... So the one possible trivial, non generic, attempt that works is to do: static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) { u16 class = dev->class >> 8; pci_info(dev, "%s\n", __func__); /* Don't touch classless devices or host bridges or IOAPICs */ if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) return; .... What is your suggestion to make the solution generic? Create a whitelist? Remove this verification? I have no idea... nothing sounds good to me... 01:00.0 Ethernet controller: Device 1ac1:089a (prog-if ff) Subsystem: Device 1ac1:089a Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K] Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M] Capabilities: <access denied> Best regards. Luís Mendes On Thu, Apr 9, 2020 at 12:05 AM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Bjorn, > > I have successfully setup a JTAG remote debug environment. > And I found this: > First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1, > which in turn calls __pci_bus_assign_resources which visits device > 1ac1:089a on that slot and calls: ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-09 15:25 ` Luís Mendes @ 2020-04-09 15:29 ` Luís Mendes 2020-04-09 16:30 ` Bjorn Helgaas 1 sibling, 0 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-09 15:29 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt Regarding my previous email please ignore the that device is being detected as Ethernet controller I have forced class change by making dev->class==0x020000, which also works. Which is also another option for possible a solution... On Thu, Apr 9, 2020 at 4:25 PM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > Hi Bjorn, > > I've good news. I've found the culprit and it is a pretty simple > issue, however the good solution is not obvious to me. > Can you help in finding the best way to patch this issue? > > So first detailing the problem in file setup_bus.c there is this *if > condition* to ignore resources from classless devices and so > it is that this Google/Coral Edge TPU is a classless device with class 0xff: > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > { > u16 class = dev->class >> 8; > > pci_info(dev, "%s\n", __func__); > /* Don't touch classless devices or host bridges or IOAPICs */ > if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) > return; > .... > > So the one possible trivial, non generic, attempt that works is to do: > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > { > u16 class = dev->class >> 8; > > pci_info(dev, "%s\n", __func__); > /* Don't touch classless devices or host bridges or IOAPICs */ > if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) > return; > .... > > What is your suggestion to make the solution generic? Create a > whitelist? Remove this verification? I have no idea... nothing sounds > good to me... > 01:00.0 Ethernet controller: Device 1ac1:089a (prog-if ff) > Subsystem: Device 1ac1:089a > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- > ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Interrupt: pin A routed to IRQ 0 > Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K] > Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M] > Capabilities: <access denied> > > > > Best regards. > Luís Mendes > > On Thu, Apr 9, 2020 at 12:05 AM Luís Mendes <luis.p.mendes@gmail.com> wrote: > > > > Hi Bjorn, > > > > I have successfully setup a JTAG remote debug environment. > > And I found this: > > First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1, > > which in turn calls __pci_bus_assign_resources which visits device > > 1ac1:089a on that slot and calls: ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-09 15:25 ` Luís Mendes 2020-04-09 15:29 ` Luís Mendes @ 2020-04-09 16:30 ` Bjorn Helgaas 2020-04-09 17:32 ` Luís Mendes 2020-04-09 18:08 ` Bjorn Helgaas 1 sibling, 2 replies; 25+ messages in thread From: Bjorn Helgaas @ 2020-04-09 16:30 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote: > Hi Bjorn, > > I've good news. I've found the culprit and it is a pretty simple > issue, however the good solution is not obvious to me. > Can you help in finding the best way to patch this issue? > > So first detailing the problem in file setup_bus.c there is this *if > condition* to ignore resources from classless devices and so > it is that this Google/Coral Edge TPU is a classless device with class 0xff: > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > { > u16 class = dev->class >> 8; > > pci_info(dev, "%s\n", __func__); > /* Don't touch classless devices or host bridges or IOAPICs */ > if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) > return; > .... > > So the one possible trivial, non generic, attempt that works is to do: > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > { > u16 class = dev->class >> 8; > > pci_info(dev, "%s\n", __func__); > /* Don't touch classless devices or host bridges or IOAPICs */ > if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) > return; > .... > > What is your suggestion to make the solution generic? Create a > whitelist? Remove this verification? I have no idea... nothing sounds > good to me... Good detective work, thanks for chasing this down! I should have seen that check when adding the debug. Guess I thought "sort", hmmm, that just re-orders things without actually changing the content. But pdev_sort_resources() in fact *adds* resources to a list, and if resources aren't on the list, we apparently don't assign space for them. In any event, I would first check to see if there's an Edge TPU firmware update that might set the class code. If not, we should probably add a quirk to override the class code, similar to quirk_eisa_bridge(), fixup_rev1_53c810(), fixup_ti816x_class(), quirk_tw686x_class(). Bjorn ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-09 16:30 ` Bjorn Helgaas @ 2020-04-09 17:32 ` Luís Mendes 2020-04-09 18:08 ` Bjorn Helgaas 1 sibling, 0 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-09 17:32 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt I have looked at their webpage and I see no signs of a firmware that can be downloaded. https://coral.ai/products/pcie-accelerator If it is acceptable for you I'll provide a patch for such a quirk. Thanks, Luís On Thu, Apr 9, 2020 at 5:30 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote: > > Hi Bjorn, > > > > I've good news. I've found the culprit and it is a pretty simple > > issue, however the good solution is not obvious to me. > > Can you help in finding the best way to patch this issue? > > > > So first detailing the problem in file setup_bus.c there is this *if > > condition* to ignore resources from classless devices and so > > it is that this Google/Coral Edge TPU is a classless device with class 0xff: > > > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > { > > u16 class = dev->class >> 8; > > > > pci_info(dev, "%s\n", __func__); > > /* Don't touch classless devices or host bridges or IOAPICs */ > > if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) > > return; > > .... > > > > So the one possible trivial, non generic, attempt that works is to do: > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > { > > u16 class = dev->class >> 8; > > > > pci_info(dev, "%s\n", __func__); > > /* Don't touch classless devices or host bridges or IOAPICs */ > > if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && > > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) > > return; > > .... > > > > What is your suggestion to make the solution generic? Create a > > whitelist? Remove this verification? I have no idea... nothing sounds > > good to me... > > Good detective work, thanks for chasing this down! > > I should have seen that check when adding the debug. Guess I thought > "sort", hmmm, that just re-orders things without actually changing the > content. But pdev_sort_resources() in fact *adds* resources to a > list, and if resources aren't on the list, we apparently don't assign > space for them. > > In any event, I would first check to see if there's an Edge TPU > firmware update that might set the class code. > > If not, we should probably add a quirk to override the class code, > similar to quirk_eisa_bridge(), fixup_rev1_53c810(), > fixup_ti816x_class(), quirk_tw686x_class(). > > Bjorn ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-09 16:30 ` Bjorn Helgaas 2020-04-09 17:32 ` Luís Mendes @ 2020-04-09 18:08 ` Bjorn Helgaas 2020-04-09 20:07 ` Luís Mendes 1 sibling, 1 reply; 25+ messages in thread From: Bjorn Helgaas @ 2020-04-09 18:08 UTC (permalink / raw) To: Luís Mendes Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt, Todd Poynor [+cc Todd] On Thu, Apr 09, 2020 at 11:30:10AM -0500, Bjorn Helgaas wrote: > On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote: > > Hi Bjorn, > > > > I've good news. I've found the culprit and it is a pretty simple > > issue, however the good solution is not obvious to me. > > Can you help in finding the best way to patch this issue? > > > > So first detailing the problem in file setup_bus.c there is this *if > > condition* to ignore resources from classless devices and so > > it is that this Google/Coral Edge TPU is a classless device with class 0xff: > > > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > { > > u16 class = dev->class >> 8; > > > > pci_info(dev, "%s\n", __func__); > > /* Don't touch classless devices or host bridges or IOAPICs */ > > if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) > > return; > > .... > > > > So the one possible trivial, non generic, attempt that works is to do: > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > { > > u16 class = dev->class >> 8; > > > > pci_info(dev, "%s\n", __func__); > > /* Don't touch classless devices or host bridges or IOAPICs */ > > if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && > > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) > > return; > > .... > > > > What is your suggestion to make the solution generic? Create a > > whitelist? Remove this verification? I have no idea... nothing sounds > > good to me... > > Good detective work, thanks for chasing this down! > > I should have seen that check when adding the debug. Guess I thought > "sort", hmmm, that just re-orders things without actually changing the > content. But pdev_sort_resources() in fact *adds* resources to a > list, and if resources aren't on the list, we apparently don't assign > space for them. > > In any event, I would first check to see if there's an Edge TPU > firmware update that might set the class code. > > If not, we should probably add a quirk to override the class code, > similar to quirk_eisa_bridge(), fixup_rev1_53c810(), > fixup_ti816x_class(), quirk_tw686x_class(). In fact, apex_pci_fixup_class() already exists! But it's in apex_driver.c. Do you happen to have CONFIG_STAGING_APEX_DRIVER=m (built as a module)? If so, that quirk won't be run until the module is loaded, and that happens long after resource assignment. Building with CONFIG_STAGING_APEX_DRIVER=y (not =m) should be a workaround. But I think the real fix would be moving apex_pci_fixup_class() from apex_driver.c to drivers/pci/quirks.c, like the following. Would you mind testing it? commit 59f3165318b3 ("PCI: Move Apex Edge TPU class quirk to fix BAR assignment") Author: Bjorn Helgaas <bhelgaas@google.com> Date: Thu Apr 9 12:43:45 2020 -0500 PCI: Move Apex Edge TPU class quirk to fix BAR assignment Some Google Apex Edge TPU devices have a class code of 0 (PCI_CLASS_NOT_DEFINED). This prevents the PCI core from assigning resources for the Apex BARs because __dev_sort_resources() ignores classless devices, host bridges, and IOAPICs. On x86, firmware typically assigns those resources, so this was not a problem. But on some architectures, firmware does *not* assign BARs, and since the PCI core didn't do it either, the Apex device didn't work correctly: apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed apex 0000:01:00.0: error enabling PCI device f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") added a quirk to fix the class code, but it was in the apex driver, and if the driver was built as a module, it was too late to help. Move the quirk to the PCI core, where it will always run early enough that the PCI core will assign resources if necessary. Link: https://lore.kernel.org/r/CAEzXK1r0Er039iERnc2KJ4jn7ySNUOG9H=Ha8TD8XroVqiZjgg@mail.gmail.com Fixes: f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") Reported-by: Luís Mendes <luis.p.mendes@gmail.com> Debugged-by: Luís Mendes <luis.p.mendes@gmail.com> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 28c9a2409c50..ca9ed5774eb1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -5567,3 +5567,10 @@ static void pci_fixup_no_d0_pme(struct pci_dev *dev) dev->pme_support &= ~(PCI_PM_CAP_PME_D0 >> PCI_PM_CAP_PME_SHIFT); } DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ASMEDIA, 0x2142, pci_fixup_no_d0_pme); + +static void apex_pci_fixup_class(struct pci_dev *pdev) +{ + pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class; +} +DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a, + PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class); diff --git a/drivers/staging/gasket/apex_driver.c b/drivers/staging/gasket/apex_driver.c index 46199c8ca441..f12f81c8dd2f 100644 --- a/drivers/staging/gasket/apex_driver.c +++ b/drivers/staging/gasket/apex_driver.c @@ -570,13 +570,6 @@ static const struct pci_device_id apex_pci_ids[] = { { PCI_DEVICE(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID) }, { 0 } }; -static void apex_pci_fixup_class(struct pci_dev *pdev) -{ - pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class; -} -DECLARE_PCI_FIXUP_CLASS_HEADER(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID, - PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class); - static int apex_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) { ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux 2020-04-09 18:08 ` Bjorn Helgaas @ 2020-04-09 20:07 ` Luís Mendes 0 siblings, 0 replies; 25+ messages in thread From: Luís Mendes @ 2020-04-09 20:07 UTC (permalink / raw) To: Bjorn Helgaas Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper, Benjamin Herrenschmidt, Todd Poynor Hi Bjorn, I've just tested with linux-next master branch 20200409. You can add: Tested-by: Luis Mendes <luis.p.mendes@gmail.com> Thanks, Luís 01:00.0 System peripheral: Device 1ac1:089a (prog-if ff) Subsystem: Device 1ac1:089a Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at e8100000 (64-bit, prefetchable) [disabled] [size=16K] Region 2: Memory at e8000000 (64-bit, prefetchable) [disabled] [size=1M] Capabilities: <access denied> On Thu, Apr 9, 2020 at 7:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > [+cc Todd] > > On Thu, Apr 09, 2020 at 11:30:10AM -0500, Bjorn Helgaas wrote: > > On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote: > > > Hi Bjorn, > > > > > > I've good news. I've found the culprit and it is a pretty simple > > > issue, however the good solution is not obvious to me. > > > Can you help in finding the best way to patch this issue? > > > > > > So first detailing the problem in file setup_bus.c there is this *if > > > condition* to ignore resources from classless devices and so > > > it is that this Google/Coral Edge TPU is a classless device with class 0xff: > > > > > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > > { > > > u16 class = dev->class >> 8; > > > > > > pci_info(dev, "%s\n", __func__); > > > /* Don't touch classless devices or host bridges or IOAPICs */ > > > if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST) > > > return; > > > .... > > > > > > So the one possible trivial, non generic, attempt that works is to do: > > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head) > > > { > > > u16 class = dev->class >> 8; > > > > > > pci_info(dev, "%s\n", __func__); > > > /* Don't touch classless devices or host bridges or IOAPICs */ > > > if ((class == PCI_CLASS_NOT_DEFINED && !(dev->vendor == 0x1ac1 && > > > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST) > > > return; > > > .... > > > > > > What is your suggestion to make the solution generic? Create a > > > whitelist? Remove this verification? I have no idea... nothing sounds > > > good to me... > > > > Good detective work, thanks for chasing this down! > > > > I should have seen that check when adding the debug. Guess I thought > > "sort", hmmm, that just re-orders things without actually changing the > > content. But pdev_sort_resources() in fact *adds* resources to a > > list, and if resources aren't on the list, we apparently don't assign > > space for them. > > > > In any event, I would first check to see if there's an Edge TPU > > firmware update that might set the class code. > > > > If not, we should probably add a quirk to override the class code, > > similar to quirk_eisa_bridge(), fixup_rev1_53c810(), > > fixup_ti816x_class(), quirk_tw686x_class(). > > In fact, apex_pci_fixup_class() already exists! But it's in > apex_driver.c. Do you happen to have CONFIG_STAGING_APEX_DRIVER=m > (built as a module)? If so, that quirk won't be run until the module > is loaded, and that happens long after resource assignment. > > Building with CONFIG_STAGING_APEX_DRIVER=y (not =m) should be a > workaround. But I think the real fix would be moving > apex_pci_fixup_class() from apex_driver.c to drivers/pci/quirks.c, > like the following. Would you mind testing it? > > > commit 59f3165318b3 ("PCI: Move Apex Edge TPU class quirk to fix BAR assignment") > Author: Bjorn Helgaas <bhelgaas@google.com> > Date: Thu Apr 9 12:43:45 2020 -0500 > > PCI: Move Apex Edge TPU class quirk to fix BAR assignment > > Some Google Apex Edge TPU devices have a class code of 0 > (PCI_CLASS_NOT_DEFINED). This prevents the PCI core from assigning > resources for the Apex BARs because __dev_sort_resources() ignores > classless devices, host bridges, and IOAPICs. > > On x86, firmware typically assigns those resources, so this was not a > problem. But on some architectures, firmware does *not* assign BARs, and > since the PCI core didn't do it either, the Apex device didn't work > correctly: > > apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed > apex 0000:01:00.0: error enabling PCI device > > f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") added a > quirk to fix the class code, but it was in the apex driver, and if the > driver was built as a module, it was too late to help. > > Move the quirk to the PCI core, where it will always run early enough that > the PCI core will assign resources if necessary. > > Link: https://lore.kernel.org/r/CAEzXK1r0Er039iERnc2KJ4jn7ySNUOG9H=Ha8TD8XroVqiZjgg@mail.gmail.com > Fixes: f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") > Reported-by: Luís Mendes <luis.p.mendes@gmail.com> > Debugged-by: Luís Mendes <luis.p.mendes@gmail.com> > Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index 28c9a2409c50..ca9ed5774eb1 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -5567,3 +5567,10 @@ static void pci_fixup_no_d0_pme(struct pci_dev *dev) > dev->pme_support &= ~(PCI_PM_CAP_PME_D0 >> PCI_PM_CAP_PME_SHIFT); > } > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ASMEDIA, 0x2142, pci_fixup_no_d0_pme); > + > +static void apex_pci_fixup_class(struct pci_dev *pdev) > +{ > + pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class; > +} > +DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a, > + PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class); > diff --git a/drivers/staging/gasket/apex_driver.c b/drivers/staging/gasket/apex_driver.c > index 46199c8ca441..f12f81c8dd2f 100644 > --- a/drivers/staging/gasket/apex_driver.c > +++ b/drivers/staging/gasket/apex_driver.c > @@ -570,13 +570,6 @@ static const struct pci_device_id apex_pci_ids[] = { > { PCI_DEVICE(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID) }, { 0 } > }; > > -static void apex_pci_fixup_class(struct pci_dev *pdev) > -{ > - pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class; > -} > -DECLARE_PCI_FIXUP_CLASS_HEADER(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID, > - PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class); > - > static int apex_pci_probe(struct pci_dev *pci_dev, > const struct pci_device_id *id) > { ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2020-04-09 20:08 UTC | newest] Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-06 14:32 Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux Luís Mendes 2020-03-06 21:47 ` Bjorn Helgaas 2020-03-07 12:11 ` Luís Mendes 2020-03-07 15:26 ` Luís Mendes 2020-03-07 21:38 ` Bjorn Helgaas 2020-03-08 5:51 ` Nicholas Johnson 2020-03-09 11:21 ` Luís Mendes 2020-03-11 14:20 ` Luís Mendes 2020-03-29 22:11 ` Luís Mendes 2020-03-30 19:49 ` Bjorn Helgaas 2020-03-31 21:28 ` Luís Mendes 2020-04-01 18:16 ` Bjorn Helgaas 2020-04-01 21:20 ` Luís Mendes 2020-04-01 21:55 ` Luís Mendes 2020-04-01 23:31 ` Bjorn Helgaas 2020-04-02 14:13 ` Luís Mendes 2020-04-04 1:32 ` Bjorn Helgaas 2020-04-04 21:39 ` Luís Mendes 2020-04-08 23:05 ` Luís Mendes 2020-04-09 15:25 ` Luís Mendes 2020-04-09 15:29 ` Luís Mendes 2020-04-09 16:30 ` Bjorn Helgaas 2020-04-09 17:32 ` Luís Mendes 2020-04-09 18:08 ` Bjorn Helgaas 2020-04-09 20:07 ` Luís Mendes
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).