From: Bjorn Helgaas <helgaas@kernel.org>
To: "Luís Mendes" <luis.p.mendes@gmail.com>
Cc: linux-pci@vger.kernel.org,
Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
Jason Cooper <jason@lakedaemon.net>,
Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>
Subject: Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
Date: Fri, 6 Mar 2020 15:47:53 -0600 [thread overview]
Message-ID: <20200306214753.GA235309@google.com> (raw)
In-Reply-To: <CAEzXK1r0Er039iERnc2KJ4jn7ySNUOG9H=Ha8TD8XroVqiZjgg@mail.gmail.com>
[+cc Thomas, Jason, Nicholas, Ben]
On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote:
> Hi,
>
> I'm trying to use Google/Coral TPU Edge modules for a project, on
> arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
> of PCIe devices and consequently pci_enable_device(...) fails on BAR0
> resource with value -22 (EINVAL) (resource has null parent) when
> loading gasket/apex driver.
>
> I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
> is not the root cause for this issue.
>
> Relevant Log extracts follow in attachment.
Hi Luís,
Thanks for the report, and sorry for the problem you're tripping over.
I cc'd a few folks who might be interested.
> [ 6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
> [ 6.993528] hub 4-1:1.0: 4 ports detected
> [ 6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
> [ 7.106741] sdb: sdb1
> [ 7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
> [ 7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
> [ 7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
> [ 7.261267] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff]
> [ 7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
> [ 7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
> [ 7.293971] PCI: bus0: Fast back to back transfers disabled
> [ 7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [ 7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> [ 7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
> [ 7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
> [ 7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
> [ 7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
> [ 7.383442] PCI: bus1: Fast back to back transfers disabled
> [ 7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> [ 7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
> [ 7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
> [ 7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
> [ 7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
> [ 7.585079] pci 0000:00:01.0: bridge window [mem 0xe8000000-0xe81fffff]
> [ 7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
>
>
> [ 11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
> [ 11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
> [ 11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
> [ 11.227825] apex 0000:01:00.0: error enabling PCI device
It looks like we did assign space for the bridge window leading to
01:00.0, but failed to assign space to the 01:00.0 BAR itself.
I don't know offhand why that would be. Can you put the entire dmesg
log somewhere we can see? That will tell us what kernel you're using
and possibly other useful things.
Does the same problem happen with other devices, or does it only
happen with the gasket/apex combination? There shouldn't be anything
device-specific in the PCI core resource assignment code.
> 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
> Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@1,0
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> I/O behind bridge: 0000f000-00000fff [empty]
> Memory behind bridge: e8000000-e81fffff [size=2M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> [virtual] Expansion ROM at e8200000 [disabled] [size=2K]
> BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0
> ExtTag- RBE+
> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s (ok), Width x1 (ok)
> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
> Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
> SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> Changed: MRL- PresDet- LinkState-
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> RootCap: CRSVisible-
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> AtomicOpsCtl: ReqEn- EgressBlck-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>
> 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
> Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@2,0
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> I/O behind bridge: 00010000-00010fff [size=4K]
> Memory behind bridge: d0000000-e7ffffff [size=384M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> [virtual] Expansion ROM at e8300000 [disabled] [size=2K]
> BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0
> ExtTag- RBE+
> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
> Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
> SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> Changed: MRL- PresDet- LinkState-
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> RootCap: CRSVisible-
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> AtomicOpsCtl: ReqEn- EgressBlck-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>
> 01:00.0 Non-VGA unclassified device: Device 1ac1:089a (prog-if ff)
> Subsystem: Device 1ac1:089a
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 48
> Region 0: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=16K]
> Region 2: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1M]
> Capabilities: [80] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s (ok), Width x1 (ok)
> TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
> AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> AtomicOpsCtl: ReqEn-
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> Capabilities: [d0] MSI-X: Enable- Count=128 Masked-
> Vector table: BAR=2 offset=00046800
> PBA: BAR=2 offset=00046068
> Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [f8] Power Management version 3
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
> Capabilities: [108 v1] Latency Tolerance Reporting
> Max snoop latency: 0ns
> Max no snoop latency: 0ns
> Capabilities: [110 v1] L1 PM Substates
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
> PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
> L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> T_CommonMode=0us LTR1.2_Threshold=0ns
> L1SubCtl2: T_PwrOn=10us
> Capabilities: [200 v2] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> HeaderLog: 00000000 00000000 00000000 00000000
> Kernel modules: apex
>
>
>
next prev parent reply other threads:[~2020-03-06 21:47 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-06 14:32 Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux Luís Mendes
2020-03-06 21:47 ` Bjorn Helgaas [this message]
2020-03-07 12:11 ` Luís Mendes
2020-03-07 15:26 ` Luís Mendes
2020-03-07 21:38 ` Bjorn Helgaas
2020-03-08 5:51 ` Nicholas Johnson
2020-03-09 11:21 ` Luís Mendes
2020-03-11 14:20 ` Luís Mendes
2020-03-29 22:11 ` Luís Mendes
2020-03-30 19:49 ` Bjorn Helgaas
2020-03-31 21:28 ` Luís Mendes
2020-04-01 18:16 ` Bjorn Helgaas
2020-04-01 21:20 ` Luís Mendes
2020-04-01 21:55 ` Luís Mendes
2020-04-01 23:31 ` Bjorn Helgaas
2020-04-02 14:13 ` Luís Mendes
2020-04-04 1:32 ` Bjorn Helgaas
2020-04-04 21:39 ` Luís Mendes
2020-04-08 23:05 ` Luís Mendes
2020-04-09 15:25 ` Luís Mendes
2020-04-09 15:29 ` Luís Mendes
2020-04-09 16:30 ` Bjorn Helgaas
2020-04-09 17:32 ` Luís Mendes
2020-04-09 18:08 ` Bjorn Helgaas
2020-04-09 20:07 ` Luís Mendes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200306214753.GA235309@google.com \
--to=helgaas@kernel.org \
--cc=benh@kernel.crashing.org \
--cc=jason@lakedaemon.net \
--cc=linux-pci@vger.kernel.org \
--cc=luis.p.mendes@gmail.com \
--cc=nicholas.johnson-opensource@outlook.com.au \
--cc=thomas.petazzoni@bootlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).