linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
@ 2020-03-06 14:32 Luís Mendes
  2020-03-06 21:47 ` Bjorn Helgaas
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-06 14:32 UTC (permalink / raw)
  To: Linux PCI

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

Hi,

I'm trying to use Google/Coral TPU Edge modules for a project, on
arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
of PCIe devices and consequently pci_enable_device(...) fails on BAR0
resource with value -22 (EINVAL) (resource has null parent) when
loading gasket/apex driver.

I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
is not the root cause for this issue.

Relevant Log extracts follow in attachment.

Regards,
Luís Mendes

[-- Attachment #2: Coral_TPU_armhf_enum.txt --]
[-- Type: text/plain, Size: 11171 bytes --]


[    6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
[    6.993528] hub 4-1:1.0: 4 ports detected
[    6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
[    7.106741]  sdb: sdb1
[    7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
[    7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
[    7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
[    7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
[    7.261267] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
[    7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
[    7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    7.293971] PCI: bus0: Fast back to back transfers disabled
[    7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
[    7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
[    7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
[    7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
[    7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
[    7.383442] PCI: bus1: Fast back to back transfers disabled
[    7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
[    7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
[    7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
[    7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
[    7.585079] pci 0000:00:01.0:   bridge window [mem 0xe8000000-0xe81fffff]
[    7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)


[   11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
[   11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
[   11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
[   11.227825] apex 0000:01:00.0: error enabling PCI device


00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
	Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@1,0
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [empty]
	Memory behind bridge: e8000000-e81fffff [size=2M]
	Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	[virtual] Expansion ROM at e8200000 [disabled] [size=2K]
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
	Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@2,0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 00010000-00010fff [size=4K]
	Memory behind bridge: d0000000-e7ffffff [size=384M]
	Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	[virtual] Expansion ROM at e8300000 [disabled] [size=2K]
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

01:00.0 Non-VGA unclassified device: Device 1ac1:089a (prog-if ff)
	Subsystem: Device 1ac1:089a
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 48
	Region 0: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=16K]
	Region 2: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1M]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [d0] MSI-X: Enable- Count=128 Masked-
		Vector table: BAR=2 offset=00046800
		PBA: BAR=2 offset=00046068
	Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
	Capabilities: [108 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Kernel modules: apex




^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-06 14:32 Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux Luís Mendes
@ 2020-03-06 21:47 ` Bjorn Helgaas
  2020-03-07 12:11   ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-03-06 21:47 UTC (permalink / raw)
  To: Luís Mendes
  Cc: linux-pci, Thomas Petazzoni, Jason Cooper, Nicholas Johnson,
	Benjamin Herrenschmidt

[+cc Thomas, Jason, Nicholas, Ben]

On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote:
> Hi,
> 
> I'm trying to use Google/Coral TPU Edge modules for a project, on
> arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
> of PCIe devices and consequently pci_enable_device(...) fails on BAR0
> resource with value -22 (EINVAL) (resource has null parent) when
> loading gasket/apex driver.
> 
> I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
> is not the root cause for this issue.
> 
> Relevant Log extracts follow in attachment.

Hi Luís,

Thanks for the report, and sorry for the problem you're tripping over.
I cc'd a few folks who might be interested.

> [    6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
> [    6.993528] hub 4-1:1.0: 4 ports detected
> [    6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
> [    7.106741]  sdb: sdb1
> [    7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
> [    7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
> [    7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
> [    7.261267] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> [    7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
> [    7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
> [    7.293971] PCI: bus0: Fast back to back transfers disabled
> [    7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> [    7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
> [    7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
> [    7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
> [    7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
> [    7.383442] PCI: bus1: Fast back to back transfers disabled
> [    7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> [    7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
> [    7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
> [    7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
> [    7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
> [    7.585079] pci 0000:00:01.0:   bridge window [mem 0xe8000000-0xe81fffff]
> [    7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> 
> 
> [   11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
> [   11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
> [   11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
> [   11.227825] apex 0000:01:00.0: error enabling PCI device

It looks like we did assign space for the bridge window leading to
01:00.0, but failed to assign space to the 01:00.0 BAR itself.

I don't know offhand why that would be.  Can you put the entire dmesg
log somewhere we can see?  That will tell us what kernel you're using
and possibly other useful things.

Does the same problem happen with other devices, or does it only
happen with the gasket/apex combination?  There shouldn't be anything
device-specific in the PCI core resource assignment code.

> 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
> 	Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@1,0
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0, Cache Line Size: 64 bytes
> 	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> 	I/O behind bridge: 0000f000-00000fff [empty]
> 	Memory behind bridge: e8000000-e81fffff [size=2M]
> 	Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> 	[virtual] Expansion ROM at e8200000 [disabled] [size=2K]
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
> 		DevCap:	MaxPayload 128 bytes, PhantFunc 0
> 			ExtTag- RBE+
> 		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
> 			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> 			MaxPayload 128 bytes, MaxReadReq 512 bytes
> 		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> 		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
> 			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> 		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> 			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> 		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
> 			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> 		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
> 			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
> 		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> 			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> 		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> 			Changed: MRL- PresDet- LinkState-
> 		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> 		RootCap: CRSVisible-
> 		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> 		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> 			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> 		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> 			 AtomicOpsCtl: ReqEn- EgressBlck-
> 		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> 			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> 			 Compliance De-emphasis: -6dB
> 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> 			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> 
> 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) (prog-if 00 [Normal decode])
> 	Device tree node: /sys/firmware/devicetree/base/soc/pcie/pcie@2,0
> 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0, Cache Line Size: 64 bytes
> 	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> 	I/O behind bridge: 00010000-00010fff [size=4K]
> 	Memory behind bridge: d0000000-e7ffffff [size=384M]
> 	Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> 	[virtual] Expansion ROM at e8300000 [disabled] [size=2K]
> 	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort+ >Reset- FastB2B-
> 		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
> 		DevCap:	MaxPayload 128 bytes, PhantFunc 0
> 			ExtTag- RBE+
> 		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
> 			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> 			MaxPayload 128 bytes, MaxReadReq 512 bytes
> 		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> 		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
> 			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> 		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> 			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> 		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (ok)
> 			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> 		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
> 			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
> 		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> 			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> 		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> 			Changed: MRL- PresDet- LinkState-
> 		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> 		RootCap: CRSVisible-
> 		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> 		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> 			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> 		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> 			 AtomicOpsCtl: ReqEn- EgressBlck-
> 		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> 			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> 			 Compliance De-emphasis: -6dB
> 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> 			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> 
> 01:00.0 Non-VGA unclassified device: Device 1ac1:089a (prog-if ff)
> 	Subsystem: Device 1ac1:089a
> 	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 48
> 	Region 0: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=16K]
> 	Region 2: [virtual] Memory at <unassigned> (64-bit, prefetchable) [size=1M]
> 	Capabilities: [80] Express (v2) Endpoint, MSI 00
> 		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> 			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
> 		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
> 			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> 			MaxPayload 128 bytes, MaxReadReq 512 bytes
> 		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> 		LnkCap:	Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
> 			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> 		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> 			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> 		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
> 			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
> 		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
> 			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> 		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> 			 AtomicOpsCtl: ReqEn-
> 		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> 			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> 			 Compliance De-emphasis: -6dB
> 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> 			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> 	Capabilities: [d0] MSI-X: Enable- Count=128 Masked-
> 		Vector table: BAR=2 offset=00046800
> 		PBA: BAR=2 offset=00046068
> 	Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
> 		Address: 0000000000000000  Data: 0000
> 	Capabilities: [f8] Power Management version 3
> 		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> 		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> 	Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
> 	Capabilities: [108 v1] Latency Tolerance Reporting
> 		Max snoop latency: 0ns
> 		Max no snoop latency: 0ns
> 	Capabilities: [110 v1] L1 PM Substates
> 		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
> 			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
> 		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> 			   T_CommonMode=0us LTR1.2_Threshold=0ns
> 		L1SubCtl2: T_PwrOn=10us
> 	Capabilities: [200 v2] Advanced Error Reporting
> 		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> 		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> 		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
> 		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
> 		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> 		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> 			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> 		HeaderLog: 00000000 00000000 00000000 00000000
> 	Kernel modules: apex
> 
> 
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-06 21:47 ` Bjorn Helgaas
@ 2020-03-07 12:11   ` Luís Mendes
  2020-03-07 15:26     ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-07 12:11 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson,
	Benjamin Herrenschmidt

Hi Bjorn,

Thanks for your help.

On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Thomas, Jason, Nicholas, Ben]
>
> On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote:
> > Hi,
> >
> > I'm trying to use Google/Coral TPU Edge modules for a project, on
> > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
> > of PCIe devices and consequently pci_enable_device(...) fails on BAR0
> > resource with value -22 (EINVAL) (resource has null parent) when
> > loading gasket/apex driver.
> >
> > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
> > is not the root cause for this issue.
> >
> > Relevant Log extracts follow in attachment.
>
> Hi Luís,
>
> Thanks for the report, and sorry for the problem you're tripping over.
> I cc'd a few folks who might be interested.
>
> > [    6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
> > [    6.993528] hub 4-1:1.0: 4 ports detected
> > [    6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
> > [    7.106741]  sdb: sdb1
> > [    7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
> > [    7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
> > [    7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [    7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
> > [    7.261267] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> > [    7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
> > [    7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
> > [    7.293971] PCI: bus0: Fast back to back transfers disabled
> > [    7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > [    7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> > [    7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
> > [    7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
> > [    7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
> > [    7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
> > [    7.383442] PCI: bus1: Fast back to back transfers disabled
> > [    7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> > [    7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
> > [    7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
> > [    7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
> > [    7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
> > [    7.585079] pci 0000:00:01.0:   bridge window [mem 0xe8000000-0xe81fffff]
> > [    7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> >
> >
> > [   11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
> > [   11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
> > [   11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
> > [   11.227825] apex 0000:01:00.0: error enabling PCI device
>
> It looks like we did assign space for the bridge window leading to
> 01:00.0, but failed to assign space to the 01:00.0 BAR itself.
>
> I don't know offhand why that would be.  Can you put the entire dmesg
> log somewhere we can see?  That will tell us what kernel you're using
> and possibly other useful things.

Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/
This is a custom built machine based on the Armada 388 armhf SoC, that
I am using, but I've also tried an arm64 machine from Toradex, an
Apalis IMX8QM with the Apalis Eval board, producing similar results
with different kernels and also different ARM architectures.
>
> Does the same problem happen with other devices, or does it only
> happen with the gasket/apex combination?  There shouldn't be anything
> device-specific in the PCI core resource assignment code.

This issue seems to happen only with the Coral Edge TPU device, but it
happens independently of whether the gasket/apex driver module is
loaded or not. The BAR 0 of the Coral device is not assigned either
way.

Luís

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-07 12:11   ` Luís Mendes
@ 2020-03-07 15:26     ` Luís Mendes
  2020-03-07 21:38       ` Bjorn Helgaas
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-07 15:26 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson,
	Benjamin Herrenschmidt

Hi,

A quick look at the logs, makes me wonder if BAR0-BAR5 are only being
assigned to IO space on device by Linux, and BAR6 is the first bar
index that Linux is assigning on armhf/arm64 for mem space. If so,
that would be wrong because registers 0x10 and and 0x18 are BAR0 and
BAR2.
The TP-Link Gigabit LAN card that is installed on the other PCIe slot
has BAR 0 enabled but it is IO space and according to the registers
for the mem space, in that device, seen in dmesg, they are regs 0x18
and 0x20, or, BAR 2 and BAR 4, but Linux is assigning them to have
indices BAR 6 and BAR 8, since they are MEM space devices.
That looks wrong.... maybe the TP-Link device is working just because
BAR 0 does happen to exist in this case, which happens to be the
minimal requirement that allows pci_enable_device(...) to work,
passing in the test 'if (!r->parent)' performed by
pci_enable_resources(...) at drivers/pci/setup-res.c. Since most PCIe
cards have IO space, this generally works.
Can it be?

Luís

On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> Thanks for your help.
>
> On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc Thomas, Jason, Nicholas, Ben]
> >
> > On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote:
> > > Hi,
> > >
> > > I'm trying to use Google/Coral TPU Edge modules for a project, on
> > > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
> > > of PCIe devices and consequently pci_enable_device(...) fails on BAR0
> > > resource with value -22 (EINVAL) (resource has null parent) when
> > > loading gasket/apex driver.
> > >
> > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
> > > is not the root cause for this issue.
> > >
> > > Relevant Log extracts follow in attachment.
> >
> > Hi Luís,
> >
> > Thanks for the report, and sorry for the problem you're tripping over.
> > I cc'd a few folks who might be interested.
> >
> > > [    6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
> > > [    6.993528] hub 4-1:1.0: 4 ports detected
> > > [    6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
> > > [    7.106741]  sdb: sdb1
> > > [    7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
> > > [    7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
> > > [    7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [    7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
> > > [    7.261267] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> > > [    7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
> > > [    7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
> > > [    7.293971] PCI: bus0: Fast back to back transfers disabled
> > > [    7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > > [    7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> > > [    7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
> > > [    7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
> > > [    7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
> > > [    7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
> > > [    7.383442] PCI: bus1: Fast back to back transfers disabled
> > > [    7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> > > [    7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
> > > [    7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
> > > [    7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
> > > [    7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
> > > [    7.585079] pci 0000:00:01.0:   bridge window [mem 0xe8000000-0xe81fffff]
> > > [    7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> > >
> > >
> > > [   11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
> > > [   11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
> > > [   11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
> > > [   11.227825] apex 0000:01:00.0: error enabling PCI device
> >
> > It looks like we did assign space for the bridge window leading to
> > 01:00.0, but failed to assign space to the 01:00.0 BAR itself.
> >
> > I don't know offhand why that would be.  Can you put the entire dmesg
> > log somewhere we can see?  That will tell us what kernel you're using
> > and possibly other useful things.
>
> Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/
> This is a custom built machine based on the Armada 388 armhf SoC, that
> I am using, but I've also tried an arm64 machine from Toradex, an
> Apalis IMX8QM with the Apalis Eval board, producing similar results
> with different kernels and also different ARM architectures.
> >
> > Does the same problem happen with other devices, or does it only
> > happen with the gasket/apex combination?  There shouldn't be anything
> > device-specific in the PCI core resource assignment code.
>
> This issue seems to happen only with the Coral Edge TPU device, but it
> happens independently of whether the gasket/apex driver module is
> loaded or not. The BAR 0 of the Coral device is not assigned either
> way.
>
> Luís

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-07 15:26     ` Luís Mendes
@ 2020-03-07 21:38       ` Bjorn Helgaas
  2020-03-08  5:51         ` Nicholas Johnson
  0 siblings, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-03-07 21:38 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Linux PCI, Thomas Petazzoni, Jason Cooper, Nicholas Johnson,
	Benjamin Herrenschmidt

On Sat, Mar 07, 2020 at 03:26:49PM +0000, Luís Mendes wrote:
> Hi,
> 
> A quick look at the logs, makes me wonder if BAR0-BAR5 are only being
> assigned to IO space on device by Linux, and BAR6 is the first bar
> index that Linux is assigning on armhf/arm64 for mem space. If so,
> that would be wrong because registers 0x10 and and 0x18 are BAR0 and
> BAR2.

BAR0-5 are the standard BARs and can be either mem or I/O space.
64-bit mem BARs take two slots, e.g.,

  pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
  pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]

These are BAR2 (at 0x18 and 0x1c) and BAR4 (at 0x20 and 0x24).

> The TP-Link Gigabit LAN card that is installed on the other PCIe slot
> has BAR 0 enabled but it is IO space and according to the registers
> for the mem space, in that device, seen in dmesg, they are regs 0x18
> and 0x20, or, BAR 2 and BAR 4, but Linux is assigning them to have
> indices BAR 6 and BAR 8, since they are MEM space devices.

Our logging is a little screwed up: sometimes we print "reg 0x10",
other times "BAR 0":

  pci 0000:02:00.0: reg 0x10: [io  0x0000-0x00ff]
  pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
  pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
  pci 0000:02:00.0: BAR 4: assigned [mem 0xd0200000-0xd0203fff 64bit pref]
  pci 0000:02:00.0: BAR 2: assigned [mem 0xd0204000-0xd0204fff 64bit]
  pci 0000:02:00.0: BAR 0: assigned [io  0x10000-0x100ff]

Anyway, we end up with this, which looks fine:

  BAR0 (reg 0x10):       [io  0x10000-0x100ff]
  BAR2 (reg 0x18, 0x1c): [mem 0xd0204000-0xd0204fff 64bit]
  BAR4 (reg 0x20, 0x24): [mem 0xd0200000-0xd0203fff 64bit pref]

"BAR 6" is for option ROMs.  You have bridges with option ROMS (seems
sort of unusual, but legal), and we assign space for them:

  pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
  pci 0000:00:01.0: BAR 6: assigned [mem 0xd0300000-0xd03007ff pref]
  pci 0000:00:01.0: BAR 8: assigned [mem 0xd0000000-0xd01fffff]
  pci 0000:00:01.0: PCI bridge to [bus 01]
  pci 0000:00:01.0:   bridge window [mem 0xd0000000-0xd01fffff]

  pci 0000:00:02.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
  pci 0000:00:02.0: BAR 6: assigned [mem 0xd0400000-0xd04007ff pref]
  pci 0000:00:02.0: BAR 8: assigned [mem 0xd0200000-0xd02fffff]
  pci 0000:00:02.0: PCI bridge to [bus 02]
  pci 0000:00:02.0:   bridge window [io  0x10000-0x10fff]
  pci 0000:00:02.0:   bridge window [mem 0xd0200000-0xd02fffff]

You don't have CONFIG_PCI_IOV enabled (see the enum in <linux/pci.h>),
so "BAR7" is is the bridge I/O window, "BAR8" is the MMIO window, and
"BAR9" is the prefetchable MMIO window.

The problematic device needs 16KB + 1MB of prefetchable memory space:

  pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
  pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]

The bridge leading to it has a 2MB non-prefetchable window:

  pci 0000:00:01.0: PCI bridge to [bus 01]
  pci 0000:00:01.0:   bridge window [mem 0xd0000000-0xd01fffff]

That *should* work -- it's not prefetchable, but if we don't have
prefetchable space, non-prefetchable space should also work (with poor
performance, of course).  But maybe the fallback from prefetchable to
non-prefetchable space is broken or something.

> That looks wrong.... maybe the TP-Link device is working just because
> BAR 0 does happen to exist in this case, which happens to be the
> minimal requirement that allows pci_enable_device(...) to work,
> passing in the test 'if (!r->parent)' performed by
> pci_enable_resources(...) at drivers/pci/setup-res.c. Since most PCIe
> cards have IO space, this generally works.
> Can it be?
> 
> Luís
> 
> On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> >
> > Hi Bjorn,
> >
> > Thanks for your help.
> >
> > On Fri, Mar 6, 2020 at 9:47 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > [+cc Thomas, Jason, Nicholas, Ben]
> > >
> > > On Fri, Mar 06, 2020 at 02:32:59PM +0000, Luís Mendes wrote:
> > > > Hi,
> > > >
> > > > I'm trying to use Google/Coral TPU Edge modules for a project, on
> > > > arm64 and armhf, but BAR0 doesn't get assigned during the enumeration
> > > > of PCIe devices and consequently pci_enable_device(...) fails on BAR0
> > > > resource with value -22 (EINVAL) (resource has null parent) when
> > > > loading gasket/apex driver.
> > > >
> > > > I'm also trying to adapt gasket/apex to run on armhf, but anyhow that
> > > > is not the root cause for this issue.
> > > >
> > > > Relevant Log extracts follow in attachment.
> > >
> > > Hi Luís,
> > >
> > > Thanks for the report, and sorry for the problem you're tripping over.
> > > I cc'd a few folks who might be interested.
> > >
> > > > [    6.983880] mvebu-pcie soc:pcie: /soc/pcie/pcie@1,0: reset gpio is active low
> > > > [    6.993528] hub 4-1:1.0: 4 ports detected
> > > > [    6.993749] mvebu-pcie soc:pcie: /soc/pcie/pcie@2,0: reset gpio is active low
> > > > [    7.106741]  sdb: sdb1
> > > > [    7.109826] sd 2:0:0:0: [sdb] Attached SCSI removable disk
> > > > [    7.242916] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
> > > > [    7.248854] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > > [    7.254370] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xefffffff]
> > > > [    7.261267] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> > > > [    7.267621] pci 0000:00:01.0: [11ab:6828] type 01 class 0x060400
> > > > [    7.273662] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
> > > > [    7.293971] PCI: bus0: Fast back to back transfers disabled
> > > > [    7.299558] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > > > [    7.315694] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> > > > [    7.321749] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
> > > > [    7.322814] usb 4-1.1: new high-speed USB device number 3 using xhci-hcd
> > > > [    7.329004] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref]
> > > > [    7.343111] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
> > > > [    7.383442] PCI: bus1: Fast back to back transfers disabled
> > > > [    7.389031] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> > > > [    7.495604] pci 0000:00:02.0: ASPM: current common clock configuration is broken, reconfiguring
> > > > [    7.552513] pci 0000:00:01.0: BAR 8: assigned [mem 0xe8000000-0xe81fffff]
> > > > [    7.565611] pci 0000:00:01.0: BAR 6: assigned [mem 0xe8200000-0xe82007ff pref]
> > > > [    7.580096] pci 0000:00:01.0: PCI bridge to [bus 01]
> > > > [    7.585079] pci 0000:00:01.0:   bridge window [mem 0xe8000000-0xe81fffff]
> > > > [    7.653228] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> > > >
> > > >
> > > > [   11.188025] gasket: module is from the staging directory, the quality is unknown, you have been warned.
> > > > [   11.217048] apex: module is from the staging directory, the quality is unknown, you have been warned.
> > > > [   11.217926] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
> > > > [   11.227825] apex 0000:01:00.0: error enabling PCI device
> > >
> > > It looks like we did assign space for the bridge window leading to
> > > 01:00.0, but failed to assign space to the 01:00.0 BAR itself.
> > >
> > > I don't know offhand why that would be.  Can you put the entire dmesg
> > > log somewhere we can see?  That will tell us what kernel you're using
> > > and possibly other useful things.
> >
> > Sure, complete dmesg is available at: https://pastebin.ubuntu.com/p/qnzJ56kM7k/
> > This is a custom built machine based on the Armada 388 armhf SoC, that
> > I am using, but I've also tried an arm64 machine from Toradex, an
> > Apalis IMX8QM with the Apalis Eval board, producing similar results
> > with different kernels and also different ARM architectures.
> > >
> > > Does the same problem happen with other devices, or does it only
> > > happen with the gasket/apex combination?  There shouldn't be anything
> > > device-specific in the PCI core resource assignment code.
> >
> > This issue seems to happen only with the Coral Edge TPU device, but it
> > happens independently of whether the gasket/apex driver module is
> > loaded or not. The BAR 0 of the Coral device is not assigned either
> > way.
> >
> > Luís

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-07 21:38       ` Bjorn Helgaas
@ 2020-03-08  5:51         ` Nicholas Johnson
  2020-03-09 11:21           ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Nicholas Johnson @ 2020-03-08  5:51 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Luís Mendes, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi,
> > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> > > This issue seems to happen only with the Coral Edge TPU device, but it
> > > happens independently of whether the gasket/apex driver module is
> > > loaded or not. The BAR 0 of the Coral device is not assigned either
> > > way.
> > >
> > > Luís

So the problem only occurs with the Coral Edge TPU device, so there is a 
possibility that it is not a problem with the platform, or something 
caused by the combination of the TPU and platform. Is it possible to put 
the TPU into an X86 system with the same kernel version(s) to add more 
evidence to this theory? If it works on X86 then we can focus on the 
differences between X86 and ARM.

Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO 
window assignment" or try with Linux v5.4 which does not have this 
commit, just to rule out the possibility of it causing issues. This 
patch helps me and also solved the problem of one other person using an 
ARM computer who came to us regarding a problem. However, it could also 
adversely affect unknown use cases - it is impossible to completely rule 
out, due to the nature of how drivers/pci/setup-bus.c is written.

Kind regards,
Nicholas

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-08  5:51         ` Nicholas Johnson
@ 2020-03-09 11:21           ` Luís Mendes
  2020-03-11 14:20             ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-09 11:21 UTC (permalink / raw)
  To: Nicholas Johnson
  Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Nicholas,

Thanks for your help.
Replies follow below.

Kind Regards,
Luís

On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson
<nicholas.johnson-opensource@outlook.com.au> wrote:
>
> Hi,
> > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> > > > This issue seems to happen only with the Coral Edge TPU device, but it
> > > > happens independently of whether the gasket/apex driver module is
> > > > loaded or not. The BAR 0 of the Coral device is not assigned either
> > > > way.
> > > >
> > > > Luís
>
> So the problem only occurs with the Coral Edge TPU device, so there is a
> possibility that it is not a problem with the platform, or something
> caused by the combination of the TPU and platform. Is it possible to put
> the TPU into an X86 system with the same kernel version(s) to add more
> evidence to this theory? If it works on X86 then we can focus on the
> differences between X86 and ARM.

I've tested two Coral TPUs on two x86_64 platforms and the BARs seem
to be assigned, however the driver fails to load, during probe and the
system is unable to shutdown cleanly, but I think that is a driver
issue when setting up sysfs entries. I can blacklist gasket/apex, if
it helps in some way, or try an older kernel.
Dmesg log for one of the x86_64 system is here:
https://pastebin.ubuntu.com/p/FHhHNN6XTF/
lspci -vvv for the same x86_64 system is here:
https://pastebin.ubuntu.com/p/xbSNWFQ9TS/

>
> Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO
> window assignment" or try with Linux v5.4 which does not have this
> commit, just to rule out the possibility of it causing issues. This
> patch helps me and also solved the problem of one other person using an
> ARM computer who came to us regarding a problem. However, it could also
> adversely affect unknown use cases - it is impossible to completely rule
> out, due to the nature of how drivers/pci/setup-bus.c is written.

On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not
assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/
I've also tried kernel 4.20.11 and the problem also exists.
I've got JTAG on this system with OpenOCD. I believe I can debug the
kernel, if needed.

>
> Kind regards,
> Nicholas

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-09 11:21           ` Luís Mendes
@ 2020-03-11 14:20             ` Luís Mendes
  2020-03-29 22:11               ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-11 14:20 UTC (permalink / raw)
  To: Nicholas Johnson
  Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Re-sending... after 48 hours the previous message still didn't hit the
mailing list...

On Mon, Mar 9, 2020 at 11:21 AM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Nicholas,
>
> Thanks for your help.
> Replies follow below.
>
> Kind Regards,
> Luís
>
> On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson
> <nicholas.johnson-opensource@outlook.com.au> wrote:
> >
> > Hi,
> > > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> > > > > This issue seems to happen only with the Coral Edge TPU device, but it
> > > > > happens independently of whether the gasket/apex driver module is
> > > > > loaded or not. The BAR 0 of the Coral device is not assigned either
> > > > > way.
> > > > >
> > > > > Luís
> >
> > So the problem only occurs with the Coral Edge TPU device, so there is a
> > possibility that it is not a problem with the platform, or something
> > caused by the combination of the TPU and platform. Is it possible to put
> > the TPU into an X86 system with the same kernel version(s) to add more
> > evidence to this theory? If it works on X86 then we can focus on the
> > differences between X86 and ARM.
>
> I've tested two Coral TPUs on two x86_64 platforms and the BARs seem
> to be assigned, however the driver fails to load, during probe and the
> system is unable to shutdown cleanly, but I think that is a driver
> issue when setting up sysfs entries. I can blacklist gasket/apex, if
> it helps in some way, or try an older kernel.
> Dmesg log for one of the x86_64 system is here:
> https://pastebin.ubuntu.com/p/FHhHNN6XTF/
> lspci -vvv for the same x86_64 system is here:
> https://pastebin.ubuntu.com/p/xbSNWFQ9TS/
>
> >
> > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO
> > window assignment" or try with Linux v5.4 which does not have this
> > commit, just to rule out the possibility of it causing issues. This
> > patch helps me and also solved the problem of one other person using an
> > ARM computer who came to us regarding a problem. However, it could also
> > adversely affect unknown use cases - it is impossible to completely rule
> > out, due to the nature of how drivers/pci/setup-bus.c is written.
>
> On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not
> assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/
> I've also tried kernel 4.20.11 and the problem also exists.
> I've got JTAG on this system with OpenOCD. I believe I can debug the
> kernel, if needed.
>
> >
> > Kind regards,
> > Nicholas

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-11 14:20             ` Luís Mendes
@ 2020-03-29 22:11               ` Luís Mendes
  2020-03-30 19:49                 ` Bjorn Helgaas
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-29 22:11 UTC (permalink / raw)
  To: Nicholas Johnson
  Cc: Bjorn Helgaas, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Nicholas, Bjorn,

I was able to make the apex driver work on a X86_64 system with the
Coral Edge TPU PCIe device.
So, now the PCI enumeration problem is now clearly an ARM and ARM64
platform issue. What are the recommended steps for debugging this? I
hava a JTAG interface and openOCD supported configuration for it.

On X86_64 the PCI device did enumerate properly, but the driver would
fail to load due to a bug. That is now fixed and it did run a couple
of examples, on X86_64 only, after applying  a patch that I submitted
to the Gasket driver maintainers:
Fix incongruency in handling of sysfs entries creation.
This issue could cause invalid memory accesses, by not properly
detecting the end of the sysfs attributes array.

Signed-off-by: Luis Mendes <luis.p.mendes@gmail.com>
---

 gasket_sysfs.c |    3 +--
 gasket_sysfs.h |    4 ----
 2 files changed, 1 insertion(+), 6 deletions(-)

Kind Regards,
Luís

> On Mon, Mar 9, 2020 at 11:21 AM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> >
> > Hi Nicholas,
> >
> > Thanks for your help.
> > Replies follow below.
> >
> > Kind Regards,
> > Luís
> >
> > On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson
> > <nicholas.johnson-opensource@outlook.com.au> wrote:
> > >
> > > Hi,
> > > > > On Sat, Mar 7, 2020 at 12:11 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> > > > > > This issue seems to happen only with the Coral Edge TPU device, but it
> > > > > > happens independently of whether the gasket/apex driver module is
> > > > > > loaded or not. The BAR 0 of the Coral device is not assigned either
> > > > > > way.
> > > > > >
> > > > > > Luís
> > >
> > > So the problem only occurs with the Coral Edge TPU device, so there is a
> > > possibility that it is not a problem with the platform, or something
> > > caused by the combination of the TPU and platform. Is it possible to put
> > > the TPU into an X86 system with the same kernel version(s) to add more
> > > evidence to this theory? If it works on X86 then we can focus on the
> > > differences between X86 and ARM.
> >
> > I've tested two Coral TPUs on two x86_64 platforms and the BARs seem
> > to be assigned, however the driver fails to load, during probe and the
> > system is unable to shutdown cleanly, but I think that is a driver
> > issue when setting up sysfs entries. I can blacklist gasket/apex, if
> > it helps in some way, or try an older kernel.
> > Dmesg log for one of the x86_64 system is here:
> > https://pastebin.ubuntu.com/p/FHhHNN6XTF/
> > lspci -vvv for the same x86_64 system is here:
> > https://pastebin.ubuntu.com/p/xbSNWFQ9TS/
> >
> > >
> > > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO
> > > window assignment" or try with Linux v5.4 which does not have this
> > > commit, just to rule out the possibility of it causing issues. This
> > > patch helps me and also solved the problem of one other person using an
> > > ARM computer who came to us regarding a problem. However, it could also
> > > adversely affect unknown use cases - it is impossible to completely rule
> > > out, due to the nature of how drivers/pci/setup-bus.c is written.
> >
> > On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not
> > assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/
> > I've also tried kernel 4.20.11 and the problem also exists.
> > I've got JTAG on this system with OpenOCD. I believe I can debug the
> > kernel, if needed.
> >
> > >
> > > Kind regards,
> > > Nicholas

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-29 22:11               ` Luís Mendes
@ 2020-03-30 19:49                 ` Bjorn Helgaas
  2020-03-31 21:28                   ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-03-30 19:49 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

On Sun, Mar 29, 2020 at 11:11:28PM +0100, Luís Mendes wrote:
> Hi Nicholas, Bjorn,
> 
> I was able to make the apex driver work on a X86_64 system with the
> Coral Edge TPU PCIe device.
> So, now the PCI enumeration problem is now clearly an ARM and ARM64
> platform issue. What are the recommended steps for debugging this? I
> hava a JTAG interface and openOCD supported configuration for it.

Thanks for the work of testing on X86_64.  I don't have any magic
ideas other than instrumenting the code and slogging through the
output.  Can you try the patch below and collect the dmesg?  This will
probably take a few iterations.


diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 8e40b3e6da77..2cdb705752de 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -166,6 +166,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 	resource_size_t max;
 
 	type_mask |= IORESOURCE_TYPE_BITS;
+	pci_info(bus, "%s: %pR type_mask %#lx\n", __func__, res, type_mask);
 
 	pci_bus_for_each_resource(bus, r, i) {
 		resource_size_t min_used = min;
@@ -173,6 +174,9 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 		if (!r)
 			continue;
 
+		pci_info(bus, "%s: from %pR res %#lx r %#lx\n", __func__,
+			r, res->flags, r->flags);
+
 		/* type_mask must match */
 		if ((res->flags ^ r->flags) & type_mask)
 			continue;
@@ -203,6 +207,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 		if (ret == 0)
 			return 0;
 	}
+	pci_info(bus, "%s: failed\n", __func__);
 	return -ENOMEM;
 }
 
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index f2461bf9243d..649aa90b8b29 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -280,8 +280,10 @@ static void assign_requested_resources_sorted(struct list_head *head,
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
 		idx = res - &dev_res->dev->resource[0];
+		pci_info(dev_res->dev, "%s BAR%d %pR\n", __func__, idx, res);
 		if (resource_size(res) &&
 		    pci_assign_resource(dev_res->dev, idx)) {
+			pci_info(dev_res->dev, "%s (failed)\n", __func__);
 			if (fail_head) {
 				/*
 				 * If the failed resource is a ROM BAR and
@@ -996,6 +998,12 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 	resource_size_t children_add_align = 0;
 	resource_size_t add_align = 0;
 
+	pci_info(bus, "%s: mask %#lx type %#lx %#lx %#lx min %#llx add %#llx b_res %pR parent %pR\n",
+		__func__, mask, type, type2, type3,
+		(unsigned long long) min_size,
+		(unsigned long long) add_size,
+		b_res, b_res ? b_res->parent : NULL);
+
 	if (!b_res)
 		return -ENOSPC;
 
@@ -1199,6 +1207,8 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head)
 	struct resource *b_res;
 	int ret;
 
+	pci_info(bus, "%s\n", __func__);
+
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		struct pci_bus *b = dev->subordinate;
 		if (!b)
@@ -1311,6 +1321,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head)
 
 void pci_bus_size_bridges(struct pci_bus *bus)
 {
+	pci_info(bus, "%s\n", __func__);
 	__pci_bus_size_bridges(bus, NULL);
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
@@ -1394,6 +1405,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus,
 
 void pci_bus_assign_resources(const struct pci_bus *bus)
 {
+	pci_info(bus, "%s\n", __func__);
 	__pci_bus_assign_resources(bus, NULL, NULL);
 }
 EXPORT_SYMBOL(pci_bus_assign_resources);
@@ -1408,6 +1420,7 @@ static void pci_claim_device_resources(struct pci_dev *dev)
 		if (!r->flags || r->parent)
 			continue;
 
+		pci_info(dev, "%s BAR%d %pR\n", __func__, i, r);
 		pci_claim_resource(dev, i);
 	}
 }
@@ -1422,6 +1435,7 @@ static void pci_claim_bridge_resources(struct pci_dev *dev)
 		if (!r->flags || r->parent)
 			continue;
 
+		pci_info(dev, "%s BAR%d %pR\n", __func__, i, r);
 		pci_claim_bridge_resource(dev, i);
 	}
 }
@@ -1460,6 +1474,7 @@ static void pci_bus_allocate_resources(struct pci_bus *b)
 
 void pci_bus_claim_resources(struct pci_bus *b)
 {
+	pci_info(bus, "%s\n", __func__);
 	pci_bus_allocate_resources(b);
 	pci_bus_allocate_dev_resources(b);
 }

^ permalink raw reply related	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-30 19:49                 ` Bjorn Helgaas
@ 2020-03-31 21:28                   ` Luís Mendes
  2020-04-01 18:16                     ` Bjorn Helgaas
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-03-31 21:28 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

Thanks for your help.

I've removed all other PCIe devices to make the analysis easier.
The dmesg with the traces can be found at:
https://paste.ubuntu.com/p/W3m2VQCYqg/

Didn't find anything new related to BAR0 or BAR2, in the dmesg,
though. Anyway I'm no expert in this, maybe it can give you some
useful information, still.

Kind Regards,
Luís Mendes

On Mon, Mar 30, 2020 at 8:49 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sun, Mar 29, 2020 at 11:11:28PM +0100, Luís Mendes wrote:
> > Hi Nicholas, Bjorn,
> >
> > I was able to make the apex driver work on a X86_64 system with the
> > Coral Edge TPU PCIe device.
> > So, now the PCI enumeration problem is now clearly an ARM and ARM64
> > platform issue. What are the recommended steps for debugging this? I
> > hava a JTAG interface and openOCD supported configuration for it.
>
> Thanks for the work of testing on X86_64.  I don't have any magic
> ideas other than instrumenting the code and slogging through the
> output.  Can you try the patch below and collect the dmesg?  This will
> probably take a few iterations.
>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-03-31 21:28                   ` Luís Mendes
@ 2020-04-01 18:16                     ` Bjorn Helgaas
  2020-04-01 21:20                       ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-04-01 18:16 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote:
> I've removed all other PCIe devices to make the analysis easier.
> The dmesg with the traces can be found at:
> https://paste.ubuntu.com/p/W3m2VQCYqg/
> 
> Didn't find anything new related to BAR0 or BAR2, in the dmesg,
> though. Anyway I'm no expert in this, maybe it can give you some
> useful information, still.

It looks like we assigned the right amount of space to the bridge, but
for some reason didn't assign it to the device *below* the bridge.

I added a few more messages in this patch.  Can you remove the first
one and replace it with this?  This is still based on v5.6-rc1.


diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 8e40b3e6da77..2cdb705752de 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -166,6 +166,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 	resource_size_t max;
 
 	type_mask |= IORESOURCE_TYPE_BITS;
+	pci_info(bus, "%s: %pR type_mask %#lx\n", __func__, res, type_mask);
 
 	pci_bus_for_each_resource(bus, r, i) {
 		resource_size_t min_used = min;
@@ -173,6 +174,9 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 		if (!r)
 			continue;
 
+		pci_info(bus, "%s: from %pR res %#lx r %#lx\n", __func__,
+			r, res->flags, r->flags);
+
 		/* type_mask must match */
 		if ((res->flags ^ r->flags) & type_mask)
 			continue;
@@ -203,6 +207,7 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 		if (ret == 0)
 			return 0;
 	}
+	pci_info(bus, "%s: failed\n", __func__);
 	return -ENOMEM;
 }
 
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index f2461bf9243d..b42f1bcab25f 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -177,6 +177,7 @@ static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
 {
 	u16 class = dev->class >> 8;
 
+	pci_info(dev, "%s\n", __func__);
 	/* Don't touch classless devices or host bridges or IOAPICs */
 	if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
 		return;
@@ -280,8 +281,10 @@ static void assign_requested_resources_sorted(struct list_head *head,
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
 		idx = res - &dev_res->dev->resource[0];
+		pci_info(dev_res->dev, "%s: BAR%d %pR\n", __func__, idx, res);
 		if (resource_size(res) &&
 		    pci_assign_resource(dev_res->dev, idx)) {
+			pci_info(dev_res->dev, "%s: failed\n", __func__);
 			if (fail_head) {
 				/*
 				 * If the failed resource is a ROM BAR and
@@ -372,6 +375,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	unsigned long fail_type;
 	resource_size_t add_align, align;
 
+	pr_info("%s\n", __func__);
 	/* Check if optional add_size is there */
 	if (!realloc_head || list_empty(realloc_head))
 		goto requested_and_reassign;
@@ -386,6 +390,7 @@ static void __assign_resources_sorted(struct list_head *head,
 
 	/* Update res in head list with add_size in realloc_head list */
 	list_for_each_entry_safe(dev_res, tmp_res, head, list) {
+		pci_info(dev_res->dev, "%s: %pR\n", __func__, dev_res->res);
 		dev_res->res->end += get_res_add_size(realloc_head,
 							dev_res->res);
 
@@ -436,6 +441,7 @@ static void __assign_resources_sorted(struct list_head *head,
 			remove_from_list(realloc_head, dev_res->res);
 		free_list(&save_head);
 		free_list(head);
+		pr_info("%s: success\n", __func__);
 		return;
 	}
 
@@ -483,6 +489,7 @@ static void pdev_assign_resources_sorted(struct pci_dev *dev,
 {
 	LIST_HEAD(head);
 
+	pci_info(dev, "%s\n", __func__);
 	__dev_sort_resources(dev, &head);
 	__assign_resources_sorted(&head, add_head, fail_head);
 
@@ -495,6 +502,7 @@ static void pbus_assign_resources_sorted(const struct pci_bus *bus,
 	struct pci_dev *dev;
 	LIST_HEAD(head);
 
+	pci_info(bus, "%s\n", __func__);
 	list_for_each_entry(dev, &bus->devices, bus_list)
 		__dev_sort_resources(dev, &head);
 
@@ -996,6 +1004,12 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 	resource_size_t children_add_align = 0;
 	resource_size_t add_align = 0;
 
+	pci_info(bus, "%s: mask %#04lx type %#04lx %#04lx %#04lx min %#llx add %#llx b_res %pR parent %pR\n",
+		__func__, mask, type, type2, type3,
+		(unsigned long long) min_size,
+		(unsigned long long) add_size,
+		b_res, b_res ? b_res->parent : NULL);
+
 	if (!b_res)
 		return -ENOSPC;
 
@@ -1089,6 +1103,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			   (unsigned long long) (size1 - size0),
 			   (unsigned long long) add_align);
 	}
+	pci_info(bus->self, "%s: %pR\n", __func__, b_res);
 	return 0;
 }
 
@@ -1199,6 +1214,8 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head)
 	struct resource *b_res;
 	int ret;
 
+	pci_info(bus, "%s\n", __func__);
+
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		struct pci_bus *b = dev->subordinate;
 		if (!b)
@@ -1311,6 +1328,7 @@ void __pci_bus_size_bridges(struct pci_bus *bus, struct list_head *realloc_head)
 
 void pci_bus_size_bridges(struct pci_bus *bus)
 {
+	pci_info(bus, "%s\n", __func__);
 	__pci_bus_size_bridges(bus, NULL);
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
@@ -1340,6 +1358,7 @@ static void pdev_assign_fixed_resources(struct pci_dev *dev)
 {
 	int i;
 
+	pci_info(dev, "%s\n", __func__);
 	for (i = 0; i <  PCI_NUM_RESOURCES; i++) {
 		struct pci_bus *b;
 		struct resource *r = &dev->resource[i];
@@ -1363,9 +1382,11 @@ void __pci_bus_assign_resources(const struct pci_bus *bus,
 	struct pci_bus *b;
 	struct pci_dev *dev;
 
+	pci_info(bus, "%s\n", __func__);
 	pbus_assign_resources_sorted(bus, realloc_head, fail_head);
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
+		pci_info(dev, "%s\n", __func__);
 		pdev_assign_fixed_resources(dev);
 
 		b = dev->subordinate;
@@ -1394,6 +1415,7 @@ void __pci_bus_assign_resources(const struct pci_bus *bus,
 
 void pci_bus_assign_resources(const struct pci_bus *bus)
 {
+	pci_info(bus, "%s\n", __func__);
 	__pci_bus_assign_resources(bus, NULL, NULL);
 }
 EXPORT_SYMBOL(pci_bus_assign_resources);
@@ -1408,6 +1430,7 @@ static void pci_claim_device_resources(struct pci_dev *dev)
 		if (!r->flags || r->parent)
 			continue;
 
+		pci_info(dev, "%s: BAR%d %pR\n", __func__, i, r);
 		pci_claim_resource(dev, i);
 	}
 }
@@ -1422,6 +1445,7 @@ static void pci_claim_bridge_resources(struct pci_dev *dev)
 		if (!r->flags || r->parent)
 			continue;
 
+		pci_info(dev, "%s: BAR%d %pR\n", __func__, i, r);
 		pci_claim_bridge_resource(dev, i);
 	}
 }
@@ -1432,6 +1456,7 @@ static void pci_bus_allocate_dev_resources(struct pci_bus *b)
 	struct pci_bus *child;
 
 	list_for_each_entry(dev, &b->devices, bus_list) {
+		pci_info(dev, "%s\n", __func__);
 		pci_claim_device_resources(dev);
 
 		child = dev->subordinate;
@@ -1460,6 +1485,7 @@ static void pci_bus_allocate_resources(struct pci_bus *b)
 
 void pci_bus_claim_resources(struct pci_bus *b)
 {
+	pci_info(bus, "%s\n", __func__);
 	pci_bus_allocate_resources(b);
 	pci_bus_allocate_dev_resources(b);
 }
@@ -1471,6 +1497,7 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 {
 	struct pci_bus *b;
 
+	pci_info(dev, "%s\n", __func__);
 	pdev_assign_resources_sorted((struct pci_dev *)bridge,
 					 add_head, fail_head);
 

^ permalink raw reply related	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-01 18:16                     ` Bjorn Helgaas
@ 2020-04-01 21:20                       ` Luís Mendes
  2020-04-01 21:55                         ` Luís Mendes
  2020-04-01 23:31                         ` Bjorn Helgaas
  0 siblings, 2 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-01 21:20 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

Here is the dmesg output with the new kernel patch:
https://paste.ubuntu.com/p/7cv7ZcyrnG/

Luís

On Wed, Apr 1, 2020 at 7:16 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote:
> > I've removed all other PCIe devices to make the analysis easier.
> > The dmesg with the traces can be found at:
> > https://paste.ubuntu.com/p/W3m2VQCYqg/
> >
> > Didn't find anything new related to BAR0 or BAR2, in the dmesg,
> > though. Anyway I'm no expert in this, maybe it can give you some
> > useful information, still.
>
> It looks like we assigned the right amount of space to the bridge, but
> for some reason didn't assign it to the device *below* the bridge.
>
> I added a few more messages in this patch.  Can you remove the first
> one and replace it with this?  This is still based on v5.6-rc1.
>
>
> diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> index 8e40b3e6da77..2cdb705752de 100644
> --- a/drivers/pci/bus.c
> +++ b/drivers/pci/bus.c
...

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-01 21:20                       ` Luís Mendes
@ 2020-04-01 21:55                         ` Luís Mendes
  2020-04-01 23:31                         ` Bjorn Helgaas
  1 sibling, 0 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-01 21:55 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjiorn

I don't see any log of __pci_bridge_assign_resources. This platform
makes use of pci-bridge-emul.c functions. Don't know if that can be
responsible for not calling __pci_bridge_assign_resources(...).

Luís

On Wed, Apr 1, 2020 at 10:20 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> Here is the dmesg output with the new kernel patch:
> https://paste.ubuntu.com/p/7cv7ZcyrnG/
>
> Luís
>
> On Wed, Apr 1, 2020 at 7:16 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Tue, Mar 31, 2020 at 10:28:51PM +0100, Luís Mendes wrote:
> > > I've removed all other PCIe devices to make the analysis easier.
> > > The dmesg with the traces can be found at:
> > > https://paste.ubuntu.com/p/W3m2VQCYqg/
> > >
> > > Didn't find anything new related to BAR0 or BAR2, in the dmesg,
> > > though. Anyway I'm no expert in this, maybe it can give you some
> > > useful information, still.
> >
> > It looks like we assigned the right amount of space to the bridge, but
> > for some reason didn't assign it to the device *below* the bridge.
> >
> > I added a few more messages in this patch.  Can you remove the first
> > one and replace it with this?  This is still based on v5.6-rc1.
> >
> >
> > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > index 8e40b3e6da77..2cdb705752de 100644
> > --- a/drivers/pci/bus.c
> > +++ b/drivers/pci/bus.c
> ...

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-01 21:20                       ` Luís Mendes
  2020-04-01 21:55                         ` Luís Mendes
@ 2020-04-01 23:31                         ` Bjorn Helgaas
  2020-04-02 14:13                           ` Luís Mendes
  1 sibling, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-04-01 23:31 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote:
> Hi Bjorn,
> 
> Here is the dmesg output with the new kernel patch:
> https://paste.ubuntu.com/p/7cv7ZcyrnG/

Thanks.  What happens if you do this:

  # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan

(I think on v5.6 that file is "bus_rescan" instead of "rescan".
There's a patch queued up to change it.)

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-01 23:31                         ` Bjorn Helgaas
@ 2020-04-02 14:13                           ` Luís Mendes
  2020-04-04  1:32                             ` Bjorn Helgaas
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-04-02 14:13 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

The command that worked was:
#  sh -c "echo 1 >
/sys/bus/pci/devices/0000\:00\:01.0/pci_bus/0000\:01/bus_rescan"

The only new information compared to yesterday dmesg are the following lines:
...
[   20.633566] mvneta f1070000.ethernet eth0: Link is Up - 1Gbps/Full
- flow control off
[   20.633598] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1048.661591] FAT-fs (sda1): Volume was not properly unmounted. Some
data may be corrupt. Please run fsck.
[ 1049.792121] EXT4-fs (sdb1): mounted filesystem with ordered data
mode. Opts: (null)
[60026.321476] pci_bus 0000:01: __pci_bus_assign_resources
[60026.321481] pci_bus 0000:01: pbus_assign_resources_sorted
[60026.321485] pci 0000:01:00.0: __dev_sort_resources
[60026.321487] __assign_resources_sorted
[60026.321490] pci 0000:01:00.0: __pci_bus_assign_resources
[60026.321493] pci 0000:01:00.0: pdev_assign_fixed_resources

Luís

On Thu, Apr 2, 2020 at 12:31 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote:
> > Hi Bjorn,
> >
> > Here is the dmesg output with the new kernel patch:
> > https://paste.ubuntu.com/p/7cv7ZcyrnG/
>
> Thanks.  What happens if you do this:
>
>   # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan
>
> (I think on v5.6 that file is "bus_rescan" instead of "rescan".
> There's a patch queued up to change it.)

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-02 14:13                           ` Luís Mendes
@ 2020-04-04  1:32                             ` Bjorn Helgaas
  2020-04-04 21:39                               ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-04-04  1:32 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

On Thu, Apr 02, 2020 at 03:13:36PM +0100, Luís Mendes wrote:
> Hi Bjorn,
> 
> The command that worked was:
> #  sh -c "echo 1 >
> /sys/bus/pci/devices/0000\:00\:01.0/pci_bus/0000\:01/bus_rescan"
> 
> The only new information compared to yesterday dmesg are the following lines:
> ...
> [   20.633566] mvneta f1070000.ethernet eth0: Link is Up - 1Gbps/Full
> - flow control off
> [   20.633598] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 1048.661591] FAT-fs (sda1): Volume was not properly unmounted. Some
> data may be corrupt. Please run fsck.
> [ 1049.792121] EXT4-fs (sdb1): mounted filesystem with ordered data
> mode. Opts: (null)
> [60026.321476] pci_bus 0000:01: __pci_bus_assign_resources
> [60026.321481] pci_bus 0000:01: pbus_assign_resources_sorted
> [60026.321485] pci 0000:01:00.0: __dev_sort_resources
> [60026.321487] __assign_resources_sorted
> [60026.321490] pci 0000:01:00.0: __pci_bus_assign_resources
> [60026.321493] pci 0000:01:00.0: pdev_assign_fixed_resources

The idea I'm pushing on is that maybe Linux don't program endpoint
BARs correctly.  This theory is weak because I'm pretty sure there are
ARM systems where firmware doesn't program the BARs, and some of them
seem to work.  But resource assignment is arch-specific and even
mvebu-specific, so maybe this is a wrinkle.

The hotplug case is much less arch-specific than the boot-time case,
so I wonder if you'll see something different if you remove the device
and then re-enumerate it?  I don't think you have a hotplug-capable
slot, but we can do something similar via sysfs.

I tried to test this on my x86 system (where BARs are assigned by
the BIOS at boot-time) by clearing the BAR of an idle device, removing
it, and re-enumerating it.  Boot-time enumeration said:

  pci 0000:00:1c.0: PCI bridge to [bus 02]
  pci 0000:02:00.0: reg 0x14: [mem 0xf1100000-0xf1100fff]

Here I cleared the BAR, removed, rescanned:

  # setpci -s02:00.0 BASE_ADDRESS_1.L=0
  # echo 1 > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/remove
  # echo 1 > /sys/devices/pci0000:00/0000:00:1c.0/rescan

and dmesg said:

  pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000
  pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
  pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff]
  pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff)

So we correctly detected the device, read the cleared BAR, and
allocated space for it, and tried to update the BAR.  On my system the
update failed, I think because of a power management issue (all config
reads now return 0xffffffff).  But there have been a lot of power
management fixes since the v5.2 kernel I'm running, so it's possible
you'd have better luck.

On your system, I think you would want something like:

  # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove
  # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan

> On Thu, Apr 2, 2020 at 12:31 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Wed, Apr 01, 2020 at 10:20:42PM +0100, Luís Mendes wrote:
> > > Hi Bjorn,
> > >
> > > Here is the dmesg output with the new kernel patch:
> > > https://paste.ubuntu.com/p/7cv7ZcyrnG/
> >
> > Thanks.  What happens if you do this:
> >
> >   # echo 1 > sys/devices/pci0000:00/0000:00:01.0/pci_bus/0000:01/rescan
> >
> > (I think on v5.6 that file is "bus_rescan" instead of "rescan".
> > There's a patch queued up to change it.)

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-04  1:32                             ` Bjorn Helgaas
@ 2020-04-04 21:39                               ` Luís Mendes
  2020-04-08 23:05                                 ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-04-04 21:39 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

Thanks again for your help.

Ok... I've tested your theory on this system and still no changes. The
BARs 0 and 1 are still not assigned. I should note that this issue
does not occur only on this particular armhf system, but also on a
Toradex Apalis IMX8QM, which in this case is an arm64 device and
doesn't make use of the mvebu infrastructure.

So I did issue the following commands:
# echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/remove
# echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/dev_rescan"

And the dmesg update after the last command is:
[   61.124696] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
[   61.124732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff
64bit pref]
[   61.124743] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff
64bit pref]
[   61.161258] pci_bus 0000:01: __pci_bus_size_bridges
[   61.161270] pci_bus 0000:01: pbus_size_mem: mask 0x2200 type 0x2200
0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
[   61.161277] pci_bus 0000:01: pbus_size_mem: mask 0x200 type 0x200
0x200 0x200 min 0x0 add 0x0 b_res [mem 0xd0000000-0xd01fffff] parent
[mem 0xd0000000-0xefffffff]
[   61.161281] pci_bus 0000:02: __pci_bus_size_bridges
[   61.161286] pci_bus 0000:02: pbus_size_mem: mask 0x2200 type 0x2200
0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
[   61.161291] pci_bus 0000:02: pbus_size_mem: mask 0x200 type 0x200
0x200 0x200 min 0x0 add 0x0 b_res [mem 0x00000000] parent (null)
[   61.161295] pci_bus 0000:00: __pci_bus_assign_resources
[   61.161298] pci_bus 0000:00: pbus_assign_resources_sorted
[   61.161302] pci 0000:00:01.0: __dev_sort_resources
[   61.161305] pci 0000:00:02.0: __dev_sort_resources
[   61.161308] __assign_resources_sorted
[   61.161311] pci 0000:00:01.0: __pci_bus_assign_resources
[   61.161314] pci 0000:00:01.0: pdev_assign_fixed_resources
[   61.161317] pci_bus 0000:01: __pci_bus_assign_resources
[   61.161319] pci_bus 0000:01: pbus_assign_resources_sorted
[   61.161323] pci 0000:01:00.0: __dev_sort_resources
[   61.161324] __assign_resources_sorted
[   61.161327] pci 0000:01:00.0: __pci_bus_assign_resources
[   61.161330] pci 0000:01:00.0: pdev_assign_fixed_resources
[   61.161333] pci 0000:00:01.0: PCI bridge to [bus 01]
[   61.161340] pci 0000:00:01.0:   bridge window [mem 0xd0000000-0xd01fffff]
[   61.161344] pci 0000:00:02.0: __pci_bus_assign_resources
[   61.161347] pci 0000:00:02.0: pdev_assign_fixed_resources
[   61.161350] pci_bus 0000:02: __pci_bus_assign_resources
[   61.161353] pci_bus 0000:02: pbus_assign_resources_sorted
[   61.161354] __assign_resources_sorted
[   61.161357] pci 0000:00:02.0: PCI bridge to [bus 02]

Luís

On Sat, Apr 4, 2020 at 2:32 AM Bjorn Helgaas <helgaas@kernel.org> wrote:

>   pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000
>   pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
>   pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff]
>   pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff)
>
> So we correctly detected the device, read the cleared BAR, and
> allocated space for it, and tried to update the BAR.  On my system the
> update failed, I think because of a power management issue (all config
> reads now return 0xffffffff).  But there have been a lot of power
> management fixes since the v5.2 kernel I'm running, so it's possible
> you'd have better luck.
>
> On your system, I think you would want something like:
>
>   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove
>   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-04 21:39                               ` Luís Mendes
@ 2020-04-08 23:05                                 ` Luís Mendes
  2020-04-09 15:25                                   ` Luís Mendes
  0 siblings, 1 reply; 25+ messages in thread
From: Luís Mendes @ 2020-04-08 23:05 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

I have successfully setup a JTAG remote debug environment.
And I found this:
First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1,
which in turn calls __pci_bus_assign_resources which visits device
1ac1:089a on that slot and calls:
/*
 * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they are
 * skipped by pbus_assign_resources_sorted().
 */
static void pdev_assign_fixed_resources(struct pci_dev *dev)
{
        int i;

       pci_info(dev, "%s\n", __func__);
        for (i = 0; i <  PCI_NUM_RESOURCES; i++) {
                struct pci_bus *b;
                struct resource *r = &dev->resource[i];

                if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) ||
                    !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
                        continue;

                b = dev->bus;
                while (b && !r->parent) {
                        assign_fixed_resource_on_bus(b, r);
                        b = b->parent;
                }
        }
}
where dev has the following data before calling
pdev_assign_fixed_resources, for some reason BAR0 and BAR2 are skipped
or assign_fixed_resource_on_bus is not called :
dev->vendorID 6849(0x1ac1)
   ->deviceID 2202(0x089a)
   ->resource={ {start = 0
         end=16383
         name="0000:01:00.0"
         flags=1319436(0x14220C)
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name=0x0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0,
         end = 1048575,
         name="0000:01:00.0"
         flags=1319436(0x14220C)
         desc=0,
         parent=0x0,
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name="0000:01:00.0"
         flags=0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name="0000:01:00.0"
         flags=0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name="0000:01:00.0"
         flags=0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name=0x0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name=0x0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name=0x0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0},
        {start = 0
         end=0
         name=0x0
         desc=0
         parent=0x0
         sibling=0x0
         child=0x0}}

Luís

On Sat, Apr 4, 2020 at 10:39 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> Thanks again for your help.
>
> Ok... I've tested your theory on this system and still no changes. The
> BARs 0 and 1 are still not assigned. I should note that this issue
> does not occur only on this particular armhf system, but also on a
> Toradex Apalis IMX8QM, which in this case is an arm64 device and
> doesn't make use of the mvebu infrastructure.
>
> So I did issue the following commands:
> # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/remove
> # echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/dev_rescan"
>
> And the dmesg update after the last command is:
> [   61.124696] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
> [   61.124732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff
> 64bit pref]
> [   61.124743] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff
> 64bit pref]
> [   61.161258] pci_bus 0000:01: __pci_bus_size_bridges
> [   61.161270] pci_bus 0000:01: pbus_size_mem: mask 0x2200 type 0x2200
> 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
> [   61.161277] pci_bus 0000:01: pbus_size_mem: mask 0x200 type 0x200
> 0x200 0x200 min 0x0 add 0x0 b_res [mem 0xd0000000-0xd01fffff] parent
> [mem 0xd0000000-0xefffffff]
> [   61.161281] pci_bus 0000:02: __pci_bus_size_bridges
> [   61.161286] pci_bus 0000:02: pbus_size_mem: mask 0x2200 type 0x2200
> 0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
> [   61.161291] pci_bus 0000:02: pbus_size_mem: mask 0x200 type 0x200
> 0x200 0x200 min 0x0 add 0x0 b_res [mem 0x00000000] parent (null)
> [   61.161295] pci_bus 0000:00: __pci_bus_assign_resources
> [   61.161298] pci_bus 0000:00: pbus_assign_resources_sorted
> [   61.161302] pci 0000:00:01.0: __dev_sort_resources
> [   61.161305] pci 0000:00:02.0: __dev_sort_resources
> [   61.161308] __assign_resources_sorted
> [   61.161311] pci 0000:00:01.0: __pci_bus_assign_resources
> [   61.161314] pci 0000:00:01.0: pdev_assign_fixed_resources
> [   61.161317] pci_bus 0000:01: __pci_bus_assign_resources
> [   61.161319] pci_bus 0000:01: pbus_assign_resources_sorted
> [   61.161323] pci 0000:01:00.0: __dev_sort_resources
> [   61.161324] __assign_resources_sorted
> [   61.161327] pci 0000:01:00.0: __pci_bus_assign_resources
> [   61.161330] pci 0000:01:00.0: pdev_assign_fixed_resources
> [   61.161333] pci 0000:00:01.0: PCI bridge to [bus 01]
> [   61.161340] pci 0000:00:01.0:   bridge window [mem 0xd0000000-0xd01fffff]
> [   61.161344] pci 0000:00:02.0: __pci_bus_assign_resources
> [   61.161347] pci 0000:00:02.0: pdev_assign_fixed_resources
> [   61.161350] pci_bus 0000:02: __pci_bus_assign_resources
> [   61.161353] pci_bus 0000:02: pbus_assign_resources_sorted
> [   61.161354] __assign_resources_sorted
> [   61.161357] pci 0000:00:02.0: PCI bridge to [bus 02]
>
> Luís
>
> On Sat, Apr 4, 2020 at 2:32 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> >   pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000
> >   pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
> >   pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff]
> >   pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff)
> >
> > So we correctly detected the device, read the cleared BAR, and
> > allocated space for it, and tried to update the BAR.  On my system the
> > update failed, I think because of a power management issue (all config
> > reads now return 0xffffffff).  But there have been a lot of power
> > management fixes since the v5.2 kernel I'm running, so it's possible
> > you'd have better luck.
> >
> > On your system, I think you would want something like:
> >
> >   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove
> >   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan
> >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-08 23:05                                 ` Luís Mendes
@ 2020-04-09 15:25                                   ` Luís Mendes
  2020-04-09 15:29                                     ` Luís Mendes
  2020-04-09 16:30                                     ` Bjorn Helgaas
  0 siblings, 2 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-09 15:25 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Hi Bjorn,

I've good news. I've found the culprit and it is a pretty simple
issue, however the good solution is not obvious to me.
Can you help in finding the best way to patch this issue?

So first detailing the problem in file setup_bus.c there is this *if
condition* to ignore resources from classless devices and so
it is that this Google/Coral Edge TPU is a classless device with class 0xff:

static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
{
    u16 class = dev->class >> 8;

       pci_info(dev, "%s\n", __func__);
    /* Don't touch classless devices or host bridges or IOAPICs */
    if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
        return;
   ....

So the one possible trivial, non generic, attempt that works is to do:
static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
{
    u16 class = dev->class >> 8;

       pci_info(dev, "%s\n", __func__);
    /* Don't touch classless devices or host bridges or IOAPICs */
    if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
        return;
   ....

What is your suggestion to make the solution generic? Create a
whitelist? Remove this verification? I have no idea... nothing sounds
good to me...
01:00.0 Ethernet controller: Device 1ac1:089a (prog-if ff)
    Subsystem: Device 1ac1:089a
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K]
    Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M]
    Capabilities: <access denied>



Best regards.
Luís Mendes

On Thu, Apr 9, 2020 at 12:05 AM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> I have successfully setup a JTAG remote debug environment.
> And I found this:
> First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1,
> which in turn calls __pci_bus_assign_resources which visits device
> 1ac1:089a on that slot and calls:

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-09 15:25                                   ` Luís Mendes
@ 2020-04-09 15:29                                     ` Luís Mendes
  2020-04-09 16:30                                     ` Bjorn Helgaas
  1 sibling, 0 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-09 15:29 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

Regarding my previous email please ignore the that device is being
detected as Ethernet controller I have forced class change by making
dev->class==0x020000, which also works. Which is also another option
for possible a solution...

On Thu, Apr 9, 2020 at 4:25 PM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> I've good news. I've found the culprit and it is a pretty simple
> issue, however the good solution is not obvious to me.
> Can you help in finding the best way to patch this issue?
>
> So first detailing the problem in file setup_bus.c there is this *if
> condition* to ignore resources from classless devices and so
> it is that this Google/Coral Edge TPU is a classless device with class 0xff:
>
> static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> {
>     u16 class = dev->class >> 8;
>
>        pci_info(dev, "%s\n", __func__);
>     /* Don't touch classless devices or host bridges or IOAPICs */
>     if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
>         return;
>    ....
>
> So the one possible trivial, non generic, attempt that works is to do:
> static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> {
>     u16 class = dev->class >> 8;
>
>        pci_info(dev, "%s\n", __func__);
>     /* Don't touch classless devices or host bridges or IOAPICs */
>     if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
> dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
>         return;
>    ....
>
> What is your suggestion to make the solution generic? Create a
> whitelist? Remove this verification? I have no idea... nothing sounds
> good to me...
> 01:00.0 Ethernet controller: Device 1ac1:089a (prog-if ff)
>     Subsystem: Device 1ac1:089a
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR+ FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin A routed to IRQ 0
>     Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K]
>     Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M]
>     Capabilities: <access denied>
>
>
>
> Best regards.
> Luís Mendes
>
> On Thu, Apr 9, 2020 at 12:05 AM Luís Mendes <luis.p.mendes@gmail.com> wrote:
> >
> > Hi Bjorn,
> >
> > I have successfully setup a JTAG remote debug environment.
> > And I found this:
> > First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1,
> > which in turn calls __pci_bus_assign_resources which visits device
> > 1ac1:089a on that slot and calls:

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-09 15:25                                   ` Luís Mendes
  2020-04-09 15:29                                     ` Luís Mendes
@ 2020-04-09 16:30                                     ` Bjorn Helgaas
  2020-04-09 17:32                                       ` Luís Mendes
  2020-04-09 18:08                                       ` Bjorn Helgaas
  1 sibling, 2 replies; 25+ messages in thread
From: Bjorn Helgaas @ 2020-04-09 16:30 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote:
> Hi Bjorn,
> 
> I've good news. I've found the culprit and it is a pretty simple
> issue, however the good solution is not obvious to me.
> Can you help in finding the best way to patch this issue?
> 
> So first detailing the problem in file setup_bus.c there is this *if
> condition* to ignore resources from classless devices and so
> it is that this Google/Coral Edge TPU is a classless device with class 0xff:
> 
> static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> {
>     u16 class = dev->class >> 8;
> 
>        pci_info(dev, "%s\n", __func__);
>     /* Don't touch classless devices or host bridges or IOAPICs */
>     if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
>         return;
>    ....
> 
> So the one possible trivial, non generic, attempt that works is to do:
> static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> {
>     u16 class = dev->class >> 8;
> 
>        pci_info(dev, "%s\n", __func__);
>     /* Don't touch classless devices or host bridges or IOAPICs */
>     if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
> dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
>         return;
>    ....
> 
> What is your suggestion to make the solution generic? Create a
> whitelist? Remove this verification? I have no idea... nothing sounds
> good to me...

Good detective work, thanks for chasing this down!

I should have seen that check when adding the debug.  Guess I thought
"sort", hmmm, that just re-orders things without actually changing the
content.  But pdev_sort_resources() in fact *adds* resources to a
list, and if resources aren't on the list, we apparently don't assign
space for them.

In any event, I would first check to see if there's an Edge TPU
firmware update that might set the class code.

If not, we should probably add a quirk to override the class code,
similar to quirk_eisa_bridge(), fixup_rev1_53c810(),
fixup_ti816x_class(), quirk_tw686x_class().

Bjorn

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-09 16:30                                     ` Bjorn Helgaas
@ 2020-04-09 17:32                                       ` Luís Mendes
  2020-04-09 18:08                                       ` Bjorn Helgaas
  1 sibling, 0 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-09 17:32 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt

I have looked at their webpage and I see no signs of a firmware that
can be downloaded.
https://coral.ai/products/pcie-accelerator

If it is acceptable for you I'll provide a patch for such a quirk.

Thanks,
Luís

On Thu, Apr 9, 2020 at 5:30 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote:
> > Hi Bjorn,
> >
> > I've good news. I've found the culprit and it is a pretty simple
> > issue, however the good solution is not obvious to me.
> > Can you help in finding the best way to patch this issue?
> >
> > So first detailing the problem in file setup_bus.c there is this *if
> > condition* to ignore resources from classless devices and so
> > it is that this Google/Coral Edge TPU is a classless device with class 0xff:
> >
> > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > {
> >     u16 class = dev->class >> 8;
> >
> >        pci_info(dev, "%s\n", __func__);
> >     /* Don't touch classless devices or host bridges or IOAPICs */
> >     if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
> >         return;
> >    ....
> >
> > So the one possible trivial, non generic, attempt that works is to do:
> > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > {
> >     u16 class = dev->class >> 8;
> >
> >        pci_info(dev, "%s\n", __func__);
> >     /* Don't touch classless devices or host bridges or IOAPICs */
> >     if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
> > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
> >         return;
> >    ....
> >
> > What is your suggestion to make the solution generic? Create a
> > whitelist? Remove this verification? I have no idea... nothing sounds
> > good to me...
>
> Good detective work, thanks for chasing this down!
>
> I should have seen that check when adding the debug.  Guess I thought
> "sort", hmmm, that just re-orders things without actually changing the
> content.  But pdev_sort_resources() in fact *adds* resources to a
> list, and if resources aren't on the list, we apparently don't assign
> space for them.
>
> In any event, I would first check to see if there's an Edge TPU
> firmware update that might set the class code.
>
> If not, we should probably add a quirk to override the class code,
> similar to quirk_eisa_bridge(), fixup_rev1_53c810(),
> fixup_ti816x_class(), quirk_tw686x_class().
>
> Bjorn

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-09 16:30                                     ` Bjorn Helgaas
  2020-04-09 17:32                                       ` Luís Mendes
@ 2020-04-09 18:08                                       ` Bjorn Helgaas
  2020-04-09 20:07                                         ` Luís Mendes
  1 sibling, 1 reply; 25+ messages in thread
From: Bjorn Helgaas @ 2020-04-09 18:08 UTC (permalink / raw)
  To: Luís Mendes
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt, Todd Poynor

[+cc Todd]

On Thu, Apr 09, 2020 at 11:30:10AM -0500, Bjorn Helgaas wrote:
> On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote:
> > Hi Bjorn,
> > 
> > I've good news. I've found the culprit and it is a pretty simple
> > issue, however the good solution is not obvious to me.
> > Can you help in finding the best way to patch this issue?
> > 
> > So first detailing the problem in file setup_bus.c there is this *if
> > condition* to ignore resources from classless devices and so
> > it is that this Google/Coral Edge TPU is a classless device with class 0xff:
> > 
> > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > {
> >     u16 class = dev->class >> 8;
> > 
> >        pci_info(dev, "%s\n", __func__);
> >     /* Don't touch classless devices or host bridges or IOAPICs */
> >     if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
> >         return;
> >    ....
> > 
> > So the one possible trivial, non generic, attempt that works is to do:
> > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > {
> >     u16 class = dev->class >> 8;
> > 
> >        pci_info(dev, "%s\n", __func__);
> >     /* Don't touch classless devices or host bridges or IOAPICs */
> >     if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
> > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
> >         return;
> >    ....
> > 
> > What is your suggestion to make the solution generic? Create a
> > whitelist? Remove this verification? I have no idea... nothing sounds
> > good to me...
> 
> Good detective work, thanks for chasing this down!
> 
> I should have seen that check when adding the debug.  Guess I thought
> "sort", hmmm, that just re-orders things without actually changing the
> content.  But pdev_sort_resources() in fact *adds* resources to a
> list, and if resources aren't on the list, we apparently don't assign
> space for them.
> 
> In any event, I would first check to see if there's an Edge TPU
> firmware update that might set the class code.
> 
> If not, we should probably add a quirk to override the class code,
> similar to quirk_eisa_bridge(), fixup_rev1_53c810(),
> fixup_ti816x_class(), quirk_tw686x_class().

In fact, apex_pci_fixup_class() already exists!  But it's in
apex_driver.c.  Do you happen to have CONFIG_STAGING_APEX_DRIVER=m
(built as a module)?  If so, that quirk won't be run until the module
is loaded, and that happens long after resource assignment.

Building with CONFIG_STAGING_APEX_DRIVER=y (not =m) should be a
workaround.  But I think the real fix would be moving
apex_pci_fixup_class() from apex_driver.c to drivers/pci/quirks.c,
like the following.  Would you mind testing it?


commit 59f3165318b3 ("PCI: Move Apex Edge TPU class quirk to fix BAR assignment")
Author: Bjorn Helgaas <bhelgaas@google.com>
Date:   Thu Apr 9 12:43:45 2020 -0500

    PCI: Move Apex Edge TPU class quirk to fix BAR assignment
    
    Some Google Apex Edge TPU devices have a class code of 0
    (PCI_CLASS_NOT_DEFINED).  This prevents the PCI core from assigning
    resources for the Apex BARs because __dev_sort_resources() ignores
    classless devices, host bridges, and IOAPICs.
    
    On x86, firmware typically assigns those resources, so this was not a
    problem.  But on some architectures, firmware does *not* assign BARs, and
    since the PCI core didn't do it either, the Apex device didn't work
    correctly:
    
      apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
      apex 0000:01:00.0: error enabling PCI device
    
    f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") added a
    quirk to fix the class code, but it was in the apex driver, and if the
    driver was built as a module, it was too late to help.
    
    Move the quirk to the PCI core, where it will always run early enough that
    the PCI core will assign resources if necessary.
    
    Link: https://lore.kernel.org/r/CAEzXK1r0Er039iERnc2KJ4jn7ySNUOG9H=Ha8TD8XroVqiZjgg@mail.gmail.com
    Fixes: f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class")
    Reported-by: Luís Mendes <luis.p.mendes@gmail.com>
    Debugged-by: Luís Mendes <luis.p.mendes@gmail.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 28c9a2409c50..ca9ed5774eb1 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5567,3 +5567,10 @@ static void pci_fixup_no_d0_pme(struct pci_dev *dev)
 	dev->pme_support &= ~(PCI_PM_CAP_PME_D0 >> PCI_PM_CAP_PME_SHIFT);
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ASMEDIA, 0x2142, pci_fixup_no_d0_pme);
+
+static void apex_pci_fixup_class(struct pci_dev *pdev)
+{
+	pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class;
+}
+DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
+			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
diff --git a/drivers/staging/gasket/apex_driver.c b/drivers/staging/gasket/apex_driver.c
index 46199c8ca441..f12f81c8dd2f 100644
--- a/drivers/staging/gasket/apex_driver.c
+++ b/drivers/staging/gasket/apex_driver.c
@@ -570,13 +570,6 @@ static const struct pci_device_id apex_pci_ids[] = {
 	{ PCI_DEVICE(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID) }, { 0 }
 };
 
-static void apex_pci_fixup_class(struct pci_dev *pdev)
-{
-	pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class;
-}
-DECLARE_PCI_FIXUP_CLASS_HEADER(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID,
-			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
-
 static int apex_pci_probe(struct pci_dev *pci_dev,
 			  const struct pci_device_id *id)
 {

^ permalink raw reply related	[flat|nested] 25+ messages in thread

* Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
  2020-04-09 18:08                                       ` Bjorn Helgaas
@ 2020-04-09 20:07                                         ` Luís Mendes
  0 siblings, 0 replies; 25+ messages in thread
From: Luís Mendes @ 2020-04-09 20:07 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Nicholas Johnson, Linux PCI, Thomas Petazzoni, Jason Cooper,
	Benjamin Herrenschmidt, Todd Poynor

Hi Bjorn,

I've just tested with linux-next master branch 20200409.

You can add:
Tested-by: Luis Mendes <luis.p.mendes@gmail.com>

Thanks,
Luís

01:00.0 System peripheral: Device 1ac1:089a (prog-if ff)
    Subsystem: Device 1ac1:089a
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    Region 0: Memory at e8100000 (64-bit, prefetchable) [disabled] [size=16K]
    Region 2: Memory at e8000000 (64-bit, prefetchable) [disabled] [size=1M]
    Capabilities: <access denied>

On Thu, Apr 9, 2020 at 7:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Todd]
>
> On Thu, Apr 09, 2020 at 11:30:10AM -0500, Bjorn Helgaas wrote:
> > On Thu, Apr 09, 2020 at 04:25:40PM +0100, Luís Mendes wrote:
> > > Hi Bjorn,
> > >
> > > I've good news. I've found the culprit and it is a pretty simple
> > > issue, however the good solution is not obvious to me.
> > > Can you help in finding the best way to patch this issue?
> > >
> > > So first detailing the problem in file setup_bus.c there is this *if
> > > condition* to ignore resources from classless devices and so
> > > it is that this Google/Coral Edge TPU is a classless device with class 0xff:
> > >
> > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > > {
> > >     u16 class = dev->class >> 8;
> > >
> > >        pci_info(dev, "%s\n", __func__);
> > >     /* Don't touch classless devices or host bridges or IOAPICs */
> > >     if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
> > >         return;
> > >    ....
> > >
> > > So the one possible trivial, non generic, attempt that works is to do:
> > > static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
> > > {
> > >     u16 class = dev->class >> 8;
> > >
> > >        pci_info(dev, "%s\n", __func__);
> > >     /* Don't touch classless devices or host bridges or IOAPICs */
> > >     if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
> > > dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
> > >         return;
> > >    ....
> > >
> > > What is your suggestion to make the solution generic? Create a
> > > whitelist? Remove this verification? I have no idea... nothing sounds
> > > good to me...
> >
> > Good detective work, thanks for chasing this down!
> >
> > I should have seen that check when adding the debug.  Guess I thought
> > "sort", hmmm, that just re-orders things without actually changing the
> > content.  But pdev_sort_resources() in fact *adds* resources to a
> > list, and if resources aren't on the list, we apparently don't assign
> > space for them.
> >
> > In any event, I would first check to see if there's an Edge TPU
> > firmware update that might set the class code.
> >
> > If not, we should probably add a quirk to override the class code,
> > similar to quirk_eisa_bridge(), fixup_rev1_53c810(),
> > fixup_ti816x_class(), quirk_tw686x_class().
>
> In fact, apex_pci_fixup_class() already exists!  But it's in
> apex_driver.c.  Do you happen to have CONFIG_STAGING_APEX_DRIVER=m
> (built as a module)?  If so, that quirk won't be run until the module
> is loaded, and that happens long after resource assignment.
>
> Building with CONFIG_STAGING_APEX_DRIVER=y (not =m) should be a
> workaround.  But I think the real fix would be moving
> apex_pci_fixup_class() from apex_driver.c to drivers/pci/quirks.c,
> like the following.  Would you mind testing it?
>
>
> commit 59f3165318b3 ("PCI: Move Apex Edge TPU class quirk to fix BAR assignment")
> Author: Bjorn Helgaas <bhelgaas@google.com>
> Date:   Thu Apr 9 12:43:45 2020 -0500
>
>     PCI: Move Apex Edge TPU class quirk to fix BAR assignment
>
>     Some Google Apex Edge TPU devices have a class code of 0
>     (PCI_CLASS_NOT_DEFINED).  This prevents the PCI core from assigning
>     resources for the Apex BARs because __dev_sort_resources() ignores
>     classless devices, host bridges, and IOAPICs.
>
>     On x86, firmware typically assigns those resources, so this was not a
>     problem.  But on some architectures, firmware does *not* assign BARs, and
>     since the PCI core didn't do it either, the Apex device didn't work
>     correctly:
>
>       apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
>       apex 0000:01:00.0: error enabling PCI device
>
>     f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class") added a
>     quirk to fix the class code, but it was in the apex driver, and if the
>     driver was built as a module, it was too late to help.
>
>     Move the quirk to the PCI core, where it will always run early enough that
>     the PCI core will assign resources if necessary.
>
>     Link: https://lore.kernel.org/r/CAEzXK1r0Er039iERnc2KJ4jn7ySNUOG9H=Ha8TD8XroVqiZjgg@mail.gmail.com
>     Fixes: f390d08d8b87 ("staging: gasket: apex: fixup undefined PCI class")
>     Reported-by: Luís Mendes <luis.p.mendes@gmail.com>
>     Debugged-by: Luís Mendes <luis.p.mendes@gmail.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 28c9a2409c50..ca9ed5774eb1 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -5567,3 +5567,10 @@ static void pci_fixup_no_d0_pme(struct pci_dev *dev)
>         dev->pme_support &= ~(PCI_PM_CAP_PME_D0 >> PCI_PM_CAP_PME_SHIFT);
>  }
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ASMEDIA, 0x2142, pci_fixup_no_d0_pme);
> +
> +static void apex_pci_fixup_class(struct pci_dev *pdev)
> +{
> +       pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class;
> +}
> +DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
> +                              PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
> diff --git a/drivers/staging/gasket/apex_driver.c b/drivers/staging/gasket/apex_driver.c
> index 46199c8ca441..f12f81c8dd2f 100644
> --- a/drivers/staging/gasket/apex_driver.c
> +++ b/drivers/staging/gasket/apex_driver.c
> @@ -570,13 +570,6 @@ static const struct pci_device_id apex_pci_ids[] = {
>         { PCI_DEVICE(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID) }, { 0 }
>  };
>
> -static void apex_pci_fixup_class(struct pci_dev *pdev)
> -{
> -       pdev->class = (PCI_CLASS_SYSTEM_OTHER << 8) | pdev->class;
> -}
> -DECLARE_PCI_FIXUP_CLASS_HEADER(APEX_PCI_VENDOR_ID, APEX_PCI_DEVICE_ID,
> -                              PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
> -
>  static int apex_pci_probe(struct pci_dev *pci_dev,
>                           const struct pci_device_id *id)
>  {

^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2020-04-09 20:08 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-06 14:32 Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux Luís Mendes
2020-03-06 21:47 ` Bjorn Helgaas
2020-03-07 12:11   ` Luís Mendes
2020-03-07 15:26     ` Luís Mendes
2020-03-07 21:38       ` Bjorn Helgaas
2020-03-08  5:51         ` Nicholas Johnson
2020-03-09 11:21           ` Luís Mendes
2020-03-11 14:20             ` Luís Mendes
2020-03-29 22:11               ` Luís Mendes
2020-03-30 19:49                 ` Bjorn Helgaas
2020-03-31 21:28                   ` Luís Mendes
2020-04-01 18:16                     ` Bjorn Helgaas
2020-04-01 21:20                       ` Luís Mendes
2020-04-01 21:55                         ` Luís Mendes
2020-04-01 23:31                         ` Bjorn Helgaas
2020-04-02 14:13                           ` Luís Mendes
2020-04-04  1:32                             ` Bjorn Helgaas
2020-04-04 21:39                               ` Luís Mendes
2020-04-08 23:05                                 ` Luís Mendes
2020-04-09 15:25                                   ` Luís Mendes
2020-04-09 15:29                                     ` Luís Mendes
2020-04-09 16:30                                     ` Bjorn Helgaas
2020-04-09 17:32                                       ` Luís Mendes
2020-04-09 18:08                                       ` Bjorn Helgaas
2020-04-09 20:07                                         ` Luís Mendes

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).