PCIe end-point on FPGA doesn't show up on PCI bus when configured

PCIe end-point on FPGA doesn't show up on PCI bus when configured

[Matias Garcia]

I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-core 
processor, and have the following conundrum: I configure the FPGA which 
brings up a PCIe interface to the processor. I scan both PCI buses on 
the system (I believe the second bus is behind the Freescale integrated 
bridge on the first), and it doesn't show up. I initiate a reset on the 
processor, and both U-boot and Linux now see the FPGA PCI device at 
0000:01:00.00. I've noticed some of the memory mappings in the PCI 
bridge windows are different between the two boot sequences. I've tried 
all manner of pci calls (including the pcibios_fixup routines) on the 
bridge device (including removing and re-scanning it), and on bus 1, 
which is otherwise empty, to no avail. Following are some debug listings 
from dmesg; any help/ideas in tracking down the problem (hardware or 
software) is greatly appreciated.

#Boot without FPGA configured:
<snip>
Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number: 0->255
PCI host bridge /pcie@8ff70a000  ranges:
  MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
   IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
/pcie@8ff70a000: PCICSRBAR @ 0xfff00000
/pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map. 
Adjusting the memory map could reduce unnecessary bounce buffering.
/pcie@8ff70a000: DMA window size is 0x80000000
MPC85xx RDB board from Freescale Semiconductor
<...>
PCI: Probing PCI hardware
pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
pci 0000:00:00.0: supports D1 D2
pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:00:00.0: PME# disabled
pci 0000:00:00.0: PCI bridge to [bus 01-ff]
pci 0000:00:00.0:   bridge window [io  0x0000-0x0000] (disabled)
pci 0000:00:00.0:   bridge window [mem 0x00000000-0x000fffff] (disabled)
pci 0000:00:00.0:   bridge window [mem 0x00000000-0x000fffff pref] 
(disabled)
PCI 0000:00 Cannot reserve Legacy IO [io  0xffbed000-0xffbedfff]
pci 0000:00:00.0: PCI bridge to [bus 01-01]
pci 0000:00:00.0:   bridge window [io  0xffbed000-0xffbfcfff]
pci 0000:00:00.0:   bridge window [mem 0x880000000-0x88fffffff]
pci 0000:00:00.0:   bridge window [mem pref disabled]
pci 0000:00:00.0: enabling device (0106 -> 0107)
pci_bus 0000:00: resource 0 [io  0xffbed000-0xffbfcfff]
pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
pci_bus 0000:01: resource 0 [io  0xffbed000-0xffbfcfff]
pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]

#Reset with FPGA configured:
<snip>
Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number: 0->255
PCI host bridge /pcie@8ff70a000  ranges:
  MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
   IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
/pcie@8ff70a000: PCICSRBAR @ 0xfff00000
/pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map. 
Adjusting the memory map could reduce unnecessary bounce buffering.
/pcie@8ff70a000: DMA window size is 0x80000000
MPC85xx RDB board from Freescale Semiconductor
<...>
PCI: Probing PCI hardware
pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
pci 0000:00:00.0: supports D1 D2
pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:00:00.0: PME# disabled
pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
pci 0000:00:00.0: PCI bridge to [bus 01-ff]
pci 0000:00:00.0:   bridge window [io  0x0000-0x0000] (disabled)
pci 0000:00:00.0:   bridge window [mem 0x80000000-0x82ffffff]
pci 0000:00:00.0:   bridge window [mem 0x10000000-0x000fffff pref] 
(disabled)
irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
PCI 0000:00 Cannot reserve Legacy IO [io  0xffbed000-0xffbedfff]
pci 0000:00:00.0: PCI bridge to [bus 01-01]
pci 0000:00:00.0:   bridge window [io  0xffbed000-0xffbfcfff]
pci 0000:00:00.0:   bridge window [mem 0x880000000-0x88fffffff]
pci 0000:00:00.0:   bridge window [mem pref disabled]
pci 0000:00:00.0: enabling device (0106 -> 0107)
pci_bus 0000:00: resource 0 [io  0xffbed000-0xffbfcfff]
pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
pci_bus 0000:01: resource 0 [io  0xffbed000-0xffbfcfff]
pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]

Cheers,
Matias

-- 
	*Matias Garcia*
/Embedded Software Developer/
Ross Video | Live Production Technology
www.rossvideo.com <http://www.rossvideo.com>
+1 (613) 228 1198 x4264

Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[Elie De Brauwer]

On 01/28/11 19:37, Matias Garcia wrote:
> I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-core
> processor, and have the following conundrum: I configure the FPGA which
> brings up a PCIe interface to the processor. I scan both PCI buses on
> the system (I believe the second bus is behind the Freescale integrated
> bridge on the first), and it doesn't show up. I initiate a reset on the
> processor, and both U-boot and Linux now see the FPGA PCI device at
> 0000:01:00.00. I've noticed some of the memory mappings in the PCI
> bridge windows are different between the two boot sequences. I've tried
> all manner of pci calls (including the pcibios_fixup routines) on the
> bridge device (including removing and re-scanning it), and on bus 1,
> which is otherwise empty, to no avail. Following are some debug listings
> from dmesg; any help/ideas in tracking down the problem (hardware or
> software) is greatly appreciated.
>
> #Boot without FPGA configured:
> <snip>
> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
> 0->255
> PCI host bridge /pcie@8ff70a000 ranges:
> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
> Adjusting the memory map could reduce unnecessary bounce buffering.
> /pcie@8ff70a000: DMA window size is 0x80000000
> MPC85xx RDB board from Freescale Semiconductor
> <...>
> PCI: Probing PCI hardware
> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
> pci 0000:00:00.0: supports D1 D2
> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> pci 0000:00:00.0: PME# disabled
> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff] (disabled)
> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff pref] (disabled)
> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
> pci 0000:00:00.0: PCI bridge to [bus 01-01]
> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
> pci 0000:00:00.0: bridge window [mem pref disabled]
> pci 0000:00:00.0: enabling device (0106 -> 0107)
> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>
> #Reset with FPGA configured:
> <snip>
> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
> 0->255
> PCI host bridge /pcie@8ff70a000 ranges:
> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
> Adjusting the memory map could reduce unnecessary bounce buffering.
> /pcie@8ff70a000: DMA window size is 0x80000000
> MPC85xx RDB board from Freescale Semiconductor
> <...>
> PCI: Probing PCI hardware
> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
> pci 0000:00:00.0: supports D1 D2
> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> pci 0000:00:00.0: PME# disabled
> pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
> pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
> pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
> pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
> pci 0000:00:00.0: bridge window [mem 0x80000000-0x82ffffff]
> pci 0000:00:00.0: bridge window [mem 0x10000000-0x000fffff pref] (disabled)
> irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
> pci 0000:00:00.0: PCI bridge to [bus 01-01]
> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
> pci 0000:00:00.0: bridge window [mem pref disabled]
> pci 0000:00:00.0: enabling device (0106 -> 0107)
> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]


Hi Mattias,

I'm doing the same on a similar setup, also a P2020 but a 2.6.36 and 
with me it works just fine. However I encountered one problem. I 
understand it as follows, if there is no physical PCIe link then 
somewhere a flag PPC_INDIRECT_TYPE_NO_PCIE_LINK gets set. This has as 
result that reading the PCIe config space will fail with a 
PCIBIOS_DEVICE_NOT_FOUND (ref 
http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/sysdev/indirect_pci.c#L24 )


At 
http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/include/asm/pci-bridge.h#L105 
they specify this as a workaround since the PCIe might hang if there is 
no physical link. So my workaround for this issue was:

- load the fpga
- travel down the pci bus to the correct bus where the fpga is attached 
  use a pci_bus_to_host() to obtain a struct pci_controller, unset the 
PPC_INDIRECT_TYPE_NO_PCIE_LINK  and call a pci_rescon_bus() on that bus.

After doing this I can find access the FPGA, and reload it if needed. 
Not a clue if this is 'the proper way' to do it, but it works for me.

gr
E.

-- 
Elie De Brauwer

Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[tiejun.chen]

Elie De Brauwer wrote:
> On 01/28/11 19:37, Matias Garcia wrote:
>> I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-core
>> processor, and have the following conundrum: I configure the FPGA which
>> brings up a PCIe interface to the processor. I scan both PCI buses on
>> the system (I believe the second bus is behind the Freescale integrated
>> bridge on the first), and it doesn't show up. I initiate a reset on the
>> processor, and both U-boot and Linux now see the FPGA PCI device at
>> 0000:01:00.00. I've noticed some of the memory mappings in the PCI
>> bridge windows are different between the two boot sequences. I've tried
>> all manner of pci calls (including the pcibios_fixup routines) on the
>> bridge device (including removing and re-scanning it), and on bus 1,
>> which is otherwise empty, to no avail. Following are some debug listings
>> from dmesg; any help/ideas in tracking down the problem (hardware or
>> software) is greatly appreciated.
>>
>> #Boot without FPGA configured:
>> <snip>
>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>> 0->255
>> PCI host bridge /pcie@8ff70a000 ranges:
>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
>> Adjusting the memory map could reduce unnecessary bounce buffering.
>> /pcie@8ff70a000: DMA window size is 0x80000000
>> MPC85xx RDB board from Freescale Semiconductor
>> <...>
>> PCI: Probing PCI hardware
>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>> pci 0000:00:00.0: supports D1 D2
>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>> pci 0000:00:00.0: PME# disabled
>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff] (disabled)
>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff pref]
>> (disabled)
>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>> pci 0000:00:00.0: bridge window [mem pref disabled]
>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>>
>> #Reset with FPGA configured:
>> <snip>
>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>> 0->255
>> PCI host bridge /pcie@8ff70a000 ranges:
>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
>> Adjusting the memory map could reduce unnecessary bounce buffering.
>> /pcie@8ff70a000: DMA window size is 0x80000000
>> MPC85xx RDB board from Freescale Semiconductor
>> <...>
>> PCI: Probing PCI hardware
>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>> pci 0000:00:00.0: supports D1 D2
>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>> pci 0000:00:00.0: PME# disabled
>> pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
>> pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
>> pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
>> pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>> pci 0000:00:00.0: bridge window [mem 0x80000000-0x82ffffff]
>> pci 0000:00:00.0: bridge window [mem 0x10000000-0x000fffff pref]
>> (disabled)
>> irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>> pci 0000:00:00.0: bridge window [mem pref disabled]
>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
> 
> 
> Hi Mattias,
> 
> I'm doing the same on a similar setup, also a P2020 but a 2.6.36 and
> with me it works just fine. However I encountered one problem. I
> understand it as follows, if there is no physical PCIe link then
> somewhere a flag PPC_INDIRECT_TYPE_NO_PCIE_LINK gets set. This has as
> result that reading the PCIe config space will fail with a
> PCIBIOS_DEVICE_NOT_FOUND (ref
> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/sysdev/indirect_pci.c#L24 )
> 
> 
> At
> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/include/asm/pci-bridge.h#L105
> they specify this as a workaround since the PCIe might hang if there is
> no physical link. So my workaround for this issue was:
> 
> - load the fpga
> - travel down the pci bus to the correct bus where the fpga is attached
>  use a pci_bus_to_host() to obtain a struct pci_controller, unset the
> PPC_INDIRECT_TYPE_NO_PCIE_LINK  and call a pci_rescon_bus() on that bus.
> 
> After doing this I can find access the FPGA, and reload it if needed.
> Not a clue if this is 'the proper way' to do it, but it works for me.

Looks this may be really related to the PCIe Link Training. So you have to reset
the PCIe after load the FPGA, but I think we should do this in the u-boot. For
more detail on this please refer to the code segments defined by
CONFIG_FSL_PCIE_RESET in the file, drivers/pci/fsl_pci_init.c.

Tiejun

> 
> gr
> E.
> 

Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[Stijn Devriendt]

As far as I know, you're violating PCIe spec.
PCIe base spec (rev1.0a) states that a device must start link training
within 80ms
after a fundamental reset and that each device must be ready to accept conf=
ig
requests within 100ms after fundamental reset.

Regards,
Stijn

On Sun, Jan 30, 2011 at 4:07 AM, tiejun.chen <tiejun.chen@windriver.com> wr=
ote:
> Elie De Brauwer wrote:
>> On 01/28/11 19:37, Matias Garcia wrote:
>>> I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-cor=
e
>>> processor, and have the following conundrum: I configure the FPGA which
>>> brings up a PCIe interface to the processor. I scan both PCI buses on
>>> the system (I believe the second bus is behind the Freescale integrated
>>> bridge on the first), and it doesn't show up. I initiate a reset on the
>>> processor, and both U-boot and Linux now see the FPGA PCI device at
>>> 0000:01:00.00. I've noticed some of the memory mappings in the PCI
>>> bridge windows are different between the two boot sequences. I've tried
>>> all manner of pci calls (including the pcibios_fixup routines) on the
>>> bridge device (including removing and re-scanning it), and on bus 1,
>>> which is otherwise empty, to no avail. Following are some debug listing=
s
>>> from dmesg; any help/ideas in tracking down the problem (hardware or
>>> software) is greatly appreciated.
>>>
>>> #Boot without FPGA configured:
>>> <snip>
>>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>>> 0->255
>>> PCI host bridge /pcie@8ff70a000 ranges:
>>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map=
.
>>> Adjusting the memory map could reduce unnecessary bounce buffering.
>>> /pcie@8ff70a000: DMA window size is 0x80000000
>>> MPC85xx RDB board from Freescale Semiconductor
>>> <...>
>>> PCI: Probing PCI hardware
>>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>>> pci 0000:00:00.0: supports D1 D2
>>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>>> pci 0000:00:00.0: PME# disabled
>>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff] (disabled)
>>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff pref]
>>> (disabled)
>>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>>> pci 0000:00:00.0: bridge window [mem pref disabled]
>>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>>>
>>> #Reset with FPGA configured:
>>> <snip>
>>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>>> 0->255
>>> PCI host bridge /pcie@8ff70a000 ranges:
>>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map=
.
>>> Adjusting the memory map could reduce unnecessary bounce buffering.
>>> /pcie@8ff70a000: DMA window size is 0x80000000
>>> MPC85xx RDB board from Freescale Semiconductor
>>> <...>
>>> PCI: Probing PCI hardware
>>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>>> pci 0000:00:00.0: supports D1 D2
>>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>>> pci 0000:00:00.0: PME# disabled
>>> pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
>>> pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
>>> pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
>>> pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
>>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>>> pci 0000:00:00.0: bridge window [mem 0x80000000-0x82ffffff]
>>> pci 0000:00:00.0: bridge window [mem 0x10000000-0x000fffff pref]
>>> (disabled)
>>> irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
>>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>>> pci 0000:00:00.0: bridge window [mem pref disabled]
>>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>>
>>
>> Hi Mattias,
>>
>> I'm doing the same on a similar setup, also a P2020 but a 2.6.36 and
>> with me it works just fine. However I encountered one problem. I
>> understand it as follows, if there is no physical PCIe link then
>> somewhere a flag PPC_INDIRECT_TYPE_NO_PCIE_LINK gets set. This has as
>> result that reading the PCIe config space will fail with a
>> PCIBIOS_DEVICE_NOT_FOUND (ref
>> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/sysdev/indirect_pci.c#L2=
4 )
>>
>>
>> At
>> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/include/asm/pci-bridge.h=
#L105
>> they specify this as a workaround since the PCIe might hang if there is
>> no physical link. So my workaround for this issue was:
>>
>> - load the fpga
>> - travel down the pci bus to the correct bus where the fpga is attached
>> =A0use a pci_bus_to_host() to obtain a struct pci_controller, unset the
>> PPC_INDIRECT_TYPE_NO_PCIE_LINK =A0and call a pci_rescon_bus() on that bu=
s.
>>
>> After doing this I can find access the FPGA, and reload it if needed.
>> Not a clue if this is 'the proper way' to do it, but it works for me.
>
> Looks this may be really related to the PCIe Link Training. So you have t=
o reset
> the PCIe after load the FPGA, but I think we should do this in the u-boot=
. For
> more detail on this please refer to the code segments defined by
> CONFIG_FSL_PCIE_RESET in the file, drivers/pci/fsl_pci_init.c.
>
> Tiejun
>
>>
>> gr
>> E.
>>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>

Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[tiejun.chen]

Stijn Devriendt wrote:
> As far as I know, you're violating PCIe spec.
> PCIe base spec (rev1.0a) states that a device must start link training
> within 80ms
> after a fundamental reset and that each device must be ready to accept config
> requests within 100ms after fundamental reset.

Nope.

>From this scenario I only doubts this problem is issued like some Freescale PCIe
errata. From that chip errata you can find this easily, "This sequence resets
the PCI Express controllers only.", and so these codes are used definitely on
u-boot.

Tiejun

> 
> Regards,
> Stijn
> 
> On Sun, Jan 30, 2011 at 4:07 AM, tiejun.chen <tiejun.chen@windriver.com> wrote:
>> Elie De Brauwer wrote:
>>> On 01/28/11 19:37, Matias Garcia wrote:
>>>> I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-core
>>>> processor, and have the following conundrum: I configure the FPGA which
>>>> brings up a PCIe interface to the processor. I scan both PCI buses on
>>>> the system (I believe the second bus is behind the Freescale integrated
>>>> bridge on the first), and it doesn't show up. I initiate a reset on the
>>>> processor, and both U-boot and Linux now see the FPGA PCI device at
>>>> 0000:01:00.00. I've noticed some of the memory mappings in the PCI
>>>> bridge windows are different between the two boot sequences. I've tried
>>>> all manner of pci calls (including the pcibios_fixup routines) on the
>>>> bridge device (including removing and re-scanning it), and on bus 1,
>>>> which is otherwise empty, to no avail. Following are some debug listings
>>>> from dmesg; any help/ideas in tracking down the problem (hardware or
>>>> software) is greatly appreciated.
>>>>
>>>> #Boot without FPGA configured:
>>>> <snip>
>>>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>>>> 0->255
>>>> PCI host bridge /pcie@8ff70a000 ranges:
>>>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>>>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>>>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>>>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
>>>> Adjusting the memory map could reduce unnecessary bounce buffering.
>>>> /pcie@8ff70a000: DMA window size is 0x80000000
>>>> MPC85xx RDB board from Freescale Semiconductor
>>>> <...>
>>>> PCI: Probing PCI hardware
>>>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>>>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>>>> pci 0000:00:00.0: supports D1 D2
>>>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>>>> pci 0000:00:00.0: PME# disabled
>>>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>>>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>>>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff] (disabled)
>>>> pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff pref]
>>>> (disabled)
>>>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>>>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>>>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>>>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>>>> pci 0000:00:00.0: bridge window [mem pref disabled]
>>>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>>>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>>>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>>>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>>>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>>>>
>>>> #Reset with FPGA configured:
>>>> <snip>
>>>> Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
>>>> 0->255
>>>> PCI host bridge /pcie@8ff70a000 ranges:
>>>> MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
>>>> IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
>>>> /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
>>>> /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
>>>> Adjusting the memory map could reduce unnecessary bounce buffering.
>>>> /pcie@8ff70a000: DMA window size is 0x80000000
>>>> MPC85xx RDB board from Freescale Semiconductor
>>>> <...>
>>>> PCI: Probing PCI hardware
>>>> pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
>>>> pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
>>>> pci 0000:00:00.0: supports D1 D2
>>>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>>>> pci 0000:00:00.0: PME# disabled
>>>> pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
>>>> pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
>>>> pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
>>>> pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
>>>> pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>>>> pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
>>>> pci 0000:00:00.0: bridge window [mem 0x80000000-0x82ffffff]
>>>> pci 0000:00:00.0: bridge window [mem 0x10000000-0x000fffff pref]
>>>> (disabled)
>>>> irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
>>>> PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
>>>> pci 0000:00:00.0: PCI bridge to [bus 01-01]
>>>> pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
>>>> pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
>>>> pci 0000:00:00.0: bridge window [mem pref disabled]
>>>> pci 0000:00:00.0: enabling device (0106 -> 0107)
>>>> pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
>>>> pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
>>>> pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
>>>> pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
>>>
>>> Hi Mattias,
>>>
>>> I'm doing the same on a similar setup, also a P2020 but a 2.6.36 and
>>> with me it works just fine. However I encountered one problem. I
>>> understand it as follows, if there is no physical PCIe link then
>>> somewhere a flag PPC_INDIRECT_TYPE_NO_PCIE_LINK gets set. This has as
>>> result that reading the PCIe config space will fail with a
>>> PCIBIOS_DEVICE_NOT_FOUND (ref
>>> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/sysdev/indirect_pci.c#L24 )
>>>
>>>
>>> At
>>> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/include/asm/pci-bridge.h#L105
>>> they specify this as a workaround since the PCIe might hang if there is
>>> no physical link. So my workaround for this issue was:
>>>
>>> - load the fpga
>>> - travel down the pci bus to the correct bus where the fpga is attached
>>> �use a pci_bus_to_host() to obtain a struct pci_controller, unset the
>>> PPC_INDIRECT_TYPE_NO_PCIE_LINK �and call a pci_rescon_bus() on that bus.
>>>
>>> After doing this I can find access the FPGA, and reload it if needed.
>>> Not a clue if this is 'the proper way' to do it, but it works for me.
>> Looks this may be really related to the PCIe Link Training. So you have to reset
>> the PCIe after load the FPGA, but I think we should do this in the u-boot. For
>> more detail on this please refer to the code segments defined by
>> CONFIG_FSL_PCIE_RESET in the file, drivers/pci/fsl_pci_init.c.
>>
>> Tiejun
>>
>>> gr
>>> E.

RFC [PATCH] fsl pci quirk to __devinit Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[Matias Garcia]

On Fri, 2011-01-28 at 21:06 +0100, Elie De Brauwer wrote:
> On 01/28/11 19:37, Matias Garcia wrote:
> > I'm running a vanilla linux 2.6.37 kernel on a Freescale P2020 dual-core
> > processor, and have the following conundrum: I configure the FPGA which
> > brings up a PCIe interface to the processor. I scan both PCI buses on
> > the system (I believe the second bus is behind the Freescale integrated
> > bridge on the first), and it doesn't show up. I initiate a reset on the
> > processor, and both U-boot and Linux now see the FPGA PCI device at
> > 0000:01:00.00. I've noticed some of the memory mappings in the PCI
> > bridge windows are different between the two boot sequences. I've tried
> > all manner of pci calls (including the pcibios_fixup routines) on the
> > bridge device (including removing and re-scanning it), and on bus 1,
> > which is otherwise empty, to no avail. Following are some debug listings
> > from dmesg; any help/ideas in tracking down the problem (hardware or
> > software) is greatly appreciated.
> >
> > #Boot without FPGA configured:
> > <snip>
> > Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
> > 0->255
> > PCI host bridge /pcie@8ff70a000 ranges:
> > MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
> > IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
> > /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
> > /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
> > Adjusting the memory map could reduce unnecessary bounce buffering.
> > /pcie@8ff70a000: DMA window size is 0x80000000
> > MPC85xx RDB board from Freescale Semiconductor
> > <...>
> > PCI: Probing PCI hardware
> > pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
> > pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
> > pci 0000:00:00.0: supports D1 D2
> > pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > pci 0000:00:00.0: PME# disabled
> > pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
> > pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff] (disabled)
> > pci 0000:00:00.0: bridge window [mem 0x00000000-0x000fffff pref] (disabled)
> > PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
> > pci 0000:00:00.0: PCI bridge to [bus 01-01]
> > pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
> > pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
> > pci 0000:00:00.0: bridge window [mem pref disabled]
> > pci 0000:00:00.0: enabling device (0106 -> 0107)
> > pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
> > pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
> > pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
> > pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
> >
> > #Reset with FPGA configured:
> > <snip>
> > Found FSL PCI host bridge at 0x00000008ff70a000. Firmware bus number:
> > 0->255
> > PCI host bridge /pcie@8ff70a000 ranges:
> > MEM 0x0000000880000000..0x000000088fffffff -> 0x0000000080000000
> > IO 0x00000008a0000000..0x00000008a000ffff -> 0x0000000000000000
> > /pcie@8ff70a000: PCICSRBAR @ 0xfff00000
> > /pcie@8ff70a000: WARNING: Outbound window cfg leaves gaps in memory map.
> > Adjusting the memory map could reduce unnecessary bounce buffering.
> > /pcie@8ff70a000: DMA window size is 0x80000000
> > MPC85xx RDB board from Freescale Semiconductor
> > <...>
> > PCI: Probing PCI hardware
> > pci 0000:00:00.0: [1957:0070] type 1 class 0x000b20
> > pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
> > pci 0000:00:00.0: supports D1 D2
> > pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> > pci 0000:00:00.0: PME# disabled
> > pci 0000:01:00.0: [1172:0004] type 0 class 0x001000
> > pci 0000:01:00.0: reg 10: [mem 0x80000000-0x80ffffff]
> > pci 0000:01:00.0: reg 14: [mem 0x81000000-0x81ffffff]
> > pci 0000:01:00.0: reg 18: [mem 0x82000000-0x82ffffff]
> > pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > pci 0000:00:00.0: bridge window [io 0x0000-0x0000] (disabled)
> > pci 0000:00:00.0: bridge window [mem 0x80000000-0x82ffffff]
> > pci 0000:00:00.0: bridge window [mem 0x10000000-0x000fffff pref] (disabled)
> > irq: irq 0 on host /soc@8ff700000/pic@40000 mapped to virtual irq 16
> > PCI 0000:00 Cannot reserve Legacy IO [io 0xffbed000-0xffbedfff]
> > pci 0000:00:00.0: PCI bridge to [bus 01-01]
> > pci 0000:00:00.0: bridge window [io 0xffbed000-0xffbfcfff]
> > pci 0000:00:00.0: bridge window [mem 0x880000000-0x88fffffff]
> > pci 0000:00:00.0: bridge window [mem pref disabled]
> > pci 0000:00:00.0: enabling device (0106 -> 0107)
> > pci_bus 0000:00: resource 0 [io 0xffbed000-0xffbfcfff]
> > pci_bus 0000:00: resource 1 [mem 0x880000000-0x88fffffff]
> > pci_bus 0000:01: resource 0 [io 0xffbed000-0xffbfcfff]
> > pci_bus 0000:01: resource 1 [mem 0x880000000-0x88fffffff]
> 
> 
> Hi Mattias,
> 
> I'm doing the same on a similar setup, also a P2020 but a 2.6.36 and 
> with me it works just fine. However I encountered one problem. I 
> understand it as follows, if there is no physical PCIe link then 
> somewhere a flag PPC_INDIRECT_TYPE_NO_PCIE_LINK gets set. This has as 
> result that reading the PCIe config space will fail with a 
> PCIBIOS_DEVICE_NOT_FOUND (ref 
> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/sysdev/indirect_pci.c#L24 )
> 
> 
> At 
> http://lxr.linux.no/#linux+v2.6.37/arch/powerpc/include/asm/pci-bridge.h#L105 
> they specify this as a workaround since the PCIe might hang if there is 
> no physical link. So my workaround for this issue was:
> 
> - load the fpga
> - travel down the pci bus to the correct bus where the fpga is attached 
>   use a pci_bus_to_host() to obtain a struct pci_controller, unset the 
> PPC_INDIRECT_TYPE_NO_PCIE_LINK  and call a pci_rescon_bus() on that bus.
> 
> After doing this I can find access the FPGA, and reload it if needed. 
> Not a clue if this is 'the proper way' to do it, but it works for me.
> 
> gr
> E.

Elie et al,

Thanks again for the find. I've been using this method successfully
until now (programming the FPGA from U-Boot and resetting the PCIe
controller as Tiejun suggested was not practical). If anyone has found a
less weird solution, I'd love to hear it. That controller just doesn't
like booting without an end-point on the bus.

We started seeing intermittent failures at the call to
quirk_fsl_pcie_header, particularly on one unit. I finally clued in that
it might be called after initialization with an __init tag. Would it be
uncouth to ask that it be changed to __devinit? I gather it's only
supposed to be called once, but in my case, it gets called more than
once when the controller is re-added in my fpga loader.

Here's the patch against 2.6.37:

Change quirk_fsl_pcie_header from __init to __devinit.

Signed-off-by: Matias Garcia <mgarcia@rossvideo.com>
---
diff --git a/arch/powerpc/sysdev/fsl_pci.c
b/arch/powerpc/sysdev/fsl_pci.c
index 818f7c6..8807d77 100644
--- a/arch/powerpc/sysdev/fsl_pci.c
+++ b/arch/powerpc/sysdev/fsl_pci.c
@@ -36,7 +36,7 @@
 
 static int fsl_pcie_bus_fixup, is_mpc83xx_pci;
 
-static void __init quirk_fsl_pcie_header(struct pci_dev *dev)
+static void __devinit quirk_fsl_pcie_header(struct pci_dev *dev)
 {
        /* if we aren't a PCIe don't bother */
        if (!pci_find_capability(dev, PCI_CAP_ID_EXP))

Re: RFC [PATCH] fsl pci quirk to __devinit Re: PCIe end-point on FPGA doesn't show up on PCI bus when configured

[Kumar Gala]

On Sep 19, 2011, at 10:35 AM, Matias Garcia wrote:

> 
> Here's the patch against 2.6.37:
> 
> Change quirk_fsl_pcie_header from __init to __devinit.
> 
> Signed-off-by: Matias Garcia <mgarcia@rossvideo.com>

applied

- k