Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Fawad Lateef]

Hello,

I am trying to figure-out an issue on our i.MX6Q platform based design
where PCIe interface goes bad.

We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
board. PCIe root-complex from i.MX6Q is attached to PLX switch
(PEX8605).

Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
in config.

PLX switch #PERST is attached to a GPIO pin and stays in disable state
until Linux is booted. So at boot time only PCIe root-complex is
initialized by kernel.

After boot if I do "lspci -v"  and see everything good from PCIe
root-complex (below):

~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 295
Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport


Then I enable the #PERST pin of PLX switch, everything is still good
(no rescan on Linux is done yet)

~ # echo 139 > /sys/class/gpio/export
~ # echo out > /sys/class/gpio/gpio139/direction
~ # echo 1 > /sys/class/gpio/gpio139/value
~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 295
Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport


Now just disable/put-in-reset the PLX switch (Linux don't see the
switch yet, as no rescan on PCIe was done). Now "lspci -v" and
root-complex goes bad.

~ # echo 0 > /sys/class/gpio/gpio139/value
~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: fast devsel, IRQ 295
Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 00000000-000fffff [size=1M]
Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport

~ # uname -a
Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
UTC 2020 armv7l GNU/Linux
~ #


I am really not sure what is going wrong here. Did I am missing
something basic?

Thanks in advance,

-- Fawad Lateef

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Fawad Lateef]

Hi again,

Can someone guide me what is going on?

Thanks,

-- Fawad Lateef

On Sat, 22 Feb 2020 at 16:25, Fawad Lateef <fawadlateef@gmail.com> wrote:
>
> Hello,
>
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
>
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
>
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> in config.
>
> PLX switch #PERST is attached to a GPIO pin and stays in disable state
> until Linux is booted. So at boot time only PCIe root-complex is
> initialized by kernel.
>
> After boot if I do "lspci -v"  and see everything good from PCIe
> root-complex (below):
>
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
>
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Now just disable/put-in-reset the PLX switch (Linux don't see the
> switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> root-complex goes bad.
>
> ~ # echo 0 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: fast devsel, IRQ 295
> Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> I/O behind bridge: 00000000-00000fff [size=4K]
> Memory behind bridge: 00000000-000fffff [size=1M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
> ~ # uname -a
> Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> UTC 2020 armv7l GNU/Linux
> ~ #
>
>
> I am really not sure what is going wrong here. Did I am missing
> something basic?
>
> Thanks in advance,
>
> -- Fawad Lateef

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Bjorn Helgaas]

On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> Hello,
> 
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
> 
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
> 
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> in config.
> 
> PLX switch #PERST is attached to a GPIO pin and stays in disable state
> until Linux is booted. So at boot time only PCIe root-complex is
> initialized by kernel.
> 
> After boot if I do "lspci -v"  and see everything good from PCIe
> root-complex (below):
> 
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
> 
> 
> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
> 
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
> 
> 
> Now just disable/put-in-reset the PLX switch (Linux don't see the
> switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> root-complex goes bad.
> 
> ~ # echo 0 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: fast devsel, IRQ 295
> Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> I/O behind bridge: 00000000-00000fff [size=4K]
> Memory behind bridge: 00000000-000fffff [size=1M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
> 
> ~ # uname -a
> Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> UTC 2020 armv7l GNU/Linux
> ~ #
> 
> 
> I am really not sure what is going wrong here. Did I am missing
> something basic?

I agree, it looks like something's wrong, but I really don't have any
ideas.

I would start by using "lspci -xxxx" to see the actual values we get
from config space.  It looks like we're reading zeros from at least
the bus and window registers.

You could also instrument the i.MX config accessors in case there's
something strange going on there.  Maybe try to reproduce this on a
current upstream kernel?

Bjorn

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Bjorn Helgaas]

[+cc Richard, Lucas]

On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > Hello,
> > 
> > I am trying to figure-out an issue on our i.MX6Q platform based design
> > where PCIe interface goes bad.
> > 
> > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > (PEX8605).
> > 
> > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > in config.
> > 
> > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > until Linux is booted. So at boot time only PCIe root-complex is
> > initialized by kernel.
> > 
> > After boot if I do "lspci -v"  and see everything good from PCIe
> > root-complex (below):
> > 
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: bus master, fast devsel, latency 0, IRQ 295
> > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: None
> > Prefetchable memory behind bridge: None
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> > 
> > 
> > Then I enable the #PERST pin of PLX switch, everything is still good
> > (no rescan on Linux is done yet)
> > 
> > ~ # echo 139 > /sys/class/gpio/export
> > ~ # echo out > /sys/class/gpio/gpio139/direction
> > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: bus master, fast devsel, latency 0, IRQ 295
> > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: None
> > Prefetchable memory behind bridge: None
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> > 
> > 
> > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > root-complex goes bad.
> > 
> > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: fast devsel, IRQ 295
> > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > I/O behind bridge: 00000000-00000fff [size=4K]
> > Memory behind bridge: 00000000-000fffff [size=1M]
> > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> > 
> > ~ # uname -a
> > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > UTC 2020 armv7l GNU/Linux
> > ~ #
> > 
> > 
> > I am really not sure what is going wrong here. Did I am missing
> > something basic?
> 
> I agree, it looks like something's wrong, but I really don't have any
> ideas.
> 
> I would start by using "lspci -xxxx" to see the actual values we get
> from config space.  It looks like we're reading zeros from at least
> the bus and window registers.
> 
> You could also instrument the i.MX config accessors in case there's
> something strange going on there.  Maybe try to reproduce this on a
> current upstream kernel?
> 
> Bjorn

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Fabio Estevam]

Hi Fawad,

On Sat, Feb 22, 2020 at 12:26 PM Fawad Lateef <fawadlateef@gmail.com> wrote:
>
> Hello,
>
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
>
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
>
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's

Does it happen with 5.4 or 5.5 too?

Which dts are you using?

> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
>
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value

Not sure why you toggle the PERST pin from userspace.

You should do it via reset-gpio property in the device tree.

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Fawad Lateef]

Hi Bjorn,

Thanks for your reply. Please see my comments below.

By the way, I have another development kit from "Embedded Artists"
with i.MX6Q SOM. I did similar test quickly (with WLAN attached to
PCIe root-complex _not_ PLX switch). This one also showed same
behavior though I have to confirm this properly (working on it). Then
at-least I can say its not exactly issue of Phytec SOM.

On Thu, 27 Feb 2020 at 00:27, Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Richard, Lucas]
>
> On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> > On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > > Hello,
> > >
> > > I am trying to figure-out an issue on our i.MX6Q platform based design
> > > where PCIe interface goes bad.
> > >
> > > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > > (PEX8605).
> > >
> > > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > > in config.
> > >
> > > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > > until Linux is booted. So at boot time only PCIe root-complex is
> > > initialized by kernel.
> > >
> > > After boot if I do "lspci -v"  and see everything good from PCIe
> > > root-complex (below):
> > >
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: None
> > > Prefetchable memory behind bridge: None
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > >
> > > Then I enable the #PERST pin of PLX switch, everything is still good
> > > (no rescan on Linux is done yet)
> > >
> > > ~ # echo 139 > /sys/class/gpio/export
> > > ~ # echo out > /sys/class/gpio/gpio139/direction
> > > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: None
> > > Prefetchable memory behind bridge: None
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > >
> > > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > > root-complex goes bad.
> > >
> > > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: fast devsel, IRQ 295
> > > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > > I/O behind bridge: 00000000-00000fff [size=4K]
> > > Memory behind bridge: 00000000-000fffff [size=1M]
> > > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > > ~ # uname -a
> > > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > > UTC 2020 armv7l GNU/Linux
> > > ~ #
> > >
> > >
> > > I am really not sure what is going wrong here. Did I am missing
> > > something basic?
> >
> > I agree, it looks like something's wrong, but I really don't have any
> > ideas.
> >
> > I would start by using "lspci -xxxx" to see the actual values we get
> > from config space.  It looks like we're reading zeros from at least
> > the bus and window registers.

Somehow "lspci -xxxx" generate kernel crash ("imprecise external
abort") on both Phytec and Embedded Artists SOMs. lspci with -xxx (3
x) works but not 4 x. Seems like i.MX6 general issue?

> >
> > You could also instrument the i.MX config accessors in case there's
> > something strange going on there.  Maybe try to reproduce this on a
> > current upstream kernel?

I will try to read i.MX PCIe config registers, but I think those will
be read through PCIe interface and when it goes bad, devmem or any
other access to root-complex memory-address hangs the full SOM, not
even sys-rq works.

I was playing with 5.2.xx kernel earlier, but didn't try it on
recently. Will do a clean build with it again and see if I can face
similar situation.

Thanks,

Fawad Lateef

> >
> > Bjorn

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Fawad Lateef]

[-- Attachment #1: Type: text/plain, Size: 3211 bytes --]

Hi Fabio,

On Thu, 27 Feb 2020 at 00:40, Fabio Estevam <festevam@gmail.com> wrote:
>
> Hi Fawad,
>
> On Sat, Feb 22, 2020 at 12:26 PM Fawad Lateef <fawadlateef@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to figure-out an issue on our i.MX6Q platform based design
> > where PCIe interface goes bad.
> >
> > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > (PEX8605).
> >
> > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
>
> Does it happen with 5.4 or 5.5 too?

I had 5.2.xx kernel working earlier but then due to other issues I
switched back to Phytec provided kernel. I will give 5.2.xx (as its
patched for SOM already on my system) a try again and see if its
better.

>
> Which dts are you using?

I attached the dts files by mail. Though I tried on another OEM som
"Embedded ARtists" i.MX6Q with their own dts (but reset-gpio setting
for mPCIe card commented out) quickly and saw similar behaviour.

>
> > Then I enable the #PERST pin of PLX switch, everything is still good
> > (no rescan on Linux is done yet)
> >
> > ~ # echo 139 > /sys/class/gpio/export
> > ~ # echo out > /sys/class/gpio/gpio139/direction
> > ~ # echo 1 > /sys/class/gpio/gpio139/value
>
> Not sure why you toggle the PERST pin from userspace.

I am trying to do this later from user-space as we are battery
operated WLAN AP device and only want to enable devices based on
different "modes"; like on batteries we do not want USB3 controller
active and also might just enable one of the two WLANs etc.

>
> You should do it via reset-gpio property in the device tree.

I tried to enable only PLX switch with reset-gpio and I see that later
when I try to enable WLANs and do pci->rescan then WLANs unable to
allocate memory in BAR regions. Likely as we do not have PCIe hot-plug
enabled. On ARM without BIOS/bootloader doing enumeration it might be
even useful, right?

~ # echo 1 > /sys/bus/pci/rescan
[ 2280.186261] pcieport 0000:02:02.0: BAR 8: no space for [mem size 0x00300000]
[ 2280.193409] pcieport 0000:02:02.0: BAR 8: failed to assign [mem
size 0x00300000]
[ 2280.200834] pcieport 0000:02:03.0: BAR 8: no space for [mem size 0x00300000]
[ 2280.207948] pcieport 0000:02:03.0: BAR 8: failed to assign [mem
size 0x00300000]
[ 2280.215456] pci 0000:04:00.0: BAR 0: no space for [mem size 0x00200000 64bit]
[ 2280.222690] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x00200000 64bit]
[ 2280.230206] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[ 2280.237321] pci 0000:04:00.0: BAR 6: failed to assign [mem size
0x00010000 pref]
[ 2280.244886] pci 0000:05:00.0: BAR 0: no space for [mem size 0x00200000 64bit]
[ 2280.252115] pci 0000:05:00.0: BAR 0: failed to assign [mem size
0x00200000 64bit]
[ 2280.259623] pci 0000:05:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[ 2280.266729] pci 0000:05:00.0: BAR 6: failed to assign [mem size
0x00010000 pref]

By the way is there way to specify multiple "gpio-reset" pins in
device tree? Is reset-gpio property can have multiple pins OR
reset-gpios is to be used and its similar to gpio-reset (without 's')
property?

Thanks,

Fawad Lateef

[-- Attachment #2: imx6q-phytec-leo-emmc.dts --]
[-- Type: audio/vnd.dts, Size: 924 bytes --]

[-- Attachment #3: imx6qdl-phytec-leo.dtsi --]
[-- Type: application/octet-stream, Size: 6790 bytes --]

// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
 * Copyright (C) 2018 PHYTEC Messtechnik GmbH
 * Author: Christian Hemp <c.hemp@phytec.de>
 */

#include <dt-bindings/leds/leds-pca9532.h>

/ {
	pcie_gpio_leds: leds {
		compatible = "gpio-leds";
		status = "disabled";

/*		pcie-wlan-reset {
			gpios = <&gpio5 22 GPIO_ACTIVE_HIGH>;
			default-state = "on";
		};
*/

		pcie-1 {
			gpios = <&gpio4 25 GPIO_ACTIVE_HIGH>;
			default-state = "on";
		};

		pcie-2 {
			gpios = <&gpio4 23 GPIO_ACTIVE_HIGH>;
			default-state = "on";
		};
	};

	usb_gpio_leds: leds {
		compatible = "gpio-leds";
		status = "disabled";

		usb-hub-en {
			gpios = <&gpio4 24 GPIO_ACTIVE_HIGH>;
			default-state = "on";
		};

		adsb-en {
			gpios = <&gpio5 8 GPIO_ACTIVE_LOW>;
			default-state = "on";
		};
	};

	reg_en_switch: regulator-en-switch {
		compatible = "regulator-fixed";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_en_switch>;
		regulator-name = "Enable Switch";
		regulator-min-microvolt = <3300000>;
		regulator-max-microvolt = <3300000>;
		enable-active-high;
		gpio = <&gpio3 4 GPIO_ACTIVE_HIGH>;
		regulator-always-on;
	};

	reg_pcie: regulator-pcie {
		compatible = "regulator-fixed";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_pcie_reg>;
		regulator-name = "mPCIe_1V5";
		regulator-min-microvolt = <1500000>;
		regulator-max-microvolt = <1500000>;
		gpio = <&gpio3 0 GPIO_ACTIVE_HIGH>;
		enable-active-high;
	};

	reg_usb_h1_vbus: usb-h1-vbus {
		compatible = "regulator-fixed";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_usbh1_vbus>;
		regulator-name = "usb_h1_vbus";
		regulator-min-microvolt = <5000000>;
		regulator-max-microvolt = <5000000>;
		gpio = <&gpio2 18 GPIO_ACTIVE_HIGH>;
		enable-active-high;
	};

	reg_usbotg_vbus: usbotg-vbus {
		compatible = "regulator-fixed";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_usbotg_vbus>;
		regulator-name = "usb_otg_vbus";
		regulator-min-microvolt = <5000000>;
		regulator-max-microvolt = <5000000>;
		gpio = <&gpio2 19 GPIO_ACTIVE_HIGH>;
		enable-active-high;
	};

};

&i2c1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_i2c1>;
	clock-frequency = <100000>;
	status = "disabled";

	i2c_pex8605: pex8605@58 {
		compatible = "plx,pex8605";
		reg = <0x58>;
		status = "disabled";
	};
};

&i2c2 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_i2c2>;
	clock-frequency = <100000>;
	status = "disabled";
};

&pcie {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_pcie>;
//	reset-gpio = <&gpio2 25 GPIO_ACTIVE_LOW>;
	vpcie-supply = <&reg_pcie>;
	status = "disabled";
};

&uart1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_uart1>;
	status = "disabled";
};

&uart2 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_uart2>;
	status = "okay";
};

&uart3 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_uart3>;
	status = "disabled";
};

&uart4 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_uart4>;
	status = "disabled";
};

&usbh1 {
	vbus-supply = <&reg_usb_h1_vbus>;
	disable-over-current;
	status = "disabled";
};

&usbotg {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_usbotg>;
	vbus-supply = <&reg_usbotg_vbus>;
	disable-over-current;
	status = "disabled";
};

&usdhc1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_usdhc1>;
	cd-gpios = <&gpio6 31 GPIO_ACTIVE_LOW>;
	no-1-8-v;
	status = "disabled";
};

&iomuxc {
	pinctrl_cam0data: cam0datagrp {
		fsl,pins = <
			MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x0001b0b0
			MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x0001b0b0
			MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC   0x0001b0b0
			MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x4001b0b0
			MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC  0x0001b0b0
		>;
	};

	pinctrl_cam0clk: cam0clkgrp {
		fsl,pins = <MX6QDL_PAD_GPIO_0__CCM_CLKO1	0x000000b9>;
	};

	pinctrl_cam0switch: cam0switchgrp {
		fsl,pins = <
			MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20	0x4001b0b0
			MX6QDL_PAD_EIM_DA9__GPIO3_IO09		0x4001b0b0
		>;
	};

	pinctrl_panel_en: panelen1grp {
		fsl,pins = <
			MX6QDL_PAD_EIM_EB0__GPIO2_IO28		0xb0b1
		>;
	};

	pinctrl_en_switch: enswitchgrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_DA4__GPIO3_IO04		0xb0b1
		>;
	};

	pinctrl_flexcan1: flexcan1grp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_7__FLEXCAN1_TX		0x1b0b0
			MX6QDL_PAD_GPIO_8__FLEXCAN1_RX		0x1b0b0
		>;
	};

	pinctrl_flexcan1_en: flexcan1engrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_A18__GPIO2_IO20		0xb0b1
		>;
	};

	pinctrl_hdmicec: hdmicecgrp {
		fsl,pins = <
			MX6QDL_PAD_KEY_ROW2__HDMI_TX_CEC_LINE	0x1f8b0
		>;
	};

	pinctrl_i2c2: i2c2grp {
		fsl,pins = <
			MX6QDL_PAD_KEY_ROW3__I2C2_SDA		0x4001b8b1
			MX6QDL_PAD_KEY_COL3__I2C2_SCL		0x4001b8b1
		>;
	};

	pinctrl_i2c1: i2c1grp {
		fsl,pins = <
			MX6QDL_PAD_EIM_D21__I2C1_SCL		0x4001b8b1
			MX6QDL_PAD_EIM_D28__I2C1_SDA		0x4001b8b1
		>;
	};

	pinctrl_pcie: pciegrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_OE__GPIO2_IO25		0xb0b1
		>;
	};

	pinctrl_pcie_reg: pciereggrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_DA0__GPIO3_IO00		0xb0b1
		>;
	};

	pinctrl_pwm1: pwm1grp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_9__PWM1_OUT		0x1b0b1
		>;
	};

	pinctrl_rtc_int: rtcintgrp {
		fsl,pins = <
			MX6QDL_PAD_SD3_RST__GPIO7_IO08		0x1b0b0
		>;
	};

	pinctrl_stmpe: stmpegrp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_17__GPIO7_IO12		0x1b0b0
		>;
	};

	pinctrl_uart1: uart1grp {
		fsl,pins = <
			MX6QDL_PAD_CSI0_DAT10__UART1_TX_DATA	0x1b0b1
			MX6QDL_PAD_CSI0_DAT11__UART1_RX_DATA	0x1b0b1
		>;
	};

	pinctrl_uart2: uart2grp {
		fsl,pins = <
			MX6QDL_PAD_EIM_D26__UART2_TX_DATA	0x1b0b1
			MX6QDL_PAD_EIM_D27__UART2_RX_DATA	0x1b0b1
		>;
	};

	pinctrl_uart3: uart3grp {
		fsl,pins = <
			MX6QDL_PAD_EIM_D24__UART3_TX_DATA	0x1b0b1
			MX6QDL_PAD_EIM_D25__UART3_RX_DATA	0x1b0b1
		>;
	};

	pinctrl_uart4: uart4grp {
		fsl,pins = <
			MX6QDL_PAD_CSI0_DAT12__UART4_TX_DATA	0x1b0b1
			MX6QDL_PAD_CSI0_DAT13__UART4_RX_DATA	0x1b0b1
		>;
	};

	pinctrl_usbh1_vbus: usbh1vbusgrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_A20__GPIO2_IO18		0xb0b1
		>;
	};

	pinctrl_usbotg: usbotggrp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_1__USB_OTG_ID		0x17059
		>;
	};

	pinctrl_usbotg_vbus: usbotgvbusgrp {
		fsl,pins = <
			MX6QDL_PAD_EIM_A19__GPIO2_IO19		0xb0b1
		>;
	};

	pinctrl_usdhc1: usdhc1grp {
		fsl,pins = <
			MX6QDL_PAD_SD1_CMD__SD1_CMD		0x170f9
			MX6QDL_PAD_SD1_CLK__SD1_CLK		0x100f9
			MX6QDL_PAD_SD1_DAT0__SD1_DATA0		0x170f9
			MX6QDL_PAD_SD1_DAT1__SD1_DATA1		0x170f9
			MX6QDL_PAD_SD1_DAT2__SD1_DATA2		0x170f9
			MX6QDL_PAD_SD1_DAT3__SD1_DATA3		0x170f9
			MX6QDL_PAD_EIM_BCLK__GPIO6_IO31		0xb0b1  /* CD */
		>;
	};
};

[-- Attachment #4: imx6qdl-phytec-phycore-som-leo.dtsi --]
[-- Type: application/octet-stream, Size: 6044 bytes --]

// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
 * Copyright (C) 2018 PHYTEC Messtechnik GmbH
 * Author: Christian Hemp <c.hemp@phytec.de>
 */

#include <dt-bindings/gpio/gpio.h>

/ {
	aliases {
		rtc1 = &da9062_rtc;
		rtc2 = &snvs_rtc;
		watchdog0 = &da9062_wdog;
		watchdog1 = &wdog1;
	};

	/*
	 * Set the minimum memory size here and
	 * let the bootloader set the real size.
	 */
	memory@10000000 {
		device_type = "memory";
		reg = <0x10000000 0x8000000>;
	};

	gpio_leds_som: somleds {
		compatible = "gpio-leds";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_gpioleds_som>;

		phycore-green {
			gpios = <&gpio1 4 GPIO_ACTIVE_HIGH>;
			linux,default-trigger = "heartbeat";
		};
	};
};

&cpu0 {
	fsl,ldo-bypass;
};

&ecspi1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_ecspi1>;
	cs-gpios = <&gpio3 19 GPIO_ACTIVE_LOW>;
	status = "okay";

	m25p80: flash@0 {
		compatible = "jedec,spi-nor";
		spi-max-frequency = <20000000>;
		reg = <0>;
		status = "disabled";
	};
};

&i2c3 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_i2c3>;
	clock-frequency = <400000>;
	status = "okay";

	eeprom@50 {
		compatible = "atmel,24c32";
		reg = <0x50>;
	};

	pmic@58 {
		compatible = "dlg,da9062";
		pinctrl-names = "default";
		pinctrl-0 = <&pinctrl_pmic>;
		reg = <0x58>;
		interrupt-parent = <&gpio1>;
		interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
		interrupt-controller;

		da9062_rtc: rtc {
			compatible = "dlg,da9062-rtc";
		};

		da9062_wdog: watchdog {
			compatible = "dlg,da9062-watchdog";
		};

		regulators {
			vdd_arm: buck1 {
				regulator-name = "vdd_arm";
				regulator-min-microvolt = <730000>;
				regulator-max-microvolt = <1500000>;
				regulator-ramp-delay = <2500>;
				regulator-initial-mode = <2>; /* SYNC */
				regulator-always-on;
			};

			vdd_soc: buck2 {
				regulator-name = "vdd_soc";
				regulator-min-microvolt = <730000>;
				regulator-max-microvolt = <1500000>;
				regulator-ramp-delay = <2500>;
				regulator-initial-mode = <2>; /* SYNC */
				regulator-always-on;
			};

			vdd_ddr3_1p5: buck3 {
				regulator-name = "vdd_ddr3";
				regulator-min-microvolt = <1500000>;
				regulator-max-microvolt = <1500000>;
				regulator-initial-mode = <2>; /* SYNC */
				regulator-always-on;
			};

			vdd_snvs: ldo1 {
				regulator-name = "vdd_snvs";
				regulator-min-microvolt = <3000000>;
				regulator-max-microvolt = <3000000>;
				regulator-always-on;
			};

			vdd_high: ldo2 {
				regulator-name = "vdd_high";
				regulator-min-microvolt = <3000000>;
				regulator-max-microvolt = <3000000>;
				regulator-always-on;
			};

			vdd_emmc_1p8: ldo4 {
				regulator-name = "vdd_emmc";
				regulator-min-microvolt = <1800000>;
				regulator-max-microvolt = <1800000>;
			};
		};
	};
};

&reg_arm {
	regulator-allow-bypass;
	vin-supply = <&vdd_arm>;
};

&reg_pu {
	regulator-allow-bypass;
	vin-supply = <&vdd_soc>;
};

&reg_soc {
	regulator-allow-bypass;
	vin-supply = <&vdd_soc>;
};

&snvs_poweroff {
	status = "okay";
};

&usdhc4 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_usdhc4>;
	bus-width = <8>;
	non-removable;
	vmmc-supply = <&vdd_emmc_1p8>;
	status = "disabled";
};

&wdog1 {
	/*
	 * Rely on PMIC reboot handler. Internal i.MX6 watchdog, that is also
	 * used for reboot, does not reset all external PMIC voltages on reset.
	 */
	status = "disabled";
};

&iomuxc {
	pinctrl_enet: enetgrp {
		fsl,pins = <
			MX6QDL_PAD_ENET_MDIO__ENET_MDIO		0x1b0b0
			MX6QDL_PAD_ENET_MDC__ENET_MDC		0x1b0b0
			MX6QDL_PAD_RGMII_TXC__RGMII_TXC		0x1b0b0
			MX6QDL_PAD_RGMII_TD0__RGMII_TD0		0x1b0b0
			MX6QDL_PAD_RGMII_TD1__RGMII_TD1		0x1b0b0
			MX6QDL_PAD_RGMII_TD2__RGMII_TD2		0x1b0b0
			MX6QDL_PAD_RGMII_TD3__RGMII_TD3		0x1b0b0
			MX6QDL_PAD_RGMII_TX_CTL__RGMII_TX_CTL	0x1b0b0
			MX6QDL_PAD_ENET_REF_CLK__ENET_TX_CLK	0x1b0b0
			MX6QDL_PAD_RGMII_RXC__RGMII_RXC		0x1b0b0
			MX6QDL_PAD_RGMII_RD0__RGMII_RD0		0x1b0b0
			MX6QDL_PAD_RGMII_RD1__RGMII_RD1		0x1b0b0
			MX6QDL_PAD_RGMII_RD2__RGMII_RD2		0x1b0b0
			MX6QDL_PAD_RGMII_RD3__RGMII_RD3		0x1b0b0
			MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL	0x1b0b0
			MX6QDL_PAD_ENET_TX_EN__ENET_TX_EN	0x1b0b0
			MX6QDL_PAD_SD2_DAT1__GPIO1_IO14		0x1b0b0
		>;
	};

	pinctrl_gpioleds_som: gpioledssomgrp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_4__GPIO1_IO04		0x1b0b0
		>;
	};

	pinctrl_gpmi_nand: gpminandgrp {
		fsl,pins = <
			MX6QDL_PAD_NANDF_CLE__NAND_CLE		0xb0b1
			MX6QDL_PAD_NANDF_ALE__NAND_ALE		0xb0b1
			MX6QDL_PAD_NANDF_WP_B__NAND_WP_B	0xb0b1
			MX6QDL_PAD_NANDF_RB0__NAND_READY_B	0xb000
			MX6QDL_PAD_NANDF_CS0__NAND_CE0_B	0xb0b1
			MX6QDL_PAD_NANDF_CS1__NAND_CE1_B	0xb0b1
			MX6QDL_PAD_NANDF_CS2__NAND_CE2_B	0xb0b1
			MX6QDL_PAD_NANDF_CS3__NAND_CE3_B	0xb0b1
			MX6QDL_PAD_SD4_CMD__NAND_RE_B		0xb0b1
			MX6QDL_PAD_SD4_CLK__NAND_WE_B		0xb0b1
			MX6QDL_PAD_NANDF_D0__NAND_DATA00	0xb0b1
			MX6QDL_PAD_NANDF_D1__NAND_DATA01	0xb0b1
			MX6QDL_PAD_NANDF_D2__NAND_DATA02	0xb0b1
			MX6QDL_PAD_NANDF_D3__NAND_DATA03	0xb0b1
			MX6QDL_PAD_NANDF_D4__NAND_DATA04	0xb0b1
			MX6QDL_PAD_NANDF_D5__NAND_DATA05	0xb0b1
			MX6QDL_PAD_NANDF_D6__NAND_DATA06	0xb0b1
			MX6QDL_PAD_NANDF_D7__NAND_DATA07	0xb0b1
			MX6QDL_PAD_SD4_DAT0__NAND_DQS		0x00b1
		>;
	};

	pinctrl_i2c3: i2c3grp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_6__I2C3_SDA		0x4001b8b1
			MX6QDL_PAD_GPIO_5__I2C3_SCL		0x4001b8b1
		>;
	};

	pinctrl_ecspi1: ecspi1grp {
		fsl,pins = <
			MX6QDL_PAD_EIM_D16__ECSPI1_SCLK		0x100b1
			MX6QDL_PAD_EIM_D17__ECSPI1_MISO		0x100b1
			MX6QDL_PAD_EIM_D18__ECSPI1_MOSI		0x100b1
			MX6QDL_PAD_EIM_D19__GPIO3_IO19		0x1b0b0
		>;
	};

	pinctrl_pmic: pmicgrp {
		fsl,pins = <
			MX6QDL_PAD_GPIO_2__GPIO1_IO02		0x1b0b0
		>;
	};

	pinctrl_usdhc4: usdhc4grp {
		fsl,pins = <
			MX6QDL_PAD_SD4_CMD__SD4_CMD		0x17059
			MX6QDL_PAD_SD4_CLK__SD4_CLK		0x10059
			MX6QDL_PAD_SD4_DAT0__SD4_DATA0		0x17059
			MX6QDL_PAD_SD4_DAT1__SD4_DATA1		0x17059
			MX6QDL_PAD_SD4_DAT2__SD4_DATA2		0x17059
			MX6QDL_PAD_SD4_DAT3__SD4_DATA3		0x17059
			MX6QDL_PAD_SD4_DAT4__SD4_DATA4		0x17059
			MX6QDL_PAD_SD4_DAT5__SD4_DATA5		0x17059
			MX6QDL_PAD_SD4_DAT6__SD4_DATA6		0x17059
			MX6QDL_PAD_SD4_DAT7__SD4_DATA7		0x17059
		>;
	};
};

Re: Help needed in understanding weird PCIe issue on imx6q (PCIe just goes bad)

[Bjorn Helgaas]

On Fri, Feb 28, 2020 at 11:16:59AM +0100, Fawad Lateef wrote:
> Hi Bjorn,
> 
> Thanks for your reply. Please see my comments below.
> 
> By the way, I have another development kit from "Embedded Artists"
> with i.MX6Q SOM. I did similar test quickly (with WLAN attached to
> PCIe root-complex _not_ PLX switch). This one also showed same
> behavior though I have to confirm this properly (working on it). Then
> at-least I can say its not exactly issue of Phytec SOM.
> 
> On Thu, 27 Feb 2020 at 00:27, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> > > On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > > > Hello,
> > > >
> > > > I am trying to figure-out an issue on our i.MX6Q platform based design
> > > > where PCIe interface goes bad.
> > > >
> > > > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > > > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > > > (PEX8605).
> > > >
> > > > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > > > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > > > in config.
> > > >
> > > > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > > > until Linux is booted. So at boot time only PCIe root-complex is
> > > > initialized by kernel.
> > > >
> > > > After boot if I do "lspci -v"  and see everything good from PCIe
> > > > root-complex (below):
> > > >
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > > I/O behind bridge: None
> > > > Memory behind bridge: None
> > > > Prefetchable memory behind bridge: None
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > >
> > > > Then I enable the #PERST pin of PLX switch, everything is still good
> > > > (no rescan on Linux is done yet)
> > > >
> > > > ~ # echo 139 > /sys/class/gpio/export
> > > > ~ # echo out > /sys/class/gpio/gpio139/direction
> > > > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > > I/O behind bridge: None
> > > > Memory behind bridge: None
> > > > Prefetchable memory behind bridge: None
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > >
> > > > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > > > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > > > root-complex goes bad.
> > > >
> > > > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: fast devsel, IRQ 295
> > > > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > > > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > > > I/O behind bridge: 00000000-00000fff [size=4K]
> > > > Memory behind bridge: 00000000-000fffff [size=1M]
> > > > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > > ~ # uname -a
> > > > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > > > UTC 2020 armv7l GNU/Linux
> > > > ~ #
> > > >
> > > >
> > > > I am really not sure what is going wrong here. Did I am missing
> > > > something basic?
> > >
> > > I agree, it looks like something's wrong, but I really don't have any
> > > ideas.
> > >
> > > I would start by using "lspci -xxxx" to see the actual values we get
> > > from config space.  It looks like we're reading zeros from at least
> > > the bus and window registers.
> 
> Somehow "lspci -xxxx" generate kernel crash ("imprecise external
> abort") on both Phytec and Embedded Artists SOMs. lspci with -xxx (3
> x) works but not 4 x. Seems like i.MX6 general issue?

Sounds like i.MX6 doesn't handle PCIe errors correctly.  "lspci -xxx"
reads the 256-byte PCI config space, while "lspci -xxxx" reads the
entire 4K extended config space.  If we read config space that a
device doesn't implement, I think we'll get an Unsupported Request
completion on PCIe.  That *should* be handled nicely (without causing
a kernel crash) and turned into a ~0 response to the read.  If that
doesn't work, it needs to be solved somewhere in the i.MX6 or ARM arch
code.

Bjorn