Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux

From: "Luís Mendes" <luis.p.mendes@gmail.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>,
	Linux PCI <linux-pci@vger.kernel.org>,
	Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Jason Cooper <jason@lakedaemon.net>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>
Subject: Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
Date: Sat, 4 Apr 2020 22:39:47 +0100	[thread overview]
Message-ID: <CAEzXK1pNiy_pjDh_=2XKpKUwEfO39rFubkviZyT_7DqEaQ7z5w@mail.gmail.com> (raw)
In-Reply-To: <20200404013233.GA30614@google.com>

Hi Bjorn,

Thanks again for your help.

Ok... I've tested your theory on this system and still no changes. The
BARs 0 and 1 are still not assigned. I should note that this issue
does not occur only on this particular armhf system, but also on a
Toradex Apalis IMX8QM, which in this case is an arm64 device and
doesn't make use of the mvebu infrastructure.

So I did issue the following commands:
# echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/remove
# echo 1 > /sys/bus/pci/devices/0000\:00\:01.0/dev_rescan"

And the dmesg update after the last command is:
[   61.124696] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff
[   61.124732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff
64bit pref]
[   61.124743] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff
64bit pref]
[   61.161258] pci_bus 0000:01: __pci_bus_size_bridges
[   61.161270] pci_bus 0000:01: pbus_size_mem: mask 0x2200 type 0x2200
0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
[   61.161277] pci_bus 0000:01: pbus_size_mem: mask 0x200 type 0x200
0x200 0x200 min 0x0 add 0x0 b_res [mem 0xd0000000-0xd01fffff] parent
[mem 0xd0000000-0xefffffff]
[   61.161281] pci_bus 0000:02: __pci_bus_size_bridges
[   61.161286] pci_bus 0000:02: pbus_size_mem: mask 0x2200 type 0x2200
0x2200 0x2200 min 0x0 add 0x0 b_res (null) parent (null)
[   61.161291] pci_bus 0000:02: pbus_size_mem: mask 0x200 type 0x200
0x200 0x200 min 0x0 add 0x0 b_res [mem 0x00000000] parent (null)
[   61.161295] pci_bus 0000:00: __pci_bus_assign_resources
[   61.161298] pci_bus 0000:00: pbus_assign_resources_sorted
[   61.161302] pci 0000:00:01.0: __dev_sort_resources
[   61.161305] pci 0000:00:02.0: __dev_sort_resources
[   61.161308] __assign_resources_sorted
[   61.161311] pci 0000:00:01.0: __pci_bus_assign_resources
[   61.161314] pci 0000:00:01.0: pdev_assign_fixed_resources
[   61.161317] pci_bus 0000:01: __pci_bus_assign_resources
[   61.161319] pci_bus 0000:01: pbus_assign_resources_sorted
[   61.161323] pci 0000:01:00.0: __dev_sort_resources
[   61.161324] __assign_resources_sorted
[   61.161327] pci 0000:01:00.0: __pci_bus_assign_resources
[   61.161330] pci 0000:01:00.0: pdev_assign_fixed_resources
[   61.161333] pci 0000:00:01.0: PCI bridge to [bus 01]
[   61.161340] pci 0000:00:01.0:   bridge window [mem 0xd0000000-0xd01fffff]
[   61.161344] pci 0000:00:02.0: __pci_bus_assign_resources
[   61.161347] pci 0000:00:02.0: pdev_assign_fixed_resources
[   61.161350] pci_bus 0000:02: __pci_bus_assign_resources
[   61.161353] pci_bus 0000:02: pbus_assign_resources_sorted
[   61.161354] __assign_resources_sorted
[   61.161357] pci 0000:00:02.0: PCI bridge to [bus 02]

Luís

On Sat, Apr 4, 2020 at 2:32 AM Bjorn Helgaas <helgaas@kernel.org> wrote:

>   pci 0000:02:00.0: [10ec:525a] type 00 class 0xff0000
>   pci 0000:02:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
>   pci 0000:02:00.0: BAR 1: assigned [mem 0xf1100000-0xf1100fff]
>   pci 0000:02:00.0: BAR 1: error updating (0xf1100000 != 0xffffffff)
>
> So we correctly detected the device, read the cleared BAR, and
> allocated space for it, and tried to update the BAR.  On my system the
> update failed, I think because of a power management issue (all config
> reads now return 0xffffffff).  But there have been a lot of power
> management fixes since the v5.2 kernel I'm running, so it's possible
> you'd have better luck.
>
> On your system, I think you would want something like:
>
>   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/remove
>   # echo 1 > /sys/devices/pci0000:00/0000:00:01.0/rescan
>