Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux

From: "Luís Mendes" <luis.p.mendes@gmail.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>,
	Linux PCI <linux-pci@vger.kernel.org>,
	Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Jason Cooper <jason@lakedaemon.net>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>
Subject: Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux
Date: Thu, 9 Apr 2020 16:25:40 +0100	[thread overview]
Message-ID: <CAEzXK1oCwx=w3S0ednwM2mJEEidqF3saEKu9OQP=i82y3Az0Aw@mail.gmail.com> (raw)
In-Reply-To: <CAEzXK1oU4KwatuXAjUwj5arZtdmDSgFsm2vn8DrAWCJu-v-r0g@mail.gmail.com>

Hi Bjorn,

I've good news. I've found the culprit and it is a pretty simple
issue, however the good solution is not obvious to me.
Can you help in finding the best way to patch this issue?

So first detailing the problem in file setup_bus.c there is this *if
condition* to ignore resources from classless devices and so
it is that this Google/Coral Edge TPU is a classless device with class 0xff:

static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
{
    u16 class = dev->class >> 8;

       pci_info(dev, "%s\n", __func__);
    /* Don't touch classless devices or host bridges or IOAPICs */
    if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
        return;
   ....

So the one possible trivial, non generic, attempt that works is to do:
static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
{
    u16 class = dev->class >> 8;

       pci_info(dev, "%s\n", __func__);
    /* Don't touch classless devices or host bridges or IOAPICs */
    if ((class == PCI_CLASS_NOT_DEFINED &&  !(dev->vendor == 0x1ac1 &&
dev->device==0x089a)) || class == PCI_CLASS_BRIDGE_HOST)
        return;
   ....

What is your suggestion to make the solution generic? Create a
whitelist? Remove this verification? I have no idea... nothing sounds
good to me...
01:00.0 Ethernet controller: Device 1ac1:089a (prog-if ff)
    Subsystem: Device 1ac1:089a
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K]
    Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M]
    Capabilities: <access denied>

Best regards.
Luís Mendes

On Thu, Apr 9, 2020 at 12:05 AM Luís Mendes <luis.p.mendes@gmail.com> wrote:
>
> Hi Bjorn,
>
> I have successfully setup a JTAG remote debug environment.
> And I found this:
> First call to __pci_bus_assign_resources visits 11ab:6828 -> SLOT 1,
> which in turn calls __pci_bus_assign_resources which visits device
> 1ac1:089a on that slot and calls: