Re: RPi4 can't deal with 64 bit PCI accesses

From: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
To: Ard Biesheuvel <ardb@kernel.org>
Cc: linux-pci <linux-pci@vger.kernel.org>,
	linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
	linux-rpi-kernel <linux-rpi-kernel@lists.infradead.org>,
	devicetree <devicetree@vger.kernel.org>,
	Rob Herring <robh@kernel.org>,
	Florian Fainelli <f.fainelli@gmail.com>,
	Bjorn Helgaas <helgaas@kernel.org>,
	Robin Murphy <robin.murphy@arm.con>,
	bcm-kernel-feedback-list <bcm-kernel-feedback-list@broadcom.com>
Subject: Re: RPi4 can't deal with 64 bit PCI accesses
Date: Mon, 22 Feb 2021 17:36:20 +0100	[thread overview]
Message-ID: <59614491333fde94da880aed328a22f9ca49171a.camel@suse.de> (raw)
In-Reply-To: <CAMj1kXG9ALnJcdzgv9805A91x-decqS1eq9oWi7Bb+pa3f6ErQ@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 2127 bytes --]

On Mon, 2021-02-22 at 17:18 +0100, Ard Biesheuvel wrote:
> On Mon, 22 Feb 2021 at 16:48, Nicolas Saenz Julienne
> <nsaenzjulienne@suse.de> wrote:
> > 
> > Hi everyone,
> > Raspberry Pi 4, a 64bit arm system on chip, contains a PCIe bus that can't
> > handle 64bit accesses to its MMIO address space, in other words, writeq() has
> > to be split into two distinct writel() operations. This isn't ideal, as it
> > misrepresents PCI's promise of being able to treat device memory as regular
> > memory, ultimately breaking a bunch of PCI device drivers[1].
> > 
> > I'd like to have a go at fixing this in a way that can be distributed in a
> > generic distro without prejudice to other users.
> > 
> > AFAIK there is no way to detect this limitation through generic PCIe
> > capabilities, so one solution would be to expose it through firmware
> > (devicetree in this case), and pass the limitations through 'struct device' so
> > as for the drivers to choose the right access method in a way that doesn't
> > affect performance much[2]. All in all, most of this doesn't need to be
> > PCI-centric as the property could be applied to any MMIO bus.
> > 
> > Thoughts? Opinions? Is it overkill just for a single SoC?
> > 
> 
> Hi Nicolas,
> 
> How does this issue manifest itself? There are other PCIe RC

Only the low bits would get written/read, as for the high bits I can't recall
if they were corrupted or simply ignored (I experienced this some time ago
while bringing up RPi's PCIe in u-boot).

> implementations suffering from the same issue, and some of the drivers
> in Linux already work around this, by using split accesses. Look at
> this one, for instance:
> 
> a310acd7a7ea ("NVMe: use split lo_hi_{read,write}q")
> 
> which switches NVMe to lo_hi_readq, which appears to be used in quite
> a few other places as well.

Indeed, XHCI does this unanimously too. But I figured forcing the split on all
drivers woudln't be a very popular solution just for RPi's 'faulty' bus. But if
it turns out to be a common problem, I guess it isn't such a bad idea.

Regards,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]