Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Thu, 2020-09-10 at 20:29 -0300, Jason Gunthorpe wrote:
> > Probably, at least on powerpc you will as well, that's the only way to
> > get write combine.
> 
> If I remove the PROT_READ in the user space mmap will it block it?

No. powerpc at least doesn't have write-only mappings.

> Read TLPs are not harmful but I suspect they would cause an
> undesirable random performance anomaly.

I suspect in practice you wont get them esp. if the code has barriers
but ... it's allowed by the architecture.

> > > Does it/could it "combine writes"?
> > 
> > I assume so for ARM, definitely for powerpc.
> 
> Various IBM PPC chips I know work, we do test that.
> 
> > > > That's why I looped you in - that's what worries me about
> > > > "enabling"
> > > > arch_can_pci_mmap_wc() on arm64. If we enable it and we have perf
> > > > regressions that's not OK.
> > > > 
> > > > Or we *can* enable arch_can_pci_mmap_wc() but force the mellanox
> > > > driver (or more broadly all drivers following this message push
> > > > semantics) to use "something else" for WC detection.
> > > 
> > > arch_can_pci_mmap_wc() really only controls the sysfs resource file
> > > and it seems very unclear who in userspace uses that these days.
> > 
> > dpdk under some circumstances afaik.
> 
> And something gross for DMA then? Not sure dpdk is useful without
> DMA. Why not use CONFIG_VFIO_NOIOMMU for such a non-secure thing?

Clint, can you elaborate on the use case ?

> > > vfio is now the right way to do that stuff. I don't see an obvious
> > > way to get WC memory in VFIO though...
> > 
> > Which would be a performance issue on a number of things I suppose...
> 
> Almost nothing uses pci_iomap_wc(), so I'd be surpried if userspace
> DPDK was an important user when an in-kernel driver for the same HW
> doesn't use it?

Hard to know how uses those files out there but I don't like arm not
providing what pretty much all relevant archs do provide since the
semantics afaik aren't that different.

Yes, "write combine" isn't a good name.... The goal is to get WC but it
comes with the whole package on several archs. We don't even have a
reasonnable definition of the semantics of readl/writel on a WC mapping
(hint: on powerpc the barriers in them will prevent WC even on a WC
mapping) nor of what barriers might work  and how on such a mapping.

I tried a while ago and ... ugh.

Cheers,
Ben.