[PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[gavinli]

From: Gavin Li <git@thegavinli.com>

On x86, calling dma_mmap_coherent() on memory allocated with
dma_alloc_coherent() causes the following warning to be issued:

x86/PAT: ... map pfn RAM range req uncached-minus for [mem 0x77f000000-0x77f210fff], got write-back

This occurs because on x86 dma_alloc_coherent() returns normal kernel
memory pages (with a write-back PAT), but dma_mmap_coherent() tries to
map the same pages into userspace with a uncached-minus PAT, even though
there is no need for the mapping to noncached.

This patch ensures that on DMA coherent architectures/devices, memory is
mapped normally rather than as noncached.

Signed-off-by: Gavin Li <git@thegavinli.com>
---
 include/linux/dma-noncoherent.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/linux/dma-noncoherent.h b/include/linux/dma-noncoherent.h
index 9741767e400f..40ff11380ec7 100644
--- a/include/linux/dma-noncoherent.h
+++ b/include/linux/dma-noncoherent.h
@@ -31,7 +31,11 @@ long arch_dma_coherent_to_pfn(struct device *dev, void *cpu_addr,
 pgprot_t arch_dma_mmap_pgprot(struct device *dev, pgprot_t prot,
 		unsigned long attrs);
 #else
-# define arch_dma_mmap_pgprot(dev, prot, attrs)	pgprot_noncached(prot)
+static inline pgprot_t arch_dma_mmap_pgprot(struct device *dev, pgprot_t prot,
+		unsigned long attrs)
+{
+	return dev_is_dma_coherent(dev) ? prot : pgprot_noncached(prot);
+}
 #endif
 
 #ifdef CONFIG_DMA_NONCOHERENT_CACHE_SYNC
-- 
2.22.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Christoph Hellwig]

See the discussion at:

https://lists.linuxfoundation.org/pipermail/iommu/2019-August/037716.html

Just curious, what driver do you use that uses dma_mmap_coherent on
x86?
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Gavin Li]

Ah, seems like it wasn't as simple of a fix as I thought :)

I intended to use dma_mmap_coherent() in the usbfs driver
(drivers/usb/core/devio.c) because its mmap() was broken on arm64 and
all the other noncoherent DMA architectures.

Patch here: https://www.spinics.net/lists/linux-usb/msg183148.html
More info: https://www.spinics.net/lists/linux-usb/msg183180.html

On Fri, Aug 2, 2019 at 11:23 PM Christoph Hellwig <hch@lst.de> wrote:
>
> See the discussion at:
>
> https://lists.linuxfoundation.org/pipermail/iommu/2019-August/037716.html
>
> Just curious, what driver do you use that uses dma_mmap_coherent on
> x86?
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Christoph Hellwig]

On Fri, Aug 02, 2019 at 11:35:48PM -0700, Gavin Li wrote:
> Ah, seems like it wasn't as simple of a fix as I thought :)

With our mess of dma flags nothing is ever as simple as it seems..

> I intended to use dma_mmap_coherent() in the usbfs driver
> (drivers/usb/core/devio.c) because its mmap() was broken on arm64 and
> all the other noncoherent DMA architectures.

Good to know, that is the right thing to do!

I'll try to expedite the fix to 5.3-rc + stable backports once we agree
on the details.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Oded Gabbay]

On Sat, Aug 3, 2019 at 9:26 AM Christoph Hellwig <hch@lst.de> wrote:
>
> See the discussion at:
>
> https://lists.linuxfoundation.org/pipermail/iommu/2019-August/037716.html
>
> Just curious, what driver do you use that uses dma_mmap_coherent on
> x86?
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Hi Christoph,
In the habanalabs driver we have moved to use dma_mmap_coherent (to
match dma_alloc_coherent - see commit
https://lkml.org/lkml/2020/8/29/252)
Since then, we are plagued by the kernel log message that gavin has
mentioned, as we are mostly running in our C/I environment with 5.4.
I wondered if you know if there was any fix to that in the more recent kernels ?
If not, can I help to fix that ?

Thanks,
oded
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Christoph Hellwig]

On Wed, Oct 28, 2020 at 01:39:17PM +0200, Oded Gabbay wrote:
> On Sat, Aug 3, 2019 at 9:26 AM Christoph Hellwig <hch@lst.de> wrote:
> >
> > See the discussion at:
> >
> > https://lists.linuxfoundation.org/pipermail/iommu/2019-August/037716.html
> >
> > Just curious, what driver do you use that uses dma_mmap_coherent on
> > x86?
> > _______________________________________________
> > iommu mailing list
> > iommu@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/iommu
> 
> Hi Christoph,
> In the habanalabs driver we have moved to use dma_mmap_coherent (to
> match dma_alloc_coherent - see commit
> https://lkml.org/lkml/2020/8/29/252)
> Since then, we are plagued by the kernel log message that gavin has
> mentioned, as we are mostly running in our C/I environment with 5.4.
> I wondered if you know if there was any fix to that in the more recent kernels ?
> If not, can I help to fix that ?

What are the kernel log messages that gaving has mentioned?
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Oded Gabbay]

On Wed, Oct 28, 2020 at 7:31 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Oct 28, 2020 at 01:39:17PM +0200, Oded Gabbay wrote:
> > On Sat, Aug 3, 2019 at 9:26 AM Christoph Hellwig <hch@lst.de> wrote:
> > >
> > > See the discussion at:
> > >
> > > https://lists.linuxfoundation.org/pipermail/iommu/2019-August/037716.html
> > >
> > > Just curious, what driver do you use that uses dma_mmap_coherent on
> > > x86?
> > > _______________________________________________
> > > iommu mailing list
> > > iommu@lists.linux-foundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/iommu
> >
> > Hi Christoph,
> > In the habanalabs driver we have moved to use dma_mmap_coherent (to
> > match dma_alloc_coherent - see commit
> > https://lkml.org/lkml/2020/8/29/252)
> > Since then, we are plagued by the kernel log message that gavin has
> > mentioned, as we are mostly running in our C/I environment with 5.4.
> > I wondered if you know if there was any fix to that in the more recent kernels ?
> > If not, can I help to fix that ?
>
> What are the kernel log messages that gaving has mentioned?

This one:
11:22:43  [Wed Oct 28 11:22:34 2020] x86/PAT: synapse_tests:29265 map
pfn RAM range req uncached-minus for [mem 0xe6236b000-0xe6236bfff],
got write-back

Thousands of the same message with different addresses of course.

Thanks,
Oded
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Christoph Hellwig]

On Wed, Oct 28, 2020 at 07:38:04PM +0200, Oded Gabbay wrote:
> > > https://lkml.org/lkml/2020/8/29/252)
> > > Since then, we are plagued by the kernel log message that gavin has
> > > mentioned, as we are mostly running in our C/I environment with 5.4.
> > > I wondered if you know if there was any fix to that in the more recent kernels ?
> > > If not, can I help to fix that ?
> >
> > What are the kernel log messages that gaving has mentioned?
> 
> This one:
> 11:22:43  [Wed Oct 28 11:22:34 2020] x86/PAT: synapse_tests:29265 map
> pfn RAM range req uncached-minus for [mem 0xe6236b000-0xe6236bfff],
> got write-back
> 
> Thousands of the same message with different addresses of course.

What kernel version is this on?  And what is the test case?
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v1] dma-mapping: normal memory for mmap() on coherent architectures

[Oded Gabbay]

On Wed, Oct 28, 2020 at 7:39 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Oct 28, 2020 at 07:38:04PM +0200, Oded Gabbay wrote:
> > > > https://lkml.org/lkml/2020/8/29/252)
> > > > Since then, we are plagued by the kernel log message that gavin has
> > > > mentioned, as we are mostly running in our C/I environment with 5.4.
> > > > I wondered if you know if there was any fix to that in the more recent kernels ?
> > > > If not, can I help to fix that ?
> > >
> > > What are the kernel log messages that gaving has mentioned?
> >
> > This one:
> > 11:22:43  [Wed Oct 28 11:22:34 2020] x86/PAT: synapse_tests:29265 map
> > pfn RAM range req uncached-minus for [mem 0xe6236b000-0xe6236bfff],
> > got write-back
> >
> > Thousands of the same message with different addresses of course.
>
> What kernel version is this on?  And what is the test case?

ok, so my previous message wasn't accurate. On 5.4 kernel it actually
does NOT happen.
It happens on 4.18 and 4.15 (kernels that are delivered in Ubuntu 18.04).
So I assume it was fixed somewhere between 4.19 to 5.4.
I'll try to dig the code to see where the change is exactly.

Thanks and sorry for the trouble,
Oded
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu