[PATCH 0/2] arm64: mm: Fix cache maintenance for non-coherent streaming DMA

[PATCH 0/2] arm64: mm: Fix cache maintenance for non-coherent streaming DMA

[Will Deacon]

Hi all,

This pair of patches follows-up on the discussion we had on linux-arch
earlier this week and addresses the reported problem for arm64 only.

Other architectures should be able to do something similar if they wish,
but this all came out of discussion and code inspection rather than a
concrete failure.

Cheers,

Will

Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Russell King <linux@armlinux.org.uk>

--->8

Will Deacon (2):
  arm64: mm: Don't invalidate FROM_DEVICE buffers at start of DMA
    transfer
  arm64: mm: Remove assembly DMA cache maintenance wrappers

 arch/arm64/include/asm/cacheflush.h |  7 -----
 arch/arm64/mm/cache.S               | 43 -----------------------------
 arch/arm64/mm/dma-mapping.c         | 19 +++++++++----
 3 files changed, 14 insertions(+), 55 deletions(-)

-- 
2.36.1.476.g0c4daa206d-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 1/2] arm64: mm: Don't invalidate FROM_DEVICE buffers at start of DMA transfer

[Will Deacon]

Invalidating the buffer memory in arch_sync_dma_for_device() for
FROM_DEVICE transfers

When using the streaming DMA API to map a buffer prior to inbound
non-coherent DMA (i.e. DMA_FROM_DEVICE), we invalidate any dirty CPU
cachelines so that they will not be written back during the transfer and
corrupt the buffer contents written by the DMA. This, however, poses two
potential problems:

  (1) If the DMA transfer does not write to every byte in the buffer,
      then the unwritten bytes will contain stale data once the transfer
      has completed.

  (2) If the buffer has a virtual alias in userspace, then stale data
      may be visible via this alias during the period between performing
      the cache invalidation and the DMA writes landing in memory.

Address both of these issues by cleaning (aka writing-back) the dirty
lines in arch_sync_dma_for_device(DMA_FROM_DEVICE) instead of discarding
them using invalidation.

Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20220606152150.GA31568@willie-the-truck
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/mm/cache.S | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S
index 0ea6cc25dc66..21c907987080 100644
--- a/arch/arm64/mm/cache.S
+++ b/arch/arm64/mm/cache.S
@@ -218,8 +218,6 @@ SYM_FUNC_ALIAS(__dma_flush_area, __pi___dma_flush_area)
  */
 SYM_FUNC_START(__pi___dma_map_area)
 	add	x1, x0, x1
-	cmp	w2, #DMA_FROM_DEVICE
-	b.eq	__pi_dcache_inval_poc
 	b	__pi_dcache_clean_poc
 SYM_FUNC_END(__pi___dma_map_area)
 SYM_FUNC_ALIAS(__dma_map_area, __pi___dma_map_area)
-- 
2.36.1.476.g0c4daa206d-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 2/2] arm64: mm: Remove assembly DMA cache maintenance wrappers

[Will Deacon]

Remove the __dma_{flush,map,unmap}_area assembly wrappers and call the
appropriate cache maintenance functions directly from the DMA mapping
callbacks.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/cacheflush.h |  7 -----
 arch/arm64/mm/cache.S               | 41 -----------------------------
 arch/arm64/mm/dma-mapping.c         | 19 +++++++++----
 3 files changed, 14 insertions(+), 53 deletions(-)

diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h
index 5a228e203ef9..37185e978aeb 100644
--- a/arch/arm64/include/asm/cacheflush.h
+++ b/arch/arm64/include/asm/cacheflush.h
@@ -104,13 +104,6 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)
 }
 #define flush_icache_range flush_icache_range
 
-/*
- * Cache maintenance functions used by the DMA API. No to be used directly.
- */
-extern void __dma_map_area(const void *, size_t, int);
-extern void __dma_unmap_area(const void *, size_t, int);
-extern void __dma_flush_area(const void *, size_t);
-
 /*
  * Copy user data from/to a page which is mapped into a different
  * processes address space.  Really, we want to allow our "user
diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S
index 21c907987080..081058d4e436 100644
--- a/arch/arm64/mm/cache.S
+++ b/arch/arm64/mm/cache.S
@@ -194,44 +194,3 @@ SYM_FUNC_START(__pi_dcache_clean_pop)
 	ret
 SYM_FUNC_END(__pi_dcache_clean_pop)
 SYM_FUNC_ALIAS(dcache_clean_pop, __pi_dcache_clean_pop)
-
-/*
- *	__dma_flush_area(start, size)
- *
- *	clean & invalidate D / U line
- *
- *	- start   - virtual start address of region
- *	- size    - size in question
- */
-SYM_FUNC_START(__pi___dma_flush_area)
-	add	x1, x0, x1
-	dcache_by_line_op civac, sy, x0, x1, x2, x3
-	ret
-SYM_FUNC_END(__pi___dma_flush_area)
-SYM_FUNC_ALIAS(__dma_flush_area, __pi___dma_flush_area)
-
-/*
- *	__dma_map_area(start, size, dir)
- *	- start	- kernel virtual start address
- *	- size	- size of region
- *	- dir	- DMA direction
- */
-SYM_FUNC_START(__pi___dma_map_area)
-	add	x1, x0, x1
-	b	__pi_dcache_clean_poc
-SYM_FUNC_END(__pi___dma_map_area)
-SYM_FUNC_ALIAS(__dma_map_area, __pi___dma_map_area)
-
-/*
- *	__dma_unmap_area(start, size, dir)
- *	- start	- kernel virtual start address
- *	- size	- size of region
- *	- dir	- DMA direction
- */
-SYM_FUNC_START(__pi___dma_unmap_area)
-	add	x1, x0, x1
-	cmp	w2, #DMA_TO_DEVICE
-	b.ne	__pi_dcache_inval_poc
-	ret
-SYM_FUNC_END(__pi___dma_unmap_area)
-SYM_FUNC_ALIAS(__dma_unmap_area, __pi___dma_unmap_area)
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 6719f9efea09..df0c488ae643 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -14,20 +14,29 @@
 #include <asm/cacheflush.h>
 
 void arch_sync_dma_for_device(phys_addr_t paddr, size_t size,
-		enum dma_data_direction dir)
+			      enum dma_data_direction dir)
 {
-	__dma_map_area(phys_to_virt(paddr), size, dir);
+	unsigned long start = (unsigned long)phys_to_virt(paddr);
+
+	dcache_clean_poc(start, start + size);
 }
 
 void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size,
-		enum dma_data_direction dir)
+			   enum dma_data_direction dir)
 {
-	__dma_unmap_area(phys_to_virt(paddr), size, dir);
+	unsigned long start = (unsigned long)phys_to_virt(paddr);
+
+	if (dir == DMA_TO_DEVICE)
+		return;
+
+	dcache_inval_poc(start, start + size);
 }
 
 void arch_dma_prep_coherent(struct page *page, size_t size)
 {
-	__dma_flush_area(page_address(page), size);
+	unsigned long start = (unsigned long)page_address(page);
+
+	dcache_clean_inval_poc(start, start + size);
 }
 
 #ifdef CONFIG_IOMMU_DMA
-- 
2.36.1.476.g0c4daa206d-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/2] arm64: mm: Remove assembly DMA cache maintenance wrappers

[Christoph Hellwig]

On Fri, Jun 10, 2022 at 04:12:28PM +0100, Will Deacon wrote:
> Remove the __dma_{flush,map,unmap}_area assembly wrappers and call the
> appropriate cache maintenance functions directly from the DMA mapping
> callbacks.

Thanks, this looks so much nicer.  Is there any good reason why
the C prototypes for the assembly function take an unsigned long
instead of a void *, though?  It seems like all or almost all of
the callers have a pointer at hand an do silly casts for the assembly
code that could work perfectly fine with a pointer and length.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/2] arm64: mm: Remove assembly DMA cache maintenance wrappers

[Catalin Marinas]

On Fri, Jun 10, 2022 at 04:12:28PM +0100, Will Deacon wrote:
> Remove the __dma_{flush,map,unmap}_area assembly wrappers and call the
> appropriate cache maintenance functions directly from the DMA mapping
> callbacks.
> 
> Signed-off-by: Will Deacon <will@kernel.org>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

(I'll queue the first patch as a fix)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/2] arm64: mm: Fix cache maintenance for non-coherent streaming DMA

[Ard Biesheuvel]

On Fri, 10 Jun 2022 at 17:12, Will Deacon <will@kernel.org> wrote:
>
> Hi all,
>
> This pair of patches follows-up on the discussion we had on linux-arch
> earlier this week and addresses the reported problem for arm64 only.
>
> Other architectures should be able to do something similar if they wish,
> but this all came out of discussion and code inspection rather than a
> concrete failure.
>
> Cheers,
>
> Will
>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Russell King <linux@armlinux.org.uk>
>
> --->8
>
> Will Deacon (2):
>   arm64: mm: Don't invalidate FROM_DEVICE buffers at start of DMA
>     transfer
>   arm64: mm: Remove assembly DMA cache maintenance wrappers
>

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: (subset) [PATCH 0/2] arm64: mm: Fix cache maintenance for non-coherent streaming DMA

[Catalin Marinas]

On Fri, 10 Jun 2022 16:12:26 +0100, Will Deacon wrote:
> This pair of patches follows-up on the discussion we had on linux-arch
> earlier this week and addresses the reported problem for arm64 only.
> 
> Other architectures should be able to do something similar if they wish,
> but this all came out of discussion and code inspection rather than a
> concrete failure.
> 
> [...]

Applied to arm64 (for-next/fixes), thanks!

[1/2] arm64: mm: Don't invalidate FROM_DEVICE buffers at start of DMA transfer
      https://git.kernel.org/arm64/c/c50f11c6196f

-- 
Catalin


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel