linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [RFC 0/4] Raspberry Pi 4 DMA addressing support
@ 2019-07-17 15:31 Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
                   ` (3 more replies)
  0 siblings, 4 replies; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-17 15:31 UTC (permalink / raw)
  To: linux-arm-kernel, Robin Murphy, iommu
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, linux-kernel,
	Jisheng.Zhang, mbrugger, will, hch, Nicolas Saenz Julienne,
	m.szyprowski

Hi all,
this series attempts to address some issues we found while bringing up the new
Raspberry Pi 4 in arm64 and it's intended to serve as a follow up to this:
https://www.spinics.net/lists/arm-kernel/msg740650.html

The new Raspberry Pi 4 has up to 4GB of ram but most devices can only address
the first GB of ram: the DMA address range is 0xc0000000-0xfc000000 which is
aliased to the first GB of memory 0x00000000-0x3c000000. Note that only some
devices have this limitations, the ARM cores, PCIe, GENET, and 40-bit DMA
channels have a wider view of the address space.

This is solved in arm32 by setting up the correct '.dma_zone_size = SZ_1G'
which takes care of the allocating the coherent memory area at the right spot
and also is taken into account in the arch specific 'dma_map_ops'.

Unfortunately there is no such thing as '.dma_zone_size' in arm64, to make
things worse it's assumed that all devices will be able to adress the first 4GB
of memory.

This raises two issues: the coherent memory reserves are located in an area not
accessible by most devices, and DMA streaming's dma_supported(), which fails
for most devices since it's min_mask isn't properly set. Note that the rest if
DMA streaming works fine thanks to the help of swiotlb.

On one hand I've implemented a function that parses the 'dma-range' on all
interconnects and tries to select a location for the coherent memory reserves
that'll fit all devices. I made the algorithm as simple as possible, based on
the existing devices limitations.

On the other I've added a new variable in dma-direct that allows modifying the
min_mask during the init process and taken care of setting it accordingly in
the arm64's init code.

Regards,
Nicolas

---

Nicolas Saenz Julienne (4):
  arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys()
  arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit
  dma-direct: add dma_direct_min_mask
  arm64: mm: set direct_dma_min_mask according to dma-ranges

 arch/arm64/mm/init.c | 69 ++++++++++++++++++++++++++++++++++++++++----
 kernel/dma/direct.c  |  4 ++-
 2 files changed, 67 insertions(+), 6 deletions(-)

-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [RFC 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys()
  2019-07-17 15:31 [RFC 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
@ 2019-07-17 15:31 ` Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit Nicolas Saenz Julienne
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-17 15:31 UTC (permalink / raw)
  To: linux-arm-kernel, linux-kernel
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, robin.murphy,
	Jisheng.Zhang, mbrugger, will, hch, Nicolas Saenz Julienne,
	m.szyprowski

By the time we call zones_sizes_init() arm64_dma_phys_limit already
contains the result of max_zone_dma_phys(). We use the variable instead
of calling the function directly to save some precious cpu time.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 arch/arm64/mm/init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index f3c795278def..6112d6c90fa8 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -181,7 +181,7 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 	unsigned long max_zone_pfns[MAX_NR_ZONES]  = {0};
 
 #ifdef CONFIG_ZONE_DMA32
-	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(max_zone_dma_phys());
+	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(arm64_dma_phys_limit);
 #endif
 	max_zone_pfns[ZONE_NORMAL] = max;
 
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit
  2019-07-17 15:31 [RFC 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
@ 2019-07-17 15:31 ` Nicolas Saenz Julienne
  2019-07-24 13:54   ` Catalin Marinas
  2019-07-17 15:31 ` [RFC 3/4] dma-direct: add dma_direct_min_mask Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 4/4] arm64: mm: set direct_dma_min_mask according to dma-ranges Nicolas Saenz Julienne
  3 siblings, 1 reply; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-17 15:31 UTC (permalink / raw)
  To: linux-arm-kernel, linux-kernel
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, robin.murphy,
	Jisheng.Zhang, mbrugger, will, hch, Nicolas Saenz Julienne,
	m.szyprowski

The dma physical limit has so far been calculated based on the memory
size and the assumption that dma would be at least able to address the
first 4 GB. This turned out no to be true with the Raspberry Pi 4
which, on it's main interconnect, can only address the first GB of
memory, even though it might have up to 4 GB.

With the current miscalculated dma physical limit the contiguous memory
reserve is located in an inaccessible area for most of the board's
devices.

To solve this we now scan the device tree for the 'dma-ranges' property
on the root or interconnect nodes, which allows us to calculate the
lowest common denominator dma physical limit. If no dma-ranges is
available, we'll default to the old scheme.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 arch/arm64/mm/init.c | 61 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 57 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 6112d6c90fa8..5708adf0db52 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -163,14 +163,67 @@ static void __init reserve_elfcorehdr(void)
 {
 }
 #endif /* CONFIG_CRASH_DUMP */
+
+static int __init early_init_dt_scan_dma_ranges(unsigned long node,
+		const char *uname, int depth, void *data)
+{
+	phys_addr_t *dma_phys_limit = data;
+	u64 phys_addr, dma_addr, size;
+	int dma_addr_cells;
+	const __be32 *reg;
+	const void *prop;
+	int len;
+
+	/* We keep looking if this isn't the root node or an interconnect */
+	if (!(depth == 0 ||
+	    (depth == 1 && of_flat_dt_is_compatible(node, "simple-bus"))))
+		return 0;
+
+	prop = of_get_flat_dt_prop(node, "#address-cells", NULL);
+	if (prop)
+		dma_addr_cells = be32_to_cpup(prop);
+	else
+		dma_addr_cells = 1; /* arm64's default addr_cell size */
+
+	reg = of_get_flat_dt_prop(node, "dma-ranges", &len);
+	if (!reg ||
+	    len < (dma_addr_cells + dt_root_addr_cells + dt_root_size_cells))
+		return 0;
+
+	dma_addr = dt_mem_next_cell(dma_addr_cells, &reg);
+	phys_addr = dt_mem_next_cell(dt_root_addr_cells, &reg);
+	size = dt_mem_next_cell(dt_root_size_cells, &reg);
+
+	/* We're in ZONE_DMA32 */
+	if (size > (1ULL << 32))
+		size = 1ULL << 32;
+
+	if (*dma_phys_limit > (phys_addr + size))
+		*dma_phys_limit = phys_addr + size;
+
+	return 0;
+}
+
 /*
- * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)). It
- * currently assumes that for memory starting above 4G, 32-bit devices will
- * use a DMA offset.
+ * Return the maximum physical address for ZONE_DMA32. It currently assumes
+ * that for memory starting above 4G, 32-bit devices will use a DMA offset.
  */
 static phys_addr_t __init max_zone_dma_phys(void)
 {
-	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
+	phys_addr_t dma_phys_limit = ~0;
+	phys_addr_t offset;
+
+	/*
+	 * We walk the whole fdt looking for nodes with dma-ranges, calculate
+	 * the max_zone_dma_phys for them and keep going. We end-up getting the
+	 * lowest common denominator of all the matches.
+	 */
+	of_scan_flat_dt(early_init_dt_scan_dma_ranges, &dma_phys_limit);
+	if (dma_phys_limit != ~0)
+		return dma_phys_limit;
+
+	/* If no dma-ranges property was found we try to infer the value */
+	offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
 	return min(offset + (1ULL << 32), memblock_end_of_DRAM());
 }
 
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-17 15:31 [RFC 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
  2019-07-17 15:31 ` [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit Nicolas Saenz Julienne
@ 2019-07-17 15:31 ` Nicolas Saenz Julienne
  2019-07-18  9:15   ` Christoph Hellwig
  2019-07-17 15:31 ` [RFC 4/4] arm64: mm: set direct_dma_min_mask according to dma-ranges Nicolas Saenz Julienne
  3 siblings, 1 reply; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-17 15:31 UTC (permalink / raw)
  To: linux-arm-kernel, Christoph Hellwig, Marek Szyprowski, Robin Murphy
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, iommu,
	linux-kernel, Jisheng.Zhang, mbrugger, will,
	Nicolas Saenz Julienne

Historically devices with ZONE_DMA32 have been assumed to be able to
address at least the lower 4GB of ram for DMA. This is still the defualt
behavior yet the Raspberry Pi 4 is limited to the first GB of memory.
This has been observed to trigger failures in dma_direct_supported() as
the 'min_mask' isn't properly set.

We create 'dma_direct_min_mask' in order for the arch init code to be
able to fine-tune dma direct's 'min_dma' mask.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 kernel/dma/direct.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index b90e1aede743..3c8cd730648b 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -23,6 +23,8 @@
 #define ARCH_ZONE_DMA_BITS 24
 #endif
 
+u64 dma_direct_min_mask __ro_after_init = DMA_BIT_MASK(32);
+
 /*
  * For AMD SEV all DMA must be to unencrypted addresses.
  */
@@ -393,7 +395,7 @@ int dma_direct_supported(struct device *dev, u64 mask)
 	if (IS_ENABLED(CONFIG_ZONE_DMA))
 		min_mask = DMA_BIT_MASK(ARCH_ZONE_DMA_BITS);
 	else
-		min_mask = DMA_BIT_MASK(32);
+		min_mask = dma_direct_min_mask;
 
 	min_mask = min_t(u64, min_mask, (max_pfn - 1) << PAGE_SHIFT);
 
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [RFC 4/4] arm64: mm: set direct_dma_min_mask according to dma-ranges
  2019-07-17 15:31 [RFC 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
                   ` (2 preceding siblings ...)
  2019-07-17 15:31 ` [RFC 3/4] dma-direct: add dma_direct_min_mask Nicolas Saenz Julienne
@ 2019-07-17 15:31 ` Nicolas Saenz Julienne
  3 siblings, 0 replies; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-17 15:31 UTC (permalink / raw)
  To: linux-arm-kernel, linux-kernel
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, robin.murphy,
	Jisheng.Zhang, mbrugger, will, hch, Nicolas Saenz Julienne,
	m.szyprowski

Now that we parse the dma-ranges during initialization we can fine-tune
the DMA mask used by the direct DMA implementation.

We set the mask based on the size of the DMA addressable memory, and if
bigger than 4GB we force it to DMA_BIT_MASK(32) as it's always been.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 arch/arm64/mm/init.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 5708adf0db52..f8af2c99667c 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -52,6 +52,8 @@ EXPORT_SYMBOL(memstart_addr);
 
 phys_addr_t arm64_dma_phys_limit __ro_after_init;
 
+extern u64 dma_direct_min_mask;
+
 #ifdef CONFIG_KEXEC_CORE
 /*
  * reserve_crashkernel() - reserves memory for crash kernel
@@ -198,8 +200,12 @@ static int __init early_init_dt_scan_dma_ranges(unsigned long node,
 	if (size > (1ULL << 32))
 		size = 1ULL << 32;
 
-	if (*dma_phys_limit > (phys_addr + size))
+	if (*dma_phys_limit > (phys_addr + size)) {
+		/* Set min DMA mask in case is was smaller than 32 */
+		dma_direct_min_mask = size - 1;
+
 		*dma_phys_limit = phys_addr + size;
+	}
 
 	return 0;
 }
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-17 15:31 ` [RFC 3/4] dma-direct: add dma_direct_min_mask Nicolas Saenz Julienne
@ 2019-07-18  9:15   ` Christoph Hellwig
  2019-07-18 11:18     ` Nicolas Saenz Julienne
  0 siblings, 1 reply; 12+ messages in thread
From: Christoph Hellwig @ 2019-07-18  9:15 UTC (permalink / raw)
  To: Nicolas Saenz Julienne
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, Robin Murphy,
	iommu, linux-kernel, Jisheng.Zhang, mbrugger, will,
	Christoph Hellwig, linux-arm-kernel, Marek Szyprowski

On Wed, Jul 17, 2019 at 05:31:34PM +0200, Nicolas Saenz Julienne wrote:
> Historically devices with ZONE_DMA32 have been assumed to be able to
> address at least the lower 4GB of ram for DMA. This is still the defualt
> behavior yet the Raspberry Pi 4 is limited to the first GB of memory.
> This has been observed to trigger failures in dma_direct_supported() as
> the 'min_mask' isn't properly set.
> 
> We create 'dma_direct_min_mask' in order for the arch init code to be
> able to fine-tune dma direct's 'min_dma' mask.

Normally we use ZONE_DMA for that case.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-18  9:15   ` Christoph Hellwig
@ 2019-07-18 11:18     ` Nicolas Saenz Julienne
  2019-07-19 13:08       ` Nicolas Saenz Julienne
  0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-18 11:18 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, Robin Murphy,
	iommu, linux-kernel, Jisheng.Zhang, mbrugger, will,
	linux-arm-kernel, Marek Szyprowski


[-- Attachment #1.1: Type: text/plain, Size: 1045 bytes --]

On Thu, 2019-07-18 at 11:15 +0200, Christoph Hellwig wrote:
> On Wed, Jul 17, 2019 at 05:31:34PM +0200, Nicolas Saenz Julienne wrote:
> > Historically devices with ZONE_DMA32 have been assumed to be able to
> > address at least the lower 4GB of ram for DMA. This is still the defualt
> > behavior yet the Raspberry Pi 4 is limited to the first GB of memory.
> > This has been observed to trigger failures in dma_direct_supported() as
> > the 'min_mask' isn't properly set.
> > 
> > We create 'dma_direct_min_mask' in order for the arch init code to be
> > able to fine-tune dma direct's 'min_dma' mask.
> 
> Normally we use ZONE_DMA for that case.

Fair enough, I didn't think of that possibility.

So would the arm64 maintainers be happy with something like this:

- ZONE_DMA: Follows standard definition, 16MB in size. ARCH_ZONE_DMA_BITS is
	    left as is.
- ZONE_DMA32: Will honor the most constraining 'dma-ranges'. Which so far for
	      most devices is 4G, except for RPi4.
- ZONE_NORMAL: The rest of the memory.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-18 11:18     ` Nicolas Saenz Julienne
@ 2019-07-19 13:08       ` Nicolas Saenz Julienne
  2019-07-24 13:51         ` Catalin Marinas
  0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-19 13:08 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: stefan.wahren, f.fainelli, catalin.marinas, phil, Robin Murphy,
	iommu, linux-kernel, Jisheng.Zhang, mbrugger, will,
	linux-arm-kernel, Marek Szyprowski


[-- Attachment #1.1: Type: text/plain, Size: 2257 bytes --]

On Thu, 2019-07-18 at 13:18 +0200, Nicolas Saenz Julienne wrote:
> On Thu, 2019-07-18 at 11:15 +0200, Christoph Hellwig wrote:
> > On Wed, Jul 17, 2019 at 05:31:34PM +0200, Nicolas Saenz Julienne wrote:
> > > Historically devices with ZONE_DMA32 have been assumed to be able to
> > > address at least the lower 4GB of ram for DMA. This is still the defualt
> > > behavior yet the Raspberry Pi 4 is limited to the first GB of memory.
> > > This has been observed to trigger failures in dma_direct_supported() as
> > > the 'min_mask' isn't properly set.
> > > 
> > > We create 'dma_direct_min_mask' in order for the arch init code to be
> > > able to fine-tune dma direct's 'min_dma' mask.
> > 
> > Normally we use ZONE_DMA for that case.
> 
> Fair enough, I didn't think of that possibility.
> 
> So would the arm64 maintainers be happy with something like this:
> 
> - ZONE_DMA: Follows standard definition, 16MB in size. ARCH_ZONE_DMA_BITS is
> 	    left as is.
> - ZONE_DMA32: Will honor the most constraining 'dma-ranges'. Which so far for
> 	      most devices is 4G, except for RPi4.
> - ZONE_NORMAL: The rest of the memory.

Never mind this suggestion, I don't think it makes any sense. If anything arm64
seems to fit the ZONE_DMA usage pattern of arm and powerpc: where ZONE_DMA's
size is decided based on ram size and/or board configuration. It was actually
set-up like this until Christoph's ad67f5a6545f7 ("arm64: replace ZONE_DMA with
ZONE_DMA32").

So the easy solution would be to simply revert that commit. On one hand I feel
it would be a step backwards as most 64 bit architectures have been moving to
use ZONE_DMA32. On the other, current ZONE_DMA32 usage seems to be heavily
rooted on having a 32 bit DMA mask*, which will no longer be the case on arm64
if we want to support the RPi 4.

So the way I see it and lacking a better solution, the argument is stronger on
moving back arm64 to using ZONE_DMA. Any comments/opinions?

Note that I've been looking at all the DMA/CMA/swiotlb code to see if this
would break anything or change behaviors and couldn't find anything obvious. I
also tested the revert on my RPi4 and nothing seems to fail.

* A good example is dma-direct's implementation.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-19 13:08       ` Nicolas Saenz Julienne
@ 2019-07-24 13:51         ` Catalin Marinas
  2019-07-24 13:56           ` Christoph Hellwig
  0 siblings, 1 reply; 12+ messages in thread
From: Catalin Marinas @ 2019-07-24 13:51 UTC (permalink / raw)
  To: Nicolas Saenz Julienne
  Cc: stefan.wahren, f.fainelli, Robin Murphy, phil, iommu,
	linux-kernel, Jisheng.Zhang, mbrugger, will, Christoph Hellwig,
	linux-arm-kernel, Marek Szyprowski

On Fri, Jul 19, 2019 at 03:08:52PM +0200, Nicolas Saenz Julienne wrote:
> On Thu, 2019-07-18 at 13:18 +0200, Nicolas Saenz Julienne wrote:
> > On Thu, 2019-07-18 at 11:15 +0200, Christoph Hellwig wrote:
> > > On Wed, Jul 17, 2019 at 05:31:34PM +0200, Nicolas Saenz Julienne wrote:
> > > > Historically devices with ZONE_DMA32 have been assumed to be able to
> > > > address at least the lower 4GB of ram for DMA. This is still the defualt
> > > > behavior yet the Raspberry Pi 4 is limited to the first GB of memory.
> > > > This has been observed to trigger failures in dma_direct_supported() as
> > > > the 'min_mask' isn't properly set.
> > > > 
> > > > We create 'dma_direct_min_mask' in order for the arch init code to be
> > > > able to fine-tune dma direct's 'min_dma' mask.
> > > 
> > > Normally we use ZONE_DMA for that case.
> > 
> > Fair enough, I didn't think of that possibility.
> > 
> > So would the arm64 maintainers be happy with something like this:
> > 
> > - ZONE_DMA: Follows standard definition, 16MB in size. ARCH_ZONE_DMA_BITS is
> > 	    left as is.
> > - ZONE_DMA32: Will honor the most constraining 'dma-ranges'. Which so far for
> > 	      most devices is 4G, except for RPi4.
> > - ZONE_NORMAL: The rest of the memory.
> 
> Never mind this suggestion, I don't think it makes any sense. If anything arm64
> seems to fit the ZONE_DMA usage pattern of arm and powerpc: where ZONE_DMA's
> size is decided based on ram size and/or board configuration. It was actually
> set-up like this until Christoph's ad67f5a6545f7 ("arm64: replace ZONE_DMA with
> ZONE_DMA32").
> 
> So the easy solution would be to simply revert that commit. On one hand I feel
> it would be a step backwards as most 64 bit architectures have been moving to
> use ZONE_DMA32. On the other, current ZONE_DMA32 usage seems to be heavily
> rooted on having a 32 bit DMA mask*, which will no longer be the case on arm64
> if we want to support the RPi 4.
> 
> So the way I see it and lacking a better solution, the argument is stronger on
> moving back arm64 to using ZONE_DMA. Any comments/opinions?

As it was suggested in this or the previous thread, I'm not keen on
limiting ZONE_DMA32 to the smalles RPi4 can cover, as the naming
implies this zone should cover 32-bit devices that can deal with a full
32-bit mask.

I think it may be better if we have both ZONE_DMA and ZONE_DMA32 on
arm64. ZONE_DMA would be based on the smallest dma-ranges as described
in the DT while DMA32 covers the first naturally aligned 4GB of RAM
(unchanged). When a smaller ZONE_DMA is not needed, it could be expanded
to cover what would normally be ZONE_DMA32 (or could we have ZONE_DMA as
0-bytes? I don't think GFP_DMA can still allocate memory in this case).

We'd probably have to define ARCH_ZONE_DMA_BITS for arm64 to something
smaller than 32-bit but sufficient to cover the known platforms like
RPi4 (the current 24 is too small, so maybe 30). AFAICT,
__dma_direct_optimal_gfp_mask() figures out whether GFP_DMA or GFP_DMA32
should be passed.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit
  2019-07-17 15:31 ` [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit Nicolas Saenz Julienne
@ 2019-07-24 13:54   ` Catalin Marinas
  0 siblings, 0 replies; 12+ messages in thread
From: Catalin Marinas @ 2019-07-24 13:54 UTC (permalink / raw)
  To: Nicolas Saenz Julienne
  Cc: stefan.wahren, f.fainelli, will, phil, linux-kernel,
	Jisheng.Zhang, mbrugger, robin.murphy, hch, linux-arm-kernel,
	m.szyprowski

On Wed, Jul 17, 2019 at 05:31:33PM +0200, Nicolas Saenz Julienne wrote:
> The dma physical limit has so far been calculated based on the memory
> size and the assumption that dma would be at least able to address the
> first 4 GB. This turned out no to be true with the Raspberry Pi 4
> which, on it's main interconnect, can only address the first GB of
> memory, even though it might have up to 4 GB.
> 
> With the current miscalculated dma physical limit the contiguous memory
> reserve is located in an inaccessible area for most of the board's
> devices.
> 
> To solve this we now scan the device tree for the 'dma-ranges' property
> on the root or interconnect nodes, which allows us to calculate the
> lowest common denominator dma physical limit. If no dma-ranges is
> available, we'll default to the old scheme.
> 
> Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> ---
>  arch/arm64/mm/init.c | 61 +++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 57 insertions(+), 4 deletions(-)

I'd rather have this parsing in the core code, returning setting the
minimum DMA mask (or range, address etc.) that covers all devices/buses
described.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-24 13:51         ` Catalin Marinas
@ 2019-07-24 13:56           ` Christoph Hellwig
  2019-07-24 14:27             ` Nicolas Saenz Julienne
  0 siblings, 1 reply; 12+ messages in thread
From: Christoph Hellwig @ 2019-07-24 13:56 UTC (permalink / raw)
  To: Catalin Marinas
  Cc: stefan.wahren, f.fainelli, Robin Murphy, phil, iommu,
	linux-kernel, Jisheng.Zhang, mbrugger, Nicolas Saenz Julienne,
	will, Christoph Hellwig, linux-arm-kernel, Marek Szyprowski

On Wed, Jul 24, 2019 at 02:51:24PM +0100, Catalin Marinas wrote:
> I think it may be better if we have both ZONE_DMA and ZONE_DMA32 on
> arm64. ZONE_DMA would be based on the smallest dma-ranges as described
> in the DT while DMA32 covers the first naturally aligned 4GB of RAM
> (unchanged). When a smaller ZONE_DMA is not needed, it could be expanded
> to cover what would normally be ZONE_DMA32 (or could we have ZONE_DMA as
> 0-bytes? I don't think GFP_DMA can still allocate memory in this case).
> 
> We'd probably have to define ARCH_ZONE_DMA_BITS for arm64 to something
> smaller than 32-bit but sufficient to cover the known platforms like
> RPi4 (the current 24 is too small, so maybe 30). AFAICT,
> __dma_direct_optimal_gfp_mask() figures out whether GFP_DMA or GFP_DMA32
> should be passed.

ARCH_ZONE_DMA_BITS should probably become a variable.  That way we can
just initialize it to the default 24 bits in kernel/dma/direct.c and
allow architectures to override it in their early boot code.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC 3/4] dma-direct: add dma_direct_min_mask
  2019-07-24 13:56           ` Christoph Hellwig
@ 2019-07-24 14:27             ` Nicolas Saenz Julienne
  0 siblings, 0 replies; 12+ messages in thread
From: Nicolas Saenz Julienne @ 2019-07-24 14:27 UTC (permalink / raw)
  To: Christoph Hellwig, Catalin Marinas
  Cc: stefan.wahren, Jisheng.Zhang, f.fainelli, will, phil,
	linux-kernel, iommu, mbrugger, Robin Murphy, linux-arm-kernel,
	Marek Szyprowski


[-- Attachment #1.1: Type: text/plain, Size: 1191 bytes --]

On Wed, 2019-07-24 at 15:56 +0200, Christoph Hellwig wrote:
> On Wed, Jul 24, 2019 at 02:51:24PM +0100, Catalin Marinas wrote:
> > I think it may be better if we have both ZONE_DMA and ZONE_DMA32 on
> > arm64. ZONE_DMA would be based on the smallest dma-ranges as described
> > in the DT while DMA32 covers the first naturally aligned 4GB of RAM
> > (unchanged). When a smaller ZONE_DMA is not needed, it could be expanded
> > to cover what would normally be ZONE_DMA32 (or could we have ZONE_DMA as
> > 0-bytes? I don't think GFP_DMA can still allocate memory in this case).
> > 
> > We'd probably have to define ARCH_ZONE_DMA_BITS for arm64 to something
> > smaller than 32-bit but sufficient to cover the known platforms like
> > RPi4 (the current 24 is too small, so maybe 30). AFAICT,
> > __dma_direct_optimal_gfp_mask() figures out whether GFP_DMA or GFP_DMA32
> > should be passed.
> 
> ARCH_ZONE_DMA_BITS should probably become a variable.  That way we can
> just initialize it to the default 24 bits in kernel/dma/direct.c and
> allow architectures to override it in their early boot code.

Thanks both for your feedback. I'll start preparing a proper series.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2019-07-24 14:28 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-17 15:31 [RFC 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
2019-07-17 15:31 ` [RFC 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
2019-07-17 15:31 ` [RFC 2/4] arm64: mm: parse dma-ranges in order to better estimate arm64_dma_phys_limit Nicolas Saenz Julienne
2019-07-24 13:54   ` Catalin Marinas
2019-07-17 15:31 ` [RFC 3/4] dma-direct: add dma_direct_min_mask Nicolas Saenz Julienne
2019-07-18  9:15   ` Christoph Hellwig
2019-07-18 11:18     ` Nicolas Saenz Julienne
2019-07-19 13:08       ` Nicolas Saenz Julienne
2019-07-24 13:51         ` Catalin Marinas
2019-07-24 13:56           ` Christoph Hellwig
2019-07-24 14:27             ` Nicolas Saenz Julienne
2019-07-17 15:31 ` [RFC 4/4] arm64: mm: set direct_dma_min_mask according to dma-ranges Nicolas Saenz Julienne

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).