linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
@ 2019-09-09  9:58 Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
                   ` (4 more replies)
  0 siblings, 5 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09  9:58 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv
  Cc: f.fainelli, will, robin.murphy, nsaenzjulienne, linux-kernel,
	mbrugger, linux-rpi-kernel, phill, m.szyprowski

Hi all,
this series attempts to address some issues we found while bringing up
the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
up of these discussions:
v4: https://lkml.org/lkml/2019/9/6/352
v3: https://lkml.org/lkml/2019/9/2/589
v2: https://lkml.org/lkml/2019/8/20/767
v1: https://lkml.org/lkml/2019/7/31/922
RFC: https://lkml.org/lkml/2019/7/17/476

The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
only address the first GB: their DMA address range is
0xc0000000-0xfc000000 which is aliased to the first GB of physical
memory 0x00000000-0x3c000000. Note that only some peripherals have these
limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
view of the address space by virtue of being hooked up trough a second
interconnect.

Part of this is solved on arm32 by setting up the machine specific
'.dma_zone_size = SZ_1G', which takes care of reserving the coherent
memory area at the right spot. That said no buffer bouncing (needed for
dma streaming) is available at the moment, but that's a story for
another series.

Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
arch code as if all peripherals where be able to address the first 4GB
of memory.

In the light of this, the series implements the following changes:

- Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
  area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
  the RPi4 is the only arm64 device with such DMA addressing limitations
  so this hardcoded solution was deemed preferable.

- Properly set ARCH_ZONE_DMA_BITS.

- Reserve the CMA area in a place suitable for all peripherals.

This series has been tested on multiple devices both by checking the
zones setup matches the expectations and by double-checking physical
addresses on pages allocated on the three relevant areas GFP_DMA,
GFP_DMA32, GFP_KERNEL:

- On an RPi4 with variations on the ram memory size. But also forcing
  the situation where all three memory zones are nonempty by setting a 3G
  ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.

- On a Synquacer box[1] with 32G of memory.

- On an ACPI based Huawei TaiShan server[2] with 256G of memory.

- On a QEMU virtual machine running arm64's OpenSUSE Tumbleweed.

That's all.

Regards,
Nicolas

[1] https://www.96boards.org/product/developerbox/
[2] https://e.huawei.com/en/products/cloud-computing-dc/servers/taishan-server/taishan-2280-v2

---

Changes in v5:
- Fix issue with swiotlb initialization

Changes in v4:
- Rebased to linux-next
- Fix issue when NUMA=n and ZONE_DMA=n
- Merge two max_zone_dma*_phys() functions

Changes in v3:
- Fixed ZONE_DMA's size to 1G
- Update mmzone.h's comment to match changes in arm64
- Remove all dma-direct patches

Changes in v2:
- Update comment to reflect new zones split
- ZONE_DMA will never be left empty
- Try another approach merging both ZONE_DMA comments into one
- Address Christoph's comments
- If this approach doesn't get much traction I'll just drop the patch
  from the series as it's not really essential

Nicolas Saenz Julienne (4):
  arm64: mm: use arm64_dma_phys_limit instead of calling
    max_zone_dma_phys()
  arm64: rename variables used to calculate ZONE_DMA32's size
  arm64: use both ZONE_DMA and ZONE_DMA32
  mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type'

 arch/arm64/Kconfig            |  4 ++
 arch/arm64/include/asm/page.h |  2 +
 arch/arm64/mm/init.c          | 71 +++++++++++++++++++++++++----------
 include/linux/mmzone.h        | 45 ++++++++++++----------
 4 files changed, 83 insertions(+), 39 deletions(-)

-- 
2.23.0



^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH v5 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys()
  2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
@ 2019-09-09  9:58 ` Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 2/4] arm64: rename variables used to calculate ZONE_DMA32's size Nicolas Saenz Julienne
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09  9:58 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv, linux-kernel
  Cc: f.fainelli, will, robin.murphy, nsaenzjulienne, mbrugger,
	linux-rpi-kernel, phill, m.szyprowski

By the time we call zones_sizes_init() arm64_dma_phys_limit already
contains the result of max_zone_dma_phys(). We use the variable instead
of calling the function directly to save some precious cpu time.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm64/mm/init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index f3c795278def..6112d6c90fa8 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -181,7 +181,7 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 	unsigned long max_zone_pfns[MAX_NR_ZONES]  = {0};
 
 #ifdef CONFIG_ZONE_DMA32
-	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(max_zone_dma_phys());
+	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(arm64_dma_phys_limit);
 #endif
 	max_zone_pfns[ZONE_NORMAL] = max;
 
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v5 2/4] arm64: rename variables used to calculate ZONE_DMA32's size
  2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
@ 2019-09-09  9:58 ` Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 Nicolas Saenz Julienne
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09  9:58 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv, linux-kernel
  Cc: f.fainelli, will, robin.murphy, nsaenzjulienne, mbrugger,
	linux-rpi-kernel, phill, m.szyprowski

Let the name indicate that they are used to calculate ZONE_DMA32's size
as opposed to ZONE_DMA.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm64/mm/init.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 098c0f5bedf6..8e9bc64c5878 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -56,7 +56,7 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
-phys_addr_t arm64_dma_phys_limit __ro_after_init;
+phys_addr_t arm64_dma32_phys_limit __ro_after_init;
 
 #ifdef CONFIG_KEXEC_CORE
 /*
@@ -174,7 +174,7 @@ static void __init reserve_elfcorehdr(void)
  * currently assumes that for memory starting above 4G, 32-bit devices will
  * use a DMA offset.
  */
-static phys_addr_t __init max_zone_dma_phys(void)
+static phys_addr_t __init max_zone_dma32_phys(void)
 {
 	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
 	return min(offset + (1ULL << 32), memblock_end_of_DRAM());
@@ -187,7 +187,7 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 	unsigned long max_zone_pfns[MAX_NR_ZONES]  = {0};
 
 #ifdef CONFIG_ZONE_DMA32
-	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(arm64_dma_phys_limit);
+	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(arm64_dma32_phys_limit);
 #endif
 	max_zone_pfns[ZONE_NORMAL] = max;
 
@@ -200,16 +200,16 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 {
 	struct memblock_region *reg;
 	unsigned long zone_size[MAX_NR_ZONES], zhole_size[MAX_NR_ZONES];
-	unsigned long max_dma = min;
+	unsigned long max_dma32 = min;
 
 	memset(zone_size, 0, sizeof(zone_size));
 
 	/* 4GB maximum for 32-bit only capable devices */
 #ifdef CONFIG_ZONE_DMA32
-	max_dma = PFN_DOWN(arm64_dma_phys_limit);
-	zone_size[ZONE_DMA32] = max_dma - min;
+	max_dma32 = PFN_DOWN(arm64_dma32_phys_limit);
+	zone_size[ZONE_DMA32] = max_dma32 - min;
 #endif
-	zone_size[ZONE_NORMAL] = max - max_dma;
+	zone_size[ZONE_NORMAL] = max - max_dma32;
 
 	memcpy(zhole_size, zone_size, sizeof(zhole_size));
 
@@ -221,14 +221,14 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 			continue;
 
 #ifdef CONFIG_ZONE_DMA32
-		if (start < max_dma) {
-			unsigned long dma_end = min(end, max_dma);
+		if (start < max_dma32) {
+			unsigned long dma_end = min(end, max_dma32);
 			zhole_size[ZONE_DMA32] -= dma_end - start;
 		}
 #endif
-		if (end > max_dma) {
+		if (end > max_dma32) {
 			unsigned long normal_end = min(end, max);
-			unsigned long normal_start = max(start, max_dma);
+			unsigned long normal_start = max(start, max_dma32);
 			zhole_size[ZONE_NORMAL] -= normal_end - normal_start;
 		}
 	}
@@ -420,9 +420,9 @@ void __init arm64_memblock_init(void)
 
 	/* 4GB maximum for 32-bit only capable devices */
 	if (IS_ENABLED(CONFIG_ZONE_DMA32))
-		arm64_dma_phys_limit = max_zone_dma_phys();
+		arm64_dma32_phys_limit = max_zone_dma32_phys();
 	else
-		arm64_dma_phys_limit = PHYS_MASK + 1;
+		arm64_dma32_phys_limit = PHYS_MASK + 1;
 
 	reserve_crashkernel();
 
@@ -430,7 +430,7 @@ void __init arm64_memblock_init(void)
 
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
-	dma_contiguous_reserve(arm64_dma_phys_limit);
+	dma_contiguous_reserve(arm64_dma32_phys_limit);
 }
 
 void __init bootmem_init(void)
@@ -534,7 +534,7 @@ static void __init free_unused_memmap(void)
 void __init mem_init(void)
 {
 	if (swiotlb_force == SWIOTLB_FORCE ||
-	    max_pfn > (arm64_dma_phys_limit >> PAGE_SHIFT))
+	    max_pfn > (arm64_dma32_phys_limit >> PAGE_SHIFT))
 		swiotlb_init(1);
 	else
 		swiotlb_force = SWIOTLB_NO_FORCE;
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32
  2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 2/4] arm64: rename variables used to calculate ZONE_DMA32's size Nicolas Saenz Julienne
@ 2019-09-09  9:58 ` Nicolas Saenz Julienne
  2019-09-11 10:54   ` Nicolas Saenz Julienne
  2019-09-09  9:58 ` [PATCH v5 4/4] mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type' Nicolas Saenz Julienne
  2019-09-09 19:33 ` [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Stefan Wahren
  4 siblings, 1 reply; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09  9:58 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv, Will Deacon
  Cc: f.fainelli, robin.murphy, nsaenzjulienne, linux-kernel, mbrugger,
	linux-rpi-kernel, phill, m.szyprowski

So far all arm64 devices have supported 32 bit DMA masks for their
peripherals. This is not true anymore for the Raspberry Pi 4 as most of
it's peripherals can only address the first GB of memory on a total of
up to 4 GB.

This goes against ZONE_DMA32's intent, as it's expected for ZONE_DMA32
to be addressable with a 32 bit mask. So it was decided to re-introduce
ZONE_DMA in arm64.

ZONE_DMA will contain the lower 1G of memory, which is currently the
memory area addressable by any peripheral on an arm64 device.
ZONE_DMA32 will contain the rest of the 32 bit addressable memory.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

---

Changes in v5:
- Fixed swiotlb initialization

Changes in v4:
- Fixed issue when NUMA=n and ZONE_DMA=n
- Merged two max_zone_dma*_phys() functions

Changes in v3:
- Used fixed size ZONE_DMA
- Fix check befor swiotlb_init()

Changes in v2:
- Update comment to reflect new zones split
- ZONE_DMA will never be left empty

 arch/arm64/Kconfig            |  4 +++
 arch/arm64/include/asm/page.h |  2 ++
 arch/arm64/mm/init.c          | 52 +++++++++++++++++++++++++----------
 3 files changed, 44 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6b6362b83004..2dbe0165bd15 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -267,6 +267,10 @@ config GENERIC_CSUM
 config GENERIC_CALIBRATE_DELAY
 	def_bool y
 
+config ZONE_DMA
+	bool "Support DMA zone" if EXPERT
+	default y
+
 config ZONE_DMA32
 	bool "Support DMA32 zone" if EXPERT
 	default y
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index d39ddb258a04..7b8c98830101 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -38,4 +38,6 @@ extern int pfn_valid(unsigned long);
 
 #include <asm-generic/getorder.h>
 
+#define ARCH_ZONE_DMA_BITS 30
+
 #endif
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 8e9bc64c5878..92c911fc2ff9 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -56,6 +56,13 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
+/*
+ * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first 1G of
+ * memory as some devices, namely the Raspberry Pi 4, have peripherals with
+ * this limited view of the memory. ZONE_DMA32 will cover the rest of the 32
+ * bit addressable memory area.
+ */
+phys_addr_t arm64_dma_phys_limit __ro_after_init;
 phys_addr_t arm64_dma32_phys_limit __ro_after_init;
 
 #ifdef CONFIG_KEXEC_CORE
@@ -169,15 +176,16 @@ static void __init reserve_elfcorehdr(void)
 {
 }
 #endif /* CONFIG_CRASH_DUMP */
+
 /*
- * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)). It
- * currently assumes that for memory starting above 4G, 32-bit devices will
- * use a DMA offset.
+ * Return the maximum physical address for a zone with a given address size
+ * limit. It currently assumes that for memory starting above 4G, 32-bit
+ * devices will use a DMA offset.
  */
-static phys_addr_t __init max_zone_dma32_phys(void)
+static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
 {
 	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
-	return min(offset + (1ULL << 32), memblock_end_of_DRAM());
+	return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
 }
 
 #ifdef CONFIG_NUMA
@@ -186,6 +194,9 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 {
 	unsigned long max_zone_pfns[MAX_NR_ZONES]  = {0};
 
+#ifdef CONFIG_ZONE_DMA
+	max_zone_pfns[ZONE_DMA] = PFN_DOWN(arm64_dma_phys_limit);
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(arm64_dma32_phys_limit);
 #endif
@@ -201,13 +212,18 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 	struct memblock_region *reg;
 	unsigned long zone_size[MAX_NR_ZONES], zhole_size[MAX_NR_ZONES];
 	unsigned long max_dma32 = min;
+	unsigned long max_dma = min;
 
 	memset(zone_size, 0, sizeof(zone_size));
 
-	/* 4GB maximum for 32-bit only capable devices */
+#ifdef CONFIG_ZONE_DMA
+	max_dma = PFN_DOWN(arm64_dma_phys_limit);
+	zone_size[ZONE_DMA] = max_dma - min;
+	max_dma32 = max_dma;
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	max_dma32 = PFN_DOWN(arm64_dma32_phys_limit);
-	zone_size[ZONE_DMA32] = max_dma32 - min;
+	zone_size[ZONE_DMA32] = max_dma32 - max_dma;
 #endif
 	zone_size[ZONE_NORMAL] = max - max_dma32;
 
@@ -219,11 +235,17 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
 
 		if (start >= max)
 			continue;
-
+#ifdef CONFIG_ZONE_DMA
+		if (start < max_dma) {
+			unsigned long dma_end = min_not_zero(end, max_dma);
+			zhole_size[ZONE_DMA] -= dma_end - start;
+		}
+#endif
 #ifdef CONFIG_ZONE_DMA32
 		if (start < max_dma32) {
-			unsigned long dma_end = min(end, max_dma32);
-			zhole_size[ZONE_DMA32] -= dma_end - start;
+			unsigned long dma32_end = min(end, max_dma32);
+			unsigned long dma32_start = max(start, max_dma);
+			zhole_size[ZONE_DMA32] -= dma32_end - dma32_start;
 		}
 #endif
 		if (end > max_dma32) {
@@ -418,9 +440,11 @@ void __init arm64_memblock_init(void)
 
 	early_init_fdt_scan_reserved_mem();
 
-	/* 4GB maximum for 32-bit only capable devices */
+	if (IS_ENABLED(CONFIG_ZONE_DMA))
+		arm64_dma_phys_limit = max_zone_phys(ARCH_ZONE_DMA_BITS);
+
 	if (IS_ENABLED(CONFIG_ZONE_DMA32))
-		arm64_dma32_phys_limit = max_zone_dma32_phys();
+		arm64_dma32_phys_limit = max_zone_phys(32);
 	else
 		arm64_dma32_phys_limit = PHYS_MASK + 1;
 
@@ -430,7 +454,7 @@ void __init arm64_memblock_init(void)
 
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
-	dma_contiguous_reserve(arm64_dma32_phys_limit);
+	dma_contiguous_reserve(arm64_dma_phys_limit ? : arm64_dma32_phys_limit);
 }
 
 void __init bootmem_init(void)
@@ -534,7 +558,7 @@ static void __init free_unused_memmap(void)
 void __init mem_init(void)
 {
 	if (swiotlb_force == SWIOTLB_FORCE ||
-	    max_pfn > (arm64_dma32_phys_limit >> PAGE_SHIFT))
+	    max_pfn > PFN_DOWN(arm64_dma_phys_limit ? : arm64_dma32_phys_limit))
 		swiotlb_init(1);
 	else
 		swiotlb_force = SWIOTLB_NO_FORCE;
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v5 4/4] mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type'
  2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
                   ` (2 preceding siblings ...)
  2019-09-09  9:58 ` [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 Nicolas Saenz Julienne
@ 2019-09-09  9:58 ` Nicolas Saenz Julienne
  2019-09-09 19:33 ` [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Stefan Wahren
  4 siblings, 0 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09  9:58 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv, Paul Walmsley,
	Palmer Dabbelt, Albert Ou
  Cc: f.fainelli, will, robin.murphy, nsaenzjulienne, linux-kernel,
	mbrugger, linux-rpi-kernel, phill, m.szyprowski

These zones usage has evolved with time and the comments were outdated.
This joins both ZONE_DMA and ZONE_DMA32 explanation and gives up to date
examples on how they are used on different architectures.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

---

Changes in v3:
- Update comment to match changes in arm64

Changes in v2:
- Try another approach merging both ZONE_DMA comments into one
- Address Christoph's comments
- If this approach doesn't get much traction I'll just drop the patch
  from the series as it's not really essential

 include/linux/mmzone.h | 45 ++++++++++++++++++++++++------------------
 1 file changed, 26 insertions(+), 19 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 3f38c30d2f13..bf1b916c9ecb 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -357,33 +357,40 @@ struct per_cpu_nodestat {
 #endif /* !__GENERATING_BOUNDS.H */
 
 enum zone_type {
-#ifdef CONFIG_ZONE_DMA
 	/*
-	 * ZONE_DMA is used when there are devices that are not able
-	 * to do DMA to all of addressable memory (ZONE_NORMAL). Then we
-	 * carve out the portion of memory that is needed for these devices.
-	 * The range is arch specific.
+	 * ZONE_DMA and ZONE_DMA32 are used when there are peripherals not able
+	 * to DMA to all of the addressable memory (ZONE_NORMAL).
+	 * On architectures where this area covers the whole 32 bit address
+	 * space ZONE_DMA32 is used. ZONE_DMA is left for the ones with smaller
+	 * DMA addressing constraints. This distinction is important as a 32bit
+	 * DMA mask is assumed when ZONE_DMA32 is defined. Some 64-bit
+	 * platforms may need both zones as they support peripherals with
+	 * different DMA addressing limitations.
+	 *
+	 * Some examples:
+	 *
+	 *  - i386 and x86_64 have a fixed 16M ZONE_DMA and ZONE_DMA32 for the
+	 *    rest of the lower 4G.
+	 *
+	 *  - arm only uses ZONE_DMA, the size, up to 4G, may vary depending on
+	 *    the specific device.
+	 *
+	 *  - arm64 has a fixed 1G ZONE_DMA and ZONE_DMA32 for the rest of the
+	 *    lower 4G.
 	 *
-	 * Some examples
+	 *  - powerpc only uses ZONE_DMA, the size, up to 2G, may vary
+	 *    depending on the specific device.
 	 *
-	 * Architecture		Limit
-	 * ---------------------------
-	 * parisc, ia64, sparc	<4G
-	 * s390, powerpc	<2G
-	 * arm			Various
-	 * alpha		Unlimited or 0-16MB.
+	 *  - s390 uses ZONE_DMA fixed to the lower 2G.
 	 *
-	 * i386, x86_64 and multiple other arches
-	 * 			<16M.
+	 *  - ia64 and riscv only use ZONE_DMA32.
+	 *
+	 *  - parisc uses neither.
 	 */
+#ifdef CONFIG_ZONE_DMA
 	ZONE_DMA,
 #endif
 #ifdef CONFIG_ZONE_DMA32
-	/*
-	 * x86_64 needs two ZONE_DMAs because it supports devices that are
-	 * only able to do DMA to the lower 16M but also 32 bit devices that
-	 * can only do DMA areas below 4G.
-	 */
 	ZONE_DMA32,
 #endif
 	/*
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
                   ` (3 preceding siblings ...)
  2019-09-09  9:58 ` [PATCH v5 4/4] mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type' Nicolas Saenz Julienne
@ 2019-09-09 19:33 ` Stefan Wahren
  2019-09-09 19:50   ` Nicolas Saenz Julienne
  2019-09-10  9:27   ` Matthias Brugger
  4 siblings, 2 replies; 20+ messages in thread
From: Stefan Wahren @ 2019-09-09 19:33 UTC (permalink / raw)
  To: Nicolas Saenz Julienne, catalin.marinas, hch, marc.zyngier,
	robh+dt, linux-arm-kernel, linux-mm, linux-riscv
  Cc: f.fainelli, will, linux-kernel, mbrugger, linux-rpi-kernel,
	phill, robin.murphy, m.szyprowski

Hi Nicolas,

Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
> Hi all,
> this series attempts to address some issues we found while bringing up
> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
> up of these discussions:
> v4: https://lkml.org/lkml/2019/9/6/352
> v3: https://lkml.org/lkml/2019/9/2/589
> v2: https://lkml.org/lkml/2019/8/20/767
> v1: https://lkml.org/lkml/2019/7/31/922
> RFC: https://lkml.org/lkml/2019/7/17/476
>
> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
> only address the first GB: their DMA address range is
> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
> memory 0x00000000-0x3c000000. Note that only some peripherals have these
> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
> view of the address space by virtue of being hooked up trough a second
> interconnect.
>
> Part of this is solved on arm32 by setting up the machine specific
> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
> memory area at the right spot. That said no buffer bouncing (needed for
> dma streaming) is available at the moment, but that's a story for
> another series.
>
> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
> arch code as if all peripherals where be able to address the first 4GB
> of memory.
>
> In the light of this, the series implements the following changes:
>
> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>   the RPi4 is the only arm64 device with such DMA addressing limitations
>   so this hardcoded solution was deemed preferable.
>
> - Properly set ARCH_ZONE_DMA_BITS.
>
> - Reserve the CMA area in a place suitable for all peripherals.
>
> This series has been tested on multiple devices both by checking the
> zones setup matches the expectations and by double-checking physical
> addresses on pages allocated on the three relevant areas GFP_DMA,
> GFP_DMA32, GFP_KERNEL:
>
> - On an RPi4 with variations on the ram memory size. But also forcing
>   the situation where all three memory zones are nonempty by setting a 3G
>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>
i like to test this series on Raspberry Pi 4 and i have some questions
to get arm64 running:

Do you use U-Boot? Which tree?
Are there any config.txt tweaks necessary?



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-09 19:33 ` [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Stefan Wahren
@ 2019-09-09 19:50   ` Nicolas Saenz Julienne
  2019-09-10  9:27   ` Matthias Brugger
  1 sibling, 0 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-09 19:50 UTC (permalink / raw)
  To: Stefan Wahren, catalin.marinas, hch, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv
  Cc: f.fainelli, will, linux-kernel, mbrugger, linux-rpi-kernel,
	phill, robin.murphy, m.szyprowski

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

On Mon, 2019-09-09 at 21:33 +0200, Stefan Wahren wrote:
> Hi Nicolas,
> 
> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
> > Hi all,
> > this series attempts to address some issues we found while bringing up
> > the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
> > up of these discussions:
> > v4: https://lkml.org/lkml/2019/9/6/352
> > v3: https://lkml.org/lkml/2019/9/2/589
> > v2: https://lkml.org/lkml/2019/8/20/767
> > v1: https://lkml.org/lkml/2019/7/31/922
> > RFC: https://lkml.org/lkml/2019/7/17/476
> > 
> > The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
> > only address the first GB: their DMA address range is
> > 0xc0000000-0xfc000000 which is aliased to the first GB of physical
> > memory 0x00000000-0x3c000000. Note that only some peripherals have these
> > limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
> > view of the address space by virtue of being hooked up trough a second
> > interconnect.
> > 
> > Part of this is solved on arm32 by setting up the machine specific
> > '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
> > memory area at the right spot. That said no buffer bouncing (needed for
> > dma streaming) is available at the moment, but that's a story for
> > another series.
> > 
> > Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
> > ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
> > arch code as if all peripherals where be able to address the first 4GB
> > of memory.
> > 
> > In the light of this, the series implements the following changes:
> > 
> > - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
> >   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
> >   the RPi4 is the only arm64 device with such DMA addressing limitations
> >   so this hardcoded solution was deemed preferable.
> > 
> > - Properly set ARCH_ZONE_DMA_BITS.
> > 
> > - Reserve the CMA area in a place suitable for all peripherals.
> > 
> > This series has been tested on multiple devices both by checking the
> > zones setup matches the expectations and by double-checking physical
> > addresses on pages allocated on the three relevant areas GFP_DMA,
> > GFP_DMA32, GFP_KERNEL:
> > 
> > - On an RPi4 with variations on the ram memory size. But also forcing
> >   the situation where all three memory zones are nonempty by setting a 3G
> >   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
> > 
> i like to test this series on Raspberry Pi 4 and i have some questions
> to get arm64 running:
> 
> Do you use U-Boot? Which tree?

No, I boot directly.

> Are there any config.txt tweaks necessary?

I'm using the foundation's arm64 stub. Though I'm not 100% it's needed anymore
with the latest firmware.

config.txt:
	arm_64bit=1
	armstub=armstub8-gic.bin
	enable_gic=1
	enable_uart=1

Apart from that the series is based on today's linux-next plus your RPi4
bringup patches.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-09 19:33 ` [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Stefan Wahren
  2019-09-09 19:50   ` Nicolas Saenz Julienne
@ 2019-09-10  9:27   ` Matthias Brugger
  2019-09-12 17:18     ` Matthias Brugger
  1 sibling, 1 reply; 20+ messages in thread
From: Matthias Brugger @ 2019-09-10  9:27 UTC (permalink / raw)
  To: Stefan Wahren, catalin.marinas, marc.zyngier, robh+dt, linux-mm,
	linux-arm-kernel, linux-riscv, hch, Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phill,
	m.szyprowski, linux-kernel



On 09/09/2019 21:33, Stefan Wahren wrote:
> Hi Nicolas,
> 
> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>> Hi all,
>> this series attempts to address some issues we found while bringing up
>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>> up of these discussions:
>> v4: https://lkml.org/lkml/2019/9/6/352
>> v3: https://lkml.org/lkml/2019/9/2/589
>> v2: https://lkml.org/lkml/2019/8/20/767
>> v1: https://lkml.org/lkml/2019/7/31/922
>> RFC: https://lkml.org/lkml/2019/7/17/476
>>
>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>> only address the first GB: their DMA address range is
>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>> view of the address space by virtue of being hooked up trough a second
>> interconnect.
>>
>> Part of this is solved on arm32 by setting up the machine specific
>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>> memory area at the right spot. That said no buffer bouncing (needed for
>> dma streaming) is available at the moment, but that's a story for
>> another series.
>>
>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>> arch code as if all peripherals where be able to address the first 4GB
>> of memory.
>>
>> In the light of this, the series implements the following changes:
>>
>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>   so this hardcoded solution was deemed preferable.
>>
>> - Properly set ARCH_ZONE_DMA_BITS.
>>
>> - Reserve the CMA area in a place suitable for all peripherals.
>>
>> This series has been tested on multiple devices both by checking the
>> zones setup matches the expectations and by double-checking physical
>> addresses on pages allocated on the three relevant areas GFP_DMA,
>> GFP_DMA32, GFP_KERNEL:
>>
>> - On an RPi4 with variations on the ram memory size. But also forcing
>>   the situation where all three memory zones are nonempty by setting a 3G
>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>
> i like to test this series on Raspberry Pi 4 and i have some questions
> to get arm64 running:
> 
> Do you use U-Boot? Which tree?

If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
to boot your kernel.

Regards,
Matthias

> Are there any config.txt tweaks necessary?
> 
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32
  2019-09-09  9:58 ` [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 Nicolas Saenz Julienne
@ 2019-09-11 10:54   ` Nicolas Saenz Julienne
  2019-09-11 14:35     ` Catalin Marinas
  0 siblings, 1 reply; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-11 10:54 UTC (permalink / raw)
  To: catalin.marinas, hch, wahrenst, marc.zyngier, robh+dt,
	linux-arm-kernel, linux-mm, linux-riscv, Will Deacon
  Cc: f.fainelli, robin.murphy, linux-kernel, mbrugger,
	linux-rpi-kernel, phill, m.szyprowski

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On Mon, 2019-09-09 at 11:58 +0200, Nicolas Saenz Julienne wrote:
> +
>  /*
> - * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)). It
> - * currently assumes that for memory starting above 4G, 32-bit devices will
> - * use a DMA offset.
> + * Return the maximum physical address for a zone with a given address size
> + * limit. It currently assumes that for memory starting above 4G, 32-bit
> + * devices will use a DMA offset.
>   */
> -static phys_addr_t __init max_zone_dma32_phys(void)
> +static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
>  {
>         phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
> -       return min(offset + (1ULL << 32), memblock_end_of_DRAM());
> +       return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
>  }

Hi all,
while testing other code on top of this series on odd arm64 machines I found an
issue: when memblock_start_of_DRAM() != 0, max_zone_phys() isn't taking into
account the offset to the beginning of memory. This doesn't matter with
zone_bits == 32 but it does when zone_bits == 30.

I'll send a follow-up series.

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32
  2019-09-11 10:54   ` Nicolas Saenz Julienne
@ 2019-09-11 14:35     ` Catalin Marinas
  2019-09-11 15:00       ` Nicolas Saenz Julienne
  0 siblings, 1 reply; 20+ messages in thread
From: Catalin Marinas @ 2019-09-11 14:35 UTC (permalink / raw)
  To: Nicolas Saenz Julienne
  Cc: hch, wahrenst, marc.zyngier, robh+dt, linux-arm-kernel, linux-mm,
	linux-riscv, Will Deacon, f.fainelli, robin.murphy, linux-kernel,
	mbrugger, linux-rpi-kernel, phill, m.szyprowski

On Wed, Sep 11, 2019 at 12:54:38PM +0200, Nicolas Saenz Julienne wrote:
> On Mon, 2019-09-09 at 11:58 +0200, Nicolas Saenz Julienne wrote:
> >  /*
> > - * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)). It
> > - * currently assumes that for memory starting above 4G, 32-bit devices will
> > - * use a DMA offset.
> > + * Return the maximum physical address for a zone with a given address size
> > + * limit. It currently assumes that for memory starting above 4G, 32-bit
> > + * devices will use a DMA offset.
> >   */
> > -static phys_addr_t __init max_zone_dma32_phys(void)
> > +static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
> >  {
> >         phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, 32);
> > -       return min(offset + (1ULL << 32), memblock_end_of_DRAM());
> > +       return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
> >  }
> 
> while testing other code on top of this series on odd arm64 machines I found an
> issue: when memblock_start_of_DRAM() != 0, max_zone_phys() isn't taking into
> account the offset to the beginning of memory. This doesn't matter with
> zone_bits == 32 but it does when zone_bits == 30.

I thought about this but I confused myself and the only case I had in
mind was an AMD Seattle system with RAM starting at 4GB.

What we need from this function is that the lowest naturally aligned
2^30 RAM is covered by ZONE_DMA while the rest to 2^32 are ZONE_DMA32.
This assumed that devices only capable of 30-bit (or 32-bit), have the
top address bits hardwired to be able access the bottom of the memory
(and this would be expressed in DT as the DMA offset).

I guess the fix here is to use GENMASK_ULL(63, zone_bits).

-- 
Catalin


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32
  2019-09-11 14:35     ` Catalin Marinas
@ 2019-09-11 15:00       ` Nicolas Saenz Julienne
  0 siblings, 0 replies; 20+ messages in thread
From: Nicolas Saenz Julienne @ 2019-09-11 15:00 UTC (permalink / raw)
  To: Catalin Marinas
  Cc: hch, wahrenst, marc.zyngier, robh+dt, linux-arm-kernel, linux-mm,
	linux-riscv, Will Deacon, f.fainelli, robin.murphy, linux-kernel,
	mbrugger, linux-rpi-kernel, phill, m.szyprowski

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]

On Wed, 2019-09-11 at 15:35 +0100, Catalin Marinas wrote:
> On Wed, Sep 11, 2019 at 12:54:38PM +0200, Nicolas Saenz Julienne wrote:
> > On Mon, 2019-09-09 at 11:58 +0200, Nicolas Saenz Julienne wrote:
> > >  /*
> > > - * Return the maximum physical address for ZONE_DMA32 (DMA_BIT_MASK(32)).
> > > It
> > > - * currently assumes that for memory starting above 4G, 32-bit devices
> > > will
> > > - * use a DMA offset.
> > > + * Return the maximum physical address for a zone with a given address
> > > size
> > > + * limit. It currently assumes that for memory starting above 4G, 32-bit
> > > + * devices will use a DMA offset.
> > >   */
> > > -static phys_addr_t __init max_zone_dma32_phys(void)
> > > +static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
> > >  {
> > >         phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63,
> > > 32);
> > > -       return min(offset + (1ULL << 32), memblock_end_of_DRAM());
> > > +       return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
> > >  }
> > 
> > while testing other code on top of this series on odd arm64 machines I found
> > an
> > issue: when memblock_start_of_DRAM() != 0, max_zone_phys() isn't taking into
> > account the offset to the beginning of memory. This doesn't matter with
> > zone_bits == 32 but it does when zone_bits == 30.
> 
> I thought about this but I confused myself and the only case I had in
> mind was an AMD Seattle system with RAM starting at 4GB.

I found the issue on a Cavium ThunderX2 server. Oddly enough the memory starts
at 0x802f0000.

> What we need from this function is that the lowest naturally aligned
> 2^30 RAM is covered by ZONE_DMA while the rest to 2^32 are ZONE_DMA32.
> This assumed that devices only capable of 30-bit (or 32-bit), have the
> top address bits hardwired to be able access the bottom of the memory
> (and this would be expressed in DT as the DMA offset).

Ok, I was testing a fix I wrote under these assumptions...

> I guess the fix here is to use GENMASK_ULL(63, zone_bits).

...but this is way cleaner than my solution. Thanks!

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-10  9:27   ` Matthias Brugger
@ 2019-09-12 17:18     ` Matthias Brugger
  2019-09-12 17:27       ` Florian Fainelli
  2019-09-12 19:32       ` Stefan Wahren
  0 siblings, 2 replies; 20+ messages in thread
From: Matthias Brugger @ 2019-09-12 17:18 UTC (permalink / raw)
  To: Matthias Brugger, Stefan Wahren, catalin.marinas, marc.zyngier,
	robh+dt, linux-mm, linux-arm-kernel, linux-riscv, hch,
	Nicolas Saenz Julienne
  Cc: f.fainelli, robin.murphy, linux-kernel, linux-rpi-kernel, phill,
	will, m.szyprowski



On 10/09/2019 11:27, Matthias Brugger wrote:
> 
> 
> On 09/09/2019 21:33, Stefan Wahren wrote:
>> Hi Nicolas,
>>
>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>> Hi all,
>>> this series attempts to address some issues we found while bringing up
>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>> up of these discussions:
>>> v4: https://lkml.org/lkml/2019/9/6/352
>>> v3: https://lkml.org/lkml/2019/9/2/589
>>> v2: https://lkml.org/lkml/2019/8/20/767
>>> v1: https://lkml.org/lkml/2019/7/31/922
>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>
>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>> only address the first GB: their DMA address range is
>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>> view of the address space by virtue of being hooked up trough a second
>>> interconnect.
>>>
>>> Part of this is solved on arm32 by setting up the machine specific
>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>> memory area at the right spot. That said no buffer bouncing (needed for
>>> dma streaming) is available at the moment, but that's a story for
>>> another series.
>>>
>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>> arch code as if all peripherals where be able to address the first 4GB
>>> of memory.
>>>
>>> In the light of this, the series implements the following changes:
>>>
>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>   so this hardcoded solution was deemed preferable.
>>>
>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>
>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>
>>> This series has been tested on multiple devices both by checking the
>>> zones setup matches the expectations and by double-checking physical
>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>> GFP_DMA32, GFP_KERNEL:
>>>
>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>
>> i like to test this series on Raspberry Pi 4 and i have some questions
>> to get arm64 running:
>>
>> Do you use U-Boot? Which tree?
> 
> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
> to boot your kernel.
> 

Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
RPi4 devicetree provided by the FW uses mostly bcm2838. U-Boot in its default
config uses the devicetree provided by the FW, mostly because this way you don't
have to do anything to find out how many RAM you really have. Secondly because
this will allow us, in the near future, to have one U-boot binary for both RPi3
and RPi4 (and as a side effect one binary for RPi1 and RPi2).

Anyway, I found at least, that the following compatibles need to be added:

"brcm,bcm2838-cprman"
"brcm,bcm2838-gpio"

Without at least the cprman driver update, you won't see anything.

"brcm,bcm2838-rng200" is also a candidate.

I also suppose we will need to add "brcm,bcm2838" to
arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.

Regards,
Matthias

> Regards,
> Matthias
> 
>> Are there any config.txt tweaks necessary?
>>
>>
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-12 17:18     ` Matthias Brugger
@ 2019-09-12 17:27       ` Florian Fainelli
  2019-09-12 19:32       ` Stefan Wahren
  1 sibling, 0 replies; 20+ messages in thread
From: Florian Fainelli @ 2019-09-12 17:27 UTC (permalink / raw)
  To: Matthias Brugger, Matthias Brugger, Stefan Wahren,
	catalin.marinas, marc.zyngier, robh+dt, linux-mm,
	linux-arm-kernel, linux-riscv, hch, Nicolas Saenz Julienne
  Cc: robin.murphy, linux-kernel, linux-rpi-kernel, phill, will, m.szyprowski

On 9/12/19 10:18 AM, Matthias Brugger wrote:
> 
> 
> On 10/09/2019 11:27, Matthias Brugger wrote:
>>
>>
>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>> Hi Nicolas,
>>>
>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>> Hi all,
>>>> this series attempts to address some issues we found while bringing up
>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>> up of these discussions:
>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>
>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>> only address the first GB: their DMA address range is
>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>> view of the address space by virtue of being hooked up trough a second
>>>> interconnect.
>>>>
>>>> Part of this is solved on arm32 by setting up the machine specific
>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>> dma streaming) is available at the moment, but that's a story for
>>>> another series.
>>>>
>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>> arch code as if all peripherals where be able to address the first 4GB
>>>> of memory.
>>>>
>>>> In the light of this, the series implements the following changes:
>>>>
>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>   so this hardcoded solution was deemed preferable.
>>>>
>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>
>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>
>>>> This series has been tested on multiple devices both by checking the
>>>> zones setup matches the expectations and by double-checking physical
>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>> GFP_DMA32, GFP_KERNEL:
>>>>
>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>
>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>> to get arm64 running:
>>>
>>> Do you use U-Boot? Which tree?
>>
>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>> to boot your kernel.
>>
> 
> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
> RPi4 devicetree provided by the FW uses mostly bcm2838. U-Boot in its default
> config uses the devicetree provided by the FW, mostly because this way you don't
> have to do anything to find out how many RAM you really have. Secondly because
> this will allow us, in the near future, to have one U-boot binary for both RPi3
> and RPi4 (and as a side effect one binary for RPi1 and RPi2).

Fairly sure we had the conversation a few weeks ago about whether to
chose bcm2711 or bcm2838 for the compatible string. In all cases, the
actual HW this designates is the same, but there was a consistency
argument that 2838, is numerically + 1 than its predecessor and might be
how the RPi would be announced, even if the chip silkscreen says 2711.

If we start adding 2711, does that mean we should also add 2708/09/10 to
existing 2835/36/37 compatible strings or has that ship sailed?
-- 
Florian


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-12 17:18     ` Matthias Brugger
  2019-09-12 17:27       ` Florian Fainelli
@ 2019-09-12 19:32       ` Stefan Wahren
  2019-09-13  7:15         ` Matthias Brugger
  2019-09-13  8:09         ` Matthias Brugger
  1 sibling, 2 replies; 20+ messages in thread
From: Stefan Wahren @ 2019-09-12 19:32 UTC (permalink / raw)
  To: Matthias Brugger, Matthias Brugger, Stefan Wahren,
	catalin.marinas, marc.zyngier, robh+dt, linux-mm,
	linux-arm-kernel, linux-riscv, hch, Nicolas Saenz Julienne
  Cc: f.fainelli, robin.murphy, linux-kernel, linux-rpi-kernel, phill,
	will, m.szyprowski


Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>
> On 10/09/2019 11:27, Matthias Brugger wrote:
>>
>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>> Hi Nicolas,
>>>
>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>> Hi all,
>>>> this series attempts to address some issues we found while bringing up
>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>> up of these discussions:
>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>
>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>> only address the first GB: their DMA address range is
>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>> view of the address space by virtue of being hooked up trough a second
>>>> interconnect.
>>>>
>>>> Part of this is solved on arm32 by setting up the machine specific
>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>> dma streaming) is available at the moment, but that's a story for
>>>> another series.
>>>>
>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>> arch code as if all peripherals where be able to address the first 4GB
>>>> of memory.
>>>>
>>>> In the light of this, the series implements the following changes:
>>>>
>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>   so this hardcoded solution was deemed preferable.
>>>>
>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>
>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>
>>>> This series has been tested on multiple devices both by checking the
>>>> zones setup matches the expectations and by double-checking physical
>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>> GFP_DMA32, GFP_KERNEL:
>>>>
>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>
>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>> to get arm64 running:
>>>
>>> Do you use U-Boot? Which tree?
>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>> to boot your kernel.
>>
> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
> RPi4 devicetree provided by the FW uses mostly bcm2838.

Do you mean the DTB provided at runtime?

You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
series?

>  U-Boot in its default
> config uses the devicetree provided by the FW, mostly because this way you don't
> have to do anything to find out how many RAM you really have. Secondly because
> this will allow us, in the near future, to have one U-boot binary for both RPi3
> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>
> Anyway, I found at least, that the following compatibles need to be added:
>
> "brcm,bcm2838-cprman"
> "brcm,bcm2838-gpio"
>
> Without at least the cprman driver update, you won't see anything.
>
> "brcm,bcm2838-rng200" is also a candidate.
>
> I also suppose we will need to add "brcm,bcm2838" to
> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
How about changing this in the downstream kernel? Which is much easier.
>
> Regards,
> Matthias
>
>> Regards,
>> Matthias
>>
>>> Are there any config.txt tweaks necessary?
>>>
>>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-12 19:32       ` Stefan Wahren
@ 2019-09-13  7:15         ` Matthias Brugger
  2019-09-13  8:09         ` Matthias Brugger
  1 sibling, 0 replies; 20+ messages in thread
From: Matthias Brugger @ 2019-09-13  7:15 UTC (permalink / raw)
  To: Stefan Wahren, Matthias Brugger, catalin.marinas, marc.zyngier,
	robh+dt, linux-mm, linux-arm-kernel, linux-riscv, hch,
	Nicolas Saenz Julienne
  Cc: f.fainelli, robin.murphy, linux-kernel, linux-rpi-kernel, phill,
	will, m.szyprowski



On 12/09/2019 21:32, Stefan Wahren wrote:
> 
> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>
>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>
>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>> Hi Nicolas,
>>>>
>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>> Hi all,
>>>>> this series attempts to address some issues we found while bringing up
>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>> up of these discussions:
>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>
>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>> only address the first GB: their DMA address range is
>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>> view of the address space by virtue of being hooked up trough a second
>>>>> interconnect.
>>>>>
>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>> dma streaming) is available at the moment, but that's a story for
>>>>> another series.
>>>>>
>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>> of memory.
>>>>>
>>>>> In the light of this, the series implements the following changes:
>>>>>
>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>   so this hardcoded solution was deemed preferable.
>>>>>
>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>
>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>
>>>>> This series has been tested on multiple devices both by checking the
>>>>> zones setup matches the expectations and by double-checking physical
>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>
>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>
>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>> to get arm64 running:
>>>>
>>>> Do you use U-Boot? Which tree?
>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>> to boot your kernel.
>>>
>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>> RPi4 devicetree provided by the FW uses mostly bcm2838.
> 
> Do you mean the DTB provided at runtime?
> 

Yes.

> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
> series?
> 

Unfortunately that is exactly the state right now.

>>  U-Boot in its default
>> config uses the devicetree provided by the FW, mostly because this way you don't
>> have to do anything to find out how many RAM you really have. Secondly because
>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>
>> Anyway, I found at least, that the following compatibles need to be added:
>>
>> "brcm,bcm2838-cprman"
>> "brcm,bcm2838-gpio"
>>
>> Without at least the cprman driver update, you won't see anything.
>>
>> "brcm,bcm2838-rng200" is also a candidate.
>>
>> I also suppose we will need to add "brcm,bcm2838" to
>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
> How about changing this in the downstream kernel? Which is much easier.
>>
>> Regards,
>> Matthias
>>
>>> Regards,
>>> Matthias
>>>
>>>> Are there any config.txt tweaks necessary?
>>>>
>>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-12 19:32       ` Stefan Wahren
  2019-09-13  7:15         ` Matthias Brugger
@ 2019-09-13  8:09         ` Matthias Brugger
  2019-09-13  8:50           ` Stefan Wahren
  1 sibling, 1 reply; 20+ messages in thread
From: Matthias Brugger @ 2019-09-13  8:09 UTC (permalink / raw)
  To: Stefan Wahren, catalin.marinas, marc.zyngier, Matthias Brugger,
	robh+dt, linux-mm, linux-arm-kernel, linux-riscv, hch,
	Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phill,
	m.szyprowski, linux-kernel



On 12/09/2019 21:32, Stefan Wahren wrote:
> 
> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>
>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>
>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>> Hi Nicolas,
>>>>
>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>> Hi all,
>>>>> this series attempts to address some issues we found while bringing up
>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>> up of these discussions:
>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>
>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>> only address the first GB: their DMA address range is
>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>> view of the address space by virtue of being hooked up trough a second
>>>>> interconnect.
>>>>>
>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>> dma streaming) is available at the moment, but that's a story for
>>>>> another series.
>>>>>
>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>> of memory.
>>>>>
>>>>> In the light of this, the series implements the following changes:
>>>>>
>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>   so this hardcoded solution was deemed preferable.
>>>>>
>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>
>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>
>>>>> This series has been tested on multiple devices both by checking the
>>>>> zones setup matches the expectations and by double-checking physical
>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>
>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>
>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>> to get arm64 running:
>>>>
>>>> Do you use U-Boot? Which tree?
>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>> to boot your kernel.
>>>
>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>> RPi4 devicetree provided by the FW uses mostly bcm2838.
> 
> Do you mean the DTB provided at runtime?
> 
> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
> series?
> 
>>  U-Boot in its default
>> config uses the devicetree provided by the FW, mostly because this way you don't
>> have to do anything to find out how many RAM you really have. Secondly because
>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>
>> Anyway, I found at least, that the following compatibles need to be added:
>>
>> "brcm,bcm2838-cprman"
>> "brcm,bcm2838-gpio"
>>
>> Without at least the cprman driver update, you won't see anything.
>>
>> "brcm,bcm2838-rng200" is also a candidate.
>>
>> I also suppose we will need to add "brcm,bcm2838" to
>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
> How about changing this in the downstream kernel? Which is much easier.

I'm not sure I understand what you want to say. My goal is to use the upstream
kernel with the device tree blob provided by the FW. If you talk about the
downstream kernel, I suppose you mean we should change this in the FW DT blob
and in the downstream kernel. That would work for me.

Did I understand you correctly?

>>
>> Regards,
>> Matthias
>>
>>> Regards,
>>> Matthias
>>>
>>>> Are there any config.txt tweaks necessary?
>>>>
>>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-13  8:09         ` Matthias Brugger
@ 2019-09-13  8:50           ` Stefan Wahren
  2019-09-13  9:25             ` Matthias Brugger
  0 siblings, 1 reply; 20+ messages in thread
From: Stefan Wahren @ 2019-09-13  8:50 UTC (permalink / raw)
  To: Matthias Brugger, catalin.marinas, marc.zyngier,
	Matthias Brugger, robh+dt, linux-mm, linux-arm-kernel,
	linux-riscv, hch, Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phill,
	m.szyprowski, linux-kernel

Am 13.09.19 um 10:09 schrieb Matthias Brugger:
>
> On 12/09/2019 21:32, Stefan Wahren wrote:
>> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>>> Hi Nicolas,
>>>>>
>>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>>> Hi all,
>>>>>> this series attempts to address some issues we found while bringing up
>>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>>> up of these discussions:
>>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>>
>>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>>> only address the first GB: their DMA address range is
>>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>>> view of the address space by virtue of being hooked up trough a second
>>>>>> interconnect.
>>>>>>
>>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>>> dma streaming) is available at the moment, but that's a story for
>>>>>> another series.
>>>>>>
>>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>>> of memory.
>>>>>>
>>>>>> In the light of this, the series implements the following changes:
>>>>>>
>>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>>   so this hardcoded solution was deemed preferable.
>>>>>>
>>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>>
>>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>>
>>>>>> This series has been tested on multiple devices both by checking the
>>>>>> zones setup matches the expectations and by double-checking physical
>>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>>
>>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>>
>>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>>> to get arm64 running:
>>>>>
>>>>> Do you use U-Boot? Which tree?
>>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>>> to boot your kernel.
>>>>
>>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>>> RPi4 devicetree provided by the FW uses mostly bcm2838.
>> Do you mean the DTB provided at runtime?
>>
>> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
>> series?
>>
>>>  U-Boot in its default
>>> config uses the devicetree provided by the FW, mostly because this way you don't
>>> have to do anything to find out how many RAM you really have. Secondly because
>>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>>
>>> Anyway, I found at least, that the following compatibles need to be added:
>>>
>>> "brcm,bcm2838-cprman"
>>> "brcm,bcm2838-gpio"
>>>
>>> Without at least the cprman driver update, you won't see anything.
>>>
>>> "brcm,bcm2838-rng200" is also a candidate.
>>>
>>> I also suppose we will need to add "brcm,bcm2838" to
>>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
>> How about changing this in the downstream kernel? Which is much easier.
> I'm not sure I understand what you want to say. My goal is to use the upstream
> kernel with the device tree blob provided by the FW.

The device tree blob you are talking is defined in this repository:

https://github.com/raspberrypi/linux

So the word FW is misleading to me.

>  If you talk about the
> downstream kernel, I suppose you mean we should change this in the FW DT blob
> and in the downstream kernel. That would work for me.
>
> Did I understand you correctly?

Yes

So i suggest to add the upstream compatibles into the repo mentioned above.

Sorry, but in case you decided as a U-Boot developer to be compatible
with a unreviewed DT, we also need to make U-Boot compatible with
upstream and downstream DT blobs.

>
>>> Regards,
>>> Matthias
>>>
>>>> Regards,
>>>> Matthias
>>>>
>>>>> Are there any config.txt tweaks necessary?
>>>>>
>>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-13  8:50           ` Stefan Wahren
@ 2019-09-13  9:25             ` Matthias Brugger
  2019-09-13 10:08               ` Stefan Wahren
  0 siblings, 1 reply; 20+ messages in thread
From: Matthias Brugger @ 2019-09-13  9:25 UTC (permalink / raw)
  To: Stefan Wahren, catalin.marinas, marc.zyngier, Matthias Brugger,
	robh+dt, linux-mm, linux-arm-kernel, linux-riscv, hch,
	Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phill,
	m.szyprowski, linux-kernel



On 13/09/2019 10:50, Stefan Wahren wrote:
> Am 13.09.19 um 10:09 schrieb Matthias Brugger:
>>
>> On 12/09/2019 21:32, Stefan Wahren wrote:
>>> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>>>> Hi Nicolas,
>>>>>>
>>>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>>>> Hi all,
>>>>>>> this series attempts to address some issues we found while bringing up
>>>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>>>> up of these discussions:
>>>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>>>
>>>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>>>> only address the first GB: their DMA address range is
>>>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>>>> view of the address space by virtue of being hooked up trough a second
>>>>>>> interconnect.
>>>>>>>
>>>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>>>> dma streaming) is available at the moment, but that's a story for
>>>>>>> another series.
>>>>>>>
>>>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>>>> of memory.
>>>>>>>
>>>>>>> In the light of this, the series implements the following changes:
>>>>>>>
>>>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>>>   so this hardcoded solution was deemed preferable.
>>>>>>>
>>>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>>>
>>>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>>>
>>>>>>> This series has been tested on multiple devices both by checking the
>>>>>>> zones setup matches the expectations and by double-checking physical
>>>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>>>
>>>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>>>
>>>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>>>> to get arm64 running:
>>>>>>
>>>>>> Do you use U-Boot? Which tree?
>>>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>>>> to boot your kernel.
>>>>>
>>>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>>>> RPi4 devicetree provided by the FW uses mostly bcm2838.
>>> Do you mean the DTB provided at runtime?
>>>
>>> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
>>> series?
>>>
>>>>  U-Boot in its default
>>>> config uses the devicetree provided by the FW, mostly because this way you don't
>>>> have to do anything to find out how many RAM you really have. Secondly because
>>>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>>>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>>>
>>>> Anyway, I found at least, that the following compatibles need to be added:
>>>>
>>>> "brcm,bcm2838-cprman"
>>>> "brcm,bcm2838-gpio"
>>>>
>>>> Without at least the cprman driver update, you won't see anything.
>>>>
>>>> "brcm,bcm2838-rng200" is also a candidate.
>>>>
>>>> I also suppose we will need to add "brcm,bcm2838" to
>>>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
>>> How about changing this in the downstream kernel? Which is much easier.
>> I'm not sure I understand what you want to say. My goal is to use the upstream
>> kernel with the device tree blob provided by the FW.
> 
> The device tree blob you are talking is defined in this repository:
> 
> https://github.com/raspberrypi/linux
> 
> So the word FW is misleading to me.
> 

No, it's part of
https://github.com/raspberrypi/firmware.git
file boot/bcm2711-rpi-4-b.dtb

>>  If you talk about the
>> downstream kernel, I suppose you mean we should change this in the FW DT blob
>> and in the downstream kernel. That would work for me.
>>
>> Did I understand you correctly?
> 
> Yes
> 
> So i suggest to add the upstream compatibles into the repo mentioned above.
> 
> Sorry, but in case you decided as a U-Boot developer to be compatible
> with a unreviewed DT, we also need to make U-Boot compatible with
> upstream and downstream DT blobs.
> 

Well RPi3 is working with the DT blob provided by the FW, as I mentioned earlier
if we can use this DTB we can work towards one binary that can boot both RPi3
and RPi4. On the other hand we can rely on the FW to detect the amount of memory
our RPi4 has.

That said, I agree that we should make sure that U-Boot can boot with both DTBs,
the upstream one and the downstream. Now the question is how to get to this. I'm
a bit puzzled that by talking about "unreviewed DT" you insinuate that bcm2711
compatible is already reviewed and can't be changed. From what I can see none of
these compatibles got merged for now, so we are still at time to change them.

Apart from the point Florian made, to stay consistent with the RPi SoC naming,
it will save us work, both in the kernel and in U-Boot, as we would need to add
both compatibles to the code-base.

Regards,
Matthias

>>
>>>> Regards,
>>>> Matthias
>>>>
>>>>> Regards,
>>>>> Matthias
>>>>>
>>>>>> Are there any config.txt tweaks necessary?
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> linux-arm-kernel@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-13  9:25             ` Matthias Brugger
@ 2019-09-13 10:08               ` Stefan Wahren
  2019-09-13 10:39                 ` Matthias Brugger
  0 siblings, 1 reply; 20+ messages in thread
From: Stefan Wahren @ 2019-09-13 10:08 UTC (permalink / raw)
  To: Matthias Brugger, catalin.marinas, marc.zyngier,
	Matthias Brugger, robh+dt, linux-mm, linux-arm-kernel,
	linux-riscv, hch, Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phil,
	m.szyprowski, linux-kernel

Am 13.09.19 um 11:25 schrieb Matthias Brugger:
>
> On 13/09/2019 10:50, Stefan Wahren wrote:
>> Am 13.09.19 um 10:09 schrieb Matthias Brugger:
>>> On 12/09/2019 21:32, Stefan Wahren wrote:
>>>> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>>>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>>>>> Hi Nicolas,
>>>>>>>
>>>>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>>>>> Hi all,
>>>>>>>> this series attempts to address some issues we found while bringing up
>>>>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>>>>> up of these discussions:
>>>>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>>>>
>>>>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>>>>> only address the first GB: their DMA address range is
>>>>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>>>>> view of the address space by virtue of being hooked up trough a second
>>>>>>>> interconnect.
>>>>>>>>
>>>>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>>>>> dma streaming) is available at the moment, but that's a story for
>>>>>>>> another series.
>>>>>>>>
>>>>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>>>>> of memory.
>>>>>>>>
>>>>>>>> In the light of this, the series implements the following changes:
>>>>>>>>
>>>>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>>>>   so this hardcoded solution was deemed preferable.
>>>>>>>>
>>>>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>>>>
>>>>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>>>>
>>>>>>>> This series has been tested on multiple devices both by checking the
>>>>>>>> zones setup matches the expectations and by double-checking physical
>>>>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>>>>
>>>>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>>>>
>>>>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>>>>> to get arm64 running:
>>>>>>>
>>>>>>> Do you use U-Boot? Which tree?
>>>>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>>>>> to boot your kernel.
>>>>>>
>>>>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>>>>> RPi4 devicetree provided by the FW uses mostly bcm2838.
>>>> Do you mean the DTB provided at runtime?
>>>>
>>>> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
>>>> series?
>>>>
>>>>>  U-Boot in its default
>>>>> config uses the devicetree provided by the FW, mostly because this way you don't
>>>>> have to do anything to find out how many RAM you really have. Secondly because
>>>>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>>>>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>>>>
>>>>> Anyway, I found at least, that the following compatibles need to be added:
>>>>>
>>>>> "brcm,bcm2838-cprman"
>>>>> "brcm,bcm2838-gpio"
>>>>>
>>>>> Without at least the cprman driver update, you won't see anything.
>>>>>
>>>>> "brcm,bcm2838-rng200" is also a candidate.
>>>>>
>>>>> I also suppose we will need to add "brcm,bcm2838" to
>>>>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
>>>> How about changing this in the downstream kernel? Which is much easier.
>>> I'm not sure I understand what you want to say. My goal is to use the upstream
>>> kernel with the device tree blob provided by the FW.
>> The device tree blob you are talking is defined in this repository:
>>
>> https://github.com/raspberrypi/linux
>>
>> So the word FW is misleading to me.
>>
> No, it's part of
> https://github.com/raspberrypi/firmware.git
> file boot/bcm2711-rpi-4-b.dtb
The compiled DT blobs incl. the kernel image are stored in this artifact
repository. But the sources for the kernel and the DT are in the Linux
repo. This is necessary to be compliant to the GPL.
>
>>>  If you talk about the
>>> downstream kernel, I suppose you mean we should change this in the FW DT blob
>>> and in the downstream kernel. That would work for me.
>>>
>>> Did I understand you correctly?
>> Yes
>>
>> So i suggest to add the upstream compatibles into the repo mentioned above.
>>
>> Sorry, but in case you decided as a U-Boot developer to be compatible
>> with a unreviewed DT, we also need to make U-Boot compatible with
>> upstream and downstream DT blobs.
>>
> Well RPi3 is working with the DT blob provided by the FW, as I mentioned earlier
> if we can use this DTB we can work towards one binary that can boot both RPi3
> and RPi4. On the other hand we can rely on the FW to detect the amount of memory
> our RPi4 has.
>
> That said, I agree that we should make sure that U-Boot can boot with both DTBs,
> the upstream one and the downstream. Now the question is how to get to this. I'm
> a bit puzzled that by talking about "unreviewed DT" you insinuate that bcm2711
> compatible is already reviewed and can't be changed. From what I can see none of
> these compatibles got merged for now, so we are still at time to change them.

Stephen Boyd was okay with clk changes except of a small nit. So i fixed
this is as he suggested in a separate series. Unfortunately this hasn't
be applied yet [1].

The i2c, pinctrl and the sdhci changes has been applied yet.

In my opinion it isn't the job of the mainline kernel to adapt to a
vendor device tree. It's the vendor device tree which needs to be fixed.

Sorry, but this is my holiday. I will back after the weekend.

Best regards
Stefan

[1] - https://www.spinics.net/lists/linux-clk/msg40534.html

>
> Apart from the point Florian made, to stay consistent with the RPi SoC naming,
> it will save us work, both in the kernel and in U-Boot, as we would need to add
> both compatibles to the code-base.
>
> Regards,
> Matthias
>
>>>>> Regards,
>>>>> Matthias
>>>>>
>>>>>> Regards,
>>>>>> Matthias
>>>>>>
>>>>>>> Are there any config.txt tweaks necessary?
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-arm-kernel mailing list
>>>>>> linux-arm-kernel@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> linux-arm-kernel@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support
  2019-09-13 10:08               ` Stefan Wahren
@ 2019-09-13 10:39                 ` Matthias Brugger
  0 siblings, 0 replies; 20+ messages in thread
From: Matthias Brugger @ 2019-09-13 10:39 UTC (permalink / raw)
  To: Stefan Wahren, Matthias Brugger, catalin.marinas, marc.zyngier,
	robh+dt, linux-mm, linux-arm-kernel, linux-riscv, hch,
	Nicolas Saenz Julienne
  Cc: robin.murphy, f.fainelli, will, linux-rpi-kernel, phil,
	m.szyprowski, linux-kernel



On 13/09/2019 12:08, Stefan Wahren wrote:
> Am 13.09.19 um 11:25 schrieb Matthias Brugger:
>>
>> On 13/09/2019 10:50, Stefan Wahren wrote:
>>> Am 13.09.19 um 10:09 schrieb Matthias Brugger:
>>>> On 12/09/2019 21:32, Stefan Wahren wrote:
>>>>> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>>>>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>>>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>>>>>> Hi Nicolas,
>>>>>>>>
>>>>>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>>>>>> Hi all,
>>>>>>>>> this series attempts to address some issues we found while bringing up
>>>>>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>>>>>> up of these discussions:
>>>>>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>>>>>
>>>>>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>>>>>> only address the first GB: their DMA address range is
>>>>>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>>>>>> view of the address space by virtue of being hooked up trough a second
>>>>>>>>> interconnect.
>>>>>>>>>
>>>>>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>>>>>> dma streaming) is available at the moment, but that's a story for
>>>>>>>>> another series.
>>>>>>>>>
>>>>>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>>>>>> of memory.
>>>>>>>>>
>>>>>>>>> In the light of this, the series implements the following changes:
>>>>>>>>>
>>>>>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>>>>>   so this hardcoded solution was deemed preferable.
>>>>>>>>>
>>>>>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>>>>>
>>>>>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>>>>>
>>>>>>>>> This series has been tested on multiple devices both by checking the
>>>>>>>>> zones setup matches the expectations and by double-checking physical
>>>>>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>>>>>
>>>>>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>>>>>
>>>>>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>>>>>> to get arm64 running:
>>>>>>>>
>>>>>>>> Do you use U-Boot? Which tree?
>>>>>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>>>>>> to boot your kernel.
>>>>>>>
>>>>>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>>>>>> RPi4 devicetree provided by the FW uses mostly bcm2838.
>>>>> Do you mean the DTB provided at runtime?
>>>>>
>>>>> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
>>>>> series?
>>>>>
>>>>>>  U-Boot in its default
>>>>>> config uses the devicetree provided by the FW, mostly because this way you don't
>>>>>> have to do anything to find out how many RAM you really have. Secondly because
>>>>>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>>>>>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>>>>>
>>>>>> Anyway, I found at least, that the following compatibles need to be added:
>>>>>>
>>>>>> "brcm,bcm2838-cprman"
>>>>>> "brcm,bcm2838-gpio"
>>>>>>
>>>>>> Without at least the cprman driver update, you won't see anything.
>>>>>>
>>>>>> "brcm,bcm2838-rng200" is also a candidate.
>>>>>>
>>>>>> I also suppose we will need to add "brcm,bcm2838" to
>>>>>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
>>>>> How about changing this in the downstream kernel? Which is much easier.
>>>> I'm not sure I understand what you want to say. My goal is to use the upstream
>>>> kernel with the device tree blob provided by the FW.
>>> The device tree blob you are talking is defined in this repository:
>>>
>>> https://github.com/raspberrypi/linux
>>>
>>> So the word FW is misleading to me.
>>>
>> No, it's part of
>> https://github.com/raspberrypi/firmware.git
>> file boot/bcm2711-rpi-4-b.dtb
> The compiled DT blobs incl. the kernel image are stored in this artifact
> repository. But the sources for the kernel and the DT are in the Linux
> repo. This is necessary to be compliant to the GPL.

Got it, thanks for clarifying.

>>
>>>>  If you talk about the
>>>> downstream kernel, I suppose you mean we should change this in the FW DT blob
>>>> and in the downstream kernel. That would work for me.
>>>>
>>>> Did I understand you correctly?
>>> Yes
>>>
>>> So i suggest to add the upstream compatibles into the repo mentioned above.
>>>
>>> Sorry, but in case you decided as a U-Boot developer to be compatible
>>> with a unreviewed DT, we also need to make U-Boot compatible with
>>> upstream and downstream DT blobs.
>>>
>> Well RPi3 is working with the DT blob provided by the FW, as I mentioned earlier
>> if we can use this DTB we can work towards one binary that can boot both RPi3
>> and RPi4. On the other hand we can rely on the FW to detect the amount of memory
>> our RPi4 has.
>>
>> That said, I agree that we should make sure that U-Boot can boot with both DTBs,
>> the upstream one and the downstream. Now the question is how to get to this. I'm
>> a bit puzzled that by talking about "unreviewed DT" you insinuate that bcm2711
>> compatible is already reviewed and can't be changed. From what I can see none of
>> these compatibles got merged for now, so we are still at time to change them.
> 
> Stephen Boyd was okay with clk changes except of a small nit. So i fixed
> this is as he suggested in a separate series. Unfortunately this hasn't
> be applied yet [1].
> 
> The i2c, pinctrl and the sdhci changes has been applied yet.
> 
> In my opinion it isn't the job of the mainline kernel to adapt to a
> vendor device tree. It's the vendor device tree which needs to be fixed.
> 

I agree with that. But if we can make this easier by choosing a compatible which
fits downstream without violating upstream and it makes sense with the naming
scheme of the RPi, I think that's a good argument.

> Sorry, but this is my holiday. I will back after the weekend.
> 

Sure, enjoy. I'll be on travel for the next two weeks but will try to keep up
with emails.

Regards,
Matthias

> Best regards
> Stefan
> 
> [1] - https://www.spinics.net/lists/linux-clk/msg40534.html
> 
>>
>> Apart from the point Florian made, to stay consistent with the RPi SoC naming,
>> it will save us work, both in the kernel and in U-Boot, as we would need to add
>> both compatibles to the code-base.
>>
>> Regards,
>> Matthias
>>
>>>>>> Regards,
>>>>>> Matthias
>>>>>>
>>>>>>> Regards,
>>>>>>> Matthias
>>>>>>>
>>>>>>>> Are there any config.txt tweaks necessary?
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> linux-arm-kernel mailing list
>>>>>>> linux-arm-kernel@lists.infradead.org
>>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-arm-kernel mailing list
>>>>>> linux-arm-kernel@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel


^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2019-09-13 10:39 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-09  9:58 [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Nicolas Saenz Julienne
2019-09-09  9:58 ` [PATCH v5 1/4] arm64: mm: use arm64_dma_phys_limit instead of calling max_zone_dma_phys() Nicolas Saenz Julienne
2019-09-09  9:58 ` [PATCH v5 2/4] arm64: rename variables used to calculate ZONE_DMA32's size Nicolas Saenz Julienne
2019-09-09  9:58 ` [PATCH v5 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 Nicolas Saenz Julienne
2019-09-11 10:54   ` Nicolas Saenz Julienne
2019-09-11 14:35     ` Catalin Marinas
2019-09-11 15:00       ` Nicolas Saenz Julienne
2019-09-09  9:58 ` [PATCH v5 4/4] mm: refresh ZONE_DMA and ZONE_DMA32 comments in 'enum zone_type' Nicolas Saenz Julienne
2019-09-09 19:33 ` [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support Stefan Wahren
2019-09-09 19:50   ` Nicolas Saenz Julienne
2019-09-10  9:27   ` Matthias Brugger
2019-09-12 17:18     ` Matthias Brugger
2019-09-12 17:27       ` Florian Fainelli
2019-09-12 19:32       ` Stefan Wahren
2019-09-13  7:15         ` Matthias Brugger
2019-09-13  8:09         ` Matthias Brugger
2019-09-13  8:50           ` Stefan Wahren
2019-09-13  9:25             ` Matthias Brugger
2019-09-13 10:08               ` Stefan Wahren
2019-09-13 10:39                 ` Matthias Brugger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).