[PATCH 0/2] arm64: Fix CMA/crashkernel reservation

[PATCH 0/2] arm64: Fix CMA/crashkernel reservation

[Nicolas Saenz Julienne]

As pointed out by Qian Cai[1] the series enabling ZONE_DMA in arm64
breaks CMA/crashkernel reservations on large devices, as it changed its
default placement. After discussing it with Catalin Marinas we're
restoring the old behavior.

The Raspberry Pi 4, being the only device that needs CMA and crashkernel
in ZONE_DMA will explicitly do so trough it's device tree.

[1] https://lkml.org/lkml/2019/10/21/725

---

Nicolas Saenz Julienne (2):
  ARM: dts: bcm2711: force CMA into first GB of memory
  arm64: mm: reserve CMA and crashkernel in ZONE_DMA32

 arch/arm/boot/dts/bcm2711-rpi-4-b.dts | 19 +++++++++++++++++++
 arch/arm64/mm/init.c                  |  4 ++--
 2 files changed, 21 insertions(+), 2 deletions(-)

-- 
2.23.0

[PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Nicolas Saenz Julienne]

arm64 places the CMA in ZONE_DMA32, which is not good enough for the
Raspberry Pi 4 since it contains peripherals that can only address the
first GB of memory. Explicitly place the CMA into that area.

Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 arch/arm/boot/dts/bcm2711-rpi-4-b.dts | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
index cccc1ccd19be..3c7833e9005a 100644
--- a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
+++ b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
@@ -19,6 +19,25 @@
 		reg = <0 0 0>;
 	};
 
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <1>;
+		ranges;
+
+		/*
+		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
+		 * that's not good enough for the Raspberry Pi 4 as some
+		 * devices can only address the lower 1G of memory (ZONE_DMA).
+		 */
+		linux,cma {
+			compatible = "shared-dma-pool";
+			size = <0x2000000>; /* 32MB */
+			alloc-ranges = <0x0 0x00000000 0x40000000>;
+			reusable;
+			linux,cma-default;
+		};
+	};
+
 	leds {
 		act {
 			gpios = <&gpio 42 GPIO_ACTIVE_HIGH>;
-- 
2.23.0

[PATCH 2/2] arm64: mm: reserve CMA and crashkernel in ZONE_DMA32

[Nicolas Saenz Julienne]

With the introduction of ZONE_DMA in arm64 we moved the default CMA and
crashkernel reservation into that area. This caused a regression on big
machines that need big CMA and crashkernel reservations. Note that
ZONE_DMA is only 1GB big.

Restore the previous behavior as the wide majority of devices are OK
with reserving these in ZONE_DMA32. The ones that need them in ZONE_DMA
will configure it explicitly.

Reported-by: Qian Cai <cai@lca.pw>
Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
---
 arch/arm64/mm/init.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 580d1052ac34..8385d3c0733f 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -88,7 +88,7 @@ static void __init reserve_crashkernel(void)
 
 	if (crash_base == 0) {
 		/* Current arm64 boot protocol requires 2MB alignment */
-		crash_base = memblock_find_in_range(0, ARCH_LOW_ADDRESS_LIMIT,
+		crash_base = memblock_find_in_range(0, arm64_dma32_phys_limit,
 				crash_size, SZ_2M);
 		if (crash_base == 0) {
 			pr_warn("cannot allocate crashkernel (size:0x%llx)\n",
@@ -454,7 +454,7 @@ void __init arm64_memblock_init(void)
 
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
-	dma_contiguous_reserve(arm64_dma_phys_limit ? : arm64_dma32_phys_limit);
+	dma_contiguous_reserve(arm64_dma32_phys_limit);
 }
 
 void __init bootmem_init(void)
-- 
2.23.0

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Stefan Wahren]

Hi Nicolas,

Am 04.11.19 um 14:54 schrieb Nicolas Saenz Julienne:
> arm64 places the CMA in ZONE_DMA32, which is not good enough for the
> Raspberry Pi 4 since it contains peripherals that can only address the
> first GB of memory. Explicitly place the CMA into that area.
>
> Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>

do you want this in Linux 5.5 via devicetree/fixes? In this case please
add an fixes tag.

Otherwise this will be queued for Linux 5.6.

> ---
>  arch/arm/boot/dts/bcm2711-rpi-4-b.dts | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> index cccc1ccd19be..3c7833e9005a 100644
> --- a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> +++ b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> @@ -19,6 +19,25 @@
>  		reg = <0 0 0>;
>  	};
>
> +	reserved-memory {
> +		#address-cells = <2>;
> +		#size-cells = <1>;
> +		ranges;
> +
> +		/*
> +		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
> +		 * that's not good enough for the Raspberry Pi 4 as some
> +		 * devices can only address the lower 1G of memory (ZONE_DMA).
> +		 */
> +		linux,cma {
> +			compatible = "shared-dma-pool";
> +			size = <0x2000000>; /* 32MB */
> +			alloc-ranges = <0x0 0x00000000 0x40000000>;
> +			reusable;
> +			linux,cma-default;
> +		};
> +	};
> +

i think this is a SoC-specific issue not a board specifc one. Please
move this to bcm2711.dtsi

Thanks
Stefan

>  	leds {
>  		act {
>  			gpios = <&gpio 42 GPIO_ACTIVE_HIGH>;

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Nicolas Saenz Julienne]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]

On Mon, 2019-11-04 at 18:09 +0100, Stefan Wahren wrote:
> Hi Nicolas,
> 
> Am 04.11.19 um 14:54 schrieb Nicolas Saenz Julienne:
> > arm64 places the CMA in ZONE_DMA32, which is not good enough for the
> > Raspberry Pi 4 since it contains peripherals that can only address the
> > first GB of memory. Explicitly place the CMA into that area.
> > 
> > Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> 
> do you want this in Linux 5.5 via devicetree/fixes? In this case please
> add an fixes tag.

This has to go into v5.5 if the second patch is accepted. That said I can't add
a fixes tag as the code being fixed isn't yet in linus' tree.

Any suggestions? Maybe go through Catalin's tree?

> Otherwise this will be queued for Linux 5.6.
> 
> > ---
> >  arch/arm/boot/dts/bcm2711-rpi-4-b.dts | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> > 
> > diff --git a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> > b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> > index cccc1ccd19be..3c7833e9005a 100644
> > --- a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> > +++ b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
> > @@ -19,6 +19,25 @@
> >  		reg = <0 0 0>;
> >  	};
> > 
> > +	reserved-memory {
> > +		#address-cells = <2>;
> > +		#size-cells = <1>;
> > +		ranges;
> > +
> > +		/*
> > +		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
> > +		 * that's not good enough for the Raspberry Pi 4 as some
> > +		 * devices can only address the lower 1G of memory (ZONE_DMA).
> > +		 */
> > +		linux,cma {
> > +			compatible = "shared-dma-pool";
> > +			size = <0x2000000>; /* 32MB */
> > +			alloc-ranges = <0x0 0x00000000 0x40000000>;
> > +			reusable;
> > +			linux,cma-default;
> > +		};
> > +	};
> > +
> 
> i think this is a SoC-specific issue not a board specifc one. Please
> move this to bcm2711.dtsi

Noted, thanks!

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Florian Fainelli]

On 11/4/19 9:09 AM, Stefan Wahren wrote:

[snip]

>> +	reserved-memory {
>> +		#address-cells = <2>;
>> +		#size-cells = <1>;
>> +		ranges;
>> +
>> +		/*
>> +		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
>> +		 * that's not good enough for the Raspberry Pi 4 as some
>> +		 * devices can only address the lower 1G of memory (ZONE_DMA).
>> +		 */
>> +		linux,cma {
>> +			compatible = "shared-dma-pool";
>> +			size = <0x2000000>; /* 32MB */
>> +			alloc-ranges = <0x0 0x00000000 0x40000000>;
>> +			reusable;
>> +			linux,cma-default;
>> +		};
>> +	};
>> +
> 
> i think this is a SoC-specific issue not a board specifc one. Please
> move this to bcm2711.dtsi

This sounds like a possibly fragile solution if someone changes
CONFIG_CMA_SIZE_MBYTES to a value greater than 32MB no?

I know we don't want machine descriptors for ARM64 kernels, but since
there is already a specific 2711 machine compatible string check, maybe
you could use that as well for determining whether arm64_dma_phys_limit
or arm64_dma32_phys_limit should be chosen?
-- 
Florian

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Florian Fainelli]

On 11/4/19 9:51 AM, Florian Fainelli wrote:
> On 11/4/19 9:09 AM, Stefan Wahren wrote:
> 
> [snip]
> 
>>> +	reserved-memory {
>>> +		#address-cells = <2>;
>>> +		#size-cells = <1>;
>>> +		ranges;
>>> +
>>> +		/*
>>> +		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
>>> +		 * that's not good enough for the Raspberry Pi 4 as some
>>> +		 * devices can only address the lower 1G of memory (ZONE_DMA).
>>> +		 */
>>> +		linux,cma {
>>> +			compatible = "shared-dma-pool";
>>> +			size = <0x2000000>; /* 32MB */
>>> +			alloc-ranges = <0x0 0x00000000 0x40000000>;
>>> +			reusable;
>>> +			linux,cma-default;
>>> +		};
>>> +	};
>>> +
>>
>> i think this is a SoC-specific issue not a board specifc one. Please
>> move this to bcm2711.dtsi
> 
> This sounds like a possibly fragile solution if someone changes
> CONFIG_CMA_SIZE_MBYTES to a value greater than 32MB no?
> 
> I know we don't want machine descriptors for ARM64 kernels, but since
> there is already a specific 2711 machine compatible string check, maybe
> you could use that as well for determining whether arm64_dma_phys_limit
> or arm64_dma32_phys_limit should be chosen?

This last sentence was referring to an earlier version of another patch
series, this is not being done right now, although ARCH_BCM2835 does
forcibly select ZONE_DMA. Nevermind then, I do not see a cleaner
solution right now either.
-- 
Florian

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Nicolas Saenz Julienne]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Mon, 2019-11-04 at 09:51 -0800, Florian Fainelli wrote:
> On 11/4/19 9:09 AM, Stefan Wahren wrote:
> 
> [snip]
> 
> > > +	reserved-memory {
> > > +		#address-cells = <2>;
> > > +		#size-cells = <1>;
> > > +		ranges;
> > > +
> > > +		/*
> > > +		 * arm64 reserves the CMA by default somewhere in ZONE_DMA32,
> > > +		 * that's not good enough for the Raspberry Pi 4 as some
> > > +		 * devices can only address the lower 1G of memory (ZONE_DMA).
> > > +		 */
> > > +		linux,cma {
> > > +			compatible = "shared-dma-pool";
> > > +			size = <0x2000000>; /* 32MB */
> > > +			alloc-ranges = <0x0 0x00000000 0x40000000>;
> > > +			reusable;
> > > +			linux,cma-default;
> > > +		};
> > > +	};
> > > +
> > 
> > i think this is a SoC-specific issue not a board specifc one. Please
> > move this to bcm2711.dtsi
> 
> This sounds like a possibly fragile solution if someone changes
> CONFIG_CMA_SIZE_MBYTES to a value greater than 32MB no?

I agree it's not the most flexible solution in the world. It also bypasses the
command line interface. But I can't see any alternatives as of today.

Overall, I suggest that we set CMA's size to whatever is needed for a sensible
desktop use. And let odd users with custom HW modify it trough overlays (which
they will most likely be forced to do anyway).

That said I'm open to suggestions.

Regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Catalin Marinas]

On Mon, Nov 04, 2019 at 06:09:39PM +0100, Stefan Wahren wrote:
> Hi Nicolas,
> 
> Am 04.11.19 um 14:54 schrieb Nicolas Saenz Julienne:
> > arm64 places the CMA in ZONE_DMA32, which is not good enough for the
> > Raspberry Pi 4 since it contains peripherals that can only address the
> > first GB of memory. Explicitly place the CMA into that area.
> >
> > Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> 
> do you want this in Linux 5.5 via devicetree/fixes? In this case please
> add an fixes tag.

That's not really a fix since AFAICT CMA never worked properly on RPi4
with mainline. For 5.5, I queued the arm64 for-next/zone-dma patches
which would allow RPi4 to get a CMA in the correct physical address
range. However, since these patches cause a regression on other
platforms that don't need a small ZOEN_DMA, my suggestion was to leave
the CMA handling for RPi4 as per the current mainline (i.e. broken) and
allow CMA from the full ZONE_DMA32 range (second patch in this series).

IIUC, this dts patch can be merged independently of the ZONE_DMA patches
for arm64 and it may be beneficial for current mainline (even without
the arm64/for-next/zone-dma patches).

[1] git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/zone-dma

> Otherwise this will be queued for Linux 5.6.

I'm happy to queue them together with your ack for 5.5, otherwise I'll
only pick the second patch in this series.

Thanks.

-- 
Catalin

Re: [PATCH 1/2] ARM: dts: bcm2711: force CMA into first GB of memory

[Stefan Wahren]

Am 05.11.19 um 15:51 schrieb Catalin Marinas:
> Otherwise this will be queued for Linux 5.6.
> I'm happy to queue them together with your ack for 5.5, otherwise I'll
> only pick the second patch in this series.

I had a comment for this patch which should be addressed in V2. After
that i'm happy to gave my Ack.

Thanks
Stefan

>
> Thanks.
>