vc4_bo_create: Failed to allocate from CMA

vc4_bo_create: Failed to allocate from CMA

[Stefan Wahren]

Hi,

while testing the mainline kernel (arm64, defconfig) on Raspberry Pi 3 B
Plus with Raspberry Pi OS - 64 bit, sometimes X doesn't start into
desktop properly (unexpected and unusable login screen instead of auto
login or mouse pointer is show shorty and than switch back to black
screen in a loop). In that case dmesg shows the following:

[   74.737106] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
CMA:
[   74.737558] vc4-drm soc:gpu: [drm]                            V3D: 
28976kb BOs (10)
[   74.737602] vc4-drm soc:gpu: [drm]                     V3D
shader:     44kb BOs (11)
[   74.737632] vc4-drm soc:gpu: [drm]                           dumb:  
4564kb BOs (5)
[   74.737664] vc4-drm soc:gpu: [drm]                         binner: 
16384kb BOs (1)
[   74.737697] vc4-drm soc:gpu: [drm]                total purged
BO:      4kb BOs (1)
[   74.739039] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
CMA:
[   74.739466] vc4-drm soc:gpu: [drm]                            V3D: 
28972kb BOs (9)
[   74.739512] vc4-drm soc:gpu: [drm]                     V3D
shader:     44kb BOs (11)
[   74.739541] vc4-drm soc:gpu: [drm]                           dumb:  
4564kb BOs (5)
[   74.739570] vc4-drm soc:gpu: [drm]                         binner: 
16384kb BOs (1)
[   74.739602] vc4-drm soc:gpu: [drm]                total purged
BO:      4kb BOs (1)
[   74.740718] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
CMA:
[   74.741138] vc4-drm soc:gpu: [drm]                            V3D: 
28972kb BOs (9)
[   74.741171] vc4-drm soc:gpu: [drm]                     V3D
shader:     44kb BOs (11)
[   74.741202] vc4-drm soc:gpu: [drm]                           dumb:  
4564kb BOs (5)
[   74.741231] vc4-drm soc:gpu: [drm]                         binner: 
16384kb BOs (1)
[   74.741263] vc4-drm soc:gpu: [drm]                total purged
BO:      4kb BOs (1)
...

I have only seen this issue on arm64 with latest mainline kernel
(5.13.0-rc5-00130-gf21b807c3cf8), but also with older kernel versions.
So it's not a regression. It seems 64 bit needs more CMA.

In case X started properly i was also able to reproduce these errors
above by dis- and reconneting HDMI.

So i increased CMA in bcm283x.dtsi and the problem disappeared:

iff --git a/arch/arm/boot/dts/bcm283x.dtsi b/arch/arm/boot/dts/bcm283x.dtsi
index b83a864..d1304cb 100644
--- a/arch/arm/boot/dts/bcm283x.dtsi
+++ b/arch/arm/boot/dts/bcm283x.dtsi
@@ -37,7 +37,7 @@
 
         cma: linux,cma {
             compatible = "shared-dma-pool";
-            size = <0x4000000>; /* 64MB */
+            size = <0x6000000>; /* 96MB */
             reusable;
             linux,cma-default;
         };

The questions are:

Is this the right way (tm) to fix this problem?

And what is a sensible value (don't have a 4K display to test)?

Best regards
Stefan

Re: vc4_bo_create: Failed to allocate from CMA

[nicolas saenz julienne]

On Sat, 2021-06-12 at 17:17 +0200, Stefan Wahren wrote:
> Hi,
> 
> while testing the mainline kernel (arm64, defconfig) on Raspberry Pi 3 B
> Plus with Raspberry Pi OS - 64 bit, sometimes X doesn't start into
> desktop properly (unexpected and unusable login screen instead of auto
> login or mouse pointer is show shorty and than switch back to black
> screen in a loop). In that case dmesg shows the following:
> 
> [   74.737106] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
> CMA:
> [   74.737558] vc4-drm soc:gpu: [drm]                            V3D: 
> 28976kb BOs (10)
> [   74.737602] vc4-drm soc:gpu: [drm]                     V3D
> shader:     44kb BOs (11)
> [   74.737632] vc4-drm soc:gpu: [drm]                           dumb:  
> 4564kb BOs (5)
> [   74.737664] vc4-drm soc:gpu: [drm]                         binner: 
> 16384kb BOs (1)
> [   74.737697] vc4-drm soc:gpu: [drm]                total purged
> BO:      4kb BOs (1)
> [   74.739039] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
> CMA:
> [   74.739466] vc4-drm soc:gpu: [drm]                            V3D: 
> 28972kb BOs (9)
> [   74.739512] vc4-drm soc:gpu: [drm]                     V3D
> shader:     44kb BOs (11)
> [   74.739541] vc4-drm soc:gpu: [drm]                           dumb:  
> 4564kb BOs (5)
> [   74.739570] vc4-drm soc:gpu: [drm]                         binner: 
> 16384kb BOs (1)
> [   74.739602] vc4-drm soc:gpu: [drm]                total purged
> BO:      4kb BOs (1)
> [   74.740718] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
> CMA:
> [   74.741138] vc4-drm soc:gpu: [drm]                            V3D: 
> 28972kb BOs (9)
> [   74.741171] vc4-drm soc:gpu: [drm]                     V3D
> shader:     44kb BOs (11)
> [   74.741202] vc4-drm soc:gpu: [drm]                           dumb:  
> 4564kb BOs (5)
> [   74.741231] vc4-drm soc:gpu: [drm]                         binner: 
> 16384kb BOs (1)
> [   74.741263] vc4-drm soc:gpu: [drm]                total purged
> BO:      4kb BOs (1)
> ...
> 
> I have only seen this issue on arm64 with latest mainline kernel
> (5.13.0-rc5-00130-gf21b807c3cf8), but also with older kernel versions.
> So it's not a regression. It seems 64 bit needs more CMA.
> 
> In case X started properly i was also able to reproduce these errors
> above by dis- and reconneting HDMI.
> 
> So i increased CMA in bcm283x.dtsi and the problem disappeared:
> 
> iff --git a/arch/arm/boot/dts/bcm283x.dtsi b/arch/arm/boot/dts/bcm283x.dtsi
> index b83a864..d1304cb 100644
> --- a/arch/arm/boot/dts/bcm283x.dtsi
> +++ b/arch/arm/boot/dts/bcm283x.dtsi
> @@ -37,7 +37,7 @@
>  
>          cma: linux,cma {
>              compatible = "shared-dma-pool";
> -            size = <0x4000000>; /* 64MB */
> +            size = <0x6000000>; /* 96MB */
>              reusable;
>              linux,cma-default;
>          };
> 
> The questions are:
> 
> Is this the right way (tm) to fix this problem?

Frankly I don't know if there is a better way. IIRC opensuse and downstream use
DT overlays to cater for this limitation. It seems reasonable to bump the
value. But it'll be in detriment of users that don't care much for graphical
interfaces. Nonetheless, I'm not familiar with how DRM handles CMA/DMA memory.
So let me have a look at it. Maybe there is a SW fix. At first glance I'm
surprised they can't defer to normal page allocations when CMA isn't capable of
honoring the request (like the dma code does).

> And what is a sensible value (don't have a 4K display to test)?

The default for downstream is 256MB. But I've read discussions in the forum
where people needed even more. IIUC it's use-case dependent, resolution is only
one variable, you might then try to run a game and run out of memory there.

Regards,
Nicolas

Re: vc4_bo_create: Failed to allocate from CMA

[Daniel Stone]

On Thu, 17 Jun 2021 at 10:36, nicolas saenz julienne <nsaenz@kernel.org> wrote:
> Frankly I don't know if there is a better way. IIRC opensuse and downstream use
> DT overlays to cater for this limitation. It seems reasonable to bump the
> value. But it'll be in detriment of users that don't care much for graphical
> interfaces. Nonetheless, I'm not familiar with how DRM handles CMA/DMA memory.
> So let me have a look at it. Maybe there is a SW fix. At first glance I'm
> surprised they can't defer to normal page allocations when CMA isn't capable of
> honoring the request (like the dma code does).

DMA transfers can be split into multiple transactions at the cost of
being a bit slower. But there isn't a fallback for graphics buffers;
you can't display a quarter of a screen at a time. If the hardware did
support buffers being backed by multiple discontiguous pages, then it
wouldn't need CMA in the first place ...

Cheers,
Daniel

Re: vc4_bo_create: Failed to allocate from CMA

[Stefan Wahren]

Hi Nicolas,

Am 17.06.21 um 11:36 schrieb nicolas saenz julienne:
> On Sat, 2021-06-12 at 17:17 +0200, Stefan Wahren wrote:
>> Hi,
>>
>> while testing the mainline kernel (arm64, defconfig) on Raspberry Pi 3 B
>> Plus with Raspberry Pi OS - 64 bit, sometimes X doesn't start into
>> desktop properly (unexpected and unusable login screen instead of auto
>> login or mouse pointer is show shorty and than switch back to black
>> screen in a loop). In that case dmesg shows the following:
>>
>> [   74.737106] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
>> CMA:
>> [   74.737558] vc4-drm soc:gpu: [drm]                            V3D: 
>> 28976kb BOs (10)
>> [   74.737602] vc4-drm soc:gpu: [drm]                     V3D
>> shader:     44kb BOs (11)
>> [   74.737632] vc4-drm soc:gpu: [drm]                           dumb:  
>> 4564kb BOs (5)
>> [   74.737664] vc4-drm soc:gpu: [drm]                         binner: 
>> 16384kb BOs (1)
>> [   74.737697] vc4-drm soc:gpu: [drm]                total purged
>> BO:      4kb BOs (1)
>> [   74.739039] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
>> CMA:
>> [   74.739466] vc4-drm soc:gpu: [drm]                            V3D: 
>> 28972kb BOs (9)
>> [   74.739512] vc4-drm soc:gpu: [drm]                     V3D
>> shader:     44kb BOs (11)
>> [   74.739541] vc4-drm soc:gpu: [drm]                           dumb:  
>> 4564kb BOs (5)
>> [   74.739570] vc4-drm soc:gpu: [drm]                         binner: 
>> 16384kb BOs (1)
>> [   74.739602] vc4-drm soc:gpu: [drm]                total purged
>> BO:      4kb BOs (1)
>> [   74.740718] [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from
>> CMA:
>> [   74.741138] vc4-drm soc:gpu: [drm]                            V3D: 
>> 28972kb BOs (9)
>> [   74.741171] vc4-drm soc:gpu: [drm]                     V3D
>> shader:     44kb BOs (11)
>> [   74.741202] vc4-drm soc:gpu: [drm]                           dumb:  
>> 4564kb BOs (5)
>> [   74.741231] vc4-drm soc:gpu: [drm]                         binner: 
>> 16384kb BOs (1)
>> [   74.741263] vc4-drm soc:gpu: [drm]                total purged
>> BO:      4kb BOs (1)
>> ...
>>
>> I have only seen this issue on arm64 with latest mainline kernel
>> (5.13.0-rc5-00130-gf21b807c3cf8), but also with older kernel versions.
>> So it's not a regression. It seems 64 bit needs more CMA.
>>
>> In case X started properly i was also able to reproduce these errors
>> above by dis- and reconneting HDMI.
>>
>> So i increased CMA in bcm283x.dtsi and the problem disappeared:
>>
>> iff --git a/arch/arm/boot/dts/bcm283x.dtsi b/arch/arm/boot/dts/bcm283x.dtsi
>> index b83a864..d1304cb 100644
>> --- a/arch/arm/boot/dts/bcm283x.dtsi
>> +++ b/arch/arm/boot/dts/bcm283x.dtsi
>> @@ -37,7 +37,7 @@
>>  
>>          cma: linux,cma {
>>              compatible = "shared-dma-pool";
>> -            size = <0x4000000>; /* 64MB */
>> +            size = <0x6000000>; /* 96MB */
>>              reusable;
>>              linux,cma-default;
>>          };
>>
>> The questions are:
>>
>> Is this the right way (tm) to fix this problem?
> Frankly I don't know if there is a better way. IIRC opensuse and downstream use
> DT overlays to cater for this limitation. It seems reasonable to bump the
> value. But it'll be in detriment of users that don't care much for graphical
> interfaces. Nonetheless, I'm not familiar with how DRM handles CMA/DMA memory.
> So let me have a look at it. Maybe there is a SW fix. At first glance I'm
> surprised they can't defer to normal page allocations when CMA isn't capable of
> honoring the request (like the dma code does).

a compromise might be to increase the CMA size based on the SoC type
(newer generations have more memory)

BCM2835 => 64 MB
BCM2836, BCM2837 => 256 MB

>
>> And what is a sensible value (don't have a 4K display to test)?
> The default for downstream is 256MB. But I've read discussions in the forum
> where people needed even more. IIUC it's use-case dependent, resolution is only
> one variable, you might then try to run a game and run out of memory there.

Sure this wasn't intend to make everybody happy. But i would expect to
start X reliable at least.

Regards
Stefan

>
> Regards,
> Nicolas
>