STM32MP1: Adding TF-A causes kernel errors

[Jan Kiszka <jan.kiszka@web.de>]

[BCC'ed TF-A only, migrating to u-boot, including folks involved there]

On 30.09.20 11:20, Yann GAUTIER wrote:
> Hi Jan,
>
> After discussing with my colleagues, it seems there are 2 issues there.
> One patch is missing in U-Boot:
> http://patchwork.ozlabs.org/project/uboot/patch/20200605092244.1.I773bf523d9f4d1a6212483d030e34113b832a779 at changeid/
>

I can confirm that this resolves the errors I've seen.

Will that patch still hit 2020.10?

> And it shows an issue in kernel, with GPU and CMA regions.
> A correction in kernel should also be done:
>
> diff --git a/arch/arm/boot/dts/stm32mp157c-ed1.dts
> b/arch/arm/boot/dts/stm32mp157c-ed1.dts
> index ca109dc18..685dd61c0 100644
> --- a/arch/arm/boot/dts/stm32mp157c-ed1.dts
> +++ b/arch/arm/boot/dts/stm32mp157c-ed1.dts
> @@ -66,8 +66,8 @@
>   			no-map;
>   		};
>
> -		gpu_reserved: gpu at e8000000 {
> -			reg = <0xe8000000 0x8000000>;
> +		gpu_reserved: gpu at f6000000 {
> +			reg = <0xf6000000 0x8000000>;
>   			no-map;
>   		};
>   	};
>

There might be more issues:

[    3.428914] stm32_rtc 5c004000.rtc: IRQ index 1 not found
[    3.432892] stm32_rtc 5c004000.rtc: alarm can't wake up the system: -6
[    3.440290] stm32_rtc 5c004000.rtc: registered as rtc0
[    3.444560] stm32_rtc 5c004000.rtc: setting system clock to 2000-01-01T00:00:07 UTC (946684807)
[    3.453530] stm32_rtc 5c004000.rtc: Date/Time must be initialized
[    3.459333] stm32_rtc 5c004000.rtc: registered rev:1.2
[    3.465993] i2c /dev entries driver
[    3.490226] stm32f7-i2c 40013000.i2c: can't request DMA tx channel
[    3.494951] stm32f7-i2c 40013000.i2c: can't use DMA
[    3.504213] stm32f7-i2c 40013000.i2c: STM32F7 I2C-0 bus adapter
[    3.529032] stm32f7-i2c 40015000.i2c: can't request DMA tx channel
[    3.533779] stm32f7-i2c 40015000.i2c: can't use DMA
[    3.539356] stm32f7-i2c 40015000.i2c: STM32F7 I2C-1 bus adapter
[    3.567561] stm32f7-i2c 5c002000.i2c: can't request DMA tx channel
[    3.572312] stm32f7-i2c 5c002000.i2c: can't use DMA

But those would be a kernel topic, I suppose.

Thanks for the quick resolution,
Jan

>
> Regards,
> Yann
>
> On 9/30/20 11:18 AM, Jan Kiszka wrote:
>> On 30.09.20 10:40, Yann GAUTIER wrote:
>>> Hi Jan,
>>>
>>> The optee nodes should be removed by U-Boot.
>>> Check arch/arm/mach-stm32mp/fdt.c file and function
>>> stm32_fdt_disable_optee().
>>
>> CONFIG_OPTEE is obviously enabled in stm32mp15_trusted_defconfig, and
>> there must be something found by tee_find_device()...
>>
>> FWIW, I've now also tested TF-A and U-Boot HEAD, just to avoid missing
>> some fix, but there is no change.
>>
>>>
>>> I'll check with my colleagues in charge of U-Boot and kernel and get
>>> back to you.
>>>
>>
>> TIA!
>> Jan
>>
>>>
>>> Regards,
>>> Yann
>>>
>>> On 9/30/20 9:05 AM, Jan Kiszka wrote:
>>>> On 29.09.20 23:42, Jan Kiszka wrote:
>>>>> Hi Yann,
>>>>>
>>>>> not sure if TF-A is the one to blame, but it's the variable that
>>>>> triggers the following on the STM32MP15x eval board for me. I think I'm
>>>>> following tfa/docs/plat/stm32mp1.rst and
>>>>> u-boot/doc/board/st/stm32mp1.rst correctly.
>>>>>
>>>>> Working:
>>>>> - U-Boot 2020.07, stm32mp15_basic_defconfig
>>>>> - Linux 5.9-rc7 (or 5.4.x), defconfig
>>>>>
>>>>> [    0.000000] Memory: 815540K/917500K available (13312K kernel code, 1800K rwdata, 5452K rodata, 2048K init, 407K bss, 36424K reserved, 65536K cma-reserved, 196604K highmem)
>>>>>
>>>>> Failing:
>>>>> - TF-A 2.3, PLAT=stm32mp1 ARCH=aarch32 ARM_ARCH_MAJOR=7 \
>>>>>     AARCH32_SP=sp_min STM32MP_SDMMC=1 STM32MP_EMMC=1 STM32MP_RAW_NAND=1 \
>>>>>     STM32MP_SPI_NAND=1 STM32MP_SPI_NOR=1 DTB_FILE_NAME=stm32mp157c-ev1.dtb
>>>>> - U-Boot 2020.07, stm32mp15_trusted_defconfig
>>>>> - Linux as above
>>>>>
>>>>> [    0.000000] Memory: 881076K/917500K available (13312K kernel code, 1800K rwdata, 5452K rodata, 2048K init, 407K bss, 4294938184K reserved, 65536K cma-reserved, 262140K highmem)
>>>>>
>>>>> which causes issues like
>>>>>
>>>>> [    0.047215] BUG: Bad page state in process swapper/0  pfn:fa000
>>>>> [    0.047236] page:(ptrval) refcount:0 mapcount:-128 mapping:00000000 index:0x1 pfn:0xfa000
>>>>> [    0.047249] flags: 0x80000000() CMA
>>>>> [    0.047273] raw: 80000000 e7f29004 e7f49004 00000000 00000001 0000000b ffffff7f 00000000
>>>>> [    0.047281] page dumped because: nonzero mapcount
>>>>> [    0.047287] Modules linked in:
>>>>> [    0.047305] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.9.0-rc7 #1
>>>>> [    0.047314] Hardware name: STM32 (Device Tree Support)
>>>>> [    0.047358] [<c0311708>] (unwind_backtrace) from [<c030b88c>] (show_stack+0x10/0x14)
>>>>> [    0.047384] [<c030b88c>] (show_stack) from [<c0718a40>] (dump_stack+0xc8/0xdc)
>>>>> [    0.047408] [<c0718a40>] (dump_stack) from [<c047b3c8>] (bad_page+0xdc/0x10c)
>>>>> [    0.047426] [<c047b3c8>] (bad_page) from [<c047c060>] (__free_pages_ok+0x2e8/0x36c)
>>>>> [    0.047447] [<c047c060>] (__free_pages_ok) from [<c1623a80>] (init_cma_reserved_pageblock+0x58/0x68)
>>>>> [    0.047469] [<c1623a80>] (init_cma_reserved_pageblock) from [<c16266c8>] (cma_init_reserved_areas+0x170/0x1c8)
>>>>> [    0.047491] [<c16266c8>] (cma_init_reserved_areas) from [<c0301ef8>] (do_one_initcall+0x54/0x22c)
>>>>> [    0.047513] [<c0301ef8>] (do_one_initcall) from [<c160102c>] (kernel_init_freeable+0x188/0x1ec)
>>>>> [    0.047537] [<c160102c>] (kernel_init_freeable) from [<c0f4a340>] (kernel_init+0x8/0x118)
>>>>> [    0.047559] [<c0f4a340>] (kernel_init) from [<c03001a8>] (ret_from_fork+0x14/0x2c)
>>>>> [    0.047570] Exception stack(0xe68b7fb0 to 0xe68b7ff8)
>>>>> [    0.047584] 7fa0:                                     00000000 00000000 00000000 00000000
>>>>> [    0.047600] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
>>>>> [    0.047614] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
>>>>> [    0.047624] Disabling lock debugging due to kernel taint
>>>>>
>>>>> Still, the system boots, and I can login.
>>>>>
>>>>> That reserved value the kernel finds is obviously off. Does it come from
>>>>> TF-A, is U-Boot causing this in the presence of TF-A, or is the kernel
>>>>> itself getting it wrong? Or am I missing some switch that is not in the
>>>>> kernel defconfig?
>>>>>
>>>>> Thanks,
>>>>> Jan
>>>>>
>>>>
>>>> Looking at the dtb diff (no-tfa -> with-tfa, based on
>>>> /sys/firmware/fdt), I'm starting to believe it's a kernel issues,
>>>> possibly some signed 32-bit -> 64-bit expansion:
>>>>
>>>> --- /dev/fd/63	2020-09-30 08:39:43.870749258 +0200
>>>> +++ /dev/fd/62	2020-09-30 08:39:43.874749249 +0200
>>>> @@ -1,6 +1,6 @@
>>>>    /dts-v1/;
>>>>
>>>> -/memreserve/	0x00000000cfbcb000 0x00000000004349c5;
>>>> +/memreserve/	0x00000000cfb5c000 0x00000000004a3faa;
>>>>    / {
>>>>    	serial-number = "003100453338511534383330";
>>>>    	#address-cells = <0x01>;
>>>> @@ -8,12 +8,19 @@
>>>>    	model = "STMicroelectronics STM32MP157C eval daughter on eval mother";
>>>>    	compatible = "st,stm32mp157c-ev1\0st,stm32mp157c-ed1\0st,stm32mp157";
>>>>
>>>> +	firmware {
>>>> +
>>>> +		optee {
>>>> +			method = "smc";
>>>> +			compatible = "linaro,optee-tz";
>>>> +		};
>>>> +	};
>>>> +
>>>>    	cpus {
>>>>    		#address-cells = <0x01>;
>>>>    		#size-cells = <0x00>;
>>>>
>>>>    		cpu at 0 {
>>>> -			enable-method = "psci";
>>>>    			compatible = "arm,cortex-a7";
>>>>    			clock-frequency = <0x26be3680>;
>>>>    			device_type = "cpu";
>>>> @@ -21,7 +28,6 @@
>>>>    		};
>>>>
>>>>    		cpu at 1 {
>>>> -			enable-method = "psci";
>>>>    			compatible = "arm,cortex-a7";
>>>>    			clock-frequency = <0x26be3680>;
>>>>    			device_type = "cpu";
>>>> @@ -30,8 +36,7 @@
>>>>    	};
>>>>
>>>>    	psci {
>>>> -		status = "okay";
>>>> -		compatible = "arm,psci-1.0\0arm,psci-0.2";
>>>> +		compatible = "arm,psci-1.0";
>>>>    		method = "smc";
>>>>    	};
>>>>
>>>> @@ -3804,9 +3809,9 @@
>>>>    	};
>>>>
>>>>    	chosen {
>>>> -		linux,initrd-end = <0xcffff9c5>;
>>>> -		linux,initrd-start = <0xcfbcb000>;
>>>> -		bootargs = "root=PARTUUID=0a553417-d224-4c81-b052-0679fb761cf5 rootwait rw console=ttySTM0,115200";
>>>> +		linux,initrd-end = <0xcfffffaa>;
>>>> +		linux,initrd-start = <0xcfb5c000>;
>>>> +		bootargs = "root=PARTUUID=a9089592-b0fe-4c14-8b27-e1b398043c47 rootwait rw console=ttySTM0,115200";
>>>>    		stdout-path = "serial0:115200n8";
>>>>    	};
>>>>
>>>> @@ -3820,6 +3825,10 @@
>>>>    		#size-cells = <0x01>;
>>>>    		ranges;
>>>>
>>>> +		optee at fe000000 {
>>>> +			reg = <0xfe000000 0x2000000>;
>>>> +		};
>>>> +
>>>>    		mcuram2 at 10000000 {
>>>>    			compatible = "shared-dma-pool";
>>>>    			reg = <0x10000000 0x40000>;
>>>>
>>>>
>>>> OTOH, I'm wondering what that optee entry is about - I have none
>>>> configured...
>>>>
>>>> Anyway, if you can confirm it's a kernel thing, I'm happy to move this
>>>> to another list.
>>>>
>>>> Thanks,
>>>> Jan
>>>>
>>>> PS: This was supposed to be quick (which worked exceptionally well for
>>>> the first, "untrusted" part, thanks to everything-is-upstream), just to
>>>> demonstrate custom TF-A packing for Debian via the image generator
>>>> Isar...