Re: [RFC PATCH 00/16] arm: Add Rockchip RK3588 support

[Jonas Karlman <jonas@kwiboo.se>]

Hi Jagan,
On 2023-01-26 20:17, Jagan Teki wrote:
> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas@kwiboo.se> wrote:
>>
>> On 2023-01-26 19:26, Jagan Teki wrote:
>>> Hi Simon,
>>>
>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg@chromium.org> wrote:
>>>>
>>>> Hi Jagan,
>>>>
>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan@edgeble.ai> wrote:
>>>>>
>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas@kwiboo.se> wrote:
>>>>>>
>>>>>> Hi Jagan,
>>>>>> On 2023-01-26 17:51, Jagan Teki wrote:
>>>>>>> Hi Jonas,
>>>>>>>
>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas@kwiboo.se> wrote:
>>>>>>>>
>>>>>>>> Hi Jagan,
>>>>>>>>
>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote:
>>>>>>>>> This series support Rockchip RK3588. All the device tree files are
>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit
>>>>>>>>> messages.
>>>>>>>>>
>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so
>>>>>>>>> it is failing to load ATF entry from SPL and hang.
>>>>>>>>>
>>>>>>>>> Verified below BL31 versions,
>>>>>>>>>   bl31-v1.15
>>>>>>>>>   bl31-v1.21
>>>>>>>>>   bl31-v1.22
>>>>>>>>>   bl31-v1.23
>>>>>>>>>   bl31-v1.24
>>>>>>>>>   bl31-v1.25
>>>>>>>>>   bl31-v1.26
>>>>>>>>>
>>>>>>>>> Rever-engineered with respect to rockchip u-boot by using the same
>>>>>>>>> FIT_GENERATOR being used in Mainline, rockchip u-boot is booting but
>>>>>>>>> mainline showing the same issue.
>>>>>>>>>
>>>>>>>>> Log:
>>>>>>>>>
>>>>>>>>> LPDDR4X, 2112MHz01-00642-g6bdfd31756-dirty (Jan 26 2023 ���3:44:34 +0530)
>>>>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
>>>>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
>>>>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
>>>>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=8 Size=4096MB
>>>>>>>>> change to F1: 528MHz
>>>>>>>>> change to F2: 1068MHz
>>>>>>>>> change to F3: 1560MHz
>>>>>>>>> change to F0: 2112MHz
>>>>>>>>> out
>>>>>>>>>
>>>>>>>>> U-Boot SPL 2023.01-00642-g6bdfd31756-dirty (Jan 26 2023 - 03:44:34 +0530)
>>>>>>>>> Trying to boot from MMC1
>>>>>>>>> bl31_entry: atf_entry start
>>>>>>>>> << hang >>
>>>>>>>>>
>>>>>>>>> Any information on BL31 for RK3588 please share.
>>>>>>>>
>>>>>>>> I had a similar strange booing issue with RK3568 and mainline U-Boot,
>>>>>>>> turned out to be related to all parts of ATF not being properly loaded
>>>>>>>> into PMU SRAM.
>>>>>>>>
>>>>>>>> Using my series at [1] I managed to get ATF to be fully loaded into
>>>>>>>> PMU SRAM. Using CONFIG_SPL_FIT_SIGNATURE=y helped me finding out that
>>>>>>>> the segment being loaded ended up corrupted.
>>>>>>>>
>>>>>>>> The use of 512 bytes alignment of the FIT helped mitigate that issue.
>>>>>>>> Vendor U-Boot use a bounce buffer for all parts that is written into
>>>>>>>> SRAM (anything loaded outside the gd->ram_base to gd->ram_top range).
>>>>>>>>
>>>>>>>> You can also find newer bl31 at [2], up to version v1.32.
>>>>>>>>
>>>>>>>> [1] https://patchwork.ozlabs.org/project/uboot/list/?series=337891 [2] https://gitlab.com/rk3588_linux/rk/rkbin/-/tree/linux-5.10-gen-rkr3.5/bin/rk35 Thanks for the details. I did apply this set on the master. No change
>>>>>>> in the behavior, used BL31 and ddr from [2] as well as in
>>>>>>> rkbin/master.
>>>>>>
>>>>>> I did some tests on my Radxa ROCK 3 Model B with CONFIG_SPL_FIT_SIGNATURE=y
>>>>>> and it looked like it failed to read data into memory, see below.
>>>>>>
>>>>>> It also looks like the sdhci compatible is not supported by the driver.
>>>>>> Something that may need to be added to driver to properly read data?
>>>>>>
>>>>>>
>>>>>> DDR V1.09 a930779e06 typ 22/11/21-17:50:56
>>>>>> LPDDR4X, 2112MHz
>>>>>> channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
>>>>>> channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
>>>>>> channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
>>>>>> channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
>>>>>> Manufacturer ID:0x6
>>>>>> CH0 RX Vref:31.7%, TX Vref:21.8%,20.8%
>>>>>> CH1 RX Vref:31.7%, TX Vref:21.8%,21.8%
>>>>>> CH2 RX Vref:32.7%, TX Vref:22.8%,21.8%
>>>>>> CH3 RX Vref:32.7%, TX Vref:21.8%,20.8%
>>>>>> change to F1: 528MHz
>>>>>> change to F2: 1068MHz
>>>>>> change to F3: 1560MHz
>>>>>> change to F0: 2112MHz
>>>>>> out
>>>>>>
>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 00:24:53 +0000)
>>>>>> Trying to boot from MMC1
>>>>>> ## Checking hash(es) for config config_1 ... OK
>>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error!
>>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node
>>>>>> mmc_load_image_raw_sector: mmc block read error
>>>>>> Trying to boot from MMC1
>>>>>> ## Checking hash(es) for config config_1 ... OK
>>>>>> ## Checking hash(es) for Image atf_1 ... sha256 error!
>>>>>> Bad hash value for 'hash' hash node in 'atf_1' image node
>>>>>
>>>>> Look like this is something wrong with your patch series or master
>>>>> changes on binman, not with the driver. I have observed the same if I
>>>>> enable CONFIG_SPL_FIT_SIGNATURE.
>>>>
>>>> There are some more changes in dm/master that I'm about to pull in.
>>>> One of them from Jonas Karlman adds hash nodes so may be involved.
>>>
>>> I found the same issue on the dm/master
>>>
>>> U-Boot SPL 2023.01-00176-gb21fb7a9c0 (Jan 26 2023 - 23:55:11 +0530)
>>> Trying to boot from MMC1
>>> ## Checking hash(es) for config config-1 ... OK
>>> ## Checking hash(es) for Image atf-1 ... sha256 error!
>>> Bad hash value for 'hash' hash node in 'atf-1' image node
>>> mmc_load_image_raw_sector: mmc block read error
>>> SPL: failed to boot from all boot devices
>>> ### ERROR ### Please RESET the board ###
>>
>> On my RK3568 ROCK 3A board this is working correctly, see below.
>>
>> This was using u-boot master 17e8e58fe62c019b2cc26af221b6defc3368229f
>> with a few patches on top, see [1].
>>
>> -------------
>> U-Boot SPL 2023.01 (Jan 26 2023 - 18:29:56 +0000)
>> Trying to boot from MMC1
>> ## Checking hash(es) for config config-1 ... OK
>> ## Checking hash(es) for Image atf-1 ... sha256+ OK
>> ## Checking hash(es) for Image u-boot ... sha256+ OK
>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK
>> ## Checking hash(es) for Image atf-2 ... sha256+ OK
>> ## Checking hash(es) for Image atf-3 ... sha256+ OK
>> ## Checking hash(es) for Image atf-4 ... sha256+ OK
>> ## Checking hash(es) for Image atf-5 ... sha256+ OK
>> ## Checking hash(es) for Image atf-6 ... sha256+ OK
>> INFO:    Preloader serial: 2
>> NOTICE:  BL31: v2.3():v2.3-460-g2c8be93f9:huan.he
>> NOTICE:  BL31: Built : 19:45:17, Nov  8 2022
>> INFO:    GICv3 without legacy support detected.
>> INFO:    ARM GICv3 driver initialized in EL3
>> INFO:    pmu v1 is valid 220114
>> INFO:    dfs DDR fsp_param[0].freq_mhz= 1560MHz
>> INFO:    dfs DDR fsp_param[1].freq_mhz= 324MHz
>> INFO:    dfs DDR fsp_param[2].freq_mhz= 528MHz
>> INFO:    dfs DDR fsp_param[3].freq_mhz= 780MHz
>> INFO:    Using opteed sec cpu_context!
>> INFO:    boot cpu mask: 0
>> INFO:    BL31: Initializing runtime services
>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
>> ERROR:   Error initializing runtime service opteed_fast
>> INFO:    BL31: Preparing for EL3 exit to normal world
>> INFO:    Entry point address = 0xa00000
>> INFO:    SPSR = 0x3c9
>>
>>
>> U-Boot 2023.01 (Jan 26 2023 - 18:29:56 +0000)
>>
>> Model: Rockchip RK3568 EVB1 DDR4 V10 Board
>> DRAM:  8 GiB (effective 7.7 GiB)
>> Core:  58 devices, 17 uclasses, devicetree: separate
>> MMC:   mmc@fe2b0000: 1, mmc@fe310000: 0
>> ...
>> -------------
>>
>> However, on my RK3588 ROCK 5B device I get a similar error you get.
>> Seems to be the mmc reading that times out all of sudden.
>>
>> It can read the FIP and config, and then some mmc command/transfer times out.
>> See below for details with LOG_DEBUG defined at top of include/log.h
>>
>> This was trying to boot from a SD-card, trying to use eMMC fails earlier.
>> Because there is no driver for emmc compatible, SPL only tries to boot from SD-card.
>>
>> I believe this is related to your << hang >>, reading atf
>> from mmc fails in the background and there is no error message.
>>
>> That is why I suggested trying with CONFIG_SPL_FIT_SIGNATURE=y,
>> to see if there is any hidden issue trying to load ATF.
>>
>> -------------
>> Trying to boot from MMC1
>> ## Checking hash(es) for config config-1 ... OK
>> mmc_load_image_raw_sector: mmc block read error
>> Trying to boot from MMC1
>> mmc_load_image_raw_sector: mmc block read error
>> SPL: failed to boot from all boot devices
>> ### ERROR ### Please RESET the board ###
>> -------------
>> Trying to boot from MMC1
>> 0
>>    - 0 'mmc@fe2c0000'
>>    - found
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> clock is disabled (0Hz)
>> Buswidth = 0, clock: 0
>> Buswidth = 1, clock: 0
>> clock is enabled (400000Hz)
>> Buswidth = 1, clock: 400000
>> Sending CMD0
>> Sending CMD8
>> Sending CMD55
>> Sending CMD41
>> Sending CMD55
>> Sending CMD41
>> Sending CMD2
>> Sending CMD3
>> Sending CMD9
>> Sending CMD7
>> Sending CMD55
>> Sending CMD51
>> Sending CMD6
>> Sending CMD55
>> Sending CMD6
>> Buswidth = 4, clock: 400000
>> Sending CMD6
>> clock is enabled (50000000Hz)
>> Buswidth = 4, clock: 50000000
>> clk_set_rate(clk=500000, rate=50000000)
>> rockchip_dwmmc_get_mmc_clk: err=-2
>> spl: mmc boot mode: raw
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> Sending CMD16
>> Sending CMD17
>> hdr read sector 4000, count=1
>> Found FIT
>> size=a00, ptr=ac0, limit=100000: aligned to 5000c0
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> Sending CMD16
>> Sending CMD18
>> Sending CMD12
>> fit read sector 4000, sectors=5, dst=5000c0, count=5, size=0xa00
>> Selecting default config 'rk3588-evb.dtb'
>> ## Checking hash(es) for config config-1 ... fit_config_verify_required_keys: No signature node found: FDT_ERR_NOTFOUND
>> OK
>> firmware: 'atf-1'
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> Sending CMD16
>> Sending CMD18
>> dwmci_data_transfer: Timeout waiting for data!
>> mmc_load_image_raw_sector: mmc block read error
>> spl: mmc boot mode: fs
>> Trying to boot from MMC1
>> 0
>>    - 0 'mmc@fe2c0000'
>>    - found
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> spl: mmc boot mode: raw
>> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
>> Sending CMD16
>> dwmci_send_cmd: Response Timeout.
>> hdr read sector 4000, count=0
>> mmc_load_image_raw_sector: mmc block read error
>> spl: mmc boot mode: fs
>> SPL: failed to boot from all boot devices
>> ### ERROR ### Please RESET the board ###
> 
> I'm thinking this is fit/binman issue rather than mmc.
> 
> U-Boot SPL 2023.01-00176-gb21fb7a9c0-dirty (Jan 27 2023 - 00:43:52 +0530)
> SPL malloc() before relocation used 0xbe0 bytes (2 KB)
>>> SPL: board_init_r()
> spl_init
> size=a0, ptr=a0, limit=100000: 500000
> size=8, ptr=a8, limit=100000: 5000a0
> fdtdec_get_addr_size_auto_parent: na=2, ns=2,
> fdtdec_get_addr_size_fixed: reg: addr=00000000x
> ofnode_read_u32_index: bus-width: x (4)
> ofnode_read_bool: non-removable: false
> ofnode_read_u32_index: fifo-depth: x (256)
> ofnode_read_bool: fifo-mode: false
> ofnode_read_bool: u-boot,spl-fifo-mode: false
> ofnode_read_u32_array: clock-freq-min-max: fdtdec_get_int_array:
> clock-freq-min-max
> get_prop_check_min_len: clock-freq-min-max
> ofnode_read_u32_index: max-frequency: x (200000000)
> clk_set_defaults(mmc@fe2c0000)
> clk_set_default_parents: could not read assigned-clock-parents for 3f60700
> ofnode_read_prop: assigned-clock-rates: <not found>
> fdtdec_get_int: #clock-cells: x (1)
> fdtdec_get_int: #clock-cells: x (1)
> Looking for clock-controller@fd7c0000
> Looking for clock-controller@fd7c0000
>       - checking clock-1
>       - checking clock-controller@fd7c0000
>    - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0)
>    - result for clock-controller@fd7c0000: clock-controller@fd7c0000 (ret=0)
> clk_of_xlate_default(clk=500000)
> clk_request(dev=3f60320, clk=500000)
> clk_set_rate(clk=500000, rate=400000)
> rockchip_dwmmc_get_mmc_clk: err=-2
> Trying to boot from MMC1
> 0
>    - 0 'mmc@fe2c0000'
>    - found
> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
> clock is disabled (0Hz)
> Buswidth = 0, clock: 0
> Buswidth = 1, clock: 0
> clock is enabled (400000Hz)
> Buswidth = 1, clock: 400000
> Sending CMD0
> Sending CMD8
> Sending CMD55
> Sending CMD41
> Sending CMD55
> Sending CMD41
> Sending CMD55
> Sending CMD41
> Sending CMD2
> Sending CMD3
> Sending CMD9
> Sending CMD7
> Sending CMD55
> Unaligned buffer length u
> size=40, ptr=100, limit=100000: aligned to 5000c0
> Sending CMD51
> Sending CMD6
> Sending CMD55
> Sending CMD6
> Buswidth = 4, clock: 400000
> Sending CMD6
> clock is enabled (50000000Hz)
> Buswidth = 4, clock: 50000000
> clk_set_rate(clk=500000, rate=50000000)
> rockchip_dwmmc_get_mmc_clk: err=-2
> spl: mmc boot mode: raw
> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
> Sending CMD16
> Sending CMD17
> hdr read sector 4000, count=1
> Found FIT
> size=a00, ptr=b00, limit=100000: aligned to 500100
> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
> Sending CMD16
> Sending CMD18
> Sending CMD12
> fit read sector 4000, sectors=5, dst=500100, count=5, size=0xa00
> Selecting default config 'rk3588-edgeble-neu6a-io.dtb'
> ## Checking hash(es) for config config-1 ...
> fit_config_verify_required_keys: No signature node found:
> FDT_ERR_NOTFOUND
> OK
> firmware: 'atf-1'
> blk_find_device: uclass_id=67, devnum=0: mmc@fe2c0000.blk, 67, 0
> Sending CMD16
> Sending CMD18
> dwmci_data_transfer: Timeout waiting for data!
> Sending CMD12
> External data: dst=40000, offset=ad200, size=2e940
> ## Checking hash(es) for Image atf-1 ...
> fit_image_verify_required_sigs: No signature node found:
> FDT_ERR_NOTFOUND
> sha256 error!
> Bad hash value for 'hash' hash node in 'atf-1' image node
> mmc_load_image_raw_sector: mmc block read error
> spl: mmc boot mode: fs
> SPL: failed to boot from all boot devices
> ### ERROR ### Please RESET the board ###

As you can see in the logs above there is timeout waiting for data.

I managed to find the issue and have a workaround that gets me longer
in the boot process, there still seem to be other issue with the rk3588
startup.

--------
U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000)
Trying to boot from MMC1
## Checking hash(es) for config config-1 ... OK
## Checking hash(es) for Image atf-1 ... sha256+ OK
## Checking hash(es) for Image u-boot ... sha256+ OK
## Checking hash(es) for Image fdt-1 ... sha256+ OK
## Checking hash(es) for Image atf-2 ... sha256+ OK
## Checking hash(es) for Image atf-3 ... sha256+ OK
INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-468-ge529a2760:derrick.huang
NOTICE:  BL31: Built : 09:59:49, Nov 21 2022
INFO:    spec: 0x1
INFO:    ext 32k is not valid
INFO:    ddr: stride-en 4CH
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0
INFO:    system boots from cpu-hwid-0
INFO:    idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001
INFO:    dfs DDR fsp_params[0].freq_mhz= 2112MHz
INFO:    dfs DDR fsp_params[1].freq_mhz= 528MHz
INFO:    dfs DDR fsp_params[2].freq_mhz= 1068MHz
INFO:    dfs DDR fsp_params[3].freq_mhz= 1560MHz
INFO:    BL31: Initialising Exception Handling Framework
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9
"Synchronous Abort" handler, esr 0x96000000
elr: 0000000000a23650 lr : 0000000000a24d9c
x0 : 0000000000b7fbe8 x1 : 350003402a0003f3
x2 : 0000000000000000 x3 : 0000000000b80ff0
x4 : 0000000000b80ff0 x5 : 0000000000b80e88
x6 : 0000000000000054 x7 : 0000000000000044
x8 : 000000000000000a x9 : 0000000000000000
x10: 0000000000000034 x11: 0000000000000002
x12: 0000000000001988 x13: 0000000000b7fadc
x14: 0000000000a7e808 x15: 0000000000a7e808
x16: 0000000000000000 x17: 0000000000000000
x18: 0000000000b7fe50 x19: 0000000000b7fbe8
x20: 000000003c14dc00 x21: 000000003c14dc00
x22: 0000000000a7e808 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000000 x27: 0000000000000000
x28: 0000000000000000 x29: 0000000000b7fb80

Code: f90013f5 f9400001 b4000201 f9400021 (f9403435)
Resetting CPU ...
--------

This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e
with some rk3568 patches and this series, see [1].

The last 3 commits contains workaround to issue with sdmmc clock.
dwmmc driver set sclk to (uint)-2, my workaround just adds a
fallback to default 400khz clock.

Next issue is the sync abort, looks it happens when u-boot
tries to set clock rates based on devicetree. this is the
last debug line before the crash.

clk_set_rate(clk=0000000000b7fba8, rate=1008000000)

[1] https://github.com/Kwiboo/u-boot-rockchip/compare/060a65e899859dcbf42049a18be20ce7118e7c0e...be21c34067d6e3a2036ce70d571700170c59d270

Regards,
Jonas

> 
> Please check this repo, it is on top of the dm/master
> https://github.com/edgeble/u-boot/tree/neu6-v1
> 
> Jagan.