Re: [PATCH RFC u-boot-mvebu 0/2] arm: mvebu: Fix eMMC boot

[Martin Rowe <martin.p.rowe@gmail.com>]

On Mon, 6 Mar 2023 at 11:53, Pali Rohár <pali@kernel.org> wrote:

> On Monday 06 March 2023 11:15:35 Martin Rowe wrote:
> > On Sun, 5 Mar 2023 at 16:04, Pali Rohár <pali@kernel.org> wrote:
> >
> > > On Sunday 05 March 2023 12:46:34 Pali Rohár wrote:
> > > > On Sunday 05 March 2023 02:24:27 Martin Rowe wrote:
> > > > > On Sat, 4 Mar 2023 at 10:40, Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > > Boot configuration stored in EXT_CSC register is completely
> ignored
> > > by
> > > > > > BootROM:
> > > > > >
> > > > > >
> > >
> https://lore.kernel.org/u-boot/CAOAjy5SYPPzWKok-BSGYwZwcKOQt_aZPgh6FTbrFd3F=8DM5ZQ@mail.gmail.com/
> > > > > >
> > > > > > Reflect this eMMC booting in documentation and in the code.
> > > > > >
> > > > > > Martin, can you test this patch series if SPL and main U-Boot is
> > > loaded
> > > > > > from the same eMMC HW partition?
> > > > > >
> > > > >
> > > > > boot0: u-boot
> > > > >
> > > > > Works fine, no issues.
> > > > >
> > > > >
> > > > > boot0: zeroed
> > > > > boot1: u-boot
> > > > > user: zeroed
> > > > >
> > > > > It succeeds, eventually...
> > > > > ==============================
> > > > > BootROM - 1.73
> > > > >
> > > > > Booting from MMC
> > > > > BootROM: Bad header at offset 00000000
> > > > > BootROM: Bad header at offset 00200000
> > > > > Switching BootPartitions.
> > > > >
> > > > > U-Boot SPL 2023.04-rc3-00159-gd1653548d2-dirty (Mar 05 2023 -
> 11:50:45
> > > > > +1000)
> > > > > High speed PHY - Version: 2.0
> > > > > EEPROM TLV detection failed: Using static config for Clearfog Pro.
> > > > > Detected Device ID 6828
> > > > > board SerDes lanes topology details:
> > > > >  | Lane # | Speed |  Type       |
> > > > >  --------------------------------
> > > > >  |   0    |   3   | SATA0       |
> > > > >  |   1    |   0   | SGMII1      |
> > > > >  |   2    |   5   | PCIe1       |
> > > > >  |   3    |   5   | USB3 HOST1  |
> > > > >  |   4    |   5   | PCIe2       |
> > > > >  |   5    |   0   | SGMII2      |
> > > > >  --------------------------------
> > > > > High speed PHY - Ended Successfully
> > > > > mv_ddr: 14.0.0
> > > > > DDR3 Training Sequence - Switching XBAR Window to FastPath Window
> > > > > mv_ddr: completed successfully
> > > > > Trying to boot from MMC1
> > > > > ERROR: Invalid kwbimage v1
> > > > > mmc_load_image_raw_sector: mmc block read error
> > > > > spl: mmc: wrong boot mode
> > > > > Trying to boot from BOOTROM
> > > > > Returning to BootROM (return address 0xffff05c4)...
> > > > > Timeout waiting card ready
> > > > > BootROM: Image checksum verification PASSED
> > > > >
> > > > >
> > > > > U-Boot 2023.04-rc3-00159-gd1653548d2-dirty (Mar 05 2023 - 11:50:45
> > > +1000)
> > > > >
> > > > > SoC:   MV88F6828-A0 at 1600 MHz
> > > > > DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
> > > > > Core:  38 devices, 22 uclasses, devicetree: separate
> > > > > MMC:   mv_sdh: 0
> > > > > Loading Environment from MMC... *** Warning - bad CRC, using
> default
> > > > > environment
> > > > >
> > > > > Model: SolidRun Clearfog A1
> > > > > Board: SolidRun Clearfog Pro
> > > > > Net:
> > > > > Warning: ethernet@70000 (eth1) using random MAC address -
> > > 12:07:8b:f9:7a:6f
> > > > > eth1: ethernet@70000
> > > > > Warning: ethernet@30000 (eth2) using random MAC address -
> > > ee:ed:f3:bb:c2:af
> > > > > , eth2: ethernet@30000
> > > > > Warning: ethernet@34000 (eth3) using random MAC address -
> > > ae:34:b9:bb:28:c6
> > > > > , eth3: ethernet@34000
> > > > > Hit any key to stop autoboot:  0
> > > > > =>
> > > > > ==============================
> > > > >
> > > > > Between "Returning to BootROM" and "Timeout waiting card ready"
> takes
> > > > > around 315 seconds. That's long enough that I thought it had hung
> > > > > completely and I only noticed it continue because I left it running
> > > while
> > > > > working on something else. I tried several things to reduce this
> > > timeout,
> > > > > including reverting to the "non-removable" dts for shdci, but
> nothing
> > > seems
> > > > > to affect it.
> > > >
> > > > Ok. So now it is in the state that it is working but is slow. Better
> > > > than nothing.
> > > >
> > > > Message "Returning to BootROM" is printed by SPL and message
> > > > "Timeout waiting card ready" is printed by BootROM. After printing
> > > > "Returning to BootROM" is SPL jumping back to the BootROM so the
> delay
> > > > is for sure in the BootROM. So seems that SPL reconfigures eMMC into
> > > > state in which BootROM cannot work with it. Something timeouts,
> BootROM
> > > > reconfigure/retry it and then it work again. It would be needed to
> > > > investigate what is happening here. My guess is that this could have
> > > > something with eMMC HW partition access, and code for switching
> > > > partitions near SPL MMCSD_MODE_EMMCBOOT.
> > >
> > > Try this change?
> > >
> > > diff --git a/arch/arm/mach-mvebu/spl.c b/arch/arm/mach-mvebu/spl.c
> > > index b20eac3dcd38..eb59c41a824e 100644
> > > --- a/arch/arm/mach-mvebu/spl.c
> > > +++ b/arch/arm/mach-mvebu/spl.c
> > > @@ -11,6 +11,7 @@
> > >  #include <image.h>
> > >  #include <init.h>
> > >  #include <log.h>
> > > +#include <mmc.h>
> > >  #include <spl.h>
> > >  #include <asm/global_data.h>
> > >  #include <asm/io.h>
> > > @@ -297,11 +298,33 @@ u32 spl_boot_device(void)
> > >
> > >  #endif
> > >
> > > +void restore_emmc_boot_part_config(void)
> > > +{
> > > +#ifdef CONFIG_SPL_MMC
> > > +       struct mmc *mmc;
> > > +       int ret;
> > > +
> > > +       mmc = find_mmc_device(0);
> > > +       if (!mmc || !mmc->has_init || mmc->part_config ==
> > > MMCPART_NOAVAILABLE)
> > > +               return;
> > > +
> > > +       ret = mmc_set_part_conf(mmc,
> > > +               EXT_CSD_EXTRACT_BOOT_ACK(mmc->part_config),
> > > +               EXT_CSD_EXTRACT_BOOT_PART(mmc->part_config),
> > > +               EXT_CSD_EXTRACT_PARTITION_ACCESS(mmc->part_config));
> > > +       if (ret)
> > > +               printf("Failed to restore eMMC boot partition
> > > configuration\n");
> > > +#endif
> > > +}
> > > +
> > >  int board_return_to_bootrom(struct spl_image_info *spl_image,
> > >                             struct spl_boot_device *bootdev)
> > >  {
> > >         u32 *regs = *(u32 **)(CONFIG_SPL_STACK + 4);
> > >
> > > +       /* restore original eMMC boot partition configuration -
> required
> > > by BootROM */
> > > +       restore_emmc_boot_part_config();
> > > +
> > >         printf("Returning to BootROM (return address 0x%08x)...\n",
> > > regs[13]);
> > >         return_to_bootrom();
> > >
> >
> > Same result, around 5 minutes wait after returning to BootROM:
>
> Could you try to print mmc->part_config (ideally as early as possible)?
>

In SPL mmc->part_config is 255
In main u-boot at the start of clearfog.c board_init() mmc->part_config is
255
In main u-boot at the start of clearfog.c checkboard() mmc->part_config is
8 (ack: 0, partition_enable: 1, access: 0)

If I set partition_enable to 2, I get the same result except the value is
16  (ack: 0, partition_enable: 2, access: 0) instead of 8 for the last value

<partition_enable 1>
BootROM - 1.73

Booting from MMC

U-Boot SPL 2023.04-rc3-00159-gd1653548d2-dirty (Mar 19 2023 - 10:05:32
+1000)
High speed PHY - Version: 2.0
EEPROM TLV detection failed: Using static config for Clearfog Pro.
Detected Device ID 6828
board SerDes lanes topology details:
 | Lane # | Speed |  Type       |
 --------------------------------
 |   0    |   3   | SATA0       |
 |   1    |   0   | SGMII1      |
 |   2    |   5   | PCIe1       |
 |   3    |   5   | USB3 HOST1  |
 |   4    |   5   | PCIe2       |
 |   5    |   0   | SGMII2      |
 --------------------------------
High speed PHY - Ended Successfully
mv_ddr: 14.0.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
mv_ddr: completed successfully
spl.c spl_boot_device part_config = 255
Trying to boot from MMC1


U-Boot 2023.04-rc3-00159-gd1653548d2-dirty (Mar 19 2023 - 10:05:32 +1000)

SoC:   MV88F6828-A0 at 1600 MHz
DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
clearfog.c board_init part_config = 255
Core:  38 devices, 22 uclasses, devicetree: separate
MMC:   mv_sdh: 0
Loading Environment from MMC... *** Warning - bad CRC, using default
environment

Model: SolidRun Clearfog A1
clearfog.c checkboard part_config = 8
Board: SolidRun Clearfog Pro
Net:
Warning: ethernet@70000 (eth1) using random MAC address - 32:16:0e:b4:d1:d8
eth1: ethernet@70000
Warning: ethernet@30000 (eth2) using random MAC address - 72:30:3f:79:07:12
, eth2: ethernet@30000
Warning: ethernet@34000 (eth3) using random MAC address - 82:fb:71:23:46:4f
, eth3: ethernet@34000
Hit any key to stop autoboot:  0
=> mmc partconf 0
EXT_CSD[179], PARTITION_CONFIG:
BOOT_ACK: 0x0
BOOT_PARTITION_ENABLE: 0x1
PARTITION_ACCESS: 0x0
</partition_enable 1>

<partition_enable 2>
BootROM - 1.73

Booting from MMC

U-Boot SPL 2023.04-rc3-00159-gd1653548d2-dirty (Mar 19 2023 - 10:05:32
+1000)
High speed PHY - Version: 2.0
EEPROM TLV detection failed: Using static config for Clearfog Pro.
Detected Device ID 6828
board SerDes lanes topology details:
 | Lane # | Speed |  Type       |
 --------------------------------
 |   0    |   3   | SATA0       |
 |   1    |   0   | SGMII1      |
 |   2    |   5   | PCIe1       |
 |   3    |   5   | USB3 HOST1  |
 |   4    |   5   | PCIe2       |
 |   5    |   0   | SGMII2      |
 --------------------------------
High speed PHY - Ended Successfully
mv_ddr: 14.0.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
mv_ddr: completed successfully
spl.c spl_boot_device part_config = 255
Trying to boot from MMC1


U-Boot 2023.04-rc3-00159-gd1653548d2-dirty (Mar 19 2023 - 10:05:32 +1000)

SoC:   MV88F6828-A0 at 1600 MHz
DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
clearfog.c board_init part_config = 255
Core:  38 devices, 22 uclasses, devicetree: separate
MMC:   mv_sdh: 0
Loading Environment from MMC... *** Warning - bad CRC, using default
environment

Model: SolidRun Clearfog A1
clearfog.c checkboard part_config = 16
Board: SolidRun Clearfog Pro
Net:
Warning: ethernet@70000 (eth1) using random MAC address - 92:5a:fc:14:e8:f6
eth1: ethernet@70000
Warning: ethernet@30000 (eth2) using random MAC address - 42:9c:d8:3a:cb:b2
, eth2: ethernet@30000
Warning: ethernet@34000 (eth3) using random MAC address - c6:99:20:f4:02:a0
, eth3: ethernet@34000
Hit any key to stop autoboot:  0
=> mmc partconf 0
EXT_CSD[179], PARTITION_CONFIG:
BOOT_ACK: 0x0
BOOT_PARTITION_ENABLE: 0x2
PARTITION_ACCESS: 0x0
</partition_enable 2>

I'm having trouble trying to find the hooks which run between board_init
and checkboard. If you can point me in the right direction I'm happy to
re-run and try to narrow down where the valid values are being set from.

> =====================================
> > BootROM - 1.73
> >
> > Booting from MMC
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> >
> > U-Boot SPL 2023.04-rc3-00159-gd1653548d2-dirty (Mar 06 2023 - 21:02:40
> > +1000)
> > High speed PHY - Version: 2.0
> > EEPROM TLV detection failed: Using static config for Clearfog Pro.
> > Detected Device ID 6828
> > board SerDes lanes topology details:
> >  | Lane # | Speed |  Type       |
> >  --------------------------------
> >  |   0    |   3   | SATA0       |
> >  |   1    |   0   | SGMII1      |
> >  |   2    |   5   | PCIe1       |
> >  |   3    |   5   | USB3 HOST1  |
> >  |   4    |   5   | PCIe2       |
> >  |   5    |   0   | SGMII2      |
> >  --------------------------------
> > High speed PHY - Ended Successfully
> > mv_ddr: 14.0.0
> > DDR3 Training Sequence - Switching XBAR Window to FastPath Window
> > mv_ddr: completed successfully
> > Trying to boot from MMC1
> > ERROR: Invalid kwbimage v1
> > mmc_load_image_raw_sector: mmc block read error
> > spl: mmc: wrong boot mode
> > Trying to boot from BOOTROM
> > Returning to BootROM (return address 0xffff05c4)...
> > Timeout waiting card ready
> > BootROM: Image checksum verification PASSED
> >
> >
> > U-Boot 2023.04-rc3-00159-gd1653548d2-dirty (Mar 06 2023 - 21:02:40 +1000)
> > =====================================
> >
> > > Could you try another test by completely erasing BOOT0, BOOT1 and USER
> > > > data? And see what BootROM prints.
> > >
> >
> > =====================================
> > BootROM - 1.73
> >
> > Booting from MMC
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad header at offset 00200000
> > Switching BootPartitions.
> > BootROM: Bad header at offset 00000000
> > BootROM: Bad h
> > Trying Uart
> > =====================================
> >
> > > >
> > > > > When bootloader is stored on Boot 0, then SPL should take care of
> > > > > > loading and executing main U-Boot. When it is stored on Boot 1 or
> > > User
> > > > > > Data then SPL should return back to BootROM and let BootROM to
> load
> > > and
> > > > > > execute main U-Boot.
> > > > > >
> > > > > > Pali Rohár (2):
> > > > > >   tools: kwboot: Fix MMC HW boot partitions info
> > > > > >   arm: mvebu: spl: Load proper U-Boot from eMMC Boot 0 partition
> > > > > >
> > > > > >  arch/arm/mach-mvebu/Kconfig |  1 +
> > > > > >  arch/arm/mach-mvebu/spl.c   | 13 +++++++------
> > > > > >  tools/kwboot.c              |  6 +++---
> > > > > >  3 files changed, 11 insertions(+), 9 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.20.1
> > > > > >
> > > > > >
> > >
>