[U-Boot] UBoot running UBoot - is it possible?

[U-Boot] UBoot running UBoot - is it possible?

[Allan Chandler]

Hello, bods.

We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.

However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.

What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:

1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).

2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.

The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.

We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.

So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.

1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
            mmcboot=run check_uboot; <rest of original mmcboot>
            other_uboot=0

2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.

3/ We defined check_uboot as:
            if test ${other_uboot} -eq 1; then
                        setenv other_uboot 0
                        saveenv
                        ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
                        go 0x17800000
            fi
The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.

4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.

Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.

I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.

So I guess my questions are as follows:

a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?
b/ Does anyone have any ideas on how I could achieve this?

I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.

Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.

Cheers,
Pax

Allan Chandler | Software Engineer
DTI Group Ltd | Integrated Transit Technologies
31 Affleck Road, Perth Airport, WA 6105, Australia
P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler@dti.com.au>
Visit our website www.dti.com.au<http://www.dti.com.au>
The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.

[U-Boot] UBoot running UBoot - is it possible?

[Simon Goldschmidt]

On Tue, Nov 13, 2018 at 1:16 PM Allan Chandler
<allan.chandler@dti.com.au> wrote:
>
> Hello, bods.
>
> We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
>
> However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
>
> What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
>
> 1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
>
> 2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
>
> The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
>
> We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
>
> So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
>
> 1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
>             mmcboot=run check_uboot; <rest of original mmcboot>
>             other_uboot=0
>
> 2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
>
> 3/ We defined check_uboot as:
>             if test ${other_uboot} -eq 1; then
>                         setenv other_uboot 0
>                         saveenv
>                         ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
>                         go 0x17800000
>             fi
> The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
>
> 4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
>
> Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
>
> I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
>
> So I guess my questions are as follows:
>
> a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?
> b/ Does anyone have any ideas on how I could achieve this?

My idea was to let SPL implement a dedicated boot counter/watchdog
that detects problems starting U-Boot and using a backup copy if it
fails multiple times. Of course you need an SPL on your board to do
this. Plus you end up with the same problem for updating SPL, but I'm
lucky here that my platform (socfpga gen5) has redundant storage for
SPL and implements a startup watchdog mechanism for the SPL.

Simon

>
> I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
>
> Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
>
> Cheers,
> Pax
>
> Allan Chandler | Software Engineer
> DTI Group Ltd | Integrated Transit Technologies
> 31 Affleck Road, Perth Airport, WA 6105, Australia
> P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler@dti.com.au>
> Visit our website www.dti.com.au<http://www.dti.com.au>
> The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

[U-Boot] UBoot running UBoot - is it possible?

[Stefano Babic]

Hi Allan,

On 13/11/18 02:38, Allan Chandler wrote:
> Hello, bods.
> 
> We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.

ok - think about that a *safe* update of the bootloader with all
components (SPL, u-boot.img, environment) must be considered and
designed at the early beginning because it could require hardware support.

The i.MX6 can boot from several devices in a sequence, and this can be
used to update the bootloader. For example, it can boot from a SPI NOR
flash if booting from eMMC fails. Of course, you have added a SPI NOR
(or another second source) in your design.

Even in this case, please consider that the BootROM searches for another
source if the first one is definely wroing, that means that the i.MX
header is corrupted. If the image (SPL) is correct, the BootROM cannot
detect it.

This means that with multiple devices where to boot you cannot be sure,
because a valid SPL was written but board does not boot for some reasons.

You could use the EXT CSD register in the eMMC (it looks like you have
just eMMC on board) to switch the boot partition. This is transparent to
the i.MX and eMMC since eMMC 4.3 supports two partitions. This can allow
to write the second bootloader and to be power-off safe, but there is no
fallback. Anyway, the rick can be mitigated with testing, but cannot be
dropped at all. If the second bootloader is valid, the BootROM will
always try to boot from it and there is no fallback if it goes wrong.

To be absolutely sure, a (hw) mechanism that switch between two SPLs,
maybe driven if watchdog expires, could be implemented. Of course,
complexity increases.

The thing is that each project must balance risks again costs. In some
cases, it is also accepted that bootloader is updated with a percentage
of risks that the board could be returned because broken. This can be
cheaper as implementing a 100% safe update for the bootloader, but of
course it cannot be used in critical applications.

You could also plan to update U-Boot, that is u-boot.img on i.MX,
without updating SPL. This is also a used path, with SPL fixed on the
board and checking for two copies of u-boot.img.

However, this is broken. SPL is responsible for the first initialization
and first of all of the DRAM controller on i.MX. If something was
broken, it cannot be fixed just by updating the u-boot.img image. I do
not know if it is worth to have it, it looks like a half-upate where
just a very limited number of issues can be fixed.

> 
> However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
> 
> What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
> 
> 1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
> 
> 2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.

Check if version differs is a *must*. I do this in SWUpdate and the
bootloader is always part of each delivery, but it is updated only if
version differs.

> 
> The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
> 
> We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
> 
> So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.

Think again: you are updating the bootloader, but you rely on u-boot
scripts that are part of u-boot itself. Scripts can be corrupted, too,
or changed by a previous u-boot version, or...

It sounds like a hack, but of course, it could work with a percentage of
risks that things can go wrong. I mean, a percentage without taking into
account Murphy's law :-D

> 
> 1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
>             mmcboot=run check_uboot; <rest of original mmcboot>
>             other_uboot=0
> 
> 2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
> 
> 3/ We defined check_uboot as:
>             if test ${other_uboot} -eq 1; then
>                         setenv other_uboot 0
>                         saveenv
>                         ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
>                         go 0x17800000
>             fi

This means that the *old* U-Boot works pretty well, because it can go
until this point and load something else...

There are a lot of assumption here, because you are not really updating
the bootloader, but simply adding a new actor in the chain. And then,
both can be buggy.....

> The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
> 
> 4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
> 
> Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.

The second bootloader must be fine tuned and changed from original
because it is not allowed to set again the DRAM controller, and maybe
even some closks, and maybe...

> 
> I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
> 
> So I guess my questions are as follows:
> 
> a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?

Define safely. 100% error prone, 97%, ..

100% safe could mean additional hardware and additional complexity, and
at the end, additional risk that something can go wrong.

That means: some projects decide to add hw support or more to be
absolutely sure that nothing can happen, some other projects take the
risk and accept the possibility to have a low RTB. You decide, at the end...

> b/ Does anyone have any ideas on how I could achieve this?

I think that each solution is quite project specific. Other SOCs allow
to switch automatically SPL (that means, the source for the bootloader).
On i.MX6, it can be reached using a sequence of devices (see BOOT_CFG in
manual), but there is no fallback.

> 
> I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
> 
> Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.

Best regards,
Stefano Babic

> 
> Cheers,
> Pax
> 
> Allan Chandler | Software Engineer
> DTI Group Ltd | Integrated Transit Technologies
> 31 Affleck Road, Perth Airport, WA 6105, Australia
> P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler@dti.com.au>
> Visit our website www.dti.com.au<http://www.dti.com.au>
> The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot
> 


-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================

[U-Boot] UBoot running UBoot - is it possible?

[Wolfgang Denk]

Dear Simon,

In message <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com> you wrote:
>
> My idea was to let SPL implement a dedicated boot counter/watchdog
> that detects problems starting U-Boot and using a backup copy if it
> fails multiple times. Of course you need an SPL on your board to do
> this. Plus you end up with the same problem for updating SPL, but I'm
> lucky here that my platform (socfpga gen5) has redundant storage for
> SPL and implements a startup watchdog mechanism for the SPL.

In other words, you are just adding enough additional complexity to
SPL to make it reasonably likely that there will be bugs that need
to be fixed later, i. e. you have to update the SPL.

And then?

You are just moving the problem, not solving it.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
I had the rare misfortune of being one of the first people to try and
implement a PL/1 compiler.                             -- T. Cheatham

[U-Boot] UBoot running UBoot - is it possible?

[Simon Goldschmidt]

On 13.11.2018 17:00, Wolfgang Denk wrote:
> Dear Simon,
>
> In message <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com> you wrote:
>> My idea was to let SPL implement a dedicated boot counter/watchdog
>> that detects problems starting U-Boot and using a backup copy if it
>> fails multiple times. Of course you need an SPL on your board to do
>> this. Plus you end up with the same problem for updating SPL, but I'm
>> lucky here that my platform (socfpga gen5) has redundant storage for
>> SPL and implements a startup watchdog mechanism for the SPL.
> In other words, you are just adding enough additional complexity to
> SPL to make it reasonably likely that there will be bugs that need
> to be fixed later, i. e. you have to update the SPL.
>
> And then?
>
> You are just moving the problem, not solving it.

That's not how I see it. As I see it, I have to implement an upgrade 
option for SPL. This is partly due to bad design of the socfpga_gen5 
platform. But also I have read multiple times on this list that you 
should use SPL from the same version as U-Boot as they might work 
combined and U-Boot might depend on SPL to do things that might change 
over time. So is it really a good idea to upgrade U-Boot without 
upgrading SPL at the same time? It seems to me this would require 
thorough testing of different version mixes...

So given that SPL must be upgradable, how is it more complex to detect 
U-Boot failure from SPL than from U-Boot itself?

Regards,
Simon

[U-Boot] UBoot running UBoot - is it possible?

[Stefano Babic]

On 13/11/18 17:07, Simon Goldschmidt wrote:
> On 13.11.2018 17:00, Wolfgang Denk wrote:
>> Dear Simon,
>>
>> In message
>> <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com>
>> you wrote:
>>> My idea was to let SPL implement a dedicated boot counter/watchdog
>>> that detects problems starting U-Boot and using a backup copy if it
>>> fails multiple times. Of course you need an SPL on your board to do
>>> this. Plus you end up with the same problem for updating SPL, but I'm
>>> lucky here that my platform (socfpga gen5) has redundant storage for
>>> SPL and implements a startup watchdog mechanism for the SPL.
>> In other words, you are just adding enough additional complexity to
>> SPL to make it reasonably likely that there will be bugs that need
>> to be fixed later, i. e. you have to update the SPL.
>>
>> And then?
>>
>> You are just moving the problem, not solving it.
> 
> That's not how I see it. As I see it, I have to implement an upgrade
> option for SPL. This is partly due to bad design of the socfpga_gen5
> platform. But also I have read multiple times on this list that you
> should use SPL from the same version as U-Boot as they might work
> combined and U-Boot might depend on SPL to do things that might change
> over time.

Right. It is not guaranteed that mixing versions works.

> So is it really a good idea to upgrade U-Boot without
> upgrading SPL at the same time?

IMHO it is a bad idea, and it forgets that the bootloader is really SPL
+ u-boot.img else just u-boot.img. It is also questionable if it makes
sense to provide an update mechanism for u-boot when most of critical
parts like clocks, DDR initialisation, etc. are in SPL.

> It seems to me this would require
> thorough testing of different version mixes...
> 

Agree, and this becomes a mess.

> So given that SPL must be upgradable, how is it more complex to detect
> U-Boot failure from SPL than from U-Boot itself?

I do not get the question - if SPL fails, it can be at any time before
you get the control. There sould be a mechanism to switch to a previous
copy of SPL, generally this is not available in hw.

Best regards,
Stefano

-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================

[U-Boot] UBoot running UBoot - is it possible?

[Simon Goldschmidt]

Am Di., 13. Nov. 2018, 17:53 hat Stefano Babic <sbabic@denx.de> geschrieben:

> On 13/11/18 17:07, Simon Goldschmidt wrote:
> > On 13.11.2018 17:00, Wolfgang Denk wrote:
> >> Dear Simon,
> >>
> >> In message
> >> <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com>
> >> you wrote:
> >>> My idea was to let SPL implement a dedicated boot counter/watchdog
> >>> that detects problems starting U-Boot and using a backup copy if it
> >>> fails multiple times. Of course you need an SPL on your board to do
> >>> this. Plus you end up with the same problem for updating SPL, but I'm
> >>> lucky here that my platform (socfpga gen5) has redundant storage for
> >>> SPL and implements a startup watchdog mechanism for the SPL.
> >> In other words, you are just adding enough additional complexity to
> >> SPL to make it reasonably likely that there will be bugs that need
> >> to be fixed later, i. e. you have to update the SPL.
> >>
> >> And then?
> >>
> >> You are just moving the problem, not solving it.
> >
> > That's not how I see it. As I see it, I have to implement an upgrade
> > option for SPL. This is partly due to bad design of the socfpga_gen5
> > platform. But also I have read multiple times on this list that you
> > should use SPL from the same version as U-Boot as they might work
> > combined and U-Boot might depend on SPL to do things that might change
> > over time.
>
> Right. It is not guaranteed that mixing versions works.
>
> > So is it really a good idea to upgrade U-Boot without
> > upgrading SPL at the same time?
>
> IMHO it is a bad idea, and it forgets that the bootloader is really SPL
> + u-boot.img else just u-boot.img. It is also questionable if it makes
> sense to provide an update mechanism for u-boot when most of critical
> parts like clocks, DDR initialisation, etc. are in SPL.
>
> > It seems to me this would require
> > thorough testing of different version mixes...
> >
>
> Agree, and this becomes a mess.
>
> > So given that SPL must be upgradable, how is it more complex to detect
> > U-Boot failure from SPL than from U-Boot itself?
>
> I do not get the question - if SPL fails, it can be at any time before
> you get the control. There sould be a mechanism to switch to a previous
> copy of SPL, generally this is not available in hw.
>

Unless I'm mistaken, the socfpga gen5 bootrom implements a timer and loads
SPL from secondary storage of it does not register as successful after some
time.

That's my arch only, of course...

Simon

[U-Boot] UBoot running UBoot - is it possible?

[Stefano Babic]

Hi Simon,

On 13/11/18 18:43, Simon Goldschmidt wrote:
> 
> 
> Am Di., 13. Nov. 2018, 17:53 hat Stefano Babic <sbabic@denx.de
> <mailto:sbabic@denx.de>> geschrieben:
> 
>     On 13/11/18 17:07, Simon Goldschmidt wrote:
>     > On 13.11.2018 17:00, Wolfgang Denk wrote:
>     >> Dear Simon,
>     >>
>     >> In message
>     >>
>     <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com
>     <mailto:PjmQANsw@mail.gmail.com>>
>     >> you wrote:
>     >>> My idea was to let SPL implement a dedicated boot counter/watchdog
>     >>> that detects problems starting U-Boot and using a backup copy if it
>     >>> fails multiple times. Of course you need an SPL on your board to do
>     >>> this. Plus you end up with the same problem for updating SPL,
>     but I'm
>     >>> lucky here that my platform (socfpga gen5) has redundant storage for
>     >>> SPL and implements a startup watchdog mechanism for the SPL.
>     >> In other words, you are just adding enough additional complexity to
>     >> SPL to make it reasonably likely that there will be bugs that need
>     >> to be fixed later, i. e. you have to update the SPL.
>     >>
>     >> And then?
>     >>
>     >> You are just moving the problem, not solving it.
>     >
>     > That's not how I see it. As I see it, I have to implement an upgrade
>     > option for SPL. This is partly due to bad design of the socfpga_gen5
>     > platform. But also I have read multiple times on this list that you
>     > should use SPL from the same version as U-Boot as they might work
>     > combined and U-Boot might depend on SPL to do things that might change
>     > over time.
> 
>     Right. It is not guaranteed that mixing versions works.
> 
>     > So is it really a good idea to upgrade U-Boot without
>     > upgrading SPL at the same time?
> 
>     IMHO it is a bad idea, and it forgets that the bootloader is really SPL
>     + u-boot.img else just u-boot.img. It is also questionable if it makes
>     sense to provide an update mechanism for u-boot when most of critical
>     parts like clocks, DDR initialisation, etc. are in SPL.
> 
>     > It seems to me this would require
>     > thorough testing of different version mixes...
>     >
> 
>     Agree, and this becomes a mess.
> 
>     > So given that SPL must be upgradable, how is it more complex to detect
>     > U-Boot failure from SPL than from U-Boot itself?
> 
>     I do not get the question - if SPL fails, it can be at any time before
>     you get the control. There sould be a mechanism to switch to a previous
>     copy of SPL, generally this is not available in hw.
> 
> 
> Unless I'm mistaken, the socfpga gen5 bootrom implements a timer and
> loads SPL from secondary storage of it does not register as successful
> after some time.

That means you *have* hardware support, then.

I am not aware there is such as feature on i.MX6.

> 
> That's my arch only, of course...

Regards,
Stefano


-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================

[U-Boot] UBoot running UBoot - is it possible?

[Simon Glass]

Hi Allan,

On 12 November 2018 at 18:38, Allan Chandler <allan.chandler@dti.com.au> wrote:
> Hello, bods.
>
> We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
>
> However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
>
> What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
>
> 1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
>
> 2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
>
> The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
>
> We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
>
> So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
>
> 1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
>             mmcboot=run check_uboot; <rest of original mmcboot>
>             other_uboot=0
>
> 2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
>
> 3/ We defined check_uboot as:
>             if test ${other_uboot} -eq 1; then
>                         setenv other_uboot 0
>                         saveenv
>                         ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
>                         go 0x17800000
>             fi
> The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
>
> 4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
>
> Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
>
> I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.

You may need to flush the cache area that you read into, or use
'dcache off' before jumping to the second U-Boot.

>
> So I guess my questions are as follows:
>
> a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?
> b/ Does anyone have any ideas on how I could achieve this?
>
> I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.

Chromium OS and Android verified boot (which is in mainline - see avb)
use an A/B system. You can have SPL chosen whether to boot from A or
B.

I'm actually working on integrating Chromium OS vboot into U-Boot
again. It uses TPL to select the SPL to use, which then boots into
UBoot proper. This is so we can update the SDRAM code. Am hoping to
have something going in January although a lot of the required patches
have been sent.

>
> Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.

You really need to sign it so you can send updated U-Boot versions and
the device in the field can verified the U-Boot image in the field.

Regards,
Simon

>
> Cheers,
> Pax
>
> Allan Chandler | Software Engineer
> DTI Group Ltd | Integrated Transit Technologies
> 31 Affleck Road, Perth Airport, WA 6105, Australia
> P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler@dti.com.au>
> Visit our website www.dti.com.au<http://www.dti.com.au>
> The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

[U-Boot] UBoot running UBoot - is it possible?

[Simon Goldschmidt]

On 13.11.2018 20:53, Simon Glass wrote:
> Hi Allan,
>
> On 12 November 2018 at 18:38, Allan Chandler <allan.chandler@dti.com.au> wrote:
>> Hello, bods.
>>
>> We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
>>
>> However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
>>
>> What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
>>
>> 1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
>>
>> 2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
>>
>> The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
>>
>> We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
>>
>> So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
>>
>> 1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
>>              mmcboot=run check_uboot; <rest of original mmcboot>
>>              other_uboot=0
>>
>> 2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
>>
>> 3/ We defined check_uboot as:
>>              if test ${other_uboot} -eq 1; then
>>                          setenv other_uboot 0
>>                          saveenv
>>                          ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
>>                          go 0x17800000
>>              fi
>> The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
>>
>> 4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
>>
>> Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
>>
>> I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
> You may need to flush the cache area that you read into, or use
> 'dcache off' before jumping to the second U-Boot.
>
>> So I guess my questions are as follows:
>>
>> a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?
>> b/ Does anyone have any ideas on how I could achieve this?
>>
>> I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
> Chromium OS and Android verified boot (which is in mainline - see avb)
> use an A/B system. You can have SPL chosen whether to boot from A or
> B.
>
> I'm actually working on integrating Chromium OS vboot into U-Boot
> again. It uses TPL to select the SPL to use, which then boots into
> UBoot proper. This is so we can update the SDRAM code. Am hoping to
> have something going in January although a lot of the required patches
> have been sent.

That sounds interesting, but would it be safe to use TPL from 2019 and 
letting it boot a, say, 2025 SPL + U-Boot? I don't have experience with 
TPL, can it check signatures for secure boot? I'd suspect Chromium OS 
would need that?

Simon

>
>> Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
> You really need to sign it so you can send updated U-Boot versions and
> the device in the field can verified the U-Boot image in the field.
>
> Regards,
> Simon
>
>> Cheers,
>> Pax
>>
>> Allan Chandler | Software Engineer
>> DTI Group Ltd | Integrated Transit Technologies
>> 31 Affleck Road, Perth Airport, WA 6105, Australia
>> P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler@dti.com.au>
>> Visit our website www.dti.com.au<http://www.dti.com.au>
>> The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
>> _______________________________________________
>> U-Boot mailing list
>> U-Boot at lists.denx.de
>> https://lists.denx.de/listinfo/u-boot
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

[U-Boot] UBoot running UBoot - is it possible?

[Simon Glass]

Hi Simon,

On 13 November 2018 at 12:17, Simon Goldschmidt
<simon.k.r.goldschmidt@gmail.com> wrote:
> On 13.11.2018 20:53, Simon Glass wrote:
>>
>> Hi Allan,
>>
>> On 12 November 2018 at 18:38, Allan Chandler <allan.chandler@dti.com.au>
>> wrote:
>>>
>>> Hello, bods.
>>>
>>> We're trying to architect a relatively safe solution for updating UBoot
>>> in the field. What we have at the moment is an iMX6-based board with two
>>> UBoot partitions and two system partitions but we only use the first UBoot
>>> one. Switching system partitions is covered well by UBoot since we use what
>>> seems like a fairly standard method involving upgrade_available, bootlimit
>>> and bootcount variables, along with bootcmd/altbootcmd scripts to try and
>>> load a new system partition while falling back if it fails.
>>>
>>> However, I'm wondering how people handle the need to update UBoot itself.
>>> Although this will happen FAR less often than normal software updates, these
>>> product have a multi-decade lifespan and it's hard to imagine we'll get
>>> through that without some update being needed. Especially since we need one
>>> now (three years in) due to a bug (with our added stuff, not UBoot
>>> baseline). With the iMX6 boards we have, once we tell it to start using the
>>> second UBoot partition, there's no UBoot/watchdog combo that will revert
>>> that change and reprogramming requires a rather expensive RTB (return to
>>> base) to fix via the serial interface.
>>>
>>> What we had hoped to do was to be able to soft-boot an alternate UBoot
>>> (i.e., without first telling the iMX6 board to commit to the change). The
>>> scenario would go like this:
>>>
>>> 1/ Have an installer package (it runs under control of the system
>>> partition) that just writes the new UBoot image into the alternate UBoot
>>> partition then soft-boots it somehow (so now running same system partition
>>> but started from the new UBoot image).
>>>
>>> 2/ While running from the system partition started from that alternate
>>> UBoot partition, have an updater package that tells the iMX6 board to commit
>>> to the changeover. This updater package would only run if it detected that
>>> the bootable UBoot and currently-used UBoot were different.
>>>
>>> The advantage of this is that, unless the new UBoot is FULLY capable of
>>> running our system partition (and also running an installer from there), no
>>> commit is possible, hence a simple power cycle would return to the previous
>>> working state.
>>>
>>> We originally tried kexec from the system partition but it seemed to want
>>> to run a Linux kernel rather than loading and executing some arbitrary boot
>>> code.
>>>
>>> So we then turned to the UBoot scripts themselves and thought we'd found
>>> a way we could do it.
>>>
>>> 1/ We changed the mmcboot script by prefixing a special check and
>>> introduced a variable for handling soft-boot:
>>>              mmcboot=run check_uboot; <rest of original mmcboot>
>>>              other_uboot=0
>>>
>>> 2/ We extracted the boot image uboot.bootimg from the IMX file by
>>> stripping off the first 0xc00 bytes (we had to put this into the /boot file
>>> system since I don't yet know how to get at raw partition data from UBoot
>>> scripts.
>>>
>>> 3/ We defined check_uboot as:
>>>              if test ${other_uboot} -eq 1; then
>>>                          setenv other_uboot 0
>>>                          saveenv
>>>                          ext4load mmc ${mmcdev}:${mmcpart} 0x17800000
>>> /boot/uboot.bootimg
>>>                          go 0x17800000
>>>              fi
>>> The way this should work is that, if alternate boot is flagged, it
>>> immediately unflags it (for recovery if the alternate fails) then loads and
>>> executes the other UBoot image. If it's not flagged, check_uboot will return
>>> without trying to soft-boot the alternate.
>>>
>>> 4/ After installing the new UBoot to the alternate partition, we set a
>>> UBoot variable (other_uboot) to 1 and rebooted.
>>>
>>> Now this seemed to work inasmuch as the alternate UBoot program started
>>> pumping out console messages but, unfortunately, it seemed to hang partway
>>> through the boot process.
>>>
>>> I suspect this is because, having already been through a portion of that
>>> boot process in the primary UBoot, it's not keen on having to do it again.
>>
>> You may need to flush the cache area that you read into, or use
>> 'dcache off' before jumping to the second U-Boot.
>>
>>> So I guess my questions are as follows:
>>>
>>> a/ How do people currently handle (if they do) the requirement that UBoot
>>> be safely updatable in the field?
>>> b/ Does anyone have any ideas on how I could achieve this?
>>>
>>> I've been told that Google does something like this for Android booting
>>> but had to heavily modify UBoot to do it. I haven't yet investigated this
>>> possibility.
>>
>> Chromium OS and Android verified boot (which is in mainline - see avb)
>> use an A/B system. You can have SPL chosen whether to boot from A or
>> B.
>>
>> I'm actually working on integrating Chromium OS vboot into U-Boot
>> again. It uses TPL to select the SPL to use, which then boots into
>> UBoot proper. This is so we can update the SDRAM code. Am hoping to
>> have something going in January although a lot of the required patches
>> have been sent.
>
>
> That sounds interesting, but would it be safe to use TPL from 2019 and
> letting it boot a, say, 2025 SPL + U-Boot? I don't have experience with TPL,
> can it check signatures for secure boot? I'd suspect Chromium OS would need
> that?

This needs to be tested with each build.

The bloblist feature is designed to handle this, although it is update
to individual users to include version info at present. It should land
in the next release.

Regards,
Simon