Build the "boot" partition image (just like the rootfs)

Build the "boot" partition image (just like the rootfs)

[Mike Looijmans]

For software updates, I want to have what would have gone into the boot 
partition of the WIC image as a separate file.

If I want to have the contents of the rootfs as an ext4 image, I can 
just specify IMAGE_TYPES="ext4" in my image recipe.

This image I can feed to SWUdate and write to the rootfs storage.

But I also want to be able to update the boot partition (for example, 
the raspberrypi has the annoying property that devicetree and kernel 
reside here).

If I create a WIC image, the boot partition is in there with the proper 
files (from IMAGE_BOOT_FILES) so I would really like to re-use that 
code. I could create the wic image and then cut out the part I want, but 
that doesn't seem particularly nice.

What I want to do at update time is to write the new boot partition to 
another location on disk, and then adjust the partition table to make 
the first partition entry point to the new copy. That way, in case of 
unexpected failure (power loss for example), the device remains bootable.


-- 
Mike Looijmans


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: [OE-core] Build the "boot" partition image (just like the rootfs)

[Stefano Babic]

Hi Mike,

On 15.03.21 14:13, Mike Looijmans wrote:
> For software updates, I want to have what would have gone into the boot 
> partition of the WIC image as a separate file.
> 
> If I want to have the contents of the rootfs as an ext4 image, I can 
> just specify IMAGE_TYPES="ext4" in my image recipe.
>  > This image I can feed to SWUdate and write to the rootfs storage.
> 
> But I also want to be able to update the boot partition (for example, 
> the raspberrypi has the annoying property that devicetree and kernel 
> reside here).

This is very annoying, but you could also get rid of it. You can install 
kernel and device tree in your rootfs (then they are located in /boot as 
usual), and you switch to U-Boot ( RPI_USE_U_BOOT = "1"). The 
proprietary bootloader will start U-Boot instead of kernel, and in 
u-boot you can load kernel and device tree from your rootfs.

> 
> If I create a WIC image, the boot partition is in there with the proper 
> files (from IMAGE_BOOT_FILES) so I would really like to re-use that 
> code. I could create the wic image and then cut out the part I want, but 
> that doesn't seem particularly nice.

You can add vfat support for IMAGE_FSTYPES, and then you can build an 
image (you can just take the files you have in IMAGE_BOOT_FILES).

> 
> What I want to do at update time is to write the new boot partition to 
> another location on disk, and then adjust the partition table to make 
> the first partition entry point to the new copy. That way, in case of 
> unexpected failure (power loss for example), the device remains bootable.

This does not seem to be atomic. It remains the risk that partition 
table gets corrupted and then even the first bootloader cannot run. If 
you want to have a power-cut safe way to update is not enough.

Best regards,
Stefano Babic

-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
=====================================================================

Re: [OE-core] Build the "boot" partition image (just like the rootfs)

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 15-03-2021 14:47, Stefano Babic wrote:
> Hi Mike,
>
> On 15.03.21 14:13, Mike Looijmans wrote:
>> For software updates, I want to have what would have gone into the 
>> boot partition of the WIC image as a separate file.
>>
>> If I want to have the contents of the rootfs as an ext4 image, I can 
>> just specify IMAGE_TYPES="ext4" in my image recipe.
>>  > This image I can feed to SWUdate and write to the rootfs storage.
>>
>> But I also want to be able to update the boot partition (for example, 
>> the raspberrypi has the annoying property that devicetree and kernel 
>> reside here).
>
> This is very annoying, but you could also get rid of it. You can 
> install kernel and device tree in your rootfs (then they are located 
> in /boot as usual), and you switch to U-Boot ( RPI_USE_U_BOOT = "1"). 
> The proprietary bootloader will start U-Boot instead of kernel, and in 
> u-boot you can load kernel and device tree from your rootfs.

That's what I did on other boards, but for the RPi that's not enough. 
There's also firmware there, which interacts with the kernel, and the 
firmware is altering the devicetree too. It's pretty likely that the 
firmware needs an update too when the kernel gets a big update.


>
>>
>> If I create a WIC image, the boot partition is in there with the 
>> proper files (from IMAGE_BOOT_FILES) so I would really like to re-use 
>> that code. I could create the wic image and then cut out the part I 
>> want, but that doesn't seem particularly nice.
>
> You can add vfat support for IMAGE_FSTYPES, and then you can build an 
> image (you can just take the files you have in IMAGE_BOOT_FILES).

But that's *exactly* my problem: How do I get these IMAGE_BOOT_FILES 
into an image of sorts? Just setting IMAGE_FSTYPE="vfat" will put the 
rootfs contents in there, not the bootfiles I'm after.

>
>>
>> What I want to do at update time is to write the new boot partition 
>> to another location on disk, and then adjust the partition table to 
>> make the first partition entry point to the new copy. That way, in 
>> case of unexpected failure (power loss for example), the device 
>> remains bootable.
>
> This does not seem to be atomic. It remains the risk that partition 
> table gets corrupted and then even the first bootloader cannot run. If 
> you want to have a power-cut safe way to update is not enough.

It's not perfect but it's close enough. The partition table resides in a 
single sector, so chances of power-out at exactly this moment are small 
enough that I'm willing to take my chances here.

My experience is that there's more chance of the SD card completely 
dying because of power-out during some internal mumbojumbo than 
corrupting that partition table. Against that SD failure there's nothing 
I can do...


-- 
Mike Looijmans

Re: [OE-core] Build the "boot" partition image (just like the rootfs)

[Stefano Babic]

Hi Mike,

On 15.03.21 15:46, Mike Looijmans wrote:
> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
> 
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> On 15-03-2021 14:47, Stefano Babic wrote:
>> Hi Mike,
>>
>> On 15.03.21 14:13, Mike Looijmans wrote:
>>> For software updates, I want to have what would have gone into the 
>>> boot partition of the WIC image as a separate file.
>>>
>>> If I want to have the contents of the rootfs as an ext4 image, I can 
>>> just specify IMAGE_TYPES="ext4" in my image recipe.
>>>  > This image I can feed to SWUdate and write to the rootfs storage.
>>>
>>> But I also want to be able to update the boot partition (for example, 
>>> the raspberrypi has the annoying property that devicetree and kernel 
>>> reside here).
>>
>> This is very annoying, but you could also get rid of it. You can 
>> install kernel and device tree in your rootfs (then they are located 
>> in /boot as usual), and you switch to U-Boot ( RPI_USE_U_BOOT = "1"). 
>> The proprietary bootloader will start U-Boot instead of kernel, and in 
>> u-boot you can load kernel and device tree from your rootfs.
> 
> That's what I did on other boards, but for the RPi that's not enough. 
> There's also firmware there, which interacts with the kernel, and the 
> firmware is altering the devicetree too. It's pretty likely that the 
> firmware needs an update too when the kernel gets a big update.
> 
> 
>>
>>>
>>> If I create a WIC image, the boot partition is in there with the 
>>> proper files (from IMAGE_BOOT_FILES) so I would really like to re-use 
>>> that code. I could create the wic image and then cut out the part I 
>>> want, but that doesn't seem particularly nice.
>>
>> You can add vfat support for IMAGE_FSTYPES, and then you can build an 
>> image (you can just take the files you have in IMAGE_BOOT_FILES).
> 
> But that's *exactly* my problem: How do I get these IMAGE_BOOT_FILES 
> into an image of sorts? Just setting IMAGE_FSTYPE="vfat" will put the 
> rootfs contents in there, not the bootfiles I'm after.

rpi-bootfiles inherit nopackage, so you should write a recipe that 
instead of simply deploying the files as this recipe does, install them. 
You should then install it into /, as this is where they are put on the 
VFAT partition. If they are in the package, you can write an image 
recipe to install it via IMAGE_INSTALL, and via post process command you 
drop what is not necessary (/etc,...).

Anyway, as far as I know, an IMAGE_FSTYPES += "vfat" is not supported, 
you have also to add support for it.

> 
>>
>>>
>>> What I want to do at update time is to write the new boot partition 
>>> to another location on disk, and then adjust the partition table to 
>>> make the first partition entry point to the new copy. That way, in 
>>> case of unexpected failure (power loss for example), the device 
>>> remains bootable.
>>
>> This does not seem to be atomic. It remains the risk that partition 
>> table gets corrupted and then even the first bootloader cannot run. If 
>> you want to have a power-cut safe way to update is not enough.
> 
> It's not perfect but it's close enough. The partition table resides in a 
> single sector, so chances of power-out at exactly this moment are small 
> enough that I'm willing to take my chances here.
> 
> My experience is that there's more chance of the SD card completely 
> dying because of power-out during some internal mumbojumbo than 
> corrupting that partition table.

True, that is. So simply the hardware does not allow a safe update, 
point. Fully agree. Or use a very expensive industrial grade SD, but any 
board with eMMC is then cheaper.

> Against that SD failure there's nothing 
> I can do...

Absolutely true.

Best regards,
Stefano Babic


-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
=====================================================================

Re: [OE-core] Build the "boot" partition image (just like the rootfs)

[Mike Looijmans]

Hi Stefano,


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 15-03-2021 15:58, Stefano Babic wrote:
> Hi Mike,
>
> On 15.03.21 15:46, Mike Looijmans wrote:
>>
>> On 15-03-2021 14:47, Stefano Babic wrote:
>>> Hi Mike,
>>>
>>> On 15.03.21 14:13, Mike Looijmans wrote:
>>>> For software updates, I want to have what would have gone into the 
>>>> boot partition of the WIC image as a separate file.
>>>>
>>>> If I want to have the contents of the rootfs as an ext4 image, I 
>>>> can just specify IMAGE_TYPES="ext4" in my image recipe.
>>>>  > This image I can feed to SWUdate and write to the rootfs storage.
>>>>
>>>> But I also want to be able to update the boot partition (for 
>>>> example, the raspberrypi has the annoying property that devicetree 
>>>> and kernel reside here).
>>>
>>> This is very annoying, but you could also get rid of it. You can 
>>> install kernel and device tree in your rootfs (then they are located 
>>> in /boot as usual), and you switch to U-Boot ( RPI_USE_U_BOOT = 
>>> "1"). The proprietary bootloader will start U-Boot instead of 
>>> kernel, and in u-boot you can load kernel and device tree from your 
>>> rootfs.
>>
>> That's what I did on other boards, but for the RPi that's not enough. 
>> There's also firmware there, which interacts with the kernel, and the 
>> firmware is altering the devicetree too. It's pretty likely that the 
>> firmware needs an update too when the kernel gets a big update.
>>
>>
>>>
>>>>
>>>> If I create a WIC image, the boot partition is in there with the 
>>>> proper files (from IMAGE_BOOT_FILES) so I would really like to 
>>>> re-use that code. I could create the wic image and then cut out the 
>>>> part I want, but that doesn't seem particularly nice.
>>>
>>> You can add vfat support for IMAGE_FSTYPES, and then you can build 
>>> an image (you can just take the files you have in IMAGE_BOOT_FILES).
>>
>> But that's *exactly* my problem: How do I get these IMAGE_BOOT_FILES 
>> into an image of sorts? Just setting IMAGE_FSTYPE="vfat" will put the 
>> rootfs contents in there, not the bootfiles I'm after.
>
> rpi-bootfiles inherit nopackage, so you should write a recipe that 
> instead of simply deploying the files as this recipe does, install 
> them. You should then install it into /, as this is where they are put 
> on the VFAT partition. If they are in the package, you can write an 
> image recipe to install it via IMAGE_INSTALL, and via post process 
> command you drop what is not necessary (/etc,...).
>
> Anyway, as far as I know, an IMAGE_FSTYPES += "vfat" is not supported, 
> you have also to add support for it.

rpi-bootfiles provides only part of the file set. The issue is that 
IMAGE_BOOT_FILES, which lists much more than what rpi-bootfiles 
provides, can be modified in the image recipe itself. It contains the 
kernel, devicetrees and configuration files as well.

>
>>
>>>
>>>>
>>>> What I want to do at update time is to write the new boot partition 
>>>> to another location on disk, and then adjust the partition table to 
>>>> make the first partition entry point to the new copy. That way, in 
>>>> case of unexpected failure (power loss for example), the device 
>>>> remains bootable.
>>>
>>> This does not seem to be atomic. It remains the risk that partition 
>>> table gets corrupted and then even the first bootloader cannot run. 
>>> If you want to have a power-cut safe way to update is not enough.
>>
>> It's not perfect but it's close enough. The partition table resides 
>> in a single sector, so chances of power-out at exactly this moment 
>> are small enough that I'm willing to take my chances here.
>>
>> My experience is that there's more chance of the SD card completely 
>> dying because of power-out during some internal mumbojumbo than 
>> corrupting that partition table.
>
> True, that is. So simply the hardware does not allow a safe update, 
> point. Fully agree. Or use a very expensive industrial grade SD, but 
> any board with eMMC is then cheaper.

I actually tried a few industrial grade SDs (usually SLC type). I had 
issues with those on other boards and the Pi is no exception. If I put 
one of those expensive SLC cards into the SD slot of the RPi4, there's 
about 30% chance that the board won't boot at all.

>
>> Against that SD failure there's nothing I can do...
>
> Absolutely true.

Amen.

I'm going to let this thing rest for a while, maybe I'll come up with 
something completely different or maybe someone in this group has some 
brilliant idea...


-- 
Mike Looijmans

Re: [OE-core] Build the "boot" partition image (just like the rootfs)

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 15-03-2021 16:17, Mike Looijmans via lists.openembedded.org wrote:
>>>>>
>>>>> If I create a WIC image, the boot partition is in there with the proper 
>>>>> files (from IMAGE_BOOT_FILES) so I would really like to re-use that code. 
>>>>> I could create the wic image and then cut out the part I want, but that 
>>>>> doesn't seem particularly nice.
>>>>
>>>> You can add vfat support for IMAGE_FSTYPES, and then you can build an 
>>>> image (you can just take the files you have in IMAGE_BOOT_FILES).
>>>
>>> But that's *exactly* my problem: How do I get these IMAGE_BOOT_FILES into 
>>> an image of sorts? Just setting IMAGE_FSTYPE="vfat" will put the rootfs 
>>> contents in there, not the bootfiles I'm after.
>>
>> rpi-bootfiles inherit nopackage, so you should write a recipe that instead 
>> of simply deploying the files as this recipe does, install them. You should 
>> then install it into /, as this is where they are put on the VFAT partition. 
>> If they are in the package, you can write an image recipe to install it via 
>> IMAGE_INSTALL, and via post process command you drop what is not necessary 
>> (/etc,...).
>>
>> Anyway, as far as I know, an IMAGE_FSTYPES += "vfat" is not supported, you 
>> have also to add support for it.
> 
> rpi-bootfiles provides only part of the file set. The issue is that 
> IMAGE_BOOT_FILES, which lists much more than what rpi-bootfiles provides, can 
> be modified in the image recipe itself. It contains the kernel, devicetrees 
> and configuration files as well.

What I did for now was just insert a task before swuimage that depends on the 
wic image, and just extracts the FAT partition using "dd". I'm not 
particularly proud of that, but it works/