RFC: Reference updater filesystem

RFC: Reference updater filesystem

[Mariano Lopez]

There has been interest in an image based software updater in Yocto 
Project. The proposed solution for a image based updater is to use 
Stefano Babic's software updater (http://sbabic.github.io/swupdate). 
This software do a binary copy, so it is needed to have at least two 
partitions, these partitions would be the rootfs and the maintenance 
partition. The rootfs it's the main partition used to boot during the 
normal device operation, on the other hand, the maintenance will be used 
to update the main partition.

To update the system, the user has to connect to device and boot in the 
maintenance partition; once in the maintenance partition the software 
updater will copy the new image in the rootfs partition. A final reboot 
into the rootfs it is necessary to complete the upgrade.

As mentioned before the the software will copy an image to the 
partition, so everything in that partition will be wiped out, including 
custom configurations. To avoid the loss of configuration I explore 
three different solutions:
1. Use a separate partition for the configuration.
   a. The pro of this method is the partition is not touched during the 
update.
   b. The con of this method is the configuration is not directly in 
rootfs (example: /etc).

2. Do the backup during the update.
   a. The pro is the configuration is directly in rootfs.
   b. The con is If the update fail most likely the configuration would 
be lost.

3. Have an OverlayFS for the rootfs or the partition that have the 
configuration.
   a. The pro is the configuration is  "directly" in rootfs.
   b. The con is there is need to provide a custom init to guarantee the 
Overlay is mounted before the boot process.

With the above information I'm proposing to use a separate partition for 
the configuration; this is because is more reliable and doesn't require 
big changes in the current architecture.

So, the idea is to have 4 partitions in the media:
1. boot. This is the usual boot partition
2. data. This will hold the configuration files. Not modified by updates.
3. maintenance. This partition will be used to update rootfs.
4. rootfs. Partition used for normal operation.

Re: RFC: Reference updater filesystem

[Jens Rehsack]

> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
> 
> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
> 
> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
> 
> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
> 1. Use a separate partition for the configuration.
>  a. The pro of this method is the partition is not touched during the update.
>  b. The con of this method is the configuration is not directly in rootfs (example: /etc).
> 
> 2. Do the backup during the update.
>  a. The pro is the configuration is directly in rootfs.
>  b. The con is If the update fail most likely the configuration would be lost.
> 
> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>  a. The pro is the configuration is  "directly" in rootfs.
>  b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
> 
> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
> 
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.

That's what we currently have implemented and running in field for a while with a small difference:

1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
3) We use

boot
rootfs
maintfs
data

This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs

4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay

Feel free to use that solution if you want.

Cheers
-- 
Jens Rehsack - rehsack@gmail.com

Re: [yocto] RFC: Reference updater filesystem

[Jens Rehsack]

> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
> 
> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
> 
> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
> 
> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
> 1. Use a separate partition for the configuration.
>  a. The pro of this method is the partition is not touched during the update.
>  b. The con of this method is the configuration is not directly in rootfs (example: /etc).
> 
> 2. Do the backup during the update.
>  a. The pro is the configuration is directly in rootfs.
>  b. The con is If the update fail most likely the configuration would be lost.
> 
> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>  a. The pro is the configuration is  "directly" in rootfs.
>  b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
> 
> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
> 
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.

That's what we currently have implemented and running in field for a while with a small difference:

1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
3) We use

boot
rootfs
maintfs
data

This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs

4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay

Feel free to use that solution if you want.

Cheers
-- 
Jens Rehsack - rehsack@gmail.com

Re: RFC: Reference updater filesystem

[Daniel.]

Hi,

Hey Jen, I was looking for an image upgrade solution and factory reset
solution using overlayfs. The idea have two partitions one read-only
with the factory image, other to hold the changes that were made by
time. The factory reset feature should be triggered by a hided button
that can be pressed with help of a clips. I was thinking in using an
init ram disk to wipe out the rw partition, making the rootfs clean as
after an image installation. The upgrader tool shold re-flash a new
image to rootfs. Old rootfs is lost. The configuration changes that
have been holded by overlayfs should be wiped-out too, I didn't think
about that, is something to take in account.

Are you using overlayfs? How is it going? What difficulties you have found?

Other solution whould be using Smart package manager to upgrade the
rootfs, but this doesn't attend my need for factory-reset.

Please tell me more about your experience with overlayfs :)

Best regards,
- dhs

2015-11-30 14:10 GMT-02:00 Jens Rehsack <rehsack@gmail.com>:
>
>> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
>>
>> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
>>
>> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
>>
>> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
>> 1. Use a separate partition for the configuration.
>>  a. The pro of this method is the partition is not touched during the update.
>>  b. The con of this method is the configuration is not directly in rootfs (example: /etc).
>>
>> 2. Do the backup during the update.
>>  a. The pro is the configuration is directly in rootfs.
>>  b. The con is If the update fail most likely the configuration would be lost.
>>
>> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>>  a. The pro is the configuration is  "directly" in rootfs.
>>  b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
>>
>> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
>>
>> So, the idea is to have 4 partitions in the media:
>> 1. boot. This is the usual boot partition
>> 2. data. This will hold the configuration files. Not modified by updates.
>> 3. maintenance. This partition will be used to update rootfs.
>> 4. rootfs. Partition used for normal operation.
>
> That's what we currently have implemented and running in field for a while with a small difference:
>
> 1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
> 2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
> 3) We use
>
> boot
> rootfs
> maintfs
> data
>
> This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs
>
> 4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay
>
> Feel free to use that solution if you want.
>
> Cheers
> --
> Jens Rehsack - rehsack@gmail.com
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



-- 
"Do or do not. There is no try"
  Yoda Master

Re: RFC: Reference updater filesystem

[Jens Rehsack]

> 2015-11-30 14:10 GMT-02:00 Jens Rehsack <rehsack@gmail.com>:
>> 
>>> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
>>> 
>>> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
>>> 
>>> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
>>> 
>>> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
>>> 1. Use a separate partition for the configuration.
>>> a. The pro of this method is the partition is not touched during the update.
>>> b. The con of this method is the configuration is not directly in rootfs (example: /etc).
>>> 
>>> 2. Do the backup during the update.
>>> a. The pro is the configuration is directly in rootfs.
>>> b. The con is If the update fail most likely the configuration would be lost.
>>> 
>>> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>>> a. The pro is the configuration is  "directly" in rootfs.
>>> b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
>>> 
>>> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
>>> 
>>> So, the idea is to have 4 partitions in the media:
>>> 1. boot. This is the usual boot partition
>>> 2. data. This will hold the configuration files. Not modified by updates.
>>> 3. maintenance. This partition will be used to update rootfs.
>>> 4. rootfs. Partition used for normal operation.
>> 
>> That's what we currently have implemented and running in field for a while with a small difference:
>> 
>> 1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
>> 2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
>> 3) We use
>> 
>> boot
>> rootfs
>> maintfs
>> data
>> 
>> This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs
>> 
>> 4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay
>> 
>> Feel free to use that solution if you want.

> Am 30.11.2015 um 17:54 schrieb Daniel. <danielhilst@gmail.com>:
> 
> Hi,
> 
> Hey Jen, I was looking for an image upgrade solution and factory reset
> solution using overlayfs. The idea have two partitions one read-only
> with the factory image, other to hold the changes that were made by
> time. The factory reset feature should be triggered by a hided button
> that can be pressed with help of a clips. I was thinking in using an
> init ram disk to wipe out the rw partition, making the rootfs clean as
> after an image installation. The upgrader tool shold re-flash a new
> image to rootfs. Old rootfs is lost. The configuration changes that
> have been holded by overlayfs should be wiped-out too, I didn't think
> about that, is something to take in account.
> 
> Are you using overlayfs? How is it going? What difficulties you have found?

Yes, we do. All difficulties we'd found are solved in referred initoverlay
recipe. Maybe one fine I write a blog post regarding that topic ;)

> Other solution whould be using Smart package manager to upgrade the
> rootfs, but this doesn't attend my need for factory-reset.

How does that fit into an ro rootfs?

> Please tell me more about your experience with overlayfs :)

It's more stable than unionfs. There was little effort updating
systems from 3.10 with overlay patch to 4.1 with overlay builtin
kernel (work dirs must be created - https://github.com/rdm-dev/meta-jens/blob/jethro/recipes-core/initoverlay/initoverlay/migrate2overlay.sh#L28-L30)

Cheers
-- 
Jens Rehsack - rehsack@gmail.com

Re: [yocto] RFC: Reference updater filesystem

[Jens Rehsack]

> 2015-11-30 14:10 GMT-02:00 Jens Rehsack <rehsack@gmail.com>:
>> 
>>> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
>>> 
>>> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
>>> 
>>> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
>>> 
>>> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
>>> 1. Use a separate partition for the configuration.
>>> a. The pro of this method is the partition is not touched during the update.
>>> b. The con of this method is the configuration is not directly in rootfs (example: /etc).
>>> 
>>> 2. Do the backup during the update.
>>> a. The pro is the configuration is directly in rootfs.
>>> b. The con is If the update fail most likely the configuration would be lost.
>>> 
>>> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>>> a. The pro is the configuration is  "directly" in rootfs.
>>> b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
>>> 
>>> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
>>> 
>>> So, the idea is to have 4 partitions in the media:
>>> 1. boot. This is the usual boot partition
>>> 2. data. This will hold the configuration files. Not modified by updates.
>>> 3. maintenance. This partition will be used to update rootfs.
>>> 4. rootfs. Partition used for normal operation.
>> 
>> That's what we currently have implemented and running in field for a while with a small difference:
>> 
>> 1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
>> 2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
>> 3) We use
>> 
>> boot
>> rootfs
>> maintfs
>> data
>> 
>> This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs
>> 
>> 4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay
>> 
>> Feel free to use that solution if you want.

> Am 30.11.2015 um 17:54 schrieb Daniel. <danielhilst@gmail.com>:
> 
> Hi,
> 
> Hey Jen, I was looking for an image upgrade solution and factory reset
> solution using overlayfs. The idea have two partitions one read-only
> with the factory image, other to hold the changes that were made by
> time. The factory reset feature should be triggered by a hided button
> that can be pressed with help of a clips. I was thinking in using an
> init ram disk to wipe out the rw partition, making the rootfs clean as
> after an image installation. The upgrader tool shold re-flash a new
> image to rootfs. Old rootfs is lost. The configuration changes that
> have been holded by overlayfs should be wiped-out too, I didn't think
> about that, is something to take in account.
> 
> Are you using overlayfs? How is it going? What difficulties you have found?

Yes, we do. All difficulties we'd found are solved in referred initoverlay
recipe. Maybe one fine I write a blog post regarding that topic ;)

> Other solution whould be using Smart package manager to upgrade the
> rootfs, but this doesn't attend my need for factory-reset.

How does that fit into an ro rootfs?

> Please tell me more about your experience with overlayfs :)

It's more stable than unionfs. There was little effort updating
systems from 3.10 with overlay patch to 4.1 with overlay builtin
kernel (work dirs must be created - https://github.com/rdm-dev/meta-jens/blob/jethro/recipes-core/initoverlay/initoverlay/migrate2overlay.sh#L28-L30)

Cheers
-- 
Jens Rehsack - rehsack@gmail.com

Re: RFC: Reference updater filesystem

[Daniel.]

2015-11-30 16:20 GMT-02:00 Jens Rehsack <rehsack@gmail.com>:
>> 2015-11-30 14:10 GMT-02:00 Jens Rehsack <rehsack@gmail.com>:
>>>
>>>> Am 23.11.2015 um 22:15 schrieb Mariano Lopez <mariano.lopez@intel.com>:
>>>>
>>>> There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition.
>>>>
>>>> To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade.
>>>>
>>>> As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions:
>>>> 1. Use a separate partition for the configuration.
>>>> a. The pro of this method is the partition is not touched during the update.
>>>> b. The con of this method is the configuration is not directly in rootfs (example: /etc).
>>>>
>>>> 2. Do the backup during the update.
>>>> a. The pro is the configuration is directly in rootfs.
>>>> b. The con is If the update fail most likely the configuration would be lost.
>>>>
>>>> 3. Have an OverlayFS for the rootfs or the partition that have the configuration.
>>>> a. The pro is the configuration is  "directly" in rootfs.
>>>> b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process.
>>>>
>>>> With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture.
>>>>
>>>> So, the idea is to have 4 partitions in the media:
>>>> 1. boot. This is the usual boot partition
>>>> 2. data. This will hold the configuration files. Not modified by updates.
>>>> 3. maintenance. This partition will be used to update rootfs.
>>>> 4. rootfs. Partition used for normal operation.
>>>
>>> That's what we currently have implemented and running in field for a while with a small difference:
>>>
>>> 1) We don't use Stefano Babic's software updater, but an own script which deals with initial software flash and later update similar - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/prd
>>> 2) We have integrated the updater with an update-service which can download the new image and install based on a manifest (signature support comes with next update) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-rdm/system-image // http://www.netbsd.org/~sno/talks/nrpm/Moo-at-System-Image-Update.pdf
>>> 3) We use
>>>
>>> boot
>>> rootfs
>>> maintfs
>>> data
>>>
>>> This layout allows us to extend data to fit the entire storage with know sizes for boot, rootfs and maintfs
>>>
>>> 4) Overlayfs with all serices is implemented (Update-wise, when coming from 3.10 to 3.18 or coming from 3.0 with unionfs to overlay ...) - https://github.com/rdm-dev/meta-jens/tree/jethro/recipes-core/initoverlay
>>>
>>> Feel free to use that solution if you want.
>
>> Am 30.11.2015 um 17:54 schrieb Daniel. <danielhilst@gmail.com>:
>>
>> Hi,
>>
>> Hey Jen, I was looking for an image upgrade solution and factory reset
>> solution using overlayfs. The idea have two partitions one read-only
>> with the factory image, other to hold the changes that were made by
>> time. The factory reset feature should be triggered by a hided button
>> that can be pressed with help of a clips. I was thinking in using an
>> init ram disk to wipe out the rw partition, making the rootfs clean as
>> after an image installation. The upgrader tool shold re-flash a new
>> image to rootfs. Old rootfs is lost. The configuration changes that
>> have been holded by overlayfs should be wiped-out too, I didn't think
>> about that, is something to take in account.
>>
>> Are you using overlayfs? How is it going? What difficulties you have found?
>
> Yes, we do. All difficulties we'd found are solved in referred initoverlay
> recipe. Maybe one fine I write a blog post regarding that topic ;)
>
>> Other solution whould be using Smart package manager to upgrade the
>> rootfs, but this doesn't attend my need for factory-reset.
>
> How does that fit into an ro rootfs?
It doesn't. If I choose this aproach I'll need an rw rootfs,
>
>> Please tell me more about your experience with overlayfs :)
>
> It's more stable than unionfs. There was little effort updating
> systems from 3.10 with overlay patch to 4.1 with overlay builtin
> kernel (work dirs must be created - https://github.com/rdm-dev/meta-jens/blob/jethro/recipes-core/initoverlay/initoverlay/migrate2overlay.sh#L28-L30)
I'm using 3.10 kernel, the overlayfs is supported as a patch for this
version of kernel? I'll take a look. Thanks for the reply
>
> Cheers
> --
> Jens Rehsack - rehsack@gmail.com
>

Regards,

- dhs


-- 
"Do or do not. There is no try"
  Yoda Master

Re: RFC: Reference updater filesystem

[Christian Ege]

On 23.11.2015 22:15, Mariano Lopez wrote:
> There has been interest in an image based software updater in Yocto
> Project. The proposed solution for a image based updater is to use
> Stefano Babic's software updater (http://sbabic.github.io/swupdate).
> This software do a binary copy, so it is needed to have at least two
> partitions, these partitions would be the rootfs and the maintenance
> partition. The rootfs it's the main partition used to boot during the
> normal device operation, on the other hand, the maintenance will be used
> to update the main partition.
> 
> To update the system, the user has to connect to device and boot in the
> maintenance partition; once in the maintenance partition the software
> updater will copy the new image in the rootfs partition. A final reboot
> into the rootfs it is necessary to complete the upgrade.
> 
> As mentioned before the the software will copy an image to the
> partition, so everything in that partition will be wiped out, including
> custom configurations. To avoid the loss of configuration I explore
> three different solutions:
> 1. Use a separate partition for the configuration.
>   a. The pro of this method is the partition is not touched during the
> update.
>   b. The con of this method is the configuration is not directly in
> rootfs (example: /etc).
> 
> 2. Do the backup during the update.
>   a. The pro is the configuration is directly in rootfs.
>   b. The con is If the update fail most likely the configuration would
> be lost.
> 
> 3. Have an OverlayFS for the rootfs or the partition that have the
> configuration.
>   a. The pro is the configuration is  "directly" in rootfs.
>   b. The con is there is need to provide a custom init to guarantee the
> Overlay is mounted before the boot process.
> 
> With the above information I'm proposing to use a separate partition for
> the configuration; this is because is more reliable and doesn't require
> big changes in the current architecture.
> 
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.
This is mostly what we have implemented for our O3D3xx 3D Camera at ifm.
We are using the swupdate tool developed by Stefano and sponsored
by ifm.

We have two flashes one small SPI NOR flash which is responsible for
booting and a larger NAND which contains the root filesystem.

Due to the restrictions to not have any buttons on the camera Stefano
implemented a boot counter which allows us to boot into swupdate from
the rootfs. Or if the rootfs is not able to boot we automatically boot
into swupdate mode itself.

To have the same ip address in both cases we have a small partition in
the SPI NOR which contains the network config and some stuff which must
retain during software update. The swupdate itself is implemented as a
kernel with an initramfs. We decided to have such an fallback to not
loose any camera during a field update. BTW this was a good decision and
we have a lot of good feedback on this feature.

Another option was the way how for example AVM Fritz.Box! implements the
update, they do download the image and install the image from the rootfs
during shutdown. But if someone removes power the box is bricked unless
you do some tftp magic. I guess swupdate is tunable to support this
usecase as-well.

Regards,
Christian

RFC: Reference updater filesystem

[Mariano Lopez]

There has been interest in an image based software updater in Yocto 
Project. The proposed solution for a image based updater is to use 
Stefano Babic's software updater (http://sbabic.github.io/swupdate). 
This software do a binary copy, so it is needed to have at least two 
partitions, these partitions would be the rootfs and the maintenance 
partition. The rootfs it's the main partition used to boot during the 
normal device operation, on the other hand, the maintenance will be used 
to update the main partition.

To update the system, the user has to connect to device and boot in the 
maintenance partition; once in the maintenance partition the software 
updater will copy the new image in the rootfs partition. A final reboot 
into the rootfs it is necessary to complete the upgrade.

As mentioned before the the software will copy an image to the 
partition, so everything in that partition will be wiped out, including 
custom configurations. To avoid the loss of configuration I explore 
three different solutions:
1. Use a separate partition for the configuration.
   a. The pro of this method is the partition is not touched during the 
update.
   b. The con of this method is the configuration is not directly in 
rootfs (example: /etc).

2. Do the backup during the update.
   a. The pro is the configuration is directly in rootfs.
   b. The con is If the update fail most likely the configuration would 
be lost.

3. Have an OverlayFS for the rootfs or the partition that have the 
configuration.
   a. The pro is the configuration is  "directly" in rootfs.
   b. The con is there is need to provide a custom init to guarantee the 
Overlay is mounted before the boot process.

With the above information I'm proposing to use a separate partition for 
the configuration; this is because is more reliable and doesn't require 
big changes in the current architecture.

So, the idea is to have 4 partitions in the media:
1. boot. This is the usual boot partition
2. data. This will hold the configuration files. Not modified by updates.
3. maintenance. This partition will be used to update rootfs.
4. rootfs. Partition used for normal operation.

Mariano

Re: RFC: Reference updater filesystem

[Anders Darander]

* Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]:

> There has been interest in an image based software updater in Yocto Project.

Ok. Sure, it might be nice with something that can be shared, instead
off everyone's building our own solutions.

> The proposed solution for a image based updater is to use Stefano Babic's
> software updater (http://sbabic.github.io/swupdate). This software do a
> binary copy, so it is needed to have at least two partitions, these
> partitions would be the rootfs and the maintenance partition. The rootfs
> it's the main partition used to boot during the normal device operation, on
> the other hand, the maintenance will be used to update the main partition.

I haven't checked the swupdate tool, though I'd suspect that it also
supports the alternating rootfs use case? (I.e. run system1 update
system2; reboot to system2. Next update is system1). This is a rather
common setup, not least when you need a remote upgrade facility.

Would your proposed inclusion to the Yocto Project support that case
too?

> To update the system, the user has to connect to device and boot in the
> maintenance partition; once in the maintenance partition the software
> updater will copy the new image in the rootfs partition. A final reboot into
> the rootfs it is necessary to complete the upgrade.

Like said above, not all system can be reached manually (at least not in
cost efficient way). Sure, the mainenance partition scheme can be made
to work anyway...

> As mentioned before the the software will copy an image to the partition, so
> everything in that partition will be wiped out, including custom
> configurations. To avoid the loss of configuration I explore three different
> solutions:
> 1. Use a separate partition for the configuration.
>   a. The pro of this method is the partition is not touched during the
> update.
>   b. The con of this method is the configuration is not directly in rootfs
> (example: /etc).

I'd vote for that as well. Though, I only keep the re-writable
configurations here. The one that are constant between all systems are
shipped in /etc in the read-only-rootfs.

> With the above information I'm proposing to use a separate partition for the
> configuration; this is because is more reliable and doesn't require big
> changes in the current architecture.

> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.

How flexible to you intend to make this system? Allow everything that
swupdate supports? Or a specific subset?

Cheers,
Anders

-- 
Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

Re: RFC: Reference updater filesystem

[Randy Witt]

[-- Attachment #1: Type: text/plain, Size: 2931 bytes --]

On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez <
mariano.lopez@linux.intel.com> wrote:

> There has been interest in an image based software updater in Yocto
> Project. The proposed solution for a image based updater is to use Stefano
> Babic's software updater (http://sbabic.github.io/swupdate). This
> software do a binary copy, so it is needed to have at least two partitions,
> these partitions would be the rootfs and the maintenance partition. The
> rootfs it's the main partition used to boot during the normal device
> operation, on the other hand, the maintenance will be used to update the
> main partition.
>
> To update the system, the user has to connect to device and boot in the
> maintenance partition; once in the maintenance partition the software
> updater will copy the new image in the rootfs partition. A final reboot
> into the rootfs it is necessary to complete the upgrade.
>
> As mentioned before the the software will copy an image to the partition,
> so everything in that partition will be wiped out, including custom
> configurations. To avoid the loss of configuration I explore three
> different solutions:
> 1. Use a separate partition for the configuration.
>   a. The pro of this method is the partition is not touched during the
> update.
>   b. The con of this method is the configuration is not directly in rootfs
> (example: /etc).
>
> Configuration files can be anywhere a package decides to install them. So
having a single partition would be difficult. If you could, you would most
likely be forced to have an initramfs to make sure /etc was mounted before
init runs.


> 2. Do the backup during the update.
>   a. The pro is the configuration is directly in rootfs.
>   b. The con is If the update fail most likely the configuration would be
> lost.
>
> Why would the configuration be lost if the update fails? Couldn't it just
be stored on the thumbdrive?


> 3. Have an OverlayFS for the rootfs or the partition that have the
> configuration.
>   a. The pro is the configuration is  "directly" in rootfs.
>   b. The con is there is need to provide a custom init to guarantee the
> Overlay is mounted before the boot process.
>
> With the above information I'm proposing to use a separate partition for
> the configuration; this is because is more reliable and doesn't require big
> changes in the current architecture.
>
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.
>
> Mariano
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3866 bytes --]

Re: RFC: Reference updater filesystem

[Lopez, Mariano]

On 11/24/2015 1:32 AM, Randy Witt wrote:
>
>
> On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez 
> <mariano.lopez@linux.intel.com <mailto:mariano.lopez@linux.intel.com>> 
> wrote:
>
>     There has been interest in an image based software updater in
>     Yocto Project. The proposed solution for a image based updater is
>     to use Stefano Babic's software updater
>     (http://sbabic.github.io/swupdate). This software do a binary
>     copy, so it is needed to have at least two partitions, these
>     partitions would be the rootfs and the maintenance partition. The
>     rootfs it's the main partition used to boot during the normal
>     device operation, on the other hand, the maintenance will be used
>     to update the main partition.
>
>     To update the system, the user has to connect to device and boot
>     in the maintenance partition; once in the maintenance partition
>     the software updater will copy the new image in the rootfs
>     partition. A final reboot into the rootfs it is necessary to
>     complete the upgrade.
>
>     As mentioned before the the software will copy an image to the
>     partition, so everything in that partition will be wiped out,
>     including custom configurations. To avoid the loss of
>     configuration I explore three different solutions:
>     1. Use a separate partition for the configuration.
>       a. The pro of this method is the partition is not touched during
>     the update.
>       b. The con of this method is the configuration is not directly
>     in rootfs (example: /etc).
>
> Configuration files can be anywhere a package decides to install them. 
> So having a single partition would be difficult. If you could, you 
> would most likely be forced to have an initramfs to make sure /etc was 
> mounted before init runs.

/etc was an example, the image should have the required files to make 
the target boot and the get the application configuration from this 
other partition. This is like openwrt does, it has a read-only rootfs 
and small read-write partition where the user can write its 
configuration and restore it at boot time.

>     2. Do the backup during the update.
>       a. The pro is the configuration is directly in rootfs.
>       b. The con is If the update fail most likely the configuration
>     would be lost.
>
> Why would the configuration be lost if the update fails? Couldn't it 
> just be stored on the thumbdrive?

If there is a power loss while the configuration is copied, the 
partition could go corrupt and would be difficult to recover. And as you 
mentioned  before the configuration files could be anywhere, so the 
script must be customized to get all those files and once the update is 
complete another script must restore those files, these could be 
cumbersome instead of the application have the config in another partition.

>     3. Have an OverlayFS for the rootfs or the partition that have the
>     configuration.
>       a. The pro is the configuration is  "directly" in rootfs.
>       b. The con is there is need to provide a custom init to
>     guarantee the Overlay is mounted before the boot process.
>
>     With the above information I'm proposing to use a separate
>     partition for the configuration; this is because is more reliable
>     and doesn't require big changes in the current architecture.
>
>     So, the idea is to have 4 partitions in the media:
>     1. boot. This is the usual boot partition
>     2. data. This will hold the configuration files. Not modified by
>     updates.
>     3. maintenance. This partition will be used to update rootfs.
>     4. rootfs. Partition used for normal operation.
>
>     Mariano
>     -- 
>     _______________________________________________
>     Openembedded-core mailing list
>     Openembedded-core@lists.openembedded.org
>     <mailto:Openembedded-core@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Re: RFC: Reference updater filesystem

[Roman Khimov]

В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал:
> 1. Use a separate partition for the configuration.
>    a. The pro of this method is the partition is not touched during the
> update.
>    b. The con of this method is the configuration is not directly in
> rootfs (example: /etc).

That's the right solution, although to do it really right (at least IMO) you 
need to implement the /usr merge [1] (and that's orthogonal to using or not 
using systemd), which can also help you make your /usr read-only (because 
that's just code and static data) with read-write / for user data of various 
sorts.

> 3. Have an OverlayFS for the rootfs or the partition that have the
> configuration.
>    a. The pro is the configuration is  "directly" in rootfs.
>    b. The con is there is need to provide a custom init to guarantee the
> Overlay is mounted before the boot process.

And this is the approach I would recommend not doing. I've used UnionFS for 
thing like that (overlaying whole root file system) some 6 years ago, it 
sounded nice and it kinda worked, but it wasn't difficult to make it fail 
(just a little playing with power), we've even seen failures on production 
devices, like when you have whiteout file for directory already written, but 
don't have new files in it yet and that can completely ruin the system.

Also, it usually works better when you don't have any changes in the lower 
layer, but we're talking about updating it here, you can easily end up in a 
situation where you have updated something in the rootfs but that was 
overriden by upper layer and thus your user doesn't see any change.

> With the above information I'm proposing to use a separate partition for
> the configuration; this is because is more reliable and doesn't require
> big changes in the current architecture.
> 
> So, the idea is to have 4 partitions in the media:
> 1. boot. This is the usual boot partition
> 2. data. This will hold the configuration files. Not modified by updates.
> 3. maintenance. This partition will be used to update rootfs.
> 4. rootfs. Partition used for normal operation.

You probably don't need to separate 1 and 3, all the code for system update 
should easily fit into initramfs and just making /boot a bit larger would 
allow you to store some backup rootfs.

Also, you can swap 4 and 2 which will be useful if you're installing on 
different sized storage devices, usually you know good enough the size of your 
rootfs, but you probably want to leave more space for user data if there is an 
opportunity to do so, that's just easier to do with data partition at the end.


[1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

Re: RFC: Reference updater filesystem

[Lopez, Mariano]

On 11/24/2015 4:30 AM, Roman Khimov wrote:
> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал:
>> 1. Use a separate partition for the configuration.
>>     a. The pro of this method is the partition is not touched during the
>> update.
>>     b. The con of this method is the configuration is not directly in
>> rootfs (example: /etc).
> That's the right solution, although to do it really right (at least IMO) you
> need to implement the /usr merge [1] (and that's orthogonal to using or not
> using systemd), which can also help you make your /usr read-only (because
> that's just code and static data) with read-write / for user data of various
> sorts.

To be honest I'm not familiar with /usr merge, I need to check on that 
to see if it is a good option with the current OE-core infrastructure.

>
>> 3. Have an OverlayFS for the rootfs or the partition that have the
>> configuration.
>>     a. The pro is the configuration is  "directly" in rootfs.
>>     b. The con is there is need to provide a custom init to guarantee the
>> Overlay is mounted before the boot process.
> And this is the approach I would recommend not doing. I've used UnionFS for
> thing like that (overlaying whole root file system) some 6 years ago, it
> sounded nice and it kinda worked, but it wasn't difficult to make it fail
> (just a little playing with power), we've even seen failures on production
> devices, like when you have whiteout file for directory already written, but
> don't have new files in it yet and that can completely ruin the system.
>
> Also, it usually works better when you don't have any changes in the lower
> layer, but we're talking about updating it here, you can easily end up in a
> situation where you have updated something in the rootfs but that was
> overriden by upper layer and thus your user doesn't see any change.

Thanks for sharing your experience, this is another big con for the 
Overlay option.

>
>> With the above information I'm proposing to use a separate partition for
>> the configuration; this is because is more reliable and doesn't require
>> big changes in the current architecture.
>>
>> So, the idea is to have 4 partitions in the media:
>> 1. boot. This is the usual boot partition
>> 2. data. This will hold the configuration files. Not modified by updates.
>> 3. maintenance. This partition will be used to update rootfs.
>> 4. rootfs. Partition used for normal operation.
> You probably don't need to separate 1 and 3, all the code for system update
> should easily fit into initramfs and just making /boot a bit larger would
> allow you to store some backup rootfs.

I left the /boot partition separate just in case there is need to 
replace the kernel or the bootloader. This way it would be easier to 
change using the same method as the upgrading the rootfs.

>
> Also, you can swap 4 and 2 which will be useful if you're installing on
> different sized storage devices, usually you know good enough the size of your
> rootfs, but you probably want to leave more space for user data if there is an
> opportunity to do so, that's just easier to do with data partition at the end.

I was thinking in the same thinking just backwards, usually 
configuration files are just small text files that don't require too 
much space. If you require a new feature in the target that will make 
rootfs to grow depending on the feature. I plan to use wic to accomplish 
the filesystem structure. A good thing about wic is that it will be very 
easy to do the swap, just need to modify two options in the .wks file.

>
>
> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

Re: RFC: Reference updater filesystem

[Roman Khimov]

В письме от 24 ноября 2015 10:30:17 пользователь Lopez, Mariano написал:
> > That's the right solution, although to do it really right (at least IMO)
> > you need to implement the /usr merge [1] (and that's orthogonal to using
> > or not using systemd), which can also help you make your /usr read-only
> > (because that's just code and static data) with read-write / for user
> > data of various sorts.
> 
> To be honest I'm not familiar with /usr merge, I need to check on that
> to see if it is a good option with the current OE-core infrastructure.

It needs some patches for current OE, I hope I'll find some time to rebase the 
set I have for this (based on fido) and send it for discussion.

-- 
 http://roman.khimov.ru
mailto: roman@khimov.ru
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3