* RFC: Reference updater filesystem @ 2015-11-23 21:41 Mariano Lopez 2015-11-24 6:06 ` Anders Darander ` (3 more replies) 0 siblings, 4 replies; 35+ messages in thread From: Mariano Lopez @ 2015-11-23 21:41 UTC (permalink / raw) To: openembedded-devel, OE-core, yocto There has been interest in an image based software updater in Yocto Project. The proposed solution for a image based updater is to use Stefano Babic's software updater (http://sbabic.github.io/swupdate). This software do a binary copy, so it is needed to have at least two partitions, these partitions would be the rootfs and the maintenance partition. The rootfs it's the main partition used to boot during the normal device operation, on the other hand, the maintenance will be used to update the main partition. To update the system, the user has to connect to device and boot in the maintenance partition; once in the maintenance partition the software updater will copy the new image in the rootfs partition. A final reboot into the rootfs it is necessary to complete the upgrade. As mentioned before the the software will copy an image to the partition, so everything in that partition will be wiped out, including custom configurations. To avoid the loss of configuration I explore three different solutions: 1. Use a separate partition for the configuration. a. The pro of this method is the partition is not touched during the update. b. The con of this method is the configuration is not directly in rootfs (example: /etc). 2. Do the backup during the update. a. The pro is the configuration is directly in rootfs. b. The con is If the update fail most likely the configuration would be lost. 3. Have an OverlayFS for the rootfs or the partition that have the configuration. a. The pro is the configuration is "directly" in rootfs. b. The con is there is need to provide a custom init to guarantee the Overlay is mounted before the boot process. With the above information I'm proposing to use a separate partition for the configuration; this is because is more reliable and doesn't require big changes in the current architecture. So, the idea is to have 4 partitions in the media: 1. boot. This is the usual boot partition 2. data. This will hold the configuration files. Not modified by updates. 3. maintenance. This partition will be used to update rootfs. 4. rootfs. Partition used for normal operation. Mariano ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem 2015-11-23 21:41 RFC: Reference updater filesystem Mariano Lopez @ 2015-11-24 6:06 ` Anders Darander 2015-11-24 7:32 ` Randy Witt ` (2 subsequent siblings) 3 siblings, 0 replies; 35+ messages in thread From: Anders Darander @ 2015-11-24 6:06 UTC (permalink / raw) To: openembedded-core, openembedded-devel, yocto * Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]: > There has been interest in an image based software updater in Yocto Project. Ok. Sure, it might be nice with something that can be shared, instead off everyone's building our own solutions. > The proposed solution for a image based updater is to use Stefano Babic's > software updater (http://sbabic.github.io/swupdate). This software do a > binary copy, so it is needed to have at least two partitions, these > partitions would be the rootfs and the maintenance partition. The rootfs > it's the main partition used to boot during the normal device operation, on > the other hand, the maintenance will be used to update the main partition. I haven't checked the swupdate tool, though I'd suspect that it also supports the alternating rootfs use case? (I.e. run system1 update system2; reboot to system2. Next update is system1). This is a rather common setup, not least when you need a remote upgrade facility. Would your proposed inclusion to the Yocto Project support that case too? > To update the system, the user has to connect to device and boot in the > maintenance partition; once in the maintenance partition the software > updater will copy the new image in the rootfs partition. A final reboot into > the rootfs it is necessary to complete the upgrade. Like said above, not all system can be reached manually (at least not in cost efficient way). Sure, the mainenance partition scheme can be made to work anyway... > As mentioned before the the software will copy an image to the partition, so > everything in that partition will be wiped out, including custom > configurations. To avoid the loss of configuration I explore three different > solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in rootfs > (example: /etc). I'd vote for that as well. Though, I only keep the re-writable configurations here. The one that are constant between all systems are shipped in /etc in the read-only-rootfs. > With the above information I'm proposing to use a separate partition for the > configuration; this is because is more reliable and doesn't require big > changes in the current architecture. > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. How flexible to you intend to make this system? Allow everything that swupdate supports? Or a specific subset? Cheers, Anders -- Anders Darander, Senior System Architect ChargeStorm AB / eStorm AB ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 6:06 ` Anders Darander 0 siblings, 0 replies; 35+ messages in thread From: Anders Darander @ 2015-11-24 6:06 UTC (permalink / raw) To: openembedded-core, openembedded-devel, yocto * Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]: > There has been interest in an image based software updater in Yocto Project. Ok. Sure, it might be nice with something that can be shared, instead off everyone's building our own solutions. > The proposed solution for a image based updater is to use Stefano Babic's > software updater (http://sbabic.github.io/swupdate). This software do a > binary copy, so it is needed to have at least two partitions, these > partitions would be the rootfs and the maintenance partition. The rootfs > it's the main partition used to boot during the normal device operation, on > the other hand, the maintenance will be used to update the main partition. I haven't checked the swupdate tool, though I'd suspect that it also supports the alternating rootfs use case? (I.e. run system1 update system2; reboot to system2. Next update is system1). This is a rather common setup, not least when you need a remote upgrade facility. Would your proposed inclusion to the Yocto Project support that case too? > To update the system, the user has to connect to device and boot in the > maintenance partition; once in the maintenance partition the software > updater will copy the new image in the rootfs partition. A final reboot into > the rootfs it is necessary to complete the upgrade. Like said above, not all system can be reached manually (at least not in cost efficient way). Sure, the mainenance partition scheme can be made to work anyway... > As mentioned before the the software will copy an image to the partition, so > everything in that partition will be wiped out, including custom > configurations. To avoid the loss of configuration I explore three different > solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in rootfs > (example: /etc). I'd vote for that as well. Though, I only keep the re-writable configurations here. The one that are constant between all systems are shipped in /etc in the read-only-rootfs. > With the above information I'm proposing to use a separate partition for the > configuration; this is because is more reliable and doesn't require big > changes in the current architecture. > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. How flexible to you intend to make this system? Allow everything that swupdate supports? Or a specific subset? Cheers, Anders -- Anders Darander, Senior System Architect ChargeStorm AB / eStorm AB ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem 2015-11-24 6:06 ` Anders Darander (?) @ 2015-11-24 18:39 ` Lopez, Mariano -1 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 18:39 UTC (permalink / raw) To: openembedded-core, openembedded-devel, yocto On 11/24/2015 12:06 AM, Anders Darander wrote: > * Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]: > >> There has been interest in an image based software updater in Yocto Project. > Ok. Sure, it might be nice with something that can be shared, instead > off everyone's building our own solutions. The idea is to integrate one, not build one from the scratch. > >> The proposed solution for a image based updater is to use Stefano Babic's >> software updater (http://sbabic.github.io/swupdate). This software do a >> binary copy, so it is needed to have at least two partitions, these >> partitions would be the rootfs and the maintenance partition. The rootfs >> it's the main partition used to boot during the normal device operation, on >> the other hand, the maintenance will be used to update the main partition. > I haven't checked the swupdate tool, though I'd suspect that it also > supports the alternating rootfs use case? (I.e. run system1 update > system2; reboot to system2. Next update is system1). This is a rather > common setup, not least when you need a remote upgrade facility. > > Would your proposed inclusion to the Yocto Project support that case > too? Yeah, it would be possible to have two "rootfs" and do the update and the just reboot one time. > >> To update the system, the user has to connect to device and boot in the >> maintenance partition; once in the maintenance partition the software >> updater will copy the new image in the rootfs partition. A final reboot into >> the rootfs it is necessary to complete the upgrade. > Like said above, not all system can be reached manually (at least not in > cost efficient way). Sure, the mainenance partition scheme can be made > to work anyway... I plan to release this in phases, in the first one it will be manually do the update. The idea is implement tools to automate the process of the update (where it can be automated). > >> As mentioned before the the software will copy an image to the partition, so >> everything in that partition will be wiped out, including custom >> configurations. To avoid the loss of configuration I explore three different >> solutions: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in rootfs >> (example: /etc). > I'd vote for that as well. Though, I only keep the re-writable > configurations here. The one that are constant between all systems are > shipped in /etc in the read-only-rootfs. > >> With the above information I'm proposing to use a separate partition for the >> configuration; this is because is more reliable and doesn't require big >> changes in the current architecture. >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > How flexible to you intend to make this system? Allow everything that > swupdate supports? Or a specific subset? If you are referring to the filesystem creation I would say very flexible. It will be implemented using wic instead of a class, so just needs to change a file to suit your needs. If you refer to the swupdate features, I plan to have a generic use case; as an example I won't use the MTD capabilities of the software. > > Cheers, > Anders > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] [OE-core] RFC: Reference updater filesystem @ 2015-11-24 18:39 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 18:39 UTC (permalink / raw) To: openembedded-core, openembedded-devel, yocto On 11/24/2015 12:06 AM, Anders Darander wrote: > * Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]: > >> There has been interest in an image based software updater in Yocto Project. > Ok. Sure, it might be nice with something that can be shared, instead > off everyone's building our own solutions. The idea is to integrate one, not build one from the scratch. > >> The proposed solution for a image based updater is to use Stefano Babic's >> software updater (http://sbabic.github.io/swupdate). This software do a >> binary copy, so it is needed to have at least two partitions, these >> partitions would be the rootfs and the maintenance partition. The rootfs >> it's the main partition used to boot during the normal device operation, on >> the other hand, the maintenance will be used to update the main partition. > I haven't checked the swupdate tool, though I'd suspect that it also > supports the alternating rootfs use case? (I.e. run system1 update > system2; reboot to system2. Next update is system1). This is a rather > common setup, not least when you need a remote upgrade facility. > > Would your proposed inclusion to the Yocto Project support that case > too? Yeah, it would be possible to have two "rootfs" and do the update and the just reboot one time. > >> To update the system, the user has to connect to device and boot in the >> maintenance partition; once in the maintenance partition the software >> updater will copy the new image in the rootfs partition. A final reboot into >> the rootfs it is necessary to complete the upgrade. > Like said above, not all system can be reached manually (at least not in > cost efficient way). Sure, the mainenance partition scheme can be made > to work anyway... I plan to release this in phases, in the first one it will be manually do the update. The idea is implement tools to automate the process of the update (where it can be automated). > >> As mentioned before the the software will copy an image to the partition, so >> everything in that partition will be wiped out, including custom >> configurations. To avoid the loss of configuration I explore three different >> solutions: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in rootfs >> (example: /etc). > I'd vote for that as well. Though, I only keep the re-writable > configurations here. The one that are constant between all systems are > shipped in /etc in the read-only-rootfs. > >> With the above information I'm proposing to use a separate partition for the >> configuration; this is because is more reliable and doesn't require big >> changes in the current architecture. >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > How flexible to you intend to make this system? Allow everything that > swupdate supports? Or a specific subset? If you are referring to the filesystem creation I would say very flexible. It will be implemented using wic instead of a class, so just needs to change a file to suit your needs. If you refer to the swupdate features, I plan to have a generic use case; as an example I won't use the MTD capabilities of the software. > > Cheers, > Anders > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] RFC: Reference updater filesystem @ 2015-11-24 18:39 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 18:39 UTC (permalink / raw) To: openembedded-core, openembedded-devel, yocto On 11/24/2015 12:06 AM, Anders Darander wrote: > * Mariano Lopez <mariano.lopez@linux.intel.com> [151123 22:41]: > >> There has been interest in an image based software updater in Yocto Project. > Ok. Sure, it might be nice with something that can be shared, instead > off everyone's building our own solutions. The idea is to integrate one, not build one from the scratch. > >> The proposed solution for a image based updater is to use Stefano Babic's >> software updater (http://sbabic.github.io/swupdate). This software do a >> binary copy, so it is needed to have at least two partitions, these >> partitions would be the rootfs and the maintenance partition. The rootfs >> it's the main partition used to boot during the normal device operation, on >> the other hand, the maintenance will be used to update the main partition. > I haven't checked the swupdate tool, though I'd suspect that it also > supports the alternating rootfs use case? (I.e. run system1 update > system2; reboot to system2. Next update is system1). This is a rather > common setup, not least when you need a remote upgrade facility. > > Would your proposed inclusion to the Yocto Project support that case > too? Yeah, it would be possible to have two "rootfs" and do the update and the just reboot one time. > >> To update the system, the user has to connect to device and boot in the >> maintenance partition; once in the maintenance partition the software >> updater will copy the new image in the rootfs partition. A final reboot into >> the rootfs it is necessary to complete the upgrade. > Like said above, not all system can be reached manually (at least not in > cost efficient way). Sure, the mainenance partition scheme can be made > to work anyway... I plan to release this in phases, in the first one it will be manually do the update. The idea is implement tools to automate the process of the update (where it can be automated). > >> As mentioned before the the software will copy an image to the partition, so >> everything in that partition will be wiped out, including custom >> configurations. To avoid the loss of configuration I explore three different >> solutions: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in rootfs >> (example: /etc). > I'd vote for that as well. Though, I only keep the re-writable > configurations here. The one that are constant between all systems are > shipped in /etc in the read-only-rootfs. > >> With the above information I'm proposing to use a separate partition for the >> configuration; this is because is more reliable and doesn't require big >> changes in the current architecture. >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > How flexible to you intend to make this system? Allow everything that > swupdate supports? Or a specific subset? If you are referring to the filesystem creation I would say very flexible. It will be implemented using wic instead of a class, so just needs to change a file to suit your needs. If you refer to the swupdate features, I plan to have a generic use case; as an example I won't use the MTD capabilities of the software. > > Cheers, > Anders > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem 2015-11-23 21:41 RFC: Reference updater filesystem Mariano Lopez @ 2015-11-24 7:32 ` Randy Witt 2015-11-24 7:32 ` Randy Witt ` (2 subsequent siblings) 3 siblings, 0 replies; 35+ messages in thread From: Randy Witt @ 2015-11-24 7:32 UTC (permalink / raw) To: Mariano Lopez; +Cc: yocto, openembedded-devel, OE-core [-- Attachment #1: Type: text/plain, Size: 2931 bytes --] On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez < mariano.lopez@linux.intel.com> wrote: > There has been interest in an image based software updater in Yocto > Project. The proposed solution for a image based updater is to use Stefano > Babic's software updater (http://sbabic.github.io/swupdate). This > software do a binary copy, so it is needed to have at least two partitions, > these partitions would be the rootfs and the maintenance partition. The > rootfs it's the main partition used to boot during the normal device > operation, on the other hand, the maintenance will be used to update the > main partition. > > To update the system, the user has to connect to device and boot in the > maintenance partition; once in the maintenance partition the software > updater will copy the new image in the rootfs partition. A final reboot > into the rootfs it is necessary to complete the upgrade. > > As mentioned before the the software will copy an image to the partition, > so everything in that partition will be wiped out, including custom > configurations. To avoid the loss of configuration I explore three > different solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in rootfs > (example: /etc). > > Configuration files can be anywhere a package decides to install them. So having a single partition would be difficult. If you could, you would most likely be forced to have an initramfs to make sure /etc was mounted before init runs. > 2. Do the backup during the update. > a. The pro is the configuration is directly in rootfs. > b. The con is If the update fail most likely the configuration would be > lost. > > Why would the configuration be lost if the update fails? Couldn't it just be stored on the thumbdrive? > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to guarantee the > Overlay is mounted before the boot process. > > With the above information I'm proposing to use a separate partition for > the configuration; this is because is more reliable and doesn't require big > changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. > > Mariano > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > [-- Attachment #2: Type: text/html, Size: 3866 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 7:32 ` Randy Witt 0 siblings, 0 replies; 35+ messages in thread From: Randy Witt @ 2015-11-24 7:32 UTC (permalink / raw) To: Mariano Lopez; +Cc: yocto, openembedded-devel, OE-core [-- Attachment #1: Type: text/plain, Size: 2931 bytes --] On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez < mariano.lopez@linux.intel.com> wrote: > There has been interest in an image based software updater in Yocto > Project. The proposed solution for a image based updater is to use Stefano > Babic's software updater (http://sbabic.github.io/swupdate). This > software do a binary copy, so it is needed to have at least two partitions, > these partitions would be the rootfs and the maintenance partition. The > rootfs it's the main partition used to boot during the normal device > operation, on the other hand, the maintenance will be used to update the > main partition. > > To update the system, the user has to connect to device and boot in the > maintenance partition; once in the maintenance partition the software > updater will copy the new image in the rootfs partition. A final reboot > into the rootfs it is necessary to complete the upgrade. > > As mentioned before the the software will copy an image to the partition, > so everything in that partition will be wiped out, including custom > configurations. To avoid the loss of configuration I explore three > different solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in rootfs > (example: /etc). > > Configuration files can be anywhere a package decides to install them. So having a single partition would be difficult. If you could, you would most likely be forced to have an initramfs to make sure /etc was mounted before init runs. > 2. Do the backup during the update. > a. The pro is the configuration is directly in rootfs. > b. The con is If the update fail most likely the configuration would be > lost. > > Why would the configuration be lost if the update fails? Couldn't it just be stored on the thumbdrive? > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to guarantee the > Overlay is mounted before the boot process. > > With the above information I'm proposing to use a separate partition for > the configuration; this is because is more reliable and doesn't require big > changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. > > Mariano > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > [-- Attachment #2: Type: text/html, Size: 3866 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem 2015-11-24 7:32 ` Randy Witt @ 2015-11-24 17:19 ` Lopez, Mariano -1 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 17:19 UTC (permalink / raw) To: Randy Witt; +Cc: yocto, openembedded-devel, OE-core On 11/24/2015 1:32 AM, Randy Witt wrote: > > > On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez > <mariano.lopez@linux.intel.com <mailto:mariano.lopez@linux.intel.com>> > wrote: > > There has been interest in an image based software updater in > Yocto Project. The proposed solution for a image based updater is > to use Stefano Babic's software updater > (http://sbabic.github.io/swupdate). This software do a binary > copy, so it is needed to have at least two partitions, these > partitions would be the rootfs and the maintenance partition. The > rootfs it's the main partition used to boot during the normal > device operation, on the other hand, the maintenance will be used > to update the main partition. > > To update the system, the user has to connect to device and boot > in the maintenance partition; once in the maintenance partition > the software updater will copy the new image in the rootfs > partition. A final reboot into the rootfs it is necessary to > complete the upgrade. > > As mentioned before the the software will copy an image to the > partition, so everything in that partition will be wiped out, > including custom configurations. To avoid the loss of > configuration I explore three different solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during > the update. > b. The con of this method is the configuration is not directly > in rootfs (example: /etc). > > Configuration files can be anywhere a package decides to install them. > So having a single partition would be difficult. If you could, you > would most likely be forced to have an initramfs to make sure /etc was > mounted before init runs. /etc was an example, the image should have the required files to make the target boot and the get the application configuration from this other partition. This is like openwrt does, it has a read-only rootfs and small read-write partition where the user can write its configuration and restore it at boot time. > 2. Do the backup during the update. > a. The pro is the configuration is directly in rootfs. > b. The con is If the update fail most likely the configuration > would be lost. > > Why would the configuration be lost if the update fails? Couldn't it > just be stored on the thumbdrive? If there is a power loss while the configuration is copied, the partition could go corrupt and would be difficult to recover. And as you mentioned before the configuration files could be anywhere, so the script must be customized to get all those files and once the update is complete another script must restore those files, these could be cumbersome instead of the application have the config in another partition. > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to > guarantee the Overlay is mounted before the boot process. > > With the above information I'm proposing to use a separate > partition for the configuration; this is because is more reliable > and doesn't require big changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by > updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. > > Mariano > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > <mailto:Openembedded-core@lists.openembedded.org> > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 17:19 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 17:19 UTC (permalink / raw) To: Randy Witt; +Cc: yocto, openembedded-devel, OE-core On 11/24/2015 1:32 AM, Randy Witt wrote: > > > On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez > <mariano.lopez@linux.intel.com <mailto:mariano.lopez@linux.intel.com>> > wrote: > > There has been interest in an image based software updater in > Yocto Project. The proposed solution for a image based updater is > to use Stefano Babic's software updater > (http://sbabic.github.io/swupdate). This software do a binary > copy, so it is needed to have at least two partitions, these > partitions would be the rootfs and the maintenance partition. The > rootfs it's the main partition used to boot during the normal > device operation, on the other hand, the maintenance will be used > to update the main partition. > > To update the system, the user has to connect to device and boot > in the maintenance partition; once in the maintenance partition > the software updater will copy the new image in the rootfs > partition. A final reboot into the rootfs it is necessary to > complete the upgrade. > > As mentioned before the the software will copy an image to the > partition, so everything in that partition will be wiped out, > including custom configurations. To avoid the loss of > configuration I explore three different solutions: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during > the update. > b. The con of this method is the configuration is not directly > in rootfs (example: /etc). > > Configuration files can be anywhere a package decides to install them. > So having a single partition would be difficult. If you could, you > would most likely be forced to have an initramfs to make sure /etc was > mounted before init runs. /etc was an example, the image should have the required files to make the target boot and the get the application configuration from this other partition. This is like openwrt does, it has a read-only rootfs and small read-write partition where the user can write its configuration and restore it at boot time. > 2. Do the backup during the update. > a. The pro is the configuration is directly in rootfs. > b. The con is If the update fail most likely the configuration > would be lost. > > Why would the configuration be lost if the update fails? Couldn't it > just be stored on the thumbdrive? If there is a power loss while the configuration is copied, the partition could go corrupt and would be difficult to recover. And as you mentioned before the configuration files could be anywhere, so the script must be customized to get all those files and once the update is complete another script must restore those files, these could be cumbersome instead of the application have the config in another partition. > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to > guarantee the Overlay is mounted before the boot process. > > With the above information I'm proposing to use a separate > partition for the configuration; this is because is more reliable > and doesn't require big changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by > updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. > > Mariano > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > <mailto:Openembedded-core@lists.openembedded.org> > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-23 21:41 RFC: Reference updater filesystem Mariano Lopez @ 2015-11-24 10:39 ` Roman Khimov 2015-11-24 7:32 ` Randy Witt ` (2 subsequent siblings) 3 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 10:39 UTC (permalink / raw) To: openembedded-devel; +Cc: yocto, OE-core В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in > rootfs (example: /etc). That's the right solution, although to do it really right (at least IMO) you need to implement the /usr merge [1] (and that's orthogonal to using or not using systemd), which can also help you make your /usr read-only (because that's just code and static data) with read-write / for user data of various sorts. > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to guarantee the > Overlay is mounted before the boot process. And this is the approach I would recommend not doing. I've used UnionFS for thing like that (overlaying whole root file system) some 6 years ago, it sounded nice and it kinda worked, but it wasn't difficult to make it fail (just a little playing with power), we've even seen failures on production devices, like when you have whiteout file for directory already written, but don't have new files in it yet and that can completely ruin the system. Also, it usually works better when you don't have any changes in the lower layer, but we're talking about updating it here, you can easily end up in a situation where you have updated something in the rootfs but that was overriden by upper layer and thus your user doesn't see any change. > With the above information I'm proposing to use a separate partition for > the configuration; this is because is more reliable and doesn't require > big changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. You probably don't need to separate 1 and 3, all the code for system update should easily fit into initramfs and just making /boot a bit larger would allow you to store some backup rootfs. Also, you can swap 4 and 2 which will be useful if you're installing on different sized storage devices, usually you know good enough the size of your rootfs, but you probably want to leave more space for user data if there is an opportunity to do so, that's just easier to do with data partition at the end. [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 10:39 ` Roman Khimov 0 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 10:39 UTC (permalink / raw) To: openembedded-devel; +Cc: yocto, OE-core В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > 1. Use a separate partition for the configuration. > a. The pro of this method is the partition is not touched during the > update. > b. The con of this method is the configuration is not directly in > rootfs (example: /etc). That's the right solution, although to do it really right (at least IMO) you need to implement the /usr merge [1] (and that's orthogonal to using or not using systemd), which can also help you make your /usr read-only (because that's just code and static data) with read-write / for user data of various sorts. > 3. Have an OverlayFS for the rootfs or the partition that have the > configuration. > a. The pro is the configuration is "directly" in rootfs. > b. The con is there is need to provide a custom init to guarantee the > Overlay is mounted before the boot process. And this is the approach I would recommend not doing. I've used UnionFS for thing like that (overlaying whole root file system) some 6 years ago, it sounded nice and it kinda worked, but it wasn't difficult to make it fail (just a little playing with power), we've even seen failures on production devices, like when you have whiteout file for directory already written, but don't have new files in it yet and that can completely ruin the system. Also, it usually works better when you don't have any changes in the lower layer, but we're talking about updating it here, you can easily end up in a situation where you have updated something in the rootfs but that was overriden by upper layer and thus your user doesn't see any change. > With the above information I'm proposing to use a separate partition for > the configuration; this is because is more reliable and doesn't require > big changes in the current architecture. > > So, the idea is to have 4 partitions in the media: > 1. boot. This is the usual boot partition > 2. data. This will hold the configuration files. Not modified by updates. > 3. maintenance. This partition will be used to update rootfs. > 4. rootfs. Partition used for normal operation. You probably don't need to separate 1 and 3, all the code for system update should easily fit into initramfs and just making /boot a bit larger would allow you to store some backup rootfs. Also, you can swap 4 and 2 which will be useful if you're installing on different sized storage devices, usually you know good enough the size of your rootfs, but you probably want to leave more space for user data if there is an opportunity to do so, that's just easier to do with data partition at the end. [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] [oe] RFC: Reference updater filesystem 2015-11-24 10:39 ` Roman Khimov (?) @ 2015-11-24 13:47 ` Mark Hatle -1 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 13:47 UTC (permalink / raw) To: Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 4:39 AM, Roman Khimov wrote: > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in >> rootfs (example: /etc). > > That's the right solution, although to do it really right (at least IMO) you > need to implement the /usr merge [1] (and that's orthogonal to using or not > using systemd), which can also help you make your /usr read-only (because > that's just code and static data) with read-write / for user data of various > sorts. Why does merging /usr have anything to do with this? I've read the case for merging /usr and / and still don't understand why it "helps". The key is that if you have separate partitions for /usr and /, then you need to update both of them in sequence. Merging these two just seems like a lazy solution to people not wanting to deal with early boot being self-contained. Also having a separate / from /usr can help with '/' be your maintenance partition in some cases. >> 3. Have an OverlayFS for the rootfs or the partition that have the >> configuration. >> a. The pro is the configuration is "directly" in rootfs. >> b. The con is there is need to provide a custom init to guarantee the >> Overlay is mounted before the boot process. > > And this is the approach I would recommend not doing. I've used UnionFS for > thing like that (overlaying whole root file system) some 6 years ago, it > sounded nice and it kinda worked, but it wasn't difficult to make it fail > (just a little playing with power), we've even seen failures on production > devices, like when you have whiteout file for directory already written, but > don't have new files in it yet and that can completely ruin the system. > > Also, it usually works better when you don't have any changes in the lower > layer, but we're talking about updating it here, you can easily end up in a > situation where you have updated something in the rootfs but that was > overriden by upper layer and thus your user doesn't see any change. When using overlayfs, I'd strongly recommend not doing it over the entire rootfs. This is generally a bad idea for the reasons stated above. However, overlaying a part of the rootfs often makes sense. /etc is a good example. This way applications that want their configurations in /etc can still have it that way -- and there is always a (hopefully) reasonable default configuration, should the configuration 'partition' get corrupted. So worst case the user can start over on configurations only. For applications and user data, these can and should be stored outside of the main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's -actually- /opt, the concept itself is good. So going back to image upgrade. The key here is that you need a way to update arbitrary images with arbitrary contents and a mechanisms that is smart enough to generate the update (vs a full image flash) to conserve bandwidth. I still contend it's up to the device to be able to configure the system on how to get the update and where to apply the update. The tooling (host and target) should simply assist with this. Delta updates need version information in order to know they're doing the right sequence of updating. Full updates don't, but should be sent in a format that limits "empty space", effectively send them as sparse files. On many devices you will need to flash as part of the download due to space limitations. And you need the ability to flash multiple partitions. maintenance / /usr data etc.. whatever it takes to either upgrade or restore the device. --Mark >> With the above information I'm proposing to use a separate partition for >> the configuration; this is because is more reliable and doesn't require >> big changes in the current architecture. >> >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > > You probably don't need to separate 1 and 3, all the code for system update > should easily fit into initramfs and just making /boot a bit larger would > allow you to store some backup rootfs. > > Also, you can swap 4 and 2 which will be useful if you're installing on > different sized storage devices, usually you know good enough the size of your > rootfs, but you probably want to leave more space for user data if there is an > opportunity to do so, that's just easier to do with data partition at the end. > > [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-11-24 13:47 ` Mark Hatle 0 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 13:47 UTC (permalink / raw) To: Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 4:39 AM, Roman Khimov wrote: > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in >> rootfs (example: /etc). > > That's the right solution, although to do it really right (at least IMO) you > need to implement the /usr merge [1] (and that's orthogonal to using or not > using systemd), which can also help you make your /usr read-only (because > that's just code and static data) with read-write / for user data of various > sorts. Why does merging /usr have anything to do with this? I've read the case for merging /usr and / and still don't understand why it "helps". The key is that if you have separate partitions for /usr and /, then you need to update both of them in sequence. Merging these two just seems like a lazy solution to people not wanting to deal with early boot being self-contained. Also having a separate / from /usr can help with '/' be your maintenance partition in some cases. >> 3. Have an OverlayFS for the rootfs or the partition that have the >> configuration. >> a. The pro is the configuration is "directly" in rootfs. >> b. The con is there is need to provide a custom init to guarantee the >> Overlay is mounted before the boot process. > > And this is the approach I would recommend not doing. I've used UnionFS for > thing like that (overlaying whole root file system) some 6 years ago, it > sounded nice and it kinda worked, but it wasn't difficult to make it fail > (just a little playing with power), we've even seen failures on production > devices, like when you have whiteout file for directory already written, but > don't have new files in it yet and that can completely ruin the system. > > Also, it usually works better when you don't have any changes in the lower > layer, but we're talking about updating it here, you can easily end up in a > situation where you have updated something in the rootfs but that was > overriden by upper layer and thus your user doesn't see any change. When using overlayfs, I'd strongly recommend not doing it over the entire rootfs. This is generally a bad idea for the reasons stated above. However, overlaying a part of the rootfs often makes sense. /etc is a good example. This way applications that want their configurations in /etc can still have it that way -- and there is always a (hopefully) reasonable default configuration, should the configuration 'partition' get corrupted. So worst case the user can start over on configurations only. For applications and user data, these can and should be stored outside of the main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's -actually- /opt, the concept itself is good. So going back to image upgrade. The key here is that you need a way to update arbitrary images with arbitrary contents and a mechanisms that is smart enough to generate the update (vs a full image flash) to conserve bandwidth. I still contend it's up to the device to be able to configure the system on how to get the update and where to apply the update. The tooling (host and target) should simply assist with this. Delta updates need version information in order to know they're doing the right sequence of updating. Full updates don't, but should be sent in a format that limits "empty space", effectively send them as sparse files. On many devices you will need to flash as part of the download due to space limitations. And you need the ability to flash multiple partitions. maintenance / /usr data etc.. whatever it takes to either upgrade or restore the device. --Mark >> With the above information I'm proposing to use a separate partition for >> the configuration; this is because is more reliable and doesn't require >> big changes in the current architecture. >> >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > > You probably don't need to separate 1 and 3, all the code for system update > should easily fit into initramfs and just making /boot a bit larger would > allow you to store some backup rootfs. > > Also, you can swap 4 and 2 which will be useful if you're installing on > different sized storage devices, usually you know good enough the size of your > rootfs, but you probably want to leave more space for user data if there is an > opportunity to do so, that's just easier to do with data partition at the end. > > [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem @ 2015-11-24 13:47 ` Mark Hatle 0 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 13:47 UTC (permalink / raw) To: Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 4:39 AM, Roman Khimov wrote: > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in >> rootfs (example: /etc). > > That's the right solution, although to do it really right (at least IMO) you > need to implement the /usr merge [1] (and that's orthogonal to using or not > using systemd), which can also help you make your /usr read-only (because > that's just code and static data) with read-write / for user data of various > sorts. Why does merging /usr have anything to do with this? I've read the case for merging /usr and / and still don't understand why it "helps". The key is that if you have separate partitions for /usr and /, then you need to update both of them in sequence. Merging these two just seems like a lazy solution to people not wanting to deal with early boot being self-contained. Also having a separate / from /usr can help with '/' be your maintenance partition in some cases. >> 3. Have an OverlayFS for the rootfs or the partition that have the >> configuration. >> a. The pro is the configuration is "directly" in rootfs. >> b. The con is there is need to provide a custom init to guarantee the >> Overlay is mounted before the boot process. > > And this is the approach I would recommend not doing. I've used UnionFS for > thing like that (overlaying whole root file system) some 6 years ago, it > sounded nice and it kinda worked, but it wasn't difficult to make it fail > (just a little playing with power), we've even seen failures on production > devices, like when you have whiteout file for directory already written, but > don't have new files in it yet and that can completely ruin the system. > > Also, it usually works better when you don't have any changes in the lower > layer, but we're talking about updating it here, you can easily end up in a > situation where you have updated something in the rootfs but that was > overriden by upper layer and thus your user doesn't see any change. When using overlayfs, I'd strongly recommend not doing it over the entire rootfs. This is generally a bad idea for the reasons stated above. However, overlaying a part of the rootfs often makes sense. /etc is a good example. This way applications that want their configurations in /etc can still have it that way -- and there is always a (hopefully) reasonable default configuration, should the configuration 'partition' get corrupted. So worst case the user can start over on configurations only. For applications and user data, these can and should be stored outside of the main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's -actually- /opt, the concept itself is good. So going back to image upgrade. The key here is that you need a way to update arbitrary images with arbitrary contents and a mechanisms that is smart enough to generate the update (vs a full image flash) to conserve bandwidth. I still contend it's up to the device to be able to configure the system on how to get the update and where to apply the update. The tooling (host and target) should simply assist with this. Delta updates need version information in order to know they're doing the right sequence of updating. Full updates don't, but should be sent in a format that limits "empty space", effectively send them as sparse files. On many devices you will need to flash as part of the download due to space limitations. And you need the ability to flash multiple partitions. maintenance / /usr data etc.. whatever it takes to either upgrade or restore the device. --Mark >> With the above information I'm proposing to use a separate partition for >> the configuration; this is because is more reliable and doesn't require >> big changes in the current architecture. >> >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > > You probably don't need to separate 1 and 3, all the code for system update > should easily fit into initramfs and just making /boot a bit larger would > allow you to store some backup rootfs. > > Also, you can swap 4 and 2 which will be useful if you're installing on > different sized storage devices, usually you know good enough the size of your > rootfs, but you probably want to leave more space for user data if there is an > opportunity to do so, that's just easier to do with data partition at the end. > > [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] [oe] RFC: Reference updater filesystem 2015-11-24 13:47 ` [oe] " Mark Hatle (?) @ 2015-11-24 17:02 ` Lopez, Mariano -1 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 17:02 UTC (permalink / raw) To: Mark Hatle, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/2015 7:47 AM, Mark Hatle wrote: > On 11/24/15 4:39 AM, Roman Khimov wrote: >> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>> 1. Use a separate partition for the configuration. >>> a. The pro of this method is the partition is not touched during the >>> update. >>> b. The con of this method is the configuration is not directly in >>> rootfs (example: /etc). >> That's the right solution, although to do it really right (at least IMO) you >> need to implement the /usr merge [1] (and that's orthogonal to using or not >> using systemd), which can also help you make your /usr read-only (because >> that's just code and static data) with read-write / for user data of various >> sorts. > Why does merging /usr have anything to do with this? I've read the case for > merging /usr and / and still don't understand why it "helps". The key is that > if you have separate partitions for /usr and /, then you need to update both of > them in sequence. Merging these two just seems like a lazy solution to people > not wanting to deal with early boot being self-contained. > > Also having a separate / from /usr can help with '/' be your maintenance > partition in some cases. > >>> 3. Have an OverlayFS for the rootfs or the partition that have the >>> configuration. >>> a. The pro is the configuration is "directly" in rootfs. >>> b. The con is there is need to provide a custom init to guarantee the >>> Overlay is mounted before the boot process. >> And this is the approach I would recommend not doing. I've used UnionFS for >> thing like that (overlaying whole root file system) some 6 years ago, it >> sounded nice and it kinda worked, but it wasn't difficult to make it fail >> (just a little playing with power), we've even seen failures on production >> devices, like when you have whiteout file for directory already written, but >> don't have new files in it yet and that can completely ruin the system. >> >> Also, it usually works better when you don't have any changes in the lower >> layer, but we're talking about updating it here, you can easily end up in a >> situation where you have updated something in the rootfs but that was >> overriden by upper layer and thus your user doesn't see any change. > When using overlayfs, I'd strongly recommend not doing it over the entire > rootfs. This is generally a bad idea for the reasons stated above. > > However, overlaying a part of the rootfs often makes sense. /etc is a good > example. This way applications that want their configurations in /etc can still > have it that way -- and there is always a (hopefully) reasonable default > configuration, should the configuration 'partition' get corrupted. So worst > case the user can start over on configurations only. Do you know a way to mount the overlay before all the services start? I tried to do this, but the only reliable way to do it was using a custom init, I couldn't accomplish this using systemd or sysvnit. > > For applications and user data, these can and should be stored outside of the > main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's > -actually- /opt, the concept itself is good. > > > So going back to image upgrade. The key here is that you need a way to update > arbitrary images with arbitrary contents and a mechanisms that is smart enough > to generate the update (vs a full image flash) to conserve bandwidth. I was concerned about this too, not just bandwidth but resources in the target. Unfortunately I couldn't find an option that is generic enough to just provide the update. The idea is to integrate the tool into YP, not to develop a new one. Some of the tools that I checked needed to use btrfs partitions, need python in the target, or other constrains that make the update system impossible for a lot of targets. > > I still contend it's up to the device to be able to configure the system on how > to get the update and where to apply the update. The tooling (host and target) > should simply assist with this. > > Delta updates need version information in order to know they're doing the right > sequence of updating. > > Full updates don't, but should be sent in a format that limits "empty space", > effectively send them as sparse files. > > On many devices you will need to flash as part of the download due to space > limitations. The tool mentioned has this capability. > > And you need the ability to flash multiple partitions. > > maintenance > / > /usr > data > > etc.. whatever it takes to either upgrade or restore the device. Yes, that would be possible, the only limitation is that is not possible to flash the partition that is being used. > > --Mark > >>> With the above information I'm proposing to use a separate partition for >>> the configuration; this is because is more reliable and doesn't require >>> big changes in the current architecture. >>> >>> So, the idea is to have 4 partitions in the media: >>> 1. boot. This is the usual boot partition >>> 2. data. This will hold the configuration files. Not modified by updates. >>> 3. maintenance. This partition will be used to update rootfs. >>> 4. rootfs. Partition used for normal operation. >> You probably don't need to separate 1 and 3, all the code for system update >> should easily fit into initramfs and just making /boot a bit larger would >> allow you to store some backup rootfs. >> >> Also, you can swap 4 and 2 which will be useful if you're installing on >> different sized storage devices, usually you know good enough the size of your >> rootfs, but you probably want to leave more space for user data if there is an >> opportunity to do so, that's just easier to do with data partition at the end. >> >> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >> ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] [OE-core] RFC: Reference updater filesystem @ 2015-11-24 17:02 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 17:02 UTC (permalink / raw) To: Mark Hatle, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/2015 7:47 AM, Mark Hatle wrote: > On 11/24/15 4:39 AM, Roman Khimov wrote: >> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>> 1. Use a separate partition for the configuration. >>> a. The pro of this method is the partition is not touched during the >>> update. >>> b. The con of this method is the configuration is not directly in >>> rootfs (example: /etc). >> That's the right solution, although to do it really right (at least IMO) you >> need to implement the /usr merge [1] (and that's orthogonal to using or not >> using systemd), which can also help you make your /usr read-only (because >> that's just code and static data) with read-write / for user data of various >> sorts. > Why does merging /usr have anything to do with this? I've read the case for > merging /usr and / and still don't understand why it "helps". The key is that > if you have separate partitions for /usr and /, then you need to update both of > them in sequence. Merging these two just seems like a lazy solution to people > not wanting to deal with early boot being self-contained. > > Also having a separate / from /usr can help with '/' be your maintenance > partition in some cases. > >>> 3. Have an OverlayFS for the rootfs or the partition that have the >>> configuration. >>> a. The pro is the configuration is "directly" in rootfs. >>> b. The con is there is need to provide a custom init to guarantee the >>> Overlay is mounted before the boot process. >> And this is the approach I would recommend not doing. I've used UnionFS for >> thing like that (overlaying whole root file system) some 6 years ago, it >> sounded nice and it kinda worked, but it wasn't difficult to make it fail >> (just a little playing with power), we've even seen failures on production >> devices, like when you have whiteout file for directory already written, but >> don't have new files in it yet and that can completely ruin the system. >> >> Also, it usually works better when you don't have any changes in the lower >> layer, but we're talking about updating it here, you can easily end up in a >> situation where you have updated something in the rootfs but that was >> overriden by upper layer and thus your user doesn't see any change. > When using overlayfs, I'd strongly recommend not doing it over the entire > rootfs. This is generally a bad idea for the reasons stated above. > > However, overlaying a part of the rootfs often makes sense. /etc is a good > example. This way applications that want their configurations in /etc can still > have it that way -- and there is always a (hopefully) reasonable default > configuration, should the configuration 'partition' get corrupted. So worst > case the user can start over on configurations only. Do you know a way to mount the overlay before all the services start? I tried to do this, but the only reliable way to do it was using a custom init, I couldn't accomplish this using systemd or sysvnit. > > For applications and user data, these can and should be stored outside of the > main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's > -actually- /opt, the concept itself is good. > > > So going back to image upgrade. The key here is that you need a way to update > arbitrary images with arbitrary contents and a mechanisms that is smart enough > to generate the update (vs a full image flash) to conserve bandwidth. I was concerned about this too, not just bandwidth but resources in the target. Unfortunately I couldn't find an option that is generic enough to just provide the update. The idea is to integrate the tool into YP, not to develop a new one. Some of the tools that I checked needed to use btrfs partitions, need python in the target, or other constrains that make the update system impossible for a lot of targets. > > I still contend it's up to the device to be able to configure the system on how > to get the update and where to apply the update. The tooling (host and target) > should simply assist with this. > > Delta updates need version information in order to know they're doing the right > sequence of updating. > > Full updates don't, but should be sent in a format that limits "empty space", > effectively send them as sparse files. > > On many devices you will need to flash as part of the download due to space > limitations. The tool mentioned has this capability. > > And you need the ability to flash multiple partitions. > > maintenance > / > /usr > data > > etc.. whatever it takes to either upgrade or restore the device. Yes, that would be possible, the only limitation is that is not possible to flash the partition that is being used. > > --Mark > >>> With the above information I'm proposing to use a separate partition for >>> the configuration; this is because is more reliable and doesn't require >>> big changes in the current architecture. >>> >>> So, the idea is to have 4 partitions in the media: >>> 1. boot. This is the usual boot partition >>> 2. data. This will hold the configuration files. Not modified by updates. >>> 3. maintenance. This partition will be used to update rootfs. >>> 4. rootfs. Partition used for normal operation. >> You probably don't need to separate 1 and 3, all the code for system update >> should easily fit into initramfs and just making /boot a bit larger would >> allow you to store some backup rootfs. >> >> Also, you can swap 4 and 2 which will be useful if you're installing on >> different sized storage devices, usually you know good enough the size of your >> rootfs, but you probably want to leave more space for user data if there is an >> opportunity to do so, that's just easier to do with data partition at the end. >> >> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >> ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] [oe] RFC: Reference updater filesystem @ 2015-11-24 17:02 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 17:02 UTC (permalink / raw) To: Mark Hatle, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/2015 7:47 AM, Mark Hatle wrote: > On 11/24/15 4:39 AM, Roman Khimov wrote: >> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>> 1. Use a separate partition for the configuration. >>> a. The pro of this method is the partition is not touched during the >>> update. >>> b. The con of this method is the configuration is not directly in >>> rootfs (example: /etc). >> That's the right solution, although to do it really right (at least IMO) you >> need to implement the /usr merge [1] (and that's orthogonal to using or not >> using systemd), which can also help you make your /usr read-only (because >> that's just code and static data) with read-write / for user data of various >> sorts. > Why does merging /usr have anything to do with this? I've read the case for > merging /usr and / and still don't understand why it "helps". The key is that > if you have separate partitions for /usr and /, then you need to update both of > them in sequence. Merging these two just seems like a lazy solution to people > not wanting to deal with early boot being self-contained. > > Also having a separate / from /usr can help with '/' be your maintenance > partition in some cases. > >>> 3. Have an OverlayFS for the rootfs or the partition that have the >>> configuration. >>> a. The pro is the configuration is "directly" in rootfs. >>> b. The con is there is need to provide a custom init to guarantee the >>> Overlay is mounted before the boot process. >> And this is the approach I would recommend not doing. I've used UnionFS for >> thing like that (overlaying whole root file system) some 6 years ago, it >> sounded nice and it kinda worked, but it wasn't difficult to make it fail >> (just a little playing with power), we've even seen failures on production >> devices, like when you have whiteout file for directory already written, but >> don't have new files in it yet and that can completely ruin the system. >> >> Also, it usually works better when you don't have any changes in the lower >> layer, but we're talking about updating it here, you can easily end up in a >> situation where you have updated something in the rootfs but that was >> overriden by upper layer and thus your user doesn't see any change. > When using overlayfs, I'd strongly recommend not doing it over the entire > rootfs. This is generally a bad idea for the reasons stated above. > > However, overlaying a part of the rootfs often makes sense. /etc is a good > example. This way applications that want their configurations in /etc can still > have it that way -- and there is always a (hopefully) reasonable default > configuration, should the configuration 'partition' get corrupted. So worst > case the user can start over on configurations only. Do you know a way to mount the overlay before all the services start? I tried to do this, but the only reliable way to do it was using a custom init, I couldn't accomplish this using systemd or sysvnit. > > For applications and user data, these can and should be stored outside of the > main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's > -actually- /opt, the concept itself is good. > > > So going back to image upgrade. The key here is that you need a way to update > arbitrary images with arbitrary contents and a mechanisms that is smart enough > to generate the update (vs a full image flash) to conserve bandwidth. I was concerned about this too, not just bandwidth but resources in the target. Unfortunately I couldn't find an option that is generic enough to just provide the update. The idea is to integrate the tool into YP, not to develop a new one. Some of the tools that I checked needed to use btrfs partitions, need python in the target, or other constrains that make the update system impossible for a lot of targets. > > I still contend it's up to the device to be able to configure the system on how > to get the update and where to apply the update. The tooling (host and target) > should simply assist with this. > > Delta updates need version information in order to know they're doing the right > sequence of updating. > > Full updates don't, but should be sent in a format that limits "empty space", > effectively send them as sparse files. > > On many devices you will need to flash as part of the download due to space > limitations. The tool mentioned has this capability. > > And you need the ability to flash multiple partitions. > > maintenance > / > /usr > data > > etc.. whatever it takes to either upgrade or restore the device. Yes, that would be possible, the only limitation is that is not possible to flash the partition that is being used. > > --Mark > >>> With the above information I'm proposing to use a separate partition for >>> the configuration; this is because is more reliable and doesn't require >>> big changes in the current architecture. >>> >>> So, the idea is to have 4 partitions in the media: >>> 1. boot. This is the usual boot partition >>> 2. data. This will hold the configuration files. Not modified by updates. >>> 3. maintenance. This partition will be used to update rootfs. >>> 4. rootfs. Partition used for normal operation. >> You probably don't need to separate 1 and 3, all the code for system update >> should easily fit into initramfs and just making /boot a bit larger would >> allow you to store some backup rootfs. >> >> Also, you can swap 4 and 2 which will be useful if you're installing on >> different sized storage devices, usually you know good enough the size of your >> rootfs, but you probably want to leave more space for user data if there is an >> opportunity to do so, that's just easier to do with data partition at the end. >> >> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >> ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] [oe] RFC: Reference updater filesystem 2015-11-24 17:02 ` [yocto] [oe] " Lopez, Mariano (?) @ 2015-11-24 17:13 ` Mark Hatle -1 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 17:13 UTC (permalink / raw) To: Lopez, Mariano, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 11:02 AM, Lopez, Mariano wrote: > > > On 11/24/2015 7:47 AM, Mark Hatle wrote: >> On 11/24/15 4:39 AM, Roman Khimov wrote: >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>> 1. Use a separate partition for the configuration. >>>> a. The pro of this method is the partition is not touched during the >>>> update. >>>> b. The con of this method is the configuration is not directly in >>>> rootfs (example: /etc). >>> That's the right solution, although to do it really right (at least IMO) you >>> need to implement the /usr merge [1] (and that's orthogonal to using or not >>> using systemd), which can also help you make your /usr read-only (because >>> that's just code and static data) with read-write / for user data of various >>> sorts. >> Why does merging /usr have anything to do with this? I've read the case for >> merging /usr and / and still don't understand why it "helps". The key is that >> if you have separate partitions for /usr and /, then you need to update both of >> them in sequence. Merging these two just seems like a lazy solution to people >> not wanting to deal with early boot being self-contained. >> >> Also having a separate / from /usr can help with '/' be your maintenance >> partition in some cases. >> >>>> 3. Have an OverlayFS for the rootfs or the partition that have the >>>> configuration. >>>> a. The pro is the configuration is "directly" in rootfs. >>>> b. The con is there is need to provide a custom init to guarantee the >>>> Overlay is mounted before the boot process. >>> And this is the approach I would recommend not doing. I've used UnionFS for >>> thing like that (overlaying whole root file system) some 6 years ago, it >>> sounded nice and it kinda worked, but it wasn't difficult to make it fail >>> (just a little playing with power), we've even seen failures on production >>> devices, like when you have whiteout file for directory already written, but >>> don't have new files in it yet and that can completely ruin the system. >>> >>> Also, it usually works better when you don't have any changes in the lower >>> layer, but we're talking about updating it here, you can easily end up in a >>> situation where you have updated something in the rootfs but that was >>> overriden by upper layer and thus your user doesn't see any change. >> When using overlayfs, I'd strongly recommend not doing it over the entire >> rootfs. This is generally a bad idea for the reasons stated above. >> >> However, overlaying a part of the rootfs often makes sense. /etc is a good >> example. This way applications that want their configurations in /etc can still >> have it that way -- and there is always a (hopefully) reasonable default >> configuration, should the configuration 'partition' get corrupted. So worst >> case the user can start over on configurations only. > > Do you know a way to mount the overlay before all the services start? I > tried to do this, but the only reliable way to do it was using a custom > init, I couldn't accomplish this using systemd or sysvnit. In the past I've done this with an initrd, with a custom /sbin/init that mounted and then did a reexec for the real init system or ordered things in such a way that the overlay happened -very- early. >> >> For applications and user data, these can and should be stored outside of the >> main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's >> -actually- /opt, the concept itself is good. >> >> >> So going back to image upgrade. The key here is that you need a way to update >> arbitrary images with arbitrary contents and a mechanisms that is smart enough >> to generate the update (vs a full image flash) to conserve bandwidth. > > I was concerned about this too, not just bandwidth but resources in the > target. Unfortunately I couldn't find an option that is generic enough > to just provide the update. The idea is to integrate the tool into YP, > not to develop a new one. Some of the tools that I checked needed to use > btrfs partitions, need python in the target, or other constrains that > make the update system impossible for a lot of targets. Yup. I just want to make sure people know one tool isn't going to do everything.. and the integration of a single tool shouldn't restrict people for doing other things with custom tooling. --Mark >> >> I still contend it's up to the device to be able to configure the system on how >> to get the update and where to apply the update. The tooling (host and target) >> should simply assist with this. >> >> Delta updates need version information in order to know they're doing the right >> sequence of updating. >> >> Full updates don't, but should be sent in a format that limits "empty space", >> effectively send them as sparse files. >> >> On many devices you will need to flash as part of the download due to space >> limitations. > > The tool mentioned has this capability. > >> >> And you need the ability to flash multiple partitions. >> >> maintenance >> / >> /usr >> data >> >> etc.. whatever it takes to either upgrade or restore the device. > > Yes, that would be possible, the only limitation is that is not possible > to flash the partition that is being used. > >> >> --Mark >> >>>> With the above information I'm proposing to use a separate partition for >>>> the configuration; this is because is more reliable and doesn't require >>>> big changes in the current architecture. >>>> >>>> So, the idea is to have 4 partitions in the media: >>>> 1. boot. This is the usual boot partition >>>> 2. data. This will hold the configuration files. Not modified by updates. >>>> 3. maintenance. This partition will be used to update rootfs. >>>> 4. rootfs. Partition used for normal operation. >>> You probably don't need to separate 1 and 3, all the code for system update >>> should easily fit into initramfs and just making /boot a bit larger would >>> allow you to store some backup rootfs. >>> >>> Also, you can swap 4 and 2 which will be useful if you're installing on >>> different sized storage devices, usually you know good enough the size of your >>> rootfs, but you probably want to leave more space for user data if there is an >>> opportunity to do so, that's just easier to do with data partition at the end. >>> >>> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >>> > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] [OE-core] RFC: Reference updater filesystem @ 2015-11-24 17:13 ` Mark Hatle 0 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 17:13 UTC (permalink / raw) To: Lopez, Mariano, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 11:02 AM, Lopez, Mariano wrote: > > > On 11/24/2015 7:47 AM, Mark Hatle wrote: >> On 11/24/15 4:39 AM, Roman Khimov wrote: >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>> 1. Use a separate partition for the configuration. >>>> a. The pro of this method is the partition is not touched during the >>>> update. >>>> b. The con of this method is the configuration is not directly in >>>> rootfs (example: /etc). >>> That's the right solution, although to do it really right (at least IMO) you >>> need to implement the /usr merge [1] (and that's orthogonal to using or not >>> using systemd), which can also help you make your /usr read-only (because >>> that's just code and static data) with read-write / for user data of various >>> sorts. >> Why does merging /usr have anything to do with this? I've read the case for >> merging /usr and / and still don't understand why it "helps". The key is that >> if you have separate partitions for /usr and /, then you need to update both of >> them in sequence. Merging these two just seems like a lazy solution to people >> not wanting to deal with early boot being self-contained. >> >> Also having a separate / from /usr can help with '/' be your maintenance >> partition in some cases. >> >>>> 3. Have an OverlayFS for the rootfs or the partition that have the >>>> configuration. >>>> a. The pro is the configuration is "directly" in rootfs. >>>> b. The con is there is need to provide a custom init to guarantee the >>>> Overlay is mounted before the boot process. >>> And this is the approach I would recommend not doing. I've used UnionFS for >>> thing like that (overlaying whole root file system) some 6 years ago, it >>> sounded nice and it kinda worked, but it wasn't difficult to make it fail >>> (just a little playing with power), we've even seen failures on production >>> devices, like when you have whiteout file for directory already written, but >>> don't have new files in it yet and that can completely ruin the system. >>> >>> Also, it usually works better when you don't have any changes in the lower >>> layer, but we're talking about updating it here, you can easily end up in a >>> situation where you have updated something in the rootfs but that was >>> overriden by upper layer and thus your user doesn't see any change. >> When using overlayfs, I'd strongly recommend not doing it over the entire >> rootfs. This is generally a bad idea for the reasons stated above. >> >> However, overlaying a part of the rootfs often makes sense. /etc is a good >> example. This way applications that want their configurations in /etc can still >> have it that way -- and there is always a (hopefully) reasonable default >> configuration, should the configuration 'partition' get corrupted. So worst >> case the user can start over on configurations only. > > Do you know a way to mount the overlay before all the services start? I > tried to do this, but the only reliable way to do it was using a custom > init, I couldn't accomplish this using systemd or sysvnit. In the past I've done this with an initrd, with a custom /sbin/init that mounted and then did a reexec for the real init system or ordered things in such a way that the overlay happened -very- early. >> >> For applications and user data, these can and should be stored outside of the >> main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's >> -actually- /opt, the concept itself is good. >> >> >> So going back to image upgrade. The key here is that you need a way to update >> arbitrary images with arbitrary contents and a mechanisms that is smart enough >> to generate the update (vs a full image flash) to conserve bandwidth. > > I was concerned about this too, not just bandwidth but resources in the > target. Unfortunately I couldn't find an option that is generic enough > to just provide the update. The idea is to integrate the tool into YP, > not to develop a new one. Some of the tools that I checked needed to use > btrfs partitions, need python in the target, or other constrains that > make the update system impossible for a lot of targets. Yup. I just want to make sure people know one tool isn't going to do everything.. and the integration of a single tool shouldn't restrict people for doing other things with custom tooling. --Mark >> >> I still contend it's up to the device to be able to configure the system on how >> to get the update and where to apply the update. The tooling (host and target) >> should simply assist with this. >> >> Delta updates need version information in order to know they're doing the right >> sequence of updating. >> >> Full updates don't, but should be sent in a format that limits "empty space", >> effectively send them as sparse files. >> >> On many devices you will need to flash as part of the download due to space >> limitations. > > The tool mentioned has this capability. > >> >> And you need the ability to flash multiple partitions. >> >> maintenance >> / >> /usr >> data >> >> etc.. whatever it takes to either upgrade or restore the device. > > Yes, that would be possible, the only limitation is that is not possible > to flash the partition that is being used. > >> >> --Mark >> >>>> With the above information I'm proposing to use a separate partition for >>>> the configuration; this is because is more reliable and doesn't require >>>> big changes in the current architecture. >>>> >>>> So, the idea is to have 4 partitions in the media: >>>> 1. boot. This is the usual boot partition >>>> 2. data. This will hold the configuration files. Not modified by updates. >>>> 3. maintenance. This partition will be used to update rootfs. >>>> 4. rootfs. Partition used for normal operation. >>> You probably don't need to separate 1 and 3, all the code for system update >>> should easily fit into initramfs and just making /boot a bit larger would >>> allow you to store some backup rootfs. >>> >>> Also, you can swap 4 and 2 which will be useful if you're installing on >>> different sized storage devices, usually you know good enough the size of your >>> rootfs, but you probably want to leave more space for user data if there is an >>> opportunity to do so, that's just easier to do with data partition at the end. >>> >>> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >>> > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [yocto] [oe] RFC: Reference updater filesystem @ 2015-11-24 17:13 ` Mark Hatle 0 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 17:13 UTC (permalink / raw) To: Lopez, Mariano, Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/15 11:02 AM, Lopez, Mariano wrote: > > > On 11/24/2015 7:47 AM, Mark Hatle wrote: >> On 11/24/15 4:39 AM, Roman Khimov wrote: >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>> 1. Use a separate partition for the configuration. >>>> a. The pro of this method is the partition is not touched during the >>>> update. >>>> b. The con of this method is the configuration is not directly in >>>> rootfs (example: /etc). >>> That's the right solution, although to do it really right (at least IMO) you >>> need to implement the /usr merge [1] (and that's orthogonal to using or not >>> using systemd), which can also help you make your /usr read-only (because >>> that's just code and static data) with read-write / for user data of various >>> sorts. >> Why does merging /usr have anything to do with this? I've read the case for >> merging /usr and / and still don't understand why it "helps". The key is that >> if you have separate partitions for /usr and /, then you need to update both of >> them in sequence. Merging these two just seems like a lazy solution to people >> not wanting to deal with early boot being self-contained. >> >> Also having a separate / from /usr can help with '/' be your maintenance >> partition in some cases. >> >>>> 3. Have an OverlayFS for the rootfs or the partition that have the >>>> configuration. >>>> a. The pro is the configuration is "directly" in rootfs. >>>> b. The con is there is need to provide a custom init to guarantee the >>>> Overlay is mounted before the boot process. >>> And this is the approach I would recommend not doing. I've used UnionFS for >>> thing like that (overlaying whole root file system) some 6 years ago, it >>> sounded nice and it kinda worked, but it wasn't difficult to make it fail >>> (just a little playing with power), we've even seen failures on production >>> devices, like when you have whiteout file for directory already written, but >>> don't have new files in it yet and that can completely ruin the system. >>> >>> Also, it usually works better when you don't have any changes in the lower >>> layer, but we're talking about updating it here, you can easily end up in a >>> situation where you have updated something in the rootfs but that was >>> overriden by upper layer and thus your user doesn't see any change. >> When using overlayfs, I'd strongly recommend not doing it over the entire >> rootfs. This is generally a bad idea for the reasons stated above. >> >> However, overlaying a part of the rootfs often makes sense. /etc is a good >> example. This way applications that want their configurations in /etc can still >> have it that way -- and there is always a (hopefully) reasonable default >> configuration, should the configuration 'partition' get corrupted. So worst >> case the user can start over on configurations only. > > Do you know a way to mount the overlay before all the services start? I > tried to do this, but the only reliable way to do it was using a custom > init, I couldn't accomplish this using systemd or sysvnit. In the past I've done this with an initrd, with a custom /sbin/init that mounted and then did a reexec for the real init system or ordered things in such a way that the overlay happened -very- early. >> >> For applications and user data, these can and should be stored outside of the >> main rootfs. The FHS/LSB recomment '/opt', but while it doesn't matter if it's >> -actually- /opt, the concept itself is good. >> >> >> So going back to image upgrade. The key here is that you need a way to update >> arbitrary images with arbitrary contents and a mechanisms that is smart enough >> to generate the update (vs a full image flash) to conserve bandwidth. > > I was concerned about this too, not just bandwidth but resources in the > target. Unfortunately I couldn't find an option that is generic enough > to just provide the update. The idea is to integrate the tool into YP, > not to develop a new one. Some of the tools that I checked needed to use > btrfs partitions, need python in the target, or other constrains that > make the update system impossible for a lot of targets. Yup. I just want to make sure people know one tool isn't going to do everything.. and the integration of a single tool shouldn't restrict people for doing other things with custom tooling. --Mark >> >> I still contend it's up to the device to be able to configure the system on how >> to get the update and where to apply the update. The tooling (host and target) >> should simply assist with this. >> >> Delta updates need version information in order to know they're doing the right >> sequence of updating. >> >> Full updates don't, but should be sent in a format that limits "empty space", >> effectively send them as sparse files. >> >> On many devices you will need to flash as part of the download due to space >> limitations. > > The tool mentioned has this capability. > >> >> And you need the ability to flash multiple partitions. >> >> maintenance >> / >> /usr >> data >> >> etc.. whatever it takes to either upgrade or restore the device. > > Yes, that would be possible, the only limitation is that is not possible > to flash the partition that is being used. > >> >> --Mark >> >>>> With the above information I'm proposing to use a separate partition for >>>> the configuration; this is because is more reliable and doesn't require >>>> big changes in the current architecture. >>>> >>>> So, the idea is to have 4 partitions in the media: >>>> 1. boot. This is the usual boot partition >>>> 2. data. This will hold the configuration files. Not modified by updates. >>>> 3. maintenance. This partition will be used to update rootfs. >>>> 4. rootfs. Partition used for normal operation. >>> You probably don't need to separate 1 and 3, all the code for system update >>> should easily fit into initramfs and just making /boot a bit larger would >>> allow you to store some backup rootfs. >>> >>> Also, you can swap 4 and 2 which will be useful if you're installing on >>> different sized storage devices, usually you know good enough the size of your >>> rootfs, but you probably want to leave more space for user data if there is an >>> opportunity to do so, that's just easier to do with data partition at the end. >>> >>> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ >>> > ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-24 13:47 ` [oe] " Mark Hatle @ 2015-11-24 18:05 ` Roman Khimov -1 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:05 UTC (permalink / raw) To: Mark Hatle, OE-core; +Cc: openembedded-devel В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: > On 11/24/15 4:39 AM, Roman Khimov wrote: > > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > >> 1. Use a separate partition for the configuration. > >> > >> a. The pro of this method is the partition is not touched during the > >> > >> update. > >> > >> b. The con of this method is the configuration is not directly in > >> > >> rootfs (example: /etc). > > > > That's the right solution, although to do it really right (at least IMO) > > you need to implement the /usr merge [1] (and that's orthogonal to using > > or not using systemd), which can also help you make your /usr read-only > > (because that's just code and static data) with read-write / for user > > data of various sorts. > > Why does merging /usr have anything to do with this? I've read the case for > merging /usr and / and still don't understand why it "helps". The key is > that if you have separate partitions for /usr and /, then you need to > update both of them in sequence. Merging these two just seems like a lazy > solution to people not wanting to deal with early boot being > self-contained. It helps in that you can achieve a clear separation of your software and users data (whatever that is, even if that's just some configuration files) and only update your part (which is /usr). > So going back to image upgrade. The key here is that you need a way to > update arbitrary images with arbitrary contents and a mechanisms that is > smart enough to generate the update (vs a full image flash) to conserve > bandwidth. In my experience, size is almost not an issue these days, at least if we're talking about something less than 100 MB, but updating the whole image is more reliable and easier to manage. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-11-24 18:05 ` Roman Khimov 0 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:05 UTC (permalink / raw) To: Mark Hatle, OE-core; +Cc: openembedded-devel В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: > On 11/24/15 4:39 AM, Roman Khimov wrote: > > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > >> 1. Use a separate partition for the configuration. > >> > >> a. The pro of this method is the partition is not touched during the > >> > >> update. > >> > >> b. The con of this method is the configuration is not directly in > >> > >> rootfs (example: /etc). > > > > That's the right solution, although to do it really right (at least IMO) > > you need to implement the /usr merge [1] (and that's orthogonal to using > > or not using systemd), which can also help you make your /usr read-only > > (because that's just code and static data) with read-write / for user > > data of various sorts. > > Why does merging /usr have anything to do with this? I've read the case for > merging /usr and / and still don't understand why it "helps". The key is > that if you have separate partitions for /usr and /, then you need to > update both of them in sequence. Merging these two just seems like a lazy > solution to people not wanting to deal with early boot being > self-contained. It helps in that you can achieve a clear separation of your software and users data (whatever that is, even if that's just some configuration files) and only update your part (which is /usr). > So going back to image upgrade. The key here is that you need a way to > update arbitrary images with arbitrary contents and a mechanisms that is > smart enough to generate the update (vs a full image flash) to conserve > bandwidth. In my experience, size is almost not an issue these days, at least if we're talking about something less than 100 MB, but updating the whole image is more reliable and easier to manage. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-24 18:05 ` [OE-core] " Roman Khimov @ 2015-11-24 18:27 ` Mark Hatle -1 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 18:27 UTC (permalink / raw) To: roman, OE-core; +Cc: openembedded-devel On 11/24/15 12:05 PM, Roman Khimov wrote: > В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: >> On 11/24/15 4:39 AM, Roman Khimov wrote: >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>> 1. Use a separate partition for the configuration. >>>> >>>> a. The pro of this method is the partition is not touched during the >>>> >>>> update. >>>> >>>> b. The con of this method is the configuration is not directly in >>>> >>>> rootfs (example: /etc). >>> >>> That's the right solution, although to do it really right (at least IMO) >>> you need to implement the /usr merge [1] (and that's orthogonal to using >>> or not using systemd), which can also help you make your /usr read-only >>> (because that's just code and static data) with read-write / for user >>> data of various sorts. >> >> Why does merging /usr have anything to do with this? I've read the case for >> merging /usr and / and still don't understand why it "helps". The key is >> that if you have separate partitions for /usr and /, then you need to >> update both of them in sequence. Merging these two just seems like a lazy >> solution to people not wanting to deal with early boot being >> self-contained. > > It helps in that you can achieve a clear separation of your software and users > data (whatever that is, even if that's just some configuration files) and only > update your part (which is /usr). This can easily be archived using different partitions like /opt as well. >> So going back to image upgrade. The key here is that you need a way to >> update arbitrary images with arbitrary contents and a mechanisms that is >> smart enough to generate the update (vs a full image flash) to conserve >> bandwidth. > > In my experience, size is almost not an issue these days, at least if we're > talking about something less than 100 MB, but updating the whole image is more > reliable and easier to manage. > Bandwidth is a big deal in areas that are not serviced by anything but very GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor as is overall bandwidth. Again, it depends on the product and what you are building if this matters. The point is we don't want to make some assumptions that preclude alternative implementations. --Mark ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-11-24 18:27 ` Mark Hatle 0 siblings, 0 replies; 35+ messages in thread From: Mark Hatle @ 2015-11-24 18:27 UTC (permalink / raw) To: roman, OE-core; +Cc: openembedded-devel On 11/24/15 12:05 PM, Roman Khimov wrote: > В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: >> On 11/24/15 4:39 AM, Roman Khimov wrote: >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>> 1. Use a separate partition for the configuration. >>>> >>>> a. The pro of this method is the partition is not touched during the >>>> >>>> update. >>>> >>>> b. The con of this method is the configuration is not directly in >>>> >>>> rootfs (example: /etc). >>> >>> That's the right solution, although to do it really right (at least IMO) >>> you need to implement the /usr merge [1] (and that's orthogonal to using >>> or not using systemd), which can also help you make your /usr read-only >>> (because that's just code and static data) with read-write / for user >>> data of various sorts. >> >> Why does merging /usr have anything to do with this? I've read the case for >> merging /usr and / and still don't understand why it "helps". The key is >> that if you have separate partitions for /usr and /, then you need to >> update both of them in sequence. Merging these two just seems like a lazy >> solution to people not wanting to deal with early boot being >> self-contained. > > It helps in that you can achieve a clear separation of your software and users > data (whatever that is, even if that's just some configuration files) and only > update your part (which is /usr). This can easily be archived using different partitions like /opt as well. >> So going back to image upgrade. The key here is that you need a way to >> update arbitrary images with arbitrary contents and a mechanisms that is >> smart enough to generate the update (vs a full image flash) to conserve >> bandwidth. > > In my experience, size is almost not an issue these days, at least if we're > talking about something less than 100 MB, but updating the whole image is more > reliable and easier to manage. > Bandwidth is a big deal in areas that are not serviced by anything but very GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor as is overall bandwidth. Again, it depends on the product and what you are building if this matters. The point is we don't want to make some assumptions that preclude alternative implementations. --Mark ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-24 18:27 ` [OE-core] " Mark Hatle @ 2015-11-24 18:47 ` Roman Khimov -1 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:47 UTC (permalink / raw) To: Mark Hatle; +Cc: openembedded-devel, OE-core В письме от 24 ноября 2015 12:27:30 пользователь Mark Hatle написал: > On 11/24/15 12:05 PM, Roman Khimov wrote: > > В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: > >> On 11/24/15 4:39 AM, Roman Khimov wrote: > >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > >>>> 1. Use a separate partition for the configuration. > >>>> > >>>> a. The pro of this method is the partition is not touched during the > >>>> > >>>> update. > >>>> > >>>> b. The con of this method is the configuration is not directly in > >>>> > >>>> rootfs (example: /etc). > >>> > >>> That's the right solution, although to do it really right (at least IMO) > >>> you need to implement the /usr merge [1] (and that's orthogonal to using > >>> or not using systemd), which can also help you make your /usr read-only > >>> (because that's just code and static data) with read-write / for user > >>> data of various sorts. > >> > >> Why does merging /usr have anything to do with this? I've read the case > >> for merging /usr and / and still don't understand why it "helps". The > >> key is that if you have separate partitions for /usr and /, then you > >> need to update both of them in sequence. Merging these two just seems > >> like a lazy solution to people not wanting to deal with early boot being > >> self-contained. > > > > It helps in that you can achieve a clear separation of your software and > > users data (whatever that is, even if that's just some configuration > > files) and only update your part (which is /usr). > > This can easily be archived using different partitions like /opt as well. Usually it leads to more complicated partitioning and more complicated startup (as well as update) procedure. Of course it can vary from system to system, but in my experience /usr merge was exactly the thing needed to clearly separate and simplify things. One way or another you want to have your code and users data to be separated, /usr merge makes it quite simple. > Bandwidth is a big deal in areas that are not serviced by anything but very > GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor > as is overall bandwidth. > > Again, it depends on the product and what you are building if this matters. > > The point is we don't want to make some assumptions that preclude > alternative implementations. Sure. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-11-24 18:47 ` Roman Khimov 0 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:47 UTC (permalink / raw) To: Mark Hatle; +Cc: openembedded-devel, OE-core В письме от 24 ноября 2015 12:27:30 пользователь Mark Hatle написал: > On 11/24/15 12:05 PM, Roman Khimov wrote: > > В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: > >> On 11/24/15 4:39 AM, Roman Khimov wrote: > >>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: > >>>> 1. Use a separate partition for the configuration. > >>>> > >>>> a. The pro of this method is the partition is not touched during the > >>>> > >>>> update. > >>>> > >>>> b. The con of this method is the configuration is not directly in > >>>> > >>>> rootfs (example: /etc). > >>> > >>> That's the right solution, although to do it really right (at least IMO) > >>> you need to implement the /usr merge [1] (and that's orthogonal to using > >>> or not using systemd), which can also help you make your /usr read-only > >>> (because that's just code and static data) with read-write / for user > >>> data of various sorts. > >> > >> Why does merging /usr have anything to do with this? I've read the case > >> for merging /usr and / and still don't understand why it "helps". The > >> key is that if you have separate partitions for /usr and /, then you > >> need to update both of them in sequence. Merging these two just seems > >> like a lazy solution to people not wanting to deal with early boot being > >> self-contained. > > > > It helps in that you can achieve a clear separation of your software and > > users data (whatever that is, even if that's just some configuration > > files) and only update your part (which is /usr). > > This can easily be archived using different partitions like /opt as well. Usually it leads to more complicated partitioning and more complicated startup (as well as update) procedure. Of course it can vary from system to system, but in my experience /usr merge was exactly the thing needed to clearly separate and simplify things. One way or another you want to have your code and users data to be separated, /usr merge makes it quite simple. > Bandwidth is a big deal in areas that are not serviced by anything but very > GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor > as is overall bandwidth. > > Again, it depends on the product and what you are building if this matters. > > The point is we don't want to make some assumptions that preclude > alternative implementations. Sure. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-24 18:47 ` [OE-core] " Roman Khimov @ 2015-12-03 16:45 ` Mariano Lopez -1 siblings, 0 replies; 35+ messages in thread From: Mariano Lopez @ 2015-12-03 16:45 UTC (permalink / raw) To: roman, Mark Hatle; +Cc: openembedded-devel, OE-core On 11/24/2015 12:47 PM, Roman Khimov wrote: > В письме от 24 ноября 2015 12:27:30 пользователь Mark Hatle написал: >> On 11/24/15 12:05 PM, Roman Khimov wrote: >>> В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: >>>> On 11/24/15 4:39 AM, Roman Khimov wrote: >>>>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>>>> 1. Use a separate partition for the configuration. >>>>>> >>>>>> a. The pro of this method is the partition is not touched during the >>>>>> >>>>>> update. >>>>>> >>>>>> b. The con of this method is the configuration is not directly in >>>>>> >>>>>> rootfs (example: /etc). >>>>> That's the right solution, although to do it really right (at least IMO) >>>>> you need to implement the /usr merge [1] (and that's orthogonal to using >>>>> or not using systemd), which can also help you make your /usr read-only >>>>> (because that's just code and static data) with read-write / for user >>>>> data of various sorts. >>>> Why does merging /usr have anything to do with this? I've read the case >>>> for merging /usr and / and still don't understand why it "helps". The >>>> key is that if you have separate partitions for /usr and /, then you >>>> need to update both of them in sequence. Merging these two just seems >>>> like a lazy solution to people not wanting to deal with early boot being >>>> self-contained. >>> It helps in that you can achieve a clear separation of your software and >>> users data (whatever that is, even if that's just some configuration >>> files) and only update your part (which is /usr). >> This can easily be archived using different partitions like /opt as well. > Usually it leads to more complicated partitioning and more complicated startup > (as well as update) procedure. Of course it can vary from system to system, > but in my experience /usr merge was exactly the thing needed to clearly > separate and simplify things. One way or another you want to have your code > and users data to be separated, /usr merge makes it quite simple. > >> Bandwidth is a big deal in areas that are not serviced by anything but very >> GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor >> as is overall bandwidth. >> >> Again, it depends on the product and what you are building if this matters. >> >> The point is we don't want to make some assumptions that preclude >> alternative implementations. > Sure. > Thanks to all for your input. The conclusion of this thread is: 1. One size doesn't fit all. 2. Most of the people was fine with the image based update. 3. The recommended way to keep the configuration is to have a separated data partition. 4. The partition scheme would be: 1. boot. This is the usual boot partition 2. rootfs. Partition used for normal operation. 3. maintenance. This partition will be used to update rootfs. 4. data. This will hold the configuration files. Not modified by updates. -- Mariano Lopez ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-12-03 16:45 ` Mariano Lopez 0 siblings, 0 replies; 35+ messages in thread From: Mariano Lopez @ 2015-12-03 16:45 UTC (permalink / raw) To: roman, Mark Hatle; +Cc: openembedded-devel, OE-core On 11/24/2015 12:47 PM, Roman Khimov wrote: > В письме от 24 ноября 2015 12:27:30 пользователь Mark Hatle написал: >> On 11/24/15 12:05 PM, Roman Khimov wrote: >>> В письме от 24 ноября 2015 07:47:38 пользователь Mark Hatle написал: >>>> On 11/24/15 4:39 AM, Roman Khimov wrote: >>>>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >>>>>> 1. Use a separate partition for the configuration. >>>>>> >>>>>> a. The pro of this method is the partition is not touched during the >>>>>> >>>>>> update. >>>>>> >>>>>> b. The con of this method is the configuration is not directly in >>>>>> >>>>>> rootfs (example: /etc). >>>>> That's the right solution, although to do it really right (at least IMO) >>>>> you need to implement the /usr merge [1] (and that's orthogonal to using >>>>> or not using systemd), which can also help you make your /usr read-only >>>>> (because that's just code and static data) with read-write / for user >>>>> data of various sorts. >>>> Why does merging /usr have anything to do with this? I've read the case >>>> for merging /usr and / and still don't understand why it "helps". The >>>> key is that if you have separate partitions for /usr and /, then you >>>> need to update both of them in sequence. Merging these two just seems >>>> like a lazy solution to people not wanting to deal with early boot being >>>> self-contained. >>> It helps in that you can achieve a clear separation of your software and >>> users data (whatever that is, even if that's just some configuration >>> files) and only update your part (which is /usr). >> This can easily be archived using different partitions like /opt as well. > Usually it leads to more complicated partitioning and more complicated startup > (as well as update) procedure. Of course it can vary from system to system, > but in my experience /usr merge was exactly the thing needed to clearly > separate and simplify things. One way or another you want to have your code > and users data to be separated, /usr merge makes it quite simple. > >> Bandwidth is a big deal in areas that are not serviced by anything but very >> GPRS or worse. So 100 MB is even too big in some cases. Cost is a factor >> as is overall bandwidth. >> >> Again, it depends on the product and what you are building if this matters. >> >> The point is we don't want to make some assumptions that preclude >> alternative implementations. > Sure. > Thanks to all for your input. The conclusion of this thread is: 1. One size doesn't fit all. 2. Most of the people was fine with the image based update. 3. The recommended way to keep the configuration is to have a separated data partition. 4. The partition scheme would be: 1. boot. This is the usual boot partition 2. rootfs. Partition used for normal operation. 3. maintenance. This partition will be used to update rootfs. 4. data. This will hold the configuration files. Not modified by updates. -- Mariano Lopez ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-12-03 16:45 ` [OE-core] " Mariano Lopez @ 2015-12-06 11:34 ` Jens Rehsack -1 siblings, 0 replies; 35+ messages in thread From: Jens Rehsack @ 2015-12-06 11:34 UTC (permalink / raw) To: Mariano Lopez; +Cc: openembedded-devel, OE-core > Am 03.12.2015 um 17:45 schrieb Mariano Lopez <mariano.lopez@linux.intel.com>: > > [...] > > Thanks to all for your input. The conclusion of this thread is: > > 1. One size doesn't fit all. > 2. Most of the people was fine with the image based update. Where is the image stored? I read over referred http://sbabic.github.io/swupdate/ and realized that the update image must include a stripped rootfs and the entire software for rootfs as well as probably the recoveryfs/maintfs and some logic for updating the data volume (eg. migrator for database of business logic). This can be very large ... and must be stored somewhere. Next question: how bullet proof is the swupdate? Out process is designed (because of typical users can't interact) to can survive surprising power outage. Unfortunately we didn't manage to split out u-boot into separate updatable parts, so we can't benefit from it - but technically (if one has implemented for heir targets), our script updates and continues after power comes back. I think, beside the question of having an image containing the update (we have such an add-on, too - for legacy devices without he modern Yocto based firmware), the question of stability should be evaluated, too. > 3. The recommended way to keep the configuration is to have a separated data partition. > 4. The partition scheme would be: > 1. boot. This is the usual boot partition > 2. rootfs. Partition used for normal operation. > 3. maintenance. This partition will be used to update rootfs. > 4. data. This will hold the configuration files. Not modified by > updates. Cheers -- Jens Rehsack - rehsack@gmail.com ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [OE-core] RFC: Reference updater filesystem @ 2015-12-06 11:34 ` Jens Rehsack 0 siblings, 0 replies; 35+ messages in thread From: Jens Rehsack @ 2015-12-06 11:34 UTC (permalink / raw) To: Mariano Lopez; +Cc: openembedded-devel, OE-core > Am 03.12.2015 um 17:45 schrieb Mariano Lopez <mariano.lopez@linux.intel.com>: > > [...] > > Thanks to all for your input. The conclusion of this thread is: > > 1. One size doesn't fit all. > 2. Most of the people was fine with the image based update. Where is the image stored? I read over referred http://sbabic.github.io/swupdate/ and realized that the update image must include a stripped rootfs and the entire software for rootfs as well as probably the recoveryfs/maintfs and some logic for updating the data volume (eg. migrator for database of business logic). This can be very large ... and must be stored somewhere. Next question: how bullet proof is the swupdate? Out process is designed (because of typical users can't interact) to can survive surprising power outage. Unfortunately we didn't manage to split out u-boot into separate updatable parts, so we can't benefit from it - but technically (if one has implemented for heir targets), our script updates and continues after power comes back. I think, beside the question of having an image containing the update (we have such an add-on, too - for legacy devices without he modern Yocto based firmware), the question of stability should be evaluated, too. > 3. The recommended way to keep the configuration is to have a separated data partition. > 4. The partition scheme would be: > 1. boot. This is the usual boot partition > 2. rootfs. Partition used for normal operation. > 3. maintenance. This partition will be used to update rootfs. > 4. data. This will hold the configuration files. Not modified by > updates. Cheers -- Jens Rehsack - rehsack@gmail.com ^ permalink raw reply [flat|nested] 35+ messages in thread
[parent not found: <60350861.Ng52JMGuMb@bancha.hex>]
* Re: [oe] RFC: Reference updater filesystem [not found] ` <60350861.Ng52JMGuMb@bancha.hex> @ 2015-11-24 16:30 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 16:30 UTC (permalink / raw) To: Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/2015 4:30 AM, Roman Khimov wrote: > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in >> rootfs (example: /etc). > That's the right solution, although to do it really right (at least IMO) you > need to implement the /usr merge [1] (and that's orthogonal to using or not > using systemd), which can also help you make your /usr read-only (because > that's just code and static data) with read-write / for user data of various > sorts. To be honest I'm not familiar with /usr merge, I need to check on that to see if it is a good option with the current OE-core infrastructure. > >> 3. Have an OverlayFS for the rootfs or the partition that have the >> configuration. >> a. The pro is the configuration is "directly" in rootfs. >> b. The con is there is need to provide a custom init to guarantee the >> Overlay is mounted before the boot process. > And this is the approach I would recommend not doing. I've used UnionFS for > thing like that (overlaying whole root file system) some 6 years ago, it > sounded nice and it kinda worked, but it wasn't difficult to make it fail > (just a little playing with power), we've even seen failures on production > devices, like when you have whiteout file for directory already written, but > don't have new files in it yet and that can completely ruin the system. > > Also, it usually works better when you don't have any changes in the lower > layer, but we're talking about updating it here, you can easily end up in a > situation where you have updated something in the rootfs but that was > overriden by upper layer and thus your user doesn't see any change. Thanks for sharing your experience, this is another big con for the Overlay option. > >> With the above information I'm proposing to use a separate partition for >> the configuration; this is because is more reliable and doesn't require >> big changes in the current architecture. >> >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > You probably don't need to separate 1 and 3, all the code for system update > should easily fit into initramfs and just making /boot a bit larger would > allow you to store some backup rootfs. I left the /boot partition separate just in case there is need to replace the kernel or the bootloader. This way it would be easier to change using the same method as the upgrading the rootfs. > > Also, you can swap 4 and 2 which will be useful if you're installing on > different sized storage devices, usually you know good enough the size of your > rootfs, but you probably want to leave more space for user data if there is an > opportunity to do so, that's just easier to do with data partition at the end. I was thinking in the same thinking just backwards, usually configuration files are just small text files that don't require too much space. If you require a new feature in the target that will make rootfs to grow depending on the feature. I plan to use wic to accomplish the filesystem structure. A good thing about wic is that it will be very easy to do the swap, just need to modify two options in the .wks file. > > > [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 16:30 ` Lopez, Mariano 0 siblings, 0 replies; 35+ messages in thread From: Lopez, Mariano @ 2015-11-24 16:30 UTC (permalink / raw) To: Roman Khimov, openembedded-devel; +Cc: yocto, OE-core On 11/24/2015 4:30 AM, Roman Khimov wrote: > В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал: >> 1. Use a separate partition for the configuration. >> a. The pro of this method is the partition is not touched during the >> update. >> b. The con of this method is the configuration is not directly in >> rootfs (example: /etc). > That's the right solution, although to do it really right (at least IMO) you > need to implement the /usr merge [1] (and that's orthogonal to using or not > using systemd), which can also help you make your /usr read-only (because > that's just code and static data) with read-write / for user data of various > sorts. To be honest I'm not familiar with /usr merge, I need to check on that to see if it is a good option with the current OE-core infrastructure. > >> 3. Have an OverlayFS for the rootfs or the partition that have the >> configuration. >> a. The pro is the configuration is "directly" in rootfs. >> b. The con is there is need to provide a custom init to guarantee the >> Overlay is mounted before the boot process. > And this is the approach I would recommend not doing. I've used UnionFS for > thing like that (overlaying whole root file system) some 6 years ago, it > sounded nice and it kinda worked, but it wasn't difficult to make it fail > (just a little playing with power), we've even seen failures on production > devices, like when you have whiteout file for directory already written, but > don't have new files in it yet and that can completely ruin the system. > > Also, it usually works better when you don't have any changes in the lower > layer, but we're talking about updating it here, you can easily end up in a > situation where you have updated something in the rootfs but that was > overriden by upper layer and thus your user doesn't see any change. Thanks for sharing your experience, this is another big con for the Overlay option. > >> With the above information I'm proposing to use a separate partition for >> the configuration; this is because is more reliable and doesn't require >> big changes in the current architecture. >> >> So, the idea is to have 4 partitions in the media: >> 1. boot. This is the usual boot partition >> 2. data. This will hold the configuration files. Not modified by updates. >> 3. maintenance. This partition will be used to update rootfs. >> 4. rootfs. Partition used for normal operation. > You probably don't need to separate 1 and 3, all the code for system update > should easily fit into initramfs and just making /boot a bit larger would > allow you to store some backup rootfs. I left the /boot partition separate just in case there is need to replace the kernel or the bootloader. This way it would be easier to change using the same method as the upgrading the rootfs. > > Also, you can swap 4 and 2 which will be useful if you're installing on > different sized storage devices, usually you know good enough the size of your > rootfs, but you probably want to leave more space for user data if there is an > opportunity to do so, that's just easier to do with data partition at the end. I was thinking in the same thinking just backwards, usually configuration files are just small text files that don't require too much space. If you require a new feature in the target that will make rootfs to grow depending on the feature. I plan to use wic to accomplish the filesystem structure. A good thing about wic is that it will be very easy to do the swap, just need to modify two options in the .wks file. > > > [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [oe] RFC: Reference updater filesystem 2015-11-24 16:30 ` Lopez, Mariano @ 2015-11-24 18:10 ` Roman Khimov -1 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:10 UTC (permalink / raw) To: openembedded-devel, OE-core В письме от 24 ноября 2015 10:30:17 пользователь Lopez, Mariano написал: > > That's the right solution, although to do it really right (at least IMO) > > you need to implement the /usr merge [1] (and that's orthogonal to using > > or not using systemd), which can also help you make your /usr read-only > > (because that's just code and static data) with read-write / for user > > data of various sorts. > > To be honest I'm not familiar with /usr merge, I need to check on that > to see if it is a good option with the current OE-core infrastructure. It needs some patches for current OE, I hope I'll find some time to rebase the set I have for this (based on fido) and send it for discussion. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: RFC: Reference updater filesystem @ 2015-11-24 18:10 ` Roman Khimov 0 siblings, 0 replies; 35+ messages in thread From: Roman Khimov @ 2015-11-24 18:10 UTC (permalink / raw) To: openembedded-devel, OE-core В письме от 24 ноября 2015 10:30:17 пользователь Lopez, Mariano написал: > > That's the right solution, although to do it really right (at least IMO) > > you need to implement the /usr merge [1] (and that's orthogonal to using > > or not using systemd), which can also help you make your /usr read-only > > (because that's just code and static data) with read-write / for user > > data of various sorts. > > To be honest I'm not familiar with /usr merge, I need to check on that > to see if it is a good option with the current OE-core infrastructure. It needs some patches for current OE, I hope I'll find some time to rebase the set I have for this (based on fido) and send it for discussion. -- http://roman.khimov.ru mailto: roman@khimov.ru gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3 ^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2015-12-06 11:34 UTC | newest] Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-11-23 21:41 RFC: Reference updater filesystem Mariano Lopez 2015-11-24 6:06 ` [OE-core] " Anders Darander 2015-11-24 6:06 ` Anders Darander 2015-11-24 18:39 ` [OE-core] " Lopez, Mariano 2015-11-24 18:39 ` [yocto] " Lopez, Mariano 2015-11-24 18:39 ` [yocto] " Lopez, Mariano 2015-11-24 7:32 ` [OE-core] " Randy Witt 2015-11-24 7:32 ` Randy Witt 2015-11-24 17:19 ` [OE-core] " Lopez, Mariano 2015-11-24 17:19 ` Lopez, Mariano 2015-11-24 10:39 ` [oe] " Roman Khimov 2015-11-24 10:39 ` Roman Khimov 2015-11-24 13:47 ` [OE-core] [oe] " Mark Hatle 2015-11-24 13:47 ` [OE-core] " Mark Hatle 2015-11-24 13:47 ` [oe] " Mark Hatle 2015-11-24 17:02 ` [OE-core] " Lopez, Mariano 2015-11-24 17:02 ` [yocto] [OE-core] " Lopez, Mariano 2015-11-24 17:02 ` [yocto] [oe] " Lopez, Mariano 2015-11-24 17:13 ` [OE-core] " Mark Hatle 2015-11-24 17:13 ` [yocto] [OE-core] " Mark Hatle 2015-11-24 17:13 ` [yocto] [oe] " Mark Hatle 2015-11-24 18:05 ` Roman Khimov 2015-11-24 18:05 ` [OE-core] " Roman Khimov 2015-11-24 18:27 ` [oe] " Mark Hatle 2015-11-24 18:27 ` [OE-core] " Mark Hatle 2015-11-24 18:47 ` [oe] " Roman Khimov 2015-11-24 18:47 ` [OE-core] " Roman Khimov 2015-12-03 16:45 ` [oe] " Mariano Lopez 2015-12-03 16:45 ` [OE-core] " Mariano Lopez 2015-12-06 11:34 ` [oe] " Jens Rehsack 2015-12-06 11:34 ` [OE-core] " Jens Rehsack [not found] ` <60350861.Ng52JMGuMb@bancha.hex> 2015-11-24 16:30 ` [oe] " Lopez, Mariano 2015-11-24 16:30 ` Lopez, Mariano 2015-11-24 18:10 ` [oe] " Roman Khimov 2015-11-24 18:10 ` Roman Khimov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.