* Contents of non-rootfs partitions @ 2016-11-22 9:05 Kristian Amlie [not found] ` <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com> ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-22 9:05 UTC (permalink / raw) To: openembedded-core Hey In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building. However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition. I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done: * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic. * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined. * Have a special recipe that users can bbappend, which specifically install to the data partition (this might also need a "data" directory, like above). It's also possible that this should be generalized further so that it doesn't only apply to a data partition, but to any non-rootfs partition. I'm willing to invest some time in this, but I thought I'd gather some opinions first. Thoughts? -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
[parent not found: <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com>]
* Re: Contents of non-rootfs partitions [not found] ` <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com> @ 2016-11-22 9:54 ` Kristian Amlie 0 siblings, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-22 9:54 UTC (permalink / raw) To: Yann Dirson; +Cc: openembedded-core On 22/11/16 10:27, Yann Dirson wrote: > Since usually such a data partition needs to be written at times, you'll anyway have to handle the event of it getting corrupted, and possibly have to reformat it. > With this in mind, one option is to include default contents in your rootfs, and populate the persistent one on first boot. That's true, and a neat solution, but I consider this somewhat orthogonal. It solves the problem at run time, whereas my proposal tries to solve it at build time. With your solution, in a dual rootfs layout situation, you're paying the space cost of the initial data partition contents three times over, once in each of the two rootfs filesystems and once in the data filesystem. Depending on whether you have logic to detect a corrupt data partition or not, this price may not be worth paying if you can have a prepopulated data partition. It also has higher complexity, particularly integrating the population step in the boot sequence. I'm not opposed to this solution, but maybe it makes sense to start smaller? -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 9:05 Contents of non-rootfs partitions Kristian Amlie [not found] ` <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com> @ 2016-11-22 11:10 ` Patrick Ohly 2016-11-22 11:54 ` Kristian Amlie 2016-11-22 23:45 ` Khem Raj 2 siblings, 1 reply; 33+ messages in thread From: Patrick Ohly @ 2016-11-22 11:10 UTC (permalink / raw) To: Kristian Amlie, Eduard Bartosh; +Cc: openembedded-core On Tue, 2016-11-22 at 10:05 +0100, Kristian Amlie wrote: > Hey > > In Mender we are using a dual rootfs partition layout (A/B layout) > with a persistent data partition on the side. We're using wic to do > the actual image building. > > However, putting files into this data partition is not a > straightforward operation ATM. All recipes in OE put files into the > rootfs only, and we'd like there to be a way to specify that files > should go into the data partition. > > I haven't seen any straightforward way to do this in OE. Some > suggestions for how it could be done: > > * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can > list files you'd like to be included. This is very straightforward, > but has the disadvantage of being a bit hairy when paths are involved, > and it doesn't allow for much logic. > > * Keep a special "data" directory, something similar to sysroot, where > recipes could install files into in order to have it included on the > data partition. We could potentially have a bbclass for use in recipes > to make this more streamlined. > > * Have a special recipe that users can bbappend, which specifically > install to the data partition (this might also need a "data" > directory, like above). All of these introduce some special mechanism. Let me propose something that might integrate better with the existing tooling: The "rootfs" directory gets redefined as representing the entire virtual file system. When creating a disk image, it gets split up into different partitions based on the image configuration. For example, the /home or /data directories in the rootfs could hold the content that in some image configurations goes into separate partitions. The advantage of this approach is that the tooling for staging content for image creation does not need to be changed. The same staged content then can be used to create different images, potentially even using different partition layouts. To implement this approach with wic, wic needs to be taught how to exclude directories from the main rootfs. Ideally, the mkfs.* tools should also support that without having to make an intermediate copy of the files for a certain partition, but initially wic could create temporary directory trees. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 11:10 ` Patrick Ohly @ 2016-11-22 11:54 ` Kristian Amlie 2016-11-23 8:40 ` Enrico Joerns 2016-11-23 12:08 ` Ed Bartosh 0 siblings, 2 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-22 11:54 UTC (permalink / raw) To: Patrick Ohly, Eduard Bartosh; +Cc: openembedded-core On 22/11/16 12:10, Patrick Ohly wrote: >> ... > > All of these introduce some special mechanism. Let me propose something > that might integrate better with the existing tooling: > > The "rootfs" directory gets redefined as representing the entire virtual > file system. When creating a disk image, it gets split up into different > partitions based on the image configuration. > > For example, the /home or /data directories in the rootfs could hold the > content that in some image configurations goes into separate partitions. > > The advantage of this approach is that the tooling for staging content > for image creation does not need to be changed. The same staged content > then can be used to create different images, potentially even using > different partition layouts. That's a very good idea. I think it beats all of my suggestions! > To implement this approach with wic, wic needs to be taught how to > exclude directories from the main rootfs. Ideally, the mkfs.* tools > should also support that without having to make an intermediate copy of > the files for a certain partition, but initially wic could create > temporary directory trees. Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 11:54 ` Kristian Amlie @ 2016-11-23 8:40 ` Enrico Joerns 2016-11-23 9:04 ` Kristian Amlie 2016-11-23 9:24 ` Maciej Borzęcki 2016-11-23 12:08 ` Ed Bartosh 1 sibling, 2 replies; 33+ messages in thread From: Enrico Joerns @ 2016-11-23 8:40 UTC (permalink / raw) To: Kristian Amlie, Patrick Ohly, Eduard Bartosh; +Cc: openembedded-core On 11/22/2016 12:54 PM, Kristian Amlie wrote: > On 22/11/16 12:10, Patrick Ohly wrote: >>> ... >> >> All of these introduce some special mechanism. Let me propose something >> that might integrate better with the existing tooling: >> >> The "rootfs" directory gets redefined as representing the entire virtual >> file system. When creating a disk image, it gets split up into different >> partitions based on the image configuration. >> >> For example, the /home or /data directories in the rootfs could hold the >> content that in some image configurations goes into separate partitions. >> >> The advantage of this approach is that the tooling for staging content >> for image creation does not need to be changed. The same staged content >> then can be used to create different images, potentially even using >> different partition layouts. > > That's a very good idea. I think it beats all of my suggestions! I totally agree with this solution, because it is the same approach we are already using for several years. Instead of wic we are using the genimage tool (http://public.pengutronix.de/software/genimage/) together with a Yocto class for easing image creation, but its pretty much the same concerning its base purpose. We called the option to split up 'mountpoint' while assuming that the splitted partition will be mounted to the same location in the file system it was taken from. An example configuration would look like this (stripped down to the essential parts): image my-disk-image { [...] partition root { image = "rootfs.ext4" [...] } partition home { image = "home.ext4" [...] } } image rootfs.ext4 { [...] mountpoint = "/" } image home.ext4 { [...] mountpoint = "/home" } Maybe a similar approach could be used in wic, too. Best regards, Enrico -- Pengutronix e.K. | Enrico Jörns | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 8:40 ` Enrico Joerns @ 2016-11-23 9:04 ` Kristian Amlie 2016-11-23 9:24 ` Maciej Borzęcki 1 sibling, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-23 9:04 UTC (permalink / raw) To: Enrico Joerns, Patrick Ohly, Eduard Bartosh; +Cc: openembedded-core On 23/11/16 09:40, Enrico Joerns wrote: > On 11/22/2016 12:54 PM, Kristian Amlie wrote: >> On 22/11/16 12:10, Patrick Ohly wrote: >>>> ... >>> >>> All of these introduce some special mechanism. Let me propose something >>> that might integrate better with the existing tooling: >>> >>> The "rootfs" directory gets redefined as representing the entire virtual >>> file system. When creating a disk image, it gets split up into different >>> partitions based on the image configuration. >>> >>> For example, the /home or /data directories in the rootfs could hold the >>> content that in some image configurations goes into separate partitions. >>> >>> The advantage of this approach is that the tooling for staging content >>> for image creation does not need to be changed. The same staged content >>> then can be used to create different images, potentially even using >>> different partition layouts. >> >> That's a very good idea. I think it beats all of my suggestions! > > I totally agree with this solution, because it is the same approach we > are already using for several years. Good! Sounds like there are several votes in favor of this proposal then. > Instead of wic we are using the genimage tool > (http://public.pengutronix.de/software/genimage/) together with a Yocto > class for easing image creation, but its pretty much the same concerning > its base purpose. > > We called the option to split up 'mountpoint' while assuming that the > splitted partition will be mounted to the same location in the file > system it was taken from. > > An example configuration would look like this (stripped down to the > essential parts): > > image my-disk-image { > [...] > partition root { > image = "rootfs.ext4" > [...] > } > > partition home { > image = "home.ext4" > [...] > } > } > > image rootfs.ext4 { > [...] > mountpoint = "/" > } > > image home.ext4 { > [...] > mountpoint = "/home" > } > > Maybe a similar approach could be used in wic, too. Yes, this looks very similar "in spirit" to what I'm thinking of right now. -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 8:40 ` Enrico Joerns 2016-11-23 9:04 ` Kristian Amlie @ 2016-11-23 9:24 ` Maciej Borzęcki 2016-11-23 10:12 ` Enrico Joerns 1 sibling, 1 reply; 33+ messages in thread From: Maciej Borzęcki @ 2016-11-23 9:24 UTC (permalink / raw) To: Enrico Joerns Cc: Eduard Bartosh, Patches and discussions about the oe-core layer On Wed, Nov 23, 2016 at 9:40 AM, Enrico Joerns <ejo@pengutronix.de> wrote: > On 11/22/2016 12:54 PM, Kristian Amlie wrote: >> >> On 22/11/16 12:10, Patrick Ohly wrote: >>>> >>>> ... >>> >>> >>> All of these introduce some special mechanism. Let me propose something >>> that might integrate better with the existing tooling: >>> >>> The "rootfs" directory gets redefined as representing the entire virtual >>> file system. When creating a disk image, it gets split up into different >>> partitions based on the image configuration. >>> >>> For example, the /home or /data directories in the rootfs could hold the >>> content that in some image configurations goes into separate partitions. >>> >>> The advantage of this approach is that the tooling for staging content >>> for image creation does not need to be changed. The same staged content >>> then can be used to create different images, potentially even using >>> different partition layouts. >> >> >> That's a very good idea. I think it beats all of my suggestions! > > > I totally agree with this solution, because it is the same approach we are > already using for several years. > > Instead of wic we are using the genimage tool > (http://public.pengutronix.de/software/genimage/) together with a Yocto > class for easing image creation, but its pretty much the same concerning its > base purpose. > > We called the option to split up 'mountpoint' while assuming that the > splitted partition will be mounted to the same location in the file system > it was taken from. > > An example configuration would look like this (stripped down to the > essential parts): > > image my-disk-image { > [...] > partition root { > image = "rootfs.ext4" > [...] > } > > partition home { > image = "home.ext4" > [...] > } > } > > image rootfs.ext4 { > [...] > mountpoint = "/" > } > > image home.ext4 { > [...] > mountpoint = "/home" > } > > Maybe a similar approach could be used in wic, too. I went through the README from genimage repo https://git.pengutronix.de/cgit/genimage/tree/README. Definitely looks interesting and covers storage medias that are not supported by wic right now. The split option does not seem to be documented though. If I understood you correctly, the partition is seeded with data from location specified by `mountpoint` inside IMAGE_ROOTFS. Is that correct? I see there's a large overlap in functionality with wic (at least for block devices), so perhaps the best way would be introduce genimage as another IMAGE_CMD. I've quickly checked at https://layers.openembedded.org and there are no recipes for genimage/libconfuse listed there. Do happen to have done integration with OE in some private repo already? Would it be possible for you to post the relevant patches? -- Maciej Borzecki RnDity ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 9:24 ` Maciej Borzęcki @ 2016-11-23 10:12 ` Enrico Joerns 2016-11-23 11:03 ` Maciej Borzęcki 0 siblings, 1 reply; 33+ messages in thread From: Enrico Joerns @ 2016-11-23 10:12 UTC (permalink / raw) To: Maciej Borzęcki Cc: Eduard Bartosh, Patches and discussions about the oe-core layer On 11/23/2016 10:24 AM, Maciej Borzęcki wrote: > On Wed, Nov 23, 2016 at 9:40 AM, Enrico Joerns <ejo@pengutronix.de> wrote: >> On 11/22/2016 12:54 PM, Kristian Amlie wrote: >>> >>> On 22/11/16 12:10, Patrick Ohly wrote: >>>>> >>>>> ... >>>> >>>> >>>> All of these introduce some special mechanism. Let me propose something >>>> that might integrate better with the existing tooling: >>>> >>>> The "rootfs" directory gets redefined as representing the entire virtual >>>> file system. When creating a disk image, it gets split up into different >>>> partitions based on the image configuration. >>>> >>>> For example, the /home or /data directories in the rootfs could hold the >>>> content that in some image configurations goes into separate partitions. >>>> >>>> The advantage of this approach is that the tooling for staging content >>>> for image creation does not need to be changed. The same staged content >>>> then can be used to create different images, potentially even using >>>> different partition layouts. >>> >>> >>> That's a very good idea. I think it beats all of my suggestions! >> >> >> I totally agree with this solution, because it is the same approach we are >> already using for several years. >> >> Instead of wic we are using the genimage tool >> (http://public.pengutronix.de/software/genimage/) together with a Yocto >> class for easing image creation, but its pretty much the same concerning its >> base purpose. >> >> We called the option to split up 'mountpoint' while assuming that the >> splitted partition will be mounted to the same location in the file system >> it was taken from. >> >> An example configuration would look like this (stripped down to the >> essential parts): >> >> image my-disk-image { >> [...] >> partition root { >> image = "rootfs.ext4" >> [...] >> } >> >> partition home { >> image = "home.ext4" >> [...] >> } >> } >> >> image rootfs.ext4 { >> [...] >> mountpoint = "/" >> } >> >> image home.ext4 { >> [...] >> mountpoint = "/home" >> } >> >> Maybe a similar approach could be used in wic, too. > > I went through the README from genimage repo > https://git.pengutronix.de/cgit/genimage/tree/README. Definitely looks > interesting and covers storage medias that are not supported by wic > right now. The split option does not seem to be documented though. If I > understood you correctly, the partition is seeded with data from > location specified by `mountpoint` inside IMAGE_ROOTFS. Is that correct? Yes, what the genimage class (see below) does, is unpacking the final rootfs.tar.xz inside a fakeroot environment and call the genimage tool with the --rootpath argument set to this path. The mountpoint="/home" tells genimage to pack all content in directory /home into a separate data ext4 image (temporary named 'home.ext4') and all other content in a rootfs ext4 image (temporary named 'rootfs.ext4'). The `image` section then specifies how to construct the final disk image using the referenced generated rootfs and datafs images (and optionally some bootloader, non-partition-data, etc.). Note that this is only one way to use genimage. Another common way to use it is to simply construct complex disk images out of different filesystem images, e.g. with partition barebox { image = "barebox-any-platform.img" size = 1M in-partition-table = false } > I see there's a large overlap in functionality with wic (at least for > block devices), so perhaps the best way would be introduce genimage as > another IMAGE_CMD. > > I've quickly checked at https://layers.openembedded.org and there are no > recipes for genimage/libconfuse listed there. Do happen to have done > integration with OE in some private repo already? Would it be possible > for you to post the relevant patches? > We have a layer for this, which is potentially publicly available but not advertised as the general plan was to move some of its content to upstream (oe-core, meta-oe) and split up other content (e.g. the rauc update tool stuff) to separate layers that will be made publicly available. But, as always, there are more plans than time left ;) Well, to make a long story short, the path to the git repository is: https://git.pengutronix.de/cgit/meta-ptx The genimage class I mentioned above you can find in it, too: https://git.pengutronix.de/cgit/meta-ptx/tree/classes/genimage.bbclass When starting with Yocto I thought about changing from genimage to wic for disk image creation as wic already seemed to become the standard solution for building disk images. But back in these days it was a very separate tool and there was no integration with the BSP itself (which is crucial when having to reproduce builds). But the main reason why I decided to stay with genimage was that wic simply did not cover most of the use cases I required for building complex disk images for different embedded platforms. I'm pretty sure that this has changed in some points, but had no time to get deeper into this topic again, yet. Hope that helps, best regards Enrico -- Pengutronix e.K. | Enrico Jörns | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 10:12 ` Enrico Joerns @ 2016-11-23 11:03 ` Maciej Borzęcki 0 siblings, 0 replies; 33+ messages in thread From: Maciej Borzęcki @ 2016-11-23 11:03 UTC (permalink / raw) To: Enrico Joerns Cc: Eduard Bartosh, Patches and discussions about the oe-core layer On Wed, Nov 23, 2016 at 11:12 AM, Enrico Joerns <ejo@pengutronix.de> wrote: > On 11/23/2016 10:24 AM, Maciej Borzęcki wrote: >> <snip> > > Yes, what the genimage class (see below) does, is unpacking the final > rootfs.tar.xz inside a fakeroot environment and call the genimage tool with > the --rootpath argument set to this path. > > The mountpoint="/home" tells genimage to pack all content in directory /home > into a separate data ext4 image (temporary named 'home.ext4') and all other > content in a rootfs ext4 image (temporary named 'rootfs.ext4'). > > The `image` section then specifies how to construct the final disk image > using the referenced generated rootfs and datafs images (and optionally some > bootloader, non-partition-data, etc.). > > > Note that this is only one way to use genimage. Another common way to use it > is to simply construct complex disk images out of different filesystem > images, e.g. with > > partition barebox { > image = "barebox-any-platform.img" > size = 1M > in-partition-table = false > } > >> I see there's a large overlap in functionality with wic (at least for >> block devices), so perhaps the best way would be introduce genimage as >> another IMAGE_CMD. >> >> I've quickly checked at https://layers.openembedded.org and there are no >> recipes for genimage/libconfuse listed there. Do happen to have done >> integration with OE in some private repo already? Would it be possible >> for you to post the relevant patches? >> > > We have a layer for this, which is potentially publicly available but not > advertised as the general plan was to move some of its content to upstream > (oe-core, meta-oe) and split up other content (e.g. the rauc update tool > stuff) to separate layers that will be made publicly available. But, as > always, there are more plans than time left ;) > > Well, to make a long story short, the path to the git repository is: > > https://git.pengutronix.de/cgit/meta-ptx > > The genimage class I mentioned above you can find in it, too: > > https://git.pengutronix.de/cgit/meta-ptx/tree/classes/genimage.bbclass > > > When starting with Yocto I thought about changing from genimage to wic for > disk image creation as wic already seemed to become the standard solution > for building disk images. But back in these days it was a very separate tool > and there was no integration with the BSP itself (which is crucial when > having to reproduce builds). But the main reason why I decided to stay with > genimage was that wic simply did not cover most of the use cases I required > for building complex disk images for different embedded platforms. I'm > pretty sure that this has changed in some points, but had no time to get > deeper into this topic again, yet. > > > Hope that helps, best regards Thanks a lot. Very helpful indeed. -- Maciej Borzecki RnDity ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 11:54 ` Kristian Amlie 2016-11-23 8:40 ` Enrico Joerns @ 2016-11-23 12:08 ` Ed Bartosh 2016-11-23 13:08 ` Kristian Amlie 1 sibling, 1 reply; 33+ messages in thread From: Ed Bartosh @ 2016-11-23 12:08 UTC (permalink / raw) To: Kristian Amlie; +Cc: Eduard Bartosh, openembedded-core On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > On 22/11/16 12:10, Patrick Ohly wrote: > >> ... > > > > All of these introduce some special mechanism. Let me propose something > > that might integrate better with the existing tooling: > > > > The "rootfs" directory gets redefined as representing the entire virtual > > file system. When creating a disk image, it gets split up into different > > partitions based on the image configuration. > > > > For example, the /home or /data directories in the rootfs could hold the > > content that in some image configurations goes into separate partitions. > > > > The advantage of this approach is that the tooling for staging content > > for image creation does not need to be changed. The same staged content > > then can be used to create different images, potentially even using > > different partition layouts. > > That's a very good idea. I think it beats all of my suggestions! > > > To implement this approach with wic, wic needs to be taught how to > > exclude directories from the main rootfs. Ideally, the mkfs.* tools > > should also support that without having to make an intermediate copy of > > the files for a certain partition, but initially wic could create > > temporary directory trees. > > Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. > I support the idea. Let's discuss the details of implementation and create a bug in bugzilla to track the development This can be done by extending existing rootfs plugin. It should be able to do 2 things: - populate content of one rootfs directory to the partition. We can extend syntax of --rootfs-dir parameter to specify optional directory path to use - exclude rootfs directories when populating partitions. I'd propose to introduce --exclude-dirs wks parser option to handle this. Example of wks file with proposed new options: part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 Does this make sense? Any other ideas? -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 12:08 ` Ed Bartosh @ 2016-11-23 13:08 ` Kristian Amlie 2016-11-23 13:22 ` Ed Bartosh 2016-11-23 13:44 ` Ed Bartosh 0 siblings, 2 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-23 13:08 UTC (permalink / raw) To: ed.bartosh; +Cc: Eduard Bartosh, openembedded-core On 23/11/16 13:08, Ed Bartosh wrote: > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: >> On 22/11/16 12:10, Patrick Ohly wrote: >>>> ... >>> >>> All of these introduce some special mechanism. Let me propose something >>> that might integrate better with the existing tooling: >>> >>> The "rootfs" directory gets redefined as representing the entire virtual >>> file system. When creating a disk image, it gets split up into different >>> partitions based on the image configuration. >>> >>> For example, the /home or /data directories in the rootfs could hold the >>> content that in some image configurations goes into separate partitions. >>> >>> The advantage of this approach is that the tooling for staging content >>> for image creation does not need to be changed. The same staged content >>> then can be used to create different images, potentially even using >>> different partition layouts. >> >> That's a very good idea. I think it beats all of my suggestions! >> >>> To implement this approach with wic, wic needs to be taught how to >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools >>> should also support that without having to make an intermediate copy of >>> the files for a certain partition, but initially wic could create >>> temporary directory trees. >> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. >> > > I support the idea. Let's discuss the details of implementation and > create a bug in bugzilla to track the development Do you want me to create the ticket? (it'll be my first, so apologies in advance if I omit something important) > This can be done by extending existing rootfs plugin. It should be able > to do 2 things: > > - populate content of one rootfs directory to the partition. We can > extend syntax of --rootfs-dir parameter to specify optional directory path to use > > - exclude rootfs directories when populating partitions. I'd propose to > introduce --exclude-dirs wks parser option to handle this. > > Example of wks file with proposed new options: > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > > Does this make sense? Looks good. The only thing I would question is that, in the interest of reducing redundancy, maybe we should omit --exclude-dirs and have wic figure this out by combining all the entries, since "--exclude-dirs <dir>" and the corresponding "part <dir>" will almost always come in pairs. Possibly we could mark the "/" partition with one single --no-overlapping-dirs to force wic to make this consideration. Or do you think that's too magical? (I haven't checked how feasible this is in the code btw) -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:08 ` Kristian Amlie @ 2016-11-23 13:22 ` Ed Bartosh 2016-11-23 13:51 ` Maciej Borzęcki 2016-11-23 15:56 ` Patrick Ohly 2016-11-23 13:44 ` Ed Bartosh 1 sibling, 2 replies; 33+ messages in thread From: Ed Bartosh @ 2016-11-23 13:22 UTC (permalink / raw) To: Kristian Amlie; +Cc: Eduard Bartosh, openembedded-core On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > On 23/11/16 13:08, Ed Bartosh wrote: > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > >> On 22/11/16 12:10, Patrick Ohly wrote: > >>>> ... > >>> > >>> All of these introduce some special mechanism. Let me propose something > >>> that might integrate better with the existing tooling: > >>> > >>> The "rootfs" directory gets redefined as representing the entire virtual > >>> file system. When creating a disk image, it gets split up into different > >>> partitions based on the image configuration. > >>> > >>> For example, the /home or /data directories in the rootfs could hold the > >>> content that in some image configurations goes into separate partitions. > >>> > >>> The advantage of this approach is that the tooling for staging content > >>> for image creation does not need to be changed. The same staged content > >>> then can be used to create different images, potentially even using > >>> different partition layouts. > >> > >> That's a very good idea. I think it beats all of my suggestions! > >> > >>> To implement this approach with wic, wic needs to be taught how to > >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools > >>> should also support that without having to make an intermediate copy of > >>> the files for a certain partition, but initially wic could create > >>> temporary directory trees. > >> > >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. > >> > > > > I support the idea. Let's discuss the details of implementation and > > create a bug in bugzilla to track the development > > Do you want me to create the ticket? (it'll be my first, so apologies in > advance if I omit something important) > > > This can be done by extending existing rootfs plugin. It should be able > > to do 2 things: > > > > - populate content of one rootfs directory to the partition. We can > > extend syntax of --rootfs-dir parameter to specify optional directory path to use > > > > - exclude rootfs directories when populating partitions. I'd propose to > > introduce --exclude-dirs wks parser option to handle this. > > > > Example of wks file with proposed new options: > > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > > > > Does this make sense? > > Looks good. The only thing I would question is that, in the interest of > reducing redundancy, maybe we should omit --exclude-dirs and have wic > figure this out by combining all the entries, since "--exclude-dirs > <dir>" and the corresponding "part <dir>" will almost always come in > pairs. Possibly we could mark the "/" partition with one single > --no-overlapping-dirs to force wic to make this consideration. Or do you > think that's too magical? > Tt's quite implicit from my point of view. However, if people like it we can implement it this way. > (I haven't checked how feasible this is in the code btw) I think it would be much easier to implement --exclude-dirs. BTW, it will also allow to exclude directories from any partition, not just from root, e.g. to use --exclude dirs this way: part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:22 ` Ed Bartosh @ 2016-11-23 13:51 ` Maciej Borzęcki 2016-11-23 14:10 ` Ed Bartosh 2016-11-23 15:56 ` Patrick Ohly 1 sibling, 1 reply; 33+ messages in thread From: Maciej Borzęcki @ 2016-11-23 13:51 UTC (permalink / raw) To: Ed Bartosh Cc: Eduard Bartosh, Patches and discussions about the oe-core layer On Wed, Nov 23, 2016 at 2:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote: > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: >> On 23/11/16 13:08, Ed Bartosh wrote: >> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: >> >> On 22/11/16 12:10, Patrick Ohly wrote: >> >>>> ... >> >>> >> >>> All of these introduce some special mechanism. Let me propose something >> >>> that might integrate better with the existing tooling: >> >>> >> >>> The "rootfs" directory gets redefined as representing the entire virtual >> >>> file system. When creating a disk image, it gets split up into different >> >>> partitions based on the image configuration. >> >>> >> >>> For example, the /home or /data directories in the rootfs could hold the >> >>> content that in some image configurations goes into separate partitions. >> >>> >> >>> The advantage of this approach is that the tooling for staging content >> >>> for image creation does not need to be changed. The same staged content >> >>> then can be used to create different images, potentially even using >> >>> different partition layouts. >> >> >> >> That's a very good idea. I think it beats all of my suggestions! >> >> >> >>> To implement this approach with wic, wic needs to be taught how to >> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools >> >>> should also support that without having to make an intermediate copy of >> >>> the files for a certain partition, but initially wic could create >> >>> temporary directory trees. >> >> >> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. >> >> >> > >> > I support the idea. Let's discuss the details of implementation and >> > create a bug in bugzilla to track the development >> >> Do you want me to create the ticket? (it'll be my first, so apologies in >> advance if I omit something important) >> >> > This can be done by extending existing rootfs plugin. It should be able >> > to do 2 things: >> > >> > - populate content of one rootfs directory to the partition. We can >> > extend syntax of --rootfs-dir parameter to specify optional directory path to use >> > >> > - exclude rootfs directories when populating partitions. I'd propose to >> > introduce --exclude-dirs wks parser option to handle this. >> > >> > Example of wks file with proposed new options: >> > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home >> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 >> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 >> > >> > Does this make sense? >> >> Looks good. The only thing I would question is that, in the interest of >> reducing redundancy, maybe we should omit --exclude-dirs and have wic >> figure this out by combining all the entries, since "--exclude-dirs >> <dir>" and the corresponding "part <dir>" will almost always come in >> pairs. Possibly we could mark the "/" partition with one single >> --no-overlapping-dirs to force wic to make this consideration. Or do you >> think that's too magical? >> > Tt's quite implicit from my point of view. However, if people like it we > can implement it this way. > >> (I haven't checked how feasible this is in the code btw) > > I think it would be much easier to implement --exclude-dirs. > > BTW, it will also allow to exclude directories from any partition, not just from > root, e.g. to use --exclude dirs this way: > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp Just to be sure, --exclude-dirs means that the contents of the directory are skipped, not the directory itself? Given your example, /home/tmp, /home/cache will be included in /data, but /home/{cache,tmp}/* not. Is this correct? Regards, -- Maciej Borzecki RnDity ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:51 ` Maciej Borzęcki @ 2016-11-23 14:10 ` Ed Bartosh 0 siblings, 0 replies; 33+ messages in thread From: Ed Bartosh @ 2016-11-23 14:10 UTC (permalink / raw) To: Maciej Borzęcki Cc: Eduard Bartosh, Patches and discussions about the oe-core layer On Wed, Nov 23, 2016 at 02:51:20PM +0100, Maciej Borzęcki wrote: > On Wed, Nov 23, 2016 at 2:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote: > > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > >> On 23/11/16 13:08, Ed Bartosh wrote: > >> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > >> >> On 22/11/16 12:10, Patrick Ohly wrote: > >> >>>> ... > >> >>> > >> >>> All of these introduce some special mechanism. Let me propose something > >> >>> that might integrate better with the existing tooling: > >> >>> > >> >>> The "rootfs" directory gets redefined as representing the entire virtual > >> >>> file system. When creating a disk image, it gets split up into different > >> >>> partitions based on the image configuration. > >> >>> > >> >>> For example, the /home or /data directories in the rootfs could hold the > >> >>> content that in some image configurations goes into separate partitions. > >> >>> > >> >>> The advantage of this approach is that the tooling for staging content > >> >>> for image creation does not need to be changed. The same staged content > >> >>> then can be used to create different images, potentially even using > >> >>> different partition layouts. > >> >> > >> >> That's a very good idea. I think it beats all of my suggestions! > >> >> > >> >>> To implement this approach with wic, wic needs to be taught how to > >> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools > >> >>> should also support that without having to make an intermediate copy of > >> >>> the files for a certain partition, but initially wic could create > >> >>> temporary directory trees. > >> >> > >> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. > >> >> > >> > > >> > I support the idea. Let's discuss the details of implementation and > >> > create a bug in bugzilla to track the development > >> > >> Do you want me to create the ticket? (it'll be my first, so apologies in > >> advance if I omit something important) > >> > >> > This can be done by extending existing rootfs plugin. It should be able > >> > to do 2 things: > >> > > >> > - populate content of one rootfs directory to the partition. We can > >> > extend syntax of --rootfs-dir parameter to specify optional directory path to use > >> > > >> > - exclude rootfs directories when populating partitions. I'd propose to > >> > introduce --exclude-dirs wks parser option to handle this. > >> > > >> > Example of wks file with proposed new options: > >> > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > >> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > >> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > >> > > >> > Does this make sense? > >> > >> Looks good. The only thing I would question is that, in the interest of > >> reducing redundancy, maybe we should omit --exclude-dirs and have wic > >> figure this out by combining all the entries, since "--exclude-dirs > >> <dir>" and the corresponding "part <dir>" will almost always come in > >> pairs. Possibly we could mark the "/" partition with one single > >> --no-overlapping-dirs to force wic to make this consideration. Or do you > >> think that's too magical? > >> > > Tt's quite implicit from my point of view. However, if people like it we > > can implement it this way. > > > >> (I haven't checked how feasible this is in the code btw) > > > > I think it would be much easier to implement --exclude-dirs. > > > > BTW, it will also allow to exclude directories from any partition, not just from > > root, e.g. to use --exclude dirs this way: > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp > > Just to be sure, --exclude-dirs means that the contents of the > directory are skipped, not the directory itself? > Thanks for pointing out to this. I didn't think about it, but after you pointed out to it I think it makes sense to keep the directory and only skip the content. Another way to do it is to skip directory and content and create mount points mentioned in wks file when generating /etc/fstab. However, in this case directory permissions and ownership will be lost. -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:22 ` Ed Bartosh 2016-11-23 13:51 ` Maciej Borzęcki @ 2016-11-23 15:56 ` Patrick Ohly 2016-11-24 6:15 ` Ulrich Ölmann 1 sibling, 1 reply; 33+ messages in thread From: Patrick Ohly @ 2016-11-23 15:56 UTC (permalink / raw) To: ed.bartosh; +Cc: Eduard Bartosh, openembedded-core On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > > On 23/11/16 13:08, Ed Bartosh wrote: > > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > > >> On 22/11/16 12:10, Patrick Ohly wrote: > > >>>> ... > > >>> > > >>> All of these introduce some special mechanism. Let me propose something > > >>> that might integrate better with the existing tooling: > > >>> > > >>> The "rootfs" directory gets redefined as representing the entire virtual > > >>> file system. When creating a disk image, it gets split up into different > > >>> partitions based on the image configuration. > > >>> > > >>> For example, the /home or /data directories in the rootfs could hold the > > >>> content that in some image configurations goes into separate partitions. > > >>> > > >>> The advantage of this approach is that the tooling for staging content > > >>> for image creation does not need to be changed. The same staged content > > >>> then can be used to create different images, potentially even using > > >>> different partition layouts. > > >> > > >> That's a very good idea. I think it beats all of my suggestions! > > >> > > >>> To implement this approach with wic, wic needs to be taught how to > > >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools > > >>> should also support that without having to make an intermediate copy of > > >>> the files for a certain partition, but initially wic could create > > >>> temporary directory trees. > > >> > > >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. > > >> > > > > > > I support the idea. Let's discuss the details of implementation and > > > create a bug in bugzilla to track the development > > > > Do you want me to create the ticket? (it'll be my first, so apologies in > > advance if I omit something important) > > > > > This can be done by extending existing rootfs plugin. It should be able > > > to do 2 things: > > > > > > - populate content of one rootfs directory to the partition. We can > > > extend syntax of --rootfs-dir parameter to specify optional directory path to use > > > > > > - exclude rootfs directories when populating partitions. I'd propose to > > > introduce --exclude-dirs wks parser option to handle this. > > > > > > Example of wks file with proposed new options: > > > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > > > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > > > > > > Does this make sense? > > > > Looks good. The only thing I would question is that, in the interest of > > reducing redundancy, maybe we should omit --exclude-dirs and have wic > > figure this out by combining all the entries, since "--exclude-dirs > > <dir>" and the corresponding "part <dir>" will almost always come in > > pairs. Possibly we could mark the "/" partition with one single > > --no-overlapping-dirs to force wic to make this consideration. Or do you > > think that's too magical? > > > Tt's quite implicit from my point of view. However, if people like it we > can implement it this way. I prefer the explicit --exclude-dirs. It's less surprising and perhaps there are usages for having the same content in different partitions (redundancy, factory reset, etc.). Excluding only the directory content but not the actual directory is indeed a good point. I'm a bit undecided. When excluding only the directory content, there's no way of building a rootfs without that mount point, if that's desired. OTOH, when excluding also the directory, the data would have to be staged under a different path in the rootfs and the mount point would have to be a separate, empty directory. I'm leaning towards excluding the directory content and keeping the directory. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 15:56 ` Patrick Ohly @ 2016-11-24 6:15 ` Ulrich Ölmann 2016-11-24 7:38 ` Patrick Ohly 2016-11-24 7:38 ` Kristian Amlie 0 siblings, 2 replies; 33+ messages in thread From: Ulrich Ölmann @ 2016-11-24 6:15 UTC (permalink / raw) To: openembedded-core Hi, On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: > On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: > > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > > > On 23/11/16 13:08, Ed Bartosh wrote: > > > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > > > > [...] > > > > This can be done by extending existing rootfs plugin. It should be able > > > > to do 2 things: > > > > > > > > - populate content of one rootfs directory to the partition. We can > > > > extend syntax of --rootfs-dir parameter to specify optional directory path to use > > > > > > > > - exclude rootfs directories when populating partitions. I'd propose to > > > > introduce --exclude-dirs wks parser option to handle this. > > > > > > > > Example of wks file with proposed new options: > > > > part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > > > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > > > > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > > > > > > > > Does this make sense? > > > > > > Looks good. The only thing I would question is that, in the interest of > > > reducing redundancy, maybe we should omit --exclude-dirs and have wic > > > figure this out by combining all the entries, since "--exclude-dirs > > > <dir>" and the corresponding "part <dir>" will almost always come in > > > pairs. Possibly we could mark the "/" partition with one single > > > --no-overlapping-dirs to force wic to make this consideration. Or do you > > > think that's too magical? > > > > > Tt's quite implicit from my point of view. However, if people like it we > > can implement it this way. > > I prefer the explicit --exclude-dirs. It's less surprising and perhaps > there are usages for having the same content in different partitions > (redundancy, factory reset, etc.). > > Excluding only the directory content but not the actual directory is > indeed a good point. I'm a bit undecided. When excluding only the > directory content, there's no way of building a rootfs without that > mount point, if that's desired. OTOH, when excluding also the directory, > the data would have to be staged under a different path in the rootfs > and the mount point would have to be a separate, empty directory. > > I'm leaning towards excluding the directory content and keeping the > directory. what about having both possibilities by leaning against the syntax that rsync uses to specify if a whole source directory or only it's contents shall be synced to some destination site (see [1])? In analogy to this to exclude only the contents of the directory named 'data' you would use --exclude-dirs data/ but to additionally exclude the dir itself as well it would read --exclude-dirs data Best regards Ulrich [1] http://man.cx/rsync(1)#heading6 -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 6:15 ` Ulrich Ölmann @ 2016-11-24 7:38 ` Patrick Ohly 2016-11-24 7:38 ` Kristian Amlie 1 sibling, 0 replies; 33+ messages in thread From: Patrick Ohly @ 2016-11-24 7:38 UTC (permalink / raw) To: Ulrich Ölmann, Eduard Bartosh; +Cc: openembedded-core On Thu, 2016-11-24 at 07:15 +0100, Ulrich Ölmann wrote: > On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: > > Excluding only the directory content but not the actual directory is > > indeed a good point. I'm a bit undecided. When excluding only the > > directory content, there's no way of building a rootfs without that > > mount point, if that's desired. OTOH, when excluding also the directory, > > the data would have to be staged under a different path in the rootfs > > and the mount point would have to be a separate, empty directory. > > > > I'm leaning towards excluding the directory content and keeping the > > directory. > > what about having both possibilities by leaning against the syntax that rsync > uses to specify if a whole source directory or only it's contents shall be > synced to some destination site (see [1])? I like that idea. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 6:15 ` Ulrich Ölmann 2016-11-24 7:38 ` Patrick Ohly @ 2016-11-24 7:38 ` Kristian Amlie 2016-11-24 13:23 ` Ed Bartosh 1 sibling, 1 reply; 33+ messages in thread From: Kristian Amlie @ 2016-11-24 7:38 UTC (permalink / raw) To: Ulrich Ölmann, openembedded-core On 24/11/16 07:15, Ulrich Ölmann wrote: > Hi, > > On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: >> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: >>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: >>>> On 23/11/16 13:08, Ed Bartosh wrote: >>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: >>>>> [...] >>>>> This can be done by extending existing rootfs plugin. It should be able >>>>> to do 2 things: >>>>> >>>>> - populate content of one rootfs directory to the partition. We can >>>>> extend syntax of --rootfs-dir parameter to specify optional directory path to use >>>>> >>>>> - exclude rootfs directories when populating partitions. I'd propose to >>>>> introduce --exclude-dirs wks parser option to handle this. >>>>> >>>>> Example of wks file with proposed new options: >>>>> part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home >>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 >>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 >>>>> >>>>> Does this make sense? >>>> >>>> Looks good. The only thing I would question is that, in the interest of >>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic >>>> figure this out by combining all the entries, since "--exclude-dirs >>>> <dir>" and the corresponding "part <dir>" will almost always come in >>>> pairs. Possibly we could mark the "/" partition with one single >>>> --no-overlapping-dirs to force wic to make this consideration. Or do you >>>> think that's too magical? >>>> >>> Tt's quite implicit from my point of view. However, if people like it we >>> can implement it this way. >> >> I prefer the explicit --exclude-dirs. It's less surprising and perhaps >> there are usages for having the same content in different partitions >> (redundancy, factory reset, etc.). >> >> Excluding only the directory content but not the actual directory is >> indeed a good point. I'm a bit undecided. When excluding only the >> directory content, there's no way of building a rootfs without that >> mount point, if that's desired. OTOH, when excluding also the directory, >> the data would have to be staged under a different path in the rootfs >> and the mount point would have to be a separate, empty directory. >> >> I'm leaning towards excluding the directory content and keeping the >> directory. > > what about having both possibilities by leaning against the syntax that rsync > uses to specify if a whole source directory or only it's contents shall be > synced to some destination site (see [1])? > > In analogy to this to exclude only the contents of the directory named 'data' > you would use > > --exclude-dirs data/ > > but to additionally exclude the dir itself as well it would read > > --exclude-dirs data This is creative, but ultimately too unintuitive IMHO. Rsync is the only tool which uses this syntax AFAIK, and it's a constant source of confusion, especially when mixed with cp or similar commands. There is some discussion in the ticket as well: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10712 -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 7:38 ` Kristian Amlie @ 2016-11-24 13:23 ` Ed Bartosh 2016-11-24 14:43 ` Kristian Amlie 0 siblings, 1 reply; 33+ messages in thread From: Ed Bartosh @ 2016-11-24 13:23 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote: > On 24/11/16 07:15, Ulrich Ölmann wrote: > > Hi, > > > > On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: > >> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: > >>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > >>>> On 23/11/16 13:08, Ed Bartosh wrote: > >>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > >>>>> [...] > >>>>> This can be done by extending existing rootfs plugin. It should be able > >>>>> to do 2 things: > >>>>> > >>>>> - populate content of one rootfs directory to the partition. We can > >>>>> extend syntax of --rootfs-dir parameter to specify optional directory path to use > >>>>> > >>>>> - exclude rootfs directories when populating partitions. I'd propose to > >>>>> introduce --exclude-dirs wks parser option to handle this. > >>>>> > >>>>> Example of wks file with proposed new options: > >>>>> part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > >>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > >>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > >>>>> > >>>>> Does this make sense? > >>>> > >>>> Looks good. The only thing I would question is that, in the interest of > >>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic > >>>> figure this out by combining all the entries, since "--exclude-dirs > >>>> <dir>" and the corresponding "part <dir>" will almost always come in > >>>> pairs. Possibly we could mark the "/" partition with one single > >>>> --no-overlapping-dirs to force wic to make this consideration. Or do you > >>>> think that's too magical? > >>>> > >>> Tt's quite implicit from my point of view. However, if people like it we > >>> can implement it this way. > >> > >> I prefer the explicit --exclude-dirs. It's less surprising and perhaps > >> there are usages for having the same content in different partitions > >> (redundancy, factory reset, etc.). > >> > >> Excluding only the directory content but not the actual directory is > >> indeed a good point. I'm a bit undecided. When excluding only the > >> directory content, there's no way of building a rootfs without that > >> mount point, if that's desired. OTOH, when excluding also the directory, > >> the data would have to be staged under a different path in the rootfs > >> and the mount point would have to be a separate, empty directory. > >> > >> I'm leaning towards excluding the directory content and keeping the > >> directory. > > > > what about having both possibilities by leaning against the syntax that rsync > > uses to specify if a whole source directory or only it's contents shall be > > synced to some destination site (see [1])? > > > > In analogy to this to exclude only the contents of the directory named 'data' > > you would use > > > > --exclude-dirs data/ > > > > but to additionally exclude the dir itself as well it would read > > > > --exclude-dirs data > > This is creative, but ultimately too unintuitive IMHO. Rsync is the only > tool which uses this syntax AFAIK, and it's a constant source of > confusion, especially when mixed with cp or similar commands. > Would this way be less intuitive? --exclude-path data/* --exclude-path data We can go even further with it allowing any level of directories: --exclude-path data/tmp/* --exclude-path data/db/tmp ... -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 13:23 ` Ed Bartosh @ 2016-11-24 14:43 ` Kristian Amlie 2016-11-24 14:51 ` Ed Bartosh 0 siblings, 1 reply; 33+ messages in thread From: Kristian Amlie @ 2016-11-24 14:43 UTC (permalink / raw) To: ed.bartosh; +Cc: openembedded-core On 24/11/16 14:23, Ed Bartosh wrote: > On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote: >> On 24/11/16 07:15, Ulrich Ölmann wrote: >>> Hi, >>> >>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: >>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: >>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: >>>>>> On 23/11/16 13:08, Ed Bartosh wrote: >>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: >>>>>>> [...] >>>>>>> This can be done by extending existing rootfs plugin. It should be able >>>>>>> to do 2 things: >>>>>>> >>>>>>> - populate content of one rootfs directory to the partition. We can >>>>>>> extend syntax of --rootfs-dir parameter to specify optional directory path to use >>>>>>> >>>>>>> - exclude rootfs directories when populating partitions. I'd propose to >>>>>>> introduce --exclude-dirs wks parser option to handle this. >>>>>>> >>>>>>> Example of wks file with proposed new options: >>>>>>> part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home >>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 >>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 >>>>>>> >>>>>>> Does this make sense? >>>>>> >>>>>> Looks good. The only thing I would question is that, in the interest of >>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic >>>>>> figure this out by combining all the entries, since "--exclude-dirs >>>>>> <dir>" and the corresponding "part <dir>" will almost always come in >>>>>> pairs. Possibly we could mark the "/" partition with one single >>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you >>>>>> think that's too magical? >>>>>> >>>>> Tt's quite implicit from my point of view. However, if people like it we >>>>> can implement it this way. >>>> >>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps >>>> there are usages for having the same content in different partitions >>>> (redundancy, factory reset, etc.). >>>> >>>> Excluding only the directory content but not the actual directory is >>>> indeed a good point. I'm a bit undecided. When excluding only the >>>> directory content, there's no way of building a rootfs without that >>>> mount point, if that's desired. OTOH, when excluding also the directory, >>>> the data would have to be staged under a different path in the rootfs >>>> and the mount point would have to be a separate, empty directory. >>>> >>>> I'm leaning towards excluding the directory content and keeping the >>>> directory. >>> >>> what about having both possibilities by leaning against the syntax that rsync >>> uses to specify if a whole source directory or only it's contents shall be >>> synced to some destination site (see [1])? >>> >>> In analogy to this to exclude only the contents of the directory named 'data' >>> you would use >>> >>> --exclude-dirs data/ >>> >>> but to additionally exclude the dir itself as well it would read >>> >>> --exclude-dirs data >> >> This is creative, but ultimately too unintuitive IMHO. Rsync is the only >> tool which uses this syntax AFAIK, and it's a constant source of >> confusion, especially when mixed with cp or similar commands. >> > > Would this way be less intuitive? > --exclude-path data/* > --exclude-path data > > We can go even further with it allowing any level of directories: > --exclude-path data/tmp/* > --exclude-path data/db/tmp > ... I agree, this is pretty unambiguous and easy to understand. But this raises the question: Should we go all the way and support wildcards? Which might make it a bit complicated. Maybe support only pure '*' for now? -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 14:43 ` Kristian Amlie @ 2016-11-24 14:51 ` Ed Bartosh 2016-11-24 15:24 ` Patrick Ohly 2016-11-24 15:28 ` Andreas Oberritter 0 siblings, 2 replies; 33+ messages in thread From: Ed Bartosh @ 2016-11-24 14:51 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote: > On 24/11/16 14:23, Ed Bartosh wrote: > > On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote: > >> On 24/11/16 07:15, Ulrich Ölmann wrote: > >>> Hi, > >>> > >>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: > >>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: > >>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > >>>>>> On 23/11/16 13:08, Ed Bartosh wrote: > >>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > >>>>>>> [...] > >>>>>>> This can be done by extending existing rootfs plugin. It should be able > >>>>>>> to do 2 things: > >>>>>>> > >>>>>>> - populate content of one rootfs directory to the partition. We can > >>>>>>> extend syntax of --rootfs-dir parameter to specify optional directory path to use > >>>>>>> > >>>>>>> - exclude rootfs directories when populating partitions. I'd propose to > >>>>>>> introduce --exclude-dirs wks parser option to handle this. > >>>>>>> > >>>>>>> Example of wks file with proposed new options: > >>>>>>> part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home > >>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 > >>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 > >>>>>>> > >>>>>>> Does this make sense? > >>>>>> > >>>>>> Looks good. The only thing I would question is that, in the interest of > >>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic > >>>>>> figure this out by combining all the entries, since "--exclude-dirs > >>>>>> <dir>" and the corresponding "part <dir>" will almost always come in > >>>>>> pairs. Possibly we could mark the "/" partition with one single > >>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you > >>>>>> think that's too magical? > >>>>>> > >>>>> Tt's quite implicit from my point of view. However, if people like it we > >>>>> can implement it this way. > >>>> > >>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps > >>>> there are usages for having the same content in different partitions > >>>> (redundancy, factory reset, etc.). > >>>> > >>>> Excluding only the directory content but not the actual directory is > >>>> indeed a good point. I'm a bit undecided. When excluding only the > >>>> directory content, there's no way of building a rootfs without that > >>>> mount point, if that's desired. OTOH, when excluding also the directory, > >>>> the data would have to be staged under a different path in the rootfs > >>>> and the mount point would have to be a separate, empty directory. > >>>> > >>>> I'm leaning towards excluding the directory content and keeping the > >>>> directory. > >>> > >>> what about having both possibilities by leaning against the syntax that rsync > >>> uses to specify if a whole source directory or only it's contents shall be > >>> synced to some destination site (see [1])? > >>> > >>> In analogy to this to exclude only the contents of the directory named 'data' > >>> you would use > >>> > >>> --exclude-dirs data/ > >>> > >>> but to additionally exclude the dir itself as well it would read > >>> > >>> --exclude-dirs data > >> > >> This is creative, but ultimately too unintuitive IMHO. Rsync is the only > >> tool which uses this syntax AFAIK, and it's a constant source of > >> confusion, especially when mixed with cp or similar commands. > >> > > > > Would this way be less intuitive? > > --exclude-path data/* > > --exclude-path data > > > > We can go even further with it allowing any level of directories: > > --exclude-path data/tmp/* > > --exclude-path data/db/tmp > > ... > > I agree, this is pretty unambiguous and easy to understand. > > But this raises the question: Should we go all the way and support > wildcards? Which might make it a bit complicated. Maybe support only > pure '*' for now? As it shouldn't be hard to implement I'd go for it. -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 14:51 ` Ed Bartosh @ 2016-11-24 15:24 ` Patrick Ohly 2016-11-24 15:52 ` Kristian Amlie 2016-11-24 15:28 ` Andreas Oberritter 1 sibling, 1 reply; 33+ messages in thread From: Patrick Ohly @ 2016-11-24 15:24 UTC (permalink / raw) To: ed.bartosh; +Cc: openembedded-core On Thu, 2016-11-24 at 16:51 +0200, Ed Bartosh wrote: > On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote: > > On 24/11/16 14:23, Ed Bartosh wrote: > > > Would this way be less intuitive? > > > --exclude-path data/* > > > --exclude-path data > > > > > > We can go even further with it allowing any level of directories: > > > --exclude-path data/tmp/* Just to clarify, that is meant to also match data/tmp/.hidden-file, right? I.e. fnmatch() without the special FNM_PERIOD flag. > > > --exclude-path data/db/tmp > > > ... > > > > I agree, this is pretty unambiguous and easy to understand. > > > > But this raises the question: Should we go all the way and support > > wildcards? Which might make it a bit complicated. Maybe support only > > pure '*' for now? > > As it shouldn't be hard to implement I'd go for it. Additional code implies additional testing. Remember that it also should better be supported by mkfs.ext4 and friends. I'd rather start simple and only add additional complexity when needed. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 15:24 ` Patrick Ohly @ 2016-11-24 15:52 ` Kristian Amlie 0 siblings, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-24 15:52 UTC (permalink / raw) To: Patrick Ohly, ed.bartosh; +Cc: openembedded-core On 24/11/16 16:24, Patrick Ohly wrote: > On Thu, 2016-11-24 at 16:51 +0200, Ed Bartosh wrote: >> On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote: >>> On 24/11/16 14:23, Ed Bartosh wrote: >>>> Would this way be less intuitive? >>>> --exclude-path data/* >>>> --exclude-path data >>>> >>>> We can go even further with it allowing any level of directories: >>>> --exclude-path data/tmp/* > > Just to clarify, that is meant to also match data/tmp/.hidden-file, > right? I.e. fnmatch() without the special FNM_PERIOD flag. Yes, I think that makes the most sense, even if it breaks with shell defaults. -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 14:51 ` Ed Bartosh 2016-11-24 15:24 ` Patrick Ohly @ 2016-11-24 15:28 ` Andreas Oberritter 2016-11-24 15:51 ` Kristian Amlie 1 sibling, 1 reply; 33+ messages in thread From: Andreas Oberritter @ 2016-11-24 15:28 UTC (permalink / raw) To: ed.bartosh, Kristian Amlie; +Cc: openembedded-core On 24.11.2016 15:51, Ed Bartosh wrote: > On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote: >> On 24/11/16 14:23, Ed Bartosh wrote: >>> On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote: >>>> On 24/11/16 07:15, Ulrich Ölmann wrote: >>>>> Hi, >>>>> >>>>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote: >>>>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote: >>>>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: >>>>>>>> On 23/11/16 13:08, Ed Bartosh wrote: >>>>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: >>>>>>>>> [...] >>>>>>>>> This can be done by extending existing rootfs plugin. It should be able >>>>>>>>> to do 2 things: >>>>>>>>> >>>>>>>>> - populate content of one rootfs directory to the partition. We can >>>>>>>>> extend syntax of --rootfs-dir parameter to specify optional directory path to use >>>>>>>>> >>>>>>>>> - exclude rootfs directories when populating partitions. I'd propose to >>>>>>>>> introduce --exclude-dirs wks parser option to handle this. >>>>>>>>> >>>>>>>>> Example of wks file with proposed new options: >>>>>>>>> part / --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home >>>>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 >>>>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024 >>>>>>>>> >>>>>>>>> Does this make sense? >>>>>>>> >>>>>>>> Looks good. The only thing I would question is that, in the interest of >>>>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic >>>>>>>> figure this out by combining all the entries, since "--exclude-dirs >>>>>>>> <dir>" and the corresponding "part <dir>" will almost always come in >>>>>>>> pairs. Possibly we could mark the "/" partition with one single >>>>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you >>>>>>>> think that's too magical? >>>>>>>> >>>>>>> Tt's quite implicit from my point of view. However, if people like it we >>>>>>> can implement it this way. >>>>>> >>>>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps >>>>>> there are usages for having the same content in different partitions >>>>>> (redundancy, factory reset, etc.). >>>>>> >>>>>> Excluding only the directory content but not the actual directory is >>>>>> indeed a good point. I'm a bit undecided. When excluding only the >>>>>> directory content, there's no way of building a rootfs without that >>>>>> mount point, if that's desired. OTOH, when excluding also the directory, >>>>>> the data would have to be staged under a different path in the rootfs >>>>>> and the mount point would have to be a separate, empty directory. >>>>>> >>>>>> I'm leaning towards excluding the directory content and keeping the >>>>>> directory. >>>>> >>>>> what about having both possibilities by leaning against the syntax that rsync >>>>> uses to specify if a whole source directory or only it's contents shall be >>>>> synced to some destination site (see [1])? >>>>> >>>>> In analogy to this to exclude only the contents of the directory named 'data' >>>>> you would use >>>>> >>>>> --exclude-dirs data/ >>>>> >>>>> but to additionally exclude the dir itself as well it would read >>>>> >>>>> --exclude-dirs data >>>> >>>> This is creative, but ultimately too unintuitive IMHO. Rsync is the only >>>> tool which uses this syntax AFAIK, and it's a constant source of >>>> confusion, especially when mixed with cp or similar commands. >>>> >>> >>> Would this way be less intuitive? >>> --exclude-path data/* >>> --exclude-path data >>> >>> We can go even further with it allowing any level of directories: >>> --exclude-path data/tmp/* >>> --exclude-path data/db/tmp >>> ... >> >> I agree, this is pretty unambiguous and easy to understand. >> >> But this raises the question: Should we go all the way and support >> wildcards? Which might make it a bit complicated. Maybe support only >> pure '*' for now? > > As it shouldn't be hard to implement I'd go for it. Note that in a shell, "data/*" doesn't include "data/.*". So this syntax avoids the confusing rsync syntax in trade for another one. I'd prefer the rsync flavour, because "behaves like rsync" is easier to remember than "behaves like a shell glob, but differently". Confusion could be prevented with good documentation, FWIW. Regards, Andreas ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 15:28 ` Andreas Oberritter @ 2016-11-24 15:51 ` Kristian Amlie 2016-11-24 16:21 ` Patrick Ohly 0 siblings, 1 reply; 33+ messages in thread From: Kristian Amlie @ 2016-11-24 15:51 UTC (permalink / raw) To: Andreas Oberritter, ed.bartosh; +Cc: openembedded-core On 24/11/16 16:28, Andreas Oberritter wrote: > Note that in a shell, "data/*" doesn't include "data/.*". So this syntax > avoids the confusing rsync syntax in trade for another one. I'd prefer > the rsync flavour, because "behaves like rsync" is easier to remember > than "behaves like a shell glob, but differently". Confusion could be > prevented with good documentation, FWIW. This is turning into a bit of a bikeshed discussion. I have a slight preference for the glob version, because I think it's less surprising, but I'm fine either way. I count myself and Ed in favor of glob version, Ulrich and Andreas in favor of rsync version. Patrick, you wanna be tie breaker? :-) -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 15:51 ` Kristian Amlie @ 2016-11-24 16:21 ` Patrick Ohly 2016-11-24 16:40 ` Kristian Amlie 0 siblings, 1 reply; 33+ messages in thread From: Patrick Ohly @ 2016-11-24 16:21 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core On Thu, 2016-11-24 at 16:51 +0100, Kristian Amlie wrote: > On 24/11/16 16:28, Andreas Oberritter wrote: > > Note that in a shell, "data/*" doesn't include "data/.*". So this syntax > > avoids the confusing rsync syntax in trade for another one. I'd prefer > > the rsync flavour, because "behaves like rsync" is easier to remember > > than "behaves like a shell glob, but differently". Confusion could be > > prevented with good documentation, FWIW. > > This is turning into a bit of a bikeshed discussion. I have a slight > preference for the glob version, because I think it's less surprising, > but I'm fine either way. > > I count myself and Ed in favor of glob version, Ulrich and Andreas in > favor of rsync version. Patrick, you wanna be tie breaker? :-) But I said I was undecided ;-} Okay, I vote for the rsync flavor. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-24 16:21 ` Patrick Ohly @ 2016-11-24 16:40 ` Kristian Amlie 0 siblings, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-24 16:40 UTC (permalink / raw) To: Patrick Ohly; +Cc: openembedded-core On 24/11/16 17:21, Patrick Ohly wrote: > On Thu, 2016-11-24 at 16:51 +0100, Kristian Amlie wrote: >> On 24/11/16 16:28, Andreas Oberritter wrote: >>> Note that in a shell, "data/*" doesn't include "data/.*". So this syntax >>> avoids the confusing rsync syntax in trade for another one. I'd prefer >>> the rsync flavour, because "behaves like rsync" is easier to remember >>> than "behaves like a shell glob, but differently". Confusion could be >>> prevented with good documentation, FWIW. >> >> This is turning into a bit of a bikeshed discussion. I have a slight >> preference for the glob version, because I think it's less surprising, >> but I'm fine either way. >> >> I count myself and Ed in favor of glob version, Ulrich and Andreas in >> favor of rsync version. Patrick, you wanna be tie breaker? :-) > > But I said I was undecided ;-} Okay, I vote for the rsync flavor. Rsync flavor it is then, unless anyone has strong objections! -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:08 ` Kristian Amlie 2016-11-23 13:22 ` Ed Bartosh @ 2016-11-23 13:44 ` Ed Bartosh 2016-11-23 14:20 ` Kristian Amlie 1 sibling, 1 reply; 33+ messages in thread From: Ed Bartosh @ 2016-11-23 13:44 UTC (permalink / raw) To: Kristian Amlie; +Cc: Eduard Bartosh, openembedded-core On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: > On 23/11/16 13:08, Ed Bartosh wrote: > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote: > >> On 22/11/16 12:10, Patrick Ohly wrote: > >>>> ... > >>> > >>> All of these introduce some special mechanism. Let me propose something > >>> that might integrate better with the existing tooling: > >>> > >>> The "rootfs" directory gets redefined as representing the entire virtual > >>> file system. When creating a disk image, it gets split up into different > >>> partitions based on the image configuration. > >>> > >>> For example, the /home or /data directories in the rootfs could hold the > >>> content that in some image configurations goes into separate partitions. > >>> > >>> The advantage of this approach is that the tooling for staging content > >>> for image creation does not need to be changed. The same staged content > >>> then can be used to create different images, potentially even using > >>> different partition layouts. > >> > >> That's a very good idea. I think it beats all of my suggestions! > >> > >>> To implement this approach with wic, wic needs to be taught how to > >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools > >>> should also support that without having to make an intermediate copy of > >>> the files for a certain partition, but initially wic could create > >>> temporary directory trees. > >> > >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing. > >> > > > > I support the idea. Let's discuss the details of implementation and > > create a bug in bugzilla to track the development > > Do you want me to create the ticket? (it'll be my first, so apologies in > advance if I omit something important) > Yes, please create an issue in bugzilla and assign it to me. Don't worry. If you miss something other people will point to in in the comments :) -- Regards, Ed ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 13:44 ` Ed Bartosh @ 2016-11-23 14:20 ` Kristian Amlie 0 siblings, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-23 14:20 UTC (permalink / raw) To: ed.bartosh; +Cc: Eduard Bartosh, openembedded-core On 23/11/16 14:44, Ed Bartosh wrote: > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote: >> On 23/11/16 13:08, Ed Bartosh wrote: >>> I support the idea. Let's discuss the details of implementation and >>> create a bug in bugzilla to track the development >> >> Do you want me to create the ticket? (it'll be my first, so apologies in >> advance if I omit something important) >> > Yes, please create an issue in bugzilla and assign it to me. Don't > worry. If you miss something other people will point to in in the > comments :) Done, I have added https://bugzilla.yoctoproject.org/show_bug.cgi?id=10712 -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 9:05 Contents of non-rootfs partitions Kristian Amlie [not found] ` <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com> 2016-11-22 11:10 ` Patrick Ohly @ 2016-11-22 23:45 ` Khem Raj 2016-11-23 7:13 ` Kristian Amlie 2 siblings, 1 reply; 33+ messages in thread From: Khem Raj @ 2016-11-22 23:45 UTC (permalink / raw) To: Kristian Amlie, openembedded-core [-- Attachment #1.1: Type: text/plain, Size: 1842 bytes --] On 11/22/16 1:05 AM, Kristian Amlie wrote: > Hey > > In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building. > > However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition. > > I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done: > > * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic. > > * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined. > Firstly, it needs to be defined what the data partition is about. 1. is it some pre-initialized/seeded data that can then be modified during subsequent boots ? 2. Is it read-only data which is then used to populate r/w data segments in RAM during boot 3. is it data that does not change over the course of install and reboots of the device > * Have a special recipe that users can bbappend, which specifically install to the data partition (this might also need a "data" directory, like above). > > It's also possible that this should be generalized further so that it doesn't only apply to a data partition, but to any non-rootfs partition. > > I'm willing to invest some time in this, but I thought I'd gather some opinions first. > > Thoughts? > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 211 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-22 23:45 ` Khem Raj @ 2016-11-23 7:13 ` Kristian Amlie 2016-11-23 7:26 ` Khem Raj 0 siblings, 1 reply; 33+ messages in thread From: Kristian Amlie @ 2016-11-23 7:13 UTC (permalink / raw) To: Khem Raj, openembedded-core On 23/11/16 00:45, Khem Raj wrote: > On 11/22/16 1:05 AM, Kristian Amlie wrote: >> Hey >> >> In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building. >> >> However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition. >> >> I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done: >> >> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic. >> >> * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined. >> > > Firstly, it needs to be defined what the data partition is about. > 1. is it some pre-initialized/seeded data that can then be modified during > subsequent boots ? > 2. Is it read-only data which is then used to populate r/w data segments in > RAM during boot > 3. is it data that does not change over the course of install and reboots of > the device It's preseeded, but modifiable data, that survives complete rootfs updates. IOW if you overwrite either of your A/B rootfs partitions, the data partition and its content survives. -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 7:13 ` Kristian Amlie @ 2016-11-23 7:26 ` Khem Raj 2016-11-23 9:01 ` Kristian Amlie 0 siblings, 1 reply; 33+ messages in thread From: Khem Raj @ 2016-11-23 7:26 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core > On Nov 22, 2016, at 11:13 PM, Kristian Amlie <kristian.amlie@mender.io> wrote: > > On 23/11/16 00:45, Khem Raj wrote: >> On 11/22/16 1:05 AM, Kristian Amlie wrote: >>> Hey >>> >>> In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building. >>> >>> However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition. >>> >>> I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done: >>> >>> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic. >>> >>> * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined. >>> >> >> Firstly, it needs to be defined what the data partition is about. >> 1. is it some pre-initialized/seeded data that can then be modified during >> subsequent boots ? >> 2. Is it read-only data which is then used to populate r/w data segments in >> RAM during boot >> 3. is it data that does not change over the course of install and reboots of >> the device > > It's preseeded, but modifiable data, that survives complete rootfs > updates. IOW if you overwrite either of your A/B rootfs partitions, the > data partition and its content survives. at some point this data segment will be seeded. think of factory case. So in such a case updates should not touch them unless user does a factory reset which should clean it up > > -- > Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Contents of non-rootfs partitions 2016-11-23 7:26 ` Khem Raj @ 2016-11-23 9:01 ` Kristian Amlie 0 siblings, 0 replies; 33+ messages in thread From: Kristian Amlie @ 2016-11-23 9:01 UTC (permalink / raw) To: Khem Raj; +Cc: openembedded-core On 23/11/16 08:26, Khem Raj wrote: > >> On Nov 22, 2016, at 11:13 PM, Kristian Amlie <kristian.amlie@mender.io> wrote: >> >> On 23/11/16 00:45, Khem Raj wrote: >>> Firstly, it needs to be defined what the data partition is about. >>> 1. is it some pre-initialized/seeded data that can then be modified during >>> subsequent boots ? >>> 2. Is it read-only data which is then used to populate r/w data segments in >>> RAM during boot >>> 3. is it data that does not change over the course of install and reboots of >>> the device >> >> It's preseeded, but modifiable data, that survives complete rootfs >> updates. IOW if you overwrite either of your A/B rootfs partitions, the >> data partition and its content survives. > > at some point this data segment will be seeded. > think of factory case. The creation of the initial partitioned image for the memory card preseeds the data partition. This is meant to be used in initial factory flashing (by which I mean literally "at the factory", not the factory reset button). Subsequent rootfs updates would not touch the data partition. > So in such a case updates should not touch them unless user does a factory reset > which should clean it up Factory reset is not covered in my suggestion. While also important, I consider this a separate feature, which would require cooperation by the running OS. I hope it's acceptable to keep this out of the first patchset? -- Kristian ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2016-11-24 16:40 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-11-22 9:05 Contents of non-rootfs partitions Kristian Amlie [not found] ` <CA+4=imZLMh7di712mejxQ6qgsDSXiCOJJDbUY99YC9DoCrqrmQ@mail.gmail.com> 2016-11-22 9:54 ` Kristian Amlie 2016-11-22 11:10 ` Patrick Ohly 2016-11-22 11:54 ` Kristian Amlie 2016-11-23 8:40 ` Enrico Joerns 2016-11-23 9:04 ` Kristian Amlie 2016-11-23 9:24 ` Maciej Borzęcki 2016-11-23 10:12 ` Enrico Joerns 2016-11-23 11:03 ` Maciej Borzęcki 2016-11-23 12:08 ` Ed Bartosh 2016-11-23 13:08 ` Kristian Amlie 2016-11-23 13:22 ` Ed Bartosh 2016-11-23 13:51 ` Maciej Borzęcki 2016-11-23 14:10 ` Ed Bartosh 2016-11-23 15:56 ` Patrick Ohly 2016-11-24 6:15 ` Ulrich Ölmann 2016-11-24 7:38 ` Patrick Ohly 2016-11-24 7:38 ` Kristian Amlie 2016-11-24 13:23 ` Ed Bartosh 2016-11-24 14:43 ` Kristian Amlie 2016-11-24 14:51 ` Ed Bartosh 2016-11-24 15:24 ` Patrick Ohly 2016-11-24 15:52 ` Kristian Amlie 2016-11-24 15:28 ` Andreas Oberritter 2016-11-24 15:51 ` Kristian Amlie 2016-11-24 16:21 ` Patrick Ohly 2016-11-24 16:40 ` Kristian Amlie 2016-11-23 13:44 ` Ed Bartosh 2016-11-23 14:20 ` Kristian Amlie 2016-11-22 23:45 ` Khem Raj 2016-11-23 7:13 ` Kristian Amlie 2016-11-23 7:26 ` Khem Raj 2016-11-23 9:01 ` Kristian Amlie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.