Contents of non-rootfs partitions

Contents of non-rootfs partitions

[Kristian Amlie]

Hey

In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building.

However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition.

I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done:

* A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic.

* Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined.

* Have a special recipe that users can bbappend, which specifically install to the data partition (this might also need a "data" directory, like above).

It's also possible that this should be generalized further so that it doesn't only apply to a data partition, but to any non-rootfs partition.

I'm willing to invest some time in this, but I thought I'd gather some opinions first.

Thoughts?

-- 
Kristian

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 22/11/16 10:27, Yann Dirson wrote:
> Since usually such a data partition needs to be written at times, you'll anyway have to handle the event of it getting corrupted, and possibly have to reformat it.
> With this in mind, one option is to include default contents in your rootfs, and populate the persistent one on first boot.

That's true, and a neat solution, but I consider this somewhat orthogonal. It solves the problem at run time, whereas my proposal tries to solve it at build time. With your solution, in a dual rootfs layout situation, you're paying the space cost of the initial data partition contents three times over, once in each of the two rootfs filesystems and once in the data filesystem. Depending on whether you have logic to detect a corrupt data partition or not, this price may not be worth paying if you can have a prepopulated data partition.

It also has higher complexity, particularly integrating the population step in the boot sequence. I'm not opposed to this solution, but maybe it makes sense to start smaller?

-- 
Kristian

Re: Contents of non-rootfs partitions

[Patrick Ohly]

On Tue, 2016-11-22 at 10:05 +0100, Kristian Amlie wrote:
> Hey
> 
> In Mender we are using a dual rootfs partition layout (A/B layout)
> with a persistent data partition on the side. We're using wic to do
> the actual image building.
> 
> However, putting files into this data partition is not a
> straightforward operation ATM. All recipes in OE put files into the
> rootfs only, and we'd like there to be a way to specify that files
> should go into the data partition.
> 
> I haven't seen any straightforward way to do this in OE. Some
> suggestions for how it could be done:
> 
> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can
> list files you'd like to be included. This is very straightforward,
> but has the disadvantage of being a bit hairy when paths are involved,
> and it doesn't allow for much logic.
> 
> * Keep a special "data" directory, something similar to sysroot, where
> recipes could install files into in order to have it included on the
> data partition. We could potentially have a bbclass for use in recipes
> to make this more streamlined.
> 
> * Have a special recipe that users can bbappend, which specifically
> install to the data partition (this might also need a "data"
> directory, like above).

All of these introduce some special mechanism. Let me propose something
that might integrate better with the existing tooling:

The "rootfs" directory gets redefined as representing the entire virtual
file system. When creating a disk image, it gets split up into different
partitions based on the image configuration.

For example, the /home or /data directories in the rootfs could hold the
content that in some image configurations goes into separate partitions.

The advantage of this approach is that the tooling for staging content
for image creation does not need to be changed. The same staged content
then can be used to create different images, potentially even using
different partition layouts.

To implement this approach with wic, wic needs to be taught how to
exclude directories from the main rootfs. Ideally, the mkfs.* tools
should also support that without having to make an intermediate copy of
the files for a certain partition, but initially wic could create
temporary directory trees.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 22/11/16 12:10, Patrick Ohly wrote:
>> ...
> 
> All of these introduce some special mechanism. Let me propose something
> that might integrate better with the existing tooling:
> 
> The "rootfs" directory gets redefined as representing the entire virtual
> file system. When creating a disk image, it gets split up into different
> partitions based on the image configuration.
> 
> For example, the /home or /data directories in the rootfs could hold the
> content that in some image configurations goes into separate partitions.
> 
> The advantage of this approach is that the tooling for staging content
> for image creation does not need to be changed. The same staged content
> then can be used to create different images, potentially even using
> different partition layouts.

That's a very good idea. I think it beats all of my suggestions!

> To implement this approach with wic, wic needs to be taught how to
> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> should also support that without having to make an intermediate copy of
> the files for a certain partition, but initially wic could create
> temporary directory trees.

Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.

-- 
Kristian

Re: Contents of non-rootfs partitions

[Enrico Joerns]

On 11/22/2016 12:54 PM, Kristian Amlie wrote:
> On 22/11/16 12:10, Patrick Ohly wrote:
>>> ...
>>
>> All of these introduce some special mechanism. Let me propose something
>> that might integrate better with the existing tooling:
>>
>> The "rootfs" directory gets redefined as representing the entire virtual
>> file system. When creating a disk image, it gets split up into different
>> partitions based on the image configuration.
>>
>> For example, the /home or /data directories in the rootfs could hold the
>> content that in some image configurations goes into separate partitions.
>>
>> The advantage of this approach is that the tooling for staging content
>> for image creation does not need to be changed. The same staged content
>> then can be used to create different images, potentially even using
>> different partition layouts.
>
> That's a very good idea. I think it beats all of my suggestions!

I totally agree with this solution, because it is the same approach we 
are already using for several years.

Instead of wic we are using the genimage tool 
(http://public.pengutronix.de/software/genimage/) together with a Yocto 
class for easing image creation, but its pretty much the same concerning 
its base purpose.

We called the option to split up 'mountpoint' while assuming that the 
splitted partition will be mounted to the same location in the file 
system it was taken from.

An example configuration would look like this (stripped down to the 
essential parts):

   image my-disk-image {
           [...]
           partition root {
                   image = "rootfs.ext4"
                   [...]
           }

           partition home {
                   image = "home.ext4"
                   [...]
           }
   }

   image rootfs.ext4 {
           [...]
           mountpoint = "/"
   }

   image home.ext4 {
           [...]
           mountpoint = "/home"
   }

Maybe a similar approach could be used in wic, too.


Best regards, Enrico

-- 
Pengutronix e.K.                           | Enrico Jörns                |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 23/11/16 09:40, Enrico Joerns wrote:
> On 11/22/2016 12:54 PM, Kristian Amlie wrote:
>> On 22/11/16 12:10, Patrick Ohly wrote:
>>>> ...
>>>
>>> All of these introduce some special mechanism. Let me propose something
>>> that might integrate better with the existing tooling:
>>>
>>> The "rootfs" directory gets redefined as representing the entire virtual
>>> file system. When creating a disk image, it gets split up into different
>>> partitions based on the image configuration.
>>>
>>> For example, the /home or /data directories in the rootfs could hold the
>>> content that in some image configurations goes into separate partitions.
>>>
>>> The advantage of this approach is that the tooling for staging content
>>> for image creation does not need to be changed. The same staged content
>>> then can be used to create different images, potentially even using
>>> different partition layouts.
>>
>> That's a very good idea. I think it beats all of my suggestions!
> 
> I totally agree with this solution, because it is the same approach we
> are already using for several years.

Good! Sounds like there are several votes in favor of this proposal then.

> Instead of wic we are using the genimage tool
> (http://public.pengutronix.de/software/genimage/) together with a Yocto
> class for easing image creation, but its pretty much the same concerning
> its base purpose.
> 
> We called the option to split up 'mountpoint' while assuming that the
> splitted partition will be mounted to the same location in the file
> system it was taken from.
> 
> An example configuration would look like this (stripped down to the
> essential parts):
> 
>   image my-disk-image {
>           [...]
>           partition root {
>                   image = "rootfs.ext4"
>                   [...]
>           }
> 
>           partition home {
>                   image = "home.ext4"
>                   [...]
>           }
>   }
> 
>   image rootfs.ext4 {
>           [...]
>           mountpoint = "/"
>   }
> 
>   image home.ext4 {
>           [...]
>           mountpoint = "/home"
>   }
> 
> Maybe a similar approach could be used in wic, too.

Yes, this looks very similar "in spirit" to what I'm thinking of right now.

-- 
Kristian

Re: Contents of non-rootfs partitions

[Maciej Borzęcki]

On Wed, Nov 23, 2016 at 9:40 AM, Enrico Joerns <ejo@pengutronix.de> wrote:
> On 11/22/2016 12:54 PM, Kristian Amlie wrote:
>>
>> On 22/11/16 12:10, Patrick Ohly wrote:
>>>>
>>>> ...
>>>
>>>
>>> All of these introduce some special mechanism. Let me propose something
>>> that might integrate better with the existing tooling:
>>>
>>> The "rootfs" directory gets redefined as representing the entire virtual
>>> file system. When creating a disk image, it gets split up into different
>>> partitions based on the image configuration.
>>>
>>> For example, the /home or /data directories in the rootfs could hold the
>>> content that in some image configurations goes into separate partitions.
>>>
>>> The advantage of this approach is that the tooling for staging content
>>> for image creation does not need to be changed. The same staged content
>>> then can be used to create different images, potentially even using
>>> different partition layouts.
>>
>>
>> That's a very good idea. I think it beats all of my suggestions!
>
>
> I totally agree with this solution, because it is the same approach we are
> already using for several years.
>
> Instead of wic we are using the genimage tool
> (http://public.pengutronix.de/software/genimage/) together with a Yocto
> class for easing image creation, but its pretty much the same concerning its
> base purpose.
>
> We called the option to split up 'mountpoint' while assuming that the
> splitted partition will be mounted to the same location in the file system
> it was taken from.
>
> An example configuration would look like this (stripped down to the
> essential parts):
>
>   image my-disk-image {
>           [...]
>           partition root {
>                   image = "rootfs.ext4"
>                   [...]
>           }
>
>           partition home {
>                   image = "home.ext4"
>                   [...]
>           }
>   }
>
>   image rootfs.ext4 {
>           [...]
>           mountpoint = "/"
>   }
>
>   image home.ext4 {
>           [...]
>           mountpoint = "/home"
>   }
>
> Maybe a similar approach could be used in wic, too.

I went through the README from genimage repo
https://git.pengutronix.de/cgit/genimage/tree/README. Definitely looks
interesting and covers storage medias that are not supported by wic
right now. The split option does not seem to be documented though. If I
understood you correctly, the partition is seeded with data from
location specified by `mountpoint` inside IMAGE_ROOTFS. Is that correct?

I see there's a large overlap in functionality with wic (at least for
block devices), so perhaps the best way would be introduce genimage as
another IMAGE_CMD.

I've quickly checked at https://layers.openembedded.org and there are no
recipes for genimage/libconfuse listed there. Do happen to have done
integration with OE in some private repo already? Would it be possible
for you to post the relevant patches?

-- 
Maciej Borzecki
RnDity

Re: Contents of non-rootfs partitions

[Enrico Joerns]

On 11/23/2016 10:24 AM, Maciej Borzęcki wrote:
> On Wed, Nov 23, 2016 at 9:40 AM, Enrico Joerns <ejo@pengutronix.de> wrote:
>> On 11/22/2016 12:54 PM, Kristian Amlie wrote:
>>>
>>> On 22/11/16 12:10, Patrick Ohly wrote:
>>>>>
>>>>> ...
>>>>
>>>>
>>>> All of these introduce some special mechanism. Let me propose something
>>>> that might integrate better with the existing tooling:
>>>>
>>>> The "rootfs" directory gets redefined as representing the entire virtual
>>>> file system. When creating a disk image, it gets split up into different
>>>> partitions based on the image configuration.
>>>>
>>>> For example, the /home or /data directories in the rootfs could hold the
>>>> content that in some image configurations goes into separate partitions.
>>>>
>>>> The advantage of this approach is that the tooling for staging content
>>>> for image creation does not need to be changed. The same staged content
>>>> then can be used to create different images, potentially even using
>>>> different partition layouts.
>>>
>>>
>>> That's a very good idea. I think it beats all of my suggestions!
>>
>>
>> I totally agree with this solution, because it is the same approach we are
>> already using for several years.
>>
>> Instead of wic we are using the genimage tool
>> (http://public.pengutronix.de/software/genimage/) together with a Yocto
>> class for easing image creation, but its pretty much the same concerning its
>> base purpose.
>>
>> We called the option to split up 'mountpoint' while assuming that the
>> splitted partition will be mounted to the same location in the file system
>> it was taken from.
>>
>> An example configuration would look like this (stripped down to the
>> essential parts):
>>
>>   image my-disk-image {
>>           [...]
>>           partition root {
>>                   image = "rootfs.ext4"
>>                   [...]
>>           }
>>
>>           partition home {
>>                   image = "home.ext4"
>>                   [...]
>>           }
>>   }
>>
>>   image rootfs.ext4 {
>>           [...]
>>           mountpoint = "/"
>>   }
>>
>>   image home.ext4 {
>>           [...]
>>           mountpoint = "/home"
>>   }
>>
>> Maybe a similar approach could be used in wic, too.
>
> I went through the README from genimage repo
> https://git.pengutronix.de/cgit/genimage/tree/README. Definitely looks
> interesting and covers storage medias that are not supported by wic
> right now. The split option does not seem to be documented though. If I
> understood you correctly, the partition is seeded with data from
> location specified by `mountpoint` inside IMAGE_ROOTFS. Is that correct?

Yes, what the genimage class (see below) does, is unpacking the final 
rootfs.tar.xz inside a fakeroot environment and call the genimage tool 
with the --rootpath argument set to this path.

The mountpoint="/home" tells genimage to pack all content in directory 
/home into a separate data ext4 image (temporary named 'home.ext4') and 
all other content in a rootfs ext4 image (temporary named 'rootfs.ext4').

The `image` section then specifies how to construct the final disk image 
using the referenced generated rootfs and datafs images (and optionally 
some bootloader, non-partition-data, etc.).


Note that this is only one way to use genimage. Another common way to 
use it is to simply construct complex disk images out of different 
filesystem images, e.g. with

   partition barebox {
           image = "barebox-any-platform.img"
           size = 1M
           in-partition-table = false
   }

> I see there's a large overlap in functionality with wic (at least for
> block devices), so perhaps the best way would be introduce genimage as
> another IMAGE_CMD.
>
> I've quickly checked at https://layers.openembedded.org and there are no
> recipes for genimage/libconfuse listed there. Do happen to have done
> integration with OE in some private repo already? Would it be possible
> for you to post the relevant patches?
>

We have a layer for this, which is potentially publicly available but 
not advertised as the general plan was to move some of its content to 
upstream (oe-core, meta-oe) and split up other content (e.g. the rauc 
update tool stuff) to separate layers that will be made publicly 
available. But, as always, there are more plans than time left ;)

Well, to make a long story short, the path to the git repository is:

   https://git.pengutronix.de/cgit/meta-ptx

The genimage class I mentioned above you can find in it, too:

   https://git.pengutronix.de/cgit/meta-ptx/tree/classes/genimage.bbclass


When starting with Yocto I thought about changing from genimage to wic 
for disk image creation as wic already seemed to become the standard 
solution for building disk images. But back in these days it was a very 
separate tool and there was no integration with the BSP itself (which is 
crucial when having to reproduce builds). But the main reason why I 
decided to stay with genimage was that wic simply did not cover most of 
the use cases I required for building complex disk images for different 
embedded platforms. I'm pretty sure that this has changed in some 
points, but had no time to get deeper into this topic again, yet.


Hope that helps, best regards

Enrico

-- 
Pengutronix e.K.                           | Enrico Jörns                |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Contents of non-rootfs partitions

[Maciej Borzęcki]

On Wed, Nov 23, 2016 at 11:12 AM, Enrico Joerns <ejo@pengutronix.de> wrote:
> On 11/23/2016 10:24 AM, Maciej Borzęcki wrote:
>>
<snip>
>
> Yes, what the genimage class (see below) does, is unpacking the final
> rootfs.tar.xz inside a fakeroot environment and call the genimage tool with
> the --rootpath argument set to this path.
>
> The mountpoint="/home" tells genimage to pack all content in directory /home
> into a separate data ext4 image (temporary named 'home.ext4') and all other
> content in a rootfs ext4 image (temporary named 'rootfs.ext4').
>
> The `image` section then specifies how to construct the final disk image
> using the referenced generated rootfs and datafs images (and optionally some
> bootloader, non-partition-data, etc.).
>
>
> Note that this is only one way to use genimage. Another common way to use it
> is to simply construct complex disk images out of different filesystem
> images, e.g. with
>
>   partition barebox {
>           image = "barebox-any-platform.img"
>           size = 1M
>           in-partition-table = false
>   }
>
>> I see there's a large overlap in functionality with wic (at least for
>> block devices), so perhaps the best way would be introduce genimage as
>> another IMAGE_CMD.
>>
>> I've quickly checked at https://layers.openembedded.org and there are no
>> recipes for genimage/libconfuse listed there. Do happen to have done
>> integration with OE in some private repo already? Would it be possible
>> for you to post the relevant patches?
>>
>
> We have a layer for this, which is potentially publicly available but not
> advertised as the general plan was to move some of its content to upstream
> (oe-core, meta-oe) and split up other content (e.g. the rauc update tool
> stuff) to separate layers that will be made publicly available. But, as
> always, there are more plans than time left ;)
>
> Well, to make a long story short, the path to the git repository is:
>
>   https://git.pengutronix.de/cgit/meta-ptx
>
> The genimage class I mentioned above you can find in it, too:
>
>   https://git.pengutronix.de/cgit/meta-ptx/tree/classes/genimage.bbclass
>
>
> When starting with Yocto I thought about changing from genimage to wic for
> disk image creation as wic already seemed to become the standard solution
> for building disk images. But back in these days it was a very separate tool
> and there was no integration with the BSP itself (which is crucial when
> having to reproduce builds). But the main reason why I decided to stay with
> genimage was that wic simply did not cover most of the use cases I required
> for building complex disk images for different embedded platforms. I'm
> pretty sure that this has changed in some points, but had no time to get
> deeper into this topic again, yet.
>
>
> Hope that helps, best regards

Thanks a lot. Very helpful indeed.

-- 
Maciej Borzecki
RnDity

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> On 22/11/16 12:10, Patrick Ohly wrote:
> >> ...
> > 
> > All of these introduce some special mechanism. Let me propose something
> > that might integrate better with the existing tooling:
> > 
> > The "rootfs" directory gets redefined as representing the entire virtual
> > file system. When creating a disk image, it gets split up into different
> > partitions based on the image configuration.
> > 
> > For example, the /home or /data directories in the rootfs could hold the
> > content that in some image configurations goes into separate partitions.
> > 
> > The advantage of this approach is that the tooling for staging content
> > for image creation does not need to be changed. The same staged content
> > then can be used to create different images, potentially even using
> > different partition layouts.
> 
> That's a very good idea. I think it beats all of my suggestions!
> 
> > To implement this approach with wic, wic needs to be taught how to
> > exclude directories from the main rootfs. Ideally, the mkfs.* tools
> > should also support that without having to make an intermediate copy of
> > the files for a certain partition, but initially wic could create
> > temporary directory trees.
> 
> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
>

I support the idea. Let's discuss the details of implementation and
create a bug in bugzilla to track the development 

This can be done by extending existing rootfs plugin. It should be able
to do 2 things:

- populate content of one rootfs directory to the partition. We can
  extend syntax of --rootfs-dir parameter to specify optional directory path to use

- exclude rootfs directories when populating partitions. I'd propose to
  introduce --exclude-dirs wks parser option to handle this.

Example of wks file with proposed new options:
part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024

Does this make sense?

Any other ideas?

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 23/11/16 13:08, Ed Bartosh wrote:
> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
>> On 22/11/16 12:10, Patrick Ohly wrote:
>>>> ...
>>>
>>> All of these introduce some special mechanism. Let me propose something
>>> that might integrate better with the existing tooling:
>>>
>>> The "rootfs" directory gets redefined as representing the entire virtual
>>> file system. When creating a disk image, it gets split up into different
>>> partitions based on the image configuration.
>>>
>>> For example, the /home or /data directories in the rootfs could hold the
>>> content that in some image configurations goes into separate partitions.
>>>
>>> The advantage of this approach is that the tooling for staging content
>>> for image creation does not need to be changed. The same staged content
>>> then can be used to create different images, potentially even using
>>> different partition layouts.
>>
>> That's a very good idea. I think it beats all of my suggestions!
>>
>>> To implement this approach with wic, wic needs to be taught how to
>>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
>>> should also support that without having to make an intermediate copy of
>>> the files for a certain partition, but initially wic could create
>>> temporary directory trees.
>>
>> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
>>
> 
> I support the idea. Let's discuss the details of implementation and
> create a bug in bugzilla to track the development 

Do you want me to create the ticket? (it'll be my first, so apologies in
advance if I omit something important)

> This can be done by extending existing rootfs plugin. It should be able
> to do 2 things:
> 
> - populate content of one rootfs directory to the partition. We can
>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> 
> - exclude rootfs directories when populating partitions. I'd propose to
>   introduce --exclude-dirs wks parser option to handle this.
> 
> Example of wks file with proposed new options:
> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> 
> Does this make sense?

Looks good. The only thing I would question is that, in the interest of
reducing redundancy, maybe we should omit --exclude-dirs and have wic
figure this out by combining all the entries, since "--exclude-dirs
<dir>" and the corresponding "part <dir>" will almost always come in
pairs. Possibly we could mark the "/" partition with one single
--no-overlapping-dirs to force wic to make this consideration. Or do you
think that's too magical?

(I haven't checked how feasible this is in the code btw)

-- 
Kristian

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> On 23/11/16 13:08, Ed Bartosh wrote:
> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >>>> ...
> >>>
> >>> All of these introduce some special mechanism. Let me propose something
> >>> that might integrate better with the existing tooling:
> >>>
> >>> The "rootfs" directory gets redefined as representing the entire virtual
> >>> file system. When creating a disk image, it gets split up into different
> >>> partitions based on the image configuration.
> >>>
> >>> For example, the /home or /data directories in the rootfs could hold the
> >>> content that in some image configurations goes into separate partitions.
> >>>
> >>> The advantage of this approach is that the tooling for staging content
> >>> for image creation does not need to be changed. The same staged content
> >>> then can be used to create different images, potentially even using
> >>> different partition layouts.
> >>
> >> That's a very good idea. I think it beats all of my suggestions!
> >>
> >>> To implement this approach with wic, wic needs to be taught how to
> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >>> should also support that without having to make an intermediate copy of
> >>> the files for a certain partition, but initially wic could create
> >>> temporary directory trees.
> >>
> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
> >>
> > 
> > I support the idea. Let's discuss the details of implementation and
> > create a bug in bugzilla to track the development 
> 
> Do you want me to create the ticket? (it'll be my first, so apologies in
> advance if I omit something important)
> 
> > This can be done by extending existing rootfs plugin. It should be able
> > to do 2 things:
> > 
> > - populate content of one rootfs directory to the partition. We can
> >   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> > 
> > - exclude rootfs directories when populating partitions. I'd propose to
> >   introduce --exclude-dirs wks parser option to handle this.
> > 
> > Example of wks file with proposed new options:
> > part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> > 
> > Does this make sense?
> 
> Looks good. The only thing I would question is that, in the interest of
> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> figure this out by combining all the entries, since "--exclude-dirs
> <dir>" and the corresponding "part <dir>" will almost always come in
> pairs. Possibly we could mark the "/" partition with one single
> --no-overlapping-dirs to force wic to make this consideration. Or do you
> think that's too magical?
>
Tt's quite implicit from my point of view. However, if people like it we
can implement it this way.

> (I haven't checked how feasible this is in the code btw)

I think it would be much easier to implement --exclude-dirs.

BTW, it will also allow to exclude directories from any partition, not just from
root, e.g. to use --exclude dirs this way:
part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Maciej Borzęcki]

On Wed, Nov 23, 2016 at 2:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
>> On 23/11/16 13:08, Ed Bartosh wrote:
>> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
>> >> On 22/11/16 12:10, Patrick Ohly wrote:
>> >>>> ...
>> >>>
>> >>> All of these introduce some special mechanism. Let me propose something
>> >>> that might integrate better with the existing tooling:
>> >>>
>> >>> The "rootfs" directory gets redefined as representing the entire virtual
>> >>> file system. When creating a disk image, it gets split up into different
>> >>> partitions based on the image configuration.
>> >>>
>> >>> For example, the /home or /data directories in the rootfs could hold the
>> >>> content that in some image configurations goes into separate partitions.
>> >>>
>> >>> The advantage of this approach is that the tooling for staging content
>> >>> for image creation does not need to be changed. The same staged content
>> >>> then can be used to create different images, potentially even using
>> >>> different partition layouts.
>> >>
>> >> That's a very good idea. I think it beats all of my suggestions!
>> >>
>> >>> To implement this approach with wic, wic needs to be taught how to
>> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
>> >>> should also support that without having to make an intermediate copy of
>> >>> the files for a certain partition, but initially wic could create
>> >>> temporary directory trees.
>> >>
>> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
>> >>
>> >
>> > I support the idea. Let's discuss the details of implementation and
>> > create a bug in bugzilla to track the development
>>
>> Do you want me to create the ticket? (it'll be my first, so apologies in
>> advance if I omit something important)
>>
>> > This can be done by extending existing rootfs plugin. It should be able
>> > to do 2 things:
>> >
>> > - populate content of one rootfs directory to the partition. We can
>> >   extend syntax of --rootfs-dir parameter to specify optional directory path to use
>> >
>> > - exclude rootfs directories when populating partitions. I'd propose to
>> >   introduce --exclude-dirs wks parser option to handle this.
>> >
>> > Example of wks file with proposed new options:
>> > part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
>> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
>> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
>> >
>> > Does this make sense?
>>
>> Looks good. The only thing I would question is that, in the interest of
>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
>> figure this out by combining all the entries, since "--exclude-dirs
>> <dir>" and the corresponding "part <dir>" will almost always come in
>> pairs. Possibly we could mark the "/" partition with one single
>> --no-overlapping-dirs to force wic to make this consideration. Or do you
>> think that's too magical?
>>
> Tt's quite implicit from my point of view. However, if people like it we
> can implement it this way.
>
>> (I haven't checked how feasible this is in the code btw)
>
> I think it would be much easier to implement --exclude-dirs.
>
> BTW, it will also allow to exclude directories from any partition, not just from
> root, e.g. to use --exclude dirs this way:
> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp

Just to be sure, --exclude-dirs means that the contents of the
directory are skipped, not the directory itself?

Given your example, /home/tmp, /home/cache will be included in /data,
but /home/{cache,tmp}/* not. Is this correct?

Regards,
-- 
Maciej Borzecki
RnDity

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:51:20PM +0100, Maciej Borzęcki wrote:
> On Wed, Nov 23, 2016 at 2:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >> On 23/11/16 13:08, Ed Bartosh wrote:
> >> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >> >>>> ...
> >> >>>
> >> >>> All of these introduce some special mechanism. Let me propose something
> >> >>> that might integrate better with the existing tooling:
> >> >>>
> >> >>> The "rootfs" directory gets redefined as representing the entire virtual
> >> >>> file system. When creating a disk image, it gets split up into different
> >> >>> partitions based on the image configuration.
> >> >>>
> >> >>> For example, the /home or /data directories in the rootfs could hold the
> >> >>> content that in some image configurations goes into separate partitions.
> >> >>>
> >> >>> The advantage of this approach is that the tooling for staging content
> >> >>> for image creation does not need to be changed. The same staged content
> >> >>> then can be used to create different images, potentially even using
> >> >>> different partition layouts.
> >> >>
> >> >> That's a very good idea. I think it beats all of my suggestions!
> >> >>
> >> >>> To implement this approach with wic, wic needs to be taught how to
> >> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >> >>> should also support that without having to make an intermediate copy of
> >> >>> the files for a certain partition, but initially wic could create
> >> >>> temporary directory trees.
> >> >>
> >> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
> >> >>
> >> >
> >> > I support the idea. Let's discuss the details of implementation and
> >> > create a bug in bugzilla to track the development
> >>
> >> Do you want me to create the ticket? (it'll be my first, so apologies in
> >> advance if I omit something important)
> >>
> >> > This can be done by extending existing rootfs plugin. It should be able
> >> > to do 2 things:
> >> >
> >> > - populate content of one rootfs directory to the partition. We can
> >> >   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> >> >
> >> > - exclude rootfs directories when populating partitions. I'd propose to
> >> >   introduce --exclude-dirs wks parser option to handle this.
> >> >
> >> > Example of wks file with proposed new options:
> >> > part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> >> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> >> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> >> >
> >> > Does this make sense?
> >>
> >> Looks good. The only thing I would question is that, in the interest of
> >> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >> figure this out by combining all the entries, since "--exclude-dirs
> >> <dir>" and the corresponding "part <dir>" will almost always come in
> >> pairs. Possibly we could mark the "/" partition with one single
> >> --no-overlapping-dirs to force wic to make this consideration. Or do you
> >> think that's too magical?
> >>
> > Tt's quite implicit from my point of view. However, if people like it we
> > can implement it this way.
> >
> >> (I haven't checked how feasible this is in the code btw)
> >
> > I think it would be much easier to implement --exclude-dirs.
> >
> > BTW, it will also allow to exclude directories from any partition, not just from
> > root, e.g. to use --exclude dirs this way:
> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp
> 
> Just to be sure, --exclude-dirs means that the contents of the
> directory are skipped, not the directory itself?
>
Thanks for pointing out to this. I didn't think about it, but after you
pointed out to it I think it makes sense to keep the directory and only skip the
content.

Another way to do it is to skip directory and content and create mount points
mentioned in wks file when generating /etc/fstab. However, in this case
directory permissions and ownership will be lost.

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Patrick Ohly]

On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> > On 23/11/16 13:08, Ed Bartosh wrote:
> > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> > >> On 22/11/16 12:10, Patrick Ohly wrote:
> > >>>> ...
> > >>>
> > >>> All of these introduce some special mechanism. Let me propose something
> > >>> that might integrate better with the existing tooling:
> > >>>
> > >>> The "rootfs" directory gets redefined as representing the entire virtual
> > >>> file system. When creating a disk image, it gets split up into different
> > >>> partitions based on the image configuration.
> > >>>
> > >>> For example, the /home or /data directories in the rootfs could hold the
> > >>> content that in some image configurations goes into separate partitions.
> > >>>
> > >>> The advantage of this approach is that the tooling for staging content
> > >>> for image creation does not need to be changed. The same staged content
> > >>> then can be used to create different images, potentially even using
> > >>> different partition layouts.
> > >>
> > >> That's a very good idea. I think it beats all of my suggestions!
> > >>
> > >>> To implement this approach with wic, wic needs to be taught how to
> > >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> > >>> should also support that without having to make an intermediate copy of
> > >>> the files for a certain partition, but initially wic could create
> > >>> temporary directory trees.
> > >>
> > >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
> > >>
> > > 
> > > I support the idea. Let's discuss the details of implementation and
> > > create a bug in bugzilla to track the development 
> > 
> > Do you want me to create the ticket? (it'll be my first, so apologies in
> > advance if I omit something important)
> > 
> > > This can be done by extending existing rootfs plugin. It should be able
> > > to do 2 things:
> > > 
> > > - populate content of one rootfs directory to the partition. We can
> > >   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> > > 
> > > - exclude rootfs directories when populating partitions. I'd propose to
> > >   introduce --exclude-dirs wks parser option to handle this.
> > > 
> > > Example of wks file with proposed new options:
> > > part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> > > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> > > 
> > > Does this make sense?
> > 
> > Looks good. The only thing I would question is that, in the interest of
> > reducing redundancy, maybe we should omit --exclude-dirs and have wic
> > figure this out by combining all the entries, since "--exclude-dirs
> > <dir>" and the corresponding "part <dir>" will almost always come in
> > pairs. Possibly we could mark the "/" partition with one single
> > --no-overlapping-dirs to force wic to make this consideration. Or do you
> > think that's too magical?
> >
> Tt's quite implicit from my point of view. However, if people like it we
> can implement it this way.

I prefer the explicit --exclude-dirs. It's less surprising and perhaps
there are usages for having the same content in different partitions
(redundancy, factory reset, etc.).

Excluding only the directory content but not the actual directory is
indeed a good point. I'm a bit undecided. When excluding only the
directory content, there's no way of building a rootfs without that
mount point, if that's desired. OTOH, when excluding also the directory,
the data would have to be staged under a different path in the rootfs
and the mount point would have to be a separate, empty directory.

I'm leaning towards excluding the directory content and keeping the
directory.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Contents of non-rootfs partitions

[Ulrich Ölmann]

Hi,

On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> > > On 23/11/16 13:08, Ed Bartosh wrote:
> > > > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> > > > [...]
> > > > This can be done by extending existing rootfs plugin. It should be able
> > > > to do 2 things:
> > > > 
> > > > - populate content of one rootfs directory to the partition. We can
> > > >   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> > > > 
> > > > - exclude rootfs directories when populating partitions. I'd propose to
> > > >   introduce --exclude-dirs wks parser option to handle this.
> > > > 
> > > > Example of wks file with proposed new options:
> > > > part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> > > > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> > > > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> > > > 
> > > > Does this make sense?
> > > 
> > > Looks good. The only thing I would question is that, in the interest of
> > > reducing redundancy, maybe we should omit --exclude-dirs and have wic
> > > figure this out by combining all the entries, since "--exclude-dirs
> > > <dir>" and the corresponding "part <dir>" will almost always come in
> > > pairs. Possibly we could mark the "/" partition with one single
> > > --no-overlapping-dirs to force wic to make this consideration. Or do you
> > > think that's too magical?
> > >
> > Tt's quite implicit from my point of view. However, if people like it we
> > can implement it this way.
> 
> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
> there are usages for having the same content in different partitions
> (redundancy, factory reset, etc.).
> 
> Excluding only the directory content but not the actual directory is
> indeed a good point. I'm a bit undecided. When excluding only the
> directory content, there's no way of building a rootfs without that
> mount point, if that's desired. OTOH, when excluding also the directory,
> the data would have to be staged under a different path in the rootfs
> and the mount point would have to be a separate, empty directory.
> 
> I'm leaning towards excluding the directory content and keeping the
> directory.

what about having both possibilities by leaning against the syntax that rsync
uses to specify if a whole source directory or only it's contents shall be
synced to some destination site (see [1])?

In analogy to this to exclude only the contents of the directory named 'data'
you would use

  --exclude-dirs data/

but to additionally exclude the dir itself as well it would read

  --exclude-dirs data

Best regards
Ulrich

[1] http://man.cx/rsync(1)#heading6
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Contents of non-rootfs partitions

[Patrick Ohly]

On Thu, 2016-11-24 at 07:15 +0100, Ulrich Ölmann wrote:
> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> > Excluding only the directory content but not the actual directory is
> > indeed a good point. I'm a bit undecided. When excluding only the
> > directory content, there's no way of building a rootfs without that
> > mount point, if that's desired. OTOH, when excluding also the directory,
> > the data would have to be staged under a different path in the rootfs
> > and the mount point would have to be a separate, empty directory.
> > 
> > I'm leaning towards excluding the directory content and keeping the
> > directory.
> 
> what about having both possibilities by leaning against the syntax that rsync
> uses to specify if a whole source directory or only it's contents shall be
> synced to some destination site (see [1])?

I like that idea.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 24/11/16 07:15, Ulrich Ölmann wrote:
> Hi,
> 
> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
>>>> On 23/11/16 13:08, Ed Bartosh wrote:
>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
>>>>> [...]
>>>>> This can be done by extending existing rootfs plugin. It should be able
>>>>> to do 2 things:
>>>>>
>>>>> - populate content of one rootfs directory to the partition. We can
>>>>>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
>>>>>
>>>>> - exclude rootfs directories when populating partitions. I'd propose to
>>>>>   introduce --exclude-dirs wks parser option to handle this.
>>>>>
>>>>> Example of wks file with proposed new options:
>>>>> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
>>>>>
>>>>> Does this make sense?
>>>>
>>>> Looks good. The only thing I would question is that, in the interest of
>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
>>>> figure this out by combining all the entries, since "--exclude-dirs
>>>> <dir>" and the corresponding "part <dir>" will almost always come in
>>>> pairs. Possibly we could mark the "/" partition with one single
>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
>>>> think that's too magical?
>>>>
>>> Tt's quite implicit from my point of view. However, if people like it we
>>> can implement it this way.
>>
>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
>> there are usages for having the same content in different partitions
>> (redundancy, factory reset, etc.).
>>
>> Excluding only the directory content but not the actual directory is
>> indeed a good point. I'm a bit undecided. When excluding only the
>> directory content, there's no way of building a rootfs without that
>> mount point, if that's desired. OTOH, when excluding also the directory,
>> the data would have to be staged under a different path in the rootfs
>> and the mount point would have to be a separate, empty directory.
>>
>> I'm leaning towards excluding the directory content and keeping the
>> directory.
> 
> what about having both possibilities by leaning against the syntax that rsync
> uses to specify if a whole source directory or only it's contents shall be
> synced to some destination site (see [1])?
> 
> In analogy to this to exclude only the contents of the directory named 'data'
> you would use
> 
>   --exclude-dirs data/
> 
> but to additionally exclude the dir itself as well it would read
> 
>   --exclude-dirs data

This is creative, but ultimately too unintuitive IMHO. Rsync is the only
tool which uses this syntax AFAIK, and it's a constant source of
confusion, especially when mixed with cp or similar commands.

There is some discussion in the ticket as well:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10712

-- 
Kristian

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
> On 24/11/16 07:15, Ulrich Ölmann wrote:
> > Hi,
> > 
> > On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> >> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> >>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >>>> On 23/11/16 13:08, Ed Bartosh wrote:
> >>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >>>>> [...]
> >>>>> This can be done by extending existing rootfs plugin. It should be able
> >>>>> to do 2 things:
> >>>>>
> >>>>> - populate content of one rootfs directory to the partition. We can
> >>>>>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> >>>>>
> >>>>> - exclude rootfs directories when populating partitions. I'd propose to
> >>>>>   introduce --exclude-dirs wks parser option to handle this.
> >>>>>
> >>>>> Example of wks file with proposed new options:
> >>>>> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> >>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>
> >>>>> Does this make sense?
> >>>>
> >>>> Looks good. The only thing I would question is that, in the interest of
> >>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >>>> figure this out by combining all the entries, since "--exclude-dirs
> >>>> <dir>" and the corresponding "part <dir>" will almost always come in
> >>>> pairs. Possibly we could mark the "/" partition with one single
> >>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
> >>>> think that's too magical?
> >>>>
> >>> Tt's quite implicit from my point of view. However, if people like it we
> >>> can implement it this way.
> >>
> >> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
> >> there are usages for having the same content in different partitions
> >> (redundancy, factory reset, etc.).
> >>
> >> Excluding only the directory content but not the actual directory is
> >> indeed a good point. I'm a bit undecided. When excluding only the
> >> directory content, there's no way of building a rootfs without that
> >> mount point, if that's desired. OTOH, when excluding also the directory,
> >> the data would have to be staged under a different path in the rootfs
> >> and the mount point would have to be a separate, empty directory.
> >>
> >> I'm leaning towards excluding the directory content and keeping the
> >> directory.
> > 
> > what about having both possibilities by leaning against the syntax that rsync
> > uses to specify if a whole source directory or only it's contents shall be
> > synced to some destination site (see [1])?
> > 
> > In analogy to this to exclude only the contents of the directory named 'data'
> > you would use
> > 
> >   --exclude-dirs data/
> > 
> > but to additionally exclude the dir itself as well it would read
> > 
> >   --exclude-dirs data
> 
> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
> tool which uses this syntax AFAIK, and it's a constant source of
> confusion, especially when mixed with cp or similar commands.
> 

Would this way be less intuitive?
--exclude-path data/*
--exclude-path data

We can go even further with it allowing any level of directories:
--exclude-path data/tmp/*
--exclude-path data/db/tmp
...

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 24/11/16 14:23, Ed Bartosh wrote:
> On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
>> On 24/11/16 07:15, Ulrich Ölmann wrote:
>>> Hi,
>>>
>>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
>>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
>>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
>>>>>> On 23/11/16 13:08, Ed Bartosh wrote:
>>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
>>>>>>> [...]
>>>>>>> This can be done by extending existing rootfs plugin. It should be able
>>>>>>> to do 2 things:
>>>>>>>
>>>>>>> - populate content of one rootfs directory to the partition. We can
>>>>>>>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
>>>>>>>
>>>>>>> - exclude rootfs directories when populating partitions. I'd propose to
>>>>>>>   introduce --exclude-dirs wks parser option to handle this.
>>>>>>>
>>>>>>> Example of wks file with proposed new options:
>>>>>>> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
>>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
>>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
>>>>>>>
>>>>>>> Does this make sense?
>>>>>>
>>>>>> Looks good. The only thing I would question is that, in the interest of
>>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
>>>>>> figure this out by combining all the entries, since "--exclude-dirs
>>>>>> <dir>" and the corresponding "part <dir>" will almost always come in
>>>>>> pairs. Possibly we could mark the "/" partition with one single
>>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
>>>>>> think that's too magical?
>>>>>>
>>>>> Tt's quite implicit from my point of view. However, if people like it we
>>>>> can implement it this way.
>>>>
>>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
>>>> there are usages for having the same content in different partitions
>>>> (redundancy, factory reset, etc.).
>>>>
>>>> Excluding only the directory content but not the actual directory is
>>>> indeed a good point. I'm a bit undecided. When excluding only the
>>>> directory content, there's no way of building a rootfs without that
>>>> mount point, if that's desired. OTOH, when excluding also the directory,
>>>> the data would have to be staged under a different path in the rootfs
>>>> and the mount point would have to be a separate, empty directory.
>>>>
>>>> I'm leaning towards excluding the directory content and keeping the
>>>> directory.
>>>
>>> what about having both possibilities by leaning against the syntax that rsync
>>> uses to specify if a whole source directory or only it's contents shall be
>>> synced to some destination site (see [1])?
>>>
>>> In analogy to this to exclude only the contents of the directory named 'data'
>>> you would use
>>>
>>>   --exclude-dirs data/
>>>
>>> but to additionally exclude the dir itself as well it would read
>>>
>>>   --exclude-dirs data
>>
>> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
>> tool which uses this syntax AFAIK, and it's a constant source of
>> confusion, especially when mixed with cp or similar commands.
>>
> 
> Would this way be less intuitive?
> --exclude-path data/*
> --exclude-path data
> 
> We can go even further with it allowing any level of directories:
> --exclude-path data/tmp/*
> --exclude-path data/db/tmp
> ...

I agree, this is pretty unambiguous and easy to understand.

But this raises the question: Should we go all the way and support
wildcards? Which might make it a bit complicated. Maybe support only
pure '*' for now?

-- 
Kristian

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote:
> On 24/11/16 14:23, Ed Bartosh wrote:
> > On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
> >> On 24/11/16 07:15, Ulrich Ölmann wrote:
> >>> Hi,
> >>>
> >>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> >>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> >>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >>>>>> On 23/11/16 13:08, Ed Bartosh wrote:
> >>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >>>>>>> [...]
> >>>>>>> This can be done by extending existing rootfs plugin. It should be able
> >>>>>>> to do 2 things:
> >>>>>>>
> >>>>>>> - populate content of one rootfs directory to the partition. We can
> >>>>>>>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
> >>>>>>>
> >>>>>>> - exclude rootfs directories when populating partitions. I'd propose to
> >>>>>>>   introduce --exclude-dirs wks parser option to handle this.
> >>>>>>>
> >>>>>>> Example of wks file with proposed new options:
> >>>>>>> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
> >>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>>>
> >>>>>>> Does this make sense?
> >>>>>>
> >>>>>> Looks good. The only thing I would question is that, in the interest of
> >>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >>>>>> figure this out by combining all the entries, since "--exclude-dirs
> >>>>>> <dir>" and the corresponding "part <dir>" will almost always come in
> >>>>>> pairs. Possibly we could mark the "/" partition with one single
> >>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
> >>>>>> think that's too magical?
> >>>>>>
> >>>>> Tt's quite implicit from my point of view. However, if people like it we
> >>>>> can implement it this way.
> >>>>
> >>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
> >>>> there are usages for having the same content in different partitions
> >>>> (redundancy, factory reset, etc.).
> >>>>
> >>>> Excluding only the directory content but not the actual directory is
> >>>> indeed a good point. I'm a bit undecided. When excluding only the
> >>>> directory content, there's no way of building a rootfs without that
> >>>> mount point, if that's desired. OTOH, when excluding also the directory,
> >>>> the data would have to be staged under a different path in the rootfs
> >>>> and the mount point would have to be a separate, empty directory.
> >>>>
> >>>> I'm leaning towards excluding the directory content and keeping the
> >>>> directory.
> >>>
> >>> what about having both possibilities by leaning against the syntax that rsync
> >>> uses to specify if a whole source directory or only it's contents shall be
> >>> synced to some destination site (see [1])?
> >>>
> >>> In analogy to this to exclude only the contents of the directory named 'data'
> >>> you would use
> >>>
> >>>   --exclude-dirs data/
> >>>
> >>> but to additionally exclude the dir itself as well it would read
> >>>
> >>>   --exclude-dirs data
> >>
> >> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
> >> tool which uses this syntax AFAIK, and it's a constant source of
> >> confusion, especially when mixed with cp or similar commands.
> >>
> > 
> > Would this way be less intuitive?
> > --exclude-path data/*
> > --exclude-path data
> > 
> > We can go even further with it allowing any level of directories:
> > --exclude-path data/tmp/*
> > --exclude-path data/db/tmp
> > ...
> 
> I agree, this is pretty unambiguous and easy to understand.
> 
> But this raises the question: Should we go all the way and support
> wildcards? Which might make it a bit complicated. Maybe support only
> pure '*' for now?

As it shouldn't be hard to implement I'd go for it.

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Patrick Ohly]

On Thu, 2016-11-24 at 16:51 +0200, Ed Bartosh wrote:
> On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote:
> > On 24/11/16 14:23, Ed Bartosh wrote:
> > > Would this way be less intuitive?
> > > --exclude-path data/*
> > > --exclude-path data
> > > 
> > > We can go even further with it allowing any level of directories:
> > > --exclude-path data/tmp/*

Just to clarify, that is meant to also match data/tmp/.hidden-file,
right? I.e. fnmatch() without the special FNM_PERIOD flag.

> > > --exclude-path data/db/tmp
> > > ...
> > 
> > I agree, this is pretty unambiguous and easy to understand.
> > 
> > But this raises the question: Should we go all the way and support
> > wildcards? Which might make it a bit complicated. Maybe support only
> > pure '*' for now?
> 
> As it shouldn't be hard to implement I'd go for it.

Additional code implies additional testing. Remember that it also should
better be supported by mkfs.ext4 and friends.

I'd rather start simple and only add additional complexity when needed.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 24/11/16 16:24, Patrick Ohly wrote:
> On Thu, 2016-11-24 at 16:51 +0200, Ed Bartosh wrote:
>> On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote:
>>> On 24/11/16 14:23, Ed Bartosh wrote:
>>>> Would this way be less intuitive?
>>>> --exclude-path data/*
>>>> --exclude-path data
>>>>
>>>> We can go even further with it allowing any level of directories:
>>>> --exclude-path data/tmp/*
> 
> Just to clarify, that is meant to also match data/tmp/.hidden-file,
> right? I.e. fnmatch() without the special FNM_PERIOD flag.

Yes, I think that makes the most sense, even if it breaks with shell
defaults.

-- 
Kristian

Re: Contents of non-rootfs partitions

[Andreas Oberritter]

On 24.11.2016 15:51, Ed Bartosh wrote:
> On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote:
>> On 24/11/16 14:23, Ed Bartosh wrote:
>>> On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
>>>> On 24/11/16 07:15, Ulrich Ölmann wrote:
>>>>> Hi,
>>>>>
>>>>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
>>>>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
>>>>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
>>>>>>>> On 23/11/16 13:08, Ed Bartosh wrote:
>>>>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
>>>>>>>>> [...]
>>>>>>>>> This can be done by extending existing rootfs plugin. It should be able
>>>>>>>>> to do 2 things:
>>>>>>>>>
>>>>>>>>> - populate content of one rootfs directory to the partition. We can
>>>>>>>>>   extend syntax of --rootfs-dir parameter to specify optional directory path to use
>>>>>>>>>
>>>>>>>>> - exclude rootfs directories when populating partitions. I'd propose to
>>>>>>>>>   introduce --exclude-dirs wks parser option to handle this.
>>>>>>>>>
>>>>>>>>> Example of wks file with proposed new options:
>>>>>>>>> part /     --source rootfs --rootfs-dir=core-image-minimal       --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
>>>>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda --fstype=ext4 --label data --align 1024
>>>>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda --fstype=ext4 --label data --align 1024
>>>>>>>>>
>>>>>>>>> Does this make sense?
>>>>>>>>
>>>>>>>> Looks good. The only thing I would question is that, in the interest of
>>>>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
>>>>>>>> figure this out by combining all the entries, since "--exclude-dirs
>>>>>>>> <dir>" and the corresponding "part <dir>" will almost always come in
>>>>>>>> pairs. Possibly we could mark the "/" partition with one single
>>>>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
>>>>>>>> think that's too magical?
>>>>>>>>
>>>>>>> Tt's quite implicit from my point of view. However, if people like it we
>>>>>>> can implement it this way.
>>>>>>
>>>>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
>>>>>> there are usages for having the same content in different partitions
>>>>>> (redundancy, factory reset, etc.).
>>>>>>
>>>>>> Excluding only the directory content but not the actual directory is
>>>>>> indeed a good point. I'm a bit undecided. When excluding only the
>>>>>> directory content, there's no way of building a rootfs without that
>>>>>> mount point, if that's desired. OTOH, when excluding also the directory,
>>>>>> the data would have to be staged under a different path in the rootfs
>>>>>> and the mount point would have to be a separate, empty directory.
>>>>>>
>>>>>> I'm leaning towards excluding the directory content and keeping the
>>>>>> directory.
>>>>>
>>>>> what about having both possibilities by leaning against the syntax that rsync
>>>>> uses to specify if a whole source directory or only it's contents shall be
>>>>> synced to some destination site (see [1])?
>>>>>
>>>>> In analogy to this to exclude only the contents of the directory named 'data'
>>>>> you would use
>>>>>
>>>>>   --exclude-dirs data/
>>>>>
>>>>> but to additionally exclude the dir itself as well it would read
>>>>>
>>>>>   --exclude-dirs data
>>>>
>>>> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
>>>> tool which uses this syntax AFAIK, and it's a constant source of
>>>> confusion, especially when mixed with cp or similar commands.
>>>>
>>>
>>> Would this way be less intuitive?
>>> --exclude-path data/*
>>> --exclude-path data
>>>
>>> We can go even further with it allowing any level of directories:
>>> --exclude-path data/tmp/*
>>> --exclude-path data/db/tmp
>>> ...
>>
>> I agree, this is pretty unambiguous and easy to understand.
>>
>> But this raises the question: Should we go all the way and support
>> wildcards? Which might make it a bit complicated. Maybe support only
>> pure '*' for now?
> 
> As it shouldn't be hard to implement I'd go for it.

Note that in a shell, "data/*" doesn't include "data/.*". So this syntax
avoids the confusing rsync syntax in trade for another one. I'd prefer
the rsync flavour, because "behaves like rsync" is easier to remember
than "behaves like a shell glob, but differently". Confusion could be
prevented with good documentation, FWIW.

Regards,
Andreas

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 24/11/16 16:28, Andreas Oberritter wrote:
> Note that in a shell, "data/*" doesn't include "data/.*". So this syntax
> avoids the confusing rsync syntax in trade for another one. I'd prefer
> the rsync flavour, because "behaves like rsync" is easier to remember
> than "behaves like a shell glob, but differently". Confusion could be
> prevented with good documentation, FWIW.

This is turning into a bit of a bikeshed discussion. I have a slight
preference for the glob version, because I think it's less surprising,
but I'm fine either way.

I count myself and Ed in favor of glob version, Ulrich and Andreas in
favor of rsync version. Patrick, you wanna be tie breaker? :-)

-- 
Kristian

Re: Contents of non-rootfs partitions

[Patrick Ohly]

On Thu, 2016-11-24 at 16:51 +0100, Kristian Amlie wrote:
> On 24/11/16 16:28, Andreas Oberritter wrote:
> > Note that in a shell, "data/*" doesn't include "data/.*". So this syntax
> > avoids the confusing rsync syntax in trade for another one. I'd prefer
> > the rsync flavour, because "behaves like rsync" is easier to remember
> > than "behaves like a shell glob, but differently". Confusion could be
> > prevented with good documentation, FWIW.
> 
> This is turning into a bit of a bikeshed discussion. I have a slight
> preference for the glob version, because I think it's less surprising,
> but I'm fine either way.
> 
> I count myself and Ed in favor of glob version, Ulrich and Andreas in
> favor of rsync version. Patrick, you wanna be tie breaker? :-)

But I said I was undecided ;-} Okay, I vote for the rsync flavor.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 24/11/16 17:21, Patrick Ohly wrote:
> On Thu, 2016-11-24 at 16:51 +0100, Kristian Amlie wrote:
>> On 24/11/16 16:28, Andreas Oberritter wrote:
>>> Note that in a shell, "data/*" doesn't include "data/.*". So this syntax
>>> avoids the confusing rsync syntax in trade for another one. I'd prefer
>>> the rsync flavour, because "behaves like rsync" is easier to remember
>>> than "behaves like a shell glob, but differently". Confusion could be
>>> prevented with good documentation, FWIW.
>>
>> This is turning into a bit of a bikeshed discussion. I have a slight
>> preference for the glob version, because I think it's less surprising,
>> but I'm fine either way.
>>
>> I count myself and Ed in favor of glob version, Ulrich and Andreas in
>> favor of rsync version. Patrick, you wanna be tie breaker? :-)
> 
> But I said I was undecided ;-} Okay, I vote for the rsync flavor.

Rsync flavor it is then, unless anyone has strong objections!

-- 
Kristian

Re: Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> On 23/11/16 13:08, Ed Bartosh wrote:
> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >>>> ...
> >>>
> >>> All of these introduce some special mechanism. Let me propose something
> >>> that might integrate better with the existing tooling:
> >>>
> >>> The "rootfs" directory gets redefined as representing the entire virtual
> >>> file system. When creating a disk image, it gets split up into different
> >>> partitions based on the image configuration.
> >>>
> >>> For example, the /home or /data directories in the rootfs could hold the
> >>> content that in some image configurations goes into separate partitions.
> >>>
> >>> The advantage of this approach is that the tooling for staging content
> >>> for image creation does not need to be changed. The same staged content
> >>> then can be used to create different images, potentially even using
> >>> different partition layouts.
> >>
> >> That's a very good idea. I think it beats all of my suggestions!
> >>
> >>> To implement this approach with wic, wic needs to be taught how to
> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >>> should also support that without having to make an intermediate copy of
> >>> the files for a certain partition, but initially wic could create
> >>> temporary directory trees.
> >>
> >> Yes, some work would be needed here, but ultimately it would be contained within wic and related tools, which is a good thing.
> >>
> > 
> > I support the idea. Let's discuss the details of implementation and
> > create a bug in bugzilla to track the development 
> 
> Do you want me to create the ticket? (it'll be my first, so apologies in
> advance if I omit something important)
> 
Yes, please create an issue in bugzilla and assign it to me. Don't
worry. If you miss something other people will point to in in the
comments :)

--
Regards,
Ed

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 23/11/16 14:44, Ed Bartosh wrote:
> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
>> On 23/11/16 13:08, Ed Bartosh wrote:
>>> I support the idea. Let's discuss the details of implementation and
>>> create a bug in bugzilla to track the development 
>>
>> Do you want me to create the ticket? (it'll be my first, so apologies in
>> advance if I omit something important)
>>
> Yes, please create an issue in bugzilla and assign it to me. Don't
> worry. If you miss something other people will point to in in the
> comments :)

Done, I have added https://bugzilla.yoctoproject.org/show_bug.cgi?id=10712

-- 
Kristian

Re: Contents of non-rootfs partitions

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1842 bytes --]



On 11/22/16 1:05 AM, Kristian Amlie wrote:
> Hey
> 
> In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building.
> 
> However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition.
> 
> I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done:
> 
> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic.
> 
> * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined.
> 

Firstly, it needs to be defined what the data partition is about.
1. is it some pre-initialized/seeded data that can then be modified during
subsequent boots ?
2. Is it read-only data which is then used to populate r/w data segments in
RAM during boot
3. is it data that does not change over the course of install and reboots of
the device

> * Have a special recipe that users can bbappend, which specifically install to the data partition (this might also need a "data" directory, like above).
> 
> It's also possible that this should be generalized further so that it doesn't only apply to a data partition, but to any non-rootfs partition.
> 
> I'm willing to invest some time in this, but I thought I'd gather some opinions first.
> 
> Thoughts?
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 23/11/16 00:45, Khem Raj wrote:
> On 11/22/16 1:05 AM, Kristian Amlie wrote:
>> Hey
>>
>> In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building.
>>
>> However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition.
>>
>> I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done:
>>
>> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic.
>>
>> * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined.
>>
> 
> Firstly, it needs to be defined what the data partition is about.
> 1. is it some pre-initialized/seeded data that can then be modified during
> subsequent boots ?
> 2. Is it read-only data which is then used to populate r/w data segments in
> RAM during boot
> 3. is it data that does not change over the course of install and reboots of
> the device

It's preseeded, but modifiable data, that survives complete rootfs
updates. IOW if you overwrite either of your A/B rootfs partitions, the
data partition and its content survives.

-- 
Kristian

Re: Contents of non-rootfs partitions

[Khem Raj]

> On Nov 22, 2016, at 11:13 PM, Kristian Amlie <kristian.amlie@mender.io> wrote:
> 
> On 23/11/16 00:45, Khem Raj wrote:
>> On 11/22/16 1:05 AM, Kristian Amlie wrote:
>>> Hey
>>> 
>>> In Mender we are using a dual rootfs partition layout (A/B layout) with a persistent data partition on the side. We're using wic to do the actual image building.
>>> 
>>> However, putting files into this data partition is not a straightforward operation ATM. All recipes in OE put files into the rootfs only, and we'd like there to be a way to specify that files should go into the data partition.
>>> 
>>> I haven't seen any straightforward way to do this in OE. Some suggestions for how it could be done:
>>> 
>>> * A "DATA_FILES" variable (like "IMAGE_BOOT_FILES"), where you can list files you'd like to be included. This is very straightforward, but has the disadvantage of being a bit hairy when paths are involved, and it doesn't allow for much logic.
>>> 
>>> * Keep a special "data" directory, something similar to sysroot, where recipes could install files into in order to have it included on the data partition. We could potentially have a bbclass for use in recipes to make this more streamlined.
>>> 
>> 
>> Firstly, it needs to be defined what the data partition is about.
>> 1. is it some pre-initialized/seeded data that can then be modified during
>> subsequent boots ?
>> 2. Is it read-only data which is then used to populate r/w data segments in
>> RAM during boot
>> 3. is it data that does not change over the course of install and reboots of
>> the device
> 
> It's preseeded, but modifiable data, that survives complete rootfs
> updates. IOW if you overwrite either of your A/B rootfs partitions, the
> data partition and its content survives.

at some point this data segment will be seeded.
think of factory case. 

So in such a case updates should not touch them unless user does a factory reset
which should clean it up

> 
> -- 
> Kristian

Re: Contents of non-rootfs partitions

[Kristian Amlie]

On 23/11/16 08:26, Khem Raj wrote:
> 
>> On Nov 22, 2016, at 11:13 PM, Kristian Amlie <kristian.amlie@mender.io> wrote:
>>
>> On 23/11/16 00:45, Khem Raj wrote:
>>> Firstly, it needs to be defined what the data partition is about.
>>> 1. is it some pre-initialized/seeded data that can then be modified during
>>> subsequent boots ?
>>> 2. Is it read-only data which is then used to populate r/w data segments in
>>> RAM during boot
>>> 3. is it data that does not change over the course of install and reboots of
>>> the device
>>
>> It's preseeded, but modifiable data, that survives complete rootfs
>> updates. IOW if you overwrite either of your A/B rootfs partitions, the
>> data partition and its content survives.
> 
> at some point this data segment will be seeded.
> think of factory case. 

The creation of the initial partitioned image for the memory card
preseeds the data partition. This is meant to be used in initial factory
flashing (by which I mean literally "at the factory", not the factory
reset button). Subsequent rootfs updates would not touch the data partition.

> So in such a case updates should not touch them unless user does a factory reset
> which should clean it up

Factory reset is not covered in my suggestion. While also important, I
consider this a separate feature, which would require cooperation by the
running OS. I hope it's acceptable to keep this out of the first patchset?

-- 
Kristian