Wic and "live" images

Wic and "live" images

[Ian Geiser]

Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.  

My desired partition layout is as follows:
      40MiB                 40MiB               300MiB
+--------------------+-----------------+-----------------------------+
|      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |  
+--------------------+-----------------+-----------------------------+

BOOT - efi boot partition with kernel and initramfs
DATA - generic fat filesystem to hold configuration files
ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file.

I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons.

Any hints?

Thanks!

Re: Wic and "live" images

[Ed Bartosh]

On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.  
> 
> My desired partition layout is as follows:
>       40MiB                 40MiB               300MiB
> +--------------------+-----------------+-----------------------------+
> |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |  
> +--------------------+-----------------+-----------------------------+
> 
> BOOT - efi boot partition with kernel and initramfs
> DATA - generic fat filesystem to hold configuration files
> ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file.
> 
> I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons.
> 
> Any hints?
I'd suggest to use wic image type and generate your image by bitbake.
You can find example wic-image-minimal.bb and wic-image-minimal.wks in ../meta-selftest/recipes-test/images/

You can probably do the same by using wic plugins, but I'd not suggest
to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation.

--
Regards,
Ed

Re: Wic and "live" images

[Ian Geiser]

 ---- On Mon, 23 May 2016 06:36:23 -0400 Ed Bartosh <ed.bartosh@linux.intel.com> wrote ---- 
 > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote: 
 > > Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.   
 > >  
 > > My desired partition layout is as follows: 
 > >       40MiB                 40MiB               300MiB 
 > > +--------------------+-----------------+-----------------------------+ 
 > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |   
 > > +--------------------+-----------------+-----------------------------+ 
 > >  
 > > BOOT - efi boot partition with kernel and initramfs 
 > > DATA - generic fat filesystem to hold configuration files 
 > > ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file. 
 > >  
 > > I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons. 
 > >  
 > > Any hints? 
 > I'd suggest to use wic image type and generate your image by bitbake. 
 > You can find example wic-image-minimal.bb and wic-image-minimal.wks in ../meta-selftest/recipes-test/images/ 
 >  
This is where I started.  I was able to make it work but not with my configuration above.  It looks like I can use a type of "fsimage" for my "ROOT" partition, but I have not been able to figure out the syntax there yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but there doesn't seem to be a way to provide an initramfs.

 > You can probably do the same by using wic plugins, but I'd not suggest 
 > to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation. 

Is using the wic image type and a plugin mutually exclusive?

Thanks!

Re: Wic and "live" images

[Sergey 'Jin' Bostandzhyan]

Hi,

I'm risking to land in the "reinvented the wheel" corner, but basically I
had a task to create images with msdos partitions, while at the same time
some things had to be in areas where parted/fdisk were not able to create a
partition yet (i.e. u-boot had to be at a very early offset in the image).

wic seemed too PC oriented and cumbersome to me, so I came up with
a custom class:

https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/classes/msdos_partition_image.bbclass

I have a multimachine configuration which then looks like this (example with
raw image outside a partition):
https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/recipes-core/images/dss20-image.inc

Or this (example with extended partition):
https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/recipes-core/images/dss11e-image.inc

The DSSIMG_TASK_DEPENDS variable lists all things that will be
written into the final image (i.e. u-boot, rescue fs, root fs).

If someone considers this interesting for OE in general, I can surely tune
a few things and submit a patch for review.

Kind regards,
Jin



On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
>  ---- On Mon, 23 May 2016 06:36:23 -0400 Ed Bartosh <ed.bartosh@linux.intel.com> wrote ---- 
>  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote: 
>  > > Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.   
>  > >  
>  > > My desired partition layout is as follows: 
>  > >       40MiB                 40MiB               300MiB 
>  > > +--------------------+-----------------+-----------------------------+ 
>  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |   
>  > > +--------------------+-----------------+-----------------------------+ 
>  > >  
>  > > BOOT - efi boot partition with kernel and initramfs 
>  > > DATA - generic fat filesystem to hold configuration files 
>  > > ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file. 
>  > >  
>  > > I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons. 
>  > >  
>  > > Any hints? 
>  > I'd suggest to use wic image type and generate your image by bitbake. 
>  > You can find example wic-image-minimal.bb and wic-image-minimal.wks in ../meta-selftest/recipes-test/images/ 
>  >  
> This is where I started.  I was able to make it work but not with my configuration above.  It looks like I can use a type of "fsimage" for my "ROOT" partition, but I have not been able to figure out the syntax there yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but there doesn't seem to be a way to provide an initramfs.
> 
>  > You can probably do the same by using wic plugins, but I'd not suggest 
>  > to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation. 
> 
> Is using the wic image type and a plugin mutually exclusive?
> 
> Thanks!
> 
> 
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Wic and "live" images

[Ian Geiser]

 ---- On Mon, 23 May 2016 09:00:28 -0400 Sergey &#39;Jin&#39; Bostandzhyan <jin@mediatomb.cc> wrote ---- 
 > Hi, 
 >  
 > I'm risking to land in the "reinvented the wheel" corner, but basically I 
 > had a task to create images with msdos partitions, while at the same time 
 > some things had to be in areas where parted/fdisk were not able to create a 
 > partition yet (i.e. u-boot had to be at a very early offset in the image). 
 >  
 > wic seemed too PC oriented and cumbersome to me, so I came up with 
 > a custom class: 
 >  
 > https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/classes/msdos_partition_image.bbclass 
 >  
This is more or less what I have now.  I was trying to move to 'wic' because the 'image-vm.bbclass' cannot handle some of the things I want.  From the manual it sounds like wic is more flexible, but I think it is only flexible in the way you can use plugins to create new partition types.  It also lacks any sort of dependency tracking with bitbake so you cannot use it from there without a lot of manual hand holding.  I think for the limited types it currently handles it is less capable than image-vm.bbclass.

 > I have a multimachine configuration which then looks like this (example with 
 > raw image outside a partition): 
 > https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/recipes-core/images/dss20-image.inc 
 >  
 > Or this (example with extended partition): 
 > https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/recipes-core/images/dss11e-image.inc 
 >  
 > The DSSIMG_TASK_DEPENDS variable lists all things that will be 
 > written into the final image (i.e. u-boot, rescue fs, root fs). 
 >  
 > If someone considers this interesting for OE in general, I can surely tune 
 > a few things and submit a patch for review. 
 
I do find this very interesting.  From the looks of it there seems to be more extensibility than can be had with image-vm.bbclass or wic.

>  
 > Kind regards, 
 > Jin 
 >  
 >  
 >  
 > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote: 
 > >  ---- On Mon, 23 May 2016 06:36:23 -0400 Ed Bartosh <ed.bartosh@linux.intel.com> wrote ----  
 > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:  
 > >  > > Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.    
 > >  > >   
 > >  > > My desired partition layout is as follows:  
 > >  > >       40MiB                 40MiB               300MiB  
 > >  > > +--------------------+-----------------+-----------------------------+  
 > >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |    
 > >  > > +--------------------+-----------------+-----------------------------+  
 > >  > >   
 > >  > > BOOT - efi boot partition with kernel and initramfs  
 > >  > > DATA - generic fat filesystem to hold configuration files  
 > >  > > ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file.  
 > >  > >   
 > >  > > I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons.  
 > >  > >   
 > >  > > Any hints?  
 > >  > I'd suggest to use wic image type and generate your image by bitbake.  
 > >  > You can find example wic-image-minimal.bb and wic-image-minimal.wks in ../meta-selftest/recipes-test/images/  
 > >  >   
 > > This is where I started.  I was able to make it work but not with my configuration above.  It looks like I can use a type of "fsimage" for my "ROOT" partition, but I have not been able to figure out the syntax there yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but there doesn't seem to be a way to provide an initramfs. 
 > >  
 > >  > You can probably do the same by using wic plugins, but I'd not suggest  
 > >  > to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation.  
 > >  
 > > Is using the wic image type and a plugin mutually exclusive? 
 > >  
 > > Thanks! 
 > >  
 > >  
 > >  
 > > --  
 > > _______________________________________________ 
 > > Openembedded-core mailing list 
 > > Openembedded-core@lists.openembedded.org 
 > > http://lists.openembedded.org/mailman/listinfo/openembedded-core 
 > 

Re: Wic and "live" images

[Ed Bartosh]

On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
>  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote: 
>  > > Greetings, I am trying to learn "wic" and have been confused as how to create a "live" style image.  I am following "http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images" but am getting confused on the target to use to create the a file system that has a single squashfs file containing my root file system.   
>  > >  
>  > > My desired partition layout is as follows: 
>  > >       40MiB                 40MiB               300MiB 
>  > > +--------------------+-----------------+-----------------------------+ 
>  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)        |   
>  > > +--------------------+-----------------+-----------------------------+ 
>  > >  
>  > > BOOT - efi boot partition with kernel and initramfs 
>  > > DATA - generic fat filesystem to hold configuration files 
>  > > ROOT - an ext4 filesystem that contains a single os.img, which is a squashfs file. 
>  > >  
>  > > I have ROOT and DATA figured out but I am at a loss as how to generate the os.img file and copy it into ROOT.  If I generate the os.img file with bitbake and then use the "-r" option to manually supply a directory structure it works, but I would rather have it done from a wks file for automation reasons. 
>  > >  
>  > > Any hints? 
>  > I'd suggest to use wic image type and generate your image by bitbake. 
>  > You can find example wic-image-minimal.bb and wic-image-minimal.wks in ../meta-selftest/recipes-test/images/ 
>  >  
> This is where I started.  I was able to make it work but not with my configuration above.  It looks like I can use a type of "fsimage" for my "ROOT" partition, but I have not been able to figure out the syntax there yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but there doesn't seem to be a way to provide an initramfs.

How about creating recipe to prepare content or your boot partition and then using --source rootfs --rootfs-dir=<your recipe> ?
This is much more generic way of creating partitioned images from my
point of view. Image recipes should take care of content and wic takes care of
putting that content into partitions according to the partitioning
scheme described in .wks

Does it make sense for you?

> 
>  > You can probably do the same by using wic plugins, but I'd not suggest 
>  > to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation. 
> 
> Is using the wic image type and a plugin mutually exclusive?
No, not at all. However, I personally found the way I described above
more consistent, flexible and easy to implement and maintain.

--
Regards,
Ed

Re: Wic and "live" images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
> >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> >  > > Greetings, I am trying to learn "wic" and have been confused as how
> to create a "live" style image.  I am following "
> http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images"
> but am getting confused on the target to use to create the a file system
> that has a single squashfs file containing my root file system.
> >  > >
> >  > > My desired partition layout is as follows:
> >  > >       40MiB                 40MiB               300MiB
> >  > >
> +--------------------+-----------------+-----------------------------+
> >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)
>   |
> >  > >
> +--------------------+-----------------+-----------------------------+
> >  > >
> >  > > BOOT - efi boot partition with kernel and initramfs
> >  > > DATA - generic fat filesystem to hold configuration files
> >  > > ROOT - an ext4 filesystem that contains a single os.img, which is a
> squashfs file.
> >  > >
> >  > > I have ROOT and DATA figured out but I am at a loss as how to
> generate the os.img file and copy it into ROOT.  If I generate the os.img
> file with bitbake and then use the "-r" option to manually supply a
> directory structure it works, but I would rather have it done from a wks
> file for automation reasons.
> >  > >
> >  > > Any hints?
> >  > I'd suggest to use wic image type and generate your image by bitbake.
> >  > You can find example wic-image-minimal.bb and wic-image-minimal.wks
> in ../meta-selftest/recipes-test/images/
> >  >
> > This is where I started.  I was able to make it work but not with my
> configuration above.  It looks like I can use a type of "fsimage" for my
> "ROOT" partition, but I have not been able to figure out the syntax there
> yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but
> there doesn't seem to be a way to provide an initramfs.
>
> How about creating recipe to prepare content or your boot partition and
> then using --source rootfs --rootfs-dir=<your recipe> ?
> This is much more generic way of creating partitioned images from my
> point of view. Image recipes should take care of content and wic takes
> care of
> putting that content into partitions according to the partitioning
> scheme described in .wks
>
> Does it make sense for you?
>
> >
> >  > You can probably do the same by using wic plugins, but I'd not suggest
> >  > to go this way. Using wic image type is simpler, more consistent,
> easier to do and provides higher level of automation.
> >
> > Is using the wic image type and a plugin mutually exclusive?
> No, not at all. However, I personally found the way I described above
> more consistent, flexible and easy to implement and maintain.
>

The thing is, it's likely the machine/bsp setting the WKS_FILE, yet in
OE/yocto we prefer machine/distro/image to be orthogonal. If you're
injecting machine specific logic into an image, that image isn't going to
be generally useful for all machines, and so violates our philosophy.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4439 bytes --]

Re: Wic and "live" images

[Ed Bartosh]

On Tue, May 24, 2016 at 12:56:39PM -0700, Christopher Larson wrote:
> On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
> 
> > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
> > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> > >  > > Greetings, I am trying to learn "wic" and have been confused as how
> > to create a "live" style image.  I am following "
> > http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images"
> > but am getting confused on the target to use to create the a file system
> > that has a single squashfs file containing my root file system.
> > >  > >
> > >  > > My desired partition layout is as follows:
> > >  > >       40MiB                 40MiB               300MiB
> > >  > >
> > +--------------------+-----------------+-----------------------------+
> > >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)
> >   |
> > >  > >
> > +--------------------+-----------------+-----------------------------+
> > >  > >
> > >  > > BOOT - efi boot partition with kernel and initramfs
> > >  > > DATA - generic fat filesystem to hold configuration files
> > >  > > ROOT - an ext4 filesystem that contains a single os.img, which is a
> > squashfs file.
> > >  > >
> > >  > > I have ROOT and DATA figured out but I am at a loss as how to
> > generate the os.img file and copy it into ROOT.  If I generate the os.img
> > file with bitbake and then use the "-r" option to manually supply a
> > directory structure it works, but I would rather have it done from a wks
> > file for automation reasons.
> > >  > >
> > >  > > Any hints?
> > >  > I'd suggest to use wic image type and generate your image by bitbake.
> > >  > You can find example wic-image-minimal.bb and wic-image-minimal.wks
> > in ../meta-selftest/recipes-test/images/
> > >  >
> > > This is where I started.  I was able to make it work but not with my
> > configuration above.  It looks like I can use a type of "fsimage" for my
> > "ROOT" partition, but I have not been able to figure out the syntax there
> > yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but
> > there doesn't seem to be a way to provide an initramfs.
> >
> > How about creating recipe to prepare content or your boot partition and
> > then using --source rootfs --rootfs-dir=<your recipe> ?
> > This is much more generic way of creating partitioned images from my
> > point of view. Image recipes should take care of content and wic takes
> > care of
> > putting that content into partitions according to the partitioning
> > scheme described in .wks
> >
> > Does it make sense for you?
> >
> > >
> > >  > You can probably do the same by using wic plugins, but I'd not suggest
> > >  > to go this way. Using wic image type is simpler, more consistent,
> > easier to do and provides higher level of automation.
> > >
> > > Is using the wic image type and a plugin mutually exclusive?
> > No, not at all. However, I personally found the way I described above
> > more consistent, flexible and easy to implement and maintain.
> >
> 
> The thing is, it's likely the machine/bsp setting the WKS_FILE, yet in
> OE/yocto we prefer machine/distro/image to be orthogonal. If you're
> injecting machine specific logic into an image, that image isn't going to
> be generally useful for all machines, and so violates our philosophy.

I'm not sure I understand why it has to be machine-dependent setting in
the .wks Can you give some example?

--
Regards,
Ed

Re: Wic and "live" images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4234 bytes --]

On Tue, May 24, 2016 at 1:16 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> On Tue, May 24, 2016 at 12:56:39PM -0700, Christopher Larson wrote:
> > On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh@linux.intel.com
> >
> > wrote:
> >
> > > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
> > > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> > > >  > > Greetings, I am trying to learn "wic" and have been confused as
> how
> > > to create a "live" style image.  I am following "
> > >
> http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images
> "
> > > but am getting confused on the target to use to create the a file
> system
> > > that has a single squashfs file containing my root file system.
> > > >  > >
> > > >  > > My desired partition layout is as follows:
> > > >  > >       40MiB                 40MiB               300MiB
> > > >  > >
> > > +--------------------+-----------------+-----------------------------+
> > > >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)
> > >   |
> > > >  > >
> > > +--------------------+-----------------+-----------------------------+
> > > >  > >
> > > >  > > BOOT - efi boot partition with kernel and initramfs
> > > >  > > DATA - generic fat filesystem to hold configuration files
> > > >  > > ROOT - an ext4 filesystem that contains a single os.img, which
> is a
> > > squashfs file.
> > > >  > >
> > > >  > > I have ROOT and DATA figured out but I am at a loss as how to
> > > generate the os.img file and copy it into ROOT.  If I generate the
> os.img
> > > file with bitbake and then use the "-r" option to manually supply a
> > > directory structure it works, but I would rather have it done from a
> wks
> > > file for automation reasons.
> > > >  > >
> > > >  > > Any hints?
> > > >  > I'd suggest to use wic image type and generate your image by
> bitbake.
> > > >  > You can find example wic-image-minimal.bb and
> wic-image-minimal.wks
> > > in ../meta-selftest/recipes-test/images/
> > > >  >
> > > > This is where I started.  I was able to make it work but not with my
> > > configuration above.  It looks like I can use a type of "fsimage" for
> my
> > > "ROOT" partition, but I have not been able to figure out the syntax
> there
> > > yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but
> > > there doesn't seem to be a way to provide an initramfs.
> > >
> > > How about creating recipe to prepare content or your boot partition and
> > > then using --source rootfs --rootfs-dir=<your recipe> ?
> > > This is much more generic way of creating partitioned images from my
> > > point of view. Image recipes should take care of content and wic takes
> > > care of
> > > putting that content into partitions according to the partitioning
> > > scheme described in .wks
> > >
> > > Does it make sense for you?
> > >
> > > >
> > > >  > You can probably do the same by using wic plugins, but I'd not
> suggest
> > > >  > to go this way. Using wic image type is simpler, more consistent,
> > > easier to do and provides higher level of automation.
> > > >
> > > > Is using the wic image type and a plugin mutually exclusive?
> > > No, not at all. However, I personally found the way I described above
> > > more consistent, flexible and easy to implement and maintain.
> > >
> >
> > The thing is, it's likely the machine/bsp setting the WKS_FILE, yet in
> > OE/yocto we prefer machine/distro/image to be orthogonal. If you're
> > injecting machine specific logic into an image, that image isn't going to
> > be generally useful for all machines, and so violates our philosophy.
>
> I'm not sure I understand why it has to be machine-dependent setting in
> the .wks Can you give some example?


I don't understand the question. The .wks is machine specific by
definition. It includes knowledge of the expectations of the boot process
on the hardware the image is used on, what the boot device name is, etc.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5726 bytes --]

Re: Wic and "live" images

[Ed Bartosh]

On Tue, May 24, 2016 at 01:30:19PM -0700, Christopher Larson wrote:
> On Tue, May 24, 2016 at 1:16 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
> 
> > On Tue, May 24, 2016 at 12:56:39PM -0700, Christopher Larson wrote:
> > > On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh@linux.intel.com
> > >
> > > wrote:
> > >
> > > > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
> > > > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> > > > >  > > Greetings, I am trying to learn "wic" and have been confused as
> > how
> > > > to create a "live" style image.  I am following "
> > > >
> > http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images
> > "
> > > > but am getting confused on the target to use to create the a file
> > system
> > > > that has a single squashfs file containing my root file system.
> > > > >  > >
> > > > >  > > My desired partition layout is as follows:
> > > > >  > >       40MiB                 40MiB               300MiB
> > > > >  > >
> > > > +--------------------+-----------------+-----------------------------+
> > > > >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT (live)
> > > >   |
> > > > >  > >
> > > > +--------------------+-----------------+-----------------------------+
> > > > >  > >
> > > > >  > > BOOT - efi boot partition with kernel and initramfs
> > > > >  > > DATA - generic fat filesystem to hold configuration files
> > > > >  > > ROOT - an ext4 filesystem that contains a single os.img, which
> > is a
> > > > squashfs file.
> > > > >  > >
> > > > >  > > I have ROOT and DATA figured out but I am at a loss as how to
> > > > generate the os.img file and copy it into ROOT.  If I generate the
> > os.img
> > > > file with bitbake and then use the "-r" option to manually supply a
> > > > directory structure it works, but I would rather have it done from a
> > wks
> > > > file for automation reasons.
> > > > >  > >
> > > > >  > > Any hints?
> > > > >  > I'd suggest to use wic image type and generate your image by
> > bitbake.
> > > > >  > You can find example wic-image-minimal.bb and
> > wic-image-minimal.wks
> > > > in ../meta-selftest/recipes-test/images/
> > > > >  >
> > > > > This is where I started.  I was able to make it work but not with my
> > > > configuration above.  It looks like I can use a type of "fsimage" for
> > my
> > > > "ROOT" partition, but I have not been able to figure out the syntax
> > there
> > > > yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi" but
> > > > there doesn't seem to be a way to provide an initramfs.
> > > >
> > > > How about creating recipe to prepare content or your boot partition and
> > > > then using --source rootfs --rootfs-dir=<your recipe> ?
> > > > This is much more generic way of creating partitioned images from my
> > > > point of view. Image recipes should take care of content and wic takes
> > > > care of
> > > > putting that content into partitions according to the partitioning
> > > > scheme described in .wks
> > > >
> > > > Does it make sense for you?
> > > >
> > > > >
> > > > >  > You can probably do the same by using wic plugins, but I'd not
> > suggest
> > > > >  > to go this way. Using wic image type is simpler, more consistent,
> > > > easier to do and provides higher level of automation.
> > > > >
> > > > > Is using the wic image type and a plugin mutually exclusive?
> > > > No, not at all. However, I personally found the way I described above
> > > > more consistent, flexible and easy to implement and maintain.
> > > >
> > >
> > > The thing is, it's likely the machine/bsp setting the WKS_FILE, yet in
> > > OE/yocto we prefer machine/distro/image to be orthogonal. If you're
> > > injecting machine specific logic into an image, that image isn't going to
> > > be generally useful for all machines, and so violates our philosophy.
> >
> > I'm not sure I understand why it has to be machine-dependent setting in
> > the .wks Can you give some example?
> 
> 
> I don't understand the question. The .wks is machine specific by
> definition. It includes knowledge of the expectations of the boot process
> on the hardware the image is used on, what the boot device name is, etc.

Sorry for not being clear. Now I understand what you mean. I didn't know
about this philosopy when I implemented handling wic image types in
image core classes. Are you suggesting to remove it and keep using wic
as only as a separate tool?

--
Regards,
Ed

Re: Wic and "live" images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 5978 bytes --]

On Wed, May 25, 2016 at 4:35 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> On Tue, May 24, 2016 at 01:30:19PM -0700, Christopher Larson wrote:
> > On Tue, May 24, 2016 at 1:16 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> > wrote:
> >
> > > On Tue, May 24, 2016 at 12:56:39PM -0700, Christopher Larson wrote:
> > > > On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh
> @linux.intel.com
> > > >
> > > > wrote:
> > > >
> > > > > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote:
> > > > > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:
> > > > > >  > > Greetings, I am trying to learn "wic" and have been
> confused as
> > > how
> > > > > to create a "live" style image.  I am following "
> > > > >
> > >
> http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images
> > > "
> > > > > but am getting confused on the target to use to create the a file
> > > system
> > > > > that has a single squashfs file containing my root file system.
> > > > > >  > >
> > > > > >  > > My desired partition layout is as follows:
> > > > > >  > >       40MiB                 40MiB               300MiB
> > > > > >  > >
> > > > >
> +--------------------+-----------------+-----------------------------+
> > > > > >  > > |      BOOT (esp)    |    DATA (fat)   |          ROOT
> (live)
> > > > >   |
> > > > > >  > >
> > > > >
> +--------------------+-----------------+-----------------------------+
> > > > > >  > >
> > > > > >  > > BOOT - efi boot partition with kernel and initramfs
> > > > > >  > > DATA - generic fat filesystem to hold configuration files
> > > > > >  > > ROOT - an ext4 filesystem that contains a single os.img,
> which
> > > is a
> > > > > squashfs file.
> > > > > >  > >
> > > > > >  > > I have ROOT and DATA figured out but I am at a loss as how
> to
> > > > > generate the os.img file and copy it into ROOT.  If I generate the
> > > os.img
> > > > > file with bitbake and then use the "-r" option to manually supply a
> > > > > directory structure it works, but I would rather have it done from
> a
> > > wks
> > > > > file for automation reasons.
> > > > > >  > >
> > > > > >  > > Any hints?
> > > > > >  > I'd suggest to use wic image type and generate your image by
> > > bitbake.
> > > > > >  > You can find example wic-image-minimal.bb and
> > > wic-image-minimal.wks
> > > > > in ../meta-selftest/recipes-test/images/
> > > > > >  >
> > > > > > This is where I started.  I was able to make it work but not
> with my
> > > > > configuration above.  It looks like I can use a type of "fsimage"
> for
> > > my
> > > > > "ROOT" partition, but I have not been able to figure out the syntax
> > > there
> > > > > yet.  For "BOOT" I am at a complete loss.  In theory "bootimg-efi"
> but
> > > > > there doesn't seem to be a way to provide an initramfs.
> > > > >
> > > > > How about creating recipe to prepare content or your boot
> partition and
> > > > > then using --source rootfs --rootfs-dir=<your recipe> ?
> > > > > This is much more generic way of creating partitioned images from
> my
> > > > > point of view. Image recipes should take care of content and wic
> takes
> > > > > care of
> > > > > putting that content into partitions according to the partitioning
> > > > > scheme described in .wks
> > > > >
> > > > > Does it make sense for you?
> > > > >
> > > > > >
> > > > > >  > You can probably do the same by using wic plugins, but I'd not
> > > suggest
> > > > > >  > to go this way. Using wic image type is simpler, more
> consistent,
> > > > > easier to do and provides higher level of automation.
> > > > > >
> > > > > > Is using the wic image type and a plugin mutually exclusive?
> > > > > No, not at all. However, I personally found the way I described
> above
> > > > > more consistent, flexible and easy to implement and maintain.
> > > > >
> > > >
> > > > The thing is, it's likely the machine/bsp setting the WKS_FILE, yet
> in
> > > > OE/yocto we prefer machine/distro/image to be orthogonal. If you're
> > > > injecting machine specific logic into an image, that image isn't
> going to
> > > > be generally useful for all machines, and so violates our philosophy.
> > >
> > > I'm not sure I understand why it has to be machine-dependent setting in
> > > the .wks Can you give some example?
> >
> >
> > I don't understand the question. The .wks is machine specific by
> > definition. It includes knowledge of the expectations of the boot process
> > on the hardware the image is used on, what the boot device name is, etc.
>
> Sorry for not being clear. Now I understand what you mean. I didn't know
> about this philosopy when I implemented handling wic image types in
> image core classes. Are you suggesting to remove it and keep using wic
> as only as a separate tool?


It's debatable. As long as we keep the logic separated, such that anything
bsp specific is in the machine or bsp layer, so the images are independent
of any bsp, then we're fine. If we need to keep certain aspects of rootfs /
image preparation outside of wic, then we'd need the machines which need
such setup to use a hook to ensure it's applied to all images, i.e. an
IMAGE_POSTPROCESS_COMMAND, and we'd need to document that.

That might be doable, but we definitely need to give careful thought to
what pieces of information about the image creation process come from
where. Certain aspects are clearly distro, i.e. extra image space, and
possibly other partitioning like splitting up the rootfs into multiple
partitions, but others are clearly bsp, i.e. setup of a boot partition if
the bootloader expects it, and image recipes need to work regardless of
what distro or machine are selected.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 8258 bytes --]

Re: Wic and "live" images

[Ian Geiser]

 ---- On Tue, 24 May 2016 15:51:45 -0400 Ed Bartosh <ed.bartosh@linux.intel.com> wrote ---- 
 > On Mon, May 23, 2016 at 08:13:28AM -0400, Ian Geiser wrote: 
 > >  > On Thu, May 19, 2016 at 05:52:45AM -0400, Ian Geiser wrote:  

[...snip...]

 > How about creating recipe to prepare content or your boot partition and then using --source rootfs --rootfs-dir=<your recipe> ? 
 > This is much more generic way of creating partitioned images from my 
 > point of view. Image recipes should take care of content and wic takes care of 
 > putting that content into partitions according to the partitioning 
 > scheme described in .wks 
 >  
 > Does it make sense for you? 

I am at a loss how to do this because it is a efi system partition.  Really all it needs is the kernel, initramfs and bootx64.  I see how to do the kernel and the bootx64 in the existing plugin.  What I get turned around with is the initramfs.  It seems like there is no way to put that in, or use a kernel with the initramfs appended.  Is this a limitation that is fixed by "send a patch" or am I missing something there.

 >  
 > >  
 > >  > You can probably do the same by using wic plugins, but I'd not suggest  
 > >  > to go this way. Using wic image type is simpler, more consistent, easier to do and provides higher level of automation.  
 > >  
 > > Is using the wic image type and a plugin mutually exclusive? 
 > No, not at all. However, I personally found the way I described above 
 > more consistent, flexible and easy to implement and maintain. 
 
I am not groking this concept because it seems without a robust library of plugins the wks files become very hard to implement.  Is this true, or am I again missing something obvious.

Thanks for your patience. 

Re: Wic and "live" images

[Ian Geiser]

 ---- On Tue, 24 May 2016 15:56:39 -0400 Christopher Larson <clarson@kergoth.com> wrote ---- 
 > 
 > On Tue, May 24, 2016 at 12:51 PM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
 > 
 > The thing is, it's likely the machine/bsp setting the WKS_FILE, yet in OE/yocto we prefer machine/distro/image to be orthogonal. If you're injecting machine specific logic into an image, that image isn't going to be generally useful for all machines, and so violates our philosophy.

Isn't the physical disk image the place that machine/distro/image all intersect though?  Maybe this is some philosophy I have always not gotten because it seems every time I make an image for a machine I am fighting the current.  Really I think at the core what I want is a partition layout that I can put a payload in.  wic seems exactly that, but I am not sure how to get from my rootfs, initramfs and kernel to there.  Thanks!

Re: Wic and "live" images

[Ian Geiser]

 ---- On Wed, 25 May 2016 16:04:50 -0400 Christopher Larson <clarson@kergoth.com> wrote ---- 
 > 
 > On Wed, May 25, 2016 at 4:35 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
 > 
 > It's debatable. As long as we keep the logic separated, such that anything bsp specific is in the machine or bsp layer, so the images are independent of any bsp, then we're fine. If we need to keep certain aspects of rootfs / image preparation outside of wic, then we'd need the machines which need such setup to use a hook to ensure it's applied to all images, i.e. an IMAGE_POSTPROCESS_COMMAND, and we'd need to document that.

If I follow in wic the logic is in the plugins.  So its up to the bsp to implement those.  Currently though we have tight coupling with efi and mbr in the oe-core libraries.  I guess this leads me into a false sense that wic is machine/bsp specific.

 > That might be doable, but we definitely need to give careful thought to what pieces of information about the image creation process come from where. Certain aspects are clearly distro, i.e. extra image space, and possibly other partitioning like splitting up the rootfs into multiple partitions, but others are clearly bsp, i.e. setup of a boot partition if the bootloader expects it, and image recipes need to work regardless of what distro or machine are selected.
 
This is what I get at by saying needing a "robust" library of common plugins that can be reused by the bsp authors.  I think this would allow the wks files to be more consistent and more reusable.   

Lastly, in the current vernacular am I confusing the terms "image" and "rootfs"?  In my mind "image" is the physical bits that will get burned into ROM/SSD/etc.   "rootfs" is the filesystem component that is injected somehow into the image.  Is this correct?

Re: Wic and "live" images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4064 bytes --]

On Tue, May 31, 2016 at 8:31 AM, Ian Geiser <geiseri@geekcentral.pub> wrote:

>  ---- On Wed, 25 May 2016 16:04:50 -0400 Christopher Larson <
> clarson@kergoth.com> wrote ----
>  >
>  > On Wed, May 25, 2016 at 4:35 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
>  >
>  > It's debatable. As long as we keep the logic separated, such that
> anything bsp specific is in the machine or bsp layer, so the images are
> independent of any bsp, then we're fine. If we need to keep certain aspects
> of rootfs / image preparation outside of wic, then we'd need the machines
> which need such setup to use a hook to ensure it's applied to all images,
> i.e. an IMAGE_POSTPROCESS_COMMAND, and we'd need to document that.
>
> If I follow in wic the logic is in the plugins.  So its up to the bsp to
> implement those.  Currently though we have tight coupling with efi and mbr
> in the oe-core libraries.  I guess this leads me into a false sense that
> wic is machine/bsp specific.
>

I'd agree with that, yes, it's the specific plugins that are bound to bsp,
and a number of the defaults are clearly for specific use cases by specific
bsps, which of course is why the wks is machine specific and in the bsp's
realm of responsibility.


>  > That might be doable, but we definitely need to give careful thought to
> what pieces of information about the image creation process come from
> where. Certain aspects are clearly distro, i.e. extra image space, and
> possibly other partitioning like splitting up the rootfs into multiple
> partitions, but others are clearly bsp, i.e. setup of a boot partition if
> the bootloader expects it, and image recipes need to work regardless of
> what distro or machine are selected.
>
> This is what I get at by saying needing a "robust" library of common
> plugins that can be reused by the bsp authors.  I think this would allow
> the wks files to be more consistent and more reusable.
>

I'd agree with that to a certain extent. Many of the current plugins encode
a lot of hardcoded logic and configuration and aren't very flexible. More
small plugins that we can mix and match with more options to configure them
would be nicer, but I'm assuming we just haven't had a lot of people
contributing to wic in that way yet. There aren't currently many ways for
the distro or image to affect what wic does, right now.

Ed wants the image recipe to be responsible for breaking up the rootfs or
otherwise prepping the partition input for use by wic, but if we do that,
we're injecting machine-specific logic into the image recipe. I think it
should either be done by a wic plugin or plugins (either in the wic core,
or as you suggest, new plugins in the bsp), not the image recipe, or the
image/rootfs classes and python package need more hooks for the
machine/distro configuration to opt-in to that sort of preparation without
hardcoding it into the image recipe itself. I don't feel too strongly
either way, I just want to make sure we follow the project philosophy and
don't tightly intertwine the distro, machine, and image recipe.

Lastly, in the current vernacular am I confusing the terms "image" and
> "rootfs"?  In my mind "image" is the physical bits that will get burned
> into ROM/SSD/etc.   "rootfs" is the filesystem component that is injected
> somehow into the image.  Is this correct?
>

It depends on context, actually, which gives the term some ambiguity. The
image *recipe* is of course a yocto construct, and its output might be just
hte rootfs or might be a disk image, depending on the configuration of
IMAGE_FSTYPES. It's this recipe notion of an image that needs the
orthogonality -- no machine or distro specifics in the image recipe.

On the other hand, wic's output is an image, that being a disk image which
gets flashed to media using the rootfs and other files as input.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5234 bytes --]