[PATCH 0/4] add wic based image installer

[PATCH 0/4] add wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.

2 partition needed for installable image
 - first partition build using source plugin bootimg-efi and configure to run install
 - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)

These patches:
 - add new source plugin for second partition
 - add new .wks
 - add new wic dependency and set default value for required variable
 - allow source plugin bootimg-efi to configure to install


To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.


Chee Yang Lee (4):
  wic: add new installer-partition plugin
  wic: bootimg-efi: add label source parameter
  wic: add new wic-installer.wks for wic based image installer
  image_types_wic: add dependency for wic based image installer

 meta/classes/image_types_wic.bbclass               |   6 +
 scripts/lib/wic/canned-wks/wic-installer.wks.in    |   7 +
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  20 ++-
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 4 files changed, 220 insertions(+), 5 deletions(-)
 create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

-- 
2.7.4

[PATCH 1/4] wic: add new installer-partition plugin

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

This patch implement 'installer-partition.py' source plugin for wic.
The plugin create an image with systemd-boot and rootfs.
The generated image can be use to create wic based image installer.

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 1 file changed, 192 insertions(+)
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

diff --git a/scripts/lib/wic/plugins/source/installer-partition.py b/scripts/lib/wic/plugins/source/installer-partition.py
new file mode 100644
index 0000000..4d8343f
--- /dev/null
+++ b/scripts/lib/wic/plugins/source/installer-partition.py
@@ -0,0 +1,192 @@
+# Copyright (c) 2019, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'installer-partition' source plugin class for 'wic'
+# This plugin prepare partition with content to be install on target machine.
+#
+# AUTHORS
+# Lee Chee Yang <Chee.Yang.Lee (at] intel.com>
+#
+
+import logging
+import os
+
+from wic import WicError
+from wic.engine import get_custom_config
+from wic.pluginbase import SourcePlugin
+from wic.misc import (exec_cmd, exec_native_cmd, get_bitbake_var)
+
+logger = logging.getLogger('wic')
+
+class InstallerImagePlugin(SourcePlugin):
+    """
+    Populate content for wic image based installer
+    """
+
+    name = 'installer-partition'
+
+    @classmethod
+    def do_configure_partition(cls, part, source_params, creator, cr_workdir,
+                               oe_builddir, bootimg_dir, kernel_dir,
+                               native_sysroot):
+        """
+        Called before do_prepare_partition(), creates loader-specific config
+        """
+        if not kernel_dir:
+            kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+            if not kernel_dir:
+                raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+        staging_kernel_dir = kernel_dir
+
+        partition_dir = "%s/%s-%s" % (cr_workdir, part.label, part.lineno)
+
+        install_cmd = "install -d %s/EFI/BOOT" % partition_dir
+        exec_cmd(install_cmd)
+
+        install_cmd = "install -d %s/loader/entries" % partition_dir
+        exec_cmd(install_cmd)
+
+        bootloader = creator.ks.bootloader
+
+        loader_conf = ""
+        loader_conf += "default boot\n"
+        loader_conf += "timeout %d\n" % bootloader.timeout
+
+        initrd = source_params.get('initrd')
+
+        if initrd:
+            cp_cmd = "cp %s/%s %s" % (kernel_dir, initrd, partition_dir)
+            exec_cmd(cp_cmd, True)
+        else:
+            logger.debug("Ignoring missing initrd")
+
+        logger.debug("Writing systemd-boot config "
+                     "%s/loader/loader.conf", partition_dir)
+        cfg = open("%s/loader/loader.conf" % partition_dir, "w")
+        cfg.write(loader_conf)
+        cfg.close()
+
+        kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+        if not kernel:
+            kernel = "bzImage"
+
+        install_cmd = "install -m 0644 %s/%s %s/%s" % \
+            (staging_kernel_dir, kernel, partition_dir, kernel)
+        exec_cmd(install_cmd)
+
+        configfile = creator.ks.bootloader.configfile
+        custom_cfg = None
+        if configfile:
+            custom_cfg = get_custom_config(configfile)
+            if custom_cfg:
+                # Use a custom configuration for systemd-boot
+                boot_conf = custom_cfg
+                logger.debug("Using custom configuration file "
+                             "%s for systemd-boots's boot.conf", configfile)
+            else:
+                raise WicError("configfile is specified but failed to "
+                               "get it from %s.", configfile)
+
+        if not custom_cfg:
+            # Create systemd-boot configuration using parameters from wks file
+            title = source_params.get('title')
+
+            boot_conf = ""
+            boot_conf += "title %s\n" % (title if title else "boot")
+            boot_conf += "linux /%s\n" % kernel
+            boot_conf += "options LABEL=Boot %s\n" % (bootloader.append)
+
+            if initrd:
+                boot_conf += "initrd /%s\n" % initrd
+
+        for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
+            cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, partition_dir, mod[8:])
+            exec_cmd(cp_cmd, True)
+
+        logger.debug("Writing systemd-boot config "
+                     "%s/loader/entries/boot.conf", partition_dir)
+        cfg = open("%s//loader/entries/boot.conf" % partition_dir, "w")
+        cfg.write(boot_conf)
+        cfg.close()
+
+
+    @classmethod
+    def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
+                             oe_builddir, bootimg_dir, kernel_dir,
+                             rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition,
+        prepare systemd bootloader and rootfs image
+        """
+        partition_dir = "%s/%s-%s" % (cr_workdir, part.label, part.lineno)
+
+        rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, part.label, part.lineno, part.fstype)
+
+        image_deploy_dir = get_bitbake_var("IMGDEPLOYDIR")
+        if not image_deploy_dir:
+            raise WicError("Couldn't find IMGDEPLOYDIR, exiting")
+
+        rootfs_img = "%s/%s.%s" % (image_deploy_dir,
+            get_bitbake_var("IMAGE_LINK_NAME"), part.fstype )
+        if not os.path.isfile(rootfs_img):
+            raise WicError("Couldn't find %s, exiting" % rootfs_img)
+
+        install_cmd = "install -m 0644 %s %s/rootfs.img" % \
+            (rootfs_img, partition_dir)
+        exec_cmd(install_cmd)
+
+        #look for actual size required for the partition
+        du_cmd = "du -ks %s" % partition_dir
+        out = exec_cmd(du_cmd)
+        part.size = cls.get_partition_size(part, int(out.split()[0]))
+        logger.debug("Set partition size to %d ", part.size)
+
+        with open(rootfs, 'w') as sparse:
+            os.ftruncate(sparse.fileno(), part.size * 1024)
+
+        extraopts = part.mkfs_extraopts or "-i 8192"
+
+        label_str = ""
+        if part.label:
+            label_str = "-L %s" % part.label
+
+        mkfs_cmd = "mkfs.%s -F %s %s -U %s %s -d %s" % \
+            (part.fstype, extraopts, label_str, part.fsuuid, rootfs, partition_dir)
+        exec_native_cmd(mkfs_cmd, native_sysroot)
+
+        part.source_file = rootfs
+
+
+    def get_partition_size(part, actual_partition_size=0):
+        """
+        Calculate the required size of the partition, taking into consideration
+        --size/--fixed-size/--extra-space flags specified in kickstart file.
+        Raises an error if the `actual_partition_size` is larger than fixed-size.
+        """
+        if part.fixed_size:
+            partition_size = part.fixed_size
+            if actual_partition_size > partition_size:
+                raise WicError("Actual partition size (%d kB) is larger than "
+                               "allowed size %d kB" %
+                               (actual_partition_size, partition_size))
+        else:
+            extra_space = 0
+            if extra_space < part.extra_space:
+                extra_space = part.extra_space
+            partition_size = actual_partition_size + extra_space
+
+        return partition_size
-- 
2.7.4

[PATCH 2/4] wic: bootimg-efi: add label source parameter

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Add new source parameter label to allow boot.conf/grub.cfg label option
to set as "install-efi", so far it's hardcoded to "Boot".

The label remain to "Boot" for systemd-boot or blank for grub-efi whenever
label are not "install-efi".

when label set to "install-efi" the image can boot into image installation.

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 83a7e18..0a0c5bd 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -86,9 +86,14 @@ class BootimgEFIPlugin(SourcePlugin):
             grubefi_conf += "menuentry '%s'{\n" % (title if title else "boot")
 
             kernel = "/bzImage"
+            label = source_params.get('label')
 
-            grubefi_conf += "linux %s root=%s rootwait %s\n" \
-                % (kernel, creator.rootdev, bootloader.append)
+            if label == "install-efi":
+                grubefi_conf += "linux %s LABEL=%s rootwait %s\n" \
+                    % (kernel, label, bootloader.append)
+            else:
+                grubefi_conf += "linux %s root=%s rootwait %s\n" \
+                    % (kernel, creator.rootdev, bootloader.append)
 
             if initrd:
                grubefi_conf += "initrd /%s\n" % initrd
@@ -154,12 +159,17 @@ class BootimgEFIPlugin(SourcePlugin):
             # Create systemd-boot configuration using parameters from wks file
             kernel = "/bzImage"
             title = source_params.get('title')
-
+            label = source_params.get('label')
             boot_conf = ""
             boot_conf += "title %s\n" % (title if title else "boot")
             boot_conf += "linux %s\n" % kernel
-            boot_conf += "options LABEL=Boot root=%s %s\n" % \
-                             (creator.rootdev, bootloader.append)
+
+            if label == "install-efi":
+                boot_conf += "options LABEL=%s %s\n" % \
+                                (label, bootloader.append)
+            else:
+                boot_conf += "options LABEL=Boot root=%s %s\n" % \
+                                 (creator.rootdev, bootloader.append)
 
             if initrd:
                 boot_conf += "initrd /%s\n" % initrd
-- 
2.7.4

[PATCH 3/4] wic: add new wic-installer.wks for wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 scripts/lib/wic/canned-wks/wic-installer.wks.in | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in

diff --git a/scripts/lib/wic/canned-wks/wic-installer.wks.in b/scripts/lib/wic/canned-wks/wic-installer.wks.in
new file mode 100644
index 0000000..a7b018d
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/wic-installer.wks.in
@@ -0,0 +1,7 @@
+# description: Create wic installable image
+
+part /boot --source bootimg-efi --sourceparams="loader=systemd-boot,title=install,label=install-efi,initrd=${INITRD_IMAGE_LIVE}-${MACHINE}.${INITRAMFS_FSTYPES}" --ondisk sda --label install --active --align 1024 --use-uuid
+
+part / --source installer-partition --ondisk sda --fstype=ext4 --label image --use-uuid
+
+bootloader --ptable gpt --timeout=5 --append="rootwait console=ttyS0,115200 console=tty0"
-- 
2.7.4

[PATCH 4/4] image_types_wic: add dependency for wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

initramfs and rootfs image are required to build wic based
image installer, add both as dependency for do_image_wic.

initiate default value for LIVE_ROOTFS_TYPE, INITRD_IMAGE_LIVE
and INITRD_LIVE

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 meta/classes/image_types_wic.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 5b40a9e..e4f2b8f 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -102,9 +102,15 @@ python () {
                 # a variable and let the metadata deal with the deps.
                 d.setVar('_WKS_TEMPLATE', body)
                 bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
+        d.appendVarFlag('do_image_wic', 'depends', ' %s:do_image_complete' % d.getVar('INITRD_IMAGE_LIVE'))
         bb.build.addtask('do_image_wic', 'do_image_complete', None, d)
 }
 
+LIVE_ROOTFS_TYPE ?= "ext4"
+INITRD_IMAGE_LIVE ?= "${MLPREFIX}core-image-minimal-initramfs"
+INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.${INITRAMFS_FSTYPES}"
+IMAGE_TYPEDEP_wic = "${LIVE_ROOTFS_TYPE}"
+
 #
 # Write environment variables used by wic
 # to tmp/sysroots/<machine>/imgdata/<image>.env
-- 
2.7.4

Re: [PATCH 0/4] add wic based image installer

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]

On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com wrote:

> From: Chee Yang Lee <chee.yang.lee@intel.com>
> 
> Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.
> 
> 2 partition needed for installable image
>  - first partition build using source plugin bootimg-efi and configure to run install
>  - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)
> 
> These patches:
>  - add new source plugin for second partition
>  - add new .wks
>  - add new wic dependency and set default value for required variable
>  - allow source plugin bootimg-efi to configure to install
> 
> 
> To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.

I'm confused as to why we need most of these changes to support the end
goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
see why we need a new plugin to do the rest, however.  What's being done
here that we don't already do today?  I say this having made installer
images for EFI machines using wic (in sumo, even).  Thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 0/4] add wic based image installer

[Lee, Chee Yang]

Any comments on these patches?

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org <openembedded-core-bounces@lists.openembedded.org> On Behalf Of chee.yang.lee@intel.com
Sent: Wednesday, May 8, 2019 2:41 PM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] [PATCH 0/4] add wic based image installer

From: Chee Yang Lee <chee.yang.lee@intel.com>

Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.

2 partition needed for installable image
 - first partition build using source plugin bootimg-efi and configure to run install
 - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)

These patches:
 - add new source plugin for second partition
 - add new .wks
 - add new wic dependency and set default value for required variable
 - allow source plugin bootimg-efi to configure to install


To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.


Chee Yang Lee (4):
  wic: add new installer-partition plugin
  wic: bootimg-efi: add label source parameter
  wic: add new wic-installer.wks for wic based image installer
  image_types_wic: add dependency for wic based image installer

 meta/classes/image_types_wic.bbclass               |   6 +
 scripts/lib/wic/canned-wks/wic-installer.wks.in    |   7 +
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  20 ++-
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 4 files changed, 220 insertions(+), 5 deletions(-)  create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 0/4] add wic based image installer

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 6583 bytes --]

Thanks!  FWIW, I really think an installer improvement would be go to
from "copy image contents to install media, update slightly" to
"write compressed image to disk (use bmaptool to handle decomp), update
a few things like UUIDs".  This would buy us a lot of space as a .xz
compressed image is going to tend to be a lot smaller than 4GB, even
with a ton of things in it.

On Tue, May 21, 2019 at 07:46:52AM +0000, Lee, Chee Yang wrote:
> Tom, Thanks for the sharing
> I try out this method and it works 
> however I notice that sdimage-bootpart.wks format bootimg-partition with vfat still limit the rootfs.img size to 4GB.
> Anyway, with the similar trick and using a new .wks with combination of 2 partition below could build the installer image for image greater than 4GB.
> - bootimg-efi with "wic: bootimg-efi: add label source parameter" patch 
> - bootimg-partition ( format in ext4 ) 
> so I think it would be great to keep the enhancement for bootimg-efi while ignore the other 3 patches. 
> 
> 
> On 5/16/19, 8:20 PM, "Tom Rini" <trini@konsulko.com> wrote:
> 
>     On Thu, May 16, 2019 at 11:48:02AM +0000, Lee, Chee Yang wrote:
>     
>     > I couldn’t figure out how to make it with sdimage-bootpart.wks, can
>     > you tell me what did you set for IMAGE_BOOT_FILES ?
>     > 
>     > To further explain the idea,  I'm trying to use the new source plugin
>     > and .wks.in to build a wic image that "mimic" a hddimg, where I can dd
>     > the wic image to an USB drive and use that usb drive to install image
>     > to my target machine (like hddimg did ). 
>     > first partition allow to boot into initramfs and it will execute the
>     > install script. The script will then install system-boot
>     > configuration, efi, kernel, rootfs.img and initrd from second
>     > partition to target machine. 
>     > 
>     > My sample image here: 
>     > https://drive.google.com/file/d/1c1FCJ2ghWNzsXK1zbOAVz1bz6G3H5zx5/view?usp=sharing
>     
>     OK, so some cleaned up snippets from the installer image recipe:
>     # List of external things we must have had complete.
>     do_rootfs[depends] += "core-image-minimal-initramfs:do_image_complete"
>     do_rootfs[depends] += "full-image-to-install:do_image_complete"
>     do_rootfs[depends] += "grub-efi:do_deploy"
>     do_rootfs[depends] += "virtual/kernel:do_deploy"
>     
>     WKS_FILE = "sdimage-bootpart.wks"
>     
>     IMAGE_BOOT_FILES_append = "\
>         ${KERNEL_IMAGETYPE};EFI/BOOT/${KERNEL_IMAGETYPE} \
>         core-image-minimal-initramfs-${MACHINE}.cpio.gz;EFI/BOOT/initrd \
>         full-image-to-install-${MACHINE}.wic.xz;rootfs.img \
>         grub-efi-bootx64.efi;EFI/BOOT/bootx64.efi \
>         grub-installer.cfg;EFI/BOOT/grub.cfg \
>         "
>     
>     And note that I reworked the installer for our case to write then update
>     the real image's wic image rather than copy contents and then modify
>     files due to other constraints (this project is also why I was pushing
>     various changes to meta-secure-core).
>     
>     > 
>     > 
>     > On 5/16/19, 10:38 AM, "Tom Rini" <trini@konsulko.com> wrote:
>     > 
>     >     On Thu, May 16, 2019 at 02:23:05AM +0000, Lee, Chee Yang wrote:
>     >     > Hi Tom, 
>     >     > 
>     >     > Would you mind to share with me how you make the installer image using wic? 
>     >     
>     >     I ended up using sdimage-bootpart.wks and IMAGE_BOOT_FILES to populate
>     >     the partition as needed.  I can't paste the exact image as it was done
>     >     for a customer.
>     >     
>     >     > 
>     >     >     -------- Forwarded Message --------
>     >     >     From: Tom Rini <trini@konsulko.com>
>     >     >     To: chee.yang.lee@intel.com
>     >     >     Cc: openembedded-core@lists.openembedded.org
>     >     >     Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
>     >     >     Date: Wed, 15 May 2019 11:33:43 -0400
>     >     >     
>     >     >     On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
>     >     >     wrote:
>     >     >     
>     >     >     > From: Chee Yang Lee <chee.yang.lee@intel.com>
>     >     >     > 
>     >     >     > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
>     >     >     > build larger installable image, we can build it in wic based image.
>     >     >     > 
>     >     >     > 2 partition needed for installable image
>     >     >     >  - first partition build using source plugin bootimg-efi and
>     >     >     > configure to run install
>     >     >     >  - second partition build with root.img, systemd-boot and kernel to
>     >     >     > be install on target machine (this partition build using new source
>     >     >     > plugin installer-partition)
>     >     >     > 
>     >     >     > These patches:
>     >     >     >  - add new source plugin for second partition
>     >     >     >  - add new .wks
>     >     >     >  - add new wic dependency and set default value for required variable
>     >     >     >  - allow source plugin bootimg-efi to configure to install
>     >     >     > 
>     >     >     > 
>     >     >     > To build the image, set WKS_FILE="wic-installer.wks.in" in
>     >     >     > local.conf.
>     >     >     
>     >     >     I'm confused as to why we need most of these changes to support the end
>     >     >     goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
>     >     >     see why we need a new plugin to do the rest, however.  What's being
>     >     >     done
>     >     >     here that we don't already do today?  I say this having made installer
>     >     >     images for EFI machines using wic (in sumo, even).  Thanks!
>     >     >     
>     >     >     -- 
>     >     >     Tom
>     >     >     -- 
>     >     >     _______________________________________________
>     >     >     Openembedded-core mailing list
>     >     >     Openembedded-core@lists.openembedded.org
>     >     >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>     >     >     
>     >     >     
>     >     > 
>     >     
>     >     -- 
>     >     Tom
>     >     
>     > 
>     
>     -- 
>     Tom
>     
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 0/4] add wic based image installer

[Lee, Chee Yang]

Tom, Thanks for the sharing
I try out this method and it works 
however I notice that sdimage-bootpart.wks format bootimg-partition with vfat still limit the rootfs.img size to 4GB.
Anyway, with the similar trick and using a new .wks with combination of 2 partition below could build the installer image for image greater than 4GB.
- bootimg-efi with "wic: bootimg-efi: add label source parameter" patch 
- bootimg-partition ( format in ext4 ) 
so I think it would be great to keep the enhancement for bootimg-efi while ignore the other 3 patches. 


On 5/16/19, 8:20 PM, "Tom Rini" <trini@konsulko.com> wrote:

    On Thu, May 16, 2019 at 11:48:02AM +0000, Lee, Chee Yang wrote:
    
    > I couldn’t figure out how to make it with sdimage-bootpart.wks, can
    > you tell me what did you set for IMAGE_BOOT_FILES ?
    > 
    > To further explain the idea,  I'm trying to use the new source plugin
    > and .wks.in to build a wic image that "mimic" a hddimg, where I can dd
    > the wic image to an USB drive and use that usb drive to install image
    > to my target machine (like hddimg did ). 
    > first partition allow to boot into initramfs and it will execute the
    > install script. The script will then install system-boot
    > configuration, efi, kernel, rootfs.img and initrd from second
    > partition to target machine. 
    > 
    > My sample image here: 
    > https://drive.google.com/file/d/1c1FCJ2ghWNzsXK1zbOAVz1bz6G3H5zx5/view?usp=sharing
    
    OK, so some cleaned up snippets from the installer image recipe:
    # List of external things we must have had complete.
    do_rootfs[depends] += "core-image-minimal-initramfs:do_image_complete"
    do_rootfs[depends] += "full-image-to-install:do_image_complete"
    do_rootfs[depends] += "grub-efi:do_deploy"
    do_rootfs[depends] += "virtual/kernel:do_deploy"
    
    WKS_FILE = "sdimage-bootpart.wks"
    
    IMAGE_BOOT_FILES_append = "\
        ${KERNEL_IMAGETYPE};EFI/BOOT/${KERNEL_IMAGETYPE} \
        core-image-minimal-initramfs-${MACHINE}.cpio.gz;EFI/BOOT/initrd \
        full-image-to-install-${MACHINE}.wic.xz;rootfs.img \
        grub-efi-bootx64.efi;EFI/BOOT/bootx64.efi \
        grub-installer.cfg;EFI/BOOT/grub.cfg \
        "
    
    And note that I reworked the installer for our case to write then update
    the real image's wic image rather than copy contents and then modify
    files due to other constraints (this project is also why I was pushing
    various changes to meta-secure-core).
    
    > 
    > 
    > On 5/16/19, 10:38 AM, "Tom Rini" <trini@konsulko.com> wrote:
    > 
    >     On Thu, May 16, 2019 at 02:23:05AM +0000, Lee, Chee Yang wrote:
    >     > Hi Tom, 
    >     > 
    >     > Would you mind to share with me how you make the installer image using wic? 
    >     
    >     I ended up using sdimage-bootpart.wks and IMAGE_BOOT_FILES to populate
    >     the partition as needed.  I can't paste the exact image as it was done
    >     for a customer.
    >     
    >     > 
    >     >     -------- Forwarded Message --------
    >     >     From: Tom Rini <trini@konsulko.com>
    >     >     To: chee.yang.lee@intel.com
    >     >     Cc: openembedded-core@lists.openembedded.org
    >     >     Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
    >     >     Date: Wed, 15 May 2019 11:33:43 -0400
    >     >     
    >     >     On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
    >     >     wrote:
    >     >     
    >     >     > From: Chee Yang Lee <chee.yang.lee@intel.com>
    >     >     > 
    >     >     > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
    >     >     > build larger installable image, we can build it in wic based image.
    >     >     > 
    >     >     > 2 partition needed for installable image
    >     >     >  - first partition build using source plugin bootimg-efi and
    >     >     > configure to run install
    >     >     >  - second partition build with root.img, systemd-boot and kernel to
    >     >     > be install on target machine (this partition build using new source
    >     >     > plugin installer-partition)
    >     >     > 
    >     >     > These patches:
    >     >     >  - add new source plugin for second partition
    >     >     >  - add new .wks
    >     >     >  - add new wic dependency and set default value for required variable
    >     >     >  - allow source plugin bootimg-efi to configure to install
    >     >     > 
    >     >     > 
    >     >     > To build the image, set WKS_FILE="wic-installer.wks.in" in
    >     >     > local.conf.
    >     >     
    >     >     I'm confused as to why we need most of these changes to support the end
    >     >     goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
    >     >     see why we need a new plugin to do the rest, however.  What's being
    >     >     done
    >     >     here that we don't already do today?  I say this having made installer
    >     >     images for EFI machines using wic (in sumo, even).  Thanks!
    >     >     
    >     >     -- 
    >     >     Tom
    >     >     -- 
    >     >     _______________________________________________
    >     >     Openembedded-core mailing list
    >     >     Openembedded-core@lists.openembedded.org
    >     >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
    >     >     
    >     >     
    >     > 
    >     
    >     -- 
    >     Tom
    >     
    > 
    
    -- 
    Tom
    

Re: [PATCH 0/4] add wic based image installer

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4602 bytes --]

On Thu, May 16, 2019 at 11:48:02AM +0000, Lee, Chee Yang wrote:

> I couldn’t figure out how to make it with sdimage-bootpart.wks, can
> you tell me what did you set for IMAGE_BOOT_FILES ?
> 
> To further explain the idea,  I'm trying to use the new source plugin
> and .wks.in to build a wic image that "mimic" a hddimg, where I can dd
> the wic image to an USB drive and use that usb drive to install image
> to my target machine (like hddimg did ). 
> first partition allow to boot into initramfs and it will execute the
> install script. The script will then install system-boot
> configuration, efi, kernel, rootfs.img and initrd from second
> partition to target machine. 
> 
> My sample image here: 
> https://drive.google.com/file/d/1c1FCJ2ghWNzsXK1zbOAVz1bz6G3H5zx5/view?usp=sharing

OK, so some cleaned up snippets from the installer image recipe:
# List of external things we must have had complete.
do_rootfs[depends] += "core-image-minimal-initramfs:do_image_complete"
do_rootfs[depends] += "full-image-to-install:do_image_complete"
do_rootfs[depends] += "grub-efi:do_deploy"
do_rootfs[depends] += "virtual/kernel:do_deploy"

WKS_FILE = "sdimage-bootpart.wks"

IMAGE_BOOT_FILES_append = "\
    ${KERNEL_IMAGETYPE};EFI/BOOT/${KERNEL_IMAGETYPE} \
    core-image-minimal-initramfs-${MACHINE}.cpio.gz;EFI/BOOT/initrd \
    full-image-to-install-${MACHINE}.wic.xz;rootfs.img \
    grub-efi-bootx64.efi;EFI/BOOT/bootx64.efi \
    grub-installer.cfg;EFI/BOOT/grub.cfg \
    "

And note that I reworked the installer for our case to write then update
the real image's wic image rather than copy contents and then modify
files due to other constraints (this project is also why I was pushing
various changes to meta-secure-core).

> 
> 
> On 5/16/19, 10:38 AM, "Tom Rini" <trini@konsulko.com> wrote:
> 
>     On Thu, May 16, 2019 at 02:23:05AM +0000, Lee, Chee Yang wrote:
>     > Hi Tom, 
>     > 
>     > Would you mind to share with me how you make the installer image using wic? 
>     
>     I ended up using sdimage-bootpart.wks and IMAGE_BOOT_FILES to populate
>     the partition as needed.  I can't paste the exact image as it was done
>     for a customer.
>     
>     > 
>     >     -------- Forwarded Message --------
>     >     From: Tom Rini <trini@konsulko.com>
>     >     To: chee.yang.lee@intel.com
>     >     Cc: openembedded-core@lists.openembedded.org
>     >     Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
>     >     Date: Wed, 15 May 2019 11:33:43 -0400
>     >     
>     >     On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
>     >     wrote:
>     >     
>     >     > From: Chee Yang Lee <chee.yang.lee@intel.com>
>     >     > 
>     >     > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
>     >     > build larger installable image, we can build it in wic based image.
>     >     > 
>     >     > 2 partition needed for installable image
>     >     >  - first partition build using source plugin bootimg-efi and
>     >     > configure to run install
>     >     >  - second partition build with root.img, systemd-boot and kernel to
>     >     > be install on target machine (this partition build using new source
>     >     > plugin installer-partition)
>     >     > 
>     >     > These patches:
>     >     >  - add new source plugin for second partition
>     >     >  - add new .wks
>     >     >  - add new wic dependency and set default value for required variable
>     >     >  - allow source plugin bootimg-efi to configure to install
>     >     > 
>     >     > 
>     >     > To build the image, set WKS_FILE="wic-installer.wks.in" in
>     >     > local.conf.
>     >     
>     >     I'm confused as to why we need most of these changes to support the end
>     >     goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
>     >     see why we need a new plugin to do the rest, however.  What's being
>     >     done
>     >     here that we don't already do today?  I say this having made installer
>     >     images for EFI machines using wic (in sumo, even).  Thanks!
>     >     
>     >     -- 
>     >     Tom
>     >     -- 
>     >     _______________________________________________
>     >     Openembedded-core mailing list
>     >     Openembedded-core@lists.openembedded.org
>     >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>     >     
>     >     
>     > 
>     
>     -- 
>     Tom
>     
> 

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 0/4] add wic based image installer

[Lee, Chee Yang]

I couldn’t figure out how to make it with sdimage-bootpart.wks, can you tell me what did you set for IMAGE_BOOT_FILES ?

To further explain the idea,  I'm trying to use the new source plugin and .wks.in to build a wic image that "mimic" a hddimg, where I can dd the wic image to an USB drive and use that usb drive to install image to my target machine (like hddimg did ). 
first partition allow to boot into initramfs and it will execute the install script. The script will then install system-boot configuration, efi, kernel, rootfs.img and initrd from second partition to target machine. 

My sample image here: 
https://drive.google.com/file/d/1c1FCJ2ghWNzsXK1zbOAVz1bz6G3H5zx5/view?usp=sharing


On 5/16/19, 10:38 AM, "Tom Rini" <trini@konsulko.com> wrote:

    On Thu, May 16, 2019 at 02:23:05AM +0000, Lee, Chee Yang wrote:
    > Hi Tom, 
    > 
    > Would you mind to share with me how you make the installer image using wic? 
    
    I ended up using sdimage-bootpart.wks and IMAGE_BOOT_FILES to populate
    the partition as needed.  I can't paste the exact image as it was done
    for a customer.
    
    > 
    >     -------- Forwarded Message --------
    >     From: Tom Rini <trini@konsulko.com>
    >     To: chee.yang.lee@intel.com
    >     Cc: openembedded-core@lists.openembedded.org
    >     Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
    >     Date: Wed, 15 May 2019 11:33:43 -0400
    >     
    >     On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
    >     wrote:
    >     
    >     > From: Chee Yang Lee <chee.yang.lee@intel.com>
    >     > 
    >     > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
    >     > build larger installable image, we can build it in wic based image.
    >     > 
    >     > 2 partition needed for installable image
    >     >  - first partition build using source plugin bootimg-efi and
    >     > configure to run install
    >     >  - second partition build with root.img, systemd-boot and kernel to
    >     > be install on target machine (this partition build using new source
    >     > plugin installer-partition)
    >     > 
    >     > These patches:
    >     >  - add new source plugin for second partition
    >     >  - add new .wks
    >     >  - add new wic dependency and set default value for required variable
    >     >  - allow source plugin bootimg-efi to configure to install
    >     > 
    >     > 
    >     > To build the image, set WKS_FILE="wic-installer.wks.in" in
    >     > local.conf.
    >     
    >     I'm confused as to why we need most of these changes to support the end
    >     goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
    >     see why we need a new plugin to do the rest, however.  What's being
    >     done
    >     here that we don't already do today?  I say this having made installer
    >     images for EFI machines using wic (in sumo, even).  Thanks!
    >     
    >     -- 
    >     Tom
    >     -- 
    >     _______________________________________________
    >     Openembedded-core mailing list
    >     Openembedded-core@lists.openembedded.org
    >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
    >     
    >     
    > 
    
    -- 
    Tom
    

Re: [PATCH 0/4] add wic based image installer

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

On Thu, May 16, 2019 at 02:23:05AM +0000, Lee, Chee Yang wrote:
> Hi Tom, 
> 
> Would you mind to share with me how you make the installer image using wic? 

I ended up using sdimage-bootpart.wks and IMAGE_BOOT_FILES to populate
the partition as needed.  I can't paste the exact image as it was done
for a customer.

> 
>     -------- Forwarded Message --------
>     From: Tom Rini <trini@konsulko.com>
>     To: chee.yang.lee@intel.com
>     Cc: openembedded-core@lists.openembedded.org
>     Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
>     Date: Wed, 15 May 2019 11:33:43 -0400
>     
>     On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
>     wrote:
>     
>     > From: Chee Yang Lee <chee.yang.lee@intel.com>
>     > 
>     > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
>     > build larger installable image, we can build it in wic based image.
>     > 
>     > 2 partition needed for installable image
>     >  - first partition build using source plugin bootimg-efi and
>     > configure to run install
>     >  - second partition build with root.img, systemd-boot and kernel to
>     > be install on target machine (this partition build using new source
>     > plugin installer-partition)
>     > 
>     > These patches:
>     >  - add new source plugin for second partition
>     >  - add new .wks
>     >  - add new wic dependency and set default value for required variable
>     >  - allow source plugin bootimg-efi to configure to install
>     > 
>     > 
>     > To build the image, set WKS_FILE="wic-installer.wks.in" in
>     > local.conf.
>     
>     I'm confused as to why we need most of these changes to support the end
>     goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
>     see why we need a new plugin to do the rest, however.  What's being
>     done
>     here that we don't already do today?  I say this having made installer
>     images for EFI machines using wic (in sumo, even).  Thanks!
>     
>     -- 
>     Tom
>     -- 
>     _______________________________________________
>     Openembedded-core mailing list
>     Openembedded-core@lists.openembedded.org
>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>     
>     
> 

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[PATCH 0/4] add wic based image installer

[Lee, Chee Yang]

Hi Tom, 

Would you mind to share with me how you make the installer image using wic? 

    -------- Forwarded Message --------
    From: Tom Rini <trini@konsulko.com>
    To: chee.yang.lee@intel.com
    Cc: openembedded-core@lists.openembedded.org
    Subject: Re: [OE-core] [PATCH 0/4] add wic based image installer
    Date: Wed, 15 May 2019 11:33:43 -0400
    
    On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com
    wrote:
    
    > From: Chee Yang Lee <chee.yang.lee@intel.com>
    > 
    > Existing installable image (hddimg/ISO) has limit of 4GB size, so to
    > build larger installable image, we can build it in wic based image.
    > 
    > 2 partition needed for installable image
    >  - first partition build using source plugin bootimg-efi and
    > configure to run install
    >  - second partition build with root.img, systemd-boot and kernel to
    > be install on target machine (this partition build using new source
    > plugin installer-partition)
    > 
    > These patches:
    >  - add new source plugin for second partition
    >  - add new .wks
    >  - add new wic dependency and set default value for required variable
    >  - allow source plugin bootimg-efi to configure to install
    > 
    > 
    > To build the image, set WKS_FILE="wic-installer.wks.in" in
    > local.conf.
    
    I'm confused as to why we need most of these changes to support the end
    goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
    see why we need a new plugin to do the rest, however.  What's being
    done
    here that we don't already do today?  I say this having made installer
    images for EFI machines using wic (in sumo, even).  Thanks!
    
    -- 
    Tom
    -- 
    _______________________________________________
    Openembedded-core mailing list
    Openembedded-core@lists.openembedded.org
    http://lists.openembedded.org/mailman/listinfo/openembedded-core