[PATCH 0/4] add wic based image installer

[PATCH 0/4] add wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.

2 partition needed for installable image
 - first partition build using source plugin bootimg-efi and configure to run install
 - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)

These patches:
 - add new source plugin for second partition
 - add new .wks
 - add new wic dependency and set default value for required variable
 - allow source plugin bootimg-efi to configure to install


To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.


Chee Yang Lee (4):
  wic: add new installer-partition plugin
  wic: bootimg-efi: add label source parameter
  wic: add new wic-installer.wks for wic based image installer
  image_types_wic: add dependency for wic based image installer

 meta/classes/image_types_wic.bbclass               |   6 +
 scripts/lib/wic/canned-wks/wic-installer.wks.in    |   7 +
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  20 ++-
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 4 files changed, 220 insertions(+), 5 deletions(-)
 create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

-- 
2.7.4

[PATCH 1/4] wic: add new installer-partition plugin

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

This patch implement 'installer-partition.py' source plugin for wic.
The plugin create an image with systemd-boot and rootfs.
The generated image can be use to create wic based image installer.

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 1 file changed, 192 insertions(+)
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

diff --git a/scripts/lib/wic/plugins/source/installer-partition.py b/scripts/lib/wic/plugins/source/installer-partition.py
new file mode 100644
index 0000000..4d8343f
--- /dev/null
+++ b/scripts/lib/wic/plugins/source/installer-partition.py
@@ -0,0 +1,192 @@
+# Copyright (c) 2019, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'installer-partition' source plugin class for 'wic'
+# This plugin prepare partition with content to be install on target machine.
+#
+# AUTHORS
+# Lee Chee Yang <Chee.Yang.Lee (at] intel.com>
+#
+
+import logging
+import os
+
+from wic import WicError
+from wic.engine import get_custom_config
+from wic.pluginbase import SourcePlugin
+from wic.misc import (exec_cmd, exec_native_cmd, get_bitbake_var)
+
+logger = logging.getLogger('wic')
+
+class InstallerImagePlugin(SourcePlugin):
+    """
+    Populate content for wic image based installer
+    """
+
+    name = 'installer-partition'
+
+    @classmethod
+    def do_configure_partition(cls, part, source_params, creator, cr_workdir,
+                               oe_builddir, bootimg_dir, kernel_dir,
+                               native_sysroot):
+        """
+        Called before do_prepare_partition(), creates loader-specific config
+        """
+        if not kernel_dir:
+            kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+            if not kernel_dir:
+                raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+        staging_kernel_dir = kernel_dir
+
+        partition_dir = "%s/%s-%s" % (cr_workdir, part.label, part.lineno)
+
+        install_cmd = "install -d %s/EFI/BOOT" % partition_dir
+        exec_cmd(install_cmd)
+
+        install_cmd = "install -d %s/loader/entries" % partition_dir
+        exec_cmd(install_cmd)
+
+        bootloader = creator.ks.bootloader
+
+        loader_conf = ""
+        loader_conf += "default boot\n"
+        loader_conf += "timeout %d\n" % bootloader.timeout
+
+        initrd = source_params.get('initrd')
+
+        if initrd:
+            cp_cmd = "cp %s/%s %s" % (kernel_dir, initrd, partition_dir)
+            exec_cmd(cp_cmd, True)
+        else:
+            logger.debug("Ignoring missing initrd")
+
+        logger.debug("Writing systemd-boot config "
+                     "%s/loader/loader.conf", partition_dir)
+        cfg = open("%s/loader/loader.conf" % partition_dir, "w")
+        cfg.write(loader_conf)
+        cfg.close()
+
+        kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+        if not kernel:
+            kernel = "bzImage"
+
+        install_cmd = "install -m 0644 %s/%s %s/%s" % \
+            (staging_kernel_dir, kernel, partition_dir, kernel)
+        exec_cmd(install_cmd)
+
+        configfile = creator.ks.bootloader.configfile
+        custom_cfg = None
+        if configfile:
+            custom_cfg = get_custom_config(configfile)
+            if custom_cfg:
+                # Use a custom configuration for systemd-boot
+                boot_conf = custom_cfg
+                logger.debug("Using custom configuration file "
+                             "%s for systemd-boots's boot.conf", configfile)
+            else:
+                raise WicError("configfile is specified but failed to "
+                               "get it from %s.", configfile)
+
+        if not custom_cfg:
+            # Create systemd-boot configuration using parameters from wks file
+            title = source_params.get('title')
+
+            boot_conf = ""
+            boot_conf += "title %s\n" % (title if title else "boot")
+            boot_conf += "linux /%s\n" % kernel
+            boot_conf += "options LABEL=Boot %s\n" % (bootloader.append)
+
+            if initrd:
+                boot_conf += "initrd /%s\n" % initrd
+
+        for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
+            cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, partition_dir, mod[8:])
+            exec_cmd(cp_cmd, True)
+
+        logger.debug("Writing systemd-boot config "
+                     "%s/loader/entries/boot.conf", partition_dir)
+        cfg = open("%s//loader/entries/boot.conf" % partition_dir, "w")
+        cfg.write(boot_conf)
+        cfg.close()
+
+
+    @classmethod
+    def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
+                             oe_builddir, bootimg_dir, kernel_dir,
+                             rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition,
+        prepare systemd bootloader and rootfs image
+        """
+        partition_dir = "%s/%s-%s" % (cr_workdir, part.label, part.lineno)
+
+        rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, part.label, part.lineno, part.fstype)
+
+        image_deploy_dir = get_bitbake_var("IMGDEPLOYDIR")
+        if not image_deploy_dir:
+            raise WicError("Couldn't find IMGDEPLOYDIR, exiting")
+
+        rootfs_img = "%s/%s.%s" % (image_deploy_dir,
+            get_bitbake_var("IMAGE_LINK_NAME"), part.fstype )
+        if not os.path.isfile(rootfs_img):
+            raise WicError("Couldn't find %s, exiting" % rootfs_img)
+
+        install_cmd = "install -m 0644 %s %s/rootfs.img" % \
+            (rootfs_img, partition_dir)
+        exec_cmd(install_cmd)
+
+        #look for actual size required for the partition
+        du_cmd = "du -ks %s" % partition_dir
+        out = exec_cmd(du_cmd)
+        part.size = cls.get_partition_size(part, int(out.split()[0]))
+        logger.debug("Set partition size to %d ", part.size)
+
+        with open(rootfs, 'w') as sparse:
+            os.ftruncate(sparse.fileno(), part.size * 1024)
+
+        extraopts = part.mkfs_extraopts or "-i 8192"
+
+        label_str = ""
+        if part.label:
+            label_str = "-L %s" % part.label
+
+        mkfs_cmd = "mkfs.%s -F %s %s -U %s %s -d %s" % \
+            (part.fstype, extraopts, label_str, part.fsuuid, rootfs, partition_dir)
+        exec_native_cmd(mkfs_cmd, native_sysroot)
+
+        part.source_file = rootfs
+
+
+    def get_partition_size(part, actual_partition_size=0):
+        """
+        Calculate the required size of the partition, taking into consideration
+        --size/--fixed-size/--extra-space flags specified in kickstart file.
+        Raises an error if the `actual_partition_size` is larger than fixed-size.
+        """
+        if part.fixed_size:
+            partition_size = part.fixed_size
+            if actual_partition_size > partition_size:
+                raise WicError("Actual partition size (%d kB) is larger than "
+                               "allowed size %d kB" %
+                               (actual_partition_size, partition_size))
+        else:
+            extra_space = 0
+            if extra_space < part.extra_space:
+                extra_space = part.extra_space
+            partition_size = actual_partition_size + extra_space
+
+        return partition_size
-- 
2.7.4

[PATCH 2/4] wic: bootimg-efi: add label source parameter

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Add new source parameter label to allow boot.conf/grub.cfg label option
to set as "install-efi", so far it's hardcoded to "Boot".

The label remain to "Boot" for systemd-boot or blank for grub-efi whenever
label are not "install-efi".

when label set to "install-efi" the image can boot into image installation.

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 83a7e18..0a0c5bd 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -86,9 +86,14 @@ class BootimgEFIPlugin(SourcePlugin):
             grubefi_conf += "menuentry '%s'{\n" % (title if title else "boot")
 
             kernel = "/bzImage"
+            label = source_params.get('label')
 
-            grubefi_conf += "linux %s root=%s rootwait %s\n" \
-                % (kernel, creator.rootdev, bootloader.append)
+            if label == "install-efi":
+                grubefi_conf += "linux %s LABEL=%s rootwait %s\n" \
+                    % (kernel, label, bootloader.append)
+            else:
+                grubefi_conf += "linux %s root=%s rootwait %s\n" \
+                    % (kernel, creator.rootdev, bootloader.append)
 
             if initrd:
                grubefi_conf += "initrd /%s\n" % initrd
@@ -154,12 +159,17 @@ class BootimgEFIPlugin(SourcePlugin):
             # Create systemd-boot configuration using parameters from wks file
             kernel = "/bzImage"
             title = source_params.get('title')
-
+            label = source_params.get('label')
             boot_conf = ""
             boot_conf += "title %s\n" % (title if title else "boot")
             boot_conf += "linux %s\n" % kernel
-            boot_conf += "options LABEL=Boot root=%s %s\n" % \
-                             (creator.rootdev, bootloader.append)
+
+            if label == "install-efi":
+                boot_conf += "options LABEL=%s %s\n" % \
+                                (label, bootloader.append)
+            else:
+                boot_conf += "options LABEL=Boot root=%s %s\n" % \
+                                 (creator.rootdev, bootloader.append)
 
             if initrd:
                 boot_conf += "initrd /%s\n" % initrd
-- 
2.7.4

[PATCH 3/4] wic: add new wic-installer.wks for wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 scripts/lib/wic/canned-wks/wic-installer.wks.in | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in

diff --git a/scripts/lib/wic/canned-wks/wic-installer.wks.in b/scripts/lib/wic/canned-wks/wic-installer.wks.in
new file mode 100644
index 0000000..a7b018d
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/wic-installer.wks.in
@@ -0,0 +1,7 @@
+# description: Create wic installable image
+
+part /boot --source bootimg-efi --sourceparams="loader=systemd-boot,title=install,label=install-efi,initrd=${INITRD_IMAGE_LIVE}-${MACHINE}.${INITRAMFS_FSTYPES}" --ondisk sda --label install --active --align 1024 --use-uuid
+
+part / --source installer-partition --ondisk sda --fstype=ext4 --label image --use-uuid
+
+bootloader --ptable gpt --timeout=5 --append="rootwait console=ttyS0,115200 console=tty0"
-- 
2.7.4

[PATCH 4/4] image_types_wic: add dependency for wic based image installer

[chee.yang.lee]

From: Chee Yang Lee <chee.yang.lee@intel.com>

initramfs and rootfs image are required to build wic based
image installer, add both as dependency for do_image_wic.

initiate default value for LIVE_ROOTFS_TYPE, INITRD_IMAGE_LIVE
and INITRD_LIVE

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
---
 meta/classes/image_types_wic.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 5b40a9e..e4f2b8f 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -102,9 +102,15 @@ python () {
                 # a variable and let the metadata deal with the deps.
                 d.setVar('_WKS_TEMPLATE', body)
                 bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
+        d.appendVarFlag('do_image_wic', 'depends', ' %s:do_image_complete' % d.getVar('INITRD_IMAGE_LIVE'))
         bb.build.addtask('do_image_wic', 'do_image_complete', None, d)
 }
 
+LIVE_ROOTFS_TYPE ?= "ext4"
+INITRD_IMAGE_LIVE ?= "${MLPREFIX}core-image-minimal-initramfs"
+INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.${INITRAMFS_FSTYPES}"
+IMAGE_TYPEDEP_wic = "${LIVE_ROOTFS_TYPE}"
+
 #
 # Write environment variables used by wic
 # to tmp/sysroots/<machine>/imgdata/<image>.env
-- 
2.7.4

Re: [PATCH 0/4] add wic based image installer

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]

On Wed, May 08, 2019 at 02:40:58PM +0800, chee.yang.lee@intel.com wrote:

> From: Chee Yang Lee <chee.yang.lee@intel.com>
> 
> Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.
> 
> 2 partition needed for installable image
>  - first partition build using source plugin bootimg-efi and configure to run install
>  - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)
> 
> These patches:
>  - add new source plugin for second partition
>  - add new .wks
>  - add new wic dependency and set default value for required variable
>  - allow source plugin bootimg-efi to configure to install
> 
> 
> To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.

I'm confused as to why we need most of these changes to support the end
goal.  The enhancement to bootimg-efi makes conceptual sense.  I don't
see why we need a new plugin to do the rest, however.  What's being done
here that we don't already do today?  I say this having made installer
images for EFI machines using wic (in sumo, even).  Thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 0/4] add wic based image installer

[Lee, Chee Yang]

Any comments on these patches?

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org <openembedded-core-bounces@lists.openembedded.org> On Behalf Of chee.yang.lee@intel.com
Sent: Wednesday, May 8, 2019 2:41 PM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] [PATCH 0/4] add wic based image installer

From: Chee Yang Lee <chee.yang.lee@intel.com>

Existing installable image (hddimg/ISO) has limit of 4GB size, so to build larger installable image, we can build it in wic based image.

2 partition needed for installable image
 - first partition build using source plugin bootimg-efi and configure to run install
 - second partition build with root.img, systemd-boot and kernel to be install on target machine (this partition build using new source plugin installer-partition)

These patches:
 - add new source plugin for second partition
 - add new .wks
 - add new wic dependency and set default value for required variable
 - allow source plugin bootimg-efi to configure to install


To build the image, set WKS_FILE="wic-installer.wks.in" in local.conf.


Chee Yang Lee (4):
  wic: add new installer-partition plugin
  wic: bootimg-efi: add label source parameter
  wic: add new wic-installer.wks for wic based image installer
  image_types_wic: add dependency for wic based image installer

 meta/classes/image_types_wic.bbclass               |   6 +
 scripts/lib/wic/canned-wks/wic-installer.wks.in    |   7 +
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  20 ++-
 .../lib/wic/plugins/source/installer-partition.py  | 192 +++++++++++++++++++++
 4 files changed, 220 insertions(+), 5 deletions(-)  create mode 100644 scripts/lib/wic/canned-wks/wic-installer.wks.in
 create mode 100644 scripts/lib/wic/plugins/source/installer-partition.py

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core