[PATCH v4 0/7] i#10073: generic EFI for wic

[PATCH v4 0/7] i#10073: generic EFI for wic

[Ed Bartosh]

Hi,

This patchset is an implementation of generic EFI approach for wic images.

Instead of introducing yet another wic plugin it uses existing APIs from
EFI_PROVIDER classes to populate ${WORKDIR}/bootfs directory with EFI artifacts
and bootloader configuration. This directory can be used by wic rootfs plugin
to put into boot partition of the image.

Example kickstart file and 2 test cases were added to wic test suite to better
illustrate the approach.

I personally like this approach much better than duplicating oe image creation
functionality in wic plugins. This way we can have the code in one place and
bootloaders can be configured the same way for oe and wic images.

Changes in v2:
 - added patch: fixed default value of GRUB_ROOT
 - fixed typo in commit message: timage_types_wic > image_types_wic

Changes in v3:
 - scheduled do_prepare_wic_build as earlier as possible

Changes in v4:
 - fixed build on platforms that don't support EFI
 - fixed non-gpl3 build
 - fixed build failure in systemd-boot code caused by rescheduled
   do_populate_bootfs to run earlier

The following changes since commit 920e1592fba8fd714b8fdf87f9792ebfa3377793:

  selftest: fix test_unsupported_subcommand test case (2017-05-17 13:35:28 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wip

Ed Bartosh (7):
  systemd-boot: create output dir if it doesn't exist
  efi: add efi_bootfs_populate API
  image_types_wic: add do_populate_bootfs task
  image_types_wic: merged 2 tasks
  oe-selftest: add wic tests for generic EFI
  grub-efi: fixed default value of GRUB_ROOT
  image_types_wic: schedule prepare_wic_build correctly

 meta-selftest/wic/test_generic_efi.wks.in |  9 ++++
 meta/classes/grub-efi.bbclass             | 14 +++--
 meta/classes/image_types_wic.bbclass      | 85 +++++++++++++++++++++++--------
 meta/classes/systemd-boot.bbclass         |  8 +++
 meta/lib/oeqa/selftest/wic.py             | 36 +++++++++++++
 5 files changed, 126 insertions(+), 26 deletions(-)
 create mode 100644 meta-selftest/wic/test_generic_efi.wks.in

-- 
Regards,
Ed

[PATCH v4 1/7] systemd-boot: create output dir if it doesn't exist

[Ed Bartosh]

build_efi_cfg function creates configuration files for
systemd-boot entries in 'S' directory. This directory
may not exist when api is called, which breaks the build.

Creating the directory if it doesn't exist should fix
this issue.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/systemd-boot.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/systemd-boot.bbclass b/meta/classes/systemd-boot.bbclass
index 4e69a2c6b5..4412fb1ef7 100644
--- a/meta/classes/systemd-boot.bbclass
+++ b/meta/classes/systemd-boot.bbclass
@@ -99,6 +99,8 @@ python build_efi_cfg() {
             bb.fatal('OVERRIDES not defined')
 
         entryfile = "%s/%s.conf" % (s, label)
+        if not os.path.exists(s):
+            os.makedirs(s)
         d.appendVar("SYSTEMD_BOOT_ENTRIES", " " + entryfile)
         try:
             entrycfg = open(entryfile, "w")
-- 
2.12.0

[PATCH v4 2/7] efi: add efi_bootfs_populate API

[Ed Bartosh]

Added API to populate ${WORKDIR}/bootfs directory with EFI
artifacts to both EFI provider classes(grub-efi and systemd-boot).

This API will be used to prepare artifacts for the wic image
build.

[YOCTO #10073]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/grub-efi.bbclass     | 6 ++++++
 meta/classes/systemd-boot.bbclass | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/meta/classes/grub-efi.bbclass b/meta/classes/grub-efi.bbclass
index df7fe18a79..bacbeb4b66 100644
--- a/meta/classes/grub-efi.bbclass
+++ b/meta/classes/grub-efi.bbclass
@@ -71,6 +71,12 @@ efi_hddimg_populate() {
 	efi_populate $1
 }
 
+efi_bootfs_populate() {
+	bootfs_dir="${WORKDIR}/bootfs"
+	efi_populate $bootfs_dir
+	cp --dereference ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE} $bootfs_dir/vmlinuz
+}
+
 python build_efi_cfg() {
     import sys
 
diff --git a/meta/classes/systemd-boot.bbclass b/meta/classes/systemd-boot.bbclass
index 4412fb1ef7..c97adce584 100644
--- a/meta/classes/systemd-boot.bbclass
+++ b/meta/classes/systemd-boot.bbclass
@@ -62,6 +62,12 @@ efi_hddimg_populate() {
         efi_populate $1
 }
 
+efi_bootfs_populate() {
+        bootfs_dir="${WORKDIR}/bootfs"
+        efi_populate $bootfs_dir
+        cp --dereference ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE} $bootfs_dir/vmlinuz
+}
+
 python build_efi_cfg() {
     s = d.getVar("S")
     labels = d.getVar('LABELS')
-- 
2.12.0

[PATCH v4 3/7] image_types_wic: add do_populate_bootfs task

[Ed Bartosh]

This task generates root partition UUID and calls efi_bootfs_populate
API of current EFI provider to populate bootfs directory with
EFI artifacts.

[YOCTO #10073]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types_wic.bbclass | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 4711c24593..3f9824d26a 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -115,3 +115,40 @@ python do_rootfs_wicenv () {
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
+
+# Set variables for efi_bootfs_populate
+GRUB_CFG ?= "${WORKDIR}/grub_wic.cfg"
+SYSTEMD_BOOT_CFG ?= "${WORKDIR}/loader_wic.conf"
+GRUB_GFXSERIAL ?= "1"
+LABELS_WIC ?= "boot install"
+LABELS ?= "${LABELS_WIC}"
+
+EFI_PROVIDER ?= "systemd-boot"
+EFI_CLASS = "${@bb.utils.contains("MACHINE_FEATURES", "efi", "${EFI_PROVIDER}", "", d)}"
+inherit ${EFI_CLASS}
+
+python do_populate_bootfs() {
+    def populate_bootfs(partuuid):
+        # remove bootfs dir as it may have files from previous build
+        bootfs = os.path.join(d.getVar("WORKDIR"), 'bootfs')
+        if os.path.exists(bootfs):
+            import shutil
+            shutil.rmtree(bootfs)
+
+        d.setVar("APPEND", "root=PARTUUID=%s" % partuuid)
+        bb.build.exec_func('build_efi_cfg', d)
+
+        bb.build.exec_func('efi_bootfs_populate', d)
+
+    if d.getVar('USING_WIC'):
+        # Generate parition UUID
+        from uuid import uuid4
+        partuuid = str(uuid4())
+        d.setVar("ROOTFS_PARTUUID", partuuid)
+
+        if d.getVar("EFI_CLASS"):
+            populate_bootfs(partuuid)
+}
+
+addtask do_populate_bootfs before do_image_wic
+do_populate_bootfs[depends] = "${MLPREFIX}${EFI_PROVIDER}:do_deploy virtual/kernel:do_deploy"
-- 
2.12.0

[PATCH v4 4/7] image_types_wic: merged 2 tasks

[Ed Bartosh]

Merged do_write_wks_template and do_populate_bootfs into
new task do_prepare_wic_build to be able to write the same
partition UUID into bootloader configuration files and
kickstart file.

[YOCTO #10073]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types_wic.bbclass | 53 +++++++++++++++++++-----------------
 1 file changed, 28 insertions(+), 25 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 3f9824d26a..d8430e49ac 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -43,27 +43,6 @@ do_image_wic[depends] += "wic-tools:do_populate_sysroot"
 WKS_FILE_DEPENDS ??= ''
 DEPENDS += "${@ '${WKS_FILE_DEPENDS}' if d.getVar('USING_WIC') else '' }"
 
-python do_write_wks_template () {
-    """Write out expanded template contents to WKS_FULL_PATH."""
-    import re
-
-    template_body = d.getVar('_WKS_TEMPLATE')
-
-    # Remove any remnant variable references left behind by the expansion
-    # due to undefined variables
-    expand_var_regexp = re.compile(r"\${[^{}@\n\t :]+}")
-    while True:
-        new_body = re.sub(expand_var_regexp, '', template_body)
-        if new_body == template_body:
-            break
-        else:
-            template_body = new_body
-
-    wks_file = d.getVar('WKS_FULL_PATH')
-    with open(wks_file, 'w') as f:
-        f.write(template_body)
-}
-
 python () {
     if d.getVar('USING_WIC'):
         wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -90,7 +69,6 @@ python () {
                 # file in process_wks_template as well, so just put it in
                 # a variable and let the metadata deal with the deps.
                 d.setVar('_WKS_TEMPLATE', body)
-                bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
 }
 
 #
@@ -127,7 +105,11 @@ EFI_PROVIDER ?= "systemd-boot"
 EFI_CLASS = "${@bb.utils.contains("MACHINE_FEATURES", "efi", "${EFI_PROVIDER}", "", d)}"
 inherit ${EFI_CLASS}
 
-python do_populate_bootfs() {
+python do_prepare_wic_build() {
+    # Prepare required artifacts for the wic image build:
+    #  - Populate {WORKDIR}/bootfs directory with EFI content
+    #  - Write wks.in template into the .wks file
+
     def populate_bootfs(partuuid):
         # remove bootfs dir as it may have files from previous build
         bootfs = os.path.join(d.getVar("WORKDIR"), 'bootfs')
@@ -140,6 +122,23 @@ python do_populate_bootfs() {
 
         bb.build.exec_func('efi_bootfs_populate', d)
 
+    def write_wks_template(template_body, wks_file):
+        """Write out expanded template contents to WKS_FULL_PATH."""
+        import re
+
+        # Remove any remnant variable references left behind by the expansion
+        # due to undefined variables
+        expand_var_regexp = re.compile(r"\${[^{}@\n\t :]+}")
+        while True:
+            new_body = re.sub(expand_var_regexp, '', template_body)
+            if new_body == template_body:
+                break
+            else:
+                template_body = new_body
+
+        with open(wks_file, 'w') as f:
+            f.write(template_body)
+
     if d.getVar('USING_WIC'):
         # Generate parition UUID
         from uuid import uuid4
@@ -148,7 +147,11 @@ python do_populate_bootfs() {
 
         if d.getVar("EFI_CLASS"):
             populate_bootfs(partuuid)
+
+        template = d.getVar("_WKS_TEMPLATE")
+        if template:
+            write_wks_template(template, d.getVar('WKS_FULL_PATH'))
 }
 
-addtask do_populate_bootfs before do_image_wic
-do_populate_bootfs[depends] = "${MLPREFIX}${EFI_PROVIDER}:do_deploy virtual/kernel:do_deploy"
+addtask do_prepare_wic_build before do_image_wic
+do_prepare_wic_build[depends] = "${MLPREFIX}${EFI_PROVIDER}:do_deploy virtual/kernel:do_deploy"
-- 
2.12.0

[PATCH v4 5/7] oe-selftest: add wic tests for generic EFI

[Ed Bartosh]

Added test_generic_efi_grub_qemu and test_generic_efi_systemd_qemu
test cases and wks template to build and boot generic EFI images in qemu.

[YOCTO #10073]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/wic/test_generic_efi.wks.in |  9 ++++++++
 meta/lib/oeqa/selftest/wic.py             | 36 +++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100644 meta-selftest/wic/test_generic_efi.wks.in

diff --git a/meta-selftest/wic/test_generic_efi.wks.in b/meta-selftest/wic/test_generic_efi.wks.in
new file mode 100644
index 0000000000..fecf3fd713
--- /dev/null
+++ b/meta-selftest/wic/test_generic_efi.wks.in
@@ -0,0 +1,9 @@
+# short-description: Create an EFI disk image
+# long-description: Creates a partitioned EFI disk image that the user
+# can directly dd to boot media.
+
+part /boot --source rootfs --rootfs-dir=${WORKDIR}/bootfs --fstype=vfat --label boot --active --align 1024
+
+part / --source rootfs --fstype=ext4 --label root --align 1024 --uuid ${ROOTFS_PARTUUID}
+
+bootloader --ptable gpt
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index cdec80c076..9cb9507830 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -789,3 +789,39 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
                 # 8 blocks is 4K (physical sector size)
                 self.assertEqual(dest_stat.st_blocks, 8)
             os.unlink(dest)
+
+    @only_for_arch(['i586', 'i686', 'x86_64'])
+    def test_generic_efi_grub_qemu(self):
+        """Test generic efi (grub-efi) image in qemu"""
+        config = 'IMAGE_FSTYPES = "wic"\nMACHINE_FEATURES_append = " efi"\n'\
+                 'EFI_PROVIDER = "grub-efi"\n'\
+                 'WKS_FILE = "test_generic_efi.wks.in"\n'
+
+        self.append_config(config)
+        self.assertEqual(0, bitbake('core-image-minimal ovmf').status)
+        self.remove_config(config)
+
+        with runqemu('core-image-minimal', ssh=False,
+                     runqemuparams='ovmf', image_fstype='wic') as qemu:
+            cmd = "grep sda. /proc/partitions  |wc -l"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+            self.assertEqual(output, '2')
+
+    @only_for_arch(['i586', 'i686', 'x86_64'])
+    def test_generic_efi_systemd_qemu(self):
+        """Test generic efi (systemd-boot) image in qemu"""
+        config = 'IMAGE_FSTYPES = "wic"\nMACHINE_FEATURES_append = " efi"\n'\
+                 'EFI_PROVIDER = "systemd-boot"\n'\
+                 'WKS_FILE = "test_generic_efi.wks.in"\n'
+
+        self.append_config(config)
+        self.assertEqual(0, bitbake('core-image-minimal ovmf').status)
+        self.remove_config(config)
+
+        with runqemu('core-image-minimal', ssh=False,
+                     runqemuparams='ovmf', image_fstype='wic') as qemu:
+            cmd = "grep sda. /proc/partitions  |wc -l"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+            self.assertEqual(output, '2')
-- 
2.12.0

[PATCH v4 6/7] grub-efi: fixed default value of GRUB_ROOT

[Ed Bartosh]

Currently GRUB_ROOT assigned a value of literal "${ROOT}" by default
if ROOT variable is not set. This causes kernel commandline to
look like this: "linux /vmlinuz LABEL=boot {ROOT}"

Used weak assignments of ROOT and GRUB_ROOT variables to make
sure GRUB_ROOT is "" by default.

Allowed empty GRUB_ROOT to be able to use only APPEND variable
for both EFI bootloaders (systemd-boot and grub-efi).

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/grub-efi.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/grub-efi.bbclass b/meta/classes/grub-efi.bbclass
index bacbeb4b66..b55ac96083 100644
--- a/meta/classes/grub-efi.bbclass
+++ b/meta/classes/grub-efi.bbclass
@@ -27,7 +27,8 @@ GRUB_TIMEOUT ?= "10"
 GRUB_OPTS ?= "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
 
 EFIDIR = "/EFI/BOOT"
-GRUB_ROOT ?= "${ROOT}"
+ROOT ??= ""
+GRUB_ROOT ??= "${ROOT}"
 APPEND ?= ""
 
 # Need UUID utility code.
@@ -121,8 +122,6 @@ python build_efi_cfg() {
         cfgfile.write('timeout=50\n')
 
     root = d.getVar('GRUB_ROOT')
-    if not root:
-        bb.fatal('GRUB_ROOT not defined')
 
     if gfxserial == "1":
         btypes = [ [ " graphics console", "" ],
@@ -146,7 +145,8 @@ python build_efi_cfg() {
                 lb = "install-efi"
             cfgfile.write('linux /vmlinuz LABEL=%s' % (lb))
 
-            cfgfile.write(' %s' % replace_rootfs_uuid(d, root))
+            if root:
+                cfgfile.write(' %s' % replace_rootfs_uuid(d, root))
 
             append = localdata.getVar('APPEND')
             initrd = localdata.getVar('INITRD')
-- 
2.12.0

[PATCH v4 7/7] image_types_wic: schedule prepare_wic_build correctly

[Ed Bartosh]

Scheduled prepare_wic_build only if wic build enabled.
Added dependencies to kernel and efi bootloader deploy
tasks only if EFI is enabled.

This should fix build failure on machines without
EFI functionality.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types_wic.bbclass | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index d8430e49ac..3b73261d5e 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -69,6 +69,11 @@ python () {
                 # file in process_wks_template as well, so just put it in
                 # a variable and let the metadata deal with the deps.
                 d.setVar('_WKS_TEMPLATE', body)
+
+        bb.build.addtask('do_prepare_wic_build', 'do_image_wic', None, d)
+        if d.getVar('EFI_CLASS'):
+            d.appendVarFlag('do_prepare_wic_build', 'depends',
+                            '%s%s:do_deploy virtual/kernel:do_deploy' % (d.getVar('MLPREFIX'), d.getVar('EFI_CLASS')))
 }
 
 #
@@ -139,19 +144,15 @@ python do_prepare_wic_build() {
         with open(wks_file, 'w') as f:
             f.write(template_body)
 
-    if d.getVar('USING_WIC'):
-        # Generate parition UUID
-        from uuid import uuid4
-        partuuid = str(uuid4())
-        d.setVar("ROOTFS_PARTUUID", partuuid)
+    # Generate parition UUID
+    from uuid import uuid4
+    partuuid = str(uuid4())
+    d.setVar("ROOTFS_PARTUUID", partuuid)
 
-        if d.getVar("EFI_CLASS"):
-            populate_bootfs(partuuid)
+    if d.getVar("EFI_CLASS"):
+        populate_bootfs(partuuid)
 
-        template = d.getVar("_WKS_TEMPLATE")
-        if template:
-            write_wks_template(template, d.getVar('WKS_FULL_PATH'))
+    template = d.getVar("_WKS_TEMPLATE")
+    if template:
+        write_wks_template(template, d.getVar('WKS_FULL_PATH'))
 }
-
-addtask do_prepare_wic_build before do_image_wic
-do_prepare_wic_build[depends] = "${MLPREFIX}${EFI_PROVIDER}:do_deploy virtual/kernel:do_deploy"
-- 
2.12.0

Re: [PATCH v4 7/7] image_types_wic: schedule prepare_wic_build correctly

[Wold, Saul]

On Wed, 2017-05-17 at 13:47 +0000, Ed Bartosh wrote:
> Scheduled prepare_wic_build only if wic build enabled.
> Added dependencies to kernel and efi bootloader deploy
> tasks only if EFI is enabled.
> 
> This should fix build failure on machines without
> EFI functionality.
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/classes/image_types_wic.bbclass | 27 ++++++++++++++----------
> ---
>  1 file changed, 14 insertions(+), 13 deletions(-)
> 
> diff --git a/meta/classes/image_types_wic.bbclass
> b/meta/classes/image_types_wic.bbclass
> index d8430e49ac..3b73261d5e 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -69,6 +69,11 @@ python () {
>                  # file in process_wks_template as well, so just put
> it in
>                  # a variable and let the metadata deal with the
> deps.
>                  d.setVar('_WKS_TEMPLATE', body)
> +
> +        bb.build.addtask('do_prepare_wic_build', 'do_image_wic',
> None, d)
> +        if d.getVar('EFI_CLASS'):
> +            d.appendVarFlag('do_prepare_wic_build', 'depends',
> +                            '%s%s:do_deploy 
> virtual/kernel:do_deploy' % (d.getVar('MLPREFIX'),
> d.getVar('EFI_CLASS')))

Ed, 
Have you tested this with any layers?  I tied recently with meta-
intel and tripped over an issue with the rmc-boot not having an actual
target for the EFI_CLASS to have a deploy task caused a failure.

Sau!

>  }
>  
>  #
> @@ -139,19 +144,15 @@ python do_prepare_wic_build() {
>          with open(wks_file, 'w') as f:
>              f.write(template_body)
>  
> -    if d.getVar('USING_WIC'):
> -        # Generate parition UUID
> -        from uuid import uuid4
> -        partuuid = str(uuid4())
> -        d.setVar("ROOTFS_PARTUUID", partuuid)
> +    # Generate parition UUID
> +    from uuid import uuid4
> +    partuuid = str(uuid4())
> +    d.setVar("ROOTFS_PARTUUID", partuuid)
>  
> -        if d.getVar("EFI_CLASS"):
> -            populate_bootfs(partuuid)
> +    if d.getVar("EFI_CLASS"):
> +        populate_bootfs(partuuid)
>  
> -        template = d.getVar("_WKS_TEMPLATE")
> -        if template:
> -            write_wks_template(template, d.getVar('WKS_FULL_PATH'))
> +    template = d.getVar("_WKS_TEMPLATE")
> +    if template:
> +        write_wks_template(template, d.getVar('WKS_FULL_PATH'))
>  }
> -
> -addtask do_prepare_wic_build before do_image_wic
> -do_prepare_wic_build[depends] =
> "${MLPREFIX}${EFI_PROVIDER}:do_deploy virtual/kernel:do_deploy"
> -- 
> 2.12.0
> 

Re: [PATCH v4 7/7] image_types_wic: schedule prepare_wic_build correctly

[Ed Bartosh]

On Fri, May 19, 2017 at 10:55:46PM +0000, Wold, Saul wrote:
> On Wed, 2017-05-17 at 13:47 +0000, Ed Bartosh wrote:
> > Scheduled prepare_wic_build only if wic build enabled.
> > Added dependencies to kernel and efi bootloader deploy
> > tasks only if EFI is enabled.
> > 
> > This should fix build failure on machines without
> > EFI functionality.
> > 
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> >  meta/classes/image_types_wic.bbclass | 27 ++++++++++++++----------
> > ---
> >  1 file changed, 14 insertions(+), 13 deletions(-)
> > 
> > diff --git a/meta/classes/image_types_wic.bbclass
> > b/meta/classes/image_types_wic.bbclass
> > index d8430e49ac..3b73261d5e 100644
> > --- a/meta/classes/image_types_wic.bbclass
> > +++ b/meta/classes/image_types_wic.bbclass
> > @@ -69,6 +69,11 @@ python () {
> >                  # file in process_wks_template as well, so just put
> > it in
> >                  # a variable and let the metadata deal with the
> > deps.
> >                  d.setVar('_WKS_TEMPLATE', body)
> > +
> > +        bb.build.addtask('do_prepare_wic_build', 'do_image_wic',
> > None, d)
> > +        if d.getVar('EFI_CLASS'):
> > +            d.appendVarFlag('do_prepare_wic_build', 'depends',
> > +                            '%s%s:do_deploy 
> > virtual/kernel:do_deploy' % (d.getVar('MLPREFIX'),
> > d.getVar('EFI_CLASS')))
> 
> Ed, 
> Have you tested this with any layers?
I tested it with poky layers as I do for all my patches.

> I tied recently with meta-
> intel and tripped over an issue with the rmc-boot not having an actual
> target for the EFI_CLASS to have a deploy task caused a failure.

I also tested it on qemuarm machine that doesn't support EFI. It worked just fine.

I'll try to test it with meta-intel. Can you provide more detailed
instructions how to reproduce this issue? Which image did you try to
build?

--
Regards,
Ed

Re: [PATCH v4 7/7] image_types_wic: schedule prepare_wic_build correctly

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On 22 May 2017 at 08:19, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:

> I'll try to test it with meta-intel. Can you provide more detailed
> instructions how to reproduce this issue? Which image did you try to
> build?
>

I just hit this.

image_types_wic does this:

+            d.appendVarFlag('do_prepare_wic_build', 'depends',
+                            '%s%s:do_deploy virtual/kernel:do_deploy' %
(d.getVar('MLPREFIX'), d.getVar('EFI_CLASS')))

Where:

meta/classes/live-vm-common.bbclass:EFI_CLASS =
"${@bb.utils.contains("MACHINE_FEATURES", "efi", "${EFI_PROVIDER}", "", d)}"

With meta-intel, EFI_PROVIDER is rmc-boot, which is a class but not a
recipe.

Ross

[-- Attachment #2: Type: text/html, Size: 1681 bytes --]

Re: [PATCH v4 0/7] i#10073: generic EFI for wic

[Wold, Saul]

On Wed, 2017-05-17 at 13:47 +0000, Ed Bartosh wrote:
> Hi,
> 
> This patchset is an implementation of generic EFI approach for wic
> images.
> 
> Instead of introducing yet another wic plugin it uses existing APIs
> from
> EFI_PROVIDER classes to populate ${WORKDIR}/bootfs directory with EFI
> artifacts
> and bootloader configuration. This directory can be used by wic
> rootfs plugin
> to put into boot partition of the image.
> 
> Example kickstart file and 2 test cases were added to wic test suite
> to better
> illustrate the approach.
> 
> I personally like this approach much better than duplicating oe image
> creation
> functionality in wic plugins. This way we can have the code in one
> place and
> bootloaders can be configured the same way for oe and wic images.
> 

Ed,

I have been looking at this set of changes over the last few days, and
while I like the basic approach, I don't think it goes far enough.  I
am looking at how one would create a generic bootfs via a DEPLOY_DIR
and an image-bootfs.bb type of recipe that does the bootfs construction
instead of the existing systemd-boot or grub-efi bbclass construction.

This will allow for more flexibility in creating labels for
bootloaders, different types of boot loaders without having to create
classes.  We should then be able to construct the equivlant of hddimg
or iso images via a recipe that depends on the various do_deploy()
tasks of the images dependent items (syslinux, grub, virtual/kernel)
and configuration.

I think we are still too locked into prepare_wic_build() calling
build_efi_cfg and efi_bootfs_populate, which means wic still needs
knowledge of the boot loaders instead of just having a recipe populate
a bootfs partition and wic creating the partition and copying files
into it.

Maybe I am missing something, so please correct me if I am wrong.  I
have an example of what I am thinking of in contrib/sgw/wic_wip

Thanks

Sau!


> Changes in v2:
>  - added patch: fixed default value of GRUB_ROOT
>  - fixed typo in commit message: timage_types_wic > image_types_wic
> 
> Changes in v3:
>  - scheduled do_prepare_wic_build as earlier as possible
> 
> Changes in v4:
>  - fixed build on platforms that don't support EFI
>  - fixed non-gpl3 build
>  - fixed build failure in systemd-boot code caused by rescheduled
>    do_populate_bootfs to run earlier
> 
> The following changes since commit
> 920e1592fba8fd714b8fdf87f9792ebfa3377793:
> 
>   selftest: fix test_unsupported_subcommand test case (2017-05-17
> 13:35:28 +0000)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib ed/wic/wip
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wip
> 
> Ed Bartosh (7):
>   systemd-boot: create output dir if it doesn't exist
>   efi: add efi_bootfs_populate API
>   image_types_wic: add do_populate_bootfs task
>   image_types_wic: merged 2 tasks
>   oe-selftest: add wic tests for generic EFI
>   grub-efi: fixed default value of GRUB_ROOT
>   image_types_wic: schedule prepare_wic_build correctly
> 
>  meta-selftest/wic/test_generic_efi.wks.in |  9 ++++
>  meta/classes/grub-efi.bbclass             | 14 +++--
>  meta/classes/image_types_wic.bbclass      | 85
> +++++++++++++++++++++++--------
>  meta/classes/systemd-boot.bbclass         |  8 +++
>  meta/lib/oeqa/selftest/wic.py             | 36 +++++++++++++
>  5 files changed, 126 insertions(+), 26 deletions(-)
>  create mode 100644 meta-selftest/wic/test_generic_efi.wks.in
> 
> -- 
> Regards,
> Ed
> 

Re: [PATCH v4 0/7] i#10073: generic EFI for wic

[Ed Bartosh]

On Thu, Jun 01, 2017 at 03:21:51PM +0000, Wold, Saul wrote:
> On Wed, 2017-05-17 at 13:47 +0000, Ed Bartosh wrote:
> > Hi,
> > 
> > This patchset is an implementation of generic EFI approach for wic
> > images.
> > 
> > Instead of introducing yet another wic plugin it uses existing APIs
> > from
> > EFI_PROVIDER classes to populate ${WORKDIR}/bootfs directory with EFI
> > artifacts
> > and bootloader configuration. This directory can be used by wic
> > rootfs plugin
> > to put into boot partition of the image.
> > 
> > Example kickstart file and 2 test cases were added to wic test suite
> > to better
> > illustrate the approach.
> > 
> > I personally like this approach much better than duplicating oe image
> > creation
> > functionality in wic plugins. This way we can have the code in one
> > place and
> > bootloaders can be configured the same way for oe and wic images.
> > 
> 
> Ed,
> 
> I have been looking at this set of changes over the last few days, and
> while I like the basic approach, I don't think it goes far enough.  I
> am looking at how one would create a generic bootfs via a DEPLOY_DIR
> and an image-bootfs.bb type of recipe that does the bootfs construction
> instead of the existing systemd-boot or grub-efi bbclass construction.
> 
> This will allow for more flexibility in creating labels for
> bootloaders, different types of boot loaders without having to create
> classes.  We should then be able to construct the equivlant of hddimg
> or iso images via a recipe that depends on the various do_deploy()
> tasks of the images dependent items (syslinux, grub, virtual/kernel)
> and configuration.
I like the idea in general. In practice it could be hard to synchronize
variables between a recipe and .wks file

Obvious example of this would be generation of partition UUID and
writing it into bootloader configs and .wks.in

I spent quite a bit of time on this. The only way I found to get them
synchronized is to merge both tasks (generation of .wks file from
.wks.in and generation of bootloader config) into one:
http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136551.html

If we find a way to solve this then I'm all for your proposal.

> I think we are still too locked into prepare_wic_build() calling
> build_efi_cfg and efi_bootfs_populate, which means wic still needs
> knowledge of the boot loaders instead of just having a recipe populate
> a bootfs partition and wic creating the partition and copying files
> into it.
Well, I tend to disagree here. Yes, my code expects that build_efi_cfg
and efi_bootfs_populate APIs are provided by bootloader class. However,
it doesn't need to know any bootloader-specific internals. It calls
those 2 APIs of current EFI provider class and expects fully populated
bootloader artifacts in bootfs directory.

BTW, even if we use recipes to generate bootfs we'll most probably end
up using some classes to avoid duplication of code in recipes. Those
classes will probably contain less code than currently, but we'll need
them unless we want to duplicate generation of bootloader configs and
copying bootloader files in all bootf recipes.

Regards,
Ed