[PATCH v2 1/5] wic: partition: Support valueless keys in sourceparams

[PATCH v2 1/5] wic: partition: Support valueless keys in sourceparams

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Accept valueless keys in sourceparams without equals sign (=) to match
the comment and support Boolean entries.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

(no changes since v1)

 scripts/lib/wic/partition.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index a25834048e..09e491dd49 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -171,7 +171,7 @@ class Partition():
             # Split sourceparams string of the form key1=val1[,key2=val2,...]
             # into a dict.  Also accepts valueless keys i.e. without =
             splitted = self.sourceparams.split(',')
-            srcparams_dict = dict(par.split('=', 1) for par in splitted if par)
+            srcparams_dict = dict((par.split('=', 1) + [None])[:2] for par in splitted if par)
 
         plugin = PluginMgr.get_plugins('source')[self.source]
         plugin.do_configure_partition(self, srcparams_dict, creator,
-- 
2.30.2

[PATCH v2 2/5] wic: rawcopy: Add support for packed images

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add support for packed images to wic rawcopy handler do minimize disk
usage in deploy directory and reuse of packed images between wic and
swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz
archives.

Example:
part / --source rawcopy --sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack"

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Reword WicError message and add compressor filename extension

 scripts/lib/wic/plugins/source/rawcopy.py | 29 ++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py
index fa7b1eb8ac..7c90cd3cf8 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -4,6 +4,8 @@
 
 import logging
 import os
+import signal
+import subprocess
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
@@ -38,6 +40,25 @@ class RawCopyPlugin(SourcePlugin):
 
         exec_cmd(cmd)
 
+    @staticmethod
+    def do_image_uncompression(src, dst, workdir):
+        def subprocess_setup():
+            # Python installs a SIGPIPE handler by default. This is usually not what
+            # non-Python subprocesses expect.
+            # SIGPIPE errors are known issues with gzip/bash
+            signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+        extension = os.path.splitext(src)[1]
+        decompressor = {
+            ".bz2": "bzip2",
+            ".gz": "gzip",
+            ".xz": "xz"
+        }.get(extension)
+        if not decompressor:
+            raise WicError("Not supported compressor filename extension: %s" % extension)
+        cmd = "%s -dc %s > %s" % (decompressor, src, dst)
+        subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=workdir)
+
     @classmethod
     def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
                              oe_builddir, bootimg_dir, kernel_dir,
@@ -56,7 +77,13 @@ class RawCopyPlugin(SourcePlugin):
         if 'file' not in source_params:
             raise WicError("No file specified")
 
-        src = os.path.join(kernel_dir, source_params['file'])
+        if 'unpack' in source_params:
+            img = os.path.join(kernel_dir, source_params['file'])
+            src = os.path.join(cr_workdir, os.path.splitext(source_params['file'])[0])
+            RawCopyPlugin.do_image_uncompression(img, src, cr_workdir)
+        else:
+            src = os.path.join(kernel_dir, source_params['file'])
+
         dst = os.path.join(cr_workdir, "%s.%s" % (os.path.basename(source_params['file']), part.lineno))
 
         if not os.path.exists(os.path.dirname(dst)):
-- 
2.30.2

[PATCH v2 3/5] selftest: wic: Remove requirement of syslinux from test_rawcopy_plugin

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Remove bootimg-pcbios from wks to eliminate requirement of syslinux from
test_rawcopy_plugin to avoid the following error.

ERROR: Couldn't find correct bootimg_dir, exiting

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

(no changes since v1)

 meta/lib/oeqa/selftest/cases/wic.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 5fc8e65142..96b3e1b6a5 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1070,10 +1070,8 @@ class Wic2(WicTestCase):
         img = 'core-image-minimal'
         machine = get_bb_var('MACHINE', img)
         with NamedTemporaryFile("w", suffix=".wks") as wks:
-            wks.writelines(['part /boot --active --source bootimg-pcbios\n',
-                            'part / --source rawcopy --sourceparams="file=%s-%s.ext4" --use-uuid\n'\
-                             % (img, machine),
-                            'bootloader --timeout=0 --append="console=ttyS0,115200n8"\n'])
+            wks.write('part / --source rawcopy --sourceparams="file=%s-%s.ext4"\n'\
+                      % (img, machine))
             wks.flush()
             cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
             runCmd(cmd)
-- 
2.30.2

[PATCH v2 4/5] selftest: wic: Add rawcopy plugin unpack test

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

(no changes since v1)

 meta/lib/oeqa/selftest/cases/wic.py | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 96b3e1b6a5..a021f8d84b 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1065,13 +1065,14 @@ class Wic2(WicTestCase):
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
             self.assertEqual(output, '2')
 
-    def test_rawcopy_plugin(self):
+    def _rawcopy_plugin(self, fstype):
         """Test rawcopy plugin"""
         img = 'core-image-minimal'
         machine = get_bb_var('MACHINE', img)
+        params = ',unpack' if fstype.endswith('.gz') else ''
         with NamedTemporaryFile("w", suffix=".wks") as wks:
-            wks.write('part / --source rawcopy --sourceparams="file=%s-%s.ext4"\n'\
-                      % (img, machine))
+            wks.write('part / --source rawcopy --sourceparams="file=%s-%s.%s%s"\n'\
+                      % (img, machine, fstype, params))
             wks.flush()
             cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
             runCmd(cmd)
@@ -1079,6 +1080,17 @@ class Wic2(WicTestCase):
             out = glob(self.resultdir + "%s-*direct" % wksname)
             self.assertEqual(1, len(out))
 
+    def test_rawcopy_plugin(self):
+        self._rawcopy_plugin('ext4')
+
+    def test_rawcopy_plugin_unpack(self):
+        fstype = 'ext4.gz'
+        config = 'IMAGE_FSTYPES = "%s"\n' % fstype
+        self.append_config(config)
+        self.assertEqual(0, bitbake('core-image-minimal').status)
+        self.remove_config(config)
+        self._rawcopy_plugin(fstype)
+
     def test_empty_plugin(self):
         """Test empty plugin"""
         config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "test_empty_plugin.wks"\n'
-- 
2.30.2

[PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Disable graphic support of qemu to support qemu tests in WSL.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Add patch

 meta/lib/oeqa/selftest/cases/wic.py | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index a021f8d84b..6f3dc27743 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -851,7 +851,7 @@ class Wic2(WicTestCase):
         self.assertEqual(0, bitbake('wic-image-minimal').status)
         self.remove_config(config)
 
-        with runqemu('wic-image-minimal', ssh=False) as qemu:
+        with runqemu('wic-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
             cmd = "mount | grep '^/dev/' | cut -f1,3 -d ' ' | egrep -c -e '/dev/sda1 /boot' " \
                   "-e '/dev/root /|/dev/sda2 /' -e '/dev/sda3 /media' -e '/dev/sda4 /mnt'"
             status, output = qemu.run_serial(cmd)
@@ -871,7 +871,7 @@ class Wic2(WicTestCase):
         self.remove_config(config)
 
         with runqemu('core-image-minimal', ssh=False,
-                     runqemuparams='ovmf', image_fstype='wic') as qemu:
+                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
             cmd = "grep sda. /proc/partitions  |wc -l"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
@@ -1059,7 +1059,8 @@ class Wic2(WicTestCase):
         self.assertEqual(0, bitbake('core-image-minimal-mtdutils').status)
         self.remove_config(config)
 
-        with runqemu('core-image-minimal-mtdutils', ssh=False, image_fstype='wic') as qemu:
+        with runqemu('core-image-minimal-mtdutils', ssh=False,
+                     runqemuparams='nographic', image_fstype='wic') as qemu:
             cmd = "grep sda. /proc/partitions  |wc -l"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
@@ -1119,7 +1120,8 @@ class Wic2(WicTestCase):
         self.assertEqual(0, bitbake('core-image-minimal').status)
         self.remove_config(config)
 
-        with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
+        with runqemu('core-image-minimal', ssh=False,
+                     runqemuparams='nographic', image_fstype='wic') as qemu:
             # Check that we have ONLY two /dev/sda* partitions (/boot and /)
             cmd = "grep sda. /proc/partitions | wc -l"
             status, output = qemu.run_serial(cmd)
@@ -1180,7 +1182,7 @@ class Wic2(WicTestCase):
         self.remove_config(config)
 
         with runqemu('core-image-minimal', ssh=False,
-                     runqemuparams='ovmf', image_fstype='wic') as qemu:
+                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
             # Check that /boot has EFI bootx64.efi (required for EFI)
             cmd = "ls /boot/EFI/BOOT/bootx64.efi | wc -l"
             status, output = qemu.run_serial(cmd)
@@ -1418,7 +1420,7 @@ class Wic2(WicTestCase):
             bb.utils.rename(new_image_path, image_path)
 
             # Check if it boots in qemu
-            with runqemu('core-image-minimal', ssh=False) as qemu:
+            with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
                 cmd = "ls /etc/"
                 status, output = qemu.run_serial('true')
                 self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
-- 
2.30.2

Re: [OE-core] [PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL

[Alexander Kanavin]

This fixes one location where the problem can occur in selftests, but
what about all the others?

Generally, it is not selftest's job to ensure qemu can be started: you
need to either tweak runqemu to detect WSL, configure
qemu-system-native from your local.conf so that it doesn't enable sdl
or gtk, or, better yet, fix the problem at the source - find out why
graphical qemu doesn't work in WSL and fix that.

Alex



On Mon, 14 Feb 2022 at 16:46, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> Disable graphic support of qemu to support qemu tests in WSL.
>
> Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> ---
>
> Changes in v2:
> - Add patch
>
>  meta/lib/oeqa/selftest/cases/wic.py | 14 ++++++++------
>  1 file changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> index a021f8d84b..6f3dc27743 100644
> --- a/meta/lib/oeqa/selftest/cases/wic.py
> +++ b/meta/lib/oeqa/selftest/cases/wic.py
> @@ -851,7 +851,7 @@ class Wic2(WicTestCase):
>          self.assertEqual(0, bitbake('wic-image-minimal').status)
>          self.remove_config(config)
>
> -        with runqemu('wic-image-minimal', ssh=False) as qemu:
> +        with runqemu('wic-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
>              cmd = "mount | grep '^/dev/' | cut -f1,3 -d ' ' | egrep -c -e '/dev/sda1 /boot' " \
>                    "-e '/dev/root /|/dev/sda2 /' -e '/dev/sda3 /media' -e '/dev/sda4 /mnt'"
>              status, output = qemu.run_serial(cmd)
> @@ -871,7 +871,7 @@ class Wic2(WicTestCase):
>          self.remove_config(config)
>
>          with runqemu('core-image-minimal', ssh=False,
> -                     runqemuparams='ovmf', image_fstype='wic') as qemu:
> +                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
>              cmd = "grep sda. /proc/partitions  |wc -l"
>              status, output = qemu.run_serial(cmd)
>              self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
> @@ -1059,7 +1059,8 @@ class Wic2(WicTestCase):
>          self.assertEqual(0, bitbake('core-image-minimal-mtdutils').status)
>          self.remove_config(config)
>
> -        with runqemu('core-image-minimal-mtdutils', ssh=False, image_fstype='wic') as qemu:
> +        with runqemu('core-image-minimal-mtdutils', ssh=False,
> +                     runqemuparams='nographic', image_fstype='wic') as qemu:
>              cmd = "grep sda. /proc/partitions  |wc -l"
>              status, output = qemu.run_serial(cmd)
>              self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
> @@ -1119,7 +1120,8 @@ class Wic2(WicTestCase):
>          self.assertEqual(0, bitbake('core-image-minimal').status)
>          self.remove_config(config)
>
> -        with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
> +        with runqemu('core-image-minimal', ssh=False,
> +                     runqemuparams='nographic', image_fstype='wic') as qemu:
>              # Check that we have ONLY two /dev/sda* partitions (/boot and /)
>              cmd = "grep sda. /proc/partitions | wc -l"
>              status, output = qemu.run_serial(cmd)
> @@ -1180,7 +1182,7 @@ class Wic2(WicTestCase):
>          self.remove_config(config)
>
>          with runqemu('core-image-minimal', ssh=False,
> -                     runqemuparams='ovmf', image_fstype='wic') as qemu:
> +                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
>              # Check that /boot has EFI bootx64.efi (required for EFI)
>              cmd = "ls /boot/EFI/BOOT/bootx64.efi | wc -l"
>              status, output = qemu.run_serial(cmd)
> @@ -1418,7 +1420,7 @@ class Wic2(WicTestCase):
>              bb.utils.rename(new_image_path, image_path)
>
>              # Check if it boots in qemu
> -            with runqemu('core-image-minimal', ssh=False) as qemu:
> +            with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
>                  cmd = "ls /etc/"
>                  status, output = qemu.run_serial('true')
>                  self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
> --
> 2.30.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161718): https://lists.openembedded.org/g/openembedded-core/message/161718
> Mute This Topic: https://lists.openembedded.org/mt/89138943/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL

[Stefan Herbrechtsmeier]

Hi Alex,

Am 14.02.2022 um 17:28 schrieb Alexander Kanavin:
> This fixes one location where the problem can occur in selftests, but
> what about all the others?

Other tests like efibootpartition, gcc, glibc or runqemu already set 
nographic.

> Generally, it is not selftest's job to ensure qemu can be started:

Why selftest requires features which it doesn't need?

> you
> need to either tweak runqemu to detect WSL, configure
> qemu-system-native from your local.conf so that it doesn't enable sdl
> or gtk, or, better yet, fix the problem at the source - find out why
> graphical qemu doesn't work in WSL and fix that.

The graphic support depends on the WSL version.

Regards
   Stefan

Re: [OE-core] [PATCH v2 5/5] selftest: wic: Disable graphic of qemu to support WSL

[Alexander Kanavin]

On Mon, 14 Feb 2022 at 18:12, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> Am 14.02.2022 um 17:28 schrieb Alexander Kanavin:
> > This fixes one location where the problem can occur in selftests, but
> > what about all the others?
>
> Other tests like efibootpartition, gcc, glibc or runqemu already set
> nographic.

Yes, some of them do. Others do not, here's a complete list:
[ak@localhost meta]$ grep -ilr "with runqemu" lib/oeqa/selftest/cases
lib/oeqa/selftest/cases/gcc.py
lib/oeqa/selftest/cases/runqemu.py
lib/oeqa/selftest/cases/efibootpartition.py
lib/oeqa/selftest/cases/package.py
lib/oeqa/selftest/cases/wic.py
lib/oeqa/selftest/cases/devtool.py
lib/oeqa/selftest/cases/imagefeatures.py
lib/oeqa/selftest/cases/overlayfs.py
lib/oeqa/selftest/cases/runtime_test.py


> > Generally, it is not selftest's job to ensure qemu can be started:
> Why selftest requires features which it doesn't need?

Selftest does not 'require' anything; it simply uses runqemu()'s
defaults. If the defaults aren't suitable, the right place to correct
them is runqemu() in lib/oeqa/utils/commands.py.

> The graphic support depends on the WSL version.

Yep, so please correct this in the right place, subject to actually
running in WSL version where nographic is required.

Alex