[PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part)

[PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part)

[Ed Bartosh]

Hi,

Increased test coverage of the wic codebase to 81%:
 - tested partitions with various fs types(btrfs, squashfs, etc)
 - removed fsimage plugin
 - removed not used code from filemap.py and runner.py
 - tested globbed source spec of image-bootpart plugin

The following changes since commit 415b72ffcbd26e5f3664370d8b2a9b8105fb6342:

  dnf: remove systemd units in nativesdk builds (2017-03-28 10:34:37 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wip

Ed Bartosh (9):
  wic-tools: add btrfs-tools squashfs-tools to DEPENDS
  oe-selftest: add wic test case test_fs_types
  filemap: remove FilemapSeek class
  oe-selftest: add kickstart_parser test case
  wic: remove fsimage plugin
  wic: use wic-tools STAGING_DATADIR as bootimg_dir
  wic: remove runner.show API
  wic: remove unused code from runner module
  oe-selftest: add test_image_bootpart_globbed test for wic

 meta/lib/oeqa/selftest/wic.py                    |  41 +++
 meta/recipes-core/meta/wic-tools.bb              |   6 +-
 scripts/lib/wic/filemap.py                       | 309 ++++-------------------
 scripts/lib/wic/partition.py                     |  13 -
 scripts/lib/wic/plugins/source/bootimg-pcbios.py |  48 ++--
 scripts/lib/wic/plugins/source/fsimage.py        |  56 ----
 scripts/lib/wic/utils/misc.py                    |  14 +-
 scripts/lib/wic/utils/runner.py                  |  74 +-----
 8 files changed, 130 insertions(+), 431 deletions(-)
 delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py

-- 
2.1.4

[PATCH 1/9] wic-tools: add btrfs-tools squashfs-tools to DEPENDS

[Ed Bartosh]

Added btrfs-tools-native and squashfs-tools-native to DEPENDS as
wic uses these tools to support btrfs and squashfs filesystems.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-core/meta/wic-tools.bb | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index bd4319a..cd494ec 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -2,7 +2,11 @@ SUMMARY = "A meta recipe to build native tools used by wic."
 
 LICENSE = "MIT"
 
-DEPENDS = "parted-native syslinux-native gptfdisk-native dosfstools-native mtools-native bmap-tools-native grub-efi-native cdrtools-native"
+DEPENDS = "\
+           parted-native syslinux-native gptfdisk-native dosfstools-native \
+           mtools-native bmap-tools-native grub-efi-native cdrtools-native \
+           btrfs-tools-native squashfs-tools-native \
+           "
 DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
 DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
 
-- 
2.1.4

[PATCH 2/9] oe-selftest: add wic test case test_fs_types

[Ed Bartosh]

Added wic test case to test all possible filesystem
types for empty and not empty partitions.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index ee633f8..2af3bf5 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -717,3 +717,22 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
             wksname = os.path.splitext(os.path.basename(wks.name))[0]
             out = glob(self.resultdir + "%s-*direct" % wksname)
             self.assertEqual(1, len(out))
+
+    def test_fs_types(self):
+        """Test filesystem types for empty and not empty partitions"""
+        img = 'core-image-minimal'
+        with NamedTemporaryFile("w", suffix=".wks") as wks:
+            wks.writelines(['part ext2   --fstype ext2     --source rootfs\n',
+                            'part btrfs  --fstype btrfs    --source rootfs --size 40M\n',
+                            'part squash --fstype squashfs --source rootfs\n',
+                            'part swap   --fstype swap --size 1M\n',
+                            'part emptyvfat   --fstype vfat   --size 1M\n',
+                            'part emptyext2   --fstype ext2   --size 1M\n',
+                            'part emptybtrfs  --fstype btrfs  --size 100M\n',
+                            'part emptysquash --fstype squash --size 1M\n'])
+            wks.flush()
+            cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
+            self.assertEqual(0, runCmd(cmd).status)
+            wksname = os.path.splitext(os.path.basename(wks.name))[0]
+            out = glob(self.resultdir + "%s-*direct" % wksname)
+            self.assertEqual(1, len(out))
-- 
2.1.4

[PATCH 3/9] filemap: remove FilemapSeek class

[Ed Bartosh]

FIEMAP API was added to Linux kernel 2.6.28 back in 2008
SEEK_HOLE and SEEK_DATA API was added much letter.
As FIEMAP is used by filemap module as a default API it's
safe to remove FileMpSeek class as it's never used.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/filemap.py | 309 +++++++--------------------------------------
 1 file changed, 49 insertions(+), 260 deletions(-)

diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
index 080668e..34523c0 100644
--- a/scripts/lib/wic/filemap.py
+++ b/scripts/lib/wic/filemap.py
@@ -54,24 +54,46 @@ class Error(Exception):
     """A class for all the other exceptions raised by this module."""
     pass
 
+# Below goes the FIEMAP ioctl implementation, which is not very readable
+# because it deals with the rather complex FIEMAP ioctl. To understand the
+# code, you need to know the FIEMAP interface, which is documented in the
+# "Documentation/filesystems/fiemap.txt" file in the Linux kernel sources.
+
+# Format string for 'struct fiemap'
+_FIEMAP_FORMAT = "=QQLLLL"
+# sizeof(struct fiemap)
+_FIEMAP_SIZE = struct.calcsize(_FIEMAP_FORMAT)
+# Format string for 'struct fiemap_extent'
+_FIEMAP_EXTENT_FORMAT = "=QQQQQLLLL"
+# sizeof(struct fiemap_extent)
+_FIEMAP_EXTENT_SIZE = struct.calcsize(_FIEMAP_EXTENT_FORMAT)
+# The FIEMAP ioctl number
+_FIEMAP_IOCTL = 0xC020660B
+# This FIEMAP ioctl flag which instructs the kernel to sync the file before
+# reading the block map
+_FIEMAP_FLAG_SYNC = 0x00000001
+# Size of the buffer for 'struct fiemap_extent' elements which will be used
+# when invoking the FIEMAP ioctl. The larger is the buffer, the less times the
+# FIEMAP ioctl will be invoked.
+_FIEMAP_BUFFER_SIZE = 256 * 1024
 
-class _FilemapBase(object):
+class FilemapFiemap:
     """
-    This is a base class for a couple of other classes in this module. This
-    class simply performs the common parts of the initialization process: opens
-    the image file, gets its size, etc. The 'log' parameter is the logger object
-    to use for printing messages.
+    This class provides API to the FIEMAP ioctl. Namely, it allows to iterate
+    over all mapped blocks and over all holes.
+
+    This class synchronizes the image file every time it invokes the FIEMAP
+    ioctl in order to work-around early FIEMAP implementation kernel bugs.
     """
 
-    def __init__(self, image, log=None):
+    def __init__(self, image):
         """
-        Initialize a class instance. The 'image' argument is full path to the
-        file or file object to operate on.
+        Initialize a class instance. The 'image' argument is full the file
+        object to operate on.
         """
+        self._log = logging.getLogger(__name__)
 
-        self._log = log
-        if self._log is None:
-            self._log = logging.getLogger(__name__)
+        self._log.debug("FilemapFiemap: initializing")
 
         self._f_image_needs_close = False
 
@@ -113,240 +135,6 @@ class _FilemapBase(object):
         self._log.debug("block size %d, blocks count %d, image size %d"
                         % (self.block_size, self.blocks_cnt, self.image_size))
 
-    def __del__(self):
-        """The class destructor which just closes the image file."""
-        if self._f_image_needs_close:
-            self._f_image.close()
-
-    def _open_image_file(self):
-        """Open the image file."""
-        try:
-            self._f_image = open(self._image_path, 'rb')
-        except IOError as err:
-            raise Error("cannot open image file '%s': %s"
-                        % (self._image_path, err))
-
-        self._f_image_needs_close = True
-
-    def block_is_mapped(self, block): # pylint: disable=W0613,R0201
-        """
-        This method has has to be implemented by child classes. It returns
-        'True' if block number 'block' of the image file is mapped and 'False'
-        otherwise.
-        """
-
-        raise Error("the method is not implemented")
-
-    def block_is_unmapped(self, block): # pylint: disable=W0613,R0201
-        """
-        This method has has to be implemented by child classes. It returns
-        'True' if block number 'block' of the image file is not mapped (hole)
-        and 'False' otherwise.
-        """
-
-        raise Error("the method is not implemented")
-
-    def get_mapped_ranges(self, start, count): # pylint: disable=W0613,R0201
-        """
-        This method has has to be implemented by child classes. This is a
-        generator which yields ranges of mapped blocks in the file. The ranges
-        are tuples of 2 elements: [first, last], where 'first' is the first
-        mapped block and 'last' is the last mapped block.
-
-        The ranges are yielded for the area of the file of size 'count' blocks,
-        starting from block 'start'.
-        """
-
-        raise Error("the method is not implemented")
-
-    def get_unmapped_ranges(self, start, count): # pylint: disable=W0613,R0201
-        """
-        This method has has to be implemented by child classes. Just like
-        'get_mapped_ranges()', but yields unmapped block ranges instead
-        (holes).
-        """
-
-        raise Error("the method is not implemented")
-
-
-# The 'SEEK_HOLE' and 'SEEK_DATA' options of the file seek system call
-_SEEK_DATA = 3
-_SEEK_HOLE = 4
-
-def _lseek(file_obj, offset, whence):
-    """This is a helper function which invokes 'os.lseek' for file object
-    'file_obj' and with specified 'offset' and 'whence'. The 'whence'
-    argument is supposed to be either '_SEEK_DATA' or '_SEEK_HOLE'. When
-    there is no more data or hole starting from 'offset', this function
-    returns '-1'.  Otherwise the data or hole position is returned."""
-
-    try:
-        return os.lseek(file_obj.fileno(), offset, whence)
-    except OSError as err:
-        # The 'lseek' system call returns the ENXIO if there is no data or
-        # hole starting from the specified offset.
-        if err.errno == os.errno.ENXIO:
-            return -1
-        elif err.errno == os.errno.EINVAL:
-            raise ErrorNotSupp("the kernel or file-system does not support "
-                               "\"SEEK_HOLE\" and \"SEEK_DATA\"")
-        else:
-            raise
-
-class FilemapSeek(_FilemapBase):
-    """
-    This class uses the 'SEEK_HOLE' and 'SEEK_DATA' to find file block mapping.
-    Unfortunately, the current implementation requires the caller to have write
-    access to the image file.
-    """
-
-    def __init__(self, image, log=None):
-        """Refer the '_FilemapBase' class for the documentation."""
-
-        # Call the base class constructor first
-        _FilemapBase.__init__(self, image, log)
-        self._log.debug("FilemapSeek: initializing")
-
-        self._probe_seek_hole()
-
-    def _probe_seek_hole(self):
-        """
-        Check whether the system implements 'SEEK_HOLE' and 'SEEK_DATA'.
-        Unfortunately, there seems to be no clean way for detecting this,
-        because often the system just fakes them by just assuming that all
-        files are fully mapped, so 'SEEK_HOLE' always returns EOF and
-        'SEEK_DATA' always returns the requested offset.
-
-        I could not invent a better way of detecting the fake 'SEEK_HOLE'
-        implementation than just to create a temporary file in the same
-        directory where the image file resides. It would be nice to change this
-        to something better.
-        """
-
-        directory = os.path.dirname(self._image_path)
-
-        try:
-            tmp_obj = tempfile.TemporaryFile("w+", dir=directory)
-        except IOError as err:
-            raise ErrorNotSupp("cannot create a temporary in \"%s\": %s"
-                              % (directory, err))
-
-        try:
-            os.ftruncate(tmp_obj.fileno(), self.block_size)
-        except OSError as err:
-            raise ErrorNotSupp("cannot truncate temporary file in \"%s\": %s"
-                               % (directory, err))
-
-        offs = _lseek(tmp_obj, 0, _SEEK_HOLE)
-        if offs != 0:
-            # We are dealing with the stub 'SEEK_HOLE' implementation which
-            # always returns EOF.
-            self._log.debug("lseek(0, SEEK_HOLE) returned %d" % offs)
-            raise ErrorNotSupp("the file-system does not support "
-                               "\"SEEK_HOLE\" and \"SEEK_DATA\" but only "
-                               "provides a stub implementation")
-
-        tmp_obj.close()
-
-    def block_is_mapped(self, block):
-        """Refer the '_FilemapBase' class for the documentation."""
-        offs = _lseek(self._f_image, block * self.block_size, _SEEK_DATA)
-        if offs == -1:
-            result = False
-        else:
-            result = (offs // self.block_size == block)
-
-        self._log.debug("FilemapSeek: block_is_mapped(%d) returns %s"
-                        % (block, result))
-        return result
-
-    def block_is_unmapped(self, block):
-        """Refer the '_FilemapBase' class for the documentation."""
-        return not self.block_is_mapped(block)
-
-    def _get_ranges(self, start, count, whence1, whence2):
-        """
-        This function implements 'get_mapped_ranges()' and
-        'get_unmapped_ranges()' depending on what is passed in the 'whence1'
-        and 'whence2' arguments.
-        """
-
-        assert whence1 != whence2
-        end = start * self.block_size
-        limit = end + count * self.block_size
-
-        while True:
-            start = _lseek(self._f_image, end, whence1)
-            if start == -1 or start >= limit or start == self.image_size:
-                break
-
-            end = _lseek(self._f_image, start, whence2)
-            if end == -1 or end == self.image_size:
-                end = self.blocks_cnt * self.block_size
-            if end > limit:
-                end = limit
-
-            start_blk = start // self.block_size
-            end_blk = end // self.block_size - 1
-            self._log.debug("FilemapSeek: yielding range (%d, %d)"
-                            % (start_blk, end_blk))
-            yield (start_blk, end_blk)
-
-    def get_mapped_ranges(self, start, count):
-        """Refer the '_FilemapBase' class for the documentation."""
-        self._log.debug("FilemapSeek: get_mapped_ranges(%d,  %d(%d))"
-                        % (start, count, start + count - 1))
-        return self._get_ranges(start, count, _SEEK_DATA, _SEEK_HOLE)
-
-    def get_unmapped_ranges(self, start, count):
-        """Refer the '_FilemapBase' class for the documentation."""
-        self._log.debug("FilemapSeek: get_unmapped_ranges(%d,  %d(%d))"
-                        % (start, count, start + count - 1))
-        return self._get_ranges(start, count, _SEEK_HOLE, _SEEK_DATA)
-
-
-# Below goes the FIEMAP ioctl implementation, which is not very readable
-# because it deals with the rather complex FIEMAP ioctl. To understand the
-# code, you need to know the FIEMAP interface, which is documented in the
-# "Documentation/filesystems/fiemap.txt" file in the Linux kernel sources.
-
-# Format string for 'struct fiemap'
-_FIEMAP_FORMAT = "=QQLLLL"
-# sizeof(struct fiemap)
-_FIEMAP_SIZE = struct.calcsize(_FIEMAP_FORMAT)
-# Format string for 'struct fiemap_extent'
-_FIEMAP_EXTENT_FORMAT = "=QQQQQLLLL"
-# sizeof(struct fiemap_extent)
-_FIEMAP_EXTENT_SIZE = struct.calcsize(_FIEMAP_EXTENT_FORMAT)
-# The FIEMAP ioctl number
-_FIEMAP_IOCTL = 0xC020660B
-# This FIEMAP ioctl flag which instructs the kernel to sync the file before
-# reading the block map
-_FIEMAP_FLAG_SYNC = 0x00000001
-# Size of the buffer for 'struct fiemap_extent' elements which will be used
-# when invoking the FIEMAP ioctl. The larger is the buffer, the less times the
-# FIEMAP ioctl will be invoked.
-_FIEMAP_BUFFER_SIZE = 256 * 1024
-
-class FilemapFiemap(_FilemapBase):
-    """
-    This class provides API to the FIEMAP ioctl. Namely, it allows to iterate
-    over all mapped blocks and over all holes.
-
-    This class synchronizes the image file every time it invokes the FIEMAP
-    ioctl in order to work-around early FIEMAP implementation kernel bugs.
-    """
-
-    def __init__(self, image, log=None):
-        """
-        Initialize a class instance. The 'image' argument is full the file
-        object to operate on.
-        """
-
-        # Call the base class constructor first
-        _FilemapBase.__init__(self, image, log)
-        self._log.debug("FilemapFiemap: initializing")
-
         self._buf_size = _FIEMAP_BUFFER_SIZE
 
         # Calculate how many 'struct fiemap_extent' elements fit the buffer
@@ -362,6 +150,21 @@ class FilemapFiemap(_FilemapBase):
         # Check if the FIEMAP ioctl is supported
         self.block_is_mapped(0)
 
+    def __del__(self):
+        """The class destructor which just closes the image file."""
+        if self._f_image_needs_close:
+            self._f_image.close()
+
+    def _open_image_file(self):
+        """Open the image file."""
+        try:
+            self._f_image = open(self._image_path, 'rb')
+        except IOError as err:
+            raise Error("cannot open image file '%s': %s"
+                        % (self._image_path, err))
+
+        self._f_image_needs_close = True
+
     def _invoke_fiemap(self, block, count):
         """
         Invoke the FIEMAP ioctl for 'count' blocks of the file starting from
@@ -515,24 +318,10 @@ class FilemapFiemap(_FilemapBase):
                             % (hole_first, start + count - 1))
             yield (hole_first, start + count - 1)
 
-def filemap(image, log=None):
-    """
-    Create and return an instance of a Filemap class - 'FilemapFiemap' or
-    'FilemapSeek', depending on what the system we run on supports. If the
-    FIEMAP ioctl is supported, an instance of the 'FilemapFiemap' class is
-    returned. Otherwise, if 'SEEK_HOLE' is supported an instance of the
-    'FilemapSeek' class is returned. If none of these are supported, the
-    function generates an 'Error' type exception.
-    """
-
-    try:
-        return FilemapFiemap(image, log)
-    except ErrorNotSupp:
-        return FilemapSeek(image, log)
 
 def sparse_copy(src_fname, dst_fname, offset=0, skip=0):
     """Efficiently copy sparse file to or into another file."""
-    fmap = filemap(src_fname)
+    fmap = FilemapFiemap(src_fname)
     try:
         dst_file = open(dst_fname, 'r+b')
     except IOError:
-- 
2.1.4

[PATCH 4/9] oe-selftest: add kickstart_parser test case

[Ed Bartosh]

Added test_kickstart_parser test case to test wks parser
options not yet covered by tests.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 2af3bf5..f0d9dd9 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -736,3 +736,15 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
             wksname = os.path.splitext(os.path.basename(wks.name))[0]
             out = glob(self.resultdir + "%s-*direct" % wksname)
             self.assertEqual(1, len(out))
+
+    def test_kickstart_parser(self):
+        """Test wks parser options"""
+        with NamedTemporaryFile("w", suffix=".wks") as wks:
+            wks.writelines(['part / --fstype ext3 --source rootfs --system-id 0xFF '\
+                            '--overhead-factor 1.2 --size 100k\n'])
+            wks.flush()
+            cmd = "wic create %s -e core-image-minimal -o %s" % (wks.name, self.resultdir)
+            self.assertEqual(0, runCmd(cmd).status)
+            wksname = os.path.splitext(os.path.basename(wks.name))[0]
+            out = glob(self.resultdir + "%s-*direct" % wksname)
+            self.assertEqual(1, len(out))
-- 
2.1.4

[PATCH 5/9] wic: remove fsimage plugin

[Ed Bartosh]

Removed fsimage plugin and prepare_rootfs_from_fs_image API as
they duplicate functionality of rawcopy plugin. Fsimage plugin makes
wic to remove the image artifact from deployment directory, which
can cause problems too.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/partition.py              | 13 -------
 scripts/lib/wic/plugins/source/fsimage.py | 56 -------------------------------
 2 files changed, 69 deletions(-)
 delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 8e32afc..647a6fb 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -193,19 +193,6 @@ class Partition():
                            "larger (%d kB) than its allowed size %d kB" %
                            (self.mountpoint, self.size, self.fixed_size))
 
-    def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
-                                     rootfs_dir):
-        """
-        Handle an already-created partition e.g. xxx.ext3
-        """
-        rootfs = oe_builddir
-        du_cmd = "du -Lbks %s" % rootfs
-        out = exec_cmd(du_cmd)
-        rootfs_size = out.split()[0]
-
-        self.size = int(rootfs_size)
-        self.source_file = rootfs
-
     def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
                        native_sysroot):
         """
diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py
deleted file mode 100644
index f781499..0000000
--- a/scripts/lib/wic/plugins/source/fsimage.py
+++ /dev/null
@@ -1,56 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-
-import logging
-import os
-
-from wic import WicError
-from wic.pluginbase import SourcePlugin
-from wic.utils.misc import get_bitbake_var
-
-logger = logging.getLogger('wic')
-
-class FSImagePlugin(SourcePlugin):
-    """
-    Add an already existing filesystem image to the partition layout.
-    """
-
-    name = 'fsimage'
-
-    @classmethod
-    def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
-                             oe_builddir, bootimg_dir, kernel_dir,
-                             rootfs_dir, native_sysroot):
-        """
-        Called to do the actual content population for a partition i.e. it
-        'prepares' the partition to be incorporated into the image.
-        """
-        if not bootimg_dir:
-            bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
-            if not bootimg_dir:
-                raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
-
-        logger.debug('Bootimg dir: %s', bootimg_dir)
-
-        if 'file' not in source_params:
-            raise WicError("No file specified")
-
-        src = os.path.join(bootimg_dir, source_params['file'])
-
-
-        logger.debug('Preparing partition using image %s', src)
-        part.prepare_rootfs_from_fs_image(cr_workdir, src, "")
-- 
2.1.4

[PATCH 6/9] wic: use wic-tools STAGING_DATADIR as bootimg_dir

[Ed Bartosh]

If bootloader artifacts are not found in default bootimg_dir
use wic-tools sysroot for the same purpose. This should
prevent wic from failing if bootloader artifacts can't be
found in image native sysroot.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/plugins/source/bootimg-pcbios.py | 42 +++++++++++-------------
 1 file changed, 20 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 11db304..bfb8ff5 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -44,19 +44,16 @@ class BootimgPcbiosPlugin(SourcePlugin):
     name = 'bootimg-pcbios'
 
     @classmethod
-    def _get_syslinux_dir(cls, bootimg_dir):
+    def _get_bootimg_dir(cls, bootimg_dir, dirname):
         """
-        Get path to syslinux from either default bootimg_dir
-        or wic-tools STAGING_DIR.
+        Check if dirname exists in default bootimg_dir or
+        in wic-tools STAGING_DIR.
         """
-        for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
-            if not path:
-                continue
-            syslinux_dir = os.path.join(path, 'syslinux')
-            if os.path.exists(syslinux_dir):
-                return syslinux_dir
+        for result in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
+            if os.path.exists("%s/%s" % (result, dirname)):
+                return result
 
-        raise WicError("Couldn't find syslinux directory, exiting")
+        raise WicError("Couldn't find correct bootimg_dir, exiting")
 
     @classmethod
     def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
@@ -65,11 +62,12 @@ class BootimgPcbiosPlugin(SourcePlugin):
         Called after all partitions have been prepared and assembled into a
         disk image.  In this case, we install the MBR.
         """
-        syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
+        bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
+        mbrfile = "%s/syslinux/" % bootimg_dir
         if creator.ptable_format == 'msdos':
-            mbrfile = os.path.join(syslinux_dir, "mbr.bin")
+            mbrfile += "mbr.bin"
         elif creator.ptable_format == 'gpt':
-            mbrfile = os.path.join(syslinux_dir, "gptmbr.bin")
+            mbrfile += "gptmbr.bin"
         else:
             raise WicError("Unsupported partition table: %s" %
                            creator.ptable_format)
@@ -155,7 +153,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
         'prepares' the partition to be incorporated into the image.
         In this case, prepare content for legacy bios boot partition.
         """
-        syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
+        bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
 
         staging_kernel_dir = kernel_dir
 
@@ -163,14 +161,14 @@ class BootimgPcbiosPlugin(SourcePlugin):
 
         cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" %
                 (staging_kernel_dir, hdddir),
-                "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" %
-                (syslinux_dir, hdddir),
-                "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" %
-                (syslinux_dir, hdddir),
-                "install -m 444 %s/libcom32.c32 %s/libcom32.c32" %
-                (syslinux_dir, hdddir),
-                "install -m 444 %s/libutil.c32 %s/libutil.c32" %
-                (syslinux_dir, hdddir))
+                "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" %
+                (bootimg_dir, hdddir),
+                "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" %
+                (bootimg_dir, hdddir),
+                "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" %
+                (bootimg_dir, hdddir),
+                "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
+                (bootimg_dir, hdddir))
 
         for install_cmd in cmds:
             exec_cmd(install_cmd)
-- 
2.1.4

[PATCH 7/9] wic: remove runner.show API

[Ed Bartosh]

Replaced runner.show call to exec_cmd call in bootimg-pcbios
plugin. Removed runner.show API as it's not used anywhere else.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/plugins/source/bootimg-pcbios.py |  6 ++----
 scripts/lib/wic/utils/runner.py                  | 24 ------------------------
 2 files changed, 2 insertions(+), 28 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index bfb8ff5..5890c12 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -81,10 +81,8 @@ class BootimgPcbiosPlugin(SourcePlugin):
         logger.debug("Installing MBR on disk %s as %s with size %s bytes",
                      disk_name, full_path, disk.min_size)
 
-        rcode = runner.show(['dd', 'if=%s' % mbrfile,
-                             'of=%s' % full_path, 'conv=notrunc'])
-        if rcode != 0:
-            raise WicError("Unable to set MBR to %s" % full_path)
+        dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
+        exec_cmd(dd_cmd, native_sysroot)
 
     @classmethod
     def do_configure_partition(cls, part, source_params, creator, cr_workdir,
diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
index 56d7ea3..348557a 100644
--- a/scripts/lib/wic/utils/runner.py
+++ b/scripts/lib/wic/utils/runner.py
@@ -82,30 +82,6 @@ def runtool(cmdln_or_args, catch=1):
 
     return (process.returncode, out)
 
-def show(cmdln_or_args):
-    """Show all messages using logger.debug."""
-
-    rcode, out = runtool(cmdln_or_args, catch=3)
-
-    if isinstance(cmdln_or_args, list):
-        cmd = ' '.join(cmdln_or_args)
-    else:
-        cmd = cmdln_or_args
-
-    msg = 'running command: "%s"' % cmd
-    if out:
-        out = out.strip()
-    if out:
-        msg += ', with output::'
-        msg += '\n  +----------------'
-        for line in out.splitlines():
-            msg += '\n  | %s' % line
-        msg += '\n  +----------------'
-
-    logger.debug(msg)
-
-    return rcode
-
 def outs(cmdln_or_args, catch=1):
     # get the outputs of tools
     return runtool(cmdln_or_args, catch)[1].strip()
-- 
2.1.4

[PATCH 8/9] wic: remove unused code from runner module

[Ed Bartosh]

Removed unused APIs 'outs' and 'quiet'.
Removed 'catch' parameter from runner.runtool API as wic
uses only one value of it. Removed the code that handles
unused values of 'catch' parameter.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/misc.py   | 14 ++++++------
 scripts/lib/wic/utils/runner.py | 50 +++++------------------------------------
 2 files changed, 13 insertions(+), 51 deletions(-)

diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
index c941112..307779a 100644
--- a/scripts/lib/wic/utils/misc.py
+++ b/scripts/lib/wic/utils/misc.py
@@ -59,7 +59,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
                   "syslinux": "syslinux"
                  }
 
-def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def _exec_cmd(cmd_and_args, as_shell=False):
     """
     Execute command, catching stderr, stdout
 
@@ -70,9 +70,9 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
     logger.debug(args)
 
     if as_shell:
-        ret, out = runner.runtool(cmd_and_args, catch)
+        ret, out = runner.runtool(cmd_and_args)
     else:
-        ret, out = runner.runtool(args, catch)
+        ret, out = runner.runtool(args)
     out = out.strip()
     if ret != 0:
         raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
@@ -84,14 +84,14 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
     return ret, out
 
 
-def exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def exec_cmd(cmd_and_args, as_shell=False):
     """
     Execute command, return output
     """
-    return _exec_cmd(cmd_and_args, as_shell, catch)[1]
+    return _exec_cmd(cmd_and_args, as_shell)[1]
 
 
-def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
+def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
     """
     Execute native command, catching stderr, stdout
 
@@ -118,7 +118,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
 
     # If the command isn't in the native sysroot say we failed.
     if spawn.find_executable(args[0], native_paths):
-        ret, out = _exec_cmd(native_cmd_and_args, True, catch)
+        ret, out = _exec_cmd(native_cmd_and_args, True)
     else:
         ret = 127
         out = "can't find native executable %s in %s" % (args[0], native_paths)
diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
index 348557a..4aa00fb 100644
--- a/scripts/lib/wic/utils/runner.py
+++ b/scripts/lib/wic/utils/runner.py
@@ -14,32 +14,17 @@
 # You should have received a copy of the GNU General Public License along
 # with this program; if not, write to the Free Software Foundation, Inc., 59
 # Temple Place - Suite 330, Boston, MA 02111-1307, USA.
-
-import logging
-import os
 import subprocess
 
 from wic import WicError
 
-logger = logging.getLogger('wic')
-
-def runtool(cmdln_or_args, catch=1):
+def runtool(cmdln_or_args):
     """ wrapper for most of the subprocess calls
     input:
         cmdln_or_args: can be both args and cmdln str (shell=True)
-        catch: 0, quitely run
-               1, only STDOUT
-               2, only STDERR
-               3, both STDOUT and STDERR
     return:
-        (rc, output)
-        if catch==0: the output will always None
+        rc, output
     """
-
-    if catch not in (0, 1, 2, 3):
-        # invalid catch selection, will cause exception, that's good
-        return None
-
     if isinstance(cmdln_or_args, list):
         cmd = cmdln_or_args[0]
         shell = False
@@ -48,26 +33,13 @@ def runtool(cmdln_or_args, catch=1):
         cmd = shlex.split(cmdln_or_args)[0]
         shell = True
 
-    if catch != 3:
-        dev_null = os.open("/dev/null", os.O_WRONLY)
-
-    if catch == 0:
-        sout = dev_null
-        serr = dev_null
-    elif catch == 1:
-        sout = subprocess.PIPE
-        serr = dev_null
-    elif catch == 2:
-        sout = dev_null
-        serr = subprocess.PIPE
-    elif catch == 3:
-        sout = subprocess.PIPE
-        serr = subprocess.STDOUT
+    sout = subprocess.PIPE
+    serr = subprocess.STDOUT
 
     try:
         process = subprocess.Popen(cmdln_or_args, stdout=sout,
                                    stderr=serr, shell=shell)
-        (sout, serr) = process.communicate()
+        sout, serr = process.communicate()
         # combine stdout and stderr, filter None out and decode
         out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
     except OSError as err:
@@ -76,15 +48,5 @@ def runtool(cmdln_or_args, catch=1):
             raise WicError('Cannot run command: %s, lost dependency?' % cmd)
         else:
             raise # relay
-    finally:
-        if catch != 3:
-            os.close(dev_null)
-
-    return (process.returncode, out)
-
-def outs(cmdln_or_args, catch=1):
-    # get the outputs of tools
-    return runtool(cmdln_or_args, catch)[1].strip()
 
-def quiet(cmdln_or_args):
-    return runtool(cmdln_or_args, catch=0)[0]
+    return process.returncode, out
-- 
2.1.4

[PATCH 9/9] oe-selftest: add test_image_bootpart_globbed test for wic

[Ed Bartosh]

Test image-bootpart wic plugin with globbed value of
IMAGE_BOOT_FILES variable to increase test coverage.

[YOCTO #10618]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index f0d9dd9..2401aaa 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -748,3 +748,13 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
             wksname = os.path.splitext(os.path.basename(wks.name))[0]
             out = glob(self.resultdir + "%s-*direct" % wksname)
             self.assertEqual(1, len(out))
+
+    def test_image_bootpart_globbed(self):
+        """Test globbed sources with image-bootpart plugin"""
+        img = "core-image-minimal"
+        cmd = "wic create sdimage-bootpart -e %s -o %s" % (img, self.resultdir)
+        config = 'IMAGE_BOOT_FILES = "%s*"' % get_bb_var('KERNEL_IMAGETYPE', img)
+        self.append_config(config)
+        self.assertEqual(0, runCmd(cmd).status)
+        self.remove_config(config)
+        self.assertEqual(1, len(glob(self.resultdir + "sdimage-bootpart-*direct")))
-- 
2.1.4