[PATCH 00/10] #11278 - wic: create an option to use entire disk

[PATCH 00/10] #11278 - wic: create an option to use entire disk

[Ed Bartosh]

Hi,

This patchset implements wic write [--expand auto|<partnum>:<size>]
subcommand to write partitioned image to the target media or file with the optional
possibility to expand partitions to either fill the target media(--expand auto)
or using specified partition sizes (--expand <partnum>:<size>)

'wic write' uses bmaptool API, so it's much faster than dd.
It doesn't require root privileges, just write permissions for the target file or
device.

'wic write' can also be used to flash unpartitioned images.

Please, review.

The following changes since commit a087e0bc765ade6386720f22d842e2fc0bd5f128:

  maintainers.inc: assign newly added recipes (2017-08-24 16:49:57 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip

Ed Bartosh (10):
  wic: add help and usage content for 'wic write'
  wic: add 'wic write' command
  wic: reimplement getting paths of used tools
  wic: get more info from the 'parted print' output
  wic: convert partition number to int
  wic: added 'fstypes' parameter to Disk.__init__
  wic: extend list of used tools
  wic: implement 'wic write' command
  wic: always read image partitions
  wic: setlftest: test expanding MBR image

 meta/lib/oeqa/selftest/cases/wic.py |  53 +++++++++
 meta/recipes-core/meta/wic-tools.bb |   3 +-
 scripts/lib/wic/engine.py           | 210 ++++++++++++++++++++++++++++++------
 scripts/lib/wic/help.py             |  40 +++++++
 scripts/wic                         |  55 ++++++++++
 5 files changed, 326 insertions(+), 35 deletions(-)

--
Regards,
Ed

[PATCH 01/10] wic: add help and usage content for 'wic write'

[Ed Bartosh]

Added wic_write_help and wic_write_usage variables to the help.py module.
These variables contain help content that will be used in 'wic write help'
and 'wic write --help' output.

[YOCTO #11278]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/help.py | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 99912cd..ccd3382 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -468,6 +468,46 @@ DESCRIPTION
     containing the tools(parted and mtools) to use.
 """
 
+wic_write_usage = """
+
+ Write image to a device
+
+ usage: wic write <image> <target device> [--expand [rules]] [--native-sysroot <path>]
+
+ This command writes wic image to a target device (USB stick, SD card etc).
+
+ See 'wic help write' for more detailed instructions.
+
+"""
+
+wic_write_help = """
+
+NAME
+    wic write - write wic image to a device
+
+SYNOPSIS
+    wic write <image> <target>
+    wic write <image> <target> --expand auto
+    wic write <image> <target> --expand 1:100M-2:300M
+    wic write <image> <target> --native-sysroot <path>
+
+DESCRIPTION
+    This command writes wic image to a target device (USB stick, SD card etc)
+
+        $ wic write ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic /dev/sdb
+
+    The --expand option is used to resize image partitions.
+    --expand auto expands partitions to occupy all free space available on the target device.
+    It's also possible to specify expansion rules in a format
+    <partition>:<size>[-<partition>:<size>...] for one or more partitions.
+    Specifying size 0 will keep partition unmodified.
+    Note: Resizing boot partition can result in non-bootable image for non-EFI images. It is
+    recommended to use size 0 for boot partition to keep image bootable.
+
+    The --native-sysroot option is used to specify the path to the native sysroot
+    containing the tools(parted, resize2fs) to use.
+"""
+
 wic_plugins_help = """
 
 NAME
-- 
2.1.4

[PATCH 02/10] wic: add 'wic write' command

[Ed Bartosh]

Added empty 'wic write' command that does nothing.
The functionality will be added by the next commits.

[YOCTO #11278]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py |  6 ++++++
 scripts/wic               | 55 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 2dc2fd5..d33c9b3 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -375,6 +375,12 @@ def wic_rm(args, native_sysroot):
     disk = Disk(args.path.image, native_sysroot)
     disk.remove(args.path.part, args.path.path)
 
+def wic_write(args, native_sysroot):
+    """
+    Write image to a target device.
+    """
+    pass
+
 def find_canned(scripts_path, file_name):
     """
     Find a file either by its path or by name in the canned files dir.
diff --git a/scripts/wic b/scripts/wic
index 02bc82c..592a0e4 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -257,6 +257,13 @@ def wic_rm_subcommand(args, usage_str):
     """
     engine.wic_rm(args, args.native_sysroot)
 
+def wic_write_subcommand(args, usage_str):
+    """
+    Command-line handling for writing images.
+    The real work is done by engine.wic_write()
+    """
+    engine.wic_write(args, args.native_sysroot)
+
 def wic_help_subcommand(args, usage_str):
     """
     Command-line handling for help subcommand to keep the current
@@ -298,6 +305,9 @@ helptopics = {
     "rm":        [wic_help_topic_subcommand,
                   wic_help_topic_usage,
                   hlp.wic_rm_help],
+    "write":     [wic_help_topic_subcommand,
+                  wic_help_topic_usage,
+                  hlp.wic_write_help],
     "list":      [wic_help_topic_subcommand,
                   wic_help_topic_usage,
                   hlp.wic_list_help]
@@ -397,6 +407,47 @@ def wic_init_parser_rm(subparser):
     subparser.add_argument("-n", "--native-sysroot",
                         help="path to the native sysroot containing the tools")
 
+def expandtype(rules):
+    """
+    Custom type for ArgumentParser
+    Converts expand rules to the dictionary {<partition>: size}
+    """
+    if rules == 'auto':
+        return {}
+    result = {}
+    for rule in rules.split('-'):
+        try:
+            part, size = rule.split(':')
+        except ValueError:
+            raise argparse.ArgumentTypeError("Incorrect rule format: %s" % rule)
+
+        if not part.isdigit():
+            raise argparse.ArgumentTypeError("Rule '%s': partition number must be integer" % rule)
+
+        # validate size
+        multiplier = 1
+        for suffix, mult in [('K', 1024), ('M', 1024 * 1024), ('G', 1024 * 1024 * 1024)]:
+            if size.upper().endswith(suffix):
+                multiplier = mult
+                size = size[:-1]
+                break
+        if not size.isdigit():
+            raise argparse.ArgumentTypeError("Rule '%s': size must be integer" % rule)
+
+        result[int(part)] = int(size) * multiplier
+
+    return result
+
+def wic_init_parser_write(subparser):
+    subparser.add_argument("image",
+                        help="path to the wic image")
+    subparser.add_argument("target",
+                        help="target file or device")
+    subparser.add_argument("-e", "--expand", type=expandtype,
+                        help="expand rules: auto or <partition>:<size>[,<partition>:<size>]")
+    subparser.add_argument("-n", "--native-sysroot",
+                        help="path to the native sysroot containing the tools")
+
 def wic_init_parser_help(subparser):
     helpparsers = subparser.add_subparsers(dest='help_topic', help=hlp.wic_usage)
     for helptopic in helptopics:
@@ -425,6 +476,10 @@ subcommands = {
                   hlp.wic_rm_usage,
                   hlp.wic_rm_help,
                   wic_init_parser_rm],
+    "write":     [wic_write_subcommand,
+                  hlp.wic_write_usage,
+                  hlp.wic_write_help,
+                  wic_init_parser_write],
     "help":      [wic_help_subcommand,
                   wic_help_topic_usage,
                   hlp.wic_help_help,
-- 
2.1.4

[PATCH 03/10] wic: reimplement getting paths of used tools

[Ed Bartosh]

So far every used tool have to have separate property and
private attribute in the Disk class. This is too verbose,
considering that there will be much more tools used.

Reimplemented getting tools paths using custom __getattr__
method. This is much more compact and readable.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 36 +++++++++---------------------------
 1 file changed, 9 insertions(+), 27 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index d33c9b3..a030ca2 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -236,10 +236,6 @@ class Disk:
         self.imagepath = imagepath
         self.native_sysroot = native_sysroot
         self._partitions = None
-        self._mdir = None
-        self._mcopy = None
-        self._mdel = None
-        self._mdeltree = None
         self._partimages = {}
 
         # find parted
@@ -270,30 +266,16 @@ class Disk:
 
         return self._partitions
 
-    def _prop(self, name):
+    def __getattr__(self, name):
         """Get path to the executable in a lazy way."""
-        aname = "_%s" % name
-        if getattr(self, aname) is None:
-            setattr(self, aname, find_executable(name, self.paths))
-            if not getattr(self, aname):
-                raise WicError("Can't find executable {}".format(name))
-        return getattr(self, aname)
-
-    @property
-    def mdir(self):
-        return self._prop('mdir')
-
-    @property
-    def mcopy(self):
-        return self._prop("mcopy")
-
-    @property
-    def mdel(self):
-        return self._prop("mdel")
-
-    @property
-    def mdeltree(self):
-        return self._prop("mdeltree")
+        if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+            aname = "_%s" % name
+            if aname not in self.__dict__:
+                setattr(self, aname, find_executable(name, self.paths))
+                if aname not in self.__dict__:
+                    raise WicError("Can't find executable {}".format(name))
+            return self.__dict__[aname]
+        return self.__dict__[name]
 
     def _get_part_image(self, pnum):
         if pnum not in self.partitions:
-- 
2.1.4

[PATCH 04/10] wic: get more info from the 'parted print' output

[Ed Bartosh]

Got partition type and sector sizes from the output
of 'parted print'. This info may be used in the implementation
of 'wic write' command.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index a030ca2..b8fd3ae 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -237,6 +237,9 @@ class Disk:
         self.native_sysroot = native_sysroot
         self._partitions = None
         self._partimages = {}
+        self._lsector_size = None
+        self._psector_size = None
+        self._ptable_format = None
 
         # find parted
         self.paths = "/bin:/usr/bin:/usr/sbin:/sbin/"
@@ -258,7 +261,11 @@ class Disk:
             self._partitions = OrderedDict()
             out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath))
             parttype = namedtuple("Part", "pnum start end size fstype")
-            for line in out.splitlines()[2:]:
+            splitted = out.splitlines()
+            lsector_size, psector_size, self._ptable_format = splitted[1].split(":")[3:6]
+            self._lsector_size = int(lsector_size)
+            self._psector_size = int(psector_size)
+            for line in splitted[2:]:
                 pnum, start, end, size, fstype = line.split(':')[:5]
                 partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
                                      int(size[:-1]), fstype)
-- 
2.1.4

[PATCH 05/10] wic: convert partition number to int

[Ed Bartosh]

Converted partition number to int in order to use
it as an index in the list of partitions.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index b8fd3ae..83e633d 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -267,7 +267,7 @@ class Disk:
             self._psector_size = int(psector_size)
             for line in splitted[2:]:
                 pnum, start, end, size, fstype = line.split(':')[:5]
-                partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
+                partition = parttype(int(pnum), int(start[:-1]), int(end[:-1]),
                                      int(size[:-1]), fstype)
                 self._partitions[pnum] = partition
 
@@ -341,7 +341,7 @@ def wic_ls(args, native_sysroot):
         if disk.partitions:
             print('Num     Start        End          Size      Fstype')
             for part in disk.partitions.values():
-                print("{:2s}  {:12d} {:12d} {:12d}  {}".format(\
+                print("{:2d}  {:12d} {:12d} {:12d}  {}".format(\
                           part.pnum, part.start, part.end,
                           part.size, part.fstype))
     else:
-- 
2.1.4

[PATCH 06/10] wic: added 'fstypes' parameter to Disk.__init__

[Ed Bartosh]

This parameter specifies list of supported filesystems.
So far only 'fat' is supported, but 'wic write' is going
to support at least 'fat', 'ext' and 'swap'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 83e633d..a6ea362 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -232,9 +232,10 @@ def wic_list(args, scripts_path):
 
 
 class Disk:
-    def __init__(self, imagepath, native_sysroot):
+    def __init__(self, imagepath, native_sysroot, fstypes=('fat',)):
         self.imagepath = imagepath
         self.native_sysroot = native_sysroot
+        self.fstypes = fstypes
         self._partitions = None
         self._partimages = {}
         self._lsector_size = None
@@ -288,7 +289,11 @@ class Disk:
         if pnum not in self.partitions:
             raise WicError("Partition %s is not in the image")
         part = self.partitions[pnum]
-        if not part.fstype.startswith("fat"):
+        # check if fstype is supported
+        for fstype in self.fstypes:
+            if part.fstype.startswith(fstype):
+                break
+        else:
             raise WicError("Not supported fstype: {}".format(part.fstype))
         if pnum not in self._partimages:
             tmpf = tempfile.NamedTemporaryFile(prefix="wic-part")
-- 
2.1.4

[PATCH 07/10] wic: extend list of used tools

[Ed Bartosh]

Added sfdisk, e2fsck, mkswap, resize2fs, mkdosfs to the
list of used tools in Disk class. They're going to be used
in 'wic write' implementation.

Added dependency to util-linux to wic-tools to ensure that
sfdisk and mkswap are available from wic-tools native sysroot.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-core/meta/wic-tools.bb | 3 ++-
 scripts/lib/wic/engine.py           | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index 428befe..57dd37a 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -5,7 +5,8 @@ LICENSE = "MIT"
 DEPENDS = "\
            parted-native syslinux-native gptfdisk-native dosfstools-native \
            mtools-native bmap-tools-native grub-efi-native cdrtools-native \
-           btrfs-tools-native squashfs-tools-native pseudo-native e2fsprogs-native \
+           btrfs-tools-native squashfs-tools-native pseudo-native \
+           e2fsprogs-native util-linux-native \
            "
 DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
 DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index a6ea362..3c1f0d3 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -276,7 +276,8 @@ class Disk:
 
     def __getattr__(self, name):
         """Get path to the executable in a lazy way."""
-        if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+        if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", "e2fsck",
+                    "resize2fs", "mkswap", "mkdosfs"):
             aname = "_%s" % name
             if aname not in self.__dict__:
                 setattr(self, aname, find_executable(name, self.paths))
-- 
2.1.4

[PATCH 08/10] wic: implement 'wic write' command

[Ed Bartosh]

This command writes image to the media or another file with
the possibility to expand partitions to fill free target space.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 142 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 141 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 3c1f0d3..303f323 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -31,6 +31,8 @@
 import logging
 import os
 import tempfile
+import json
+import subprocess
 
 from collections import namedtuple, OrderedDict
 from distutils.spawn import find_executable
@@ -340,6 +342,143 @@ class Disk:
                 raise err
         self._put_part_image(pnum)
 
+    def write(self, target, expand):
+        """Write disk image to the media or file."""
+        def write_sfdisk_script(outf, parts):
+            for key, val in parts['partitiontable'].items():
+                if key in ("partitions", "device", "firstlba", "lastlba"):
+                    continue
+                if key == "id":
+                    key = "label-id"
+                outf.write("{}: {}\n".format(key, val))
+            outf.write("\n")
+            for part in parts['partitiontable']['partitions']:
+                line = ''
+                for name in ('attrs', 'name', 'size', 'type', 'uuid'):
+                    if name == 'size' and part['type'] == 'f':
+                        # don't write size for extended partition
+                        continue
+                    val = part.get(name)
+                    if val:
+                        line += '{}={}, '.format(name, val)
+                if line:
+                    line = line[:-2] # strip ', '
+                if part.get('bootable'):
+                    line += ' ,bootable'
+                outf.write("{}\n".format(line))
+            outf.flush()
+
+        def read_ptable(path):
+            out = exec_cmd("{} -dJ {}".format(self.sfdisk, path))
+            return json.loads(out)
+
+        def write_ptable(parts, target):
+            with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", mode='w') as outf:
+                write_sfdisk_script(outf, parts)
+                cmd = "{} --no-reread {} < {} 2>/dev/null".format(self.sfdisk, target, outf.name)
+                try:
+                    subprocess.check_output(cmd, shell=True)
+                except subprocess.CalledProcessError as err:
+                    raise WicError("Can't run '{}' command: {}".format(cmd, err))
+
+        if expand is None:
+            sparse_copy(self.imagepath, target)
+        else:
+            # copy first sectors that may contain bootloader
+            sparse_copy(self.imagepath, target, length=2048 * self._lsector_size)
+
+            # copy source partition table to the target
+            parts = read_ptable(self.imagepath)
+            write_ptable(parts, target)
+
+            # get size of unpartitioned space
+            free = None
+            for line in exec_cmd("{} -F {}".format(self.sfdisk, target)).splitlines():
+                if line.startswith("Unpartitioned space ") and line.endswith("sectors"):
+                    free = int(line.split()[-2])
+            if free is None:
+                raise WicError("Can't get size of unpartitioned space")
+
+            # calculate expanded partitions sizes
+            sizes = {}
+            for num, part in enumerate(parts['partitiontable']['partitions'], 1):
+                if num in expand:
+                    if expand[num] != 0: # don't resize partition if size is set to 0
+                        sectors = expand[num] // self._lsector_size
+                        free -= sectors - part['size']
+                        part['size'] = sectors
+                        sizes[num] = sectors
+                elif part['type'] != 'f':
+                    sizes[num] = -1
+
+            for num, part in enumerate(parts['partitiontable']['partitions'], 1):
+                if sizes.get(num) == -1:
+                    part['size'] += free // len(sizes)
+
+            # write resized partition table to the target
+            write_ptable(parts, target)
+
+            # read resized partition table
+            parts = read_ptable(target)
+
+            # copy partitions content
+            for num, part in enumerate(parts['partitiontable']['partitions'], 1):
+                pnum = str(num)
+                fstype = self.partitions[pnum].fstype
+
+                # copy unchanged partition
+                if part['size'] == self.partitions[pnum].size // self._lsector_size:
+                    logger.info("copying unchanged partition {}".format(pnum))
+                    sparse_copy(self._get_part_image(pnum), target, seek=part['start'] * self._lsector_size)
+                    continue
+
+                # resize or re-create partitions
+                if fstype.startswith('ext') or fstype.startswith('fat') or \
+                   fstype.startswith('linux-swap'):
+
+                    partfname = None
+                    with tempfile.NamedTemporaryFile(prefix="wic-part{}-".format(pnum)) as partf:
+                        partfname = partf.name
+
+                    if fstype.startswith('ext'):
+                        logger.info("resizing ext partition {}".format(pnum))
+                        partimg = self._get_part_image(pnum)
+                        sparse_copy(partimg, partfname)
+                        exec_cmd("{} -pf {}".format(self.e2fsck, partfname))
+                        exec_cmd("{} {} {}s".format(\
+                                 self.resize2fs, partfname, part['size']))
+                    elif fstype.startswith('fat'):
+                        logger.info("copying content of the fat partition {}".format(pnum))
+                        with tempfile.TemporaryDirectory(prefix='wic-fatdir-') as tmpdir:
+                            # copy content to the temporary directory
+                            cmd = "{} -snompi {} :: {}".format(self.mcopy,
+                                                               self._get_part_image(pnum),
+                                                               tmpdir)
+                            exec_cmd(cmd)
+                            # create new msdos partition
+                            label = part.get("name")
+                            label_str = "-n {}".format(label) if label else ''
+
+                            cmd = "{} {} -C {} {}".format(self.mkdosfs, label_str, partfname,
+                                                          part['size'])
+                            exec_cmd(cmd)
+                            # copy content from the temporary directory to the new partition
+                            cmd = "{} -snompi {} {}/* ::".format(self.mcopy, partfname, tmpdir)
+                            exec_cmd(cmd, as_shell=True)
+                    elif fstype.startswith('linux-swap'):
+                        logger.info("creating swap partition {}".format(pnum))
+                        label = part.get("name")
+                        label_str = "-L {}".format(label) if label else ''
+                        uuid = part.get("uuid")
+                        uuid_str = "-U {}".format(uuid) if uuid else ''
+                        with open(partfname, 'w') as sparse:
+                            os.ftruncate(sparse.fileno(), part['size'] * self._lsector_size)
+                        exec_cmd("{} {} {} {}".format(self.mkswap, label_str, uuid_str, partfname))
+                    sparse_copy(partfname, target, seek=part['start'] * self._lsector_size)
+                    os.unlink(partfname)
+                elif part['type'] != 'f':
+                    logger.warn("skipping partition {}: unsupported fstype {}".format(pnum, fstype))
+
 def wic_ls(args, native_sysroot):
     """List contents of partitioned image or vfat partition."""
     disk = Disk(args.path.image, native_sysroot)
@@ -374,7 +513,8 @@ def wic_write(args, native_sysroot):
     """
     Write image to a target device.
     """
-    pass
+    disk = Disk(args.image, native_sysroot, ('fat', 'ext', 'swap'))
+    disk.write(args.target, args.expand)
 
 def find_canned(scripts_path, file_name):
     """
-- 
2.1.4

[PATCH 09/10] wic: always read image partitions

[Ed Bartosh]

Got rid of lazy evaluation of self.partitions property.
It's not needed because partitions of the source image should
be always read.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/engine.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 303f323..eafc6c7 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -254,12 +254,13 @@ class Disk:
         if not self.parted:
             raise WicError("Can't find executable parted")
 
+        self.partitions = self.get_partitions()
+
     def __del__(self):
         for path in self._partimages.values():
             os.unlink(path)
 
-    @property
-    def partitions(self):
+    def get_partitions(self):
         if self._partitions is None:
             self._partitions = OrderedDict()
             out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath))
-- 
2.1.4

[PATCH 10/10] wic: setlftest: test expanding MBR image

[Ed Bartosh]

Added test_expand_mbr_image test case to the wic oe-selftest suite.
The test expands directrisk wic image to 1Gb target, checks if it's
expanded correctly and boots it in qemu to make sure the image
is bootable, i.e. bootloader, kernel, boot and root partitions
are still functional.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/wic.py | 53 +++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index aaefd4f..aa73ba4 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -938,3 +938,56 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
             wksname = os.path.splitext(os.path.basename(wks.name))[0]
             out = glob(self.resultdir + "%s-*direct" % wksname)
             self.assertEqual(1, len(out))
+
+    def test_expand_mbr_image(self):
+        """Test wic write --expand command for mbr image"""
+        # build an image
+        config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "directdisk.wks"\n'
+        self.append_config(config)
+        self.assertEqual(0, bitbake('core-image-minimal').status)
+
+        # get path to the image
+        bb_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'MACHINE'])
+        deploy_dir = bb_vars['DEPLOY_DIR_IMAGE']
+        machine = bb_vars['MACHINE']
+        image_path = os.path.join(deploy_dir, 'core-image-minimal-%s.wic' % machine)
+
+        self.remove_config(config)
+
+        try:
+            # expand image to 1G
+            new_image_path = None
+            with NamedTemporaryFile(mode='wb', suffix='.wic.exp',
+                                    dir=deploy_dir, delete=False) as sparse:
+                sparse.truncate(1024 ** 3)
+                new_image_path = sparse.name
+
+            sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+            cmd = "wic write -n %s --expand 1:0 %s %s" % (sysroot, image_path, new_image_path)
+            self.assertEqual(0, runCmd(cmd).status)
+
+            # check if partitions are expanded
+            orig = runCmd("wic ls %s -n %s" % (image_path, sysroot))
+            exp = runCmd("wic ls %s -n %s" % (new_image_path, sysroot))
+            orig_sizes = [int(line.split()[3]) for line in orig.output.split('\n')[1:]]
+            exp_sizes = [int(line.split()[3]) for line in exp.output.split('\n')[1:]]
+            self.assertEqual(orig_sizes[0], exp_sizes[0]) # first partition is not resized
+            self.assertTrue(orig_sizes[1] < exp_sizes[1])
+
+            # Check if all free space is partitioned
+            result = runCmd("%s/usr/sbin/sfdisk -F %s" % (sysroot, new_image_path))
+            self.assertTrue("0 B, 0 bytes, 0 sectors" in result.output)
+
+            os.rename(image_path, image_path + '.bak')
+            os.rename(new_image_path, image_path)
+
+            # Check if it boots in qemu
+            with runqemu('core-image-minimal', ssh=False) as qemu:
+                cmd = "ls /etc/"
+                status, output = qemu.run_serial('true')
+                self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+        finally:
+            if os.path.exists(new_image_path):
+                os.unlink(new_image_path)
+            if os.path.exists(image_path + '.bak'):
+                os.rename(image_path + '.bak', image_path)
-- 
2.1.4

Re: [PATCH 00/10] #11278 - wic: create an option to use entire disk

[Otavio Salvador]

Hello Ed,

On Fri, Aug 25, 2017 at 10:41 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
...
>   wic: add help and usage content for 'wic write'
>   wic: add 'wic write' command
>   wic: implement 'wic write' command

Those should be squashed as they are in fact one logical unit.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 10/10] wic: setlftest: test expanding MBR image

[Otavio Salvador]

On Fri, Aug 25, 2017 at 10:41 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> Added test_expand_mbr_image test case to the wic oe-selftest suite.
> The test expands directrisk wic image to 1Gb target, checks if it's

directdisk

> expanded correctly and boots it in qemu to make sure the image
> is bootable, i.e. bootloader, kernel, boot and root partitions
> are still functional.




-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750