[wic][PATCH v3 00/14] Support for wic images in runqemu and oe-selftest

[wic][PATCH v3 00/14] Support for wic images in runqemu and oe-selftest

[Ed Bartosh]

Hi,

This patchset contains set of fixes for wic, runqemu and oe-selftest to
support booting and testing wic images in qemu.

This work is based on Cristian's work from poky-contrib:ciorga/wic-testing
Thank you Cristian!

The patchset also includes bugfix for wic --use-uuid option and a bit of
code cleanup for wic tests.

[YOCTO #8691]
[YOCTO #8498]
[YOCTO #8499]

Changes in v2: Dropped "runqemu: don't use virtio for wic images"
               as suggested by reviewers.

Changes in v3: Fixed test_wic_image_type test case

The following changes since commit 971fafba6a04531acc73ed348f04a6315be6d0ef:

  maintainers.inc: include libjpeg-turbo and mmc-utils (2016-02-12 18:03:47 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/qemuRunner-wic-8498
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/qemuRunner-wic-8498

Ed Bartosh (14):
  runqemu: add support for wic images
  runqemu: don't set KERNEL for wic images
  runqemu: support path/to/<image>-<machine>.wic
  oe-selftest: remove unused parameter
  qemurunner: add parameter to method 'start'
  oeqa/targetcontrol: make ssh control optional
  oeqa/targetcontrol: support wic image type
  wic-image-minimal: change IMAGE_FSTYPES
  wic-image-minimal: update .wks to boot by qemu
  oe-selftest: add new wic testcase
  wic: fix processing of --use-uuid
  wic-image-minimal: use uuid for root partition
  oe-selftest: pylinted wic tests
  oe-selftest: don't use specific tasks

 .../recipes-test/images/wic-image-minimal.bb       |  4 +-
 .../recipes-test/images/wic-image-minimal.wks      | 10 ++---
 meta/lib/oeqa/selftest/devtool.py                  |  2 +-
 meta/lib/oeqa/selftest/imagefeatures.py            |  6 +--
 meta/lib/oeqa/selftest/wic.py                      | 34 ++++++++++------
 meta/lib/oeqa/targetcontrol.py                     | 20 +++++-----
 meta/lib/oeqa/utils/commands.py                    |  4 +-
 meta/lib/oeqa/utils/qemurunner.py                  | 45 +++++++++++-----------
 scripts/lib/wic/partition.py                       |  3 ++
 scripts/runqemu                                    | 15 +++++---
 scripts/runqemu-internal                           | 18 ++++-----
 11 files changed, 90 insertions(+), 71 deletions(-)

--
Regards,
Ed

[wic][PATCH v3 01/14] runqemu: add support for wic images

[Ed Bartosh]

Quemu should be able to run wic images this way:
    runqemu <machine> <image recipe> wic

Tested with 'runqemu qemux86-64 wic-image-minimal wic'

(From OE-Core rev: 8716be799949cb8bde7fa49cbea61312a3a93bb7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/runqemu          |  3 ++-
 scripts/runqemu-internal | 16 ++++++++--------
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 46918e2..4bb3bb6 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -40,6 +40,7 @@ usage() {
     echo "Examples:"
     echo "  $MYNAME qemuarm"
     echo "  $MYNAME qemux86-64 core-image-sato ext4"
+    echo "  $MYNAME qemux86-64 wic-image-minimal wic"
     echo "  $MYNAME path/to/bzImage-qemux86.bin path/to/nfsrootdir/ serial"
     echo "  $MYNAME qemux86 ramfs"
     echo "  $MYNAME qemux86 iso"
@@ -116,7 +117,7 @@ while true; do
             [ -z "$MACHINE" -o "$MACHINE" = "$arg" ] && MACHINE=$arg || \
                 error "conflicting MACHINE types [$MACHINE] and [$arg]"
             ;;
-        "ext2" | "ext3" | "ext4" | "jffs2" | "nfs" | "btrfs" | "hddimg" | "hdddirect" )
+        "ext2" | "ext3" | "ext4" | "jffs2" | "nfs" | "btrfs" | "hddimg" | "hdddirect" | "wic" )
             [ -z "$FSTYPE" -o "$FSTYPE" = "$arg" ] && FSTYPE=$arg || \
                 error "conflicting FSTYPE types [$FSTYPE] and [$arg]"
             ;;
diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index fb98a6e..2926024 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -349,7 +349,7 @@ if [ "$MACHINE" = "qemuarm" -o "$MACHINE" = "qemuarmv6" -o "$MACHINE" = "qemuarm
     export QEMU_AUDIO_DRV="none"
     QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS"
     # QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -force-pointer"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyAMA0,115200 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY highres=off"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M ${MACHINE_SUBTYPE} $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
     fi
@@ -383,7 +383,7 @@ if [ "$MACHINE" = "qemuarm64" ]; then
     else
         QEMU_UI_OPTIONS=""
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyAMA0,38400 mem=$QEMU_MEMORY highres=off $KERNEL_NETWORK_CMD"
         # qemu-system-aarch64 only support '-machine virt -cpu cortex-a57' for now
         QEMUOPTIONS="$QEMU_NETWORK_CMD -machine virt -cpu cortex-a57 $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
@@ -411,7 +411,7 @@ if [ "$MACHINE" = "qemux86" ]; then
     else
         QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -vga vmware"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="vga=0 uvesafb.mode_option=640x480-32 root=$DROOT rw mem=$QEMU_MEMORY $KERNEL_NETWORK_CMD"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
     fi
@@ -449,7 +449,7 @@ if [ "$MACHINE" = "qemux86-64" ]; then
     else
         QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -vga vmware"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="vga=0 uvesafb.mode_option=640x480-32 root=$DROOT rw mem=$QEMU_MEMORY $KERNEL_NETWORK_CMD"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
     fi
@@ -495,7 +495,7 @@ if [ "$MACHINE" = "qemumips" -o "$MACHINE" = "qemumipsel" -o "$MACHINE" = "qemum
     esac
     MACHINE_SUBTYPE=malta
     QEMU_UI_OPTIONS="-vga cirrus $QEMU_UI_OPTIONS"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         #KERNCMDLINE="root=/dev/hda console=ttyS0 console=tty0 $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         KERNCMDLINE="root=$DROOT rw console=ttyS0 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M $MACHINE_SUBTYPE $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
@@ -521,7 +521,7 @@ if [ "$MACHINE" = "qemuppc" ]; then
     else
         QEMU_NETWORK_CMD="-net nic,model=pcnet $QEMU_TAP_CMD"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyS0 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE -M $MACHINE_SUBTYPE $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
     fi
@@ -540,7 +540,7 @@ if [ "$MACHINE" = "qemush4" ]; then
     QEMU=qemu-system-sh4
     MACHINE_SUBTYPE=r2d
     QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         #KERNCMDLINE="root=/dev/hda console=ttyS0 console=tty0 $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         KERNCMDLINE="root=/dev/hda rw console=ttySC1 noiotrap earlyprintk=sh-sci.1 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M $MACHINE_SUBTYPE -hda $ROOTFS -no-reboot $QEMU_UI_OPTIONS -monitor null -serial vc -serial stdio"
@@ -726,7 +726,7 @@ echo "Running $QEMU..."
 if [ "$FSTYPE" = "vmdk" -o "$FSTYPE" = "hddimg" -o "$FSTYPE" = "hdddirect" ]; then
     echo $QEMUBIN $VM $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
     LD_PRELOAD="$GL_LD_PRELOAD" $QEMUBIN $VM $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
-elif [ "$FSTYPE" = "iso" ]; then
+elif [ "$FSTYPE" = "iso" -o "$FSTYPE" = "wic" ]; then
     echo $QEMUBIN $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
     LD_PRELOAD="$GL_LD_PRELOAD" $QEMUBIN $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
 else
-- 
2.1.4

[wic][PATCH v3 02/14] runqemu: don't set KERNEL for wic images

[Ed Bartosh]

Wic images should be boot as is, without pointing qemu to the kernel
binary. Current code doesn't use kernel, but sets KERNEL variable and
shows kernel path in the console output. This can confuse users.

Changed runqemu and runqemu-internal code to avoid setting KERNEL
variable and show kernel path.

(From OE-Core rev: 474caa7ed5ff05caa5d49d270b283882fa616ed1)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/runqemu          | 2 +-
 scripts/runqemu-internal | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 4bb3bb6..0830901 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -454,7 +454,7 @@ if [ -e "$ROOTFS" -a -z "$FSTYPE" ]; then
     fi
 fi
 
-if [ -z "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" ]; then
+if [ -z "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" -a "x$FSTYPE" != "xwic" ]; then
     setup_path_vars 1
     eval kernel_file=\$${machine2}_DEFAULT_KERNEL
     KERNEL=$DEPLOY_DIR_IMAGE/$kernel_file
diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index 2926024..ad854d1 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -301,7 +301,7 @@ case "$MACHINE" in
     ;;
 esac
 
-if [ ! -f "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" ]; then
+if [ ! -f "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" -a "x$FSTYPE" != "xwic" ]; then
     echo "Error: Kernel image file $KERNEL doesn't exist"
     cleanup
     return 1
-- 
2.1.4

[wic][PATCH v3 03/14] runqemu: support path/to/<image>-<machine>.wic

[Ed Bartosh]

Supported providing wic image path to runqemu:
  runquemu path/to/<image>-<machine>.wic

[YOCTO #8691]

(From OE-Core rev: 58a3bfb1e4b493200820cdf0bf3fc79e31e792de)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/runqemu | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 0830901..48b7551 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -47,6 +47,7 @@ usage() {
     echo "  $MYNAME qemux86 qemuparams=\"-m 256\""
     echo "  $MYNAME qemux86 bootparams=\"psplash=false\""
     echo "  $MYNAME path/to/<image>-<machine>.vmdk"
+    echo "  $MYNAME path/to/<image>-<machine>.wic"
     exit 1
 }
 
@@ -88,7 +89,7 @@ process_filename() {
 		[ -z "$KERNEL" ] && KERNEL=$filename || \
 		    error "conflicting KERNEL args [$KERNEL] and [$filename]"
 		;;
-	/ext[234]/|/jffs2/|/btrfs/)
+	/ext[234]/|/jffs2/|/btrfs/|/wic/)
 		# A file ending in a supportted fs type is a rootfs image
 		if [ -z "$FSTYPE" -o "$FSTYPE" = "$EXT" ]; then
 		    FSTYPE=$EXT
@@ -235,7 +236,7 @@ elif [ ! -w /dev/net/tun ] ; then
 fi
 
 # Report errors for missing combinations of options
-if [ -z "$MACHINE" -a -z "$KERNEL" -a -z "$VM" ]; then
+if [ -z "$MACHINE" -a -z "$KERNEL" -a -z "$VM" -a "$FSTYPE" != "wic" ]; then
     error "you must specify at least a MACHINE, VM, or KERNEL argument"
 fi
 if [ "$FSTYPE" = "nfs" -a -z "$ROOTFS" ]; then
@@ -243,8 +244,9 @@ if [ "$FSTYPE" = "nfs" -a -z "$ROOTFS" ]; then
 fi
 
 if [ -z "$MACHINE" ]; then
-    if [ "x$FSTYPE" = "xvmdk" ] || [ "x$FSTYPE" = "xhddimg" ] || [ "x$FSTYPE" = "xhdddirect" ]; then
-        MACHINE=`basename $VM | sed -n 's/.*\(qemux86-64\|qemux86\|qemuarm64\|qemuarm\|qemumips64\|qemumips\|qemuppc\|qemush4\).*/\1/p'`
+    if [ "x$FSTYPE" = "xvmdk" ] || [ "x$FSTYPE" = "xhddimg" ] || [ "x$FSTYPE" = "xhdddirect" ] || [ "x$FSTYPE" = "xwic" ]; then
+        [ "x$FSTYPE" = "xwic" ] && filename=$ROOTFS || filename=$VM
+        MACHINE=`basename $filename | sed -n 's/.*\(qemux86-64\|qemux86\|qemuarm64\|qemuarm\|qemumips64\|qemumips\|qemuppc\|qemush4\).*/\1/p'`
         if [ -z "$MACHINE" ]; then
             error "Unable to set MACHINE from image filename [$VM]"
         fi
-- 
2.1.4

[wic][PATCH v3 04/14] oe-selftest: remove unused parameter

[Ed Bartosh]

From: Ed Bartosh <bartosh@gmail.com>

Removed unused parameter 'test' from runqemu function.

(From OE-Core rev: c688b3bcbb57099fa72a9728bc708b109802f7fc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/devtool.py       | 2 +-
 meta/lib/oeqa/selftest/imagefeatures.py | 6 +++---
 meta/lib/oeqa/utils/commands.py         | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index e5a2134..0361464 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -1025,7 +1025,7 @@ class DevtoolTests(DevtoolBase):
         result = runCmd('devtool deploy-target -n %s root@localhost' % testrecipe)
         self.assertIn('  %s' % testfile, result.output)
         # Boot the image
-        with runqemu(testimage, self) as qemu:
+        with runqemu(testimage) as qemu:
             # Now really test deploy-target
             result = runCmd('devtool deploy-target -c %s root@%s' % (testrecipe, qemu.ip))
             # Run a test command to see if it was installed properly
diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 62ddc52..1c08ce2 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -31,7 +31,7 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             # Attempt to ssh with each user into qemu with empty password
             for user in [self.root_user, self.test_user]:
                 ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
@@ -58,7 +58,7 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             # Attempt to ssh with each user into qemu with empty password
             for user in [self.root_user, self.test_user]:
                 ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
@@ -112,7 +112,7 @@ class ImageFeatures(oeSelfTest):
                 self.fail('No rpm package found in image')
 
         # Now do a couple of runtime tests
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             command = "rpm --version"
             status, output = qemu.run(command)
             self.assertEqual(0, status, 'Failed to run command "%s": %s' % (command, output))
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index dfb6c21..93a0e48 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -178,7 +178,7 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
 
 
 @contextlib.contextmanager
-def runqemu(pn, test):
+def runqemu(pn):
 
     import bb.tinfoil
     import bb.build
-- 
2.1.4

[wic][PATCH v3 05/14] qemurunner: add parameter to method 'start'

[Ed Bartosh]

QemuRunner requires pair of ip addresses provided through kernel
commandline for method 'start' to work. These ip addresses are
used to connect to the image using ssh and run tests there.
However, this functionality should not be mandatory as testing
doesn't always require ssh connection. Some tests can be run using
serial console.

Added new parameter 'get_ip' to QemuRunner.start to make it possible
to skip getting pair of ip addresses from kernel command line. This
should allow oe-selftest to test images without modifying kernel
command line.

[YOCTO #8498]

(From OE-Core rev: 3f8b734ebb81d035849288091bb0b97b9c4fba34)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/utils/qemurunner.py | 45 ++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index def8ace..784cf96 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -91,7 +91,7 @@ class QemuRunner:
                 self._dump_host()
                 raise SystemExit
 
-    def start(self, qemuparams = None):
+    def start(self, qemuparams = None, get_ip = True):
         if self.display:
             os.environ["DISPLAY"] = self.display
             # Set this flag so that Qemu doesn't do any grabs as SDL grabs
@@ -178,27 +178,28 @@ class QemuRunner:
 
         if self.is_alive():
             logger.info("qemu started - qemu procces pid is %s" % self.qemupid)
-            cmdline = ''
-            with open('/proc/%s/cmdline' % self.qemupid) as p:
-                cmdline = p.read()
-                # It is needed to sanitize the data received
-                # because is possible to have control characters
-                cmdline = re_control_char.sub('', cmdline)
-            try:
-                ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
-                if not ips or len(ips) != 3:
-                    raise ValueError
-                else:
-                    self.ip = ips[0]
-                    self.server_ip = ips[1]
-            except IndexError, ValueError:
-                logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used:\n%s\nand output from runqemu:\n%s" % (cmdline, self.getOutput(output)))
-                self._dump_host()
-                self.stop()
-                return False
-            logger.info("qemu cmdline used:\n{}".format(cmdline))
-            logger.info("Target IP: %s" % self.ip)
-            logger.info("Server IP: %s" % self.server_ip)
+            if get_ip:
+                cmdline = ''
+                with open('/proc/%s/cmdline' % self.qemupid) as p:
+                    cmdline = p.read()
+                    # It is needed to sanitize the data received
+                    # because is possible to have control characters
+                    cmdline = re_control_char.sub('', cmdline)
+                try:
+                    ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
+                    if not ips or len(ips) != 3:
+                        raise ValueError
+                    else:
+                        self.ip = ips[0]
+                        self.server_ip = ips[1]
+                except IndexError, ValueError:
+                    logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used:\n%s\nand output from runqemu:\n%s" % (cmdline, self.getOutput(output)))
+                    self._dump_host()
+                    self.stop()
+                    return False
+                logger.info("qemu cmdline used:\n{}".format(cmdline))
+                logger.info("Target IP: %s" % self.ip)
+                logger.info("Server IP: %s" % self.server_ip)
 
             self.thread = LoggingThread(self.log, threadsock, logger)
             self.thread.start()
-- 
2.1.4

[wic][PATCH v3 06/14] oeqa/targetcontrol: make ssh control optional

[Ed Bartosh]

Added new parameter 'ssh' to targetcontrol 'start' method
to be able to test images without running ssh server.

[YOCTO #8498]

(From OE-Core rev: 1c3c66aadd43092bc19242b0651ee810cc31fe7c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/targetcontrol.py  | 18 ++++++++++--------
 meta/lib/oeqa/utils/commands.py |  4 ++--
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index edc0d01..f461651 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -68,7 +68,7 @@ class BaseTarget(object):
         bb.note("SSH log file: %s" %  self.sshlog)
 
     @abstractmethod
-    def start(self, params=None):
+    def start(self, params=None, ssh=True):
         pass
 
     @abstractmethod
@@ -176,11 +176,12 @@ class QemuTarget(BaseTarget):
         bb.note("Qemu log file: %s" % self.qemulog)
         super(QemuTarget, self).deploy()
 
-    def start(self, params=None):
-        if self.runner.start(params):
-            self.ip = self.runner.ip
-            self.server_ip = self.runner.server_ip
-            self.connection = SSHControl(ip=self.ip, logfile=self.sshlog)
+    def start(self, params=None, ssh=True):
+        if self.runner.start(params, get_ip=ssh):
+            if ssh:
+                self.ip = self.runner.ip
+                self.server_ip = self.runner.server_ip
+                self.connection = SSHControl(ip=self.ip, logfile=self.sshlog)
         else:
             self.stop()
             if os.path.exists(self.qemulog):
@@ -231,8 +232,9 @@ class SimpleRemoteTarget(BaseTarget):
     def deploy(self):
         super(SimpleRemoteTarget, self).deploy()
 
-    def start(self, params=None):
-        self.connection = SSHControl(self.ip, logfile=self.sshlog, port=self.port)
+    def start(self, params=None, ssh=True):
+        if ssh:
+            self.connection = SSHControl(self.ip, logfile=self.sshlog, port=self.port)
 
     def stop(self):
         self.connection = None
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 93a0e48..32e001c 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -178,7 +178,7 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
 
 
 @contextlib.contextmanager
-def runqemu(pn):
+def runqemu(pn, ssh=True):
 
     import bb.tinfoil
     import bb.build
@@ -222,7 +222,7 @@ def runqemu(pn):
     try:
         qemu.deploy()
         try:
-            qemu.start()
+            qemu.start(ssh=ssh)
         except bb.build.FuncFailed:
             raise Exception('Failed to start QEMU - see the logs in %s' % logdir)
 
-- 
2.1.4

[wic][PATCH v3 07/14] oeqa/targetcontrol: support wic image type

[Ed Bartosh]

Added 'wic' to the list image types supported by targetcontrol.

This is a preparation for booting and testing wic images
with oe-selftest.

[YOCTO #8498]

(From OE-Core rev: 7dda053fbd1ea1354b7720cfa691470ba88ef5b9)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/targetcontrol.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index f461651..5422a61 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -113,7 +113,7 @@ class BaseTarget(object):
 
 class QemuTarget(BaseTarget):
 
-    supported_image_fstypes = ['ext3', 'ext4', 'cpio.gz']
+    supported_image_fstypes = ['ext3', 'ext4', 'cpio.gz', 'wic']
 
     def __init__(self, d):
 
-- 
2.1.4

[wic][PATCH v3 08/14] wic-image-minimal: change IMAGE_FSTYPES

[Ed Bartosh]

Replaced wic.bz2 -> wic in the recipe.
Replaced wic.bz2 -> wic in oe-selftest test case.

wic-image-minimal is going to be booted in qemu and tested.
As runqemu doesn't support compressed images this recipe needs
to produce raw wic image.

(From OE-Core rev: 3ce91db94f4b921566dce0dc6f91a422009be06b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.bb | 2 +-
 meta/lib/oeqa/selftest/wic.py                          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb b/meta-selftest/recipes-test/images/wic-image-minimal.bb
index 95c7a9b..ced22c8 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.bb
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
@@ -4,7 +4,7 @@ SRC_URI = "file://${FILE_DIRNAME}/${BPN}.wks"
 
 IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 
-IMAGE_FSTYPES = "wic.bz2"
+IMAGE_FSTYPES = "wic"
 RM_OLD_IMAGE = "1"
 
 DEPENDS = "syslinux syslinux-native parted-native dosfstools-native mtools-native"
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index df826ed..a739f69 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -228,7 +228,7 @@ class Wic(oeSelfTest):
         prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
         # check if we have result image and manifests symlinks
         # pointing to existing files
-        for suffix in ('wic.bz2', 'manifest'):
+        for suffix in ('wic', 'manifest'):
             path = prefix + suffix
             self.assertTrue(os.path.islink(path))
             self.assertTrue(os.path.isfile(os.path.realpath(path)))
-- 
2.1.4

[wic][PATCH v3 09/14] wic-image-minimal: update .wks to boot by qemu

[Ed Bartosh]

With below list of changes wic-image-minimal can be boot by qemu:
    - Changed device name sda -> vda.
    - Added root=/dev/vda2 to the kernel command line.
    - Changed mount point for core partition to /mnt to have it
      mounted on boot time.

(From OE-Core rev: f085274711f561f97a2c89f40bb0e89094973054)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.wks b/meta-selftest/recipes-test/images/wic-image-minimal.wks
index 8f9be09..191244b 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.wks
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.wks
@@ -2,9 +2,9 @@
 # long-description: This image contains boot partition and 3 rootfs partitions
 # created from core-image-minimal and wic-image-minimal image recipes.
 
-part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
-part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label core --align 1024
-part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext4 --label backup --align 1024
+part /boot --source bootimg-pcbios --ondisk vda --label boot --active --align 1024
+part / --source rootfs --ondisk vda --fstype=ext4 --label platform --align 1024
+part /mnt --source rootfs --rootfs-dir=core-image-minimal --ondisk vda --fstype=ext4 --label core --align 1024
+part backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk vda --fstype=ext4 --label backup --align 1024
 
-bootloader  --timeout=0  --append="rootwait console=tty0"
+bootloader --timeout=0  --append="rootwait console=tty0 root=/dev/vda2"
-- 
2.1.4

[wic][PATCH v3 10/14] oe-selftest: add new wic testcase

[Ed Bartosh]

Added test_qemu testcase to boot wic-image-minimal and test
that 2 partitions mentioned in .wks are mounted.

[YOCTO #8499]

(From OE-Core rev: 6fb015d0847fe7d259d654d4a99bf4c328f810ab)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/wic.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index a739f69..7b173e4 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -30,7 +30,7 @@ from glob import glob
 from shutil import rmtree
 
 from oeqa.selftest.base import oeSelfTest
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 from oeqa.utils.decorators import testcase
 
 
@@ -264,3 +264,13 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_qemu(self):
+        """Test wic-image-minimal under qemu"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        with runqemu('wic-image-minimal', ssh=False) as qemu:
+            command = "mount |grep '^/dev/' | cut -f1,3 -d ' '"
+            status, output = qemu.run_serial(command)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (command, output))
+            self.assertEqual(output, '/dev/root /\r\n/dev/vda3 /mnt')
-- 
2.1.4

[wic][PATCH v3 11/14] wic: fix processing of --use-uuid

[Ed Bartosh]

Processing of this option was lost during recent change of
wks parsting. It was discovered during the work on booting
wic images under qemu. Now, when -use-uuid is fixed it's
possible to specify root partition by partition uuid.
This will be done in the following commit.

(From OE-Core rev: b4882e0b84d7fd4c85ee95386e94722485eafc2b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/wic/partition.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 9bb1a0a..55bcfba 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -57,7 +57,10 @@ class Partition(object):
         self.size = args.size
         self.source = args.source
         self.sourceparams = args.sourceparams
+        self.use_uuid = args.use_uuid
         self.uuid = args.uuid
+        if args.use_uuid and not self.uuid:
+            self.uuid = str(uuid.uuid4())
 
         self.lineno = lineno
         self.source_file = ""
-- 
2.1.4

[wic][PATCH v3 12/14] wic-image-minimal: use uuid for root partition

[Ed Bartosh]

Used --uuid option to generate PARTUUID for root partion and
gpt partition type. As this image is used in runtime testing
under qemu it makes sense to ensure that this way of specifying
root partition works.

Added dependency to gptfdisk-native to wic-image-minimal recipe to
be able to create gpt partition table.

(From OE-Core rev: ba82867bfda54e86e1de5dc4a6ad7246254cf5c3)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta-selftest/recipes-test/images/wic-image-minimal.bb  | 2 +-
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb b/meta-selftest/recipes-test/images/wic-image-minimal.bb
index ced22c8..9c50ddb 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.bb
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
@@ -7,7 +7,7 @@ IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 IMAGE_FSTYPES = "wic"
 RM_OLD_IMAGE = "1"
 
-DEPENDS = "syslinux syslinux-native parted-native dosfstools-native mtools-native"
+DEPENDS = "syslinux syslinux-native parted-native dosfstools-native mtools-native gptfdisk-native"
 
 # core-image-minimal is referenced in .wks, so we need its rootfs
 # to be ready before our rootfs
diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.wks b/meta-selftest/recipes-test/images/wic-image-minimal.wks
index 191244b..d55075d 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.wks
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.wks
@@ -3,8 +3,8 @@
 # created from core-image-minimal and wic-image-minimal image recipes.
 
 part /boot --source bootimg-pcbios --ondisk vda --label boot --active --align 1024
-part / --source rootfs --ondisk vda --fstype=ext4 --label platform --align 1024
+part / --source rootfs --ondisk vda --fstype=ext4 --label platform --align 1024 --use-uuid
 part /mnt --source rootfs --rootfs-dir=core-image-minimal --ondisk vda --fstype=ext4 --label core --align 1024
 part backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk vda --fstype=ext4 --label backup --align 1024
 
-bootloader --timeout=0  --append="rootwait console=tty0 root=/dev/vda2"
+bootloader --ptable gpt --timeout=0  --append="rootwait console=tty0"
-- 
2.1.4

[wic][PATCH v3 13/14] oe-selftest: pylinted wic tests

[Ed Bartosh]

Fixed the following pylint warnings:
  C: 45, 0: Line too long (91/80) (line-too-long)
  C:105, 0: Wrong continued indentation.
                            ignore_status=True).status)
                            ^         | (bad-continuation)
  W: 83, 8: Redefining built-in 'vars' (redefined-builtin)
  W:175, 8: Redefining built-in 'vars' (redefined-builtin)
  W: 27, 0: Unused import sys (unused-import)

(From OE-Core rev: ecb5320e766d0fa031c269d6d9b2a24c81c7aa7e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/wic.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 7b173e4..ed03018 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -24,7 +24,6 @@
 """Test cases for wic."""
 
 import os
-import sys
 
 from glob import glob
 from shutil import rmtree
@@ -42,7 +41,8 @@ class Wic(oeSelfTest):
 
     def setUpLocal(self):
         """This code is executed before each test method."""
-        self.write_config('IMAGE_FSTYPES += " hddimg"\nMACHINE_FEATURES_append = " efi"\n')
+        self.write_config('IMAGE_FSTYPES += " hddimg"\n'
+                          'MACHINE_FEATURES_append = " efi"\n')
 
         # Do this here instead of in setUpClass as the base setUp does some
         # clean up which can result in the native tools built earlier in
@@ -80,14 +80,14 @@ class Wic(oeSelfTest):
     @testcase(1212)
     def test_build_artifacts(self):
         """Test wic create directdisk providing all artifacts."""
-        vars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
+        bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
                         for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE',
                                     'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS'))
         status = runCmd("wic create directdisk "
                         "-b %(staging_datadir)s "
                         "-k %(deploy_dir_image)s "
                         "-n %(staging_dir_native)s "
-                        "-r %(image_rootfs)s" % vars).status
+                        "-r %(image_rootfs)s" % bbvars).status
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct")))
 
@@ -102,7 +102,7 @@ class Wic(oeSelfTest):
     def test_unsupported_subcommand(self):
         """Test unsupported subcommand"""
         self.assertEqual(1, runCmd('wic unsupported',
-                         ignore_status=True).status)
+                                   ignore_status=True).status)
 
     @testcase(1214)
     def test_no_command(self):
@@ -172,20 +172,20 @@ class Wic(oeSelfTest):
     @testcase(1269)
     def test_rootfs_artifacts(self):
         """Test usage of rootfs plugin with rootfs paths"""
-        vars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
+        bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
                         for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE',
                                     'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS'))
-        vars['wks'] = "directdisk-multi-rootfs"
+        bbvars['wks'] = "directdisk-multi-rootfs"
         status = runCmd("wic create %(wks)s "
                         "-b %(staging_datadir)s "
                         "-k %(deploy_dir_image)s "
                         "-n %(staging_dir_native)s "
                         "--rootfs-dir rootfs1=%(image_rootfs)s "
                         "--rootfs-dir rootfs2=%(image_rootfs)s" \
-                        % vars).status
+                        % bbvars).status
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + \
-                                     "%(wks)s-*.direct" % vars)))
+                                     "%(wks)s-*.direct" % bbvars)))
 
     @testcase(1346)
     def test_iso_image(self):
-- 
2.1.4

[wic][PATCH v3 14/14] oe-selftest: don't use specific tasks

[Ed Bartosh]

Image should be build usual way, i.e. using bitbake <image>.
Specifying do_mage_complete and do_rootfs_wicenv tasks should not
be needed anymore as those tasks should be run by bitbake for
every image.

Removed specifying do_mage_complete and do_rootfs_wicenv tasks
from bitbake calls.

(From OE-Core rev: d8d7bd1b41eb846f18378a2581ff172cb2cc52b8)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/wic.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index ed03018..b3f9f1e 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -50,7 +50,7 @@ class Wic(oeSelfTest):
         if not Wic.image_is_ready:
             bitbake('syslinux syslinux-native parted-native gptfdisk-native '
                     'dosfstools-native mtools-native')
-            bitbake('core-image-minimal:do_image_complete core-image-minimal:do_rootfs_wicenv')
+            bitbake('core-image-minimal')
             Wic.image_is_ready = True
 
         rmtree(self.resultdir, ignore_errors=True)
-- 
2.1.4