[wic][PATCH 00/15] Support for wic images in runqemu and oe-selftest

[wic][PATCH 00/15] Support for wic images in runqemu and oe-selftest

[Ed Bartosh]

Hi,

This patchset contains set of fixes for wic, runqemu and oe-selftest to
support booting and testing wic images in qemu.

This work is based on Cristian's work from poky-contrib:ciorga/wic-testing
Thank you Cristian!

The patchset also includes bugfix for wic --use-uuid option and a bit of
code cleanup for wic tests.

[YOCTO #8691]
[YOCTO #8498]
[YOCTO #8499]

The following changes since commit 1ba62f9e53f5e2d0da9fe6ee52007562deb77f32:

  pth: Remove dead code (2016-02-01 17:16:05 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/qemuRunner-wic-8498
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/qemuRunner-wic-8498

Ed Bartosh (15):
  runqemu: add support for wic images
  runqemu: don't use virtio for wic images
  runqemu: don't set KERNEL for wic images
  runqemu: support path/to/<image>-<machine>.wic
  oe-selftest: remove unused parameter
  qemurunner: add parameter to method 'start'
  oeqa/targetcontrol: make ssh control optional
  oeqa/targetcontrol: support wic image type
  wic-image-minimal: change IMAGE_FSTYPES
  wic-image-minimal: updated .wks to boot by qemu
  oe-selftest: add new wic testcase
  wic: fix processing of --use-uuid
  wic-image-minimal: use uuid for root partition
  oe-selftest: pylinted wic tests
  oe-selftest: don't use specific tasks

 .../recipes-test/images/wic-image-minimal.bb       |  2 +-
 .../recipes-test/images/wic-image-minimal.wks      | 10 ++---
 meta/lib/oeqa/selftest/devtool.py                  |  2 +-
 meta/lib/oeqa/selftest/imagefeatures.py            |  6 +--
 meta/lib/oeqa/selftest/wic.py                      | 32 +++++++++------
 meta/lib/oeqa/targetcontrol.py                     | 20 +++++-----
 meta/lib/oeqa/utils/commands.py                    |  4 +-
 meta/lib/oeqa/utils/qemurunner.py                  | 45 +++++++++++-----------
 scripts/lib/wic/partition.py                       |  3 ++
 scripts/runqemu                                    | 15 +++++---
 scripts/runqemu-internal                           | 21 +++++-----
 11 files changed, 90 insertions(+), 70 deletions(-)

--
Regards,
Ed

[wic][PATCH 01/15] runqemu: add support for wic images

[Ed Bartosh]

Quemu should be able to run wic images this way:
    runqemu <machine> <image recipe> wic

Tested with 'runqemu qemux86-64 wic-image-minimal wic'

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu          |  3 ++-
 scripts/runqemu-internal | 16 ++++++++--------
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 46918e2..4bb3bb6 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -40,6 +40,7 @@ usage() {
     echo "Examples:"
     echo "  $MYNAME qemuarm"
     echo "  $MYNAME qemux86-64 core-image-sato ext4"
+    echo "  $MYNAME qemux86-64 wic-image-minimal wic"
     echo "  $MYNAME path/to/bzImage-qemux86.bin path/to/nfsrootdir/ serial"
     echo "  $MYNAME qemux86 ramfs"
     echo "  $MYNAME qemux86 iso"
@@ -116,7 +117,7 @@ while true; do
             [ -z "$MACHINE" -o "$MACHINE" = "$arg" ] && MACHINE=$arg || \
                 error "conflicting MACHINE types [$MACHINE] and [$arg]"
             ;;
-        "ext2" | "ext3" | "ext4" | "jffs2" | "nfs" | "btrfs" | "hddimg" | "hdddirect" )
+        "ext2" | "ext3" | "ext4" | "jffs2" | "nfs" | "btrfs" | "hddimg" | "hdddirect" | "wic" )
             [ -z "$FSTYPE" -o "$FSTYPE" = "$arg" ] && FSTYPE=$arg || \
                 error "conflicting FSTYPE types [$FSTYPE] and [$arg]"
             ;;
diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index c2787c3..4d5f6ad 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -349,7 +349,7 @@ if [ "$MACHINE" = "qemuarm" -o "$MACHINE" = "qemuarmv6" -o "$MACHINE" = "qemuarm
     export QEMU_AUDIO_DRV="none"
     QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS"
     # QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -force-pointer"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyAMA0,115200 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY highres=off"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M ${MACHINE_SUBTYPE} $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
     fi
@@ -383,7 +383,7 @@ if [ "$MACHINE" = "qemuarm64" ]; then
     else
         QEMU_UI_OPTIONS=""
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyAMA0,38400 mem=$QEMU_MEMORY highres=off $KERNEL_NETWORK_CMD"
         # qemu-system-aarch64 only support '-machine virt -cpu cortex-a57' for now
         QEMUOPTIONS="$QEMU_NETWORK_CMD -machine virt -cpu cortex-a57 $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
@@ -411,7 +411,7 @@ if [ "$MACHINE" = "qemux86" ]; then
     else
         QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -vga vmware"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="vga=0 uvesafb.mode_option=640x480-32 root=$DROOT rw mem=$QEMU_MEMORY $KERNEL_NETWORK_CMD"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
     fi
@@ -449,7 +449,7 @@ if [ "$MACHINE" = "qemux86-64" ]; then
     else
         QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS -vga vmware"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="vga=0 uvesafb.mode_option=640x480-32 root=$DROOT rw mem=$QEMU_MEMORY $KERNEL_NETWORK_CMD"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE $ROOTFS_OPTIONS $QEMU_UI_OPTIONS"
     fi
@@ -495,7 +495,7 @@ if [ "$MACHINE" = "qemumips" -o "$MACHINE" = "qemumipsel" -o "$MACHINE" = "qemum
     esac
     MACHINE_SUBTYPE=malta
     QEMU_UI_OPTIONS="-vga cirrus $QEMU_UI_OPTIONS"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         #KERNCMDLINE="root=/dev/hda console=ttyS0 console=tty0 $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         KERNCMDLINE="root=$DROOT rw console=ttyS0 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M $MACHINE_SUBTYPE $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
@@ -521,7 +521,7 @@ if [ "$MACHINE" = "qemuppc" ]; then
     else
         QEMU_NETWORK_CMD="-net nic,model=pcnet $QEMU_TAP_CMD"
     fi
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         KERNCMDLINE="root=$DROOT rw console=ttyS0 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -cpu $CPU_SUBTYPE -M $MACHINE_SUBTYPE $ROOTFS_OPTIONS -no-reboot $QEMU_UI_OPTIONS"
     fi
@@ -540,7 +540,7 @@ if [ "$MACHINE" = "qemush4" ]; then
     QEMU=qemu-system-sh4
     MACHINE_SUBTYPE=r2d
     QEMU_UI_OPTIONS="$QEMU_UI_OPTIONS"
-    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" ]; then
+    if [ "${FSTYPE:0:3}" = "ext" -o "$FSTYPE" = "btrfs" -o "$FSTYPE" = "wic" ]; then
         #KERNCMDLINE="root=/dev/hda console=ttyS0 console=tty0 $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         KERNCMDLINE="root=/dev/hda rw console=ttySC1 noiotrap earlyprintk=sh-sci.1 console=tty $KERNEL_NETWORK_CMD mem=$QEMU_MEMORY"
         QEMUOPTIONS="$QEMU_NETWORK_CMD -M $MACHINE_SUBTYPE -hda $ROOTFS -no-reboot $QEMU_UI_OPTIONS -monitor null -serial vc -serial stdio"
@@ -726,7 +726,7 @@ echo "Running $QEMU..."
 if [ "$FSTYPE" = "vmdk" -o "$FSTYPE" = "hddimg" -o "$FSTYPE" = "hdddirect" ]; then
     echo $QEMUBIN $VM $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
     LD_PRELOAD="$GL_LD_PRELOAD" $QEMUBIN $VM $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
-elif [ "$FSTYPE" = "iso" ]; then
+elif [ "$FSTYPE" = "iso" -o "$FSTYPE" = "wic" ]; then
     echo $QEMUBIN $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
     LD_PRELOAD="$GL_LD_PRELOAD" $QEMUBIN $QEMUOPTIONS $SERIALOPTS -no-reboot $SCRIPT_QEMU_OPT $SCRIPT_QEMU_EXTRA_OPT
 else
-- 
2.1.4

[wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Ed Bartosh]

qemu hands with 'waiting for root device' message during the
boot of wic images if '-drive if=virtio' option is used.

Removed if=virtio from qemu command line for wic images. This
should fix above mentioned boot issue.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu-internal | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index 4d5f6ad..8afcccf 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -265,7 +265,8 @@ else
             QEMU_NETWORK_CMD="-net nic,model=virtio $QEMU_TAP_CMD"
         fi
         DROOT="/dev/vda"
-        ROOTFS_OPTIONS="-drive file=$ROOTFS,if=virtio,format=raw"
+        ROOTFS_OPTIONS="-drive file=$ROOTFS,format=raw"
+        [ "$FSTYPE" != "wic" ] && ROOTFS_OPTIONS="$ROOTFS_OPTIONS,if=virtio"
 
         KERNCMDLINE="mem=$QEMU_MEMORY"
         QEMU_UI_OPTIONS="-show-cursor -usb -usbdevice tablet"
-- 
2.1.4

[wic][PATCH 03/15] runqemu: don't set KERNEL for wic images

[Ed Bartosh]

Wic images should be boot as is, without pointing qemu to the kernel
binary. Current code doesn't use kernel, but sets KERNEL variable and
shows kernel path in the console output. This can confuse users.

Changed runqemu and runqemu-internal code to avoid setting KERNEL
variable and show kernel path.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu          | 2 +-
 scripts/runqemu-internal | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 4bb3bb6..0830901 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -454,7 +454,7 @@ if [ -e "$ROOTFS" -a -z "$FSTYPE" ]; then
     fi
 fi
 
-if [ -z "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" ]; then
+if [ -z "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" -a "x$FSTYPE" != "xwic" ]; then
     setup_path_vars 1
     eval kernel_file=\$${machine2}_DEFAULT_KERNEL
     KERNEL=$DEPLOY_DIR_IMAGE/$kernel_file
diff --git a/scripts/runqemu-internal b/scripts/runqemu-internal
index 8afcccf..dfffd4d 100755
--- a/scripts/runqemu-internal
+++ b/scripts/runqemu-internal
@@ -302,7 +302,7 @@ case "$MACHINE" in
     ;;
 esac
 
-if [ ! -f "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" ]; then
+if [ ! -f "$KERNEL" -a "x$FSTYPE" != "xvmdk" -a "x$FSTYPE" != "xhddimg" -a "x$FSTYPE" != "xhdddirect" -a "x$FSTYPE" != "xwic" ]; then
     echo "Error: Kernel image file $KERNEL doesn't exist"
     cleanup
     return 1
-- 
2.1.4

[wic][PATCH 04/15] runqemu: support path/to/<image>-<machine>.wic

[Ed Bartosh]

Supported providing wic image path to runqemu:
  runquemu path/to/<image>-<machine>.wic

[YOCTO #8691]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/runqemu | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 0830901..48b7551 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -47,6 +47,7 @@ usage() {
     echo "  $MYNAME qemux86 qemuparams=\"-m 256\""
     echo "  $MYNAME qemux86 bootparams=\"psplash=false\""
     echo "  $MYNAME path/to/<image>-<machine>.vmdk"
+    echo "  $MYNAME path/to/<image>-<machine>.wic"
     exit 1
 }
 
@@ -88,7 +89,7 @@ process_filename() {
 		[ -z "$KERNEL" ] && KERNEL=$filename || \
 		    error "conflicting KERNEL args [$KERNEL] and [$filename]"
 		;;
-	/ext[234]/|/jffs2/|/btrfs/)
+	/ext[234]/|/jffs2/|/btrfs/|/wic/)
 		# A file ending in a supportted fs type is a rootfs image
 		if [ -z "$FSTYPE" -o "$FSTYPE" = "$EXT" ]; then
 		    FSTYPE=$EXT
@@ -235,7 +236,7 @@ elif [ ! -w /dev/net/tun ] ; then
 fi
 
 # Report errors for missing combinations of options
-if [ -z "$MACHINE" -a -z "$KERNEL" -a -z "$VM" ]; then
+if [ -z "$MACHINE" -a -z "$KERNEL" -a -z "$VM" -a "$FSTYPE" != "wic" ]; then
     error "you must specify at least a MACHINE, VM, or KERNEL argument"
 fi
 if [ "$FSTYPE" = "nfs" -a -z "$ROOTFS" ]; then
@@ -243,8 +244,9 @@ if [ "$FSTYPE" = "nfs" -a -z "$ROOTFS" ]; then
 fi
 
 if [ -z "$MACHINE" ]; then
-    if [ "x$FSTYPE" = "xvmdk" ] || [ "x$FSTYPE" = "xhddimg" ] || [ "x$FSTYPE" = "xhdddirect" ]; then
-        MACHINE=`basename $VM | sed -n 's/.*\(qemux86-64\|qemux86\|qemuarm64\|qemuarm\|qemumips64\|qemumips\|qemuppc\|qemush4\).*/\1/p'`
+    if [ "x$FSTYPE" = "xvmdk" ] || [ "x$FSTYPE" = "xhddimg" ] || [ "x$FSTYPE" = "xhdddirect" ] || [ "x$FSTYPE" = "xwic" ]; then
+        [ "x$FSTYPE" = "xwic" ] && filename=$ROOTFS || filename=$VM
+        MACHINE=`basename $filename | sed -n 's/.*\(qemux86-64\|qemux86\|qemuarm64\|qemuarm\|qemumips64\|qemumips\|qemuppc\|qemush4\).*/\1/p'`
         if [ -z "$MACHINE" ]; then
             error "Unable to set MACHINE from image filename [$VM]"
         fi
-- 
2.1.4

[wic][PATCH 05/15] oe-selftest: remove unused parameter

[Ed Bartosh]

From: Ed Bartosh <bartosh@gmail.com>

Removed unused parameter 'test' from runqemu function.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py       | 2 +-
 meta/lib/oeqa/selftest/imagefeatures.py | 6 +++---
 meta/lib/oeqa/utils/commands.py         | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index d95cb08..623662b 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -987,7 +987,7 @@ class DevtoolTests(DevtoolBase):
         result = runCmd('devtool deploy-target -n %s root@localhost' % testrecipe)
         self.assertIn('  %s' % testfile, result.output)
         # Boot the image
-        with runqemu(testimage, self) as qemu:
+        with runqemu(testimage) as qemu:
             # Now really test deploy-target
             result = runCmd('devtool deploy-target -c %s root@%s' % (testrecipe, qemu.ip))
             # Run a test command to see if it was installed properly
diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 62ddc52..1c08ce2 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -31,7 +31,7 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             # Attempt to ssh with each user into qemu with empty password
             for user in [self.root_user, self.test_user]:
                 ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
@@ -58,7 +58,7 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-minimal
         bitbake('core-image-minimal')
 
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             # Attempt to ssh with each user into qemu with empty password
             for user in [self.root_user, self.test_user]:
                 ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user=user)
@@ -112,7 +112,7 @@ class ImageFeatures(oeSelfTest):
                 self.fail('No rpm package found in image')
 
         # Now do a couple of runtime tests
-        with runqemu("core-image-minimal", self) as qemu:
+        with runqemu("core-image-minimal") as qemu:
             command = "rpm --version"
             status, output = qemu.run(command)
             self.assertEqual(0, status, 'Failed to run command "%s": %s' % (command, output))
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 08e2cbb..48f00f5 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -177,7 +177,7 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
 
 
 @contextlib.contextmanager
-def runqemu(pn, test):
+def runqemu(pn):
 
     import bb.tinfoil
     import bb.build
-- 
2.1.4

[wic][PATCH 06/15] qemurunner: add parameter to method 'start'

[Ed Bartosh]

QemuRunner requires pair of ip addresses provided through kernel
commandline for method 'start' to work. These ip addresses are
used to connect to the image using ssh and run tests there.
However, this functionality should not be mandatory as testing
doesn't always require ssh connection. Some tests can be run using
serial console.

Added new parameter 'get_ip' to QemuRunner.start to make it possible
to skip getting pair of ip addresses from kernel command line. This
should allow oe-selftest to test images without modifying kernel
command line.

[YOCTO #8498]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 45 ++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index def8ace..784cf96 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -91,7 +91,7 @@ class QemuRunner:
                 self._dump_host()
                 raise SystemExit
 
-    def start(self, qemuparams = None):
+    def start(self, qemuparams = None, get_ip = True):
         if self.display:
             os.environ["DISPLAY"] = self.display
             # Set this flag so that Qemu doesn't do any grabs as SDL grabs
@@ -178,27 +178,28 @@ class QemuRunner:
 
         if self.is_alive():
             logger.info("qemu started - qemu procces pid is %s" % self.qemupid)
-            cmdline = ''
-            with open('/proc/%s/cmdline' % self.qemupid) as p:
-                cmdline = p.read()
-                # It is needed to sanitize the data received
-                # because is possible to have control characters
-                cmdline = re_control_char.sub('', cmdline)
-            try:
-                ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
-                if not ips or len(ips) != 3:
-                    raise ValueError
-                else:
-                    self.ip = ips[0]
-                    self.server_ip = ips[1]
-            except IndexError, ValueError:
-                logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used:\n%s\nand output from runqemu:\n%s" % (cmdline, self.getOutput(output)))
-                self._dump_host()
-                self.stop()
-                return False
-            logger.info("qemu cmdline used:\n{}".format(cmdline))
-            logger.info("Target IP: %s" % self.ip)
-            logger.info("Server IP: %s" % self.server_ip)
+            if get_ip:
+                cmdline = ''
+                with open('/proc/%s/cmdline' % self.qemupid) as p:
+                    cmdline = p.read()
+                    # It is needed to sanitize the data received
+                    # because is possible to have control characters
+                    cmdline = re_control_char.sub('', cmdline)
+                try:
+                    ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
+                    if not ips or len(ips) != 3:
+                        raise ValueError
+                    else:
+                        self.ip = ips[0]
+                        self.server_ip = ips[1]
+                except IndexError, ValueError:
+                    logger.info("Couldn't get ip from qemu process arguments! Here is the qemu command line used:\n%s\nand output from runqemu:\n%s" % (cmdline, self.getOutput(output)))
+                    self._dump_host()
+                    self.stop()
+                    return False
+                logger.info("qemu cmdline used:\n{}".format(cmdline))
+                logger.info("Target IP: %s" % self.ip)
+                logger.info("Server IP: %s" % self.server_ip)
 
             self.thread = LoggingThread(self.log, threadsock, logger)
             self.thread.start()
-- 
2.1.4

[wic][PATCH 07/15] oeqa/targetcontrol: make ssh control optional

[Ed Bartosh]

Added new parameter 'ssh' to targetcontrol 'start' method
to be able to test images without running ssh server.

[YOCTO #8498]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py  | 18 ++++++++++--------
 meta/lib/oeqa/utils/commands.py |  4 ++--
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index edc0d01..f461651 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -68,7 +68,7 @@ class BaseTarget(object):
         bb.note("SSH log file: %s" %  self.sshlog)
 
     @abstractmethod
-    def start(self, params=None):
+    def start(self, params=None, ssh=True):
         pass
 
     @abstractmethod
@@ -176,11 +176,12 @@ class QemuTarget(BaseTarget):
         bb.note("Qemu log file: %s" % self.qemulog)
         super(QemuTarget, self).deploy()
 
-    def start(self, params=None):
-        if self.runner.start(params):
-            self.ip = self.runner.ip
-            self.server_ip = self.runner.server_ip
-            self.connection = SSHControl(ip=self.ip, logfile=self.sshlog)
+    def start(self, params=None, ssh=True):
+        if self.runner.start(params, get_ip=ssh):
+            if ssh:
+                self.ip = self.runner.ip
+                self.server_ip = self.runner.server_ip
+                self.connection = SSHControl(ip=self.ip, logfile=self.sshlog)
         else:
             self.stop()
             if os.path.exists(self.qemulog):
@@ -231,8 +232,9 @@ class SimpleRemoteTarget(BaseTarget):
     def deploy(self):
         super(SimpleRemoteTarget, self).deploy()
 
-    def start(self, params=None):
-        self.connection = SSHControl(self.ip, logfile=self.sshlog, port=self.port)
+    def start(self, params=None, ssh=True):
+        if ssh:
+            self.connection = SSHControl(self.ip, logfile=self.sshlog, port=self.port)
 
     def stop(self):
         self.connection = None
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 48f00f5..c60294d 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -177,7 +177,7 @@ def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec=
 
 
 @contextlib.contextmanager
-def runqemu(pn):
+def runqemu(pn, ssh=True):
 
     import bb.tinfoil
     import bb.build
@@ -211,7 +211,7 @@ def runqemu(pn):
     try:
         qemu.deploy()
         try:
-            qemu.start()
+            qemu.start(ssh=ssh)
         except bb.build.FuncFailed:
             raise Exception('Failed to start QEMU - see the logs in %s' % logdir)
 
-- 
2.1.4

[wic][PATCH 08/15] oeqa/targetcontrol: support wic image type

[Ed Bartosh]

Added 'wic' to the list image types supported by targetcontrol.

This is a preparation for booting and testing wic images
with oe-selftest.

[YOCTO #8498]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index f461651..5422a61 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -113,7 +113,7 @@ class BaseTarget(object):
 
 class QemuTarget(BaseTarget):
 
-    supported_image_fstypes = ['ext3', 'ext4', 'cpio.gz']
+    supported_image_fstypes = ['ext3', 'ext4', 'cpio.gz', 'wic']
 
     def __init__(self, d):
 
-- 
2.1.4

[wic][PATCH 09/15] wic-image-minimal: change IMAGE_FSTYPES

[Ed Bartosh]

Replaced wic.bz2 -> wic in IMAGE_FSTYPES.

wic-image-minimal is going to be booted in qemu and tested.
As runqemu doesn't support compressed images this recipe needs
to produce raw wic image.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb b/meta-selftest/recipes-test/images/wic-image-minimal.bb
index 95c7a9b..ced22c8 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.bb
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
@@ -4,7 +4,7 @@ SRC_URI = "file://${FILE_DIRNAME}/${BPN}.wks"
 
 IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 
-IMAGE_FSTYPES = "wic.bz2"
+IMAGE_FSTYPES = "wic"
 RM_OLD_IMAGE = "1"
 
 DEPENDS = "syslinux syslinux-native parted-native dosfstools-native mtools-native"
-- 
2.1.4

[wic][PATCH 10/15] wic-image-minimal: updated .wks to boot by qemu

[Ed Bartosh]

With below list of changes wic-image-minimal can be boot by qemu:
    - Changed device name sda -> hda.
    - Added root=/dev/hda2 to the kernel command line.
    - Changed mount point for core partition to /mnt to have it
      mounted on boot time.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.wks b/meta-selftest/recipes-test/images/wic-image-minimal.wks
index 8f9be09..6501a82 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.wks
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.wks
@@ -2,9 +2,9 @@
 # long-description: This image contains boot partition and 3 rootfs partitions
 # created from core-image-minimal and wic-image-minimal image recipes.
 
-part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
-part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext4 --label core --align 1024
-part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext4 --label backup --align 1024
+part /boot --source bootimg-pcbios --ondisk hda --label boot --active --align 1024
+part / --source rootfs --ondisk hda --fstype=ext4 --label platform --align 1024
+part /mnt --source rootfs --rootfs-dir=core-image-minimal --ondisk hda --fstype=ext4 --label core --align 1024
+part backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk hda --fstype=ext4 --label backup --align 1024
 
-bootloader  --timeout=0  --append="rootwait console=tty0"
+bootloader --timeout=0  --append="rootwait console=tty0 root=/dev/hda2"
-- 
2.1.4

[wic][PATCH 11/15] oe-selftest: add new wic testcase

[Ed Bartosh]

Added test_qemu testcase to boot wic-image-minimal and test
that 2 partitions mentioned in .wks are mounted.

[YOCTO #8499]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index df826ed..4985714 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -30,7 +30,7 @@ from glob import glob
 from shutil import rmtree
 
 from oeqa.selftest.base import oeSelfTest
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 from oeqa.utils.decorators import testcase
 
 
@@ -264,3 +264,13 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_qemu(self):
+        """Test wic-image-minimal under qemu"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        with runqemu('wic-image-minimal', ssh=False) as qemu:
+            command = "mount |grep '^/dev/' | cut -f1,3 -d ' '"
+            status, output = qemu.run_serial(command)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (command, output))
+            self.assertEqual(output, '/dev/root /\r\n/dev/hda3 /mnt')
-- 
2.1.4

[wic][PATCH 12/15] wic: fix processing of --use-uuid

[Ed Bartosh]

Processing of this option was lost during recent change of
wks parsting. It was discovered during the work on booting
wic images under qemu. Now, when -use-uuid is fixed it's
possible to specify root partition by partition uuid.
This will be done in the following commit.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/partition.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 9bb1a0a..55bcfba 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -57,7 +57,10 @@ class Partition(object):
         self.size = args.size
         self.source = args.source
         self.sourceparams = args.sourceparams
+        self.use_uuid = args.use_uuid
         self.uuid = args.uuid
+        if args.use_uuid and not self.uuid:
+            self.uuid = str(uuid.uuid4())
 
         self.lineno = lineno
         self.source_file = ""
-- 
2.1.4

[wic][PATCH 13/15] wic-image-minimal: use uuid for root partition

[Ed Bartosh]

Used --uuid option to generate PARTUUID for root partion and
gpt partition type. As this image is used in runtime testing
under qemu it makes sense to ensure that this way of specifying
root partition works.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.wks b/meta-selftest/recipes-test/images/wic-image-minimal.wks
index 6501a82..0d6035e 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.wks
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.wks
@@ -3,8 +3,8 @@
 # created from core-image-minimal and wic-image-minimal image recipes.
 
 part /boot --source bootimg-pcbios --ondisk hda --label boot --active --align 1024
-part / --source rootfs --ondisk hda --fstype=ext4 --label platform --align 1024
+part / --source rootfs --ondisk hda --fstype=ext4 --label platform --align 1024 --use-uuid
 part /mnt --source rootfs --rootfs-dir=core-image-minimal --ondisk hda --fstype=ext4 --label core --align 1024
 part backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk hda --fstype=ext4 --label backup --align 1024
 
-bootloader --timeout=0  --append="rootwait console=tty0 root=/dev/hda2"
+bootloader --ptable gpt --timeout=0 --append="rootwait console=tty0"
-- 
2.1.4

[wic][PATCH 14/15] oe-selftest: pylinted wic tests

[Ed Bartosh]

Fixed the following pylint warnings:
  C: 45, 0: Line too long (91/80) (line-too-long)
  C:105, 0: Wrong continued indentation.
                            ignore_status=True).status)
                            ^         | (bad-continuation)
  W: 83, 8: Redefining built-in 'vars' (redefined-builtin)
  W:175, 8: Redefining built-in 'vars' (redefined-builtin)
  W: 27, 0: Unused import sys (unused-import)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 4985714..964b954 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -24,7 +24,6 @@
 """Test cases for wic."""
 
 import os
-import sys
 
 from glob import glob
 from shutil import rmtree
@@ -42,7 +41,8 @@ class Wic(oeSelfTest):
 
     def setUpLocal(self):
         """This code is executed before each test method."""
-        self.write_config('IMAGE_FSTYPES += " hddimg"\nMACHINE_FEATURES_append = " efi"\n')
+        self.write_config('IMAGE_FSTYPES += " hddimg"\n'
+                          'MACHINE_FEATURES_append = " efi"\n')
 
         # Do this here instead of in setUpClass as the base setUp does some
         # clean up which can result in the native tools built earlier in
@@ -80,14 +80,14 @@ class Wic(oeSelfTest):
     @testcase(1212)
     def test_build_artifacts(self):
         """Test wic create directdisk providing all artifacts."""
-        vars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
+        bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
                         for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE',
                                     'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS'))
         status = runCmd("wic create directdisk "
                         "-b %(staging_datadir)s "
                         "-k %(deploy_dir_image)s "
                         "-n %(staging_dir_native)s "
-                        "-r %(image_rootfs)s" % vars).status
+                        "-r %(image_rootfs)s" % bbvars).status
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct")))
 
@@ -102,7 +102,7 @@ class Wic(oeSelfTest):
     def test_unsupported_subcommand(self):
         """Test unsupported subcommand"""
         self.assertEqual(1, runCmd('wic unsupported',
-                         ignore_status=True).status)
+                                   ignore_status=True).status)
 
     @testcase(1214)
     def test_no_command(self):
@@ -172,20 +172,20 @@ class Wic(oeSelfTest):
     @testcase(1269)
     def test_rootfs_artifacts(self):
         """Test usage of rootfs plugin with rootfs paths"""
-        vars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
+        bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
                         for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE',
                                     'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS'))
-        vars['wks'] = "directdisk-multi-rootfs"
+        bbvars['wks'] = "directdisk-multi-rootfs"
         status = runCmd("wic create %(wks)s "
                         "-b %(staging_datadir)s "
                         "-k %(deploy_dir_image)s "
                         "-n %(staging_dir_native)s "
                         "--rootfs-dir rootfs1=%(image_rootfs)s "
                         "--rootfs-dir rootfs2=%(image_rootfs)s" \
-                        % vars).status
+                        % bbvars).status
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + \
-                                     "%(wks)s-*.direct" % vars)))
+                                     "%(wks)s-*.direct" % bbvars)))
 
     @testcase(1346)
     def test_iso_image(self):
-- 
2.1.4

[wic][PATCH 15/15] oe-selftest: don't use specific tasks

[Ed Bartosh]

Image should be build usual way, i.e. using bitbake <image>.
Specifying do_mage_complete and do_rootfs_wicenv tasks should not
be needed anymore as those tasks should be run by bitbake for
every image.

Removed specifying do_mage_complete and do_rootfs_wicenv tasks
from bitbake calls.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 964b954..f500f1c 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -50,7 +50,7 @@ class Wic(oeSelfTest):
         if not Wic.image_is_ready:
             bitbake('syslinux syslinux-native parted-native gptfdisk-native '
                     'dosfstools-native mtools-native')
-            bitbake('core-image-minimal:do_image_complete core-image-minimal:do_rootfs_wicenv')
+            bitbake('core-image-minimal')
             Wic.image_is_ready = True
 
         rmtree(self.resultdir, ignore_errors=True)
-- 
2.1.4

Re: [wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Ed Bartosh]

On Tue, Feb 02, 2016 at 02:50:19PM +0000, Richard Purdie wrote:
> On Tue, 2016-02-02 at 09:27 +0200, Ed Bartosh wrote:
> > qemu hands with 'waiting for root device' message during the
> > boot of wic images if '-drive if=virtio' option is used.
> > 
> > Removed if=virtio from qemu command line for wic images. This
> > should fix above mentioned boot issue.
> 
> Does this not change the command for other image types too?
No, it doesn't. This patch changes qemu command line only for wic images.

--
Regards,
Ed

Re: [wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Ed Bartosh]

On Tue, Feb 02, 2016 at 03:57:08PM +0000, Richard Purdie wrote:
> On Tue, 2016-02-02 at 14:55 +0200, Ed Bartosh wrote:
> > On Tue, Feb 02, 2016 at 02:50:19PM +0000, Richard Purdie wrote:
> > > On Tue, 2016-02-02 at 09:27 +0200, Ed Bartosh wrote:
> > > > qemu hands with 'waiting for root device' message during the
> > > > boot of wic images if '-drive if=virtio' option is used.
> > > > 
> > > > Removed if=virtio from qemu command line for wic images. This
> > > > should fix above mentioned boot issue.
> > > 
> > > Does this not change the command for other image types too?
> > No, it doesn't. This patch changes qemu command line only for wic
> > images.
> 
> Sorry, somehow I didn't see the second line in that diff which adds it
> back.
> 
> I do worry about creating too many codepaths in this code though. Any
> idea why virtio doesn't work?

I'll look at it and update patchset. Thanks for review!

--
Regards,
Ed

Re: [wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Richard Purdie]

On Tue, 2016-02-02 at 09:27 +0200, Ed Bartosh wrote:
> qemu hands with 'waiting for root device' message during the
> boot of wic images if '-drive if=virtio' option is used.
> 
> Removed if=virtio from qemu command line for wic images. This
> should fix above mentioned boot issue.

Does this not change the command for other image types too? What side
effects might that have?

Cheers,

Richard

Re: [wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

On 2 February 2016 at 12:55, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:

> > > Removed if=virtio from qemu command line for wic images. This
> > > should fix above mentioned boot issue.
> >
> > Does this not change the command for other image types too?
> No, it doesn't. This patch changes qemu command line only for wic images.
>

Any idea why?  virtio is better, surely?

Ross

[-- Attachment #2: Type: text/html, Size: 805 bytes --]

Re: [wic][PATCH 02/15] runqemu: don't use virtio for wic images

[Richard Purdie]

On Tue, 2016-02-02 at 14:55 +0200, Ed Bartosh wrote:
> On Tue, Feb 02, 2016 at 02:50:19PM +0000, Richard Purdie wrote:
> > On Tue, 2016-02-02 at 09:27 +0200, Ed Bartosh wrote:
> > > qemu hands with 'waiting for root device' message during the
> > > boot of wic images if '-drive if=virtio' option is used.
> > > 
> > > Removed if=virtio from qemu command line for wic images. This
> > > should fix above mentioned boot issue.
> > 
> > Does this not change the command for other image types too?
> No, it doesn't. This patch changes qemu command line only for wic
> images.

Sorry, somehow I didn't see the second line in that diff which adds it
back.

I do worry about creating too many codepaths in this code though. Any
idea why virtio doesn't work?

Cheers,

Richard