[RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[Stefan Stanacar]

Hello,

This is the proposed implementation of a new runtime tests framework based on python unittest. 
It's mean to ease qemu image testing and encourage developers to add more tests similar to the example tests provided (all tests are basically commands ran over ssh)

You can try it out like this:
 - first build a qemu core-image-sato (a minimal wouldn't be interesting at all)
 - add INHERIT += "testimage" in local.conf
 - then bitbake core-image-sato -c testimage. That will run a standard suite of tests.

You can set TEST_SUITES = "ping ssh <test name>" in local.conf to force run only certain tests (order matters here, it's the order in which tests run). You can also append "auto" and it will also run whatever tests are suitable for the image (if that was a sato-sdk image and you set TEST_SUITE = "ping ssh rpm auto" you force run ping, ssh and rpm but you also get smart, connman and gcc tests).

Check the task log (log.do_testimage) in WORKDIR to see the results. Also a ssh log (what command is running, output and return codes) and qemu boot log are kept in WORKDIR/testimage/

There are some areas for improvement/limitations right now:
 - a better way of getting the list of installed packages in a image
 - qemu is started with the -snapshot option, we should create a copy of the original rootfs instead.
 - when using TEST_SUITES = "auto" there is no dependency at all between tests (rpm would run before ssh test), so better use TEST_SUITES = "ping ssh auto"
 - better logging and error reporting for qemu problems (runqemu erros, qemu doesn't start, etc)
 - more helper methods for tests
 - others I can't think of right now :)

I hope this is going in the right direction and we could improve from here.
Comments and feedback are most welcome!

Regards,
Stefan



The following changes since commit 042203531b10b37ac9a8201b376f5dec403e51d8:

  sanity.bbclass: Fix COREBASE sanity tests (2013-06-27 12:48:56 +0100)

are available in the git repository at:

  git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib stefans/oeqa5

for you to fetch changes up to e6e567982f4b28822afdc8ea55401513df7c9466:

  lib/oeqa/runtime: add gcc test (2013-06-28 13:00:45 +0300)

----------------------------------------------------------------
Radu Moisan (2):
      lib/oeqa/utils/qemurunner.py: class to handle qemu instance
      lib/oeqa/utils/decorators.py: decorators for test methods

Stefan Stanacar (6):
      classes/testimage.bbclass: new class for image tests
      lib/oeqa/oetest.py: base module for all runtime unittests
      lib/oeqa/utils/sshcontrol.py: helper module for running remote commands
      lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix domain sockets
      lib/oeqa/runtime: image sanity tests
      lib/oeqa/runtime: add gcc test

 meta/classes/testimage.bbclass           |  95 ++++++++++++++++++
 meta/lib/oeqa/__init__.py                |   0
 meta/lib/oeqa/oetest.py                  |  94 ++++++++++++++++++
 meta/lib/oeqa/runtime/__init__.py        |   0
 meta/lib/oeqa/runtime/connman.py         |  34 +++++++
 meta/lib/oeqa/runtime/dmesg.py           |  10 ++
 meta/lib/oeqa/runtime/files/test.c       |  26 +++++
 meta/lib/oeqa/runtime/files/testmakefile |   5 +
 meta/lib/oeqa/runtime/gcc.py             |  36 +++++++
 meta/lib/oeqa/runtime/multilib.py        |  14 +++
 meta/lib/oeqa/runtime/ping.py            |  11 +++
 meta/lib/oeqa/runtime/rpm.py             |  25 +++++
 meta/lib/oeqa/runtime/smart.py           |  23 +++++
 meta/lib/oeqa/runtime/ssh.py             |  16 ++++
 meta/lib/oeqa/utils/__init__.py          |   0
 meta/lib/oeqa/utils/decorators.py        |  40 ++++++++
 meta/lib/oeqa/utils/oetelnetlib.py       |  49 ++++++++++
 meta/lib/oeqa/utils/qemurunner.py        | 160 +++++++++++++++++++++++++++++++
 meta/lib/oeqa/utils/sshcontrol.py        | 100 +++++++++++++++++++
 scripts/runqemu                          |   2 +-
 20 files changed, 739 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/testimage.bbclass
 create mode 100644 meta/lib/oeqa/__init__.py
 create mode 100644 meta/lib/oeqa/oetest.py
 create mode 100644 meta/lib/oeqa/runtime/__init__.py
 create mode 100644 meta/lib/oeqa/runtime/connman.py
 create mode 100644 meta/lib/oeqa/runtime/dmesg.py
 create mode 100644 meta/lib/oeqa/runtime/files/test.c
 create mode 100644 meta/lib/oeqa/runtime/files/testmakefile
 create mode 100644 meta/lib/oeqa/runtime/gcc.py
 create mode 100644 meta/lib/oeqa/runtime/multilib.py
 create mode 100644 meta/lib/oeqa/runtime/ping.py
 create mode 100644 meta/lib/oeqa/runtime/rpm.py
 create mode 100644 meta/lib/oeqa/runtime/smart.py
 create mode 100644 meta/lib/oeqa/runtime/ssh.py
 create mode 100644 meta/lib/oeqa/utils/__init__.py
 create mode 100644 meta/lib/oeqa/utils/decorators.py
 create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py
 create mode 100644 meta/lib/oeqa/utils/qemurunner.py
 create mode 100644 meta/lib/oeqa/utils/sshcontrol.py

Radu Moisan (2):
  lib/oeqa/utils/qemurunner.py: class to handle qemu instance
  lib/oeqa/utils/decorators.py: decorators for test methods

Stefan Stanacar (6):
  classes/testimage.bbclass: new class for image tests
  lib/oeqa/oetest.py: base module for all runtime unittests
  lib/oeqa/utils/sshcontrol.py: helper module for running remote
    commands
  lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix
    domain sockets
  lib/oeqa/runtime: image sanity tests
  lib/oeqa/runtime: add gcc test

 meta/classes/testimage.bbclass           |  95 ++++++++++++++++++
 meta/lib/oeqa/__init__.py                |   0
 meta/lib/oeqa/oetest.py                  |  94 ++++++++++++++++++
 meta/lib/oeqa/runtime/__init__.py        |   0
 meta/lib/oeqa/runtime/connman.py         |  34 +++++++
 meta/lib/oeqa/runtime/dmesg.py           |  10 ++
 meta/lib/oeqa/runtime/files/test.c       |  26 +++++
 meta/lib/oeqa/runtime/files/testmakefile |   5 +
 meta/lib/oeqa/runtime/gcc.py             |  36 +++++++
 meta/lib/oeqa/runtime/multilib.py        |  14 +++
 meta/lib/oeqa/runtime/ping.py            |  11 +++
 meta/lib/oeqa/runtime/rpm.py             |  25 +++++
 meta/lib/oeqa/runtime/smart.py           |  23 +++++
 meta/lib/oeqa/runtime/ssh.py             |  16 ++++
 meta/lib/oeqa/utils/__init__.py          |   0
 meta/lib/oeqa/utils/decorators.py        |  40 ++++++++
 meta/lib/oeqa/utils/oetelnetlib.py       |  49 ++++++++++
 meta/lib/oeqa/utils/qemurunner.py        | 160 +++++++++++++++++++++++++++++++
 meta/lib/oeqa/utils/sshcontrol.py        | 100 +++++++++++++++++++
 scripts/runqemu                          |   2 +-
 20 files changed, 739 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/testimage.bbclass
 create mode 100644 meta/lib/oeqa/__init__.py
 create mode 100644 meta/lib/oeqa/oetest.py
 create mode 100644 meta/lib/oeqa/runtime/__init__.py
 create mode 100644 meta/lib/oeqa/runtime/connman.py
 create mode 100644 meta/lib/oeqa/runtime/dmesg.py
 create mode 100644 meta/lib/oeqa/runtime/files/test.c
 create mode 100644 meta/lib/oeqa/runtime/files/testmakefile
 create mode 100644 meta/lib/oeqa/runtime/gcc.py
 create mode 100644 meta/lib/oeqa/runtime/multilib.py
 create mode 100644 meta/lib/oeqa/runtime/ping.py
 create mode 100644 meta/lib/oeqa/runtime/rpm.py
 create mode 100644 meta/lib/oeqa/runtime/smart.py
 create mode 100644 meta/lib/oeqa/runtime/ssh.py
 create mode 100644 meta/lib/oeqa/utils/__init__.py
 create mode 100644 meta/lib/oeqa/utils/decorators.py
 create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py
 create mode 100644 meta/lib/oeqa/utils/qemurunner.py
 create mode 100644 meta/lib/oeqa/utils/sshcontrol.py

-- 
1.8.1.4

[RFC PATCH 1/8] classes/testimage.bbclass: new class for image tests

[Stefan Stanacar]

Replacement class for imagetest-qemu.bbclass. It launches a qemu instance and
runs test modules defined in TEST_SUITES.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Radu Moisan <radu.moisan@intel.com>
---
 meta/classes/testimage.bbclass | 95 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
 create mode 100644 meta/classes/testimage.bbclass

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
new file mode 100644
index 0000000..82d0491
--- /dev/null
+++ b/meta/classes/testimage.bbclass
@@ -0,0 +1,95 @@
+TEST_LOG_DIR ?= "${WORKDIR}/testimage"
+
+DEFAULT_TEST_SUITES = "ping ssh auto"
+DEFAULT_TEST_SUITES_pn-core-image-minimal = "ping"
+DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh connman rpm smart dmesg"
+DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "ping ssh connman rpm smart gcc dmesg"
+
+TEST_SUITES ?= "${DEFAULT_TEST_SUITES}"
+
+python do_testimage() {
+    testimage_main(d)
+}
+addtask testimage
+do_testimage[nostamp] = "1"
+do_testimage[depends] += "qemu-native:do_populate_sysroot"
+do_testimage[depends] += "qemu-helper-native:do_populate_sysroot"
+
+def testimage_main(d):
+    import unittest
+    import os
+    import oeqa.runtime
+    import re
+    from oeqa.oetest import createTestContext
+    from oeqa.utils.sshcontrol import SSHControl
+    from oeqa.utils.qemurunner import QemuRunner
+
+    testdir = d.getVar("TEST_LOG_DIR", True)
+    bb.utils.mkdirhier(testdir)
+    sshlog = os.path.join(testdir, "ssh_target_log.%s" % d.getVar('DATETIME', True))
+    sshloglink = os.path.join(testdir, "ssh_target_log")
+    if os.path.islink(sshloglink):
+        os.unlink(sshloglink)
+    os.symlink(sshlog, sshloglink)
+
+    # testsuite are modules - they are what the user wants (required tests)
+    testsuites = d.getVar("TEST_SUITES", True)
+    # testslist is what we'll run - order matters
+    # perhaps let the user decide if it wants the ssh test or not - it should be the first one though
+    #testslist = [ "ssh" ]
+    testslist = [ x for x in testsuites.split() if x != "auto" ]
+    # if we have auto search for other modules
+    if "auto" in testsuites:
+        for f in os.listdir(os.path.dirname(os.path.abspath(oeqa.runtime.__file__))):
+            if f.endswith('.py') and not f.startswith('_')  and f[:-3] not in testslist:
+                testslist.append(f[:-3])
+
+    testslist = [ "oeqa.runtime." + x for x in testslist ]
+
+    bb.note("Test modules  %s" % testslist)
+
+    class TestContext:
+        def __init__(self):
+            self.d = d
+            self.testslist = testslist
+            self.testsrequired = testsuites.split()
+            self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
+
+    # test context
+    tc = TestContext()
+
+    # prepare qemu instance
+    # and boot each supported fs type
+    machine=d.getVar("MACHINE", True)
+    #will handle fs type eventually, stick with ext3 for now
+    rootfs=d.getVar("DEPLOY_DIR_IMAGE", True) + '/' + d.getVar("IMAGE_BASENAME",True) + '-' + machine + '.ext3'
+
+    qemu = QemuRunner(machine, rootfs)
+    qemu.tmpdir = d.getVar("TMPDIR", True)
+    qemu.display = d.getVar("BB_ORIGENV", False).getVar("DISPLAY", True)
+    qemu.logfile = os.path.join(testdir, "qemu_boot_log.%s" % d.getVar('DATETIME', True))
+
+    #catch exceptions when loading or running tests (mostly our own errors)
+    try:
+        if qemu.launch():
+
+            # prepare test suite, loader and runner
+            suite = unittest.TestSuite()
+            runner = unittest.TextTestRunner(verbosity=2)
+            # set more context - ssh instance and qemu - we do this here because we needed the qemu ip
+            tc.qemu = qemu
+            tc.target = SSHControl(host=qemu.ip,logfile=sshlog)
+            # create suite
+            suite = createTestContext(tc)
+            bb.note("Found %s tests" % suite.countTestCases())
+            result = runner.run(suite)
+
+            if result.wasSuccessful():
+                bb.note("All required tests passed")
+            else:
+                raise bb.build.FuncFailed("Some tests failed. You should check the task log and the ssh log. (ssh log is %s" % sshlog)
+
+        else:
+            raise bb.build.FuncFailed("Failed to start qemu. You should check the task log and the qemu boot log (qemu log is %s)" % qemu.logfile)
+    finally:
+        qemu.kill()
-- 
1.8.1.4

[RFC PATCH 2/8] lib/oeqa/oetest.py: base module for all runtime unittests

[Stefan Stanacar]

This module contains the base class for all runtime tests
and some helper methods.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Radu Moisan <radu.moisan@intel.com>
---
 meta/lib/oeqa/__init__.py |  0
 meta/lib/oeqa/oetest.py   | 94 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)
 create mode 100644 meta/lib/oeqa/__init__.py
 create mode 100644 meta/lib/oeqa/oetest.py

diff --git a/meta/lib/oeqa/__init__.py b/meta/lib/oeqa/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
new file mode 100644
index 0000000..d59109e
--- /dev/null
+++ b/meta/lib/oeqa/oetest.py
@@ -0,0 +1,94 @@
+import os, re, mmap
+import unittest
+import inspect
+import bb
+from oeqa.utils.sshcontrol import SSHControl
+
+
+def createTestContext(tc):
+
+    # set the context object passed from the test class
+    setattr(oeRuntimeTest, "tc", tc)
+
+    # load the tests and return the suite to the runner
+    testloader = unittest.TestLoader()
+    testloader.sortTestMethodsUsing = None
+    suite = unittest.TestSuite()
+    suite = testloader.loadTestsFromNames(tc.testslist)
+
+    return suite
+
+
+
+class oeRuntimeTest(unittest.TestCase):
+    testFailures = []
+    testSkipped = []
+    testErrors = []
+    pscmd = "ps"
+
+    def __init__(self, methodName='runTest'):
+        self.target = oeRuntimeTest.tc.target
+        super(oeRuntimeTest, self).__init__(methodName)
+
+
+    def run(self, result=None):
+        super(oeRuntimeTest, self).run(result)
+
+        # we add to our own lists the results, we use those for decorators
+        if len(result.failures) > len(oeRuntimeTest.testFailures):
+            oeRuntimeTest.testFailures.append(str(result.failures[-1][0]).split()[0])
+        if len(result.skipped) > len(oeRuntimeTest.testSkipped):
+            oeRuntimeTest.testSkipped.append(str(result.skipped[-1][0]).split()[0])
+        if len(result.errors) > len(oeRuntimeTest.testErrors):
+            oeRuntimeTest.testErrors.append(str(result.errors[-1][0]).split()[0])
+
+    @classmethod
+    def hasPackage(self, pkg):
+
+        pkgfile = os.path.join(oeRuntimeTest.tc.d.getVar("WORKDIR", True), "installed_pkgs.txt")
+
+        with open(pkgfile) as f:
+            data = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
+            match = re.search(pkg, data)
+            data.close()
+
+        if match:
+            return True
+
+        return False
+
+    @classmethod
+    def hasFeature(self,feature):
+
+        if feature in oeRuntimeTest.tc.d.getVar("IMAGE_FEATURES", True).split():
+            return True
+        else:
+            return False
+
+
+
+
+def getmodule(pos=2):
+    # stack returns a list of tuples containg frame information
+    # First element of the list the is current frame, caller is 1
+    frameinfo = inspect.stack()[pos]
+    modname = inspect.getmodulename(frameinfo[1])
+    #modname = inspect.getmodule(frameinfo[0]).__name__
+    return modname
+
+def skipModule(reason, pos=2):
+    modname = getmodule(pos)
+    if modname not in oeRuntimeTest.tc.testsrequired:
+        raise unittest.SkipTest("%s: %s" % (modname, reason))
+    else:
+        bb.warn("Test %s is required, not skipping" % modname)
+
+def skipModuleIf(cond, reason):
+
+    if cond:
+        skipModule(reason, 3)
+
+def skipModuleUnless(cond, reason):
+
+    if not cond:
+        skipModule(reason, 3)
-- 
1.8.1.4

[RFC PATCH 3/8] lib/oeqa/utils/sshcontrol.py: helper module for running remote commands

[Stefan Stanacar]

Provides a class for setting up ssh connections,
running commands and copying files to/from a target.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/utils/__init__.py   |   0
 meta/lib/oeqa/utils/sshcontrol.py | 100 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 100 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/__init__.py
 create mode 100644 meta/lib/oeqa/utils/sshcontrol.py

diff --git a/meta/lib/oeqa/utils/__init__.py b/meta/lib/oeqa/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/meta/lib/oeqa/utils/sshcontrol.py b/meta/lib/oeqa/utils/sshcontrol.py
new file mode 100644
index 0000000..ebdf7e8
--- /dev/null
+++ b/meta/lib/oeqa/utils/sshcontrol.py
@@ -0,0 +1,100 @@
+import subprocess
+import time
+import os
+import shlex
+
+class SSHControl(object):
+
+    def __init__(self, host=None, timeout=120, logfile=None):
+        self.host = host
+        self.timeout = timeout
+        self._out = ''
+        self._ret = 126
+        self.logfile = logfile
+
+    def log(self, msg):
+        if self.logfile:
+            with open(self.logfile, "a") as f:
+                f.write("%s\n" % msg)
+
+    def _internal_run(self, cmd):
+        # ssh hangs without os.setsid
+        proc = subprocess.Popen(shlex.split(cmd), shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, preexec_fn=os.setsid)
+        return proc
+
+    def run(self, cmd, timeout=None):
+        """Run cmd and get it's return code and output.
+        Let it run for timeout seconds and then terminate/kill it,
+        if time is 0 will let cmd run until it finishes.
+        Time can be passed to here or can be set per class instance."""
+
+
+
+        actualcmd = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root %s '%s'" % (self.host, cmd)
+        if self.host:
+            sshconn = self._internal_run(actualcmd)
+        else:
+            raise Exception("Remote IP hasn't been set: '%s'" % actualcmd)
+
+        if timeout == 0:
+            self.log("[SSH run without timeout]$ %s" % actualcmd)
+            self.log("  # %s" % cmd)
+            self._out = sshconn.communicate()[0]
+            self._ret = sshconn.poll()
+        else:
+            if timeout is None:
+                endtime = time.time() + self.timeout
+            else:
+                endtime = time.time() + timeout
+            while sshconn.poll() is None and time.time() < endtime:
+                time.sleep(0.5)
+            self.log("[SSH run with timeout]$ %s" % actualcmd)
+            self.log("  # %s" % cmd)
+            # process hasn't returned yet
+            if sshconn.poll() is None:
+                self._ret = 255
+                sshconn.terminate()
+                sshconn.kill()
+                self._out = sshconn.stdout.read()
+                sshconn.stdout.close()
+                self.log("[!!! process killed]")
+            else:
+                self._out = sshconn.stdout.read()
+                self._ret = sshconn.poll()
+        # remove first line from output which is always smth like (unless return code is 255 - which is a ssh error):
+        # Warning: Permanently added '192.168.7.2' (RSA) to the list of known hosts.
+        if self._ret != 255:
+            self._out = '\n'.join(self._out.splitlines()[1:])
+        self.log("%s" % self._out)
+        self.log("[SSH command returned]: %s" % self._ret)
+        return (self._ret, self._out)
+
+    def _internal_scp(self, cmd):
+        self.log("[SCP]$ %s" % cmd)
+        scpconn = self._internal_run(cmd)
+        try:
+            self._out = scpconn.communicate()[0]
+            self._ret = scpconn.poll()
+            if self._ret != 0:
+                self.log("%s" % self._out)
+                raise Exception("Error copying file")
+        except Exception as e:
+            print("Execution failed: %s :" % cmd)
+            print e
+        self.log("%s" % self._out)
+        return (self._ret, self._out)
+
+    def copy_to(self, localpath, remotepath):
+        actualcmd = "scp -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no %s root@%s:%s" % (localpath, self.host, remotepath)
+        return self._internal_scp(actualcmd)
+
+
+    def copy_from(self, remotepath, localpath):
+        actualcmd = "scp -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no root@%s:%s %s" % (self.host, remotepath, localpath)
+        return self._internal_scp(actualcmd)
+
+    def get_status(self):
+        return self._ret
+
+    def get_output(self):
+        return self._out
-- 
1.8.1.4

[RFC PATCH 4/8] lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix domain sockets

[Stefan Stanacar]

Python's telnetlib Telnet class connects only to AF_INET sockets, but we
want to use Unix domain socket for the qemu serial connection.
Also add a new read_all_timeout method similar to Telnet's read_all

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/utils/oetelnetlib.py | 49 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py

diff --git a/meta/lib/oeqa/utils/oetelnetlib.py b/meta/lib/oeqa/utils/oetelnetlib.py
new file mode 100644
index 0000000..cdebac1
--- /dev/null
+++ b/meta/lib/oeqa/utils/oetelnetlib.py
@@ -0,0 +1,49 @@
+import socket
+import time
+import re
+from telnetlib import Telnet
+
+class oeTelnet(Telnet):
+
+    """
+    Override Telnet class to use unix domain sockets,
+    Telnet uses AF_INET for socket, we don't want that.
+    Also, provide a read_all variant with timeout, that
+    returns whatever output there is.
+    """
+
+    def __init__(self, stream=None, logfile=None):
+
+        Telnet.__init__(self, host=None)
+        self.stream = stream
+        self.logfile = logfile
+        if stream is not None:
+            self.open(stream)
+
+    def log(self, msg):
+        if self.logfile:
+            with open(self.logfile, "a") as f:
+                f.write("%s\n" % msg)
+
+    def open(self, stream):
+
+        self.eof = 0
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        self.sock.connect(stream)
+
+    def read_all_timeout(self, match, timeout=120):
+        """Read until EOF or until timeout or until match.
+        """
+        ret = False
+        self.process_rawq()
+        endtime = time.time() + timeout
+        while not self.eof and time.time() < endtime:
+            self.fill_rawq()
+            self.process_rawq()
+            if re.search(match, self.cookedq):
+                ret = True
+                break
+        buf = self.cookedq
+        self.cookedq = ''
+        self.log(buf)
+        return (ret, buf)
-- 
1.8.1.4

[RFC PATCH 5/8] lib/oeqa/utils/qemurunner.py: class to handle qemu instance

[Stefan Stanacar]

From: Radu Moisan <radu.moisan@intel.com>

Handles qemu instances (launch, kill, restart, serial connection, logging)
Launch is blocking until login prompt and returns to the task. A qemu
serial connection is used to save the boot log and get the ip from the image.
Changed runqemu script not to error out when using custom serial option.

Signed-off-by: Radu Moisan <radu.moisan@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 160 ++++++++++++++++++++++++++++++++++++++
 scripts/runqemu                   |   2 +-
 2 files changed, 161 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/utils/qemurunner.py

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
new file mode 100644
index 0000000..e201e25
--- /dev/null
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -0,0 +1,160 @@
+import subprocess
+import optparse
+import sys
+import os
+import time
+import signal
+import re
+import bb
+from oeqa.utils.oetelnetlib import oeTelnet
+
+class QemuRunner:
+
+    def __init__(self, machine, rootfs, display = None, tmpdir = None, logfile = None):
+        # Popen object
+        self.runqemu = None
+
+        self.machine = machine
+        self.rootfs = rootfs
+
+        self.streampath = '/tmp/qemuconnection.%s' % os.getpid()
+        self.qemuparams = 'bootparams="console=ttyS0" qemuparams="-snapshot -serial unix:%s,server,nowait"' % self.streampath
+        self.qemupid = None
+        self.ip = None
+
+        self.display = display
+        self.tmpdir = tmpdir
+        self.logfile = logfile
+
+    def launch(self, qemuparams = None):
+
+        if qemuparams:
+            self.qemuparams = self.qemuparams[:-1] + " " + qemuparams + " " + '\"'
+
+        if self.display:
+            os.environ["DISPLAY"] = self.display
+        if not os.path.exists(self.rootfs):
+            bb.error("Invalid rootfs %s" % self.rootfs)
+            return False
+        if not os.path.exists(self.tmpdir):
+            bb.error("Invalid TMPDIR path %s" % self.tmpdir)
+            return False
+        else:
+            os.environ["OE_TMPDIR"] = self.tmpdir
+
+        launch_cmd = 'runqemu %s %s %s' % (self.machine, self.rootfs, self.qemuparams)
+        self.runqemu = subprocess.Popen(launch_cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,preexec_fn=os.setpgrp)
+
+        bb.note("runqemu started, pid is %s" % self.runqemu.pid)
+        # wait at most 30 seconds until qemu pid appears
+        bb.note("waiting at most 30 seconds for qemu pid")
+        endtime = time.time() + 30
+        while not self.is_alive() and time.time() < endtime:
+            time.sleep(0.5)
+
+        if self.is_alive():
+            bb.note("qemu started - qemu procces pid is %s" % self.qemupid)
+
+            console = oeTelnet(self.streampath, self.logfile)
+            bb.note("Waiting at most 120 seconds for login banner")
+            (match, text) = console.read_all_timeout("login:", 120)
+
+            if match:
+                bb.note("Reached login banner")
+                console.write("root\n")
+                (index, match, text) = console.expect([r"(root@[\w-]+:~#)"],10)
+                if not match:
+                    bb.note("Couldn't get prompt, all I got was:\n%s" % match.group(0))
+                    return False
+                console.write("ip addr show eth0 | sed -n '3p' | awk '{ print $2 }' | cut -f 1 -d \"/\"\n")
+                (index, match, text) = console.expect([r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})"],10)
+                console.close()
+                if match:
+                    self.ip = match.group(0)
+                    bb.note("Ip found: %s" % self.ip)
+                else:
+                    bb.note("Couldn't determine ip, all I got was:\n%s" % text)
+                    return False
+            else:
+                console.close()
+                bb.note("Target didn't reached login boot in 120 seconds")
+                lines = "\n".join(text.splitlines()[-5:])
+                bb.note("Last 5 lines of text:\n%s" % lines)
+                bb.note("Check full boot log: %s" % self.logfile)
+                return False
+        else:
+            bb.note("Qemu pid didn't appeared in 30 seconds")
+            self.runqemu.terminate()
+            self.runqemu.kill()
+            bb.note("Output from runqemu: %s " % self.runqemu.stdout.read())
+            self.runqemu.stdout.close()
+            return False
+
+        return self.is_alive()
+
+
+    def kill(self):
+        if self.runqemu:
+            os.kill(-self.runqemu.pid,signal.SIGTERM)
+        self.qemupid = None
+        self.ip = None
+        if os.path.exists(self.streampath):
+            os.remove(self.streampath)
+
+    def restart(self, qemuparams = None):
+        if self.is_alive():
+            self.kill()
+        bb.note("Qemu Restart required...")
+        return self.launch(qemuparams)
+
+    def is_alive(self):
+        qemu_child = self.find_child(str(self.runqemu.pid))
+        if qemu_child:
+            self.qemupid = qemu_child[0]
+            return os.path.exists("/proc/" + str(self.qemupid))
+        return False
+
+    def find_child(self,parent_pid):
+        #
+        # Walk the process tree from the process specified looking for a qemu-system. Return its [pid'cmd]
+        #
+        ps = subprocess.Popen(['ps', 'axww', '-o', 'pid,ppid,command'], stdout=subprocess.PIPE).communicate()[0]
+        processes = ps.split('\n')
+        nfields = len(processes[0].split()) - 1
+        pids = {}
+        commands = {}
+        for row in processes[1:]:
+            data = row.split(None, nfields)
+            if len(data) != 3:
+                continue
+            if data[1] not in pids:
+                pids[data[1]] = []
+
+            pids[data[1]].append(data[0])
+            commands[data[0]] = data[2]
+
+        if parent_pid not in pids:
+            sys.stderr.write("No children found matching %s\n" % parent_pid)
+            return []
+
+        parents = []
+        newparents = pids[parent_pid]
+        while newparents:
+            next = []
+            for p in newparents:
+                if p in pids:
+                    for n in pids[p]:
+                        if n not in parents and n not in next:
+                            next.append(n)
+                if p not in parents:
+                    parents.append(p)
+                    newparents = next
+        #print "Children matching %s:" % str(parents)
+        for p in parents:
+            # Need to be careful here since runqemu-internal runs "ldd qemu-system-xxxx"
+            # Also, old versions of ldd (2.11) run "LD_XXXX qemu-system-xxxx"
+            basecmd = commands[p].split()[0]
+            basecmd = os.path.basename(basecmd)
+            if "qemu-system" in basecmd and "192.168" in commands[p]:
+                return [int(p),commands[p]]
+
diff --git a/scripts/runqemu b/scripts/runqemu
index f2eb2e1..406092b 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -156,7 +156,7 @@ while true; do
             serial_option=`expr "$SCRIPT_QEMU_EXTRA_OPT" : '.*\(-serial\)'`
             kvm_option=`expr "$SCRIPT_QEMU_EXTRA_OPT" : '.*\(-enable-kvm\)'`
             [ ! -z "$serial_option" -o ! -z "$kvm_option" ] && \
-                error "Please use simplified serial or kvm options instead"
+                echo "Please use simplified serial or kvm options instead"
             ;;
         "bootparams="*)
             SCRIPT_KERNEL_OPT="$SCRIPT_KERNEL_OPT ${arg##bootparams=}"
-- 
1.8.1.4

[RFC PATCH 6/8] lib/oeqa/utils/decorators.py: decorators for test methods

[Stefan Stanacar]

From: Radu Moisan <radu.moisan@intel.com>

Some skip decorators meant only for test methods, providing
some kind of test methods dependency.
They are used together with a test method name not a condition.
These are complementary to python's unittest skip decorators.

Signed-off-by: Radu Moisan <radu.moisan@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/utils/decorators.py | 40 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/decorators.py

diff --git a/meta/lib/oeqa/utils/decorators.py b/meta/lib/oeqa/utils/decorators.py
new file mode 100644
index 0000000..4b7a53d
--- /dev/null
+++ b/meta/lib/oeqa/utils/decorators.py
@@ -0,0 +1,40 @@
+from oeqa.oetest import *
+
+class skipIfFailure(object):
+
+    def __init__(self,testcase):
+        self.testcase = testcase
+
+    def __call__(self,f):
+        def wrapped_f(*args):
+            if self.testcase in oeRuntimeTest.testFailures:
+                raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase)
+            f(*args)
+        wrapped_f.__name__ = f.__name__
+        return wrapped_f
+
+class skipIfSkipped(object):
+
+    def __init__(self,testcase):
+        self.testcase = testcase
+
+    def __call__(self,f):
+        def wrapped_f(*args):
+            if self.testcase in oeRuntimeTest.testSkipped:
+                raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase)
+            f(*args)
+        wrapped_f.__name__ = f.__name__
+        return wrapped_f
+
+class skipUnlessPassed(object):
+
+    def __init__(self,testcase):
+        self.testcase = testcase
+
+    def __call__(self,f):
+        def wrapped_f(*args):
+            if self.testcase in (oeRuntimeTest.testSkipped, oeRuntimeTest.testFailures, oeRuntimeTest.testErrors):
+                raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase)
+            f(*args)
+        wrapped_f.__name__ = f.__name__
+        return wrapped_f
-- 
1.8.1.4

[RFC PATCH 7/8] lib/oeqa/runtime: image sanity tests

[Stefan Stanacar]

These are basic sanity tests. A test can be force run by setting
TEST_SUITES = "ping ssh <module-name>" in local.conf.
By default there are suites for minimal, sato and sato-sdk images.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Radu Moisan <radu.moisan@intel.com>
---
 meta/lib/oeqa/runtime/__init__.py |  0
 meta/lib/oeqa/runtime/connman.py  | 34 ++++++++++++++++++++++++++++++++++
 meta/lib/oeqa/runtime/dmesg.py    | 10 ++++++++++
 meta/lib/oeqa/runtime/multilib.py | 14 ++++++++++++++
 meta/lib/oeqa/runtime/ping.py     | 11 +++++++++++
 meta/lib/oeqa/runtime/rpm.py      | 25 +++++++++++++++++++++++++
 meta/lib/oeqa/runtime/smart.py    | 23 +++++++++++++++++++++++
 meta/lib/oeqa/runtime/ssh.py      | 16 ++++++++++++++++
 8 files changed, 133 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/__init__.py
 create mode 100644 meta/lib/oeqa/runtime/connman.py
 create mode 100644 meta/lib/oeqa/runtime/dmesg.py
 create mode 100644 meta/lib/oeqa/runtime/multilib.py
 create mode 100644 meta/lib/oeqa/runtime/ping.py
 create mode 100644 meta/lib/oeqa/runtime/rpm.py
 create mode 100644 meta/lib/oeqa/runtime/smart.py
 create mode 100644 meta/lib/oeqa/runtime/ssh.py

diff --git a/meta/lib/oeqa/runtime/__init__.py b/meta/lib/oeqa/runtime/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/meta/lib/oeqa/runtime/connman.py b/meta/lib/oeqa/runtime/connman.py
new file mode 100644
index 0000000..835e135
--- /dev/null
+++ b/meta/lib/oeqa/runtime/connman.py
@@ -0,0 +1,34 @@
+import unittest
+from oeqa.oetest import oeRuntimeTest, skipModule
+from oeqa.utils.decorators import *
+
+def setUpModule():
+    if not oeRuntimeTest.hasPackage("connman"):
+        skipModule("No connman package in image")
+
+
+class ConnmanTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_ssh')
+    def test_connmand_help(self):
+        (status, output) = self.target.run('/usr/sbin/connmand --help')
+        self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output))
+
+
+    @skipUnlessPassed('test_connmand_help')
+    def test_connmand_running(self):
+        status = self.target.run('ls -l `which ps` | grep busybox')[0]
+        if status == 0:
+            oeRuntimeTest.pscmd = 'ps'
+        else:
+            oeRuntimeTest.pscmd = 'ps -ef'
+        (status, output) = self.target.run(oeRuntimeTest.pscmd + ' | grep [c]onnmand')
+        self.assertEqual(status, 0, msg="no connmand process, ps output: %s" % self.target.run(oeRuntimeTest.pscmd)[1])
+
+    @skipUnlessPassed('test_connmand_running')
+    def test_connmand_unique(self):
+        self.target.run('/usr/sbin/connmand')
+        output = self.target.run(oeRuntimeTest.pscmd + ' | grep -c [c]onnmand')[1]
+        self.assertEqual(output, "1", msg="more than one connmand running in background, ps output: %s\n%s" % (output, self.target.run(oeRuntimeTest.pscmd)[1]))
+
+
diff --git a/meta/lib/oeqa/runtime/dmesg.py b/meta/lib/oeqa/runtime/dmesg.py
new file mode 100644
index 0000000..3acf09e
--- /dev/null
+++ b/meta/lib/oeqa/runtime/dmesg.py
@@ -0,0 +1,10 @@
+import unittest
+from oeqa.oetest import oeRuntimeTest
+from oeqa.utils.decorators import *
+
+class DmesgTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_ssh')
+    def test_dmesg(self):
+        (status, output) = self.target.run('dmesg | grep -v mmci-pl18x | grep -i error')
+        self.assertEqual(status, 1, msg = "Error messages in dmesg log: %s" % output)
diff --git a/meta/lib/oeqa/runtime/multilib.py b/meta/lib/oeqa/runtime/multilib.py
new file mode 100644
index 0000000..397d075
--- /dev/null
+++ b/meta/lib/oeqa/runtime/multilib.py
@@ -0,0 +1,14 @@
+import unittest
+from oeqa.oetest import oeRuntimeTest, skipModule
+
+def setUpModule():
+    multilibs = oeRuntimeTest.tc.d.getVar("MULTILIBS", True) or ""
+    if "multlib:lib32" not in multilibs:
+        skipModule("this isn't a multilib:lib32 image")
+
+
+class MultilibFileTest(oeRuntimeTest):
+
+    def test_file_connman(self):
+        (status, output) = self.target.run('file -L /usr/sbin/connmand | grep "ELF 32-bit LSB executable"')
+        self.assertEqual(status, 0, msg="status and output : %s and %s" % (status,output))
diff --git a/meta/lib/oeqa/runtime/ping.py b/meta/lib/oeqa/runtime/ping.py
new file mode 100644
index 0000000..d6a0c28
--- /dev/null
+++ b/meta/lib/oeqa/runtime/ping.py
@@ -0,0 +1,11 @@
+import subprocess
+import unittest
+import sys
+from oeqa.oetest import oeRuntimeTest
+
+class PingTest(oeRuntimeTest):
+
+    def test_ping(self):
+        status = subprocess.call("ping -w 30 -c 1 %s" % oeRuntimeTest.tc.qemu.ip, shell=True, stdout=subprocess.PIPE)
+        self.assertEqual(status, 0)
+
diff --git a/meta/lib/oeqa/runtime/rpm.py b/meta/lib/oeqa/runtime/rpm.py
new file mode 100644
index 0000000..57101b0
--- /dev/null
+++ b/meta/lib/oeqa/runtime/rpm.py
@@ -0,0 +1,25 @@
+import unittest
+from oeqa.oetest import oeRuntimeTest, skipModule
+from oeqa.utils.decorators import *
+
+def setUpModule():
+    if not oeRuntimeTest.hasFeature("package-management"):
+            skipModule("rpm module skipped: target doesn't have package-management in IMAGE_FEATURES")
+    if "package_rpm" != oeRuntimeTest.tc.d.getVar("PACKAGE_CLASSES", True).split()[0]:
+            skipModule("rpm module skipped: target doesn't have rpm as primary package manager")
+
+
+class RpmHelpTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_ssh')
+    def test_rpm_help(self):
+        (status, output) = self.target.run('rpm --help')
+        self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output))
+
+class RpmQueryTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_rpm_help')
+    def test_rpm_query(self):
+        (status, output) = self.target.run('rpm -q rpm')
+        self.assertEqual(status, 0, msg="status and output: %s and %s" % (status,output))
+
diff --git a/meta/lib/oeqa/runtime/smart.py b/meta/lib/oeqa/runtime/smart.py
new file mode 100644
index 0000000..b7d4e3e
--- /dev/null
+++ b/meta/lib/oeqa/runtime/smart.py
@@ -0,0 +1,23 @@
+import unittest
+from oeqa.oetest import oeRuntimeTest
+from oeqa.utils.decorators import *
+
+def setUpModule():
+    if not oeRuntimeTest.hasFeature("package-management"):
+        skipModule("Image doesn't have package management feature")
+    if not oeRuntimeTest.hasPackage("smart"):
+        skipModule("Image doesn't have smart installed")
+
+class SmartHelpTest(oeRuntimeTest):
+
+    def test_smart_help(self):
+        status = self.target.run('smart --help')[0]
+        self.assertEqual(status, 0)
+
+class SmartQueryTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_smart_help')
+    def test_smart_query(self):
+        (status, output) = self.target.run('smart query rpm')
+        self.assertEqual(status, 0, msg="status and output : %s and %s" % (status,output))
+
diff --git a/meta/lib/oeqa/runtime/ssh.py b/meta/lib/oeqa/runtime/ssh.py
new file mode 100644
index 0000000..8c96020
--- /dev/null
+++ b/meta/lib/oeqa/runtime/ssh.py
@@ -0,0 +1,16 @@
+import subprocess
+import unittest
+import sys
+from oeqa.oetest import oeRuntimeTest, skipModule
+from oeqa.utils.decorators import *
+
+def setUpModule():
+    if not (oeRuntimeTest.hasPackage("dropbear") or oeRuntimeTest.hasPackage("openssh")):
+        skipModule("No ssh package in image")
+
+class SshTest(oeRuntimeTest):
+
+    @skipUnlessPassed('test_ping')
+    def test_ssh(self):
+        (status, output) = self.target.run('uname -a')
+        self.assertEqual(status, 0, msg="SSH Test failed: %s" % output)
-- 
1.8.1.4

[RFC PATCH 8/8] lib/oeqa/runtime: add gcc test

[Stefan Stanacar]

gcc compile test and support files.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/runtime/files/test.c       | 26 +++++++++++++++++++++++
 meta/lib/oeqa/runtime/files/testmakefile |  5 +++++
 meta/lib/oeqa/runtime/gcc.py             | 36 ++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/files/test.c
 create mode 100644 meta/lib/oeqa/runtime/files/testmakefile
 create mode 100644 meta/lib/oeqa/runtime/gcc.py

diff --git a/meta/lib/oeqa/runtime/files/test.c b/meta/lib/oeqa/runtime/files/test.c
new file mode 100644
index 0000000..2d8389c
--- /dev/null
+++ b/meta/lib/oeqa/runtime/files/test.c
@@ -0,0 +1,26 @@
+#include <stdio.h>
+#include <math.h>
+#include <stdlib.h>
+
+double convert(long long l)
+{
+  return (double)l;
+}
+
+int main(int argc, char * argv[]) {
+
+  long long l = 10;
+  double f;
+  double check = 10.0;
+
+  f = convert(l);
+  printf("convert: %lld => %f\n", l, f);
+  if ( f != check ) exit(1);
+
+  f = 1234.67;
+  check = 1234.0;
+  printf("floorf(%f) = %f\n", f, floorf(f));
+  if ( floorf(f) != check) exit(1);
+
+  return 0;
+}
diff --git a/meta/lib/oeqa/runtime/files/testmakefile b/meta/lib/oeqa/runtime/files/testmakefile
new file mode 100644
index 0000000..ca1844e
--- /dev/null
+++ b/meta/lib/oeqa/runtime/files/testmakefile
@@ -0,0 +1,5 @@
+test: test.o
+	gcc -o test test.o -lm
+test.o: test.c
+	gcc -c test.c
+
diff --git a/meta/lib/oeqa/runtime/gcc.py b/meta/lib/oeqa/runtime/gcc.py
new file mode 100644
index 0000000..b63badd
--- /dev/null
+++ b/meta/lib/oeqa/runtime/gcc.py
@@ -0,0 +1,36 @@
+import unittest
+import os
+from oeqa.oetest import oeRuntimeTest, skipModule
+from oeqa.utils.decorators import *
+
+def setUpModule():
+    if not oeRuntimeTest.hasFeature("tools-sdk"):
+        skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES")
+
+
+class GccCompileTest(oeRuntimeTest):
+
+    @classmethod
+    def setUpClass(self):
+        oeRuntimeTest.tc.target.copy_to(os.path.join(oeRuntimeTest.tc.filesdir, "test.c"), "/tmp/test.c")
+        oeRuntimeTest.tc.target.copy_to(os.path.join(oeRuntimeTest.tc.filesdir, "testmakefile"), "/tmp/testmakefile")
+
+    def test_gcc_compile(self):
+        (status, output) = self.target.run('gcc /tmp/test.c -o /tmp/test -lm')
+        self.assertEqual(status, 0, msg="gcc compile failed, output: %s" % output)
+        (status, output) = self.target.run('/tmp/test')
+        self.assertEqual(status, 0, msg="running compiled file failed, output %s" % output)
+
+    def test_gpp_compile(self):
+        (status, output) = self.target.run('g++ /tmp/test.c -o /tmp/test -lm')
+        self.assertEqual(status, 0, msg="g++ compile failed, output: %s" % output)
+        (status, output) = self.target.run('/tmp/test')
+        self.assertEqual(status, 0, msg="running compiled file failed, output %s" % output)
+
+    def test_make(self):
+        (status, output) = self.target.run('cd /tmp; make -f testmakefile')
+        self.assertEqual(status, 0, msg="running make failed, output %s" % output)
+
+    @classmethod
+    def tearDownClass(self):
+        oeRuntimeTest.tc.target.run("rm /tmp/test.c /tmp/test.o /tmp/test /tmp/testmakefile")
-- 
1.8.1.4

Re: [RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[Otavio Salvador]

On Fri, Jun 28, 2013 at 7:04 AM, Stefan Stanacar
<stefanx.stanacar@intel.com> wrote:
...
> There are some areas for improvement/limitations right now:
>  - a better way of getting the list of installed packages in a image
>  - qemu is started with the -snapshot option, we should create a copy of the original rootfs instead.
>  - when using TEST_SUITES = "auto" there is no dependency at all between tests (rpm would run before ssh test), so better use TEST_SUITES = "ping ssh auto"
>  - better logging and error reporting for qemu problems (runqemu erros, qemu doesn't start, etc)
>  - more helper methods for tests
>  - others I can't think of right now :)

Did you think about adding something like an overidable set of hooks
which starts, setup image and shutdown the machine? This would allow
same framework to be used for real hardware test.

Regards,

--
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[Paul Eggleton]

On Friday 28 June 2013 19:06:04 Otavio Salvador wrote:
> Did you think about adding something like an overidable set of hooks
> which starts, setup image and shutdown the machine? This would allow
> same framework to be used for real hardware test.

Yes, we did think about this; the way this would be done would be to simply 
swap out the QemuRunner instance at runtime with some other class that has the 
same interface which would provision a real machine. However there is no 
established framework for doing the latter, and at the moment we just want to 
concentrate on getting runtime tests working. As long as the tests themselves 
know as little as possible about the machine they are running in then 
switching over to real hardware is pretty straightforward, and we've tried to 
design things in that manner.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC PATCH 4/8] lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix domain sockets

[Paul Eggleton]

On Friday 28 June 2013 13:04:39 Stefan Stanacar wrote:
> Python's telnetlib Telnet class connects only to AF_INET sockets, but we
> want to use Unix domain socket for the qemu serial connection.
> Also add a new read_all_timeout method similar to Telnet's read_all
> 
> Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
> ---
>  meta/lib/oeqa/utils/oetelnetlib.py | 49
> ++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
>  create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py
> 
> diff --git a/meta/lib/oeqa/utils/oetelnetlib.py
> b/meta/lib/oeqa/utils/oetelnetlib.py new file mode 100644
> index 0000000..cdebac1
> --- /dev/null
> +++ b/meta/lib/oeqa/utils/oetelnetlib.py
> @@ -0,0 +1,49 @@
> +import socket
> +import time
> +import re
> +from telnetlib import Telnet
> +
> +class oeTelnet(Telnet):
> +
> +    """
> +    Override Telnet class to use unix domain sockets,
> +    Telnet uses AF_INET for socket, we don't want that.
> +    Also, provide a read_all variant with timeout, that
> +    returns whatever output there is.
> +    """
> +
> +    def __init__(self, stream=None, logfile=None):
> +
> +        Telnet.__init__(self, host=None)
> +        self.stream = stream
> +        self.logfile = logfile
> +        if stream is not None:
> +            self.open(stream)
> +
> +    def log(self, msg):
> +        if self.logfile:
> +            with open(self.logfile, "a") as f:
> +                f.write("%s\n" % msg)
> +
> +    def open(self, stream):
> +
> +        self.eof = 0
> +        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> +        self.sock.connect(stream)
> +
> +    def read_all_timeout(self, match, timeout=120):
> +        """Read until EOF or until timeout or until match.
> +        """
> +        ret = False
> +        self.process_rawq()
> +        endtime = time.time() + timeout
> +        while not self.eof and time.time() < endtime:
> +            self.fill_rawq()
> +            self.process_rawq()
> +            if re.search(match, self.cookedq):
> +                ret = True
> +                break
> +        buf = self.cookedq
> +        self.cookedq = ''
> +        self.log(buf)
> +        return (ret, buf)

Just a minor bit of feedback about this; I think it's probably best not to 
call it something relating to "telnet" since it's no longer really telnet when 
using unix domain sockets.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[Saul Wold]

On 06/28/2013 03:04 AM, Stefan Stanacar wrote:
>
> Hello,
>
> This is the proposed implementation of a new runtime tests framework based on python unittest.
> It's mean to ease qemu image testing and encourage developers to add more tests similar to the example tests provided (all tests are basically commands ran over ssh)
>
> You can try it out like this:
>   - first build a qemu core-image-sato (a minimal wouldn't be interesting at all)
>   - add INHERIT += "testimage" in local.conf
>   - then bitbake core-image-sato -c testimage. That will run a standard suite of tests.
>
> You can set TEST_SUITES = "ping ssh <test name>" in local.conf to force run only certain tests (order matters here, it's the order in which tests run). You can also append "auto" and it will also run whatever tests are suitable for the image (if that was a sato-sdk image and you set TEST_SUITE = "ping ssh rpm auto" you force run ping, ssh and rpm but you also get smart, connman and gcc tests).
>
> Check the task log (log.do_testimage) in WORKDIR to see the results. Also a ssh log (what command is running, output and return codes) and qemu boot log are kept in WORKDIR/testimage/
>
> There are some areas for improvement/limitations right now:
>   - a better way of getting the list of installed packages in a image
>   - qemu is started with the -snapshot option, we should create a copy of the original rootfs instead.
>   - when using TEST_SUITES = "auto" there is no dependency at all between tests (rpm would run before ssh test), so better use TEST_SUITES = "ping ssh auto"
>   - better logging and error reporting for qemu problems (runqemu erros, qemu doesn't start, etc)
>   - more helper methods for tests
>   - others I can't think of right now :)
>
> I hope this is going in the right direction and we could improve from here.
> Comments and feedback are most welcome!
>

Tired this out today and got the following:
> ERROR: Function failed: Failed to start qemu. You should check the task log and the qemu boot log (qemu log is /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130704065001)
> ERROR: Logfile of failure stored in: /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp/log.do_testimage.18354
> Log data follows:
> | DEBUG: Executing python function do_testimage
> | NOTE: Test modules  ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.connman', 'oeqa.runtime.rpm', 'oeqa.runtime.smart', 'oeqa.runtime.xorg', 'oeqa.runtime.dmesg']
> | NOTE: DISPLAY value: None
> | NOTE: rootfs file: /srv/ssd/sgw/builds/world/tmp/deploy/images/core-image-sato-qemux86-64.ext3
> | NOTE: Qemu logfile: /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130704065001
> | NOTE: runqemu started, pid is 18389
> | NOTE: waiting at most 30 seconds for qemu pid
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | No children found matching 18389
> | NOTE: Qemu pid didn't appeared in 30 seconds
> | NOTE: Output from runqemu: Please use simplified serial or kvm options instead
> |
> | Continuing with the following parameters:
> | KERNEL: [/srv/ssd/sgw/builds/world/tmp/deploy/images/bzImage-qemux86-64.bin]
> | ROOTFS: [/srv/ssd/sgw/builds/world/tmp/deploy/images/core-image-sato-qemux86-64.ext3]
> | FSTYPE: [ext3]
> | stty: standard input: Inappropriate ioctl for device
> | Acquiring lockfile for tap0...
> | Using preconfigured tap device 'tap0'
> | Running qemu-system-x86_64...
> | /srv/ssd/sgw/builds/world/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64 -kernel /srv/ssd/sgw/builds/world/tmp/deploy/images/bzImage-qemux86-64.bin -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -hda /srv/ssd/sgw/builds/world/tmp/deploy/images/core-image-sato-qemux86-64.ext3 -show-cursor -usb -usbdevice wacom-tablet -vga vmware -no-reboot -snapshot -serial unix:/tmp/qemuconnection.18354,server,nowait -m 128 --append "vga=0 uvesafb.mode_option=640x480-32 root=/dev/hda rw mem=128M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 console=ttyS0"
> | Could not initialize SDL(No available video device) - exiting
> | Releasing lockfile of preconfigured tap device 'tap0'
> | stty: standard input: Inappropriate ioctl for device
> | stty: standard input: Inappropriate ioctl for device
> |
> | DEBUG: Python function do_testimage finished
> | ERROR: Function failed: Failed to start qemu. You should check the task log and the qemu boot log (qemu log is /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130704065001)
> ERROR: Task 0 (/srv/ssd/sgw/poky/meta/recipes-sato/images/core-image-sato.bb, do_testimage) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 164 tasks of which 163 didn't need to be rerun and 1 failed.
>


Sau!


> Regards,
> Stefan
>
>
>
> The following changes since commit 042203531b10b37ac9a8201b376f5dec403e51d8:
>
>    sanity.bbclass: Fix COREBASE sanity tests (2013-06-27 12:48:56 +0100)
>
> are available in the git repository at:
>
>    git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib stefans/oeqa5
>
> for you to fetch changes up to e6e567982f4b28822afdc8ea55401513df7c9466:
>
>    lib/oeqa/runtime: add gcc test (2013-06-28 13:00:45 +0300)
>
> ----------------------------------------------------------------
> Radu Moisan (2):
>        lib/oeqa/utils/qemurunner.py: class to handle qemu instance
>        lib/oeqa/utils/decorators.py: decorators for test methods
>
> Stefan Stanacar (6):
>        classes/testimage.bbclass: new class for image tests
>        lib/oeqa/oetest.py: base module for all runtime unittests
>        lib/oeqa/utils/sshcontrol.py: helper module for running remote commands
>        lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix domain sockets
>        lib/oeqa/runtime: image sanity tests
>        lib/oeqa/runtime: add gcc test
>
>   meta/classes/testimage.bbclass           |  95 ++++++++++++++++++
>   meta/lib/oeqa/__init__.py                |   0
>   meta/lib/oeqa/oetest.py                  |  94 ++++++++++++++++++
>   meta/lib/oeqa/runtime/__init__.py        |   0
>   meta/lib/oeqa/runtime/connman.py         |  34 +++++++
>   meta/lib/oeqa/runtime/dmesg.py           |  10 ++
>   meta/lib/oeqa/runtime/files/test.c       |  26 +++++
>   meta/lib/oeqa/runtime/files/testmakefile |   5 +
>   meta/lib/oeqa/runtime/gcc.py             |  36 +++++++
>   meta/lib/oeqa/runtime/multilib.py        |  14 +++
>   meta/lib/oeqa/runtime/ping.py            |  11 +++
>   meta/lib/oeqa/runtime/rpm.py             |  25 +++++
>   meta/lib/oeqa/runtime/smart.py           |  23 +++++
>   meta/lib/oeqa/runtime/ssh.py             |  16 ++++
>   meta/lib/oeqa/utils/__init__.py          |   0
>   meta/lib/oeqa/utils/decorators.py        |  40 ++++++++
>   meta/lib/oeqa/utils/oetelnetlib.py       |  49 ++++++++++
>   meta/lib/oeqa/utils/qemurunner.py        | 160 +++++++++++++++++++++++++++++++
>   meta/lib/oeqa/utils/sshcontrol.py        | 100 +++++++++++++++++++
>   scripts/runqemu                          |   2 +-
>   20 files changed, 739 insertions(+), 1 deletion(-)
>   create mode 100644 meta/classes/testimage.bbclass
>   create mode 100644 meta/lib/oeqa/__init__.py
>   create mode 100644 meta/lib/oeqa/oetest.py
>   create mode 100644 meta/lib/oeqa/runtime/__init__.py
>   create mode 100644 meta/lib/oeqa/runtime/connman.py
>   create mode 100644 meta/lib/oeqa/runtime/dmesg.py
>   create mode 100644 meta/lib/oeqa/runtime/files/test.c
>   create mode 100644 meta/lib/oeqa/runtime/files/testmakefile
>   create mode 100644 meta/lib/oeqa/runtime/gcc.py
>   create mode 100644 meta/lib/oeqa/runtime/multilib.py
>   create mode 100644 meta/lib/oeqa/runtime/ping.py
>   create mode 100644 meta/lib/oeqa/runtime/rpm.py
>   create mode 100644 meta/lib/oeqa/runtime/smart.py
>   create mode 100644 meta/lib/oeqa/runtime/ssh.py
>   create mode 100644 meta/lib/oeqa/utils/__init__.py
>   create mode 100644 meta/lib/oeqa/utils/decorators.py
>   create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py
>   create mode 100644 meta/lib/oeqa/utils/qemurunner.py
>   create mode 100644 meta/lib/oeqa/utils/sshcontrol.py
>
> Radu Moisan (2):
>    lib/oeqa/utils/qemurunner.py: class to handle qemu instance
>    lib/oeqa/utils/decorators.py: decorators for test methods
>
> Stefan Stanacar (6):
>    classes/testimage.bbclass: new class for image tests
>    lib/oeqa/oetest.py: base module for all runtime unittests
>    lib/oeqa/utils/sshcontrol.py: helper module for running remote
>      commands
>    lib/oeqa/utils/oetelnetlib.py: override Telnet class to use Unix
>      domain sockets
>    lib/oeqa/runtime: image sanity tests
>    lib/oeqa/runtime: add gcc test
>
>   meta/classes/testimage.bbclass           |  95 ++++++++++++++++++
>   meta/lib/oeqa/__init__.py                |   0
>   meta/lib/oeqa/oetest.py                  |  94 ++++++++++++++++++
>   meta/lib/oeqa/runtime/__init__.py        |   0
>   meta/lib/oeqa/runtime/connman.py         |  34 +++++++
>   meta/lib/oeqa/runtime/dmesg.py           |  10 ++
>   meta/lib/oeqa/runtime/files/test.c       |  26 +++++
>   meta/lib/oeqa/runtime/files/testmakefile |   5 +
>   meta/lib/oeqa/runtime/gcc.py             |  36 +++++++
>   meta/lib/oeqa/runtime/multilib.py        |  14 +++
>   meta/lib/oeqa/runtime/ping.py            |  11 +++
>   meta/lib/oeqa/runtime/rpm.py             |  25 +++++
>   meta/lib/oeqa/runtime/smart.py           |  23 +++++
>   meta/lib/oeqa/runtime/ssh.py             |  16 ++++
>   meta/lib/oeqa/utils/__init__.py          |   0
>   meta/lib/oeqa/utils/decorators.py        |  40 ++++++++
>   meta/lib/oeqa/utils/oetelnetlib.py       |  49 ++++++++++
>   meta/lib/oeqa/utils/qemurunner.py        | 160 +++++++++++++++++++++++++++++++
>   meta/lib/oeqa/utils/sshcontrol.py        | 100 +++++++++++++++++++
>   scripts/runqemu                          |   2 +-
>   20 files changed, 739 insertions(+), 1 deletion(-)
>   create mode 100644 meta/classes/testimage.bbclass
>   create mode 100644 meta/lib/oeqa/__init__.py
>   create mode 100644 meta/lib/oeqa/oetest.py
>   create mode 100644 meta/lib/oeqa/runtime/__init__.py
>   create mode 100644 meta/lib/oeqa/runtime/connman.py
>   create mode 100644 meta/lib/oeqa/runtime/dmesg.py
>   create mode 100644 meta/lib/oeqa/runtime/files/test.c
>   create mode 100644 meta/lib/oeqa/runtime/files/testmakefile
>   create mode 100644 meta/lib/oeqa/runtime/gcc.py
>   create mode 100644 meta/lib/oeqa/runtime/multilib.py
>   create mode 100644 meta/lib/oeqa/runtime/ping.py
>   create mode 100644 meta/lib/oeqa/runtime/rpm.py
>   create mode 100644 meta/lib/oeqa/runtime/smart.py
>   create mode 100644 meta/lib/oeqa/runtime/ssh.py
>   create mode 100644 meta/lib/oeqa/utils/__init__.py
>   create mode 100644 meta/lib/oeqa/utils/decorators.py
>   create mode 100644 meta/lib/oeqa/utils/oetelnetlib.py
>   create mode 100644 meta/lib/oeqa/utils/qemurunner.py
>   create mode 100644 meta/lib/oeqa/utils/sshcontrol.py
>

Re: [RFC PATCH 0/8] Proposed implementation of a new runtime tests framework

[Stanacar, StefanX]

On Wed, 2013-07-03 at 23:51 -0700, Saul Wold wrote:
> Tired this out today and got the following:
> > ERROR: Function failed: Failed to start qemu. You should check the task log and the qemu boot log (qemu log is /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130704065001)
> > ERROR: Logfile of failure stored in: /srv/ssd/sgw/builds/world/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp/log.do_testimage.18354
> > Log data follows:
> > | DEBUG: Executing python function do_testimage
> > | NOTE: Test modules  ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.connman', 'oeqa.runtime.rpm', 'oeqa.runtime.smart', 'oeqa.runtime.xorg', 'oeqa.runtime.dmesg']

On a second look, this shouldn't be none:

> > | NOTE: DISPLAY value: None

Because this happens:

> > | Running qemu-system-x86_64...
> > | /srv/ssd/sgw/builds/world/tmp/sysroots/x86_64-linux/usr/bin/qemu-system-x86_64 -kernel /srv/ssd/sgw/builds/world/tmp/deploy/images/bzImage-qemux86-64.bin -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -hda /srv/ssd/sgw/builds/world/tmp/deploy/images/core-image-sato-qemux86-64.ext3 -show-cursor -usb -usbdevice wacom-tablet -vga vmware -no-reboot -snapshot -serial unix:/tmp/qemuconnection.18354,server,nowait -m 128 --append "vga=0 uvesafb.mode_option=640x480-32 root=/dev/hda rw mem=128M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 console=ttyS0"
> > | Could not initialize SDL(No available video device) - exiting

Can you export DISPLAY before running the task? But don't use
DISPLAY=localhost:1 use DISPLAY=:1 (see my other mail).


Cheers, 
Stefan

Re: [RFC PATCH 5/8] lib/oeqa/utils/qemurunner.py: class to handle qemu instance

[Colin Walters]

On Fri, 2013-06-28 at 13:04 +0300, Stefan Stanacar wrote:

> +        self.streampath = '/tmp/qemuconnection.%s' % os.getpid()

That's a security problem on shared machines.

> +                bb.note("Reached login banner")
> +                console.write("root\n")
> +                (index, match, text) = console.expect([r"(root@[\w-]+:~#)"],10)

So I forget if I've mentioned this here, but what I do for the
gnome-ostree testing is at boot time, use a qcow2 overlay disk to write
a custom systemd service that exports the journal over a virtio-serial
channel.  Then I look for specific MESSAGE_IDs in the journal.

This is extremely reliable, no parsing of log messages etc.

See:
https://rwmj.wordpress.com/2013/07/19/half-baked-ideas-ocr-vm-console-to-diagnose-state-and-errors/

Re: [RFC PATCH 5/8] lib/oeqa/utils/qemurunner.py: class to handle qemu instance

[Stanacar, StefanX]

Hi Colin,

On Mon, 2013-08-05 at 21:50 +0200, Colin Walters wrote:
> On Fri, 2013-06-28 at 13:04 +0300, Stefan Stanacar wrote:
> 
> > +        self.streampath = '/tmp/qemuconnection.%s' % os.getpid()
> 
> That's a security problem on shared machines.

I know this is a late reply, sorry, I missed the email back then.
This has been changed since then to something else (but the reasons were
different)
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0ba78c1162bb125850a0ee504ca6fbe5bf21247f
That's a tcp socket localhost only now, random high port (calls bind
with 127.0.0.1, 0 so the os chooses the port).


> 
> > +                bb.note("Reached login banner")
> > +                console.write("root\n")
> > +                (index, match, text) = console.expect([r"(root@[\w-]+:~#)"],10)
> 
> So I forget if I've mentioned this here, but what I do for the
> gnome-ostree testing is at boot time, use a qcow2 overlay disk to write
> a custom systemd service that exports the journal over a virtio-serial
> channel.  Then I look for specific MESSAGE_IDs in the journal.
> 
> This is extremely reliable, no parsing of log messages etc.
> 

That sounds really cool, nice job! 
I might be wrong but doesn't that require virtio support in the target
kernel (which is something we can't expect to have)?

Cheers,
Stefan

> See:
> https://rwmj.wordpress.com/2013/07/19/half-baked-ideas-ocr-vm-console-to-diagnose-state-and-errors/
> 
> 

Re: [RFC PATCH 5/8] lib/oeqa/utils/qemurunner.py: class to handle qemu instance

[Colin Walters]

On Thu, 2013-08-29 at 11:18 +0000, Stanacar, StefanX wrote:

> That sounds really cool, nice job! 
> I might be wrong but doesn't that require virtio support in the target
> kernel (which is something we can't expect to have)?

Yes...although you could do the same over TCP, it'd just require more
gymnastics on the host and guest side.   The driver is just 12k here on
this RHEL6 box; I have it built in to the kernel for my OE target so I
don't know how large it is offhand there.