[[PATCH][qa-tools] 00/16] Add Toaster test suite support

[[PATCH][qa-tools] 00/16] Add Toaster test suite support

[Aníbal Limón]

This set of changes adds toaster test suite to qa-tools repo.

Summary,
	- Add ToasterHelper class for clone/setup/start/stop toaster.
	- Add Toaster test suite script, mainly work did by QA Rumania team.
	- Add COPYRIGHT and improve instructions for install/usage.

Next steps are for create an buildset for our AB in order to run toaster tests.

Also the changes can be found at,
	http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/log/?h=devel
	
Aníbal Limón (16):
  README.md: Update instructions on how install host deps on debian.
  utils/run/toaster.sh: Get rid of toaster shell script.
  toaster: Add helpers for clone, setup, start and stop.
  tests/toaster/__init__.py: Add support for clone/setup/start/stop
  tests/toaster/helpers.py: When execute bash remove interactive mode.
  tests/toaster/helpers.py: Fix toaster_start deadlocks.
  ts/toaster/helpers.py: Add new class ToasterHelper instead functions.
  toaster/helper.py: Add force mode to stop method.
  toaster/__init__.py: Update toaster test to match new ToasterHelper.
  Reorder source code tree,
  README.md: Update with information about setup, toaster and
    maintainence.
  toaster/toaster.py: Add missing shebang to call python.
  toaster: Add header to specify license to MIT.
  toaster/toaster.py: Add command line options.
  toaster/helpers.py: Fix ToasterHelper setup method git checkout.
  README.md: Add recommendation to use vncserver for run Toaster tests.

 COPYING.MIT               |  17 +++++
 README.md                 |  55 +++++++++++++++-
 external/__init__.py      |   0
 requeriments.txt          |   1 +
 tests/toaster/__init__.py | 109 -------------------------------
 toaster/__init__.py       |   0
 toaster/helpers.py        | 113 ++++++++++++++++++++++++++++++++
 toaster/toaster.py        | 160 ++++++++++++++++++++++++++++++++++++++++++++++
 utils/run/toaster.sh      |  20 ------
 9 files changed, 343 insertions(+), 132 deletions(-)
 create mode 100644 COPYING.MIT
 create mode 100644 external/__init__.py
 delete mode 100755 tests/toaster/__init__.py
 create mode 100644 toaster/__init__.py
 create mode 100644 toaster/helpers.py
 create mode 100755 toaster/toaster.py
 delete mode 100755 utils/run/toaster.sh

-- 
2.1.4

[[PATCH][qa-tools] 01/16] README.md: Update instructions on how install host deps on debian.

[Aníbal Limón]

From: Aníbal Limón <limon.anibal@gmail.com>

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
 README.md | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 8d40c47..b5e6c8f 100644
--- a/README.md
+++ b/README.md
@@ -7,5 +7,9 @@ Scripts used by Yocto QA Team for automatize QA process.
 == Toaster ==
 
 - virtualenv for python
+- pip for python
 - firefox web browser for use selenium
-- vncserver
+
+=== Debian install ===
+
+# apt-get install iceweasel python-virtualenv python-pip
-- 
2.1.4

[[PATCH][qa-tools] 02/16] utils/run/toaster.sh: Get rid of toaster shell script.

[Aníbal Limón]

From: Aníbal Limón <limon.anibal@gmail.com>

A module in python will be created for have toaster helpers
with clone, setup, start, stop.

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
 utils/run/toaster.sh | 20 --------------------
 1 file changed, 20 deletions(-)
 delete mode 100755 utils/run/toaster.sh

diff --git a/utils/run/toaster.sh b/utils/run/toaster.sh
deleted file mode 100755
index ba6d537..0000000
--- a/utils/run/toaster.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-
-base_work_dir=/tmp
-work_dir=$base_work_dir/toaster
-poky_dir=$work_dir/poky
-poky_url=git://git.yoctoproject.org/poky.git
-
-if [ -d $work_dir ]; then
-	rm -rf $work_dir
-fi
-
-mkdir -p $work_dir
-git clone git://git.yoctoproject.org/poky.git $poky_dir
-
-cd $poky_dir
-virtualenv $poky_dir/venv
-. $poky_dir/venv/bin/activate
-. $poky_dir/oe-init-build-env
-pip install -r $poky_dir/bitbake/toaster-requirements.txt
-TOASTER_CONF=$poky_dir/meta-yocto/conf/toasterconf.json . $poky_dir/bitbake/bin/toaster start
-- 
2.1.4

[[PATCH][qa-tools] 03/16] toaster: Add helpers for clone, setup, start and stop.

[Aníbal Limón]

From: Aníbal Limón <limon.anibal@gmail.com>

README.md: Toaster tests requires bash so add it.
TODO is add a sanity module for test this kind of issues
like shell compatibility.

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
 README.md                |  4 ++++
 tests/toaster/helpers.py | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
 create mode 100644 tests/toaster/helpers.py

diff --git a/README.md b/README.md
index b5e6c8f..4efe1a8 100644
--- a/README.md
+++ b/README.md
@@ -4,6 +4,10 @@ Scripts used by Yocto QA Team for automatize QA process.
 
 = Host requirements =
 
+All the test components REQUIRE bash as your default shell, you
+need to review if /bin/sh points to bash because some problems are
+detected when use dash.
+
 == Toaster ==
 
 - virtualenv for python
diff --git a/tests/toaster/helpers.py b/tests/toaster/helpers.py
new file mode 100644
index 0000000..49f34da
--- /dev/null
+++ b/tests/toaster/helpers.py
@@ -0,0 +1,43 @@
+import subprocess
+import os
+import shutil
+import signal
+
+TOASTER_TEST_BRANCH = 'toaster_tests'
+VENV_NAME = 'venv'
+SHELL_CMD = os.environ['SHELL'] if 'SHELL' in os.environ else "/bin/bash"
+
+def _execute_command(directory, cmd):
+    subprocess.check_call([SHELL_CMD, "-i", "-c", "cd %s; %s" % \
+        (directory, cmd)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+def _execute_command_venv(directory, venv, cmd):
+    _execute_command(directory, "source %s/%s/bin/activate; %s" % \
+        (directory, venv, cmd))
+
+def toaster_clone(directory, repo, ref='master', rm=False):
+    if os.path.exists(directory):
+        if rm:
+            shutil.rmtree(directory)
+        else:
+            raise IOError
+
+    subprocess.check_output([SHELL_CMD, "-i", "-c", "git clone %s %s" % \
+        (repo, directory)], stderr=subprocess.STDOUT)
+    _execute_command(directory, "git checkout %s -b %s" % \
+        (ref, TOASTER_TEST_BRANCH))
+
+def toaster_setup(directory):
+    _execute_command(directory, "virtualenv %s" % VENV_NAME)
+    _execute_command_venv(directory, VENV_NAME, "pip install -r" \
+        " bitbake/toaster-requirements.txt")
+
+def toaster_start(directory):
+    _execute_command_venv(directory, VENV_NAME,
+        "source %s/oe-init-build-env; source %s/bitbake/bin/toaster start" % \
+        (directory, directory))
+
+def toaster_stop(directory):
+    _execute_command_venv(directory, VENV_NAME,
+        "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
+        (directory, directory))
-- 
2.1.4

[[PATCH][qa-tools] 04/16] tests/toaster/__init__.py: Add support for clone/setup/start/stop

[Aníbal Limón]

From: Aníbal Limón <limon.anibal@gmail.com>

The main idea is to have an script for execute tests against toaster
so for now the arguments to execute toaster tests are fixed but
will be configured with args in the console.

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
 tests/toaster/__init__.py | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/tests/toaster/__init__.py b/tests/toaster/__init__.py
index 659faa8..3762094 100755
--- a/tests/toaster/__init__.py
+++ b/tests/toaster/__init__.py
@@ -1,5 +1,6 @@
 import unittest, time, re, sys, getopt, os, logging, string, errno, exceptions
 import shutil, argparse, ConfigParser, platform
+
 from selenium import webdriver
 from selenium.common.exceptions import NoSuchElementException
 from selenium import selenium
@@ -7,11 +8,28 @@ from selenium.webdriver.common.by import By
 from selenium.webdriver.common.keys import Keys
 from selenium.webdriver.support.ui import Select
 
+sys.path.insert(0, os.path.join(os.path.dirname(
+    os.path.abspath(__file__)), '../'))
+from toaster.helpers import *
+
+WORK_DIRECTORY = '/tmp/toaster'
+POKY_URL = 'http://git.yoctoproject.org/git/poky.git'
+
 class InitToaster(unittest.TestCase):
-    def setUp(self):
+    @classmethod
+    def setUpClass(self):
+        toaster_clone(WORK_DIRECTORY, POKY_URL, rm=True)
+        toaster_setup(WORK_DIRECTORY)
+        toaster_start(WORK_DIRECTORY)
+
         self.driver = webdriver.Firefox()
         self.timeout = 320
 
+    @classmethod
+    def tearDownClass(self):
+        self.driver.close()
+        toaster_stop(WORK_DIRECTORY)
+
     def is_text_present (self, patterns):
         for pattern in patterns:
             if str(pattern) not in self.driver.page_source:
@@ -102,8 +120,5 @@ class InitToaster(unittest.TestCase):
                 self.fail(msg="Builds did not complete successfully.")
         print "Builds complete!"
 
-    def tearDown(self):
-        self.driver.close()
-
 if __name__ == "__main__":
     unittest.main()
-- 
2.1.4

[[PATCH][qa-tools] 05/16] tests/toaster/helpers.py: When execute bash remove interactive mode.

[Aníbal Limón]

From: Aníbal Limón <limon.anibal@gmail.com>

Bash interactive mode isn't useful so remove it.

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
 tests/toaster/helpers.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/toaster/helpers.py b/tests/toaster/helpers.py
index 49f34da..7b82e88 100644
--- a/tests/toaster/helpers.py
+++ b/tests/toaster/helpers.py
@@ -8,8 +8,8 @@ VENV_NAME = 'venv'
 SHELL_CMD = os.environ['SHELL'] if 'SHELL' in os.environ else "/bin/bash"
 
 def _execute_command(directory, cmd):
-    subprocess.check_call([SHELL_CMD, "-i", "-c", "cd %s; %s" % \
-        (directory, cmd)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    subprocess.check_call([SHELL_CMD, "-c", "cd %s; %s" % \
+        (directory, cmd)], stderr=subprocess.STDOUT)
 
 def _execute_command_venv(directory, venv, cmd):
     _execute_command(directory, "source %s/%s/bin/activate; %s" % \
@@ -22,7 +22,7 @@ def toaster_clone(directory, repo, ref='master', rm=False):
         else:
             raise IOError
 
-    subprocess.check_output([SHELL_CMD, "-i", "-c", "git clone %s %s" % \
+    subprocess.check_output([SHELL_CMD, "-c", "git clone %s %s" % \
         (repo, directory)], stderr=subprocess.STDOUT)
     _execute_command(directory, "git checkout %s -b %s" % \
         (ref, TOASTER_TEST_BRANCH))
-- 
2.1.4

[[PATCH][qa-tools] 06/16] tests/toaster/helpers.py: Fix toaster_start deadlocks.

[Aníbal Limón]

When call toaster_start on failing scenarios (already started) it
becomes blocked on process.communicate() function so add my own
function for solve this issue.

The new function uses a tempfile instead of PIPE for avoid deadlocks.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 tests/toaster/helpers.py | 38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/tests/toaster/helpers.py b/tests/toaster/helpers.py
index 7b82e88..5e54723 100644
--- a/tests/toaster/helpers.py
+++ b/tests/toaster/helpers.py
@@ -2,17 +2,47 @@ import subprocess
 import os
 import shutil
 import signal
+import tempfile
 
 TOASTER_TEST_BRANCH = 'toaster_tests'
 VENV_NAME = 'venv'
 SHELL_CMD = os.environ['SHELL'] if 'SHELL' in os.environ else "/bin/bash"
 
+def _check_output1(*popenargs, **kwargs):
+    """
+        Almost the same as subprocess.check_output but change the stdout from
+        PIPE to tempfile to avoid deadlocks when trying to read the PIPE using
+        communicate(). This scenario can be seen calling toaster_start on failure
+        scenarios.
+
+        This causes a little overhead by the tempfile.
+    """
+
+    f = tempfile.TemporaryFile(mode='rw+')
+    if 'stdout' in kwargs:
+        raise ValueError('stdout argument not allowed, it will be overridden.')
+    process = subprocess.Popen(stdout=f, *popenargs, **kwargs)
+    retcode = process.wait()
+
+    f.flush()
+    os.fsync(f.fileno())
+    f.seek(0, 0)
+    output = f.read()
+    f.close()
+
+    if retcode:
+        cmd = kwargs.get("args")
+        if cmd is None:
+            cmd = popenargs[0]
+        raise subprocess.CalledProcessError(retcode, cmd, output=output)
+    return output
+
 def _execute_command(directory, cmd):
-    subprocess.check_call([SHELL_CMD, "-c", "cd %s; %s" % \
+    return _check_output1([SHELL_CMD, "-c", "cd %s; %s" % \
         (directory, cmd)], stderr=subprocess.STDOUT)
 
 def _execute_command_venv(directory, venv, cmd):
-    _execute_command(directory, "source %s/%s/bin/activate; %s" % \
+    return _execute_command(directory, "source %s/%s/bin/activate; %s" % \
         (directory, venv, cmd))
 
 def toaster_clone(directory, repo, ref='master', rm=False):
@@ -33,11 +63,11 @@ def toaster_setup(directory):
         " bitbake/toaster-requirements.txt")
 
 def toaster_start(directory):
-    _execute_command_venv(directory, VENV_NAME,
+    return _execute_command_venv(directory, VENV_NAME,
         "source %s/oe-init-build-env; source %s/bitbake/bin/toaster start" % \
         (directory, directory))
 
 def toaster_stop(directory):
-    _execute_command_venv(directory, VENV_NAME,
+    return _execute_command_venv(directory, VENV_NAME,
         "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
         (directory, directory))
-- 
2.1.4

[[PATCH][qa-tools] 07/16] ts/toaster/helpers.py: Add new class ToasterHelper instead functions.

[Aníbal Limón]

In order to provide a better interface add a ToasterHelper class the
functionality is the same so only modified functions to match method
definitions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 tests/toaster/helpers.py | 62 ++++++++++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 28 deletions(-)

diff --git a/tests/toaster/helpers.py b/tests/toaster/helpers.py
index 5e54723..98b65e0 100644
--- a/tests/toaster/helpers.py
+++ b/tests/toaster/helpers.py
@@ -37,37 +37,43 @@ def _check_output1(*popenargs, **kwargs):
         raise subprocess.CalledProcessError(retcode, cmd, output=output)
     return output
 
-def _execute_command(directory, cmd):
-    return _check_output1([SHELL_CMD, "-c", "cd %s; %s" % \
-        (directory, cmd)], stderr=subprocess.STDOUT)
+class ToasterHelper(object):
+    def __init__(self, directory, repo, repo_ref='master'):
+        self.directory = directory
+        self.repo = repo
+        self.repo_ref = repo_ref
 
-def _execute_command_venv(directory, venv, cmd):
-    return _execute_command(directory, "source %s/%s/bin/activate; %s" % \
-        (directory, venv, cmd))
+    def _execute_command(self, cmd):
+        return _check_output1([SHELL_CMD, "-c", "cd %s; %s" % \
+            (self.directory, cmd)], stderr=subprocess.STDOUT)
 
-def toaster_clone(directory, repo, ref='master', rm=False):
-    if os.path.exists(directory):
-        if rm:
-            shutil.rmtree(directory)
-        else:
-            raise IOError
+    def _execute_command_venv(self, venv, cmd):
+        return self._execute_command("source %s/%s/bin/activate; %s"\
+                % (self.directory, venv, cmd))
 
-    subprocess.check_output([SHELL_CMD, "-c", "git clone %s %s" % \
-        (repo, directory)], stderr=subprocess.STDOUT)
-    _execute_command(directory, "git checkout %s -b %s" % \
-        (ref, TOASTER_TEST_BRANCH))
+    def clone(self, rm=False):
+        if os.path.exists(self.directory):
+            if rm:
+                shutil.rmtree(self.directory)
+            else:
+                raise IOError
 
-def toaster_setup(directory):
-    _execute_command(directory, "virtualenv %s" % VENV_NAME)
-    _execute_command_venv(directory, VENV_NAME, "pip install -r" \
-        " bitbake/toaster-requirements.txt")
+        subprocess.check_output([SHELL_CMD, "-c", "git clone %s %s" % \
+            (self.repo, self.directory)], stderr=subprocess.STDOUT)
+        self._execute_command("git checkout %s -b %s" % \
+            (self.repo_ref, TOASTER_TEST_BRANCH))
 
-def toaster_start(directory):
-    return _execute_command_venv(directory, VENV_NAME,
-        "source %s/oe-init-build-env; source %s/bitbake/bin/toaster start" % \
-        (directory, directory))
+    def setup(self):
+        self._execute_command("virtualenv %s" % VENV_NAME)
+        self._execute_command_venv(VENV_NAME, "pip install -r" \
+            " bitbake/toaster-requirements.txt")
 
-def toaster_stop(directory):
-    return _execute_command_venv(directory, VENV_NAME,
-        "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
-        (directory, directory))
+    def start(self):
+        return self._execute_command_venv(VENV_NAME,
+            "source %s/oe-init-build-env; source %s/bitbake/bin/toaster start" % \
+            (self.directory, self.directory))
+
+    def stop(self):
+        return self._execute_command_venv(VENV_NAME,
+            "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
+            (self.directory, self.directory))
-- 
2.1.4

[[PATCH][qa-tools] 08/16] toaster/helper.py: Add force mode to stop method.

[Aníbal Limón]

The stop method have force mode because toaster without production
setup have known issues when is on load, the server response 503
service unavailable.

This mode iterates over process to found PID's of toaster instance
then sends SIGKILL to it.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 requeriments.txt         |  1 +
 tests/toaster/helpers.py | 38 ++++++++++++++++++++++++++++++++++----
 2 files changed, 35 insertions(+), 4 deletions(-)

diff --git a/requeriments.txt b/requeriments.txt
index 9ee5047..89a735d 100644
--- a/requeriments.txt
+++ b/requeriments.txt
@@ -2,3 +2,4 @@ python-bugzilla==1.2.2
 
 # toaster requirements
 selenium
+proc
diff --git a/tests/toaster/helpers.py b/tests/toaster/helpers.py
index 98b65e0..753bbf0 100644
--- a/tests/toaster/helpers.py
+++ b/tests/toaster/helpers.py
@@ -4,6 +4,8 @@ import shutil
 import signal
 import tempfile
 
+from proc.core import find_processes
+
 TOASTER_TEST_BRANCH = 'toaster_tests'
 VENV_NAME = 'venv'
 SHELL_CMD = os.environ['SHELL'] if 'SHELL' in os.environ else "/bin/bash"
@@ -73,7 +75,35 @@ class ToasterHelper(object):
             "source %s/oe-init-build-env; source %s/bitbake/bin/toaster start" % \
             (self.directory, self.directory))
 
-    def stop(self):
-        return self._execute_command_venv(VENV_NAME,
-            "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
-            (self.directory, self.directory))
+    def _stop_force(self):
+        """
+            The _stop_force method iterates over the /proc and search for toaster path
+            in the process cmdline then send SIGKILL for every matched process.
+        """
+        pids = []
+        for p in find_processes():
+            if len(p.cmdline) > 1 and \
+                os.path.basename(p.cmdline[0]) == 'python' and \
+                p.cmdline[1].startswith(self.directory):
+                    pids.append(p.pid)
+
+        for pid in pids:
+            try:
+                os.kill(pid, signal.SIGKILL)
+            except:
+                pass
+
+        return ''
+
+    def stop(self, force=False):
+        """
+            The stop method have force mode because toaster without production 
+            setup have known issues when is on load, the server response 503
+            service unavailable.
+        """
+        if force:
+            return self._stop_force()
+        else:
+            return self._execute_command_venv(VENV_NAME,
+                "source %s/oe-init-build-env; source %s/bitbake/bin/toaster stop" % \
+                (self.directory, self.directory))
-- 
2.1.4

[[PATCH][qa-tools] 09/16] toaster/__init__.py: Update toaster test to match new ToasterHelper.

[Aníbal Limón]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 tests/toaster/__init__.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tests/toaster/__init__.py b/tests/toaster/__init__.py
index 3762094..d3c5fff 100755
--- a/tests/toaster/__init__.py
+++ b/tests/toaster/__init__.py
@@ -10,7 +10,7 @@ from selenium.webdriver.support.ui import Select
 
 sys.path.insert(0, os.path.join(os.path.dirname(
     os.path.abspath(__file__)), '../'))
-from toaster.helpers import *
+from toaster.helpers import ToasterHelper
 
 WORK_DIRECTORY = '/tmp/toaster'
 POKY_URL = 'http://git.yoctoproject.org/git/poky.git'
@@ -18,17 +18,18 @@ POKY_URL = 'http://git.yoctoproject.org/git/poky.git'
 class InitToaster(unittest.TestCase):
     @classmethod
     def setUpClass(self):
-        toaster_clone(WORK_DIRECTORY, POKY_URL, rm=True)
-        toaster_setup(WORK_DIRECTORY)
-        toaster_start(WORK_DIRECTORY)
+        self.toaster_helper = ToasterHelper(WORK_DIRECTORY, POKY_URL)
+        self.toaster_helper.clone(rm=True)
+        self.toaster_helper.setup()
+        self.toaster_helper.start()
 
         self.driver = webdriver.Firefox()
         self.timeout = 320
 
     @classmethod
     def tearDownClass(self):
+        self.toaster_helper.stop(force=True)
         self.driver.close()
-        toaster_stop(WORK_DIRECTORY)
 
     def is_text_present (self, patterns):
         for pattern in patterns:
-- 
2.1.4