[Buildroot] [PATCH 0/9] support/testing: run tests in parallel

[Buildroot] [PATCH 0/9] support/testing: run tests in parallel

[Ricardo Martincoski]

Hello,

This patch series unleashes the power of nose2 to run testcases simultaneously.

Patches 1 to 5 gradually change the emulator from using subprocess + telnetlib
+ 'qemu -serial telnet' to use pexpect + 'qemu -serial stdio'. They allow to
run testcases in parallel without random failures caused by two emulators trying
to use the same telnet port (1234).

Patch 6 fixes 2 remaining code style issues in the same file.

Patch 7 adds option -j to run-tests to configure the BR2_JLEVEL used when
building stuff for the testcases. This parameter defaults to Buildroot default 0
that means to use cpu_count + 1. It can be used i.e. to run the tests in
background with 1 core while using the other cores from the host computer for
other work.

Patch 8 adds option -t to run-tests to configure the number of testcases to run
in parallel (default=1). When -j is not used, -t calculates it to split the
available cores between the simultaneous testcases. But the user can override
this auto calculated value.

Patch 9 makes one last change to emulator to allow many instances to run in
parallel.

Actually patches 1 to 6 and also patch 9 could all be squashed in a (not so big)
patch "Improve emulator using pexpect+stdio" but in separate patches I hope the
review and testing become easier. Anyway, I can squash them easily and resend.

This series was tested using Ubuntu 16.04.2 in a computer with 4 cores and
following package versions:
Python 2.7.12
nose2 0.6.4-1
QEMU emulator version 2.5.0

Regards,
Ricardo


Ricardo Martincoski (9):
  support/testing: use pexpect in emulator
  support/testing: use qemu stdio in emulator
  support/testing: let pexpect write stdout to log
  support/testing: use pexpect.sendline directly
  support/testing: use pexpect.expect directly
  support/testing: fix code style in emulator
  support/testing: allow to set BR2_JLEVEL
  support/testing: run testcases in parallel
  support/testing: large timeout for login prompt

 support/testing/conf/unittest.cfg |  1 -
 support/testing/infra/basetest.py |  6 +++-
 support/testing/infra/emulator.py | 71 ++++++++++++++-------------------------
 support/testing/run-tests         | 28 +++++++++++++++
 4 files changed, 59 insertions(+), 47 deletions(-)

-- 
2.11.0

[Buildroot] [PATCH 1/9] support/testing: use pexpect in emulator

[Ricardo Martincoski]

Replace subprocess + telnetlib with pexpect.

Use the telnet installed on the host machine instead of telnetlib, while
the serial from qemu is not yet redirected to stdio.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
Notice this patch adds some lines that will be replaced in the next
patches of the series.
---
 support/testing/infra/emulator.py | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 2480b46540..14a8bef890 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -1,6 +1,4 @@
-import socket
-import subprocess
-import telnetlib
+import pexpect
 
 import infra
 import infra.basetest
@@ -71,25 +69,24 @@ class Emulator(object):
             qemu_cmd += ["-append", " ".join(kernel_cmdline)]
 
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
-        self.qemu = subprocess.Popen(qemu_cmd, stdout=self.logfile, stderr=self.logfile)
+        self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:])
 
         # Wait for the telnet port to appear and connect to it.
-        while True:
-            try:
-                self.__tn = telnetlib.Telnet("localhost", 1234)
-                if self.__tn:
-                    break
-            except socket.error:
-                continue
+        self.qemu.expect("waiting for connection")
+        telnet_cmd = ["telnet", "localhost", "1234"]
+        self.__tn = pexpect.spawn(telnet_cmd[0], telnet_cmd[1:])
 
     def __read_until(self, waitstr, timeout=5):
-        data = self.__tn.read_until(waitstr, timeout)
+        index = self.__tn.expect([waitstr, pexpect.TIMEOUT], timeout=timeout)
+        data = self.__tn.before
+        if index == 0:
+            data += self.__tn.after
         self.log += data
         self.logfile.write(data)
         return data
 
     def __write(self, wstr):
-        self.__tn.write(wstr)
+        self.__tn.send(wstr)
 
     # Wait for the login prompt to appear, and then login as root with
     # the provided password, or no password if not specified.
@@ -124,7 +121,8 @@ class Emulator(object):
         return output, exit_code
 
     def stop(self):
+        if self.__tn:
+            self.__tn.terminate(force=True)
         if self.qemu is None:
             return
-        self.qemu.terminate()
-        self.qemu.kill()
+        self.qemu.terminate(force=True)
-- 
2.11.0

[Buildroot] [PATCH 2/9] support/testing: use qemu stdio in emulator

[Ricardo Martincoski]

Instead of redirecting qemu serial to telnet, redirect it to stdio.

It allows to run testcases in parallel without random failing caused by
two emulators trying to use the same telnet port (1234).

'qemu -serial stdio' returns some extra <CR> characters, so remove them
from the log.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
Using this patch, one can happily change the config file
'support/testing/conf/unittest.cfg' to use
[multiprocess]
processes = 2

To go further than 2, I needed the last patch of this series. This limit
certainly depends on the host machine.
---
 support/testing/infra/emulator.py | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 14a8bef890..aa1e9e5b2c 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -3,13 +3,10 @@ import pexpect
 import infra
 import infra.basetest
 
-# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
-# with the qemu machine.
 class Emulator(object):
 
     def __init__(self, builddir, downloaddir, logtofile):
         self.qemu = None
-        self.__tn = None
         self.downloaddir = downloaddir
         self.log = ""
         self.logfile = infra.open_log_file(builddir, "run", logtofile)
@@ -37,7 +34,7 @@ class Emulator(object):
             qemu_arch = arch
 
         qemu_cmd = ["qemu-system-{}".format(qemu_arch),
-                    "-serial", "telnet::1234,server",
+                    "-serial", "stdio",
                     "-display", "none"]
 
         if options:
@@ -71,22 +68,19 @@ class Emulator(object):
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
         self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:])
 
-        # Wait for the telnet port to appear and connect to it.
-        self.qemu.expect("waiting for connection")
-        telnet_cmd = ["telnet", "localhost", "1234"]
-        self.__tn = pexpect.spawn(telnet_cmd[0], telnet_cmd[1:])
-
     def __read_until(self, waitstr, timeout=5):
-        index = self.__tn.expect([waitstr, pexpect.TIMEOUT], timeout=timeout)
-        data = self.__tn.before
+        index = self.qemu.expect([waitstr, pexpect.TIMEOUT], timeout=timeout)
+        data = self.qemu.before
         if index == 0:
-            data += self.__tn.after
+            data += self.qemu.after
         self.log += data
         self.logfile.write(data)
-        return data
+        # Remove double carriage return from qemu stdout so str.splitlines()
+        # works as expected.
+        return data.replace("\r\r", "\r")
 
     def __write(self, wstr):
-        self.__tn.send(wstr)
+        self.qemu.send(wstr)
 
     # Wait for the login prompt to appear, and then login as root with
     # the provided password, or no password if not specified.
@@ -121,8 +115,6 @@ class Emulator(object):
         return output, exit_code
 
     def stop(self):
-        if self.__tn:
-            self.__tn.terminate(force=True)
         if self.qemu is None:
             return
         self.qemu.terminate(force=True)
-- 
2.11.0

[Buildroot] [PATCH 3/9] support/testing: let pexpect write stdout to log

[Ricardo Martincoski]

When the parameter logfile is passed to spawn(), pexpect sends both
stdin and stdout to the logfile and it creates a double echo effect.

One way to avoid the double echo in the logfile would be to disable the
echo on the terminal just after login ("stty -echo"), but double echo of
user and password would remain.

Instead of that, send only the stdout to the logfile using the
logfile_read property.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/infra/emulator.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index aa1e9e5b2c..55110db401 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -67,6 +67,8 @@ class Emulator(object):
 
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
         self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:])
+        # We want only stdout into the log to avoid double echo
+        self.qemu.logfile_read = self.logfile
 
     def __read_until(self, waitstr, timeout=5):
         index = self.qemu.expect([waitstr, pexpect.TIMEOUT], timeout=timeout)
@@ -74,7 +76,6 @@ class Emulator(object):
         if index == 0:
             data += self.qemu.after
         self.log += data
-        self.logfile.write(data)
         # Remove double carriage return from qemu stdout so str.splitlines()
         # works as expected.
         return data.replace("\r\r", "\r")
-- 
2.11.0

[Buildroot] [PATCH 4/9] support/testing: use pexpect.sendline directly

[Ricardo Martincoski]

When using pexpect there is no need for a helper function. Just use
sendline() directly everywhere.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/infra/emulator.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 55110db401..e488d8ac6c 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -80,9 +80,6 @@ class Emulator(object):
         # works as expected.
         return data.replace("\r\r", "\r")
 
-    def __write(self, wstr):
-        self.qemu.send(wstr)
-
     # Wait for the login prompt to appear, and then login as root with
     # the provided password, or no password if not specified.
     def login(self, password=None):
@@ -91,10 +88,10 @@ class Emulator(object):
             self.logfile.write("==> System does not boot")
             raise SystemError("System does not boot")
 
-        self.__write("root\n")
+        self.qemu.sendline("root")
         if password:
             self.__read_until("Password:")
-            self.__write(password + "\n")
+            self.qemu.sendline(password)
         self.__read_until("# ")
         if "# " not in self.log:
             raise SystemError("Cannot login")
@@ -103,12 +100,12 @@ class Emulator(object):
     # Run the given 'cmd' on the target
     # return a tuple (output, exit_code)
     def run(self, cmd):
-        self.__write(cmd + "\n")
+        self.qemu.sendline(cmd)
         output = self.__read_until("# ")
         output = output.strip().splitlines()
         output = output[1:len(output)-1]
 
-        self.__write("echo $?\n")
+        self.qemu.sendline("echo $?")
         exit_code = self.__read_until("# ")
         exit_code = exit_code.strip().splitlines()[1]
         exit_code = int(exit_code)
-- 
2.11.0

[Buildroot] [PATCH 5/9] support/testing: use pexpect.expect directly

[Ricardo Martincoski]

When using pexpect there is no need for a helper function. Just use
expect() directly everywhere.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/infra/emulator.py | 35 +++++++++++++----------------------
 1 file changed, 13 insertions(+), 22 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index e488d8ac6c..7407309a29 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -8,7 +8,6 @@ class Emulator(object):
     def __init__(self, builddir, downloaddir, logtofile):
         self.qemu = None
         self.downloaddir = downloaddir
-        self.log = ""
         self.logfile = infra.open_log_file(builddir, "run", logtofile)
 
     # Start Qemu to boot the system
@@ -66,34 +65,25 @@ class Emulator(object):
             qemu_cmd += ["-append", " ".join(kernel_cmdline)]
 
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
-        self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:])
+        self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5)
         # We want only stdout into the log to avoid double echo
         self.qemu.logfile_read = self.logfile
 
-    def __read_until(self, waitstr, timeout=5):
-        index = self.qemu.expect([waitstr, pexpect.TIMEOUT], timeout=timeout)
-        data = self.qemu.before
-        if index == 0:
-            data += self.qemu.after
-        self.log += data
-        # Remove double carriage return from qemu stdout so str.splitlines()
-        # works as expected.
-        return data.replace("\r\r", "\r")
-
     # Wait for the login prompt to appear, and then login as root with
     # the provided password, or no password if not specified.
     def login(self, password=None):
-        self.__read_until("buildroot login:", 10)
-        if "buildroot login:" not in self.log:
+        index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT],
+                                 timeout=10)
+        if index != 0:
             self.logfile.write("==> System does not boot")
             raise SystemError("System does not boot")
 
         self.qemu.sendline("root")
         if password:
-            self.__read_until("Password:")
+            self.qemu.expect("Password:")
             self.qemu.sendline(password)
-        self.__read_until("# ")
-        if "# " not in self.log:
+        index = self.qemu.expect(["# ", pexpect.TIMEOUT])
+        if index != 0:
             raise SystemError("Cannot login")
         self.run("dmesg -n 1")
 
@@ -101,13 +91,14 @@ class Emulator(object):
     # return a tuple (output, exit_code)
     def run(self, cmd):
         self.qemu.sendline(cmd)
-        output = self.__read_until("# ")
-        output = output.strip().splitlines()
-        output = output[1:len(output)-1]
+        self.qemu.expect("# ")
+        # Remove double carriage return from qemu stdout so str.splitlines()
+        # works as expected.
+        output = self.qemu.before.replace("\r\r", "\r").splitlines()[1:]
 
         self.qemu.sendline("echo $?")
-        exit_code = self.__read_until("# ")
-        exit_code = exit_code.strip().splitlines()[1]
+        self.qemu.expect("# ")
+        exit_code = self.qemu.before.splitlines()[2]
         exit_code = int(exit_code)
 
         return output, exit_code
-- 
2.11.0

[Buildroot] [PATCH 6/9] support/testing: fix code style in emulator

[Ricardo Martincoski]

Remove unused import.
Use 2 empty lines before a class.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
This patch depends on previous ones because some code with style issues
were replaced by equivalent code.

After this patch
$ flake8 support/testing/infra/emulator.py
shows no error or warning.
---
 support/testing/infra/emulator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 7407309a29..0f925fc9fb 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -1,7 +1,7 @@
 import pexpect
 
 import infra
-import infra.basetest
+
 
 class Emulator(object):
 
-- 
2.11.0

[Buildroot] [PATCH 7/9] support/testing: allow to set BR2_JLEVEL

[Ricardo Martincoski]

Let the user to override the default BR2_JLEVEL used for each testcase.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/infra/basetest.py |  6 +++++-
 support/testing/run-tests         | 10 ++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py
index 557baa215c..d75458a02c 100644
--- a/support/testing/infra/basetest.py
+++ b/support/testing/infra/basetest.py
@@ -34,6 +34,7 @@ class BRTest(unittest.TestCase):
     outputdir = None
     logtofile = True
     keepbuilds = False
+    jlevel = None
 
     def show_msg(self, msg):
         print "{} {:40s} {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
@@ -43,7 +44,10 @@ class BRTest(unittest.TestCase):
         self.builddir = os.path.join(self.__class__.outputdir, self.testname)
         self.emulator = None
         self.show_msg("Starting")
-        self.b = Builder(self.__class__.config, self.builddir, self.logtofile)
+        config = self.__class__.config
+        if self.jlevel:
+            config += "BR2_JLEVEL={}\n".format(self.jlevel)
+        self.b = Builder(config, self.builddir, self.logtofile)
 
         if not self.keepbuilds:
             self.b.delete()
diff --git a/support/testing/run-tests b/support/testing/run-tests
index 07dad0d8b9..e560ec7720 100755
--- a/support/testing/run-tests
+++ b/support/testing/run-tests
@@ -23,6 +23,8 @@ def main():
     parser.add_argument('-k', '--keep',
                         help='keep build directories',
                         action='store_true')
+    parser.add_argument('-j', '--jlevel', type=int,
+                        help='BR2_JLEVEL to use for each testcase')
 
     args = parser.parse_args()
 
@@ -70,6 +72,14 @@ def main():
 
     BRTest.keepbuilds = args.keep
 
+    if args.jlevel:
+        if args.jlevel < 0:
+            print "Invalid BR2_JLEVEL to use for each testcase"
+            print ""
+            parser.print_help()
+            return 1
+        BRTest.jlevel = args.jlevel
+
     nose2_args = ["-v",
                   "-s", "support/testing",
                   "-c", "support/testing/conf/unittest.cfg"]
-- 
2.11.0

[Buildroot] [PATCH 8/9] support/testing: run testcases in parallel

[Ricardo Martincoski]

Let the user to pass -t to set the number of testcases to run
simultaneously.

When -j is not specified, calculate it to split the available cores
between the simultaneous testcases.

Example of auto calculated -j for cpu_count 8:
  -t -j  total
   1  9  9
   2  4  8
   3  3  9
   4  2  8
 >=5  1  t

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/conf/unittest.cfg |  1 -
 support/testing/run-tests         | 18 ++++++++++++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg
index 6eaa234fc9..4f516fb80a 100644
--- a/support/testing/conf/unittest.cfg
+++ b/support/testing/conf/unittest.cfg
@@ -2,5 +2,4 @@
 plugins = nose2.plugins.mp
 
 [multiprocess]
-processes = 1
 always-on = True
diff --git a/support/testing/run-tests b/support/testing/run-tests
index e560ec7720..0cb673c61f 100755
--- a/support/testing/run-tests
+++ b/support/testing/run-tests
@@ -3,6 +3,7 @@ import argparse
 import sys
 import os
 import nose2
+import multiprocessing
 
 from infra.basetest import BRTest
 
@@ -23,6 +24,8 @@ def main():
     parser.add_argument('-k', '--keep',
                         help='keep build directories',
                         action='store_true')
+    parser.add_argument('-t', '--testcases', type=int, default=1,
+                        help='number of testcases to run simultaneously')
     parser.add_argument('-j', '--jlevel', type=int,
                         help='BR2_JLEVEL to use for each testcase')
 
@@ -72,15 +75,30 @@ def main():
 
     BRTest.keepbuilds = args.keep
 
+    if args.testcases != 1:
+        if args.testcases < 1:
+            print "Invalid number of testcases to run simultaneously"
+            print ""
+            parser.print_help()
+            return 1
+        # same default BR2_JLEVEL as package/Makefile.in
+        br2_jlevel = 1 + multiprocessing.cpu_count()
+        each_testcase = br2_jlevel / args.testcases
+        if each_testcase < 1:
+            each_testcase = 1
+        BRTest.jlevel = each_testcase
+
     if args.jlevel:
         if args.jlevel < 0:
             print "Invalid BR2_JLEVEL to use for each testcase"
             print ""
             parser.print_help()
             return 1
+        # the user can override the auto calculated value
         BRTest.jlevel = args.jlevel
 
     nose2_args = ["-v",
+                  "-N", str(args.testcases),
                   "-s", "support/testing",
                   "-c", "support/testing/conf/unittest.cfg"]
 
-- 
2.11.0

[Buildroot] [PATCH 9/9] support/testing: large timeout for login prompt

[Ricardo Martincoski]

When running multiple instances of emulator in parallel, the login
prompt can take some time to appear.

Use a large timeout when waiting for the prompt to avoid random
failures.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
The issue can be reproduced easily by running the emulator of all
current tests in parallel, e.g. run 'run-tests -k --all' and then
'run-tests -k -t 99 --all'
---
 support/testing/infra/emulator.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 0f925fc9fb..a39d59b881 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -72,8 +72,10 @@ class Emulator(object):
     # Wait for the login prompt to appear, and then login as root with
     # the provided password, or no password if not specified.
     def login(self, password=None):
+        # The login prompt can take some time to appear when running multiple
+        # instances in parallel, so set the timeout to a large value
         index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT],
-                                 timeout=10)
+                                 timeout=60)
         if index != 0:
             self.logfile.write("==> System does not boot")
             raise SystemError("System does not boot")
-- 
2.11.0

[Buildroot] [PATCH 0/9] support/testing: run tests in parallel

[Thomas Petazzoni]

Hello,

On Wed, 28 Jun 2017 23:45:40 -0300, Ricardo Martincoski wrote:

> Ricardo Martincoski (9):
>   support/testing: use pexpect in emulator
>   support/testing: use qemu stdio in emulator
>   support/testing: let pexpect write stdout to log
>   support/testing: use pexpect.sendline directly
>   support/testing: use pexpect.expect directly
>   support/testing: fix code style in emulator
>   support/testing: allow to set BR2_JLEVEL
>   support/testing: run testcases in parallel
>   support/testing: large timeout for login prompt

All applied to master. Many thanks for this work, definitely very
useful!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com