* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
@ 2017-03-20 20:36 ` Thomas Petazzoni
2017-03-22 8:11 ` Thomas De Schampheleire
` (3 more replies)
2017-03-20 20:36 ` [Buildroot] [PATCH v3 2/5] support/testing: add core tests Thomas Petazzoni
` (5 subsequent siblings)
6 siblings, 4 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-20 20:36 UTC (permalink / raw)
To: buildroot
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/conf/unittest.cfg | 6 ++
support/testing/infra/__init__.py | 89 +++++++++++++++++++++++++
support/testing/infra/basetest.py | 66 +++++++++++++++++++
support/testing/infra/builder.py | 49 ++++++++++++++
support/testing/infra/emulator.py | 135 ++++++++++++++++++++++++++++++++++++++
support/testing/run-tests | 83 +++++++++++++++++++++++
support/testing/tests/__init__.py | 0
7 files changed, 428 insertions(+)
create mode 100644 support/testing/conf/unittest.cfg
create mode 100644 support/testing/infra/__init__.py
create mode 100644 support/testing/infra/basetest.py
create mode 100644 support/testing/infra/builder.py
create mode 100644 support/testing/infra/emulator.py
create mode 100755 support/testing/run-tests
create mode 100644 support/testing/tests/__init__.py
diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg
new file mode 100644
index 0000000..6eaa234
--- /dev/null
+++ b/support/testing/conf/unittest.cfg
@@ -0,0 +1,6 @@
+[unittest]
+plugins = nose2.plugins.mp
+
+[multiprocess]
+processes = 1
+always-on = True
diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
new file mode 100644
index 0000000..c3f645c
--- /dev/null
+++ b/support/testing/infra/__init__.py
@@ -0,0 +1,89 @@
+import contextlib
+import os
+import re
+import sys
+import tempfile
+import subprocess
+from urllib2 import urlopen, HTTPError, URLError
+
+ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
+
+ at contextlib.contextmanager
+def smart_open(filename=None):
+ """
+ Return a file-like object that can be written to using the 'with'
+ keyword, as in the example:
+ with infra.smart_open("test.log") as outfile:
+ outfile.write("Hello, world!\n")
+ """
+ if filename and filename != '-':
+ fhandle = open(filename, 'a+')
+ else:
+ fhandle = sys.stdout
+
+ try:
+ yield fhandle
+ finally:
+ if fhandle is not sys.stdout:
+ fhandle.close()
+
+def filepath(relpath):
+ return os.path.join(os.getcwd(), "support/testing", relpath)
+
+def download(dldir, filename):
+ finalpath = os.path.join(dldir, filename)
+ if os.path.exists(finalpath):
+ return finalpath
+
+ if not os.path.exists(dldir):
+ os.makedirs(dldir)
+
+ tmpfile = tempfile.mktemp(dir=dldir)
+ print "Downloading to {}".format(tmpfile)
+
+ try:
+ url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename))
+ with open(tmpfile, "w+") as tmpfile_fh:
+ tmpfile_fh.write(url_fh.read())
+ except (HTTPError, URLError), err:
+ os.unlink(tmpfile)
+ raise err
+
+ print "Renaming from %s to %s" % (tmpfile, finalpath)
+ os.rename(tmpfile, finalpath)
+ return finalpath
+
+def get_elf_arch_tag(builddir, prefix, fpath, tag):
+ """
+ Runs the cross readelf on 'fpath', then extracts the value of tag 'tag'.
+ Example:
+ >>> get_elf_arch_tag('output', 'arm-none-linux-gnueabi-',
+ 'bin/busybox', 'Tag_CPU_arch')
+ v5TEJ
+ >>>
+ """
+ cmd = ["host/usr/bin/{}-readelf".format(prefix),
+ "-A", os.path.join("target", fpath)]
+ out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
+ regexp = re.compile("^ {}: (.*)$".format(tag))
+ for line in out.splitlines():
+ m = regexp.match(line)
+ if not m:
+ continue
+ return m.group(1)
+ return None
+
+def get_file_arch(builddir, prefix, fpath):
+ return get_elf_arch_tag(builddir, prefix, fpath, "Tag_CPU_arch")
+
+def get_elf_prog_interpreter(builddir, prefix, fpath):
+ cmd = ["host/usr/bin/{}-readelf".format(prefix),
+ "-l", os.path.join("target", fpath)]
+ out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
+ regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$")
+ for line in out.splitlines():
+ m = regexp.match(line)
+ if not m:
+ continue
+ return m.group(1)
+ return None
diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py
new file mode 100644
index 0000000..eb9da90
--- /dev/null
+++ b/support/testing/infra/basetest.py
@@ -0,0 +1,66 @@
+import unittest
+import os
+import datetime
+
+from infra.builder import Builder
+from infra.emulator import Emulator
+
+BASIC_TOOLCHAIN_CONFIG = \
+"""
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2015.05-1190-g4a48479.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_4_7=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
+BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+"""
+
+MINIMAL_CONFIG = \
+"""
+BR2_INIT_NONE=y
+BR2_SYSTEM_BIN_SH_NONE=y
+# BR2_PACKAGE_BUSYBOX is not set
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+class BRTest(unittest.TestCase):
+ config = None
+ downloaddir = None
+ outputdir = None
+ logtofile = True
+ keepbuilds = False
+
+ def show_msg(self, msg):
+ print "[%s/%s/%s] %s" % (os.path.basename(self.__class__.outputdir),
+ self.testname,
+ datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+ msg)
+ def setUp(self):
+ self.testname = self.__class__.__name__
+ self.builddir = os.path.join(self.__class__.outputdir, self.testname)
+ self.runlog = self.builddir + "-run.log"
+ self.emulator = None
+ self.show_msg("Starting")
+ self.b = Builder(self.__class__.config, self.builddir, self.logtofile)
+
+ if not self.keepbuilds:
+ self.b.delete()
+
+ if not self.b.is_finished():
+ self.show_msg("Building")
+ self.b.build()
+ self.show_msg("Building done")
+
+ self.emulator = Emulator(self.builddir, self.downloaddir, self.logtofile)
+
+ def tearDown(self):
+ self.show_msg("Cleaning up")
+ if self.emulator:
+ self.emulator.stop()
+ if self.b and not self.keepbuilds:
+ self.b.delete()
diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
new file mode 100644
index 0000000..105da01
--- /dev/null
+++ b/support/testing/infra/builder.py
@@ -0,0 +1,49 @@
+import os
+import shutil
+import subprocess
+
+import infra
+
+class Builder(object):
+ def __init__(self, config, builddir, logtofile):
+ self.config = config
+ self.builddir = builddir
+ self.logtofile = logtofile
+
+ def build(self):
+ if not os.path.isdir(self.builddir):
+ os.makedirs(self.builddir)
+
+ log = "{}-build.log".format(self.builddir)
+ if not self.logtofile:
+ log = None
+
+ config_file = os.path.join(self.builddir, ".config")
+ with open(config_file, "w+") as cf:
+ cf.write(self.config)
+
+ cmd = ["make",
+ "O={}".format(self.builddir),
+ "olddefconfig"]
+ with infra.smart_open(log) as log_fh:
+ ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh)
+ if ret != 0:
+ raise SystemError("Cannot olddefconfig")
+
+ cmd = ["make", "-C", self.builddir]
+ with infra.smart_open(log) as log_fh:
+ ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh)
+ if ret != 0:
+ raise SystemError("Build failed")
+
+ open(self.stamp_path(), 'a').close()
+
+ def stamp_path(self):
+ return os.path.join(self.builddir, "build-done")
+
+ def is_finished(self):
+ return os.path.exists(self.stamp_path())
+
+ def delete(self):
+ if os.path.exists(self.builddir):
+ shutil.rmtree(self.builddir)
diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
new file mode 100644
index 0000000..7c476cb
--- /dev/null
+++ b/support/testing/infra/emulator.py
@@ -0,0 +1,135 @@
+import socket
+import subprocess
+import telnetlib
+
+import infra
+import infra.basetest
+
+# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
+# with the qemu machine.
+class Emulator(object):
+
+ def __init__(self, builddir, downloaddir, logtofile):
+ self.qemu = None
+ self.__tn = None
+ self.downloaddir = downloaddir
+ self.log = ""
+ self.log_file = "{}-run.log".format(builddir)
+ if logtofile is None:
+ self.log_file = None
+
+ # Start Qemu to boot the system
+ #
+ # arch: Qemu architecture to use
+ #
+ # kernel: path to the kernel image, or the special string
+ # 'builtin'. 'builtin' means a pre-built kernel image will be
+ # downloaded from ARTEFACTS_URL and suitable options are
+ # automatically passed to qemu and added to the kernel cmdline. So
+ # far only armv5, armv7 and i386 builtin kernels are available.
+ # If None, then no kernel is used, and we assume a bootable device
+ # will be specified.
+ #
+ # kernel_cmdline: array of kernel arguments to pass to Qemu -append option
+ #
+ # options: array of command line options to pass to Qemu
+ #
+ def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
+ if arch in ["armv7", "armv5"]:
+ qemu_arch = "arm"
+ else:
+ qemu_arch = arch
+
+ qemu_cmd = ["qemu-system-{}".format(qemu_arch),
+ "-serial", "telnet::1234,server",
+ "-display", "none"]
+
+ if options:
+ qemu_cmd += options
+
+ if kernel_cmdline is None:
+ kernel_cmdline = []
+
+ if kernel:
+ if kernel == "builtin":
+ if arch in ["armv7", "armv5"]:
+ kernel_cmdline.append("console=ttyAMA0")
+
+ if arch == "armv7":
+ kernel = infra.download(self.downloaddir,
+ "kernel-vexpress")
+ dtb = infra.download(self.downloaddir,
+ "vexpress-v2p-ca9.dtb")
+ qemu_cmd += ["-dtb", dtb]
+ qemu_cmd += ["-M", "vexpress-a9"]
+ elif arch == "armv5":
+ kernel = infra.download(self.downloaddir,
+ "kernel-versatile")
+ qemu_cmd += ["-M", "versatilepb"]
+
+ qemu_cmd += ["-kernel", kernel]
+
+ if kernel_cmdline:
+ qemu_cmd += ["-append", " ".join(kernel_cmdline)]
+
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
+ self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh)
+
+ # Wait for the telnet port to appear and connect to it.
+ while True:
+ try:
+ self.__tn = telnetlib.Telnet("localhost", 1234)
+ if self.__tn:
+ break
+ except socket.error:
+ continue
+
+ def __read_until(self, waitstr, timeout=5):
+ data = self.__tn.read_until(waitstr, timeout)
+ self.log += data
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write(data)
+ return data
+
+ def __write(self, wstr):
+ self.__tn.write(wstr)
+
+ # Wait for the login prompt to appear, and then login as root with
+ # the provided password, or no password if not specified.
+ def login(self, password=None):
+ self.__read_until("buildroot login:", 10)
+ if "buildroot login:" not in self.log:
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("==> System does not boot")
+ raise SystemError("System does not boot")
+
+ self.__write("root\n")
+ if password:
+ self.__read_until("Password:")
+ self.__write(password + "\n")
+ self.__read_until("# ")
+ if "# " not in self.log:
+ raise SystemError("Cannot login")
+ self.run("dmesg -n 1")
+
+ # Run the given 'cmd' on the target
+ # return a tuple (output, exit_code)
+ def run(self, cmd):
+ self.__write(cmd + "\n")
+ output = self.__read_until("# ")
+ output = output.strip().splitlines()
+ output = output[1:len(output)-1]
+
+ self.__write("echo $?\n")
+ exit_code = self.__read_until("# ")
+ exit_code = exit_code.strip().splitlines()[1]
+ exit_code = int(exit_code)
+
+ return output, exit_code
+
+ def stop(self):
+ if self.qemu is None:
+ return
+ self.qemu.terminate()
+ self.qemu.kill()
diff --git a/support/testing/run-tests b/support/testing/run-tests
new file mode 100755
index 0000000..339bb66
--- /dev/null
+++ b/support/testing/run-tests
@@ -0,0 +1,83 @@
+#!/usr/bin/env python2
+import argparse
+import sys
+import os
+import nose2
+
+from infra.basetest import BRTest
+
+def main():
+ parser = argparse.ArgumentParser(description='Run Buildroot tests')
+ parser.add_argument('testname', nargs='*',
+ help='list of test cases to execute')
+ parser.add_argument('--list', '-l', action='store_true',
+ help='list of available test cases')
+ parser.add_argument('--all', '-a', action='store_true',
+ help='execute all test cases')
+ parser.add_argument('--stdout', '-s', action='store_true',
+ help='log everything to stdout')
+ parser.add_argument('--output', '-o',
+ help='output directory')
+ parser.add_argument('--download', '-d',
+ help='download directory')
+ parser.add_argument('--keep', '-k',
+ help='keep build directories',
+ action='store_true')
+
+ args = parser.parse_args()
+
+ script_path = os.path.realpath(__file__)
+ test_dir = os.path.dirname(script_path)
+
+ if args.stdout:
+ BRTest.logtofile = False
+
+ if args.list:
+ print "List of tests"
+ nose2.discover(argv=[script_path,
+ "-s", test_dir,
+ "-v",
+ "--collect-only"],
+ plugins=["nose2.plugins.collect"])
+ return 0
+
+ if args.download is None:
+ args.download = os.getenv("BR2_DL_DIR")
+ if args.download is None:
+ print "Missing download directory, please use -d/--download"
+ print ""
+ parser.print_help()
+ return 1
+
+ BRTest.downloaddir = os.path.abspath(args.download)
+
+ if args.output is None:
+ print "Missing output directory, please use -o/--output"
+ print ""
+ parser.print_help()
+ return 1
+
+ if not os.path.exists(args.output):
+ os.mkdir(args.output)
+
+ BRTest.outputdir = os.path.abspath(args.output)
+
+ if args.all is False and len(args.testname) == 0:
+ print "No test selected"
+ print ""
+ parser.print_help()
+ return 1
+
+ BRTest.keepbuilds = args.keep
+
+ nose2_args = ["-v",
+ "-s", "support/testing",
+ "-c", "support/testing/conf/unittest.cfg"]
+
+ if len(args.testname) != 0:
+ nose2_args += args.testname
+
+ nose2.discover(argv=nose2_args)
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/support/testing/tests/__init__.py b/support/testing/tests/__init__.py
new file mode 100644
index 0000000..e69de29
--
2.7.4
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
@ 2017-03-22 8:11 ` Thomas De Schampheleire
2017-03-23 3:02 ` Ricardo Martincoski
` (2 subsequent siblings)
3 siblings, 0 replies; 24+ messages in thread
From: Thomas De Schampheleire @ 2017-03-22 8:11 UTC (permalink / raw)
To: buildroot
On Mon, Mar 20, 2017 at 9:36 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> This commit adds the core of a new testing infrastructure that allows to
> perform runtime testing of Buildroot generated systems. This
> infrastructure uses the Python unittest logic as its foundation.
>
> This core infrastructure commit includes the following aspects:
>
> - A base test class, called BRTest, defined in
> support/testing/infra/basetest.py. This base test class inherited
> from the Python provided unittest.TestCase, and must be subclassed by
> all Buildroot test cases.
>
> Its main purpose is to provide the Python unittest setUp() and
> tearDown() methods. In our case, setUp() takes care of building the
> Buildroot system described in the test case, and instantiate the
> Emulator object in case runtime testing is needed. The tearDown()
> method simply cleans things up (stop the emulator, remove the output
> directory).
>
> - A Builder class, defined in support/testing/infra/builder.py, simply
> responsible for building the Buildroot system in each test case.
>
> - An Emulator class, defined in support/testing/infra/emulator.py,
> responsible for running the generated system under Qemu, allowing
> each test case to run arbitrary commands inside the emulated system.
>
> - A run-tests script, which is the entry point to start the tests.
>
> Even though I wrote the original version of this small infrastructure, a
> huge amount of rework and improvement has been done by Maxime
> Hadjinlian, and squashed into this patch. So many thanks to Maxime for
> cleaning up and improving my Python code!
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/conf/unittest.cfg | 6 ++
> support/testing/infra/__init__.py | 89 +++++++++++++++++++++++++
> support/testing/infra/basetest.py | 66 +++++++++++++++++++
> support/testing/infra/builder.py | 49 ++++++++++++++
> support/testing/infra/emulator.py | 135 ++++++++++++++++++++++++++++++++++++++
> support/testing/run-tests | 83 +++++++++++++++++++++++
> support/testing/tests/__init__.py | 0
> 7 files changed, 428 insertions(+)
> create mode 100644 support/testing/conf/unittest.cfg
> create mode 100644 support/testing/infra/__init__.py
> create mode 100644 support/testing/infra/basetest.py
> create mode 100644 support/testing/infra/builder.py
> create mode 100644 support/testing/infra/emulator.py
> create mode 100755 support/testing/run-tests
> create mode 100644 support/testing/tests/__init__.py
>
[..]
> diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
> new file mode 100644
> index 0000000..7c476cb
> --- /dev/null
> +++ b/support/testing/infra/emulator.py
> @@ -0,0 +1,135 @@
> +import socket
> +import subprocess
> +import telnetlib
> +
> +import infra
> +import infra.basetest
> +
> +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
> +# with the qemu machine.
> +class Emulator(object):
> +
> + def __init__(self, builddir, downloaddir, logtofile):
> + self.qemu = None
> + self.__tn = None
> + self.downloaddir = downloaddir
> + self.log = ""
> + self.log_file = "{}-run.log".format(builddir)
> + if logtofile is None:
> + self.log_file = None
> +
> + # Start Qemu to boot the system
> + #
> + # arch: Qemu architecture to use
> + #
> + # kernel: path to the kernel image, or the special string
> + # 'builtin'. 'builtin' means a pre-built kernel image will be
> + # downloaded from ARTEFACTS_URL and suitable options are
There is a stale reference to ARTEFACTS_URL here (ARTIFACTS_URL)
[..]
> + # Wait for the login prompt to appear, and then login as root with
> + # the provided password, or no password if not specified.
> + def login(self, password=None):
> + self.__read_until("buildroot login:", 10)
> + if "buildroot login:" not in self.log:
If we want people to use this testing infrastructure on their own
defconfigs too, then encoding the hostname in this string is a bit
restrictive. What about searching for 'login:' only ? Do you think it
is too generic?
[..]
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
2017-03-22 8:11 ` Thomas De Schampheleire
@ 2017-03-23 3:02 ` Ricardo Martincoski
2017-03-23 8:07 ` Thomas Petazzoni
2017-03-26 21:54 ` Luca Ceresoli
2017-05-07 21:07 ` Luca Ceresoli
3 siblings, 1 reply; 24+ messages in thread
From: Ricardo Martincoski @ 2017-03-23 3:02 UTC (permalink / raw)
To: buildroot
Thomas,
There are still 2 things that I think can be solved together.
On Mon, Mar 20, 2017 at 05:36 PM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/conf/unittest.cfg
> @@ -0,0 +1,6 @@
> +[unittest]
> +plugins = nose2.plugins.mp
> +
> +[multiprocess]
> +processes = 1
1) If I increase the value of processes in unittest.cfg to unleash the power of
nose2, emulator fails sporadically because telnet port is fixed.
[snip]
> +++ b/support/testing/infra/emulator.py
[snip]
> + qemu_cmd = ["qemu-system-{}".format(qemu_arch),
> + "-serial", "telnet::1234,server",
> + "-display", "none"]
2) If I do this (not restricted to these testcases) ...
for i in $(seq 100); do \
support/testing/run-tests -d dl1 -o output -k tests.package || break ; done
... while performing other resource-consuming tasks on my PC, such as starting
an unrelated build that uses all the cores, I eventually get this:
ERROR: test_run (tests.package.test_dropbear.TestDropbear)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v3/support/testing/tests/package/test_dropbear.py", line 22, in test_run
self.emulator.login("testpwd")
File "/tmp/testing-v3/support/testing/infra/emulator.py", line 101, in login
self.__read_until("buildroot login:", 10)
File "/tmp/testing-v3/support/testing/infra/emulator.py", line 89, in __read_until
data = self.__tn.read_until(waitstr, timeout)
File "/usr/lib/python2.7/telnetlib.py", line 294, in read_until
return self._read_until_with_poll(match, timeout)
File "/usr/lib/python2.7/telnetlib.py", line 329, in _read_until_with_poll
self.fill_rawq()
File "/usr/lib/python2.7/telnetlib.py", line 576, in fill_rawq
buf = self.sock.recv(50)
error: [Errno 104] Connection reset by peer
Something like this patch could fix those:
+++ b/support/testing/infra/emulator.py
@@ -1,3 +1,3 @@
+import pexpect
import socket
-import subprocess
import telnetlib
@@ -7,2 +7,5 @@ import infra.basetest
+TELNET_PORT_INITIAL = 1234
+TELNET_PORT_LAST = TELNET_PORT_INITIAL + 20
+
# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
@@ -13,2 +16,3 @@ class Emulator(object):
self.qemu = None
+ self.port = None
self.__tn = None
@@ -43,3 +47,2 @@ class Emulator(object):
qemu_cmd = ["qemu-system-{}".format(qemu_arch),
- "-serial", "telnet::1234,server",
"-display", "none"]
@@ -75,4 +78,15 @@ class Emulator(object):
with infra.smart_open(self.log_file) as lfh:
- lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
- self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh)
+ for port in range(TELNET_PORT_INITIAL, TELNET_PORT_LAST):
+ telnet_arg = ["-serial", "telnet::{},server".format(port)]
+ cmd = qemu_cmd + telnet_arg
+ lfh.write("> starting qemu with '{}'\n".format(" ".join(cmd)))
+ self.qemu = pexpect.spawn(cmd[0], cmd[1:], logfile=lfh)
+ ret = self.qemu.expect(["waiting for connection",
+ "Address already in use"])
+ if ret == 0:
+ self.port = port
+ break
+ self.qemu.read()
+ if not self.port:
+ raise SystemError("Could not find a free port to run emulator")
@@ -81,3 +95,3 @@ class Emulator(object):
try:
- self.__tn = telnetlib.Telnet("localhost", 1234)
+ self.__tn = telnetlib.Telnet("localhost", self.port)
if self.__tn:
@@ -100,6 +114,8 @@ class Emulator(object):
def login(self, password=None):
- self.__read_until("buildroot login:", 10)
+ # The login prompt can take some time to appear when running multiple
+ # instances in parallel, so set the timeout to a large value
+ self.__read_until("buildroot login:", 30)
if "buildroot login:" not in self.log:
with infra.smart_open(self.log_file) as lfh:
- lfh.write("==> System does not boot")
+ lfh.write("==> System does not boot\n")
raise SystemError("System does not boot")
@@ -133,3 +149,2 @@ class Emulator(object):
return
- self.qemu.terminate()
- self.qemu.kill()
+ self.qemu.terminate(force=True)
Regards,
Ricardo
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-23 3:02 ` Ricardo Martincoski
@ 2017-03-23 8:07 ` Thomas Petazzoni
2017-03-24 2:19 ` Ricardo Martincoski
0 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-23 8:07 UTC (permalink / raw)
To: buildroot
Hello,
On Thu, 23 Mar 2017 00:02:30 -0300, Ricardo Martincoski wrote:
> 1) If I increase the value of processes in unittest.cfg to unleash the power of
> nose2, emulator fails sporadically because telnet port is fixed.
Yes, this is a known problem, and there's a TODO about this in
emulator.py:
# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
# with the qemu machine.
> 2) If I do this (not restricted to these testcases) ...
>
> for i in $(seq 100); do \
> support/testing/run-tests -d dl1 -o output -k tests.package || break ; done
>
> ... while performing other resource-consuming tasks on my PC, such as starting
> an unrelated build that uses all the cores, I eventually get this:
>
> ERROR: test_run (tests.package.test_dropbear.TestDropbear)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/tmp/testing-v3/support/testing/tests/package/test_dropbear.py", line 22, in test_run
> self.emulator.login("testpwd")
> File "/tmp/testing-v3/support/testing/infra/emulator.py", line 101, in login
> self.__read_until("buildroot login:", 10)
> File "/tmp/testing-v3/support/testing/infra/emulator.py", line 89, in __read_until
> data = self.__tn.read_until(waitstr, timeout)
> File "/usr/lib/python2.7/telnetlib.py", line 294, in read_until
> return self._read_until_with_poll(match, timeout)
> File "/usr/lib/python2.7/telnetlib.py", line 329, in _read_until_with_poll
> self.fill_rawq()
> File "/usr/lib/python2.7/telnetlib.py", line 576, in fill_rawq
> buf = self.sock.recv(50)
> error: [Errno 104] Connection reset by peer
Hum, do you know why this one happens? I definitely understand that
will a qemu is running, another qemu cannot bind its telnet server on
the same TCP port, but I don't see why it would cause a "Connection
reset by peer".
Or is the logic of the test bogus, and if it doesn't manage to start
qemu, the test still continues, and it in fact connects to an already
running qemu instance from a different test, which then gets killed in
the middle ?
> Something like this patch could fix those:
It would probably be nicer to interact through stdio and pexpect, but
in the mean time, this does the job. I'll integrate that into my
series. Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-23 8:07 ` Thomas Petazzoni
@ 2017-03-24 2:19 ` Ricardo Martincoski
0 siblings, 0 replies; 24+ messages in thread
From: Ricardo Martincoski @ 2017-03-24 2:19 UTC (permalink / raw)
To: buildroot
Hello,
On Thu, Mar 23, 2017 at 05:07 AM, Thomas Petazzoni wrote:
>> error: [Errno 104] Connection reset by peer
>
> Hum, do you know why this one happens? I definitely understand that
> will a qemu is running, another qemu cannot bind its telnet server on
> the same TCP port, but I don't see why it would cause a "Connection
> reset by peer".
It is not clear to me why this one happens.
> Or is the logic of the test bogus, and if it doesn't manage to start
> qemu, the test still continues, and it in fact connects to an already
> running qemu instance from a different test, which then gets killed in
> the middle ?
I redirected the output from Popen to stdout and stderr and when the error
occurs only the messages from the first qemu appear before the Traceback:
[output/TestPythonBase/2017-03-23 20:52:29] Starting
QEMU waiting for connection on: disconnected:telnet::1234,server
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
[output/TestPythonBase/2017-03-23 20:52:37] Cleaning up
.[output/TestDropbear/2017-03-23 20:52:37] Starting
[output/TestDropbear/2017-03-23 20:52:37] Cleaning up
E
So I suppose it could be a bogus logic.
Anyway I think waiting for "waiting for connection" before connecting should be
more reliable to know the qemu started than just trying to connect using
telnet.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
2017-03-22 8:11 ` Thomas De Schampheleire
2017-03-23 3:02 ` Ricardo Martincoski
@ 2017-03-26 21:54 ` Luca Ceresoli
2017-05-07 21:07 ` Luca Ceresoli
3 siblings, 0 replies; 24+ messages in thread
From: Luca Ceresoli @ 2017-03-26 21:54 UTC (permalink / raw)
To: buildroot
Hi,
only a minor nit that I noticed while reviewing patch 5.
On 20/03/2017 21:36, Thomas Petazzoni wrote:
[...]
> diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
> new file mode 100644
> index 0000000..c3f645c
> --- /dev/null
> +++ b/support/testing/infra/__init__.py
[...]
> +def get_elf_prog_interpreter(builddir, prefix, fpath):
> + cmd = ["host/usr/bin/{}-readelf".format(prefix),
> + "-l", os.path.join("target", fpath)]
> + out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
> + regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$")
> + for line in out.splitlines():
> + m = regexp.match(line)
> + if not m:
> + continue
> + return m.group(1)
> + return None
This should be documented, just like get_elf_arch_tag() above. Proposal
(unchecked):
"""
Runs the cross readelf on 'fpath' to extract the program interpreter
name and returns it.
Example:
>>> get_elf_prog_interpreter('output', 'arm-none-linux-gnueabi-',
'bin/busybox')
/lib/ld-uClibc.so.0
>>>
"""
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
` (2 preceding siblings ...)
2017-03-26 21:54 ` Luca Ceresoli
@ 2017-05-07 21:07 ` Luca Ceresoli
2017-05-07 21:38 ` Thomas Petazzoni
3 siblings, 1 reply; 24+ messages in thread
From: Luca Ceresoli @ 2017-05-07 21:07 UTC (permalink / raw)
To: buildroot
Hi,
this the review I had written a few days ago. Since the patch has just
been applied, I've reworded it as a list of proposed improvements and
request for clarifications.
On 20/03/2017 21:36, Thomas Petazzoni wrote:
> This commit adds the core of a new testing infrastructure that allows to
> perform runtime testing of Buildroot generated systems. This
> infrastructure uses the Python unittest logic as its foundation.
Good we have this on master now!
Things that I already commented about and for which I plan to send patches:
- remove smart_open
- document code
- better reporting
[...]
> diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
> new file mode 100644
> index 0000000..c3f645c
> --- /dev/null
> +++ b/support/testing/infra/__init__.py
> @@ -0,0 +1,89 @@
> +import contextlib
> +import os
> +import re
> +import sys
> +import tempfile
> +import subprocess
> +from urllib2 import urlopen, HTTPError, URLError
> +
> +ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
Since you renamed artefacts to artifacts, it's probably good to rename
in the URL too (and update the server). Sure it's a minor nit, but since
it's an URL that should stay available for long, better having it fixed
as soon as possible.
[...]
> diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
> new file mode 100644
> index 0000000..7c476cb
> --- /dev/null
> +++ b/support/testing/infra/emulator.py
> @@ -0,0 +1,135 @@
> +import socket
> +import subprocess
> +import telnetlib
> +
> +import infra
> +import infra.basetest
> +
> +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
> +# with the qemu machine.
> +class Emulator(object):
> +
> + def __init__(self, builddir, downloaddir, logtofile):
> + self.qemu = None
> + self.__tn = None
> + self.downloaddir = downloaddir
> + self.log = ""
> + self.log_file = "{}-run.log".format(builddir)
> + if logtofile is None:
> + self.log_file = None
> +
> + # Start Qemu to boot the system
> + #
> + # arch: Qemu architecture to use
> + #
> + # kernel: path to the kernel image, or the special string
> + # 'builtin'. 'builtin' means a pre-built kernel image will be
> + # downloaded from ARTEFACTS_URL and suitable options are
> + # automatically passed to qemu and added to the kernel cmdline. So
> + # far only armv5, armv7 and i386 builtin kernels are available.
> + # If None, then no kernel is used, and we assume a bootable device
> + # will be specified.
> + #
> + # kernel_cmdline: array of kernel arguments to pass to Qemu -append option
> + #
> + # options: array of command line options to pass to Qemu
> + #
> + def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
> + if arch in ["armv7", "armv5"]:
> + qemu_arch = "arm"
> + else:
> + qemu_arch = arch
> +
> + qemu_cmd = ["qemu-system-{}".format(qemu_arch),
> + "-serial", "telnet::1234,server",
> + "-display", "none"]
> +
> + if options:
> + qemu_cmd += options
> +
> + if kernel_cmdline is None:
> + kernel_cmdline = []
> +
> + if kernel:
> + if kernel == "builtin":
> + if arch in ["armv7", "armv5"]:
> + kernel_cmdline.append("console=ttyAMA0")
> +
> + if arch == "armv7":
> + kernel = infra.download(self.downloaddir,
> + "kernel-vexpress")
> + dtb = infra.download(self.downloaddir,
> + "vexpress-v2p-ca9.dtb")
> + qemu_cmd += ["-dtb", dtb]
> + qemu_cmd += ["-M", "vexpress-a9"]
> + elif arch == "armv5":
> + kernel = infra.download(self.downloaddir,
> + "kernel-versatile")
> + qemu_cmd += ["-M", "versatilepb"]
> +
> + qemu_cmd += ["-kernel", kernel]
I'm OK with the "builtin" logic, but I really dislike it being
hard-coded in Emulator.boot(). It's acceptable for the moment since we
have only very few builtin kernels. Should we add more in the future, I
think we should at least have a sort of "database" of builtin kernels,
perhaps in the form of an associative array.
> diff --git a/support/testing/run-tests b/support/testing/run-tests
> new file mode 100755
> index 0000000..339bb66
> --- /dev/null
> +++ b/support/testing/run-tests
> @@ -0,0 +1,83 @@
> +#!/usr/bin/env python2
> +import argparse
> +import sys
> +import os
> +import nose2
> +
> +from infra.basetest import BRTest
> +
> +def main():
> + parser = argparse.ArgumentParser(description='Run Buildroot tests')
> + parser.add_argument('testname', nargs='*',
> + help='list of test cases to execute')
> + parser.add_argument('--list', '-l', action='store_true',
> + help='list of available test cases')
> + parser.add_argument('--all', '-a', action='store_true',
> + help='execute all test cases')
> + parser.add_argument('--stdout', '-s', action='store_true',
> + help='log everything to stdout')
> + parser.add_argument('--output', '-o',
> + help='output directory')
> + parser.add_argument('--download', '-d',
> + help='download directory')
> + parser.add_argument('--keep', '-k',
> + help='keep build directories',
> + action='store_true')
Stylish note: I would put the one-letter form before the long form. This
is not only my personal taste, but also what the manpages usually do,
and what Python does with the automatically-added -h/--help parameter:
$ ./support/testing/run-tests
[...]
optional arguments:
-h, --help show this help message and exit
--list, -l list of available test cases
I'll send a patch.
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 1/5] support/testing: core testing infrastructure
2017-05-07 21:07 ` Luca Ceresoli
@ 2017-05-07 21:38 ` Thomas Petazzoni
0 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2017-05-07 21:38 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, 7 May 2017 23:07:31 +0200, Luca Ceresoli wrote:
> Things that I already commented about and for which I plan to send patches:
> - remove smart_open
> - document code
> - better reporting
Great!
> > +ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
>
> Since you renamed artefacts to artifacts, it's probably good to rename
> in the URL too (and update the server). Sure it's a minor nit, but since
> it's an URL that should stay available for long, better having it fixed
> as soon as possible.
Correct, I'll fix that up.
> > + qemu_cmd += ["-kernel", kernel]
>
> I'm OK with the "builtin" logic, but I really dislike it being
> hard-coded in Emulator.boot(). It's acceptable for the moment since we
> have only very few builtin kernels. Should we add more in the future, I
> think we should at least have a sort of "database" of builtin kernels,
> perhaps in the form of an associative array.
Fully agreed. What we have now is OK for a few kernel/qemu setups, but
clearly does not scale, and will have to be improved.
> Stylish note: I would put the one-letter form before the long form. This
> is not only my personal taste, but also what the manpages usually do,
> and what Python does with the automatically-added -h/--help parameter:
Indeed. Patch welcome! :-)
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 2/5] support/testing: add core tests
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
@ 2017-03-20 20:36 ` Thomas Petazzoni
2017-03-22 8:17 ` Thomas De Schampheleire
2017-03-20 20:36 ` [Buildroot] [PATCH v3 3/5] support/testing: add fs tests Thomas Petazzoni
` (4 subsequent siblings)
6 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-20 20:36 UTC (permalink / raw)
To: buildroot
This commit adds a few Buildroot "core" tests, testing functionalities
such as:
- post-build and post-image scripts
- root filesystem overlays
- timezone support
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/core/__init__.py | 0
support/testing/tests/core/post-build.sh | 12 ++++
support/testing/tests/core/post-image.sh | 12 ++++
.../testing/tests/core/rootfs-overlay1/test-file1 | 1 +
.../tests/core/rootfs-overlay2/etc/test-file2 | 1 +
support/testing/tests/core/test_post_scripts.py | 40 +++++++++++++
support/testing/tests/core/test_rootfs_overlay.py | 27 +++++++++
support/testing/tests/core/test_timezone.py | 66 ++++++++++++++++++++++
8 files changed, 159 insertions(+)
create mode 100644 support/testing/tests/core/__init__.py
create mode 100755 support/testing/tests/core/post-build.sh
create mode 100755 support/testing/tests/core/post-image.sh
create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
create mode 100644 support/testing/tests/core/test_post_scripts.py
create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
create mode 100644 support/testing/tests/core/test_timezone.py
diff --git a/support/testing/tests/core/__init__.py b/support/testing/tests/core/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/core/post-build.sh b/support/testing/tests/core/post-build.sh
new file mode 100755
index 0000000..b1fb834
--- /dev/null
+++ b/support/testing/tests/core/post-build.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+(
+printf "arg1,%s\n" "${1}"
+printf "arg2,%s\n" "${2}"
+printf "arg3,%s\n" "${3}"
+printf "TARGET_DIR,%s\n" "${TARGET_DIR}"
+printf "BUILD_DIR,%s\n" "${BUILD_DIR}"
+printf "HOST_DIR,%s\n" "${HOST_DIR}"
+printf "STAGING_DIR,%s\n" "${STAGING_DIR}"
+printf "BINARIES_DIR,%s\n" "${BINARIES_DIR}"
+printf "BR2_CONFIG,%s\n" "${BR2_CONFIG}"
+) > ${BUILD_DIR}/post-build.log
diff --git a/support/testing/tests/core/post-image.sh b/support/testing/tests/core/post-image.sh
new file mode 100755
index 0000000..1c2a0f2
--- /dev/null
+++ b/support/testing/tests/core/post-image.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+(
+printf "arg1,%s\n" "${1}"
+printf "arg2,%s\n" "${2}"
+printf "arg3,%s\n" "${3}"
+printf "TARGET_DIR,%s\n" "${TARGET_DIR}"
+printf "BUILD_DIR,%s\n" "${BUILD_DIR}"
+printf "HOST_DIR,%s\n" "${HOST_DIR}"
+printf "STAGING_DIR,%s\n" "${STAGING_DIR}"
+printf "BINARIES_DIR,%s\n" "${BINARIES_DIR}"
+printf "BR2_CONFIG,%s\n" "${BR2_CONFIG}"
+) > ${BUILD_DIR}/post-image.log
diff --git a/support/testing/tests/core/rootfs-overlay1/test-file1 b/support/testing/tests/core/rootfs-overlay1/test-file1
new file mode 100644
index 0000000..323fae0
--- /dev/null
+++ b/support/testing/tests/core/rootfs-overlay1/test-file1
@@ -0,0 +1 @@
+foobar
diff --git a/support/testing/tests/core/rootfs-overlay2/etc/test-file2 b/support/testing/tests/core/rootfs-overlay2/etc/test-file2
new file mode 100644
index 0000000..34a3ec2
--- /dev/null
+++ b/support/testing/tests/core/rootfs-overlay2/etc/test-file2
@@ -0,0 +1 @@
+barfoo
diff --git a/support/testing/tests/core/test_post_scripts.py b/support/testing/tests/core/test_post_scripts.py
new file mode 100644
index 0000000..d2a0be1
--- /dev/null
+++ b/support/testing/tests/core/test_post_scripts.py
@@ -0,0 +1,40 @@
+import os
+import csv
+
+import infra.basetest
+
+class TestPostScripts(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_INIT_NONE=y
+BR2_SYSTEM_BIN_SH_NONE=y
+# BR2_PACKAGE_BUSYBOX is not set
+BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+BR2_ROOTFS_POST_IMAGE_SCRIPT="{}"
+BR2_ROOTFS_POST_SCRIPT_ARGS="foobar baz"
+""".format(infra.filepath("tests/core/post-build.sh"),
+ infra.filepath("tests/core/post-image.sh"))
+
+ def check_post_log_file(self, path, what):
+ lines = {}
+ with open(path, 'rb') as csvfile:
+ r = csv.reader(csvfile, delimiter=',')
+ for row in r:
+ lines[row[0]] = row[1]
+
+ self.assertEqual(lines["arg1"], os.path.join(self.builddir, what))
+ self.assertEqual(lines["arg2"], "foobar")
+ self.assertEqual(lines["arg3"], "baz")
+ self.assertEqual(lines["TARGET_DIR"], os.path.join(self.builddir, "target"))
+ self.assertEqual(lines["BUILD_DIR"], os.path.join(self.builddir, "build"))
+ self.assertEqual(lines["HOST_DIR"], os.path.join(self.builddir, "host"))
+ staging = os.readlink(os.path.join(self.builddir, "staging"))
+ self.assertEqual(lines["STAGING_DIR"], staging)
+ self.assertEqual(lines["BINARIES_DIR"], os.path.join(self.builddir, "images"))
+ self.assertEqual(lines["BR2_CONFIG"], os.path.join(self.builddir, ".config"))
+
+ def test_run(self):
+ f = os.path.join(self.builddir, "build", "post-build.log")
+ self.check_post_log_file(f, "target")
+ f = os.path.join(self.builddir, "build", "post-image.log")
+ self.check_post_log_file(f, "images")
diff --git a/support/testing/tests/core/test_rootfs_overlay.py b/support/testing/tests/core/test_rootfs_overlay.py
new file mode 100644
index 0000000..42d890b
--- /dev/null
+++ b/support/testing/tests/core/test_rootfs_overlay.py
@@ -0,0 +1,27 @@
+import os
+import subprocess
+
+import infra.basetest
+
+def compare_file(file1, file2):
+ return subprocess.call(["cmp", file1, file2])
+
+class TestRootfsOverlay(infra.basetest.BRTest):
+
+ rootfs_overlay_path = infra.filepath("tests/core/rootfs-overlay")
+ rootfs_overlay = "BR2_ROOTFS_OVERLAY=\"{0}1 {0}2\"".format(rootfs_overlay_path)
+
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ infra.basetest.MINIMAL_CONFIG + \
+ rootfs_overlay
+
+ def test_run(self):
+ target_file = os.path.join(self.builddir, "target", "test-file1")
+ overlay_file = "{}1/test-file1".format(self.rootfs_overlay_path)
+ ret = compare_file(overlay_file, target_file)
+ self.assertEqual(ret, 0)
+
+ target_file = os.path.join(self.builddir, "target", "etc", "test-file2")
+ overlay_file = "{}2/etc/test-file2".format(self.rootfs_overlay_path)
+ ret = compare_file(overlay_file, target_file)
+ self.assertEqual(ret, 0)
diff --git a/support/testing/tests/core/test_timezone.py b/support/testing/tests/core/test_timezone.py
new file mode 100644
index 0000000..9776b4b
--- /dev/null
+++ b/support/testing/tests/core/test_timezone.py
@@ -0,0 +1,66 @@
+import os
+
+import infra.basetest
+
+def boot_armv5_cpio(emulator, builddir):
+ img = os.path.join(builddir, "images", "rootfs.cpio")
+ emulator.boot(arch="armv5", kernel="builtin",
+ options=["-initrd", img])
+ emulator.login()
+
+class TestNoTimezone(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ """
+# BR2_TARGET_TZ_INFO is not set
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+
+class TestGlibcAllTimezone(infra.basetest.BRTest):
+ config = """
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TARGET_TZ_INFO=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "PST")
+ tz, _ = self.emulator.run("TZ=Europe/Paris date +%Z")
+ self.assertEqual(tz[0].strip(), "CET")
+
+class TestGlibcNonDefaultLimitedTimezone(infra.basetest.BRTest):
+ config = """
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TARGET_TZ_INFO=y
+BR2_TARGET_TZ_ZONELIST="northamerica"
+BR2_TARGET_LOCALTIME="America/New_York"
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("date +%Z")
+ self.assertEqual(tz[0].strip(), "EST")
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "PST")
+ tz, _ = self.emulator.run("TZ=Europe/Paris date +%Z")
+ self.assertEqual(tz[0].strip(), "Europe")
--
2.7.4
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 2/5] support/testing: add core tests
2017-03-20 20:36 ` [Buildroot] [PATCH v3 2/5] support/testing: add core tests Thomas Petazzoni
@ 2017-03-22 8:17 ` Thomas De Schampheleire
2017-03-22 8:25 ` Thomas Petazzoni
0 siblings, 1 reply; 24+ messages in thread
From: Thomas De Schampheleire @ 2017-03-22 8:17 UTC (permalink / raw)
To: buildroot
On Mon, Mar 20, 2017 at 9:36 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> This commit adds a few Buildroot "core" tests, testing functionalities
> such as:
>
> - post-build and post-image scripts
> - root filesystem overlays
> - timezone support
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/tests/core/__init__.py | 0
> support/testing/tests/core/post-build.sh | 12 ++++
> support/testing/tests/core/post-image.sh | 12 ++++
> .../testing/tests/core/rootfs-overlay1/test-file1 | 1 +
> .../tests/core/rootfs-overlay2/etc/test-file2 | 1 +
> support/testing/tests/core/test_post_scripts.py | 40 +++++++++++++
> support/testing/tests/core/test_rootfs_overlay.py | 27 +++++++++
> support/testing/tests/core/test_timezone.py | 66 ++++++++++++++++++++++
> 8 files changed, 159 insertions(+)
> create mode 100644 support/testing/tests/core/__init__.py
> create mode 100755 support/testing/tests/core/post-build.sh
> create mode 100755 support/testing/tests/core/post-image.sh
> create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
> create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
> create mode 100644 support/testing/tests/core/test_post_scripts.py
> create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
> create mode 100644 support/testing/tests/core/test_timezone.py
Two minor comments:
I think it would be cleaner to move support files for tests, like the
post-build.sh and post-image.sh and rootfs-overlays, to a subdirectory
instead of in the same location as the tests.
Also, I would argue that the rootfs-overlays do not need to be real
files in git but can be generated on the fly during the test.
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 2/5] support/testing: add core tests
2017-03-22 8:17 ` Thomas De Schampheleire
@ 2017-03-22 8:25 ` Thomas Petazzoni
2017-03-22 8:35 ` Thomas De Schampheleire
0 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-22 8:25 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 22 Mar 2017 09:17:59 +0100, Thomas De Schampheleire wrote:
> Two minor comments:
>
> I think it would be cleaner to move support files for tests, like the
> post-build.sh and post-image.sh and rootfs-overlays, to a subdirectory
> instead of in the same location as the tests.
Can you suggest a name for this subdirectory ? :-)
> Also, I would argue that the rootfs-overlays do not need to be real
> files in git but can be generated on the fly during the test.
I tried to do that, but:
1. It complicates quite a bit the test scenario.
2. There are plenty of other test cases where we will really need some
data that cannot be generated: a set of patches to test if patches
on packages are properly applied, a set of dummy BR2_EXTERNAL
trees, etc. So we will anyway have some "supporting" artifacts in
the tree, and therefore I thought it wasn't worth the effort making
this test more complicated.
Yann did the exact same comment while reviewing the previous version,
so I tried generating the files automatically, and went through this
reasoning before giving up and keeping things as they are.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 2/5] support/testing: add core tests
2017-03-22 8:25 ` Thomas Petazzoni
@ 2017-03-22 8:35 ` Thomas De Schampheleire
0 siblings, 0 replies; 24+ messages in thread
From: Thomas De Schampheleire @ 2017-03-22 8:35 UTC (permalink / raw)
To: buildroot
On Wed, Mar 22, 2017 at 9:25 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Wed, 22 Mar 2017 09:17:59 +0100, Thomas De Schampheleire wrote:
>
>> Two minor comments:
>>
>> I think it would be cleaner to move support files for tests, like the
>> post-build.sh and post-image.sh and rootfs-overlays, to a subdirectory
>> instead of in the same location as the tests.
>
> Can you suggest a name for this subdirectory ? :-)
support_files
test_data
?
>
>> Also, I would argue that the rootfs-overlays do not need to be real
>> files in git but can be generated on the fly during the test.
>
> I tried to do that, but:
>
> 1. It complicates quite a bit the test scenario.
>
> 2. There are plenty of other test cases where we will really need some
> data that cannot be generated: a set of patches to test if patches
> on packages are properly applied, a set of dummy BR2_EXTERNAL
> trees, etc. So we will anyway have some "supporting" artifacts in
> the tree, and therefore I thought it wasn't worth the effort making
> this test more complicated.
>
> Yann did the exact same comment while reviewing the previous version,
> so I tried generating the files automatically, and went through this
> reasoning before giving up and keeping things as they are.
Ok, I see. No issue...
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 3/5] support/testing: add fs tests
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
2017-03-20 20:36 ` [Buildroot] [PATCH v3 1/5] support/testing: core " Thomas Petazzoni
2017-03-20 20:36 ` [Buildroot] [PATCH v3 2/5] support/testing: add core tests Thomas Petazzoni
@ 2017-03-20 20:36 ` Thomas Petazzoni
2017-05-07 20:55 ` Luca Ceresoli
2017-03-20 20:36 ` [Buildroot] [PATCH v3 4/5] support/testing: add package tests Thomas Petazzoni
` (3 subsequent siblings)
6 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-20 20:36 UTC (permalink / raw)
To: buildroot
This commit adds a number of test cases for various filesystem formats:
ext2/3/4, iso9660, jffs2, squashfs, ubi/ubifs and yaffs2. All of them
except yaffs2 are runtime tested. The iso9660 set of test cases is
particularly rich, testing the proper operation of the iso9660 support
with all of grub, grub2 and isolinux.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
| 20 +++
support/testing/conf/grub2.cfg | 7 +
support/testing/conf/isolinux.cfg | 5 +
.../testing/conf/minimal-x86-qemu-kernel.config | 23 +++
support/testing/tests/fs/__init__.py | 0
support/testing/tests/fs/test_ext.py | 119 +++++++++++++++
support/testing/tests/fs/test_iso9660.py | 162 +++++++++++++++++++++
support/testing/tests/fs/test_jffs2.py | 45 ++++++
support/testing/tests/fs/test_squashfs.py | 37 +++++
support/testing/tests/fs/test_ubi.py | 39 +++++
support/testing/tests/fs/test_yaffs2.py | 12 ++
11 files changed, 469 insertions(+)
create mode 100644 support/testing/conf/grub-menu.lst
create mode 100644 support/testing/conf/grub2.cfg
create mode 100644 support/testing/conf/isolinux.cfg
create mode 100644 support/testing/conf/minimal-x86-qemu-kernel.config
create mode 100644 support/testing/tests/fs/__init__.py
create mode 100644 support/testing/tests/fs/test_ext.py
create mode 100644 support/testing/tests/fs/test_iso9660.py
create mode 100644 support/testing/tests/fs/test_jffs2.py
create mode 100644 support/testing/tests/fs/test_squashfs.py
create mode 100644 support/testing/tests/fs/test_ubi.py
create mode 100644 support/testing/tests/fs/test_yaffs2.py
--git a/support/testing/conf/grub-menu.lst b/support/testing/conf/grub-menu.lst
new file mode 100644
index 0000000..6143d80
--- /dev/null
+++ b/support/testing/conf/grub-menu.lst
@@ -0,0 +1,20 @@
+default 0
+timeout 1
+
+# Used when no splashimage is used
+color cyan/blue white/blue
+
+# Gets enabled/disabled depending on Grub support for splashimage
+splashimage /boot/grub/splash.xpm.gz
+
+# Used when a splashimage is enabled
+foreground 000000
+background cccccc
+
+title Buildroot ISO9660 image
+kernel __KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+initrd __INITRD_PATH__
+
+title Hard Drive (first partition)
+rootnoverify (hd0)
+chainloader +1
diff --git a/support/testing/conf/grub2.cfg b/support/testing/conf/grub2.cfg
new file mode 100644
index 0000000..a982d0b
--- /dev/null
+++ b/support/testing/conf/grub2.cfg
@@ -0,0 +1,7 @@
+set default="0"
+set timeout="1"
+
+menuentry "Buildroot" {
+ linux __KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+ initrd __INITRD_PATH__
+}
diff --git a/support/testing/conf/isolinux.cfg b/support/testing/conf/isolinux.cfg
new file mode 100644
index 0000000..ba031a6
--- /dev/null
+++ b/support/testing/conf/isolinux.cfg
@@ -0,0 +1,5 @@
+default 1
+label 1
+ kernel __KERNEL_PATH__
+ initrd __INITRD_PATH__
+ append root=/dev/sr0 console=ttyS0,115200
diff --git a/support/testing/conf/minimal-x86-qemu-kernel.config b/support/testing/conf/minimal-x86-qemu-kernel.config
new file mode 100644
index 0000000..8f6ceef
--- /dev/null
+++ b/support/testing/conf/minimal-x86-qemu-kernel.config
@@ -0,0 +1,23 @@
+# CONFIG_64BIT is not set
+CONFIG_SYSVIPC=y
+CONFIG_BLK_DEV_INITRD=y
+# CONFIG_SUSPEND is not set
+# CONFIG_ACPI is not set
+CONFIG_CPU_IDLE=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_BLK_DEV_SR=y
+CONFIG_ATA=y
+CONFIG_ATA_PIIX=y
+CONFIG_INPUT_EVDEV=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_POWER_SUPPLY=y
+CONFIG_THERMAL=y
+CONFIG_EXT4_FS=y
+CONFIG_ISO9660_FS=y
+CONFIG_JOLIET=y
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+# CONFIG_VIRTUALIZATION is not set
diff --git a/support/testing/tests/fs/__init__.py b/support/testing/tests/fs/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
new file mode 100644
index 0000000..f7e2e85
--- /dev/null
+++ b/support/testing/tests/fs/test_ext.py
@@ -0,0 +1,119 @@
+import os
+import subprocess
+
+import infra.basetest
+
+VOLNAME_PROP = "Filesystem volume name"
+REVISION_PROP = "Filesystem revision #"
+FEATURES_PROP = "Filesystem features"
+BLOCKCNT_PROP = "Block count"
+INODECNT_PROP = "Inode count"
+RESBLKCNT_PROP = "Reserved block count"
+
+CHECK_FS_TYPE_CMD = "mount | grep '/dev/root on / type {}'"
+
+def dumpe2fs_run(builddir, image):
+ cmd = ["host/usr/sbin/dumpe2fs", os.path.join("images", image)]
+ ret = subprocess.check_output(cmd,
+ stderr=open(os.devnull, "w"),
+ cwd=builddir,
+ env={"LANG": "C"})
+ return ret.strip().splitlines()
+
+def dumpe2fs_getprop(out, prop):
+ for lines in out:
+ lines = lines.split(": ")
+ if lines[0] == prop:
+ return lines[1].strip()
+
+def boot_img_and_check_fs_type(emulator, builddir, fs_type):
+ img = os.path.join(builddir, "images", "rootfs.{}".format(fs_type))
+ emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mmcblk0",
+ "rootfstype={}".format(fs_type)],
+ options=["-drive", "file={},if=sd".format(img)])
+ emulator.login()
+ _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
+ return exit_code
+
+class TestExt2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_2r0=y
+BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext2")
+ self.assertEqual(dumpe2fs_getprop(out, VOLNAME_PROP), "foobaz")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "0 (original)")
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext2")
+ self.assertEqual(exit_code, 0)
+
+class TestExt2r1(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_2r1=y
+BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext2")
+ self.assertEqual(dumpe2fs_getprop(out, VOLNAME_PROP), "foobar")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertNotIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext2")
+ self.assertEqual(exit_code, 0)
+
+class TestExt3(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_3=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext3")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext3")
+ self.assertEqual(exit_code, 0)
+
+class TestExt4(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_4=y
+BR2_TARGET_ROOTFS_EXT2_BLOCKS=16384
+BR2_TARGET_ROOTFS_EXT2_INODES=3000
+BR2_TARGET_ROOTFS_EXT2_RESBLKS=10
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext4")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertEqual(dumpe2fs_getprop(out, BLOCKCNT_PROP), "16384")
+ # Yes there are 8 more inodes than requested
+ self.assertEqual(dumpe2fs_getprop(out, INODECNT_PROP), "3008")
+ self.assertEqual(dumpe2fs_getprop(out, RESBLKCNT_PROP), "1638")
+ self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+ self.assertIn("extent", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext4")
+ self.assertEqual(exit_code, 0)
+
+
diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
new file mode 100644
index 0000000..eec6e89
--- /dev/null
+++ b/support/testing/tests/fs/test_iso9660.py
@@ -0,0 +1,162 @@
+import os
+
+import infra.basetest
+
+BASIC_CONFIG = \
+"""
+BR2_x86_pentium4=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-i386-pentium4-full-2015.05-496-g85945aa.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_2=y
+BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
+BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
+BR2_LINUX_KERNEL=y
+BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.0"
+BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="{}"
+# BR2_TARGET_ROOTFS_TAR is not set
+""".format(infra.filepath("conf/minimal-x86-qemu-kernel.config"))
+
+def test_mount_internal_external(emulator, builddir, internal=True):
+ img = os.path.join(builddir, "images", "rootfs.iso9660")
+ emulator.boot(arch="i386", options=["-cdrom", img])
+ emulator.login()
+
+ if internal:
+ cmd = "mount | grep 'rootfs on / type rootfs'"
+ else:
+ cmd = "mount | grep '/dev/root on / type iso9660'"
+
+ _, exit_code = emulator.run(cmd)
+ return exit_code
+
+def test_touch_file(emulator):
+ _, exit_code = emulator.run("touch test")
+ return exit_code
+
+#
+# Grub 2
+#
+
+class TestIso9660Grub2External(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_GRUB2=y
+BR2_TARGET_GRUB2_BOOT_PARTITION="cd"
+BR2_TARGET_GRUB2_BUILTIN_MODULES="boot linux ext2 fat part_msdos part_gpt normal biosdisk iso9660"
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub2.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660Grub2Internal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_ROOTFS_ISO9660_INITRD=y
+BR2_TARGET_GRUB2=y
+BR2_TARGET_GRUB2_BOOT_PARTITION="cd"
+BR2_TARGET_GRUB2_BUILTIN_MODULES="boot linux ext2 fat part_msdos part_gpt normal biosdisk iso9660"
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub2.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 0)
+
+#
+# Grub
+#
+
+class TestIso9660GrubExternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_GRUB=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub-menu.lst"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660GrubInternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_GRUB=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub-menu.lst"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 0)
+
+#
+# Syslinux
+#
+
+class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_ROOTFS_ISO9660_HYBRID=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+BR2_TARGET_SYSLINUX=y
+""".format(infra.filepath("conf/isolinux.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_ROOTFS_ISO9660_INITRD=y
+BR2_TARGET_ROOTFS_ISO9660_HYBRID=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+BR2_TARGET_SYSLINUX=y
+""".format(infra.filepath("conf/isolinux.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_jffs2.py b/support/testing/tests/fs/test_jffs2.py
new file mode 100644
index 0000000..0d45af2
--- /dev/null
+++ b/support/testing/tests/fs/test_jffs2.py
@@ -0,0 +1,45 @@
+import os
+import subprocess
+
+import infra.basetest
+
+def jffs2dump_find_file(files_list, fname):
+ for file_name in files_list:
+ file_name = file_name.strip()
+ if file_name.startswith("Dirent") and file_name.endswith(fname):
+ return True
+ return False
+
+class TestJffs2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_JFFS2=y
+BR2_TARGET_ROOTFS_JFFS2_CUSTOM=y
+BR2_TARGET_ROOTFS_JFFS2_CUSTOM_EBSIZE=0x80000
+BR2_TARGET_ROOTFS_JFFS2_NOCLEANMARKER=y
+BR2_TARGET_ROOTFS_JFFS2_PAD=y
+BR2_TARGET_ROOTFS_JFFS2_PADSIZE=0x4000000
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ # TODO: there are some scary JFFS2 messages when one starts to
+ # write files in the rootfs: "jffs2: Newly-erased block contained
+ # word 0x0 at offset 0x046c0000". To be investigated.
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.jffs2")
+ out = subprocess.check_output(["host/usr/sbin/jffs2dump", "-c", img],
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+ self.assertTrue(jffs2dump_find_file(out, "busybox"))
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mtdblock0",
+ "rootfstype=jffs2"],
+ options=["-drive", "file={},if=pflash".format(img)])
+ self.emulator.login()
+ cmd = "mount | grep '/dev/root on / type jffs2'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
new file mode 100644
index 0000000..edaa087
--- /dev/null
+++ b/support/testing/tests/fs/test_squashfs.py
@@ -0,0 +1,37 @@
+import os
+import subprocess
+
+import infra.basetest
+
+class TestSquashfs(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_SQUASHFS=y
+# BR2_TARGET_ROOTFS_SQUASHFS4_GZIP is not set
+BR2_TARGET_ROOTFS_SQUASHFS4_LZ4=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ unsquashfs_cmd = ["host/usr/bin/unsquashfs", "-s", "images/rootfs.squashfs"]
+ out = subprocess.check_output(unsquashfs_cmd,
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+ self.assertEqual(out[0],
+ "Found a valid SQUASHFS 4:0 superblock on images/rootfs.squashfs.")
+ self.assertEqual(out[3], "Compression lz4")
+
+ img = os.path.join(self.builddir, "images", "rootfs.squashfs")
+ subprocess.call(["truncate", "-s", "%1M", img])
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mmcblk0",
+ "rootfstype=squashfs"],
+ options=["-drive", "file={},if=sd,format=raw".format(img)])
+ self.emulator.login()
+
+ cmd = "mount | grep '/dev/root on / type squashfs'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_ubi.py b/support/testing/tests/fs/test_ubi.py
new file mode 100644
index 0000000..ede4999
--- /dev/null
+++ b/support/testing/tests/fs/test_ubi.py
@@ -0,0 +1,39 @@
+import subprocess
+import os
+
+import infra.basetest
+
+class TestUbi(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_UBIFS=y
+BR2_TARGET_ROOTFS_UBIFS_LEBSIZE=0x7ff80
+BR2_TARGET_ROOTFS_UBIFS_MINIOSIZE=0x1
+BR2_TARGET_ROOTFS_UBI=y
+BR2_TARGET_ROOTFS_UBI_PEBSIZE=0x80000
+BR2_TARGET_ROOTFS_UBI_SUBSIZE=1
+"""
+
+ # TODO: if you boot Qemu twice on the same UBI image, it fails to
+ # attach the image the second time, with "ubi0 error:
+ # ubi_read_volume_table: the layout volume was not found".
+ # To be investigated.
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.ubi")
+ out = subprocess.check_output(["file", img],
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+
+ subprocess.call(["truncate", "-s 128M", img])
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=ubi0:rootfs",
+ "ubi.mtd=0",
+ "rootfstype=ubifs"],
+ options=["-drive", "file={},if=pflash".format(img)])
+ self.emulator.login()
+ cmd = "mount | grep 'ubi0:rootfs on / type ubifs'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_yaffs2.py b/support/testing/tests/fs/test_yaffs2.py
new file mode 100644
index 0000000..0ffb758
--- /dev/null
+++ b/support/testing/tests/fs/test_yaffs2.py
@@ -0,0 +1,12 @@
+import os
+
+import infra.basetest
+
+class TestYaffs2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ infra.basetest.MINIMAL_CONFIG + \
+ "BR2_TARGET_ROOTFS_YAFFS2=y"
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.yaffs2")
+ self.assertTrue(os.path.exists(img))
--
2.7.4
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 3/5] support/testing: add fs tests
2017-03-20 20:36 ` [Buildroot] [PATCH v3 3/5] support/testing: add fs tests Thomas Petazzoni
@ 2017-05-07 20:55 ` Luca Ceresoli
0 siblings, 0 replies; 24+ messages in thread
From: Luca Ceresoli @ 2017-05-07 20:55 UTC (permalink / raw)
To: buildroot
Hi,
this the review I had written a few days ago. Since the patch has just
been applied, I've reworded it as a list of proposed improvements and
request for clarifications.
On 20/03/2017 21:36, Thomas Petazzoni wrote:
> This commit adds a number of test cases for various filesystem formats:
> ext2/3/4, iso9660, jffs2, squashfs, ubi/ubifs and yaffs2. All of them
> except yaffs2 are runtime tested. The iso9660 set of test cases is
> particularly rich, testing the proper operation of the iso9660 support
> with all of grub, grub2 and isolinux.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/conf/grub-menu.lst | 20 +++
> support/testing/conf/grub2.cfg | 7 +
> support/testing/conf/isolinux.cfg | 5 +
> .../testing/conf/minimal-x86-qemu-kernel.config | 23 +++
> support/testing/tests/fs/__init__.py | 0
> support/testing/tests/fs/test_ext.py | 119 +++++++++++++++
> support/testing/tests/fs/test_iso9660.py | 162 +++++++++++++++++++++
> support/testing/tests/fs/test_jffs2.py | 45 ++++++
> support/testing/tests/fs/test_squashfs.py | 37 +++++
> support/testing/tests/fs/test_ubi.py | 39 +++++
> support/testing/tests/fs/test_yaffs2.py | 12 ++
> 11 files changed, 469 insertions(+)
> create mode 100644 support/testing/conf/grub-menu.lst
> create mode 100644 support/testing/conf/grub2.cfg
> create mode 100644 support/testing/conf/isolinux.cfg
I suspect it's already been suggested, but I would rather put these
files in the 'fs' subdirectory, unless we expect to use them for non-fs
tests.
Do other people agree? Should I send a patch?
> diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
> new file mode 100644
> index 0000000..f7e2e85
> --- /dev/null
> +++ b/support/testing/tests/fs/test_ext.py
[...]
> +def dumpe2fs_getprop(out, prop):
> + for lines in out:
Here 'lines' is set to a string (one line)...
> + lines = lines.split(": ")
...and here it is reassigned to a list (the line fields, as they are
called in awk. Or is the second assignment of 'lines' creating a new
variable with the same name? Whether the reply, I don't like this.
> + if lines[0] == prop:
> + return lines[1].strip()
My proposal (tested):
def dumpe2fs_getprop(out, prop):
for line in out:
fields = line.split(": ")
if fields[0] == prop:
return fields[1].strip()
Unless there's a good reason to keep the code as is, I'll send a patch
anyway, to improve readability at least.
> +def boot_img_and_check_fs_type(emulator, builddir, fs_type):
> + img = os.path.join(builddir, "images", "rootfs.{}".format(fs_type))
Here (and similarly in several other places) this looks a lot more
intuitive and concise to me:
"rootfs." + fs_type
Any good reason the initial form is preferred? Pythonists out there?
[...]
> +class TestExt3(infra.basetest.BRTest):
> + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +"""
> +BR2_TARGET_ROOTFS_EXT2=y
> +BR2_TARGET_ROOTFS_EXT2_3=y
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
> +
> + def test_run(self):
> + out = dumpe2fs_run(self.builddir, "rootfs.ext3")
> + self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
> + self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
> +
> + exit_code = boot_img_and_check_fs_type(self.emulator,
> + self.builddir, "ext3")
> + self.assertEqual(exit_code, 0)
> +
> +class TestExt4(infra.basetest.BRTest):
> + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +"""
> +BR2_TARGET_ROOTFS_EXT2=y
> +BR2_TARGET_ROOTFS_EXT2_4=y
> +BR2_TARGET_ROOTFS_EXT2_BLOCKS=16384
> +BR2_TARGET_ROOTFS_EXT2_INODES=3000
> +BR2_TARGET_ROOTFS_EXT2_RESBLKS=10
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
> +
> + def test_run(self):
> + out = dumpe2fs_run(self.builddir, "rootfs.ext4")
> + self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
> + self.assertEqual(dumpe2fs_getprop(out, BLOCKCNT_PROP), "16384")
> + # Yes there are 8 more inodes than requested
> + self.assertEqual(dumpe2fs_getprop(out, INODECNT_PROP), "3008")
> + self.assertEqual(dumpe2fs_getprop(out, RESBLKCNT_PROP), "1638")
> + self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
> + self.assertIn("extent", dumpe2fs_getprop(out, FEATURES_PROP))
It would be nice to have both positive and a negative test for each
tested item. Specifically, I'd add self.assertIn("extent", ...) for ext3.
I'll send a patch for this.
> diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
> new file mode 100644
> index 0000000..eec6e89
> --- /dev/null
> +++ b/support/testing/tests/fs/test_iso9660.py
[...]
> +#
> +# Syslinux
> +#
> +
> +class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
The two syslinux tests are failing on top of current master. This is
since commit 6e432d5ecb46 ("syslinux: build with the target toolchain").
The error message is:
.../i686-linux-ld: unrecognized option '--no-dynamic-linker'
> diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
> new file mode 100644
> index 0000000..edaa087
> --- /dev/null
> +++ b/support/testing/tests/fs/test_squashfs.py
[...]
> + def test_run(self):
> + unsquashfs_cmd = ["host/usr/bin/unsquashfs", "-s", "images/rootfs.squashfs"]
> + out = subprocess.check_output(unsquashfs_cmd,
> + cwd=self.builddir,
> + env={"LANG": "C"})
> + out = out.splitlines()
> + self.assertEqual(out[0],
> + "Found a valid SQUASHFS 4:0 superblock on images/rootfs.squashfs.")
> + self.assertEqual(out[3], "Compression lz4")
> +
> + img = os.path.join(self.builddir, "images", "rootfs.squashfs")
> + subprocess.call(["truncate", "-s", "%1M", img])
Why do we need this call to truncate?
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 4/5] support/testing: add package tests
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
` (2 preceding siblings ...)
2017-03-20 20:36 ` [Buildroot] [PATCH v3 3/5] support/testing: add fs tests Thomas Petazzoni
@ 2017-03-20 20:36 ` Thomas Petazzoni
2017-03-22 8:26 ` Thomas De Schampheleire
2017-03-20 20:36 ` [Buildroot] [PATCH v3 5/5] support/testing: add toolchain tests Thomas Petazzoni
` (2 subsequent siblings)
6 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-20 20:36 UTC (permalink / raw)
To: buildroot
This commit adds some basic tests for two Buildroot packages: python and
dropbear. These tests are by no mean meant to be exhaustive, but mainly
to serve as initial examples for other tests.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/package/__init__.py | 0
support/testing/tests/package/test_dropbear.py | 28 +++++++++++++++++++++
support/testing/tests/package/test_python.py | 35 ++++++++++++++++++++++++++
3 files changed, 63 insertions(+)
create mode 100644 support/testing/tests/package/__init__.py
create mode 100644 support/testing/tests/package/test_dropbear.py
create mode 100644 support/testing/tests/package/test_python.py
diff --git a/support/testing/tests/package/__init__.py b/support/testing/tests/package/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/package/test_dropbear.py b/support/testing/tests/package/test_dropbear.py
new file mode 100644
index 0000000..a64b782
--- /dev/null
+++ b/support/testing/tests/package/test_dropbear.py
@@ -0,0 +1,28 @@
+import os
+
+import infra.basetest
+
+class TestDropbear(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_GENERIC_ROOT_PASSWD="testpwd"
+BR2_SYSTEM_DHCP="eth0"
+BR2_PACKAGE_DROPBEAR=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img,
+ "-net", "nic",
+ "-net", "user,hostfwd=tcp::2222-:22"])
+ self.emulator.login("testpwd")
+ cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:22"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+ # Would be useful to try to login through SSH here, through
+ # localhost:2222, though it is not easy to pass the ssh
+ # password on the command line.
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
new file mode 100644
index 0000000..5532fb5
--- /dev/null
+++ b/support/testing/tests/package/test_python.py
@@ -0,0 +1,35 @@
+import os
+
+import infra.basetest
+
+class TestPythonBase(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_PACKAGE_PYTHON=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", cpio_file])
+ self.emulator.login()
+ cmd = "python --version 2>&1 | grep '^Python 2'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import math; math.floor(12.3)'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import ctypes;"
+ cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
+ cmd += "print libc.time(None)'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import zlib'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 1)
--
2.7.4
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 4/5] support/testing: add package tests
2017-03-20 20:36 ` [Buildroot] [PATCH v3 4/5] support/testing: add package tests Thomas Petazzoni
@ 2017-03-22 8:26 ` Thomas De Schampheleire
2017-03-22 13:08 ` Thomas Petazzoni
0 siblings, 1 reply; 24+ messages in thread
From: Thomas De Schampheleire @ 2017-03-22 8:26 UTC (permalink / raw)
To: buildroot
On Mon, Mar 20, 2017 at 9:36 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> This commit adds some basic tests for two Buildroot packages: python and
> dropbear. These tests are by no mean meant to be exhaustive, but mainly
> to serve as initial examples for other tests.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/tests/package/__init__.py | 0
> support/testing/tests/package/test_dropbear.py | 28 +++++++++++++++++++++
> support/testing/tests/package/test_python.py | 35 ++++++++++++++++++++++++++
> 3 files changed, 63 insertions(+)
> create mode 100644 support/testing/tests/package/__init__.py
> create mode 100644 support/testing/tests/package/test_dropbear.py
> create mode 100644 support/testing/tests/package/test_python.py
>
> diff --git a/support/testing/tests/package/__init__.py b/support/testing/tests/package/__init__.py
> new file mode 100644
> index 0000000..e69de29
> diff --git a/support/testing/tests/package/test_dropbear.py b/support/testing/tests/package/test_dropbear.py
> new file mode 100644
> index 0000000..a64b782
> --- /dev/null
> +++ b/support/testing/tests/package/test_dropbear.py
> @@ -0,0 +1,28 @@
> +import os
> +
> +import infra.basetest
> +
> +class TestDropbear(infra.basetest.BRTest):
> + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +"""
> +BR2_TARGET_GENERIC_ROOT_PASSWD="testpwd"
> +BR2_SYSTEM_DHCP="eth0"
> +BR2_PACKAGE_DROPBEAR=y
> +BR2_TARGET_ROOTFS_CPIO=y
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
> +
> + def test_run(self):
> + img = os.path.join(self.builddir, "images", "rootfs.cpio")
> + self.emulator.boot(arch="armv5",
> + kernel="builtin",
> + options=["-initrd", img,
> + "-net", "nic",
> + "-net", "user,hostfwd=tcp::2222-:22"])
> + self.emulator.login("testpwd")
> + cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:22"
> + _, exit_code = self.emulator.run(cmd)
> + self.assertEqual(exit_code, 0)
> + # Would be useful to try to login through SSH here, through
> + # localhost:2222, though it is not easy to pass the ssh
> + # password on the command line.
This can be done using the paramiko python module, basically:
import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(host, username=user, password=pass)
If you insist on shell commands, you can use the tool 'sshpass' for it.
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 4/5] support/testing: add package tests
2017-03-22 8:26 ` Thomas De Schampheleire
@ 2017-03-22 13:08 ` Thomas Petazzoni
0 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-22 13:08 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 22 Mar 2017 09:26:01 +0100, Thomas De Schampheleire wrote:
> This can be done using the paramiko python module, basically:
>
> import paramiko
> client = paramiko.SSHClient()
> client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
> client.connect(host, username=user, password=pass)
Good idea. Let's leave this as a future improvement :-)
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 5/5] support/testing: add toolchain tests
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
` (3 preceding siblings ...)
2017-03-20 20:36 ` [Buildroot] [PATCH v3 4/5] support/testing: add package tests Thomas Petazzoni
@ 2017-03-20 20:36 ` Thomas Petazzoni
2017-03-26 21:53 ` Luca Ceresoli
2017-03-22 8:30 ` [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas De Schampheleire
2017-05-07 20:09 ` Thomas Petazzoni
6 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-03-20 20:36 UTC (permalink / raw)
To: buildroot
This commit adds an initial toolchain test case, testing the ARM
CodeSourcery toolchain, just checking that the proper sysroot is used,
and that a minimal Linux system boots fine under Qemu.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/toolchain/__init__.py | 0
support/testing/tests/toolchain/test_external.py | 156 +++++++++++++++++++++++
2 files changed, 156 insertions(+)
create mode 100644 support/testing/tests/toolchain/__init__.py
create mode 100644 support/testing/tests/toolchain/test_external.py
diff --git a/support/testing/tests/toolchain/__init__.py b/support/testing/tests/toolchain/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/toolchain/test_external.py b/support/testing/tests/toolchain/test_external.py
new file mode 100644
index 0000000..1fbf81f
--- /dev/null
+++ b/support/testing/tests/toolchain/test_external.py
@@ -0,0 +1,156 @@
+import os
+import infra
+
+BASIC_CONFIG = \
+"""
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+def check_broken_links(path):
+ for root, dirs, files in os.walk(path):
+ for f in files:
+ fpath = os.path.join(root, f)
+ if not os.path.exists(fpath):
+ return True
+ return False
+
+class TestExternalToolchain(infra.basetest.BRTest):
+ def common_check(self):
+ # Check for broken symlinks
+ for d in ["lib", "usr/lib"]:
+ path = os.path.join(self.builddir, "staging", d)
+ self.assertFalse(check_broken_links(path))
+ path = os.path.join(self.builddir, "target", d)
+ self.assertFalse(check_broken_links(path))
+
+ interp = infra.get_elf_prog_interpreter(self.builddir,
+ self.toolchain_prefix,
+ "bin/busybox")
+ interp_path = os.path.join(self.builddir, "target", interp[1:])
+ self.assertTrue(os.path.exists(interp_path))
+
+class TestExternalToolchainSourceryArmv4(TestExternalToolchain):
+ config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_arm920t=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+"""
+ toolchain_prefix = "arm-none-linux-gnueabi"
+
+ def test_run(self):
+ TestExternalToolchain.common_check(self)
+
+ # Check the architecture variant
+ arch = infra.get_file_arch(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6")
+ self.assertEqual(arch, "v4T")
+
+ # Check the sysroot symlink
+ symlink = os.path.join(self.builddir, "staging", "armv4t")
+ self.assertTrue(os.path.exists(symlink))
+ self.assertEqual(os.readlink(symlink), "./")
+
+ # Boot the system
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login()
+
+class TestExternalToolchainSourceryArmv5(TestExternalToolchain):
+ config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+"""
+ toolchain_prefix = "arm-none-linux-gnueabi"
+
+ def test_run(self):
+ TestExternalToolchain.common_check(self)
+
+ # Check the architecture variant
+ arch = infra.get_file_arch(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6")
+ self.assertEqual(arch, "v5TE")
+
+ # Boot the system
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login()
+
+class TestExternalToolchainSourceryArmv7(TestExternalToolchain):
+ config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_cortex_a8=y
+BR2_ARM_EABI=y
+BR2_ARM_INSTRUCTIONS_THUMB2=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+"""
+ toolchain_prefix = "arm-none-linux-gnueabi"
+
+ def test_run(self):
+ TestExternalToolchain.common_check(self)
+
+ # Check the architecture variant
+ arch = infra.get_file_arch(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6")
+ self.assertEqual(arch, "v7")
+ isa = infra.get_elf_arch_tag(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6",
+ "Tag_THUMB_ISA_use")
+ self.assertEqual(isa, "Thumb-2")
+
+ # Check we have the sysroot symlink
+ symlink = os.path.join(self.builddir, "staging", "thumb2")
+ self.assertTrue(os.path.exists(symlink))
+ self.assertEqual(os.readlink(symlink), "./")
+
+ # Boot the system
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login()
+
+class TestExternalToolchainLinaroArm(TestExternalToolchain):
+ config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_cortex_a8=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
+"""
+ toolchain_prefix = "arm-linux-gnueabihf"
+
+ def test_run(self):
+ TestExternalToolchain.common_check(self)
+
+ # Check the architecture variant
+ arch = infra.get_file_arch(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6")
+ self.assertEqual(arch, "v7")
+ isa = infra.get_elf_arch_tag(self.builddir,
+ self.toolchain_prefix,
+ "lib/libc.so.6",
+ "Tag_THUMB_ISA_use")
+ self.assertEqual(isa, "Thumb-2")
+
+ # Boot the system
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login()
--
2.7.4
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 5/5] support/testing: add toolchain tests
2017-03-20 20:36 ` [Buildroot] [PATCH v3 5/5] support/testing: add toolchain tests Thomas Petazzoni
@ 2017-03-26 21:53 ` Luca Ceresoli
0 siblings, 0 replies; 24+ messages in thread
From: Luca Ceresoli @ 2017-03-26 21:53 UTC (permalink / raw)
To: buildroot
Hi,
On 20/03/2017 21:36, Thomas Petazzoni wrote:
> This commit adds an initial toolchain test case, testing the ARM
> CodeSourcery toolchain, just checking that the proper sysroot is used,
> and that a minimal Linux system boots fine under Qemu.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/tests/toolchain/__init__.py | 0
> support/testing/tests/toolchain/test_external.py | 156 +++++++++++++++++++++++
> 2 files changed, 156 insertions(+)
> create mode 100644 support/testing/tests/toolchain/__init__.py
> create mode 100644 support/testing/tests/toolchain/test_external.py
>
> diff --git a/support/testing/tests/toolchain/__init__.py b/support/testing/tests/toolchain/__init__.py
> new file mode 100644
> index 0000000..e69de29
> diff --git a/support/testing/tests/toolchain/test_external.py b/support/testing/tests/toolchain/test_external.py
> new file mode 100644
> index 0000000..1fbf81f
> --- /dev/null
> +++ b/support/testing/tests/toolchain/test_external.py
> @@ -0,0 +1,156 @@
> +import os
> +import infra
> +
> +BASIC_CONFIG = \
> +"""
> +BR2_TARGET_ROOTFS_CPIO=y
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
> +
> +def check_broken_links(path):
> + for root, dirs, files in os.walk(path):
> + for f in files:
> + fpath = os.path.join(root, f)
> + if not os.path.exists(fpath):
> + return True
> + return False
For predicate functions I find it a lot more understandable when the
name contains a predicate verb. IOW I suggest renaming to
has_broken_links(). This would make it immediately understandable to me,
while with the current name I had to go through the Python docs to
understand what it does.
The rest looks OK, but I haven't checked it in depth enough to give any
formal tag, sorry.
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
` (4 preceding siblings ...)
2017-03-20 20:36 ` [Buildroot] [PATCH v3 5/5] support/testing: add toolchain tests Thomas Petazzoni
@ 2017-03-22 8:30 ` Thomas De Schampheleire
2017-05-07 20:09 ` Thomas Petazzoni
6 siblings, 0 replies; 24+ messages in thread
From: Thomas De Schampheleire @ 2017-03-22 8:30 UTC (permalink / raw)
To: buildroot
On Mon, Mar 20, 2017 at 9:36 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> Here is a third version of a small runtime testing infrastructure for
> Buildroot.
>
> I have not tried to take into account all suggestions from the review,
> only the ones that really must be fixed before it gets applied. I'd
> like this stuff to be applied so that more people can start working on
> the testing infrastructure and submit patches.
>
> The first patch introduces the infrastructure itself, the other 4
> patches just add a small number of test cases.
>
> By far and large the most featureful set of tests are the filesystem
> tests, testing a lot of features of our filesystem image support.
>
> There are definitely lots of possible improvements to the
> infrastructure, and lots of possible tests to add, but we need to get
> started at some point, and that's an initial proposal.
>
I gave some minor comments/suggestions, but nothing blocking. I fully
agree to not hold on to this much longer...
Well done!
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure
2017-03-20 20:36 [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas Petazzoni
` (5 preceding siblings ...)
2017-03-22 8:30 ` [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure Thomas De Schampheleire
@ 2017-05-07 20:09 ` Thomas Petazzoni
2017-05-07 20:41 ` Luca Ceresoli
6 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2017-05-07 20:09 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 20 Mar 2017 21:36:49 +0100, Thomas Petazzoni wrote:
> Thomas Petazzoni (5):
> support/testing: core testing infrastructure
> support/testing: add core tests
> support/testing: add fs tests
> support/testing: add package tests
> support/testing: add toolchain tests
Since this has been around for a long time, I've finally applied this
series.
I know Luca, Ricardo and Thomas DS had some comments, but I believe it
would be easier to address them as follow-up patches. So Luca, Ricardo,
Thomas, your patches to improve this testing infrastructure are more
than welcome. I'm sure it needs to be extended and improved in many
different ways to test more Buildroot functionalities.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure
2017-05-07 20:09 ` Thomas Petazzoni
@ 2017-05-07 20:41 ` Luca Ceresoli
2017-05-07 21:10 ` Thomas Petazzoni
0 siblings, 1 reply; 24+ messages in thread
From: Luca Ceresoli @ 2017-05-07 20:41 UTC (permalink / raw)
To: buildroot
Hi,
On 07/05/2017 22:09, Thomas Petazzoni wrote:
> Hello,
>
> On Mon, 20 Mar 2017 21:36:49 +0100, Thomas Petazzoni wrote:
>
>> Thomas Petazzoni (5):
>> support/testing: core testing infrastructure
>> support/testing: add core tests
>> support/testing: add fs tests
>> support/testing: add package tests
>> support/testing: add toolchain tests
>
> Since this has been around for a long time, I've finally applied this
> series.
Bad timing on my side: I was reviewing all this work in detail a couple
of days ago, but my reviews to 2 patches are in my drafts folder waiting
for the remaining ones... Ok, I'll send them anyway right noe as a list
of proposed improvements.
> I know Luca, Ricardo and Thomas DS had some comments, but I believe it
> would be easier to address them as follow-up patches. So Luca, Ricardo,
> Thomas, your patches to improve this testing infrastructure are more
> than welcome. I'm sure it needs to be extended and improved in many
> different ways to test more Buildroot functionalities.
I totally agree this is the best way to proceed. Now everybody can
easily send small, incremental patches to improve the work. Thanks for
having applied these patches.
Any plan to have the tests run automatically (on the autobuilders maybe)?
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v3 0/5] Runtime testing infrastructure
2017-05-07 20:41 ` Luca Ceresoli
@ 2017-05-07 21:10 ` Thomas Petazzoni
0 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2017-05-07 21:10 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, 7 May 2017 22:41:09 +0200, Luca Ceresoli wrote:
> Bad timing on my side: I was reviewing all this work in detail a couple
> of days ago, but my reviews to 2 patches are in my drafts folder waiting
> for the remaining ones... Ok, I'll send them anyway right noe as a list
> of proposed improvements.
Thanks for your review!
> > I know Luca, Ricardo and Thomas DS had some comments, but I believe it
> > would be easier to address them as follow-up patches. So Luca, Ricardo,
> > Thomas, your patches to improve this testing infrastructure are more
> > than welcome. I'm sure it needs to be extended and improved in many
> > different ways to test more Buildroot functionalities.
>
> I totally agree this is the best way to proceed. Now everybody can
> easily send small, incremental patches to improve the work. Thanks for
> having applied these patches.
Yes, it will allow everyone to send small, incremental patches, me
included.
> Any plan to have the tests run automatically (on the autobuilders maybe)?
Yes, we will definitely want to do that. I don't think the autobuilders
will do it. Instead I believe running the tests on Gitlab CI would be
easier and more appropriate.
I'm sure Arnout can help :-)
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread