* [Buildroot] [PATCH 0/5] Runtime testing infrastructure
@ 2017-02-07 11:35 Thomas Petazzoni
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
` (5 more replies)
0 siblings, 6 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:35 UTC (permalink / raw)
To: buildroot
Hello,
At last, here is a first version of a small runtime testing
infrastructure for Buildroot. The first patch introduces the
infrastructure itself, the other 4 patches just add a few patches.
By far and large the most featureful set of tests are the filesystem
tests, testing a lot of features of our filesystem image support.
There are definitely lots of possible improvements to the
infrastructure, and lots of possible tests to add, but we need to get
started at some point, and that's an initial proposal.
Here is a quick start:
1. List the tests
./support/testing/run-tests -l
2. Run a test:
./support/testing/run-tests -d <download dir> -o <output dir> <name of test>
e.g:
./support/testing/run-tests -d ~/dl/ -o ../outputs/ tests.fs.test_ubi.TestUbi
This patch series is also available at:
http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=runtime-tests
Best regards,
Thomas Petazzoni
Thomas Petazzoni (5):
support/testing: core testing infrastructure
support/testing: add core tests
support/testing: add fs tests
support/testing: add package tests
support/testing: add toolchain tests
support/testing/conf/grub-menu.lst | 20 +++
support/testing/conf/grub2.cfg | 7 +
support/testing/conf/isolinux.cfg | 5 +
.../testing/conf/minimal-x86-qemu-kernel.config | 23 +++
support/testing/conf/unittest.cfg | 6 +
support/testing/infra/__init__.py | 61 ++++++++
support/testing/infra/basetest.py | 66 +++++++++
support/testing/infra/builder.py | 50 +++++++
support/testing/infra/emulator.py | 144 ++++++++++++++++++
support/testing/run-tests | 77 ++++++++++
support/testing/tests/__init__.py | 0
support/testing/tests/core/__init__.py | 0
support/testing/tests/core/post-build.sh | 10 ++
support/testing/tests/core/post-image.sh | 10 ++
.../testing/tests/core/rootfs-overlay1/test-file1 | Bin 0 -> 4096 bytes
.../tests/core/rootfs-overlay2/etc/test-file2 | Bin 0 -> 8192 bytes
support/testing/tests/core/test_post_scripts.py | 36 +++++
support/testing/tests/core/test_rootfs_overlay.py | 27 ++++
support/testing/tests/core/test_timezone.py | 66 +++++++++
support/testing/tests/fs/__init__.py | 0
support/testing/tests/fs/test_ext.py | 119 +++++++++++++++
support/testing/tests/fs/test_iso9660.py | 161 +++++++++++++++++++++
support/testing/tests/fs/test_jffs2.py | 45 ++++++
support/testing/tests/fs/test_squashfs.py | 37 +++++
support/testing/tests/fs/test_ubi.py | 39 +++++
support/testing/tests/fs/test_yaffs2.py | 12 ++
support/testing/tests/package/__init__.py | 0
support/testing/tests/package/test_dropbear.py | 28 ++++
support/testing/tests/package/test_python.py | 35 +++++
support/testing/tests/toolchain/__init__.py | 0
support/testing/tests/toolchain/test_external.py | 77 ++++++++++
31 files changed, 1161 insertions(+)
create mode 100644 support/testing/conf/grub-menu.lst
create mode 100644 support/testing/conf/grub2.cfg
create mode 100644 support/testing/conf/isolinux.cfg
create mode 100644 support/testing/conf/minimal-x86-qemu-kernel.config
create mode 100644 support/testing/conf/unittest.cfg
create mode 100644 support/testing/infra/__init__.py
create mode 100644 support/testing/infra/basetest.py
create mode 100644 support/testing/infra/builder.py
create mode 100644 support/testing/infra/emulator.py
create mode 100755 support/testing/run-tests
create mode 100644 support/testing/tests/__init__.py
create mode 100644 support/testing/tests/core/__init__.py
create mode 100755 support/testing/tests/core/post-build.sh
create mode 100755 support/testing/tests/core/post-image.sh
create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
create mode 100644 support/testing/tests/core/test_post_scripts.py
create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
create mode 100644 support/testing/tests/core/test_timezone.py
create mode 100644 support/testing/tests/fs/__init__.py
create mode 100644 support/testing/tests/fs/test_ext.py
create mode 100644 support/testing/tests/fs/test_iso9660.py
create mode 100644 support/testing/tests/fs/test_jffs2.py
create mode 100644 support/testing/tests/fs/test_squashfs.py
create mode 100644 support/testing/tests/fs/test_ubi.py
create mode 100644 support/testing/tests/fs/test_yaffs2.py
create mode 100644 support/testing/tests/package/__init__.py
create mode 100644 support/testing/tests/package/test_dropbear.py
create mode 100644 support/testing/tests/package/test_python.py
create mode 100644 support/testing/tests/toolchain/__init__.py
create mode 100644 support/testing/tests/toolchain/test_external.py
--
2.7.4
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
@ 2017-02-07 11:35 ` Thomas Petazzoni
2017-02-08 9:50 ` Thomas De Schampheleire
` (2 more replies)
2017-02-07 11:36 ` [Buildroot] [PATCH 2/5] support/testing: add core tests Thomas Petazzoni
` (4 subsequent siblings)
5 siblings, 3 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:35 UTC (permalink / raw)
To: buildroot
This commit adds the core of a new testing infrastructure that allows to
perform runtime testing of Buildroot generated systems. This
infrastructure uses the Python unittest logic as its foundation.
This core infrastructure commit includes the following aspects:
- A base test class, called BRTest, defined in
support/testing/infra/basetest.py. This base test class inherited
from the Python provided unittest.TestCase, and must be subclassed by
all Buildroot test cases.
Its main purpose is to provide the Python unittest setUp() and
tearDown() methods. In our case, setUp() takes care of building the
Buildroot system described in the test case, and instantiate the
Emulator object in case runtime testing is needed. The tearDown()
method simply cleans things up (stop the emulator, remove the output
directory).
- A Builder class, defined in support/testing/infra/builder.py, simply
responsible for building the Buildroot system in each test case.
- An Emulator class, defined in support/testing/infra/emulator.py,
responsible for running the generated system under Qemu, allowing
each test case to run arbitrary commands inside the emulated system.
- A run-tests script, which is the entry point to start the tests.
Even though I wrote the original version of this small infrastructure, a
huge amount of rework and improvement has been done by Maxime
Hadjinlian, and squashed into this patch. So many thanks to Maxime for
cleaning up and improving my Python code!
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/conf/unittest.cfg | 6 ++
support/testing/infra/__init__.py | 61 ++++++++++++++++
support/testing/infra/basetest.py | 66 +++++++++++++++++
support/testing/infra/builder.py | 50 +++++++++++++
support/testing/infra/emulator.py | 144 ++++++++++++++++++++++++++++++++++++++
support/testing/run-tests | 77 ++++++++++++++++++++
support/testing/tests/__init__.py | 0
7 files changed, 404 insertions(+)
create mode 100644 support/testing/conf/unittest.cfg
create mode 100644 support/testing/infra/__init__.py
create mode 100644 support/testing/infra/basetest.py
create mode 100644 support/testing/infra/builder.py
create mode 100644 support/testing/infra/emulator.py
create mode 100755 support/testing/run-tests
create mode 100644 support/testing/tests/__init__.py
diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg
new file mode 100644
index 0000000..6eaa234
--- /dev/null
+++ b/support/testing/conf/unittest.cfg
@@ -0,0 +1,6 @@
+[unittest]
+plugins = nose2.plugins.mp
+
+[multiprocess]
+processes = 1
+always-on = True
diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
new file mode 100644
index 0000000..880e642
--- /dev/null
+++ b/support/testing/infra/__init__.py
@@ -0,0 +1,61 @@
+import contextlib
+import os
+import sys
+import tempfile
+import subprocess
+from urllib2 import urlopen, HTTPError, URLError
+
+ARTEFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
+
+ at contextlib.contextmanager
+def smart_open(filename=None):
+ if filename and filename != '-':
+ fhandle = open(filename, 'a+')
+ else:
+ fhandle = sys.stdout
+
+ try:
+ yield fhandle
+ finally:
+ if fhandle is not sys.stdout:
+ fhandle.close()
+
+def filepath(relpath):
+ return os.path.join(os.getcwd(), "support/testing", relpath)
+
+def download(dldir, filename):
+ finalpath = os.path.join(dldir, filename)
+ if os.path.exists(finalpath):
+ return finalpath
+
+ if not os.path.exists(dldir):
+ os.makedirs(dldir)
+
+ tmpfile = tempfile.mktemp(dir=dldir)
+ print "Downloading to {}".format(tmpfile)
+
+ try:
+ url_fh = urlopen(os.path.join(ARTEFACTS_URL, filename))
+ with open(tmpfile, "w+") as tmpfile_fh:
+ tmpfile_fh.write(url_fh.read())
+ except (HTTPError, URLError), err:
+ os.unlink(tmpfile)
+ raise err
+
+ print "Renaming from %s to %s" % (tmpfile, finalpath)
+ os.rename(tmpfile, finalpath)
+ return finalpath
+
+def get_elf_arch_tag(builddir, prefix, fpath, tag):
+ cmd = ["host/usr/bin/{}-readelf".format(prefix),
+ "-A", os.path.join("target", fpath)]
+ out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
+ for line in out.splitlines():
+ line = line.strip()
+ if not line.startswith(tag):
+ continue
+ return line.split(":")[1].strip()
+ return None
+
+def get_file_arch(builddir, prefix, fpath):
+ return get_elf_arch_tag(builddir, prefix, fpath, "Tag_CPU_arch:")
diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py
new file mode 100644
index 0000000..eb9da90
--- /dev/null
+++ b/support/testing/infra/basetest.py
@@ -0,0 +1,66 @@
+import unittest
+import os
+import datetime
+
+from infra.builder import Builder
+from infra.emulator import Emulator
+
+BASIC_TOOLCHAIN_CONFIG = \
+"""
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2015.05-1190-g4a48479.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_4_7=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
+BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+"""
+
+MINIMAL_CONFIG = \
+"""
+BR2_INIT_NONE=y
+BR2_SYSTEM_BIN_SH_NONE=y
+# BR2_PACKAGE_BUSYBOX is not set
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+class BRTest(unittest.TestCase):
+ config = None
+ downloaddir = None
+ outputdir = None
+ logtofile = True
+ keepbuilds = False
+
+ def show_msg(self, msg):
+ print "[%s/%s/%s] %s" % (os.path.basename(self.__class__.outputdir),
+ self.testname,
+ datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+ msg)
+ def setUp(self):
+ self.testname = self.__class__.__name__
+ self.builddir = os.path.join(self.__class__.outputdir, self.testname)
+ self.runlog = self.builddir + "-run.log"
+ self.emulator = None
+ self.show_msg("Starting")
+ self.b = Builder(self.__class__.config, self.builddir, self.logtofile)
+
+ if not self.keepbuilds:
+ self.b.delete()
+
+ if not self.b.is_finished():
+ self.show_msg("Building")
+ self.b.build()
+ self.show_msg("Building done")
+
+ self.emulator = Emulator(self.builddir, self.downloaddir, self.logtofile)
+
+ def tearDown(self):
+ self.show_msg("Cleaning up")
+ if self.emulator:
+ self.emulator.stop()
+ if self.b and not self.keepbuilds:
+ self.b.delete()
diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
new file mode 100644
index 0000000..4f72e2f
--- /dev/null
+++ b/support/testing/infra/builder.py
@@ -0,0 +1,50 @@
+import os
+import shutil
+import subprocess
+
+import infra
+
+class Builder(object):
+ def __init__(self, config, builddir, logtofile):
+ self.config = config
+ self.builddir = builddir
+ self.logtofile = logtofile
+
+ def build(self):
+ if not os.path.isdir(self.builddir):
+ os.makedirs(self.builddir)
+
+ log = "{}-build.log".format(self.builddir)
+ if self.logtofile is None:
+ log = None
+
+ config_file = os.path.join(self.builddir, ".config")
+ with open(config_file, "w+") as cfd:
+ cfd.write(self.config)
+
+ cmd = ["make",
+ "-C", os.getcwd(),
+ "O={}".format(self.builddir),
+ "olddefconfig"]
+ with infra.smart_open(log) as log_fh:
+ ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh)
+ if ret != 0:
+ raise SystemError("Cannot olddefconfig")
+
+ cmd = ["make", "-C", self.builddir]
+ with infra.smart_open(log) as log_fh:
+ ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh)
+ if ret != 0:
+ raise SystemError("Build failed")
+
+ open(self.stamp_path(), 'a').close()
+
+ def stamp_path(self):
+ return os.path.join(self.builddir, "build-done")
+
+ def is_finished(self):
+ return os.path.exists(self.stamp_path())
+
+ def delete(self):
+ if os.path.exists(self.builddir):
+ shutil.rmtree(self.builddir)
diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
new file mode 100644
index 0000000..5340f50
--- /dev/null
+++ b/support/testing/infra/emulator.py
@@ -0,0 +1,144 @@
+import socket
+import subprocess
+import telnetlib
+
+import infra
+import infra.basetest
+
+# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
+# with the qemu machine.
+class Emulator(object):
+
+ def __init__(self, builddir, downloaddir, logtofile):
+ self.qemu = None
+ self.__tn = None
+ self.downloaddir = downloaddir
+ self.log = ""
+ self.log_file = "{}-run.log".format(builddir)
+ if logtofile is None:
+ self.log_file = None
+
+ # Start Qemu to boot the system
+ #
+ # arch: Qemu architecture to use
+ #
+ # kernel: path to the kernel image, or the special string
+ # 'builtin' in which case a pre-built kernel image will be used
+ # (so far only armv5 and armv7 kernels are available). If None,
+ # then no kernel is used, and we assume a bootable device will be
+ # specified.
+ #
+ # kernel_cmdline: array of kernel arguments to pass to Qemu -append option
+ #
+ # options: array of command line options to pass to Qemu
+ #
+ def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
+ if arch in ["armv7", "armv5"]:
+ qemu_arch = "arm"
+
+ qemu_cmd = ["qemu-system-{}".format(qemu_arch),
+ "-serial", "telnet::1234,server",
+ "-display", "none"]
+
+ if options:
+ qemu_cmd += options
+
+ if kernel_cmdline is None:
+ kernel_cmdline = []
+
+ if kernel:
+ if kernel == "builtin":
+ if arch in ["armv7", "armv5"]:
+ kernel_cmdline.append("console=ttyAMA0")
+
+ if arch == "armv7":
+ kernel = infra.download(self.downloaddir,
+ "kernel-vexpress")
+ dtb = infra.download(self.downloaddir,
+ "vexpress-v2p-ca9.dtb")
+ qemu_cmd += ["-dtb", dtb]
+ qemu_cmd += ["-M", "vexpress-a9"]
+ elif arch == "armv5":
+ kernel = infra.download(self.downloaddir,
+ "kernel-versatile")
+ qemu_cmd += ["-M", "versatilepb"]
+
+ qemu_cmd += ["-kernel", kernel]
+
+ if kernel_cmdline:
+ qemu_cmd += ["-append", " ".join(kernel_cmdline)]
+
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
+ self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh)
+
+ # Wait for the telnet port to appear and connect to it.
+ while True:
+ try:
+ self.__tn = telnetlib.Telnet("localhost", 1234)
+ if self.__tn:
+ break
+ except socket.error:
+ continue
+
+ def __read_until(self, waitstr, timeout=5):
+ data = self.__tn.read_until(waitstr, timeout)
+ self.log += data
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write(data)
+ return data
+
+ def __write(self, wstr):
+ self.__tn.write(wstr)
+
+ # Wait for the login prompt to appear, and then login as root with
+ # the provided password, or no password if not specified.
+ def login(self, password=None):
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> waiting for login\n")
+
+ self.__read_until("buildroot login:", 10)
+ if "buildroot login:" not in self.log:
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("==> System does not boot")
+ raise SystemError("System does not boot")
+
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> log in\n")
+
+ self.__write("root\n")
+ if password:
+ self.__read_until("Password:")
+ self.__write(password + "\n")
+ self.__read_until("# ")
+
+ # Run the given 'cmd' on the target
+ # return a tuple (output, exit_code)
+ def run(self, cmd):
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> running '{}'\n".format(cmd))
+
+ self.__write("{}\n".format(cmd))
+ output = self.__read_until("# ")
+ output = output.strip().splitlines()
+ output = output[1:len(output)-1]
+
+ self.__write("echo $?\n")
+ exit_code = self.__read_until("# ")
+ exit_code = exit_code.strip().splitlines()[1]
+ exit_code = int(exit_code)
+ with infra.smart_open(self.log_file) as lfh:
+ lfh.write("> command terminated, status {}\n".format(exit_code))
+
+ return output, exit_code
+
+ def stop(self):
+ if self.qemu is None:
+ return
+ self.qemu.terminate()
+ self.qemu.kill()
+
+ def showlog(self):
+ print "=== Full log ==="
+ print self.log
+ print "================"
diff --git a/support/testing/run-tests b/support/testing/run-tests
new file mode 100755
index 0000000..0155b71
--- /dev/null
+++ b/support/testing/run-tests
@@ -0,0 +1,77 @@
+#!/usr/bin/env python
+import argparse
+import sys
+import os
+import string
+import nose2
+
+from infra.basetest import BRTest
+
+def main():
+ parser = argparse.ArgumentParser(description='Run Buildroot tests')
+ parser.add_argument('testname', nargs='*',
+ help='list of test cases to execute')
+ parser.add_argument('--list', '-l', action='store_true',
+ help='list of available test cases')
+ parser.add_argument('--all', '-a', action='store_true',
+ help='execute all test cases')
+ parser.add_argument('--stdout', '-s', action='store_true',
+ help='log everything to stdout')
+ parser.add_argument('--output', '-o',
+ help='output directory')
+ parser.add_argument('--download', '-d',
+ help='download directory')
+ parser.add_argument('--keep', '-k',
+ help='keep build directories',
+ action='store_true')
+
+ args = parser.parse_args()
+
+ script_path = os.path.realpath(__file__)
+ test_dir = os.path.dirname(script_path)
+
+ if args.stdout:
+ BRTest.logtofile = False
+
+ if args.list:
+ print "List of tests"
+ nose2.discover(argv=[script_path,
+ "-s", test_dir,
+ "-v",
+ "--collect-only"],
+ plugins=["nose2.plugins.collect"])
+ return 0
+
+ if args.download is None:
+ print "Missing download directory, please use -d/--download"
+ return 1
+
+ BRTest.downloaddir = args.download
+
+ if args.output is None:
+ print "Missing output directory, please use -o/--output"
+ return 1
+
+ if not os.path.exists(args.output):
+ print "The selected output directory does not exist"
+ return 1
+
+ BRTest.outputdir = args.output
+
+ if args.all is False and len(args.testname) == 0:
+ print "No test selected"
+ return 1
+
+ BRTest.keepbuilds = args.keep
+
+ nose2_args = ["-v",
+ "-s", "support/testing",
+ "-c", "support/testing/conf/unittest.cfg"]
+
+ if len(args.testname) != 0:
+ nose2_args += args.testname
+
+ nose2.discover(argv=nose2_args)
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/support/testing/tests/__init__.py b/support/testing/tests/__init__.py
new file mode 100644
index 0000000..e69de29
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 2/5] support/testing: add core tests
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
@ 2017-02-07 11:36 ` Thomas Petazzoni
2017-03-02 1:47 ` Ricardo Martincoski
2017-02-07 11:36 ` [Buildroot] [PATCH 3/5] support/testing: add fs tests Thomas Petazzoni
` (3 subsequent siblings)
5 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:36 UTC (permalink / raw)
To: buildroot
This commit adds a few Buildroot "core" tests, testing functionalities
such as:
- post-build and post-image scripts
- root filesystem overlays
- timezone support
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/core/__init__.py | 0
support/testing/tests/core/post-build.sh | 10 ++++
support/testing/tests/core/post-image.sh | 10 ++++
.../testing/tests/core/rootfs-overlay1/test-file1 | Bin 0 -> 4096 bytes
.../tests/core/rootfs-overlay2/etc/test-file2 | Bin 0 -> 8192 bytes
support/testing/tests/core/test_post_scripts.py | 36 +++++++++++
support/testing/tests/core/test_rootfs_overlay.py | 27 +++++++++
support/testing/tests/core/test_timezone.py | 66 +++++++++++++++++++++
8 files changed, 149 insertions(+)
create mode 100644 support/testing/tests/core/__init__.py
create mode 100755 support/testing/tests/core/post-build.sh
create mode 100755 support/testing/tests/core/post-image.sh
create mode 100644 support/testing/tests/core/rootfs-overlay1/test-file1
create mode 100644 support/testing/tests/core/rootfs-overlay2/etc/test-file2
create mode 100644 support/testing/tests/core/test_post_scripts.py
create mode 100644 support/testing/tests/core/test_rootfs_overlay.py
create mode 100644 support/testing/tests/core/test_timezone.py
diff --git a/support/testing/tests/core/__init__.py b/support/testing/tests/core/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/core/post-build.sh b/support/testing/tests/core/post-build.sh
new file mode 100755
index 0000000..fbea726
--- /dev/null
+++ b/support/testing/tests/core/post-build.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+echo $1 > ${BUILD_DIR}/post-build.log
+echo $2 >> ${BUILD_DIR}/post-build.log
+echo $3 >> ${BUILD_DIR}/post-build.log
+echo ${TARGET_DIR} >> ${BUILD_DIR}/post-build.log
+echo ${BUILD_DIR} >> ${BUILD_DIR}/post-build.log
+echo ${HOST_DIR} >> ${BUILD_DIR}/post-build.log
+echo ${STAGING_DIR} >> ${BUILD_DIR}/post-build.log
+echo ${BINARIES_DIR} >> ${BUILD_DIR}/post-build.log
+echo ${BR2_CONFIG} >> ${BUILD_DIR}/post-build.log
diff --git a/support/testing/tests/core/post-image.sh b/support/testing/tests/core/post-image.sh
new file mode 100755
index 0000000..5856c0f
--- /dev/null
+++ b/support/testing/tests/core/post-image.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+echo $1 > ${BUILD_DIR}/post-image.log
+echo $2 >> ${BUILD_DIR}/post-image.log
+echo $3 >> ${BUILD_DIR}/post-image.log
+echo ${TARGET_DIR} >> ${BUILD_DIR}/post-image.log
+echo ${BUILD_DIR} >> ${BUILD_DIR}/post-image.log
+echo ${HOST_DIR} >> ${BUILD_DIR}/post-image.log
+echo ${STAGING_DIR} >> ${BUILD_DIR}/post-image.log
+echo ${BINARIES_DIR} >> ${BUILD_DIR}/post-image.log
+echo ${BR2_CONFIG} >> ${BUILD_DIR}/post-image.log
diff --git a/support/testing/tests/core/rootfs-overlay1/test-file1 b/support/testing/tests/core/rootfs-overlay1/test-file1
new file mode 100644
index 0000000000000000000000000000000000000000..19f4220f2c6a1bdbd927637d19867b7d04ff5752
GIT binary patch
literal 4096
zcmV+b5dZJLh5%0erUIG?ui=pzQ*dtx%8BVBkmb`@#KxT3$@S==e)~Kz at 8$8?C9JM<
zgZ}g(q`BIWGUM)Mw!eGZ|BkfMYV2E+f-x$|bq~~pEXwsYpqRPO%P@v;(j<=1ez;5#
zZ~WjJvTLT0wZO3%`?A#ZG(ddm$u!2O>52n83+sWg#{-UbkKNBEmF)k*txQiMz&WpU
z*tHG}E!J3qC3rOv$sdbB-vAIszFfqhx6^8XeGad0G}07gs~yQO{gn~5j*NRU6X7hv
zWEU)MCx^1FW6Q#CKPc$`uxWlKi?FYlqU_T*By54Qv5pEG+ib)$Xq^Ne1zDIPUhYFr
zjuQ7{>WZ{|Y#Z8|NuZvEkJz*$ox#7 at b%m^q at dQ1}$dt4L?&bReNv?I)BDPBCUS;XH
z_($(8&H>s~xkxk)&T{IpGCyIzZ}0Y};hL!bxz<Aifng8?&621|1(|`&)ZA1mr>eql
z%SpO{Su_7iB3CGp`ds&oAu8#e9zD4OBLn9*^p|6K3Pod6x^1-%;HR$PnLMzSk1kgV
z0fP5W#^0kHP%V_#CG at X61t=n)YHYhPHqC`C$lyC$gIpqAxZ)VOy+kf#+YcscV%Gog
z==vpqA)C7#+U1{^&F%y9VsY({(wUl+rYn=&_nAnC0YuSpWJeVup~^4mW&8XFiE@$7
zeUotq=@_z(mZ~GPdq`xB13s)L321UgM at mg>%>M*DpXW<H^OV^0tOHYSfn&EaKwj_8
zn#!puZKY^b2P>J)w`|JXIXYK!?o1ppD40v&YzeeMpwAkSnjUUj+Om-NvDw6B)86}b
zCzU0UBWd(!5{6<<LlQ*Jj at 4jY9m~U@dL)<f_2}vo%6^=#Uv@(NNvS!dvuNxpN9jDy
zs3FTTfm>Q^MQ3Pm<W~uP6mD7{QA=49tU#Z|)VR;N10|MstCZ~WU3p%?;@+exe9uWV
z*0Si%3e9wLVJ&NRORA{lX-C{ITALQRfDXkwrZ)MylD)p=iU&ix5EycTe~bSXxs=-r
z&h|_6UrKb|#(Hjv9 at Ivu`AC_W6&Jg*;$@#?TNmn8+xJrcmg4J}SLV1R3I=fXj~SIF
z2WiZIo-MNW92lXCZf{<S89PkD8n#U%Bp!`Y7qx4`2@j<ybDI)+v~;UP!+@!sqVuQ1
zir-nbKfy3d^q|$$iSal7R)$aon7+*~cakQ8Y;EKfnoe?(&J&c)m0+f)8AC{*>NNg?
zejL2R%F{YMaAWjdIDE#$<^O(Q45&mb;|TB;C2Pt5Hm%MaE1gK)Acf$H?{@a>8HDwj
zS&h~nC9QlwS}l8_0l^T`i0zaBT>M;prsY<xB^$Rb`txSCYw-L70IEUn9L$>rhX1id
ztuSX48b9@<o)_psforxjf!)7W!8gZ3>L69}DIN`6?;4li_?G>bmM at U`HO0WEDqg2%
zKB1pE^<8{h)5v9g_fKY&fA@7ehHKM}J2v+)l}rCZ{z7=JN;)R}_d#47Rd%mbJT`C@
zO>U^WJHbm|$@fY}jwS`HZvL~cjEBP(m4WbLpurwTKWu5g?Tq9FtL1eZ3TkJ}N=i05
z3_}Yf8s9b%eB-9Djw`Xq;~`$1pyWPl2eSl|vo7))Ddc^Cx)V|n at 2t=>Z#6A!z_Q!&
zX}R)0myqn#kv4QaA`thIf at 3t4PmpH@#|HO0St_Apxk`<-RdB9?D+qf7GFjH+XCV*q
z@{jt?KvB!x3@-~qj|u(Jf<B#w4c|FDk8a1qo7DS2$purn$y&Pdi`>D}@O`ezC;&_$
zfF}rS$g+o-ZFh%3=^fVXmp%6s+L4z<JWtclq{%mjSIcqDZ7X-R_r>$`v8YwORrw$a
z- at XzBlZqplC6+BF36Mo(j4k8-^+mJ9NJC5u<^O2axNSSGKC@rxbSvn2alR~+O=rL^
z&wSZZNnUXn%|M_A{gT%ds*0C2CjUw@>2*!lX5wvL<*OcLI#;#)f7}?uqP|a687Nwb
z=v(DykeahZ@c1_!F_=c4PVEP)3r1fvCUqO_91tP4>t_<%QG8+d4mIQ7mpoT%V=-gX
z8y_^6#{~y+?%pz#Sf00D&htcuF1{L4fYd0h<+Uk*FgLYpMLE3u!D*h(qR~xDT=#s@
zM}9m@GK(GkDPpnr<Xz<K&T=zo2YPvn*Na_KPU^7oZo;{%qYchmlrQh=tt7!-xjqYc
zL2`_N^zx#tJSzQQ43fa65&0RbJ1Oimo_DNiXrr?=wyh{TbC~fBJf3H?FB8b1>{FD!
zDY%h>Ote*=zK3(oqxzHabdZq;dNzyy7e<p+3MAZJTGm+Q#`ddlr$)xKc1rYI5;54V
zN@t>ADqB*qF3(YAFp^fBJ{4-u+*d>9Ava^jB0C at X^S at ze9Z8_#Hs?!t6ATMXxab>d
z5toWlca^E>+l!pMB at Ew#<ZBv6?KjG<OSN!_sCVd)AN$de{J}txABcqQhsqvnOi-nH
z>;;AShN4EH_1WWn03ePHa|Fr1GvgiyKkMl>LivQfA7cfGdaJCDu9A)qU68)5-(^tw
z$3+dk08mW}8dnK6&5&&2`)(0EV_3cH5B+f0sv&IPAp$*suown%3EK&(%Hj6`Tl$`=
z{6Vv9eg_VB|KAoQ$m&AIV{iy3jwB7*mkTcITX;tDF at fO!mcSkbt_Y)?XQ)h;o6XQ6
z`O3SifmMlLaKrAy<Jr-2SX6nN7KyL_R}R|axuYIxYUts<I&w>&YHYcL9WR9&-o_U2
zpbhVjLulU7TYRinD5wMikEUba*E^u2f837y)GSqT3n4tmh9m4eZG8DtB2l;K4hIE#
z<uzDE&n1>auHPm_u5>TttBecwv+mJIUQQ^ve9dcfG*9VskfB1LAMky~${Hy5k*vZz
zA(wC7k_4^9^}f^g3#WP6J_+;pjyRW_k>7Od%o3umlCnsb6d<3t86<KO5F&8104ms(
zL+(xUU=6aUwz%J>&9&kW{@SqFl|7n`sGi~4t$6w#`PK>nr;xQjSLnau3 at w9T*;Ku?
z`yqAz_OP!kJeQZugm<B<BY}^bg=?(EmUKdI;pR(!Gj`JCJN!_G!T5H=j}r@png=}y
zR!0#NzlTo9I<!<CH#NT!a>y?K?>+`<{Y)4bs&91CAji{B?6f)ut(Ag;BuD;Fhd73q
zGzVe29ZBhX8D>*%pvBRNm*~}`3-D(S0CIQwpxQ-`-ulxWGI5aQxepumvzec<7ue;9
z1yEIR>4|<r>2RByi=e>6)aO-A0FK`jO-(50le<n^*T?`F<cG$vg9I|CZLhvSqFQ7&
z^JwGyqVT!AZgj&iAZ9-o3|e6-s;V~M|9y{q#0rylZLG^V$IIo~Ny)AUJXFHl(`t69
zIMoVi>Pu|Dr;8lRU$j5IobSCrZEZHyLy(Xts_D4+@=I~KPNzl0^+MUjef3VlwT!v{
z(!tV);VkrUbB2gDJXqRv!z)@_05vtrW!&eVuE0{|v$U))mM?T}w_x_2cFm;xk3~mj
z07H1{K$%c0QdvU4Z#^Ss@*Z{(4%5F`9sG>23cqvVS$5WP_klIzIz7;~^aKx6VZ$?p
z%BoJ1+STV~-PvS6c#mLKw2grzF#mH48E25jY>F~D&&a?(*b5{a?p}T-IQ*?W=b~$e
z?7QCLlPp0XCH6|?=lJ8pRxLH?Q)OWb at lzs^6QwN$R;+;{$}+2B52f3iuo2pauz}0t
zyJRb{m5H?(MAS(x+b7wZ(?(AWR;(2CvIYn7GwNA_CZS~y+Ss`w6(F{#>@gA-(Lk9b
zw($>0=oFcT&dWY+frZ&vQwo<5Z&_BWa#16A0g!z<msJ>RRAU2d>TkrzSkI>HeNaiR
zp3>R$w8}jWfE2ekBAKn-w{**c&P{+`$Ms0rMp?(}X*!oRA9R`vf7&q{+wo7V#-W<X
z@fd(z<~L(c;35_)>)9v>hkKO|&_P14q<4DQ*05MW-C-dpiZ|v>j>aucVMQ5GMO#c~
zus}1!PrYg;>C=Z3aD?+V#c%FIRE$J|t=^%VbJ79sb+M-9<V5MTR%IL2<bheCuHu8M
zqnNq{nv+8WUTE%9P0VAq<s&9!^)rJ&Okxi*UybNP|7C}|C&6|LobnroKa7zERx!t&
zNo!dp*%C!pVVcVBL&UMO>ksCkdh$V5Dar{*6%c8Wc+1|^&yX!>VKD5*H3Mi?k<c?N
z0BkXzdyELV2R!ABw(=g{8E6nptL5r6!+dc$633BZz~Lno90d8ax1eDe`TMFZ%@C}F
z9Jl3T-f&-M@$*5LAEmv7pB;X=`Y#SL`JX>yf at EH2Bf=Syv1%>=mco~&ZEJXBWRcFv
z9Je%vRStQ-ws~PxeWIOna8HtkX_xcL+uJ_W?lS)WOiT{{+IF5{`YV>Xd0BFn*8FWj
zEejeLjo&Ypol5|g<M_)n-s7$`Y<G9U{dnAQMimf%lU<%~j)<Rn8DIzpH^uI}rvdR0
z$Nqk7lwnpFe%g)Z_AtPe<a?CwBIWt%DoImpm*KdxPbmBYMLcQ3L<gZRi46GK-OCZ7
z^`V6JG#n%W{1L^B%jWO_%QKs$Hl9C0+upPJ6b5^9=yUSriHDKb8_`&=<8~|zYALWy
zh_#lOC$RGr_JO%+9tpLfshpEamHipfEC#P`oA&NSGpOA?SZ<3-BdrHI&`slrY|E=V
z(E-xMTMOp2l0~ddb9^L~c#H})Rg;Bq>MKrjsM4JOAHcM3oibOOdho at EpTws|JnO+X
zNPmsIg?zx!wlm-Qs9_g*++m;laFYOP{{RGseZTjFz4HA6`diVlhd^W|raX@$t}@<#
zQZe)L>QZ(dW$(e)%1Q{CCD{_iBv)h#A;T$Z>}IUnNlMn_Qd3c_J~8yE=Pd_lzUu(F
ze@%|1GwXq96U$-8AINpXaC*hcxU%Y^G8`;C`&Tmvhd30p^V&SbMkwS9b&(W2b0rlS
zxckLp9YB<3{nCDT1gQdLky?uCY;Ch?N$Ee!avymbEY|M^R(^cWWe@`DZzd)=JLH5~
z8+RE*(RF5R+s?Ywjyc&74$bS5bIy&b%~@t}Hw62c^Ft}_{FGci8k#-+|93%>v=zI0
zPAIGax8Vxq+4us8oc6gb4r_MByH3!@sqkDZxR4XZiRmUHZ*fxTCb#^)(h8_?;(9-A
zuee1D;K)!Q-8h(}73$Wck!nr1x&)V`|EHlDcg{}|hH*4qW6B~Fj)|sjI%*eo$N{(|
zAsmU`%E(q>%@ggi$)x${jLmsBLHz1Pm;|Q~FB=Fsa5X))yuX!rIH(5{jIewiNY?iz
zAR+tqc6hpAUMrVa*O#qOo6Hx=Z6jtQ$bL(y;Y=u~h2pEL1PA42BOy!*DMwNO{5*aH
yd72ypwr~zUn!(0u`mVHL_09fxtD!&;2{0oQv?aNzEh2fT7Xn_FK9_17BKHGZI1GRQ
literal 0
HcmV?d00001
diff --git a/support/testing/tests/core/rootfs-overlay2/etc/test-file2 b/support/testing/tests/core/rootfs-overlay2/etc/test-file2
new file mode 100644
index 0000000000000000000000000000000000000000..c9488dff0f6435f9dee3a8ac10daa990d459c718
GIT binary patch
literal 8192
zcmV+bAphT`=ZRSOtwRO%f-ZHlb7n2&a0Zq$4CTxrv)!=BU5zokcvSluU$5e%M-1&#
z6s{iRJ$&+MDE};loza`X`c?dXsac}S%!xi5|GOoAM-D3?ryn(C0{Ie2ua{pVsPA&t
zM9ew*#SBcMavA^tGDU8UCYCWHCgt7yEj?up?d1Gb074GAprMC#vJUO2V^2=(<@DM4
z0PiQ5#|T&al~I6BZ-*KwQdSxsAZi`VX2m}T4sO0@nUqgI6Y3)1XWUex>3rjrvuJj`
z$xaB(i6o2>A;_7VbQeiILXvPpPLEBz+WTus84$201|3JmZMSp&pbdKmp4Wp at X~&Kx
z(ALdi7|oA`IKmztSI2pbDGs7NSr-KXL*8Ua^Aq_zR0Z^8X7if;vEf~`0-d=Q=9s`J
z7bR4i36<VoXgJiaIq|bJhCl6U%HLFT_Ur+9LD&ntWtxG@kOdOH9Jg=c4Or|}^#wDf
za0E2#iFFG`g=ki6FeNDJ<$;Y(<-uMPIo}@O{!9L>Em{={BitO)i7b;NA0SdzW*$wd
z!_K^x20YPFBFTuNcDBNUpo~>0L*%+^S|{YaV`a_xCe&mYE7d9P*i0wqx0V+0+>qB&
z7$)vlc?~x28po=Q&&Dp3suM6-=xKMg>;Wtr{1iE{UZ8VdcHvXPlbZTwX*dS|Z>cC;
zmm6dTQ<c_!rTK&FWXCZwh~lY}h!NgA@IT3DKyqyVx{Dxt8P>oKy|!&oy3gei8OK>#
zQymAVKGf8A>Mr!pv|ZNlcTc6DxPk0C$^k({JP0xlML5&kQLismies<w4GPxDEOB7R
zyozBVGY!);IO+TxK5~M~M`HfKNVpwK7W{z-UohFN8h$1&R$o5Ek-h+`lt<TQfx6XY
zXYURxHxGLV2!G?QsBH2U#VuOur(_bEelc57V_uaPOHl3s(jvlAqY`({Rl8$fT?2<C
zk<><7UTD+mCoNo0J@mA`Q7?<~shOiMvuD=TuT&V`@*b#<z_qKx#*=>Rb`y7b;8o)*
zRK2O6$#AaccEezP{@hfi$VGsWqNCJcRFdKWAAxOqJENkkPzOITq}NV3dBydbsoUiW
zO at GT7YHKx;NGAOW_joRw)@OMpc^P}a`rQigvT)5VBldfCPC1M}`&YO-t2|#&v=p?U
zBh7_>+w7^Svetg~2QpW7Ji$-pW#rL+>=n=~>dss3t*}iMxPC8M8OR~vLQ|HK(Ti8O
zf`Nu&Oj`w#FLq*$m-)8I2<3lEeSeSp5X8@h_Io+W9B*@bQSQV?$*-)c#y)Wfk>;y7
zGP7M{tt>z`uzM7`&}D6~%FPIlbYYb?BXsblO(8ZZ5r91hMIx+$#(tuyq*Q}#faClZ
zMC!+8{m#>ejyGzGV>~?V?X*Q>tr!19+uhe)=MgU|^GmcJ<h8Wfay<XRVU7v8;A1fd
zat7io%?Pq6p^DQp^wT>Vd(2a=j3dp*U_*$m8~}>kPpWx6hFg*h(mcf*#vW6~b?vPV
zh;!oyO+N!eWIA7KM_JVG-7%#0S8gRV99&>mx_7FrGtum7dI6&2;SSrPbGNxv6|3k#
zt{=Y2{<69=NIrLW7CnXbb$PCeK&Lsa?7$l2Uz~8PQ)<oGhBV&V`-TU?p%L5Y@!vW%
z!>%ez{QnKn5scQG8gQ6Yx`I5%odSr$jS?J1rQ)#zA?2ZnY5mxpOfV<LHn!C8tKdZq
zcBE*QYk55&o2+-o28ZvUT?m-LYM`)niDUJ=;HiJ-ZaMIW_R?7$K@v@s$xO^O4+j&4
zLZ~%M4=E)4*=;5;>esp3rd at Cy@=>m982KPKx!Ey_;xgmxH>C8?wISPa!MCO#iVbDC
z0WdDwa5RQA5V2Rgtk4UI6c6%OsIVWcQH(~*Q`JiiSBPVE;^Yj!A%Xe_egi#q%M%}(
zJ3NGlsEvbg^2;U(9u!`IM1db4POV7_l9-il>v_jzyU7blqiKjCeP}p=$HnsFG*Nt?
zwpY4&U=1M*8H9i^frF%Tuhlw@`X1+?FUf$Iudz*w!j_|vI^2r5 at S~5N`_i`DT3=Pe
zH3pA%6(HQGMT%QJ+z?fEJsr5`QH6l09<Q>pXs6G&>fBVye&-vkRgl+nV4`DKkTq7S
z{Rk)`jX*(YOVuwL!QDE6idTqnx{VVM>k*>$42gs{ETsrgb7P%{e~cs<Z1vWgXvTC?
zTAX>r?`yWuJ1BWt8}#tfW3Z-<40y4 at iHv-RDD~^Q*vq+xd_}_x{CG~vkyY&!(joUF
zT&y{nxv4;mnE;N!91*;be_*VI`$OO)XvFO6&7DW8RZSa{iLVE*Su6DO|0v3zc7yu+
zV$~;?df(W_L2Mw+@v5_a1fj~~Up7%sqU3`ySi7o&RpHrm3T0RV1px>6_E~xlQ-N;=
z1X15xD;lUL_Wzc!slKun{7ddRg)rk2w3)EvfoxQf)Rl57e5=ksJ5;qjY1&P~+UkJ5
zKds_^zJtf+)#ud4fn5(NoQxoi5#&2w<Wpcr(jIjjoHL}WG(bwNQP(6Md%AJ#N}BGS
zqBT3rGZpwWBSZ0taV*$2qCT&){C6rq!4T)NoOH7{QtH`4O@0-c{j2#^ObDILu$kED
zN{!xwap%5+U?}9>*+^{L?ZE1Dg&zg2;^ws7{aG`b`%X=@Vw~(SRqm(WszFVFehha!
z;>;|5OmM|MY6QJV=dfF_h)K^A$Qr91YiShT-ChTTp*#RZZJ3!k34K^~1iS&;R@|p|
zTQ_n6nui(=52kZ|9aAeI5P4$K(2$)02(mNvB=#wdquVz&Ew_mq05FwjDTlq-1y>y{
za=VX1Zut$NoGWJp_*uB|5+d^2a90UL1Zjs7D*_K->_su%*5XD8WM at o5$Kz at _2;4b+
ze6RS`LUL5=(|wf2cpCPqIt at UY%%+VYcw2$q2rD5neWQD(#W1xaIzI(vwzymh)?(cI
zJU5f+6{Tg(anU(}m0rbk|5p at GLJ{=j7GqUS;xq9Kd_)PPVZJ^Q!WiJe+Ky~yS9rDD
zV89v*)w`YRU|us?h#iGRMJ1eB2f5zh+l^lnh8C^WPuQ+XLjI4i^y=radipVf_-E17
zDipgs>EXmZB!@sn^9U1uhz!pqnrU%?BE{yW6pL9EASgt39y2BRRlU6QU=H6a&07J1
zg5EonMmy$YI)Wh at 5;4o4x`4DYI~=t#r8K*bAja$M8dfe=-RZpGaGlbiK at H(XG*S6Y
zkI73W2GjKiED}i5)gzdZ<YGl-k5rt^aOJdXUtw$t$*h9-fZ=21RB+(D6atmoe|Kuc
zCLjyYD8eFYS}J9#c?JEN(Z0;F!AKoeQ{5DXlf(_dMruYzQPq&k28iPAX<@>i>=!mc
zNc=ziAAMVt&>S~B=6TyQbZSBjhM!U{N6s{#%;|;DkN|>OWl;$#&yv80L$f9n`lRJt
z6&45h>2?5W at w%Bvk5biurrkuUgT($KM+A_#V1+MPl(o2jFtI;^GWuvpN{gkAPY)ht
z at eyV_rFj66MZdKS0zgGJKhc=YrG58pB`kK at P)L9iBVM^CZJJ}vX*Cp2J1bBuEbL!e
zx+c+=w&5Lm&@6ZXKGu2mw0&U}yb2%G+RT1DAy>1*3}a1SdLfr~d020js1>b0;s at 9Z
zy5M<xgreQ#JD-cAijKm~OUmh)SO3hfapYt&nj}b73&!FoUo|{J{r|C}Oz|4@J<#F{
zw-OxV4Ht=%xS-NovbhyT#o+Tw>V-DN7<Z0_H`8Z(4<~ZCg%c~5BEdVQV^B|lTAM0M
zI%r1<{2}RP?z at 5C9*UNxu_bzt2}0humHQQIs60bkj8(I7=7O=-DR`wheun=%j;tXH
zK(XhANHx<2M4~Q}zw|<%)e_tACW!gUoZ+FCW#xQFOq(ua+}gCmEpI%uUehOxfc4Bf
zizH;Y^?j$CV3XXX^u01u1(&h9<3-oRqj!OC6JmNFb75Q(dlLtgLa4j<k-II4INp1J
z0fr?B=Wf01;F$QYj$x%+xNnyXXZPOO&@EC*rKr@Mns3HJ<i)zYDZp3jhaeO<!_DEo
z%vu^cHda{;KQW4?xvUEI3$HZ6yb@#N?m>gu{&MfZ0XEauhKh?vfoVxXz$60j^^pmw
zi7)BbE8%6XHA^jhYJ++4Cu4P&`)4v{cqw7~>SuJK%7G+t^a*o(pEKX~+%7lqQGXWg
zIf7p<vE`K2zLY3|O$V!j85Th+3mtC=pQc~}bHY`V2t6^Q1ba6?!(9DS(WcgDsIE_U
zPE&e_Pxyu*{^G0nMHrq6ZFWkfvI6>Y{>{9gawBIG)(1999(Nhuq_C71EZ+dx)QIRS
zmafB{`GzjaXs`~En at 8Gcf^GwMFs(oSYL`LNoE5<uASb2xu9(-x!SNYB;T=EqNl<H<
zVy`)P1KfdIK<Y@x_s%CLqA_do6Mx$C{AAqkM9}^MO)&%i>af{279aKw$8Rvukv_1d
zl at KG;x=)<!6r3eY_S*+5M^jW}k-kfcZ<3B|x9g=aK6yd{G!{pNQ*Opxz!nj&5@~lT
z+=UEq+yW7cu!4PtUN>b|BgasX6MR$ZBrSYcr}KmcP}g^Ktzv2h+)XgrU!Fa}9M_zl
z^IJ~Ky0-o=L4W^96JTf(&>4oxYstn+lBB|Uca^12BP+Gx9N{vZ+&{=agpSG%m$4M7
zhY-J0hHP=6R45!ucrbT`491Vq?-LmS2OR)WKg5 at H*`}V~l0Lrcd{9d*C<vcuvdyt{
znk&J%k_e7c)O`QHXu-UKCW)$O1VpKXkzNsTxoqk}tu|L&j<ZB*FUn`_izz+#cpvQX
z=YZUpG7&}!7M#Vc7cTnNRF3g9BDFX+I(1yfBAa0~ssSzcP+{f>(f;sz_Dva=EF1Xp
zY{C`8pVa}5j13bijK7$1K?T35&d?PEs(jZ&LLf`a-lN!GeU5vOr&rBL at DnUBYt&&t
zCC+6-hx$&AH)fi5nn7_YZA{%w`VpwNj4;3u at 61jR`bhylw3l<f&WTw%=U9WApV|YT
zhqP1C;i(IrWJWi~{fZ7-RRwB|X8(M22Alt!_}IxAWp++`=Jv(BR|NYW2te$oh}}F3
zKyW%ngm5;r4_Zl4_m=$qVRY#F;d*kkt;XX`JPOl`bb4NfWVegazL~y-Wck0chmZ(|
zDFsm3$DGZ<UVeW^arr~Tb=T<4!o<2OV|us(2mb%glTtm+UU?fdL$o-q+TN5Nj3(`Q
z{B($i<fj`iU2NrZa7Is4*U8;N(9Y{f!Th+4Sw5oVM55TE%?5d%JuG$p(}BsnFjnZA
z6CJfPIEvmg&jL at _#-cN#7{=5XAQ9U_(n0r8`t-j_am#nX&#hN6{#_iK2d}$LKzH&{
zrnOt%%fSU2g+l%)g;9jvtgY7bY6lyH$8TC%I0{MRG{{02`Y!nUJyXEoJHQ8{ZisI`
zouXin1arhSK7WR@V7a0H0v{EQ728*XE%SA81Ek(+2gh<X^PqBs^%m_9ozEE~MoRV=
zU7gNSYRX(G;Hyb~iWC1hm9V9oWdIc+v{~!kZz39<5?|i<Q?AB8zUfhDz$E<VufI&j
zN1w-`NgcQwswa=GfCr>szees|Y!4Q$4^NzcxTO(OF7c#iJ7~-$PIXA{eQ^$&UYPN^
zM*K@~Yk-wm$a%k<IM|1p_;Lrv$?sGR<`&Q0B}dIW(dpLZLbF&zWb-F#pzeUU%L0@L
zdPYtFf7}Ci+iEwu6%?)RU~_>YOW{f86Z}J%_dty#P4%)f%Bf^+0fMrk<{-v6Bz4d2
ztsubH995<**xzK8*qOjs(wDdfuCZi@uCp@^bM5ut1n6$uOa^Mokq}9KkU$jLcZgAK
z9NGuTO*VRTLl4X6{0oD2>@CG$`v5K8g-Cnt|DUkl9%2OY(p;;P=_ at v7qM8Y~@ZQ-5
zF!wvWH6gm*G<vQo>JI|$IGd3GB3$AAAcYbJb1IBX0D at h&o-sj`6k-M+v&cQPi^XzU
zyQb=2nHYpR%kEgy3ux4CF%+nM?Lpvg8pL at s_Ey0@43`$gU}#Rnb%+Ce0X#)PEJ9S4
zj&ywF6m7ds;RmbHir%xY6tess+~J-KQM+)!cQm;-7*NZ)_=H;KOd at n*ve=WTGoL&7
z+(}IOu0{5b%PfAnhc42+oW2tAii1LJBmetz8QV01gATnf+v&%W>xj{ov2qEjkFMfB
z(`3B<<f<m<jS5_usC}%y9tN43RWc1MmcI_2Lw}`hGVm`RHhlkQrX>*=UEvxQ-9#c)
zs14=wmCnrIV#uQM&w=}FT$VT)xO%|LnlRed at F)2&zT0B37 at e4KCz(g4E{&AT2AA)Q
zT6LLy!l634+;_cOevHYlIxRGT!0H_xtLY1LvW(%?wuq+jUGT3Qr<dMNs4V<{=&JE}
zWZ<5Nv|~e=`}g?!N*z$S8MIFML0jV{1?(#{Bz@D>MXHW8GQ6 at S5E&SY=}lq^NNk+v
zR}Y=8ZBrZC|7+`7lsIRS(`A$#I2!FO_Kt7*FM&6G|Kt#{%)q(@U6XHs at BV4~UF#ar
zDz8K4Q)<BY`sEs$Z4iOVWM2S+zI(cv#?QS>fi`AGS8*9a&>$^t7gyf2&|AnH??U$H
zv|Y2bk!IXLc^Niyg81_KRN+3|x#KpOX7;ajdx|@$gt8nC(o5Z?;{6uuW=T%4)wH_i
z at r-Yt@K0$CAvm)7b_Vz<1Vy_YY#zAP<XaC}&~sR`R56qAN4DPhKrX|f)gOW4lRN*g
z>@y_K^CRElZURDT`-9s0*omQW2Rcl)cVodC>Pt_6D>-Fs{~O(6i}2~u5BN4sq|$kl
zpZJSANbX;pI9t9;VD&JsfrWr7>-MHV1hi_k0Hd|0$2B<>(F5%y`Bi+Ix&F-&sFAh)
zwsInf`G?uJB5)|f{cDVUcEo$`R`xq1olE?#7-;v;gzv_A*Z$zlC197rz^(XHA0%(<
z_lY>KQAJVWUVQ>QBfOzuuDP-FsMUKJiNy}IlM$|GII39W7;=pk&`X3U++K5OU%n5P
z=zhYuj~IBq>fEak?+4!;f3}_RN+H4|2We9mLw(VwIDREs at nQBSO`^h;wkYFzrP-RK
z_IB?cvgi*}tc5mF+e|A4v#bJgz5gfVLhH3KPdf=i^QK*=P#^NDI3r79{c|jFyG**H
zJ5tGJk?9sJAC`qLDi<0~O;v64PEr?XT4-BAS?G~69l+Znl@IXF6Da|4myfbm>V?!|
zLy%ViMHE<Bp|#)~XHJ3?<v(sA@sLx$)oYE&=mKh?Bu6WK1vQyK-kjs2{GJ}yxrSaz
z<-%*if-cdp$>*`ti%08<uq|_R6TjeltI(@2#*GD3V!pi?vZBmq)~U}30h3CAYNY7F
zs2<ado*rhPM?{5*jNP9XKy=nuRCGP)%tXi at VNClorQ(*c^%rDELk}?d(6mV7aHoQf
z+=ja9(V9S}zytmiovkR3Dr at o=?%+!yoL@8TW$0U`(X}d+qyf4bDSbrHuS>jM`M^}i
zOB}|Pb0Bs~WTU#7xDo}#hCodPx9-#BkPG8f2#gY;2Pt7)3a_!F=2u*ra5zC47c>@H
zgk~$&1GRR1yDfMV#kjDu`#qzCyYp%m^mqRYJL4E|g46O=4+>WtBS=o|2OcV|<gme$
z#ET^uvA(<eA;GP)GCO!uwA+-yflia@1X%+n<qA=rVuS_o>O;aLP7o&r>0a~CRF;>I
z!w$88N}VZVF{rk%Dq6(*m9CU29D0KBM0-=*WsdORa{wisBg8E;muJu0^?{nsw1NPz
zXKs$vOVY~xZoQ<vJn;hh)gmZB=5cqCRo#LsHJp$+Gla>dg<R|E>&a)bPl-NJPzy&W
zT67B|;S3A?C6eP9u2Smdo;yM=T&!;23O5bvZw>DO9y7Iz!$KrJ<%>9hOuIc=If+H7
z8eKm_Ot+~jH}1N}pscgz=nc at gGR{^nuvZo-cJvwLDHN{na3(Wi20~zRjT#~-zI^Tc
z{$wnlP&pNp=ZIN7-C0G2O at GyaS`x!m{5j#~5b^<`PBFU4^Dm!=;x<|dt=GW@<PLUs
z_agC~k)Gz8R!icGNh(6H4!;ch@tyylizpul7|a&FtFLoQcie6_(69d0=YD=Dh7>TQ
z4+o_)d|J!0L+i`q(6DObos9LKjtr;dx93x?**zi_PVVye at +FT7c#72zJEC at IIRecm
zIkTC(Hk&X`2jubpByY#0znG<K*$&JRRTf=Y;LLp at Ymvmuv6KLz7p8Jna)<kXUii*%
z`$7Rl_d0`YbKMDC9+i{z0CsPVGp^v%FA at YMt6Im{6e|s at JJ82v8DSD{!QM@nUSB)+
zXf-P6c%iCp2B2Z;H!AeVB~;%IH!Tt7ezMXSaXO|SmpH41gF5jw)r3u$uMvcd`5Wj*
z4;&d%7NOX~>71ihU*OW)09(f3<C)QYn9ffQAGo!AZ<}`vZkM<d^}`Ke3)Od~L*vS1
zghA7#VT}mAsEuTXnpwd8^#{aX^cXWk*w{py|4H}-?-Ksh0UnhKB9w8<7=ZKt&Ncv)
z#3?fWdHIX)HE)K=y$$=ORF&Re!fo&Rcr~vg14@ToT?pp1J643SK9h}p8A;PiBLTQX
z1JReAKO&$<Z^p+m1gU+3aGO9GYhH#=<g2rim&|b-jA>Vrrvi29=W=(_r5BTsEG$4#
zv8`$PQ-JN)ft}*i&XQ`SRSnmNC6=_^uIZ-YZ8cM8A0=W*E?N!l%%GWqNUMvOH3dFE
zsM{<daNA7YJIxmX1#mQ7mK8Hdv01#b4}l(_^02w)Pa;b9=uw3t&rS>I3Kb>@G<zn9
zDd*f)zp<ax`gkbR_K7XZGg`+YGF6c?uVZsyJ@wZs0_P|9MN*T*j<eOr!{@{t{S$Tl
z_aI}J75?F=0dw;MPAK+%&1dS7UZ}}$J`co6GXHpz;+NrC;a5s%H6&`a9#bl{3F^rz
zAvzxOvUz>TW8-Pq92_#_T#GxMi@}HHu$mTz((9_Mr^_;?5jNTb1xVnB4f&JlE=LaU
zz>%$P=hh)bIQ6gCS4cpy at EW6emkoMLX!`h7m!+N*BU-KCu`V<lf;GC=4F{|Zs+h=%
z?dP^#-e;d+4|UUIf7BK$!fS*h8zJThVZdg!&U%nDQ@|9fD7aY+Q&{Sl=AqJV$Oi~4
zWgJLsf98Y!1xtr4(Q=^GKi%lB^f<_@{792f==|pA5;{^tNeX7#W^)jlKlbx^)v;%d
z=~lU6=Xhdp`-<kFVS{5%tpP_Xt!WX$XHQ-;7$mr?eUYs06>ts1fjCH at b1g3R?viAn
zLli0+0s;IWcVxaAx&R8p4ah1D^6Kq5<ZxQk6F3L<-)Zr^v@Su<lF`2?lo<>y>xPck
zQ2TRO5dQKaN at V%c*I8}vjW28+jTX(8ygch%fkfipsOWKdW|}X%V<hgjp}n%-Kf#eS
z735U<M7{@mbztMzKdqp~nDW!}j}pPq<9LLmBxIA}4^kz$y5dBfX)R8gj);-<S%eP>
z%wg|ykHG<z-_v&BysDn$tDQ?X8~nsgH^b8_Y6^WmOgWo$`E9cR5b%FPWK)+cshU_9
zn~57LOesB&rKU1e*VSpL&x?A>e)`X21o)6|bjdzybFdg}pIl<fRyq;l`&0t=+pH@m
zUqW4tLP46(u0>%NOtCef5ABjgyuM$i{Bilo?AsTBaX8MQST|neFEe2`W(@_(ma9eC
zy*vv^DeiIBA{as<mg+7(6wDSO;`JIh%TcI`zb!#nVIZdP;G1=hw`-KvVNNbs8csW!
zY#Kf`Y=5CwS!p~~E;-K&5wAu$_C4+_67j+=6J#@WcN7~oI?Wyxe?apLznS-L(-)@U
zz?|-^3z74R2HF}3oQD?3(QY$3O%H(*gG|Aq7s@YxRfAYLdz<{+v->+N5rb|I`u`14
z#?S5gH_F$oziYab<|uy$Jyc+$ODDmh+yOd&wpC~gVmogB^4$0_i7@>=1)#l{w6d1E
zc+1Q!=Tv&}6~U!}Kqk9 at XH7z5t8mD-x^={<l1E*ZxwZ%QKG}gTnWamljY_5}$9*5^
zJt4gdk>cVZQ+ at c?&Q&oKYP{9IH6(ZK#N7Od%Xk}3Tp_vt)1B2mc`IL|w8*qI#!yN-
z!NaV}NoMq7Nb^em+$)U4ikS5V at SO0+*@wR|;OR8;m3%PqFH^U~cw;p-?R3*-9l(xL
zJw2=(+B{8}_alo!lylNU_{`fI-t-GFMT_>)(8h*|@sAjd3}5x1juwehUYIgLZza>5
zAAVAJ;3FzegIdWNAq>@+>n8q!TjPkB%ZK?x3!&U=X)O=3T2|v<wGvg$)6t)s{|uI<
zI?e+M)>kd^Z%<Ki!37Ol#_EAORLvJ(rws)6G}U&Ir8}Tts+7=Z7`rtJ0UvAlgmIE>
zfOzN$_fOyPeS{%CH#o|hcIf`jWk^z#9<@n at N~rZcmApQ^m<c{Ol3~VM%uN%(BB+-!
zBGBZqL9#6ceG8e4$IU0(uY(Y5?6X9G1D)nr8F%H)+m?NDVrjR<jEG<YeY<oSvq*uV
zcpONjqYkPi@K!d6HFVpo8>co)ai^P)-^-!H)(aVCgCfh<g(q@W-pEa-Y$9%&?TSXO
zcaNB*->744Y!-sVb7Q}ySmfjcuna_BX!zdiu2tq7vr#kMhrkbV5sAvZRTYOp!c=-7
za4pgoql&GElYzE&CGw;Q3l&y}qx&ljRN8PaJ@&IBOiJSr!R3 at k(sMz5eEp|J5Sp!4
z8bYOol|CIWXG`A|l$=%I{4h;Ut1{GT$BL+llrFnY3X(#A<eaVuBnp#_@HblWgn?H=
mc#N<=BgLV1-)U!8j}zr`9=5D9AA!_osZSLWra6pxTf7y>1=ySb
literal 0
HcmV?d00001
diff --git a/support/testing/tests/core/test_post_scripts.py b/support/testing/tests/core/test_post_scripts.py
new file mode 100644
index 0000000..1485f28
--- /dev/null
+++ b/support/testing/tests/core/test_post_scripts.py
@@ -0,0 +1,36 @@
+import os
+
+import infra.basetest
+
+class TestPostScripts(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_INIT_NONE=y
+BR2_SYSTEM_BIN_SH_NONE=y
+# BR2_PACKAGE_BUSYBOX is not set
+BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+BR2_ROOTFS_POST_IMAGE_SCRIPT="{}"
+BR2_ROOTFS_POST_SCRIPT_ARGS="foobar baz"
+""".format(infra.filepath("tests/core/post-build.sh"),
+ infra.filepath("tests/core/post-image.sh"))
+
+ # TODO
+ def check_post_log_file(self, path, what):
+ with open(path, "r") as f:
+ lines = f.read().splitlines()
+ self.assertEqual(lines[0], os.path.join(self.builddir, what))
+ self.assertEqual(lines[1], "foobar")
+ self.assertEqual(lines[2], "baz")
+ self.assertEqual(lines[3], os.path.join(self.builddir, "target"))
+ self.assertEqual(lines[4], os.path.join(self.builddir, "build"))
+ self.assertEqual(lines[5], os.path.join(self.builddir, "host"))
+ staging = os.readlink(os.path.join(self.builddir, "staging"))
+ self.assertEqual(lines[6], staging)
+ self.assertEqual(lines[7], os.path.join(self.builddir, "images"))
+ self.assertEqual(lines[8], os.path.join(self.builddir, ".config"))
+
+ def test_run(self):
+ f = os.path.join(self.builddir, "build", "post-build.log")
+ self.check_post_log_file(f, "target")
+ f = os.path.join(self.builddir, "build", "post-image.log")
+ self.check_post_log_file(f, "images")
diff --git a/support/testing/tests/core/test_rootfs_overlay.py b/support/testing/tests/core/test_rootfs_overlay.py
new file mode 100644
index 0000000..42d890b
--- /dev/null
+++ b/support/testing/tests/core/test_rootfs_overlay.py
@@ -0,0 +1,27 @@
+import os
+import subprocess
+
+import infra.basetest
+
+def compare_file(file1, file2):
+ return subprocess.call(["cmp", file1, file2])
+
+class TestRootfsOverlay(infra.basetest.BRTest):
+
+ rootfs_overlay_path = infra.filepath("tests/core/rootfs-overlay")
+ rootfs_overlay = "BR2_ROOTFS_OVERLAY=\"{0}1 {0}2\"".format(rootfs_overlay_path)
+
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ infra.basetest.MINIMAL_CONFIG + \
+ rootfs_overlay
+
+ def test_run(self):
+ target_file = os.path.join(self.builddir, "target", "test-file1")
+ overlay_file = "{}1/test-file1".format(self.rootfs_overlay_path)
+ ret = compare_file(overlay_file, target_file)
+ self.assertEqual(ret, 0)
+
+ target_file = os.path.join(self.builddir, "target", "etc", "test-file2")
+ overlay_file = "{}2/etc/test-file2".format(self.rootfs_overlay_path)
+ ret = compare_file(overlay_file, target_file)
+ self.assertEqual(ret, 0)
diff --git a/support/testing/tests/core/test_timezone.py b/support/testing/tests/core/test_timezone.py
new file mode 100644
index 0000000..51cf912
--- /dev/null
+++ b/support/testing/tests/core/test_timezone.py
@@ -0,0 +1,66 @@
+import os
+
+import infra.basetest
+
+def boot_armv5_cpio(emulator, builddir):
+ img = os.path.join(builddir, "images", "rootfs.cpio")
+ emulator.boot(arch="armv5", kernel="builtin",
+ options=["-initrd", img])
+ emulator.login("root")
+
+class TestNoTimezone(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ """
+# BR2_TARGET_TZ_INFO is not set
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+
+class TestGlibcAllTimezone(infra.basetest.BRTest):
+ config = """
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TARGET_TZ_INFO=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "PST")
+ tz, _ = self.emulator.run("TZ=Europe/Paris date +%Z")
+ self.assertEqual(tz[0].strip(), "CET")
+
+class TestGlibcNonDefaultLimitedTimezone(infra.basetest.BRTest):
+ config = """
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TARGET_TZ_INFO=y
+BR2_TARGET_TZ_ZONELIST="northamerica"
+BR2_TARGET_LOCALTIME="America/New_York"
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ boot_armv5_cpio(self.emulator, self.builddir)
+ tz, _ = self.emulator.run("date +%Z")
+ self.assertEqual(tz[0].strip(), "EST")
+ tz, _ = self.emulator.run("TZ=UTC date +%Z")
+ self.assertEqual(tz[0].strip(), "UTC")
+ tz, _ = self.emulator.run("TZ=America/Los_Angeles date +%Z")
+ self.assertEqual(tz[0].strip(), "PST")
+ tz, _ = self.emulator.run("TZ=Europe/Paris date +%Z")
+ self.assertEqual(tz[0].strip(), "Europe")
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 3/5] support/testing: add fs tests
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
2017-02-07 11:36 ` [Buildroot] [PATCH 2/5] support/testing: add core tests Thomas Petazzoni
@ 2017-02-07 11:36 ` Thomas Petazzoni
2017-03-02 1:51 ` Ricardo Martincoski
2017-02-07 11:36 ` [Buildroot] [PATCH 4/5] support/testing: add package tests Thomas Petazzoni
` (2 subsequent siblings)
5 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:36 UTC (permalink / raw)
To: buildroot
This commit adds a number of test cases for various filesystem formats:
ext2/3/4, iso9660, jffs2, squashfs, ubi/ubifs and yaffs2. All of them
except yaffs2 are runtime tested. The iso9660 set of test cases is
particularly rich, testing the proper operation of the iso9660 support
with all of grub, grub2 and isolinux.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
| 20 +++
support/testing/conf/grub2.cfg | 7 +
support/testing/conf/isolinux.cfg | 5 +
.../testing/conf/minimal-x86-qemu-kernel.config | 23 +++
support/testing/tests/fs/__init__.py | 0
support/testing/tests/fs/test_ext.py | 119 +++++++++++++++
support/testing/tests/fs/test_iso9660.py | 161 +++++++++++++++++++++
support/testing/tests/fs/test_jffs2.py | 45 ++++++
support/testing/tests/fs/test_squashfs.py | 37 +++++
support/testing/tests/fs/test_ubi.py | 39 +++++
support/testing/tests/fs/test_yaffs2.py | 12 ++
11 files changed, 468 insertions(+)
create mode 100644 support/testing/conf/grub-menu.lst
create mode 100644 support/testing/conf/grub2.cfg
create mode 100644 support/testing/conf/isolinux.cfg
create mode 100644 support/testing/conf/minimal-x86-qemu-kernel.config
create mode 100644 support/testing/tests/fs/__init__.py
create mode 100644 support/testing/tests/fs/test_ext.py
create mode 100644 support/testing/tests/fs/test_iso9660.py
create mode 100644 support/testing/tests/fs/test_jffs2.py
create mode 100644 support/testing/tests/fs/test_squashfs.py
create mode 100644 support/testing/tests/fs/test_ubi.py
create mode 100644 support/testing/tests/fs/test_yaffs2.py
--git a/support/testing/conf/grub-menu.lst b/support/testing/conf/grub-menu.lst
new file mode 100644
index 0000000..6143d80
--- /dev/null
+++ b/support/testing/conf/grub-menu.lst
@@ -0,0 +1,20 @@
+default 0
+timeout 1
+
+# Used when no splashimage is used
+color cyan/blue white/blue
+
+# Gets enabled/disabled depending on Grub support for splashimage
+splashimage /boot/grub/splash.xpm.gz
+
+# Used when a splashimage is enabled
+foreground 000000
+background cccccc
+
+title Buildroot ISO9660 image
+kernel __KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+initrd __INITRD_PATH__
+
+title Hard Drive (first partition)
+rootnoverify (hd0)
+chainloader +1
diff --git a/support/testing/conf/grub2.cfg b/support/testing/conf/grub2.cfg
new file mode 100644
index 0000000..a982d0b
--- /dev/null
+++ b/support/testing/conf/grub2.cfg
@@ -0,0 +1,7 @@
+set default="0"
+set timeout="1"
+
+menuentry "Buildroot" {
+ linux __KERNEL_PATH__ root=/dev/sr0 console=ttyS0,115200
+ initrd __INITRD_PATH__
+}
diff --git a/support/testing/conf/isolinux.cfg b/support/testing/conf/isolinux.cfg
new file mode 100644
index 0000000..ba031a6
--- /dev/null
+++ b/support/testing/conf/isolinux.cfg
@@ -0,0 +1,5 @@
+default 1
+label 1
+ kernel __KERNEL_PATH__
+ initrd __INITRD_PATH__
+ append root=/dev/sr0 console=ttyS0,115200
diff --git a/support/testing/conf/minimal-x86-qemu-kernel.config b/support/testing/conf/minimal-x86-qemu-kernel.config
new file mode 100644
index 0000000..8f6ceef
--- /dev/null
+++ b/support/testing/conf/minimal-x86-qemu-kernel.config
@@ -0,0 +1,23 @@
+# CONFIG_64BIT is not set
+CONFIG_SYSVIPC=y
+CONFIG_BLK_DEV_INITRD=y
+# CONFIG_SUSPEND is not set
+# CONFIG_ACPI is not set
+CONFIG_CPU_IDLE=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_BLK_DEV_SR=y
+CONFIG_ATA=y
+CONFIG_ATA_PIIX=y
+CONFIG_INPUT_EVDEV=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_POWER_SUPPLY=y
+CONFIG_THERMAL=y
+CONFIG_EXT4_FS=y
+CONFIG_ISO9660_FS=y
+CONFIG_JOLIET=y
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+# CONFIG_VIRTUALIZATION is not set
diff --git a/support/testing/tests/fs/__init__.py b/support/testing/tests/fs/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
new file mode 100644
index 0000000..f7e2e85
--- /dev/null
+++ b/support/testing/tests/fs/test_ext.py
@@ -0,0 +1,119 @@
+import os
+import subprocess
+
+import infra.basetest
+
+VOLNAME_PROP = "Filesystem volume name"
+REVISION_PROP = "Filesystem revision #"
+FEATURES_PROP = "Filesystem features"
+BLOCKCNT_PROP = "Block count"
+INODECNT_PROP = "Inode count"
+RESBLKCNT_PROP = "Reserved block count"
+
+CHECK_FS_TYPE_CMD = "mount | grep '/dev/root on / type {}'"
+
+def dumpe2fs_run(builddir, image):
+ cmd = ["host/usr/sbin/dumpe2fs", os.path.join("images", image)]
+ ret = subprocess.check_output(cmd,
+ stderr=open(os.devnull, "w"),
+ cwd=builddir,
+ env={"LANG": "C"})
+ return ret.strip().splitlines()
+
+def dumpe2fs_getprop(out, prop):
+ for lines in out:
+ lines = lines.split(": ")
+ if lines[0] == prop:
+ return lines[1].strip()
+
+def boot_img_and_check_fs_type(emulator, builddir, fs_type):
+ img = os.path.join(builddir, "images", "rootfs.{}".format(fs_type))
+ emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mmcblk0",
+ "rootfstype={}".format(fs_type)],
+ options=["-drive", "file={},if=sd".format(img)])
+ emulator.login()
+ _, exit_code = emulator.run(CHECK_FS_TYPE_CMD.format(fs_type))
+ return exit_code
+
+class TestExt2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_2r0=y
+BR2_TARGET_ROOTFS_EXT2_LABEL="foobaz"
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext2")
+ self.assertEqual(dumpe2fs_getprop(out, VOLNAME_PROP), "foobaz")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "0 (original)")
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext2")
+ self.assertEqual(exit_code, 0)
+
+class TestExt2r1(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_2r1=y
+BR2_TARGET_ROOTFS_EXT2_LABEL="foobar"
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext2")
+ self.assertEqual(dumpe2fs_getprop(out, VOLNAME_PROP), "foobar")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertNotIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext2")
+ self.assertEqual(exit_code, 0)
+
+class TestExt3(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_3=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext3")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext3")
+ self.assertEqual(exit_code, 0)
+
+class TestExt4(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_EXT2=y
+BR2_TARGET_ROOTFS_EXT2_4=y
+BR2_TARGET_ROOTFS_EXT2_BLOCKS=16384
+BR2_TARGET_ROOTFS_EXT2_INODES=3000
+BR2_TARGET_ROOTFS_EXT2_RESBLKS=10
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ out = dumpe2fs_run(self.builddir, "rootfs.ext4")
+ self.assertEqual(dumpe2fs_getprop(out, REVISION_PROP), "1 (dynamic)")
+ self.assertEqual(dumpe2fs_getprop(out, BLOCKCNT_PROP), "16384")
+ # Yes there are 8 more inodes than requested
+ self.assertEqual(dumpe2fs_getprop(out, INODECNT_PROP), "3008")
+ self.assertEqual(dumpe2fs_getprop(out, RESBLKCNT_PROP), "1638")
+ self.assertIn("has_journal", dumpe2fs_getprop(out, FEATURES_PROP))
+ self.assertIn("extent", dumpe2fs_getprop(out, FEATURES_PROP))
+
+ exit_code = boot_img_and_check_fs_type(self.emulator,
+ self.builddir, "ext4")
+ self.assertEqual(exit_code, 0)
+
+
diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
new file mode 100644
index 0000000..b56ff07
--- /dev/null
+++ b/support/testing/tests/fs/test_iso9660.py
@@ -0,0 +1,161 @@
+import os
+
+import infra.basetest
+
+BASIC_CONFIG = \
+"""
+BR2_x86_pentium4=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-i386-pentium4-full-2015.05-496-g85945aa.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_2=y
+BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
+BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
+BR2_LINUX_KERNEL=y
+BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.0"
+BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
+BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="{}"
+# BR2_TARGET_ROOTFS_TAR is not set
+""".format(infra.filepath("conf/minimal-x86-qemu-kernel.config"))
+
+def test_mount_internal_external(emulator, builddir, internal=True):
+ img = os.path.join(builddir, "images", "rootfs.iso9660")
+ emulator.boot(arch="i386", options=["-cdrom", img])
+ emulator.login()
+
+ cmd = "mount | grep '/dev/root on / type iso9660'"
+ if internal:
+ cmd = "mount | grep 'rootfs on / type rootfs'"
+
+ _, exit_code = emulator.run(cmd)
+ return exit_code
+
+def test_touch_file(emulator):
+ _, exit_code = emulator.run("touch test")
+ return exit_code
+
+#
+# Grub 2
+#
+
+class TestIso9660Grub2External(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_GRUB2=y
+BR2_TARGET_GRUB2_BOOT_PARTITION="cd"
+BR2_TARGET_GRUB2_BUILTIN_MODULES="boot linux ext2 fat part_msdos part_gpt normal biosdisk iso9660"
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub2.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660Grub2Internal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_ROOTFS_ISO9660_INITRD=y
+BR2_TARGET_GRUB2=y
+BR2_TARGET_GRUB2_BOOT_PARTITION="cd"
+BR2_TARGET_GRUB2_BUILTIN_MODULES="boot linux ext2 fat part_msdos part_gpt normal biosdisk iso9660"
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub2.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+#
+# Grub
+#
+
+class TestIso9660GrubExternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_GRUB=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub-menu.lst"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660GrubInternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_GRUB=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+""".format(infra.filepath("conf/grub-menu.lst"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+#
+# Syslinux
+#
+
+class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+# BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+BR2_TARGET_ROOTFS_ISO9660_HYBRID=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+BR2_TARGET_SYSLINUX=y
+""".format(infra.filepath("conf/isolinux.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=False)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
+
+class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
+ config = BASIC_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_ISO9660=y
+BR2_TARGET_ROOTFS_ISO9660_INITRD=y
+BR2_TARGET_ROOTFS_ISO9660_HYBRID=y
+BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+BR2_TARGET_SYSLINUX=y
+""".format(infra.filepath("conf/isolinux.cfg"))
+
+ def test_run(self):
+ exit_code = test_mount_internal_external(self.emulator,
+ self.builddir, internal=True)
+ self.assertEqual(exit_code, 0)
+
+ exit_code = test_touch_file(self.emulator)
+ self.assertEqual(exit_code, 1)
diff --git a/support/testing/tests/fs/test_jffs2.py b/support/testing/tests/fs/test_jffs2.py
new file mode 100644
index 0000000..0d45af2
--- /dev/null
+++ b/support/testing/tests/fs/test_jffs2.py
@@ -0,0 +1,45 @@
+import os
+import subprocess
+
+import infra.basetest
+
+def jffs2dump_find_file(files_list, fname):
+ for file_name in files_list:
+ file_name = file_name.strip()
+ if file_name.startswith("Dirent") and file_name.endswith(fname):
+ return True
+ return False
+
+class TestJffs2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_JFFS2=y
+BR2_TARGET_ROOTFS_JFFS2_CUSTOM=y
+BR2_TARGET_ROOTFS_JFFS2_CUSTOM_EBSIZE=0x80000
+BR2_TARGET_ROOTFS_JFFS2_NOCLEANMARKER=y
+BR2_TARGET_ROOTFS_JFFS2_PAD=y
+BR2_TARGET_ROOTFS_JFFS2_PADSIZE=0x4000000
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ # TODO: there are some scary JFFS2 messages when one starts to
+ # write files in the rootfs: "jffs2: Newly-erased block contained
+ # word 0x0 at offset 0x046c0000". To be investigated.
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.jffs2")
+ out = subprocess.check_output(["host/usr/sbin/jffs2dump", "-c", img],
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+ self.assertTrue(jffs2dump_find_file(out, "busybox"))
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mtdblock0",
+ "rootfstype=jffs2"],
+ options=["-drive", "file={},if=pflash".format(img)])
+ self.emulator.login()
+ cmd = "mount | grep '/dev/root on / type jffs2'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
new file mode 100644
index 0000000..edaa087
--- /dev/null
+++ b/support/testing/tests/fs/test_squashfs.py
@@ -0,0 +1,37 @@
+import os
+import subprocess
+
+import infra.basetest
+
+class TestSquashfs(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_SQUASHFS=y
+# BR2_TARGET_ROOTFS_SQUASHFS4_GZIP is not set
+BR2_TARGET_ROOTFS_SQUASHFS4_LZ4=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ unsquashfs_cmd = ["host/usr/bin/unsquashfs", "-s", "images/rootfs.squashfs"]
+ out = subprocess.check_output(unsquashfs_cmd,
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+ self.assertEqual(out[0],
+ "Found a valid SQUASHFS 4:0 superblock on images/rootfs.squashfs.")
+ self.assertEqual(out[3], "Compression lz4")
+
+ img = os.path.join(self.builddir, "images", "rootfs.squashfs")
+ subprocess.call(["truncate", "-s", "%1M", img])
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=/dev/mmcblk0",
+ "rootfstype=squashfs"],
+ options=["-drive", "file={},if=sd,format=raw".format(img)])
+ self.emulator.login()
+
+ cmd = "mount | grep '/dev/root on / type squashfs'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_ubi.py b/support/testing/tests/fs/test_ubi.py
new file mode 100644
index 0000000..ede4999
--- /dev/null
+++ b/support/testing/tests/fs/test_ubi.py
@@ -0,0 +1,39 @@
+import subprocess
+import os
+
+import infra.basetest
+
+class TestUbi(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_ROOTFS_UBIFS=y
+BR2_TARGET_ROOTFS_UBIFS_LEBSIZE=0x7ff80
+BR2_TARGET_ROOTFS_UBIFS_MINIOSIZE=0x1
+BR2_TARGET_ROOTFS_UBI=y
+BR2_TARGET_ROOTFS_UBI_PEBSIZE=0x80000
+BR2_TARGET_ROOTFS_UBI_SUBSIZE=1
+"""
+
+ # TODO: if you boot Qemu twice on the same UBI image, it fails to
+ # attach the image the second time, with "ubi0 error:
+ # ubi_read_volume_table: the layout volume was not found".
+ # To be investigated.
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.ubi")
+ out = subprocess.check_output(["file", img],
+ cwd=self.builddir,
+ env={"LANG": "C"})
+ out = out.splitlines()
+
+ subprocess.call(["truncate", "-s 128M", img])
+
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ kernel_cmdline=["root=ubi0:rootfs",
+ "ubi.mtd=0",
+ "rootfstype=ubifs"],
+ options=["-drive", "file={},if=pflash".format(img)])
+ self.emulator.login()
+ cmd = "mount | grep 'ubi0:rootfs on / type ubifs'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
diff --git a/support/testing/tests/fs/test_yaffs2.py b/support/testing/tests/fs/test_yaffs2.py
new file mode 100644
index 0000000..0ffb758
--- /dev/null
+++ b/support/testing/tests/fs/test_yaffs2.py
@@ -0,0 +1,12 @@
+import os
+
+import infra.basetest
+
+class TestYaffs2(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+ infra.basetest.MINIMAL_CONFIG + \
+ "BR2_TARGET_ROOTFS_YAFFS2=y"
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.yaffs2")
+ self.assertTrue(os.path.exists(img))
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 4/5] support/testing: add package tests
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
` (2 preceding siblings ...)
2017-02-07 11:36 ` [Buildroot] [PATCH 3/5] support/testing: add fs tests Thomas Petazzoni
@ 2017-02-07 11:36 ` Thomas Petazzoni
2017-03-02 1:53 ` Ricardo Martincoski
2017-02-07 11:36 ` [Buildroot] [PATCH 5/5] support/testing: add toolchain tests Thomas Petazzoni
2017-03-02 1:45 ` [Buildroot] [PATCH 0/5] Runtime testing infrastructure Ricardo Martincoski
5 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:36 UTC (permalink / raw)
To: buildroot
This commit adds some basic tests for two Buildroot packages: python and
dropbear. These tests are by no mean meant to be exhaustive, but mainly
to serve as initial examples for other tests.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/package/__init__.py | 0
support/testing/tests/package/test_dropbear.py | 28 +++++++++++++++++++++
support/testing/tests/package/test_python.py | 35 ++++++++++++++++++++++++++
3 files changed, 63 insertions(+)
create mode 100644 support/testing/tests/package/__init__.py
create mode 100644 support/testing/tests/package/test_dropbear.py
create mode 100644 support/testing/tests/package/test_python.py
diff --git a/support/testing/tests/package/__init__.py b/support/testing/tests/package/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/package/test_dropbear.py b/support/testing/tests/package/test_dropbear.py
new file mode 100644
index 0000000..6f27ecd
--- /dev/null
+++ b/support/testing/tests/package/test_dropbear.py
@@ -0,0 +1,28 @@
+import os
+
+import infra.basetest
+
+class TestDropbear(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_TARGET_GENERIC_ROOT_PASSWD="root"
+BR2_SYSTEM_DHCP="eth0"
+BR2_PACKAGE_DROPBEAR=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img,
+ "-net", "nic",
+ "-net", "user,hostfwd=tcp::2222-:22"])
+ self.emulator.login("root")
+ cmd = "netstat -ltn 2>/dev/null | grep -q 0.0.0.0:22"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+ # Would be useful to try to login through SSH here, through
+ # localhost:2222, though it is not easy to pass the ssh
+ # password on the command line.
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
new file mode 100644
index 0000000..80e79be
--- /dev/null
+++ b/support/testing/tests/package/test_python.py
@@ -0,0 +1,35 @@
+import os
+
+import infra.basetest
+
+class TestPythonBase(infra.basetest.BRTest):
+ config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+"""
+BR2_PACKAGE_PYTHON=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", cpio_file])
+ self.emulator.login()
+ cmd = "python --version 2>&1 | grep -q '^Python 2'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import math; math.floor(12.3)'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import ctypes;"
+ cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
+ cmd += "print libc.time(None)'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 0)
+
+ cmd = "python -c 'import zlib'"
+ _, exit_code = self.emulator.run(cmd)
+ self.assertEqual(exit_code, 1)
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 5/5] support/testing: add toolchain tests
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
` (3 preceding siblings ...)
2017-02-07 11:36 ` [Buildroot] [PATCH 4/5] support/testing: add package tests Thomas Petazzoni
@ 2017-02-07 11:36 ` Thomas Petazzoni
2017-03-02 1:54 ` Ricardo Martincoski
2017-03-02 1:45 ` [Buildroot] [PATCH 0/5] Runtime testing infrastructure Ricardo Martincoski
5 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-07 11:36 UTC (permalink / raw)
To: buildroot
This commit adds an initial toolchain test case, testing the ARM
CodeSourcery toolchain, just checking that the proper sysroot is used,
and that a minimal Linux system boots fine under Qemu.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
support/testing/tests/toolchain/__init__.py | 0
support/testing/tests/toolchain/test_external.py | 77 ++++++++++++++++++++++++
2 files changed, 77 insertions(+)
create mode 100644 support/testing/tests/toolchain/__init__.py
create mode 100644 support/testing/tests/toolchain/test_external.py
diff --git a/support/testing/tests/toolchain/__init__.py b/support/testing/tests/toolchain/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/support/testing/tests/toolchain/test_external.py b/support/testing/tests/toolchain/test_external.py
new file mode 100644
index 0000000..5d55d38
--- /dev/null
+++ b/support/testing/tests/toolchain/test_external.py
@@ -0,0 +1,77 @@
+import os
+
+import infra
+
+class TestExternalToolchainSourceryArmv4(infra.basetest.BRTest):
+ config = \
+"""
+BR2_arm=y
+BR2_arm920t=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ arch = infra.get_file_arch(self.builddir,
+ "arm-none-linux-gnueabi", "lib/libc.so.6")
+ self.assertEqual(arch, "v4T")
+ symlink = os.path.join(self.builddir, "staging", "armv4t")
+ self.assertTrue(os.path.exists(symlink))
+ self.assertEqual(os.readlink(symlink), "./")
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login("root")
+
+class TestExternalToolchainSourceryArmv5(infra.basetest.BRTest):
+ config = \
+"""
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ arch = infra.get_file_arch(self.builddir,
+ "arm-none-linux-gnueabi", "lib/libc.so.6")
+ self.assertEqual(arch, "v5TE")
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv5",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login("root")
+
+class TestExternalToolchainSourceryArmv7(infra.basetest.BRTest):
+ config = \
+"""
+BR2_arm=y
+BR2_cortex_a8=y
+BR2_ARM_EABI=y
+BR2_ARM_INSTRUCTIONS_THUMB2=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
+BR2_TARGET_ROOTFS_CPIO=y
+# BR2_TARGET_ROOTFS_TAR is not set
+"""
+
+ def test_run(self):
+ arch = infra.get_file_arch(self.builddir,
+ "arm-none-linux-gnueabi", "lib/libc.so.6")
+ self.assertEqual(arch, "v7")
+ isa = infra.get_elf_arch_tag(self.builddir,
+ "arm-none-linux-gnueabi", "lib/libc.so.6",
+ "Tag_THUMB_ISA_use")
+ self.assertEqual(isa, "Thumb-2")
+ symlink = os.path.join(self.builddir, "staging", "thumb2")
+ self.assertTrue(os.path.exists(symlink))
+ self.assertEqual(os.readlink(symlink), "./")
+ img = os.path.join(self.builddir, "images", "rootfs.cpio")
+ self.emulator.boot(arch="armv7",
+ kernel="builtin",
+ options=["-initrd", img])
+ self.emulator.login("root")
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
@ 2017-02-08 9:50 ` Thomas De Schampheleire
2017-02-08 10:05 ` Thomas Petazzoni
2017-03-02 1:46 ` Ricardo Martincoski
2017-03-04 23:26 ` Luca Ceresoli
2 siblings, 1 reply; 29+ messages in thread
From: Thomas De Schampheleire @ 2017-02-08 9:50 UTC (permalink / raw)
To: buildroot
On Tue, Feb 7, 2017 at 12:35 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> This commit adds the core of a new testing infrastructure that allows to
> perform runtime testing of Buildroot generated systems. This
> infrastructure uses the Python unittest logic as its foundation.
>
> This core infrastructure commit includes the following aspects:
>
> - A base test class, called BRTest, defined in
> support/testing/infra/basetest.py. This base test class inherited
> from the Python provided unittest.TestCase, and must be subclassed by
> all Buildroot test cases.
>
> Its main purpose is to provide the Python unittest setUp() and
> tearDown() methods. In our case, setUp() takes care of building the
> Buildroot system described in the test case, and instantiate the
> Emulator object in case runtime testing is needed. The tearDown()
> method simply cleans things up (stop the emulator, remove the output
> directory).
>
> - A Builder class, defined in support/testing/infra/builder.py, simply
> responsible for building the Buildroot system in each test case.
>
> - An Emulator class, defined in support/testing/infra/emulator.py,
> responsible for running the generated system under Qemu, allowing
> each test case to run arbitrary commands inside the emulated system.
>
> - A run-tests script, which is the entry point to start the tests.
>
> Even though I wrote the original version of this small infrastructure, a
> huge amount of rework and improvement has been done by Maxime
> Hadjinlian, and squashed into this patch. So many thanks to Maxime for
> cleaning up and improving my Python code!
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> support/testing/conf/unittest.cfg | 6 ++
> support/testing/infra/__init__.py | 61 ++++++++++++++++
> support/testing/infra/basetest.py | 66 +++++++++++++++++
> support/testing/infra/builder.py | 50 +++++++++++++
> support/testing/infra/emulator.py | 144 ++++++++++++++++++++++++++++++++++++++
> support/testing/run-tests | 77 ++++++++++++++++++++
> support/testing/tests/__init__.py | 0
> 7 files changed, 404 insertions(+)
> create mode 100644 support/testing/conf/unittest.cfg
> create mode 100644 support/testing/infra/__init__.py
> create mode 100644 support/testing/infra/basetest.py
> create mode 100644 support/testing/infra/builder.py
> create mode 100644 support/testing/infra/emulator.py
> create mode 100755 support/testing/run-tests
> create mode 100644 support/testing/tests/__init__.py
>
Is there a particular reason why you use unittest and nose2, rather
than the popular pytest ?
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-08 9:50 ` Thomas De Schampheleire
@ 2017-02-08 10:05 ` Thomas Petazzoni
2017-02-08 11:42 ` Thomas De Schampheleire
0 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-02-08 10:05 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 8 Feb 2017 10:50:13 +0100, Thomas De Schampheleire wrote:
> Is there a particular reason why you use unittest and nose2, rather
> than the popular pytest ?
First because I'm not a Pythonist, so I have no idea what is "popular"
in the Python world.
What lead me to use nose2 is because it is capable of executing tests
in parallel. In practice, this currently doesn't work due to how the
Qemu emulation is started (I use a fixed TCP port for the telnet
connection used for the serial port), but if we fix that, tests can be
executed in parallel, this is something I had tested when choosing
nose2.
Apparently, pytest can also run tests in parallel, according to
http://pytest.org/dev/xdist.html.
I honestly don't care which Python testing infrastructure is used, as
long as there is one. If you think pytest is better (and indeed its
website seems to be nicely documented and it seems to offer lots of
features), then patches are welcome :-)
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-08 10:05 ` Thomas Petazzoni
@ 2017-02-08 11:42 ` Thomas De Schampheleire
2017-03-04 18:12 ` Luca Ceresoli
0 siblings, 1 reply; 29+ messages in thread
From: Thomas De Schampheleire @ 2017-02-08 11:42 UTC (permalink / raw)
To: buildroot
On Wed, Feb 8, 2017 at 11:05 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Wed, 8 Feb 2017 10:50:13 +0100, Thomas De Schampheleire wrote:
>
>> Is there a particular reason why you use unittest and nose2, rather
>> than the popular pytest ?
>
> First because I'm not a Pythonist, so I have no idea what is "popular"
> in the Python world.
>
> What lead me to use nose2 is because it is capable of executing tests
> in parallel. In practice, this currently doesn't work due to how the
> Qemu emulation is started (I use a fixed TCP port for the telnet
> connection used for the serial port), but if we fix that, tests can be
> executed in parallel, this is something I had tested when choosing
> nose2.
>
> Apparently, pytest can also run tests in parallel, according to
> http://pytest.org/dev/xdist.html.
>
> I honestly don't care which Python testing infrastructure is used, as
> long as there is one. If you think pytest is better (and indeed its
> website seems to be nicely documented and it seems to offer lots of
> features), then patches are welcome :-)
>
We have switched from nose (1) to pytest in Kallithea a while ago.
We are not using xdist at the moment though, because our setup code is
not ready for it.
To switch to pytest now shouldn't be complex, it can off-the-shelve
continue to support unittest-style tests. However, pytest 'fixtures'
are the alternative to such setup/teardown methods and are more
powerful. But you don't _have_ to use them.
Anyway, I agree that it is not crucial at all, and the actual impact
of nose on the code is currently minimal, so switching can be done at
a later time.
/Thomas
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 0/5] Runtime testing infrastructure
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
` (4 preceding siblings ...)
2017-02-07 11:36 ` [Buildroot] [PATCH 5/5] support/testing: add toolchain tests Thomas Petazzoni
@ 2017-03-02 1:45 ` Ricardo Martincoski
2017-03-05 14:08 ` Thomas Petazzoni
5 siblings, 1 reply; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:45 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:35 AM, Thomas Petazzoni wrote:
> At last, here is a first version of a small runtime testing
> infrastructure for Buildroot. The first patch introduces the
> infrastructure itself, the other 4 patches just add a few patches.
Thank you very much for this series.
In overall, the infra works very well.
I tested it and I will reply to each patch.
Here is a summary of what I found:
1) emulator supports only arm, making test_iso9660 testcases to fail;
2) testcases using i386 emulated system can fail occasionally due to a message
from the kernel;
3) the testcases *Internal from test_iso9660 expect the rootfs to be read-only,
I am not sure this is correct or not, but it makes the tests fail and it is
different from your previous iteration (before the stylistic rework);
4) output dir needs to be an absolute path;
5) there are some emulator.login("root") that I think should be .login()
because BR2_TARGET_GENERIC_ROOT_PASSWD it not set, except for dropbear;
6) using "-q" with commands to the emulated system is unnecessary and make
debug harder when a testcase fails;
7) the code is not compatible to python 3
- print without parenthesis
- exception handling using comma
- importing urllib2 (split in libraries in python3)
- probably will need .encode() and .decode() when using telnetlib because of
the bytes vs. string introduced in python 3
This is what I think about this list:
(1) and (2) should be fixed;
(3) should be checked and fixed if it is not correct;
(4), (5) and (6) can wait;
(7) should *not* be fixed now, but in a followup patch (it can be postponed
until 2020, when python 2 probably will retire);
> ./support/testing/run-tests -d ~/dl/ -o ../outputs/ tests.fs.test_ubi.TestUbi
Using relative path for output dir seems to cause problems with few testcases,
as tests.core.test_post_scripts.TestPostScripts .
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
2017-02-08 9:50 ` Thomas De Schampheleire
@ 2017-03-02 1:46 ` Ricardo Martincoski
2017-03-05 14:09 ` Thomas Petazzoni
2017-03-04 23:26 ` Luca Ceresoli
2 siblings, 1 reply; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:46 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:35 AM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/infra/emulator.py
[snip]
> + def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
> + if arch in ["armv7", "armv5"]:
> + qemu_arch = "arm"
Support for arch="i386" from test_iso9660 is missing here:
else:
qemu_arch = arch
ERROR: test_run (tests.fs.test_iso9660.TestIso9660SyslinuxInternal)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v1/support/testing/tests/fs/test_iso9660.py", line 157, in test_run
self.builddir, internal=True)
File "/tmp/testing-v1/support/testing/tests/fs/test_iso9660.py", line 30, in test_mount_internal_external
emulator.boot(arch="i386", options=["-cdrom", img])
File "/tmp/testing-v1/support/testing/infra/emulator.py", line 39, in boot
qemu_cmd = ["qemu-system-{}".format(qemu_arch),
UnboundLocalError: local variable 'qemu_arch' referenced before assignment
> +
> + qemu_cmd = ["qemu-system-{}".format(qemu_arch),
> + "-serial", "telnet::1234,server",
> + "-display", "none"]
[snip]
> +++ b/support/testing/run-tests
> @@ -0,0 +1,77 @@
> +#!/usr/bin/env python
Maybe we can do for now as scanpypi does:
#!/usr/bin/python2
[snip]
> +import string
Not used.
[snip]
> + if not os.path.exists(args.output):
> + print "The selected output directory does not exist"
> + return 1
> +
> + BRTest.outputdir = args.output
Maybe changing this line to:
BRTest.outputdir = os.path.abspath(args.output)
would fix the use of relative path with some testcases.
$ support/testing/run-tests -d dl -o $(readlink -f output) \
tests.core.test_post_scripts.TestPostScripts
...
OK
$ support/testing/run-tests -d dl -o output \
tests.core.test_post_scripts.TestPostScripts
...
FAIL: test_run (tests.core.test_post_scripts.TestPostScripts)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v1/support/testing/tests/core/test_post_scripts.py", line 34, in test_run
self.check_post_log_file(f, "target")
File "/tmp/testing-v1/support/testing/tests/core/test_post_scripts.py", line 21, in check_post_log_file
self.assertEqual(lines[0], os.path.join(self.builddir, what))
AssertionError: '/tmp/testing-v1/output/TestPostScripts/target' != 'output/TestPostScripts/target'
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 2/5] support/testing: add core tests
2017-02-07 11:36 ` [Buildroot] [PATCH 2/5] support/testing: add core tests Thomas Petazzoni
@ 2017-03-02 1:47 ` Ricardo Martincoski
2017-03-05 14:11 ` Thomas Petazzoni
0 siblings, 1 reply; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:47 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/tests/core/test_post_scripts.py
[snip]
> + # TODO
This comment can be removed. The method is already implemented.
> + def check_post_log_file(self, path, what):
> + with open(path, "r") as f:
> + lines = f.read().splitlines()
> + self.assertEqual(lines[0], os.path.join(self.builddir, what))
> + self.assertEqual(lines[1], "foobar")
> + self.assertEqual(lines[2], "baz")
> + self.assertEqual(lines[3], os.path.join(self.builddir, "target"))
> + self.assertEqual(lines[4], os.path.join(self.builddir, "build"))
> + self.assertEqual(lines[5], os.path.join(self.builddir, "host"))
> + staging = os.readlink(os.path.join(self.builddir, "staging"))
> + self.assertEqual(lines[6], staging)
> + self.assertEqual(lines[7], os.path.join(self.builddir, "images"))
> + self.assertEqual(lines[8], os.path.join(self.builddir, ".config"))
> +
[snip]
> +++ b/support/testing/tests/core/test_timezone.py
> @@ -0,0 +1,66 @@
> +import os
> +
> +import infra.basetest
> +
> +def boot_armv5_cpio(emulator, builddir):
> + img = os.path.join(builddir, "images", "rootfs.cpio")
> + emulator.boot(arch="armv5", kernel="builtin",
> + options=["-initrd", img])
> + emulator.login("root")
Shouldn't it be like this?
emulator.login()
> +
> +class TestNoTimezone(infra.basetest.BRTest):
> + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> + """
> +# BR2_TARGET_TZ_INFO is not set
> +BR2_TARGET_ROOTFS_CPIO=y
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 3/5] support/testing: add fs tests
2017-02-07 11:36 ` [Buildroot] [PATCH 3/5] support/testing: add fs tests Thomas Petazzoni
@ 2017-03-02 1:51 ` Ricardo Martincoski
2017-03-05 14:13 ` Thomas Petazzoni
0 siblings, 1 reply; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:51 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/conf/minimal-x86-qemu-kernel.config
[snip]
I observed sometimes a testcase fails when it shouldn't.
This emulated system displays some time after boot this message:
Switched to clocksource tsc
When the testcase is small there is a chance the message arrives when 'echo $?'
is being issued.
I took a look at the kernel and I didn't find compile-time configs to avoid
this message.
There are some options for bootargs that maybe could help:
clocksource=tsc
loglevel=0
quiet
But qemu can only use -append when -kernel is used too.
The only config I found that could workaround the error is
# CONFIG_PRINTK is not set
of course it is a big change, and it would need CONFIG_EXPERT=y and it would
also change few more lines in the defconfig.
But I will be happy with any other solution.
Here is how I reproduce the error, but I think its occurrence will depend a lot
on the host machine.
running (after fixing the arch in emulator and also changing the expected
return code to 0):
$ for i in $(seq 1 100); do echo $i ; support/testing/run-tests -d dl \
-o $(readlink -f output) -k tests.fs.test_iso9660.TestIso9660GrubInternal \
|| break; done
eventually leads to:
======================================================================
ERROR: test_run (tests.fs.test_iso9660.TestIso9660GrubInternal)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v1/support/testing/tests/fs/test_iso9660.py", line 120, in test_run
exit_code = test_touch_file(self.emulator)
File "/tmp/testing-v1/support/testing/tests/fs/test_iso9660.py", line 41, in test_touch_file
_, exit_code = emulator.run("touch test")
File "/tmp/testing-v1/support/testing/infra/emulator.py", line 131, in run
exit_code = int(exit_code)
ValueError: invalid literal for int() with base 10: 'echo $?'
----------------------------------------------------------------------
Ran 1 test in 2.789s
log when the test is OK (most of times)
---
mount | grep 'rootfs on / type rootfs'
rootfs on / type rootfs (rw)
# echo $?
0
# > command terminated, status 0
> running 'touch test'
touch test
# echo $?
0
# > command terminated, status 0
---
log when the test FAILS
---
rootfs on / type rootfs (rw)
# echo $?
0
# > command terminated, status 0
> running 'touch test'
touch test
# Switched to clocksource tsc
echo $?
0
#
---
[snip]
> +++ b/support/testing/tests/fs/test_iso9660.py
[snip]
> + emulator.boot(arch="i386", options=["-cdrom", img])
See my comment in patch 1.
[snip]
> +def test_touch_file(emulator):
> + _, exit_code = emulator.run("touch test")
> + return exit_code
[snip]
> +class TestIso9660Grub2Internal(infra.basetest.BRTest):
[snip]
> + exit_code = test_touch_file(self.emulator)
> + self.assertEqual(exit_code, 1)
Same comment as in TestIso9660SyslinuxInternal below.
> +
> +class TestIso9660GrubInternal(infra.basetest.BRTest):
[snip]
> + exit_code = test_touch_file(self.emulator)
> + self.assertEqual(exit_code, 1)
Same comment as in TestIso9660SyslinuxInternal below.
[snip]
> +class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
[snip]
> + exit_code = test_touch_file(self.emulator)
> + self.assertEqual(exit_code, 1)
> +
> +class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
[snip]
> + exit_code = test_touch_file(self.emulator)
> + self.assertEqual(exit_code, 1)
This testcase fails for me like this (after fixing arch in emulator):
======================================================================
FAIL: test_run (tests.fs.test_iso9660.TestIso9660SyslinuxInternal)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v1/support/testing/tests/fs/test_iso9660.py", line 161, in test_run
self.assertEqual(exit_code, 1)
AssertionError: 0 != 1
I am not sure if in this case the rootfs should be read-only or not, but this
differs from your previous iteration:
def run_external(self):
[snip]
(r, s) = self.s.run("touch test")
self.assertEqual(s, 1)
def run_internal(self):
[snip]
(r, s) = self.s.run("touch test")
self.assertEqual(s, 0)
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 4/5] support/testing: add package tests
2017-02-07 11:36 ` [Buildroot] [PATCH 4/5] support/testing: add package tests Thomas Petazzoni
@ 2017-03-02 1:53 ` Ricardo Martincoski
2017-03-05 14:14 ` Thomas Petazzoni
0 siblings, 1 reply; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:53 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/tests/package/test_dropbear.py
[snip]
> + cmd = "netstat -ltn 2>/dev/null | grep -q 0.0.0.0:22"
You could drop the '-q' here.
Since the log is sent to a file it will not bother the user.
And when a testcase fails a verbose log helps to locate the problem.
Also it will be consistent to the use of 'grep' in all other testcases.
[snip]
> +++ b/support/testing/tests/package/test_python.py
[snip]
> + cmd = "python --version 2>&1 | grep -q '^Python 2'"
[snip]
Same here.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 5/5] support/testing: add toolchain tests
2017-02-07 11:36 ` [Buildroot] [PATCH 5/5] support/testing: add toolchain tests Thomas Petazzoni
@ 2017-03-02 1:54 ` Ricardo Martincoski
0 siblings, 0 replies; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-02 1:54 UTC (permalink / raw)
To: buildroot
Thomas,
On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
[snip]
> +++ b/support/testing/tests/toolchain/test_external.py
> @@ -0,0 +1,77 @@
> +import os
> +
> +import infra
> +
> +class TestExternalToolchainSourceryArmv4(infra.basetest.BRTest):
> + config = \
> +"""
> +BR2_arm=y
> +BR2_arm920t=y
> +BR2_TOOLCHAIN_EXTERNAL=y
> +BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
> +BR2_TARGET_ROOTFS_CPIO=y
> +# BR2_TARGET_ROOTFS_TAR is not set
> +"""
[snip]
> + self.emulator.login("root")
Shouldn't it be like this?
self.emulator.login()
> +
> +class TestExternalToolchainSourceryArmv5(infra.basetest.BRTest):
[snip]
> + self.emulator.login("root")
Same here.
> +
> +class TestExternalToolchainSourceryArmv7(infra.basetest.BRTest):
[snip]
> + self.emulator.login("root")
Same here.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-08 11:42 ` Thomas De Schampheleire
@ 2017-03-04 18:12 ` Luca Ceresoli
2017-03-05 7:27 ` Thomas De Schampheleire
2017-03-05 8:29 ` Thomas Petazzoni
0 siblings, 2 replies; 29+ messages in thread
From: Luca Ceresoli @ 2017-03-04 18:12 UTC (permalink / raw)
To: buildroot
Hi,
On 08/02/2017 12:42, Thomas De Schampheleire wrote:
> On Wed, Feb 8, 2017 at 11:05 AM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Hello,
>>
>> On Wed, 8 Feb 2017 10:50:13 +0100, Thomas De Schampheleire wrote:
>>
>>> Is there a particular reason why you use unittest and nose2, rather
>>> than the popular pytest ?
>>
>> First because I'm not a Pythonist, so I have no idea what is "popular"
>> in the Python world.
>>
>> What lead me to use nose2 is because it is capable of executing tests
>> in parallel. In practice, this currently doesn't work due to how the
>> Qemu emulation is started (I use a fixed TCP port for the telnet
>> connection used for the serial port), but if we fix that, tests can be
>> executed in parallel, this is something I had tested when choosing
>> nose2.
>>
>> Apparently, pytest can also run tests in parallel, according to
>> http://pytest.org/dev/xdist.html.
>>
>> I honestly don't care which Python testing infrastructure is used, as
>> long as there is one. If you think pytest is better (and indeed its
>> website seems to be nicely documented and it seems to offer lots of
>> features), then patches are welcome :-)
>>
>
> We have switched from nose (1) to pytest in Kallithea a while ago.
> We are not using xdist at the moment though, because our setup code is
> not ready for it.
>
> To switch to pytest now shouldn't be complex, it can off-the-shelve
> continue to support unittest-style tests. However, pytest 'fixtures'
> are the alternative to such setup/teardown methods and are more
> powerful. But you don't _have_ to use them.
>
> Anyway, I agree that it is not crucial at all, and the actual impact
> of nose on the code is currently minimal, so switching can be done at
> a later time.
I'm no pythonist at all, so I don't know any of the mentioned
infrastructures. But one thing I dislike in the current implementation
is the unpleasant way failures are reported.
ThomasDS, MaximeH, Do you think another infrastructure allows better
reporting? Or that nose2 allows it with small changes?
--
Luca
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
2017-02-08 9:50 ` Thomas De Schampheleire
2017-03-02 1:46 ` Ricardo Martincoski
@ 2017-03-04 23:26 ` Luca Ceresoli
2017-03-05 14:44 ` Thomas Petazzoni
2 siblings, 1 reply; 29+ messages in thread
From: Luca Ceresoli @ 2017-03-04 23:26 UTC (permalink / raw)
To: buildroot
Hi,
On 07/02/2017 12:35, Thomas Petazzoni wrote:
> This commit adds the core of a new testing infrastructure that allows to
> perform runtime testing of Buildroot generated systems. This
> infrastructure uses the Python unittest logic as its foundation.
>
> This core infrastructure commit includes the following aspects:
>
> - A base test class, called BRTest, defined in
> support/testing/infra/basetest.py. This base test class inherited
> from the Python provided unittest.TestCase, and must be subclassed by
> all Buildroot test cases.
>
> Its main purpose is to provide the Python unittest setUp() and
> tearDown() methods. In our case, setUp() takes care of building the
> Buildroot system described in the test case, and instantiate the
> Emulator object in case runtime testing is needed. The tearDown()
> method simply cleans things up (stop the emulator, remove the output
> directory).
>
> - A Builder class, defined in support/testing/infra/builder.py, simply
> responsible for building the Buildroot system in each test case.
>
> - An Emulator class, defined in support/testing/infra/emulator.py,
> responsible for running the generated system under Qemu, allowing
> each test case to run arbitrary commands inside the emulated system.
>
> - A run-tests script, which is the entry point to start the tests.
>
> Even though I wrote the original version of this small infrastructure, a
> huge amount of rework and improvement has been done by Maxime
> Hadjinlian, and squashed into this patch. So many thanks to Maxime for
> cleaning up and improving my Python code!
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
I finally had a closer look to this work, and generally speaking the
infrastructure quite straightforward. I have several remarks (see
below), but most of them are minor, coding style and the like.
Disclaimer: I know python only at a very very basic level.
And since it's likely that several Buildroot users have little python
experience, I think documenting the code here is more important than in
most make and kconfig code. So I propose a few lines of documentation
below. I'd propose more, but for that I'll wait for a consensus on the
general approach.
The only thing I don't like very much is the output format. It's very
unpleasant and I'd like it to be nicer. But since we are using a testing
framework I guess this is non trivial. In this case I'll gladly accept
what we have: runtime testing with unpleasant output is _a lot_ better
that no runtime testing.
> +++ b/support/testing/infra/__init__.py
> @@ -0,0 +1,61 @@
> +import contextlib
> +import os
> +import sys
> +import tempfile
> +import subprocess
> +from urllib2 import urlopen, HTTPError, URLError
> +
> +ARTEFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
I think the usual spelling is "artifacts".
> + at contextlib.contextmanager
> +def smart_open(filename=None):
> + if filename and filename != '-':
> + fhandle = open(filename, 'a+')
> + else:
> + fhandle = sys.stdout
> +
> + try:
> + yield fhandle
> + finally:
> + if fhandle is not sys.stdout:
> + fhandle.close()
contextmanager and yield took me some time and research to understand
what this function is. For non-pythonists, I'd add a comment such as:
"""
Return a file-like object that can be written to using the 'with'
keyword, as in the example:
with infra.smart_open("test.log") as outfile:
outfile.write("Hello, world!\n")
"""
Also, since it creates a writable thing (not a readable one), a name
like open_for_writing() would be more understandable.
> +def filepath(relpath):
> + return os.path.join(os.getcwd(), "support/testing", relpath)
> +
> +def download(dldir, filename):
> + finalpath = os.path.join(dldir, filename)
> + if os.path.exists(finalpath):
> + return finalpath
> +
> + if not os.path.exists(dldir):
> + os.makedirs(dldir)
> +
> + tmpfile = tempfile.mktemp(dir=dldir)
> + print "Downloading to {}".format(tmpfile)
> +
> + try:
> + url_fh = urlopen(os.path.join(ARTEFACTS_URL, filename))
> + with open(tmpfile, "w+") as tmpfile_fh:
> + tmpfile_fh.write(url_fh.read())
I think this code read()s the whole file in RAM, then writes it all out.
This might be a problem if we download large files. Is there any
pythonic way to read-and-write in chunks without much added code? Come
on pythonists, I'm sure you can do it with at most one extra line... :)
> +def get_elf_arch_tag(builddir, prefix, fpath, tag):
> + cmd = ["host/usr/bin/{}-readelf".format(prefix),
> + "-A", os.path.join("target", fpath)]
> + out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
> + for line in out.splitlines():
> + line = line.strip()
> + if not line.startswith(tag):
> + continue
> + return line.split(":")[1].strip()
> + return None
A small example as documentation would help. Draft proposal:
"""
Runs the cross readelf on 'fpath', then extracts the value of tag 'tag'.
Example:
>>> get_elf_arch_tag('output', 'arm-none-linux-gnueabi-',
'bin/busybox', 'Tag_CPU_arch')
v5TEJ
>>>
"""
> diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
> new file mode 100644
> index 0000000..4f72e2f
> --- /dev/null
> +++ b/support/testing/infra/builder.py
> @@ -0,0 +1,50 @@
> +import os
> +import shutil
> +import subprocess
> +
> +import infra
> +
> +class Builder(object):
> + def __init__(self, config, builddir, logtofile):
> + self.config = config
> + self.builddir = builddir
> + self.logtofile = logtofile
> +
> + def build(self):
> + if not os.path.isdir(self.builddir):
> + os.makedirs(self.builddir)
> +
> + log = "{}-build.log".format(self.builddir)
> + if self.logtofile is None:
> + log = None
I think there's a bug here. The 'logtofile' variable in BRTest is always
assigned to True or False, then passed to Builder. So self.logfile here
is never None, and this if never matches.
> +
> + config_file = os.path.join(self.builddir, ".config")
> + with open(config_file, "w+") as cfd:
> + cfd.write(self.config)
I wonder what the 'd' in "cfd" means...
> + cmd = ["make",
> + "-C", os.getcwd(),
> + "O={}".format(self.builddir),
> + "olddefconfig"]
Why do we need to add "-C" + os.getcwd()? I think this is equivalent to
"-C ." in shell, which is the default make behaviour.
> +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss
> +# with the qemu machine.
> +class Emulator(object):
> +
> + def __init__(self, builddir, downloaddir, logtofile):
> + self.qemu = None
> + self.__tn = None
> + self.downloaddir = downloaddir
> + self.log = ""
> + self.log_file = "{}-run.log".format(builddir)
> + if logtofile is None:
> + self.log_file = None
> +
> + # Start Qemu to boot the system
> + #
> + # arch: Qemu architecture to use
> + #
> + # kernel: path to the kernel image, or the special string
> + # 'builtin' in which case a pre-built kernel image will be used
> + # (so far only armv5 and armv7 kernels are available). If None,
> + # then no kernel is used, and we assume a bootable device will be
> + # specified.
When first reading this, I understood that the 'builtin' kernels are
provided by qemu. Instead, they are provided by some server that is
related to the Buildroot project (currently free-electrons.com).
So I'd rephrase as:
# kernel: path to the kernel image, or the special string
# 'builtin'.
# 'builtin' means a pre-built kernel image will be
# downloaded from <BASEURL> and suitable options are automatically
# passed to qemu and added to the kernel cmdline. So far only armv5
# and armv7 builtin kernels are available.
# If None, then no kernel is used, and we assume a bootable device
# will be specified.
> + #
> + # kernel_cmdline: array of kernel arguments to pass to Qemu -append option
> + #
> + # options: array of command line options to pass to Qemu
> + #
> + def boot(self, arch, kernel=None, kernel_cmdline=None, options=None):
> + if arch in ["armv7", "armv5"]:
> + qemu_arch = "arm"
> +
> + qemu_cmd = ["qemu-system-{}".format(qemu_arch),
> + "-serial", "telnet::1234,server",
> + "-display", "none"]
> +
> + if options:
> + qemu_cmd += options
If the default value for 'options' were '[]', the if would not be
needed. Not it is because 'None' is not iterable.
> +
> + if kernel_cmdline is None:
> + kernel_cmdline = []
If the default value for 'kernel_cmdline' were '[]', these two lines
could be removed entrely.
> + # Wait for the login prompt to appear, and then login as root with
> + # the provided password, or no password if not specified.
> + def login(self, password=None):
> + with infra.smart_open(self.log_file) as lfh:
> + lfh.write("> waiting for login\n")
> +
> + self.__read_until("buildroot login:", 10)
> + if "buildroot login:" not in self.log:
> + with infra.smart_open(self.log_file) as lfh:
> + lfh.write("==> System does not boot")
> + raise SystemError("System does not boot")
> +
> + with infra.smart_open(self.log_file) as lfh:
> + lfh.write("> log in\n")
smart_open() is called 3 times in login(), 2 in run() and a few more
timer elsewhere in this class. Can't we open the file once in the
constructor and the just write to it?
> + # Run the given 'cmd' on the target
> + # return a tuple (output, exit_code)
> + def run(self, cmd):
> + with infra.smart_open(self.log_file) as lfh:
> + lfh.write("> running '{}'\n".format(cmd))
> +
> + self.__write("{}\n".format(cmd))
I think this simpler form is equally legal:
self.__write(cmd + "\n")
> + def showlog(self):
> + print "=== Full log ==="
> + print self.log
> + print "================"
Unused function.
> +def main():
> + parser = argparse.ArgumentParser(description='Run Buildroot tests')
[...]
> + if args.download is None:
> + print "Missing download directory, please use -d/--download"
> + return 1
It would be nice to use an environment variable as a default for the
download dir. It might be BR2_DL_DIR itself, i.e. sharing the existing
Buildroot package download dir. I bet most people set BR2_DL_DIR in
their ~/.bashrc.
> + if not os.path.exists(args.output):
> + print "The selected output directory does not exist"
> + return 1
If the download directory does not exist is is created. Why is the
output directory handled differently?
And again, it would be nice to default to an environment variable, e.g.
BR2_RUNTIME_TESTS_OUTPUT_DIR.
> + BRTest.outputdir = args.output
> +
> + if args.all is False and len(args.testname) == 0:
> + print "No test selected"
> + return 1
I like programs that print their usage whenever the command line has
errors (in addition to telling what the error is).
IOW (untested):
if args.all is False and len(args.testname) == 0:
print "No test selected"
+ print ""
+ parser.print_help()
return 1
--
Luca
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-04 18:12 ` Luca Ceresoli
@ 2017-03-05 7:27 ` Thomas De Schampheleire
2017-03-05 8:29 ` Thomas Petazzoni
1 sibling, 0 replies; 29+ messages in thread
From: Thomas De Schampheleire @ 2017-03-05 7:27 UTC (permalink / raw)
To: buildroot
On Mar 4, 2017 19:12, "Luca Ceresoli" <luca@lucaceresoli.net> wrote:
Hi,
On 08/02/2017 12:42, Thomas De Schampheleire wrote:
> On Wed, Feb 8, 2017 at 11:05 AM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Hello,
>>
>> On Wed, 8 Feb 2017 10:50:13 +0100, Thomas De Schampheleire wrote:
>>
>>> Is there a particular reason why you use unittest and nose2, rather
>>> than the popular pytest ?
>>
>> First because I'm not a Pythonist, so I have no idea what is "popular"
>> in the Python world.
>>
>> What lead me to use nose2 is because it is capable of executing tests
>> in parallel. In practice, this currently doesn't work due to how the
>> Qemu emulation is started (I use a fixed TCP port for the telnet
>> connection used for the serial port), but if we fix that, tests can be
>> executed in parallel, this is something I had tested when choosing
>> nose2.
>>
>> Apparently, pytest can also run tests in parallel, according to
>> http://pytest.org/dev/xdist.html.
>>
>> I honestly don't care which Python testing infrastructure is used, as
>> long as there is one. If you think pytest is better (and indeed its
>> website seems to be nicely documented and it seems to offer lots of
>> features), then patches are welcome :-)
>>
>
> We have switched from nose (1) to pytest in Kallithea a while ago.
> We are not using xdist at the moment though, because our setup code is
> not ready for it.
>
> To switch to pytest now shouldn't be complex, it can off-the-shelve
> continue to support unittest-style tests. However, pytest 'fixtures'
> are the alternative to such setup/teardown methods and are more
> powerful. But you don't _have_ to use them.
>
> Anyway, I agree that it is not crucial at all, and the actual impact
> of nose on the code is currently minimal, so switching can be done at
> a later time.
I'm no pythonist at all, so I don't know any of the mentioned
infrastructures. But one thing I dislike in the current implementation
is the unpleasant way failures are reported.
ThomasDS, MaximeH, Do you think another infrastructure allows better
reporting? Or that nose2 allows it with small changes?
Could you give some example of the output you get now?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170305/f0cd0c07/attachment.html>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-04 18:12 ` Luca Ceresoli
2017-03-05 7:27 ` Thomas De Schampheleire
@ 2017-03-05 8:29 ` Thomas Petazzoni
2017-03-07 21:59 ` Luca Ceresoli
1 sibling, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 8:29 UTC (permalink / raw)
To: buildroot
Hello,
On Sat, 4 Mar 2017 19:12:44 +0100, Luca Ceresoli wrote:
> I'm no pythonist at all, so I don't know any of the mentioned
> infrastructures. But one thing I dislike in the current implementation
> is the unpleasant way failures are reported.
Are you talking about:
* Failures because there is a syntax error in the test script
* Failures of the test itself, because the generated Buildroot system
doesn't behave as expected
To be honest, I don't think the unittest stuff is bringing us that much
stuff, so we could just as well roll our own "test runner", giving us
more freedom in our errors are handled and reported.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 0/5] Runtime testing infrastructure
2017-03-02 1:45 ` [Buildroot] [PATCH 0/5] Runtime testing infrastructure Ricardo Martincoski
@ 2017-03-05 14:08 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:08 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 01 Mar 2017 22:45:00 -0300, Ricardo Martincoski wrote:
> On Tue, Feb 07, 2017 at 09:35 AM, Thomas Petazzoni wrote:
>
> > At last, here is a first version of a small runtime testing
> > infrastructure for Buildroot. The first patch introduces the
> > infrastructure itself, the other 4 patches just add a few patches.
>
> Thank you very much for this series.
And thanks a lot to you for taking the time to test and review it! I'll
reply to all your comments in the individual patches.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-02 1:46 ` Ricardo Martincoski
@ 2017-03-05 14:09 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:09 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 01 Mar 2017 22:46:17 -0300, Ricardo Martincoski wrote:
> Support for arch="i386" from test_iso9660 is missing here:
> else:
> qemu_arch = arch
Re-added this. I guess it had been lost during one of the cleanup
sessions. I've retested one of the iso9660 test cases, and it now
passes correctly.
> Maybe we can do for now as scanpypi does:
> #!/usr/bin/python2
Fixed.
> > +import string
>
> Not used.
Ditto.
> > + if not os.path.exists(args.output):
> > + print "The selected output directory does not exist"
> > + return 1
> > +
> > + BRTest.outputdir = args.output
>
> Maybe changing this line to:
> BRTest.outputdir = os.path.abspath(args.output)
> would fix the use of relative path with some testcases.
Indeed, this fixes the problem you mentioned.
All of this will be in v2. Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 2/5] support/testing: add core tests
2017-03-02 1:47 ` Ricardo Martincoski
@ 2017-03-05 14:11 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:11 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 01 Mar 2017 22:47:33 -0300, Ricardo Martincoski wrote:
> Thomas,
>
> On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
>
> [snip]
> > +++ b/support/testing/tests/core/test_post_scripts.py
> [snip]
> > + # TODO
>
> This comment can be removed. The method is already implemented.
Indeed.
> > + emulator.login("root")
>
> Shouldn't it be like this?
> emulator.login()
Correct. I did the mistake in a number of places. I guess the fact that
I chose "root" as the root password for the Dropbear test confused me
into thinking that the argument was the user name, and not the
password. So I've also changed the root password of the Dropbear test
case to be "testpwd" instead.
I've fixed all instances of emulator.login("root") that should have
been emulator.login().
While doing this, I also noticed that the emulator.login() method did
not ensure we reached the shell prompt. So I've changed it to make sure
we reached the prompt, and if not, abort.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 3/5] support/testing: add fs tests
2017-03-02 1:51 ` Ricardo Martincoski
@ 2017-03-05 14:13 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:13 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 01 Mar 2017 22:51:31 -0300, Ricardo Martincoski wrote:
> I observed sometimes a testcase fails when it shouldn't.
>
> This emulated system displays some time after boot this message:
> Switched to clocksource tsc
>
> When the testcase is small there is a chance the message arrives when 'echo $?'
> is being issued.
>
> I took a look at the kernel and I didn't find compile-time configs to avoid
> this message.
> There are some options for bootargs that maybe could help:
> clocksource=tsc
> loglevel=0
> quiet
> But qemu can only use -append when -kernel is used too.
>
> The only config I found that could workaround the error is
> # CONFIG_PRINTK is not set
> of course it is a big change, and it would need CONFIG_EXPERT=y and it would
> also change few more lines in the defconfig.
>
> But I will be happy with any other solution.
Thanks for having debugged this issue! I was unable to reproduce on my
side, even for a 100 iterations loop. However, I've fixed it by running
"dmesg -n 1" as the first command after log in. This tells the kernel
to only display emergency kernel messages to the console, while the
"Switching to clocksource tsc" message has the info level, which is a
lower priority.
> I am not sure if in this case the rootfs should be read-only or not, but this
> differs from your previous iteration:
That's a mistake that was introduced while refactoring/cleaning up the
whole thing. Indeed in the "internal" case, we're running in an
initramfs, so the rootfs is read/write, and the "touch" is expected to
succeed.
Fixed as well, thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 4/5] support/testing: add package tests
2017-03-02 1:53 ` Ricardo Martincoski
@ 2017-03-05 14:14 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:14 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 01 Mar 2017 22:53:08 -0300, Ricardo Martincoski wrote:
> On Tue, Feb 07, 2017 at 09:36 AM, Thomas Petazzoni wrote:
>
> [snip]
> > +++ b/support/testing/tests/package/test_dropbear.py
> [snip]
> > + cmd = "netstat -ltn 2>/dev/null | grep -q 0.0.0.0:22"
>
> You could drop the '-q' here.
> Since the log is sent to a file it will not bother the user.
> And when a testcase fails a verbose log helps to locate the problem.
>
> Also it will be consistent to the use of 'grep' in all other testcases.
>
> [snip]
> > +++ b/support/testing/tests/package/test_python.py
> [snip]
> > + cmd = "python --version 2>&1 | grep -q '^Python 2'"
> [snip]
>
> Same here.
Agreed, I've fixed both.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-04 23:26 ` Luca Ceresoli
@ 2017-03-05 14:44 ` Thomas Petazzoni
0 siblings, 0 replies; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-05 14:44 UTC (permalink / raw)
To: buildroot
Hello Luca,
Thanks a lot for your review!
On Sun, 5 Mar 2017 00:26:56 +0100, Luca Ceresoli wrote:
> The only thing I don't like very much is the output format. It's very
> unpleasant and I'd like it to be nicer. But since we are using a testing
> framework I guess this is non trivial. In this case I'll gladly accept
> what we have: runtime testing with unpleasant output is _a lot_ better
> that no runtime testing.
Agreed.
> > +ARTEFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
>
> I think the usual spelling is "artifacts".
artifact is US, artefact is UK apparently. So I've changed to artifact,
like you suggested :)
> > + at contextlib.contextmanager
> > +def smart_open(filename=None):
> > + if filename and filename != '-':
> > + fhandle = open(filename, 'a+')
> > + else:
> > + fhandle = sys.stdout
> > +
> > + try:
> > + yield fhandle
> > + finally:
> > + if fhandle is not sys.stdout:
> > + fhandle.close()
>
> contextmanager and yield took me some time and research to understand
> what this function is. For non-pythonists, I'd add a comment such as:
>
> """
> Return a file-like object that can be written to using the 'with'
> keyword, as in the example:
> with infra.smart_open("test.log") as outfile:
> outfile.write("Hello, world!\n")
> """
Added. But to be honest, I dislike this Pythonic construct. Originally,
I was opening the log file at the beginning of the test, keeping the
file handle in an object member, and simply writing to it (exactly like
you suggest below).
But then Maxime did a pass on my code to make it more Pythonic, and
replaced it with this construct. To me, this is a lot more horrible
than it was:
- It's not understandable by non-Python experts
- Each write to the log takes two lines instead of just one previously
- It opens and closes the log file each time you want to write a
single line to it
So overall, I'm not at all impressed by this, and seeing that you are
also not impressed makes me considering reverting to my previous
solution.
> Also, since it creates a writable thing (not a readable one), a name
> like open_for_writing() would be more understandable.
Makes the name a bit too long IMO, and I'm not sure it's really worth
the effort.
> > + try:
> > + url_fh = urlopen(os.path.join(ARTEFACTS_URL, filename))
> > + with open(tmpfile, "w+") as tmpfile_fh:
> > + tmpfile_fh.write(url_fh.read())
>
> I think this code read()s the whole file in RAM, then writes it all out.
> This might be a problem if we download large files. Is there any
> pythonic way to read-and-write in chunks without much added code? Come
> on pythonists, I'm sure you can do it with at most one extra line... :)
urllib.retrieve() automatically does the download, but apparently,
there's no feedback on whether it was successful or not.
Anyway, I'll leave this for the future, the current code works fine,
and it can be improved by Pythonists later.
> A small example as documentation would help. Draft proposal:
>
> """
> Runs the cross readelf on 'fpath', then extracts the value of tag 'tag'.
> Example:
> >>> get_elf_arch_tag('output', 'arm-none-linux-gnueabi-',
> 'bin/busybox', 'Tag_CPU_arch')
> v5TEJ
> >>>
> """
Added.
> > + log = "{}-build.log".format(self.builddir)
> > + if self.logtofile is None:
> > + log = None
>
> I think there's a bug here. The 'logtofile' variable in BRTest is always
> assigned to True or False, then passed to Builder. So self.logfile here
> is never None, and this if never matches.
Agreed, fixed.
> > +
> > + config_file = os.path.join(self.builddir, ".config")
> > + with open(config_file, "w+") as cfd:
> > + cfd.write(self.config)
>
> I wonder what the 'd' in "cfd" means...
Don't remember. Maybe "descriptor". But I've changed that to just "cf"
anyway.
> > + cmd = ["make",
> > + "-C", os.getcwd(),
> > + "O={}".format(self.builddir),
> > + "olddefconfig"]
>
> Why do we need to add "-C" + os.getcwd()? I think this is equivalent to
> "-C ." in shell, which is the default make behaviour.
Don't remember why this was added, I've dropped it now.
> When first reading this, I understood that the 'builtin' kernels are
> provided by qemu. Instead, they are provided by some server that is
> related to the Buildroot project (currently free-electrons.com).
>
> So I'd rephrase as:
>
> # kernel: path to the kernel image, or the special string
> # 'builtin'.
> # 'builtin' means a pre-built kernel image will be
> # downloaded from <BASEURL> and suitable options are automatically
> # passed to qemu and added to the kernel cmdline. So far only armv5
> # and armv7 builtin kernels are available.
> # If None, then no kernel is used, and we assume a bootable device
> # will be specified.
Fixed. Probably all comments should be moved to Python docstrings at
some point.
> > + if options:
> > + qemu_cmd += options
>
> If the default value for 'options' were '[]', the if would not be
> needed. Not it is because 'None' is not iterable.
>
> > +
> > + if kernel_cmdline is None:
> > + kernel_cmdline = []
>
> If the default value for 'kernel_cmdline' were '[]', these two lines
> could be removed entrely.
To be honest, for the kernel_cmdline, it took me a while to get the
proper construct that ends up with the quotes at the right place. So
I'd prefer to keep it as-is for now, and if you want to clean that up
later, while making sure it still works at runtime, be my guest :)
> > + with infra.smart_open(self.log_file) as lfh:
> > + lfh.write("> log in\n")
>
> smart_open() is called 3 times in login(), 2 in run() and a few more
> timer elsewhere in this class. Can't we open the file once in the
> constructor and the just write to it?
See above: I also *hate* this smart_open() thing. It is not smart at
all. It is stupid like crazy.
> > + # Run the given 'cmd' on the target
> > + # return a tuple (output, exit_code)
> > + def run(self, cmd):
> > + with infra.smart_open(self.log_file) as lfh:
> > + lfh.write("> running '{}'\n".format(cmd))
> > +
> > + self.__write("{}\n".format(cmd))
>
> I think this simpler form is equally legal:
>
> self.__write(cmd + "\n")
Fixed.
> > + def showlog(self):
> > + print "=== Full log ==="
> > + print self.log
> > + print "================"
>
> Unused function.
Indeed, removed.
> > +def main():
> > + parser = argparse.ArgumentParser(description='Run Buildroot tests')
> [...]
> > + if args.download is None:
> > + print "Missing download directory, please use -d/--download"
> > + return 1
>
> It would be nice to use an environment variable as a default for the
> download dir. It might be BR2_DL_DIR itself, i.e. sharing the existing
> Buildroot package download dir. I bet most people set BR2_DL_DIR in
> their ~/.bashrc.
Good point, added.
> > + if not os.path.exists(args.output):
> > + print "The selected output directory does not exist"
> > + return 1
>
> If the download directory does not exist is is created. Why is the
> output directory handled differently?
Fixed.
> And again, it would be nice to default to an environment variable, e.g.
> BR2_RUNTIME_TESTS_OUTPUT_DIR.
Not sure about this one, I don't really like environment variables in
general. I've kept it as an option only for the moment, we can revisit
this later.
> > + BRTest.outputdir = args.output
> > +
> > + if args.all is False and len(args.testname) == 0:
> > + print "No test selected"
> > + return 1
>
> I like programs that print their usage whenever the command line has
> errors (in addition to telling what the error is).
>
> IOW (untested):
>
> if args.all is False and len(args.testname) == 0:
> print "No test selected"
> + print ""
> + parser.print_help()
> return 1
Agreed, fixed!
Thanks again for the review!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-05 8:29 ` Thomas Petazzoni
@ 2017-03-07 21:59 ` Luca Ceresoli
2017-03-07 22:09 ` Thomas Petazzoni
0 siblings, 1 reply; 29+ messages in thread
From: Luca Ceresoli @ 2017-03-07 21:59 UTC (permalink / raw)
To: buildroot
Hi,
On 05/03/2017 09:29, Thomas Petazzoni wrote:
> Hello,
>
> On Sat, 4 Mar 2017 19:12:44 +0100, Luca Ceresoli wrote:
>
>> I'm no pythonist at all, so I don't know any of the mentioned
>> infrastructures. But one thing I dislike in the current implementation
>> is the unpleasant way failures are reported.
>
> Are you talking about:
>
> * Failures because there is a syntax error in the test script
>
> * Failures of the test itself, because the generated Buildroot system
> doesn't behave as expected
Is there a way to understand which kind of failure we had?
But my point was mostly about the first part of the process:
$ ./support/testing/run-tests -d ../br-test-dl -o ../br-test-out -a
[br-test-out/TestPythonBase/2017-03-03 23:22:46] Starting
[br-test-out/TestPythonBase/2017-03-03 23:22:46] Building
[br-test-out/TestPythonBase/2017-03-03 23:27:37] Building done
[br-test-out/TestPythonBase/2017-03-03 23:27:45] Cleaning up
.[br-test-out/TestDropbear/2017-03-03 23:27:46] Starting
[br-test-out/TestDropbear/2017-03-03 23:27:46] Building
[br-test-out/TestDropbear/2017-03-03 23:29:00] Building done
[br-test-out/TestDropbear/2017-03-03 23:29:06] Cleaning up
[...]
F[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Starting
[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Building
[br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Building done
[br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Cleaning up
.[br-test-out/TestSquashfs/2017-03-03 23:37:27] Starting
[br-test-out/TestSquashfs/2017-03-03 23:37:27] Building
[br-test-out/TestSquashfs/2017-03-03 23:39:38] Building done
[br-test-out/TestSquashfs/2017-03-03 23:39:41] Cleaning up
.[br-test-out/TestJffs2/2017-03-03 23:39:41] Starting
[br-test-out/TestJffs2/2017-03-03 23:39:41] Building
[br-test-out/TestJffs2/2017-03-03 23:42:40] Building done
open input file: No such file or directory
[br-test-out/TestJffs2/2017-03-03 23:42:40] Cleaning up
There's a character before each "Starting" line, I guess it's nose2
reporting the test outcome. It seems to be a '.' for success, 'F' or 'E'
for failures. What's the difference between 'E' and 'F'? Can we remove
these characters, which interfere with the rest of the output?
Also, the rest of the output is a bit more verbose than I'd like to
have. Can we have something like:
23:37:13 TestRootfsOverlay Starting
23:37:13 TestRootfsOverlay Building
23:37:27 TestRootfsOverlay Building done
23:37:27 TestRootfsOverlay Cleaning up
23:37:13 TestFoo Starting
23:37:13 TestFoo Building
23:37:27 TestFoo Building done
23:37:27 TestFoo *** FAILED! ***
Or, even better, something similar to the output of
./support/scripts/test-pkg:
TestRootfsOverlay: OK
Foo: run error: assertion "a == b" failed
That's just random ideas...
--
Luca
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-07 21:59 ` Luca Ceresoli
@ 2017-03-07 22:09 ` Thomas Petazzoni
2017-03-07 23:10 ` Luca Ceresoli
0 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2017-03-07 22:09 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, 7 Mar 2017 22:59:35 +0100, Luca Ceresoli wrote:
> > Are you talking about:
> >
> > * Failures because there is a syntax error in the test script
> >
> > * Failures of the test itself, because the generated Buildroot system
> > doesn't behave as expected
>
> Is there a way to understand which kind of failure we had?
Well the former will give you Python exceptions, the latter will give
you a failure of the test itself.
> But my point was mostly about the first part of the process:
>
> $ ./support/testing/run-tests -d ../br-test-dl -o ../br-test-out -a
> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Starting
> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Building
> [br-test-out/TestPythonBase/2017-03-03 23:27:37] Building done
> [br-test-out/TestPythonBase/2017-03-03 23:27:45] Cleaning up
> .[br-test-out/TestDropbear/2017-03-03 23:27:46] Starting
> [br-test-out/TestDropbear/2017-03-03 23:27:46] Building
> [br-test-out/TestDropbear/2017-03-03 23:29:00] Building done
> [br-test-out/TestDropbear/2017-03-03 23:29:06] Cleaning up
> [...]
> F[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Starting
> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Building
> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Building done
> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Cleaning up
> .[br-test-out/TestSquashfs/2017-03-03 23:37:27] Starting
> [br-test-out/TestSquashfs/2017-03-03 23:37:27] Building
> [br-test-out/TestSquashfs/2017-03-03 23:39:38] Building done
> [br-test-out/TestSquashfs/2017-03-03 23:39:41] Cleaning up
> .[br-test-out/TestJffs2/2017-03-03 23:39:41] Starting
> [br-test-out/TestJffs2/2017-03-03 23:39:41] Building
> [br-test-out/TestJffs2/2017-03-03 23:42:40] Building done
> open input file: No such file or directory
> [br-test-out/TestJffs2/2017-03-03 23:42:40] Cleaning up
>
> There's a character before each "Starting" line, I guess it's nose2
> reporting the test outcome. It seems to be a '.' for success, 'F' or 'E'
> for failures. What's the difference between 'E' and 'F'? Can we remove
> these characters, which interfere with the rest of the output?
Don't know, we would need to look at the internals of nose2/unittest to
know what's possible. Not sure how much customization is possible here.
> Also, the rest of the output is a bit more verbose than I'd like to
> have. Can we have something like:
>
> 23:37:13 TestRootfsOverlay Starting
> 23:37:13 TestRootfsOverlay Building
> 23:37:27 TestRootfsOverlay Building done
> 23:37:27 TestRootfsOverlay Cleaning up
> 23:37:13 TestFoo Starting
> 23:37:13 TestFoo Building
> 23:37:27 TestFoo Building done
> 23:37:27 TestFoo *** FAILED! ***
Indeed, would be nice.
The more I think of it, the less I believe re-using the existing
"runners" for Python unit tests is useful in our case. It doesn't bring
much, and prevents doing the customizations we need.
But I'm Cc'ing Ricardo, who has much more Python experience than I do.
Perhaps he has some ideas.
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-07 22:09 ` Thomas Petazzoni
@ 2017-03-07 23:10 ` Luca Ceresoli
2017-03-08 1:48 ` Ricardo Martincoski
0 siblings, 1 reply; 29+ messages in thread
From: Luca Ceresoli @ 2017-03-07 23:10 UTC (permalink / raw)
To: buildroot
On 07/03/2017 23:09, Thomas Petazzoni wrote:
> Hello,
>
> On Tue, 7 Mar 2017 22:59:35 +0100, Luca Ceresoli wrote:
>
>>> Are you talking about:
>>>
>>> * Failures because there is a syntax error in the test script
>>>
>>> * Failures of the test itself, because the generated Buildroot system
>>> doesn't behave as expected
>>
>> Is there a way to understand which kind of failure we had?
>
> Well the former will give you Python exceptions, the latter will give
> you a failure of the test itself.
Oh, sorry for misreading your original e-mail. Yeah, you're right, they
are clearly different.
I was speaking about the 2nd point. When a test is syntactically OK it
should stay so in the future until it's modified. But the test could
start failing later in time (which is why we want these tests).
>
>> But my point was mostly about the first part of the process:
>>
>> $ ./support/testing/run-tests -d ../br-test-dl -o ../br-test-out -a
>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Starting
>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Building
>> [br-test-out/TestPythonBase/2017-03-03 23:27:37] Building done
>> [br-test-out/TestPythonBase/2017-03-03 23:27:45] Cleaning up
>> .[br-test-out/TestDropbear/2017-03-03 23:27:46] Starting
>> [br-test-out/TestDropbear/2017-03-03 23:27:46] Building
>> [br-test-out/TestDropbear/2017-03-03 23:29:00] Building done
>> [br-test-out/TestDropbear/2017-03-03 23:29:06] Cleaning up
>> [...]
>> F[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Starting
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Building
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Building done
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Cleaning up
>> .[br-test-out/TestSquashfs/2017-03-03 23:37:27] Starting
>> [br-test-out/TestSquashfs/2017-03-03 23:37:27] Building
>> [br-test-out/TestSquashfs/2017-03-03 23:39:38] Building done
>> [br-test-out/TestSquashfs/2017-03-03 23:39:41] Cleaning up
>> .[br-test-out/TestJffs2/2017-03-03 23:39:41] Starting
>> [br-test-out/TestJffs2/2017-03-03 23:39:41] Building
>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Building done
>> open input file: No such file or directory
>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Cleaning up
>>
>> There's a character before each "Starting" line, I guess it's nose2
>> reporting the test outcome. It seems to be a '.' for success, 'F' or 'E'
>> for failures. What's the difference between 'E' and 'F'? Can we remove
>> these characters, which interfere with the rest of the output?
>
> Don't know, we would need to look at the internals of nose2/unittest to
> know what's possible. Not sure how much customization is possible here.
>
>> Also, the rest of the output is a bit more verbose than I'd like to
>> have. Can we have something like:
>>
>> 23:37:13 TestRootfsOverlay Starting
>> 23:37:13 TestRootfsOverlay Building
>> 23:37:27 TestRootfsOverlay Building done
>> 23:37:27 TestRootfsOverlay Cleaning up
>> 23:37:13 TestFoo Starting
>> 23:37:13 TestFoo Building
>> 23:37:27 TestFoo Building done
>> 23:37:27 TestFoo *** FAILED! ***
>
> Indeed, would be nice.
>
> The more I think of it, the less I believe re-using the existing
> "runners" for Python unit tests is useful in our case. It doesn't bring
> much, and prevents doing the customizations we need.
I don't know how much effort it would be to write our own test runner.
It the effort is little, I agree with you.
--
Luca
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 1/5] support/testing: core testing infrastructure
2017-03-07 23:10 ` Luca Ceresoli
@ 2017-03-08 1:48 ` Ricardo Martincoski
0 siblings, 0 replies; 29+ messages in thread
From: Ricardo Martincoski @ 2017-03-08 1:48 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, Mar 07, 2017 at 08:10 PM, Luca Ceresoli wrote:
> On 07/03/2017 23:09, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Tue, 7 Mar 2017 22:59:35 +0100, Luca Ceresoli wrote:
>>
>>>> Are you talking about:
>>>>
>>>> * Failures because there is a syntax error in the test script
>>>>
>>>> * Failures of the test itself, because the generated Buildroot system
>>>> doesn't behave as expected
>>>
>>> Is there a way to understand which kind of failure we had?
>>
>> Well the former will give you Python exceptions, the latter will give
>> you a failure of the test itself.
>
> Oh, sorry for misreading your original e-mail. Yeah, you're right, they
> are clearly different.
>
> I was speaking about the 2nd point. When a test is syntactically OK it
> should stay so in the future until it's modified. But the test could
> start failing later in time (which is why we want these tests).
>
>>
>>> But my point was mostly about the first part of the process:
>>>
>>> $ ./support/testing/run-tests -d ../br-test-dl -o ../br-test-out -a
>>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Starting
>>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Building
>>> [br-test-out/TestPythonBase/2017-03-03 23:27:37] Building done
>>> [br-test-out/TestPythonBase/2017-03-03 23:27:45] Cleaning up
>>> .[br-test-out/TestDropbear/2017-03-03 23:27:46] Starting
>>> [br-test-out/TestDropbear/2017-03-03 23:27:46] Building
>>> [br-test-out/TestDropbear/2017-03-03 23:29:00] Building done
>>> [br-test-out/TestDropbear/2017-03-03 23:29:06] Cleaning up
>>> [...]
>>> F[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Starting
>>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Building
>>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Building done
>>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Cleaning up
>>> .[br-test-out/TestSquashfs/2017-03-03 23:37:27] Starting
>>> [br-test-out/TestSquashfs/2017-03-03 23:37:27] Building
>>> [br-test-out/TestSquashfs/2017-03-03 23:39:38] Building done
>>> [br-test-out/TestSquashfs/2017-03-03 23:39:41] Cleaning up
>>> .[br-test-out/TestJffs2/2017-03-03 23:39:41] Starting
>>> [br-test-out/TestJffs2/2017-03-03 23:39:41] Building
>>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Building done
>>> open input file: No such file or directory
>>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Cleaning up
>>>
>>> There's a character before each "Starting" line, I guess it's nose2
>>> reporting the test outcome. It seems to be a '.' for success, 'F' or 'E'
>>> for failures. What's the difference between 'E' and 'F'? Can we remove
>>> these characters, which interfere with the rest of the output?
>>
>> Don't know, we would need to look at the internals of nose2/unittest to
>> know what's possible. Not sure how much customization is possible here.
Here are some tests. Notice I forced the dropbear test to fail and also I
previously ran the tests keeping the build (-k).
The current output is
$ support/testing/run-tests -d dl -o output -k tests.package
[output/TestPythonBase/2017-03-07 22:21:41] Starting
[output/TestPythonBase/2017-03-07 22:21:47] Cleaning up
.[output/TestDropbear/2017-03-07 22:21:47] Starting
[output/TestDropbear/2017-03-07 22:21:52] Cleaning up
F
======================================================================
FAIL: test_run (tests.package.test_dropbear.TestDropbear)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testing-v2/support/testing/tests/package/test_dropbear.py", line 25, in test_run
self.assertEqual(exit_code, 1) # make it fail
AssertionError: 0 != 1
Now using
+++ support/testing/conf/unittest.cfg
@@ -1,5 +1,6 @@
[unittest]
plugins = nose2.plugins.mp
+exclude-plugins = nose2.plugins.result
$ support/testing/run-tests -d dl -o output -k tests.package
[output/TestPythonBase/2017-03-07 22:26:31] Starting
[output/TestPythonBase/2017-03-07 22:26:37] Cleaning up
[output/TestDropbear/2017-03-07 22:26:37] Starting
[output/TestDropbear/2017-03-07 22:26:37] Cleaning up
Notice this way we lose the results. But see at the end.
>>
>>> Also, the rest of the output is a bit more verbose than I'd like to
>>> have. Can we have something like:
>>>
>>> 23:37:13 TestRootfsOverlay Starting
>>> 23:37:13 TestRootfsOverlay Building
>>> 23:37:27 TestRootfsOverlay Building done
>>> 23:37:27 TestRootfsOverlay Cleaning up
>>> 23:37:13 TestFoo Starting
>>> 23:37:13 TestFoo Building
>>> 23:37:27 TestFoo Building done
>>> 23:37:27 TestFoo *** FAILED! ***
>>
>> Indeed, would be nice.
The left part is straightforward:
+++ support/testing/infra/basetest.py
@@ -38,6 +38,5 @@ class BRTest(unittest.TestCase):
def show_msg(self, msg):
- print "[%s/%s/%s] %s" % (os.path.basename(self.__class__.outputdir),
- self.testname,
- datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
- msg)
+ print "{} {:20s} {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
+ self.testname,
+ msg)
$ support/testing/run-tests -d dl -o output -k tests.package
22:27:58 TestPythonBase Starting
22:28:04 TestPythonBase Cleaning up
22:28:04 TestDropbear Starting
22:28:04 TestDropbear Cleaning up
The result is the part I am not sure how to get working.
>>
>> The more I think of it, the less I believe re-using the existing
>> "runners" for Python unit tests is useful in our case. It doesn't bring
>> much, and prevents doing the customizations we need.
>
> I don't know how much effort it would be to write our own test runner.
> It the effort is little, I agree with you.
I use python mainly for tests together to Robot Framework, so I don't know much
about nose2.
I guess we need a plugin to replace the default one:
https://nose2.readthedocs.io/en/latest/plugins/result.html
Maybe there is some plugin out there that already does that.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2017-03-08 1:48 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-07 11:35 [Buildroot] [PATCH 0/5] Runtime testing infrastructure Thomas Petazzoni
2017-02-07 11:35 ` [Buildroot] [PATCH 1/5] support/testing: core " Thomas Petazzoni
2017-02-08 9:50 ` Thomas De Schampheleire
2017-02-08 10:05 ` Thomas Petazzoni
2017-02-08 11:42 ` Thomas De Schampheleire
2017-03-04 18:12 ` Luca Ceresoli
2017-03-05 7:27 ` Thomas De Schampheleire
2017-03-05 8:29 ` Thomas Petazzoni
2017-03-07 21:59 ` Luca Ceresoli
2017-03-07 22:09 ` Thomas Petazzoni
2017-03-07 23:10 ` Luca Ceresoli
2017-03-08 1:48 ` Ricardo Martincoski
2017-03-02 1:46 ` Ricardo Martincoski
2017-03-05 14:09 ` Thomas Petazzoni
2017-03-04 23:26 ` Luca Ceresoli
2017-03-05 14:44 ` Thomas Petazzoni
2017-02-07 11:36 ` [Buildroot] [PATCH 2/5] support/testing: add core tests Thomas Petazzoni
2017-03-02 1:47 ` Ricardo Martincoski
2017-03-05 14:11 ` Thomas Petazzoni
2017-02-07 11:36 ` [Buildroot] [PATCH 3/5] support/testing: add fs tests Thomas Petazzoni
2017-03-02 1:51 ` Ricardo Martincoski
2017-03-05 14:13 ` Thomas Petazzoni
2017-02-07 11:36 ` [Buildroot] [PATCH 4/5] support/testing: add package tests Thomas Petazzoni
2017-03-02 1:53 ` Ricardo Martincoski
2017-03-05 14:14 ` Thomas Petazzoni
2017-02-07 11:36 ` [Buildroot] [PATCH 5/5] support/testing: add toolchain tests Thomas Petazzoni
2017-03-02 1:54 ` Ricardo Martincoski
2017-03-02 1:45 ` [Buildroot] [PATCH 0/5] Runtime testing infrastructure Ricardo Martincoski
2017-03-05 14:08 ` Thomas Petazzoni
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.