* [PATCH v2 00/12] Add gnu testsuite execution for OEQA
@ 2019-09-03 16:56 Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 01/12] dejagnu: Add dejagnu for binutils/gcc test suites Nathan Rossi
` (11 more replies)
0 siblings, 12 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
This series adds support to execute the gnu test suites for binutils,
gcc and glibc. With the intention for enabling automated test running of
these test suites within the OEQA framework such that they can be
executed by the Yocto Autobuilder.
The test suites covered need significant resources or build artifacts
such that running them on the target is undesirable which rules out the
use of ptest. Because of this the test suites can be run on the build
host and call out to the target for execution (via ssh or using qemu
usermode).
The following implementation adds a do_check task to gcc-runtime, and
creates recipes for binutils and glibc in order to execute the test
suite on the build host. For binutils the reason for a separate recipe
is due to test suite dependence on target libraries (libgcc, glibc) and
the issues associated with a -cross recipe depending on target
libraries. For glibc the reason for a recipe is due to the dependency
chain (libgcc -> glibc -> libgcc-initial) and the test suite dependence
on libgcc.
Target execution is another important issue specifically since target
execution is slow to extremely slow depending on the physical or
emulated target performance. In order to provide faster execution
performance qemu linux-user is implemented alongside qemu system
emulation (via ssh+nfs). In initial testing qemu linux user vs qemu
system emulation provided performance gains of between 10x to 60x whilst
initially having small pass/fail differences. Further work as covered in
this series reduces the pass/fail differences for gcc/gcc-runtime close
to 0, which can be further reduced by marking known failures with test
result filtering.
However glibc is more strict with its expectations of CPU implementation
as well as syscall behaviour. Additionally glibc expects to be able to
execute OS tools such as 'sh' and 'echo' of which is not easy to provide
in the recipe-sysroot (bindir is not populated into the sysroot). As
such correct test results for glibc rely on execution with qemu system
emulation or on a physical target. This series however still includes
qemu user execution for glibc which can be useful (especially on slow
targets) assuming the known failing tests are excluded.
This series also introduces some OEQA test cases which cover running the
test suites. The test cases are split into binutils, gcc and glibc.
Individual test cases provide execution of the sub-suites of tests
within each target. For example binutils has binutils, gas and ld and
suites which can each be executed independently.
The test cases populate the individual test suite test cases into
testresults similar to ptestresults so that resulttool can analyse them.
Filtering of the failed test cases where expected failures occur is not
included in this series.
The OEQA test cases implement execution on qemu linux user by default.
Subclasses implement qemu system emulation setup and configuration for
running the tests.
Changes in v2:
- Dropped merged patches from series:
- scripts/lib/resulttool/report.py: Add more result types
- oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server
- binutils: Fix mips patch which changes default emulation
- Added dejagnu recipe to oe-core, imported from meta-oe
- Changes from binutils-cross to binutils-cross-testsuite recipe
- Dropped gold and libiberty test suite execution
- Changes from gcc-cross to only gcc-runtime for gcc/g++ tests
- BUILD_TEST_* to TOOLCHAIN_TEST_*
- New changes to oeqa/core for tagging tests, and patches to fix up
broken tests in the oeqa/core testsuite
- New changes to oeqa/selftest for '--run-only-tags' and
'--run-exclude-tags'
- Changes to new selftest tests to populate results like ptestresults
- Mark new selftest tests with 'machine' tag
---
Nathan Rossi (12):
dejagnu: Add dejagnu for binutils/gcc test suites
binutils-cross-testsuite: Create recipe for test suite execution
gcc-runtime: Add do_check task for executing gcc test suites
gcc-common.inc: Process staging fixme with correct target/native
sysroot
glibc-testsuite: Create a recipe to implement glibc test suite
oeqa/core/tests: Skip test_fail_duplicated_module
oeqa/core/tests: Fix test_data module tests
oeqa/core: Rework OETestTag and remove unused OETestFilter
oeqa/selftest: Add test run filtering based on test tags
oeqa/selftest/binutils: Create selftest case for binutils test suite
oeqa/selftest/gcc: Create selftest case for gcc test suite
oeqa/selftest/glibc: Create selftest case for glibc test suite
meta/lib/oeqa/core/context.py | 4 +-
meta/lib/oeqa/core/decorator/__init__.py | 20 ++--
meta/lib/oeqa/core/decorator/oetag.py | 27 -----
meta/lib/oeqa/core/loader.py | 61 +++-------
meta/lib/oeqa/core/tests/cases/data.py | 2 +-
meta/lib/oeqa/core/tests/cases/oetag.py | 21 +++-
meta/lib/oeqa/core/tests/common.py | 4 +-
meta/lib/oeqa/core/tests/test_data.py | 10 +-
meta/lib/oeqa/core/tests/test_decorators.py | 77 +++++++-----
meta/lib/oeqa/core/tests/test_loader.py | 26 +---
meta/lib/oeqa/selftest/cases/binutils.py | 64 ++++++++++
meta/lib/oeqa/selftest/cases/gcc.py | 111 ++++++++++++++++++
meta/lib/oeqa/selftest/cases/glibc.py | 97 +++++++++++++++
meta/lib/oeqa/selftest/context.py | 11 ++
.../glibc/glibc-testsuite_2.30.bb | 51 ++++++++
.../glibc/glibc/check-test-wrapper | 71 +++++++++++
.../binutils/binutils-cross-testsuite_2.32.bb | 83 +++++++++++++
.../recipes-devtools/dejagnu/dejagnu_1.6.2.bb | 15 +++
meta/recipes-devtools/gcc/gcc-common.inc | 2 +-
meta/recipes-devtools/gcc/gcc-runtime.inc | 42 +++++++
meta/recipes-devtools/gcc/gcc-testsuite.inc | 106 +++++++++++++++++
21 files changed, 762 insertions(+), 143 deletions(-)
delete mode 100644 meta/lib/oeqa/core/decorator/oetag.py
create mode 100644 meta/lib/oeqa/selftest/cases/binutils.py
create mode 100644 meta/lib/oeqa/selftest/cases/gcc.py
create mode 100644 meta/lib/oeqa/selftest/cases/glibc.py
create mode 100644 meta/recipes-core/glibc/glibc-testsuite_2.30.bb
create mode 100644 meta/recipes-core/glibc/glibc/check-test-wrapper
create mode 100644 meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb
create mode 100644 meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb
create mode 100644 meta/recipes-devtools/gcc/gcc-testsuite.inc
---
2.23.0.rc1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 01/12] dejagnu: Add dejagnu for binutils/gcc test suites
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 09/12] oeqa/selftest: Add test run filtering based on test tags Nathan Rossi
` (10 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
The binutils and gcc test suites use dejagnu for execution. Copy the
dejagnu recipe from meta-oe, and update it to 1.6.2.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- New
---
meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb | 15 +++++++++++++++
1 file changed, 15 insertions(+)
create mode 100644 meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb
diff --git a/meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb b/meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb
new file mode 100644
index 0000000000..7dc4e417f6
--- /dev/null
+++ b/meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb
@@ -0,0 +1,15 @@
+SUMMARY = "GNU unit testing framework, written in Expect and Tcl"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
+SECTION = "devel"
+
+DEPENDS += "expect-native"
+
+inherit autotools
+
+SRC_URI = "${GNU_MIRROR}/${BPN}/${BP}.tar.gz"
+
+SRC_URI[md5sum] = "e1b07516533f351b3aba3423fafeffd6"
+SRC_URI[sha256sum] = "0d0671e1b45189c5fc8ade4b3b01635fb9eeab45cf54f57db23e4c4c1a17d261"
+
+BBCLASSEXTEND = "native"
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 02/12] binutils-cross-testsuite: Create recipe for test suite execution
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (10 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 06/12] oeqa/core/tests: Skip test_fail_duplicated_module Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Create the do_check task in a new recipe 'binutils-cross-testsuite'.
This recipe is built within a target recipe (not -cross) to ensure
correct testing against target specific libraries/etc. The do_check task
is used to execute the binutils test suite for the cross target
binutils. By default this executes tests for binutils, gas and ld. This
can however be changed by setting CHECK_TARGETS to the desired test
suite target (e.g. 'gas').
The binutils test suites do not require any target execution, as such
the check task can be run without QEMU or a target device. However
since the binutils tests do rely on a C compiler there is dependence on
both gcc and libc in order to run the tests.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- Moved implementation of do_check from binutils-cross to
binutils-cross-testsuite. This is to fix issues with target dependency
from a cross recipe.
- Execute dejagnu directly instead of running via make
- Drop support for executing the gold and libiberty testsuites, the gold
suite was not executing correctly and the -cross variant of libiberty
is not used by target or -native
- No longer dependent on build directory tools, using as,ld,* from
sysroot
- (mips64) Hack/workaround gas defaulting to n32 despite ld defaulting
to 64
---
.../binutils/binutils-cross-testsuite_2.32.bb | 83 ++++++++++++++++++++++
1 file changed, 83 insertions(+)
create mode 100644 meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb
diff --git a/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb b/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb
new file mode 100644
index 0000000000..e62e64e8ae
--- /dev/null
+++ b/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb
@@ -0,0 +1,83 @@
+require binutils.inc
+require binutils-${PV}.inc
+
+BPN = "binutils"
+
+DEPENDS += "dejagnu-native expect-native"
+DEPENDS += "binutils-native"
+
+deltask do_compile
+deltask do_install
+
+do_configure[dirs] += "${B}/ld ${B}/bfd"
+do_configure() {
+ # create config.h, oe enables initfini-array by default
+ echo "#define HAVE_INITFINI_ARRAY" > ${B}/ld/config.h
+ # use the bfd_stdint.h from binutils-native, this is the same of the one
+ # generated by binutils-cross
+ cp ${RECIPE_SYSROOT_NATIVE}/usr/include/bfd_stdint.h ${B}/bfd/
+}
+
+# target depends
+DEPENDS += "virtual/${MLPREFIX}${TARGET_PREFIX}binutils"
+DEPENDS += "virtual/${MLPREFIX}${TARGET_PREFIX}gcc"
+DEPENDS += "virtual/${MLPREFIX}${TARGET_PREFIX}compilerlibs"
+DEPENDS += "virtual/${MLPREFIX}libc"
+
+python check_prepare() {
+ def suffix_sys(sys):
+ if sys.endswith("-linux"):
+ return sys + "-gnu"
+ return sys
+
+ def generate_site_exp(d, suite):
+ content = []
+ content.append('set srcdir "{0}/{1}"'.format(d.getVar("S"), suite))
+ content.append('set objdir "{0}/{1}"'.format(d.getVar("B"), suite))
+ content.append('set build_alias "{0}"'.format(d.getVar("BUILD_SYS")))
+ content.append('set build_triplet {0}'.format(d.getVar("BUILD_SYS")))
+ # use BUILD here since HOST=TARGET
+ content.append('set host_alias "{0}"'.format(d.getVar("BUILD_SYS")))
+ content.append('set host_triplet {0}'.format(d.getVar("BUILD_SYS")))
+ content.append('set target_alias "{0}"'.format(d.getVar("TARGET_SYS")))
+ content.append('set target_triplet {0}'.format(suffix_sys(d.getVar("TARGET_SYS"))))
+ content.append("set development true")
+ content.append("set experimental false")
+
+ content.append(d.expand('set CXXFILT "${TARGET_PREFIX}c++filt"'))
+ content.append(d.expand('set CC "${TARGET_PREFIX}gcc --sysroot=${STAGING_DIR_TARGET} ${TUNE_CCARGS}"'))
+ content.append(d.expand('set CXX "${TARGET_PREFIX}g++ --sysroot=${STAGING_DIR_TARGET} ${TUNE_CCARGS}"'))
+ content.append(d.expand('set CFLAGS_FOR_TARGET "--sysroot=${STAGING_DIR_TARGET} ${TUNE_CCARGS}"'))
+
+ if suite == "ld" and d.getVar("TUNE_ARCH") == "mips64":
+ # oe patches binutils to have the default mips64 abi as 64bit, but
+ # skips gas causing issues with the ld test suite (which uses gas)
+ content.append('set ASFLAGS "-64"')
+
+ return "\n".join(content)
+
+ for i in ["binutils", "gas", "ld"]:
+ builddir = os.path.join(d.getVar("B"), i)
+ if not os.path.isdir(builddir):
+ os.makedirs(builddir)
+ with open(os.path.join(builddir, "site.exp"), "w") as f:
+ f.write(generate_site_exp(d, i))
+}
+
+CHECK_TARGETS ??= "binutils gas ld"
+
+do_check[dirs] = "${B} ${B}/binutils ${B}/gas ${B}/ld"
+do_check[prefuncs] += "check_prepare"
+do_check[nostamp] = "1"
+do_check() {
+ export LC_ALL=C
+ for i in ${CHECK_TARGETS}; do
+ (cd ${B}/$i; runtest \
+ --tool $i \
+ --srcdir ${S}/$i/testsuite \
+ --ignore 'plugin.exp' \
+ || true)
+ done
+}
+addtask check after do_configure
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 03/12] gcc-runtime: Add do_check task for executing gcc test suites
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (2 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 04/12] gcc-common.inc: Process staging fixme with correct target/native sysroot Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 11/12] oeqa/selftest/gcc: Create selftest case for gcc test suite Nathan Rossi
` (7 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Add a do_check task to implement execution of the gcc component test
suites. The component test suites require execution of compiled programs
on the target.
The implementation provided allows for execution testing against a host
via SSH or within the local build environment using qemu linux-user
execution. The selection of execution is done via the
TOOLCHAIN_TEST_TARGET variable, and configuration of the remote host is
done with the TOOLCHAIN_TEST_HOST, TOOLCHAIN_TEST_HOST_USER and
TOOLCHAIN_TEST_HOST_PORT variables.
By default the do_check task will execute all check targets, this can be
changed by setting MAKE_CHECK_TARGETS to the desired test suite target
(e.g. check-gcc or check-target-libatomic).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- Dropped gcc-cross changes, all execution from gcc-runtime. Resolves
potential issues with -cross depending on target libraries/etc.
- Due to merge, running gcc and g++ test suites separately is no longer
supported through MAKE_CHECK_TARGETS.
- Changed BUILD_TEST_ to TOOLCHAIN_TEST_
- Added workaround for issue with configargs.h being modified by
builddir extraction which causes issues with plugins.
---
meta/recipes-devtools/gcc/gcc-runtime.inc | 42 +++++++++++
meta/recipes-devtools/gcc/gcc-testsuite.inc | 106 ++++++++++++++++++++++++++++
2 files changed, 148 insertions(+)
create mode 100644 meta/recipes-devtools/gcc/gcc-testsuite.inc
diff --git a/meta/recipes-devtools/gcc/gcc-runtime.inc b/meta/recipes-devtools/gcc/gcc-runtime.inc
index 22c1d78dd1..2da3c02ef0 100644
--- a/meta/recipes-devtools/gcc/gcc-runtime.inc
+++ b/meta/recipes-devtools/gcc/gcc-runtime.inc
@@ -277,3 +277,45 @@ FILES_libitm-dev = "\
SUMMARY_libitm-dev = "GNU transactional memory support library - development files"
FILES_libitm-staticdev = "${libdir}/libitm.a"
SUMMARY_libitm-staticdev = "GNU transactional memory support library - static development files"
+
+require gcc-testsuite.inc
+
+EXTRA_OEMAKE_prepend_task-check = "${PARALLEL_MAKE} "
+
+MAKE_CHECK_TARGETS ??= "check-gcc ${@" ".join("check-target-" + i for i in d.getVar("RUNTIMETARGET").split())}"
+# prettyprinters and xmethods require gdb tooling
+MAKE_CHECK_IGNORE ??= "prettyprinters.exp xmethods.exp"
+MAKE_CHECK_RUNTESTFLAGS ??= "${MAKE_CHECK_BOARDARGS} --ignore '${MAKE_CHECK_IGNORE}'"
+
+# specific host and target dependencies required for test suite running
+do_check[depends] += "dejagnu-native:do_populate_sysroot expect-native:do_populate_sysroot"
+do_check[depends] += "virtual/libc:do_populate_sysroot"
+# only depend on qemu if targeting linux user execution
+do_check[depends] += "${@'qemu-native:do_populate_sysroot' if "user" in d.getVar('TOOLCHAIN_TEST_TARGET') else ''}"
+# extend the recipe sysroot to include the built libraries (for qemu usermode)
+do_check[prefuncs] += "extend_recipe_sysroot"
+do_check[prefuncs] += "check_prepare"
+do_check[dirs] = "${WORKDIR}/dejagnu ${B}"
+do_check[nostamp] = "1"
+do_check() {
+ export DEJAGNU="${WORKDIR}/dejagnu/site.exp"
+
+ # HACK: this works around the configure setting CXX with -nostd* args
+ sed -i 's/-nostdinc++ -nostdlib++//g' $(find ${B} -name testsuite_flags | head -1)
+ # HACK: this works around the de-stashing changes to configargs.h, as well as recipe-sysroot changing the content
+ sed -i '/static const char configuration_arguments/d' ${B}/gcc/configargs.h
+ ${CC} -v 2>&1 | grep "^Configured with:" | \
+ sed 's/Configured with: \(.*\)/static const char configuration_arguments[] = "\1";/g' >> ${B}/gcc/configargs.h
+
+ if [ "${TOOLCHAIN_TEST_TARGET}" = "user" ]; then
+ # qemu user has issues allocating large amounts of memory
+ export G_SLICE=always-malloc
+ # no test should need more that 10G of memory, this prevents tests like pthread7-rope from leaking memory
+ ulimit -m 4194304
+ ulimit -v 10485760
+ fi
+
+ oe_runmake -i ${MAKE_CHECK_TARGETS} RUNTESTFLAGS="${MAKE_CHECK_RUNTESTFLAGS}"
+}
+addtask check after do_compile do_populate_sysroot
+
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
new file mode 100644
index 0000000000..b383a358d8
--- /dev/null
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -0,0 +1,106 @@
+inherit qemu
+
+TOOLCHAIN_TEST_TARGET ??= "user"
+TOOLCHAIN_TEST_HOST ??= "localhost"
+TOOLCHAIN_TEST_HOST_USER ??= "root"
+TOOLCHAIN_TEST_HOST_PORT ??= "2222"
+
+MAKE_CHECK_BOARDFLAGS ??= ""
+MAKE_CHECK_BOARDARGS ??= "--target_board=${TOOLCHAIN_TEST_TARGET}${MAKE_CHECK_BOARDFLAGS}"
+
+python () {
+ # Provide the targets compiler args via targets options. This allows dejagnu to
+ # correctly mark incompatible tests as UNSUPPORTED (e.g. needs soft-float
+ # but running on hard-float target).
+ #
+ # These options are called "multilib_flags" within the gcc test suite. Most
+ # architectures handle these options in a sensible way such that tests that
+ # are incompatible with the provided multilib are marked as UNSUPPORTED.
+ #
+ # Note: multilib flags are added to the compile command after the args
+ # provided by any test (through dg-options), CFLAGS_FOR_TARGET is always
+ # added to the compile command before any other args but is not interpted
+ # as options like multilib flags.
+ #
+ # i686, x86-64 and aarch64 are special, since most toolchains built for
+ # these targets don't do multilib the tests do not get correctly marked as
+ # UNSUPPORTED. More importantly the test suite itself does not handle
+ # overriding the multilib flags where it could (like other archs do). As
+ # such do not pass the target compiler args for these targets.
+ args = d.getVar("TUNE_CCARGS").split()
+ if d.getVar("TUNE_ARCH") in ["i686", "x86_64", "aarch64"]:
+ args = []
+ d.setVar("MAKE_CHECK_BOARDFLAGS", ("/" + "/".join(args)) if len(args) != 0 else "")
+}
+
+python check_prepare() {
+ def generate_qemu_linux_user_config(d):
+ content = []
+ content.append('load_generic_config "sim"')
+ content.append('load_base_board_description "basic-sim"')
+ content.append('process_multilib_options ""')
+
+ # qemu args
+ qemu_binary = qemu_target_binary(d)
+ if not qemu_binary:
+ bb.fatal("Missing target qemu linux-user binary")
+
+ args = []
+ # QEMU_OPTIONS is not always valid due to -cross recipe
+ args += ["-r", d.getVar("OLDEST_KERNEL")]
+ # enable all valid instructions, since the test suite itself does not
+ # limit itself to the target cpu options.
+ # - valid for x86*, powerpc, arm, arm64
+ if qemu_binary.lstrip("qemu-") in ["x86_64", "i386", "ppc", "arm", "aarch64"]:
+ args += ["-cpu", "max"]
+
+ sysroot = d.getVar("RECIPE_SYSROOT")
+ args += ["-L", sysroot]
+ # lib paths are static here instead of using $libdir since this is used by a -cross recipe
+ libpaths = [sysroot + "/usr/lib", sysroot + "/lib"]
+ args += ["-E", "LD_LIBRARY_PATH={0}".format(":".join(libpaths))]
+
+ content.append('set_board_info is_simulator 1')
+ content.append('set_board_info sim "{0}"'.format(qemu_binary))
+ content.append('set_board_info sim,options "{0}"'.format(" ".join(args)))
+
+ # target build/test config
+ content.append('set_board_info target_install {%s}' % d.getVar("TARGET_SYS"))
+ content.append('set_board_info ldscript ""')
+ #content.append('set_board_info needs_status_wrapper 1') # qemu-linux-user return codes work, and abort works fine
+ content.append('set_board_info gcc,stack_size 16834')
+ content.append('set_board_info gdb,nosignals 1')
+ content.append('set_board_info gcc,timeout 60')
+
+ return "\n".join(content)
+
+ def generate_remote_ssh_linux_config(d):
+ content = []
+ content.append('load_generic_config "unix"')
+ content.append("set_board_info hostname {0}".format(d.getVar("TOOLCHAIN_TEST_HOST")))
+ content.append("set_board_info username {0}".format(d.getVar("TOOLCHAIN_TEST_HOST_USER")))
+
+ port = d.getVar("TOOLCHAIN_TEST_HOST_PORT")
+ content.append("set_board_info rsh_prog \"/usr/bin/ssh -p {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
+ content.append("set_board_info rcp_prog \"/usr/bin/scp -P {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
+
+ return "\n".join(content)
+
+ dejagnudir = d.expand("${WORKDIR}/dejagnu")
+ if not os.path.isdir(dejagnudir):
+ os.makedirs(dejagnudir)
+
+ # write out target qemu board config
+ with open(os.path.join(dejagnudir, "user.exp"), "w") as f:
+ f.write(generate_qemu_linux_user_config(d))
+
+ # write out target ssh board config
+ with open(os.path.join(dejagnudir, "ssh.exp"), "w") as f:
+ f.write(generate_remote_ssh_linux_config(d))
+
+ # generate site.exp to provide boards
+ with open(os.path.join(dejagnudir, "site.exp"), "w") as f:
+ f.write("lappend boards_dir {0}\n".format(dejagnudir))
+ f.write("set CFLAGS_FOR_TARGET \"{0}\"\n".format(d.getVar("TOOLCHAIN_OPTIONS")))
+}
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 04/12] gcc-common.inc: Process staging fixme with correct target/native sysroot
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 01/12] dejagnu: Add dejagnu for binutils/gcc test suites Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 09/12] oeqa/selftest: Add test run filtering based on test tags Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 03/12] gcc-runtime: Add do_check task for executing gcc test suites Nathan Rossi
` (8 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Correct the 'staging_processfixme' call so that target sysroot and
native sysroot paths are corrected when extracting the stashed build
directory. This is required for 'make check' to work correctly due paths
used in configuration and scripts which point at the native sysroot.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
meta/recipes-devtools/gcc/gcc-common.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/gcc/gcc-common.inc b/meta/recipes-devtools/gcc/gcc-common.inc
index 89273a7745..44cba287f3 100644
--- a/meta/recipes-devtools/gcc/gcc-common.inc
+++ b/meta/recipes-devtools/gcc/gcc-common.inc
@@ -17,7 +17,7 @@ python extract_stashed_builddir () {
src = d.expand("${COMPONENTS_DIR}/${BUILD_ARCH}/gcc-stashed-builddir-${TARGET_SYS}")
dest = d.getVar("B")
oe.path.copyhardlinktree(src, dest)
- staging_processfixme([src + "/fixmepath"], dest, dest, dest, d)
+ staging_processfixme([src + "/fixmepath"], dest, d.getVar("RECIPE_SYSROOT"), d.getVar("RECIPE_SYSROOT_NATIVE"), d)
}
def get_gcc_float_setting(bb, d):
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc test suite
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (8 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 10/12] oeqa/selftest/binutils: Create selftest case for binutils " Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-04 15:35 ` Khem Raj
2019-09-03 16:56 ` [PATCH v2 06/12] oeqa/core/tests: Skip test_fail_duplicated_module Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 02/12] binutils-cross-testsuite: Create recipe for test suite execution Nathan Rossi
11 siblings, 1 reply; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
A recipe needs to be created for the test suite due to the dependency
chain between libgcc -> glibc -> libgcc-initial, and the requirements of
the test suite to have libgcc for compilation and execution.
The glibc test suite does not use dejagnu like the gcc test suites do.
Instead a test wrapper script is used along with the assumed dependency
of having the same filesystem available on build host and target. For
qemu linux-user the same filesystem is inherently available, for remote
targets NFS is used. Separate test wrapper scripts are created for qemu
linux-user or ssh targets, with the same TOOLCHAIN_TEST_* variables used for
configuration.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- Renamed BUILD_TEST_* to TOOLCHAIN_TEST_*
---
meta/recipes-core/glibc/glibc-testsuite_2.30.bb | 51 +++++++++++++++++
meta/recipes-core/glibc/glibc/check-test-wrapper | 71 ++++++++++++++++++++++++
2 files changed, 122 insertions(+)
create mode 100644 meta/recipes-core/glibc/glibc-testsuite_2.30.bb
create mode 100644 meta/recipes-core/glibc/glibc/check-test-wrapper
diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.30.bb b/meta/recipes-core/glibc/glibc-testsuite_2.30.bb
new file mode 100644
index 0000000000..88764d9e2b
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc-testsuite_2.30.bb
@@ -0,0 +1,51 @@
+require glibc_${PV}.bb
+
+# handle PN differences
+FILESEXTRAPATHS_prepend := "${THISDIR}/glibc:"
+
+# strip provides
+PROVIDES = ""
+# setup depends
+INHIBIT_DEFAULT_DEPS = ""
+
+DEPENDS += "glibc-locale libgcc gcc-runtime"
+
+# remove the initial depends
+DEPENDS_remove = "libgcc-initial"
+
+inherit qemu
+
+SRC_URI += "file://check-test-wrapper"
+
+DEPENDS += "${@'qemu-native' if d.getVar('TOOLCHAIN_TEST_TARGET') == 'user' else ''}"
+
+TOOLCHAIN_TEST_TARGET ??= "user"
+TOOLCHAIN_TEST_HOST ??= "localhost"
+TOOLCHAIN_TEST_HOST_USER ??= "root"
+TOOLCHAIN_TEST_HOST_PORT ??= "2222"
+
+do_check[dirs] += "${B}"
+do_check[nostamp] = "1"
+do_check () {
+ chmod 0755 ${WORKDIR}/check-test-wrapper
+
+ # clean out previous test results
+ oe_runmake tests-clean
+ # makefiles don't clean entirely (and also sometimes fails due to too many args)
+ find ${B} -type f -name "*.out" -delete
+ find ${B} -type f -name "*.test-result" -delete
+ find ${B}/catgets -name "*.cat" -delete
+ find ${B}/conform -name "symlist-*" -delete
+ [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
+
+ oe_runmake -i \
+ QEMU_SYSROOT="${RECIPE_SYSROOT}" \
+ QEMU_OPTIONS="${@qemu_target_binary(d)} ${QEMU_OPTIONS}" \
+ SSH_HOST="${TOOLCHAIN_TEST_HOST}" \
+ SSH_HOST_USER="${TOOLCHAIN_TEST_HOST_USER}" \
+ SSH_HOST_PORT="${TOOLCHAIN_TEST_HOST_PORT}" \
+ test-wrapper="${WORKDIR}/check-test-wrapper ${TOOLCHAIN_TEST_TARGET}" \
+ check
+}
+addtask do_check after do_compile
+
diff --git a/meta/recipes-core/glibc/glibc/check-test-wrapper b/meta/recipes-core/glibc/glibc/check-test-wrapper
new file mode 100644
index 0000000000..f8e04e02d2
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc/check-test-wrapper
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+import sys
+import os
+import subprocess
+
+env = os.environ.copy()
+args = sys.argv[1:]
+targettype = args.pop(0)
+
+if targettype == "user":
+ qemuargs = os.environ.get("QEMU_OPTIONS", "").split()
+ if not os.path.exists(qemuargs[0]):
+ # ensure qemu args has a valid absolute path
+ for i in os.environ.get("PATH", "").split(":"):
+ if os.path.exists(os.path.join(i, qemuargs[0])):
+ qemuargs[0] = os.path.join(i, qemuargs[0])
+ break
+ sysroot = os.environ.get("QEMU_SYSROOT", None)
+ if not sysroot:
+ sys.exit(-1)
+ libpaths = [sysroot + "/usr/lib", sysroot + "/lib"]
+
+ if args[0] == "env":
+ args.pop(0)
+ if len(args) == 0:
+ args = ["env"]
+ else:
+ # process options
+ while args[0].startswith("-"):
+ opt = args.pop(0).lstrip("-")
+ if "i" in opt:
+ env.clear()
+ # process environment vars
+ while "=" in args[0]:
+ key, val = args.pop(0).split("=", 1)
+ if key == "LD_LIBRARY_PATH":
+ libpaths += val.split(":")
+ else:
+ env[key] = val
+ if args[0] == "cp":
+ # ignore copies, the filesystem is the same
+ sys.exit(0)
+
+ qemuargs += ["-L", sysroot]
+ qemuargs += ["-E", "LD_LIBRARY_PATH={}".format(":".join(libpaths))]
+ command = qemuargs + args
+elif targettype == "ssh":
+ host = os.environ.get("SSH_HOST", None)
+ user = os.environ.get("SSH_HOST_USER", None)
+ port = os.environ.get("SSH_HOST_PORT", None)
+
+ command = ["ssh", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no"]
+ if port:
+ command += ["-p", str(port)]
+ if not host:
+ sys.exit(-1)
+ command += ["{}@{}".format(user, host) if user else host]
+
+ # wrap and replace quotes for correct transformation on ssh
+ wrapped = " ".join(["'{0}'".format(i.replace("'", r"'\''")) for i in ["cd", os.getcwd()]]) + "; "
+ wrapped += " ".join(["'{0}'".format(i.replace("'", r"'\''")) for i in args])
+ command += ["sh", "-c", "\"{}\"".format(wrapped)]
+else:
+ sys.exit(-1)
+
+try:
+ r = subprocess.run(command, timeout = 1800, env = env)
+ sys.exit(r.returncode)
+except subprocess.TimeoutExpired:
+ sys.exit(-1)
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 06/12] oeqa/core/tests: Skip test_fail_duplicated_module
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (9 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc " Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 02/12] binutils-cross-testsuite: Create recipe for test suite execution Nathan Rossi
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
At some point the modules in tests/cases/loader/invalid/ were removed,
this test relied on having tests/cases/loader/invalid/oetag.py to
trigger a ImportError in the loader module. Skip this test.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- New
---
meta/lib/oeqa/core/tests/test_loader.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/lib/oeqa/core/tests/test_loader.py b/meta/lib/oeqa/core/tests/test_loader.py
index 519ba96091..e73c91b141 100755
--- a/meta/lib/oeqa/core/tests/test_loader.py
+++ b/meta/lib/oeqa/core/tests/test_loader.py
@@ -40,6 +40,7 @@ class TestLoader(TestBase):
result = True if expect in str(e) else False
self.assertTrue(result, msg=msg)
+ @unittest.skip("invalid directory is missing oetag.py")
def test_fail_duplicated_module(self):
cases_path = self.cases_path
invalid_path = os.path.join(cases_path, 'loader', 'invalid')
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 07/12] oeqa/core/tests: Fix test_data module tests
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (5 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 08/12] oeqa/core: Rework OETestTag and remove unused OETestFilter Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 12/12] oeqa/selftest/glibc: Create selftest case for glibc test suite Nathan Rossi
` (4 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
These two tests relied on the context containing the results information.
This was moved into the OETestResults class.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- New
---
meta/lib/oeqa/core/tests/test_data.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oeqa/core/tests/test_data.py b/meta/lib/oeqa/core/tests/test_data.py
index 50811bb381..ac74098b78 100755
--- a/meta/lib/oeqa/core/tests/test_data.py
+++ b/meta/lib/oeqa/core/tests/test_data.py
@@ -22,8 +22,9 @@ class TestData(TestBase):
expectedException = "oeqa.core.exception.OEQAMissingVariable"
tc = self._testLoader(modules=self.modules)
- self.assertEqual(False, tc.runTests().wasSuccessful())
- for test, data in tc.errors:
+ results = tc.runTests()
+ self.assertFalse(results.wasSuccessful())
+ for test, data in results.errors:
expect = False
if expectedException in data:
expect = True
@@ -35,8 +36,9 @@ class TestData(TestBase):
d = {'IMAGE' : 'core-image-sato', 'ARCH' : 'arm'}
tc = self._testLoader(d=d, modules=self.modules)
- self.assertEqual(False, tc.runTests().wasSuccessful())
- for test, data in tc.failures:
+ results = tc.runTests()
+ self.assertFalse(results.wasSuccessful())
+ for test, data in results.failures:
expect = False
if expectedError in data:
expect = True
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 08/12] oeqa/core: Rework OETestTag and remove unused OETestFilter
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (4 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 11/12] oeqa/selftest/gcc: Create selftest case for gcc test suite Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 07/12] oeqa/core/tests: Fix test_data module tests Nathan Rossi
` (5 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Rework OETestTag so that it does not rely on the existing decorator code
base and instead inserts the tags into an attribute on the decorated
target (e.g. class/type or method). This allows the use of OETestTag on
classes and method.
In order to filter tagged tests rework the loaders filtering code,
removing the generic-ness (with validation and attributes/etc.) and
replace it with a "tags_filter" parameter which is a function that
filters a test based on the tags it has. This allows the loader user to
filter on tags in more specific ways (e.g. include all untagged tests
and any tests tagged with foo). Plumb all this through the context code
and testing code.
Update the associated tests to pass correctly with the changes.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- New
---
meta/lib/oeqa/core/context.py | 4 +-
meta/lib/oeqa/core/decorator/__init__.py | 20 +++++---
meta/lib/oeqa/core/decorator/oetag.py | 27 ----------
meta/lib/oeqa/core/loader.py | 61 +++++++----------------
meta/lib/oeqa/core/tests/cases/data.py | 2 +-
meta/lib/oeqa/core/tests/cases/oetag.py | 21 +++++++-
meta/lib/oeqa/core/tests/common.py | 4 +-
meta/lib/oeqa/core/tests/test_decorators.py | 77 +++++++++++++++++++----------
meta/lib/oeqa/core/tests/test_loader.py | 25 ----------
9 files changed, 103 insertions(+), 138 deletions(-)
delete mode 100644 meta/lib/oeqa/core/decorator/oetag.py
diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 68819cc338..14fc6a54f4 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -64,12 +64,12 @@ class OETestContext(object):
setattr(tclass, 'setUpHooker', skipfuncgen('Skip by the command line argument "%s"' % skip))
def loadTests(self, module_paths, modules=[], tests=[],
- modules_manifest="", modules_required=[], filters={}):
+ modules_manifest="", modules_required=[], **kwargs):
if modules_manifest:
modules = self._read_modules_from_manifest(modules_manifest)
self.loader = self.loaderClass(self, module_paths, modules, tests,
- modules_required, filters)
+ modules_required, **kwargs)
self.suites = self.loader.discover()
def runTests(self, processes=None, skips=[]):
diff --git a/meta/lib/oeqa/core/decorator/__init__.py b/meta/lib/oeqa/core/decorator/__init__.py
index 923b218266..1a5ac40134 100644
--- a/meta/lib/oeqa/core/decorator/__init__.py
+++ b/meta/lib/oeqa/core/decorator/__init__.py
@@ -6,6 +6,7 @@
from functools import wraps
from abc import abstractmethod, ABCMeta
+from oeqa.core.utils.misc import strToList
decoratorClasses = set()
@@ -63,12 +64,15 @@ class OETestDiscover(OETestDecorator):
def discover(registry):
return registry['cases']
-class OETestFilter(OETestDecorator):
+def OETestTag(*tags):
+ expandedtags = []
+ for tag in tags:
+ expandedtags += strToList(tag)
+ def decorator(item):
+ if hasattr(item, "__oeqa_testtags"):
+ item.__oeqa_testtags += expandedtags
+ else:
+ item.__oeqa_testtags = expandedtags
+ return item
+ return decorator
- # OETestLoader call it while loading the tests
- # in loadTestsFromTestCase method, it needs to
- # return a bool, True if needs to be filtered.
- # This method must consume the filter used.
- @abstractmethod
- def filtrate(self, filters):
- return False
diff --git a/meta/lib/oeqa/core/decorator/oetag.py b/meta/lib/oeqa/core/decorator/oetag.py
deleted file mode 100644
index 8c31138dac..0000000000
--- a/meta/lib/oeqa/core/decorator/oetag.py
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Copyright (C) 2016 Intel Corporation
-#
-# SPDX-License-Identifier: MIT
-#
-
-from . import OETestFilter, registerDecorator
-from oeqa.core.utils.misc import strToList
-
-def _tagFilter(tags, filters):
- return False if set(tags) & set(filters) else True
-
-@registerDecorator
-class OETestTag(OETestFilter):
- attrs = ('oetag',)
-
- def bind(self, registry, case):
- super(OETestTag, self).bind(registry, case)
- self.oetag = strToList(self.oetag, 'oetag')
-
- def filtrate(self, filters):
- if filters.get('oetag'):
- filterx = strToList(filters['oetag'], 'oetag')
- del filters['oetag']
- if _tagFilter(self.oetag, filterx):
- return True
- return False
diff --git a/meta/lib/oeqa/core/loader.py b/meta/lib/oeqa/core/loader.py
index 7fea0585c7..0d7970d49e 100644
--- a/meta/lib/oeqa/core/loader.py
+++ b/meta/lib/oeqa/core/loader.py
@@ -16,7 +16,7 @@ from oeqa.core.utils.test import getSuiteModules, getCaseID
from oeqa.core.exception import OEQATestNotFound
from oeqa.core.case import OETestCase
from oeqa.core.decorator import decoratorClasses, OETestDecorator, \
- OETestFilter, OETestDiscover
+ OETestDiscover
# When loading tests, the unittest framework stores any exceptions and
# displays them only when the run method is called.
@@ -68,7 +68,7 @@ class OETestLoader(unittest.TestLoader):
'_top_level_dir']
def __init__(self, tc, module_paths, modules, tests, modules_required,
- filters, *args, **kwargs):
+ *args, **kwargs):
self.tc = tc
self.modules = _built_modules_dict(modules)
@@ -76,13 +76,7 @@ class OETestLoader(unittest.TestLoader):
self.tests = tests
self.modules_required = modules_required
- self.filters = filters
- self.decorator_filters = [d for d in decoratorClasses if \
- issubclass(d, OETestFilter)]
- self._validateFilters(self.filters, self.decorator_filters)
- self.used_filters = [d for d in self.decorator_filters
- for f in self.filters
- if f in d.attrs]
+ self.tags_filter = kwargs.get("tags_filter", None)
if isinstance(module_paths, str):
module_paths = [module_paths]
@@ -104,28 +98,6 @@ class OETestLoader(unittest.TestLoader):
setattr(testCaseClass, 'td', self.tc.td)
setattr(testCaseClass, 'logger', self.tc.logger)
- def _validateFilters(self, filters, decorator_filters):
- # Validate if filter isn't empty
- for key,value in filters.items():
- if not value:
- raise TypeError("Filter %s specified is empty" % key)
-
- # Validate unique attributes
- attr_filters = [attr for clss in decorator_filters \
- for attr in clss.attrs]
- dup_attr = [attr for attr in attr_filters
- if attr_filters.count(attr) > 1]
- if dup_attr:
- raise TypeError('Detected duplicated attribute(s) %s in filter'
- ' decorators' % ' ,'.join(dup_attr))
-
- # Validate if filter is supported
- for f in filters:
- if f not in attr_filters:
- classes = ', '.join([d.__name__ for d in decorator_filters])
- raise TypeError('Found "%s" filter but not declared in any of '
- '%s decorators' % (f, classes))
-
def _registerTestCase(self, case):
case_id = case.id()
self.tc._registry['cases'][case_id] = case
@@ -188,19 +160,20 @@ class OETestLoader(unittest.TestLoader):
return True
# Decorator filters
- if self.filters and isinstance(case, OETestCase):
- filters = self.filters.copy()
- case_decorators = [cd for cd in case.decorators
- if cd.__class__ in self.used_filters]
-
- # Iterate over case decorators to check if needs to be filtered.
- for cd in case_decorators:
- if cd.filtrate(filters):
- return True
-
- # Case is missing one or more decorators for all the filters
- # being used, so filter test case.
- if filters:
+ if self.tags_filter is not None and callable(self.tags_filter):
+ alltags = set()
+ # pull tags from the case class
+ if hasattr(case, "__oeqa_testtags"):
+ for t in getattr(case, "__oeqa_testtags"):
+ alltags.add(t)
+ # pull tags from the method itself
+ if hasattr(case, test_name):
+ method = getattr(case, test_name)
+ if hasattr(method, "__oeqa_testtags"):
+ for t in getattr(method, "__oeqa_testtags"):
+ alltags.add(t)
+
+ if self.tags_filter(alltags):
return True
return False
diff --git a/meta/lib/oeqa/core/tests/cases/data.py b/meta/lib/oeqa/core/tests/cases/data.py
index 0d8de87ae7..61f88547f7 100644
--- a/meta/lib/oeqa/core/tests/cases/data.py
+++ b/meta/lib/oeqa/core/tests/cases/data.py
@@ -5,7 +5,7 @@
#
from oeqa.core.case import OETestCase
-from oeqa.core.decorator.oetag import OETestTag
+from oeqa.core.decorator import OETestTag
from oeqa.core.decorator.data import OETestDataDepends
class DataTest(OETestCase):
diff --git a/meta/lib/oeqa/core/tests/cases/oetag.py b/meta/lib/oeqa/core/tests/cases/oetag.py
index 4e1d080985..52f97dfda6 100644
--- a/meta/lib/oeqa/core/tests/cases/oetag.py
+++ b/meta/lib/oeqa/core/tests/cases/oetag.py
@@ -5,10 +5,9 @@
#
from oeqa.core.case import OETestCase
-from oeqa.core.decorator.oetag import OETestTag
+from oeqa.core.decorator import OETestTag
class TagTest(OETestCase):
-
@OETestTag('goodTag')
def testTagGood(self):
self.assertTrue(True, msg='How is this possible?')
@@ -17,5 +16,23 @@ class TagTest(OETestCase):
def testTagOther(self):
self.assertTrue(True, msg='How is this possible?')
+ @OETestTag('otherTag', 'multiTag')
+ def testTagOtherMulti(self):
+ self.assertTrue(True, msg='How is this possible?')
+
def testTagNone(self):
self.assertTrue(True, msg='How is this possible?')
+
+@OETestTag('classTag')
+class TagClassTest(OETestCase):
+ @OETestTag('otherTag')
+ def testTagOther(self):
+ self.assertTrue(True, msg='How is this possible?')
+
+ @OETestTag('otherTag', 'multiTag')
+ def testTagOtherMulti(self):
+ self.assertTrue(True, msg='How is this possible?')
+
+ def testTagNone(self):
+ self.assertTrue(True, msg='How is this possible?')
+
diff --git a/meta/lib/oeqa/core/tests/common.py b/meta/lib/oeqa/core/tests/common.py
index 39efd504c0..88cc758ad3 100644
--- a/meta/lib/oeqa/core/tests/common.py
+++ b/meta/lib/oeqa/core/tests/common.py
@@ -30,9 +30,9 @@ class TestBase(unittest.TestCase):
directory = os.path.dirname(os.path.abspath(__file__))
self.cases_path = os.path.join(directory, 'cases')
- def _testLoader(self, d={}, modules=[], tests=[], filters={}):
+ def _testLoader(self, d={}, modules=[], tests=[], **kwargs):
from oeqa.core.context import OETestContext
tc = OETestContext(d, self.logger)
tc.loadTests(self.cases_path, modules=modules, tests=tests,
- filters=filters)
+ **kwargs)
return tc
diff --git a/meta/lib/oeqa/core/tests/test_decorators.py b/meta/lib/oeqa/core/tests/test_decorators.py
index 499cd66ff3..b798bf7d33 100755
--- a/meta/lib/oeqa/core/tests/test_decorators.py
+++ b/meta/lib/oeqa/core/tests/test_decorators.py
@@ -14,35 +14,58 @@ setup_sys_path()
from oeqa.core.exception import OEQADependency
from oeqa.core.utils.test import getCaseMethod, getSuiteCasesNames, getSuiteCasesIDs
-class TestFilterDecorator(TestBase):
-
- def _runFilterTest(self, modules, filters, expect, msg):
- tc = self._testLoader(modules=modules, filters=filters)
- test_loaded = set(getSuiteCasesNames(tc.suites))
- self.assertEqual(expect, test_loaded, msg=msg)
+class TestTagDecorator(TestBase):
+ def _runTest(self, modules, filterfn, expect):
+ tc = self._testLoader(modules = modules, tags_filter = filterfn)
+ test_loaded = set(getSuiteCasesIDs(tc.suites))
+ self.assertEqual(expect, test_loaded)
def test_oetag(self):
- # Get all cases without filtering.
- filter_all = {}
- test_all = {'testTagGood', 'testTagOther', 'testTagNone'}
- msg_all = 'Failed to get all oetag cases without filtering.'
-
- # Get cases with 'goodTag'.
- filter_good = {'oetag':'goodTag'}
- test_good = {'testTagGood'}
- msg_good = 'Failed to get just one test filtering with "goodTag" oetag.'
-
- # Get cases with an invalid tag.
- filter_invalid = {'oetag':'invalidTag'}
- test_invalid = set()
- msg_invalid = 'Failed to filter all test using an invalid oetag.'
-
- tests = ((filter_all, test_all, msg_all),
- (filter_good, test_good, msg_good),
- (filter_invalid, test_invalid, msg_invalid))
-
- for test in tests:
- self._runFilterTest(['oetag'], test[0], test[1], test[2])
+ # get all cases without any filtering
+ self._runTest(['oetag'], None, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagOther',
+ 'oetag.TagTest.testTagOtherMulti',
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # exclude any case with tags
+ self._runTest(['oetag'], lambda tags: tags, {
+ 'oetag.TagTest.testTagNone',
+ })
+
+ # exclude any case with otherTag
+ self._runTest(['oetag'], lambda tags: "otherTag" in tags, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # exclude any case with classTag
+ self._runTest(['oetag'], lambda tags: "classTag" in tags, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagOther',
+ 'oetag.TagTest.testTagOtherMulti',
+ 'oetag.TagTest.testTagNone',
+ })
+
+ # include any case with classTag
+ self._runTest(['oetag'], lambda tags: "classTag" not in tags, {
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # include any case with classTag or no tags
+ self._runTest(['oetag'], lambda tags: tags and "classTag" not in tags, {
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
class TestDependsDecorator(TestBase):
modules = ['depends']
diff --git a/meta/lib/oeqa/core/tests/test_loader.py b/meta/lib/oeqa/core/tests/test_loader.py
index e73c91b141..cb38ac845e 100755
--- a/meta/lib/oeqa/core/tests/test_loader.py
+++ b/meta/lib/oeqa/core/tests/test_loader.py
@@ -15,31 +15,6 @@ from oeqa.core.exception import OEQADependency
from oeqa.core.utils.test import getSuiteModules, getSuiteCasesIDs
class TestLoader(TestBase):
-
- def test_fail_empty_filter(self):
- filters = {'oetag' : ''}
- expect = 'Filter oetag specified is empty'
- msg = 'Expected TypeError exception for having invalid filter'
- try:
- # Must throw TypeError because empty filter
- tc = self._testLoader(filters=filters)
- self.fail(msg)
- except TypeError as e:
- result = True if expect in str(e) else False
- self.assertTrue(result, msg=msg)
-
- def test_fail_invalid_filter(self):
- filters = {'invalid' : 'good'}
- expect = 'filter but not declared in any of'
- msg = 'Expected TypeError exception for having invalid filter'
- try:
- # Must throw TypeError because invalid filter
- tc = self._testLoader(filters=filters)
- self.fail(msg)
- except TypeError as e:
- result = True if expect in str(e) else False
- self.assertTrue(result, msg=msg)
-
@unittest.skip("invalid directory is missing oetag.py")
def test_fail_duplicated_module(self):
cases_path = self.cases_path
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 09/12] oeqa/selftest: Add test run filtering based on test tags
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 01/12] dejagnu: Add dejagnu for binutils/gcc test suites Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 04/12] gcc-common.inc: Process staging fixme with correct target/native sysroot Nathan Rossi
` (9 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Add '--run-only-tags' for running tests which match any of the provided
tags, and '--run-exclude-tags' for running all tests except those that
have any of the provided tags.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- New
---
meta/lib/oeqa/selftest/context.py | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index d279994ddf..47de08e3f2 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -61,6 +61,12 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
group.add_argument('-r', '--run-tests', required=False, action='store',
nargs='+', dest="run_tests", default=None,
help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
+ group.add_argument('-t', '--run-only-tags', action='store',
+ nargs='+', dest="run_only_tags", default=None,
+ help='Run all (unhidden) tests which match any of the specified tags.')
+ group.add_argument('-T', '--run-exclude-tags', action='store',
+ nargs='+', dest="run_exclude_tags", default=None,
+ help='Run all (unhidden) tests excluding any that match any of the specified tags.')
group.add_argument('-m', '--list-modules', required=False,
action="store_true", default=False,
@@ -149,6 +155,11 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
copyfile(self.tc_kwargs['init']['config_paths']['bblayers'],
self.tc_kwargs['init']['config_paths']['bblayers_backup'])
+ if args.run_only_tags:
+ self.tc_kwargs['load']['tags_filter'] = lambda tags: not tags or not any(tag in args.run_only_tags for tag in tags)
+ if args.run_exclude_tags:
+ self.tc_kwargs['load']['tags_filter'] = lambda tags: any(tag in args.run_exclude_tags for tag in tags)
+
self.tc_kwargs['run']['skips'] = args.skips
self.tc_kwargs['run']['processes'] = args.processes
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 10/12] oeqa/selftest/binutils: Create selftest case for binutils test suite
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (7 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 12/12] oeqa/selftest/glibc: Create selftest case for glibc test suite Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc " Nathan Rossi
` (2 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Create a oeqa selftest test case to execute the binutils test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- Update to use binutils-cross-testsuite and CHECK_TARGETS
- Drop gold and libiberty test cases (and result parsing code)
- Populate test results as ptestresults
- Mark tests with 'machine' tag
---
meta/lib/oeqa/selftest/cases/binutils.py | 64 ++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/binutils.py
diff --git a/meta/lib/oeqa/selftest/cases/binutils.py b/meta/lib/oeqa/selftest/cases/binutils.py
new file mode 100644
index 0000000000..4edee09390
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/binutils.py
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: MIT
+import os
+import sys
+import re
+import logging
+from oeqa.core.decorator import OETestTag
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars
+
+def parse_values(content):
+ for i in content:
+ for v in ["PASS", "FAIL", "XPASS", "XFAIL", "UNRESOLVED", "UNSUPPORTED", "UNTESTED", "ERROR", "WARNING"]:
+ if i.startswith(v + ": "):
+ yield i[len(v) + 2:].strip(), v
+ break
+
+@OETestTag("machine")
+class BinutilsCrossSelfTest(OESelftestTestCase):
+ @classmethod
+ def setUpClass(cls):
+ super().setUpClass()
+ if not hasattr(cls.tc, "extraresults"):
+ cls.tc.extraresults = {}
+
+ if "ptestresult.sections" not in cls.tc.extraresults:
+ cls.tc.extraresults["ptestresult.sections"] = {}
+
+ def test_binutils(self):
+ self.run_binutils("binutils")
+
+ def test_gas(self):
+ self.run_binutils("gas")
+
+ def test_ld(self):
+ self.run_binutils("ld")
+
+ def run_binutils(self, suite):
+ features = []
+ features.append('CHECK_TARGETS = "{0}"'.format(suite))
+ self.write_config("\n".join(features))
+
+ recipe = "binutils-cross-testsuite"
+ bb_vars = get_bb_vars(["B", "TARGET_SYS", "T"], recipe)
+ builddir, target_sys, tdir = bb_vars["B"], bb_vars["TARGET_SYS"], bb_vars["T"]
+
+ bitbake("{0} -c check".format(recipe))
+
+ ptestsuite = "binutils-{}".format(suite) if suite != "binutils" else suite
+ self.tc.extraresults["ptestresult.sections"][ptestsuite] = {}
+
+ sumspath = os.path.join(builddir, suite, "{0}.sum".format(suite))
+ if not os.path.exists(sumspath):
+ sumspath = os.path.join(builddir, suite, "testsuite", "{0}.sum".format(suite))
+
+ failed = 0
+ with open(sumspath, "r") as f:
+ for test, result in parse_values(f):
+ self.tc.extraresults["ptestresult.{}.{}".format(ptestsuite, test)] = {"status" : result}
+ if result == "FAIL":
+ self.logger.info("failed: '{}'".format(test))
+ failed += 1
+
+ self.assertEqual(failed, 0)
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 11/12] oeqa/selftest/gcc: Create selftest case for gcc test suite
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (3 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 03/12] gcc-runtime: Add do_check task for executing gcc test suites Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 08/12] oeqa/core: Rework OETestTag and remove unused OETestFilter Nathan Rossi
` (6 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Create a oeqa selftest test case to execute the gcc test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GccSelfTestSystemEmulated test case
handles setup of the target image as well as execution with runqemu.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changed in v2:
- Change to using only gcc-runtime recipe and associated check targets
- Remove gxx/g++ testcase, this is part of the "check-gcc" target
- Populate test results as ptestresults
- Rename BUILD_TEST_* to TOOLCHAIN_TEST_*
- Mark tests with 'machine' tag
---
meta/lib/oeqa/selftest/cases/gcc.py | 111 ++++++++++++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/gcc.py
diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
new file mode 100644
index 0000000000..0ad6e21d6b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -0,0 +1,111 @@
+# SPDX-License-Identifier: MIT
+import os
+from oeqa.core.decorator import OETestTag
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runqemu, Command
+
+def parse_values(content):
+ for i in content:
+ for v in ["PASS", "FAIL", "XPASS", "XFAIL", "UNRESOLVED", "UNSUPPORTED", "UNTESTED", "ERROR", "WARNING"]:
+ if i.startswith(v + ": "):
+ yield i[len(v) + 2:].strip(), v
+ break
+
+@OETestTag("machine")
+class GccSelfTest(OESelftestTestCase):
+ @classmethod
+ def setUpClass(cls):
+ super().setUpClass()
+ if not hasattr(cls.tc, "extraresults"):
+ cls.tc.extraresults = {}
+
+ if "ptestresult.sections" not in cls.tc.extraresults:
+ cls.tc.extraresults["ptestresult.sections"] = {}
+
+ def gcc_runtime_check_skip(self, suite):
+ targets = get_bb_var("RUNTIMETARGET", "gcc-runtime").split()
+ if suite not in targets:
+ self.skipTest("Target does not use {0}".format(suite))
+
+ def test_cross_gcc(self):
+ self.gcc_run_check("gcc", "g++")
+
+ def test_libatomic(self):
+ self.gcc_run_check("libatomic")
+
+ def test_libgomp(self):
+ self.gcc_run_check("libgomp")
+
+ def test_libstdcxx(self):
+ self.gcc_run_check("libstdc++-v3")
+
+ def test_libssp(self):
+ self.gcc_runtime_check_skip("libssp")
+ self.gcc_run_check("libssp")
+
+ def test_libitm(self):
+ self.gcc_runtime_check_skip("libitm")
+ self.gcc_run_check("libitm")
+
+ def gcc_run_check(self, *suites, ssh = None):
+ targets = set()
+ for s in suites:
+ if s in ["gcc", "g++"]:
+ targets.add("check-gcc")
+ else:
+ targets.add("check-target-{}".format(s))
+
+ # configure ssh target
+ features = []
+ features.append('MAKE_CHECK_TARGETS = "{0}"'.format(" ".join(targets)))
+ if ssh is not None:
+ features.append('TOOLCHAIN_TEST_TARGET = "ssh"')
+ features.append('TOOLCHAIN_TEST_HOST = "{0}"'.format(ssh))
+ features.append('TOOLCHAIN_TEST_HOST_USER = "root"')
+ features.append('TOOLCHAIN_TEST_HOST_PORT = "22"')
+ self.write_config("\n".join(features))
+
+ recipe = "gcc-runtime"
+ bitbake("{} -c check".format(recipe))
+
+ bb_vars = get_bb_vars(["B", "TARGET_SYS"], recipe)
+ builddir, target_sys = bb_vars["B"], bb_vars["TARGET_SYS"]
+
+ failed = 0
+ for suite in suites:
+ sumspath = os.path.join(builddir, "gcc", "testsuite", suite, "{0}.sum".format(suite))
+ if not os.path.exists(sumspath): # check in target dirs
+ sumspath = os.path.join(builddir, target_sys, suite, "testsuite", "{0}.sum".format(suite))
+ if not os.path.exists(sumspath): # handle libstdc++-v3 -> libstdc++
+ sumspath = os.path.join(builddir, target_sys, suite, "testsuite", "{0}.sum".format(suite.split("-")[0]))
+
+ ptestsuite = "gcc-{}".format(suite) if suite != "gcc" else suite
+ self.tc.extraresults["ptestresult.sections"][ptestsuite] = {}
+ with open(sumspath, "r") as f:
+ for test, result in parse_values(f):
+ self.tc.extraresults["ptestresult.{}.{}".format(ptestsuite, test)] = {"status" : result}
+ if result == "FAIL":
+ self.logger.info("failed: '{}'".format(test))
+ failed += 1
+
+ self.assertEqual(failed, 0)
+
+class GccSelfTestSystemEmulated(GccSelfTest):
+ default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
+
+ def gcc_run_check(self, *args, **kwargs):
+ # build core-image-minimal with required packages
+ features = []
+ features.append('IMAGE_FEATURES += "ssh-server-openssh"')
+ features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(self.default_installed_packages)))
+ self.write_config("\n".join(features))
+ bitbake("core-image-minimal")
+
+ # wrap the execution with a qemu instance
+ with runqemu("core-image-minimal", runqemuparams = "nographic") as qemu:
+ # validate that SSH is working
+ status, _ = qemu.run("uname")
+ self.assertEqual(status, 0)
+
+ return super().gcc_run_check(*args, **kwargs, ssh = qemu.ip)
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 12/12] oeqa/selftest/glibc: Create selftest case for glibc test suite
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
` (6 preceding siblings ...)
2019-09-03 16:56 ` [PATCH v2 07/12] oeqa/core/tests: Fix test_data module tests Nathan Rossi
@ 2019-09-03 16:56 ` Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 10/12] oeqa/selftest/binutils: Create selftest case for binutils " Nathan Rossi
` (3 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Nathan Rossi @ 2019-09-03 16:56 UTC (permalink / raw)
To: openembedded-core
Create a oeqa selftest test case to execute the glibc test suite and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GlibcSelfTestSystemEmulated test case
handles setup of the target image, setup of and NFS server as well as
execution with runqemu.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
---
Changes in v2:
- Populate test results as ptestresults
- Rename BUILD_TEST_* to TOOLCHAIN_TEST_*
- Mark tests with 'machine' tag
---
meta/lib/oeqa/selftest/cases/glibc.py | 97 +++++++++++++++++++++++++++++++++++
1 file changed, 97 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/glibc.py
diff --git a/meta/lib/oeqa/selftest/cases/glibc.py b/meta/lib/oeqa/selftest/cases/glibc.py
new file mode 100644
index 0000000000..f836367094
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/glibc.py
@@ -0,0 +1,97 @@
+# SPDX-License-Identifier: MIT
+import os
+import contextlib
+from oeqa.core.decorator import OETestTag
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runqemu, Command
+from oeqa.utils.nfs import unfs_server
+
+def parse_values(content):
+ for i in content:
+ for v in ["PASS", "FAIL", "XPASS", "XFAIL", "UNRESOLVED", "UNSUPPORTED", "UNTESTED", "ERROR", "WARNING"]:
+ if i.startswith(v + ": "):
+ yield i[len(v) + 2:].strip(), v
+ break
+
+@OETestTag("machine")
+class GlibcSelfTest(OESelftestTestCase):
+ @classmethod
+ def setUpClass(cls):
+ super().setUpClass()
+ if not hasattr(cls.tc, "extraresults"):
+ cls.tc.extraresults = {}
+
+ if "ptestresult.sections" not in cls.tc.extraresults:
+ cls.tc.extraresults["ptestresult.sections"] = {}
+
+ def test_glibc(self):
+ self.glibc_run_check()
+
+ def glibc_run_check(self, ssh = None):
+ # configure ssh target
+ features = []
+ if ssh is not None:
+ features.append('TOOLCHAIN_TEST_TARGET = "ssh"')
+ features.append('TOOLCHAIN_TEST_HOST = "{0}"'.format(ssh))
+ features.append('TOOLCHAIN_TEST_HOST_USER = "root"')
+ features.append('TOOLCHAIN_TEST_HOST_PORT = "22"')
+ # force single threaded test execution
+ features.append('EGLIBCPARALLELISM_task-check_pn-glibc-testsuite = "PARALLELMFLAGS="-j1""')
+ self.write_config("\n".join(features))
+
+ bitbake("glibc-testsuite -c check")
+
+ builddir = get_bb_var("B", "glibc-testsuite")
+
+ failed = 0
+ self.tc.extraresults["ptestresult.sections"]["glibc"] = {}
+ with open(os.path.join(builddir, "tests.sum"), "r") as f:
+ for test, result in parse_values(f):
+ self.tc.extraresults["ptestresult.glibc.{}".format(test)] = {"status" : result}
+ if result == "FAIL":
+ self.logger.info("failed: '{}'".format(test))
+ failed += 1
+ self.assertEqual(failed, 0)
+
+class GlibcSelfTestSystemEmulated(GlibcSelfTest):
+ default_installed_packages = [
+ "glibc-charmaps",
+ "libgcc",
+ "libstdc++",
+ "libatomic",
+ "libgomp",
+ "python3",
+ "python3-pexpect",
+ "nfs-utils",
+ ]
+
+ def glibc_run_check(self):
+ with contextlib.ExitStack() as s:
+ # use the base work dir, as the nfs mount, since the recipe directory may not exist
+ tmpdir = get_bb_var("BASE_WORKDIR")
+ nfsport, mountport = s.enter_context(unfs_server(tmpdir))
+
+ # build core-image-minimal with required packages
+ features = []
+ features.append('IMAGE_FEATURES += "ssh-server-openssh"')
+ features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(self.default_installed_packages)))
+ self.write_config("\n".join(features))
+ bitbake("core-image-minimal")
+
+ # start runqemu
+ qemu = s.enter_context(runqemu("core-image-minimal", runqemuparams = "nographic"))
+
+ # validate that SSH is working
+ status, _ = qemu.run("uname")
+ self.assertEqual(status, 0)
+
+ # setup nfs mount
+ if qemu.run("mkdir -p \"{0}\"".format(tmpdir))[0] != 0:
+ raise Exception("Failed to setup NFS mount directory on target")
+ mountcmd = "mount -o noac,nfsvers=3,port={0},udp,mountport={1} \"{2}:{3}\" \"{3}\"".format(nfsport, mountport, qemu.server_ip, tmpdir)
+ status, output = qemu.run(mountcmd)
+ if status != 0:
+ raise Exception("Failed to setup NFS mount on target ({})".format(repr(output)))
+
+ super().glibc_run_check(ssh = qemu.ip)
+
---
2.23.0.rc1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc test suite
2019-09-03 16:56 ` [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc " Nathan Rossi
@ 2019-09-04 15:35 ` Khem Raj
0 siblings, 0 replies; 14+ messages in thread
From: Khem Raj @ 2019-09-04 15:35 UTC (permalink / raw)
To: Nathan Rossi; +Cc: Patches and discussions about the oe-core layer
https://errors.yoctoproject.org/Errors/Details/268225/
so I guess we need TOOLCHAIN = "gcc" set for this
On Tue, Sep 3, 2019 at 9:57 AM Nathan Rossi <nathan@nathanrossi.com> wrote:
>
> A recipe needs to be created for the test suite due to the dependency
> chain between libgcc -> glibc -> libgcc-initial, and the requirements of
> the test suite to have libgcc for compilation and execution.
>
> The glibc test suite does not use dejagnu like the gcc test suites do.
> Instead a test wrapper script is used along with the assumed dependency
> of having the same filesystem available on build host and target. For
> qemu linux-user the same filesystem is inherently available, for remote
> targets NFS is used. Separate test wrapper scripts are created for qemu
> linux-user or ssh targets, with the same TOOLCHAIN_TEST_* variables used for
> configuration.
>
> Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
> ---
> Changes in v2:
> - Renamed BUILD_TEST_* to TOOLCHAIN_TEST_*
> ---
> meta/recipes-core/glibc/glibc-testsuite_2.30.bb | 51 +++++++++++++++++
> meta/recipes-core/glibc/glibc/check-test-wrapper | 71 ++++++++++++++++++++++++
> 2 files changed, 122 insertions(+)
> create mode 100644 meta/recipes-core/glibc/glibc-testsuite_2.30.bb
> create mode 100644 meta/recipes-core/glibc/glibc/check-test-wrapper
>
> diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.30.bb b/meta/recipes-core/glibc/glibc-testsuite_2.30.bb
> new file mode 100644
> index 0000000000..88764d9e2b
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc-testsuite_2.30.bb
> @@ -0,0 +1,51 @@
> +require glibc_${PV}.bb
> +
> +# handle PN differences
> +FILESEXTRAPATHS_prepend := "${THISDIR}/glibc:"
> +
> +# strip provides
> +PROVIDES = ""
> +# setup depends
> +INHIBIT_DEFAULT_DEPS = ""
> +
> +DEPENDS += "glibc-locale libgcc gcc-runtime"
> +
> +# remove the initial depends
> +DEPENDS_remove = "libgcc-initial"
> +
> +inherit qemu
> +
> +SRC_URI += "file://check-test-wrapper"
> +
> +DEPENDS += "${@'qemu-native' if d.getVar('TOOLCHAIN_TEST_TARGET') == 'user' else ''}"
> +
> +TOOLCHAIN_TEST_TARGET ??= "user"
> +TOOLCHAIN_TEST_HOST ??= "localhost"
> +TOOLCHAIN_TEST_HOST_USER ??= "root"
> +TOOLCHAIN_TEST_HOST_PORT ??= "2222"
> +
> +do_check[dirs] += "${B}"
> +do_check[nostamp] = "1"
> +do_check () {
> + chmod 0755 ${WORKDIR}/check-test-wrapper
> +
> + # clean out previous test results
> + oe_runmake tests-clean
> + # makefiles don't clean entirely (and also sometimes fails due to too many args)
> + find ${B} -type f -name "*.out" -delete
> + find ${B} -type f -name "*.test-result" -delete
> + find ${B}/catgets -name "*.cat" -delete
> + find ${B}/conform -name "symlist-*" -delete
> + [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
> +
> + oe_runmake -i \
> + QEMU_SYSROOT="${RECIPE_SYSROOT}" \
> + QEMU_OPTIONS="${@qemu_target_binary(d)} ${QEMU_OPTIONS}" \
> + SSH_HOST="${TOOLCHAIN_TEST_HOST}" \
> + SSH_HOST_USER="${TOOLCHAIN_TEST_HOST_USER}" \
> + SSH_HOST_PORT="${TOOLCHAIN_TEST_HOST_PORT}" \
> + test-wrapper="${WORKDIR}/check-test-wrapper ${TOOLCHAIN_TEST_TARGET}" \
> + check
> +}
> +addtask do_check after do_compile
> +
> diff --git a/meta/recipes-core/glibc/glibc/check-test-wrapper b/meta/recipes-core/glibc/glibc/check-test-wrapper
> new file mode 100644
> index 0000000000..f8e04e02d2
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc/check-test-wrapper
> @@ -0,0 +1,71 @@
> +#!/usr/bin/env python3
> +import sys
> +import os
> +import subprocess
> +
> +env = os.environ.copy()
> +args = sys.argv[1:]
> +targettype = args.pop(0)
> +
> +if targettype == "user":
> + qemuargs = os.environ.get("QEMU_OPTIONS", "").split()
> + if not os.path.exists(qemuargs[0]):
> + # ensure qemu args has a valid absolute path
> + for i in os.environ.get("PATH", "").split(":"):
> + if os.path.exists(os.path.join(i, qemuargs[0])):
> + qemuargs[0] = os.path.join(i, qemuargs[0])
> + break
> + sysroot = os.environ.get("QEMU_SYSROOT", None)
> + if not sysroot:
> + sys.exit(-1)
> + libpaths = [sysroot + "/usr/lib", sysroot + "/lib"]
> +
> + if args[0] == "env":
> + args.pop(0)
> + if len(args) == 0:
> + args = ["env"]
> + else:
> + # process options
> + while args[0].startswith("-"):
> + opt = args.pop(0).lstrip("-")
> + if "i" in opt:
> + env.clear()
> + # process environment vars
> + while "=" in args[0]:
> + key, val = args.pop(0).split("=", 1)
> + if key == "LD_LIBRARY_PATH":
> + libpaths += val.split(":")
> + else:
> + env[key] = val
> + if args[0] == "cp":
> + # ignore copies, the filesystem is the same
> + sys.exit(0)
> +
> + qemuargs += ["-L", sysroot]
> + qemuargs += ["-E", "LD_LIBRARY_PATH={}".format(":".join(libpaths))]
> + command = qemuargs + args
> +elif targettype == "ssh":
> + host = os.environ.get("SSH_HOST", None)
> + user = os.environ.get("SSH_HOST_USER", None)
> + port = os.environ.get("SSH_HOST_PORT", None)
> +
> + command = ["ssh", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no"]
> + if port:
> + command += ["-p", str(port)]
> + if not host:
> + sys.exit(-1)
> + command += ["{}@{}".format(user, host) if user else host]
> +
> + # wrap and replace quotes for correct transformation on ssh
> + wrapped = " ".join(["'{0}'".format(i.replace("'", r"'\''")) for i in ["cd", os.getcwd()]]) + "; "
> + wrapped += " ".join(["'{0}'".format(i.replace("'", r"'\''")) for i in args])
> + command += ["sh", "-c", "\"{}\"".format(wrapped)]
> +else:
> + sys.exit(-1)
> +
> +try:
> + r = subprocess.run(command, timeout = 1800, env = env)
> + sys.exit(r.returncode)
> +except subprocess.TimeoutExpired:
> + sys.exit(-1)
> +
> ---
> 2.23.0.rc1
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-09-04 15:35 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-03 16:56 [PATCH v2 00/12] Add gnu testsuite execution for OEQA Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 01/12] dejagnu: Add dejagnu for binutils/gcc test suites Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 09/12] oeqa/selftest: Add test run filtering based on test tags Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 04/12] gcc-common.inc: Process staging fixme with correct target/native sysroot Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 03/12] gcc-runtime: Add do_check task for executing gcc test suites Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 11/12] oeqa/selftest/gcc: Create selftest case for gcc test suite Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 08/12] oeqa/core: Rework OETestTag and remove unused OETestFilter Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 07/12] oeqa/core/tests: Fix test_data module tests Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 12/12] oeqa/selftest/glibc: Create selftest case for glibc test suite Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 10/12] oeqa/selftest/binutils: Create selftest case for binutils " Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 05/12] glibc-testsuite: Create a recipe to implement glibc " Nathan Rossi
2019-09-04 15:35 ` Khem Raj
2019-09-03 16:56 ` [PATCH v2 06/12] oeqa/core/tests: Skip test_fail_duplicated_module Nathan Rossi
2019-09-03 16:56 ` [PATCH v2 02/12] binutils-cross-testsuite: Create recipe for test suite execution Nathan Rossi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.