* [PATCH v3 00/22] Add Extensible SDK test suite
@ 2016-02-04 22:29 Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 01/22] testimage: Modularize helper functions for get test lists Aníbal Limón
` (22 more replies)
0 siblings, 23 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Summary of changes,
- Add new class called testsdk that now have the tests for SDK and eSDK.
- TestContext -> {Image, SDK, SDKExt}: Remove all duplicate code inside
testimage and testsdk class move into oeqa/oetest and create class per
type of Test.
- Extensible SDK fixes usage with proxies, when eSDK is configuring it
executes setscene tasks that needs network acces if was build with sstate
mirrors.
Testing was made building/running SDK and eSDK for core-image-minimal and core-image-sato,
also testimage was run in core-image-minimal and core-image-sato with QemuRemote and SimpleRemote
to ensure that nothing was break.
This patchset DEPENDS on bitbake change to export proxies and NEEDS to be MERGED before, see:
bb/fetch2: Move export_proxies function from wget to utils.
These v3 is the same of v2 on code only fixes issues of squashing and one for don't make reverts
in the patch series.
The following changes since commit b33e440cc6fbd703e8045d94b806790343e72eb6:
libical: Work around hardcoded paths in pkgconfig file (2016-02-04 13:09:56 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib alimon/esdk_testsuite_v3
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/esdk_testsuite_v3
Aníbal Limón (22):
testimage: Modularize helper functions for get test lists.
classes/testsdk: Add new class testsdk.
classes/testimage: Add defeault inherit for testsdk.
populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME.
get_test_suites: Add sdkext type for load test suites.
classes/testsdk: Add testsdkext task only install.
bb/fetch2: Move export_proxies function from wget to utils.
classes/testsdk: Add call to export_proxies on testsdkext.
toolchain-shar-extract.sh: Add proxy variable to new env.
testimage/testsdk: Modularize TestContext.
testimage/testsdk: Move get test suites routine inside TestContext.
oetest.py/TestContext: Move loadTests and runTests inside it.
classes/testsdk: Add function run_test_context
oeqa/oetest: oeSDKTest when run a command redirect env output to null
classes/testsdk: Add compatibility SDK testsuite to eSDK
testsdkext: Add skeleton for support Extensible SDK tests.
classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and
SDK_EXT_HOST_MANIFEST
oeqa/oetest: Fix compatibility SDK tests using eSDK.
oeqa/sdkext: Add devtool basic tests for eSDK.
oeqa/sdkext/devtool.py: Add location test to ensure that devtool is
the eSDK one.
classes/testsdk: Add help information on how to run tests.
classes/testimage: Fix exportTests function.
bitbake/lib/bb/fetch2/wget.py | 17 +-
bitbake/lib/bb/utils.py | 19 ++
meta/classes/populate_sdk_ext.bbclass | 6 +-
meta/classes/testimage.bbclass | 219 +----------------
meta/classes/testsdk.bbclass | 144 +++++++++++
meta/files/toolchain-shar-extract.sh | 4 +-
meta/lib/oeqa/oetest.py | 388 ++++++++++++++++++++++--------
meta/lib/oeqa/sdkext/__init__.py | 3 +
meta/lib/oeqa/sdkext/devtool.py | 32 +++
meta/lib/oeqa/sdkext/files/myapp/Makefile | 10 +
meta/lib/oeqa/sdkext/files/myapp/myapp.c | 9 +
11 files changed, 518 insertions(+), 333 deletions(-)
create mode 100644 meta/classes/testsdk.bbclass
create mode 100644 meta/lib/oeqa/sdkext/__init__.py
create mode 100644 meta/lib/oeqa/sdkext/devtool.py
create mode 100644 meta/lib/oeqa/sdkext/files/myapp/Makefile
create mode 100644 meta/lib/oeqa/sdkext/files/myapp/myapp.c
--
2.1.4
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v3 01/22] testimage: Modularize helper functions for get test lists.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 02/22] classes/testsdk: Add new class testsdk Aníbal Limón
` (21 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Test lists functions can be used in other parts so modularize it and
move to oeqa/oetest.py library.
Testimage class was updated to meet the new sign of the functions.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testimage.bbclass | 78 +++---------------------------------------
meta/lib/oeqa/oetest.py | 74 +++++++++++++++++++++++++++++++++++++++
2 files changed, 78 insertions(+), 74 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 37af46f..a4026d3 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -106,74 +106,6 @@ do_testsdk[nostamp] = "1"
do_testsdk[depends] += "${TESTIMAGEDEPENDS}"
do_testsdk[lockfiles] += "${TESTIMAGELOCK}"
-# get testcase list from specified file
-# if path is a relative path, then relative to build/conf/
-def read_testlist(d, fpath):
- if not os.path.isabs(fpath):
- builddir = d.getVar("TOPDIR", True)
- fpath = os.path.join(builddir, "conf", fpath)
- if not os.path.exists(fpath):
- bb.fatal("No such manifest file: ", fpath)
- tcs = []
- for line in open(fpath).readlines():
- line = line.strip()
- if line and not line.startswith("#"):
- tcs.append(line)
- return " ".join(tcs)
-
-def get_tests_list(d, type="runtime"):
- testsuites = []
- testslist = []
- manifests = d.getVar("TEST_SUITES_MANIFEST", True)
- if manifests is not None:
- manifests = manifests.split()
- for manifest in manifests:
- testsuites.extend(read_testlist(d, manifest).split())
- else:
- testsuites = d.getVar("TEST_SUITES", True).split()
- if type == "sdk":
- testsuites = (d.getVar("TEST_SUITES_SDK", True) or "auto").split()
- bbpath = d.getVar("BBPATH", True).split(':')
-
- # This relies on lib/ under each directory in BBPATH being added to sys.path
- # (as done by default in base.bbclass)
- for testname in testsuites:
- if testname != "auto":
- if testname.startswith("oeqa."):
- testslist.append(testname)
- continue
- found = False
- for p in bbpath:
- if os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname + '.py')):
- testslist.append("oeqa." + type + "." + testname)
- found = True
- break
- elif os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname.split(".")[0] + '.py')):
- testslist.append("oeqa." + type + "." + testname)
- found = True
- break
- if not found:
- bb.fatal('Test %s specified in TEST_SUITES could not be found in lib/oeqa/runtime under BBPATH' % testname)
-
- if "auto" in testsuites:
- def add_auto_list(path):
- if not os.path.exists(os.path.join(path, '__init__.py')):
- bb.fatal('Tests directory %s exists but is missing __init__.py' % path)
- files = sorted([f for f in os.listdir(path) if f.endswith('.py') and not f.startswith('_')])
- for f in files:
- module = 'oeqa.' + type + '.' + f[:-3]
- if module not in testslist:
- testslist.append(module)
-
- for p in bbpath:
- testpath = os.path.join(p, 'lib', 'oeqa', type)
- bb.debug(2, 'Searching for tests in %s' % testpath)
- if os.path.exists(testpath):
- add_auto_list(testpath)
-
- return testslist
-
-
def exportTests(d,tc):
import json
import shutil
@@ -272,7 +204,7 @@ def testimage_main(d):
import oeqa.runtime
import time
import signal
- from oeqa.oetest import loadTests, runTests
+ from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
from oeqa.targetcontrol import get_target_controller
from oeqa.utils.dump import get_host_dumper
@@ -286,7 +218,7 @@ def testimage_main(d):
# tests in TEST_SUITES become required tests
# they won't be skipped even if they aren't suitable for a image (like xorg for minimal)
# testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(d)
+ testslist = get_tests_list(get_test_suites(d), d.getVar("BBPATH", True).split(':'))
testsrequired = [t for t in d.getVar("TEST_SUITES", True).split() if t != "auto"]
tagexp = d.getVar("TEST_SUITES_TAGS", True)
@@ -368,7 +300,6 @@ def testimage_main(d):
testimage_main[vardepsexclude] =+ "BB_ORIGENV"
-
def testsdk_main(d):
import unittest
import os
@@ -377,7 +308,7 @@ def testsdk_main(d):
import oeqa.sdk
import time
import subprocess
- from oeqa.oetest import loadTests, runTests
+ from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
@@ -385,7 +316,7 @@ def testsdk_main(d):
# tests in TEST_SUITES become required tests
# they won't be skipped even if they aren't suitable.
# testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(d, "sdk")
+ testslist = get_tests_list(get_test_suites(d, "sdk"), d.getVar("BBPATH", True).split(':'), "sdk")
testsrequired = [t for t in (d.getVar("TEST_SUITES_SDK", True) or "auto").split() if t != "auto"]
tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh")
@@ -457,4 +388,3 @@ def testsdk_main(d):
bb.utils.remove(sdktestdir, True)
testsdk_main[vardepsexclude] =+ "BB_ORIGENV"
-
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 6f9edec..18b2209 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -250,3 +250,77 @@ def skipModuleUnless(cond, reason):
if not cond:
skipModule(reason, 3)
+
+# get testcase list from specified file
+# if path is a relative path, then relative to build/conf/
+def read_testlist(fpath, builddir):
+ if not os.path.isabs(fpath):
+ fpath = os.path.join(builddir, "conf", fpath)
+ if not os.path.exists(fpath):
+ bb.fatal("No such manifest file: ", fpath)
+ tcs = []
+ for line in open(fpath).readlines():
+ line = line.strip()
+ if line and not line.startswith("#"):
+ tcs.append(line)
+ return " ".join(tcs)
+
+# get test suites, returns test suites based on d variables
+def get_test_suites(d, type='runtime'):
+ testsuites = []
+
+ if type == "sdk":
+ testsuites = (d.getVar("TEST_SUITES_SDK", True) or "auto").split()
+ else:
+ manifests = (d.getVar("TEST_SUITES_MANIFEST", True) or '').split()
+ if manifests:
+ for manifest in manifests:
+ testsuites.extend(read_testlist(manifest,
+ d.getVar("TOPDIR", True)).split())
+
+ else:
+ testsuites = d.getVar("TEST_SUITES", True).split()
+
+ return testsuites
+
+# return test list by type also filter if TEST_SUITES is specified
+def get_tests_list(testsuites, bbpath, type="runtime"):
+ testslist = []
+
+ # This relies on lib/ under each directory in BBPATH being added to sys.path
+ # (as done by default in base.bbclass)
+ for testname in testsuites:
+ if testname != "auto":
+ if testname.startswith("oeqa."):
+ testslist.append(testname)
+ continue
+ found = False
+ for p in bbpath:
+ if os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname + '.py')):
+ testslist.append("oeqa." + type + "." + testname)
+ found = True
+ break
+ elif os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname.split(".")[0] + '.py')):
+ testslist.append("oeqa." + type + "." + testname)
+ found = True
+ break
+ if not found:
+ bb.fatal('Test %s specified in TEST_SUITES could not be found in lib/oeqa/runtime under BBPATH' % testname)
+
+ if "auto" in testsuites:
+ def add_auto_list(path):
+ if not os.path.exists(os.path.join(path, '__init__.py')):
+ bb.fatal('Tests directory %s exists but is missing __init__.py' % path)
+ files = sorted([f for f in os.listdir(path) if f.endswith('.py') and not f.startswith('_')])
+ for f in files:
+ module = 'oeqa.' + type + '.' + f[:-3]
+ if module not in testslist:
+ testslist.append(module)
+
+ for p in bbpath:
+ testpath = os.path.join(p, 'lib', 'oeqa', type)
+ bb.debug(2, 'Searching for tests in %s' % testpath)
+ if os.path.exists(testpath):
+ add_auto_list(testpath)
+
+ return testslist
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 02/22] classes/testsdk: Add new class testsdk.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 01/22] testimage: Modularize helper functions for get test lists Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 03/22] classes/testimage: Add defeault inherit for testsdk Aníbal Limón
` (20 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Moves all the testsdk code from testimage in order to have it's own
class because new tests will be added for extensible SDK.
The old paths for store logs "${WORKDIR}/testimage" and sdk
"${WORKDIR}/testimage-sdk" was maintained for compatibility may be
change to point testsdk after review the codebase.
The dependency of QEMU was removed because isn't needed.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testimage.bbclass | 98 ----------------------------------------
meta/classes/testsdk.bbclass | 100 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 100 insertions(+), 98 deletions(-)
create mode 100644 meta/classes/testsdk.bbclass
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index a4026d3..ce71209 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -98,14 +98,6 @@ do_testimage[nostamp] = "1"
do_testimage[depends] += "${TESTIMAGEDEPENDS}"
do_testimage[lockfiles] += "${TESTIMAGELOCK}"
-python do_testsdk() {
- testsdk_main(d)
-}
-addtask testsdk
-do_testsdk[nostamp] = "1"
-do_testsdk[depends] += "${TESTIMAGEDEPENDS}"
-do_testsdk[lockfiles] += "${TESTIMAGELOCK}"
-
def exportTests(d,tc):
import json
import shutil
@@ -197,7 +189,6 @@ def exportTests(d,tc):
bb.plain("Exported tests to: %s" % exportpath)
-
def testimage_main(d):
import unittest
import os
@@ -299,92 +290,3 @@ def testimage_main(d):
target.stop()
testimage_main[vardepsexclude] =+ "BB_ORIGENV"
-
-def testsdk_main(d):
- import unittest
- import os
- import glob
- import oeqa.runtime
- import oeqa.sdk
- import time
- import subprocess
- from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
-
- pn = d.getVar("PN", True)
- bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
-
- # tests in TEST_SUITES become required tests
- # they won't be skipped even if they aren't suitable.
- # testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(get_test_suites(d, "sdk"), d.getVar("BBPATH", True).split(':'), "sdk")
- testsrequired = [t for t in (d.getVar("TEST_SUITES_SDK", True) or "auto").split() if t != "auto"]
-
- tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh")
- if not os.path.exists(tcname):
- bb.fatal("The toolchain is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .")
-
- class TestContext(object):
- def __init__(self):
- self.d = d
- self.testslist = testslist
- self.testsrequired = testsrequired
- self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
- self.sdktestdir = sdktestdir
- self.sdkenv = sdkenv
- self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
- self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
- manifest = d.getVar("SDK_TARGET_MANIFEST", True)
- try:
- with open(manifest) as f:
- self.pkgmanifest = f.read()
- except IOError as e:
- bb.fatal("No package manifest file found. Did you build the sdk image?\n%s" % e)
- hostmanifest = d.getVar("SDK_HOST_MANIFEST", True)
- try:
- with open(hostmanifest) as f:
- self.hostpkgmanifest = f.read()
- except IOError as e:
- bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
-
- sdktestdir = d.expand("${WORKDIR}/testimage-sdk/")
- bb.utils.remove(sdktestdir, True)
- bb.utils.mkdirhier(sdktestdir)
- try:
- subprocess.check_output("cd %s; %s <<EOF\n./tc\nY\nEOF" % (sdktestdir, tcname), shell=True)
- except subprocess.CalledProcessError as e:
- bb.fatal("Couldn't install the SDK:\n%s" % e.output)
-
- try:
- targets = glob.glob(d.expand(sdktestdir + "/tc/environment-setup-*"))
- bb.warn(str(targets))
- for sdkenv in targets:
- bb.plain("Testing %s" % sdkenv)
- # test context
- tc = TestContext()
-
- # this is a dummy load of tests
- # we are doing that to find compile errors in the tests themselves
- # before booting the image
- try:
- loadTests(tc, "sdk")
- except Exception as e:
- import traceback
- bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
-
-
- starttime = time.time()
- result = runTests(tc, "sdk")
- stoptime = time.time()
- if result.wasSuccessful():
- bb.plain("%s SDK(%s):%s - Ran %d test%s in %.3fs" % (pn, os.path.basename(tcname), os.path.basename(sdkenv),result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
- msg = "%s - OK - All required tests passed" % pn
- skipped = len(result.skipped)
- if skipped:
- msg += " (skipped=%d)" % skipped
- bb.plain(msg)
- else:
- raise bb.build.FuncFailed("%s - FAILED - check the task log and the commands log" % pn )
- finally:
- bb.utils.remove(sdktestdir, True)
-
-testsdk_main[vardepsexclude] =+ "BB_ORIGENV"
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
new file mode 100644
index 0000000..42c4913
--- /dev/null
+++ b/meta/classes/testsdk.bbclass
@@ -0,0 +1,100 @@
+# Copyright (C) 2013 - 2016 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+
+TEST_LOG_DIR ?= "${WORKDIR}/testimage"
+TESTSDKLOCK = "${TMPDIR}/testsdk.lock"
+
+def testsdk_main(d):
+ import unittest
+ import os
+ import glob
+ import oeqa.runtime
+ import oeqa.sdk
+ import time
+ import subprocess
+ from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+
+ pn = d.getVar("PN", True)
+ bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
+
+ # tests in TEST_SUITES become required tests
+ # they won't be skipped even if they aren't suitable.
+ # testslist is what we'll actually pass to the unittest loader
+ testslist = get_tests_list(get_test_suites(d, "sdk"), d.getVar("BBPATH", True).split(':'), "sdk")
+ testsrequired = [t for t in (d.getVar("TEST_SUITES_SDK", True) or "auto").split() if t != "auto"]
+
+ tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh")
+ if not os.path.exists(tcname):
+ bb.fatal("The toolchain is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .")
+
+ class TestContext(object):
+ def __init__(self):
+ self.d = d
+ self.testslist = testslist
+ self.testsrequired = testsrequired
+ self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
+ self.sdktestdir = sdktestdir
+ self.sdkenv = sdkenv
+ self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
+ self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
+ manifest = d.getVar("SDK_TARGET_MANIFEST", True)
+ try:
+ with open(manifest) as f:
+ self.pkgmanifest = f.read()
+ except IOError as e:
+ bb.fatal("No package manifest file found. Did you build the sdk image?\n%s" % e)
+ hostmanifest = d.getVar("SDK_HOST_MANIFEST", True)
+ try:
+ with open(hostmanifest) as f:
+ self.hostpkgmanifest = f.read()
+ except IOError as e:
+ bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
+
+ sdktestdir = d.expand("${WORKDIR}/testimage-sdk/")
+ bb.utils.remove(sdktestdir, True)
+ bb.utils.mkdirhier(sdktestdir)
+ try:
+ subprocess.check_output("cd %s; %s <<EOF\n./tc\nY\nEOF" % (sdktestdir, tcname), shell=True)
+ except subprocess.CalledProcessError as e:
+ bb.fatal("Couldn't install the SDK:\n%s" % e.output)
+
+ try:
+ targets = glob.glob(d.expand(sdktestdir + "/tc/environment-setup-*"))
+ for sdkenv in targets:
+ bb.plain("Testing %s" % sdkenv)
+ # test context
+ tc = TestContext()
+
+ # this is a dummy load of tests
+ # we are doing that to find compile errors in the tests themselves
+ # before booting the image
+ try:
+ loadTests(tc, "sdk")
+ except Exception as e:
+ import traceback
+ bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
+
+ starttime = time.time()
+ result = runTests(tc, "sdk")
+ stoptime = time.time()
+ if result.wasSuccessful():
+ bb.plain("%s SDK(%s):%s - Ran %d test%s in %.3fs" % (pn, os.path.basename(tcname), os.path.basename(sdkenv),result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
+ msg = "%s - OK - All required tests passed" % pn
+ skipped = len(result.skipped)
+ if skipped:
+ msg += " (skipped=%d)" % skipped
+ bb.plain(msg)
+ else:
+ raise bb.build.FuncFailed("%s - FAILED - check the task log and the commands log" % pn )
+ finally:
+ bb.utils.remove(sdktestdir, True)
+
+testsdk_main[vardepsexclude] =+ "BB_ORIGENV"
+
+python do_testsdk() {
+ testsdk_main(d)
+}
+addtask testsdk
+do_testsdk[nostamp] = "1"
+do_testsdk[lockfiles] += "${TESTSDKLOCK}"
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 03/22] classes/testimage: Add defeault inherit for testsdk.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 01/22] testimage: Modularize helper functions for get test lists Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 02/22] classes/testsdk: Add new class testsdk Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 04/22] populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME Aníbal Limón
` (19 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
For compatibility adds default inherit of the new class testsdk
for now, we need to review the code base.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testimage.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index ce71209..cb3314d 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -290,3 +290,5 @@ def testimage_main(d):
target.stop()
testimage_main[vardepsexclude] =+ "BB_ORIGENV"
+
+inherit testsdk
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 04/22] populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (2 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 03/22] classes/testimage: Add defeault inherit for testsdk Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 05/22] get_test_suites: Add sdkext type for load test suites Aníbal Limón
` (18 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
This variable is needed by testextsdk to known the name of extensible
sdk file generated.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index da7fdfb..394c70c 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -45,7 +45,8 @@ COREBASE_FILES ?= " \
SDK_DIR_task-populate-sdk-ext = "${WORKDIR}/sdk-ext"
B_task-populate-sdk-ext = "${SDK_DIR}"
-TOOLCHAIN_OUTPUTNAME_task-populate-sdk-ext = "${SDK_NAME}-toolchain-ext-${SDK_VERSION}"
+TOOLCHAINEXT_OUTPUTNAME = "${SDK_NAME}-toolchain-ext-${SDK_VERSION}"
+TOOLCHAIN_OUTPUTNAME_task-populate-sdk-ext = "${TOOLCHAINEXT_OUTPUTNAME}"
SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} Extensible SDK"
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 05/22] get_test_suites: Add sdkext type for load test suites.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (3 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 04/22] populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 06/22] classes/testsdk: Add testsdkext task only install Aníbal Limón
` (17 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/lib/oeqa/oetest.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 18b2209..6470129 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -271,6 +271,8 @@ def get_test_suites(d, type='runtime'):
if type == "sdk":
testsuites = (d.getVar("TEST_SUITES_SDK", True) or "auto").split()
+ elif type == "sdkext":
+ testsuites = (d.getVar("TEST_SUITES_SDKEXT", True) or "auto").split()
else:
manifests = (d.getVar("TEST_SUITES_MANIFEST", True) or '').split()
if manifests:
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 06/22] classes/testsdk: Add testsdkext task only install.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (4 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 05/22] get_test_suites: Add sdkext type for load test suites Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 07/22] bb/fetch2: Move export_proxies function from wget to utils Aníbal Limón
` (16 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Add task for test extensible sdk for now only install the SDK.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testsdk.bbclass | 45 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 42c4913..8fe6d0f 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -98,3 +98,48 @@ python do_testsdk() {
addtask testsdk
do_testsdk[nostamp] = "1"
do_testsdk[lockfiles] += "${TESTSDKLOCK}"
+
+TEST_LOG_SDKEXT_DIR ?= "${WORKDIR}/testsdkext"
+TESTSDKEXTLOCK = "${TMPDIR}/testsdkext.lock"
+
+def testsdkext_main(d):
+ import unittest
+ import os
+ import glob
+ import oeqa.sdkext
+ import time
+ import subprocess
+ from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+
+ pn = d.getVar("PN", True)
+ bb.utils.mkdirhier(d.getVar("TEST_LOG_SDKEXT_DIR", True))
+
+ # tests in TEST_SUITES become required tests
+ # they won't be skipped even if they aren't suitable.
+ # testslist is what we'll actually pass to the unittest loader
+ testslist = get_tests_list(get_test_suites(d, "sdkext"),
+ d.getVar("BBPATH", True).split(':'), "sdkext")
+ testsrequired = [t for t in (d.getVar("TEST_SUITES_SDKEXT", True) or \
+ "auto").split() if t != "auto"]
+
+ tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.sh")
+ if not os.path.exists(tcname):
+ bb.fatal("The toolchain ext is not built. Build it before running the" \
+ " tests: 'bitbake <image> -c populate_sdk_ext' .")
+
+ testdir = d.expand("${WORKDIR}/testsdkext/")
+ bb.utils.remove(testdir, True)
+ bb.utils.mkdirhier(testdir)
+ try:
+ subprocess.check_output("%s -y -d %s" % (tcname, testdir), shell=True)
+ except subprocess.CalledProcessError as e:
+ bb.fatal("Couldn't install the SDK EXT:\n%s" % e.output)
+
+testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
+
+python do_testsdkext() {
+ testsdkext_main(d)
+}
+addtask testsdkext
+do_testsdkext[nostamp] = "1"
+do_testsdkext[lockfiles] += "${TESTSDKEXTLOCK}"
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 07/22] bb/fetch2: Move export_proxies function from wget to utils.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (5 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 06/22] classes/testsdk: Add testsdkext task only install Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 08/22] classes/testsdk: Add call to export_proxies on testsdkext Aníbal Limón
` (15 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
In order to use in other modules since is a common function
when needs to get proxies working.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 17 +----------------
bitbake/lib/bb/utils.py | 19 +++++++++++++++++++
2 files changed, 20 insertions(+), 16 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 5a31730..fd25c42 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -37,6 +37,7 @@ from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
+from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
@@ -219,22 +220,6 @@ class Wget(FetchMethod):
return resp
- def export_proxies(d):
- variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY',
- 'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY']
- exported = False
-
- for v in variables:
- if v in os.environ.keys():
- exported = True
- else:
- v_proxy = d.getVar(v, True)
- if v_proxy is not None:
- os.environ[v] = v_proxy
- exported = True
-
- return exported
-
class HTTPMethodFallback(urllib2.BaseHandler):
"""
Fallback to GET if HEAD is not allowed (405 HTTP error)
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index ae10213..70b42f3 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1406,3 +1406,22 @@ def set_process_name(name):
libc.prctl(15, byref(buff), 0, 0, 0)
except:
pass
+
+# export common proxies variables from datastore to environment
+def export_proxies(d):
+ import os
+
+ variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY',
+ 'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY']
+ exported = False
+
+ for v in variables:
+ if v in os.environ.keys():
+ exported = True
+ else:
+ v_proxy = d.getVar(v, True)
+ if v_proxy is not None:
+ os.environ[v] = v_proxy
+ exported = True
+
+ return exported
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 08/22] classes/testsdk: Add call to export_proxies on testsdkext.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (6 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 07/22] bb/fetch2: Move export_proxies function from wget to utils Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 09/22] toolchain-shar-extract.sh: Add proxy variable to new env Aníbal Limón
` (14 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Extensible SDK needs to use network and some networks requires
proxies then export it.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testsdk.bbclass | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 8fe6d0f..104c7a1 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -110,6 +110,9 @@ def testsdkext_main(d):
import time
import subprocess
from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+ from bb.utils import export_proxies
+
+ export_proxies(d)
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_SDKEXT_DIR", True))
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 09/22] toolchain-shar-extract.sh: Add proxy variable to new env.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (7 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 08/22] classes/testsdk: Add call to export_proxies on testsdkext Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 10/22] testimage/testsdk: Modularize TestContext Aníbal Limón
` (13 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Extensible SDK needs to do network operations so add proxies to
environment.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
| 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index d844771..12d39c3 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -1,6 +1,8 @@
#!/bin/sh
-[ -z "$ENVCLEANED" ] && exec /usr/bin/env -i ENVCLEANED=1 HOME="$HOME" "$0" "$@"
+[ -z "$ENVCLEANED" ] && exec /usr/bin/env -i ENVCLEANED=1 HOME="$HOME" \
+ http_proxy="$http_proxy" https_proxy="$https_proxy" ftp_proxy="$ftp_proxy" \
+ no_proxy="$no_proxy" GIT_PROXY_COMMAND="$GIT_PROXY_COMMAND" "$0" "$@"
[ -f /etc/environment ] && . /etc/environment
export PATH=`echo "$PATH" | sed -e 's/:\.//' -e 's/::/:/'`
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 10/22] testimage/testsdk: Modularize TestContext.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (8 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 09/22] toolchain-shar-extract.sh: Add proxy variable to new env Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 11/22] testimage/testsdk: Move get test suites routine inside TestContext Aníbal Limón
` (12 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Move anonymous duplicated class TestContext from testimage/testsdk to
oeqa/oetest now we have two new classes ImageTestContext and
SDKTestContext with common code in TestContext class.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testimage.bbclass | 40 ++-------------------------
meta/classes/testsdk.bbclass | 29 ++-----------------
meta/lib/oeqa/oetest.py | 63 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 69 insertions(+), 63 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index cb3314d..2e5bf44 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -195,7 +195,8 @@ def testimage_main(d):
import oeqa.runtime
import time
import signal
- from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+ from oeqa.oetest import loadTests, runTests, \
+ get_test_suites, get_tests_list, ImageTestContext
from oeqa.targetcontrol import get_target_controller
from oeqa.utils.dump import get_host_dumper
@@ -212,48 +213,14 @@ def testimage_main(d):
testslist = get_tests_list(get_test_suites(d), d.getVar("BBPATH", True).split(':'))
testsrequired = [t for t in d.getVar("TEST_SUITES", True).split() if t != "auto"]
- tagexp = d.getVar("TEST_SUITES_TAGS", True)
-
# we need the host dumper in test context
host_dumper = get_host_dumper(d)
# the robot dance
target = get_target_controller(d)
- class TestContext(object):
- def __init__(self):
- self.d = d
- self.testslist = testslist
- self.tagexp = tagexp
- self.testsrequired = testsrequired
- self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
- self.target = target
- self.host_dumper = host_dumper
- self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
- self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
- manifest = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("IMAGE_LINK_NAME", True) + ".manifest")
- nomanifest = d.getVar("IMAGE_NO_MANIFEST", True)
-
- self.sigterm = False
- self.origsigtermhandler = signal.getsignal(signal.SIGTERM)
- signal.signal(signal.SIGTERM, self.sigterm_exception)
-
- if nomanifest is None or nomanifest != "1":
- try:
- with open(manifest) as f:
- self.pkgmanifest = f.read()
- except IOError as e:
- bb.fatal("No package manifest file found. Did you build the image?\n%s" % e)
- else:
- self.pkgmanifest = ""
-
- def sigterm_exception(self, signum, stackframe):
- bb.warn("TestImage received SIGTERM, shutting down...")
- self.sigterm = True
- self.target.stop()
-
# test context
- tc = TestContext()
+ tc = ImageTestContext(d, testslist, testsrequired, target, host_dumper)
# this is a dummy load of tests
# we are doing that to find compile errors in the tests themselves
@@ -264,7 +231,6 @@ def testimage_main(d):
import traceback
bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
-
if export:
signal.signal(signal.SIGTERM, tc.origsigtermhandler)
tc.origsigtermhandler = None
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 104c7a1..06e7a77 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -13,7 +13,8 @@ def testsdk_main(d):
import oeqa.sdk
import time
import subprocess
- from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+ from oeqa.oetest import loadTests, runTests, \
+ get_test_suites, get_tests_list, SDKTestContext
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
@@ -28,29 +29,6 @@ def testsdk_main(d):
if not os.path.exists(tcname):
bb.fatal("The toolchain is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .")
- class TestContext(object):
- def __init__(self):
- self.d = d
- self.testslist = testslist
- self.testsrequired = testsrequired
- self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
- self.sdktestdir = sdktestdir
- self.sdkenv = sdkenv
- self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
- self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
- manifest = d.getVar("SDK_TARGET_MANIFEST", True)
- try:
- with open(manifest) as f:
- self.pkgmanifest = f.read()
- except IOError as e:
- bb.fatal("No package manifest file found. Did you build the sdk image?\n%s" % e)
- hostmanifest = d.getVar("SDK_HOST_MANIFEST", True)
- try:
- with open(hostmanifest) as f:
- self.hostpkgmanifest = f.read()
- except IOError as e:
- bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
-
sdktestdir = d.expand("${WORKDIR}/testimage-sdk/")
bb.utils.remove(sdktestdir, True)
bb.utils.mkdirhier(sdktestdir)
@@ -63,8 +41,7 @@ def testsdk_main(d):
targets = glob.glob(d.expand(sdktestdir + "/tc/environment-setup-*"))
for sdkenv in targets:
bb.plain("Testing %s" % sdkenv)
- # test context
- tc = TestContext()
+ tc = SDKTestContext(d, testslist, testsrequired, sdktestdir, sdkenv)
# this is a dummy load of tests
# we are doing that to find compile errors in the tests themselves
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 6470129..9951a6f 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -11,11 +11,14 @@ import os, re, mmap
import unittest
import inspect
import subprocess
+import signal
try:
import bb
except ImportError:
pass
import logging
+
+import oeqa.runtime
from oeqa.utils.decorators import LogResults, gettag, getResults
logger = logging.getLogger("BitBake")
@@ -326,3 +329,63 @@ def get_tests_list(testsuites, bbpath, type="runtime"):
add_auto_list(testpath)
return testslist
+
+class TestContext(object):
+ def __init__(self, d, testslist, testsrequired):
+ self.d = d
+ self.testslist = testslist
+ self.testsrequired = testsrequired
+
+ self.filesdir = os.path.join(os.path.dirname(os.path.abspath(
+ oeqa.runtime.__file__)), "files")
+ self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
+ self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
+
+class ImageTestContext(TestContext):
+ def __init__(self, d, testslist, testsrequired, target, host_dumper):
+ super(ImageTestContext, self).__init__(d, testslist, testsrequired)
+
+ self.tagexp = d.getVar("TEST_SUITES_TAGS", True)
+
+ self.target = target
+ self.host_dumper = host_dumper
+
+ manifest = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True),
+ d.getVar("IMAGE_LINK_NAME", True) + ".manifest")
+ nomanifest = d.getVar("IMAGE_NO_MANIFEST", True)
+ if nomanifest is None or nomanifest != "1":
+ try:
+ with open(manifest) as f:
+ self.pkgmanifest = f.read()
+ except IOError as e:
+ bb.fatal("No package manifest file found. Did you build the image?\n%s" % e)
+ else:
+ self.pkgmanifest = ""
+
+ self.sigterm = False
+ self.origsigtermhandler = signal.getsignal(signal.SIGTERM)
+ signal.signal(signal.SIGTERM, self._sigterm_exception)
+
+ def _sigterm_exception(self, signum, stackframe):
+ bb.warn("TestImage received SIGTERM, shutting down...")
+ self.sigterm = True
+ self.target.stop()
+
+class SDKTestContext(TestContext):
+ def __init__(self, d, testslist, testsrequired, sdktestdir, sdkenv):
+ super(SDKTestContext, self).__init__(d, testslist, testsrequired)
+
+ self.sdktestdir = sdktestdir
+ self.sdkenv = sdkenv
+
+ try:
+ with open(d.getVar("SDK_TARGET_MANIFEST", True)) as f:
+ self.pkgmanifest = f.read()
+ except IOError as e:
+ bb.fatal("No package manifest file found. Did you build the sdk image?\n%s" % e)
+
+ try:
+ with open(d.getVar("SDK_HOST_MANIFEST", True)) as f:
+ self.hostpkgmanifest = f.read()
+ except IOError as e:
+ bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 11/22] testimage/testsdk: Move get test suites routine inside TestContext.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (9 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 10/22] testimage/testsdk: Modularize TestContext Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 12/22] oetest.py/TestContext: Move loadTests and runTests inside it Aníbal Limón
` (11 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
In order to provide better abstraction move functions to get the test
suite inside the TestContext.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testimage.bbclass | 11 +--
meta/classes/testsdk.bbclass | 19 +---
meta/lib/oeqa/oetest.py | 192 +++++++++++++++++++++++------------------
3 files changed, 112 insertions(+), 110 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 2e5bf44..5ffa8a5 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -195,8 +195,7 @@ def testimage_main(d):
import oeqa.runtime
import time
import signal
- from oeqa.oetest import loadTests, runTests, \
- get_test_suites, get_tests_list, ImageTestContext
+ from oeqa.oetest import loadTests, runTests, ImageTestContext
from oeqa.targetcontrol import get_target_controller
from oeqa.utils.dump import get_host_dumper
@@ -207,12 +206,6 @@ def testimage_main(d):
bb.utils.remove(d.getVar("TEST_EXPORT_DIR", True), recurse=True)
bb.utils.mkdirhier(d.getVar("TEST_EXPORT_DIR", True))
- # tests in TEST_SUITES become required tests
- # they won't be skipped even if they aren't suitable for a image (like xorg for minimal)
- # testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(get_test_suites(d), d.getVar("BBPATH", True).split(':'))
- testsrequired = [t for t in d.getVar("TEST_SUITES", True).split() if t != "auto"]
-
# we need the host dumper in test context
host_dumper = get_host_dumper(d)
@@ -220,7 +213,7 @@ def testimage_main(d):
target = get_target_controller(d)
# test context
- tc = ImageTestContext(d, testslist, testsrequired, target, host_dumper)
+ tc = ImageTestContext(d, target, host_dumper)
# this is a dummy load of tests
# we are doing that to find compile errors in the tests themselves
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 06e7a77..47bad29 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -13,18 +13,11 @@ def testsdk_main(d):
import oeqa.sdk
import time
import subprocess
- from oeqa.oetest import loadTests, runTests, \
- get_test_suites, get_tests_list, SDKTestContext
+ from oeqa.oetest import loadTests, runTests, SDKTestContext
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
- # tests in TEST_SUITES become required tests
- # they won't be skipped even if they aren't suitable.
- # testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(get_test_suites(d, "sdk"), d.getVar("BBPATH", True).split(':'), "sdk")
- testsrequired = [t for t in (d.getVar("TEST_SUITES_SDK", True) or "auto").split() if t != "auto"]
-
tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh")
if not os.path.exists(tcname):
bb.fatal("The toolchain is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .")
@@ -41,7 +34,7 @@ def testsdk_main(d):
targets = glob.glob(d.expand(sdktestdir + "/tc/environment-setup-*"))
for sdkenv in targets:
bb.plain("Testing %s" % sdkenv)
- tc = SDKTestContext(d, testslist, testsrequired, sdktestdir, sdkenv)
+ tc = SDKTestContext(d, sdktestdir, sdkenv)
# this is a dummy load of tests
# we are doing that to find compile errors in the tests themselves
@@ -94,14 +87,6 @@ def testsdkext_main(d):
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_SDKEXT_DIR", True))
- # tests in TEST_SUITES become required tests
- # they won't be skipped even if they aren't suitable.
- # testslist is what we'll actually pass to the unittest loader
- testslist = get_tests_list(get_test_suites(d, "sdkext"),
- d.getVar("BBPATH", True).split(':'), "sdkext")
- testsrequired = [t for t in (d.getVar("TEST_SUITES_SDKEXT", True) or \
- "auto").split() if t != "auto"]
-
tcname = d.expand("${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.sh")
if not os.path.exists(tcname):
bb.fatal("The toolchain ext is not built. Build it before running the" \
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 9951a6f..7f07037 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -33,7 +33,6 @@ def getVar(obj):
def checkTags(tc, tagexp):
return eval(tagexp, None, getVar(tc))
-
def filterByTagExp(testsuite, tagexp):
if not tagexp:
return testsuite
@@ -254,96 +253,80 @@ def skipModuleUnless(cond, reason):
if not cond:
skipModule(reason, 3)
-# get testcase list from specified file
-# if path is a relative path, then relative to build/conf/
-def read_testlist(fpath, builddir):
- if not os.path.isabs(fpath):
- fpath = os.path.join(builddir, "conf", fpath)
- if not os.path.exists(fpath):
- bb.fatal("No such manifest file: ", fpath)
- tcs = []
- for line in open(fpath).readlines():
- line = line.strip()
- if line and not line.startswith("#"):
- tcs.append(line)
- return " ".join(tcs)
-
-# get test suites, returns test suites based on d variables
-def get_test_suites(d, type='runtime'):
- testsuites = []
-
- if type == "sdk":
- testsuites = (d.getVar("TEST_SUITES_SDK", True) or "auto").split()
- elif type == "sdkext":
- testsuites = (d.getVar("TEST_SUITES_SDKEXT", True) or "auto").split()
- else:
- manifests = (d.getVar("TEST_SUITES_MANIFEST", True) or '').split()
- if manifests:
- for manifest in manifests:
- testsuites.extend(read_testlist(manifest,
- d.getVar("TOPDIR", True)).split())
-
- else:
- testsuites = d.getVar("TEST_SUITES", True).split()
-
- return testsuites
-
-# return test list by type also filter if TEST_SUITES is specified
-def get_tests_list(testsuites, bbpath, type="runtime"):
- testslist = []
-
- # This relies on lib/ under each directory in BBPATH being added to sys.path
- # (as done by default in base.bbclass)
- for testname in testsuites:
- if testname != "auto":
- if testname.startswith("oeqa."):
- testslist.append(testname)
- continue
- found = False
- for p in bbpath:
- if os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname + '.py')):
- testslist.append("oeqa." + type + "." + testname)
- found = True
- break
- elif os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname.split(".")[0] + '.py')):
- testslist.append("oeqa." + type + "." + testname)
- found = True
- break
- if not found:
- bb.fatal('Test %s specified in TEST_SUITES could not be found in lib/oeqa/runtime under BBPATH' % testname)
-
- if "auto" in testsuites:
- def add_auto_list(path):
- if not os.path.exists(os.path.join(path, '__init__.py')):
- bb.fatal('Tests directory %s exists but is missing __init__.py' % path)
- files = sorted([f for f in os.listdir(path) if f.endswith('.py') and not f.startswith('_')])
- for f in files:
- module = 'oeqa.' + type + '.' + f[:-3]
- if module not in testslist:
- testslist.append(module)
-
- for p in bbpath:
- testpath = os.path.join(p, 'lib', 'oeqa', type)
- bb.debug(2, 'Searching for tests in %s' % testpath)
- if os.path.exists(testpath):
- add_auto_list(testpath)
-
- return testslist
-
class TestContext(object):
- def __init__(self, d, testslist, testsrequired):
+ def __init__(self, d):
self.d = d
- self.testslist = testslist
- self.testsrequired = testsrequired
+
+ self.testsuites = self._get_test_suites()
+ self.testslist = self._get_tests_list(d.getVar("BBPATH", True).split(':'))
+ self.testsrequired = self._get_test_suites_required()
self.filesdir = os.path.join(os.path.dirname(os.path.abspath(
oeqa.runtime.__file__)), "files")
self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
+ # get testcase list from specified file
+ # if path is a relative path, then relative to build/conf/
+ def _read_testlist(self, fpath, builddir):
+ if not os.path.isabs(fpath):
+ fpath = os.path.join(builddir, "conf", fpath)
+ if not os.path.exists(fpath):
+ bb.fatal("No such manifest file: ", fpath)
+ tcs = []
+ for line in open(fpath).readlines():
+ line = line.strip()
+ if line and not line.startswith("#"):
+ tcs.append(line)
+ return " ".join(tcs)
+
+ # return test list by type also filter if TEST_SUITES is specified
+ def _get_tests_list(self, bbpath):
+ testslist = []
+
+ type = self._get_test_namespace()
+
+ # This relies on lib/ under each directory in BBPATH being added to sys.path
+ # (as done by default in base.bbclass)
+ for testname in self.testsuites:
+ if testname != "auto":
+ if testname.startswith("oeqa."):
+ testslist.append(testname)
+ continue
+ found = False
+ for p in bbpath:
+ if os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname + '.py')):
+ testslist.append("oeqa." + type + "." + testname)
+ found = True
+ break
+ elif os.path.exists(os.path.join(p, 'lib', 'oeqa', type, testname.split(".")[0] + '.py')):
+ testslist.append("oeqa." + type + "." + testname)
+ found = True
+ break
+ if not found:
+ bb.fatal('Test %s specified in TEST_SUITES could not be found in lib/oeqa/runtime under BBPATH' % testname)
+
+ if "auto" in self.testsuites:
+ def add_auto_list(path):
+ if not os.path.exists(os.path.join(path, '__init__.py')):
+ bb.fatal('Tests directory %s exists but is missing __init__.py' % path)
+ files = sorted([f for f in os.listdir(path) if f.endswith('.py') and not f.startswith('_')])
+ for f in files:
+ module = 'oeqa.' + type + '.' + f[:-3]
+ if module not in testslist:
+ testslist.append(module)
+
+ for p in bbpath:
+ testpath = os.path.join(p, 'lib', 'oeqa', type)
+ bb.debug(2, 'Searching for tests in %s' % testpath)
+ if os.path.exists(testpath):
+ add_auto_list(testpath)
+
+ return testslist
+
class ImageTestContext(TestContext):
- def __init__(self, d, testslist, testsrequired, target, host_dumper):
- super(ImageTestContext, self).__init__(d, testslist, testsrequired)
+ def __init__(self, d, target, host_dumper):
+ super(ImageTestContext, self).__init__(d)
self.tagexp = d.getVar("TEST_SUITES_TAGS", True)
@@ -371,9 +354,29 @@ class ImageTestContext(TestContext):
self.sigterm = True
self.target.stop()
+ def _get_test_namespace(self):
+ return "runtime"
+
+ def _get_test_suites(self):
+ testsuites = []
+
+ manifests = (self.d.getVar("TEST_SUITES_MANIFEST", True) or '').split()
+ if manifests:
+ for manifest in manifests:
+ testsuites.extend(self._read_testlist(manifest,
+ self.d.getVar("TOPDIR", True)).split())
+
+ else:
+ testsuites = self.d.getVar("TEST_SUITES", True).split()
+
+ return testsuites
+
+ def _get_test_suites_required(self):
+ return [t for t in self.d.getVar("TEST_SUITES", True).split() if t != "auto"]
+
class SDKTestContext(TestContext):
- def __init__(self, d, testslist, testsrequired, sdktestdir, sdkenv):
- super(SDKTestContext, self).__init__(d, testslist, testsrequired)
+ def __init__(self, d, sdktestdir, sdkenv):
+ super(SDKTestContext, self).__init__(d)
self.sdktestdir = sdktestdir
self.sdkenv = sdkenv
@@ -389,3 +392,24 @@ class SDKTestContext(TestContext):
self.hostpkgmanifest = f.read()
except IOError as e:
bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
+
+ def _get_test_namespace(self):
+ return "sdk"
+
+ def _get_test_suites(self):
+ return (self.d.getVar("TEST_SUITES_SDK", True) or "auto").split()
+
+ def _get_test_suites_required(self):
+ return [t for t in (self.d.getVar("TEST_SUITES_SDK", True) or \
+ "auto").split() if t != "auto"]
+
+class SDKExtTestContext(TestContext):
+ def _get_test_namespace(self):
+ return "sdkext"
+
+ def _get_test_suites(self):
+ return (self.d.getVar("TEST_SUITES_SDK_EXT", True) or "auto").split()
+
+ def _get_test_suites_required(self):
+ return [t for t in (self.d.getVar("TEST_SUITES_SDK_EXT", True) or \
+ "auto").split() if t != "auto"]
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 12/22] oetest.py/TestContext: Move loadTests and runTests inside it.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (10 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 11/22] testimage/testsdk: Move get test suites routine inside TestContext Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 13/22] classes/testsdk: Add function run_test_context Aníbal Limón
` (10 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Method's for loadTests and runTests make sense to define
inside TestContext because it can be different around
Image, SDK, SDKExt.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testimage.bbclass | 6 +-
meta/classes/testsdk.bbclass | 6 +-
meta/lib/oeqa/oetest.py | 197 ++++++++++++++++++++---------------------
3 files changed, 103 insertions(+), 106 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 5ffa8a5..5fafda1 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -195,7 +195,7 @@ def testimage_main(d):
import oeqa.runtime
import time
import signal
- from oeqa.oetest import loadTests, runTests, ImageTestContext
+ from oeqa.oetest import ImageTestContext
from oeqa.targetcontrol import get_target_controller
from oeqa.utils.dump import get_host_dumper
@@ -219,7 +219,7 @@ def testimage_main(d):
# we are doing that to find compile errors in the tests themselves
# before booting the image
try:
- loadTests(tc)
+ tc.loadTests()
except Exception as e:
import traceback
bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
@@ -233,7 +233,7 @@ def testimage_main(d):
try:
target.start()
starttime = time.time()
- result = runTests(tc)
+ result = tc.runTests()
stoptime = time.time()
if result.wasSuccessful():
bb.plain("%s - Ran %d test%s in %.3fs" % (pn, result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 47bad29..ba8897e 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -13,7 +13,7 @@ def testsdk_main(d):
import oeqa.sdk
import time
import subprocess
- from oeqa.oetest import loadTests, runTests, SDKTestContext
+ from oeqa.oetest import SDKTestContext
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
@@ -40,13 +40,13 @@ def testsdk_main(d):
# we are doing that to find compile errors in the tests themselves
# before booting the image
try:
- loadTests(tc, "sdk")
+ tc.loadTests()
except Exception as e:
import traceback
bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
starttime = time.time()
- result = runTests(tc, "sdk")
+ result = tc.runTests()
stoptime = time.time()
if result.wasSuccessful():
bb.plain("%s SDK(%s):%s - Ran %d test%s in %.3fs" % (pn, os.path.basename(tcname), os.path.basename(sdkenv),result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 7f07037..16705cc 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -45,106 +45,6 @@ def filterByTagExp(testsuite, tagexp):
caseList.append(filterByTagExp(each, tagexp))
return testsuite.__class__(caseList)
-def loadTests(tc, type="runtime"):
- if type == "runtime":
- # set the context object passed from the test class
- setattr(oeTest, "tc", tc)
- # set ps command to use
- setattr(oeRuntimeTest, "pscmd", "ps -ef" if oeTest.hasPackage("procps") else "ps")
- # prepare test suite, loader and runner
- suite = unittest.TestSuite()
- elif type == "sdk":
- # set the context object passed from the test class
- setattr(oeTest, "tc", tc)
- testloader = unittest.TestLoader()
- testloader.sortTestMethodsUsing = None
- suites = [testloader.loadTestsFromName(name) for name in tc.testslist]
- suites = filterByTagExp(suites, getattr(tc, "tagexp", None))
-
- def getTests(test):
- '''Return all individual tests executed when running the suite.'''
- # Unfortunately unittest does not have an API for this, so we have
- # to rely on implementation details. This only needs to work
- # for TestSuite containing TestCase.
- method = getattr(test, '_testMethodName', None)
- if method:
- # leaf case: a TestCase
- yield test
- else:
- # Look into TestSuite.
- tests = getattr(test, '_tests', [])
- for t1 in tests:
- for t2 in getTests(t1):
- yield t2
-
- # Determine dependencies between suites by looking for @skipUnlessPassed
- # method annotations. Suite A depends on suite B if any method in A
- # depends on a method on B.
- for suite in suites:
- suite.dependencies = []
- suite.depth = 0
- for test in getTests(suite):
- methodname = getattr(test, '_testMethodName', None)
- if methodname:
- method = getattr(test, methodname)
- depends_on = getattr(method, '_depends_on', None)
- if depends_on:
- for dep_suite in suites:
- if depends_on in [getattr(t, '_testMethodName', None) for t in getTests(dep_suite)]:
- if dep_suite not in suite.dependencies and \
- dep_suite is not suite:
- suite.dependencies.append(dep_suite)
- break
- else:
- logger.warning("Test %s was declared as @skipUnlessPassed('%s') but that test is either not defined or not active. Will run the test anyway." %
- (test, depends_on))
- # Use brute-force topological sort to determine ordering. Sort by
- # depth (higher depth = must run later), with original ordering to
- # break ties.
- def set_suite_depth(suite):
- for dep in suite.dependencies:
- new_depth = set_suite_depth(dep) + 1
- if new_depth > suite.depth:
- suite.depth = new_depth
- return suite.depth
- for index, suite in enumerate(suites):
- set_suite_depth(suite)
- suite.index = index
- suites.sort(cmp=lambda a,b: cmp((a.depth, a.index), (b.depth, b.index)))
- return testloader.suiteClass(suites)
-
-_buffer = ""
-
-def custom_verbose(msg, *args, **kwargs):
- global _buffer
- if msg[-1] != "\n":
- _buffer += msg
- else:
- _buffer += msg
- try:
- bb.plain(_buffer.rstrip("\n"), *args, **kwargs)
- except NameError:
- logger.info(_buffer.rstrip("\n"), *args, **kwargs)
- _buffer = ""
-
-def runTests(tc, type="runtime"):
-
- suite = loadTests(tc, type)
- logger.info("Test modules %s" % tc.testslist)
- if hasattr(tc, "tagexp") and tc.tagexp:
- logger.info("Filter test cases by tags: %s" % tc.tagexp)
- logger.info("Found %s tests" % suite.countTestCases())
- runner = unittest.TextTestRunner(verbosity=2)
- try:
- if bb.msg.loggerDefaultVerbose:
- runner.stream.write = custom_verbose
- except NameError:
- # Not in bb environment?
- pass
- result = runner.run(suite)
-
- return result
-
@LogResults
class oeTest(unittest.TestCase):
@@ -253,6 +153,19 @@ def skipModuleUnless(cond, reason):
if not cond:
skipModule(reason, 3)
+_buffer_logger = ""
+def custom_verbose(msg, *args, **kwargs):
+ global _buffer_logger
+ if msg[-1] != "\n":
+ _buffer_logger += msg
+ else:
+ _buffer_logger += msg
+ try:
+ bb.plain(_buffer_logger.rstrip("\n"), *args, **kwargs)
+ except NameError:
+ logger.info(_buffer_logger.rstrip("\n"), *args, **kwargs)
+ _buffer_logger = ""
+
class TestContext(object):
def __init__(self, d):
self.d = d
@@ -324,6 +237,86 @@ class TestContext(object):
return testslist
+ def loadTests(self):
+ setattr(oeTest, "tc", self)
+
+ testloader = unittest.TestLoader()
+ testloader.sortTestMethodsUsing = None
+ suites = [testloader.loadTestsFromName(name) for name in self.testslist]
+ suites = filterByTagExp(suites, getattr(self, "tagexp", None))
+
+ def getTests(test):
+ '''Return all individual tests executed when running the suite.'''
+ # Unfortunately unittest does not have an API for this, so we have
+ # to rely on implementation details. This only needs to work
+ # for TestSuite containing TestCase.
+ method = getattr(test, '_testMethodName', None)
+ if method:
+ # leaf case: a TestCase
+ yield test
+ else:
+ # Look into TestSuite.
+ tests = getattr(test, '_tests', [])
+ for t1 in tests:
+ for t2 in getTests(t1):
+ yield t2
+
+ # Determine dependencies between suites by looking for @skipUnlessPassed
+ # method annotations. Suite A depends on suite B if any method in A
+ # depends on a method on B.
+ for suite in suites:
+ suite.dependencies = []
+ suite.depth = 0
+ for test in getTests(suite):
+ methodname = getattr(test, '_testMethodName', None)
+ if methodname:
+ method = getattr(test, methodname)
+ depends_on = getattr(method, '_depends_on', None)
+ if depends_on:
+ for dep_suite in suites:
+ if depends_on in [getattr(t, '_testMethodName', None) for t in getTests(dep_suite)]:
+ if dep_suite not in suite.dependencies and \
+ dep_suite is not suite:
+ suite.dependencies.append(dep_suite)
+ break
+ else:
+ logger.warning("Test %s was declared as @skipUnlessPassed('%s') but that test is either not defined or not active. Will run the test anyway." %
+ (test, depends_on))
+
+ # Use brute-force topological sort to determine ordering. Sort by
+ # depth (higher depth = must run later), with original ordering to
+ # break ties.
+ def set_suite_depth(suite):
+ for dep in suite.dependencies:
+ new_depth = set_suite_depth(dep) + 1
+ if new_depth > suite.depth:
+ suite.depth = new_depth
+ return suite.depth
+
+ for index, suite in enumerate(suites):
+ set_suite_depth(suite)
+ suite.index = index
+ suites.sort(cmp=lambda a,b: cmp((a.depth, a.index), (b.depth, b.index)))
+
+ self.suite = testloader.suiteClass(suites)
+
+ return self.suite
+
+ def runTests(self):
+ logger.info("Test modules %s" % self.testslist)
+ if hasattr(self, "tagexp") and self.tagexp:
+ logger.info("Filter test cases by tags: %s" % self.tagexp)
+ logger.info("Found %s tests" % self.suite.countTestCases())
+ runner = unittest.TextTestRunner(verbosity=2)
+ try:
+ if bb.msg.loggerDefaultVerbose:
+ runner.stream.write = custom_verbose
+ except NameError:
+ # Not in bb environment?
+ pass
+
+ return runner.run(self.suite)
+
class ImageTestContext(TestContext):
def __init__(self, d, target, host_dumper):
super(ImageTestContext, self).__init__(d)
@@ -374,6 +367,10 @@ class ImageTestContext(TestContext):
def _get_test_suites_required(self):
return [t for t in self.d.getVar("TEST_SUITES", True).split() if t != "auto"]
+ def loadTests(self):
+ super(ImageTestContext, self).loadTests()
+ setattr(oeRuntimeTest, "pscmd", "ps -ef" if oeTest.hasPackage("procps") else "ps")
+
class SDKTestContext(TestContext):
def __init__(self, d, sdktestdir, sdkenv):
super(SDKTestContext, self).__init__(d)
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 13/22] classes/testsdk: Add function run_test_context
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (11 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 12/22] oetest.py/TestContext: Move loadTests and runTests inside it Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 14/22] oeqa/oetest: oeSDKTest when run a command redirect env output to null Aníbal Limón
` (9 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
This helper functions will be serve as well to run extensible
sdk tests so generalize it to get function context as arg.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testsdk.bbclass | 62 +++++++++++++++++++++++---------------------
1 file changed, 32 insertions(+), 30 deletions(-)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index ba8897e..88057e5 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -5,13 +5,40 @@
TEST_LOG_DIR ?= "${WORKDIR}/testimage"
TESTSDKLOCK = "${TMPDIR}/testsdk.lock"
+def run_test_context(CTestContext, d, testdir, tcname, pn):
+ import glob
+ import time
+
+ targets = glob.glob(d.expand(testdir + "/tc/environment-setup-*"))
+ for sdkenv in targets:
+ bb.plain("Testing %s" % sdkenv)
+ tc = CTestContext(d, testdir, sdkenv)
+
+ # this is a dummy load of tests
+ # we are doing that to find compile errors in the tests themselves
+ # before booting the image
+ try:
+ tc.loadTests()
+ except Exception as e:
+ import traceback
+ bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
+
+ starttime = time.time()
+ result = tc.runTests()
+ stoptime = time.time()
+ if result.wasSuccessful():
+ bb.plain("%s SDK(%s):%s - Ran %d test%s in %.3fs" % (pn, os.path.basename(tcname), os.path.basename(sdkenv),result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
+ msg = "%s - OK - All required tests passed" % pn
+ skipped = len(result.skipped)
+ if skipped:
+ msg += " (skipped=%d)" % skipped
+ bb.plain(msg)
+ else:
+ raise bb.build.FuncFailed("%s - FAILED - check the task log and the commands log" % pn )
+
def testsdk_main(d):
- import unittest
import os
- import glob
- import oeqa.runtime
import oeqa.sdk
- import time
import subprocess
from oeqa.oetest import SDKTestContext
@@ -31,32 +58,7 @@ def testsdk_main(d):
bb.fatal("Couldn't install the SDK:\n%s" % e.output)
try:
- targets = glob.glob(d.expand(sdktestdir + "/tc/environment-setup-*"))
- for sdkenv in targets:
- bb.plain("Testing %s" % sdkenv)
- tc = SDKTestContext(d, sdktestdir, sdkenv)
-
- # this is a dummy load of tests
- # we are doing that to find compile errors in the tests themselves
- # before booting the image
- try:
- tc.loadTests()
- except Exception as e:
- import traceback
- bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
-
- starttime = time.time()
- result = tc.runTests()
- stoptime = time.time()
- if result.wasSuccessful():
- bb.plain("%s SDK(%s):%s - Ran %d test%s in %.3fs" % (pn, os.path.basename(tcname), os.path.basename(sdkenv),result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
- msg = "%s - OK - All required tests passed" % pn
- skipped = len(result.skipped)
- if skipped:
- msg += " (skipped=%d)" % skipped
- bb.plain(msg)
- else:
- raise bb.build.FuncFailed("%s - FAILED - check the task log and the commands log" % pn )
+ run_test_context(SDKTestContext, d, sdktestdir, tcname, pn)
finally:
bb.utils.remove(sdktestdir, True)
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 14/22] oeqa/oetest: oeSDKTest when run a command redirect env output to null
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (12 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 13/22] classes/testsdk: Add function run_test_context Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 15/22] classes/testsdk: Add compatibility SDK testsuite to eSDK Aníbal Limón
` (8 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Some tests are failing because the eSDK env load script shows a help
message on the output so redirect this to /dev/null.
There was a discussion with Paul Eggleton [1], he don't want to have env
variables to change the behaviour of what the env script needs to print
so redirect the output in oeSDK.run() method. I didn't agree because
it can hides another messages like error ones and with the variable you
can control what prints or not.
[1] http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116744.html
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/lib/oeqa/oetest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 16705cc..47ea3b2 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -124,7 +124,7 @@ class oeSDKTest(oeTest):
return False
def _run(self, cmd):
- return subprocess.check_output(". %s; " % self.tc.sdkenv + cmd, shell=True)
+ return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True)
def getmodule(pos=2):
# stack returns a list of tuples containg frame information
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 15/22] classes/testsdk: Add compatibility SDK testsuite to eSDK
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (13 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 14/22] oeqa/oetest: oeSDKTest when run a command redirect env output to null Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 16/22] testsdkext: Add skeleton for support Extensible SDK tests Aníbal Limón
` (7 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Extensible SDK is compatible with SDK test suite so it need
to execute the same tests over it.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testsdk.bbclass | 24 +++++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 88057e5..a0c50fe 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -75,17 +75,25 @@ TEST_LOG_SDKEXT_DIR ?= "${WORKDIR}/testsdkext"
TESTSDKEXTLOCK = "${TMPDIR}/testsdkext.lock"
def testsdkext_main(d):
- import unittest
import os
- import glob
import oeqa.sdkext
- import time
import subprocess
- from oeqa.oetest import loadTests, runTests, get_test_suites, get_tests_list
+ from oeqa.oetest import SDKTestContext, SDKExtTestContext
from bb.utils import export_proxies
+ # extensible sdk use network
export_proxies(d)
+ # extensible sdk shows a warning if found bitbake in the path
+ # because can cause problems so clean it
+ new_path = ''
+ for p in os.environ['PATH'].split(':'):
+ if 'bitbake/bin' in p or 'poky/scripts' in p:
+ continue
+ new_path = new_path + p + ':'
+ new_path = new_path[:-1]
+ os.environ['PATH'] = new_path
+
pn = d.getVar("PN", True)
bb.utils.mkdirhier(d.getVar("TEST_LOG_SDKEXT_DIR", True))
@@ -98,10 +106,16 @@ def testsdkext_main(d):
bb.utils.remove(testdir, True)
bb.utils.mkdirhier(testdir)
try:
- subprocess.check_output("%s -y -d %s" % (tcname, testdir), shell=True)
+ subprocess.check_output("%s -y -d %s/tc" % (tcname, testdir), shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Couldn't install the SDK EXT:\n%s" % e.output)
+ try:
+ bb.plain("Running SDK Compatibility tests ...")
+ run_test_context(SDKTestContext, d, testdir, tcname, pn)
+ finally:
+ bb.utils.remove(testdir, True)
+
testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
python do_testsdkext() {
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 16/22] testsdkext: Add skeleton for support Extensible SDK tests.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (14 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 15/22] classes/testsdk: Add compatibility SDK testsuite to eSDK Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 17/22] classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST Aníbal Limón
` (6 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
oeqa/sdkext: Add module and __init__.py will contain eSDK tests.
classes/testsdk: Add support for run eSDK tests.
oeqa/oetest: Create oeSDKExtTest for now only inherit oeSDKTest,
modified SDKExtTestContext now inherit SDKTestContext
and set sdkext filesdir for store data fixtures.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testsdk.bbclass | 10 +++++++++-
meta/lib/oeqa/oetest.py | 12 +++++++++++-
meta/lib/oeqa/sdkext/__init__.py | 3 +++
3 files changed, 23 insertions(+), 2 deletions(-)
create mode 100644 meta/lib/oeqa/sdkext/__init__.py
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index a0c50fe..417510c 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -114,7 +114,15 @@ def testsdkext_main(d):
bb.plain("Running SDK Compatibility tests ...")
run_test_context(SDKTestContext, d, testdir, tcname, pn)
finally:
- bb.utils.remove(testdir, True)
+ pass
+
+ try:
+ bb.plain("Running Extensible SDK tests ...")
+ run_test_context(SDKExtTestContext, d, testdir, tcname, pn)
+ finally:
+ pass
+
+ bb.utils.remove(testdir, True)
testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 47ea3b2..620617c 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -19,6 +19,7 @@ except ImportError:
import logging
import oeqa.runtime
+import oeqa.sdkext
from oeqa.utils.decorators import LogResults, gettag, getResults
logger = logging.getLogger("BitBake")
@@ -126,6 +127,9 @@ class oeSDKTest(oeTest):
def _run(self, cmd):
return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True)
+class oeSDKExtTest(oeSDKTest):
+ pass
+
def getmodule(pos=2):
# stack returns a list of tuples containg frame information
# First element of the list the is current frame, caller is 1
@@ -400,7 +404,13 @@ class SDKTestContext(TestContext):
return [t for t in (self.d.getVar("TEST_SUITES_SDK", True) or \
"auto").split() if t != "auto"]
-class SDKExtTestContext(TestContext):
+class SDKExtTestContext(SDKTestContext):
+ def __init__(self, d, sdktestdir, sdkenv):
+ super(SDKExtTestContext, self).__init__(d, sdktestdir, sdkenv)
+
+ self.sdkextfilesdir = os.path.join(os.path.dirname(os.path.abspath(
+ oeqa.sdkext.__file__)), "files")
+
def _get_test_namespace(self):
return "sdkext"
diff --git a/meta/lib/oeqa/sdkext/__init__.py b/meta/lib/oeqa/sdkext/__init__.py
new file mode 100644
index 0000000..4cf3fa7
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/__init__.py
@@ -0,0 +1,3 @@
+# Enable other layers to have tests in the same named directory
+from pkgutil import extend_path
+__path__ = extend_path(__path__, __name__)
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 17/22] classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (15 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 16/22] testsdkext: Add skeleton for support Extensible SDK tests Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 18/22] oeqa/oetest: Fix compatibility SDK tests using eSDK Aníbal Limón
` (5 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Extensible SDK needs to point to the correct manifest so add
SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST variables.
oeqa/oetest.py: Fix SDKExtTestContext for load the correct manifests.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/populate_sdk_ext.bbclass | 3 +++
meta/lib/oeqa/oetest.py | 11 +++++++++--
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 394c70c..c47dd0b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -48,6 +48,9 @@ B_task-populate-sdk-ext = "${SDK_DIR}"
TOOLCHAINEXT_OUTPUTNAME = "${SDK_NAME}-toolchain-ext-${SDK_VERSION}"
TOOLCHAIN_OUTPUTNAME_task-populate-sdk-ext = "${TOOLCHAINEXT_OUTPUTNAME}"
+SDK_EXT_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.target.manifest"
+SDK_EXT_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.host.manifest"
+
SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} Extensible SDK"
python copy_buildsystem () {
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 620617c..80b40b6 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -382,14 +382,18 @@ class SDKTestContext(TestContext):
self.sdktestdir = sdktestdir
self.sdkenv = sdkenv
+ if not hasattr(self, 'target_manifest'):
+ self.target_manifest = d.getVar("SDK_TARGET_MANIFEST", True)
try:
- with open(d.getVar("SDK_TARGET_MANIFEST", True)) as f:
+ with open(self.target_manifest) as f:
self.pkgmanifest = f.read()
except IOError as e:
bb.fatal("No package manifest file found. Did you build the sdk image?\n%s" % e)
+ if not hasattr(self, 'host_manifest'):
+ self.host_manifest = d.getVar("SDK_HOST_MANIFEST", True)
try:
- with open(d.getVar("SDK_HOST_MANIFEST", True)) as f:
+ with open(self.host_manifest) as f:
self.hostpkgmanifest = f.read()
except IOError as e:
bb.fatal("No host package manifest file found. Did you build the sdk image?\n%s" % e)
@@ -406,6 +410,9 @@ class SDKTestContext(TestContext):
class SDKExtTestContext(SDKTestContext):
def __init__(self, d, sdktestdir, sdkenv):
+ self.target_manifest = d.getVar("SDK_EXT_TARGET_MANIFEST", True)
+ self.host_manifest = d.getVar("SDK_EXT_HOST_MANIFEST", True)
+
super(SDKExtTestContext, self).__init__(d, sdktestdir, sdkenv)
self.sdkextfilesdir = os.path.join(os.path.dirname(os.path.abspath(
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 18/22] oeqa/oetest: Fix compatibility SDK tests using eSDK.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (16 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 17/22] classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 19/22] oeqa/sdkext: Add devtool basic tests for eSDK Aníbal Limón
` (4 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
When run SDK tests over eSDK we need to use SDKExtTestContext
instead of SDKTestContext because if we use SDKTestContext search
for SDK manifest and depends on the SDK manifest generation so
populate_sdk needs to be executed.
Adds a compatibility mode flag to SDKExtTestContext for search tests
over sdk module instead of sdkext module and change testsdk calls
to comply with this new param.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testsdk.bbclass | 6 +++---
meta/lib/oeqa/oetest.py | 13 ++++++++++---
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 417510c..453ce40 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -5,14 +5,14 @@
TEST_LOG_DIR ?= "${WORKDIR}/testimage"
TESTSDKLOCK = "${TMPDIR}/testsdk.lock"
-def run_test_context(CTestContext, d, testdir, tcname, pn):
+def run_test_context(CTestContext, d, testdir, tcname, pn, *args):
import glob
import time
targets = glob.glob(d.expand(testdir + "/tc/environment-setup-*"))
for sdkenv in targets:
bb.plain("Testing %s" % sdkenv)
- tc = CTestContext(d, testdir, sdkenv)
+ tc = CTestContext(d, testdir, sdkenv, args)
# this is a dummy load of tests
# we are doing that to find compile errors in the tests themselves
@@ -112,7 +112,7 @@ def testsdkext_main(d):
try:
bb.plain("Running SDK Compatibility tests ...")
- run_test_context(SDKTestContext, d, testdir, tcname, pn)
+ run_test_context(SDKExtTestContext, d, testdir, tcname, pn, True)
finally:
pass
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 80b40b6..3e2ea0f 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -376,7 +376,7 @@ class ImageTestContext(TestContext):
setattr(oeRuntimeTest, "pscmd", "ps -ef" if oeTest.hasPackage("procps") else "ps")
class SDKTestContext(TestContext):
- def __init__(self, d, sdktestdir, sdkenv):
+ def __init__(self, d, sdktestdir, sdkenv, *args):
super(SDKTestContext, self).__init__(d)
self.sdktestdir = sdktestdir
@@ -409,9 +409,13 @@ class SDKTestContext(TestContext):
"auto").split() if t != "auto"]
class SDKExtTestContext(SDKTestContext):
- def __init__(self, d, sdktestdir, sdkenv):
+ def __init__(self, d, sdktestdir, sdkenv, *args):
self.target_manifest = d.getVar("SDK_EXT_TARGET_MANIFEST", True)
self.host_manifest = d.getVar("SDK_EXT_HOST_MANIFEST", True)
+ if args:
+ self.cm = args[0] # Compatibility mode for run SDK tests
+ else:
+ self.cm = False
super(SDKExtTestContext, self).__init__(d, sdktestdir, sdkenv)
@@ -419,7 +423,10 @@ class SDKExtTestContext(SDKTestContext):
oeqa.sdkext.__file__)), "files")
def _get_test_namespace(self):
- return "sdkext"
+ if self.cm:
+ return "sdk"
+ else:
+ return "sdkext"
def _get_test_suites(self):
return (self.d.getVar("TEST_SUITES_SDK_EXT", True) or "auto").split()
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 19/22] oeqa/sdkext: Add devtool basic tests for eSDK.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (17 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 18/22] oeqa/oetest: Fix compatibility SDK tests using eSDK Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 20/22] oeqa/sdkext/devtool.py: Add location test to ensure that devtool is the eSDK one Aníbal Limón
` (3 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Add simple myapp application is a C app that prints hello world
and exit.
Add devtool test for that this app to the workspace, build and
reset it.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/lib/oeqa/sdkext/devtool.py | 25 +++++++++++++++++++++++++
meta/lib/oeqa/sdkext/files/myapp/Makefile | 10 ++++++++++
meta/lib/oeqa/sdkext/files/myapp/myapp.c | 9 +++++++++
3 files changed, 44 insertions(+)
create mode 100644 meta/lib/oeqa/sdkext/devtool.py
create mode 100644 meta/lib/oeqa/sdkext/files/myapp/Makefile
create mode 100644 meta/lib/oeqa/sdkext/files/myapp/myapp.c
diff --git a/meta/lib/oeqa/sdkext/devtool.py b/meta/lib/oeqa/sdkext/devtool.py
new file mode 100644
index 0000000..0262ed3
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/devtool.py
@@ -0,0 +1,25 @@
+import shutil
+
+from oeqa.oetest import oeSDKExtTest
+from oeqa.utils.decorators import *
+
+class DevtoolTest(oeSDKExtTest):
+
+ @classmethod
+ def setUpClass(self):
+ self.myapp_src = os.path.join(self.tc.sdkextfilesdir, "myapp")
+ self.myapp_dst = os.path.join(self.tc.sdktestdir, "myapp")
+ shutil.copytree(self.myapp_src, self.myapp_dst)
+
+ def test_devtool_add_reset(self):
+ self._run('devtool add myapp %s' % self.myapp_dst)
+ self._run('devtool reset myapp')
+
+ def test_devtool_build(self):
+ self._run('devtool add myapp %s' % self.myapp_dst)
+ self._run('devtool build myapp')
+ self._run('devtool reset myapp')
+
+ @classmethod
+ def tearDownClass(self):
+ shutil.rmtree(self.myapp_dst)
diff --git a/meta/lib/oeqa/sdkext/files/myapp/Makefile b/meta/lib/oeqa/sdkext/files/myapp/Makefile
new file mode 100644
index 0000000..abd91be
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/files/myapp/Makefile
@@ -0,0 +1,10 @@
+all: myapp
+
+myapp: myapp.o
+ $(CC) $(LDFLAGS) $< -o $@
+
+myapp.o: myapp.c
+ $(CC) $(CFLAGS) -c $< -o $@
+
+clean:
+ rm -rf myapp.o myapp
diff --git a/meta/lib/oeqa/sdkext/files/myapp/myapp.c b/meta/lib/oeqa/sdkext/files/myapp/myapp.c
new file mode 100644
index 0000000..f0b63f0
--- /dev/null
+++ b/meta/lib/oeqa/sdkext/files/myapp/myapp.c
@@ -0,0 +1,9 @@
+#include <stdio.h>
+
+int
+main(int argc, char *argv[])
+{
+ printf("Hello world\n");
+
+ return 0;
+}
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 20/22] oeqa/sdkext/devtool.py: Add location test to ensure that devtool is the eSDK one.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (18 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 19/22] oeqa/sdkext: Add devtool basic tests for eSDK Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 21/22] classes/testsdk: Add help information on how to run tests Aníbal Limón
` (2 subsequent siblings)
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/lib/oeqa/sdkext/devtool.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/meta/lib/oeqa/sdkext/devtool.py b/meta/lib/oeqa/sdkext/devtool.py
index 0262ed3..c5bb310 100644
--- a/meta/lib/oeqa/sdkext/devtool.py
+++ b/meta/lib/oeqa/sdkext/devtool.py
@@ -11,10 +11,17 @@ class DevtoolTest(oeSDKExtTest):
self.myapp_dst = os.path.join(self.tc.sdktestdir, "myapp")
shutil.copytree(self.myapp_src, self.myapp_dst)
+ def test_devtool_location(self):
+ output = self._run('which devtool')
+ self.assertEqual(output.startswith(self.tc.sdktestdir), True, \
+ msg="Seems that devtool isn't the eSDK one: %s" % output)
+
+ @skipUnlessPassed('test_devtool_location')
def test_devtool_add_reset(self):
self._run('devtool add myapp %s' % self.myapp_dst)
self._run('devtool reset myapp')
+ @skipUnlessPassed('test_devtool_location')
def test_devtool_build(self):
self._run('devtool add myapp %s' % self.myapp_dst)
self._run('devtool build myapp')
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 21/22] classes/testsdk: Add help information on how to run tests.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (19 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 20/22] oeqa/sdkext/devtool.py: Add location test to ensure that devtool is the eSDK one Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 22/22] classes/testimage: Fix exportTests function Aníbal Limón
2016-02-04 22:49 ` [PATCH v3 00/22] Add Extensible SDK test suite Paul Eggleton
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core
Cc: paul.eggleton, benjamin.esquivel, Aníbal Limón
From: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
---
meta/classes/testsdk.bbclass | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 453ce40..487f40c 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -2,6 +2,16 @@
#
# Released under the MIT license (see COPYING.MIT)
+# testsdk.bbclass enables testing for SDK and Extensible SDK
+#
+# For run SDK tests you need to do,
+# - bitbake core-image-sato -c populate_sdk
+# - bitbake core-image-sato -c testsdk
+#
+# For run eSDK tests you need to do,
+# - bitbake core-image-sato -c populate_sdk_ext
+# - bitbake core-image-sato -c testsdkext
+
TEST_LOG_DIR ?= "${WORKDIR}/testimage"
TESTSDKLOCK = "${TMPDIR}/testsdk.lock"
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v3 22/22] classes/testimage: Fix exportTests function.
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (20 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 21/22] classes/testsdk: Add help information on how to run tests Aníbal Limón
@ 2016-02-04 22:29 ` Aníbal Limón
2016-02-04 22:49 ` [PATCH v3 00/22] Add Extensible SDK test suite Paul Eggleton
22 siblings, 0 replies; 24+ messages in thread
From: Aníbal Limón @ 2016-02-04 22:29 UTC (permalink / raw)
To: openembedded-core; +Cc: paul.eggleton, benjamin.esquivel
With new structure of TestContext now holds suite and variable
that contains unittest instances, it can't be exported using
JSON causing and exception.
Adds the suite variable for avoid export it.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/testimage.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 5fafda1..7fff29f 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -112,7 +112,7 @@ def exportTests(d,tc):
savedata["host_dumper"] = {}
for key in tc.__dict__:
# special cases
- if key != "d" and key != "target" and key != "host_dumper":
+ if key not in ['d', 'target', 'host_dumper', 'suite']:
savedata[key] = getattr(tc, key)
savedata["target"]["ip"] = tc.target.ip or d.getVar("TEST_TARGET_IP", True)
savedata["target"]["server_ip"] = tc.target.server_ip or d.getVar("TEST_SERVER_IP", True)
--
2.1.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v3 00/22] Add Extensible SDK test suite
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
` (21 preceding siblings ...)
2016-02-04 22:29 ` [PATCH v3 22/22] classes/testimage: Fix exportTests function Aníbal Limón
@ 2016-02-04 22:49 ` Paul Eggleton
22 siblings, 0 replies; 24+ messages in thread
From: Paul Eggleton @ 2016-02-04 22:49 UTC (permalink / raw)
To: Aníbal Limón, openembedded-core; +Cc: benjamin.esquivel
On Thu, 04 Feb 2016 16:29:17 Aníbal Limón wrote:
> Summary of changes,
>
> - Add new class called testsdk that now have the tests for SDK and eSDK.
> - TestContext -> {Image, SDK, SDKExt}: Remove all duplicate code inside
> testimage and testsdk class move into oeqa/oetest and create class per
> type of Test.
> - Extensible SDK fixes usage with proxies, when eSDK is configuring it
> executes setscene tasks that needs network acces if was build with
sstate
> mirrors.
>
> Testing was made building/running SDK and eSDK for core-image-minimal and
> core-image-sato, also testimage was run in core-image-minimal and
> core-image-sato with QemuRemote and SimpleRemote to ensure that nothing was
> break.
>
> This patchset DEPENDS on bitbake change to export proxies and NEEDS to be
> MERGED before, see: bb/fetch2: Move export_proxies function from wget to
> utils.
>
> These v3 is the same of v2 on code only fixes issues of squashing and one
> for don't make reverts in the patch series.
>
> The following changes since commit b33e440cc6fbd703e8045d94b806790343e72eb6:
>
> libical: Work around hardcoded paths in pkgconfig file (2016-02-04
> 13:09:56 +0000)
>
> are available in the git repository at:
>
> git://git.yoctoproject.org/poky-contrib alimon/esdk_testsuite_v3
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/esdk_testsu
> ite_v3
>
> Aníbal Limón (22):
> testimage: Modularize helper functions for get test lists.
> classes/testsdk: Add new class testsdk.
> classes/testimage: Add defeault inherit for testsdk.
> populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME.
> get_test_suites: Add sdkext type for load test suites.
> classes/testsdk: Add testsdkext task only install.
> bb/fetch2: Move export_proxies function from wget to utils.
> classes/testsdk: Add call to export_proxies on testsdkext.
> toolchain-shar-extract.sh: Add proxy variable to new env.
> testimage/testsdk: Modularize TestContext.
> testimage/testsdk: Move get test suites routine inside TestContext.
> oetest.py/TestContext: Move loadTests and runTests inside it.
> classes/testsdk: Add function run_test_context
> oeqa/oetest: oeSDKTest when run a command redirect env output to null
> classes/testsdk: Add compatibility SDK testsuite to eSDK
> testsdkext: Add skeleton for support Extensible SDK tests.
> classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and
> SDK_EXT_HOST_MANIFEST
> oeqa/oetest: Fix compatibility SDK tests using eSDK.
> oeqa/sdkext: Add devtool basic tests for eSDK.
> oeqa/sdkext/devtool.py: Add location test to ensure that devtool is
> the eSDK one.
> classes/testsdk: Add help information on how to run tests.
> classes/testimage: Fix exportTests function.
Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2016-02-04 22:49 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-04 22:29 [PATCH v3 00/22] Add Extensible SDK test suite Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 01/22] testimage: Modularize helper functions for get test lists Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 02/22] classes/testsdk: Add new class testsdk Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 03/22] classes/testimage: Add defeault inherit for testsdk Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 04/22] populate_sdk_ext: Set TOOLCHAINEXT_OUTPUTNAME Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 05/22] get_test_suites: Add sdkext type for load test suites Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 06/22] classes/testsdk: Add testsdkext task only install Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 07/22] bb/fetch2: Move export_proxies function from wget to utils Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 08/22] classes/testsdk: Add call to export_proxies on testsdkext Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 09/22] toolchain-shar-extract.sh: Add proxy variable to new env Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 10/22] testimage/testsdk: Modularize TestContext Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 11/22] testimage/testsdk: Move get test suites routine inside TestContext Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 12/22] oetest.py/TestContext: Move loadTests and runTests inside it Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 13/22] classes/testsdk: Add function run_test_context Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 14/22] oeqa/oetest: oeSDKTest when run a command redirect env output to null Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 15/22] classes/testsdk: Add compatibility SDK testsuite to eSDK Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 16/22] testsdkext: Add skeleton for support Extensible SDK tests Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 17/22] classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 18/22] oeqa/oetest: Fix compatibility SDK tests using eSDK Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 19/22] oeqa/sdkext: Add devtool basic tests for eSDK Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 20/22] oeqa/sdkext/devtool.py: Add location test to ensure that devtool is the eSDK one Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 21/22] classes/testsdk: Add help information on how to run tests Aníbal Limón
2016-02-04 22:29 ` [PATCH v3 22/22] classes/testimage: Fix exportTests function Aníbal Limón
2016-02-04 22:49 ` [PATCH v3 00/22] Add Extensible SDK test suite Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.