[PATCH 01/12] build-appliance: Drop test which is already automated

[PATCH 01/12] build-appliance: Drop test which is already automated

[Richard Purdie]

The nightly-build-appliance target already tests this particular build.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/manual/build-appliance.json | 26 -----------------------
 1 file changed, 26 deletions(-)

diff --git a/meta/lib/oeqa/manual/build-appliance.json b/meta/lib/oeqa/manual/build-appliance.json
index b8f89275d08..70f8c72c9bb 100644
--- a/meta/lib/oeqa/manual/build-appliance.json
+++ b/meta/lib/oeqa/manual/build-appliance.json
@@ -1,30 +1,4 @@
 [
-    {
-        "test": {
-            "@alias": "build-appliance.build-appliance.Bitbake_build-appliance-image",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "Get poky source code and prepare the build environment",
-                    "expected_results": "bitbake build-appliance-image is successful  "
-                },
-                "2": {
-                    "action": "Set MACHINE to qemux86 and add the following line to conf/local.conf :  SRCREV_pn-build-appliance-image = \"${AUTOREV}\"",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "Run \"bitbake build-appliance-image\" \n \n",
-                    "expected_results": ""
-                }
-            },
-            "summary": "Bitbake_build-appliance-image"
-        }
-    },
     {
         "test": {
             "@alias": "build-appliance.build-appliance.Build_core-image-minimal_with_build-appliance-image",
-- 
2.17.1

[PATCH 02/12] metadata_scm: Drop orphaned monotone functions

[Richard Purdie]

Its been 'a while' since we used monotone, drop these revision
function remnants.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/metadata_scm.bbclass | 24 ------------------------
 1 file changed, 24 deletions(-)

diff --git a/meta/classes/metadata_scm.bbclass b/meta/classes/metadata_scm.bbclass
index 658e2f883b0..58bb4c555a2 100644
--- a/meta/classes/metadata_scm.bbclass
+++ b/meta/classes/metadata_scm.bbclass
@@ -12,30 +12,6 @@ def base_detect_branch(d):
 def base_get_scmbasepath(d):
     return os.path.join(d.getVar('COREBASE'), 'meta')
 
-def base_get_metadata_monotone_branch(path, d):
-    monotone_branch = "<unknown>"
-    try:
-        with open("%s/_MTN/options" % path) as f:
-            monotone_branch = f.read().strip()
-            if monotone_branch.startswith( "database" ):
-                monotone_branch_words = monotone_branch.split()
-                monotone_branch = monotone_branch_words[ monotone_branch_words.index( "branch" )+1][1:-1]
-    except:
-        pass
-    return monotone_branch
-
-def base_get_metadata_monotone_revision(path, d):
-    monotone_revision = "<unknown>"
-    try:
-        with open("%s/_MTN/revision" % path) as f:
-            monotone_revision = f.read().strip()
-            if monotone_revision.startswith( "format_version" ):
-                monotone_revision_words = monotone_revision.split()
-                monotone_revision = monotone_revision_words[ monotone_revision_words.index( "old_revision" )+1][1:-1]
-    except IOError:
-        pass
-    return monotone_revision
-
 def base_get_metadata_svn_revision(path, d):
     # This only works with older subversion. For newer versions 
     # this function will need to be fixed by someone interested
-- 
2.17.1

[PATCH 03/12] distcc: Handle upstream branch deletion for old releases

[Richard Purdie]

distcc upstream dropped the 3.2 branch which we reference in older project releases
the revisions are there, just the branch is not. In order to be able to continue
to build those old releases, adjust any mirror tarball to contain the missing branch.
We have updated the  newer point releases but the previous tags were failing to build.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/distcc/distcc_3.3.2.bb | 24 ++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/meta/recipes-devtools/distcc/distcc_3.3.2.bb b/meta/recipes-devtools/distcc/distcc_3.3.2.bb
index 0d069631653..a93604bc471 100644
--- a/meta/recipes-devtools/distcc/distcc_3.3.2.bb
+++ b/meta/recipes-devtools/distcc/distcc_3.3.2.bb
@@ -69,3 +69,27 @@ FILES_${PN} = " ${sysconfdir} \
 		${systemd_unitdir}/system/distcc.service"
 FILES_distcc-distmon-gnome = "  ${bindir}/distccmon-gnome \
 				${datadir}/distcc"
+
+
+#
+# distcc upstream dropped the 3.2 branch which we reference in older project releases
+# the revisions are there, just the branch is not. In order to be able to continue
+# to build those old releases, adjust any mirror tarball to contain the missing branch
+#
+fixup_distcc_mirror_tarball () {
+	TBALL=${DL_DIR}/git2_github.com.distcc.distcc.git.tar.gz
+	if [ -f $TBALL ]; then
+		TDIR=`mktemp -d`
+		cd $TDIR
+		tar -xzf $TBALL
+		set +e
+		git rev-parse --verify 3.2
+		if [ "$?" != "0" ]; then
+			git branch 3.2 d8b18df3e9dcbe4f092bed565835d3975e99432c
+			tar -czf $TBALL *
+		fi
+		set -e
+		rm -rf $TDIR/*
+	fi
+}
+do_fetch[postfuncs] += "fixup_distcc_mirror_tarball"
-- 
2.17.1

[PATCH 04/12] oeqa/selftest/recipetool: Fix problems from changing upstream source

[Richard Purdie]

The upstream source tarball checksums changed. Use the copy from our source
mirror to avoid failures.

[YOCTO #12979]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index cd15df11a93..06f980e1b0e 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -433,12 +433,12 @@ class RecipetoolTests(RecipetoolBase):
         temprecipe = os.path.join(self.tempdir, 'recipe')
         os.makedirs(temprecipe)
         recipefile = os.path.join(temprecipe, 'navit_0.5.0.bb')
-        srcuri = 'http://downloads.sourceforge.net/project/navit/v0.5.0/navit-0.5.0.tar.gz'
+        srcuri = 'http://downloads.yoctoproject.org/mirror/sources/navit-0.5.0.tar.gz'
         result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
         self.assertTrue(os.path.isfile(recipefile))
         checkvars = {}
         checkvars['LICENSE'] = set(['Unknown', 'GPLv2', 'LGPLv2'])
-        checkvars['SRC_URI'] = 'http://downloads.sourceforge.net/project/navit/v${PV}/navit-${PV}.tar.gz'
+        checkvars['SRC_URI'] = 'http://downloads.yoctoproject.org/mirror/sources/navit-${PV}.tar.gz'
         checkvars['SRC_URI[md5sum]'] = '242f398e979a6b8c0f3c802b63435b68'
         checkvars['SRC_URI[sha256sum]'] = '13353481d7fc01a4f64e385dda460b51496366bba0fd2cc85a89a0747910e94d'
         checkvars['DEPENDS'] = set(['freetype', 'zlib', 'openssl', 'glib-2.0', 'virtual/libgl', 'virtual/egl', 'gtk+', 'libpng', 'libsdl', 'freeglut', 'dbus-glib', 'fribidi'])
-- 
2.17.1

[PATCH 05/12] oeqa/runtime/cases: Improve test dependency information

[Richard Purdie]

Add the OEHasPackage decorator to a variety of tests so they determine
automatically if they should run against a given image.

To ensure tests can do this we need to move target operations such
as scp commands into the tests and out of the class startup/teardown.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/runtime/cases/date.py   |  2 ++
 meta/lib/oeqa/runtime/cases/df.py     |  2 ++
 meta/lib/oeqa/runtime/cases/ptest.py  |  2 ++
 meta/lib/oeqa/runtime/cases/python.py |  8 ++------
 meta/lib/oeqa/runtime/cases/rpm.py    | 26 ++++++++++----------------
 meta/lib/oeqa/runtime/cases/scp.py    |  2 ++
 meta/lib/oeqa/runtime/cases/ssh.py    |  2 ++
 meta/lib/oeqa/runtime/cases/stap.py   |  2 ++
 meta/lib/oeqa/runtime/cases/xorg.py   |  2 ++
 9 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/date.py b/meta/lib/oeqa/runtime/cases/date.py
index ece7338de73..0887b831f42 100644
--- a/meta/lib/oeqa/runtime/cases/date.py
+++ b/meta/lib/oeqa/runtime/cases/date.py
@@ -3,6 +3,7 @@ import re
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class DateTest(OERuntimeTestCase):
 
@@ -18,6 +19,7 @@ class DateTest(OERuntimeTestCase):
 
     @OETestID(211)
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['coreutils', 'busybox'])
     def test_date(self):
         (status, output) = self.target.run('date +"%Y-%m-%d %T"')
         msg = 'Failed to get initial date, output: %s' % output
diff --git a/meta/lib/oeqa/runtime/cases/df.py b/meta/lib/oeqa/runtime/cases/df.py
index aecc32d7ce3..e0b6bb839d1 100644
--- a/meta/lib/oeqa/runtime/cases/df.py
+++ b/meta/lib/oeqa/runtime/cases/df.py
@@ -1,11 +1,13 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class DfTest(OERuntimeTestCase):
 
     @OETestID(234)
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['coreutils', 'busybox'])
     def test_df(self):
         cmd = "df / | sed -n '2p' | awk '{print $4}'"
         (status,output) = self.target.run(cmd)
diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index 77ae7b6b863..0972a583eeb 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -5,6 +5,7 @@ from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 from oeqa.utils.logparser import Lparser, Result
 
 class PtestRunnerTest(OERuntimeTestCase):
@@ -52,6 +53,7 @@ class PtestRunnerTest(OERuntimeTestCase):
     @OETestID(1600)
     @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['ptest-runner'])
     @unittest.expectedFailure
     def test_ptestrunner(self):
         status, output = self.target.run('which ptest-runner', 0)
diff --git a/meta/lib/oeqa/runtime/cases/python.py b/meta/lib/oeqa/runtime/cases/python.py
index 4419a9f6394..66ab4d25f37 100644
--- a/meta/lib/oeqa/runtime/cases/python.py
+++ b/meta/lib/oeqa/runtime/cases/python.py
@@ -1,16 +1,12 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class PythonTest(OERuntimeTestCase):
-    @classmethod
-    def setUpClass(cls):
-        import unittest
-        if "python3-core" not in cls.tc.image_packages:
-            raise unittest.SkipTest("Python3 not on target")
-
     @OETestID(965)
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['python3-core'])
     def test_python3(self):
         cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
         status, output = self.target.run(cmd)
diff --git a/meta/lib/oeqa/runtime/cases/rpm.py b/meta/lib/oeqa/runtime/cases/rpm.py
index 1e5e4631d37..0213c4b57e8 100644
--- a/meta/lib/oeqa/runtime/cases/rpm.py
+++ b/meta/lib/oeqa/runtime/cases/rpm.py
@@ -10,11 +10,6 @@ from oeqa.core.utils.path import findFile
 
 class RpmBasicTest(OERuntimeTestCase):
 
-    @classmethod
-    def setUpClass(cls):
-        if cls.tc.td['PACKAGE_CLASSES'].split()[0] != 'package_rpm':
-            cls.skipTest('Tests require image to be build from rpm')
-
     @OETestID(960)
     @OEHasPackage(['rpm'])
     @OETestDepends(['ssh.SSHTest.test_ssh'])
@@ -34,30 +29,25 @@ class RpmInstallRemoveTest(OERuntimeTestCase):
 
     @classmethod
     def setUpClass(cls):
-        if cls.tc.td['PACKAGE_CLASSES'].split()[0] != 'package_rpm':
-            cls.skipTest('Tests require image to be build from rpm')
-
         pkgarch = cls.td['TUNE_PKGARCH'].replace('-', '_')
         rpmdir = os.path.join(cls.tc.td['DEPLOY_DIR'], 'rpm', pkgarch)
         # Pick base-passwd-doc as a test file to get installed, because it's small
         # and it will always be built for standard targets
         rpm_doc = 'base-passwd-doc-*.%s.rpm' % pkgarch
+        if not os.path.exists(rpmdir):
+            return
         for f in fnmatch.filter(os.listdir(rpmdir), rpm_doc):
-            test_file = os.path.join(rpmdir, f)
-        dst = '/tmp/base-passwd-doc.rpm'
-        cls.tc.target.copyTo(test_file, dst)
-
-    @classmethod
-    def tearDownClass(cls):
-        dst = '/tmp/base-passwd-doc.rpm'
-        cls.tc.target.run('rm -f %s' % dst)
+            cls.test_file = os.path.join(rpmdir, f)
+        cls.dst = '/tmp/base-passwd-doc.rpm'
 
     @OETestID(192)
     @OETestDepends(['rpm.RpmBasicTest.test_rpm_help'])
     def test_rpm_install(self):
+        self.tc.target.copyTo(self.test_file, self.dst)
         status, output = self.target.run('rpm -ivh /tmp/base-passwd-doc.rpm')
         msg = 'Failed to install base-passwd-doc package: %s' % output
         self.assertEqual(status, 0, msg=msg)
+        self.tc.target.run('rm -f %s' % self.dst)
 
     @OETestID(194)
     @OETestDepends(['rpm.RpmInstallRemoveTest.test_rpm_install'])
@@ -118,6 +108,8 @@ class RpmInstallRemoveTest(OERuntimeTestCase):
         msg =  'Failed to find database files under /var/lib/rpm/ as __db.xxx'
         self.assertEqual(0, status, msg=msg)
 
+        self.tc.target.copyTo(self.test_file, self.dst)
+
         # Remove the package just in case
         self.target.run('rpm -e base-passwd-doc')
 
@@ -131,6 +123,8 @@ class RpmInstallRemoveTest(OERuntimeTestCase):
             msg = 'Failed to remove base-passwd-doc package. Reason: {}'.format(output)
             self.assertEqual(0, status, msg=msg)
 
+        self.tc.target.run('rm -f %s' % self.dst)
+
         # if using systemd this should ensure all entries are flushed to /var
         status, output = self.target.run("journalctl --sync")
         # Get the amount of entries in the log file
diff --git a/meta/lib/oeqa/runtime/cases/scp.py b/meta/lib/oeqa/runtime/cases/scp.py
index f488a6175bb..8f895da95a2 100644
--- a/meta/lib/oeqa/runtime/cases/scp.py
+++ b/meta/lib/oeqa/runtime/cases/scp.py
@@ -4,6 +4,7 @@ from tempfile import mkstemp
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class ScpTest(OERuntimeTestCase):
 
@@ -20,6 +21,7 @@ class ScpTest(OERuntimeTestCase):
 
     @OETestID(220)
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['openssh-scp', 'dropbear'])
     def test_scp_file(self):
         dst = '/tmp/test_scp_file'
 
diff --git a/meta/lib/oeqa/runtime/cases/ssh.py b/meta/lib/oeqa/runtime/cases/ssh.py
index eca167969a7..0b1ea7bcc25 100644
--- a/meta/lib/oeqa/runtime/cases/ssh.py
+++ b/meta/lib/oeqa/runtime/cases/ssh.py
@@ -1,11 +1,13 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class SSHTest(OERuntimeTestCase):
 
     @OETestID(224)
     @OETestDepends(['ping.PingTest.test_ping'])
+    @OEHasPackage(['dropbear', 'openssh-sshd'])
     def test_ssh(self):
         (status, output) = self.target.run('uname -a')
         self.assertEqual(status, 0, msg='SSH Test failed: %s' % output)
diff --git a/meta/lib/oeqa/runtime/cases/stap.py b/meta/lib/oeqa/runtime/cases/stap.py
index 96e197a2d68..747c1d95177 100644
--- a/meta/lib/oeqa/runtime/cases/stap.py
+++ b/meta/lib/oeqa/runtime/cases/stap.py
@@ -4,6 +4,7 @@ from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class StapTest(OERuntimeTestCase):
 
@@ -22,6 +23,7 @@ class StapTest(OERuntimeTestCase):
     @skipIfNotFeature('tools-profile',
                       'Test requires tools-profile to be in IMAGE_FEATURES')
     @OETestDepends(['kernelmodule.KernelModuleTest.test_kernel_module'])
+    @OEHasPackage(['systemtap'])
     def test_stap(self):
         cmds = [
             'cd /usr/src/kernel && make scripts prepare',
diff --git a/meta/lib/oeqa/runtime/cases/xorg.py b/meta/lib/oeqa/runtime/cases/xorg.py
index 2124813e3c2..82521c69ac2 100644
--- a/meta/lib/oeqa/runtime/cases/xorg.py
+++ b/meta/lib/oeqa/runtime/cases/xorg.py
@@ -2,6 +2,7 @@ from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class XorgTest(OERuntimeTestCase):
 
@@ -9,6 +10,7 @@ class XorgTest(OERuntimeTestCase):
     @skipIfNotFeature('x11-base',
                       'Test requires x11 to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['xserver-nodm-init'])
     def test_xorg_running(self):
         cmd ='%s | grep -v xinit | grep [X]org' % self.tc.target_cmds['ps']
         status, output = self.target.run(cmd)
-- 
2.17.1

[PATCH 06/12] oeqa/utils/buildproject: Only clean files if we've done something

[Richard Purdie]

We should only be wiping out things on target if the tests have
actually run.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/utils/buildproject.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/lib/oeqa/utils/buildproject.py b/meta/lib/oeqa/utils/buildproject.py
index 721f35d996e..7e9b84955f5 100644
--- a/meta/lib/oeqa/utils/buildproject.py
+++ b/meta/lib/oeqa/utils/buildproject.py
@@ -24,9 +24,12 @@ class BuildProject(metaclass=ABCMeta):
             self.fname = foldername
         else:
             self.fname = re.sub(r'\.tar\.bz2$|\.tar\.gz$|\.tar\.xz$', '', self.archive)
+        self.needclean = False
 
     # Download self.archive to self.localarchive
     def _download_archive(self):
+
+        self.needclean = True
         if self.dl_dir and os.path.exists(os.path.join(self.dl_dir, self.archive)):
             shutil.copyfile(os.path.join(self.dl_dir, self.archive), self.localarchive)
             return
@@ -51,5 +54,7 @@ class BuildProject(metaclass=ABCMeta):
         return self._run('cd %s; make install %s' % (self.targetdir, install_args))
 
     def clean(self):
+        if not self.needclean:
+             return
         self._run('rm -rf %s' % self.targetdir)
         subprocess.check_call('rm -f %s' % self.localarchive, shell=True)
-- 
2.17.1

[PATCH 07/12] oeqa/runtime/utils/targetbuildproject: Use a subdir within ~/

[Richard Purdie]

Without doing this, the code can and sometimes does try and do "rm ~/"
which I think we'd all prefer it didn't.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/runtime/utils/targetbuildproject.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/utils/targetbuildproject.py b/meta/lib/oeqa/runtime/utils/targetbuildproject.py
index 5af55d736e4..de17ba01f89 100644
--- a/meta/lib/oeqa/runtime/utils/targetbuildproject.py
+++ b/meta/lib/oeqa/runtime/utils/targetbuildproject.py
@@ -7,10 +7,12 @@ class TargetBuildProject(BuildProject):
 
     def __init__(self, target, uri, foldername=None, dl_dir=None):
         self.target = target
-        self.targetdir = "~/"
+        self.targetdir = "~/buildtest/"
         BuildProject.__init__(self, uri, foldername, dl_dir=dl_dir)
 
     def download_archive(self):
+        self.target.run("mkdir " + self.targetdir + " || true")
+
         self._download_archive()
 
         status, output = self.target.copyTo(self.localarchive, self.targetdir)
-- 
2.17.1

[PATCH 08/12] testimage: Simplfy DEFAULT_TEST_SUITES logic

[Richard Purdie]

Now that the tests have correct markup to automatically determine which images
they can run against, clean up the default test suites logic to be simpler
and not image specific.

Some cleanup of the compiler tests still needs to be completed but this
is a good first step.

The only downside to this is more noise during testing as we now see
many skipped messages for simple images like core-image-minimal.

The auto type is being removed since it currently breaks badly due to the
socat mandatory inclusion from the meta-selftest layer which is a problem
which needs to be addressed seperately.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/testimage.bbclass | 31 ++++++-------------------------
 1 file changed, 6 insertions(+), 25 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index ba2d9c491b2..b8acd3ce782 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -40,31 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?= "${WORKDIR}/testimage/packages"
 TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
 TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
 
-PKGMANTESTSUITE = "\
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'dnf rpm', '', d)} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'ipk', 'opkg', '', d)} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'deb', 'apt', '', d)} \
-    "
-SYSTEMDSUITE = "${@bb.utils.filter('DISTRO_FEATURES', 'systemd', d)}"
-MINTESTSUITE = "ping"
-NETTESTSUITE = "${MINTESTSUITE} ssh df date scp oe_syslog ${SYSTEMDSUITE}"
-DEVTESTSUITE = "gcc kernelmodule ldd"
-PTESTTESTSUITE = "${MINTESTSUITE} ssh scp ptest"
-
-DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto"
-DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest"
-DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${PKGMANTESTSUITE} ptest"
-DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${PKGMANTESTSUITE} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest gi"
-DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate perl parselogs python ${PKGMANTESTSUITE} xorg ptest gi stap"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${PKGMANTESTSUITE} ptest gi"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${PKGMANTESTSUITE} ptest gi stap"
-DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto"
+BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"
+DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
+
+DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
+DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
+DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
 
 # aarch64 has no graphics
 DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
-- 
2.17.1

[PATCH 09/12] testimage: Remove duplicate dependencies

[Richard Purdie]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/testimage.bbclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index b8acd3ce782..b00d47441b2 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -66,12 +66,10 @@ TEST_TARGET ?= "qemu"
 TESTIMAGEDEPENDS = ""
 TESTIMAGEDEPENDS_append_qemuall = " qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot qemu-helper-native:do_addto_recipe_sysroot"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'cpio-native:do_populate_sysroot', '', d)}"
-TESTIMAGEDEPENDS_append_qemuall = " ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'cpio-native:do_populate_sysroot', '', d)}"
-TESTIMAGEDEPENDS_append_qemuall = " ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'createrepo-c-native:do_populate_sysroot', '', d)}"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'dnf-native:do_populate_sysroot', '', d)}"
+TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'createrepo-c-native:do_populate_sysroot', '', d)}"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'ipk', 'opkg-utils-native:do_populate_sysroot package-index:do_package_index', '', d)}"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'deb', 'apt-native:do_populate_sysroot  package-index:do_package_index', '', d)}"
-TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'createrepo-c-native:do_populate_sysroot', '', d)}"
 
 TESTIMAGELOCK = "${TMPDIR}/testimage.lock"
 TESTIMAGELOCK_qemuall = ""
-- 
2.17.1

[PATCH 10/12] oeqa/runtime/cases: Improve dependencies of kernel/gcc/build tests

[Richard Purdie]

Mark up these tests as needing a compiler, make and kernel source code
as appropriate, the image feature requirements can then be retired.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/runtime/cases/buildcpio.py      |  9 +++++----
 .../lib/oeqa/runtime/cases/buildgalculator.py |  9 +++++----
 meta/lib/oeqa/runtime/cases/buildlzip.py      | 12 +++++-------
 meta/lib/oeqa/runtime/cases/gcc.py            | 19 ++++++++-----------
 meta/lib/oeqa/runtime/cases/kernelmodule.py   |  8 ++++++--
 meta/lib/oeqa/runtime/cases/stap.py           |  4 ++--
 6 files changed, 31 insertions(+), 30 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/buildcpio.py b/meta/lib/oeqa/runtime/cases/buildcpio.py
index 79b22d04dd0..a61d1e03043 100644
--- a/meta/lib/oeqa/runtime/cases/buildcpio.py
+++ b/meta/lib/oeqa/runtime/cases/buildcpio.py
@@ -1,7 +1,7 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
-from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 from oeqa.runtime.utils.targetbuildproject import TargetBuildProject
 
@@ -13,17 +13,18 @@ class BuildCpioTest(OERuntimeTestCase):
         cls.project = TargetBuildProject(cls.tc.target,
                                          uri,
                                          dl_dir = cls.tc.td['DL_DIR'])
-        cls.project.download_archive()
 
     @classmethod
     def tearDownClass(cls):
         cls.project.clean()
 
     @OETestID(205)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['gcc'])
+    @OEHasPackage(['make'])
+    @OEHasPackage(['autoconf'])
     def test_cpio(self):
+        self.project.download_archive()
         self.project.run_configure()
         self.project.run_make()
         self.project.run_install()
diff --git a/meta/lib/oeqa/runtime/cases/buildgalculator.py b/meta/lib/oeqa/runtime/cases/buildgalculator.py
index 7c9d4a392bb..a0a00320835 100644
--- a/meta/lib/oeqa/runtime/cases/buildgalculator.py
+++ b/meta/lib/oeqa/runtime/cases/buildgalculator.py
@@ -1,7 +1,7 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
-from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 from oeqa.runtime.utils.targetbuildproject import TargetBuildProject
 
@@ -13,16 +13,17 @@ class GalculatorTest(OERuntimeTestCase):
         cls.project = TargetBuildProject(cls.tc.target,
                                          uri,
                                          dl_dir = cls.tc.td['DL_DIR'])
-        cls.project.download_archive()
 
     @classmethod
     def tearDownClass(cls):
         cls.project.clean()
 
     @OETestID(1526)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['gcc'])
+    @OEHasPackage(['make'])
+    @OEHasPackage(['autoconf'])
     def test_galculator(self):
+        self.project.download_archive()
         self.project.run_configure()
         self.project.run_make()
diff --git a/meta/lib/oeqa/runtime/cases/buildlzip.py b/meta/lib/oeqa/runtime/cases/buildlzip.py
index ca3fead2e49..5b455a07906 100644
--- a/meta/lib/oeqa/runtime/cases/buildlzip.py
+++ b/meta/lib/oeqa/runtime/cases/buildlzip.py
@@ -1,7 +1,7 @@
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
-from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 from oeqa.runtime.utils.targetbuildproject import TargetBuildProject
 
@@ -14,21 +14,19 @@ class BuildLzipTest(OERuntimeTestCase):
         cls.project = TargetBuildProject(cls.tc.target,
                                          uri,
                                          dl_dir = cls.tc.td['DL_DIR'])
-        cls.project.download_archive()
 
     @classmethod
     def tearDownClass(cls):
         cls.project.clean()
 
     @OETestID(206)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['gcc'])
+    @OEHasPackage(['make'])
+    @OEHasPackage(['autoconf'])
     def test_lzip(self):
+        self.project.download_archive()
         self.project.run_configure()
         self.project.run_make()
         self.project.run_install()
 
-    @classmethod
-    def tearDownClass(self):
-        self.project.clean()
diff --git a/meta/lib/oeqa/runtime/cases/gcc.py b/meta/lib/oeqa/runtime/cases/gcc.py
index 911083156fb..8265c59f239 100644
--- a/meta/lib/oeqa/runtime/cases/gcc.py
+++ b/meta/lib/oeqa/runtime/cases/gcc.py
@@ -3,12 +3,12 @@ import os
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
-from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class GccCompileTest(OERuntimeTestCase):
 
     @classmethod
-    def setUpClass(cls):
+    def setUp(cls):
         dst = '/tmp/'
         src = os.path.join(cls.tc.files_dir, 'test.c')
         cls.tc.target.copyTo(src, dst)
@@ -20,14 +20,13 @@ class GccCompileTest(OERuntimeTestCase):
         cls.tc.target.copyTo(src, dst)
 
     @classmethod
-    def tearDownClass(cls):
+    def tearDown(cls):
         files = '/tmp/test.c /tmp/test.o /tmp/test /tmp/testmakefile'
         cls.tc.target.run('rm %s' % files)
 
     @OETestID(203)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['gcc'])
     def test_gcc_compile(self):
         status, output = self.target.run('gcc /tmp/test.c -o /tmp/test -lm')
         msg = 'gcc compile failed, output: %s' % output
@@ -38,9 +37,8 @@ class GccCompileTest(OERuntimeTestCase):
         self.assertEqual(status, 0, msg=msg)
 
     @OETestID(200)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['g++'])
     def test_gpp_compile(self):
         status, output = self.target.run('g++ /tmp/test.c -o /tmp/test -lm')
         msg = 'g++ compile failed, output: %s' % output
@@ -51,9 +49,8 @@ class GccCompileTest(OERuntimeTestCase):
         self.assertEqual(status, 0, msg=msg)
 
     @OETestID(1142)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['g++'])
     def test_gpp2_compile(self):
         status, output = self.target.run('g++ /tmp/test.cpp -o /tmp/test -lm')
         msg = 'g++ compile failed, output: %s' % output
@@ -64,9 +61,9 @@ class GccCompileTest(OERuntimeTestCase):
         self.assertEqual(status, 0, msg=msg)
 
     @OETestID(204)
-    @skipIfNotFeature('tools-sdk',
-                      'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['gcc'])
+    @OEHasPackage(['make'])
     def test_make(self):
         status, output = self.target.run('cd /tmp; make -f testmakefile')
         msg = 'running make failed, output %s' % output
diff --git a/meta/lib/oeqa/runtime/cases/kernelmodule.py b/meta/lib/oeqa/runtime/cases/kernelmodule.py
index de1a5aa4457..27a2c35b711 100644
--- a/meta/lib/oeqa/runtime/cases/kernelmodule.py
+++ b/meta/lib/oeqa/runtime/cases/kernelmodule.py
@@ -4,11 +4,12 @@ from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.core.decorator.data import skipIfNotFeature
+from oeqa.runtime.decorator.package import OEHasPackage
 
 class KernelModuleTest(OERuntimeTestCase):
 
     @classmethod
-    def setUpClass(cls):
+    def setUp(cls):
         src = os.path.join(cls.tc.runtime_files_dir, 'hellomod.c')
         dst = '/tmp/hellomod.c'
         cls.tc.target.copyTo(src, dst)
@@ -18,7 +19,7 @@ class KernelModuleTest(OERuntimeTestCase):
         cls.tc.target.copyTo(src, dst)
 
     @classmethod
-    def tearDownClass(cls):
+    def tearDown(cls):
         files = '/tmp/Makefile /tmp/hellomod.c'
         cls.tc.target.run('rm %s' % files)
 
@@ -26,6 +27,9 @@ class KernelModuleTest(OERuntimeTestCase):
     @skipIfNotFeature('tools-sdk',
                       'Test requires tools-sdk to be in IMAGE_FEATURES')
     @OETestDepends(['gcc.GccCompileTest.test_gcc_compile'])
+    @OEHasPackage(['kernel-devsrc'])
+    @OEHasPackage(['make'])
+    @OEHasPackage(['gcc'])
     def test_kernel_module(self):
         cmds = [
             'cd /usr/src/kernel && make scripts prepare',
diff --git a/meta/lib/oeqa/runtime/cases/stap.py b/meta/lib/oeqa/runtime/cases/stap.py
index 747c1d95177..c492caffd67 100644
--- a/meta/lib/oeqa/runtime/cases/stap.py
+++ b/meta/lib/oeqa/runtime/cases/stap.py
@@ -9,13 +9,13 @@ from oeqa.runtime.decorator.package import OEHasPackage
 class StapTest(OERuntimeTestCase):
 
     @classmethod
-    def setUpClass(cls):
+    def setUp(cls):
         src = os.path.join(cls.tc.runtime_files_dir, 'hello.stp')
         dst = '/tmp/hello.stp'
         cls.tc.target.copyTo(src, dst)
 
     @classmethod
-    def tearDownClass(cls):
+    def tearDown(cls):
         files = '/tmp/hello.stp'
         cls.tc.target.run('rm %s' % files)
 
-- 
2.17.1

[PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[Richard Purdie]

Now the test markup of the development tools tests is complete, this
can be further tweaked to auto run the correct tests.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/testimage.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index b00d47441b2..4e9494296e1 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?= "${WORKDIR}/testimage/packages"
 TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
 TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
 
-BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"
-DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
+BASICTESTSUITE = "\
+    ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs \
+    logrotate connman systemd oe_syslog pam stap ldd xorg kernelmodule gcc \
+    buildcpio buildlzip buildgalculator"
 
 DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
 
 # aarch64 has no graphics
 DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
-- 
2.17.1

[PATCH 12/12] image-buildinfo, oeqa/selftest/containerimage: Ensure image-buildinfo doesn't break tests

[Richard Purdie]

Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
  ['./',
   './etc/',
-  './etc/build',
   './etc/default/',
   './etc/default/postinst',

Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/image-buildinfo.bbclass           | 2 ++
 meta/lib/oeqa/selftest/cases/containerimage.py | 1 +
 2 files changed, 3 insertions(+)

diff --git a/meta/classes/image-buildinfo.bbclass b/meta/classes/image-buildinfo.bbclass
index 213fb9cf9b6..87a6a1a4893 100644
--- a/meta/classes/image-buildinfo.bbclass
+++ b/meta/classes/image-buildinfo.bbclass
@@ -64,6 +64,8 @@ def buildinfo_target(d):
 
 # Write build information to target filesystem
 python buildinfo () {
+    if not d.getVar('IMAGE_BUILDINFO_FILE'):
+        return
     with open(d.expand('${IMAGE_ROOTFS}${IMAGE_BUILDINFO_FILE}'), 'w') as build:
         build.writelines((
             '''-----------------------
diff --git a/meta/lib/oeqa/selftest/cases/containerimage.py b/meta/lib/oeqa/selftest/cases/containerimage.py
index 99a5cc9e575..8deaae75d8a 100644
--- a/meta/lib/oeqa/selftest/cases/containerimage.py
+++ b/meta/lib/oeqa/selftest/cases/containerimage.py
@@ -39,6 +39,7 @@ class ContainerImageTests(OESelftestTestCase):
 IMAGE_FSTYPES = "container"
 PACKAGE_CLASSES = "package_ipk"
 IMAGE_FEATURES = ""
+IMAGE_BUILDINFO_FILE = ""
 """)
 
         bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
-- 
2.17.1

Re: [PATCH 08/12] testimage: Simplfy DEFAULT_TEST_SUITES logic

[akuster808]

On 11/9/18 6:29 AM, Richard Purdie wrote:
> Now that the tests have correct markup to automatically determine which images
> they can run against, clean up the default test suites logic to be simpler
> and not image specific.
> Some cleanup of the compiler tests still needs to be completed but this
> is a good first step.
>
> The only downside to this is more noise during testing as we now see
> many skipped messages for simple images like core-image-minimal.
>
> The auto type is being removed since it currently breaks badly due to the
> socat mandatory inclusion from the meta-selftest layer which is a problem
> which needs to be addressed seperately.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/testimage.bbclass | 31 ++++++-------------------------
>  1 file changed, 6 insertions(+), 25 deletions(-)
>
> diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
> index ba2d9c491b2..b8acd3ce782 100644
> --- a/meta/classes/testimage.bbclass
> +++ b/meta/classes/testimage.bbclass
> @@ -40,31 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?= "${WORKDIR}/testimage/packages"
>  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
>  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
>  
> -PKGMANTESTSUITE = "\
> -    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'dnf rpm', '', d)} \
> -    ${@bb.utils.contains('IMAGE_PKGTYPE', 'ipk', 'opkg', '', d)} \
> -    ${@bb.utils.contains('IMAGE_PKGTYPE', 'deb', 'apt', '', d)} \
> -    "
> -SYSTEMDSUITE = "${@bb.utils.filter('DISTRO_FEATURES', 'systemd', d)}"
> -MINTESTSUITE = "ping"
> -NETTESTSUITE = "${MINTESTSUITE} ssh df date scp oe_syslog ${SYSTEMDSUITE}"
> -DEVTESTSUITE = "gcc kernelmodule ldd"
> -PTESTTESTSUITE = "${MINTESTSUITE} ssh scp ptest"
> -
> -DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto"
> -DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest"
> -DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${PKGMANTESTSUITE} ptest"
> -DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${PKGMANTESTSUITE} \
> -    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest gi"
> -DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
> -    connman ${DEVTESTSUITE} logrotate perl parselogs python ${PKGMANTESTSUITE} xorg ptest gi stap"
> -DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${PKGMANTESTSUITE} ptest gi"
> -DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
> -    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${PKGMANTESTSUITE} ptest gi stap"
> -DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto"
> +BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"

So now you are forcing me to run all those test if I just want to run
"Ptests" for my layer that don't have any of those additional pieces.

can we re-instate a basic "PTEST" variable so the rest of use can
leverage a common define? if not, i will have do it locally in my layer.

Nak for now.

- armin

> +DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
> +
> +DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
> +DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
> +DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
>  
>  # aarch64 has no graphics
>  DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"

Re: [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[akuster808]

On 11/9/18 6:29 AM, Richard Purdie wrote:
> Now the test markup of the development tools tests is complete, this
> can be further tweaked to auto run the correct tests.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/testimage.bbclass | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
> index b00d47441b2..4e9494296e1 100644
> --- a/meta/classes/testimage.bbclass
> +++ b/meta/classes/testimage.bbclass
> @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?= "${WORKDIR}/testimage/packages"
>  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
>  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
>  
> -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"
> -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
> +BASICTESTSUITE = "\
> +    ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs \
> +    logrotate connman systemd oe_syslog pam stap ldd xorg kernelmodule gcc \
> +    buildcpio buildlzip buildgalculator"

Again Nak. You are forcing folks to have x11, systemd etc in the "Basic". 


- armin

>  
>  DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
>  
>  # aarch64 has no graphics
>  DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[akuster808]

On 11/9/18 6:29 AM, Richard Purdie wrote:
> The nightly-build-appliance target already tests this particular build.

If I am reading the AB code correctly, it is building for x86-64 where
as this is for x86.  A QA person would say they are not the same.

Are the resulting build results testing the same thing?

- armin

>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/lib/oeqa/manual/build-appliance.json | 26 -----------------------
>  1 file changed, 26 deletions(-)
>
> diff --git a/meta/lib/oeqa/manual/build-appliance.json b/meta/lib/oeqa/manual/build-appliance.json
> index b8f89275d08..70f8c72c9bb 100644
> --- a/meta/lib/oeqa/manual/build-appliance.json
> +++ b/meta/lib/oeqa/manual/build-appliance.json
> @@ -1,30 +1,4 @@
>  [
> -    {
> -        "test": {
> -            "@alias": "build-appliance.build-appliance.Bitbake_build-appliance-image",
> -            "author": [
> -                {
> -                    "email": "alexandru.c.georgescu@intel.com",
> -                    "name": "alexandru.c.georgescu@intel.com"
> -                }
> -            ],
> -            "execution": {
> -                "1": {
> -                    "action": "Get poky source code and prepare the build environment",
> -                    "expected_results": "bitbake build-appliance-image is successful  "
> -                },
> -                "2": {
> -                    "action": "Set MACHINE to qemux86 and add the following line to conf/local.conf :  SRCREV_pn-build-appliance-image = \"${AUTOREV}\"",
> -                    "expected_results": ""
> -                },
> -                "3": {
> -                    "action": "Run \"bitbake build-appliance-image\" \n \n",
> -                    "expected_results": ""
> -                }
> -            },
> -            "summary": "Bitbake_build-appliance-image"
> -        }
> -    },
>      {
>          "test": {
>              "@alias": "build-appliance.build-appliance.Build_core-image-minimal_with_build-appliance-image",

Re: [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[richard.purdie]

On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
> On 11/9/18 6:29 AM, Richard Purdie wrote:
> > Now the test markup of the development tools tests is complete,
> > this
> > can be further tweaked to auto run the correct tests.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/classes/testimage.bbclass | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> > 
> > diff --git a/meta/classes/testimage.bbclass
> > b/meta/classes/testimage.bbclass
> > index b00d47441b2..4e9494296e1 100644
> > --- a/meta/classes/testimage.bbclass
> > +++ b/meta/classes/testimage.bbclass
> > @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> > "${WORKDIR}/testimage/packages"
> >  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> >  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
> >  
> > -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
> > ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
> > stap ldd xorg"
> > -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
> > buildgalculator "
> > +BASICTESTSUITE = "\
> > +    ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> > parselogs \
> > +    logrotate connman systemd oe_syslog pam stap ldd xorg
> > kernelmodule gcc \
> > +    buildcpio buildlzip buildgalculator"
> 
> Again Nak. You are forcing folks to have x11, systemd etc in the
> "Basic". 

No, I'm not. If the image doesn't have x11 in it, the test will show
"skipped". If the image doesn't have systemd in it, it will show those
tests as "skipped". If you just want to test ptest, you can specify
that, which is what you had to do before this change anyway *just* to
run ptest.

So I'm not really seeing the problem?

The only risk with doing this is that a test may be skipped when it
shouldn't be. The plan is to verify the right set of test results were
obtained when the QA overall results from a release build are compared
against a known list of tests which should have run.

Cheers,

Richard

Re: [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[akuster808]

On 11/11/18 1:18 PM, richard.purdie@linuxfoundation.org wrote:
> On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
>> On 11/9/18 6:29 AM, Richard Purdie wrote:
>>> Now the test markup of the development tools tests is complete,
>>> this
>>> can be further tweaked to auto run the correct tests.
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> ---
>>>  meta/classes/testimage.bbclass | 8 ++++----
>>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/meta/classes/testimage.bbclass
>>> b/meta/classes/testimage.bbclass
>>> index b00d47441b2..4e9494296e1 100644
>>> --- a/meta/classes/testimage.bbclass
>>> +++ b/meta/classes/testimage.bbclass
>>> @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
>>> "${WORKDIR}/testimage/packages"
>>>  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
>>>  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
>>>  
>>> -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
>>> ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
>>> stap ldd xorg"
>>> -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
>>> buildgalculator "
>>> +BASICTESTSUITE = "\
>>> +    ping date df ssh scp python perl apt opkg gi ptest dnf rpm
>>> parselogs \
>>> +    logrotate connman systemd oe_syslog pam stap ldd xorg
>>> kernelmodule gcc \
>>> +    buildcpio buildlzip buildgalculator"
>> Again Nak. You are forcing folks to have x11, systemd etc in the
>> "Basic". 
> No, I'm not. If the image doesn't have x11 in it, the test will show
> "skipped". If the image doesn't have systemd in it, it will show those
> tests as "skipped". If you just want to test ptest, you can specify
> that, which is what you had to do before this change anyway *just* to
> run ptest.
>
> So I'm not really seeing the problem?

So for meta-security, since it needs python and perl, will it re-run
those tests every time I run the meta-security ptests?  if it does, one
may interpret those changes as a negative to the overall testing experience.

- armin

>
> The only risk with doing this is that a test may be skipped when it
> shouldn't be. The plan is to verify the right set of test results were
> obtained when the QA overall results from a release build are compared
> against a known list of tests which should have run.
>
> Cheers,
>
> Richard
>

Re: [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES

[richard.purdie]

On Sun, 2018-11-11 at 13:34 -0800, akuster808 wrote:
> On 11/11/18 1:18 PM, richard.purdie@linuxfoundation.org wrote:
> > On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
> > > On 11/9/18 6:29 AM, Richard Purdie wrote:
> > > > Now the test markup of the development tools tests is complete,
> > > > this
> > > > can be further tweaked to auto run the correct tests.
> > > > 
> > > > Signed-off-by: Richard Purdie <
> > > > richard.purdie@linuxfoundation.org>
> > > > ---
> > > >  meta/classes/testimage.bbclass | 8 ++++----
> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/testimage.bbclass
> > > > b/meta/classes/testimage.bbclass
> > > > index b00d47441b2..4e9494296e1 100644
> > > > --- a/meta/classes/testimage.bbclass
> > > > +++ b/meta/classes/testimage.bbclass
> > > > @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> > > > "${WORKDIR}/testimage/packages"
> > > >  TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> > > >  TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
> > > >  
> > > > -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
> > > > ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
> > > > stap ldd xorg"
> > > > -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
> > > > buildgalculator "
> > > > +BASICTESTSUITE = "\
> > > > +    ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> > > > parselogs \
> > > > +    logrotate connman systemd oe_syslog pam stap ldd xorg
> > > > kernelmodule gcc \
> > > > +    buildcpio buildlzip buildgalculator"
> > > 
> > > Again Nak. You are forcing folks to have x11, systemd etc in the
> > > "Basic". 
> > 
> > No, I'm not. If the image doesn't have x11 in it, the test will
> > show
> > "skipped". If the image doesn't have systemd in it, it will show
> > those
> > tests as "skipped". If you just want to test ptest, you can specify
> > that, which is what you had to do before this change anyway *just*
> > to
> > run ptest.
> > 
> > So I'm not really seeing the problem?
> 
> So for meta-security, since it needs python and perl, will it re-run
> those tests every time I run the meta-security ptests?  if it does,
> one may interpret those changes as a negative to the overall testing
> experience.

It depends. It will probably continue to do what it did before which is
to "auto" run tests, unless you're building one of the core-image-XXX
images which had specific tests defined.

If you want to just run the ptest tests, just set:

TEST_SUITES = "ptest"

There are likely some "testing experience" issues but I think these
changes are a key step in improving it, by making the tests much more
automatic without having to define specific test suites for specific
images.

You can still specify specific test subsets just as you've always been
able to.

I agree the output from testing is confusing and I have some ideas on
how to improve that too, but one step (patch) at a time.

Cheers,

Richard

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[richard.purdie]

On Sun, 2018-11-11 at 13:09 -0800, akuster808 wrote:
> On 11/9/18 6:29 AM, Richard Purdie wrote:
> > The nightly-build-appliance target already tests this particular
> > build.
> 
> If I am reading the AB code correctly, it is building for x86-64
> where
> as this is for x86.  A QA person would say they are not the same.
> 
> Are the resulting build results testing the same thing?

I have to admit I did not spot the x86 verses x86_64 difference. They
are different however I think a b-a image makes most sense as a 64 bit
one and it makes more sense to test that of the two. I'm not sure there
is much marginal utility in testing the other?

I can update the commit message to state its a conscious decision?

Cheers,

Richard

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[akuster808]

On 11/11/18 1:59 PM, richard.purdie@linuxfoundation.org wrote:
> On Sun, 2018-11-11 at 13:09 -0800, akuster808 wrote:
>> On 11/9/18 6:29 AM, Richard Purdie wrote:
>>> The nightly-build-appliance target already tests this particular
>>> build.
>> If I am reading the AB code correctly, it is building for x86-64
>> where
>> as this is for x86.  A QA person would say they are not the same.
>>
>> Are the resulting build results testing the same thing?
> I have to admit I did not spot the x86 verses x86_64 difference. They
> are different however I think a b-a image makes most sense as a 64 bit
> one and it makes more sense to test that of the two. I'm not sure there
> is much marginal utility in testing the other?
Nope. x86-64 is fine (until we have an non-x86-64 builders)
>
> I can update the commit message to state its a conscious decision?

I am fine as it is.

thanks,

>
> Cheers,

With every 'Cheers' I raise my glass.

- armin

>
> Richard
>

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[Yeoh, Ee Peng]

Hi Richard,

I realize the execution action inside "build-appliance.build-appliance.Bitbake_build-appliance-image" was not complete. Actually this particular test was testing that using build-appliance-image and vmware, one can bitbake build-applianc-image. This was different compare the automated tests inside AB where it test that bitbake can build-appliance-image. 

The correct steps for "build-appliance.build-appliance.Bitbake_build-appliance-image" shall be below. Allow us to submit a patch to add the missing detail for this test. 

Steps to Reproduce :
1. Get build appliance image from following link
https://autobuilder.yocto.io/pub/releases/yocto-2.6.rc1/build-appliance/Yocto_Build_Appliance.zip

2. boot the build appliance image in vmware.

3. Set MACHINE to qemux86 and add the following line to conf/local.conf :
SRCREV_pn-build-appliance-image = "$

{AUTOREV}
"
4. Run "bitbake build-appliance-image"

I am sorry for the confusion it created.

Best regards,
Yeoh Ee Peng 

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Richard Purdie
Sent: Friday, November 9, 2018 10:29 PM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

The nightly-build-appliance target already tests this particular build.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/manual/build-appliance.json | 26 -----------------------
 1 file changed, 26 deletions(-)

diff --git a/meta/lib/oeqa/manual/build-appliance.json b/meta/lib/oeqa/manual/build-appliance.json
index b8f89275d08..70f8c72c9bb 100644
--- a/meta/lib/oeqa/manual/build-appliance.json
+++ b/meta/lib/oeqa/manual/build-appliance.json
@@ -1,30 +1,4 @@
 [
-    {
-        "test": {
-            "@alias": "build-appliance.build-appliance.Bitbake_build-appliance-image",
-            "author": [
-                {
-                    "email": "alexandru.c.georgescu@intel.com",
-                    "name": "alexandru.c.georgescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "Get poky source code and prepare the build environment",
-                    "expected_results": "bitbake build-appliance-image is successful  "
-                },
-                "2": {
-                    "action": "Set MACHINE to qemux86 and add the following line to conf/local.conf :  SRCREV_pn-build-appliance-image = \"${AUTOREV}\"",
-                    "expected_results": ""
-                },
-                "3": {
-                    "action": "Run \"bitbake build-appliance-image\" \n \n",
-                    "expected_results": ""
-                }
-            },
-            "summary": "Bitbake_build-appliance-image"
-        }
-    },
     {
         "test": {
             "@alias": "build-appliance.build-appliance.Build_core-image-minimal_with_build-appliance-image",
-- 
2.17.1

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[richard.purdie]

On Mon, 2018-11-12 at 01:41 +0000, Yeoh, Ee Peng wrote:
> I realize the execution action inside "build-appliance.build-
> appliance.Bitbake_build-appliance-image" was not complete. Actually
> this particular test was testing that using build-appliance-image and
> vmware, one can bitbake build-applianc-image. This was different
> compare the automated tests inside AB where it test that bitbake can
> build-appliance-image. 
> 
> The correct steps for "build-appliance.build-appliance.Bitbake_build-
> appliance-image" shall be below. Allow us to submit a patch to add
> the missing detail for this test. 
> 
> Steps to Reproduce :
> 1. Get build appliance image from following link
> 
https://autobuilder.yocto.io/pub/releases/yocto-2.6.rc1/build-appliance/Yocto_Build_Appliance.zip
> 
> 2. boot the build appliance image in vmware.
> 
> 3. Set MACHINE to qemux86 and add the following line to
> conf/local.conf :
> SRCREV_pn-build-appliance-image = "$
> 
> {AUTOREV}
> "
> 4. Run "bitbake build-appliance-image"
> 
> I am sorry for the confusion it created.

In that case wouldn't it make more sense to combine these steps with
some of the other build appliance tests? We could add an extra step
where the tester builds build-appliance rather than having multiple
tests which start "launch build appliance"?

Cheers,

Richard

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[Yeoh, Ee Peng]

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Monday, November 12, 2018 7:44 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

On Mon, 2018-11-12 at 01:41 +0000, Yeoh, Ee Peng wrote:
> I realize the execution action inside "build-appliance.build- 
> appliance.Bitbake_build-appliance-image" was not complete. Actually 
> this particular test was testing that using build-appliance-image and 
> vmware, one can bitbake build-applianc-image. This was different 
> compare the automated tests inside AB where it test that bitbake can 
> build-appliance-image.
> 
> The correct steps for "build-appliance.build-appliance.Bitbake_build-
> appliance-image" shall be below. Allow us to submit a patch to add the 
> missing detail for this test.
> 
> Steps to Reproduce :
> 1. Get build appliance image from following link
> 
https://autobuilder.yocto.io/pub/releases/yocto-2.6.rc1/build-appliance/Yocto_Build_Appliance.zip
> 
> 2. boot the build appliance image in vmware.
> 
> 3. Set MACHINE to qemux86 and add the following line to 
> conf/local.conf :
> SRCREV_pn-build-appliance-image = "$
> 
> {AUTOREV}
> "
> 4. Run "bitbake build-appliance-image"
> 
> I am sorry for the confusion it created.

In that case wouldn't it make more sense to combine these steps with some of the other build appliance tests? We could add an extra step where the tester builds build-appliance rather than having multiple tests which start "launch build appliance"?

In general, Pros & Cons for merging test cases, Pros: reduced the number of test cases & making it simpler for reading, Cons: loss of lower level tracking on the test status. 
Example: currently we have multiple test cases for build-appliance where each test cases testing the ability to use build-appliance-image to bitbake different type of images (recently we detect that build-appliance-image was able to build core-image-minimal & core-image-sato-sdk but it failed to build the build-appliance-image), if we merge multiple test cases into general test_build_appliance_can_build_images, then we will loss the visibilities on which type of images failed when this test failed, unless we looked into the test log if test log were generated.  

Cheers,

Richard

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[richard.purdie]

On Wed, 2018-11-14 at 08:05 +0000, Yeoh, Ee Peng wrote:
> > In that case wouldn't it make more sense to combine these steps 
> > with  
> > some of the other build appliance tests? We could add an extra step
> > where the tester builds build-appliance rather than having multiple
> > tests which start "launch build appliance"?
> 
> In general, Pros & Cons for merging test cases, Pros: reduced the
> number of test cases & making it simpler for reading, Cons: loss of
> lower level tracking on the test status. 
> Example: currently we have multiple test cases for build-appliance
> where each test cases testing the ability to use build-appliance-
> image to bitbake different type of images (recently we detect that
> build-appliance-image was able to build core-image-minimal & core-
> image-sato-sdk but it failed to build the build-appliance-image), if
> we merge multiple test cases into general
> test_build_appliance_can_build_images, then we will loss the
> visibilities on which type of images failed when this test failed,
> unless we looked into the test log if test log were generated.  

If a step fails, QA will write a bug report for the failure. I'd have
thought it would be quite clear in this case what the failing step was?
QA would also always include a failing build log in such a case which
should also make the problem quite clear.

Even if the step was "bitbake core-image-minimal core-image-sato build-
appliance-image -k", the logs should still clearly show what it was
that failed?

Cheers,

Richard

Re: [PATCH 01/12] build-appliance: Drop test which is already automated

[Yeoh, Ee Peng]

Ok, noted, I agreed. Let me initialize the build-appliance testcases consolidation. Thank you for your inputs and feedbacks. 

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, November 14, 2018 5:32 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

On Wed, 2018-11-14 at 08:05 +0000, Yeoh, Ee Peng wrote:
> > In that case wouldn't it make more sense to combine these steps with 
> > some of the other build appliance tests? We could add an extra step 
> > where the tester builds build-appliance rather than having multiple 
> > tests which start "launch build appliance"?
> 
> In general, Pros & Cons for merging test cases, Pros: reduced the 
> number of test cases & making it simpler for reading, Cons: loss of 
> lower level tracking on the test status.
> Example: currently we have multiple test cases for build-appliance 
> where each test cases testing the ability to use build-appliance- 
> image to bitbake different type of images (recently we detect that 
> build-appliance-image was able to build core-image-minimal & core- 
> image-sato-sdk but it failed to build the build-appliance-image), if 
> we merge multiple test cases into general 
> test_build_appliance_can_build_images, then we will loss the 
> visibilities on which type of images failed when this test failed, 
> unless we looked into the test log if test log were generated.

If a step fails, QA will write a bug report for the failure. I'd have thought it would be quite clear in this case what the failing step was?
QA would also always include a failing build log in such a case which should also make the problem quite clear.

Even if the step was "bitbake core-image-minimal core-image-sato build- appliance-image -k", the logs should still clearly show what it was that failed?

Cheers,

Richard