[PATCH V3 0/4] Skip selftests depending on distro and its features

[PATCH V3 0/4] Skip selftests depending on distro and its features

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Originally, the selftests were defined for the 'poky' distro. However, there
are more supported distros, including nodistro, poky-lsb and poky-tiny.
These series takes into consideration the distro name and its features
to skip certain tests.

[YOCTO #8525]

The following changes since commit e016eb10b075e280b4e78a04e47b59a173386421:

  bitbake: bb/cooker: BBCooker stops notifier at shutdown (2017-01-06 00:01:07 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lsandov1/oe-selftest-distro-agnostic
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/oe-selftest-distro-agnostic

Leonardo Sandoval (4):
  selftest: devtool: use distro agnostic recipes for devtool checks
  selftest: imagefeatures: skip tests based on distro features and
    supported fstypes
  selftest: runtime-test: skip image-install test for poky-tiny
  selftest: sstatetests: skip glibc-initial tests and use a more generic
    image

 meta/lib/oeqa/selftest/devtool.py       | 19 +++++++++--------
 meta/lib/oeqa/selftest/imagefeatures.py | 11 ++++++++++
 meta/lib/oeqa/selftest/runtime-test.py  |  2 ++
 meta/lib/oeqa/selftest/sstate.py        |  1 +
 meta/lib/oeqa/selftest/sstatetests.py   | 36 +++++++++++++++++++++++----------
 5 files changed, 48 insertions(+), 21 deletions(-)

-- 
2.1.4

[PATCH V3 1/4] selftest: devtool: use distro agnostic recipes for devtool checks

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

The recipes being replaced are not compatible with all distros, so
use mraa and virtual/make for some checks.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 19c5ccf..bbdd8c6 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -340,12 +340,11 @@ class DevtoolTests(DevtoolBase):
 
     @testcase(1161)
     def test_devtool_add_fetch_git(self):
-        # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
-        url = 'git://git.yoctoproject.org/libmatchbox'
-        checkrev = '462f0652055d89c648ddd54fd7b03f175c2c6973'
-        testrecipe = 'libmatchbox2'
+        url = 'gitsm://git.yoctoproject.org/mraa'
+        checkrev = 'ae127b19a50aa54255e4330ccfdd9a5d058e581d'
+        testrecipe = 'mraa'
         srcdir = os.path.join(tempdir, testrecipe)
         # Test devtool add
         self.track_for_cleanup(self.workspacedir)
@@ -353,7 +352,7 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         result = runCmd('devtool add %s %s -a -f %s' % (testrecipe, srcdir, url))
         self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created: %s' % result.output)
-        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure.ac')), 'Unable to find configure.ac in source directory')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'imraa', 'imraa.c')), 'Unable to find configure.ac in source directory')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)
@@ -363,7 +362,7 @@ class DevtoolTests(DevtoolBase):
         self.assertIn('_git.bb', recipefile, 'Recipe file incorrectly named')
         checkvars = {}
         checkvars['S'] = '${WORKDIR}/git'
-        checkvars['PV'] = '1.12+git${SRCPV}'
+        checkvars['PV'] = '1.0+git${SRCPV}'
         checkvars['SRC_URI'] = url
         checkvars['SRCREV'] = '${AUTOREV}'
         self._test_recipe_contents(recipefile, checkvars, [])
@@ -372,7 +371,7 @@ class DevtoolTests(DevtoolBase):
         shutil.rmtree(srcdir)
         url_rev = '%s;rev=%s' % (url, checkrev)
         result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev))
-        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure.ac')), 'Unable to find configure.ac in source directory')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'imraa', 'imraa.c')), 'Unable to find imraa/imraa.c in source directory')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)
@@ -590,8 +589,8 @@ class DevtoolTests(DevtoolBase):
     @testcase(1378)
     def test_devtool_modify_virtual(self):
         # Try modifying a virtual recipe
-        virtrecipe = 'virtual/libx11'
-        realrecipe = 'libx11'
+        virtrecipe = 'virtual/make'
+        realrecipe = 'make'
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
         self.track_for_cleanup(self.workspacedir)
@@ -1025,7 +1024,7 @@ class DevtoolTests(DevtoolBase):
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         # Try devtool extract
         self.track_for_cleanup(tempdir)
-        result = runCmd('devtool extract virtual/libx11 %s' % tempdir)
+        result = runCmd('devtool extract virtual/make %s' % tempdir)
         self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile.am')), 'Extracted source could not be found')
         # devtool extract shouldn't create the workspace
         self.assertFalse(os.path.exists(self.workspacedir))
-- 
2.1.4

[PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Some images (clutter and weston) requires certain distros features to be present,
so check presence before proceeding. Some distros does not support the ext4* fstypes
(poky-tiny only supports cpio.gz) so skip the test if this is the case.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index d015c49..5d50c69 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
 
+        if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
+            self.skipTest('opengl not present on DISTRO_FEATURES so core-image-clutter cannot be built')
+
         # Build a core-image-clutter
         bitbake('core-image-clutter')
 
@@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
         """
 
+        distro_features = get_bb_var('DISTRO_FEATURES')
+        if not ('opengl' in distro_features and 'wayland' in distro_features):
+            self.skipTest('neither opengl nor wayland present on DISTRO_FEATURES so core-image-weston cannot be built')
+
         features = 'DISTRO_FEATURES_append = " wayland"\n'
         features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"'
         self.write_config(features)
@@ -110,6 +117,10 @@ class ImageFeatures(oeSelfTest):
         features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"'
         self.write_config(features)
 
+        image_fstypes = get_bb_var('IMAGE_FSTYPES')
+        if 'ext4 ext4.bmap' not in image_fstypes:
+            self.skipTest('Current distro does not allow "ext4 ext4.bmap" filesystem type. Supported: %s' % image_fstypes)
+
         image_name = 'core-image-minimal'
         bitbake(image_name)
 
-- 
2.1.4

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

On 9 January 2017 at 17:45, <leonardo.sandoval.gonzalez@linux.intel.com>
wrote:

> +++ b/meta/lib/oeqa/selftest/imagefeatures.py
> @@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
>          AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
>          """
>
> +        if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
> +            self.skipTest('opengl not present on DISTRO_FEATURES so
> core-image-clutter cannot be built')
> +
>          # Build a core-image-clutter
>          bitbake('core-image-clutter')
>

This test is pretty pointless, lets just delete it.


> @@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
>          AutomatedBy: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
>          """
>
> +        distro_features = get_bb_var('DISTRO_FEATURES')
> +        if not ('opengl' in distro_features and 'wayland' in
> distro_features):
> +            self.skipTest('neither opengl nor wayland present on
> DISTRO_FEATURES so core-image-weston cannot be built')
> +
>          features = 'DISTRO_FEATURES_append = " wayland"\n'
>          features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"'
>          self.write_config(features)
>

This test is 90% nonsense, let's fix it whilst we're here.  After checking
that opengl and wayland are in DISTRO_FEATURES it can just build
core-image-weston: DISTRO_FEATURES doesn't need appending as it already has
wayland in, and CORE_IMAGE_EXTRA_INSTALL doesn't need extending as
core-image-weston obviously already contains weston.

Ross

[-- Attachment #2: Type: text/html, Size: 2608 bytes --]

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

[Leonardo Sandoval]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]



On 01/11/2017 11:27 AM, Burton, Ross wrote:
>
> On 9 January 2017 at 17:45, 
> <leonardo.sandoval.gonzalez@linux.intel.com 
> <mailto:leonardo.sandoval.gonzalez@linux.intel.com>> wrote:
>
>     +++ b/meta/lib/oeqa/selftest/imagefeatures.py
>     @@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
>              AutomatedBy: Daniel Istrate
>     <daniel.alexandrux.istrate@intel.com
>     <mailto:daniel.alexandrux.istrate@intel.com>>
>              """
>
>     +        if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
>     +            self.skipTest('opengl not present on DISTRO_FEATURES
>     so core-image-clutter cannot be built')
>     +
>              # Build a core-image-clutter
>              bitbake('core-image-clutter')
>
>
> This test is pretty pointless, lets just delete it.

why is this pointless? if this is the case, we should then remove the 
core-image-clutter.bb target.


>     @@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
>              AutomatedBy: Daniel Istrate
>     <daniel.alexandrux.istrate@intel.com
>     <mailto:daniel.alexandrux.istrate@intel.com>>
>              """
>
>     +        distro_features = get_bb_var('DISTRO_FEATURES')
>     +        if not ('opengl' in distro_features and 'wayland' in
>     distro_features):
>     +            self.skipTest('neither opengl nor wayland present on
>     DISTRO_FEATURES so core-image-weston cannot be built')
>     +
>              features = 'DISTRO_FEATURES_append = " wayland"\n'
>              features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"'
>              self.write_config(features)
>
>
> This test is 90% nonsense, let's fix it whilst we're here.  After 
> checking that opengl and wayland are in DISTRO_FEATURES it can just 
> build core-image-weston: DISTRO_FEATURES doesn't need appending as it 
> already has wayland in, and CORE_IMAGE_EXTRA_INSTALL doesn't need 
> extending as core-image-weston obviously already contains weston.
>
> Ross


[-- Attachment #2: Type: text/html, Size: 4356 bytes --]

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

On 11 January 2017 at 20:06, Leonardo Sandoval <
leonardo.sandoval.gonzalez@linux.intel.com> wrote:

> This test is pretty pointless, lets just delete it.
>
>
> why is this pointless? if this is the case, we should then remove the
> core-image-clutter.bb target
>

The test just builds an image that contains clutter, but core-image-weston
already does that and we dont really need to treat clutter any differently
to any other library.

Yes, removing core-image-clutter would make sense too.

Ross

[-- Attachment #2: Type: text/html, Size: 1150 bytes --]

[PATCH V3 3/4] selftest: runtime-test: skip image-install test for poky-tiny

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

poky-tiny cannot build full-cmdline image, so skip this test in this case.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/lib/oeqa/selftest/runtime-test.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/lib/oeqa/selftest/runtime-test.py b/meta/lib/oeqa/selftest/runtime-test.py
index 20caa97..c0e283f 100644
--- a/meta/lib/oeqa/selftest/runtime-test.py
+++ b/meta/lib/oeqa/selftest/runtime-test.py
@@ -95,6 +95,8 @@ class TestImage(oeSelfTest):
         Product: oe-core
         Author: Mariano Lopez <mariano.lopez@intel.com>
         """
+        if self.distro == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
         features += 'TEST_SUITES = "ping ssh selftest"\n'
-- 
2.1.4

[PATCH V3 4/4] selftest: sstatetests: skip glibc-initial tests and use a more generic image

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

There are mainly two changes: 1) Some distros uses non-glibc so skip
those tests requiring glibc-initial as target 2) instead of using core-image-sato,
use a more generic image (core-image-minimal) for hash tests.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/lib/oeqa/selftest/sstate.py      |  1 +
 meta/lib/oeqa/selftest/sstatetests.py | 36 ++++++++++++++++++++++++-----------
 2 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/meta/lib/oeqa/selftest/sstate.py b/meta/lib/oeqa/selftest/sstate.py
index d27a45c..742f4d0 100644
--- a/meta/lib/oeqa/selftest/sstate.py
+++ b/meta/lib/oeqa/selftest/sstate.py
@@ -16,6 +16,7 @@ class SStateBase(oeSelfTest):
         self.sstate_path = get_bb_var('SSTATE_DIR')
         self.hostdistro = get_bb_var('NATIVELSBSTRING')
         self.distro_specific_sstate = os.path.join(self.sstate_path, self.hostdistro)
+        self.tclibc = get_bb_var('TCLIBC')
 
     # Creates a special sstate configuration with the option to add sstate mirrors
     def config_sstate(self, temp_sstate_location=False, add_local_mirrors=[]):
diff --git a/meta/lib/oeqa/selftest/sstatetests.py b/meta/lib/oeqa/selftest/sstatetests.py
index 6642539..2f2b942 100644
--- a/meta/lib/oeqa/selftest/sstatetests.py
+++ b/meta/lib/oeqa/selftest/sstatetests.py
@@ -51,12 +51,15 @@ class SStateTests(SStateBase):
 
     @testcase(976)
     def test_sstate_creation_distro_nonspecific_pass(self):
-        self.run_test_sstate_creation(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_sstate_creation(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1375)
     def test_sstate_creation_distro_nonspecific_fail(self):
-        self.run_test_sstate_creation(['glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True, should_pass=False)
-
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_sstate_creation(['glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True, should_pass=False)
 
     # Test the sstate files deletion part of the do_cleansstate task
     def run_test_cleansstate_task(self, targets, distro_specific=True, distro_nonspecific=True, temp_sstate_location=True):
@@ -78,16 +81,26 @@ class SStateTests(SStateBase):
     @testcase(977)
     def test_cleansstate_task_distro_specific_nonspecific(self):
         targetarch = get_bb_var('TUNE_ARCH')
-        self.run_test_cleansstate_task(['binutils-cross-' + targetarch, 'binutils-native', 'glibc-initial'], distro_specific=True, distro_nonspecific=True, temp_sstate_location=True)
+        targets = ['binutils-cross-'+ targetarch, 'binutils-native']
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            targets.append('glibc-initial')
+        self.run_test_cleansstate_task(targets, distro_specific=True, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1376)
     def test_cleansstate_task_distro_nonspecific(self):
-        self.run_test_cleansstate_task(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_cleansstate_task(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1377)
     def test_cleansstate_task_distro_specific(self):
         targetarch = get_bb_var('TUNE_ARCH')
-        self.run_test_cleansstate_task(['binutils-cross-'+ targetarch, 'binutils-native', 'glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True)
+        targets = ['binutils-cross-'+ targetarch, 'binutils-native']
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            targets.append('glibc-initial')
+        self.run_test_cleansstate_task(targets, distro_specific=True, distro_nonspecific=False, temp_sstate_location=True)
 
 
     # Test rebuilding of distro-specific sstate files
@@ -240,7 +253,7 @@ SDKMACHINE = "x86_64"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
         self.write_config("""
 MACHINE = "qemux86"
 TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"
@@ -250,16 +263,17 @@ SDKMACHINE = "i686"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash2")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
 
         def get_files(d):
             f = []
             for root, dirs, files in os.walk(d):
-                if "core-image-sato" in root:
+                if "core-image-minimal" in root:
                     # SDKMACHINE changing will change
                     # do_rootfs/do_testimage/do_build stamps of images which
                     # is safe to ignore.
                     continue
+
                 f.extend(os.path.join(root, name) for name in files)
             return f
         files1 = get_files(topdir + "/tmp-sstatesamehash/stamps/")
@@ -283,13 +297,13 @@ TMPDIR = \"${TOPDIR}/tmp-sstatesamehash\"
 NATIVELSBSTRING = \"DistroA\"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
         self.write_config("""
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash2\"
 NATIVELSBSTRING = \"DistroB\"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash2")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
 
         def get_files(d):
             f = []
-- 
2.1.4

Re: [PATCH V3 4/4] selftest: sstatetests: skip glibc-initial tests and use a more generic image

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

On 9 January 2017 at 17:45, <leonardo.sandoval.gonzalez@linux.intel.com>
wrote:

> -        bitbake("core-image-sato -S none")
> +        bitbake("core-image-minimal -S none")
>

As this is just stamp calculation, could we just do "world" instead...

Ross

[-- Attachment #2: Type: text/html, Size: 766 bytes --]

Re: [PATCH V3 4/4] selftest: sstatetests: skip glibc-initial tests and use a more generic image

[Leonardo Sandoval]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]



On 01/11/2017 11:29 AM, Burton, Ross wrote:
>
> On 9 January 2017 at 17:45, 
> <leonardo.sandoval.gonzalez@linux.intel.com 
> <mailto:leonardo.sandoval.gonzalez@linux.intel.com>> wrote:
>
>     -   bitbake("core-image-sato -S none")
>     +        bitbake("core-image-minimal -S none")
>
>
> As this is just stamp calculation, could we just do "world" instead...
>
I tried s/core-image-sato/world/ but asserts are not prepared for the 
latter. Let me double check and provide log.

> Ross


[-- Attachment #2: Type: text/html, Size: 1804 bytes --]