All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] oeqa/runtime: Add OERequirePackage decorator
@ 2020-06-13 10:19 Konrad Weihmann
  2020-06-15 14:02 ` [OE-core] " Richard Purdie
  0 siblings, 1 reply; 7+ messages in thread
From: Konrad Weihmann @ 2020-06-13 10:19 UTC (permalink / raw)
  To: openembedded-core; +Cc: Konrad Weihmann

Add new decorator which behaves like OEHasPackage, but
fails the testcase if a dependency isn't met.

This helps to identify missing packages in the image
under test when using static test suite lists, otherwise
a missing package won't fail the overall test suite and
errors might slip through unnoticed

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
---
 meta/lib/oeqa/runtime/decorator/package.py | 50 ++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/meta/lib/oeqa/runtime/decorator/package.py b/meta/lib/oeqa/runtime/decorator/package.py
index 4c5ca198b0..b3d3fdbec2 100644
--- a/meta/lib/oeqa/runtime/decorator/package.py
+++ b/meta/lib/oeqa/runtime/decorator/package.py
@@ -54,3 +54,53 @@ class OEHasPackage(OETestDecorator):
             if self.case.tc.image_packages.isdisjoint(need_pkgs):
                 msg = "Test requires %s to be installed" % ', or'.join(need_pkgs)
                 self.case.skipTest(msg)
+
+@registerDecorator
+class OERequirePackage(OETestDecorator):
+    """
+        Checks if image has packages (un)installed.
+        It is almost the same as OEHasPackage, but if dependencies are missing
+        the test case fails.
+
+        The argument must be a string, set, or list of packages that must be
+        installed or not present in the image.
+
+        The way to tell a package must not be in an image is using an
+        exclamation point ('!') before the name of the package.
+
+        If test depends on pkg1 or pkg2 you need to use:
+        @OERequirePackage({'pkg1', 'pkg2'})
+
+        If test depends on pkg1 and pkg2 you need to use:
+        @OERequirePackage('pkg1')
+        @OERequirePackage('pkg2')
+
+        If test depends on pkg1 but pkg2 must not be present use:
+        @OERequirePackage({'pkg1', '!pkg2'})
+    """
+
+    attrs = ('need_pkgs',)
+
+    def setUpDecorator(self):
+        need_pkgs = set()
+        unneed_pkgs = set()
+        pkgs = strToSet(self.need_pkgs)
+        for pkg in pkgs:
+            if pkg.startswith('!'):
+                unneed_pkgs.add(pkg[1:])
+            else:
+                need_pkgs.add(pkg)
+
+        if unneed_pkgs:
+            msg = 'Checking if %s is not installed' % ', '.join(unneed_pkgs)
+            self.logger.debug(msg)
+            if not self.case.tc.image_packages.isdisjoint(unneed_pkgs):
+                msg = "Test can't run with %s installed" % ', or'.join(unneed_pkgs)
+                self.case.fail(msg)
+
+        if need_pkgs:
+            msg = 'Checking if at least one of %s is installed' % ', '.join(need_pkgs)
+            self.logger.debug(msg)
+            if self.case.tc.image_packages.isdisjoint(need_pkgs):
+                msg = "Test requires %s to be installed" % ', or'.join(need_pkgs)
+                self.case.fail(msg)
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
  2020-06-13 10:19 [PATCH] oeqa/runtime: Add OERequirePackage decorator Konrad Weihmann
@ 2020-06-15 14:02 ` Richard Purdie
  2020-06-15 16:41   ` Alexander Kanavin
       [not found]   ` <1618C532747F3046.29686@lists.openembedded.org>
  0 siblings, 2 replies; 7+ messages in thread
From: Richard Purdie @ 2020-06-15 14:02 UTC (permalink / raw)
  To: Konrad Weihmann, openembedded-core

On Sat, 2020-06-13 at 12:19 +0200, Konrad Weihmann wrote:
> Add new decorator which behaves like OEHasPackage, but
> fails the testcase if a dependency isn't met.
> 
> This helps to identify missing packages in the image
> under test when using static test suite lists, otherwise
> a missing package won't fail the overall test suite and
> errors might slip through unnoticed
> 
> Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
> ---
>  meta/lib/oeqa/runtime/decorator/package.py | 50 ++++++++++++++++++++++
>  1 file changed, 50 insertions(+)
> 
> diff --git a/meta/lib/oeqa/runtime/decorator/package.py b/meta/lib/oeqa/runtime/decorator/package.py
> index 4c5ca198b0..b3d3fdbec2 100644
> --- a/meta/lib/oeqa/runtime/decorator/package.py
> +++ b/meta/lib/oeqa/runtime/decorator/package.py
> @@ -54,3 +54,53 @@ class OEHasPackage(OETestDecorator):
>              if self.case.tc.image_packages.isdisjoint(need_pkgs):
>                  msg = "Test requires %s to be installed" % ', or'.join(need_pkgs)
>                  self.case.skipTest(msg)
> +
> +@registerDecorator
> +class OERequirePackage(OETestDecorator):
> +    """
> +        Checks if image has packages (un)installed.
> +        It is almost the same as OEHasPackage, but if dependencies are missing
> +        the test case fails.
> +
> +        The argument must be a string, set, or list of packages that must be
> +        installed or not present in the image.
> +
> +        The way to tell a package must not be in an image is using an
> +        exclamation point ('!') before the name of the package.
> +
> +        If test depends on pkg1 or pkg2 you need to use:
> +        @OERequirePackage({'pkg1', 'pkg2'})
> +
> +        If test depends on pkg1 and pkg2 you need to use:
> +        @OERequirePackage('pkg1')
> +        @OERequirePackage('pkg2')
> +
> +        If test depends on pkg1 but pkg2 must not be present use:
> +        @OERequirePackage({'pkg1', '!pkg2'})
> +    """
> +
> +    attrs = ('need_pkgs',)
> +
> +    def setUpDecorator(self):
> +        need_pkgs = set()
> +        unneed_pkgs = set()
> +        pkgs = strToSet(self.need_pkgs)
> +        for pkg in pkgs:
> +            if pkg.startswith('!'):
> +                unneed_pkgs.add(pkg[1:])
> +            else:
> +                need_pkgs.add(pkg)
> +
> +        if unneed_pkgs:
> +            msg = 'Checking if %s is not installed' % ', '.join(unneed_pkgs)
> +            self.logger.debug(msg)
> +            if not self.case.tc.image_packages.isdisjoint(unneed_pkgs):
> +                msg = "Test can't run with %s installed" % ', or'.join(unneed_pkgs)
> +                self.case.fail(msg)
> +
> +        if need_pkgs:
> +            msg = 'Checking if at least one of %s is installed' % ', '.join(need_pkgs)
> +            self.logger.debug(msg)
> +            if self.case.tc.image_packages.isdisjoint(need_pkgs):
> +                msg = "Test requires %s to be installed" % ', or'.join(need_pkgs)
> +                self.case.fail(msg)

I can see the use case, I'm a bit torn on whether we should fail in
these cases, or whether we should enourage people to check the tests
they expected to run really did.

With the complexity on the autobuilder we've had to rely on the latter,
comparing that all tests that ran previously, still run.

With regard to the patch itself, I'm worried about the code duplication
with the other skip decorator, is there a way to reduce that and make
things a bit neater?

Cheers,

Richard






^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
  2020-06-15 14:02 ` [OE-core] " Richard Purdie
@ 2020-06-15 16:41   ` Alexander Kanavin
  2020-06-15 16:57     ` Richard Purdie
       [not found]   ` <1618C532747F3046.29686@lists.openembedded.org>
  1 sibling, 1 reply; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-15 16:41 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Konrad Weihmann, OE-core

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

On Mon, 15 Jun 2020 at 16:02, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> I can see the use case, I'm a bit torn on whether we should fail in
> these cases, or whether we should enourage people to check the tests
> they expected to run really did.
>
> With the complexity on the autobuilder we've had to rely on the latter,
> comparing that all tests that ran previously, still run.
>

Do we have some kind of tooling to check that tests that are expected to
run, did run, and were not skipped?
This patch came from our internal situation where due to debian renaming,
@OEHasPackage started skipping tests
that it should not have. It wasn't immediately noticed - test logs are
hidden inside the build logs, and the build logs are not
usually looked at if the overall build does not fail.

People were very baffled by that, and it took 'yocto experts' (Konrad :) to
sort the issue.

Alex

[-- Attachment #2: Type: text/html, Size: 1328 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
       [not found]   ` <1618C532747F3046.29686@lists.openembedded.org>
@ 2020-06-15 16:54     ` Alexander Kanavin
  2020-06-15 16:56       ` Richard Purdie
  0 siblings, 1 reply; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-15 16:54 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Richard Purdie, Konrad Weihmann, OE-core

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

Come to think of it, maybe it's okay in this situation to actually remove
OEHasPackage - the test will simply always run, and will still fail if the
package is absent, but the failure would be less specific than a missing
package.

Alex

On Mon, 15 Jun 2020 at 18:41, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> On Mon, 15 Jun 2020 at 16:02, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> I can see the use case, I'm a bit torn on whether we should fail in
>> these cases, or whether we should enourage people to check the tests
>> they expected to run really did.
>>
>> With the complexity on the autobuilder we've had to rely on the latter,
>> comparing that all tests that ran previously, still run.
>>
>
> Do we have some kind of tooling to check that tests that are expected to
> run, did run, and were not skipped?
> This patch came from our internal situation where due to debian renaming,
> @OEHasPackage started skipping tests
> that it should not have. It wasn't immediately noticed - test logs are
> hidden inside the build logs, and the build logs are not
> usually looked at if the overall build does not fail.
>
> People were very baffled by that, and it took 'yocto experts' (Konrad :)
> to sort the issue.
>
> Alex
> 
>

[-- Attachment #2: Type: text/html, Size: 2102 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
  2020-06-15 16:54     ` Alexander Kanavin
@ 2020-06-15 16:56       ` Richard Purdie
  2020-06-15 18:04         ` Alexander Kanavin
  0 siblings, 1 reply; 7+ messages in thread
From: Richard Purdie @ 2020-06-15 16:56 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Konrad Weihmann, OE-core

On Mon, 2020-06-15 at 18:54 +0200, Alexander Kanavin wrote:
> Come to think of it, maybe it's okay in this situation to actually
> remove OEHasPackage - the test will simply always run, and will still
> fail if the package is absent, but the failure would be less specific
> than a missing package.

Lighting up the autobuilder with failures will just mean people no
longer know which failures matter, so no, this isn't an option.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
  2020-06-15 16:41   ` Alexander Kanavin
@ 2020-06-15 16:57     ` Richard Purdie
  0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2020-06-15 16:57 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: Konrad Weihmann, OE-core

On Mon, 2020-06-15 at 18:41 +0200, Alexander Kanavin wrote:
> On Mon, 15 Jun 2020 at 16:02, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > I can see the use case, I'm a bit torn on whether we should fail in
> > these cases, or whether we should enourage people to check the
> > tests
> > they expected to run really did.
> > 
> > With the complexity on the autobuilder we've had to rely on the
> > latter,
> > comparing that all tests that ran previously, still run.
> 
> Do we have some kind of tooling to check that tests that are expected
> to run, did run, and were not skipped? 
> This patch came from our internal situation where due to debian
> renaming, @OEHasPackage started skipping tests
> that it should not have. It wasn't immediately noticed - test logs
> are hidden inside the build logs, and the build logs are not
> usually looked at if the overall build does not fail.
> 
> People were very baffled by that, and it took 'yocto experts' (Konrad
> :) to sort the issue.

We have "resulttool regression" which is meant to cover this. We're
struggling a little to get the automated comparisons right (its hard to
know what to compare against exactly) but if you have a known correct
set of tests it should work.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH] oeqa/runtime: Add OERequirePackage decorator
  2020-06-15 16:56       ` Richard Purdie
@ 2020-06-15 18:04         ` Alexander Kanavin
  0 siblings, 0 replies; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-15 18:04 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Konrad Weihmann, OE-core

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

On Mon, 15 Jun 2020 at 18:56, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2020-06-15 at 18:54 +0200, Alexander Kanavin wrote:
> > Come to think of it, maybe it's okay in this situation to actually
> > remove OEHasPackage - the test will simply always run, and will still
> > fail if the package is absent, but the failure would be less specific
> > than a missing package.
>
> Lighting up the autobuilder with failures will just mean people no
> longer know which failures matter, so no, this isn't an option.
>

I didn't mean the AB (which has a universal list of test suites for all
images, which means the tests have to adapt via OeHasPackage), but rather
what we have here: several lists of tests that are tightly coupled to their
respective test images. In this situation OEHasPackage is
counter-productive.

Alex

[-- Attachment #2: Type: text/html, Size: 1215 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2020-06-15 18:04 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-13 10:19 [PATCH] oeqa/runtime: Add OERequirePackage decorator Konrad Weihmann
2020-06-15 14:02 ` [OE-core] " Richard Purdie
2020-06-15 16:41   ` Alexander Kanavin
2020-06-15 16:57     ` Richard Purdie
     [not found]   ` <1618C532747F3046.29686@lists.openembedded.org>
2020-06-15 16:54     ` Alexander Kanavin
2020-06-15 16:56       ` Richard Purdie
2020-06-15 18:04         ` Alexander Kanavin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.