[RFC][PATCH 2/2] OETestContext: order test cases by prio & depends

[RFC][PATCH 2/2] OETestContext: order test cases by prio & depends

[Konrad Weihmann]

Order the overall test case list in the resulting test suite
by priority (by default all test cases do have a priority of 50).
In a second round move test cases with test dependencies behind
the maximum priority of priorities.
New method 'orderByPriority' can be explictly overridden to disable
usage of the new 'priority' decorator.

Resulting test cases list will be passed back to the calling
test runner instance

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
---
 meta/lib/oeqa/core/context.py | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 2abe353d27..8c5589b512 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -66,6 +66,34 @@ class OETestContext(object):
                 if (cid + '.').startswith(skip + '.'):
                     setattr(tclass, 'setUpHooker', skipfuncgen('Skip by the command line argument "%s"' % skip))
 
+    def orderByPriority(self, suites):
+        _ordered = {}
+        for _test in suites._tests:
+            _prio = 50
+            _depends = []
+            for x in _test.decorators:
+                if hasattr(x, "depends"):
+                    _depends += getattr(x, "depends")
+                if hasattr(x, "priority"):
+                    _prio = getattr(x, "priority")
+            _ordered[_test.id()] = {"prio": _prio, "deps": _depends, "ref": _test}
+
+        def set_suite_prio(suites, suite):
+            for dep in suite["deps"]:
+                if dep not in suites:
+                    try:
+                        dep = [x for x in suites.keys() if x.endswith(dep)][0]
+                    except:
+                        continue
+                suite["prio"] = max(suite["prio"], set_suite_prio(suites, suites[dep]) + 1)
+            return suite["prio"]
+
+        for k, v in _ordered.items():
+            _ordered[k]["prio"] = set_suite_prio(_ordered, v)
+
+        suites._tests = [x[1]["ref"] for x in sorted(_ordered.items(), key=lambda tup: tup[1]["prio"])]
+        return suites
+
     def loadTests(self, module_paths, modules=[], tests=[],
             modules_manifest="", modules_required=[], **kwargs):
         if modules_manifest:
@@ -73,7 +101,7 @@ class OETestContext(object):
 
         self.loader = self.loaderClass(self, module_paths, modules, tests,
                 modules_required, **kwargs)
-        self.suites = self.loader.discover()
+        self.suites = self.orderByPriority(self.loader.discover())
 
     def prepareSuite(self, suites, processes):
         return suites
-- 
2.25.1

Re: [OE-core] [RFC][PATCH 2/2] OETestContext: order test cases by prio & depends

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3488 bytes --]

I wonder if some test cases can be written to check the ordering outcome
without actually executing the tests?

Particularly, I wonder about interaction between priority and dependency,
when one contradicts the other.

Otherwise, I do support this: we have test cases that for example check log
sanity and absence of coredumps which
are beneficial to run at the end, and test cases that check that services
started successfully and overall boot looks sane
(no sudden increase in boot time) which are best executed up front.

Alex

On Tue, 5 Jan 2021 at 20:17, Konrad Weihmann <kweihmann@outlook.com> wrote:

> Order the overall test case list in the resulting test suite
> by priority (by default all test cases do have a priority of 50).
> In a second round move test cases with test dependencies behind
> the maximum priority of priorities.
> New method 'orderByPriority' can be explictly overridden to disable
> usage of the new 'priority' decorator.
>
> Resulting test cases list will be passed back to the calling
> test runner instance
>
> Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
> ---
>  meta/lib/oeqa/core/context.py | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
> index 2abe353d27..8c5589b512 100644
> --- a/meta/lib/oeqa/core/context.py
> +++ b/meta/lib/oeqa/core/context.py
> @@ -66,6 +66,34 @@ class OETestContext(object):
>                  if (cid + '.').startswith(skip + '.'):
>                      setattr(tclass, 'setUpHooker', skipfuncgen('Skip by
> the command line argument "%s"' % skip))
>
> +    def orderByPriority(self, suites):
> +        _ordered = {}
> +        for _test in suites._tests:
> +            _prio = 50
> +            _depends = []
> +            for x in _test.decorators:
> +                if hasattr(x, "depends"):
> +                    _depends += getattr(x, "depends")
> +                if hasattr(x, "priority"):
> +                    _prio = getattr(x, "priority")
> +            _ordered[_test.id()] = {"prio": _prio, "deps": _depends,
> "ref": _test}
> +
> +        def set_suite_prio(suites, suite):
> +            for dep in suite["deps"]:
> +                if dep not in suites:
> +                    try:
> +                        dep = [x for x in suites.keys() if
> x.endswith(dep)][0]
> +                    except:
> +                        continue
> +                suite["prio"] = max(suite["prio"], set_suite_prio(suites,
> suites[dep]) + 1)
> +            return suite["prio"]
> +
> +        for k, v in _ordered.items():
> +            _ordered[k]["prio"] = set_suite_prio(_ordered, v)
> +
> +        suites._tests = [x[1]["ref"] for x in sorted(_ordered.items(),
> key=lambda tup: tup[1]["prio"])]
> +        return suites
> +
>      def loadTests(self, module_paths, modules=[], tests=[],
>              modules_manifest="", modules_required=[], **kwargs):
>          if modules_manifest:
> @@ -73,7 +101,7 @@ class OETestContext(object):
>
>          self.loader = self.loaderClass(self, module_paths, modules, tests,
>                  modules_required, **kwargs)
> -        self.suites = self.loader.discover()
> +        self.suites = self.orderByPriority(self.loader.discover())
>
>      def prepareSuite(self, suites, processes):
>          return suites
> --
> 2.25.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 4581 bytes --]