[PATCH v2 1/2] oeqa: add utils/data.py with get_data() function

[PATCH v2 1/2] oeqa: add utils/data.py with get_data() function

[Mikko Rapeli]

get_data() uses oeqa test method name and an optional
key to get data from image specific "testimage_data.json"
file located in image deploy directory. Image recipes can
provide custom versions of this file which configures
generic tests for a specific image when testing with
testimage.bbclass

For example, the parselogs.py runtime test needs image
specific configuration when the image has new errors from
the kernel which acceptable and can be ignored.

Same machine can be used to generate multiple images with different
runtime behavior so using image as the key and not machine.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/utils/data.py | 41 +++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/data.py

v2: no changes

diff --git a/meta/lib/oeqa/utils/data.py b/meta/lib/oeqa/utils/data.py
new file mode 100644
index 0000000000..539ca4a026
--- /dev/null
+++ b/meta/lib/oeqa/utils/data.py
@@ -0,0 +1,41 @@
+# Copyright (C) 2022 Linaro Limited
+#
+# SPDX-License-Identifier: MIT
+
+import os
+import json
+
+from oeqa.core.utils.test import getCaseID, getCaseFile, getCaseMethod
+
+
+def get_data(self, key = None):
+    """get_data() returns test case specific data to the test case implementation.
+    data is stored in image specific json file called "testimage_data.json" in
+    image deploy directory. Data matching test method name and an optional key
+    is returned to the test case. This data can then be used by generic test
+    cases to match image specific functionality and expected behavior. For example
+    list of expected kernel error strings, list of active systemd services etc.
+    can be image specific while the test case implementation to check them is
+    generic. Example json file for runtime test parselogs.py to ignore image
+    specific kernel error strings in dmesg:
+
+    {"test_parselogs":{"ignore_errors":[
+        "Error to be ignored in dmesg"
+    ]}}
+    """
+    test_method = getCaseMethod(self)
+    self.logger.info("%s: get_data() called by test_method =  %s, key = %s" % (__file__, test_method, key))
+
+    json_file_name = os.path.join(self.td['DEPLOY_DIR_IMAGE'], "testimage_data.json")
+    self.logger.debug("%s: json_file_name = %s" % (__file__, json_file_name))
+
+    with open(json_file_name) as json_file:
+        self.logger.debug("%s: json_file = %s" % (__file__, json_file))
+        json_data = json.load(json_file)
+        self.logger.debug("%s: json_data = %s" % (__file__, json_data))
+        if key:
+            data = json_data[test_method][key]
+        else:
+            data = json_data[test_method]
+        self.logger.debug("%s: data = %s" % (__file__, data))
+        return data
-- 
2.34.1

[PATCH v2 2/2] oeqa parselogs.py: use get_data() to fetch image specific error list

[Mikko Rapeli]

Runtime oeqa test parselogs.py checks dmesg output for errors. It has
hard coded machine specific exceptions for errors which can be ignored.
To re-use of this test on other machine targets and images, use
get_data() function to get the list of error strings to ignore
"ignore_errors" from image specific "testimage_data.json" file.
The json file stores this data as list under test method name and key
"ignore_errors. For example:

{"test_parselogs":{"ignore_errors":[
    "error strings which will be ignored",
    "another error strings which will be ignored"
]}}

If the json file does not exist, parselogs.py still falls back to using
the hardcoded defaults.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/runtime/cases/parselogs.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

v2: switch from logger.warn() to logger.info() to avoid extra warnings
in logs

v1: https://lists.openembedded.org/g/openembedded-core/message/173406

diff --git a/meta/lib/oeqa/runtime/cases/parselogs.py b/meta/lib/oeqa/runtime/cases/parselogs.py
index e67d3750da..2b89909cd7 100644
--- a/meta/lib/oeqa/runtime/cases/parselogs.py
+++ b/meta/lib/oeqa/runtime/cases/parselogs.py
@@ -12,6 +12,7 @@ from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.data import skipIfDataVar
 from oeqa.runtime.decorator.package import OEHasPackage
+from oeqa.utils.data import get_data
 
 #in the future these lists could be moved outside of module
 errors = ["error", "cannot", "can\'t", "failed"]
@@ -316,10 +317,18 @@ class ParseLogsTest(OERuntimeTestCase):
         grepcmd += '" ' + str(log) + " | grep -Eiv \'"
 
         try:
-            errorlist = ignore_errors[self.getMachine()]
-        except KeyError:
-            self.msg += 'No ignore list found for this machine, using default\n'
-            errorlist = ignore_errors['default']
+            # get list of strings to ignore from image specific testimage_data.json with format:
+            # {"test_parselogs": {"ignore_errors":["string to ignore", "second string to ignore"]}}
+            errorlist = get_data(self, key = "ignore_errors")
+        except Exception as e:
+            self.logger.debug("%s: Exception e = %s" % (__file__, e))
+            try:
+                errorlist = ignore_errors[self.getMachine()]
+            except KeyError:
+                warning_string = 'No ignore list found for this machine and no valid testimage_data.json, using defaults\n'
+                self.msg += '%s\n' % (warning_string)
+                self.logger.info("%s" % (warning_string))
+                errorlist = ignore_errors['default']
 
         for ignore_error in errorlist:
             ignore_error = ignore_error.replace('(', r'\(')
-- 
2.34.1

Re: [OE-core] [PATCH v2 1/2] oeqa: add utils/data.py with get_data() function

[Ross Burton]

On 18 Nov 2022, at 16:08, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
> 
> get_data() uses oeqa test method name and an optional
> key to get data from image specific "testimage_data.json"
> file located in image deploy directory. Image recipes can
> provide custom versions of this file which configures
> generic tests for a specific image when testing with
> testimage.bbclass
> 
> For example, the parselogs.py runtime test needs image
> specific configuration when the image has new errors from
> the kernel which acceptable and can be ignored.
> 
> Same machine can be used to generate multiple images with different
> runtime behavior so using image as the key and not machine.

Note that there’s a very related bug:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14604

In this case I need to extend parselogs *per machine*.

Does this patch series support this?

Ross

Re: [OE-core] [PATCH v2 1/2] oeqa: add utils/data.py with get_data() function

[Richard Purdie]

On Fri, 2022-11-18 at 18:08 +0200, Mikko Rapeli wrote:
> get_data() uses oeqa test method name and an optional
> key to get data from image specific "testimage_data.json"
> file located in image deploy directory. Image recipes can
> provide custom versions of this file which configures
> generic tests for a specific image when testing with
> testimage.bbclass
> 
> For example, the parselogs.py runtime test needs image
> specific configuration when the image has new errors from
> the kernel which acceptable and can be ignored.
> 
> Same machine can be used to generate multiple images with different
> runtime behavior so using image as the key and not machine.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/lib/oeqa/utils/data.py | 41 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
>  create mode 100644 meta/lib/oeqa/utils/data.py
> 
> v2: no changes

I'm afraid I'm really against adding this json approach to adding data
and I'm not planning to merge it.

I don't like having two different files containing recipe data in
different formats and I don't like the datastore not being definitive
for a given recipe.

I agree we need to be able to extend this data on a per recipe basis
but I think we need to do that from the recipes themselves.

Just to be clear, even done with json as in this patch, it isn't
something which would be an acceptable backport to an LTS.

There are two other options. One is to 'abuse' variable flags:

OEQA_PARSELOGS_IGNORELIST[qemux86] = "xxx"
OEQA_PARSELOGS_IGNORELIST[qemuarm] = "yyy"

which I don't like but could likely be made to work today.

The other is to introduce some dedicated dict/list like data syntax to
support more structured data which I think would be the better solution
in the long run but is a bit more work to implement up front.

Cheers,

Richard