XML test runner, or how to override Poky's python code

XML test runner, or how to override Poky's python code

[Mardegan, Alberto]

Hi there!
  Since Yocto "thud", the test suite no longer supports outputting the
test report in XML format.

While I'm not especially fond of that format, it had one feature which
was quite important for our further processing of test results: the
stdout from the test was captured and included in the test report (now
the JSON test report only contains the stdout and stderr for the failing
tests).

Is it possible from a layer to alter the implementation of the
OETestRunner, so that the bitbake testimage command would behave the way
we want to?

Ciao,
  Alberto

________________________________

This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you.

Re: XML test runner, or how to override Poky's python code

[Richard Purdie]

On Thu, 2019-04-18 at 06:31 +0000, Mardegan, Alberto wrote:
> Hi there!
>   Since Yocto "thud", the test suite no longer supports outputting
> the
> test report in XML format.
> 
> While I'm not especially fond of that format, it had one feature
> which
> was quite important for our further processing of test results: the
> stdout from the test was captured and included in the test report
> (now
> the JSON test report only contains the stdout and stderr for the
> failing
> tests).
> 
> Is it possible from a layer to alter the implementation of the
> OETestRunner, so that the bitbake testimage command would behave the
> way
> we want to?

Its certainly possible to alter pretty much any of the code but it
would probably involve duplication which would be ugly. A better way
may be to send patches which optionally allow the test output to be
stored in the json output based upon configuration?

Cheers,

Richard

[PATCH 1/1] XML test runner, or how to override Poky's python code

[Mardegan, Alberto]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

Hi,

On 18/04/19 11:31, Richard Purdie wrote:
> Its certainly possible to alter pretty much any of the code but it
> would probably involve duplication which would be ugly. A better way
> may be to send patches which optionally allow the test output to be
> stored in the json output based upon configuration?

Here's the patch; I hope that Thunderbird won't play too much with it :-)

Ciao,
  Alberto


________________________________

This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-oeqa-core-runner-dump-stdout-and-stderr-of-each-test.patch --]
[-- Type: text/x-patch; name="0001-oeqa-core-runner-dump-stdout-and-stderr-of-each-test.patch", Size: 2778 bytes --]

From ef01f73c61b7f202e905fc4d9ae7fb5057210805 Mon Sep 17 00:00:00 2001
From: Alberto Mardegan <amardegan@luxoft.com>
Date: Tue, 23 Apr 2019 15:25:36 +0300
Subject: [PATCH] oeqa/core/runner: dump stdout and stderr of each test case

Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.

We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports.

Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
---
 meta/lib/oeqa/core/runner.py | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index df88b85f1c..288b3ae614 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -7,6 +7,7 @@ import unittest
 import logging
 import re
 import json
+import sys
 
 from unittest import TextTestResult as _TestResult
 from unittest import TextTestRunner as _TestRunner
@@ -45,6 +46,9 @@ class OETestResult(_TestResult):
 
         self.tc = tc
 
+        # stdout and stderr for each test case
+        self.logged_output = {}
+
     def startTest(self, test):
         # May have been set by concurrencytest
         if test.id() not in self.starttime:
@@ -53,6 +57,9 @@ class OETestResult(_TestResult):
 
     def stopTest(self, test):
         self.endtime[test.id()] = time.time()
+        if self.buffer:
+            self.logged_output[test.id()] = (
+                    sys.stdout.getvalue(), sys.stderr.getvalue())
         super(OETestResult, self).stopTest(test)
         if test.id() in self.progressinfo:
             self.tc.logger.info(self.progressinfo[test.id()])
@@ -144,10 +151,14 @@ class OETestResult(_TestResult):
             if status not in logs:
                 logs[status] = []
             logs[status].append("RESULTS - %s - Testcase %s: %s%s" % (case.id(), oeid, status, t))
+            report = {'status': status}
             if log:
-                result[case.id()] = {'status': status, 'log': log}
-            else:
-                result[case.id()] = {'status': status}
+                report['log'] = log
+            if case.id() in self.logged_output:
+                (stdout, stderr) = self.logged_output[case.id()]
+                report['stdout'] = stdout
+                report['stderr'] = stderr
+            result[case.id()] = report
 
         for i in ['PASSED', 'SKIPPED', 'EXPECTEDFAIL', 'ERROR', 'FAILED', 'UNKNOWN']:
             if i not in logs:
-- 
2.17.1

Re: [PATCH 1/1] XML test runner, or how to override Poky's python code

[Alexander Kanavin]

Please send the patch using 'git send-email' to oe-core mailing list.

Also, this should be optional, subject to a bitbake variable (which
you can pass in via a bool parameter that defaults to False). Not
everyone wants the full stdout/stderr content in the json.

Alex


On Tue, 23 Apr 2019 at 14:37, Mardegan, Alberto <AMardegan@luxoft.com> wrote:
>
> Hi,
>
> On 18/04/19 11:31, Richard Purdie wrote:
> > Its certainly possible to alter pretty much any of the code but it
> > would probably involve duplication which would be ugly. A better way
> > may be to send patches which optionally allow the test output to be
> > stored in the json output based upon configuration?
>
> Here's the patch; I hope that Thunderbird won't play too much with it :-)
>
> Ciao,
>   Alberto
>
>
> ________________________________
>
> This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you.
> --
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky