[xenomai-images][PATCH 0/2] Fix pipeline failure detection

[xenomai-images][PATCH 0/2] Fix pipeline failure detection

[Florian Bezdeka]

Hi,

a failing xenomai (smokey) test was not properly reported by gitlab-ci.
In addition some bitbake warnings were triggered by the option file used
for CI builds.

It seems that LAVA decided to report test failures now as "failures"
instead of "errors". I did some research but could not identify any
recent LAVA change that triggers this. Might be the case that I
overlooked something or the error detection has been broken for a long
time now.

I have one more patch pending: Enable compat testing for the stable
3.2.x branch. Let me know if that is of general interest. It helped to
detect and fix some compat issues for this branch.

Best regards,
Florian

Florian Bezdeka (2):
  ci: Fix handling of lava test failures
  ci: Fix some bitbake warnings triggered by opt-ci.yml

 opt-ci.yml                |  4 ++--
 scripts/run-lava-tests.sh | 17 +++++++++++++----
 2 files changed, 15 insertions(+), 6 deletions(-)

-- 
2.39.1

[xenomai-images][PATCH 1/2] ci: Fix handling of lava test failures

[Florian Bezdeka]

There are two kinds of errors that can happen:
 - errors: Like LAVA infrastructure problems
 - failures: Like failing testsuites

We should check both of them to get the right signal back to gitlab-ci.

In my case I was expecting some test failures but was getting a "green"
pipeline result. The test log clearly showed that the tests failed as
expected but the gitlab-ci pipeline result was wrong.

A simplified junit test report (provided by LAVA) is provided below.
The first matches of "errors=" and "failures=" provide the full test
job result.

<?xml version="1.0" encoding="utf-8"?>
<testsuites disabled="0" errors="0" failures="1" tests="25" time="176.49">
	...
	<testsuite ...></testsuite>
	...
</testsuites>

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 scripts/run-lava-tests.sh | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 70deb36..10c60bc 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -2,7 +2,7 @@
 #
 # Xenomai Real-Time System
 #
-# Copyright (c) Siemens AG, 2019-2020
+# Copyright (c) Siemens AG, 2019-2023
 #
 # Authors:
 #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
@@ -16,6 +16,14 @@ if [ -z "${TARGET}" ]; then
     exit 1
 fi
 
+get_first_xml_attr_value()
+{
+	file=${1}
+	tag=${2}
+
+	grep -m 1 -o "${tag}=\".*\"" ${file} | cut -d\" -f2
+}
+
 # S3 artifacts
 artifact_url="${LAVA_ARTIFACTS_URL:-https://d1a54g2tyy2hl.cloudfront.net/artifacts}"
 # public master
@@ -51,8 +59,9 @@ url="${api}/${auth}"
 curl --silent "${url}" -o report.xml
 
 # change return code to generate a error in gitlab-ci if a test is failed
-number_of_fails="$(grep -o "errors=\"[0-9]*\"" report.xml | head -1  | cut -d\" -f2 )"
+errors=$(get_first_xml_attr_value report.xml errors)
+failures=$(get_first_xml_attr_value report.xml failures)
 
-if [ "${number_of_fails}" -gt "0" ]; then
-    exit 1
+if [ "${errors}" -gt "0" ] || [ "${failures}" -gt "0" ]; then
+	exit 1
 fi
-- 
2.39.1

[xenomai-images][PATCH 2/2] ci: Fix some bitbake warnings triggered by opt-ci.yml

[Florian Bezdeka]

The WARNING:
WARNING: IMAGE_INSTALL:remove += is not a recommended operator combination, please replace it.
WARNING: IMAGE_INSTALL:remove += is not a recommended operator combination, please replace it.

We can savely remove the "+".

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 opt-ci.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/opt-ci.yml b/opt-ci.yml
index 32de022..80a3970 100644
--- a/opt-ci.yml
+++ b/opt-ci.yml
@@ -21,6 +21,6 @@ header:
 local_conf_header:
   xenomai-ci: |
     ROOTFS_POSTPROCESS_COMMAND:remove = "image_postprocess_sshd_key_regen"
-    IMAGE_INSTALL:remove += "sshd-regen-keys"
-    IMAGE_INSTALL:remove += "expand-on-first-boot"
+    IMAGE_INSTALL:remove = "sshd-regen-keys"
+    IMAGE_INSTALL:remove = "expand-on-first-boot"
     IMAGE_FSTYPES = "${@ 'ext4.gz' if d.getVar('MACHINE', True).startswith('qemu') else 'tar.gz' }"
-- 
2.39.1

Re: [xenomai-images][PATCH 0/2] Fix pipeline failure detection

[Jan Kiszka]

On 23.02.23 08:40, Florian Bezdeka wrote:
> Hi,
> 
> a failing xenomai (smokey) test was not properly reported by gitlab-ci.
> In addition some bitbake warnings were triggered by the option file used
> for CI builds.
> 
> It seems that LAVA decided to report test failures now as "failures"
> instead of "errors". I did some research but could not identify any
> recent LAVA change that triggers this. Might be the case that I
> overlooked something or the error detection has been broken for a long
> time now.
> 
> I have one more patch pending: Enable compat testing for the stable
> 3.2.x branch. Let me know if that is of general interest. It helped to
> detect and fix some compat issues for this branch.
> 
> Best regards,
> Florian
> 
> Florian Bezdeka (2):
>   ci: Fix handling of lava test failures
>   ci: Fix some bitbake warnings triggered by opt-ci.yml
> 
>  opt-ci.yml                |  4 ++--
>  scripts/run-lava-tests.sh | 17 +++++++++++++----
>  2 files changed, 15 insertions(+), 6 deletions(-)
> 

Thanks, applied.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux