* [xenomai-images][PATCH 1/4] ci: Add option to upload artifacts to aws bucket
2021-07-08 15:02 [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Q. Gylstorff
@ 2021-07-08 15:02 ` Q. Gylstorff
2021-07-08 15:02 ` [xenomai-images][PATCH 2/4] ci: adapt to public lava-server Q. Gylstorff
` (3 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Q. Gylstorff @ 2021-07-08 15:02 UTC (permalink / raw)
To: jan.kiszka, xenomai
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
The current build time is doubled by uploading the build artifacts
to the lava-server. By moving the storage of the build artifacts
to aws The build time should be reduces significantly.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
ci/gitlab-ci-base.yml | 3 ++-
scripts/deploy_to_aws.sh | 46 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 48 insertions(+), 1 deletion(-)
create mode 100755 scripts/deploy_to_aws.sh
diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
index cf91379..352e023 100644
--- a/ci/gitlab-ci-base.yml
+++ b/ci/gitlab-ci-base.yml
@@ -44,7 +44,8 @@ default:
script:
- echo "Building kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}"
- kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
- - if [ -z "${USE_GITLAB_ARTIFACTS}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
+ - if [ -z "${USE_GITLAB_ARTIFACTS}" -a -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
+ - if [ -n "${USE_S3_BUCKET}" ]; then scripts/deploy_to_aws.sh ${TARGET}; fi
.test:
extends: .add-lava-ssh-config
diff --git a/scripts/deploy_to_aws.sh b/scripts/deploy_to_aws.sh
new file mode 100755
index 0000000..814aec7
--- /dev/null
+++ b/scripts/deploy_to_aws.sh
@@ -0,0 +1,46 @@
+#!/bin/sh
+
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2019-2021
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+set -e
+target="$1"
+
+if [ -z "${target}" ]; then
+ exit 1
+fi
+images_dir=build/tmp/deploy/images
+
+if [ -z "${S3_BUCKET_URL}" ] || [ -z "${AWS_ACCESS_KEY_ID}" ] || [ -z "${AWS_SECRET_ACCESS_KEY}" ]; then
+ echo "environment not available or incomplete - do not deploy"
+ exit 0
+fi
+
+destination="${S3_BUCKET_URL}"
+
+aws_args="--acl=public-read"
+
+isar_base_name="${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${target}"
+deploy_dir="${destination}${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
+#KERNEL
+aws s3 cp "${aws_args}" "${images_dir}/${target}/${isar_base_name}-vmlinuz" "${deploy_dir}/${isar_base_name}-vmlinuz"
+# INITRD
+aws s3 cp "${aws_args}" "${images_dir}/${target}/${isar_base_name}-initrd.img" "${deploy_dir}/${isar_base_name}-initrd.img"
+# ROOTFS
+if ! ls "${images_dir}/${target}/${isar_base_name}".*.gz ; then
+ gzip "${images_dir}/${target}/${isar_base_name}.*"
+fi
+image=$(ls "${images_dir}/${target}/${isar_base_name}".*.gz)
+aws s3 cp "${aws_args}" "${image}" "${deploy_dir}/$(basename "${image}")"
+# DTB
+dtb="$(ls "${images_dir}/${target}/*.dtb" 2> /dev/null)"
+if [ -n "${dtb}" ] ; then
+ aws s3 cp "${aws_args}" "${dtb}" "${deploy_dir}/$(basename "${dtb}")"
+fi
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [xenomai-images][PATCH 2/4] ci: adapt to public lava-server
2021-07-08 15:02 [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Q. Gylstorff
2021-07-08 15:02 ` [xenomai-images][PATCH 1/4] ci: Add option to upload artifacts to aws bucket Q. Gylstorff
@ 2021-07-08 15:02 ` Q. Gylstorff
2021-07-08 15:03 ` [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings Q. Gylstorff
` (2 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Q. Gylstorff @ 2021-07-08 15:02 UTC (permalink / raw)
To: jan.kiszka, xenomai
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Adapt ci scripts to the new lava-server[1].
[1]: https://lava.xenomai.org
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
ci/gitlab-ci-base.yml | 3 ++-
scripts/run-lava-tests.sh | 57 ++++++++++++++++++++++-----------------
2 files changed, 35 insertions(+), 25 deletions(-)
diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
index 352e023..96b9c2c 100644
--- a/ci/gitlab-ci-base.yml
+++ b/ci/gitlab-ci-base.yml
@@ -22,6 +22,7 @@ variables:
LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
ISAR_IMAGE: demo-image
ISAR_DISTRIBUTION: xenomai-demo
+ LAVA_TESTS_ENABLED: "true"
default:
image: ghcr.io/siemens/kas/kas-isar:2.4
@@ -58,7 +59,7 @@ default:
BUILD_JOB_NAME: "build-${BUILD_IDENTIFIER}:${TARGET}${TARGET_EXTENSION}"
only:
variables:
- - $LAVA_SSH_USER
+ - $LAVA_TESTS_ENABLED == "true"
.build:qemu-amd64:
extends: .build
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 764f3a9..e8c7377 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -16,28 +16,36 @@ if [ -z "${TARGET}" ]; then
exit -1
fi
-lava_master_port="${LAVA_MASTER_PORT:-28080}"
-if [ -n "${LAVA_SSH_PORT}" ]; then
- lava_ssh_port="-p ${LAVA_SSH_PORT}"
+if [ -n "${LAVA_SSH_HOST}" ]; then
+ echo "use SSH to connect to lava server"
+ # private master with ssh forward access
+ lava_master_port="${LAVA_MASTER_PORT:-28080}"
+ lava_ssh_port="-p ${LAVA_SSH_PORT:-22}"
+ lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
+ lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
+ # open connection for ssh port forwarding
+ ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
+ ssh_pid="$!"
+ # wait for connection
+ interval=1
+ timeout=60
+ until ss -tulw | grep -q "${lava_master_port}"
+ do
+ if [ ${timeout} -le 0 ]; then
+ echo "could not open connection to LAVA Master"
+ exit 1
+ fi
+ sleep ${interval}
+ timeout=$(( timeout - interval ))
+ done
+ artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
+ lava_master_uri="${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}"
+else
+ # S3 artifacts
+ artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
+ # public master
+ lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
fi
-lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
-lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
-# open connection for ssh port forwarding
-ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
-ssh_pid=$!
-# wait for connection
-interval=1
-timeout=60
-until ss -tulw | grep -q "${lava_master_port}"
-do
- if [ ${timeout} -le 0 ]; then
- echo "could not open connection to LAVA Master"
- exit 1
- fi
- sleep ${interval}
- timeout=$(expr ${timeout} - ${interval})
-done
-lava_master_uri=${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}
# connect to lava master
lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
@@ -45,7 +53,6 @@ lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --u
if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
else
- artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
fi
job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
@@ -61,8 +68,10 @@ test_id=$(lavacli jobs submit ${template})
lavacli jobs logs ${test_id}
lavacli results ${test_id}
# change return code to generate a error in gitlab-ci if a test is failed
-number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l)
-kill ${ssh_pid}
+number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
+if [ -n "${ssh_pid}" ];then
+ kill "${ssh_pid}"
+fi
if [ "${number_of_fails}" -gt "0" ]; then
exit 1
fi
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:02 [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Q. Gylstorff
2021-07-08 15:02 ` [xenomai-images][PATCH 1/4] ci: Add option to upload artifacts to aws bucket Q. Gylstorff
2021-07-08 15:02 ` [xenomai-images][PATCH 2/4] ci: adapt to public lava-server Q. Gylstorff
@ 2021-07-08 15:03 ` Q. Gylstorff
2021-07-08 15:43 ` Bezdeka, Florian
2021-07-08 15:50 ` Bezdeka, Florian
2021-07-08 15:03 ` [xenomai-images][PATCH 4/4] README: Add information about LAVA Lab Q. Gylstorff
2021-07-08 15:43 ` [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Jan Kiszka
4 siblings, 2 replies; 12+ messages in thread
From: Q. Gylstorff @ 2021-07-08 15:03 UTC (permalink / raw)
To: jan.kiszka, xenomai
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Add missing quotes and simplify statements.
Also add messages to indicate progress.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
1 file changed, 22 insertions(+), 15 deletions(-)
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index e8c7377..72138c3 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -9,11 +9,10 @@
#
# SPDX-License-Identifier: MIT
#
-set -e
TARGET=$1
if [ -z "${TARGET}" ]; then
echo "no target was given"
- exit -1
+ exit 1
fi
if [ -n "${LAVA_SSH_HOST}" ]; then
@@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
# open connection for ssh port forwarding
- ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
+ ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
ssh_pid="$!"
# wait for connection
interval=1
@@ -47,28 +46,36 @@ else
lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
fi
+echo "connect to lava server: ${lava_master_uri}"
+
# connect to lava master
-lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
+lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
#generate lava job description from template
if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
else
DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
fi
+
+echo "Deploy artifacts from '${artifact_url}'"
+
job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
tmp_dir=$(mktemp -d)
-template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
-cp ${job_template_path}-${TARGET}.yml ${template}
-sed -i "s|\${TARGET}|${TARGET}|g" $template
-sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
-sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
-sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
-sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
-test_id=$(lavacli jobs submit ${template})
-lavacli jobs logs ${test_id}
-lavacli results ${test_id}
+template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
+cp "${job_template_path}-${TARGET}.yml" "${template}"
+sed -i "s|\${TARGET}|${TARGET}|g" "$template"
+sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
+sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
+sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
+sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
+
+echo "submit lava job"
+
+test_id=$(lavacli jobs submit "${template}")
+lavacli jobs logs "${test_id}"
+lavacli results "${test_id}"
# change return code to generate a error in gitlab-ci if a test is failed
-number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
+number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
if [ -n "${ssh_pid}" ];then
kill "${ssh_pid}"
fi
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:03 ` [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings Q. Gylstorff
@ 2021-07-08 15:43 ` Bezdeka, Florian
2021-07-08 15:45 ` Jan Kiszka
2021-07-08 15:50 ` Bezdeka, Florian
1 sibling, 1 reply; 12+ messages in thread
From: Bezdeka, Florian @ 2021-07-08 15:43 UTC (permalink / raw)
To: quirin.gylstorff, jan.kiszka, xenomai
On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>
> Add missing quotes and simplify statements.
> Also add messages to indicate progress.
>
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
> 1 file changed, 22 insertions(+), 15 deletions(-)
>
> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
> index e8c7377..72138c3 100755
> --- a/scripts/run-lava-tests.sh
> +++ b/scripts/run-lava-tests.sh
> @@ -9,11 +9,10 @@
> #
> # SPDX-License-Identifier: MIT
> #
> -set -e
> TARGET=$1
> if [ -z "${TARGET}" ]; then
> echo "no target was given"
> - exit -1
> + exit 1
> fi
>
> if [ -n "${LAVA_SSH_HOST}" ]; then
> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
> # open connection for ssh port forwarding
> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
> ssh_pid="$!"
> # wait for connection
> interval=1
> @@ -47,28 +46,36 @@ else
> lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
> fi
>
> +echo "connect to lava server: ${lava_master_uri}"
> +
> # connect to lava master
> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
> #generate lava job description from template
> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
> else
> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
> fi
> +
> +echo "Deploy artifacts from '${artifact_url}'"
> +
> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
> tmp_dir=$(mktemp -d)
> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
> -cp ${job_template_path}-${TARGET}.yml ${template}
> -sed -i "s|\${TARGET}|${TARGET}|g" $template
> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
> -test_id=$(lavacli jobs submit ${template})
> -lavacli jobs logs ${test_id}
> -lavacli results ${test_id}
> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
> +cp "${job_template_path}-${TARGET}.yml" "${template}"
> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
Can't we simply use something like envsubst?
> +
> +echo "submit lava job"
> +
> +test_id=$(lavacli jobs submit "${template}")
> +lavacli jobs logs "${test_id}"
> +lavacli results "${test_id}"
> # change return code to generate a error in gitlab-ci if a test is failed
> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
> if [ -n "${ssh_pid}" ];then
> kill "${ssh_pid}"
> fi
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:43 ` Bezdeka, Florian
@ 2021-07-08 15:45 ` Jan Kiszka
0 siblings, 0 replies; 12+ messages in thread
From: Jan Kiszka @ 2021-07-08 15:45 UTC (permalink / raw)
To: Bezdeka, Florian (T RDA IOT SES-DE),
Gylstorff, Quirin (T RDA IOT SES-DE),
xenomai
On 08.07.21 17:43, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> Add missing quotes and simplify statements.
>> Also add messages to indicate progress.
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
>> 1 file changed, 22 insertions(+), 15 deletions(-)
>>
>> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
>> index e8c7377..72138c3 100755
>> --- a/scripts/run-lava-tests.sh
>> +++ b/scripts/run-lava-tests.sh
>> @@ -9,11 +9,10 @@
>> #
>> # SPDX-License-Identifier: MIT
>> #
>> -set -e
>> TARGET=$1
>> if [ -z "${TARGET}" ]; then
>> echo "no target was given"
>> - exit -1
>> + exit 1
>> fi
>>
>> if [ -n "${LAVA_SSH_HOST}" ]; then
>> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
>> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>> # open connection for ssh port forwarding
>> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
>> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
>> ssh_pid="$!"
>> # wait for connection
>> interval=1
>> @@ -47,28 +46,36 @@ else
>> lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org/}"
>> fi
>>
>> +echo "connect to lava server: ${lava_master_uri}"
>> +
>> # connect to lava master
>> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
>> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
>> #generate lava job description from template
>> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
>> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
>> else
>> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
>> fi
>> +
>> +echo "Deploy artifacts from '${artifact_url}'"
>> +
>> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
>> tmp_dir=$(mktemp -d)
>> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
>> -cp ${job_template_path}-${TARGET}.yml ${template}
>> -sed -i "s|\${TARGET}|${TARGET}|g" $template
>> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
>> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
>> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
>> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
>> -test_id=$(lavacli jobs submit ${template})
>> -lavacli jobs logs ${test_id}
>> -lavacli results ${test_id}
>> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
>> +cp "${job_template_path}-${TARGET}.yml" "${template}"
>> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
>> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
>> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
>> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
>> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
>
> Can't we simply use something like envsubst?
>
Valid point - for a cleanup patch on top.
Jan
>> +
>> +echo "submit lava job"
>> +
>> +test_id=$(lavacli jobs submit "${template}")
>> +lavacli jobs logs "${test_id}"
>> +lavacli results "${test_id}"
>> # change return code to generate a error in gitlab-ci if a test is failed
>> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
>> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
>> if [ -n "${ssh_pid}" ];then
>> kill "${ssh_pid}"
>> fi
>
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:03 ` [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings Q. Gylstorff
2021-07-08 15:43 ` Bezdeka, Florian
@ 2021-07-08 15:50 ` Bezdeka, Florian
2021-07-08 15:56 ` Jan Kiszka
2021-07-09 11:40 ` Gylstorff Quirin
1 sibling, 2 replies; 12+ messages in thread
From: Bezdeka, Florian @ 2021-07-08 15:50 UTC (permalink / raw)
To: quirin.gylstorff, jan.kiszka, xenomai
On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>
> Add missing quotes and simplify statements.
> Also add messages to indicate progress.
>
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
> 1 file changed, 22 insertions(+), 15 deletions(-)
>
> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
> index e8c7377..72138c3 100755
> --- a/scripts/run-lava-tests.sh
> +++ b/scripts/run-lava-tests.sh
> @@ -9,11 +9,10 @@
> #
> # SPDX-License-Identifier: MIT
> #
> -set -e
> TARGET=$1
> if [ -z "${TARGET}" ]; then
> echo "no target was given"
> - exit -1
> + exit 1
> fi
>
>
>
>
> if [ -n "${LAVA_SSH_HOST}" ]; then
> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
> # open connection for ssh port forwarding
> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
> ssh_pid="$!"
> # wait for connection
> interval=1
> @@ -47,28 +46,36 @@ else
> lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
> fi
>
>
>
>
> +echo "connect to lava server: ${lava_master_uri}"
> +
> # connect to lava master
> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
> #generate lava job description from template
> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
> else
> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
> fi
> +
> +echo "Deploy artifacts from '${artifact_url}'"
> +
> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
> tmp_dir=$(mktemp -d)
> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
> -cp ${job_template_path}-${TARGET}.yml ${template}
> -sed -i "s|\${TARGET}|${TARGET}|g" $template
> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
> -test_id=$(lavacli jobs submit ${template})
> -lavacli jobs logs ${test_id}
> -lavacli results ${test_id}
> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
> +cp "${job_template_path}-${TARGET}.yml" "${template}"
> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
> +
> +echo "submit lava job"
> +
> +test_id=$(lavacli jobs submit "${template}")
> +lavacli jobs logs "${test_id}"
> +lavacli results "${test_id}"
> # change return code to generate a error in gitlab-ci if a test is failed
> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
In addition to fetching the number of failed tests it would be possible
to download a junit test report from the lava master.
Something like the following could do it:
api="${lava_master_uri}/api/v0.2/jobs/${test_id}/junit"
auth="?token=${LAVA_MASTER_TOKEN}"
url="${api}/${auth}"
curl "${url}" -o test-report.xml
This xml can be integrated as artifact into gitlab, so we can see
details about test duration, state, ..., in gitlab directly. No need to
visit the LAVA instance for details.
> if [ -n "${ssh_pid}" ];then
> kill "${ssh_pid}"
> fi
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:50 ` Bezdeka, Florian
@ 2021-07-08 15:56 ` Jan Kiszka
2021-07-09 11:40 ` Gylstorff Quirin
1 sibling, 0 replies; 12+ messages in thread
From: Jan Kiszka @ 2021-07-08 15:56 UTC (permalink / raw)
To: Bezdeka, Florian (T RDA IOT SES-DE),
Gylstorff, Quirin (T RDA IOT SES-DE),
xenomai
On 08.07.21 17:50, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> Add missing quotes and simplify statements.
>> Also add messages to indicate progress.
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
>> 1 file changed, 22 insertions(+), 15 deletions(-)
>>
>> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
>> index e8c7377..72138c3 100755
>> --- a/scripts/run-lava-tests.sh
>> +++ b/scripts/run-lava-tests.sh
>> @@ -9,11 +9,10 @@
>> #
>> # SPDX-License-Identifier: MIT
>> #
>> -set -e
>> TARGET=$1
>> if [ -z "${TARGET}" ]; then
>> echo "no target was given"
>> - exit -1
>> + exit 1
>> fi
>>
>>
>>
>>
>> if [ -n "${LAVA_SSH_HOST}" ]; then
>> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
>> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>> # open connection for ssh port forwarding
>> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
>> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
>> ssh_pid="$!"
>> # wait for connection
>> interval=1
>> @@ -47,28 +46,36 @@ else
>> lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org/}"
>> fi
>>
>>
>>
>>
>> +echo "connect to lava server: ${lava_master_uri}"
>> +
>> # connect to lava master
>> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
>> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
>> #generate lava job description from template
>> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
>> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
>> else
>> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
>> fi
>> +
>> +echo "Deploy artifacts from '${artifact_url}'"
>> +
>> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
>> tmp_dir=$(mktemp -d)
>> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
>> -cp ${job_template_path}-${TARGET}.yml ${template}
>> -sed -i "s|\${TARGET}|${TARGET}|g" $template
>> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
>> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
>> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
>> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
>> -test_id=$(lavacli jobs submit ${template})
>> -lavacli jobs logs ${test_id}
>> -lavacli results ${test_id}
>> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
>> +cp "${job_template_path}-${TARGET}.yml" "${template}"
>> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
>> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
>> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
>> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
>> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
>> +
>> +echo "submit lava job"
>> +
>> +test_id=$(lavacli jobs submit "${template}")
>> +lavacli jobs logs "${test_id}"
>> +lavacli results "${test_id}"
>> # change return code to generate a error in gitlab-ci if a test is failed
>> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
>> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
>
> In addition to fetching the number of failed tests it would be possible
> to download a junit test report from the lava master.
>
> Something like the following could do it:
>
> api="${lava_master_uri}/api/v0.2/jobs/${test_id}/junit"
> auth="?token=${LAVA_MASTER_TOKEN}"
> url="${api}/${auth}"
> curl "${url}" -o test-report.xml
>
> This xml can be integrated as artifact into gitlab, so we can see
> details about test duration, state, ..., in gitlab directly. No need to
> visit the LAVA instance for details.
>
Sound interesting. Do you have an example for such visualization at hand?
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-08 15:50 ` Bezdeka, Florian
2021-07-08 15:56 ` Jan Kiszka
@ 2021-07-09 11:40 ` Gylstorff Quirin
2021-07-09 11:48 ` Jan Kiszka
1 sibling, 1 reply; 12+ messages in thread
From: Gylstorff Quirin @ 2021-07-09 11:40 UTC (permalink / raw)
To: Bezdeka, Florian (T RDA IOT SES-DE), Kiszka, Jan (T RDA IOT), xenomai
On 7/8/21 5:50 PM, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
> On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> Add missing quotes and simplify statements.
>> Also add messages to indicate progress.
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
>> 1 file changed, 22 insertions(+), 15 deletions(-)
>>
>> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
>> index e8c7377..72138c3 100755
>> --- a/scripts/run-lava-tests.sh
>> +++ b/scripts/run-lava-tests.sh
>> @@ -9,11 +9,10 @@
>> #
>> # SPDX-License-Identifier: MIT
>> #
>> -set -e
>> TARGET=$1
>> if [ -z "${TARGET}" ]; then
>> echo "no target was given"
>> - exit -1
>> + exit 1
>> fi
>>
>>
>>
>>
>> if [ -n "${LAVA_SSH_HOST}" ]; then
>> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
>> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>> # open connection for ssh port forwarding
>> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward localhost:'"${lava_master_port}"' localhost:80' "${lava_ssh_destination}" &
>> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
>> ssh_pid="$!"
>> # wait for connection
>> interval=1
>> @@ -47,28 +46,36 @@ else
>> lava_master_uri="${LAVA_MASTER_URL:-https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.xenomai.org%2F&data=04%7C01%7C6549bcd2-981c-4c06-8e1b-b5c6cc3441b4%40ad011.siemens.com%7Cbb6b436ee86c4149281908d94228280e%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637613562491292282%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DDVQCcSXv8grFeFCLbCl3voK%2BOkQll8dRW2q4%2Fnozx0%3D&reserved=0}"
>> fi
>>
>>
>>
>>
>> +echo "connect to lava server: ${lava_master_uri}"
>> +
>> # connect to lava master
>> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
>> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
>> #generate lava job description from template
>> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
>> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
>> else
>> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
>> fi
>> +
>> +echo "Deploy artifacts from '${artifact_url}'"
>> +
>> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
>> tmp_dir=$(mktemp -d)
>> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
>> -cp ${job_template_path}-${TARGET}.yml ${template}
>> -sed -i "s|\${TARGET}|${TARGET}|g" $template
>> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
>> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
>> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
>> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
>> -test_id=$(lavacli jobs submit ${template})
>> -lavacli jobs logs ${test_id}
>> -lavacli results ${test_id}
>> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
>> +cp "${job_template_path}-${TARGET}.yml" "${template}"
>> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
>> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
>> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
>> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
>> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
>> +
>> +echo "submit lava job"
>> +
>> +test_id=$(lavacli jobs submit "${template}")
>> +lavacli jobs logs "${test_id}"
>> +lavacli results "${test_id}"
>> # change return code to generate a error in gitlab-ci if a test is failed
>> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
>> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
>
> In addition to fetching the number of failed tests it would be possible
> to download a junit test report from the lava master.
>
> Something like the following could do it:
>
> api="${lava_master_uri}/api/v0.2/jobs/${test_id}/junit"
> auth="?token=${LAVA_MASTER_TOKEN}"
> url="${api}/${auth}"
> curl "${url}" -o test-report.xml
>
> This xml can be integrated as artifact into gitlab, so we can see
> details about test duration, state, ..., in gitlab directly. No need to
> visit the LAVA instance for details.
Do we have artifacts on https://source.denx.de? If not we can use AWS
for storage.
Quirin
>
>
>> if [ -n "${ssh_pid}" ];then
>> kill "${ssh_pid}"
>> fi
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings
2021-07-09 11:40 ` Gylstorff Quirin
@ 2021-07-09 11:48 ` Jan Kiszka
0 siblings, 0 replies; 12+ messages in thread
From: Jan Kiszka @ 2021-07-09 11:48 UTC (permalink / raw)
To: Gylstorff Quirin, Bezdeka, Florian (T RDA IOT SES-DE), xenomai
On 09.07.21 13:40, Gylstorff Quirin wrote:
>
>
> On 7/8/21 5:50 PM, Bezdeka, Florian (T RDA IOT SES-DE) wrote:
>> On Thu, 2021-07-08 at 17:03 +0200, Q. Gylstorff via Xenomai wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> Add missing quotes and simplify statements.
>>> Also add messages to indicate progress.
>>>
>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> ---
>>> scripts/run-lava-tests.sh | 37 ++++++++++++++++++++++---------------
>>> 1 file changed, 22 insertions(+), 15 deletions(-)
>>>
>>> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
>>> index e8c7377..72138c3 100755
>>> --- a/scripts/run-lava-tests.sh
>>> +++ b/scripts/run-lava-tests.sh
>>> @@ -9,11 +9,10 @@
>>> #
>>> # SPDX-License-Identifier: MIT
>>> #
>>> -set -e
>>> TARGET=$1
>>> if [ -z "${TARGET}" ]; then
>>> echo "no target was given"
>>> - exit -1
>>> + exit 1
>>> fi
>>>
>>>
>>>
>>>
>>> if [ -n "${LAVA_SSH_HOST}" ]; then
>>> @@ -24,7 +23,7 @@ if [ -n "${LAVA_SSH_HOST}" ]; then
>>> lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>>> lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>>> # open connection for ssh port forwarding
>>> - ssh -N "${lava_ssh_port}" "${lava_identity}" -o 'LocalForward
>>> localhost:'"${lava_master_port}"' localhost:80'
>>> "${lava_ssh_destination}" &
>>> + ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward
>>> localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
>>> ssh_pid="$!"
>>> # wait for connection
>>> interval=1
>>> @@ -47,28 +46,36 @@ else
>>>
>>> lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org/}"
>>>
>>> fi
>>>
>>>
>>>
>>>
>>> +echo "connect to lava server: ${lava_master_uri}"
>>> +
>>> # connect to lava master
>>> -lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri
>>> ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
>>> +lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri
>>> "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
>>> #generate lava job description from template
>>> if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
>>>
>>> DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
>>>
>>> else
>>>
>>> DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
>>> fi
>>> +
>>> +echo "Deploy artifacts from '${artifact_url}'"
>>> +
>>> job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
>>> tmp_dir=$(mktemp -d)
>>> -template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
>>> -cp ${job_template_path}-${TARGET}.yml ${template}
>>> -sed -i "s|\${TARGET}|${TARGET}|g" $template
>>> -sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" $template
>>> -sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" $template
>>> -sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" $template
>>> -sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" $template
>>> -test_id=$(lavacli jobs submit ${template})
>>> -lavacli jobs logs ${test_id}
>>> -lavacli results ${test_id}
>>> +template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
>>> +cp "${job_template_path}-${TARGET}.yml" "${template}"
>>> +sed -i "s|\${TARGET}|${TARGET}|g" "$template"
>>> +sed -i "s|\${BUILD_ARCH}|${BUILD_ARCH}|g" "$template"
>>> +sed -i "s|\${DEPLOY_URL}|${DEPLOY_URL}|g" "$template"
>>> +sed -i "s|\${ISAR_IMAGE}|${ISAR_IMAGE}|g" "$template"
>>> +sed -i "s|\${ISAR_DISTRIBUTION}|${ISAR_DISTRIBUTION}|g" "$template"
>>> +
>>> +echo "submit lava job"
>>> +
>>> +test_id=$(lavacli jobs submit "${template}")
>>> +lavacli jobs logs "${test_id}"
>>> +lavacli results "${test_id}"
>>> # change return code to generate a error in gitlab-ci if a test is
>>> failed
>>> -number_of_fails=$(lavacli results ${test_id} | grep fail | wc -l )
>>> +number_of_fails="$(lavacli results "${test_id}" | grep -c fail )"
>>
>> In addition to fetching the number of failed tests it would be possible
>> to download a junit test report from the lava master.
>>
>> Something like the following could do it:
>>
>> api="${lava_master_uri}/api/v0.2/jobs/${test_id}/junit"
>> auth="?token=${LAVA_MASTER_TOKEN}"
>> url="${api}/${auth}"
>> curl "${url}" -o test-report.xml
>>
>> This xml can be integrated as artifact into gitlab, so we can see
>> details about test duration, state, ..., in gitlab directly. No need to
>> visit the LAVA instance for details.
>
> Do we have artifacts on https://source.denx.de? If not we can use AWS
> for storage.
>
I don't see anything that should prevent storing a few small files along
the usual logs. Not sure how long the latter are kept, but those test
reports should probably have a similar lifetime.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 12+ messages in thread
* [xenomai-images][PATCH 4/4] README: Add information about LAVA Lab
2021-07-08 15:02 [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Q. Gylstorff
` (2 preceding siblings ...)
2021-07-08 15:03 ` [xenomai-images][PATCH 3/4] scripts/run_lava_tests.sh: cleanup shellcheck warnings Q. Gylstorff
@ 2021-07-08 15:03 ` Q. Gylstorff
2021-07-08 15:43 ` [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Jan Kiszka
4 siblings, 0 replies; 12+ messages in thread
From: Q. Gylstorff @ 2021-07-08 15:03 UTC (permalink / raw)
To: jan.kiszka, xenomai
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
README.md | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/README.md b/README.md
index 69b3e13..c195c0c 100644
--- a/README.md
+++ b/README.md
@@ -53,6 +53,20 @@ gitlab-runner exec docker --docker-privileged \
--env "NO_PROXY=$NO_PROXY" build:qemu-armhf
```
+## LAVA Lab
+
+The [Xenomai project](https://www.xenomai.org/) maintains a
+[LAVA lab](https://lava.xenomai.org) for integration testing.
+
+A LAVA lab can be created by following the
+[installation instructions](https://docs.lavasoftware.org/lava/first-installation.html)
+or use
+[lava docker](https://github.com/kernelci/lava-docker) for a faster setup.
+
+It is also possible to connect a lab to lava.xenomai.org. Please get in
+touch with the maintainers via mailing list to join a new lab with
+lava.xenomai.org.
+
## Community Resources
See [Xenomai project](https://www.xenomai.org/).
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server
2021-07-08 15:02 [xenomai-images][PATCH 0/4] Prepare xenomai-images for public LAVA Server Q. Gylstorff
` (3 preceding siblings ...)
2021-07-08 15:03 ` [xenomai-images][PATCH 4/4] README: Add information about LAVA Lab Q. Gylstorff
@ 2021-07-08 15:43 ` Jan Kiszka
4 siblings, 0 replies; 12+ messages in thread
From: Jan Kiszka @ 2021-07-08 15:43 UTC (permalink / raw)
To: Q. Gylstorff, xenomai
On 08.07.21 17:02, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>
> Add option to upload the image under test to AWS.
> Add option to use AWS as image download source and use
> [1] as default LAVA server.
>
> After this patchset is merged the private lava lab
> can be migrated.
>
> [1]: https://lava.xenomai.org
>
> Quirin Gylstorff (4):
> ci: Add option to upload artifacts to aws bucket
> ci: adapt to public lava-server
> scripts/run_lava_tests.sh: cleanup shellcheck warnings
> README: Add information about LAVA Lab
>
> README.md | 14 ++++++
> ci/gitlab-ci-base.yml | 6 ++-
> scripts/deploy_to_aws.sh | 46 ++++++++++++++++++++
> scripts/run-lava-tests.sh | 90 +++++++++++++++++++++++----------------
> 4 files changed, 117 insertions(+), 39 deletions(-)
> create mode 100755 scripts/deploy_to_aws.sh
>
> - '-
> 2.20.1
>
Thanks, applied to next.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 12+ messages in thread