[xenomai-images][PATCH 0/3] ci: cleanup

[xenomai-images][PATCH 0/3] ci: cleanup

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

Remove artifact deployment to lava-master via ssh and to gitlab artifacts.

The current CI builds use AWS S3 for object storage. The objects used by the
test can be found for 5 days at [1].

[1]: https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts/<GITLAB PIPELINE ID>/<kernel version>/<target>.

Quirin Gylstorff (3):
  ci: Remove artifact deployment
  ci: Remove deploy with ssh
  ci/gitlab-ci-base: use rules

 .gitlab-ci-artifacts.yml            | 13 --------
 .gitlab-ci.yml                      | 20 ++++++++++-
 ci/artifacts.yml                    | 20 -----------
 ci/child_pipelines_artifacts.yml    | 34 -------------------
 ci/child_pipelines_no_artifacts.yml | 34 -------------------
 ci/gitlab-ci-base.yml               |  9 ++---
 ci/no-artifacts.yml                 | 13 --------
 scripts/deploy_for_testing.sh       | 51 -----------------------------
 scripts/get-build-id.sh             | 34 -------------------
 scripts/run-lava-tests.sh           | 45 ++++---------------------
 10 files changed, 28 insertions(+), 245 deletions(-)
 delete mode 100644 .gitlab-ci-artifacts.yml
 delete mode 100644 ci/artifacts.yml
 delete mode 100644 ci/child_pipelines_artifacts.yml
 delete mode 100644 ci/child_pipelines_no_artifacts.yml
 delete mode 100644 ci/no-artifacts.yml
 delete mode 100755 scripts/deploy_for_testing.sh
 delete mode 100755 scripts/get-build-id.sh

-- 
2.20.1

[xenomai-images][PATCH 1/3] ci: Remove artifact deployment

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

We will use AWS S3 deployment for LAVA CI Artifacts. Therefore
remove the Deployment via gitlab artifacts as the LAVA Lab is now public
available.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 .gitlab-ci-artifacts.yml            | 13 -----------
 .gitlab-ci.yml                      | 20 ++++++++++++++++-
 ci/artifacts.yml                    | 20 -----------------
 ci/child_pipelines_artifacts.yml    | 34 -----------------------------
 ci/child_pipelines_no_artifacts.yml | 34 -----------------------------
 ci/gitlab-ci-base.yml               |  5 ++---
 ci/no-artifacts.yml                 | 13 -----------
 scripts/get-build-id.sh             | 34 -----------------------------
 scripts/run-lava-tests.sh           |  8 ++-----
 9 files changed, 23 insertions(+), 158 deletions(-)
 delete mode 100644 .gitlab-ci-artifacts.yml
 delete mode 100644 ci/artifacts.yml
 delete mode 100644 ci/child_pipelines_artifacts.yml
 delete mode 100644 ci/child_pipelines_no_artifacts.yml
 delete mode 100644 ci/no-artifacts.yml
 delete mode 100755 scripts/get-build-id.sh

diff --git a/.gitlab-ci-artifacts.yml b/.gitlab-ci-artifacts.yml
deleted file mode 100644
index 85345b8..0000000
--- a/.gitlab-ci-artifacts.yml
+++ /dev/null
@@ -1,13 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019 - 2021
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-include:
-  - local: '/ci/child_pipelines_artifacts.yml'
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a808859..b5a3a35 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -10,4 +10,22 @@
 #
 
 include:
-  - local: '/ci/child_pipelines_no_artifacts.yml'
+  - local: '/ci/child_pipelines_common.yml'
+
+stable/v3.0.x:
+  extends: .parent
+  trigger:
+    include:
+      - local: '/ci/xenomai_3_0_x.yml'
+
+stable/v3.1.x:
+  extends: .parent
+  trigger:
+    include:
+      - local: '/ci/xenomai_3_1_x.yml'
+
+next:
+  extends: .parent
+  trigger:
+    include:
+      - local: '/ci/xenomai_next.yml'
diff --git a/ci/artifacts.yml b/ci/artifacts.yml
deleted file mode 100644
index ba851cf..0000000
--- a/ci/artifacts.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019 - 2021
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-variables:
-  USE_GITLAB_ARTIFACTS: "true"
-
-.build:
-  extends: .build-base
-  artifacts:
-    paths:
-    - build/tmp/deploy/images/${TARGET}/
-    expire_in: 1 week
diff --git a/ci/child_pipelines_artifacts.yml b/ci/child_pipelines_artifacts.yml
deleted file mode 100644
index 1dd856c..0000000
--- a/ci/child_pipelines_artifacts.yml
+++ /dev/null
@@ -1,34 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019 - 2021
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-include:
-  - local: '/ci/child_pipelines_common.yml'
-
-stable/v3.0.x:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/artifacts.yml'
-      - local: '/ci/xenomai_3_0_x.yml'
-
-stable/v3.1.x:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/artifacts.yml'
-      - local: '/ci/xenomai_3_1_x.yml'
-
-next:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/artifacts.yml'
-      - local: '/ci/xenomai_next.yml'
diff --git a/ci/child_pipelines_no_artifacts.yml b/ci/child_pipelines_no_artifacts.yml
deleted file mode 100644
index c90204d..0000000
--- a/ci/child_pipelines_no_artifacts.yml
+++ /dev/null
@@ -1,34 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019 - 2021
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-include:
-  - local: '/ci/child_pipelines_common.yml'
-
-stable/v3.0.x:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/no-artifacts.yml'
-      - local: '/ci/xenomai_3_0_x.yml'
-
-stable/v3.1.x:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/no-artifacts.yml'
-      - local: '/ci/xenomai_3_1_x.yml'
-
-next:
-  extends: .parent
-  trigger:
-    include:
-      - local: '/ci/no-artifacts.yml'
-      - local: '/ci/xenomai_next.yml'
diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
index 7f245e3..362a636 100644
--- a/ci/gitlab-ci-base.yml
+++ b/ci/gitlab-ci-base.yml
@@ -39,13 +39,13 @@ default:
     - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa && chmod 600 ~/.ssh/lava_id_rsa
     - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
 
-.build-base:
+.build:
   extends: .add-lava-ssh-config
   stage: build
   script:
     - echo "Building kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}"
     - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
-    - if [ -z "${USE_GITLAB_ARTIFACTS}" -a -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
+    - if [ -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
     - if [ -n "${USE_S3_BUCKET}" ]; then scripts/deploy_to_aws.sh ${TARGET}; fi
 
 .test:
@@ -53,7 +53,6 @@ default:
   stage: test
   script:
     - scripts/install-lavacli.sh
-    - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
     - scripts/run-lava-tests.sh ${TARGET}
   variables:
     BUILD_JOB_NAME: "build-${BUILD_IDENTIFIER}:${TARGET}${TARGET_EXTENSION}"
diff --git a/ci/no-artifacts.yml b/ci/no-artifacts.yml
deleted file mode 100644
index a820f89..0000000
--- a/ci/no-artifacts.yml
+++ /dev/null
@@ -1,13 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019 - 2021
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-.build:
-  extends: .build-base
diff --git a/scripts/get-build-id.sh b/scripts/get-build-id.sh
deleted file mode 100755
index 1a0dcaa..0000000
--- a/scripts/get-build-id.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/sh
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2020
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-name=$1
-if [ -z "${name}" ]; then
-    echo "no build name was given"
-    exit -1
-fi
-index=0
-JSON=$(curl -s --header "PRIVATE-TOKEN: ${API_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs?scope[]=success"  )
-while build_name=$(echo "${JSON}" | tr '\r\n' ' ' | jq -re ".[${index}].name")
-do
-    if [ "${build_name}" = "${name}" ]; then
-        break
-    fi
-     index=$((index+1))
-done
-id=$(echo "${JSON}"  | tr '\r\n' ' ' | jq -r ".[${index}].id")
-if [ "$id" = "null" ]; then
-    >&2 echo "ERROR: did not find build with name: $name"
-    return 1
-else
-    echo $id
-    return 0
-fi
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 7972279..13fac31 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -52,12 +52,9 @@ echo "connect to lava server: ${lava_master_uri}"
 lavacli identities add --token "${LAVA_MASTER_TOKEN}" --uri "${lava_master_uri}" --username "${LAVA_MASTER_ACCOUNT}" default
 
 #generate lava job description from template
-if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
-    DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
-else
-    DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
-fi
+DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
 export DEPLOY_URL
+
 echo "Deploy artifacts from '${artifact_url}'"
 
 job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
@@ -66,7 +63,6 @@ template="${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml"
 
 cat "${job_template_path}-${TARGET}.yml" | envsubst > "$template"
 
-
 echo "submit lava job"
 
 test_id=$(lavacli jobs submit "${template}")
-- 
2.20.1

[xenomai-images][PATCH 2/3] ci: Remove deploy with ssh

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

We will use AWS S3 deployment for LAVA Build Artifacts. Therefore
remove the Deployment via ssh as the LAVA Lab is now public
available.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 ci/gitlab-ci-base.yml         |  1 -
 scripts/deploy_for_testing.sh | 51 -----------------------------------
 scripts/run-lava-tests.sh     | 37 +++----------------------
 3 files changed, 4 insertions(+), 85 deletions(-)
 delete mode 100755 scripts/deploy_for_testing.sh

diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
index 362a636..84bbdd4 100644
--- a/ci/gitlab-ci-base.yml
+++ b/ci/gitlab-ci-base.yml
@@ -45,7 +45,6 @@ default:
   script:
     - echo "Building kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}"
     - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
-    - if [ -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
     - if [ -n "${USE_S3_BUCKET}" ]; then scripts/deploy_to_aws.sh ${TARGET}; fi
 
 .test:
diff --git a/scripts/deploy_for_testing.sh b/scripts/deploy_for_testing.sh
deleted file mode 100755
index 3878e7d..0000000
--- a/scripts/deploy_for_testing.sh
+++ /dev/null
@@ -1,51 +0,0 @@
-#!/bin/sh
-
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2019-2020
-#
-# Authors:
-#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-set -e
-target="$1"
-
-if [ -z "${target}" ]; then
-    exit 1
-fi
-images_dir=build/tmp/deploy/images
-
-if [ -z "${LAVA_SSH_USER}" ] || [ -z "${LAVA_SSH_HOST}" ]  || [ -z "${LAVA_SSH_PORT}" ]; then
-    echo "Lava environment not available or incomplete - do not deploy"
-    exit 0
-fi
-
-lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
-
-isar_base_name="${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${target}"
-lava_deploy_dir="${LAVA_DEPLOY_DIR:-/var/lib/lava/artifacts}"
-deploy_dir="${lava_deploy_dir}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
-lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
-if [ -n "${CI_PIPELINE_ID}" ]; then
-    ssh -p ${LAVA_SSH_PORT} ${lava_identity} ${lava_ssh_destination} 'install -d -m 755 "'${deploy_dir}'"'
-fi
-#KERNEL
-scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-vmlinuz \
-    ${lava_ssh_destination}:${deploy_dir}
-# INITRD
-scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-initrd.img \
-    ${lava_ssh_destination}:${deploy_dir}
-# ROOTFS
-if [ ! -e ${images_dir}/${target}/${isar_base_name}.*.gz  ]; then
-    gzip ${images_dir}/${target}/${isar_base_name}.*
-fi
-scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}.* \
-    ${lava_ssh_destination}:${deploy_dir}
-# DTB
-dtb="${images_dir}/${target}/*.dtb"
-if [ -e ${dtb} ]; then
-    scp -P ${LAVA_SSH_PORT} ${lava_identity} ${dtb} ${lava_ssh_destination}:${deploy_dir}
-fi
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 13fac31..b1b51bb 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -15,36 +15,10 @@ if [ -z "${TARGET}" ]; then
     exit 1
 fi
 
-if [ -n "${LAVA_SSH_HOST}" ]; then
-    echo "use SSH to connect to lava server"
-    # private master with ssh forward access
-    lava_master_port="${LAVA_MASTER_PORT:-28080}"
-    lava_ssh_port="-p ${LAVA_SSH_PORT:-22}"
-    lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
-    lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
-    # open connection for ssh port forwarding
-    ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
-    ssh_pid="$!"
-    # wait for connection
-    interval=1
-    timeout=60
-    until ss -tulw | grep -q "${lava_master_port}"
-    do
-        if [ ${timeout} -le 0 ]; then
-            echo "could not open connection to LAVA Master"
-            exit 1
-        fi
-        sleep ${interval}
-        timeout=$(( timeout - interval ))
-    done
-    artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
-    lava_master_uri="${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}"
-else
-    # S3 artifacts
-    artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
-    # public master
-    lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
-fi
+# S3 artifacts
+artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
+# public master
+lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
 
 echo "connect to lava server: ${lava_master_uri}"
 
@@ -79,9 +53,6 @@ curl --silent "${url}" -o report.xml
 # change return code to generate a error in gitlab-ci if a test is failed
 number_of_fails="$(grep -o "errors=\"[0-9]*\"" report.xml | head -1  | cut -d\" -f2 )"
 
-if [ -n "${ssh_pid}" ];then
-   kill "${ssh_pid}"
-fi
 if [ "${number_of_fails}" -gt "0" ]; then
     exit 1
 fi
-- 
2.20.1

[xenomai-images][PATCH 3/3] ci/gitlab-ci-base: use rules

[Q. Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

Replace `only` and `except` with rules as
'`only` and `except` are no longer actively developed. `rules`
is the preferred keyword to control ...'[1]

[1]: https://docs.gitlab.com/ee/ci/yaml/#only--except

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 ci/gitlab-ci-base.yml | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
index 84bbdd4..535b972 100644
--- a/ci/gitlab-ci-base.yml
+++ b/ci/gitlab-ci-base.yml
@@ -61,9 +61,8 @@ default:
       junit:
         - report.xml
     expire_in: 1 week
-  only:
-    variables:
-      - $LAVA_TESTS_ENABLED == "true"
+  rules:
+    - if: '$LAVA_TESTS_ENABLED == "true"'
 
 .build:qemu-amd64:
   extends: .build
-- 
2.20.1

Re: [xenomai-images][PATCH 1/3] ci: Remove artifact deployment

[Jan Kiszka]

On 19.07.21 16:29, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> 
> We will use AWS S3 deployment for LAVA CI Artifacts. Therefore
> remove the Deployment via gitlab artifacts as the LAVA Lab is now public
> available.
> 
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
>  .gitlab-ci-artifacts.yml            | 13 -----------
>  .gitlab-ci.yml                      | 20 ++++++++++++++++-
>  ci/artifacts.yml                    | 20 -----------------
>  ci/child_pipelines_artifacts.yml    | 34 -----------------------------
>  ci/child_pipelines_no_artifacts.yml | 34 -----------------------------
>  ci/gitlab-ci-base.yml               |  5 ++---
>  ci/no-artifacts.yml                 | 13 -----------
>  scripts/get-build-id.sh             | 34 -----------------------------
>  scripts/run-lava-tests.sh           |  8 ++-----
>  9 files changed, 23 insertions(+), 158 deletions(-)
>  delete mode 100644 .gitlab-ci-artifacts.yml
>  delete mode 100644 ci/artifacts.yml
>  delete mode 100644 ci/child_pipelines_artifacts.yml
>  delete mode 100644 ci/child_pipelines_no_artifacts.yml
>  delete mode 100644 ci/no-artifacts.yml
>  delete mode 100755 scripts/get-build-id.sh
> 
> diff --git a/.gitlab-ci-artifacts.yml b/.gitlab-ci-artifacts.yml
> deleted file mode 100644
> index 85345b8..0000000
> --- a/.gitlab-ci-artifacts.yml
> +++ /dev/null
> @@ -1,13 +0,0 @@
> -#
> -# Xenomai Real-Time System
> -#
> -# Copyright (c) Siemens AG, 2019 - 2021
> -#
> -# Authors:
> -#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> -#
> -# SPDX-License-Identifier: MIT
> -#
> -
> -include:
> -  - local: '/ci/child_pipelines_artifacts.yml'
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index a808859..b5a3a35 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -10,4 +10,22 @@
>  #
>  
>  include:
> -  - local: '/ci/child_pipelines_no_artifacts.yml'
> +  - local: '/ci/child_pipelines_common.yml'

Is common still needed? Or should this be migrated here as well?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [xenomai-images][PATCH 2/3] ci: Remove deploy with ssh

[Jan Kiszka]

On 19.07.21 16:29, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> 
> We will use AWS S3 deployment for LAVA Build Artifacts. Therefore
> remove the Deployment via ssh as the LAVA Lab is now public
> available.
> 
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
>  ci/gitlab-ci-base.yml         |  1 -
>  scripts/deploy_for_testing.sh | 51 -----------------------------------
>  scripts/run-lava-tests.sh     | 37 +++----------------------
>  3 files changed, 4 insertions(+), 85 deletions(-)
>  delete mode 100755 scripts/deploy_for_testing.sh
> 
> diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
> index 362a636..84bbdd4 100644
> --- a/ci/gitlab-ci-base.yml
> +++ b/ci/gitlab-ci-base.yml
> @@ -45,7 +45,6 @@ default:
>    script:
>      - echo "Building kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}"
>      - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
> -    - if [ -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
>      - if [ -n "${USE_S3_BUCKET}" ]; then scripts/deploy_to_aws.sh ${TARGET}; fi
>  
>  .test:
> diff --git a/scripts/deploy_for_testing.sh b/scripts/deploy_for_testing.sh
> deleted file mode 100755
> index 3878e7d..0000000
> --- a/scripts/deploy_for_testing.sh
> +++ /dev/null
> @@ -1,51 +0,0 @@
> -#!/bin/sh
> -
> -#
> -# Xenomai Real-Time System
> -#
> -# Copyright (c) Siemens AG, 2019-2020
> -#
> -# Authors:
> -#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> -#
> -# SPDX-License-Identifier: MIT
> -#
> -set -e
> -target="$1"
> -
> -if [ -z "${target}" ]; then
> -    exit 1
> -fi
> -images_dir=build/tmp/deploy/images
> -
> -if [ -z "${LAVA_SSH_USER}" ] || [ -z "${LAVA_SSH_HOST}" ]  || [ -z "${LAVA_SSH_PORT}" ]; then
> -    echo "Lava environment not available or incomplete - do not deploy"
> -    exit 0
> -fi
> -
> -lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
> -
> -isar_base_name="${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${target}"
> -lava_deploy_dir="${LAVA_DEPLOY_DIR:-/var/lib/lava/artifacts}"
> -deploy_dir="${lava_deploy_dir}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
> -lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
> -if [ -n "${CI_PIPELINE_ID}" ]; then
> -    ssh -p ${LAVA_SSH_PORT} ${lava_identity} ${lava_ssh_destination} 'install -d -m 755 "'${deploy_dir}'"'
> -fi
> -#KERNEL
> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-vmlinuz \
> -    ${lava_ssh_destination}:${deploy_dir}
> -# INITRD
> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-initrd.img \
> -    ${lava_ssh_destination}:${deploy_dir}
> -# ROOTFS
> -if [ ! -e ${images_dir}/${target}/${isar_base_name}.*.gz  ]; then
> -    gzip ${images_dir}/${target}/${isar_base_name}.*
> -fi
> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}.* \
> -    ${lava_ssh_destination}:${deploy_dir}
> -# DTB
> -dtb="${images_dir}/${target}/*.dtb"
> -if [ -e ${dtb} ]; then
> -    scp -P ${LAVA_SSH_PORT} ${lava_identity} ${dtb} ${lava_ssh_destination}:${deploy_dir}
> -fi
> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
> index 13fac31..b1b51bb 100755
> --- a/scripts/run-lava-tests.sh
> +++ b/scripts/run-lava-tests.sh
> @@ -15,36 +15,10 @@ if [ -z "${TARGET}" ]; then
>      exit 1
>  fi
>  
> -if [ -n "${LAVA_SSH_HOST}" ]; then
> -    echo "use SSH to connect to lava server"
> -    # private master with ssh forward access
> -    lava_master_port="${LAVA_MASTER_PORT:-28080}"
> -    lava_ssh_port="-p ${LAVA_SSH_PORT:-22}"
> -    lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
> -    lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
> -    # open connection for ssh port forwarding
> -    ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
> -    ssh_pid="$!"
> -    # wait for connection
> -    interval=1
> -    timeout=60
> -    until ss -tulw | grep -q "${lava_master_port}"
> -    do
> -        if [ ${timeout} -le 0 ]; then
> -            echo "could not open connection to LAVA Master"
> -            exit 1
> -        fi
> -        sleep ${interval}
> -        timeout=$(( timeout - interval ))
> -    done
> -    artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
> -    lava_master_uri="${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}"
> -else
> -    # S3 artifacts
> -    artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
> -    # public master
> -    lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
> -fi
> +# S3 artifacts
> +artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
> +# public master
> +lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
>  
>  echo "connect to lava server: ${lava_master_uri}"
>  
> @@ -79,9 +53,6 @@ curl --silent "${url}" -o report.xml
>  # change return code to generate a error in gitlab-ci if a test is failed
>  number_of_fails="$(grep -o "errors=\"[0-9]*\"" report.xml | head -1  | cut -d\" -f2 )"
>  
> -if [ -n "${ssh_pid}" ];then
> -   kill "${ssh_pid}"
> -fi
>  if [ "${number_of_fails}" -gt "0" ]; then
>      exit 1
>  fi
> 

Maybe do another "grep -i ssh" after this, if anything related to ssh
remains. I think add-lava-ssh-config is still in.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [xenomai-images][PATCH 2/3] ci: Remove deploy with ssh

[Gylstorff Quirin]

On 7/19/21 4:51 PM, Jan Kiszka wrote:
> On 19.07.21 16:29, Q. Gylstorff wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> We will use AWS S3 deployment for LAVA Build Artifacts. Therefore
>> remove the Deployment via ssh as the LAVA Lab is now public
>> available.
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>>   ci/gitlab-ci-base.yml         |  1 -
>>   scripts/deploy_for_testing.sh | 51 -----------------------------------
>>   scripts/run-lava-tests.sh     | 37 +++----------------------
>>   3 files changed, 4 insertions(+), 85 deletions(-)
>>   delete mode 100755 scripts/deploy_for_testing.sh
>>
>> diff --git a/ci/gitlab-ci-base.yml b/ci/gitlab-ci-base.yml
>> index 362a636..84bbdd4 100644
>> --- a/ci/gitlab-ci-base.yml
>> +++ b/ci/gitlab-ci-base.yml
>> @@ -45,7 +45,6 @@ default:
>>     script:
>>       - echo "Building kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}"
>>       - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
>> -    - if [ -z "${USE_S3_BUCKET}" ]; then scripts/deploy_for_testing.sh ${TARGET}; fi
>>       - if [ -n "${USE_S3_BUCKET}" ]; then scripts/deploy_to_aws.sh ${TARGET}; fi
>>   
>>   .test:
>> diff --git a/scripts/deploy_for_testing.sh b/scripts/deploy_for_testing.sh
>> deleted file mode 100755
>> index 3878e7d..0000000
>> --- a/scripts/deploy_for_testing.sh
>> +++ /dev/null
>> @@ -1,51 +0,0 @@
>> -#!/bin/sh
>> -
>> -#
>> -# Xenomai Real-Time System
>> -#
>> -# Copyright (c) Siemens AG, 2019-2020
>> -#
>> -# Authors:
>> -#  Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> -#
>> -# SPDX-License-Identifier: MIT
>> -#
>> -set -e
>> -target="$1"
>> -
>> -if [ -z "${target}" ]; then
>> -    exit 1
>> -fi
>> -images_dir=build/tmp/deploy/images
>> -
>> -if [ -z "${LAVA_SSH_USER}" ] || [ -z "${LAVA_SSH_HOST}" ]  || [ -z "${LAVA_SSH_PORT}" ]; then
>> -    echo "Lava environment not available or incomplete - do not deploy"
>> -    exit 0
>> -fi
>> -
>> -lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>> -
>> -isar_base_name="${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${target}"
>> -lava_deploy_dir="${LAVA_DEPLOY_DIR:-/var/lib/lava/artifacts}"
>> -deploy_dir="${lava_deploy_dir}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
>> -lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>> -if [ -n "${CI_PIPELINE_ID}" ]; then
>> -    ssh -p ${LAVA_SSH_PORT} ${lava_identity} ${lava_ssh_destination} 'install -d -m 755 "'${deploy_dir}'"'
>> -fi
>> -#KERNEL
>> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-vmlinuz \
>> -    ${lava_ssh_destination}:${deploy_dir}
>> -# INITRD
>> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-initrd.img \
>> -    ${lava_ssh_destination}:${deploy_dir}
>> -# ROOTFS
>> -if [ ! -e ${images_dir}/${target}/${isar_base_name}.*.gz  ]; then
>> -    gzip ${images_dir}/${target}/${isar_base_name}.*
>> -fi
>> -scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}.* \
>> -    ${lava_ssh_destination}:${deploy_dir}
>> -# DTB
>> -dtb="${images_dir}/${target}/*.dtb"
>> -if [ -e ${dtb} ]; then
>> -    scp -P ${LAVA_SSH_PORT} ${lava_identity} ${dtb} ${lava_ssh_destination}:${deploy_dir}
>> -fi
>> diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
>> index 13fac31..b1b51bb 100755
>> --- a/scripts/run-lava-tests.sh
>> +++ b/scripts/run-lava-tests.sh
>> @@ -15,36 +15,10 @@ if [ -z "${TARGET}" ]; then
>>       exit 1
>>   fi
>>   
>> -if [ -n "${LAVA_SSH_HOST}" ]; then
>> -    echo "use SSH to connect to lava server"
>> -    # private master with ssh forward access
>> -    lava_master_port="${LAVA_MASTER_PORT:-28080}"
>> -    lava_ssh_port="-p ${LAVA_SSH_PORT:-22}"
>> -    lava_identity="-i ${LAVA_SSH_KEY_PATH:-~/.ssh/lava_id_rsa}"
>> -    lava_ssh_destination="${LAVA_SSH_USER}@${LAVA_SSH_HOST}"
>> -    # open connection for ssh port forwarding
>> -    ssh -N ${lava_ssh_port} ${lava_identity} -o 'LocalForward localhost:'${lava_master_port}' localhost:80' ${lava_ssh_destination} &
>> -    ssh_pid="$!"
>> -    # wait for connection
>> -    interval=1
>> -    timeout=60
>> -    until ss -tulw | grep -q "${lava_master_port}"
>> -    do
>> -        if [ ${timeout} -le 0 ]; then
>> -            echo "could not open connection to LAVA Master"
>> -            exit 1
>> -        fi
>> -        sleep ${interval}
>> -        timeout=$(( timeout - interval ))
>> -    done
>> -    artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
>> -    lava_master_uri="${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}"
>> -else
>> -    # S3 artifacts
>> -    artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
>> -    # public master
>> -    lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
>> -fi
>> +# S3 artifacts
>> +artifact_url="${LAVA_ARTIFACTS_URL:-https://xenomai-images-artifacts.s3.eu-central-1.amazonaws.com/artifacts}"
>> +# public master
>> +lava_master_uri="${LAVA_MASTER_URL:-https://lava.xenomai.org}"
>>   
>>   echo "connect to lava server: ${lava_master_uri}"
>>   
>> @@ -79,9 +53,6 @@ curl --silent "${url}" -o report.xml
>>   # change return code to generate a error in gitlab-ci if a test is failed
>>   number_of_fails="$(grep -o "errors=\"[0-9]*\"" report.xml | head -1  | cut -d\" -f2 )"
>>   
>> -if [ -n "${ssh_pid}" ];then
>> -   kill "${ssh_pid}"
>> -fi
>>   if [ "${number_of_fails}" -gt "0" ]; then
>>       exit 1
>>   fi
>>
> 
> Maybe do another "grep -i ssh" after this, if anything related to ssh
> remains. I think add-lava-ssh-config is still in.

I also forget to adapt the README - send a V2


> 
> Jan
>

Quirin