* [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com
@ 2020-04-21 8:31 Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com Q. Gylstorff
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Q. Gylstorff @ 2020-04-21 8:31 UTC (permalink / raw)
To: xenomai, Jan.Kiszka
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Add the configration to run the ci builds on gitlab.com or on-premise setups with
gitlab artifacts enabled. Each completed pipeline creates around 1.5 GB aritfacts.
An active pipeline can be seen at https://gitlab.com/Quirin.Gy/xenomai-images-test/pipelines.
Quirin Gylstorff (4):
ci: Build and test from gitlab.com
ci: Add option to compress ext4-images.
test/README: Update
scripts/run-lava-tests: Add LAVA_MASTER_URL
.ci/gitlab-ci-base.yml | 156 ++++++++++++++++++
.ci/gitlab-com.yml | 26 +++
.ci/siemens-ci.yml | 22 +++
.gitlab-ci.yml | 145 +---------------
.../0001-classes-ext4.img-Compress.patch | 31 ++++
opt-ext4-gz.yml | 24 +++
scripts/deploy_for_testing.sh | 4 +-
scripts/get-build-id.sh | 29 ++++
scripts/install-lavacli.sh | 5 +-
scripts/run-lava-tests.sh | 10 +-
tests/README.md | 60 +++++--
...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
tests/jobs/xenomai-hikey.yml | 4 +-
tests/jobs/xenomai-ipc227e.yml | 6 +-
tests/jobs/xenomai-minnow.yml | 6 +-
tests/jobs/xenomai-qemu-amd64.yml | 6 +-
tests/jobs/xenomai-qemu-arm64.yml | 6 +-
tests/jobs/xenomai-qemu-armhf.yml | 6 +-
...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
19 files changed, 369 insertions(+), 187 deletions(-)
create mode 100644 .ci/gitlab-ci-base.yml
create mode 100644 .ci/gitlab-com.yml
create mode 100644 .ci/siemens-ci.yml
create mode 100644 isar-patches/0001-classes-ext4.img-Compress.patch
create mode 100644 opt-ext4-gz.yml
create mode 100755 scripts/get-build-id.sh
rename tests/jobs/{xenomai-beaglebone.yml => xenomai-beagle-bone-black.yml} (91%)
rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml} (86%)
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com
2020-04-21 8:31 [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com Q. Gylstorff
@ 2020-04-21 8:31 ` Q. Gylstorff
2020-04-21 9:09 ` Jan Kiszka
2020-04-21 8:31 ` [Xenomai-images PATCH 2/4] ci: Add option to compress ext4-images Q. Gylstorff
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Q. Gylstorff @ 2020-04-21 8:31 UTC (permalink / raw)
To: xenomai, Jan.Kiszka
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Adapt scripts and gitlab-ci.yml to build on public available
gitlab infrastructure.
As the internal gitlab system has no artifact storage and
it is not possible to deactivate arifact upload with a variable
split the gitlab-ci.yml. The .gitlab-ci.yml in the root points
to .ci/gitlab-com.yml.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
.ci/gitlab-ci-base.yml | 153 ++++++++++++++++++
.ci/gitlab-com.yml | 26 +++
.ci/siemens-ci.yml | 22 +++
.gitlab-ci.yml | 145 +----------------
scripts/deploy_for_testing.sh | 4 +-
scripts/get-build-id.sh | 29 ++++
scripts/install-lavacli.sh | 5 +-
scripts/run-lava-tests.sh | 8 +-
...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
tests/jobs/xenomai-hikey.yml | 4 +-
tests/jobs/xenomai-ipc227e.yml | 6 +-
tests/jobs/xenomai-minnow.yml | 6 +-
tests/jobs/xenomai-qemu-amd64.yml | 6 +-
tests/jobs/xenomai-qemu-arm64.yml | 6 +-
tests/jobs/xenomai-qemu-armhf.yml | 6 +-
...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
16 files changed, 265 insertions(+), 171 deletions(-)
create mode 100644 .ci/gitlab-ci-base.yml
create mode 100644 .ci/gitlab-com.yml
create mode 100644 .ci/siemens-ci.yml
create mode 100755 scripts/get-build-id.sh
rename tests/jobs/{xenomai-beaglebone.yml => xenomai-beagle-bone-black.yml} (91%)
rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml} (86%)
diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
new file mode 100644
index 0000000..e8d259f
--- /dev/null
+++ b/.ci/gitlab-ci-base.yml
@@ -0,0 +1,153 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2019 - 2020
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+stages:
+ - build
+ - lava-test
+
+variables:
+ GIT_STRATEGY: clone
+ http_proxy: "$HTTP_PROXY"
+ https_proxy: "$HTTPS_PROXY"
+ ftp_proxy: "$FTP_PROXY"
+ no_proxy: "$NO_PROXY"
+ XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
+ LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
+ ISAR_IMAGE: demo-image
+ ISAR_DISTRIBUTION: xenomai-demo
+
+default:
+ image: kasproject/kas-isar:latest
+
+# add lavacli to the container and install the ssh keys
+# for the test infrastructure
+.add-lava-ssh-config:
+ before_script:
+ - mkdir -p -m=700 ~/.ssh
+ - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat - PROXY:$(echo $https_proxy | sed 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600 ~/.ssh/config; fi;
+ - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa && chmod 600 ~/.ssh/lava_id_rsa
+ - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
+
+.test:
+ extends: .add-lava-ssh-config
+ stage: lava-test
+ script:
+ - scripts/install-lavacli.sh
+ - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
+ - scripts/run-lava-tests.sh ${TARGET}
+ variables:
+ BUILD_JOB_NAME: "build:board-${TARGET}${TARGET_EXTENSION}"
+ only:
+ variables:
+ - $LAVA_SSH_USER
+
+
+build:board-qemu-amd64:
+ extends: .build
+ variables:
+ TARGET: qemu-amd64
+
+lava-test:qemu-amd64:
+ needs: [ "build:board-qemu-amd64" ]
+ dependencies: [ "build:board-qemu-amd64" ]
+ extends: .test
+ variables:
+ TARGET: qemu-amd64
+
+build:board-qemu-armhf:
+ extends: .build
+ variables:
+ TARGET: qemu-armhf
+
+lava-test:qemu-armhf:
+ needs: [ "build:board-qemu-armhf" ]
+ extends: .test
+ variables:
+ TARGET: qemu-armhf
+
+build:board-qemu-arm64:
+ extends: .build
+ variables:
+ TARGET: qemu-arm64
+
+lava-test:qemu-arm64:
+ needs: [ "build:board-qemu-arm64" ]
+ extends: .test
+ variables:
+ TARGET: qemu-arm64
+
+build:board-hikey:
+ extends: .build
+ variables:
+ TARGET: hikey
+ BUILD_OPTIONS: ":opt-lava-test.yml"
+
+lava-test:board-hikey:
+ needs: [ "build:board-hikey" ]
+ extends: .test
+ variables:
+ TARGET: hikey
+
+build:board-beagle-bone-black:
+ extends: .build
+ variables:
+ TARGET: beagle-bone-black
+ BUILD_OPTIONS: ":opt-lava-test.yml"
+
+lava-test:board-beagle-bone-black:
+ needs: [ "build:board-beagle-bone-black" ]
+ extends: .test
+ variables:
+ TARGET: beagle-bone-black
+
+build:board-x86-64-efi:
+ extends: .build
+ variables:
+ TARGET: x86-64-efi
+ BUILD_OPTIONS: ":opt-lava-test.yml"
+
+lava-test:board-x86-64-efi:
+ needs: [ "build:board-x86-64-efi" ]
+ extends: .test
+ variables:
+ TARGET: x86-64-efi
+
+build:board-beagle-bone-black:xenomai-3.0.x:
+ extends: .build
+ variables:
+ TARGET: beagle-bone-black
+ XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
+ BUILD_OPTIONS: ":opt-lava-test.yml"
+ DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
+
+lava-test:board-beagle-bone-black:xenomai-3.0.x:
+ needs: [ "build:board-beagle-bone-black:xenomai-3.0.x" ]
+ extends: .test
+ variables:
+ TARGET: beagle-bone-black
+ DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
+ TARGET_EXTENSION: ":xenomai-3.0.x"
+
+build:board-x86-64-efi:xenomai-3.0.x:
+ extends: .build
+ variables:
+ TARGET: x86-64-efi
+ XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
+ BUILD_OPTIONS: ":opt-lava-test.yml"
+ DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
+
+lava-test:board-x86-64-efi:xenomai-3.0.x:
+ needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
+ extends: .test
+ variables:
+ TARGET: x86-64-efi
+ DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
+ TARGET_EXTENSION: ":xenomai-3.0.x"
+
diff --git a/.ci/gitlab-com.yml b/.ci/gitlab-com.yml
new file mode 100644
index 0000000..0abb3fb
--- /dev/null
+++ b/.ci/gitlab-com.yml
@@ -0,0 +1,26 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2019 - 2020
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+.build:
+ extends: .add-lava-ssh-config
+ stage: build
+ script:
+ - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
+ artifacts:
+ paths:
+ - build/tmp/deploy/images/${TARGET}/
+ expire_in: 1 week
+ only:
+ variables:
+ - $USE_GITLAB_ARTIFACTS
+include:
+ - local: '/.ci/gitlab-ci-base.yml'
+
+
diff --git a/.ci/siemens-ci.yml b/.ci/siemens-ci.yml
new file mode 100644
index 0000000..58b5fcd
--- /dev/null
+++ b/.ci/siemens-ci.yml
@@ -0,0 +1,22 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2019 - 2020
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+.build:
+ extends: .add-lava-ssh-config
+ stage: build
+ script:
+ - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
+ - scripts/deploy_for_testing.sh ${TARGET}
+ except:
+ variables:
+ - $USE_GITLAB_ARTIFACTS
+include:
+ - local: '/.ci/gitlab-ci-base.yml'
+
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 45fccd0..0eb3abc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -8,146 +8,5 @@
#
# SPDX-License-Identifier: MIT
#
-stages:
- - build
- - lava-test
-
-variables:
- GIT_STRATEGY: clone
- http_proxy: "$HTTP_PROXY"
- https_proxy: "$HTTPS_PROXY"
- ftp_proxy: "$FTP_PROXY"
- no_proxy: "$NO_PROXY"
- XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
- LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
- ISAR_IMAGE: demo-image
- ISAR_DISTRIBUTION: xenomai-demo
-
-default:
- image: kasproject/kas-isar:latest
-
-# add lavacli to the container and install the ssh keys
-# for the test infrastructure
-.add-lava-ssh-config:
- before_script:
- - mkdir -p -m=700 ~/.ssh
- - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat - PROXY:$(echo $https_proxy | sed 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600 ~/.ssh/config; fi;
- - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa && chmod 600 ~/.ssh/lava_id_rsa
- - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
-
-.build:
- extends: .add-lava-ssh-config
- stage: build
- script:
- - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
- - scripts/deploy_for_testing.sh ${TARGET}
-
-.test:
- extends: .add-lava-ssh-config
- stage: lava-test
- script:
- - scripts/install-lavacli.sh
- - scripts/run-lava-tests.sh ${TARGET}
- only:
- variables:
- - $LAVA_SSH_USER
-
-
-build:qemu-amd64:
- extends: .build
- variables:
- TARGET: qemu-amd64
-
-lava-test:qemu-amd64:
- needs: [ "build:qemu-amd64" ]
- extends: .test
- variables:
- TARGET: qemu-amd64
-
-build:qemu-armhf:
- extends: .build
- variables:
- TARGET: qemu-armhf
-
-lava-test:qemu-armhf:
- needs: [ "build:qemu-armhf" ]
- extends: .test
- variables:
- TARGET: qemu-armhf
-
-build:qemu-arm64:
- extends: .build
- variables:
- TARGET: qemu-arm64
-
-lava-test:qemu-arm64:
- needs: [ "build:qemu-arm64" ]
- extends: .test
- variables:
- TARGET: qemu-arm64
-
-build:board-hikey:
- extends: .build
- variables:
- TARGET: hikey
- BUILD_OPTIONS: ":opt-lava-test.yml"
-lava-test:board-hikey:
- needs: [ "build:board-hikey" ]
- extends: .test
- variables:
- TARGET: hikey
-
-build:board-beaglebone:
- extends: .build
- variables:
- TARGET: beagle-bone-black
- BUILD_OPTIONS: ":opt-lava-test.yml"
-
-
-lava-test:board-beaglebone:
- needs: [ "build:board-beaglebone" ]
- extends: .test
- variables:
- TARGET: beaglebone
-
-build:board-x86-64-efi:
- extends: .build
- variables:
- TARGET: x86-64-efi
- BUILD_OPTIONS: ":opt-lava-test.yml"
-
-lava-test:board-x86-64-efi:
- needs: [ "build:board-x86-64-efi" ]
- extends: .test
- variables:
- TARGET: x86-64
-
-build:board-beaglebone:xenomai-3.0.x:
- extends: .build
- variables:
- TARGET: beagle-bone-black
- XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
- BUILD_OPTIONS: ":opt-lava-test.yml"
- DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
-
-lava-test:board-beaglebone:xenomai-3.0.x:
- needs: [ "build:board-beaglebone:xenomai-3.0.x" ]
- extends: .test
- variables:
- TARGET: beaglebone
- DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
-
-build:board-x86-64-efi:xenomai-3.0.x:
- extends: .build
- variables:
- TARGET: x86-64-efi
- XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
- BUILD_OPTIONS: ":opt-lava-test.yml"
- DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
-
-lava-test:board-x86-64-efi:xenomai-3.0.x:
- needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
- extends: .test
- variables:
- TARGET: ipc227e
- DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
+include:
+ - local: '/.ci/gitlab-com.yml'
diff --git a/scripts/deploy_for_testing.sh b/scripts/deploy_for_testing.sh
index b3d565f..3878e7d 100755
--- a/scripts/deploy_for_testing.sh
+++ b/scripts/deploy_for_testing.sh
@@ -14,7 +14,7 @@ set -e
target="$1"
if [ -z "${target}" ]; then
- exit -1
+ exit 1
fi
images_dir=build/tmp/deploy/images
@@ -39,7 +39,7 @@ scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_nam
scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}-initrd.img \
${lava_ssh_destination}:${deploy_dir}
# ROOTFS
-if [ -n ${images_dir}/${target}/${isar_base_name}.*.gz ]; then
+if [ ! -e ${images_dir}/${target}/${isar_base_name}.*.gz ]; then
gzip ${images_dir}/${target}/${isar_base_name}.*
fi
scp -P ${LAVA_SSH_PORT} ${lava_identity} ${images_dir}/${target}/${isar_base_name}.* \
diff --git a/scripts/get-build-id.sh b/scripts/get-build-id.sh
new file mode 100755
index 0000000..ace70c6
--- /dev/null
+++ b/scripts/get-build-id.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2020
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+name=$1
+if [ -z "${name}" ]; then
+ echo "no build name was given"
+ exit -1
+ fi
+index=0
+JSON=$(curl -s --header "PRIVATE-TOKEN: ${API_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs?scope[]=success" )
+while [ ${index} -le ${MAX_TEST:-16} ]
+do
+ build_name=$(echo "${JSON}" | tr '\r\n' ' ' | jq -r ".[${index}].name")
+ if [ "$build_name" = "${name}" ]; then
+ break
+ fi
+ index=$((index+1))
+done
+id=$(echo "${JSON}" | tr '\r\n' ' ' | jq -r ".[${index}].id")
+echo $id
diff --git a/scripts/install-lavacli.sh b/scripts/install-lavacli.sh
index f31a394..7f7a052 100755
--- a/scripts/install-lavacli.sh
+++ b/scripts/install-lavacli.sh
@@ -13,9 +13,10 @@
# of lavacli. The apt version from stretch-backports cannot communicate
# with the lava master on Debian 10.
os_version_id=$(sed -nr 's/VERSION_ID="([0-9]+)"/\1/p' /etc/os-release)
+sudo apt-get update
+sudo apt-get install -y jq
if [ "${os_version_id}" -ge "10" ]; then
- sudo apt update
- sudo apt install -y lavacli
+ sudo apt-get install -y lavacli
else
sudo pip3 install wheel
sudo pip3 install lavacli
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 1f502c4..8566493 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -42,8 +42,12 @@ lava_master_uri=http://localhost:${lava_master_port}
# connect to lava master
lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
#generate lava job description from template
-artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
-DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
+if [ -n "${USE_GITLAB_ARTIFACTS}" ]; then
+ DEPLOY_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${BUILD_JOB_ID}/artifacts/build/tmp/deploy/images/${TARGET}"
+else
+ artifact_url="${LAVA_ARTIFACTS_URL:-http://localhost/artifacts}"
+ DEPLOY_URL="${artifact_url}/${CI_PIPELINE_ID}/${DEPLOY_DIR_EXTENSION}"
+fi
job_template_path="${JOB_TEMPLATE_PATH:-tests/jobs/xenomai}"
tmp_dir=$(mktemp -d)
template=${tmp_dir}/job_${TARGET}_${CI_PIPELINE_ID}.yml
diff --git a/tests/jobs/xenomai-beaglebone.yml b/tests/jobs/xenomai-beagle-bone-black.yml
similarity index 91%
rename from tests/jobs/xenomai-beaglebone.yml
rename to tests/jobs/xenomai-beagle-bone-black.yml
index ceb4d0d..91cb138 100644
--- a/tests/jobs/xenomai-beaglebone.yml
+++ b/tests/jobs/xenomai-beagle-bone-black.yml
@@ -32,12 +32,12 @@ actions:
minutes: 5
to: tftp
kernel:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-beagle-bone-black-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
type: zimage
dtb:
url: ${DEPLOY_URL}/am335x-boneblack.dtb
nfsrootfs:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-beagle-bone-black.tar.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.tar.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-hikey.yml b/tests/jobs/xenomai-hikey.yml
index 3e52212..b33cef6 100644
--- a/tests/jobs/xenomai-hikey.yml
+++ b/tests/jobs/xenomai-hikey.yml
@@ -31,12 +31,12 @@ actions:
minutes: 30
to: tftp
kernel:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-hikey-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
type: image
dtb:
url: ${DEPLOY_URL}/hi6220-hikey.dtb
nfsrootfs:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-hikey.tar.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.tar.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-ipc227e.yml b/tests/jobs/xenomai-ipc227e.yml
index e5a2f35..d283c00 100644
--- a/tests/jobs/xenomai-ipc227e.yml
+++ b/tests/jobs/xenomai-ipc227e.yml
@@ -32,13 +32,13 @@ actions:
minutes: 5
to: tftp
kernel:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
type: bzimage
ramdisk:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
compression: gz
nfsrootfs:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi.tar.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.tar.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-minnow.yml b/tests/jobs/xenomai-minnow.yml
index 27a0f81..82c1468 100644
--- a/tests/jobs/xenomai-minnow.yml
+++ b/tests/jobs/xenomai-minnow.yml
@@ -31,13 +31,13 @@ actions:
minutes: 5
to: tftp
kernel:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
type: bzimage
ramdisk:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
compression: gz
nfsrootfs:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi.tar.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.tar.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-qemu-amd64.yml b/tests/jobs/xenomai-qemu-amd64.yml
index 33ff1b2..68d849f 100644
--- a/tests/jobs/xenomai-qemu-amd64.yml
+++ b/tests/jobs/xenomai-qemu-amd64.yml
@@ -47,13 +47,13 @@ actions:
images:
kernel:
image_arg: -kernel {kernel} -append "root=/dev/sdb console=ttyS0,115200 console=tty0"
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-amd64-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
initrd:
image_arg: -initrd {initrd}
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-amd64-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
rootfs:
image_arg: -drive file={rootfs},discard=unmap,if=none,id=disk,format=raw
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-amd64.ext4.img.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.ext4.img.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-qemu-arm64.yml b/tests/jobs/xenomai-qemu-arm64.yml
index 2a98889..cd0ca9f 100644
--- a/tests/jobs/xenomai-qemu-arm64.yml
+++ b/tests/jobs/xenomai-qemu-arm64.yml
@@ -51,13 +51,13 @@ actions:
images:
kernel:
image_arg: -kernel {kernel} -append "root=/dev/vda"
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-arm64-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
initrd:
image_arg: -initrd {initrd}
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-arm64-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
rootfs:
image_arg: -drive file={rootfs},discard=unmap,if=none,id=disk,format=raw
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-arm64.ext4.img.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.ext4.img.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-qemu-armhf.yml b/tests/jobs/xenomai-qemu-armhf.yml
index 2949cd2..18e3bf8 100644
--- a/tests/jobs/xenomai-qemu-armhf.yml
+++ b/tests/jobs/xenomai-qemu-armhf.yml
@@ -47,13 +47,13 @@ actions:
images:
kernel:
image_arg: -kernel {kernel} -append "root=/dev/vdb"
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-armhf-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
initrd:
image_arg: -initrd {initrd}
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-armhf-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
rootfs:
image_arg: -drive file={rootfs},discard=unmap,if=none,id=disk,format=raw
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-qemu-armhf.ext4.img.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.ext4.img.gz
compression: gz
# BOOT_BLOCK
- boot:
diff --git a/tests/jobs/xenomai-x86-64.yml b/tests/jobs/xenomai-x86-64-efi.yml
similarity index 86%
rename from tests/jobs/xenomai-x86-64.yml
rename to tests/jobs/xenomai-x86-64-efi.yml
index c7424c0..2471cce 100644
--- a/tests/jobs/xenomai-x86-64.yml
+++ b/tests/jobs/xenomai-x86-64-efi.yml
@@ -32,13 +32,13 @@ actions:
minutes: 5
to: tftp
kernel:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-vmlinuz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-vmlinuz
type: bzimage
ramdisk:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi-initrd.img
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}-initrd.img
compression: gz
nfsrootfs:
- url: ${DEPLOY_URL}/demo-image-xenomai-demo-x86-64-efi.tar.gz
+ url: ${DEPLOY_URL}/${ISAR_IMAGE}-${ISAR_DISTRIBUTION}-${TARGET}.tar.gz
compression: gz
# BOOT_BLOCK
- boot:
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Xenomai-images PATCH 2/4] ci: Add option to compress ext4-images.
2020-04-21 8:31 [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com Q. Gylstorff
@ 2020-04-21 8:31 ` Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 3/4] test/README: Update Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 4/4] scripts/run-lava-tests: Add LAVA_MASTER_URL Q. Gylstorff
3 siblings, 0 replies; 9+ messages in thread
From: Q. Gylstorff @ 2020-04-21 8:31 UTC (permalink / raw)
To: xenomai, Jan.Kiszka
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
The qemu test targets create uncompressed ext4-img around 1GB. This
increases the time to uplaod and test a image with qemu. Reduce the
time by adding a compress option to ext4-img.bbclass.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
.ci/gitlab-ci-base.yml | 3 ++
.../0001-classes-ext4.img-Compress.patch | 31 +++++++++++++++++++
opt-ext4-gz.yml | 24 ++++++++++++++
3 files changed, 58 insertions(+)
create mode 100644 isar-patches/0001-classes-ext4.img-Compress.patch
create mode 100644 opt-ext4-gz.yml
diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
index e8d259f..6978ae9 100644
--- a/.ci/gitlab-ci-base.yml
+++ b/.ci/gitlab-ci-base.yml
@@ -53,6 +53,7 @@ build:board-qemu-amd64:
extends: .build
variables:
TARGET: qemu-amd64
+ BUILD_OPTIONS: ":opt-ext4-gz.yml"
lava-test:qemu-amd64:
needs: [ "build:board-qemu-amd64" ]
@@ -65,6 +66,7 @@ build:board-qemu-armhf:
extends: .build
variables:
TARGET: qemu-armhf
+ BUILD_OPTIONS: ":opt-ext4-gz.yml"
lava-test:qemu-armhf:
needs: [ "build:board-qemu-armhf" ]
@@ -76,6 +78,7 @@ build:board-qemu-arm64:
extends: .build
variables:
TARGET: qemu-arm64
+ BUILD_OPTIONS: ":opt-ext4-gz.yml"
lava-test:qemu-arm64:
needs: [ "build:board-qemu-arm64" ]
diff --git a/isar-patches/0001-classes-ext4.img-Compress.patch b/isar-patches/0001-classes-ext4.img-Compress.patch
new file mode 100644
index 0000000..6b67295
--- /dev/null
+++ b/isar-patches/0001-classes-ext4.img-Compress.patch
@@ -0,0 +1,31 @@
+From 915eec01cf660b160b1eab551c60fba4aac13b4b Mon Sep 17 00:00:00 2001
+From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
+Date: Fri, 17 Apr 2020 11:32:39 +0200
+Subject: [PATCH] classes/ext4.img: Compress
+
+change owner and add compress option
+
+Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
+---
+ meta/classes/ext4-img.bbclass | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
+index 334dc64..dc091d9 100644
+--- a/meta/classes/ext4-img.bbclass
++++ b/meta/classes/ext4-img.bbclass
+@@ -17,6 +17,11 @@ do_ext4_image() {
+
+ sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \
+ -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}'
++ sudo chown $(id -u):$(id -g) ${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}
++
++ if [ "${COMPRESS}" = "gz" ]; then
++ gzip ${DEPLOY_DIR_IMAGE}/${EXT4_IMAGE_FILE}
++ fi
+ }
+
+ addtask ext4_image before do_image after do_image_tools
+--
+2.20.1
+
diff --git a/opt-ext4-gz.yml b/opt-ext4-gz.yml
new file mode 100644
index 0000000..1a6502f
--- /dev/null
+++ b/opt-ext4-gz.yml
@@ -0,0 +1,24 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2018
+#
+# Authors:
+# Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+header:
+ version: 8
+
+repos:
+ isar:
+ patches:
+ 01-compress-ext4:
+ path: isar-patches/0001-classes-ext4.img-Compress.patch
+ repo: xenomai
+
+local_conf_header:
+ compress: |
+ COMPRESS = "gz"
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Xenomai-images PATCH 3/4] test/README: Update
2020-04-21 8:31 [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 2/4] ci: Add option to compress ext4-images Q. Gylstorff
@ 2020-04-21 8:31 ` Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 4/4] scripts/run-lava-tests: Add LAVA_MASTER_URL Q. Gylstorff
3 siblings, 0 replies; 9+ messages in thread
From: Q. Gylstorff @ 2020-04-21 8:31 UTC (permalink / raw)
To: xenomai, Jan.Kiszka
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Add missing targets and update CI variable description.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
tests/README.md | 59 ++++++++++++++++++++++++++++++++++++-------------
1 file changed, 44 insertions(+), 15 deletions(-)
diff --git a/tests/README.md b/tests/README.md
index 01b956e..8bf97a5 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -8,9 +8,11 @@ sends them to the lava testlab.
## Test targets
following images are test:
+- qemu-armhf
- qemu-arm64
- qemu-amd64
- x86-64-efi
+- hikey
- beagle-bone-black
## Deploy test
@@ -85,20 +87,47 @@ Setup a lava environment by following the
[installation guide](https://docs.lavasoftware.org/lava/first-installation.html)
or use [lava-docker](https://github.com/kernelci/lava-docker).
-# CI Variables
+# CI Files
+
+To support different ci setups all ci files are stored in `.ci`. `.gitlab-ci.yml` in
+the repository root use the settings to build on gitlab.com. To use another setup adapt
+the [Custom CI configuration path](https://code.siemens.com/help/ci/pipelines/settings#custom-ci-configuration-path)
+
+
+## CI Variables
The following variables are used and set by the ci system:
-- `HTTP_PROXY` : proxy settings
-- `HTTPS_PROXY` : proxy settings
-- `FTP_PROXY` : proxy settings
-- `NO_PROXY` : proxy settings
-- `LAVA_SSH_USER` : ssh user to connect to the LAVA master
-- `LAVA_SSH_HOST` : ssh host name to connect to the LAVA master
-- `LAVA_SSH_PORT` : ssh port used for the ssh tunnel
-- `LAVA_SSH_UPLOAD_KEY` : private ssh key to connect to the LAVA master
-- `LAVA_SSH_KNOWN_HOSTS` : Known hosts to connect via ssh to the host given by `LAVA_SSH_HOST`
-- `LAVA_MASTER_ACCOUNT` : lava master account name to register lavacli for test execution
-- `LAVA_MASTER_TOKEN` : token to connect with the lava master
-- `LAVA_DEPLOY_DIR` : optional variable to define directory to store the build artifacts
-- `LAVA_ARTIFACTS_URL` : optional variable where to get the artifacts for testing
-- `BUILD_OPTIONS` : optional parameter. Used for triggers. Overwrite to build the newest ipipe together with xenomai or other combinations.
+- proxy settings:
+ - `HTTP_PROXY` : http proxy
+ - `HTTPS_PROXY` : https proxy
+ - `FTP_PROXY` : ftp proxy
+ - `NO_PROXY` : no proxy
+
+- SSH settings:
+ - `LAVA_SSH_USER` : ssh user to connect to the LAVA master
+ - `LAVA_SSH_HOST` : ssh host name to connect to the LAVA master
+ - `LAVA_SSH_PORT` : ssh port used for the ssh tunnel
+ - `LAVA_SSH_UPLOAD_KEY` : private ssh key to connect to the LAVA master
+ - `LAVA_SSH_KNOWN_HOSTS` : Known hosts to connect via ssh to the host given by `LAVA_SSH_HOST`
+
+- LAVA settings:
+ - `LAVA_MASTER_ACCOUNT` : lava master account name to register lavacli for test execution
+ - `LAVA_MASTER_TOKEN` : token to connect with the lava master
+ - `LAVA_DEPLOY_DIR` : optional variable to define directory to store the build artifacts
+ - `LAVA_ARTIFACTS_URL` : optional variable where to get the artifacts for testing
+ - `LAVA_DEPLOY_DIR`: Directory to deploy the Artifacts in a build. Default /var/lib/lava/artifacts
+ - `LAVA_MASTER_PORT`: PORT to access the LAVA Master
+
+- General build settings
+ - `BUILD_OPTIONS` : optional parameter. Used for triggers. Overwrite to build the newest ipipe together with xenomai or other combinations.
+ - `DEPLOY_DIR_EXTENSION`: Extension path to store artifacts for the same target
+ - `USE_GITLAB_ARTIFACTS`: If set this changes the deploy url to a gitlab api URL
+ - `JOB_TEMPLATE_PATH`: Path to the test templates - Default tests/jobs/xenomai
+ - `TARGET_EXTENSION`: Extension of the build job name
+## LAVA Template variable
+
+- `TARGET`: Name of the target in the LAVA Lab
+- `BUILD_ARCH`: Architecture of the Target
+- `DEPLOY_DIR`: Path to the test components will be generate by the script `scripts/run-lava-tests.sh`
+- `ISAR_IMAGE`: Name of the ISAR image (e.g. demo-image)
+- `ISAR_DISTRIBUTION`: Name of the ISAR DISTRIBUTION (e.g. xenomai-demo)
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Xenomai-images PATCH 4/4] scripts/run-lava-tests: Add LAVA_MASTER_URL
2020-04-21 8:31 [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com Q. Gylstorff
` (2 preceding siblings ...)
2020-04-21 8:31 ` [Xenomai-images PATCH 3/4] test/README: Update Q. Gylstorff
@ 2020-04-21 8:31 ` Q. Gylstorff
3 siblings, 0 replies; 9+ messages in thread
From: Q. Gylstorff @ 2020-04-21 8:31 UTC (permalink / raw)
To: xenomai, Jan.Kiszka
From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Allow to set the LAVA master URL.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
scripts/run-lava-tests.sh | 2 +-
tests/README.md | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/run-lava-tests.sh b/scripts/run-lava-tests.sh
index 8566493..764f3a9 100755
--- a/scripts/run-lava-tests.sh
+++ b/scripts/run-lava-tests.sh
@@ -37,7 +37,7 @@ do
sleep ${interval}
timeout=$(expr ${timeout} - ${interval})
done
-lava_master_uri=http://localhost:${lava_master_port}
+lava_master_uri=${LAVA_MASTER_URL:-http://localhost}:${lava_master_port}
# connect to lava master
lavacli identities add --token ${LAVA_MASTER_TOKEN} --uri ${lava_master_uri} --username ${LAVA_MASTER_ACCOUNT} default
diff --git a/tests/README.md b/tests/README.md
index 8bf97a5..119f90f 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -117,6 +117,7 @@ The following variables are used and set by the ci system:
- `LAVA_ARTIFACTS_URL` : optional variable where to get the artifacts for testing
- `LAVA_DEPLOY_DIR`: Directory to deploy the Artifacts in a build. Default /var/lib/lava/artifacts
- `LAVA_MASTER_PORT`: PORT to access the LAVA Master
+ - `LAVA_MASTER_URL`: URL to the LAVA Master
- General build settings
- `BUILD_OPTIONS` : optional parameter. Used for triggers. Overwrite to build the newest ipipe together with xenomai or other combinations.
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com
2020-04-21 8:31 ` [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com Q. Gylstorff
@ 2020-04-21 9:09 ` Jan Kiszka
2020-04-21 9:19 ` Gylstorff Quirin
0 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2020-04-21 9:09 UTC (permalink / raw)
To: Q. Gylstorff, xenomai
On 21.04.20 10:31, Q. Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>
> Adapt scripts and gitlab-ci.yml to build on public available
> gitlab infrastructure.
>
> As the internal gitlab system has no artifact storage and
> it is not possible to deactivate arifact upload with a variable
> split the gitlab-ci.yml. The .gitlab-ci.yml in the root points
> to .ci/gitlab-com.yml.
Why hidden ".ci/" and not just "ci/"?
>
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
> .ci/gitlab-ci-base.yml | 153 ++++++++++++++++++
> .ci/gitlab-com.yml | 26 +++
> .ci/siemens-ci.yml | 22 +++
> .gitlab-ci.yml | 145 +----------------
> scripts/deploy_for_testing.sh | 4 +-
> scripts/get-build-id.sh | 29 ++++
> scripts/install-lavacli.sh | 5 +-
> scripts/run-lava-tests.sh | 8 +-
> ...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
> tests/jobs/xenomai-hikey.yml | 4 +-
> tests/jobs/xenomai-ipc227e.yml | 6 +-
> tests/jobs/xenomai-minnow.yml | 6 +-
> tests/jobs/xenomai-qemu-amd64.yml | 6 +-
> tests/jobs/xenomai-qemu-arm64.yml | 6 +-
> tests/jobs/xenomai-qemu-armhf.yml | 6 +-
> ...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
> 16 files changed, 265 insertions(+), 171 deletions(-)
> create mode 100644 .ci/gitlab-ci-base.yml
> create mode 100644 .ci/gitlab-com.yml
> create mode 100644 .ci/siemens-ci.yml
> create mode 100755 scripts/get-build-id.sh
> rename tests/jobs/{xenomai-beaglebone.yml => xenomai-beagle-bone-black.yml} (91%)
> rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml} (86%)
>
> diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
> new file mode 100644
> index 0000000..e8d259f
> --- /dev/null
> +++ b/.ci/gitlab-ci-base.yml
> @@ -0,0 +1,153 @@
> +#
> +# Xenomai Real-Time System
> +#
> +# Copyright (c) Siemens AG, 2019 - 2020
> +#
> +# Authors:
> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +stages:
> + - build
> + - lava-test
> +
> +variables:
> + GIT_STRATEGY: clone
> + http_proxy: "$HTTP_PROXY"
> + https_proxy: "$HTTPS_PROXY"
> + ftp_proxy: "$FTP_PROXY"
> + no_proxy: "$NO_PROXY"
> + XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
> + LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
> + ISAR_IMAGE: demo-image
> + ISAR_DISTRIBUTION: xenomai-demo
> +
> +default:
> + image: kasproject/kas-isar:latest
> +
> +# add lavacli to the container and install the ssh keys
> +# for the test infrastructure
> +.add-lava-ssh-config:
> + before_script:
> + - mkdir -p -m=700 ~/.ssh
> + - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat - PROXY:$(echo $https_proxy | sed 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600 ~/.ssh/config; fi;
> + - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa && chmod 600 ~/.ssh/lava_id_rsa
> + - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
> +
> +.test:
> + extends: .add-lava-ssh-config
> + stage: lava-test
> + script:
> + - scripts/install-lavacli.sh
> + - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
> + - scripts/run-lava-tests.sh ${TARGET}
> + variables:
> + BUILD_JOB_NAME: "build:board-${TARGET}${TARGET_EXTENSION}"
> + only:
> + variables:
> + - $LAVA_SSH_USER
> +
> +
> +build:board-qemu-amd64:
> + extends: .build
> + variables:
> + TARGET: qemu-amd64
> +
> +lava-test:qemu-amd64:
> + needs: [ "build:board-qemu-amd64" ]
> + dependencies: [ "build:board-qemu-amd64" ]
> + extends: .test
> + variables:
> + TARGET: qemu-amd64
> +
> +build:board-qemu-armhf:
> + extends: .build
> + variables:
> + TARGET: qemu-armhf
> +
> +lava-test:qemu-armhf:
> + needs: [ "build:board-qemu-armhf" ]
> + extends: .test
> + variables:
> + TARGET: qemu-armhf
> +
> +build:board-qemu-arm64:
> + extends: .build
> + variables:
> + TARGET: qemu-arm64
> +
> +lava-test:qemu-arm64:
> + needs: [ "build:board-qemu-arm64" ]
> + extends: .test
> + variables:
> + TARGET: qemu-arm64
> +
> +build:board-hikey:
> + extends: .build
> + variables:
> + TARGET: hikey
> + BUILD_OPTIONS: ":opt-lava-test.yml"
> +
> +lava-test:board-hikey:
> + needs: [ "build:board-hikey" ]
> + extends: .test
> + variables:
> + TARGET: hikey
> +
> +build:board-beagle-bone-black:
> + extends: .build
> + variables:
> + TARGET: beagle-bone-black
> + BUILD_OPTIONS: ":opt-lava-test.yml"
> +
> +lava-test:board-beagle-bone-black:
> + needs: [ "build:board-beagle-bone-black" ]
> + extends: .test
> + variables:
> + TARGET: beagle-bone-black
> +
> +build:board-x86-64-efi:
> + extends: .build
> + variables:
> + TARGET: x86-64-efi
> + BUILD_OPTIONS: ":opt-lava-test.yml"
> +
> +lava-test:board-x86-64-efi:
> + needs: [ "build:board-x86-64-efi" ]
> + extends: .test
> + variables:
> + TARGET: x86-64-efi
> +
> +build:board-beagle-bone-black:xenomai-3.0.x:
> + extends: .build
> + variables:
> + TARGET: beagle-bone-black
> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
> + BUILD_OPTIONS: ":opt-lava-test.yml"
> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
> +
> +lava-test:board-beagle-bone-black:xenomai-3.0.x:
> + needs: [ "build:board-beagle-bone-black:xenomai-3.0.x" ]
> + extends: .test
> + variables:
> + TARGET: beagle-bone-black
> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
> + TARGET_EXTENSION: ":xenomai-3.0.x"
> +
> +build:board-x86-64-efi:xenomai-3.0.x:
> + extends: .build
> + variables:
> + TARGET: x86-64-efi
> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
> + BUILD_OPTIONS: ":opt-lava-test.yml"
> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
> +
> +lava-test:board-x86-64-efi:xenomai-3.0.x:
> + needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
> + extends: .test
> + variables:
> + TARGET: x86-64-efi
> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
> + TARGET_EXTENSION: ":xenomai-3.0.x"
> +
> diff --git a/.ci/gitlab-com.yml b/.ci/gitlab-com.yml
> new file mode 100644
> index 0000000..0abb3fb
> --- /dev/null
> +++ b/.ci/gitlab-com.yml
> @@ -0,0 +1,26 @@
> +#
> +# Xenomai Real-Time System
> +#
> +# Copyright (c) Siemens AG, 2019 - 2020
> +#
> +# Authors:
> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +.build:
> + extends: .add-lava-ssh-config
> + stage: build
> + script:
> + - kas build kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
> + artifacts:
> + paths:
> + - build/tmp/deploy/images/${TARGET}/
Can't we put that path completely into a variable and leave that empty
for CI setups that do not support (large) artifacts?
Jan
> + expire_in: 1 week
> + only:
> + variables:
> + - $USE_GITLAB_ARTIFACTS
> +include:
> + - local: '/.ci/gitlab-ci-base.yml'
> +
> +
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com
2020-04-21 9:09 ` Jan Kiszka
@ 2020-04-21 9:19 ` Gylstorff Quirin
2020-04-21 9:29 ` Jan Kiszka
0 siblings, 1 reply; 9+ messages in thread
From: Gylstorff Quirin @ 2020-04-21 9:19 UTC (permalink / raw)
To: Jan Kiszka, xenomai
On 4/21/20 11:09 AM, Jan Kiszka wrote:
> On 21.04.20 10:31, Q. Gylstorff wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> Adapt scripts and gitlab-ci.yml to build on public available
>> gitlab infrastructure.
>>
>> As the internal gitlab system has no artifact storage and
>> it is not possible to deactivate arifact upload with a variable
>> split the gitlab-ci.yml. The .gitlab-ci.yml in the root points
>> to .ci/gitlab-com.yml.
>
> Why hidden ".ci/" and not just "ci/"?
I modeled it after the .gitlab-ci.yml. If it is better to have it
visible a will ajust that in a second patch set.
>
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>> .ci/gitlab-ci-base.yml | 153 ++++++++++++++++++
>> .ci/gitlab-com.yml | 26 +++
>> .ci/siemens-ci.yml | 22 +++
>> .gitlab-ci.yml | 145 +----------------
>> scripts/deploy_for_testing.sh | 4 +-
>> scripts/get-build-id.sh | 29 ++++
>> scripts/install-lavacli.sh | 5 +-
>> scripts/run-lava-tests.sh | 8 +-
>> ...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
>> tests/jobs/xenomai-hikey.yml | 4 +-
>> tests/jobs/xenomai-ipc227e.yml | 6 +-
>> tests/jobs/xenomai-minnow.yml | 6 +-
>> tests/jobs/xenomai-qemu-amd64.yml | 6 +-
>> tests/jobs/xenomai-qemu-arm64.yml | 6 +-
>> tests/jobs/xenomai-qemu-armhf.yml | 6 +-
>> ...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
>> 16 files changed, 265 insertions(+), 171 deletions(-)
>> create mode 100644 .ci/gitlab-ci-base.yml
>> create mode 100644 .ci/gitlab-com.yml
>> create mode 100644 .ci/siemens-ci.yml
>> create mode 100755 scripts/get-build-id.sh
>> rename tests/jobs/{xenomai-beaglebone.yml =>
>> xenomai-beagle-bone-black.yml} (91%)
>> rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml} (86%)
>>
>> diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
>> new file mode 100644
>> index 0000000..e8d259f
>> --- /dev/null
>> +++ b/.ci/gitlab-ci-base.yml
>> @@ -0,0 +1,153 @@
>> +#
>> +# Xenomai Real-Time System
>> +#
>> +# Copyright (c) Siemens AG, 2019 - 2020
>> +#
>> +# Authors:
>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +stages:
>> + - build
>> + - lava-test
>> +
>> +variables:
>> + GIT_STRATEGY: clone
>> + http_proxy: "$HTTP_PROXY"
>> + https_proxy: "$HTTPS_PROXY"
>> + ftp_proxy: "$FTP_PROXY"
>> + no_proxy: "$NO_PROXY"
>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
>> + LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
>> + ISAR_IMAGE: demo-image
>> + ISAR_DISTRIBUTION: xenomai-demo
>> +
>> +default:
>> + image: kasproject/kas-isar:latest
>> +
>> +# add lavacli to the container and install the ssh keys
>> +# for the test infrastructure
>> +.add-lava-ssh-config:
>> + before_script:
>> + - mkdir -p -m=700 ~/.ssh
>> + - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat -
>> PROXY:$(echo $https_proxy | sed
>> 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed
>> 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600 ~/.ssh/config; fi;
>> + - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa
>> && chmod 600 ~/.ssh/lava_id_rsa
>> + - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod 644
>> ~/.ssh/known_hosts
>> +
>> +.test:
>> + extends: .add-lava-ssh-config
>> + stage: lava-test
>> + script:
>> + - scripts/install-lavacli.sh
>> + - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
>> + - scripts/run-lava-tests.sh ${TARGET}
>> + variables:
>> + BUILD_JOB_NAME: "build:board-${TARGET}${TARGET_EXTENSION}"
>> + only:
>> + variables:
>> + - $LAVA_SSH_USER
>> +
>> +
>> +build:board-qemu-amd64:
>> + extends: .build
>> + variables:
>> + TARGET: qemu-amd64
>> +
>> +lava-test:qemu-amd64:
>> + needs: [ "build:board-qemu-amd64" ]
>> + dependencies: [ "build:board-qemu-amd64" ]
>> + extends: .test
>> + variables:
>> + TARGET: qemu-amd64
>> +
>> +build:board-qemu-armhf:
>> + extends: .build
>> + variables:
>> + TARGET: qemu-armhf
>> +
>> +lava-test:qemu-armhf:
>> + needs: [ "build:board-qemu-armhf" ]
>> + extends: .test
>> + variables:
>> + TARGET: qemu-armhf
>> +
>> +build:board-qemu-arm64:
>> + extends: .build
>> + variables:
>> + TARGET: qemu-arm64
>> +
>> +lava-test:qemu-arm64:
>> + needs: [ "build:board-qemu-arm64" ]
>> + extends: .test
>> + variables:
>> + TARGET: qemu-arm64
>> +
>> +build:board-hikey:
>> + extends: .build
>> + variables:
>> + TARGET: hikey
>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>> +
>> +lava-test:board-hikey:
>> + needs: [ "build:board-hikey" ]
>> + extends: .test
>> + variables:
>> + TARGET: hikey
>> +
>> +build:board-beagle-bone-black:
>> + extends: .build
>> + variables:
>> + TARGET: beagle-bone-black
>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>> +
>> +lava-test:board-beagle-bone-black:
>> + needs: [ "build:board-beagle-bone-black" ]
>> + extends: .test
>> + variables:
>> + TARGET: beagle-bone-black
>> +
>> +build:board-x86-64-efi:
>> + extends: .build
>> + variables:
>> + TARGET: x86-64-efi
>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>> +
>> +lava-test:board-x86-64-efi:
>> + needs: [ "build:board-x86-64-efi" ]
>> + extends: .test
>> + variables:
>> + TARGET: x86-64-efi
>> +
>> +build:board-beagle-bone-black:xenomai-3.0.x:
>> + extends: .build
>> + variables:
>> + TARGET: beagle-bone-black
>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>> +
>> +lava-test:board-beagle-bone-black:xenomai-3.0.x:
>> + needs: [ "build:board-beagle-bone-black:xenomai-3.0.x" ]
>> + extends: .test
>> + variables:
>> + TARGET: beagle-bone-black
>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>> +
>> +build:board-x86-64-efi:xenomai-3.0.x:
>> + extends: .build
>> + variables:
>> + TARGET: x86-64-efi
>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>> +
>> +lava-test:board-x86-64-efi:xenomai-3.0.x:
>> + needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
>> + extends: .test
>> + variables:
>> + TARGET: x86-64-efi
>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>> +
>> diff --git a/.ci/gitlab-com.yml b/.ci/gitlab-com.yml
>> new file mode 100644
>> index 0000000..0abb3fb
>> --- /dev/null
>> +++ b/.ci/gitlab-com.yml
>> @@ -0,0 +1,26 @@
>> +#
>> +# Xenomai Real-Time System
>> +#
>> +# Copyright (c) Siemens AG, 2019 - 2020
>> +#
>> +# Authors:
>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +.build:
>> + extends: .add-lava-ssh-config
>> + stage: build
>> + script:
>> + - kas build
>> kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
>>
>> + artifacts:
>> + paths:
>> + - build/tmp/deploy/images/${TARGET}/
>
> Can't we put that path completely into a variable and leave that empty
> for CI setups that do not support (large) artifacts?
>
> Jan
I tried that and it leads to an invalid gitlab-ci.yml. This is kind of a
workaround to work with artifacts. If we use another storage backend
(aws) with our own upload scripts we can use your sugestion.
Quirin
>
>> + expire_in: 1 week
>> + only:
>> + variables:
>> + - $USE_GITLAB_ARTIFACTS
>> +include:
>> + - local: '/.ci/gitlab-ci-base.yml'
>> +
>> +
>
--
Quirin Gylstorff
Siemens AG
Corporate Technology
Research in Digitalization and Automation
Smart Embedded Systems
CT RDA IOT SES-DE
Otto-Hahn-Ring 6
81739 Muenchen, Germany
Mobile: +49 173 3746683
mailto:quirin.gylstorff@siemens.com
www.siemens.com/ingenuityforlife
Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Jim
Hagemann Snabe; Managing Board: Joe Kaeser, Chairman, President and
Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich,
Cedrik Neike, Michael Sen, Ralf P. Thomas; Registered offices: Berlin
and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB
12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322
Important notice: This e-mail and any attachment thereof contain
corporate proprietary information. If you have received it by mistake,
please notify us immediately by reply e-mail and delete this e-mail and
its attachments from your system. Thank you.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com
2020-04-21 9:19 ` Gylstorff Quirin
@ 2020-04-21 9:29 ` Jan Kiszka
2020-04-21 10:17 ` Gylstorff Quirin
0 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2020-04-21 9:29 UTC (permalink / raw)
To: Gylstorff Quirin, xenomai
On 21.04.20 11:19, Gylstorff Quirin wrote:
>
>
> On 4/21/20 11:09 AM, Jan Kiszka wrote:
>> On 21.04.20 10:31, Q. Gylstorff wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> Adapt scripts and gitlab-ci.yml to build on public available
>>> gitlab infrastructure.
>>>
>>> As the internal gitlab system has no artifact storage and
>>> it is not possible to deactivate arifact upload with a variable
>>> split the gitlab-ci.yml. The .gitlab-ci.yml in the root points
>>> to .ci/gitlab-com.yml.
>>
>> Why hidden ".ci/" and not just "ci/"?
>
> I modeled it after the .gitlab-ci.yml. If it is better to have it
> visible a will ajust that in a second patch set.
>
>>
>>>
>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> ---
>>> .ci/gitlab-ci-base.yml | 153 ++++++++++++++++++
>>> .ci/gitlab-com.yml | 26 +++
>>> .ci/siemens-ci.yml | 22 +++
>>> .gitlab-ci.yml | 145 +----------------
>>> scripts/deploy_for_testing.sh | 4 +-
>>> scripts/get-build-id.sh | 29 ++++
>>> scripts/install-lavacli.sh | 5 +-
>>> scripts/run-lava-tests.sh | 8 +-
>>> ...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
>>> tests/jobs/xenomai-hikey.yml | 4 +-
>>> tests/jobs/xenomai-ipc227e.yml | 6 +-
>>> tests/jobs/xenomai-minnow.yml | 6 +-
>>> tests/jobs/xenomai-qemu-amd64.yml | 6 +-
>>> tests/jobs/xenomai-qemu-arm64.yml | 6 +-
>>> tests/jobs/xenomai-qemu-armhf.yml | 6 +-
>>> ...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
>>> 16 files changed, 265 insertions(+), 171 deletions(-)
>>> create mode 100644 .ci/gitlab-ci-base.yml
>>> create mode 100644 .ci/gitlab-com.yml
>>> create mode 100644 .ci/siemens-ci.yml
>>> create mode 100755 scripts/get-build-id.sh
>>> rename tests/jobs/{xenomai-beaglebone.yml =>
>>> xenomai-beagle-bone-black.yml} (91%)
>>> rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml} (86%)
>>>
>>> diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
>>> new file mode 100644
>>> index 0000000..e8d259f
>>> --- /dev/null
>>> +++ b/.ci/gitlab-ci-base.yml
>>> @@ -0,0 +1,153 @@
>>> +#
>>> +# Xenomai Real-Time System
>>> +#
>>> +# Copyright (c) Siemens AG, 2019 - 2020
>>> +#
>>> +# Authors:
>>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +#
>>> +stages:
>>> + - build
>>> + - lava-test
>>> +
>>> +variables:
>>> + GIT_STRATEGY: clone
>>> + http_proxy: "$HTTP_PROXY"
>>> + https_proxy: "$HTTPS_PROXY"
>>> + ftp_proxy: "$FTP_PROXY"
>>> + no_proxy: "$NO_PROXY"
>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
>>> + LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
>>> + ISAR_IMAGE: demo-image
>>> + ISAR_DISTRIBUTION: xenomai-demo
>>> +
>>> +default:
>>> + image: kasproject/kas-isar:latest
>>> +
>>> +# add lavacli to the container and install the ssh keys
>>> +# for the test infrastructure
>>> +.add-lava-ssh-config:
>>> + before_script:
>>> + - mkdir -p -m=700 ~/.ssh
>>> + - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat -
>>> PROXY:$(echo $https_proxy | sed
>>> 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed
>>> 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600 ~/.ssh/config;
>>> fi;
>>> + - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa
>>> && chmod 600 ~/.ssh/lava_id_rsa
>>> + - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod
>>> 644 ~/.ssh/known_hosts
>>> +
>>> +.test:
>>> + extends: .add-lava-ssh-config
>>> + stage: lava-test
>>> + script:
>>> + - scripts/install-lavacli.sh
>>> + - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
>>> + - scripts/run-lava-tests.sh ${TARGET}
>>> + variables:
>>> + BUILD_JOB_NAME: "build:board-${TARGET}${TARGET_EXTENSION}"
>>> + only:
>>> + variables:
>>> + - $LAVA_SSH_USER
>>> +
>>> +
>>> +build:board-qemu-amd64:
>>> + extends: .build
>>> + variables:
>>> + TARGET: qemu-amd64
>>> +
>>> +lava-test:qemu-amd64:
>>> + needs: [ "build:board-qemu-amd64" ]
>>> + dependencies: [ "build:board-qemu-amd64" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: qemu-amd64
>>> +
>>> +build:board-qemu-armhf:
>>> + extends: .build
>>> + variables:
>>> + TARGET: qemu-armhf
>>> +
>>> +lava-test:qemu-armhf:
>>> + needs: [ "build:board-qemu-armhf" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: qemu-armhf
>>> +
>>> +build:board-qemu-arm64:
>>> + extends: .build
>>> + variables:
>>> + TARGET: qemu-arm64
>>> +
>>> +lava-test:qemu-arm64:
>>> + needs: [ "build:board-qemu-arm64" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: qemu-arm64
>>> +
>>> +build:board-hikey:
>>> + extends: .build
>>> + variables:
>>> + TARGET: hikey
>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>> +
>>> +lava-test:board-hikey:
>>> + needs: [ "build:board-hikey" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: hikey
>>> +
>>> +build:board-beagle-bone-black:
>>> + extends: .build
>>> + variables:
>>> + TARGET: beagle-bone-black
>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>> +
>>> +lava-test:board-beagle-bone-black:
>>> + needs: [ "build:board-beagle-bone-black" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: beagle-bone-black
>>> +
>>> +build:board-x86-64-efi:
>>> + extends: .build
>>> + variables:
>>> + TARGET: x86-64-efi
>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>> +
>>> +lava-test:board-x86-64-efi:
>>> + needs: [ "build:board-x86-64-efi" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: x86-64-efi
>>> +
>>> +build:board-beagle-bone-black:xenomai-3.0.x:
>>> + extends: .build
>>> + variables:
>>> + TARGET: beagle-bone-black
>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>> +
>>> +lava-test:board-beagle-bone-black:xenomai-3.0.x:
>>> + needs: [ "build:board-beagle-bone-black:xenomai-3.0.x" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: beagle-bone-black
>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>>> +
>>> +build:board-x86-64-efi:xenomai-3.0.x:
>>> + extends: .build
>>> + variables:
>>> + TARGET: x86-64-efi
>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>> +
>>> +lava-test:board-x86-64-efi:xenomai-3.0.x:
>>> + needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
>>> + extends: .test
>>> + variables:
>>> + TARGET: x86-64-efi
>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>>> +
>>> diff --git a/.ci/gitlab-com.yml b/.ci/gitlab-com.yml
>>> new file mode 100644
>>> index 0000000..0abb3fb
>>> --- /dev/null
>>> +++ b/.ci/gitlab-com.yml
>>> @@ -0,0 +1,26 @@
>>> +#
>>> +# Xenomai Real-Time System
>>> +#
>>> +# Copyright (c) Siemens AG, 2019 - 2020
>>> +#
>>> +# Authors:
>>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +#
>>> +.build:
>>> + extends: .add-lava-ssh-config
>>> + stage: build
>>> + script:
>>> + - kas build
>>> kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
>>>
>>> + artifacts:
>>> + paths:
>>> + - build/tmp/deploy/images/${TARGET}/
>>
>> Can't we put that path completely into a variable and leave that empty
>> for CI setups that do not support (large) artifacts?
>>
>> Jan
>
>
> I tried that and it leads to an invalid gitlab-ci.yml. This is kind of a
> workaround to work with artifacts. If we use another storage backend
> (aws) with our own upload scripts we can use your sugestion.
OK, was just a quick idea.
Then I would suggest to move your new config for gitlab.com to
.gitlab-ci.yml and provide a .gitlab-ci-no-artifacts.yml (or so) for our
special case. That way, the default setup will become artifacts-based,
and no one need to fiddle with the related CI settings when cloning the
repo on gitlab.com.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com
2020-04-21 9:29 ` Jan Kiszka
@ 2020-04-21 10:17 ` Gylstorff Quirin
0 siblings, 0 replies; 9+ messages in thread
From: Gylstorff Quirin @ 2020-04-21 10:17 UTC (permalink / raw)
To: Jan Kiszka, xenomai
On 4/21/20 11:29 AM, Jan Kiszka wrote:
> On 21.04.20 11:19, Gylstorff Quirin wrote:
>>
>>
>> On 4/21/20 11:09 AM, Jan Kiszka wrote:
>>> On 21.04.20 10:31, Q. Gylstorff wrote:
>>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>>
>>>> Adapt scripts and gitlab-ci.yml to build on public available
>>>> gitlab infrastructure.
>>>>
>>>> As the internal gitlab system has no artifact storage and
>>>> it is not possible to deactivate arifact upload with a variable
>>>> split the gitlab-ci.yml. The .gitlab-ci.yml in the root points
>>>> to .ci/gitlab-com.yml.
>>>
>>> Why hidden ".ci/" and not just "ci/"?
>>
>> I modeled it after the .gitlab-ci.yml. If it is better to have it
>> visible a will ajust that in a second patch set.
>>
>>>
>>>>
>>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>> ---
>>>> .ci/gitlab-ci-base.yml | 153
>>>> ++++++++++++++++++
>>>> .ci/gitlab-com.yml | 26 +++
>>>> .ci/siemens-ci.yml | 22 +++
>>>> .gitlab-ci.yml | 145 +----------------
>>>> scripts/deploy_for_testing.sh | 4 +-
>>>> scripts/get-build-id.sh | 29 ++++
>>>> scripts/install-lavacli.sh | 5 +-
>>>> scripts/run-lava-tests.sh | 8 +-
>>>> ...bone.yml => xenomai-beagle-bone-black.yml} | 4 +-
>>>> tests/jobs/xenomai-hikey.yml | 4 +-
>>>> tests/jobs/xenomai-ipc227e.yml | 6 +-
>>>> tests/jobs/xenomai-minnow.yml | 6 +-
>>>> tests/jobs/xenomai-qemu-amd64.yml | 6 +-
>>>> tests/jobs/xenomai-qemu-arm64.yml | 6 +-
>>>> tests/jobs/xenomai-qemu-armhf.yml | 6 +-
>>>> ...omai-x86-64.yml => xenomai-x86-64-efi.yml} | 6 +-
>>>> 16 files changed, 265 insertions(+), 171 deletions(-)
>>>> create mode 100644 .ci/gitlab-ci-base.yml
>>>> create mode 100644 .ci/gitlab-com.yml
>>>> create mode 100644 .ci/siemens-ci.yml
>>>> create mode 100755 scripts/get-build-id.sh
>>>> rename tests/jobs/{xenomai-beaglebone.yml =>
>>>> xenomai-beagle-bone-black.yml} (91%)
>>>> rename tests/jobs/{xenomai-x86-64.yml => xenomai-x86-64-efi.yml}
>>>> (86%)
>>>>
>>>> diff --git a/.ci/gitlab-ci-base.yml b/.ci/gitlab-ci-base.yml
>>>> new file mode 100644
>>>> index 0000000..e8d259f
>>>> --- /dev/null
>>>> +++ b/.ci/gitlab-ci-base.yml
>>>> @@ -0,0 +1,153 @@
>>>> +#
>>>> +# Xenomai Real-Time System
>>>> +#
>>>> +# Copyright (c) Siemens AG, 2019 - 2020
>>>> +#
>>>> +# Authors:
>>>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +#
>>>> +stages:
>>>> + - build
>>>> + - lava-test
>>>> +
>>>> +variables:
>>>> + GIT_STRATEGY: clone
>>>> + http_proxy: "$HTTP_PROXY"
>>>> + https_proxy: "$HTTPS_PROXY"
>>>> + ftp_proxy: "$FTP_PROXY"
>>>> + no_proxy: "$NO_PROXY"
>>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-next.yml"
>>>> + LINUX_BUILD_OPTION: ":opt-linux-latest.yml"
>>>> + ISAR_IMAGE: demo-image
>>>> + ISAR_DISTRIBUTION: xenomai-demo
>>>> +
>>>> +default:
>>>> + image: kasproject/kas-isar:latest
>>>> +
>>>> +# add lavacli to the container and install the ssh keys
>>>> +# for the test infrastructure
>>>> +.add-lava-ssh-config:
>>>> + before_script:
>>>> + - mkdir -p -m=700 ~/.ssh
>>>> + - if [ -n "$https_proxy" ]; then echo "ProxyCommand socat -
>>>> PROXY:$(echo $https_proxy | sed
>>>> 's|.*://\([^:]*\).*|\1|'):%h:%p,proxyport=$(echo $https_proxy | sed
>>>> 's|.*:\([0-9]*\)$|\1|')" >> ~/.ssh/config && chmod 600
>>>> ~/.ssh/config; fi;
>>>> + - echo "$LAVA_SSH_UPLOAD_KEY" | tr -d '\r' > ~/.ssh/lava_id_rsa
>>>> && chmod 600 ~/.ssh/lava_id_rsa
>>>> + - echo "$LAVA_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts && chmod
>>>> 644 ~/.ssh/known_hosts
>>>> +
>>>> +.test:
>>>> + extends: .add-lava-ssh-config
>>>> + stage: lava-test
>>>> + script:
>>>> + - scripts/install-lavacli.sh
>>>> + - export BUILD_JOB_ID=$(scripts/get-build-id.sh ${BUILD_JOB_NAME})
>>>> + - scripts/run-lava-tests.sh ${TARGET}
>>>> + variables:
>>>> + BUILD_JOB_NAME: "build:board-${TARGET}${TARGET_EXTENSION}"
>>>> + only:
>>>> + variables:
>>>> + - $LAVA_SSH_USER
>>>> +
>>>> +
>>>> +build:board-qemu-amd64:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: qemu-amd64
>>>> +
>>>> +lava-test:qemu-amd64:
>>>> + needs: [ "build:board-qemu-amd64" ]
>>>> + dependencies: [ "build:board-qemu-amd64" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: qemu-amd64
>>>> +
>>>> +build:board-qemu-armhf:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: qemu-armhf
>>>> +
>>>> +lava-test:qemu-armhf:
>>>> + needs: [ "build:board-qemu-armhf" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: qemu-armhf
>>>> +
>>>> +build:board-qemu-arm64:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: qemu-arm64
>>>> +
>>>> +lava-test:qemu-arm64:
>>>> + needs: [ "build:board-qemu-arm64" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: qemu-arm64
>>>> +
>>>> +build:board-hikey:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: hikey
>>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>>> +
>>>> +lava-test:board-hikey:
>>>> + needs: [ "build:board-hikey" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: hikey
>>>> +
>>>> +build:board-beagle-bone-black:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: beagle-bone-black
>>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>>> +
>>>> +lava-test:board-beagle-bone-black:
>>>> + needs: [ "build:board-beagle-bone-black" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: beagle-bone-black
>>>> +
>>>> +build:board-x86-64-efi:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: x86-64-efi
>>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>>> +
>>>> +lava-test:board-x86-64-efi:
>>>> + needs: [ "build:board-x86-64-efi" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: x86-64-efi
>>>> +
>>>> +build:board-beagle-bone-black:xenomai-3.0.x:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: beagle-bone-black
>>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>>> +
>>>> +lava-test:board-beagle-bone-black:xenomai-3.0.x:
>>>> + needs: [ "build:board-beagle-bone-black:xenomai-3.0.x" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: beagle-bone-black
>>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>>>> +
>>>> +build:board-x86-64-efi:xenomai-3.0.x:
>>>> + extends: .build
>>>> + variables:
>>>> + TARGET: x86-64-efi
>>>> + XENOMAI_BUILD_OPTION: ":opt-xenomai-3.0.x.yml"
>>>> + BUILD_OPTIONS: ":opt-lava-test.yml"
>>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>>> +
>>>> +lava-test:board-x86-64-efi:xenomai-3.0.x:
>>>> + needs: [ "build:board-x86-64-efi:xenomai-3.0.x" ]
>>>> + extends: .test
>>>> + variables:
>>>> + TARGET: x86-64-efi
>>>> + DEPLOY_DIR_EXTENSION: "xenomai-3.0.x"
>>>> + TARGET_EXTENSION: ":xenomai-3.0.x"
>>>> +
>>>> diff --git a/.ci/gitlab-com.yml b/.ci/gitlab-com.yml
>>>> new file mode 100644
>>>> index 0000000..0abb3fb
>>>> --- /dev/null
>>>> +++ b/.ci/gitlab-com.yml
>>>> @@ -0,0 +1,26 @@
>>>> +#
>>>> +# Xenomai Real-Time System
>>>> +#
>>>> +# Copyright (c) Siemens AG, 2019 - 2020
>>>> +#
>>>> +# Authors:
>>>> +# Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +#
>>>> +.build:
>>>> + extends: .add-lava-ssh-config
>>>> + stage: build
>>>> + script:
>>>> + - kas build
>>>> kas.yml:board-${TARGET}.yml${XENOMAI_BUILD_OPTION}${LINUX_BUILD_OPTION}${BUILD_OPTIONS}
>>>>
>>>> + artifacts:
>>>> + paths:
>>>> + - build/tmp/deploy/images/${TARGET}/
>>>
>>> Can't we put that path completely into a variable and leave that
>>> empty for CI setups that do not support (large) artifacts?
>>>
>>> Jan
>>
>>
>> I tried that and it leads to an invalid gitlab-ci.yml. This is kind of
>> a workaround to work with artifacts. If we use another storage backend
>> (aws) with our own upload scripts we can use your sugestion.
>
> OK, was just a quick idea.
>
> Then I would suggest to move your new config for gitlab.com to
> .gitlab-ci.yml and provide a .gitlab-ci-no-artifacts.yml (or so) for our
> special case. That way, the default setup will become artifacts-based,
> and no one need to fiddle with the related CI settings when cloning the
> repo on gitlab.com.
>
> Jan
>
The behaviour of the patchset is currently as you describe it ->
.gitlab-ci.yml includes the gitlab-com.yml and the internal builds
needs to point to the new siemens-ci.yml. But I can rename these yml
files to artifacts.yml and no-artifacts.yml in a version 2.
--
Quirin
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-04-21 10:17 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-21 8:31 [Xenomai-images PATCH 0/4] ci: Add configuration to run on gitlab.com Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 1/4] ci: Build and test from gitlab.com Q. Gylstorff
2020-04-21 9:09 ` Jan Kiszka
2020-04-21 9:19 ` Gylstorff Quirin
2020-04-21 9:29 ` Jan Kiszka
2020-04-21 10:17 ` Gylstorff Quirin
2020-04-21 8:31 ` [Xenomai-images PATCH 2/4] ci: Add option to compress ext4-images Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 3/4] test/README: Update Q. Gylstorff
2020-04-21 8:31 ` [Xenomai-images PATCH 4/4] scripts/run-lava-tests: Add LAVA_MASTER_URL Q. Gylstorff
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.