[PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Philippe Mathieu-Daudé]

Hi,

This series is a rework of the 'Add Dockerfile for hexagon' patch
from Alessandro/Brian that Taylor sent in v8:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
but adapted to mainstream.

Gitlab shared runner are usually capped to 2/4 cores and timeout
after 2h30 to 3h. We need a beefy custom runner to get the image
built and stored in the registry. For now the image is added in
manual mode, hopping we'll have magic runners added some day to
build the image, then developers can directly pull it from the
registry.

Alessandro Di Federico (2):
  docker: Add Hexagon image
  tests/tcg: Use Hexagon Docker image

Philippe Mathieu-Daud=C3=A9 (2):
  gitlab-ci: Pass optional EXTRA_FILES when building docker images
  gitlab-ci: Build Hexagon cross-toolchain

 .gitlab-ci.d/containers.yml                   |  11 ++
 tests/docker/Makefile.include                 |   2 +
 .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
 .../build-toolchain.sh                        | 141 ++++++++++++++++++
 tests/tcg/configure.sh                        |   4 +
 5 files changed, 181 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
 create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/bu=
ild-toolchain.sh

--=20
2.26.2

[PATCH 1/4] docker: Add Hexagon image

[Philippe Mathieu-Daudé]

From: Alessandro Di Federico <ale@rev.ng>

Signed-off-by: Alessandro Di Federico <ale@rev.ng>
[PMD: Base on qemu/debian10, add missing EXTRA_FILES, remove X86]
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                 |   2 +
 .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
 .../build-toolchain.sh                        | 141 ++++++++++++++++++
 3 files changed, 166 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
 create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7cab761bf5b..3aee031c4e9 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -144,6 +144,8 @@ docker-image-debian-riscv64-cross: docker-image-debian10
 docker-image-debian-s390x-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
+docker-image-debian-hexagon-cross: \
+	EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
 
 # Specialist build images, sometimes very limited tools
 docker-image-debian-tricore-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
new file mode 100644
index 00000000000..7aa28767311
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -0,0 +1,23 @@
+FROM qemu/debian10
+
+RUN apt update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt install -y --no-install-recommends \
+        bison \
+        cmake \
+        flex \
+        lld \
+        rsync \
+        wget
+
+ENV TOOLCHAIN_INSTALL /usr
+ENV ROOTFS /usr
+
+ENV LLVM_URL https://github.com/llvm/llvm-project/archive/3d8149c2a1228609fd7d7c91a04681304a2f0ca9.tar.gz
+ENV MUSL_URL https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1af6c0778c.tar.gz
+ENV LINUX_URL https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz
+
+ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh
+
+RUN ( cd /root/hexagon-toolchain && ./build-toolchain.sh ) && \
+    rm -rf /root/hexagon-toolchain
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
new file mode 100755
index 00000000000..19b1c9f83e1
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
@@ -0,0 +1,141 @@
+#!/bin/bash
+
+set -e
+
+BASE=$(readlink -f ${PWD})
+
+TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL")
+ROOTFS=$(readlink -f "$ROOTFS")
+
+TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin
+HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl
+HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr
+
+function cdp() {
+  DIR="$1"
+  mkdir -p "$DIR"
+  cd "$DIR"
+}
+
+function fetch() {
+  DIR="$1"
+  URL="$2"
+  TEMP="$(readlink -f "$PWD/tmp.tar.gz")"
+  wget --quiet "$URL" -O "$TEMP"
+  cdp "$DIR"
+  tar xaf "$TEMP" --strip-components=1
+  rm "$TEMP"
+  cd -
+}
+
+build_llvm_clang() {
+  fetch "$BASE/llvm-project" "$LLVM_URL"
+  cdp "$BASE/build-llvm"
+
+  cmake -G Ninja \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \
+    -DLLVM_ENABLE_LLD=ON \
+    -DLLVM_TARGETS_TO_BUILD="Hexagon" \
+    -DLLVM_ENABLE_PROJECTS="clang;lld" \
+    "$BASE/llvm-project/llvm"
+  ninja all install
+  cd ${TOOLCHAIN_BIN}
+  ln -sf clang hexagon-unknown-linux-musl-clang
+  ln -sf clang++ hexagon-unknown-linux-musl-clang++
+  ln -sf llvm-ar hexagon-unknown-linux-musl-ar
+  ln -sf llvm-objdump hexagon-unknown-linux-musl-objdump
+  ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy
+  ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf
+  ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib
+
+  # workaround for now:
+  cat <<EOF > hexagon-unknown-linux-musl.cfg
+-G0 --sysroot=${HEX_SYSROOT}
+EOF
+}
+
+build_clang_rt() {
+  cdp "$BASE/build-clang_rt"
+  cmake -G Ninja \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \
+    -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-unknown-linux-musl " \
+    -DCMAKE_SYSTEM_NAME=Linux \
+    -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
+    -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
+    -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \
+    -DCMAKE_CROSSCOMPILING=ON \
+    -DCMAKE_C_COMPILER_FORCED=ON \
+    -DCMAKE_CXX_COMPILER_FORCED=ON \
+    -DCOMPILER_RT_BUILD_BUILTINS=ON \
+    -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \
+    -DCMAKE_SIZEOF_VOID_P=4 \
+    -DCOMPILER_RT_OS_DIR= \
+    -DCAN_TARGET_hexagon=1 \
+    -DCAN_TARGET_x86_64=0 \
+    -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \
+    -DLLVM_ENABLE_PROJECTS="compiler-rt" \
+    "$BASE/llvm-project/compiler-rt"
+  ninja install-compiler-rt
+}
+
+build_musl_headers() {
+  fetch "$BASE/musl" "$MUSL_URL"
+  cd "$BASE/musl"
+  make clean
+  CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
+    CROSS_COMPILE=hexagon-unknown-linux-musl \
+    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
+    CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \
+    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
+  PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers
+
+  cd ${HEX_SYSROOT}/..
+  ln -sf hexagon-unknown-linux-musl hexagon
+}
+
+build_kernel_headers() {
+  fetch "$BASE/linux" "$LINUX_URL"
+  mkdir -p "$BASE/build-linux"
+  cd "$BASE/linux"
+  make O=../build-linux ARCH=hexagon \
+   KBUILD_CFLAGS_KERNEL="-mlong-calls" \
+   CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
+   LD=${TOOLCHAIN_BIN}/ld.lld \
+   KBUILD_VERBOSE=1 comet_defconfig
+  make mrproper
+
+  cd "$BASE/build-linux"
+  make \
+    ARCH=hexagon \
+    CC=${TOOLCHAIN_BIN}/clang \
+    INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \
+    V=1 \
+    headers_install
+}
+
+build_musl() {
+  cd "$BASE/musl"
+  make clean
+  CROSS_COMPILE=hexagon-unknown-linux-musl- \
+    AR=llvm-ar \
+    RANLIB=llvm-ranlib \
+    STRIP=llvm-strip \
+    CC=clang \
+    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
+    CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \
+    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
+  PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install
+  cd ${HEX_TOOLS_TARGET_BASE}/lib
+  ln -sf libc.so ld-musl-hexagon.so
+  ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1
+  cdp ${HEX_TOOLS_TARGET_BASE}/../lib
+  ln -sf ../usr/lib/ld-musl-hexagon.so.1
+}
+
+build_llvm_clang
+build_kernel_headers
+build_musl_headers
+build_clang_rt
+build_musl
-- 
2.26.2

[PATCH 2/4] gitlab-ci: Pass optional EXTRA_FILES when building docker images

[Philippe Mathieu-Daudé]

Pass EXTRA_FILES to tests/docker/docker.py to use its --extra-files
command line option.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/containers.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 33e4046e233..7137cc4184d 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -16,6 +16,7 @@
     - ./tests/docker/docker.py --engine docker build
           -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
           -r $CI_REGISTRY_IMAGE
+          $(test -n "$EXTRA_FILES" && echo "--extra-files $EXTRA_FILES")
     - docker tag "qemu/$NAME" "$TAG"
     - docker push "$TAG"
   after_script:
-- 
2.26.2

[PATCH 3/4] gitlab-ci: Build Hexagon cross-toolchain

[Philippe Mathieu-Daudé]

Add a job to build the Debian based Hexagon cross-toolchain image.

This image requires a lot of compute time, too much for the common
shared runners. To avoid having the job to timeout, it has to be
built with custom unlimited runner. For this reason we restrict this
job to manual runs.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/containers.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 7137cc4184d..ed57e02e769 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -103,6 +103,16 @@ armhf-debian-cross-container:
   variables:
     NAME: debian-armhf-cross
 
+hexagon-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  when: manual
+  timeout: 3h
+  variables:
+    NAME: debian-hexagon-cross
+    EXTRA_FILES: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
+
 hppa-debian-cross-container:
   <<: *container_job_definition
   stage: containers-layer2
-- 
2.26.2

[PATCH 4/4] tests/tcg: Use Hexagon Docker image

[Philippe Mathieu-Daudé]

From: Alessandro Di Federico <ale@rev.ng>

Signed-off-by: Alessandro Di Federico <ale@rev.ng>
[PMD: Split from 'Add Hexagon Docker image' patch]
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/tcg/configure.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
index 551c02f4691..489ff8e53e2 100755
--- a/tests/tcg/configure.sh
+++ b/tests/tcg/configure.sh
@@ -124,6 +124,10 @@ for target in $target_list; do
       container_image=fedora-cris-cross
       container_cross_cc=cris-linux-gnu-gcc
       ;;
+    hexagon-*)
+      container_image=debian-hexagon-cross
+      container_cross_cc=hexagon-unknown-linux-musl-clang
+      ;;
     hppa-*)
       container_image=debian-hppa-cross
       container_cross_cc=hppa-linux-gnu-gcc
-- 
2.26.2

Re: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[no-reply]

Patchew URL: https://patchew.org/QEMU/20210228222314.304787-1-f4bug@amsat.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20210228222314.304787-1-f4bug@amsat.org
Subject: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20210227232519.222663-1-richard.henderson@linaro.org -> patchew/20210227232519.222663-1-richard.henderson@linaro.org
 - [tag update]      patchew/20210228050609.24779-1-bmeng.cn@gmail.com -> patchew/20210228050609.24779-1-bmeng.cn@gmail.com
 - [tag update]      patchew/20210228111657.23240-1-ashe@kivikakk.ee -> patchew/20210228111657.23240-1-ashe@kivikakk.ee
 * [new tag]         patchew/20210228222314.304787-1-f4bug@amsat.org -> patchew/20210228222314.304787-1-f4bug@amsat.org
Switched to a new branch 'test'
4307c0a tests/tcg: Use Hexagon Docker image
5979932 gitlab-ci: Build Hexagon cross-toolchain
ef6031b gitlab-ci: Pass optional EXTRA_FILES when building docker images
65568ad docker: Add Hexagon image

=== OUTPUT BEGIN ===
1/4 Checking commit 65568ad6da35 (docker: Add Hexagon image)
Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#31: 
new file mode 100644

WARNING: line over 80 characters
#127: FILE: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh:63:
+    -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-unknown-linux-musl " \

ERROR: line over 90 characters
#154: FILE: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh:90:
+    CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \

ERROR: line over 90 characters
#191: FILE: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh:127:
+    CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \

total: 2 errors, 2 warnings, 172 lines checked

Patch 1/4 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

2/4 Checking commit ef6031b9a8b8 (gitlab-ci: Pass optional EXTRA_FILES when building docker images)
3/4 Checking commit 597993295a13 (gitlab-ci: Build Hexagon cross-toolchain)
4/4 Checking commit 4307c0a16c21 (tests/tcg: Use Hexagon Docker image)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20210228222314.304787-1-f4bug@amsat.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Alessandro Di Federico via]

On Sun, 28 Feb 2021 23:23:10 +0100
Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:

> This series is a rework of the 'Add Dockerfile for hexagon' patch
> from Alessandro/Brian that Taylor sent in v8:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
> but adapted to mainstream.

All of this looks good to me.

Sadly, we've been doing internally exactly the same thing and were close
to push it upstream. We'll be more explicit on what we're doing.

Anyway, thanks for taking care of this!

-- 
Alessandro Di Federico
rev.ng

Re: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Philippe Mathieu-Daudé]

On 3/1/21 2:25 PM, Alessandro Di Federico wrote:
> On Sun, 28 Feb 2021 23:23:10 +0100
> Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> 
>> This series is a rework of the 'Add Dockerfile for hexagon' patch
>> from Alessandro/Brian that Taylor sent in v8:
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
>> but adapted to mainstream.
> 
> All of this looks good to me.
> 
> Sadly, we've been doing internally exactly the same thing and were close
> to push it upstream. We'll be more explicit on what we're doing.

Great. I don't have much time for this, but I'd rather not have
the hexagon port start to bitrot because untested, which is why
I tried to get the testing loop started. I'll let you following
with it then, thanks!

RE: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Taylor Simpson]

> -----Original Message-----
> From: Philippe Mathieu-Daudé <philippe.mathieu.daude@gmail.com> On
> Behalf Of Philippe Mathieu-Daudé
> Sent: Sunday, February 28, 2021 4:23 PM
> To: qemu-devel@nongnu.org
> Cc: Fam Zheng <fam@euphon.net>; Taylor Simpson
> <tsimpson@quicinc.com>; Philippe Mathieu-Daudé <philmd@redhat.com>;
> Alex Bennée <alex.bennee@linaro.org>; Alessandro Di Federico
> <ale.qemu@rev.ng>; Brian Cain <bcain@quicinc.com>; Philippe Mathieu-
> Daudé <f4bug@amsat.org>
> Subject: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci
>
> Hi,
>
> This series is a rework of the 'Add Dockerfile for hexagon' patch
> from Alessandro/Brian that Taylor sent in v8:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
> but adapted to mainstream.
>
> Gitlab shared runner are usually capped to 2/4 cores and timeout
> after 2h30 to 3h. We need a beefy custom runner to get the image
> built and stored in the registry. For now the image is added in
> manual mode, hopping we'll have magic runners added some day to
> build the image, then developers can directly pull it from the
> registry.

Thanks a ton for doing this.

Do I understand correctly that someone will manually run the toolchain build scripts to create the container and others will be able to use the prebuilt container after that?

Taylor

RE: [PATCH 1/4] docker: Add Hexagon image

[Brian Cain]

Reviewed-by: Brian Cain <bcain@quicinc.com>

> -----Original Message-----
> From: Philippe Mathieu-Daudé <philippe.mathieu.daude@gmail.com> On
> Behalf Of Philippe Mathieu-Daudé
> Sent: Sunday, February 28, 2021 4:23 PM
> To: qemu-devel@nongnu.org
> Cc: Fam Zheng <fam@euphon.net>; Taylor Simpson
> <tsimpson@quicinc.com>; Philippe Mathieu-Daudé <philmd@redhat.com>;
> Alex Bennée <alex.bennee@linaro.org>; Alessandro Di Federico
> <ale.qemu@rev.ng>; Brian Cain <bcain@quicinc.com>; Alessandro Di
> Federico <ale@rev.ng>; Philippe Mathieu-Daudé <f4bug@amsat.org>
> Subject: [EXT] [PATCH 1/4] docker: Add Hexagon image
>
> From: Alessandro Di Federico <ale@rev.ng>
>
> Signed-off-by: Alessandro Di Federico <ale@rev.ng>
> [PMD: Base on qemu/debian10, add missing EXTRA_FILES, remove X86]
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include                 |   2 +
>  .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
>  .../build-toolchain.sh                        | 141 ++++++++++++++++++
>  3 files changed, 166 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
>  create mode 100755 tests/docker/dockerfiles/debian-hexagon-
> cross.docker.d/build-toolchain.sh
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 7cab761bf5b..3aee031c4e9 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -144,6 +144,8 @@ docker-image-debian-riscv64-cross: docker-image-
> debian10
>  docker-image-debian-s390x-cross: docker-image-debian10
>  docker-image-debian-sh4-cross: docker-image-debian10
>  docker-image-debian-sparc64-cross: docker-image-debian10
> +docker-image-debian-hexagon-cross: \
> +
> +EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-hexagon-
> cross.
> +docker.d/build-toolchain.sh
>
>  # Specialist build images, sometimes very limited tools
>  docker-image-debian-tricore-cross: docker-image-debian10 diff --git
> a/tests/docker/dockerfiles/debian-hexagon-cross.docker
> b/tests/docker/dockerfiles/debian-hexagon-cross.docker
> new file mode 100644
> index 00000000000..7aa28767311
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
> @@ -0,0 +1,23 @@
> +FROM qemu/debian10
> +
> +RUN apt update && \
> +    DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt install -y --no-install-recommends \
> +        bison \
> +        cmake \
> +        flex \
> +        lld \
> +        rsync \
> +        wget
> +
> +ENV TOOLCHAIN_INSTALL /usr
> +ENV ROOTFS /usr
> +
> +ENV LLVM_URL
> +https://github.com/llvm/llvm-
> project/archive/3d8149c2a1228609fd7d7c91a0
> +4681304a2f0ca9.tar.gz ENV MUSL_URL
> +https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1
> af6
> +c0778c.tar.gz ENV LINUX_URL
> +https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz
> +
> +ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh
> +
> +RUN ( cd /root/hexagon-toolchain && ./build-toolchain.sh ) && \
> +    rm -rf /root/hexagon-toolchain
> diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-
> toolchain.sh b/tests/docker/dockerfiles/debian-hexagon-
> cross.docker.d/build-toolchain.sh
> new file mode 100755
> index 00000000000..19b1c9f83e1
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolc
> +++ hain.sh
> @@ -0,0 +1,141 @@
> +#!/bin/bash
> +
> +set -e
> +
> +BASE=$(readlink -f ${PWD})
> +
> +TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL")
> ROOTFS=$(readlink
> +-f "$ROOTFS")
> +
> +TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin
> +HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl
> +HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr
> +
> +function cdp() {
> +  DIR="$1"
> +  mkdir -p "$DIR"
> +  cd "$DIR"
> +}
> +
> +function fetch() {
> +  DIR="$1"
> +  URL="$2"
> +  TEMP="$(readlink -f "$PWD/tmp.tar.gz")"
> +  wget --quiet "$URL" -O "$TEMP"
> +  cdp "$DIR"
> +  tar xaf "$TEMP" --strip-components=1
> +  rm "$TEMP"
> +  cd -
> +}
> +
> +build_llvm_clang() {
> +  fetch "$BASE/llvm-project" "$LLVM_URL"
> +  cdp "$BASE/build-llvm"
> +
> +  cmake -G Ninja \
> +    -DCMAKE_BUILD_TYPE=Release \
> +    -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \
> +    -DLLVM_ENABLE_LLD=ON \
> +    -DLLVM_TARGETS_TO_BUILD="Hexagon" \
> +    -DLLVM_ENABLE_PROJECTS="clang;lld" \
> +    "$BASE/llvm-project/llvm"
> +  ninja all install
> +  cd ${TOOLCHAIN_BIN}
> +  ln -sf clang hexagon-unknown-linux-musl-clang  ln -sf clang++
> + hexagon-unknown-linux-musl-clang++
> +  ln -sf llvm-ar hexagon-unknown-linux-musl-ar  ln -sf llvm-objdump
> + hexagon-unknown-linux-musl-objdump
> +  ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy
> +  ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf
> +  ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib
> +
> +  # workaround for now:
> +  cat <<EOF > hexagon-unknown-linux-musl.cfg
> +-G0 --sysroot=${HEX_SYSROOT}
> +EOF
> +}
> +
> +build_clang_rt() {
> +  cdp "$BASE/build-clang_rt"
> +  cmake -G Ninja \
> +    -DCMAKE_BUILD_TYPE=Release \
> +    -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \
> +    -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-
> unknown-linux-musl " \
> +    -DCMAKE_SYSTEM_NAME=Linux \
> +    -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-
> musl-clang" \
> +    -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-
> linux-musl-clang" \
> +    -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \
> +    -DCMAKE_CROSSCOMPILING=ON \
> +    -DCMAKE_C_COMPILER_FORCED=ON \
> +    -DCMAKE_CXX_COMPILER_FORCED=ON \
> +    -DCOMPILER_RT_BUILD_BUILTINS=ON \
> +    -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \
> +    -DCMAKE_SIZEOF_VOID_P=4 \
> +    -DCOMPILER_RT_OS_DIR= \
> +    -DCAN_TARGET_hexagon=1 \
> +    -DCAN_TARGET_x86_64=0 \
> +    -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \
> +    -DLLVM_ENABLE_PROJECTS="compiler-rt" \
> +    "$BASE/llvm-project/compiler-rt"
> +  ninja install-compiler-rt
> +}
> +
> +build_musl_headers() {
> +  fetch "$BASE/musl" "$MUSL_URL"
> +  cd "$BASE/musl"
> +  make clean
> +  CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
> +    CROSS_COMPILE=hexagon-unknown-linux-musl \
> +    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
> +    CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --
> target=hexagon-unknown-linux-musl" \
> +    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
> +  PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers
> +
> +  cd ${HEX_SYSROOT}/..
> +  ln -sf hexagon-unknown-linux-musl hexagon }
> +
> +build_kernel_headers() {
> +  fetch "$BASE/linux" "$LINUX_URL"
> +  mkdir -p "$BASE/build-linux"
> +  cd "$BASE/linux"
> +  make O=../build-linux ARCH=hexagon \
> +   KBUILD_CFLAGS_KERNEL="-mlong-calls" \
> +   CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
> +   LD=${TOOLCHAIN_BIN}/ld.lld \
> +   KBUILD_VERBOSE=1 comet_defconfig
> +  make mrproper
> +
> +  cd "$BASE/build-linux"
> +  make \
> +    ARCH=hexagon \
> +    CC=${TOOLCHAIN_BIN}/clang \
> +    INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \
> +    V=1 \
> +    headers_install
> +}
> +
> +build_musl() {
> +  cd "$BASE/musl"
> +  make clean
> +  CROSS_COMPILE=hexagon-unknown-linux-musl- \
> +    AR=llvm-ar \
> +    RANLIB=llvm-ranlib \
> +    STRIP=llvm-strip \
> +    CC=clang \
> +    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
> +    CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --
> target=hexagon-unknown-linux-musl" \
> +    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
> +  PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install
> +  cd ${HEX_TOOLS_TARGET_BASE}/lib
> +  ln -sf libc.so ld-musl-hexagon.so
> +  ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1
> +  cdp ${HEX_TOOLS_TARGET_BASE}/../lib
> +  ln -sf ../usr/lib/ld-musl-hexagon.so.1 }
> +
> +build_llvm_clang
> +build_kernel_headers
> +build_musl_headers
> +build_clang_rt
> +build_musl
> --
> 2.26.2

Re: [PATCH 1/4] docker: Add Hexagon image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> From: Alessandro Di Federico <ale@rev.ng>
>
> Signed-off-by: Alessandro Di Federico <ale@rev.ng>
> [PMD: Base on qemu/debian10, add missing EXTRA_FILES, remove X86]
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include                 |   2 +
>  .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
>  .../build-toolchain.sh                        | 141 ++++++++++++++++++
>  3 files changed, 166 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
>  create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 7cab761bf5b..3aee031c4e9 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -144,6 +144,8 @@ docker-image-debian-riscv64-cross: docker-image-debian10
>  docker-image-debian-s390x-cross: docker-image-debian10
>  docker-image-debian-sh4-cross: docker-image-debian10
>  docker-image-debian-sparc64-cross: docker-image-debian10
> +docker-image-debian-hexagon-cross: \
> +	EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
>  
>  # Specialist build images, sometimes very limited tools
>  docker-image-debian-tricore-cross: docker-image-debian10
> diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
> new file mode 100644
> index 00000000000..7aa28767311
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
> @@ -0,0 +1,23 @@
> +FROM qemu/debian10
> +
> +RUN apt update && \
> +    DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt install -y --no-install-recommends \
> +        bison \
> +        cmake \
> +        flex \
> +        lld \
> +        rsync \
> +        wget
> +
> +ENV TOOLCHAIN_INSTALL /usr
> +ENV ROOTFS /usr
> +
> +ENV LLVM_URL https://github.com/llvm/llvm-project/archive/3d8149c2a1228609fd7d7c91a04681304a2f0ca9.tar.gz
> +ENV MUSL_URL https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1af6c0778c.tar.gz
> +ENV LINUX_URL https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz
> +
> +ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh
> +
> +RUN ( cd /root/hexagon-toolchain && ./build-toolchain.sh ) && \
> +    rm -rf /root/hexagon-toolchain

So this is attempting to everything in one step so we don't inflate the
layers too much?

I believe this is something multi-stage builds are meant to deal with:

  https://docs.docker.com/develop/develop-images/multistage-build/

but I've not gotten one to work in practice. For now I'm trying this out
on my beefy machine and I'll see if I have the permissions required to
upload it to gitlab.

> diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
> new file mode 100755
> index 00000000000..19b1c9f83e1
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
> @@ -0,0 +1,141 @@
> +#!/bin/bash
> +
> +set -e
> +
> +BASE=$(readlink -f ${PWD})
> +
> +TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL")
> +ROOTFS=$(readlink -f "$ROOTFS")
> +
> +TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin
> +HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl
> +HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr
> +
> +function cdp() {
> +  DIR="$1"
> +  mkdir -p "$DIR"
> +  cd "$DIR"
> +}
> +
> +function fetch() {
> +  DIR="$1"
> +  URL="$2"
> +  TEMP="$(readlink -f "$PWD/tmp.tar.gz")"
> +  wget --quiet "$URL" -O "$TEMP"
> +  cdp "$DIR"
> +  tar xaf "$TEMP" --strip-components=1
> +  rm "$TEMP"
> +  cd -
> +}
> +
> +build_llvm_clang() {
> +  fetch "$BASE/llvm-project" "$LLVM_URL"
> +  cdp "$BASE/build-llvm"
> +
> +  cmake -G Ninja \
> +    -DCMAKE_BUILD_TYPE=Release \
> +    -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \
> +    -DLLVM_ENABLE_LLD=ON \
> +    -DLLVM_TARGETS_TO_BUILD="Hexagon" \
> +    -DLLVM_ENABLE_PROJECTS="clang;lld" \
> +    "$BASE/llvm-project/llvm"
> +  ninja all install
> +  cd ${TOOLCHAIN_BIN}
> +  ln -sf clang hexagon-unknown-linux-musl-clang
> +  ln -sf clang++ hexagon-unknown-linux-musl-clang++
> +  ln -sf llvm-ar hexagon-unknown-linux-musl-ar
> +  ln -sf llvm-objdump hexagon-unknown-linux-musl-objdump
> +  ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy
> +  ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf
> +  ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib
> +
> +  # workaround for now:
> +  cat <<EOF > hexagon-unknown-linux-musl.cfg
> +-G0 --sysroot=${HEX_SYSROOT}
> +EOF
> +}
> +
> +build_clang_rt() {
> +  cdp "$BASE/build-clang_rt"
> +  cmake -G Ninja \
> +    -DCMAKE_BUILD_TYPE=Release \
> +    -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \
> +    -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic --target=hexagon-unknown-linux-musl " \
> +    -DCMAKE_SYSTEM_NAME=Linux \
> +    -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
> +    -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
> +    -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \
> +    -DCMAKE_CROSSCOMPILING=ON \
> +    -DCMAKE_C_COMPILER_FORCED=ON \
> +    -DCMAKE_CXX_COMPILER_FORCED=ON \
> +    -DCOMPILER_RT_BUILD_BUILTINS=ON \
> +    -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \
> +    -DCMAKE_SIZEOF_VOID_P=4 \
> +    -DCOMPILER_RT_OS_DIR= \
> +    -DCAN_TARGET_hexagon=1 \
> +    -DCAN_TARGET_x86_64=0 \
> +    -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \
> +    -DLLVM_ENABLE_PROJECTS="compiler-rt" \
> +    "$BASE/llvm-project/compiler-rt"
> +  ninja install-compiler-rt
> +}
> +
> +build_musl_headers() {
> +  fetch "$BASE/musl" "$MUSL_URL"
> +  cd "$BASE/musl"
> +  make clean
> +  CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
> +    CROSS_COMPILE=hexagon-unknown-linux-musl \
> +    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
> +    CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \
> +    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
> +  PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers
> +
> +  cd ${HEX_SYSROOT}/..
> +  ln -sf hexagon-unknown-linux-musl hexagon
> +}
> +
> +build_kernel_headers() {
> +  fetch "$BASE/linux" "$LINUX_URL"
> +  mkdir -p "$BASE/build-linux"
> +  cd "$BASE/linux"
> +  make O=../build-linux ARCH=hexagon \
> +   KBUILD_CFLAGS_KERNEL="-mlong-calls" \
> +   CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
> +   LD=${TOOLCHAIN_BIN}/ld.lld \
> +   KBUILD_VERBOSE=1 comet_defconfig
> +  make mrproper
> +
> +  cd "$BASE/build-linux"
> +  make \
> +    ARCH=hexagon \
> +    CC=${TOOLCHAIN_BIN}/clang \
> +    INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \
> +    V=1 \
> +    headers_install
> +}
> +
> +build_musl() {
> +  cd "$BASE/musl"
> +  make clean
> +  CROSS_COMPILE=hexagon-unknown-linux-musl- \
> +    AR=llvm-ar \
> +    RANLIB=llvm-ranlib \
> +    STRIP=llvm-strip \
> +    CC=clang \
> +    LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
> +    CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math --target=hexagon-unknown-linux-musl" \
> +    ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
> +  PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install
> +  cd ${HEX_TOOLS_TARGET_BASE}/lib
> +  ln -sf libc.so ld-musl-hexagon.so
> +  ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1
> +  cdp ${HEX_TOOLS_TARGET_BASE}/../lib
> +  ln -sf ../usr/lib/ld-musl-hexagon.so.1
> +}
> +
> +build_llvm_clang
> +build_kernel_headers
> +build_musl_headers
> +build_clang_rt
> +build_musl


-- 
Alex Bennée

Re: [PATCH 3/4] gitlab-ci: Build Hexagon cross-toolchain

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Add a job to build the Debian based Hexagon cross-toolchain image.
>
> This image requires a lot of compute time, too much for the common
> shared runners. To avoid having the job to timeout, it has to be
> built with custom unlimited runner. For this reason we restrict this
> job to manual runs.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/containers.yml | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> index 7137cc4184d..ed57e02e769 100644
> --- a/.gitlab-ci.d/containers.yml
> +++ b/.gitlab-ci.d/containers.yml
> @@ -103,6 +103,16 @@ armhf-debian-cross-container:
>    variables:
>      NAME: debian-armhf-cross
>  
> +hexagon-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  when: manual
> +  timeout: 3h
> +  variables:
> +    NAME: debian-hexagon-cross
> +    EXTRA_FILES: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
> +

We don't really want this - what we need is the ability to run the tests
for hexagon when the registry contains a built container that we have
manually pushed from a build machine:

  https://gitlab.com/qemu-project/qemu/container_registry/1759846

>  hppa-debian-cross-container:
>    <<: *container_job_definition
>    stage: containers-layer2


-- 
Alex Bennée

Re: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
> This series is a rework of the 'Add Dockerfile for hexagon' patch
> from Alessandro/Brian that Taylor sent in v8:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
> but adapted to mainstream.

Queued patches 1 and 2 to testing/next, thanks.

>
> Gitlab shared runner are usually capped to 2/4 cores and timeout
> after 2h30 to 3h. We need a beefy custom runner to get the image
> built and stored in the registry. For now the image is added in
> manual mode, hopping we'll have magic runners added some day to
> build the image, then developers can directly pull it from the
> registry.
>
> Alessandro Di Federico (2):
>   docker: Add Hexagon image
>   tests/tcg: Use Hexagon Docker image
>
> Philippe Mathieu-Daud=C3=A9 (2):
>   gitlab-ci: Pass optional EXTRA_FILES when building docker images
>   gitlab-ci: Build Hexagon cross-toolchain
>
>  .gitlab-ci.d/containers.yml                   |  11 ++
>  tests/docker/Makefile.include                 |   2 +
>  .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
>  .../build-toolchain.sh                        | 141 ++++++++++++++++++
>  tests/tcg/configure.sh                        |   4 +
>  5 files changed, 181 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
>  create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/bu=
> ild-toolchain.sh
>
> --=20
> 2.26.2


-- 
Alex Bennée

Re: [PATCH 0/4] hexagon: Add Docker image & testing to gitlab-ci

[Alex Bennée]

I of course meant 1 and 4 (skipping the gitlab bits)

On Thu, 4 Mar 2021 at 13:45, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
> > Hi,
> >
> > This series is a rework of the 'Add Dockerfile for hexagon' patch
> > from Alessandro/Brian that Taylor sent in v8:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg780330.html
> > but adapted to mainstream.
>
> Queued patches 1 and 2 to testing/next, thanks.
>
> >
> > Gitlab shared runner are usually capped to 2/4 cores and timeout
> > after 2h30 to 3h. We need a beefy custom runner to get the image
> > built and stored in the registry. For now the image is added in
> > manual mode, hopping we'll have magic runners added some day to
> > build the image, then developers can directly pull it from the
> > registry.
> >
> > Alessandro Di Federico (2):
> >   docker: Add Hexagon image
> >   tests/tcg: Use Hexagon Docker image
> >
> > Philippe Mathieu-Daud=C3=A9 (2):
> >   gitlab-ci: Pass optional EXTRA_FILES when building docker images
> >   gitlab-ci: Build Hexagon cross-toolchain
> >
> >  .gitlab-ci.d/containers.yml                   |  11 ++
> >  tests/docker/Makefile.include                 |   2 +
> >  .../dockerfiles/debian-hexagon-cross.docker   |  23 +++
> >  .../build-toolchain.sh                        | 141 ++++++++++++++++++
> >  tests/tcg/configure.sh                        |   4 +
> >  5 files changed, 181 insertions(+)
> >  create mode 100644 tests/docker/dockerfiles/debian-hexagon-cross.docker
> >  create mode 100755 tests/docker/dockerfiles/debian-hexagon-cross.docker.d/bu=
> > ild-toolchain.sh
> >
> > --=20
> > 2.26.2
>
>
> --
> Alex Bennée



-- 
Alex Bennée
KVM/QEMU Hacker for Linaro

Re: [PATCH 3/4] gitlab-ci: Build Hexagon cross-toolchain

[Paolo Montesel via]

On Thu, Mar 4, 2021 at 1:10 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
> > Add a job to build the Debian based Hexagon cross-toolchain image.
> >
> > This image requires a lot of compute time, too much for the common
> > shared runners. To avoid having the job to timeout, it has to be
> > built with custom unlimited runner. For this reason we restrict this
> > job to manual runs.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> >  .gitlab-ci.d/containers.yml | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > index 7137cc4184d..ed57e02e769 100644
> > --- a/.gitlab-ci.d/containers.yml
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -103,6 +103,16 @@ armhf-debian-cross-container:
> >    variables:
> >      NAME: debian-armhf-cross
> >
> > +hexagon-debian-cross-container:
> > +  <<: *container_job_definition
> > +  stage: containers-layer2
> > +  needs: ['amd64-debian10-container']
> > +  when: manual
> > +  timeout: 3h
> > +  variables:
> > +    NAME: debian-hexagon-cross
> > +    EXTRA_FILES: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
> > +
>
> We don't really want this - what we need is the ability to run the tests
> for hexagon when the registry contains a built container that we have
> manually pushed from a build machine:

Ok, so, just to sync, should we submit a minimal patch that only
addresses the docker part of our stuff and then worry about the rest?
That way you can build & upload a container to the registry, and then
we can add a job to .gitlab-ci.yml with the rest of the patchset.

Or, if you plan to mess with the CI yourself, please tell me so we can
avoid spending time on the same thing (should be 5 lines of yaml
anyway).

- Paolo

Re: [PATCH 3/4] gitlab-ci: Build Hexagon cross-toolchain

[Alex Bennée]

Yes, exactly. Already done:

  https://gitlab.com/stsquad/qemu/-/jobs/1073206662

On Thu, 4 Mar 2021 at 17:43, Paolo Montesel <babush@rev.ng> wrote:
>
> On Thu, Mar 4, 2021 at 1:10 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >
> > Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >
> > > Add a job to build the Debian based Hexagon cross-toolchain image.
> > >
> > > This image requires a lot of compute time, too much for the common
> > > shared runners. To avoid having the job to timeout, it has to be
> > > built with custom unlimited runner. For this reason we restrict this
> > > job to manual runs.
> > >
> > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > ---
> > >  .gitlab-ci.d/containers.yml | 10 ++++++++++
> > >  1 file changed, 10 insertions(+)
> > >
> > > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > > index 7137cc4184d..ed57e02e769 100644
> > > --- a/.gitlab-ci.d/containers.yml
> > > +++ b/.gitlab-ci.d/containers.yml
> > > @@ -103,6 +103,16 @@ armhf-debian-cross-container:
> > >    variables:
> > >      NAME: debian-armhf-cross
> > >
> > > +hexagon-debian-cross-container:
> > > +  <<: *container_job_definition
> > > +  stage: containers-layer2
> > > +  needs: ['amd64-debian10-container']
> > > +  when: manual
> > > +  timeout: 3h
> > > +  variables:
> > > +    NAME: debian-hexagon-cross
> > > +    EXTRA_FILES: tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
> > > +
> >
> > We don't really want this - what we need is the ability to run the tests
> > for hexagon when the registry contains a built container that we have
> > manually pushed from a build machine:
>
> Ok, so, just to sync, should we submit a minimal patch that only
> addresses the docker part of our stuff and then worry about the rest?
> That way you can build & upload a container to the registry, and then
> we can add a job to .gitlab-ci.yml with the rest of the patchset.
>
> Or, if you plan to mess with the CI yourself, please tell me so we can
> avoid spending time on the same thing (should be 5 lines of yaml
> anyway).
>
> - Paolo



-- 
Alex Bennée
KVM/QEMU Hacker for Linaro