meta-arm.lists.yoctoproject.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 1/2] ci: make get-binary-toolchain run per build
@ 2021-09-14 15:43 Jon Mason
  2021-09-14 15:43 ` [PATCH v2 2/2] ci: update binary gcc version to 10.3 Jon Mason
  2021-09-15  5:59 ` [meta-arm] [PATCH v2 1/2] ci: make get-binary-toolchain run per build Sumit Garg
  0 siblings, 2 replies; 3+ messages in thread
From: Jon Mason @ 2021-09-14 15:43 UTC (permalink / raw)
  To: meta-arm

In a distributed, non-homogeneous CI setup, the binary-toolchain setup
script might not run on the machine that needs the toolchains.  Make
this per-build and it will always be there, at the expense of running on
builds that might not need it (though it still should be fast).

Also, there is an issue with the directory where the binary toolchain is
located being global, and racing against other systems using, setting up,
and tearing down.  Link this to a local directory to avoid any races.

Signed-off-by: Jon Mason <jon.mason@arm.com>
---
 .gitlab-ci.yml           | 12 ++++--------
 ci/external-gccarm.yml   |  2 +-
 ci/get-binary-toolchains | 24 +++++++++---------------
 3 files changed, 14 insertions(+), 24 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 697603b..ae2b346 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,12 +18,15 @@ stages:
     BB_LOGCONFIG: $CI_PROJECT_DIR/ci/logging.yml
     TOOLCHAIN_DIR: $CI_BUILDS_DIR/persist/toolchains
     IMAGE_DIR: $CI_PROJECT_DIR/work/build/tmp/deploy/images
+    TOOLCHAIN_LINK_DIR: $CI_PROJECT_DIR/work/build/toolchains
   before_script:
     - echo KAS_WORK_DIR = $KAS_WORK_DIR
     - echo SSTATE_DIR = $SSTATE_DIR
     - echo DL_DIR = $DL_DIR
     - rm -rf $KAS_WORK_DIR
-    - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR
+    - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
+    # Must do this here, as it's the only way to make sure the toolchain is installed on the same builder
+    - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
 
 # Generalised fragment to do a Kas build
 .build:
@@ -43,13 +46,6 @@ update-repos:
   script:
   - flock --verbose --timeout 60 $KAS_REPO_REF_DIR ./ci/update-repos
 
-get-binary-toolchains:
-  extends: .setup
-  stage: prep
-  script:
-    - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR
-
-
 #
 # Bootstrap stage, bootstrap and machine coverage
 #
diff --git a/ci/external-gccarm.yml b/ci/external-gccarm.yml
index ad03298..0ce4795 100644
--- a/ci/external-gccarm.yml
+++ b/ci/external-gccarm.yml
@@ -5,4 +5,4 @@ local_conf_header:
   cc: |
     PNBLACKLIST[gcc-cross-arm] = "Using external toolchain"
     TCMODE = "external-arm"
-    EXTERNAL_TOOLCHAIN = "${TOOLCHAIN_DIR}/${TARGET_ARCH}"
+    EXTERNAL_TOOLCHAIN = "${TOPDIR}/toolchains/${TARGET_ARCH}"
diff --git a/ci/get-binary-toolchains b/ci/get-binary-toolchains
index 6fa7ff4..e860ec2 100755
--- a/ci/get-binary-toolchains
+++ b/ci/get-binary-toolchains
@@ -6,9 +6,10 @@ VER="10.2-2020.11"
 
 DOWNLOAD_DIR=$1
 TOOLCHAIN_DIR=$2
+TOOLCHAIN_LINK_DIR=$3
 
-# These should be already created by .bitlab-ci.yml, but do here if run outside of that env
-mkdir -p $DOWNLOAD_DIR $TOOLCHAIN_DIR
+# These should be already created by .gitlab-ci.yml, but do here if run outside of that env
+mkdir -p $DOWNLOAD_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
 
 if [ $HOST_ARCH = "aarch64" ]; then
 	#AArch64 Linux hosted cross compilers
@@ -32,21 +33,14 @@ else
 fi
 
 for i in arm aarch64 aarch64_be; do
-	if [ ! -f $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz ]; then
-		continue
-	fi
-
-	if [ -d $TOOLCHAIN_DIR/$i ]; then
-		echo "$TOOLCHAIN_DIR/$i EXISTS!"
-		MANIFEST=$(ls $TOOLCHAIN_DIR/$i | grep txt)
-		if [[ $MANIFEST != $VER-$HOST_ARCH-$i-none-linux-gnu*.txt ]]; then
-			echo "Removing old $MANIFEST for $VER-$HOST_ARCH-$i-*.txt toolchain"
-			rm -rf $TOOLCHAIN_DIR/$i
+	if [ ! -d $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*/ ]; then
+		if [ ! -f $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz ]; then
+			continue
 		fi
-	fi
 
-	if [ ! -d $TOOLCHAIN_DIR/$i ]; then
 		tar -C $TOOLCHAIN_DIR -axvf $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz
-		mv $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*/ $TOOLCHAIN_DIR/$i
 	fi
+
+	# Setup a link for the toolchain to use local to the building machine (e.g., not in a shared location)
+	ln -s $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu* $TOOLCHAIN_LINK_DIR/$i
 done
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 3+ messages in thread
* [PATCH v2 2/2] ci: update binary gcc version to 10.3
  2021-09-14 15:43 [PATCH v2 1/2] ci: make get-binary-toolchain run per build Jon Mason
@ 2021-09-14 15:43 ` Jon Mason
  2021-09-15  5:59 ` [meta-arm] [PATCH v2 1/2] ci: make get-binary-toolchain run per build Sumit Garg
  1 sibling, 0 replies; 3+ messages in thread
From: Jon Mason @ 2021-09-14 15:43 UTC (permalink / raw)
  To: meta-arm

Signed-off-by: Jon Mason <jon.mason@arm.com>
---
 ci/get-binary-toolchains | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/get-binary-toolchains b/ci/get-binary-toolchains
index e860ec2..838342a 100755
--- a/ci/get-binary-toolchains
+++ b/ci/get-binary-toolchains
@@ -2,7 +2,7 @@
 set -u
 
 HOST_ARCH=$(uname -m)
-VER="10.2-2020.11"
+VER="10.3-2021.07"
 
 DOWNLOAD_DIR=$1
 TOOLCHAIN_DIR=$2
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 3+ messages in thread
* Re: [meta-arm] [PATCH v2 1/2] ci: make get-binary-toolchain run per build
  2021-09-14 15:43 [PATCH v2 1/2] ci: make get-binary-toolchain run per build Jon Mason
  2021-09-14 15:43 ` [PATCH v2 2/2] ci: update binary gcc version to 10.3 Jon Mason
@ 2021-09-15  5:59 ` Sumit Garg
  1 sibling, 0 replies; 3+ messages in thread
From: Sumit Garg @ 2021-09-15  5:59 UTC (permalink / raw)
  To: Jon Mason; +Cc: meta-arm

On Tue, 14 Sept 2021 at 21:13, Jon Mason <jon.mason@arm.com> wrote:
>
> In a distributed, non-homogeneous CI setup, the binary-toolchain setup
> script might not run on the machine that needs the toolchains.  Make
> this per-build and it will always be there, at the expense of running on
> builds that might not need it (though it still should be fast).
>
> Also, there is an issue with the directory where the binary toolchain is
> located being global, and racing against other systems using, setting up,
> and tearing down.  Link this to a local directory to avoid any races.
>
> Signed-off-by: Jon Mason <jon.mason@arm.com>
> ---
>  .gitlab-ci.yml           | 12 ++++--------
>  ci/external-gccarm.yml   |  2 +-
>  ci/get-binary-toolchains | 24 +++++++++---------------
>  3 files changed, 14 insertions(+), 24 deletions(-)
>

For the series:
Acked-by: Sumit Garg <sumit.garg@linaro.org>

-Sumit

> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 697603b..ae2b346 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -18,12 +18,15 @@ stages:
>      BB_LOGCONFIG: $CI_PROJECT_DIR/ci/logging.yml
>      TOOLCHAIN_DIR: $CI_BUILDS_DIR/persist/toolchains
>      IMAGE_DIR: $CI_PROJECT_DIR/work/build/tmp/deploy/images
> +    TOOLCHAIN_LINK_DIR: $CI_PROJECT_DIR/work/build/toolchains
>    before_script:
>      - echo KAS_WORK_DIR = $KAS_WORK_DIR
>      - echo SSTATE_DIR = $SSTATE_DIR
>      - echo DL_DIR = $DL_DIR
>      - rm -rf $KAS_WORK_DIR
> -    - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR
> +    - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
> +    # Must do this here, as it's the only way to make sure the toolchain is installed on the same builder
> +    - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
>
>  # Generalised fragment to do a Kas build
>  .build:
> @@ -43,13 +46,6 @@ update-repos:
>    script:
>    - flock --verbose --timeout 60 $KAS_REPO_REF_DIR ./ci/update-repos
>
> -get-binary-toolchains:
> -  extends: .setup
> -  stage: prep
> -  script:
> -    - ./ci/get-binary-toolchains $DL_DIR $TOOLCHAIN_DIR
> -
> -
>  #
>  # Bootstrap stage, bootstrap and machine coverage
>  #
> diff --git a/ci/external-gccarm.yml b/ci/external-gccarm.yml
> index ad03298..0ce4795 100644
> --- a/ci/external-gccarm.yml
> +++ b/ci/external-gccarm.yml
> @@ -5,4 +5,4 @@ local_conf_header:
>    cc: |
>      PNBLACKLIST[gcc-cross-arm] = "Using external toolchain"
>      TCMODE = "external-arm"
> -    EXTERNAL_TOOLCHAIN = "${TOOLCHAIN_DIR}/${TARGET_ARCH}"
> +    EXTERNAL_TOOLCHAIN = "${TOPDIR}/toolchains/${TARGET_ARCH}"
> diff --git a/ci/get-binary-toolchains b/ci/get-binary-toolchains
> index 6fa7ff4..e860ec2 100755
> --- a/ci/get-binary-toolchains
> +++ b/ci/get-binary-toolchains
> @@ -6,9 +6,10 @@ VER="10.2-2020.11"
>
>  DOWNLOAD_DIR=$1
>  TOOLCHAIN_DIR=$2
> +TOOLCHAIN_LINK_DIR=$3
>
> -# These should be already created by .bitlab-ci.yml, but do here if run outside of that env
> -mkdir -p $DOWNLOAD_DIR $TOOLCHAIN_DIR
> +# These should be already created by .gitlab-ci.yml, but do here if run outside of that env
> +mkdir -p $DOWNLOAD_DIR $TOOLCHAIN_DIR $TOOLCHAIN_LINK_DIR
>
>  if [ $HOST_ARCH = "aarch64" ]; then
>         #AArch64 Linux hosted cross compilers
> @@ -32,21 +33,14 @@ else
>  fi
>
>  for i in arm aarch64 aarch64_be; do
> -       if [ ! -f $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz ]; then
> -               continue
> -       fi
> -
> -       if [ -d $TOOLCHAIN_DIR/$i ]; then
> -               echo "$TOOLCHAIN_DIR/$i EXISTS!"
> -               MANIFEST=$(ls $TOOLCHAIN_DIR/$i | grep txt)
> -               if [[ $MANIFEST != $VER-$HOST_ARCH-$i-none-linux-gnu*.txt ]]; then
> -                       echo "Removing old $MANIFEST for $VER-$HOST_ARCH-$i-*.txt toolchain"
> -                       rm -rf $TOOLCHAIN_DIR/$i
> +       if [ ! -d $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*/ ]; then
> +               if [ ! -f $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz ]; then
> +                       continue
>                 fi
> -       fi
>
> -       if [ ! -d $TOOLCHAIN_DIR/$i ]; then
>                 tar -C $TOOLCHAIN_DIR -axvf $DOWNLOAD_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*.tar.xz
> -               mv $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu*/ $TOOLCHAIN_DIR/$i
>         fi
> +
> +       # Setup a link for the toolchain to use local to the building machine (e.g., not in a shared location)
> +       ln -s $TOOLCHAIN_DIR/gcc-arm-$VER-$HOST_ARCH-$i-none-linux-gnu* $TOOLCHAIN_LINK_DIR/$i
>  done
> --
> 2.17.1
>
>
> 
>

^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-09-15  5:59 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-14 15:43 [PATCH v2 1/2] ci: make get-binary-toolchain run per build Jon Mason
2021-09-14 15:43 ` [PATCH v2 2/2] ci: update binary gcc version to 10.3 Jon Mason
2021-09-15  5:59 ` [meta-arm] [PATCH v2 1/2] ci: make get-binary-toolchain run per build Sumit Garg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).