Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Wei Liu <wl@xen.org>]

On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> GitLab have a caching capability, see [1]. Let's use it to avoid using
> Internet too often.
> 
> The cache is setup so that when xen.git/Config.mk is changed, the
> cache will need to be recreated. This has been chosen because that is
> where the information about how to clone sub-project trees is encoded
> (revisions). That may not work for qemu-xen tree which usually is
> `master', but that should be fine for now.
> 
> The cache is populated of "git bundle" which will contain a mirror of
> the original repo, and can be cloned from. If the bundle exist, the
> script have the Xen makefiles clone from it, otherwise it will clone
> from the original URL and the bundles will be created just after.
> 
> We have more than one runner in GitLab, and no shared cache between
> them, so every build jobs will be responsible to create the cache.
> 
> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> 
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>

This is a good improvement.

Have you run this in Gitlab CI? Can you point me to a run?

> ---
>  automation/gitlab-ci/build.yaml     |  8 +++++
>  automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
>  create mode 100755 automation/scripts/prepare-cache.sh
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 1e61d30c8545..8f9f53a4222f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -1,6 +1,14 @@
>  .build-tmpl: &build
>    stage: build
>    image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> +  cache:
> +    key:
> +      files:
> +        - Config.mk
> +    paths:
> +      - ci_cache
> +  before_script:
> +    - ./automation/scripts/prepare-cache.sh
>    script:
>      - ./automation/scripts/build 2>&1 | tee build.log
>    artifacts:
> diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> new file mode 100755
> index 000000000000..017f1b8f0672
> --- /dev/null
> +++ b/automation/scripts/prepare-cache.sh
> @@ -0,0 +1,52 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> +mkdir -p "$cachedir"
> +
> +declare -A r
> +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL

Does this mean if in the future we add or remove trees we will need to
modify this part in the same commit?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel