[Xen-devel] [XEN PATCH 0/2] Start using GitLab caching capability

[Xen-devel] [XEN PATCH 0/2] Start using GitLab caching capability

[Anthony PERARD]

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.ci-caching-v1

This is only a small improvement to the GitLab CI. It only caches git clone
that Xen's makefile creates and not submodules needed by those clones.

It doesn't cache the different tarballs that the makefiles downloads, that
could be another improvement.

Cheers,

Anthony PERARD (2):
  tools: Allow to make *-dir-force-update without ./configure
  automation: Cache sub-project git tree in build jobs

 automation/gitlab-ci/build.yaml     |  8 +++++
 automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++
 tools/Rules.mk                      |  3 +-
 3 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100755 automation/scripts/prepare-cache.sh

-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [XEN PATCH 1/2] tools: Allow to make *-dir-force-update without ./configure

[Anthony PERARD]

This also allows to run `make src-tarball` without first having to run
`./configure`.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 tools/Rules.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/Rules.mk b/tools/Rules.mk
index cf8935d6a3ea..31cf419ef4f5 100644
--- a/tools/Rules.mk
+++ b/tools/Rules.mk
@@ -239,7 +239,8 @@ subdir-all-% subdir-clean-% subdir-install-% subdir-uninstall-%: .phony
 subdir-distclean-%: .phony
 	$(MAKE) -C $* distclean
 
-ifeq (,$(findstring clean,$(MAKECMDGOALS)))
+no-configure-targets := clean subtree-force-update-all %-dir-force-update
+ifeq (,$(filter $(no-configure-targets),$(MAKECMDGOALS)))
 $(XEN_ROOT)/config/Tools.mk:
 	$(error You have to run ./configure before building or installing the tools)
 endif
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [XEN PATCH 1/2] tools: Allow to make *-dir-force-update without ./configure

[Wei Liu]

On Thu, Dec 19, 2019 at 02:42:16PM +0000, Anthony PERARD wrote:
> This also allows to run `make src-tarball` without first having to run
> `./configure`.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>

Acked-by: Wei Liu <wl@xen.org>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Anthony PERARD]

GitLab have a caching capability, see [1]. Let's use it to avoid using
Internet too often.

The cache is setup so that when xen.git/Config.mk is changed, the
cache will need to be recreated. This has been chosen because that is
where the information about how to clone sub-project trees is encoded
(revisions). That may not work for qemu-xen tree which usually is
`master', but that should be fine for now.

The cache is populated of "git bundle" which will contain a mirror of
the original repo, and can be cloned from. If the bundle exist, the
script have the Xen makefiles clone from it, otherwise it will clone
from the original URL and the bundles will be created just after.

We have more than one runner in GitLab, and no shared cache between
them, so every build jobs will be responsible to create the cache.

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/gitlab-ci/build.yaml     |  8 +++++
 automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100755 automation/scripts/prepare-cache.sh

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 1e61d30c8545..8f9f53a4222f 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -1,6 +1,14 @@
 .build-tmpl: &build
   stage: build
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  cache:
+    key:
+      files:
+        - Config.mk
+    paths:
+      - ci_cache
+  before_script:
+    - ./automation/scripts/prepare-cache.sh
   script:
     - ./automation/scripts/build 2>&1 | tee build.log
   artifacts:
diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
new file mode 100755
index 000000000000..017f1b8f0672
--- /dev/null
+++ b/automation/scripts/prepare-cache.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+
+set -ex
+
+cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
+mkdir -p "$cachedir"
+
+declare -A r
+r[extras/mini-os]=MINIOS_UPSTREAM_URL
+r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
+r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
+r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
+r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
+
+bundle_loc() {
+    echo "$cachedir/${1//\//_}.git.bundle"
+}
+for d in ${!r[@]}; do
+    if [ -e $(bundle_loc $d) ]; then
+        export ${r[$d]}=$(bundle_loc $d)
+    fi
+done
+
+if ! make subtree-force-update-all; then
+    # There's maybe an issue with one of the git bundle, just clear the cache
+    # and allow it to be rebuilt by a different jobs.
+    # Make will reclone missing clones from original URLs instead of from the
+    # bundle.
+    for d in ${!r[@]}; do
+        rm -f "$(bundle_loc $d)"
+    done
+    exit
+fi
+
+
+tmpdir=$(mktemp -d "$CI_PROJECT_DIR/ci-tmp.XXX")
+for d in ${!r[@]}; do
+    bundle=$(bundle_loc $d)
+    if [ -e $bundle ]; then
+        # We didn't download anything new
+        continue
+    fi
+    # We create a mirror to be able to create a bundle that is a mirror of
+    # upstream. Otherwise, the bundle may not have refs that the build system
+    # will want, i.e. refs/heads/master would be missing from the bundle.
+    url=$(git --git-dir=$d/.git config remote.origin.url)
+    repo_mirrored="$tmpdir/${d//\//_}"
+    git clone --bare --mirror --reference "$d" "$url" "$repo_mirrored"
+    git --git-dir="$repo_mirrored" bundle create $bundle --all
+    rm -rf "$repo_mirrored"
+done
+rmdir "$tmpdir"
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Wei Liu]

On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> GitLab have a caching capability, see [1]. Let's use it to avoid using
> Internet too often.
> 
> The cache is setup so that when xen.git/Config.mk is changed, the
> cache will need to be recreated. This has been chosen because that is
> where the information about how to clone sub-project trees is encoded
> (revisions). That may not work for qemu-xen tree which usually is
> `master', but that should be fine for now.
> 
> The cache is populated of "git bundle" which will contain a mirror of
> the original repo, and can be cloned from. If the bundle exist, the
> script have the Xen makefiles clone from it, otherwise it will clone
> from the original URL and the bundles will be created just after.
> 
> We have more than one runner in GitLab, and no shared cache between
> them, so every build jobs will be responsible to create the cache.
> 
> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> 
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>

This is a good improvement.

Have you run this in Gitlab CI? Can you point me to a run?

> ---
>  automation/gitlab-ci/build.yaml     |  8 +++++
>  automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
>  create mode 100755 automation/scripts/prepare-cache.sh
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 1e61d30c8545..8f9f53a4222f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -1,6 +1,14 @@
>  .build-tmpl: &build
>    stage: build
>    image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> +  cache:
> +    key:
> +      files:
> +        - Config.mk
> +    paths:
> +      - ci_cache
> +  before_script:
> +    - ./automation/scripts/prepare-cache.sh
>    script:
>      - ./automation/scripts/build 2>&1 | tee build.log
>    artifacts:
> diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> new file mode 100755
> index 000000000000..017f1b8f0672
> --- /dev/null
> +++ b/automation/scripts/prepare-cache.sh
> @@ -0,0 +1,52 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> +mkdir -p "$cachedir"
> +
> +declare -A r
> +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL

Does this mean if in the future we add or remove trees we will need to
modify this part in the same commit?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Anthony PERARD]

On Fri, Jan 03, 2020 at 02:29:07PM +0000, Wei Liu wrote:
> On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > Internet too often.
> > 
> > The cache is setup so that when xen.git/Config.mk is changed, the
> > cache will need to be recreated. This has been chosen because that is
> > where the information about how to clone sub-project trees is encoded
> > (revisions). That may not work for qemu-xen tree which usually is
> > `master', but that should be fine for now.
> > 
> > The cache is populated of "git bundle" which will contain a mirror of
> > the original repo, and can be cloned from. If the bundle exist, the
> > script have the Xen makefiles clone from it, otherwise it will clone
> > from the original URL and the bundles will be created just after.
> > 
> > We have more than one runner in GitLab, and no shared cache between
> > them, so every build jobs will be responsible to create the cache.
> > 
> > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> 
> This is a good improvement.
> 
> Have you run this in Gitlab CI? Can you point me to a run?

I have use the CI to develop the patch, so yes I have a run of it. But
it is a run made with my wip branch, still it should be the same result
if it was done with the final patch:
https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621

> > diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> > new file mode 100755
> > index 000000000000..017f1b8f0672
> > --- /dev/null
> > +++ b/automation/scripts/prepare-cache.sh
> > @@ -0,0 +1,52 @@
> > +#!/bin/bash
> > +
> > +set -ex
> > +
> > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > +mkdir -p "$cachedir"
> > +
> > +declare -A r
> > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
> 
> Does this mean if in the future we add or remove trees we will need to
> modify this part in the same commit?

We would need to modify the script when trees are removed, because I
haven't thought of that. But when trees are added, the script can be
changed in a follow-up.

Ideally, we would use the Makefiles to discovers the git clones that can
be cached, but that's not possible just yet.

In the mean time, I think I should make the script more robust against
removal of trees, so it doesn't have to be modified in the same commit.

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Wei Liu]

On Mon, Jan 06, 2020 at 02:34:02PM +0000, Anthony PERARD wrote:
> On Fri, Jan 03, 2020 at 02:29:07PM +0000, Wei Liu wrote:
> > On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> > > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > > Internet too often.
> > > 
> > > The cache is setup so that when xen.git/Config.mk is changed, the
> > > cache will need to be recreated. This has been chosen because that is
> > > where the information about how to clone sub-project trees is encoded
> > > (revisions). That may not work for qemu-xen tree which usually is
> > > `master', but that should be fine for now.
> > > 
> > > The cache is populated of "git bundle" which will contain a mirror of
> > > the original repo, and can be cloned from. If the bundle exist, the
> > > script have the Xen makefiles clone from it, otherwise it will clone
> > > from the original URL and the bundles will be created just after.
> > > 
> > > We have more than one runner in GitLab, and no shared cache between
> > > them, so every build jobs will be responsible to create the cache.
> > > 
> > > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> > > 
> > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> > 
> > This is a good improvement.
> > 
> > Have you run this in Gitlab CI? Can you point me to a run?
> 
> I have use the CI to develop the patch, so yes I have a run of it. But
> it is a run made with my wip branch, still it should be the same result
> if it was done with the final patch:
> https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621

This looks good to me.

> 
> > > diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> > > new file mode 100755
> > > index 000000000000..017f1b8f0672
> > > --- /dev/null
> > > +++ b/automation/scripts/prepare-cache.sh
> > > @@ -0,0 +1,52 @@
> > > +#!/bin/bash
> > > +
> > > +set -ex
> > > +
> > > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > > +mkdir -p "$cachedir"
> > > +
> > > +declare -A r
> > > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
> > 
> > Does this mean if in the future we add or remove trees we will need to
> > modify this part in the same commit?
> 
> We would need to modify the script when trees are removed, because I
> haven't thought of that. But when trees are added, the script can be
> changed in a follow-up.
> 
> Ideally, we would use the Makefiles to discovers the git clones that can
> be cached, but that's not possible just yet.
> 
> In the mean time, I think I should make the script more robust against
> removal of trees, so it doesn't have to be modified in the same commit.

OK. I'm expecting a new version then.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel