Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Anthony PERARD <anthony.perard@citrix.com>]

On Fri, Jan 03, 2020 at 02:29:07PM +0000, Wei Liu wrote:
> On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > Internet too often.
> > 
> > The cache is setup so that when xen.git/Config.mk is changed, the
> > cache will need to be recreated. This has been chosen because that is
> > where the information about how to clone sub-project trees is encoded
> > (revisions). That may not work for qemu-xen tree which usually is
> > `master', but that should be fine for now.
> > 
> > The cache is populated of "git bundle" which will contain a mirror of
> > the original repo, and can be cloned from. If the bundle exist, the
> > script have the Xen makefiles clone from it, otherwise it will clone
> > from the original URL and the bundles will be created just after.
> > 
> > We have more than one runner in GitLab, and no shared cache between
> > them, so every build jobs will be responsible to create the cache.
> > 
> > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> 
> This is a good improvement.
> 
> Have you run this in Gitlab CI? Can you point me to a run?

I have use the CI to develop the patch, so yes I have a run of it. But
it is a run made with my wip branch, still it should be the same result
if it was done with the final patch:
https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621

> > diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> > new file mode 100755
> > index 000000000000..017f1b8f0672
> > --- /dev/null
> > +++ b/automation/scripts/prepare-cache.sh
> > @@ -0,0 +1,52 @@
> > +#!/bin/bash
> > +
> > +set -ex
> > +
> > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > +mkdir -p "$cachedir"
> > +
> > +declare -A r
> > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
> 
> Does this mean if in the future we add or remove trees we will need to
> modify this part in the same commit?

We would need to modify the script when trees are removed, because I
haven't thought of that. But when trees are added, the script can be
changed in a follow-up.

Ideally, we would use the Makefiles to discovers the git clones that can
be cached, but that's not possible just yet.

In the mean time, I think I should make the script more robust against
removal of trees, so it doesn't have to be modified in the same commit.

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel