Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Wei Liu <wl@xen.org>]

On Mon, Jan 06, 2020 at 02:34:02PM +0000, Anthony PERARD wrote:
> On Fri, Jan 03, 2020 at 02:29:07PM +0000, Wei Liu wrote:
> > On Thu, Dec 19, 2019 at 02:42:17PM +0000, Anthony PERARD wrote:
> > > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > > Internet too often.
> > > 
> > > The cache is setup so that when xen.git/Config.mk is changed, the
> > > cache will need to be recreated. This has been chosen because that is
> > > where the information about how to clone sub-project trees is encoded
> > > (revisions). That may not work for qemu-xen tree which usually is
> > > `master', but that should be fine for now.
> > > 
> > > The cache is populated of "git bundle" which will contain a mirror of
> > > the original repo, and can be cloned from. If the bundle exist, the
> > > script have the Xen makefiles clone from it, otherwise it will clone
> > > from the original URL and the bundles will be created just after.
> > > 
> > > We have more than one runner in GitLab, and no shared cache between
> > > them, so every build jobs will be responsible to create the cache.
> > > 
> > > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> > > 
> > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> > 
> > This is a good improvement.
> > 
> > Have you run this in Gitlab CI? Can you point me to a run?
> 
> I have use the CI to develop the patch, so yes I have a run of it. But
> it is a run made with my wip branch, still it should be the same result
> if it was done with the final patch:
> https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621

This looks good to me.

> 
> > > diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepare-cache.sh
> > > new file mode 100755
> > > index 000000000000..017f1b8f0672
> > > --- /dev/null
> > > +++ b/automation/scripts/prepare-cache.sh
> > > @@ -0,0 +1,52 @@
> > > +#!/bin/bash
> > > +
> > > +set -ex
> > > +
> > > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > > +mkdir -p "$cachedir"
> > > +
> > > +declare -A r
> > > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
> > 
> > Does this mean if in the future we add or remove trees we will need to
> > modify this part in the same commit?
> 
> We would need to modify the script when trees are removed, because I
> haven't thought of that. But when trees are added, the script can be
> changed in a follow-up.
> 
> Ideally, we would use the Makefiles to discovers the git clones that can
> be cached, but that's not possible just yet.
> 
> In the mean time, I think I should make the script more robust against
> removal of trees, so it doesn't have to be modified in the same commit.

OK. I'm expecting a new version then.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel