Re: Solving the gitlab-ci git fetch issue, was: [PATCH 3/3] automation: add a QEMU based x86_64 Dom0/DomU test

From: Anthony PERARD <anthony.perard@citrix.com>
To: Stefano Stabellini <sstabellini@kernel.org>
Cc: <cardoe@cardoe.com>, <xen-devel@lists.xenproject.org>,
	<iwj@xenproject.org>, <wl@xen.org>, <andrew.cooper3@citrix.com>,
	"Stefano Stabellini" <stefano.stabellini@xilinx.com>
Subject: Re: Solving the gitlab-ci git fetch issue, was: [PATCH 3/3] automation: add a QEMU based x86_64 Dom0/DomU test
Date: Thu, 28 Oct 2021 10:48:20 +0100	[thread overview]
Message-ID: <YXpx5PJwZoJQ6YIy@perard> (raw)
In-Reply-To: <alpine.DEB.2.21.2110271437120.20134@sstabellini-ThinkPad-T480s>

On Wed, Oct 27, 2021 at 02:43:46PM -0700, Stefano Stabellini wrote:
> On Wed, 27 Oct 2021, Anthony PERARD wrote:
> > > But we do have a severe problem at the moment with external sources: our
> > > "git clones" keep failing during the build on x86. That is definitely
> > > something worth improving (see my other email thread on the subject) and
> > > it is the main problem affecting gitlab-ci at the moment, I keep having
> > > to restart jobs almost daily to get the overall pipeline to "pass".
> > > 
> > > If you have any ideas on how to stop fetching things using "git" from
> > > external repositories in gitlab-ci that would be fantastic :-)
> > > The only thing I could think of to "fix it" is moving all external repos
> > > to gitlab repositories mirrors.
> > 
> > I don't think that would work, I've seen the initial clone/fetch of a
> > job fail as well, so from gitlab. If we could have a cache of those
> > external resources closer to the runners, that would be better.
> 
> You mean like a git repository mirror inside the Rackspace network (the
> provider of the x86 runner), right? Then we would force the git client
> to go to the Rackspace mirror instead of directly to the target using
> "insteadOf".

That would seems the best to me. If we could install Ian's
git-cache-proxy that is used in osstest, that would be good I think.
Having a mirror instead might work too but that would mean figure out
which repo we would need a mirror of.

I did try a different alternative a while back, I tried to use gitlab's
caching capability:
    automation: Cache sub-project git tree in build jobs
    https://lore.kernel.org/xen-devel/20191219144217.305851-3-anthony.perard@citrix.com/
It mostly works but I'm not sure how useful it is as it seems there is
10 computers that would maintain 10 different caches, and most of them
for a short while.

> Is that what you meant? Doug, do you think it would work?

-- 
Anthony PERARD