Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs

From: "Daniel P. Berrangé" <berrange@redhat.com>
To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Erik Skultety" <eskultet@redhat.com>,
	qemu-devel@nongnu.org,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Willian Rampazzo" <willianr@redhat.com>,
	"Stefan Hajnoczi" <stefanha@redhat.com>,
	"Cleber Rosa" <crosa@redhat.com>,
	"Miroslav Rezanina" <mrezanin@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
Date: Tue, 11 May 2021 15:21:44 +0100	[thread overview]
Message-ID: <YJqS+FqVttnrw6zb@redhat.com> (raw)
In-Reply-To: <a0e83ef7-13ee-6f45-96b5-b9d848bb8a43@amsat.org>

On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote:
> +Stefan/Peter
> 
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
> 
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure
> - run less testing
> - do not run testing
> 
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
> 
> So far I don't have better idea than this series.
> 
> Who is interested in sending patches to improve our workflow?

So we have a few scenarios for using the CI

 1. Running gating CI before merging to master
 2. Subsystem maintainers running CI before sending a PULL req
 3. Contributors running CI before sending a patch series

Right now we have the same jobs running in all three scenarios.

Given the increasing restrictions on usage, we clearly need to cut
down in general and also make it so that it is harder to accidentally
burn all your available CI allowance.

Currently we always run CI whenever pushing to gitlab. This is
convenient but in retrospect it is overkill. People often push
to gitlab simply as their backup strategy and thus don't need
CI run every time.

Not all changes require all possible jobs to by run, but it is hard
to filter jobs when we're triggering them based on pushes, as the
baseline against which file changes are identified is ill-defined.

For scenario (1) we need all the jobs run to maximise quality.
This is also a case where we're most likely to have custom runners
available, so CI allowance is less of a concern. The job count still
needs to be reasonable to avoid hitting issues at times when the
merges are frequent (just before freeze).

For scenario (2) subsys maintainers, we want them to minimize
the liklihood that a pull request will fail scenario (1) and
require a respin.  Running all jobs achieces this but it is
likely overkill.

eg we have 24 cross compiler builds. If we expect most maintainers
will have either x86-64 or aarch64 hardware for their primary dev
platform, then the key benefit of cross compilers is getting coverage
of

 - 32-bit
 - big endian
 - windows

We don't need 24 jobs todo that. We could simply pick armel as the
most relevant 32-bit arch and s390x as the most relevant big endian
arch, and then the win32/64 platforms. IOW we could potentially only
run 4-6 jobs instead of 24, and still get excellant arch coverage.

Similarly for native builds we test quite alot of different distros.
I think we probably can rationalize that down to just 2 distros,
one covering oldest packages (Debian Stretch) and one covering newest
(Fedora 34), and a "build everything" config.

We have many other jobs that are testing various obscure combinations
of configure args. I'd suggest these rarely fail for most pull requests
so are overkill.

For subsystem maintainers we could potentially get down to just 10-15
jobs if we're ambitious. Leave everything else as manual trigger only.

Perhaps set all the jobs to only run on certain branch name patterns.
eg perhaps "*-next" filter is common for subsystem maintainer's pending
branches ?

For general contributors a similarly short set of jobs to subsystem
maintainers is viable. Perhaps again just let then use a "-next"
branch.

If we can enable manual triggers on any other branches that's good.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|