Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5600 bytes --]

On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote:
> +Stefan/Peter
> 
> On 4/19/21 12:59 PM, Thomas Huth wrote:
> > On 19/04/2021 12.51, Daniel P. Berrangé wrote:
> >> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
> >>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
> >>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> >>>>> On 19/04/2021 12.10, Erik Skultety wrote:
> >>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> >>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé
> >>>>>>> wrote:
> >>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
> >>>>>>>> Allow them to easily rebase their custom set, while keeping using
> >>>>>>>> the mainstream templates, and ability to pick specific jobs from
> >>>>>>>> the mainstream set.
> >>>>>>>>
> >>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
> >>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where
> >>>>>>>> CI_PROJECT_NAMESPACE
> >>>>>>>> is your gitlab 'namespace', usually username). This file will be
> >>>>>>>> used instead of the default mainstream set.
> >>>>>>>
> >>>>>>> I find this approach undesirable, because AFAICT, it means you have
> >>>>>>> to commit this extra file to any of your downstream branches that
> >>>>>>> you want this to be used for.  Then you have to be either delete it
> >>>>>>> again before sending patches upstream, or tell git-publish to
> >>>>>>> exclude the commit that adds this.
> >>>>>>>
> >>>>>>> IMHO any per-contributor overhead needs to not involve committing
> >>>>>>> stuff to their git branches, that isn't intended to go upstream.
> >>>>>>
> >>>>>> Not just that, ideally, they should also run all the upstream
> >>>>>> workloads before
> >>>>>> submitting a PR or posting patches because they'd have to respin
> >>>>>> because of a
> >>>>>> potential failure in upstream pipelines anyway.
> >>>>>
> >>>>> It's pretty clear that you want to run the full QEMU CI before
> >>>>> submitting
> >>>>> patches to the QEMU project, but I think we are rather talking
> >>>>> about forks
> >>>>> here that are meant not meant for immediately contributing to upstream
> >>>>> again, like RHEL where we only build the KVM-related targets and
> >>>>> certainly
> >>>>> do not want to test other things like CPUs that are not capable of
> >>>>> KVM, or a
> >>>>> branch where Philippe only wants to check his MIPS-related work during
> >>>>> development.
> >>>>> For contributing patches to upstream, you certainly have to run the
> >>>>> full CI,
> >>>>> but for other things, it's sometimes really useful to cut down the CI
> >>>>> machinery (I'm also doing this in my development branches manually
> >>>>> some
> >>>>> times to speed up the CI), so I think this series make sense, indeed.
> >>>>
> >>>> For a downstream like RHEL, I'd just expect them to replace the main
> >>>> .gitlab-ci.yml entirely to suit their downstream needs.
> >>>
> >>> But that still means that we should clean up the main .gitlab-ci.yml
> >>> file
> >>> anyway, like it is done in this series, to avoid that you always get
> >>> conflicts in this big file with your downstream-only modifications.
> >>> So at
> >>> least up to patch 11 or 12, I think this is a very valuable work that
> >>> Philippe is doing here.
> >>
> >> I don't see a real issue with downstream conflicts. They'll just
> >> periodically pick a release to base themselves off and change once
> >> every 6 months or more.
> > 
> > It's not only downstream distros that rebase every 6 month. Like
> > Philippe, I'm sometimes hacking my .gitlab-ci.yml of my development
> > branch to speed up the CI during my development cycles (i.e. I'm
> > removing the jobs that I do not need). And I'm regularly rebasing my
> > development branchs. Conflicts in .gitlab-ci.yml are then always
> > painful, so a leaner main .gitlab-ci.yml file would be helpful for me,
> > too, indeed.
> 
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
> 
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure
> - run less testing
> - do not run testing
> 
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
> 
> So far I don't have better idea than this series.
> 
> Who is interested in sending patches to improve our workflow?
> 
> Thanks,
> 
> Phil.
> 
> [*] NB I have 3 extra runners added to my namespace, but it didn't
> help, as per [1] I got blocked by reaching an API rate limit.

The easiest short-term workaround seems to be disabling testing when you
push certain branches.

In the long term I think GitLab CI should allow unlimited jobs on
dedicated runners. It may be necessary to get in touch with GitLab
support and figure out how to stop it blocking dedicated runner jobs.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]