qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
From: Stefan Hajnoczi <stefanha@redhat.com>
To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Daniel P. Berrangé" <berrange@redhat.com>,
	"Erik Skultety" <eskultet@redhat.com>,
	qemu-devel@nongnu.org,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Willian Rampazzo" <willianr@redhat.com>,
	"Cleber Rosa" <crosa@redhat.com>,
	"Miroslav Rezanina" <mrezanin@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
Date: Tue, 11 May 2021 14:55:22 +0100	[thread overview]
Message-ID: <YJqMytvO4TNNYL0O@stefanha-x1.localdomain> (raw)
In-Reply-To: <a0e83ef7-13ee-6f45-96b5-b9d848bb8a43@amsat.org>

[-- Attachment #1: Type: text/plain, Size: 5600 bytes --]

On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote:
> +Stefan/Peter
> 
> On 4/19/21 12:59 PM, Thomas Huth wrote:
> > On 19/04/2021 12.51, Daniel P. Berrangé wrote:
> >> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
> >>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
> >>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> >>>>> On 19/04/2021 12.10, Erik Skultety wrote:
> >>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> >>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé
> >>>>>>> wrote:
> >>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
> >>>>>>>> Allow them to easily rebase their custom set, while keeping using
> >>>>>>>> the mainstream templates, and ability to pick specific jobs from
> >>>>>>>> the mainstream set.
> >>>>>>>>
> >>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
> >>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where
> >>>>>>>> CI_PROJECT_NAMESPACE
> >>>>>>>> is your gitlab 'namespace', usually username). This file will be
> >>>>>>>> used instead of the default mainstream set.
> >>>>>>>
> >>>>>>> I find this approach undesirable, because AFAICT, it means you have
> >>>>>>> to commit this extra file to any of your downstream branches that
> >>>>>>> you want this to be used for.  Then you have to be either delete it
> >>>>>>> again before sending patches upstream, or tell git-publish to
> >>>>>>> exclude the commit that adds this.
> >>>>>>>
> >>>>>>> IMHO any per-contributor overhead needs to not involve committing
> >>>>>>> stuff to their git branches, that isn't intended to go upstream.
> >>>>>>
> >>>>>> Not just that, ideally, they should also run all the upstream
> >>>>>> workloads before
> >>>>>> submitting a PR or posting patches because they'd have to respin
> >>>>>> because of a
> >>>>>> potential failure in upstream pipelines anyway.
> >>>>>
> >>>>> It's pretty clear that you want to run the full QEMU CI before
> >>>>> submitting
> >>>>> patches to the QEMU project, but I think we are rather talking
> >>>>> about forks
> >>>>> here that are meant not meant for immediately contributing to upstream
> >>>>> again, like RHEL where we only build the KVM-related targets and
> >>>>> certainly
> >>>>> do not want to test other things like CPUs that are not capable of
> >>>>> KVM, or a
> >>>>> branch where Philippe only wants to check his MIPS-related work during
> >>>>> development.
> >>>>> For contributing patches to upstream, you certainly have to run the
> >>>>> full CI,
> >>>>> but for other things, it's sometimes really useful to cut down the CI
> >>>>> machinery (I'm also doing this in my development branches manually
> >>>>> some
> >>>>> times to speed up the CI), so I think this series make sense, indeed.
> >>>>
> >>>> For a downstream like RHEL, I'd just expect them to replace the main
> >>>> .gitlab-ci.yml entirely to suit their downstream needs.
> >>>
> >>> But that still means that we should clean up the main .gitlab-ci.yml
> >>> file
> >>> anyway, like it is done in this series, to avoid that you always get
> >>> conflicts in this big file with your downstream-only modifications.
> >>> So at
> >>> least up to patch 11 or 12, I think this is a very valuable work that
> >>> Philippe is doing here.
> >>
> >> I don't see a real issue with downstream conflicts. They'll just
> >> periodically pick a release to base themselves off and change once
> >> every 6 months or more.
> > 
> > It's not only downstream distros that rebase every 6 month. Like
> > Philippe, I'm sometimes hacking my .gitlab-ci.yml of my development
> > branch to speed up the CI during my development cycles (i.e. I'm
> > removing the jobs that I do not need). And I'm regularly rebasing my
> > development branchs. Conflicts in .gitlab-ci.yml are then always
> > painful, so a leaner main .gitlab-ci.yml file would be helpful for me,
> > too, indeed.
> 
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
> 
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure
> - run less testing
> - do not run testing
> 
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
> 
> So far I don't have better idea than this series.
> 
> Who is interested in sending patches to improve our workflow?
> 
> Thanks,
> 
> Phil.
> 
> [*] NB I have 3 extra runners added to my namespace, but it didn't
> help, as per [1] I got blocked by reaching an API rate limit.

The easiest short-term workaround seems to be disabling testing when you
push certain branches.

In the long term I think GitLab CI should allow unlimited jobs on
dedicated runners. It may be necessary to get in touch with GitLab
support and figure out how to stop it blocking dedicated runner jobs.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply	other threads:[~2021-05-11 13:57 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 02/15] gitlab-ci: Replace YAML anchors by extends (native_build_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 03/15] gitlab-ci: Replace YAML anchors by extends (native_test_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) Philippe Mathieu-Daudé
2021-05-03  9:22   ` Thomas Huth
2021-05-03  9:45     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job Philippe Mathieu-Daudé
2021-04-19  5:19   ` Thomas Huth
2021-04-23 17:18     ` Willian Rampazzo
2021-04-18 23:34 ` [PATCH 06/15] gitlab-ci: Extract container job template to container-template.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 07/15] gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml Philippe Mathieu-Daudé
2021-04-19  5:26   ` Thomas Huth
2021-04-19 13:44     ` Wainer dos Santos Moschetta
2021-04-18 23:34 ` [PATCH 09/15] gitlab-ci: Extract build stages to stages.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
2021-04-19  5:39   ` Thomas Huth
2021-04-19 15:11   ` Alex Bennée
2021-05-11  7:19     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml Philippe Mathieu-Daudé
2021-04-19  5:42   ` Thomas Huth
2021-04-18 23:34 ` [PATCH 12/15] gitlab-ci: Move current job set to qemu-project.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [RFC PATCH 13/15] gitlab-ci: Switch to dynamically generated pipelines Philippe Mathieu-Daudé
2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
2021-04-19  5:48   ` Thomas Huth
2021-04-19  9:40   ` Daniel P. Berrangé
2021-04-19 10:09     ` Philippe Mathieu-Daudé
2021-04-19 10:10     ` Erik Skultety
2021-04-19 10:20       ` Thomas Huth
2021-04-19 10:36         ` Daniel P. Berrangé
2021-04-19 10:48           ` Thomas Huth
2021-04-19 10:51             ` Daniel P. Berrangé
2021-04-19 10:59               ` Thomas Huth
2021-05-11  6:48                 ` Philippe Mathieu-Daudé
2021-05-11 13:55                   ` Stefan Hajnoczi [this message]
2021-05-11 14:00                   ` Alex Bennée
2021-05-11 14:21                   ` Daniel P. Berrangé
2021-05-13 19:01                   ` Philippe Mathieu-Daudé
2021-04-19 10:47         ` Daniel P. Berrangé
2021-04-19 10:44       ` Philippe Mathieu-Daudé
2021-04-19 15:57   ` Alex Bennée
2021-04-19 16:22     ` Daniel P. Berrangé
2021-04-19 16:46       ` Philippe Mathieu-Daudé
2021-04-19 16:58         ` Daniel P. Berrangé
2021-04-19 16:39     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [NOTFORMERGE PATCH 15/15] gitlab-ci: Use my own set of jobs for CI pipeline Philippe Mathieu-Daudé

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YJqMytvO4TNNYL0O@stefanha-x1.localdomain \
    --to=stefanha@redhat.com \
    --cc=alex.bennee@linaro.org \
    --cc=berrange@redhat.com \
    --cc=crosa@redhat.com \
    --cc=eskultet@redhat.com \
    --cc=f4bug@amsat.org \
    --cc=mrezanin@redhat.com \
    --cc=peter.maydell@linaro.org \
    --cc=qemu-devel@nongnu.org \
    --cc=thuth@redhat.com \
    --cc=wainersm@redhat.com \
    --cc=willianr@redhat.com \
    --subject='Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox