gitlab-ci: Only build /staging branch?

gitlab-ci: Only build /staging branch?

[Philippe Mathieu-Daudé]

Hi,

Peter's current workflow is push to /staging and if his
testing succeeds, he pushes the same commit as /master.

IMO there is no point in building /master branch, as it
has already been built earlier as /staging.

Regards,

Phil.

Re: gitlab-ci: Only build /staging branch?

[Philippe Mathieu-Daudé]

On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Peter's current workflow is push to /staging and if his
> testing succeeds, he pushes the same commit as /master.
> 
> IMO there is no point in building /master branch, as it
> has already been built earlier as /staging.

Also this might be the reason why the /staging "deploy" job
fails with "input/output error".

Re: gitlab-ci: Only build /staging branch?

[Philippe Mathieu-Daudé]

On 3/24/21 7:33 PM, Philippe Mathieu-Daudé wrote:
> On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> Peter's current workflow is push to /staging and if his
>> testing succeeds, he pushes the same commit as /master.
>>
>> IMO there is no point in building /master branch, as it
>> has already been built earlier as /staging.

Similarly with tags. Although we don't tag often.

Suggestion:
- staging: always build
- tags: on-demand build (manual)
- master: no build

Thinking about reproducible artifacts, we could have a tag job
to produce the archive with submodules, but someone still have
to sign it.

Thoughts?

Re: gitlab-ci: Only build /staging branch?

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

On Wed, Mar 24, 2021 at 07:01:12PM +0100, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Peter's current workflow is push to /staging and if his
> testing succeeds, he pushes the same commit as /master.
> 
> IMO there is no point in building /master branch, as it
> has already been built earlier as /staging.
>

Hi,

I agree, I currently see no point in running the entire pipeline
again.

But we need to watch out for the job conditional rules. For instance,
both of these pipelines are about the exact same commit:

   https://gitlab.com/qemu-project/qemu/-/pipelines/275925246
   https://gitlab.com/qemu-project/qemu/-/pipelines/275925289

But one has 113, while the other has 119 jobs.  Also, we may want
to check if tags also trigger new (unnecessary) pipelines.

> Regards,
> 
> Phil.
> 

Best,
- Cleber

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: gitlab-ci: Only build /staging branch?

[Thomas Huth]

On 24/03/2021 22.58, Philippe Mathieu-Daudé wrote:
> On 3/24/21 7:33 PM, Philippe Mathieu-Daudé wrote:
>> On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:
>>> Hi,
>>>
>>> Peter's current workflow is push to /staging and if his
>>> testing succeeds, he pushes the same commit as /master.
>>>
>>> IMO there is no point in building /master branch, as it
>>> has already been built earlier as /staging.
> 
> Similarly with tags. Although we don't tag often.

Tags are used for pull-requests. So I think we should run the whole CI for 
tags, to make it clear that a pull-request always includes code that builds 
fine.

  Thomas

Re: gitlab-ci: Only build /staging branch?

[Philippe Mathieu-Daudé]

On 3/25/21 6:43 AM, Thomas Huth wrote:
> On 24/03/2021 22.58, Philippe Mathieu-Daudé wrote:
>> On 3/24/21 7:33 PM, Philippe Mathieu-Daudé wrote:
>>> On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:
>>>> Hi,
>>>>
>>>> Peter's current workflow is push to /staging and if his
>>>> testing succeeds, he pushes the same commit as /master.
>>>>
>>>> IMO there is no point in building /master branch, as it
>>>> has already been built earlier as /staging.
>>
>> Similarly with tags. Although we don't tag often.
> 
> Tags are used for pull-requests. So I think we should run the whole CI
> for tags, to make it clear that a pull-request always includes code that
> builds fine.

Sorry the context was not clear :/

This is only relevant for the qemu-project/qemu gitlab namespace.

v6.0 is at the door and I was wondering what is missing to have the
CI used as a gate.

- Stefan/Paolo moved the main repository location.

- Alex made yet another effort to get the CI pipeline green again.

- IIRC Peter said waiting 2h after pushing /staging is too long.
Currently worst case it takes ~2h25 between one /staging and the
next one, simply because /master is rebuilt in the middle. If we
remove /master we have ~1h15 per /staging pipeline.

- I don't remember what is missing from Cleber script, maybe we can
use it as it without waiting for a respin?

We have never been that close, but we are not there yet...

Regards,

Phil.

Re: gitlab-ci: Only build /staging branch?

[Philippe Mathieu-Daudé]

On 3/25/21 10:29 AM, Philippe Mathieu-Daudé wrote:
> On 3/25/21 6:43 AM, Thomas Huth wrote:
>> On 24/03/2021 22.58, Philippe Mathieu-Daudé wrote:
>>> On 3/24/21 7:33 PM, Philippe Mathieu-Daudé wrote:
>>>> On 3/24/21 7:01 PM, Philippe Mathieu-Daudé wrote:
>>>>> Hi,
>>>>>
>>>>> Peter's current workflow is push to /staging and if his
>>>>> testing succeeds, he pushes the same commit as /master.
>>>>>
>>>>> IMO there is no point in building /master branch, as it
>>>>> has already been built earlier as /staging.
>>>
>>> Similarly with tags. Although we don't tag often.
>>
>> Tags are used for pull-requests. So I think we should run the whole CI
>> for tags, to make it clear that a pull-request always includes code that
>> builds fine.
> 
> Sorry the context was not clear :/
> 
> This is only relevant for the qemu-project/qemu gitlab namespace.
> 
> v6.0 is at the door and I was wondering what is missing to have the
> CI used as a gate.
> 
> - Stefan/Paolo moved the main repository location.
> 
> - Alex made yet another effort to get the CI pipeline green again.
> 
> - IIRC Peter said waiting 2h after pushing /staging is too long.
> Currently worst case it takes ~2h25 between one /staging and the
> next one, simply because /master is rebuilt in the middle. If we
> remove /master we have ~1h15 per /staging pipeline.
> 
> - I don't remember what is missing from Cleber script, maybe we can
> use it as it without waiting for a respin?

As someone else is caring about this, please disregard this thread
and its questions/suggestions.

Regards,

Phil.

Re: gitlab-ci: Only build /staging branch?

[Peter Maydell]

On Thu, 25 Mar 2021 at 09:33, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> v6.0 is at the door and I was wondering what is missing to have the
> CI used as a gate.

It needs to be faster. Mostly I do check the gitlab CI pipeline
status, but in the run-up to getting rc0 out I stopped waiting
for the gitlab CI job to finish, because I was continually finding
that I kicked off a run, my local build-tests would complete within
an hour or so, and the gitlab CI jobs were still pending, barely
started, etc. Turnaround on testing a merge must be 90 minutes or
less, especially during release periods, because there are always
a huge number of merges that arrive for me to test in the last
couple of days before freeze.

thanks
-- PMM

Re: gitlab-ci: Only build /staging branch?

[Paolo Bonzini]

On 25/03/21 11:34, Peter Maydell wrote:
> It needs to be faster. Mostly I do check the gitlab CI pipeline
> status, but in the run-up to getting rc0 out I stopped waiting
> for the gitlab CI job to finish, because I was continually finding
> that I kicked off a run, my local build-tests would complete within
> an hour or so, and the gitlab CI jobs were still pending, barely
> started, etc. Turnaround on testing a merge must be 90 minutes or
> less, especially during release periods, because there are always
> a huge number of merges that arrive for me to test in the last
> couple of days before freeze.

Perhaps we could script it so that if the pipeline passes the merge to 
master is done automatically.

Paolo

Re: gitlab-ci: Only build /staging branch?

[Peter Maydell]

On Thu, 25 Mar 2021 at 11:05, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 25/03/21 11:34, Peter Maydell wrote:
> > It needs to be faster. Mostly I do check the gitlab CI pipeline
> > status, but in the run-up to getting rc0 out I stopped waiting
> > for the gitlab CI job to finish, because I was continually finding
> > that I kicked off a run, my local build-tests would complete within
> > an hour or so, and the gitlab CI jobs were still pending, barely
> > started, etc. Turnaround on testing a merge must be 90 minutes or
> > less, especially during release periods, because there are always
> > a huge number of merges that arrive for me to test in the last
> > couple of days before freeze.
>
> Perhaps we could script it so that if the pipeline passes the merge to
> master is done automatically.

That would be nice eventually, but we can't do it until the gitlab
CI is the *only* gating criterion.

thanks
-- PMM

Re: gitlab-ci: Only build /staging branch?

[Daniel P. Berrangé]

On Thu, Mar 25, 2021 at 12:05:32PM +0100, Paolo Bonzini wrote:
> On 25/03/21 11:34, Peter Maydell wrote:
> > It needs to be faster. Mostly I do check the gitlab CI pipeline
> > status, but in the run-up to getting rc0 out I stopped waiting
> > for the gitlab CI job to finish, because I was continually finding
> > that I kicked off a run, my local build-tests would complete within
> > an hour or so, and the gitlab CI jobs were still pending, barely
> > started, etc. Turnaround on testing a merge must be 90 minutes or
> > less, especially during release periods, because there are always
> > a huge number of merges that arrive for me to test in the last
> > couple of days before freeze.
> 
> Perhaps we could script it so that if the pipeline passes the merge to
> master is done automatically.

No need to script it, that functionality already exists in GitLab.

Push to the staging branch, and open a merge request for applying
staging -> master, and enable "merge when pipeline succeeds".

You can actually do this all in one command

https://docs.gitlab.com/ee/user/project/push_options.html

  git push \
     -o merge_request.create \
     -o merge_request.target=master \
     -o merge_request.merge_when_pipeline_succeeds \
     origin staging


The gitlab-ci.yml file could then be configured so that pipeline
jobs are associated with a merge request, rather than push event.
This will avoid the pipeline being re-run on master after the
merge.

If you enable "merge trains" option in the repo, then you can
even push to multiple branches concurrently, and gitlab will
serialize the CI pipelines from each merge request in turn,
(assuming no conflicts between then).

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|