no more pullreq processing til February

no more pullreq processing til February

[Peter Maydell]

Hi; we've run out of gitlab CI pipeline minutes for this month.
This leaves us the choice of:
 (a) don't process any more pullreqs til we get more minutes in Feb
 (b) merge pullreqs blindly without CI testing
 (c) buy more minutes

For the moment I propose to take option (a). My mail filter will
continue to track pullreqs that get sent to the list, but I won't
do anything with them.

If anybody has a better suggestion feel free :-)

thanks
-- PMM

Re: no more pullreq processing til February

[Eldon Stegall]

On Thu, Jan 26, 2023 at 01:22:32PM +0000, Peter Maydell wrote:
> Hi; we've run out of gitlab CI pipeline minutes for this month.
> This leaves us the choice of:
>  (a) don't process any more pullreqs til we get more minutes in Feb
>  (b) merge pullreqs blindly without CI testing
>  (c) buy more minutes
> 
> For the moment I propose to take option (a). My mail filter will
> continue to track pullreqs that get sent to the list, but I won't
> do anything with them.
> 
> If anybody has a better suggestion feel free :-)

Would it be possible if (d) were to run self-hosted instances of the
runner? I am not sure how gitlab pricing works, but I believe on github
self-hosted runners are free.

I have several baremetal machines colocated that I could dedicate to
execute these runs, dual processor xeons with a couple hundred gigs of
RAM. I would need approx 48 hours notice to initially provision the
machines. I would be happy to provide root credentials and work out IPMI
access if that becomes necessary.

If this offering isn't suitable, let me know how we can consider
adapting something to the project's needs.

Thanks,
Eldon

Re: no more pullreq processing til February

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> Hi; we've run out of gitlab CI pipeline minutes for this month.
> This leaves us the choice of:
>  (a) don't process any more pullreqs til we get more minutes in Feb
>  (b) merge pullreqs blindly without CI testing
>  (c) buy more minutes
>
> For the moment I propose to take option (a). My mail filter will
> continue to track pullreqs that get sent to the list, but I won't
> do anything with them.
>
> If anybody has a better suggestion feel free :-)

I've submitted a support request (#366644) to GitLab to see if they will
give us more minutes for this month. Longer term ideas:

 * Reduce compile time by reducing number of identical object files we
   build for specific_ss
 * Move more tests over to custom runners (don't we have an x86 box
   somewhere?)
 * Carry out an audit of code coverage for different test sets and
   rationalise our CI set

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: no more pullreq processing til February

[Daniel Henrique Barboza]

On 1/26/23 10:57, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> Hi; we've run out of gitlab CI pipeline minutes for this month.
>> This leaves us the choice of:
>>   (a) don't process any more pullreqs til we get more minutes in Feb
>>   (b) merge pullreqs blindly without CI testing
>>   (c) buy more minutes
>>
>> For the moment I propose to take option (a). My mail filter will
>> continue to track pullreqs that get sent to the list, but I won't
>> do anything with them.
>>
>> If anybody has a better suggestion feel free :-)
> 
> I've submitted a support request (#366644) to GitLab to see if they will
> give us more minutes for this month. Longer term ideas:
> 
>   * Reduce compile time by reducing number of identical object files we
>     build for specific_ss
>   * Move more tests over to custom runners (don't we have an x86 box
>     somewhere?)
>   * Carry out an audit of code coverage for different test sets and
>     rationalise our CI set

What about sub-maintainers running the CI jobs before sending PRs? Should we
stop it? I usually do a full CI run before sending a ppc queue but if we're
having problems with gitlab pipeline minutes then perhaps we should stop
doing that.


Thanks,

Daniel

> 

Re: no more pullreq processing til February

[Alex Bennée]

Eldon Stegall <eldon-qemu@eldondev.com> writes:

> On Thu, Jan 26, 2023 at 01:22:32PM +0000, Peter Maydell wrote:
>> Hi; we've run out of gitlab CI pipeline minutes for this month.
>> This leaves us the choice of:
>>  (a) don't process any more pullreqs til we get more minutes in Feb
>>  (b) merge pullreqs blindly without CI testing
>>  (c) buy more minutes
>> 
>> For the moment I propose to take option (a). My mail filter will
>> continue to track pullreqs that get sent to the list, but I won't
>> do anything with them.
>> 
>> If anybody has a better suggestion feel free :-)
>
> Would it be possible if (d) were to run self-hosted instances of the
> runner? I am not sure how gitlab pricing works, but I believe on github
> self-hosted runners are free.

Yes running more stuff on custom runners would be great (and also
possibly not as slow as the heavily loaded shared runners).

> I have several baremetal machines colocated that I could dedicate to
> execute these runs, dual processor xeons with a couple hundred gigs of
> RAM. I would need approx 48 hours notice to initially provision the
> machines. I would be happy to provide root credentials and work out IPMI
> access if that becomes necessary.

I think we would need:

  - provisioning scripts in scripts/ci/setup (if existing not already
    good enough)
  - tweak to handle multiple runner instances (or more -j on the build)
  - changes to .gitlab-ci.d/ so we can use those machines while keeping
    ability to run on shared runners for those outside the project

> If this offering isn't suitable, let me know how we can consider
> adapting something to the project's needs.
>
> Thanks,
> Eldon


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 01:52:39PM +0000, Eldon Stegall wrote:
> On Thu, Jan 26, 2023 at 01:22:32PM +0000, Peter Maydell wrote:
> > Hi; we've run out of gitlab CI pipeline minutes for this month.
> > This leaves us the choice of:
> >  (a) don't process any more pullreqs til we get more minutes in Feb
> >  (b) merge pullreqs blindly without CI testing
> >  (c) buy more minutes
> > 
> > For the moment I propose to take option (a). My mail filter will
> > continue to track pullreqs that get sent to the list, but I won't
> > do anything with them.
> > 
> > If anybody has a better suggestion feel free :-)
> 
> Would it be possible if (d) were to run self-hosted instances of the
> runner? I am not sure how gitlab pricing works, but I believe on github
> self-hosted runners are free.
> 
> I have several baremetal machines colocated that I could dedicate to
> execute these runs, dual processor xeons with a couple hundred gigs of
> RAM. I would need approx 48 hours notice to initially provision the
> machines. I would be happy to provide root credentials and work out IPMI
> access if that becomes necessary.

We do currently have some private runners registered against the
/qemu-project namespace, so we can do some non-x86 native testing.

The challenge is the integration and configuration. The GitLab CI
YAML config rules need to be written such that specific jobs  get
targetted for the right private runners, instead of the shared
runners, by including the 'tags' element in the job config, and
some 'rules' logic.

Any job we switch to work against private runners though, now become
inaccessible to our contributors who are running pipelines in their
forks, because the tags we add won't match against public shared
runners. So we'd be putting a burden on our contributors to run
private runners two, which is not desirable.

The alternative is to duplicate all our jobs, once for private
runners and once for shared runners. It is a bit repetative but
with job inheritance it isn't a 100% copy+paste job, just about
20-30% tedious boilerplate perhaps.

eg


avocado-system-debian:
  extends: .avocado_test_job_template
  needs:
    - job: build-system-debian
      artifacts: true
  variables:
    IMAGE: debian-amd64
    MAKE_CHECK_ARGS: check-avocado

would have to be replaced with


.avocado-system-debian_base:
  extends: .avocado_test_job_template
  needs:
    - job: build-system-debian
      artifacts: true
  variables:
    IMAGE: debian-amd64
    MAKE_CHECK_ARGS: check-avocado

avocado-system-debian-shared:
  extends: .avocado-system-debian_base
  rules:
    - if '$CI_PROJECT_NAMESPACE == "qemu-project"'
      when: never
    - if '$CI_PROJECT_NAMESPACE != "qemu-project"'
      when: on_success

avocado-system-debian-private:
  extends: .avocado-system-debian_base
  tags:
    - qemu-private-runner-x86
  rules:
    - if '$CI_PROJECT_NAMESPACE == "qemu-project"'
      when: on_success
    - if '$CI_PROJECT_NAMESPACE != "qemu-project"'
      when: never


there's many variations, that's just a crude example off top of my head.
This example wouldn't work if the base project incldues 'rules' as the
parent rules don't get merged. So actually we would need to play some
further games to get this to work in most cases.

Anyway, private runners are potentially useful, especially if this becomes
a long term problems for QEMU. They just aren't a quickly insertable
solution we can deploy in a matter of days, as we need much YML config
work first AFAICT.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Stefan Hajnoczi]

Are you batching pull requests? I used that approach last release
cycle. CI takes so long to run that I didn't want to run it for every
pull request. Batching worked well overall.

You could try the apply-pullreq --apply-many option in the modified
script I sent in <Y5nom0ji4S/CmvZL@fedora> on Wed, 14 Dec 2022.

The workflow was:
1. Apply multiple requests on the staging branch locally.
2. Push staging to GitLab when you're ready to run CI.
3. When there is a failure, drop the pull request most likely
responsible for the failure and run CI again with the remaining pull
requests.

Stefan

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 11:07:37AM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 1/26/23 10:57, Alex Bennée wrote:
> > 
> > Peter Maydell <peter.maydell@linaro.org> writes:
> > 
> > > Hi; we've run out of gitlab CI pipeline minutes for this month.
> > > This leaves us the choice of:
> > >   (a) don't process any more pullreqs til we get more minutes in Feb
> > >   (b) merge pullreqs blindly without CI testing
> > >   (c) buy more minutes
> > > 
> > > For the moment I propose to take option (a). My mail filter will
> > > continue to track pullreqs that get sent to the list, but I won't
> > > do anything with them.
> > > 
> > > If anybody has a better suggestion feel free :-)
> > 
> > I've submitted a support request (#366644) to GitLab to see if they will
> > give us more minutes for this month. Longer term ideas:
> > 
> >   * Reduce compile time by reducing number of identical object files we
> >     build for specific_ss
> >   * Move more tests over to custom runners (don't we have an x86 box
> >     somewhere?)
> >   * Carry out an audit of code coverage for different test sets and
> >     rationalise our CI set
> 
> What about sub-maintainers running the CI jobs before sending PRs? Should we
> stop it? I usually do a full CI run before sending a ppc queue but if we're
> having problems with gitlab pipeline minutes then perhaps we should stop
> doing that.

Any CI you run in your repo fork has no impact on QEMU CI allowance
upstream.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Peter Maydell]

On Thu, 26 Jan 2023 at 14:16, Alex Bennée <alex.bennee@linaro.org> wrote:
> Eldon Stegall <eldon-qemu@eldondev.com> writes:
> > I have several baremetal machines colocated that I could dedicate to
> > execute these runs, dual processor xeons with a couple hundred gigs of
> > RAM. I would need approx 48 hours notice to initially provision the
> > machines. I would be happy to provide root credentials and work out IPMI
> > access if that becomes necessary.
>
> I think we would need:
>
>   - provisioning scripts in scripts/ci/setup (if existing not already
>     good enough)
>   - tweak to handle multiple runner instances (or more -j on the build)
>   - changes to .gitlab-ci.d/ so we can use those machines while keeping
>     ability to run on shared runners for those outside the project

Also
  - the provider of the private runner agrees that they're doing
    the system administration for it (i.e. keeping the OS distro
    on it up to date, installing security fixes, etc)

(We've historically had a severe lack of people with the time
and expertise to do sysadmin work, which is part of why we
moved towards doing CI on gitlab in the first place).

thanks
-- PMM

Re: no more pullreq processing til February

[Peter Maydell]

On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> Are you batching pull requests? I used that approach last release
> cycle. CI takes so long to run that I didn't want to run it for every
> pull request. Batching worked well overall.

No, I just do one test per pullreq. IME the CI is flaky
enough that I don't really want to batch it up, and it
isn't so slow that I build up a backlog of unprocessed
requests.

-- PMM

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 02:18:23PM +0000, Daniel P. Berrangé wrote:
> On Thu, Jan 26, 2023 at 01:52:39PM +0000, Eldon Stegall wrote:
> > On Thu, Jan 26, 2023 at 01:22:32PM +0000, Peter Maydell wrote:
> > > Hi; we've run out of gitlab CI pipeline minutes for this month.
> > > This leaves us the choice of:
> > >  (a) don't process any more pullreqs til we get more minutes in Feb
> > >  (b) merge pullreqs blindly without CI testing
> > >  (c) buy more minutes
> > > 
> > > For the moment I propose to take option (a). My mail filter will
> > > continue to track pullreqs that get sent to the list, but I won't
> > > do anything with them.
> > > 
> > > If anybody has a better suggestion feel free :-)
> > 
> > Would it be possible if (d) were to run self-hosted instances of the
> > runner? I am not sure how gitlab pricing works, but I believe on github
> > self-hosted runners are free.
> > 
> > I have several baremetal machines colocated that I could dedicate to
> > execute these runs, dual processor xeons with a couple hundred gigs of
> > RAM. I would need approx 48 hours notice to initially provision the
> > machines. I would be happy to provide root credentials and work out IPMI
> > access if that becomes necessary.
> 
> We do currently have some private runners registered against the
> /qemu-project namespace, so we can do some non-x86 native testing.
> 
> The challenge is the integration and configuration. The GitLab CI
> YAML config rules need to be written such that specific jobs  get
> targetted for the right private runners, instead of the shared
> runners, by including the 'tags' element in the job config, and
> some 'rules' logic.

Scratch that, it is actually possible to configure private runners
to pick up un-tagged jobs

https://docs.gitlab.com/ee/ci/runners/configure_runners.html#runner-is-allowed-to-run-untagged-jobs

i'm not sure what the prioritization  is between shared and private
runners when using untagged jobs though. If a share runners will
pick up untagged jobs and then error them due to lack of CI credits
that might prevent our private runner picking up the untagged jobs.
We could turn off shared runners entirely potentially.

We would need the private runner to be configured with the docker
engine, so it can handle our container based approach.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 01:57:02PM +0000, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > Hi; we've run out of gitlab CI pipeline minutes for this month.
> > This leaves us the choice of:
> >  (a) don't process any more pullreqs til we get more minutes in Feb
> >  (b) merge pullreqs blindly without CI testing
> >  (c) buy more minutes
> >
> > For the moment I propose to take option (a). My mail filter will
> > continue to track pullreqs that get sent to the list, but I won't
> > do anything with them.
> >
> > If anybody has a better suggestion feel free :-)
> 
> I've submitted a support request (#366644) to GitLab to see if they will
> give us more minutes for this month. Longer term ideas:
> 
>  * Reduce compile time by reducing number of identical object files we
>    build for specific_ss
>  * Move more tests over to custom runners (don't we have an x86 box
>    somewhere?)

NB, we don't want to sacrifice coverage for contributors fork CI.

The current private runners don't do that because they're testing
scenarios that were already impossible with shared runners.

>  * Carry out an audit of code coverage for different test sets and
>    rationalise our CI set

I'm confident we can rationalize our jobs, especially the cross
compilation ones.

For each non-x86 arch we've got two sets of jobs, one for system
emulators and one for user emulators.

IMHO the most interesting part of non-x86 testing is the TCG
host target. We don't need 2 jobs to cover that, either system
or user emulators would cover TCG build / test. Most of the rest
of code is not heavily host arch dependant.

So for cross compilation we could limit ourselves to

  - 1 job per TCG host target 
  - Some full coverage job(s) for a big endian arch
  - Some full coverage job(s) for a 32-bit arch

I expect that could eliminate quite a few cross arch jobs.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 02:13:22PM +0000, Alex Bennée wrote:
> 
> Eldon Stegall <eldon-qemu@eldondev.com> writes:
> 
> > On Thu, Jan 26, 2023 at 01:22:32PM +0000, Peter Maydell wrote:
> >> Hi; we've run out of gitlab CI pipeline minutes for this month.
> >> This leaves us the choice of:
> >>  (a) don't process any more pullreqs til we get more minutes in Feb
> >>  (b) merge pullreqs blindly without CI testing
> >>  (c) buy more minutes
> >> 
> >> For the moment I propose to take option (a). My mail filter will
> >> continue to track pullreqs that get sent to the list, but I won't
> >> do anything with them.
> >> 
> >> If anybody has a better suggestion feel free :-)
> >
> > Would it be possible if (d) were to run self-hosted instances of the
> > runner? I am not sure how gitlab pricing works, but I believe on github
> > self-hosted runners are free.
> 
> Yes running more stuff on custom runners would be great (and also
> possibly not as slow as the heavily loaded shared runners).
> 
> > I have several baremetal machines colocated that I could dedicate to
> > execute these runs, dual processor xeons with a couple hundred gigs of
> > RAM. I would need approx 48 hours notice to initially provision the
> > machines. I would be happy to provide root credentials and work out IPMI
> > access if that becomes necessary.
> 
> I think we would need:
> 
>   - provisioning scripts in scripts/ci/setup (if existing not already
>     good enough)

The current approach for provisioning our private runners is highly
undesirable IMHO. We are installing the full set of build deps on
the host OS install, at time of provisioning the runner.

We should instead be provisioning the hosts exclusively to have
docker, and then use containers for the build + test environment,
so we don't need to have sysadmin intervention on the runners when
a merge request adds a build dep.

If we want to new private runners to replace the shared runners
transparently, then the use of docker is basically a must have.

>   - tweak to handle multiple runner instances (or more -j on the build)
>   - changes to .gitlab-ci.d/ so we can use those machines while keeping
>     ability to run on shared runners for those outside the project

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Peter Maydell]

On Thu, 26 Jan 2023 at 14:35, Daniel P. Berrangé <berrange@redhat.com> wrote:
> I'm confident we can rationalize our jobs, especially the cross
> compilation ones.
>
> For each non-x86 arch we've got two sets of jobs, one for system
> emulators and one for user emulators.
>
> IMHO the most interesting part of non-x86 testing is the TCG
> host target. We don't need 2 jobs to cover that, either system
> or user emulators would cover TCG build / test. Most of the rest
> of code is not heavily host arch dependant.

I'm not super enthusiastic about cutting this down.
I find the non-x86 testing is the most interesting part
of the CI -- most patch submitters and system submaintainers
have already done local compile-and-build with the common
x86_64 recent-distro target, so those parts pretty much
always succeed. The benefit of the auto CI is in keeping
the platforms that aren't so widely used by developers
working (both different-host-arch and different-OS).

thanks
-- PMM

Re: no more pullreq processing til February

[Thomas Huth]

On 26/01/2023 15.41, Peter Maydell wrote:
> On Thu, 26 Jan 2023 at 14:35, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> I'm confident we can rationalize our jobs, especially the cross
>> compilation ones.
>>
>> For each non-x86 arch we've got two sets of jobs, one for system
>> emulators and one for user emulators.
>>
>> IMHO the most interesting part of non-x86 testing is the TCG
>> host target. We don't need 2 jobs to cover that, either system
>> or user emulators would cover TCG build / test. Most of the rest
>> of code is not heavily host arch dependant.
> 
> I'm not super enthusiastic about cutting this down.
> I find the non-x86 testing is the most interesting part
> of the CI -- most patch submitters and system submaintainers
> have already done local compile-and-build with the common
> x86_64 recent-distro target, so those parts pretty much
> always succeed. The benefit of the auto CI is in keeping
> the platforms that aren't so widely used by developers
> working (both different-host-arch and different-OS).

I mostly agree. Question is whether we really need all of them, e.g. do we 
really need both, the *-armel and the *-armhf jobs for both, the -user and 
the -system part? Or would it be still ok to e.g. only have a -armel-user 
and a -armhf-system job and drop the other two?

I think there are also other possibilities where we could cut down CI 
minutes, e.g.:

- Avoid that some of the -softmmu targets get build multiple
   times

- Re-arrange the Avocodo jobs: We should maybe rather sort them
   by target system instead of host distro to avoid that some
   targets get tested twice here.

- Do we really need Linux-based clang jobs if we test Clang
   compilation with macOS and FreeBSD, too?

- Would it be OK to merge the merge the build-without-default-
   devices and build-without-default-features jobs?

And after all, I'd like to raise one question again: Could we finally stop 
supporting 32-bit hosts? ... that would really help to get rid of both, some 
CI minutes and some maintainer burden.

  Thomas

Re: no more pullreq processing til February

[Eldon Stegall]

On Thu, Jan 26, 2023 at 02:38:23PM +0000, Daniel P. Berrangé wrote:
> The current approach for provisioning our private runners is highly
> undesirable IMHO. We are installing the full set of build deps on
> the host OS install, at time of provisioning the runner.
> 
> We should instead be provisioning the hosts exclusively to have
> docker, and then use containers for the build + test environment,
> so we don't need to have sysadmin intervention on the runners when
> a merge request adds a build dep.
> 
> If we want to new private runners to replace the shared runners
> transparently, then the use of docker is basically a must have.

This is how I currently do yocto builds for my current primary concern,
provision the machines, and the docker-compose run the tests and builds.

As far as baremetal goes, I find authenticated IPXE scripts work well
for a number of these scenarios, and permit very dynamic allocation of
resources. I have been a fan of the ignition/coreos/fcos strategy for
baremetal deployment due to the capability to run the full system in
memory, as writing packaging to disk can waste time and flash in my
opinion. I strongly agree with the benefits of managing these components
in the repo. Dockerfile, ignition config, or cloud-config would probably
work.  Dockerfile makes sense to me if existing work in that direction
has interest and docker is sufficiently flexible for the tests. That
said, it may be easier to generate an appropriate cloud-config if no
work is yet done on running tests inside docker.

I have looked through the .gitlab-cl.d directory in the repo, and it
seems that there is existing work done with containers in the
container-template.yml. Do we also incur minutes for our cirrus builds
equivalent to the duration of the build on cirrus? Maybe relocation
those builds would be the most effective? It seems that a number of
builds unrelated to cirrus use containers already, or I am missing
something?

I will try to familiarize myself further with this directory. I will try
adding some runners to my fork and seeing what the tests look like
there.

Thanks,
Eldon

Re: no more pullreq processing til February

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 26/01/2023 15.41, Peter Maydell wrote:
>> On Thu, 26 Jan 2023 at 14:35, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> I'm confident we can rationalize our jobs, especially the cross
>>> compilation ones.
>>>
>>> For each non-x86 arch we've got two sets of jobs, one for system
>>> emulators and one for user emulators.
>>>
>>> IMHO the most interesting part of non-x86 testing is the TCG
>>> host target. We don't need 2 jobs to cover that, either system
>>> or user emulators would cover TCG build / test. Most of the rest
>>> of code is not heavily host arch dependant.
>> I'm not super enthusiastic about cutting this down.
>> I find the non-x86 testing is the most interesting part
>> of the CI -- most patch submitters and system submaintainers
>> have already done local compile-and-build with the common
>> x86_64 recent-distro target, so those parts pretty much
>> always succeed. The benefit of the auto CI is in keeping
>> the platforms that aren't so widely used by developers
>> working (both different-host-arch and different-OS).
>
> I mostly agree. Question is whether we really need all of them, e.g.
> do we really need both, the *-armel and the *-armhf jobs for both, the
> -user and the -system part? Or would it be still ok to e.g. only have
> a -armel-user and a -armhf-system job and drop the other two?

I suspect just the armhf target is good enough but as you say later...

> I think there are also other possibilities where we could cut down CI
> minutes, e.g.:
>
> - Avoid that some of the -softmmu targets get build multiple
>   times
>
> - Re-arrange the Avocodo jobs: We should maybe rather sort them
>   by target system instead of host distro to avoid that some
>   targets get tested twice here.

We can use tags to select groups of avocado tests I think.

> - Do we really need Linux-based clang jobs if we test Clang
>   compilation with macOS and FreeBSD, too?

Depends - is there any version drift between them?

> - Would it be OK to merge the merge the build-without-default-
>   devices and build-without-default-features jobs?

Sure

>
> And after all, I'd like to raise one question again: Could we finally
> stop supporting 32-bit hosts? ... that would really help to get rid of
> both, some CI minutes and some maintainer burden.

I'm totally down for that. Most distros have put 32 bit onto life
support haven't they?


>
>  Thomas


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: no more pullreq processing til February

[Thomas Huth]

On 26/01/2023 15.28, Peter Maydell wrote:
> On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>
>> Are you batching pull requests? I used that approach last release
>> cycle. CI takes so long to run that I didn't want to run it for every
>> pull request. Batching worked well overall.
> 
> No, I just do one test per pullreq. IME the CI is flaky
> enough that I don't really want to batch it up, and it
> isn't so slow that I build up a backlog of unprocessed
> requests.

Just an idea: Maybe you could at least batch up the PRs from the people who 
have their repo on gitlab.com and already use the gitlab CI? ... in those 
cases you can be pretty sure that at least a huge part should pass the CI. 
(and if you're worried about the non-x86 hosts, you could ask the 
maintainers to supply an URL to Travis builds, too - we still have the 
possibility to test s390x, ppc64le and aarch64 there)

  Thomas

Re: no more pullreq processing til February

[Gerd Hoffmann]

  Hi,

> Scratch that, it is actually possible to configure private runners
> to pick up un-tagged jobs
> 
> https://docs.gitlab.com/ee/ci/runners/configure_runners.html#runner-is-allowed-to-run-untagged-jobs
> 
> i'm not sure what the prioritization  is between shared and private
> runners when using untagged jobs though. If a share runners will
> pick up untagged jobs and then error them due to lack of CI credits
> that might prevent our private runner picking up the untagged jobs.

Both will pick up jobs, the shared runners are usually faster.

> We would need the private runner to be configured with the docker
> engine, so it can handle our container based approach.

Yep, then you can also use the containerized gitlab-runner.

take care,
  Gerd

Re: no more pullreq processing til February

[Markus Armbruster]

If this backpressure leads us to less waste of time & energy (close &
personal: faster make check), then I <3 gitlab!

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Thu, Jan 26, 2023 at 06:41:50PM +0000, Eldon Stegall wrote:

> As far as baremetal goes, I find authenticated IPXE scripts work well
> for a number of these scenarios, and permit very dynamic allocation of
> resources. I have been a fan of the ignition/coreos/fcos strategy for
> baremetal deployment due to the capability to run the full system in
> memory, as writing packaging to disk can waste time and flash in my
> opinion. I strongly agree with the benefits of managing these components
> in the repo. Dockerfile, ignition config, or cloud-config would probably
> work.  Dockerfile makes sense to me if existing work in that direction
> has interest and docker is sufficiently flexible for the tests. That
> said, it may be easier to generate an appropriate cloud-config if no
> work is yet done on running tests inside docker.

One of the critical factors for QEMU CI is reproducability by
contributors. This is a critical reason why we want do CI
inside containers to the greatest extent possible. It lets
the maintainer eithuer pull down the same container build, or
rebuild the container image locally. This has given us a much
better ability to reproduce CI failures than we have before
we used containers so widely.

> I have looked through the .gitlab-cl.d directory in the repo, and it
> seems that there is existing work done with containers in the
> container-template.yml. Do we also incur minutes for our cirrus builds
> equivalent to the duration of the build on cirrus? Maybe relocation
> those builds would be the most effective? It seems that a number of
> builds unrelated to cirrus use containers already, or I am missing
> something?

We have a two phase CI pipeline. In the first phase we build all
the container images that we need. This uses cache, reusing layers
from containers in the previous build to reduce time spent. In
the second phase we run the actual QEMU build jobs inside the
containers we built in the first phase.

The cirrus jobs are special. We want gitlab to act as the single
frontend for all CI jobs. So we use a tool called cirrus-run in
the gitlab job to spawn a job on Cirrus CI and pull back the
results. This is only used for FreeBSD/macOS/Windows, which is
a pretty small part of our set of jobs.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Kevin Wolf]

Am 26.01.2023 um 15:28 hat Peter Maydell geschrieben:
> On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > Are you batching pull requests? I used that approach last release
> > cycle. CI takes so long to run that I didn't want to run it for every
> > pull request. Batching worked well overall.
> 
> No, I just do one test per pullreq. IME the CI is flaky
> enough that I don't really want to batch it up, and it
> isn't so slow that I build up a backlog of unprocessed
> requests.

But obviously so slow that we've run out of minutes. It would be good if
this didn't happen every month in the future.

If it worked well enough for Stefan, I think it would be worth trying to
batch some pull requests going forward. What is the downside of it? If
CI fails and flaky tests seem to be at fault, I assume you just re-run
the job, no matter whether it tests a single pull request or two or
three of them?

Kevin

Re: no more pullreq processing til February

[Daniel P. Berrangé]

On Fri, Jan 27, 2023 at 01:39:08PM +0100, Kevin Wolf wrote:
> Am 26.01.2023 um 15:28 hat Peter Maydell geschrieben:
> > On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >
> > > Are you batching pull requests? I used that approach last release
> > > cycle. CI takes so long to run that I didn't want to run it for every
> > > pull request. Batching worked well overall.
> > 
> > No, I just do one test per pullreq. IME the CI is flaky
> > enough that I don't really want to batch it up, and it
> > isn't so slow that I build up a backlog of unprocessed
> > requests.
> 
> But obviously so slow that we've run out of minutes. It would be good if
> this didn't happen every month in the future.

Note that January has been an outlier because there was no pullreq
processing for 2 weeks due to holidays, so we were merging at a
somewhat higher rate than in previous months.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: no more pullreq processing til February

[Peter Maydell]

On Fri, 27 Jan 2023 at 12:39, Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 26.01.2023 um 15:28 hat Peter Maydell geschrieben:
> > On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >
> > > Are you batching pull requests? I used that approach last release
> > > cycle. CI takes so long to run that I didn't want to run it for every
> > > pull request. Batching worked well overall.
> >
> > No, I just do one test per pullreq. IME the CI is flaky
> > enough that I don't really want to batch it up, and it
> > isn't so slow that I build up a backlog of unprocessed
> > requests.
>
> But obviously so slow that we've run out of minutes. It would be good if
> this didn't happen every month in the future.
>
> If it worked well enough for Stefan, I think it would be worth trying to
> batch some pull requests going forward. What is the downside of it? If
> CI fails and flaky tests seem to be at fault, I assume you just re-run
> the job, no matter whether it tests a single pull request or two or
> three of them?

It means that if something fails it's harder to see whether
it was pullreq A or pullreq B. It also means there's a higher
cost to "abandon processing the merge and try a different one
to see if that one goes through and come back to this one later",
which is also something I sometimes do in an attempt to figure
out whether a problem is the usual flaky CI or not.

-- PMM

Re: no more pullreq processing til February

[Peter Maydell]

On Fri, 27 Jan 2023 at 13:11, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 27 Jan 2023 at 12:39, Kevin Wolf <kwolf@redhat.com> wrote:
> >
> > Am 26.01.2023 um 15:28 hat Peter Maydell geschrieben:
> > > On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > >
> > > > Are you batching pull requests? I used that approach last release
> > > > cycle. CI takes so long to run that I didn't want to run it for every
> > > > pull request. Batching worked well overall.
> > >
> > > No, I just do one test per pullreq. IME the CI is flaky
> > > enough that I don't really want to batch it up, and it
> > > isn't so slow that I build up a backlog of unprocessed
> > > requests.
> >
> > But obviously so slow that we've run out of minutes. It would be good if
> > this didn't happen every month in the future.
> >
> > If it worked well enough for Stefan, I think it would be worth trying to
> > batch some pull requests going forward. What is the downside of it? If
> > CI fails and flaky tests seem to be at fault, I assume you just re-run
> > the job, no matter whether it tests a single pull request or two or
> > three of them?
>
> It means that if something fails it's harder to see whether
> it was pullreq A or pullreq B. It also means there's a higher
> cost to "abandon processing the merge and try a different one
> to see if that one goes through and come back to this one later",
> which is also something I sometimes do in an attempt to figure
> out whether a problem is the usual flaky CI or not.

Put another way, I think that an important thing we need to do
is to look at all the CI failures we get and track down exactly
why we have a bunch of intermittent failures and squash them.
A lot of these problems are secondary things that we've ended up
with because we have a lot of flakiness.

-- PMM

Re: no more pullreq processing til February

[Peter Maydell]

On Fri, 27 Jan 2023 at 12:39, Kevin Wolf <kwolf@redhat.com> wrote:
> If it worked well enough for Stefan, I think it would be worth trying to
> batch some pull requests going forward. What is the downside of it? If
> CI fails and flaky tests seem to be at fault, I assume you just re-run
> the job, no matter whether it tests a single pull request or two or
> three of them?

In defence of "don't roll up multiple pullreqs", both the
pullreqs I have tried to merge so far today have had different
kinds of CI failure, so doing them in combination either
with each other or with other pullreqs would have been
counter-productive...

thanks
-- PMM