Serious doubts about Gitlab CI

Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

Hi,

For some (unclear) reason I got my free tier Gitlab account renewed and
lost the privilege for users opening account before the quota limit.

I pushed a single branch to my namespace repo to trigger a pipeline.
1h later I was surprised to see the pipeline was stuck, having completed
99 jobs of 119. Looking closer there is a red comment on top of the
pipeline:

 philmd has exceeded its pipeline minutes quota. Unless you buy
 additional pipeline minutes, no new jobs or pipelines in its projects
 will run. [Buy more Pipelines minutes]

So I exhausted my 400 monthly minutes credit.

From this FAQ:
https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage

Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
purchase additional CI/CD Minutes?

A. You will not be able to run new jobs until you purchase additional
CI/CD Minutes, or until the next month when you receive your monthly
allotted CI/CD Minutes.

Q. Will I be notified before I hit my limit on CI/CD Minutes?

A. You will receive notification banners in-app when your group has less
than 30%, 5% or exceeded your total allotted CI/CD minutes.

I indeed received 3 warnings in 7 minutes.

Now I'm having serious doubts about Gitlab usefulness for the QEMU
community...

Some screenshots FWIW:

https://pasteboard.co/JT51wGR.png
https://pasteboard.co/JT51Rqq.png
https://pasteboard.co/JT52fNL.png

Regards,

Phil.

Re: Serious doubts about Gitlab CI

[Bin Meng]

On Thu, Mar 18, 2021 at 4:32 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Hi,
>
> For some (unclear) reason I got my free tier Gitlab account renewed and
> lost the privilege for users opening account before the quota limit.
>
> I pushed a single branch to my namespace repo to trigger a pipeline.
> 1h later I was surprised to see the pipeline was stuck, having completed
> 99 jobs of 119. Looking closer there is a red comment on top of the
> pipeline:
>
>  philmd has exceeded its pipeline minutes quota. Unless you buy
>  additional pipeline minutes, no new jobs or pipelines in its projects
>  will run. [Buy more Pipelines minutes]
>
> So I exhausted my 400 monthly minutes credit.
>
> From this FAQ:
> https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage
>
> Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
> purchase additional CI/CD Minutes?
>
> A. You will not be able to run new jobs until you purchase additional
> CI/CD Minutes, or until the next month when you receive your monthly
> allotted CI/CD Minutes.
>
> Q. Will I be notified before I hit my limit on CI/CD Minutes?
>
> A. You will receive notification banners in-app when your group has less
> than 30%, 5% or exceeded your total allotted CI/CD minutes.
>
> I indeed received 3 warnings in 7 minutes.
>
> Now I'm having serious doubts about Gitlab usefulness for the QEMU
> community...
>
> Some screenshots FWIW:
>
> https://pasteboard.co/JT51wGR.png
> https://pasteboard.co/JT51Rqq.png

This snapshot shows that 2278 / 400 minutes (569%) were used. Strange?

> https://pasteboard.co/JT52fNL.png

I checked my gitlab settings, and it shows 0 / 400 minutes. However I
am pretty sure I have been using gitlab CI this month several times.

Regards,
Bin

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/18/21 2:28 AM, Bin Meng wrote:
> On Thu, Mar 18, 2021 at 4:32 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>>
>> Hi,
>>
>> For some (unclear) reason I got my free tier Gitlab account renewed and
>> lost the privilege for users opening account before the quota limit.
>>
>> I pushed a single branch to my namespace repo to trigger a pipeline.
>> 1h later I was surprised to see the pipeline was stuck, having completed
>> 99 jobs of 119. Looking closer there is a red comment on top of the
>> pipeline:
>>
>>  philmd has exceeded its pipeline minutes quota. Unless you buy
>>  additional pipeline minutes, no new jobs or pipelines in its projects
>>  will run. [Buy more Pipelines minutes]
>>
>> So I exhausted my 400 monthly minutes credit.
>>
>> From this FAQ:
>> https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage
>>
>> Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
>> purchase additional CI/CD Minutes?
>>
>> A. You will not be able to run new jobs until you purchase additional
>> CI/CD Minutes, or until the next month when you receive your monthly
>> allotted CI/CD Minutes.
>>
>> Q. Will I be notified before I hit my limit on CI/CD Minutes?
>>
>> A. You will receive notification banners in-app when your group has less
>> than 30%, 5% or exceeded your total allotted CI/CD minutes.
>>
>> I indeed received 3 warnings in 7 minutes.
>>
>> Now I'm having serious doubts about Gitlab usefulness for the QEMU
>> community...
>>
>> Some screenshots FWIW:
>>
>> https://pasteboard.co/JT51wGR.png
>> https://pasteboard.co/JT51Rqq.png
> 
> This snapshot shows that 2278 / 400 minutes (569%) were used. Strange?

Checking the timestamps, the last job was created *before* the 400min
limit. Gitlab didn't killed the running jobs once the limit was reached,
so the counter kept incrementing. Now I wonder if I buy 400 new minutes
I'll have new credit or it will be used to pay my debt...

Anyhow we have more than 100 jobs, so that mean if we use Gitlab once
a month, each job can't take more than 4min...

>> https://pasteboard.co/JT52fNL.png
> 
> I checked my gitlab settings, and it shows 0 / 400 minutes. However I
> am pretty sure I have been using gitlab CI this month several times.

You likely created your account more than 6 months ago. See
previous discussions:

https://forum.gitlab.com/t/ci-cd-minutes-for-gitlab-saas-free-tier/40241/42
https://forum.gitlab.com/t/ci-cd-minutes-for-gitlab-saas-free-tier/40241/52
https://forum.gitlab.com/t/ci-cd-minutes-for-gitlab-saas-free-tier/40241/59
https://forum.gitlab.com/t/ci-cd-minutes-for-gitlab-saas-free-tier/40241/63

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> For some (unclear) reason I got my free tier Gitlab account renewed and
> lost the privilege for users opening account before the quota limit.
> 
> I pushed a single branch to my namespace repo to trigger a pipeline.
> 1h later I was surprised to see the pipeline was stuck, having completed
> 99 jobs of 119. Looking closer there is a red comment on top of the
> pipeline:
> 
>  philmd has exceeded its pipeline minutes quota. Unless you buy
>  additional pipeline minutes, no new jobs or pipelines in its projects
>  will run. [Buy more Pipelines minutes]
> 
> So I exhausted my 400 monthly minutes credit.
> 
> From this FAQ:
> https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage
> 
> Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
> purchase additional CI/CD Minutes?
> 
> A. You will not be able to run new jobs until you purchase additional
> CI/CD Minutes, or until the next month when you receive your monthly
> allotted CI/CD Minutes.
> 
> Q. Will I be notified before I hit my limit on CI/CD Minutes?
> 
> A. You will receive notification banners in-app when your group has less
> than 30%, 5% or exceeded your total allotted CI/CD minutes.
> 
> I indeed received 3 warnings in 7 minutes.
> 
> Now I'm having serious doubts about Gitlab usefulness for the QEMU
> community...

Per the discussions in the related Forum postings about CI limites, the
400 minute limit is still only intended to apply to projects that are
marked as private.  Public projects are not even being tracked for
accounting, let alone have a limit enforced. They also said they want
to make sure they don't impact ability of users to contribute to OSS
projects hosted on GitLab that require use of CI.

It feels like what you hit here is fallout from your account accidentally
getting blocked, rather than something which is hitting every contributor
to QEMU. Did they restore projects as private perhaps ?


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/18/21 10:33 AM, Daniel P. Berrangé wrote:
> On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> For some (unclear) reason I got my free tier Gitlab account renewed and
>> lost the privilege for users opening account before the quota limit.
>>
>> I pushed a single branch to my namespace repo to trigger a pipeline.
>> 1h later I was surprised to see the pipeline was stuck, having completed
>> 99 jobs of 119. Looking closer there is a red comment on top of the
>> pipeline:
>>
>>  philmd has exceeded its pipeline minutes quota. Unless you buy
>>  additional pipeline minutes, no new jobs or pipelines in its projects
>>  will run. [Buy more Pipelines minutes]
>>
>> So I exhausted my 400 monthly minutes credit.
>>
>> From this FAQ:
>> https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage
>>
>> Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
>> purchase additional CI/CD Minutes?
>>
>> A. You will not be able to run new jobs until you purchase additional
>> CI/CD Minutes, or until the next month when you receive your monthly
>> allotted CI/CD Minutes.
>>
>> Q. Will I be notified before I hit my limit on CI/CD Minutes?
>>
>> A. You will receive notification banners in-app when your group has less
>> than 30%, 5% or exceeded your total allotted CI/CD minutes.
>>
>> I indeed received 3 warnings in 7 minutes.
>>
>> Now I'm having serious doubts about Gitlab usefulness for the QEMU
>> community...
> 
> Per the discussions in the related Forum postings about CI limites, the
> 400 minute limit is still only intended to apply to projects that are
> marked as private.  Public projects are not even being tracked for
> accounting, let alone have a limit enforced. They also said they want
> to make sure they don't impact ability of users to contribute to OSS
> projects hosted on GitLab that require use of CI.
> 
> It feels like what you hit here is fallout from your account accidentally
> getting blocked, rather than something which is hitting every contributor
> to QEMU. Did they restore projects as private perhaps ?

Yes my repository was restored as private and I had to switch it to
public. I'll try to blew everything (after backing it up) and recreate
it as public from start, and see if I get the unlimited minutes back.

Thanks,

Phil.

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/18/21 10:50 AM, Philippe Mathieu-Daudé wrote:
> On 3/18/21 10:33 AM, Daniel P. Berrangé wrote:
>> On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
>>> Hi,
>>>
>>> For some (unclear) reason I got my free tier Gitlab account renewed and
>>> lost the privilege for users opening account before the quota limit.
>>>
>>> I pushed a single branch to my namespace repo to trigger a pipeline.
>>> 1h later I was surprised to see the pipeline was stuck, having completed
>>> 99 jobs of 119. Looking closer there is a red comment on top of the
>>> pipeline:
>>>
>>>  philmd has exceeded its pipeline minutes quota. Unless you buy
>>>  additional pipeline minutes, no new jobs or pipelines in its projects
>>>  will run. [Buy more Pipelines minutes]
>>>
>>> So I exhausted my 400 monthly minutes credit.
>>>
>>> From this FAQ:
>>> https://about.gitlab.com/pricing/faq-consumption-cicd/#managing-your-cicd-minutes-usage
>>>
>>> Q. What happens if I hit the CI/CD Minutes allotted limit and forget to
>>> purchase additional CI/CD Minutes?
>>>
>>> A. You will not be able to run new jobs until you purchase additional
>>> CI/CD Minutes, or until the next month when you receive your monthly
>>> allotted CI/CD Minutes.
>>>
>>> Q. Will I be notified before I hit my limit on CI/CD Minutes?
>>>
>>> A. You will receive notification banners in-app when your group has less
>>> than 30%, 5% or exceeded your total allotted CI/CD minutes.
>>>
>>> I indeed received 3 warnings in 7 minutes.
>>>
>>> Now I'm having serious doubts about Gitlab usefulness for the QEMU
>>> community...
>>
>> Per the discussions in the related Forum postings about CI limites, the
>> 400 minute limit is still only intended to apply to projects that are
>> marked as private.  Public projects are not even being tracked for
>> accounting, let alone have a limit enforced. They also said they want
>> to make sure they don't impact ability of users to contribute to OSS
>> projects hosted on GitLab that require use of CI.
>>
>> It feels like what you hit here is fallout from your account accidentally
>> getting blocked, rather than something which is hitting every contributor
>> to QEMU. Did they restore projects as private perhaps ?
> 
> Yes my repository was restored as private and I had to switch it to
> public. I'll try to blew everything (after backing it up) and recreate
> it as public from start, and see if I get the unlimited minutes back.

You were right, I forked the project again as public and can run CI
pipelines. I note this is different that my previous account, I am
restricted at 15 jobs at a time, so this is slower.

A also added '##.gl-alert-danger.gl-alert' rule to uBlock Origin plugin
to remove the annoying big red panel saying I'm out of minutes and have
to buy more.

💔

Re: Serious doubts about Gitlab CI

[Thomas Huth]

On 18/03/2021 11.28, Philippe Mathieu-Daudé wrote:
> On 3/18/21 10:50 AM, Philippe Mathieu-Daudé wrote:
>> On 3/18/21 10:33 AM, Daniel P. Berrangé wrote:
[...]
>>> It feels like what you hit here is fallout from your account accidentally
>>> getting blocked, rather than something which is hitting every contributor
>>> to QEMU. Did they restore projects as private perhaps ?
>>
>> Yes my repository was restored as private and I had to switch it to
>> public. I'll try to blew everything (after backing it up) and recreate
>> it as public from start, and see if I get the unlimited minutes back.
> 
> You were right, I forked the project again as public and can run CI
> pipelines. I note this is different that my previous account, I am
> restricted at 15 jobs at a time, so this is slower.

That's not related to your new account. It's a new behaviour that gitlab 
introduced for everybody using the shared runners recently. At least it's 
happening for me, too. I sometimes even only get 5 jobs running at the same 
time, but sometimes also more, I think ... it likely depends on the day and 
the amount of free runners.

Anyway, I think we should try to somehow decrease the amount of 
shared-runner jobs in our CI again. It's grown too big. Some long-running 
jobs should maybe be moved to a dedicated runner instead... Cleber, where 
are we with the custom runners? Any news here?

  Thomas

Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
> Now I'm having serious doubts about Gitlab usefulness for the QEMU
> community...

The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
GitLab Merge Requests so that anyone can submit a merge request and get
CI coverage.

I think we need to expect free tiers to be insufficient for full CI
coverage. In the longer term we probably need to rely on dedicated
runners, although I'm not sure how much of the issue is QEMU's huge CI
pipeline versus the performance limitations of shared runners.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Serious doubts about Gitlab CI

[John Snow]

On 3/18/21 3:46 PM, Stefan Hajnoczi wrote:
> On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
>> Now I'm having serious doubts about Gitlab usefulness for the QEMU
>> community...
> 
> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
> GitLab Merge Requests so that anyone can submit a merge request and get
> CI coverage.
> 

How does this workflow work?

I push to my branch, I submit a MR, CI runs?

I suppose there must be a way for me to disable a CI run on my branch if 
I intend to trigger it via a MR, to avoid eating minutes twice.

--js

> I think we need to expect free tiers to be insufficient for full CI
> coverage. In the longer term we probably need to rely on dedicated
> runners, although I'm not sure how much of the issue is QEMU's huge CI
> pipeline versus the performance limitations of shared runners.
> 
> Stefan
> 

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/18/21 8:52 PM, John Snow wrote:
> On 3/18/21 3:46 PM, Stefan Hajnoczi wrote:
>> On Wed, Mar 17, 2021 at 09:29:32PM +0100, Philippe Mathieu-Daudé wrote:
>>> Now I'm having serious doubts about Gitlab usefulness for the QEMU
>>> community...
>>
>> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
>> GitLab Merge Requests so that anyone can submit a merge request and get
>> CI coverage.
>>
> 
> How does this workflow work?
> 
> I push to my branch, I submit a MR, CI runs?
> 
> I suppose there must be a way for me to disable a CI run on my branch if
> I intend to trigger it via a MR, to avoid eating minutes twice.
I use that alias in ~/.gitconfig:

[alias]
    skip-ci-push = push --push-option=ci.skip

Then:

$ git skip-ci-push [-f] myrepo mybranch

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 18/03/21 20:46, Stefan Hajnoczi wrote:
> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
> GitLab Merge Requests so that anyone can submit a merge request and get
> CI coverage.

Each merge request consumes about 2500.  That won't last long.

Paolo

Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 558 bytes --]

On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
> On 18/03/21 20:46, Stefan Hajnoczi wrote:
> > The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
> > GitLab Merge Requests so that anyone can submit a merge request and get
> > CI coverage.
> 
> Each merge request consumes about 2500.  That won't last long.

Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
on slow machines or if we'll hit the same issue with dedicated runners.
It seems like CI optimization will be necessary...

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 19/03/21 10:33, Stefan Hajnoczi wrote:
> On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
>> On 18/03/21 20:46, Stefan Hajnoczi wrote:
>>> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
>>> GitLab Merge Requests so that anyone can submit a merge request and get
>>> CI coverage.
>>
>> Each merge request consumes about 2500.  That won't last long.
> 
> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
> on slow machines or if we'll hit the same issue with dedicated runners.
> It seems like CI optimization will be necessary...

Shared runners are 1 vCPU, so it's really 41 CPU hours per CI run. 
That's a lot but not unheard of.

Almost every 2-socket server these days will have at least 50 CPUs; with 
some optimization we probably can get it down to half an hour of real 
time, on a single server running 3-4 runners with 16 vCPUs.

Paolo

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/19/21 10:41 AM, Paolo Bonzini wrote:
> On 19/03/21 10:33, Stefan Hajnoczi wrote:
>> On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
>>> On 18/03/21 20:46, Stefan Hajnoczi wrote:
>>>> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
>>>> GitLab Merge Requests so that anyone can submit a merge request and get
>>>> CI coverage.
>>>
>>> Each merge request consumes about 2500.  That won't last long.
>>
>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>> on slow machines or if we'll hit the same issue with dedicated runners.
>> It seems like CI optimization will be necessary...
> 
> Shared runners are 1 vCPU, so it's really 41 CPU hours per CI run.
> That's a lot but not unheard of.
> 
> Almost every 2-socket server these days will have at least 50 CPUs; with
> some optimization we probably can get it down to half an hour of real
> time, on a single server running 3-4 runners with 16 vCPUs.

Yesterday I tried to add my wife's computer she use at home to
my gitlab namespace to test Laurent latest series.

Specs:

- Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
- SSD 256GB
- 16GB RAM

So 1 runner with 4 vCPUs.

With 9 failed jobs, and 2 not run (due to previous stage failure),
the pipeline summary is:

130 jobs for m68k-iotests in 623 minutes and 49 seconds (queued for 31
seconds)

Network bandwidth/latency isn't an issue, I have a decent connection
IMO.

# du -chs /var/lib/docker
67G     /var/lib/docker

^ This is a lot (fresh docker install)

This matches your "41 CPU hours per CI run." comment.

Regards,

Phil.

Re: Serious doubts about Gitlab CI

[Andrew Jones]

On Fri, Mar 19, 2021 at 09:33:43AM +0000, Stefan Hajnoczi wrote:
> On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
> > On 18/03/21 20:46, Stefan Hajnoczi wrote:
> > > The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
> > > GitLab Merge Requests so that anyone can submit a merge request and get
> > > CI coverage.
> > 
> > Each merge request consumes about 2500.  That won't last long.
> 
> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
> on slow machines or if we'll hit the same issue with dedicated runners.
> It seems like CI optimization will be necessary...
>

We need to reduce the amount of CI we do, not only because we can't afford
it, but because it's wasteful. I hate to think of all the kWhs spent
testing the exact same code in the exact same way, since everyone runs
everything with a simple 'git push'. IMHO, 'git push' shouldn't trigger
anything. Starting CI should be an explicit step. Also, the default CI
should only trigger tests associated with the code changed. One should
have to explicitly trigger a complete CI when they deem it worthwhile.

Do we already have some sort of CI limiters? Or is anybody looking at
that for QEMU? Are people aware of other projects that try to limit
their CI and how they do it?

Thanks,
drew

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 19/03/21 11:18, Andrew Jones wrote:
>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>> on slow machines or if we'll hit the same issue with dedicated runners.
>> It seems like CI optimization will be necessary...
>>
> We need to reduce the amount of CI we do, not only because we can't afford
> it, but because it's wasteful. I hate to think of all the kWhs spent
> testing the exact same code in the exact same way, since everyone runs
> everything with a simple 'git push'.

Yes, I thought the same.

> IMHO, 'git push' shouldn't trigger
> anything. Starting CI should be an explicit step.

It is possible to do that on a project that uses merge requests, for 
example like this:

workflow:
   rules:
     - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
     - if: '$CI_COMMIT_BRANCH
       when: never

For us it's a bit more complicated (no merge requests).

Another common feature is failing the pipeline immediately if one of the 
jobs fail, but GitLab does not support it 
(https://gitlab.com/gitlab-org/gitlab/-/issues/23605).

> Also, the default CI
> should only trigger tests associated with the code changed. One should
> have to explicitly trigger a complete CI when they deem it worthwhile.

This is interesting.  We could add a stage that looks for changed files 
using "git diff" and sets some variables (e.g. softmmu, user, TCG, 
various targets) based on the results.  Then you use those to skip some 
jobs or some tests, for example skipping check-tcg.  See 
https://docs.gitlab.com/ee/ci/variables/#inherit-cicd-variables for more 
information.

Paolo

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/19/21 11:59 AM, Paolo Bonzini wrote:
> On 19/03/21 11:18, Andrew Jones wrote:
>>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>>> on slow machines or if we'll hit the same issue with dedicated runners.
>>> It seems like CI optimization will be necessary...
>>>
>> We need to reduce the amount of CI we do, not only because we can't
>> afford
>> it, but because it's wasteful. I hate to think of all the kWhs spent
>> testing the exact same code in the exact same way, since everyone runs
>> everything with a simple 'git push'.
> 
> Yes, I thought the same.
> 
>> IMHO, 'git push' shouldn't trigger
>> anything. Starting CI should be an explicit step.

* tests/acceptance: Only run tests tagged 'gating-ci' on GitLab CI
https://www.mail-archive.com/qemu-devel@nongnu.org/msg756464.html

* gitlab-ci: Allow forks to select & restrict build jobs
https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html

> It is possible to do that on a project that uses merge requests, for
> example like this:
> 
> workflow:
>   rules:
>     - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
>     - if: '$CI_COMMIT_BRANCH
>       when: never
> 
> For us it's a bit more complicated (no merge requests).
> 
> Another common feature is failing the pipeline immediately if one of the
> jobs fail, but GitLab does not support it
> (https://gitlab.com/gitlab-org/gitlab/-/issues/23605).
> 
>> Also, the default CI
>> should only trigger tests associated with the code changed. One should
>> have to explicitly trigger a complete CI when they deem it worthwhile.
> 
> This is interesting.  We could add a stage that looks for changed files
> using "git diff" and sets some variables (e.g. softmmu, user, TCG,
> various targets) based on the results.  Then you use those to skip some
> jobs or some tests, for example skipping check-tcg.  See
> https://docs.gitlab.com/ee/ci/variables/#inherit-cicd-variables for more
> information.
> 
> Paolo
> 
> 

Re: Serious doubts about Gitlab CI

[Wainer dos Santos Moschetta]

Hi,

On 3/19/21 8:34 AM, Philippe Mathieu-Daudé wrote:
> On 3/19/21 11:59 AM, Paolo Bonzini wrote:
>> On 19/03/21 11:18, Andrew Jones wrote:
>>>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>>>> on slow machines or if we'll hit the same issue with dedicated runners.
>>>> It seems like CI optimization will be necessary...
>>>>
>>> We need to reduce the amount of CI we do, not only because we can't
>>> afford
>>> it, but because it's wasteful. I hate to think of all the kWhs spent
>>> testing the exact same code in the exact same way, since everyone runs
>>> everything with a simple 'git push'.
>> Yes, I thought the same.
>>
>>> IMHO, 'git push' shouldn't trigger
>>> anything. Starting CI should be an explicit step.
> * tests/acceptance: Only run tests tagged 'gating-ci' on GitLab CI
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg756464.html
>
> * gitlab-ci: Allow forks to select & restrict build jobs
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html

In my opinion that series is the first step towards a smart CI. It got 
some reviews of Thomas and myself already but it didn't move ahead. If 
Philippe for some reason cannot continue that work, I'm volunteering to 
take it over.

- Wainer

>
>> It is possible to do that on a project that uses merge requests, for
>> example like this:
>>
>> workflow:
>>    rules:
>>      - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
>>      - if: '$CI_COMMIT_BRANCH
>>        when: never
>>
>> For us it's a bit more complicated (no merge requests).
>>
>> Another common feature is failing the pipeline immediately if one of the
>> jobs fail, but GitLab does not support it
>> (https://gitlab.com/gitlab-org/gitlab/-/issues/23605).
>>
>>> Also, the default CI
>>> should only trigger tests associated with the code changed. One should
>>> have to explicitly trigger a complete CI when they deem it worthwhile.
>> This is interesting.  We could add a stage that looks for changed files
>> using "git diff" and sets some variables (e.g. softmmu, user, TCG,
>> various targets) based on the results.  Then you use those to skip some
>> jobs or some tests, for example skipping check-tcg.  See
>> https://docs.gitlab.com/ee/ci/variables/#inherit-cicd-variables for more
>> information.
>>
>> Paolo
>>
>>

Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3310 bytes --]

On Fri, Mar 19, 2021 at 12:27:10PM -0300, Wainer dos Santos Moschetta wrote:
> Hi,
> 
> On 3/19/21 8:34 AM, Philippe Mathieu-Daudé wrote:
> > On 3/19/21 11:59 AM, Paolo Bonzini wrote:
> > > On 19/03/21 11:18, Andrew Jones wrote:
> > > > > Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
> > > > > on slow machines or if we'll hit the same issue with dedicated runners.
> > > > > It seems like CI optimization will be necessary...
> > > > > 
> > > > We need to reduce the amount of CI we do, not only because we can't
> > > > afford
> > > > it, but because it's wasteful. I hate to think of all the kWhs spent
> > > > testing the exact same code in the exact same way, since everyone runs
> > > > everything with a simple 'git push'.
> > > Yes, I thought the same.
> > > 
> > > > IMHO, 'git push' shouldn't trigger
> > > > anything. Starting CI should be an explicit step.
> > * tests/acceptance: Only run tests tagged 'gating-ci' on GitLab CI
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg756464.html
> > 
> > * gitlab-ci: Allow forks to select & restrict build jobs
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html
> 
> In my opinion that series is the first step towards a smart CI. It got some
> reviews of Thomas and myself already but it didn't move ahead. If Philippe
> for some reason cannot continue that work, I'm volunteering to take it over.

Hi,
I wanted to follow up with a summary of the CI jobs:

1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
2. Builds - ~50 minutes/job x 61 jobs
3. Tests - ~12 minutes/job x 20 jobs
4. Deploy - 52 minutes x 1 job

The Builds phase consumes the most CI minutes. If we can optimize this
phase then we'll achieve the biggest impact.

In the short term builds could be disabled. However, in the long term I
think full build coverage is desirable to prevent merging code that
breaks certain host OSes/architectures (e.g. stable Linux distros,
macOS, etc).

Traditionally ccache (https://ccache.dev/) was used to detect
recompilation of the same compiler input files. This is trickier to do
in GitLab CI since it would be necessary to share and update a cache,
potentially between untrusted users. Unfortunately this shifts the
bottleneck from CPU to network in a CI-as-a-Service environment since
the cached build output needs to be accessed by the linker on the CI
runner but is stored remotely.

A complementary approach is avoiding compilation altogether when code
changes do not affect a build target. For example, a change to
qemu-storage-daemon.c does not require rebuilding the system emulator
targets. Either the compiler or the build system could produce a
manifest of source files that went into a build target, and that
information is what's needed to avoid compiling unchanged targets.

Ideally the CI would look at the code changes and only launch jobs that
were affected. Those jobs would use a C compiler cache to avoid
rebuilding compiler input that has not changed. Basically, we need
incremental builds.

This is as far as I've gotten with thinking about CI efficiency. Do you
think these optimizations are worth investigating or should we keep it
simple and just disable many builds by default?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
> Hi,
> I wanted to follow up with a summary of the CI jobs:
> 
> 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
> 2. Builds - ~50 minutes/job x 61 jobs
> 3. Tests - ~12 minutes/job x 20 jobs
> 4. Deploy - 52 minutes x 1 job
> 
> The Builds phase consumes the most CI minutes. If we can optimize this
> phase then we'll achieve the biggest impact.
> 
> In the short term builds could be disabled. However, in the long term I
> think full build coverage is desirable to prevent merging code that
> breaks certain host OSes/architectures (e.g. stable Linux distros,
> macOS, etc).

The notion of "full build coverage" doesn't really exist in reality.
The number of platforms that QEMU is targetting, combined with the
number of features that can be turned on/off in QEMU configure
means that the matrix for "full build coverage" is too huge to ever
contemplate.

So far we've been adding new jobs whenever we hit some situation
where we found a build problem that wasn't previously detected by
CI. In theory this is more reasonable as a strategy, than striving
for full build coverage, as it targets only places where we've hit
real world problems. I think we're seeing though, that even the
incremental new coverage approach is not sustainable in the real
world. Or rather it is only sustainable if CI resources are
essentially free.


Traditionally the biggest amount of testing would be done in a
freeze period leading upto a release. WIth GitLab CI we've tried
to move to a model where testing is continuous, such that we
have git master in a so called "always ready" state. This is
very good in general, but it comes with significant hardware
resource costs. We've relied on free service for this and this
is being less viable.



I think a challenges we have with our incremental approach is that
we're not really taking into account relative importance of the
different build scenarios, and often don't look at the big picture
of what the new job adds in terms of quality, compared to existing
jobs.

eg Consider we have

  build-system-alpine:
  build-system-ubuntu:
  build-system-debian:
  build-system-fedora:
  build-system-centos:
  build-system-opensuse:

  build-trace-multi-user:
  build-trace-ftrace-system:
  build-trace-ust-system:

I'd question whether we really need any of those 'build-trace'
jobs. Instead, we could have build-system-ubuntu pass
--enable-trace-backends=log,simple,syslog, build-system-debian
pass --enable-trace-backends=ust and build-system-fedora
pass --enable-trace-backends=ftrace, etc. 

Another example, is that we test builds on centos7 with
three different combos of crypto backend settings. This was
to exercise bugs we've seen in old crypto packages in RHEL-7
but in reality, it is probably overkill, because downstream
RHEL-7 only cares about one specific combination.

We don't really have a clearly defined plan to identify what
the most important things are in our testing coverage, so we
tend to accept anything without questioning its value add.
This really feeds back into the idea I've brought up many
times in the past, that we need to better define what we aim
to support in QEMU and its quality level, which will influence
what are the scenarios we care about testing.


> Traditionally ccache (https://ccache.dev/) was used to detect
> recompilation of the same compiler input files. This is trickier to do
> in GitLab CI since it would be necessary to share and update a cache,
> potentially between untrusted users. Unfortunately this shifts the
> bottleneck from CPU to network in a CI-as-a-Service environment since
> the cached build output needs to be accessed by the linker on the CI
> runner but is stored remotely.

Our docker containers install ccache already and I could have sworn
that we use that in gitlab, but now I'm not so sure. We're only
saving the "build/" directory as an artifact between jobs, and I'm
not sure that directory holds the ccache cache.

> A complementary approach is avoiding compilation altogether when code
> changes do not affect a build target. For example, a change to
> qemu-storage-daemon.c does not require rebuilding the system emulator
> targets. Either the compiler or the build system could produce a
> manifest of source files that went into a build target, and that
> information is what's needed to avoid compiling unchanged targets.

I think we want to be pretty wary of making the CI jobs too complex
in what they do. We want them to accurately reflect the way that our
developers and end users build the system in general. Trying to add
clever logic to the CI system to skip building certain pieces will
make the CI system more complex and fragile which will increase the
burden of keeping CI working reliably.

> Ideally the CI would look at the code changes and only launch jobs that
> were affected. Those jobs would use a C compiler cache to avoid
> rebuilding compiler input that has not changed. Basically, we need
> incremental builds.

If we want to consider "code changes" between CI runs, then we need
to establish as baseline. If we're triggering GitLab jobs on "push"
events, then the baseline is whatever content already exists in
the remote server. eg if you have a branch with 10 commits delta
on top of "master", but 8 of those commits already exist in the
branch on gitlab, then the push event baseline is those 8 commits,
so it'll only look at changes in the 2 top commits, rather than
the entire 10 commits of that branch.  This is generally *not*
what we want for testing, because we can't assume that the 8
commits which already exist have successfully passed CI. We've
seen this cause us problems for CI already, when we tried to
filter out jobs rebuilding container images, so they only ran
when a tests/docker/* file was modified. 

If we want to consider code changes where "master" is the baseline,
then we need to trigger CI pipelines from merge requests, because
merge requests have an explicit baseline associated with them. Of
course this means we need to be using merge requests in some way
which is a big can of worms.

> This is as far as I've gotten with thinking about CI efficiency. Do you
> think these optimizations are worth investigating or should we keep it
> simple and just disable many builds by default?

ccache is a no-brainer and assuming it isn't already working with
our gitlab jobs, we must fix that asap.


Aside from optimizing CI, we should consider whether there's more we
can do to optimize build process itself. We've done alot of work, but
there's still plenty of stuff we build multiple times, once for each
target. Perhaps there's scope for cutting this down in some manner ?

I'm unclear how many jobs in CI are build submodules, but if there's
more scope for using the pre-built distro packages that's going to
be beneficial in build time.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Thomas Huth]

On 30/03/2021 13.19, Daniel P. Berrangé wrote:
> On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
>> Hi,
>> I wanted to follow up with a summary of the CI jobs:
>>
>> 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
>> 2. Builds - ~50 minutes/job x 61 jobs
>> 3. Tests - ~12 minutes/job x 20 jobs
>> 4. Deploy - 52 minutes x 1 job

I hope that 52 was just a typo ... ?

> I think a challenges we have with our incremental approach is that
> we're not really taking into account relative importance of the
> different build scenarios, and often don't look at the big picture
> of what the new job adds in terms of quality, compared to existing
> jobs.
> 
> eg Consider we have
> 
>    build-system-alpine:
>    build-system-ubuntu:
>    build-system-debian:
>    build-system-fedora:
>    build-system-centos:
>    build-system-opensuse:

I guess we could go through that list of jobs and remove the duplicated 
target CPUs, e.g. it should be enough to test x86_64-softmmu only once.

>    build-trace-multi-user:
>    build-trace-ftrace-system:
>    build-trace-ust-system:
> 
> I'd question whether we really need any of those 'build-trace'
> jobs. Instead, we could have build-system-ubuntu pass
> --enable-trace-backends=log,simple,syslog, build-system-debian
> pass --enable-trace-backends=ust and build-system-fedora
> pass --enable-trace-backends=ftrace, etc.

I recently had the very same idea already:

  https://gitlab.com/qemu-project/qemu/-/commit/65aff82076a9bbfdf7

:-)

> Another example, is that we test builds on centos7 with
> three different combos of crypto backend settings. This was
> to exercise bugs we've seen in old crypto packages in RHEL-7
> but in reality, it is probably overkill, because downstream
> RHEL-7 only cares about one specific combination.

Care to send a patch? Or shall we just wait one more months and then remove 
these jobs (since we won't support RHEL7 after QEMU 6.0 anymore)?

> We don't really have a clearly defined plan to identify what
> the most important things are in our testing coverage, so we
> tend to accept anything without questioning its value add.
> This really feeds back into the idea I've brought up many
> times in the past, that we need to better define what we aim
> to support in QEMU and its quality level, which will influence
> what are the scenarios we care about testing.

But code that we have in the repository should get at least some basic test 
coverage, otherwise it bitrots soon ... so it's maybe rather the other old 
problem that we struggle with, that we should deprecate more code and remove 
it if nobody cares about it...

>> Traditionally ccache (https://ccache.dev/) was used to detect
>> recompilation of the same compiler input files. This is trickier to do
>> in GitLab CI since it would be necessary to share and update a cache,
>> potentially between untrusted users. Unfortunately this shifts the
>> bottleneck from CPU to network in a CI-as-a-Service environment since
>> the cached build output needs to be accessed by the linker on the CI
>> runner but is stored remotely.
> 
> Our docker containers install ccache already and I could have sworn
> that we use that in gitlab, but now I'm not so sure. We're only
> saving the "build/" directory as an artifact between jobs, and I'm
> not sure that directory holds the ccache cache.

AFAIK we never really enabled ccache in the gitlab-CI, only in Travis.

>> This is as far as I've gotten with thinking about CI efficiency. Do you
>> think these optimizations are worth investigating or should we keep it
>> simple and just disable many builds by default?
> 
> ccache is a no-brainer and assuming it isn't already working with
> our gitlab jobs, we must fix that asap.

I've found some nice instructions here:

https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/

... and just kicked off a build with these modifications, let's see how it 
goes...

> Aside from optimizing CI, we should consider whether there's more we
> can do to optimize build process itself. We've done alot of work, but
> there's still plenty of stuff we build multiple times, once for each
> target. Perhaps there's scope for cutting this down in some manner ?

Right, I think we should also work more towards consolidating the QEMU 
binaries, to avoid that we have to always build sooo many target binaries 
again and again. E.g.:

- Do we still need to support 32-bit hosts? If not we could
   finally get rid of qemu-system-i386, qemu-system-ppc,
   qemu-system-arm, etc. and just provide the 64-bit variants

- Could we maybe somehow unify the targets that have both, big
   and little endian versions? Then we could merge e.g.
   qemu-system-microblaze and qemu-system-microblazeel etc.

- Or could we maybe even build a unified qemu-system binary that
   contains all target CPUs? ... that would also allow e.g.
   machines with a x86 main CPU and an ARM-based board management
   controller...

> I'm unclear how many jobs in CI are build submodules, but if there's
> more scope for using the pre-built distro packages that's going to
> be beneficial in build time.

Since the build system has been converted to meson, I think the configure 
script prefers to use the submodules instead of the distro packages. I've 
tried to remedy this a little bit here:

https://gitlab.com/qemu-project/qemu/-/commit/db0108d5d846e9a8

... but new jobs of course will use the submodules again if the author is 
not careful.

I think we should tweak that behavior again to use the system version of 
capstone, slirp and dtc instead if these are available. Paolo, what do you 
think?

Also I wonder whether we could maybe even get rid of the capstone and slirp 
submodules in QEMU now ... these libraries should be available in the most 
distros by now, and otherwise people could also install them manually instead?

  Thomas

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 30/03/21 13:55, Thomas Huth wrote:
> 
> Since the build system has been converted to meson, I think the 
> configure script prefers to use the submodules instead of the distro 
> packages. I've tried to remedy this a little bit here:
> 
> https://gitlab.com/qemu-project/qemu/-/commit/db0108d5d846e9a8
> 
> ... but new jobs of course will use the submodules again if the author 
> is not careful.

Hmm... it should be the same (or if not it's a bug).

> Also I wonder whether we could maybe even get rid of the capstone and slirp submodules in QEMU now

At least for slirp, we probably want to stay more on the bleeding edge 
which implies having to keep the submodule.  Capstone and libfdt 
probably can go.

Paolo

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/30/21 2:09 PM, Paolo Bonzini wrote:
> On 30/03/21 13:55, Thomas Huth wrote:
>>
>> Also I wonder whether we could maybe even get rid of the capstone and
>> slirp submodules in QEMU now
> 
> At least for slirp, we probably want to stay more on the bleeding edge
> which implies having to keep the submodule.

FYI QEMU libSLiRP submodule doesn't point to bleeding edge branch but to
the stable branch (which should be what distributions package).

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 30/03/21 14:23, Philippe Mathieu-Daudé wrote:
> On 3/30/21 2:09 PM, Paolo Bonzini wrote:
>> On 30/03/21 13:55, Thomas Huth wrote:
>>>
>>> Also I wonder whether we could maybe even get rid of the capstone and
>>> slirp submodules in QEMU now
>>
>> At least for slirp, we probably want to stay more on the bleeding edge
>> which implies having to keep the submodule.
> 
> FYI QEMU libSLiRP submodule doesn't point to bleeding edge branch but to
> the stable branch (which should be what distributions package).

Now, but that may change already in 6.1 in order to add CFI support.

Paolo

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Tue, Mar 30, 2021 at 02:45:43PM +0200, Paolo Bonzini wrote:
> On 30/03/21 14:23, Philippe Mathieu-Daudé wrote:
> > On 3/30/21 2:09 PM, Paolo Bonzini wrote:
> > > On 30/03/21 13:55, Thomas Huth wrote:
> > > > 
> > > > Also I wonder whether we could maybe even get rid of the capstone and
> > > > slirp submodules in QEMU now
> > > 
> > > At least for slirp, we probably want to stay more on the bleeding edge
> > > which implies having to keep the submodule.
> > 
> > FYI QEMU libSLiRP submodule doesn't point to bleeding edge branch but to
> > the stable branch (which should be what distributions package).
> 
> Now, but that may change already in 6.1 in order to add CFI support.

We can bundle a newer version, but we don't need to require a newer
version. Simply conditional compile for the bits we need. If distro
slirp is too old, then sorry, you can't enable CFI + slirp at the
same time. If the distro really wants that combination we don't have
to own the solution - the distro should update their slirp.

Or to put it another way, QEMU doesn't need to go out of its way to
enable new features on old distros. We merely need to not regress
in the features we previously offered.  We bundled slirp as a submodule
so that old distros didn't loose slirp entirely. We don't need to
offer CFI on those distros.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 30/03/21 15:12, Daniel P. Berrangé wrote:
>> Now, but that may change already in 6.1 in order to add CFI support.
> We can bundle a newer version, but we don't need to require a newer
> version. Simply conditional compile for the bits we need. If distro
> slirp is too old, then sorry, you can't enable CFI + slirp at the
> same time. If the distro really wants that combination we don't have
> to own the solution - the distro should update their slirp.
> 
> Or to put it another way, QEMU doesn't need to go out of its way to
> enable new features on old distros. We merely need to not regress
> in the features we previously offered.  We bundled slirp as a submodule
> so that old distros didn't loose slirp entirely. We don't need to
> offer CFI on those distros.

This is true, on the other hand only having to support one API version 
has its benefits.  The complication in the build system is minimal once 
slirp is made into a subproject; therefore it is appealing to keep the 
QEMU code simple.

Paolo

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Tue, Mar 30, 2021 at 03:19:49PM +0200, Paolo Bonzini wrote:
> On 30/03/21 15:12, Daniel P. Berrangé wrote:
> > > Now, but that may change already in 6.1 in order to add CFI support.
> > We can bundle a newer version, but we don't need to require a newer
> > version. Simply conditional compile for the bits we need. If distro
> > slirp is too old, then sorry, you can't enable CFI + slirp at the
> > same time. If the distro really wants that combination we don't have
> > to own the solution - the distro should update their slirp.
> > 
> > Or to put it another way, QEMU doesn't need to go out of its way to
> > enable new features on old distros. We merely need to not regress
> > in the features we previously offered.  We bundled slirp as a submodule
> > so that old distros didn't loose slirp entirely. We don't need to
> > offer CFI on those distros.
> 
> This is true, on the other hand only having to support one API version has
> its benefits.  The complication in the build system is minimal once slirp is
> made into a subproject; therefore it is appealing to keep the QEMU code
> simple.

I don't think slirp is special in this regard. The benefit you're promoting
here applies to any dependancy we have, but I think the benefit is not big
enough to justify.

The use of submodules has imposed significant pain on QEMU developers over
the years, and as such I think our general goal should be to have zero git
submodules over the long term. Usage of submodules ought to be considered
a short term workaround only, with a clear criteria for removal. We should
continually introduce dependancies on newer & newer versions, as that means
we'll never have any opportunity to remove them and reduce the cost on
QEMU.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Tue, Mar 30, 2021 at 7:33 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> The use of submodules has imposed significant pain on QEMU developers over
> the years, and as such I think our general goal should be to have zero git
> submodules over the long term. Usage of submodules ought to be considered
> a short term workaround only, with a clear criteria for removal. We should
> continually introduce dependancies on newer & newer versions, as that means
> we'll never have any opportunity to remove them and reduce the cost on
> QEMU.
>

submodules have caused me significant pain in rebasing the bsd-user work.
The way QEMU does things, you wind up with unclean trees after a build,
which causes grief at times... I for one, would shed no tears at the number
of
submodules dropping to 0.

Warner

[-- Attachment #2: Type: text/html, Size: 1219 bytes --]

Re: Serious doubts about Gitlab CI

[Peter Maydell]

On Tue, 30 Mar 2021 at 17:00, Warner Losh <imp@bsdimp.com> wrote:
> submodules have caused me significant pain in rebasing the bsd-user work.
> The way QEMU does things, you wind up with unclean trees after a build,
> which causes grief at times... I for one, would shed no tears at the number of
> submodules dropping to 0.

I agree that submodules are bad, but in general you shouldn't have
an unclean tree after build as a result of them. (The exception I'm
aware of is if you have to 'git checkout' an older revision where a
submodule that exists now didn't exist back then; git leaves the
submodule directory in the source tree and you have to rm it manually.
But that's "after checkout", not "after build".)

The main bear-trap IMHO is that you have to remember to
"git submodule update" when checking out different revs of
trunk to keep the submodules at the right version. I wish
git just automatically kept submodules at the right rev for
whatever trunk rev you're currently on.

-- PMM

Re: Serious doubts about Gitlab CI

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

On Tue, Mar 30, 2021 at 10:11 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 30 Mar 2021 at 17:00, Warner Losh <imp@bsdimp.com> wrote:
> > submodules have caused me significant pain in rebasing the bsd-user work.
> > The way QEMU does things, you wind up with unclean trees after a build,
> > which causes grief at times... I for one, would shed no tears at the
> number of
> > submodules dropping to 0.
>
> I agree that submodules are bad, but in general you shouldn't have
> an unclean tree after build as a result of them. (The exception I'm
> aware of is if you have to 'git checkout' an older revision where a
> submodule that exists now didn't exist back then; git leaves the
> submodule directory in the source tree and you have to rm it manually.
> But that's "after checkout", not "after build".)
>

It may be that. I routinely have issues when switching back and forth
between
3.1 based rebase and tip of master. I chalked it up to some submodule update
that was done as part of the built, but I could well be wrong. subrepos are
a
thing I've not managed so my subrepo foo is weak.


> The main bear-trap IMHO is that you have to remember to
> "git submodule update" when checking out different revs of
> trunk to keep the submodules at the right version. I wish
> git just automatically kept submodules at the right rev for
> whatever trunk rev you're currently on.


I'll have to try it next time. Maybe it was the only hassle and I was
unaware.
But hopefully I'll be focused on doing a 'logical rebase' of committable
things
that I've started and that will ultimately render this problem obsolete for
me.

Warner

[-- Attachment #2: Type: text/html, Size: 2422 bytes --]

Re: Serious doubts about Gitlab CI

[Thomas Huth]

On 30/03/2021 15.27, Daniel P. Berrangé wrote:
> On Tue, Mar 30, 2021 at 03:19:49PM +0200, Paolo Bonzini wrote:
>> On 30/03/21 15:12, Daniel P. Berrangé wrote:
>>>> Now, but that may change already in 6.1 in order to add CFI support.
>>> We can bundle a newer version, but we don't need to require a newer
>>> version. Simply conditional compile for the bits we need. If distro
>>> slirp is too old, then sorry, you can't enable CFI + slirp at the
>>> same time. If the distro really wants that combination we don't have
>>> to own the solution - the distro should update their slirp.
>>>
>>> Or to put it another way, QEMU doesn't need to go out of its way to
>>> enable new features on old distros. We merely need to not regress
>>> in the features we previously offered.  We bundled slirp as a submodule
>>> so that old distros didn't loose slirp entirely. We don't need to
>>> offer CFI on those distros.
>>
>> This is true, on the other hand only having to support one API version has
>> its benefits.  The complication in the build system is minimal once slirp is
>> made into a subproject; therefore it is appealing to keep the QEMU code
>> simple.
> 
> I don't think slirp is special in this regard. The benefit you're promoting
> here applies to any dependancy we have, but I think the benefit is not big
> enough to justify.
> 
> The use of submodules has imposed significant pain on QEMU developers over
> the years, and as such I think our general goal should be to have zero git
> submodules over the long term. Usage of submodules ought to be considered
> a short term workaround only, with a clear criteria for removal.
I agree with Daniel. Submodules keep being a pain. For all libs that are 
mature and widely available enough in the distros now, we should consider to 
drop the submodules rather sooner than later.

  Thomas

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Tue, Mar 30, 2021 at 02:09:38PM +0200, Paolo Bonzini wrote:
> On 30/03/21 13:55, Thomas Huth wrote:
> > 
> > Since the build system has been converted to meson, I think the
> > configure script prefers to use the submodules instead of the distro
> > packages. I've tried to remedy this a little bit here:
> > 
> > https://gitlab.com/qemu-project/qemu/-/commit/db0108d5d846e9a8
> > 
> > ... but new jobs of course will use the submodules again if the author
> > is not careful.
> 
> Hmm... it should be the same (or if not it's a bug).
> 
> > Also I wonder whether we could maybe even get rid of the capstone and slirp submodules in QEMU now
> 
> At least for slirp, we probably want to stay more on the bleeding edge which
> implies having to keep the submodule.  Capstone and libfdt probably can go.

I don't think we need to stay on the bleeding edge per-se in terms of
what we build against

We have a declared minimum version of libslirp that we absolutely must
have in order to get the API we need for core featureset. If new APIs
are introduced, it is quite reasonable for us to make their usage in
QEMU conditional, just as we would for any other 3rd party library we
use.

The reason to have slirp as a submodule is just to avoid a functional
regression on distros which don't have slirp available at all, and
which we don't expect to introduce it.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 30/03/21 13:55, Thomas Huth wrote:
> 
> Since the build system has been converted to meson, I think the 
> configure script prefers to use the submodules instead of the distro 
> packages. I've tried to remedy this a little bit here:
> 
> https://gitlab.com/qemu-project/qemu/-/commit/db0108d5d846e9a8
> 
> ... but new jobs of course will use the submodules again if the author 
> is not careful.

Hmm... it should be the same (or if not it's a bug).

> Also I wonder whether we could maybe even get rid of the capstone and slirp submodules in QEMU now

At least for slirp, we probably want to stay more on the bleeding edge 
which implies having to keep the submodule.  Capstone and libfdt 
probably can go, though at least libfdt may be more useful on Windows.

Paolo

Re: Serious doubts about Gitlab CI

[Peter Maydell]

On Tue, 30 Mar 2021 at 12:56, Thomas Huth <thuth@redhat.com> wrote:
> Right, I think we should also work more towards consolidating the QEMU
> binaries, to avoid that we have to always build sooo many target binaries
> again and again. E.g.:
>
> - Do we still need to support 32-bit hosts? If not we could
>    finally get rid of qemu-system-i386, qemu-system-ppc,
>    qemu-system-arm, etc. and just provide the 64-bit variants

We could drop qemu-system-i386 &c without dropping 32-bit host
support (except for the special case of wanting to use KVM):
32-bit host TCG happily runs the qemu-system-foo64 binary.
This does depend on the target arch having been set up so that
the 64-bit version works exactly like the 32-bit one for 32-bit
guest boards, though -- arm does this. I think x86 mostly does
except for differences like the default guest CPU type. riscv
used to have a "32 bit cpus only in the qemu-system-foo64 binary"
setup but I think that is either fixed or being fixed. There's
also the issue that it breaks existing working user commandlines,
of course.

> - Could we maybe somehow unify the targets that have both, big
>    and little endian versions? Then we could merge e.g.
>    qemu-system-microblaze and qemu-system-microblazeel etc.
>
> - Or could we maybe even build a unified qemu-system binary that
>    contains all target CPUs? ... that would also allow e.g.
>    machines with a x86 main CPU and an ARM-based board management
>    controller...

I would like to see this one day, but it's a pretty non-trivial
amount of engineering work to identify all the places where we
currently hard-code a compile-time setting about the target
architecture and make them runtime instead (in a way that doesn't
torpedo performance). "is the target CPU big-endian" is one of those...

> Also I wonder whether we could maybe even get rid of the capstone and slirp
> submodules in QEMU now ... these libraries should be available in the most
> distros by now, and otherwise people could also install them manually instead?

I suspect that's rather overoptimistic, but how widely available they
are is a question of fact that we can check.

thanks
-- PMM

Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/30/21 1:55 PM, Thomas Huth wrote:
> On 30/03/2021 13.19, Daniel P. Berrangé wrote:
>> On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:

>>> Traditionally ccache (https://ccache.dev/) was used to detect
>>> recompilation of the same compiler input files. This is trickier to do
>>> in GitLab CI since it would be necessary to share and update a cache,
>>> potentially between untrusted users. Unfortunately this shifts the
>>> bottleneck from CPU to network in a CI-as-a-Service environment since
>>> the cached build output needs to be accessed by the linker on the CI
>>> runner but is stored remotely.
>>
>> Our docker containers install ccache already and I could have sworn
>> that we use that in gitlab, but now I'm not so sure. We're only
>> saving the "build/" directory as an artifact between jobs, and I'm
>> not sure that directory holds the ccache cache.
> 
> AFAIK we never really enabled ccache in the gitlab-CI, only in Travis.

Back then the Travis setup was simpler, and it took me 2 to 3 weeks
to get it right (probably spending 3 to 4h a day on it).

>>> This is as far as I've gotten with thinking about CI efficiency. Do you
>>> think these optimizations are worth investigating or should we keep it
>>> simple and just disable many builds by default?
>>
>> ccache is a no-brainer and assuming it isn't already working with
>> our gitlab jobs, we must fix that asap.
> 
> I've found some nice instructions here:
> 
> https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/
> 
> ... and just kicked off a build with these modifications, let's see how
> it goes...

But we cross-build in Docker containers, so you need to mount the
cache dir in the container and set the CCACHE_DIR env var, isn't it?

Watch out about custom runners. If we do too many changes on the
free-tier runners, we'll never have the custom runner series integrated.

My 2 cents.

Regards,

Phil.

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
> On 30/03/2021 13.19, Daniel P. Berrangé wrote:

> > Another example, is that we test builds on centos7 with
> > three different combos of crypto backend settings. This was
> > to exercise bugs we've seen in old crypto packages in RHEL-7
> > but in reality, it is probably overkill, because downstream
> > RHEL-7 only cares about one specific combination.
> 
> Care to send a patch? Or shall we just wait one more months and then remove
> these jobs (since we won't support RHEL7 after QEMU 6.0 anymore)?

Yeah, we'll be able to cull this entirely very soon, including
both the C backcompat code and CI jobs at the same time, so I'll
just wait.


> > Our docker containers install ccache already and I could have sworn
> > that we use that in gitlab, but now I'm not so sure. We're only
> > saving the "build/" directory as an artifact between jobs, and I'm
> > not sure that directory holds the ccache cache.
> 
> AFAIK we never really enabled ccache in the gitlab-CI, only in Travis.
> 
> > > This is as far as I've gotten with thinking about CI efficiency. Do you
> > > think these optimizations are worth investigating or should we keep it
> > > simple and just disable many builds by default?
> > 
> > ccache is a no-brainer and assuming it isn't already working with
> > our gitlab jobs, we must fix that asap.
> 
> I've found some nice instructions here:
> 
> https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/
> 
> ... and just kicked off a build with these modifications, let's see how it
> goes...

Yep, that looks similar to what we do in libvirt, though we don't override
the compiler at the job level. Instead we just ensure the dir containing
ccache symlinks appears first in $PATH.

So in containers we have this:

https://gitlab.com/libvirt/libvirt/-/blob/master/ci/containers/centos-8.Dockerfile

and in gitlab-ci.yml we have env vars set

  export CCACHE_BASEDIR="$(pwd)"
  export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
  export CCACHE_MAXSIZE="500M"
  export PATH="$CCACHE_WRAPPERSDIR:$PATH"

And per-job caches:

  cache:
    paths:
      - ccache/
    key: "$CI_JOB_NAME"

note the "key" is important to avoid clashing caches from different
envs.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Thomas Huth]

On 30/03/2021 15.19, Daniel P. Berrangé wrote:
> On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
>> On 30/03/2021 13.19, Daniel P. Berrangé wrote:
[...]
>>> ccache is a no-brainer and assuming it isn't already working with
>>> our gitlab jobs, we must fix that asap.
>>
>> I've found some nice instructions here:
>>
>> https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/
>>
>> ... and just kicked off a build with these modifications, let's see how it
>> goes...
> 
> Yep, that looks similar to what we do in libvirt, though we don't override
> the compiler at the job level. Instead we just ensure the dir containing
> ccache symlinks appears first in $PATH.

That's of course a nice idea, too, but it does not seem to work with the 
clang builds:

https://gitlab.com/thuth/libvirt/-/jobs/1142239591#L3780
https://gitlab.com/thuth/libvirt/-/jobs/1142239581#L3738

  Thomas

Re: Serious doubts about Gitlab CI

[Andrea Bolognani]

On Wed, 2021-03-31 at 09:54 +0200, Thomas Huth wrote:
> On 30/03/2021 15.19, Daniel P. Berrangé wrote:
> > Yep, that looks similar to what we do in libvirt, though we don't override
> > the compiler at the job level. Instead we just ensure the dir containing
> > ccache symlinks appears first in $PATH.
> 
> That's of course a nice idea, too, but it does not seem to work with the 
> clang builds:
> 
> https://gitlab.com/thuth/libvirt/-/jobs/1142239591#L3780
> https://gitlab.com/thuth/libvirt/-/jobs/1142239581#L3738

That's because the corresponding Dockerfile doesn't contain
instructions to create a symlink for clang:

  https://gitlab.com/libvirt/libvirt/-/blob/master/ci/containers/fedora-rawhide.Dockerfile#L105-107

It's simply a bug in lcitool that needs to be addressed.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
> On 30/03/2021 13.19, Daniel P. Berrangé wrote:
> > On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
> > > Hi,
> > > I wanted to follow up with a summary of the CI jobs:
> > > 
> > > 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
> > > 2. Builds - ~50 minutes/job x 61 jobs
> > > 3. Tests - ~12 minutes/job x 20 jobs
> > > 4. Deploy - 52 minutes x 1 job
> 
> I hope that 52 was just a typo ... ?

No, but I think Dan already found this issue a little while ago. The
deploy job uses "make install":

  # Prepare for GitLab pages deployment. Anything copied into the
  # "public" directory will be deployed to $USER.gitlab.io/$PROJECT
  pages:
    image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
    stage: test
    needs:
      - job: build-tools-and-docs-debian
    script:
      - mkdir -p public
      # HTML-ised source tree
      - make gtags
      - htags -anT --tree-view=filetree -m qemu_init
          -t "Welcome to the QEMU sourcecode"
      - mv HTML public/src
      # Project documentation
      - make -C build install DESTDIR=$(pwd)/temp-install
      - mv temp-install/usr/local/share/doc/qemu/* public/
    artifacts:
      paths:
        - public

Do we have/need a docs-only install target?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Serious doubts about Gitlab CI

[Paolo Bonzini]

On 30/03/21 16:13, Stefan Hajnoczi wrote:
> On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
>> On 30/03/2021 13.19, Daniel P. Berrangé wrote:
>>> On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
>>>> Hi,
>>>> I wanted to follow up with a summary of the CI jobs:
>>>>
>>>> 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
>>>> 2. Builds - ~50 minutes/job x 61 jobs
>>>> 3. Tests - ~12 minutes/job x 20 jobs
>>>> 4. Deploy - 52 minutes x 1 job
>>
>> I hope that 52 was just a typo ... ?
> 
> No, but I think Dan already found this issue a little while ago. The
> deploy job uses "make install":
> 
>    # Prepare for GitLab pages deployment. Anything copied into the
>    # "public" directory will be deployed to $USER.gitlab.io/$PROJECT
>    pages:
>      image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
>      stage: test
>      needs:
>        - job: build-tools-and-docs-debian
>      script:
>        - mkdir -p public
>        # HTML-ised source tree
>        - make gtags
>        - htags -anT --tree-view=filetree -m qemu_init
>            -t "Welcome to the QEMU sourcecode"
>        - mv HTML public/src
>        # Project documentation
>        - make -C build install DESTDIR=$(pwd)/temp-install
>        - mv temp-install/usr/local/share/doc/qemu/* public/
>      artifacts:
>        paths:
>          - public
> 
> Do we have/need a docs-only install target?

The problem is that after "git clone" the artifacts from the previous 
stage are stale.  We can use the "NINJA=:" hack/workaround on the make 
command line that we already use for tests.

Paolo

Re: Serious doubts about Gitlab CI

[Daniel P. Berrangé]

On Tue, Mar 30, 2021 at 04:23:35PM +0200, Paolo Bonzini wrote:
> On 30/03/21 16:13, Stefan Hajnoczi wrote:
> > On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
> > > On 30/03/2021 13.19, Daniel P. Berrangé wrote:
> > > > On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
> > > > > Hi,
> > > > > I wanted to follow up with a summary of the CI jobs:
> > > > > 
> > > > > 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
> > > > > 2. Builds - ~50 minutes/job x 61 jobs
> > > > > 3. Tests - ~12 minutes/job x 20 jobs
> > > > > 4. Deploy - 52 minutes x 1 job
> > > 
> > > I hope that 52 was just a typo ... ?
> > 
> > No, but I think Dan already found this issue a little while ago. The
> > deploy job uses "make install":
> > 
> >    # Prepare for GitLab pages deployment. Anything copied into the
> >    # "public" directory will be deployed to $USER.gitlab.io/$PROJECT
> >    pages:
> >      image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
> >      stage: test
> >      needs:
> >        - job: build-tools-and-docs-debian
> >      script:
> >        - mkdir -p public
> >        # HTML-ised source tree
> >        - make gtags
> >        - htags -anT --tree-view=filetree -m qemu_init
> >            -t "Welcome to the QEMU sourcecode"
> >        - mv HTML public/src
> >        # Project documentation
> >        - make -C build install DESTDIR=$(pwd)/temp-install
> >        - mv temp-install/usr/local/share/doc/qemu/* public/
> >      artifacts:
> >        paths:
> >          - public
> > 
> > Do we have/need a docs-only install target?
> 
> The problem is that after "git clone" the artifacts from the previous stage
> are stale.  We can use the "NINJA=:" hack/workaround on the make command
> line that we already use for tests.

The problem we have is that the publishing job needs to be called "pages"
in the gitlab-ci.yml file.  We didn't want to rebuild everything, so we
have the dependancy on the build-tools-and-docs-debian job to do the
build phase, and the "pages" job just does "make install" + list the
artifacts.

We can avoid the accidental rebuild if we just change what we do in the
"pages" job.  Move the "make install" command into the
"build-tools-and-docs-debian" job.

The "pages" job then reduces to only

 - mv temp-install/usr/local/share/doc/qemu/* public/

and thus is guaranteed to not trigger a rebuild.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]

On Tue, Mar 30, 2021 at 12:19:38PM +0100, Daniel P. Berrangé wrote:
> On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:
> > Hi,
> > I wanted to follow up with a summary of the CI jobs:
> > 
> > 1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
> > 2. Builds - ~50 minutes/job x 61 jobs
> > 3. Tests - ~12 minutes/job x 20 jobs
> > 4. Deploy - 52 minutes x 1 job
> > 
> > The Builds phase consumes the most CI minutes. If we can optimize this
> > phase then we'll achieve the biggest impact.
> > 
> > In the short term builds could be disabled. However, in the long term I
> > think full build coverage is desirable to prevent merging code that
> > breaks certain host OSes/architectures (e.g. stable Linux distros,
> > macOS, etc).
> 
> The notion of "full build coverage" doesn't really exist in reality.
> The number of platforms that QEMU is targetting, combined with the
> number of features that can be turned on/off in QEMU configure
> means that the matrix for "full build coverage" is too huge to ever
> contemplate.

Good point. We will never cover the full build matrix. I do think that
it's important to cover real-world builds, especially ones that tend to
expose issues (e.g. macOS, Windows, stable Linux distros, etc).

> I think a challenges we have with our incremental approach is that
> we're not really taking into account relative importance of the
> different build scenarios, and often don't look at the big picture
> of what the new job adds in terms of quality, compared to existing
> jobs.
> 
> eg Consider we have
> 
>   build-system-alpine:
>   build-system-ubuntu:
>   build-system-debian:
>   build-system-fedora:
>   build-system-centos:
>   build-system-opensuse:
> 
>   build-trace-multi-user:
>   build-trace-ftrace-system:
>   build-trace-ust-system:
> 
> I'd question whether we really need any of those 'build-trace'
> jobs. Instead, we could have build-system-ubuntu pass
> --enable-trace-backends=log,simple,syslog, build-system-debian
> pass --enable-trace-backends=ust and build-system-fedora
> pass --enable-trace-backends=ftrace, etc. 

Yes, I agree. The trace builds could be collapsed into various other
builds.

> > Traditionally ccache (https://ccache.dev/) was used to detect
> > recompilation of the same compiler input files. This is trickier to do
> > in GitLab CI since it would be necessary to share and update a cache,
> > potentially between untrusted users. Unfortunately this shifts the
> > bottleneck from CPU to network in a CI-as-a-Service environment since
> > the cached build output needs to be accessed by the linker on the CI
> > runner but is stored remotely.
> 
> Our docker containers install ccache already and I could have sworn
> that we use that in gitlab, but now I'm not so sure. We're only
> saving the "build/" directory as an artifact between jobs, and I'm
> not sure that directory holds the ccache cache.

It seems we're not benefitting much from ccache at the moment since the
build takes 50 minutes. Maybe this is a good area to investigate further
and find out what can be improved.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Serious doubts about Gitlab CI

[Markus Armbruster]

Andrew Jones <drjones@redhat.com> writes:

> On Fri, Mar 19, 2021 at 09:33:43AM +0000, Stefan Hajnoczi wrote:
>> On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
>> > On 18/03/21 20:46, Stefan Hajnoczi wrote:
>> > > The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
>> > > GitLab Merge Requests so that anyone can submit a merge request and get
>> > > CI coverage.
>> > 
>> > Each merge request consumes about 2500.  That won't last long.
>> 
>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>> on slow machines or if we'll hit the same issue with dedicated runners.
>> It seems like CI optimization will be necessary...
>>
>
> We need to reduce the amount of CI we do, not only because we can't afford
> it, but because it's wasteful. I hate to think of all the kWhs spent
> testing the exact same code in the exact same way, since everyone runs
> everything with a simple 'git push'.

I normally refrain from posting +1s, but I feel this message really
needs "plussing": right you are!  This kind of wastefulness has bothered
me a lot.

[...]

Re: Serious doubts about Gitlab CI

[Thomas Huth]

On 19/03/2021 13.07, Markus Armbruster wrote:
> Andrew Jones <drjones@redhat.com> writes:
> 
>> On Fri, Mar 19, 2021 at 09:33:43AM +0000, Stefan Hajnoczi wrote:
>>> On Thu, Mar 18, 2021 at 09:30:41PM +0100, Paolo Bonzini wrote:
>>>> On 18/03/21 20:46, Stefan Hajnoczi wrote:
>>>>> The QEMU Project has 50,000 minutes of GitLab CI quota. Let's enable
>>>>> GitLab Merge Requests so that anyone can submit a merge request and get
>>>>> CI coverage.
>>>>
>>>> Each merge request consumes about 2500.  That won't last long.
>>>
>>> Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes are
>>> on slow machines or if we'll hit the same issue with dedicated runners.
>>> It seems like CI optimization will be necessary...
>>>
>>
>> We need to reduce the amount of CI we do, not only because we can't afford
>> it, but because it's wasteful. I hate to think of all the kWhs spent
>> testing the exact same code in the exact same way, since everyone runs
>> everything with a simple 'git push'.
> 
> I normally refrain from posting +1s, but I feel this message really
> needs "plussing": right you are!  This kind of wastefulness has bothered
> me a lot.

Patches for a more intelligent CI setup are certainly welcome!

  Thomas