Cirrus-CI all red

Cirrus-CI all red

[Philippe Mathieu-Daudé]

FYI, as of today, the latest merge history is red (last 10 days):
https://cirrus-ci.com/github/qemu/qemu

If we want to keep using this, we should somehow plug it to
GitLab-CI (i.e. Travis-CI is run as an external job there) so
the project maintainer can notice job failures.

Alternatively the windows job could be passed to GitLab:
https://docs.gitlab.com/ee/ci/runners/runner_cloud/windows_runner_cloud.html

Regards,

Phil.

Re: Cirrus-CI all red

[Thomas Huth]

On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
> FYI, as of today, the latest merge history is red (last 10 days):
> https://cirrus-ci.com/github/qemu/qemu
> 
> If we want to keep using this, we should somehow plug it to
> GitLab-CI (i.e. Travis-CI is run as an external job there) so
> the project maintainer can notice job failures.

Well, considering that all the cirrus-run based jobs are currently failing 
due to the non-working API token, that does not seem to work very well either.

> Alternatively the windows job could be passed to GitLab:
> https://docs.gitlab.com/ee/ci/runners/runner_cloud/windows_runner_cloud.html

See:

  https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg02474.html

... the problem is again that the shared runners are only single-threaded, 
so it's incredibly slow, especially if you have to re-install MSYS2 each 
time. I once tried to improve the patch by caching the MSYS2 installation, 
but it did not work that well either... (but if somebody wants to continue 
my work, I can rebase and send it out again, just let me know).

  Thomas

Re: Cirrus-CI all red

[Daniel P. Berrangé]

On Tue, Nov 09, 2021 at 10:45:18AM +0100, Thomas Huth wrote:
> On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
> > FYI, as of today, the latest merge history is red (last 10 days):
> > https://cirrus-ci.com/github/qemu/qemu
> > 
> > If we want to keep using this, we should somehow plug it to
> > GitLab-CI (i.e. Travis-CI is run as an external job there) so
> > the project maintainer can notice job failures.
> 
> Well, considering that all the cirrus-run based jobs are currently failing
> due to the non-working API token, that does not seem to work very well
> either.

Who owns the API token ? For other projects, this was addressed a while
ago by refreshing the token. I would have tried this myself for QEMU
but I don't have privileges on the QEMU projects in github/gitlab.

> > Alternatively the windows job could be passed to GitLab:
> > https://docs.gitlab.com/ee/ci/runners/runner_cloud/windows_runner_cloud.html
> 
> See:
> 
>  https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg02474.html
> 
> ... the problem is again that the shared runners are only single-threaded,
> so it's incredibly slow, especially if you have to re-install MSYS2 each
> time. I once tried to improve the patch by caching the MSYS2 installation,
> but it did not work that well either... (but if somebody wants to continue
> my work, I can rebase and send it out again, just let me know).

Potentially this is something where we can make sure of the Azure credits
QEMU is getting, though it would require someone to figure out shared
runner integration and maintain it...

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Cirrus-CI all red

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Nov 09, 2021 at 10:45:18AM +0100, Thomas Huth wrote:
>> On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
>> > FYI, as of today, the latest merge history is red (last 10 days):
>> > https://cirrus-ci.com/github/qemu/qemu
>> > 
>> > If we want to keep using this, we should somehow plug it to
>> > GitLab-CI (i.e. Travis-CI is run as an external job there) so
>> > the project maintainer can notice job failures.
>> 
>> Well, considering that all the cirrus-run based jobs are currently failing
>> due to the non-working API token, that does not seem to work very well
>> either.
>
> Who owns the API token ? For other projects, this was addressed a while
> ago by refreshing the token. I would have tried this myself for QEMU
> but I don't have privileges on the QEMU projects in github/gitlab.

OK I've updated the token (after I figured out the path to it):

  - top right, Settings
  - scroll to bottom "Your GitHub Organizations"
  - click gear icon
  - scroll to API settings, click Generate New Token

It seems to be triggering the builds now although GitLab still reports
failures for some other reason now.

>
>> > Alternatively the windows job could be passed to GitLab:
>> > https://docs.gitlab.com/ee/ci/runners/runner_cloud/windows_runner_cloud.html
>> 
>> See:
>> 
>>  https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg02474.html
>> 
>> ... the problem is again that the shared runners are only single-threaded,
>> so it's incredibly slow, especially if you have to re-install MSYS2 each
>> time. I once tried to improve the patch by caching the MSYS2 installation,
>> but it did not work that well either... (but if somebody wants to continue
>> my work, I can rebase and send it out again, just let me know).
>
> Potentially this is something where we can make sure of the Azure credits
> QEMU is getting, though it would require someone to figure out shared
> runner integration and maintain it...
>
> Regards,
> Daniel


-- 
Alex Bennée

Re: Cirrus-CI all red

[Daniel P. Berrangé]

On Tue, Nov 09, 2021 at 11:27:42AM +0000, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Tue, Nov 09, 2021 at 10:45:18AM +0100, Thomas Huth wrote:
> >> On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
> >> > FYI, as of today, the latest merge history is red (last 10 days):
> >> > https://cirrus-ci.com/github/qemu/qemu
> >> > 
> >> > If we want to keep using this, we should somehow plug it to
> >> > GitLab-CI (i.e. Travis-CI is run as an external job there) so
> >> > the project maintainer can notice job failures.
> >> 
> >> Well, considering that all the cirrus-run based jobs are currently failing
> >> due to the non-working API token, that does not seem to work very well
> >> either.
> >
> > Who owns the API token ? For other projects, this was addressed a while
> > ago by refreshing the token. I would have tried this myself for QEMU
> > but I don't have privileges on the QEMU projects in github/gitlab.
> 
> OK I've updated the token (after I figured out the path to it):
> 
>   - top right, Settings
>   - scroll to bottom "Your GitHub Organizations"
>   - click gear icon
>   - scroll to API settings, click Generate New Token
> 
> It seems to be triggering the builds now although GitLab still reports
> failures for some other reason now.

The cirrus-run image we're using is lockde to version 0.3.0. I'm
testing an update to version 0.5.0 which has various reliability
fixes, essentially around making it retry on transient errors.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Cirrus-CI all red

[Thomas Huth]

On 09/11/2021 12.32, Daniel P. Berrangé wrote:
> On Tue, Nov 09, 2021 at 11:27:42AM +0000, Alex Bennée wrote:
>>
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>>> On Tue, Nov 09, 2021 at 10:45:18AM +0100, Thomas Huth wrote:
>>>> On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
>>>>> FYI, as of today, the latest merge history is red (last 10 days):
>>>>> https://cirrus-ci.com/github/qemu/qemu
>>>>>
>>>>> If we want to keep using this, we should somehow plug it to
>>>>> GitLab-CI (i.e. Travis-CI is run as an external job there) so
>>>>> the project maintainer can notice job failures.
>>>>
>>>> Well, considering that all the cirrus-run based jobs are currently failing
>>>> due to the non-working API token, that does not seem to work very well
>>>> either.
>>>
>>> Who owns the API token ? For other projects, this was addressed a while
>>> ago by refreshing the token. I would have tried this myself for QEMU
>>> but I don't have privileges on the QEMU projects in github/gitlab.
>>
>> OK I've updated the token (after I figured out the path to it):
>>
>>    - top right, Settings
>>    - scroll to bottom "Your GitHub Organizations"
>>    - click gear icon
>>    - scroll to API settings, click Generate New Token
>>
>> It seems to be triggering the builds now although GitLab still reports
>> failures for some other reason now.
> 
> The cirrus-run image we're using is lockde to version 0.3.0. I'm
> testing an update to version 0.5.0 which has various reliability
> fixes, essentially around making it retry on transient errors.

We should maybe also simply bump the timeout for the cirrus jobs ... 
sometimes they get delayed and just need a little bit more than 60 minutes 
to finish ... so would it be OK to us 70 or 80 minutes here?

  Thomas

Re: Cirrus-CI all red

[Philippe Mathieu-Daudé]

On 11/16/21 10:15, Thomas Huth wrote:
> On 09/11/2021 12.32, Daniel P. Berrangé wrote:
>> On Tue, Nov 09, 2021 at 11:27:42AM +0000, Alex Bennée wrote:
>>> Daniel P. Berrangé <berrange@redhat.com> writes:
[...]
>>> It seems to be triggering the builds now although GitLab still reports
>>> failures for some other reason now.
>>
>> The cirrus-run image we're using is lockde to version 0.3.0. I'm
>> testing an update to version 0.5.0 which has various reliability
>> fixes, essentially around making it retry on transient errors.
> 
> We should maybe also simply bump the timeout for the cirrus jobs ...
> sometimes they get delayed and just need a little bit more than 60
> minutes to finish ... so would it be OK to us 70 or 80 minutes here?

Sounds a good idea, this would be more useful that what we have now.

Re: Cirrus-CI all red

[Daniel P. Berrangé]

On Tue, Nov 09, 2021 at 11:32:23AM +0000, Daniel P. Berrangé wrote:
> On Tue, Nov 09, 2021 at 11:27:42AM +0000, Alex Bennée wrote:
> > 
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > 
> > > On Tue, Nov 09, 2021 at 10:45:18AM +0100, Thomas Huth wrote:
> > >> On 09/11/2021 10.39, Philippe Mathieu-Daudé wrote:
> > >> > FYI, as of today, the latest merge history is red (last 10 days):
> > >> > https://cirrus-ci.com/github/qemu/qemu
> > >> > 
> > >> > If we want to keep using this, we should somehow plug it to
> > >> > GitLab-CI (i.e. Travis-CI is run as an external job there) so
> > >> > the project maintainer can notice job failures.
> > >> 
> > >> Well, considering that all the cirrus-run based jobs are currently failing
> > >> due to the non-working API token, that does not seem to work very well
> > >> either.
> > >
> > > Who owns the API token ? For other projects, this was addressed a while
> > > ago by refreshing the token. I would have tried this myself for QEMU
> > > but I don't have privileges on the QEMU projects in github/gitlab.
> > 
> > OK I've updated the token (after I figured out the path to it):
> > 
> >   - top right, Settings
> >   - scroll to bottom "Your GitHub Organizations"
> >   - click gear icon
> >   - scroll to API settings, click Generate New Token
> > 
> > It seems to be triggering the builds now although GitLab still reports
> > failures for some other reason now.
> 
> The cirrus-run image we're using is lockde to version 0.3.0. I'm
> testing an update to version 0.5.0 which has various reliability
> fixes, essentially around making it retry on transient errors.

Forgot to respond that I did get this updated to 0.5.0 in the end

  https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/190

and QEMU is automatically using the newly published docker image
generated from this merge request with cirrus 0.5.0.

I've also just sent a change to block cirrus jobs from master and
stable branches, which will cut the number of jobs in 1/2 and
thus reduce chances of jobs being queued for so long that they
reach timeout.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|