Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes

From: Thomas Huth <thuth@redhat.com>
To: "Daniel P. Berrangé" <berrange@redhat.com>,
	"Richard Henderson" <richard.henderson@linaro.org>
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	qemu-devel@nongnu.org,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Willian Rampazzo" <willianr@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>
Subject: Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes
Date: Wed, 17 Nov 2021 08:03:16 +0100	[thread overview]
Message-ID: <fc453873-6d33-7289-8227-4dd39ed92beb@redhat.com> (raw)
In-Reply-To: <YZP2esl/18WDQHlQ@redhat.com>

On 16/11/2021 19.20, Daniel P. Berrangé wrote:
> On Tue, Nov 16, 2021 at 06:36:50PM +0100, Richard Henderson wrote:
>> On 11/16/21 6:22 PM, Thomas Huth wrote:
>>> On 16/11/2021 18.09, Philippe Mathieu-Daudé wrote:
>>>> On 11/16/21 17:49, Daniel P. Berrangé wrote:
>>>>> On Tue, Nov 16, 2021 at 05:33:09PM +0100, Thomas Huth wrote:
>>>>>> The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
>>>>>> be scheduled, so while the build test itself finishes within 60 minutes,
>>>>>> the total run time of the jobs can be longer due to this waiting time.
>>>>>> Thus let's increase the timeout on the gitlab side a little bit, so
>>>>>> that these jobs are not marked as failing just because of the delay.
>>> ...>>> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
>>>>>> index e7b25e7427..22d42585e4 100644
>>>>>> --- a/.gitlab-ci.d/cirrus.yml
>>>>>> +++ b/.gitlab-ci.d/cirrus.yml
>>>>>> @@ -14,6 +14,7 @@
>>>>>>      stage: build
>>>>>>      image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>>>>>>      needs: []
>>>>>> +  timeout: 80m
>>>>>>      allow_failure: true
>>>>>>      script:
>>>>>>        - source .gitlab-ci.d/cirrus/$NAME.vars
>>>>>
>>>>> Whether 80 or 100 minute, consider it
>>>>>
>>>>> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> This pipeline took 1h51m09s:
>>>> https://gitlab.com/qemu-project/qemu/-/pipelines/409666733/builds
>>>> But Richard restarted unstable jobs, which probably added time
>>>> to the total.
>>>>
>>>> IIRC from a maintainer perspective 1h15 is the upper limit.
>>>> 80m fits, 100m is over.
>>>
>>> I think I agree ... I normally don't want to wait more than a little bit
>>> more than one hour, so 100 minutes feels too long already. We already
>>> have some 70m timeouts in other jobs, and one 80 minute timeout in
>>> .gitlab-ci.d/crossbuild-template.yml, so I'd say 80 minutes are really
>>> the upper boundary that we should use.
>>
>> We are also talking apples and oranges:
>> Gitlab timeouts are on the amount of time the job runs.
>> Cirrus timeouts appear to be on the amount of time the job is queued.
>>
>> If cirrus would just not start accounting until the thing runs we'd be fine.
> 
> Unfortunately it isn't that easy. Our cirrus CI jobs are launched using
> the cirrus-run tool, from a gitlab job. The timeouts we're usually
> hitting are from the gitlab job which is sitting around waiting for
> the cirrus job it launched to finish, so it can report back stats.
> 
> Cirrus CI does itself have a job timeout, but I'm not aware of us
> hitting that typically, unless i'm misinterpreting something.

Right, the problem is the timeout on the gitlab-CI side, not the timeout on 
the Cirrus-CI side. I've never seen us hitting the timeout on the Cirrus 
side either.

  Thomas