Re: "make check-acceptance" takes way too long

[Thomas Huth <thuth@redhat.com>]

On 31/07/2021 00.04, Cleber Rosa wrote:
> On Fri, Jul 30, 2021 at 11:43 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Fri, 30 Jul 2021 at 16:12, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>
>>> "make check-acceptance" takes way way too long. I just did a run
>>> on an arm-and-aarch64-targets-only debug build and it took over
>>> half an hour, and this despite it skipping or cancelling 26 out
>>> of 58 tests!
>>>
>>> I think that ~10 minutes runtime is reasonable. 30 is not;
>>> ideally no individual test would take more than a minute or so.
>>
>> Side note, can check-acceptance run multiple tests in parallel?
> 
> Yes, it can, but it's not currently enabled to do so, but I'm planning
> to.  As a matter of fact, Yesterday I was trying out Avocado's
> parallel capable runner on a GitLab CI pipeline[1] and it went well.

Was this one of the shared gitlab CI runners? ... well, those feature only a 
single CPU, so the run was likely not very different compared to a single run.

> But the environment on GitLab CI is fluid, and I bet there's already
> some level of overcommit of (at least) CPUs there.  The only pipeline
> I ran there with tests running in parallel, resulted in some jobs with
> improvements, and others with regressions in runtime.  Additionally,
> lack of adequate resources can make more tests time out, and thus give
> out false negatives.

It certainly does not make sense to enable parallel tests for the shared 
runners there.

  Thomas