Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Thomas Huth <thuth@redhat.com>]

On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote:
> On 11/27/20 6:47 PM, Thomas Huth wrote:
>> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
>>> We lately realized that the Avocado framework was not designed
>>> to be regularly run on CI environments. Therefore, as of 5.2
>>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
>>> current users, it is possible to keep the current behavior by
>>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
>>> (see [*]).
>>> From now on, using these jobs (or adding new tests to them)
>>> is strongly discouraged.
>>>
>>> Tests based on Avocado will be ported to new job schemes during
>>> the next releases, with better documentation and templates.
>>
>> Why should we disable the jobs by default as long as there is no replacement
>> available yet?
> 
> Why keep it enabled if it is failing randomly

We can still disable single jobs if they are not stable, but that's no
reason to disable all of them by default, is it?

> if images hardcoded
> in tests are being removed from public servers, etc...?

That's independent from Avocado, you'll always have that problem if you want
to test with external images, unless you mirror them into a repository on
the same server (ie. gitlab), which, however, might not always be possible...

> They are not disabled, they are still runnable manually or setting
> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...

And who do you think is going to set that variable? Hardly anybody, I guess.
So you could also simply remove the stuff from the yml file completely instead.

> We realized by default Avocado runs all tests on the CI jobs,
> triggering failures and complains. Developer stopped to contribute/
> review integration tests because of that.

Did anybody really stop contributing "acceptance" test since they were
afraid of the gitlab-CI running them? That's new to me, do you have a pointer?

> We want developers be
> able to contribute tests to the repository fearlessly.

You can always mark your test with @skipIf(os.getenv('GITLAB_CI')) if you
don't want to see it running in the gitlab-CI, so that's not a reason to be
afraid.

> If we don't change anything, we'll keep having CI failures due
> to Avocado design issues (artifacts removed from remote servers,
> etc...).

I fail to see the relation between Avocado and vanishing artifacts from 3rd
party servers... what do you plan to do instead if something gets (re-)moved
from a server that is not under our control?

> I haven't seen any attempt on this list to improve the current
> fragile situation, but better suggestions are certainly welcome.

At least I am hoping that the "check-acceptance" tests will break a little
bit less often once Peter uses the gitlab-CI for his gating tests, too. That
will at least prevent that one of the tests gets completely broken by a new
merged pull request. Of course there's still the risk that tests only fail
occasionally due to new bugs, but that can also happen for all other test
suites (unit, qtest, iotests, ...), too.

 Thomas