Re: Proposal for a regular upstream performance testing

["Lukáš Doktor" <ldoktor@redhat.com>]

Dne 30. 11. 20 v 14:23 Stefan Hajnoczi napsal(a):
> On Thu, Nov 26, 2020 at 09:43:38AM +0000, Daniel P. Berrangé wrote:
>> On Thu, Nov 26, 2020 at 09:10:14AM +0100, Lukáš Doktor wrote:
>>> Ideally the community should have a way to also issue their custom builds
>>> in order to verify their patches so they can debug and address issues
>>> better than just commit to qemu-master.
>>
>> Allowing community builds certainly adds an extra dimension of complexity
>> to the problem, as you need some kind of permissions control, as you can't
>> allow any arbitrary user on the web to trigger jobs with arbitrary code,
>> as that is a significant security risk to your infra.
> 
> syzkaller and other upstream CI/fuzzing systems do this, so it may be
> hard but not impossible.
> 

Sure, not impossible, but could not be offered by me at this point. I can't promise anything but maybe in the future this can change, or in solution 2 someone else might resolve the perm issues and I can only assist with the setup (if needed).

>> I think I'd just suggest providing a mechanism for the user to easily spin
>> up performance test jobs on their own hardware. This could be as simple
>> as providing a docker container recipe that users can deploy on some
>> arbitrary machine of their choosing that contains the test rig. All they
>> should need do is provide a git ref, and then launching the container and
>> running jobs should be a single command. They can simply run the tests
>> twice, with and without the patch series in question.
> 
> As soon as developers need to recreate an environment it becomes
> time-consuming and there is a risk that the issue won't be reproduced.
> That doesn't mean the system is useless - big regressions will still be
> tackled - but I think it's too much friction and we should aim to run
> community builds.
> 

I do understand but unfortunately at this point I can not serve.

>>> The problem with those is that we can not simply use travis/gitlab/...
>>> machines for running those tests, because we are measuring in-guest
>>> actual performance.
>>
>> As mentioned above - distinguish between the CI framework, and the
>> actual test runner.
> 
> Does the CI framework or the test runner handle detecting regressions
> and providing historical data? I ask because I'm not sure if GitLab CI
> provides any of this functionality or whether we'd need to write a
> custom CI tool to track and report regressions.
> 

Currently I am using Jenkins which allows to publish result (number of failures and total checks) and store artifacts. I am storing the pbench json results with metadata (few MBs) and html report (also few MBs). Each html report contains a timeline of usually 14 previous builds using them as a reference.

Provided GitLab can do that similarly we should be able to see the number of tests run/failed somewhere and then browse the builds html reports. Last but not least we can fetch the pbench json results and issue another comparison cherry-picking individual results (internally I have a pipeline to do that for me, I could add a helper to do that via cmdline/container for others as well).

Regards,
Lukáš

> Stefan
>