Test Meetup at Developer Summit

[Lars Kurth <lars.kurth.xen@gmail.com>]

== Attendees ==
Lars Kurth
George Dunlap
Doug Goldstein
Andrew Cooper
Paul Durrant

There were a few others, which I may have missed

I tried to transcribe from a recording we had at lunch, but due to background noise I didn't get everything. Please add/correct, if I got something wrong. There was a bit of discussion, but I tried to capture the key points only (e.g. I omitted clarifying questions)

== Discussion: Travis Improvements ==

Doug, wanted to discuss changes to the project's Travis set-up
* Reconfiguring Travis set-up such that it launches docker containers to do very basic smoke testing of individual commits on a number of different build configurations
* Right now this would require running a script running inside a script
* The key question is what matrix to support: e.g. which compiler versions, whether it should cover deb builds, ...
* It is basically just a build test, with a very basic run-test within an emulated qemu environment to check whether we boot. Aka booting a small image that is cross-compiled against various environments that produces "Hello World" on the first available serial-port
* If "Hello World" was produced, then Travis would decide the build was successful

George: over OSSTEST what value would this add? 
Doug: feedback about 7-8 minutes after a push with some confidence that a build was not broken, faster than OSSTEST does now. Obviously this would not replace OSSTEST, but should reduce the number of build-related issues in OSSTEST
Andrew: This should help quite a lot with avoiding spotting build related issues earlier
Doug: Would help avoid unnecessary manual build testing
George: Should help committers also 
Lars: Are there any extra requirements for people to use?
Andrew: Needs an account and set-up on GitHub

In summary: such a set-up would not compete with OSSTEST, but should decrease the number of build related OSSTEST failures, wasting fewer people's time if there are build related failures and increases OSSTEST throughput

Andrew, is not 100% sure how much extra confidence the "Hello World" test part will actually provide. However, Andrew also states that he does boot-testing before committing a major series. 

Random boot-tests on random config's (e.g. KConfigs) may also be valuable. But the challenge is what to test against, in OSSTEST, and how to know which test would succeed for which config. 

Paul: couldn't we build in something into XTF? ... there was a bit of discussion around this, but the audio was very hard to understand ... there was a discussion around skipping tests, but not quite knowing which ones should legitimately fail in a given configuration, or whether a skip was wrongly configured, or a git time-out. We would need to have more information about the "intent" of an OSSTEST to better help with random config tests. Andrew: thinks that for random build config tests, Travis CI may be more suitable than OSSTEST and require fewer plumbing changes, due to the faster turn-around and less complexity and scope for infrastructure related time-outs, etc. That is not to say that we shouldn't look are OSSTEST for functional config tests.

Andy: suggested a "presence test" in XTF for testing whether a specific config exists in a given binary, which could be used as a building block in OSSTEST for functional config tests. 

ASIDE: I would say that Doug and other participants, fill out this section, as I lost a lot of it, and may have misrepresented. 

Summary: In summary, we discussed
- Improvements to the Travis setup in particular for various build configs
- Possible improvements to OSSTEST / XTF for (random) config tests 
- There was no major objection

== Discussion: GitHub vs. GitLab for Travis ==
The second set of topics was to move from GitHub to GitLab for Build-testing due to the code availability issues with GitHub. Doug uses GitLab internally and is also more functional that GitHub. 

GitLab can be used hosted (for free projects) or host it locally on a VM

Doug: would probably require taking the github yaml and rewrite as gitlab yam. There are a number of advantages on OS support and docker integration, which makes set-up a lot easier compared to github.

Andy: if we were going to do this, the project should probably get a support license

Summary:
- Gather community input on GitHub vs. GitLab
- GitLab code is public, and thus should address 

There was also a bit of discussion about the review work-flow functionality in GitLab: just to clarify, we were not discussing any changes in this area. Merely to use GitLab as framework for Travis / Build testing and addressing some of the past issues that were raised around GitHub code availability.

== Supported / Unsupported / Preview / Experimental / ... ==

We should have some minimum tests and requirements for testing and configuration.

Andy: example - Intel MPX support is currently broken, because it was added when there was no available support. Because it is not compile-time disabled, the feature breaks existing users. This is a recurring theme, and we need to do something about it.

Proposal strawman:
- Would need a separate level of supported, it has to be manually configured in 
- At the moment the problem is that a previously working images often break on new HW 
- Standardise the terms: preview / experimental

Might have several rough buckets
- incomplete or no HW to test on: switch feature off via KConfig
- experimental : should be off by default in xml
- tech-preview : could be switched on by default
- to be fully supported, we need a minimum amount of tests, but we have an issue with grandfathering features that are supported and do not yet have them

Lars to re-start the lifecycle proposal again, taking new information and functionality such as KConfig into account and trying to get committer agreement. 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel