Test Meetup at Developer Summit

Test Meetup at Developer Summit

[Lars Kurth]

== Attendees ==
Lars Kurth
George Dunlap
Doug Goldstein
Andrew Cooper
Paul Durrant

There were a few others, which I may have missed

I tried to transcribe from a recording we had at lunch, but due to background noise I didn't get everything. Please add/correct, if I got something wrong. There was a bit of discussion, but I tried to capture the key points only (e.g. I omitted clarifying questions)

== Discussion: Travis Improvements ==

Doug, wanted to discuss changes to the project's Travis set-up
* Reconfiguring Travis set-up such that it launches docker containers to do very basic smoke testing of individual commits on a number of different build configurations
* Right now this would require running a script running inside a script
* The key question is what matrix to support: e.g. which compiler versions, whether it should cover deb builds, ...
* It is basically just a build test, with a very basic run-test within an emulated qemu environment to check whether we boot. Aka booting a small image that is cross-compiled against various environments that produces "Hello World" on the first available serial-port
* If "Hello World" was produced, then Travis would decide the build was successful

George: over OSSTEST what value would this add? 
Doug: feedback about 7-8 minutes after a push with some confidence that a build was not broken, faster than OSSTEST does now. Obviously this would not replace OSSTEST, but should reduce the number of build-related issues in OSSTEST
Andrew: This should help quite a lot with avoiding spotting build related issues earlier
Doug: Would help avoid unnecessary manual build testing
George: Should help committers also 
Lars: Are there any extra requirements for people to use?
Andrew: Needs an account and set-up on GitHub

In summary: such a set-up would not compete with OSSTEST, but should decrease the number of build related OSSTEST failures, wasting fewer people's time if there are build related failures and increases OSSTEST throughput

Andrew, is not 100% sure how much extra confidence the "Hello World" test part will actually provide. However, Andrew also states that he does boot-testing before committing a major series. 

Random boot-tests on random config's (e.g. KConfigs) may also be valuable. But the challenge is what to test against, in OSSTEST, and how to know which test would succeed for which config. 

Paul: couldn't we build in something into XTF? ... there was a bit of discussion around this, but the audio was very hard to understand ... there was a discussion around skipping tests, but not quite knowing which ones should legitimately fail in a given configuration, or whether a skip was wrongly configured, or a git time-out. We would need to have more information about the "intent" of an OSSTEST to better help with random config tests. Andrew: thinks that for random build config tests, Travis CI may be more suitable than OSSTEST and require fewer plumbing changes, due to the faster turn-around and less complexity and scope for infrastructure related time-outs, etc. That is not to say that we shouldn't look are OSSTEST for functional config tests.

Andy: suggested a "presence test" in XTF for testing whether a specific config exists in a given binary, which could be used as a building block in OSSTEST for functional config tests. 

ASIDE: I would say that Doug and other participants, fill out this section, as I lost a lot of it, and may have misrepresented. 

Summary: In summary, we discussed
- Improvements to the Travis setup in particular for various build configs
- Possible improvements to OSSTEST / XTF for (random) config tests 
- There was no major objection

== Discussion: GitHub vs. GitLab for Travis ==
The second set of topics was to move from GitHub to GitLab for Build-testing due to the code availability issues with GitHub. Doug uses GitLab internally and is also more functional that GitHub. 

GitLab can be used hosted (for free projects) or host it locally on a VM

Doug: would probably require taking the github yaml and rewrite as gitlab yam. There are a number of advantages on OS support and docker integration, which makes set-up a lot easier compared to github.

Andy: if we were going to do this, the project should probably get a support license

Summary:
- Gather community input on GitHub vs. GitLab
- GitLab code is public, and thus should address 

There was also a bit of discussion about the review work-flow functionality in GitLab: just to clarify, we were not discussing any changes in this area. Merely to use GitLab as framework for Travis / Build testing and addressing some of the past issues that were raised around GitHub code availability.

== Supported / Unsupported / Preview / Experimental / ... ==

We should have some minimum tests and requirements for testing and configuration.

Andy: example - Intel MPX support is currently broken, because it was added when there was no available support. Because it is not compile-time disabled, the feature breaks existing users. This is a recurring theme, and we need to do something about it.

Proposal strawman:
- Would need a separate level of supported, it has to be manually configured in 
- At the moment the problem is that a previously working images often break on new HW 
- Standardise the terms: preview / experimental

Might have several rough buckets
- incomplete or no HW to test on: switch feature off via KConfig
- experimental : should be off by default in xml
- tech-preview : could be switched on by default
- to be fully supported, we need a minimum amount of tests, but we have an issue with grandfathering features that are supported and do not yet have them

Lars to re-start the lifecycle proposal again, taking new information and functionality such as KConfig into account and trying to get committer agreement. 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Test Meetup at Developer Summit

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 4765 bytes --]

On 9/5/16 6:19 AM, Lars Kurth wrote:
> == Attendees ==
> Lars Kurth
> George Dunlap
> Doug Goldstein
> Andrew Cooper
> Paul Durrant
> 
> There were a few others, which I may have missed

I know Jonathan Creekmore was involved as well.

> 
> I tried to transcribe from a recording we had at lunch, but due to background noise I didn't get everything. Please add/correct, if I got something wrong. There was a bit of discussion, but I tried to capture the key points only (e.g. I omitted clarifying questions)

Thanks Lars! Sorry for the delay in responses I went immediately from
the Summit to vacation for the past week.

> 
> == Discussion: Travis Improvements ==
> 
> Doug, wanted to discuss changes to the project's Travis set-up
> * Reconfiguring Travis set-up such that it launches docker containers to do very basic smoke testing of individual commits on a number of different build configurations
> * Right now this would require running a script running inside a script
> * The key question is what matrix to support: e.g. which compiler versions, whether it should cover deb builds, ...
> * It is basically just a build test, with a very basic run-test within an emulated qemu environment to check whether we boot. Aka booting a small image that is cross-compiled against various environments that produces "Hello World" on the first available serial-port
> * If "Hello World" was produced, then Travis would decide the build was successful
> 
> George: over OSSTEST what value would this add? 
> Doug: feedback about 7-8 minutes after a push with some confidence that a build was not broken, faster than OSSTEST does now. Obviously this would not replace OSSTEST, but should reduce the number of build-related issues in OSSTEST
> Andrew: This should help quite a lot with avoiding spotting build related issues earlier
> Doug: Would help avoid unnecessary manual build testing
> George: Should help committers also 
> Lars: Are there any extra requirements for people to use?
> Andrew: Needs an account and set-up on GitHub

If we decide to stick with GitHub and Travis then I plan on documenting
this so its easier to people to do this on their own.

> 
> == Discussion: GitHub vs. GitLab for Travis ==
> The second set of topics was to move from GitHub to GitLab for Build-testing due to the code availability issues with GitHub. Doug uses GitLab internally and is also more functional that GitHub. 

The concern when I initiated the GitHub + Travis bits were that GitHub
and Travis were not free as in speech but only free as in beer. GitLab
is free as in speech [1] and if you use their hosted service [2] its
free as in beer as well.

They've made outstanding strides this year with their integrated CI
service called GitLab CI. At my company we now use this for all of our
projects since it allows you to define multiple steps and dependencies
of those steps. Steps can be flagged to have to run within specific
Docker containers which allows us to have a matrix of build environments
to build test with. It also allows you to capture artifacts from the
build which is what we do to actually run real tests against a built
object (ala OSSTest). Travis + GitHub at this point allow you to specify
that you need sudo and then from there you can script up Docker commands
by hand which we can certainly do but there is no parallelism and its a
lot more we need to maintain in our scripts.

> 
> GitLab can be used hosted (for free projects) or host it locally on a VM
> 
> Doug: would probably require taking the github yaml and rewrite as gitlab yam. There are a number of advantages on OS support and docker integration, which makes set-up a lot easier compared to github.
> 
> Andy: if we were going to do this, the project should probably get a support license
> 
> Summary:
> - Gather community input on GitHub vs. GitLab
> - GitLab code is public, and thus should address 

I think this was suppose to end with "address concerns about software
licensing of the software we use"

> 
> There was also a bit of discussion about the review work-flow functionality in GitLab: just to clarify, we were not discussing any changes in this area. Merely to use GitLab as framework for Travis / Build testing and addressing some of the past issues that were raised around GitHub code availability.

I'm not proposing using any of the merge request or issue tracker bits.
Just the CI portion for doing builds and allowing people to push a
branch and have it built on a variety of compilers to lessen the chance
that "it worked on my machine" situations strike.


[1] https://gitlab.com/gitlab-org/gitlab-ce
[2] https://about.gitlab.com/gitlab-com/

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel