[RFC PATCH 0/6] Using GitLab CI for build testing

[RFC PATCH 0/6] Using GitLab CI for build testing

[Doug Goldstein]

Really early work on switching over to using GitLab CI over
Travis CI. GitLab is a competitor to GitHub with some advantages
such as an integrated CI system with a lot more flexibility
and control. It additionally is fully open sourced unlike GitHub
and Travis CI. We can even run an instance if that is preferred
over using the hosted instance.

This change uses GitLab CI's ability to use Docker based runners
for running tests. With GitHub we also use a Docker based runner
but we are limited to one Docker container that is then morphed
a number of different ways. With this approach we can specify
different Docker containers for every run (or use the same). By
using different Docker containers we can build environments that
match systems where Xen can and should build. Using this
approach we should be able to cutdown on the number of surpise
build failures encountered by users.

An example run can be seen here:
https://gitlab.com/cardoe/xen/pipelines/18789907

If there is interest in this I will move it over to the "xen-project"
name space in the next version.

Doug Goldstein (6):
  ci: add Dockerfile for CentOS 7.2
  ci: add Dockerfile for Ubuntu 14.04
  ci: add Dockerfile for Ubuntu 16.04
  ci: add Dockerfile for Debian jessie
  ci: add cfg to use GitLab CI to build
  ci: add a README about the containers

 .gitlab-ci.yml                          | 34 ++++++++++++++++++++++-
 extras/testing/README.md                | 29 ++++++++++++++++++-
 extras/testing/centos/CentOS-7.2.repo   | 35 ++++++++++++++++++++++-
 extras/testing/centos/Dockerfile.7.2    | 41 ++++++++++++++++++++++++++-
 extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++-
 extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++-
 extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++-
 7 files changed, 202 insertions(+)
 create mode 100644 .gitlab-ci.yml
 create mode 100644 extras/testing/README.md
 create mode 100644 extras/testing/centos/CentOS-7.2.repo
 create mode 100644 extras/testing/centos/Dockerfile.7.2
 create mode 100644 extras/testing/debian/Dockerfile.jessie
 create mode 100644 extras/testing/ubuntu/Dockerfile.trusty
 create mode 100644 extras/testing/ubuntu/Dockerfile.xenial

base-commit: 966f154c58bacf07690135d7da3f1d5281d84ab0
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 1/6] ci: add Dockerfile for CentOS 7.2

[Doug Goldstein]

Added a Dockerfile which captures all the necessary dependencies to
build Xen on a CentOS 7.2 system.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 extras/testing/centos/CentOS-7.2.repo | 35 ++++++++++++++++++++++++-
 extras/testing/centos/Dockerfile.7.2  | 41 ++++++++++++++++++++++++++++-
 2 files changed, 76 insertions(+)
 create mode 100644 extras/testing/centos/CentOS-7.2.repo
 create mode 100644 extras/testing/centos/Dockerfile.7.2

diff --git a/extras/testing/centos/CentOS-7.2.repo b/extras/testing/centos/CentOS-7.2.repo
new file mode 100644
index 0000000..4da27fa
--- /dev/null
+++ b/extras/testing/centos/CentOS-7.2.repo
@@ -0,0 +1,35 @@
+# CentOS-Base.repo
+#
+# This is a replacement file that pins things to just use CentOS 7.2
+# from the CentOS Vault.
+#
+
+[base]
+name=CentOS-7.2.1511 - Base
+baseurl=http://vault.centos.org/7.2.1511/os/$basearch/
+gpgcheck=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+
+#released updates 
+[updates]
+name=CentOS-7.2.1511 - Updates
+baseurl=http://vault.centos.org/7.2.1511/updates/$basearch/
+gpgcheck=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+
+#additional packages that may be useful
+[extras]
+name=CentOS-7.2.1511 - Extras
+baseurl=http://vault.centos.org/7.2.1511/extras/$basearch/
+gpgcheck=1
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+
+#additional packages that extend functionality of existing packages
+[centosplus]
+name=CentOS-7.2.1511 - Plus
+baseurl=http://vault.centos.org/7.2.1511/centosplus/$basearch/
+gpgcheck=1
+gpgcheck=1
+enabled=0
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+
diff --git a/extras/testing/centos/Dockerfile.7.2 b/extras/testing/centos/Dockerfile.7.2
new file mode 100644
index 0000000..40780b4
--- /dev/null
+++ b/extras/testing/centos/Dockerfile.7.2
@@ -0,0 +1,41 @@
+FROM centos:7.2.1511
+LABEL maintainer.name="Doug Goldstein" \
+      maintainer.email="cardoe@cardoe.com"
+
+# ensure we only get bits from the vault for
+# the version we want
+COPY CentOS-7.2.repo /etc/yum.repos.d/CentOS-Base.repo
+
+RUN mkdir /build
+WORKDIR /build
+
+# work around https://github.com/moby/moby/issues/10180
+# and install Xen depends
+RUN rpm --rebuilddb && \
+    yum -y install \
+        yum-plugin-ovl \
+        gcc \
+        gcc-c++ \
+        ncurses-devel \
+        zlib-devel \
+        openssl-devel \
+        python-devel \
+        libuuid-devel \
+        pkgconfig \
+        gettext \
+        flex \
+        bison \
+        libaio-devel \
+        glib2-devel \
+        yajl-devel \
+        pixman-devel \
+        glibc-devel \
+        glibc-devel.i686 \
+        make \
+        binutils \
+        git \
+        wget \
+        acpica-tools \
+        python-markdown \
+        patch \
+    && yum clean all
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 2/6] ci: add Dockerfile for Ubuntu 14.04

[Doug Goldstein]

Added a Dockerfile which captures all the necessary dependencies to
build Xen on a Ubuntu 14.04 system.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 extras/testing/ubuntu/Dockerfile.trusty

diff --git a/extras/testing/ubuntu/Dockerfile.trusty b/extras/testing/ubuntu/Dockerfile.trusty
new file mode 100644
index 0000000..c5d761e
--- /dev/null
+++ b/extras/testing/ubuntu/Dockerfile.trusty
@@ -0,0 +1,21 @@
+FROM ubuntu:14.04
+LABEL maintainer.name="Doug Goldstein" \
+      maintainer.email="cardoe@cardoe.com"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+    apt-get --quiet --yes install \
+        build-essential zlib1g-dev libncurses5-dev libssl-dev python2.7-dev \
+        xorg-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev clang \
+        libpixman-1-dev pkg-config flex bison gettext acpica-tools bin86 \
+        bcc libc6-dev-i386 libnl-3-dev ocaml-nox libfindlib-ocaml-dev \
+        markdown transfig pandoc checkpolicy wget git && \
+        apt-get autoremove -y && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 3/6] ci: add Dockerfile for Ubuntu 16.04

[Doug Goldstein]

Added a Dockerfile which captures all the necessary dependencies to
build Xen on a Ubuntu 16.04 system.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 extras/testing/ubuntu/Dockerfile.xenial

diff --git a/extras/testing/ubuntu/Dockerfile.xenial b/extras/testing/ubuntu/Dockerfile.xenial
new file mode 100644
index 0000000..b9ce27f
--- /dev/null
+++ b/extras/testing/ubuntu/Dockerfile.xenial
@@ -0,0 +1,21 @@
+FROM ubuntu:16.04
+LABEL maintainer.name="Doug Goldstein" \
+      maintainer.email="cardoe@cardoe.com"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+    apt-get --quiet --yes install \
+        build-essential zlib1g-dev libncurses5-dev libssl-dev python2.7-dev \
+        xorg-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev clang \
+        libpixman-1-dev pkg-config flex bison gettext acpica-tools bin86 \
+        bcc libc6-dev-i386 libnl-3-dev ocaml-nox libfindlib-ocaml-dev \
+        markdown transfig pandoc checkpolicy wget git && \
+        apt-get autoremove -y && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 4/6] ci: add Dockerfile for Debian jessie

[Doug Goldstein]

Added a Dockerfile which captures all the necessary dependencies to
build Xen on a Debian jessie system.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 extras/testing/debian/Dockerfile.jessie

diff --git a/extras/testing/debian/Dockerfile.jessie b/extras/testing/debian/Dockerfile.jessie
new file mode 100644
index 0000000..be0dccc
--- /dev/null
+++ b/extras/testing/debian/Dockerfile.jessie
@@ -0,0 +1,21 @@
+FROM debian:jessie
+LABEL maintainer.name="Doug Goldstein" \
+      maintainer.email="cardoe@cardoe.com"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+    apt-get --quiet --yes install \
+        build-essential zlib1g-dev libncurses5-dev libssl-dev python2.7-dev \
+        xorg-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev clang \
+        libpixman-1-dev pkg-config flex bison gettext acpica-tools bin86 \
+        bcc libc6-dev-i386 libnl-3-dev ocaml-nox libfindlib-ocaml-dev \
+        markdown transfig pandoc checkpolicy wget git && \
+        apt-get autoremove -y && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 5/6] ci: add cfg to use GitLab CI to build

[Doug Goldstein]

Added a GitLab CI config which has a lot more flexibility to allow us to
test a lot more distro configurations than Travis can and even build
test on FreeBSD.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 .gitlab-ci.yml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
new file mode 100644
index 0000000..ccc2237
--- /dev/null
+++ b/.gitlab-ci.yml
@@ -0,0 +1,34 @@
+stages:
+  - build
+
+centos-7-2:
+  stage: build
+  image: registry.gitlab.com/cardoe/xen/centos:7.2
+  variables:
+    CC: gcc
+  script:
+    - ./scripts/travis-build
+
+debian-jessie:
+  stage: build
+  image: registry.gitlab.com/cardoe/xen/debian:jessie
+  variables:
+    CC: gcc
+  script:
+    - ./scripts/travis-build
+
+ubuntu-trusty:
+  stage: build
+  image: registry.gitlab.com/cardoe/xen/ubuntu:trusty
+  variables:
+    CC: gcc
+  script:
+    - ./scripts/travis-build
+
+ubuntu-xenial:
+  stage: build
+  image: registry.gitlab.com/cardoe/xen/ubuntu:xenial
+  variables:
+    CC: gcc
+  script:
+    - ./scripts/travis-build
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 6/6] ci: add a README about the containers

[Doug Goldstein]

Add a basic README explaining the containers and how people can use them
to locally test with if they see an error in CI and want to reproduce it
locally.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
---
 extras/testing/README.md | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 extras/testing/README.md

diff --git a/extras/testing/README.md b/extras/testing/README.md
new file mode 100644
index 0000000..0908a66
--- /dev/null
+++ b/extras/testing/README.md
@@ -0,0 +1,29 @@
+Docker Containers
+=================
+
+These Docker containers should make it possible to build Xen in
+any of the available environments on any system that supports
+running Docker. They are organized by distro and tagged with
+the version of that distro. They are available from the GitLab
+Container Registry under the Xen project at:
+
+registry.gitlab.com/cardoe/xen/DISTRO:VERSION
+
+The available ones are:
+- centos:7.2
+- debian:jessie
+- ubuntu:trusty
+- ubuntu:xenial
+
+To Build Xen
+------------
+
+From the top level of the source tree it should be possible to
+run the following:
+
+docker run --rm -it -v $(PWD):/build -u $(id -u) -e CC=gcc $(CONTAINER) make
+
+There are other modifications that can be made but this will run
+the `make` command inside the specified container. It will use your
+currently checked out source tree to build with, ensure that file
+permissions remain consistent and clean up after itself.
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 2177 bytes --]

On 3/12/18 10:31 PM, Doug Goldstein wrote:
> Really early work on switching over to using GitLab CI over
> Travis CI. GitLab is a competitor to GitHub with some advantages
> such as an integrated CI system with a lot more flexibility
> and control. It additionally is fully open sourced unlike GitHub
> and Travis CI. We can even run an instance if that is preferred
> over using the hosted instance.
> 
> This change uses GitLab CI's ability to use Docker based runners
> for running tests. With GitHub we also use a Docker based runner
> but we are limited to one Docker container that is then morphed
> a number of different ways. With this approach we can specify
> different Docker containers for every run (or use the same). By
> using different Docker containers we can build environments that
> match systems where Xen can and should build. Using this
> approach we should be able to cutdown on the number of surpise
> build failures encountered by users.
> 
> An example run can be seen here:
> https://gitlab.com/cardoe/xen/pipelines/18789907
> 
> If there is interest in this I will move it over to the "xen-project"
> name space in the next version.

Worth noting another advantage is that builders can be VMs or even
physical hosts as well. So we can have a FreeBSD VM that can be a build
environment.

Further more the above link is to a GitLab pipeline, pipelines are made
of stages which are further composed of jobs. Currently the example uses
one stage called build and all the different distros are different jobs.
But there's a lot of flexibility as to what can be done here. There can
be stages that check code style or other pre-flight checks that people
may be interested. There can be stages that happen after the build stage
as well such as some simple tests (e.g. I use it to run the just built
xen.gz with an initramfs only dom0 that contains a small Alpine Linux VM
that spits out a string to an HTTP endpoint which decides that Xen build
is good enough to allow it to be merged into our testing branch).

Overall there are a lot more possibilities than what I've put together
so far.
-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 1/6] ci: add Dockerfile for CentOS 7.2

[Julien Grall]

Hi Doug,

On 13/03/18 03:31, Doug Goldstein wrote:
> +# work around https://github.com/moby/moby/issues/10180
> +# and install Xen depends
> +RUN rpm --rebuilddb && \
> +    yum -y install \
> +        yum-plugin-ovl \
> +        gcc \
> +        gcc-c++ \
> +        ncurses-devel \
> +        zlib-devel \
> +        openssl-devel \
> +        python-devel \
> +        libuuid-devel \
> +        pkgconfig \
> +        gettext \
> +        flex \
> +        bison \
> +        libaio-devel \
> +        glib2-devel \
> +        yajl-devel \
> +        pixman-devel \
> +        glibc-devel \
> +        glibc-devel.i686 \

This looks fairly x86. Do you have any plan to add arm support?

In any case, the README should explain what architectures are supported.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[Wei Liu]

On Mon, Mar 12, 2018 at 11:29:47PM -0500, Doug Goldstein wrote:
> On 3/12/18 10:31 PM, Doug Goldstein wrote:
> > Really early work on switching over to using GitLab CI over
> > Travis CI. GitLab is a competitor to GitHub with some advantages
> > such as an integrated CI system with a lot more flexibility
> > and control. It additionally is fully open sourced unlike GitHub
> > and Travis CI. We can even run an instance if that is preferred
> > over using the hosted instance.
> > 
> > This change uses GitLab CI's ability to use Docker based runners
> > for running tests. With GitHub we also use a Docker based runner
> > but we are limited to one Docker container that is then morphed
> > a number of different ways. With this approach we can specify
> > different Docker containers for every run (or use the same). By
> > using different Docker containers we can build environments that
> > match systems where Xen can and should build. Using this
> > approach we should be able to cutdown on the number of surpise
> > build failures encountered by users.
> > 
> > An example run can be seen here:
> > https://gitlab.com/cardoe/xen/pipelines/18789907
> > 
> > If there is interest in this I will move it over to the "xen-project"
> > name space in the next version.
> 
> Worth noting another advantage is that builders can be VMs or even
> physical hosts as well. So we can have a FreeBSD VM that can be a build
> environment.
> 
> Further more the above link is to a GitLab pipeline, pipelines are made
> of stages which are further composed of jobs. Currently the example uses
> one stage called build and all the different distros are different jobs.
> But there's a lot of flexibility as to what can be done here. There can
> be stages that check code style or other pre-flight checks that people
> may be interested. There can be stages that happen after the build stage
> as well such as some simple tests (e.g. I use it to run the just built
> xen.gz with an initramfs only dom0 that contains a small Alpine Linux VM
> that spits out a string to an HTTP endpoint which decides that Xen build
> is good enough to allow it to be merged into our testing branch).
> 
> Overall there are a lot more possibilities than what I've put together
> so far.

Mostly looks fine. And I think having this new dockerised build is good.
Please document what is needed to add ARM support. I suppose we just
need to install the cross-toolchain and write the right command in
dockerfile.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 6/6] ci: add a README about the containers

[George Dunlap]

On 03/13/2018 03:31 AM, Doug Goldstein wrote:
> Add a basic README explaining the containers and how people can use them
> to locally test with if they see an error in CI and want to reproduce it
> locally.
> 
> Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
> ---
>  extras/testing/README.md | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
>  create mode 100644 extras/testing/README.md
> 
> diff --git a/extras/testing/README.md b/extras/testing/README.md
> new file mode 100644
> index 0000000..0908a66
> --- /dev/null
> +++ b/extras/testing/README.md
> @@ -0,0 +1,29 @@
> +Docker Containers
> +=================
> +
> +These Docker containers should make it possible to build Xen in
> +any of the available environments on any system that supports
> +running Docker. They are organized by distro and tagged with
> +the version of that distro. They are available from the GitLab
> +Container Registry under the Xen project at:
> +
> +registry.gitlab.com/cardoe/xen/DISTRO:VERSION

If we were to check something like this into the upstream tree, it would
be better if this were some sort of official Xen account (of which I'd
be happy for you to be the maintainer).  Is that possible?

Other than that, this looks really good.  As you say, having FreeBSD
would be a good thing to add, and with my CentOS hat on, I'd like to add
a hook to build the CentOS Virt SIG packages against master as well (so
that I could get early notification of issues).

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 1/6] ci: add Dockerfile for CentOS 7.2

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 1486 bytes --]

On 3/13/18 9:52 AM, Julien Grall wrote:
> Hi Doug,
> 
> On 13/03/18 03:31, Doug Goldstein wrote:
>> +# work around https://github.com/moby/moby/issues/10180
>> +# and install Xen depends
>> +RUN rpm --rebuilddb && \
>> +    yum -y install \
>> +        yum-plugin-ovl \
>> +        gcc \
>> +        gcc-c++ \
>> +        ncurses-devel \
>> +        zlib-devel \
>> +        openssl-devel \
>> +        python-devel \
>> +        libuuid-devel \
>> +        pkgconfig \
>> +        gettext \
>> +        flex \
>> +        bison \
>> +        libaio-devel \
>> +        glib2-devel \
>> +        yajl-devel \
>> +        pixman-devel \
>> +        glibc-devel \
>> +        glibc-devel.i686 \
> 
> This looks fairly x86. Do you have any plan to add arm support?
> 
> In any case, the README should explain what architectures are supported.
> 
> Cheers,
> 

Yes it definitely x86. I plan on adding ARM support I just haven't had
enough time to gather up what the dependencies need to be. Since its
going to be a cross compile environment I need to take the time to setup
one. Do you have a recommendation? I'm sure there are docs I just
haven't looked at them.

I'm hoping some people approve of this and we can potentially land this
for 4.11 so we can get some build testing for that branch.
-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 6/6] ci: add a README about the containers

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 2005 bytes --]

On 3/13/18 10:38 AM, George Dunlap wrote:
> On 03/13/2018 03:31 AM, Doug Goldstein wrote:
>> Add a basic README explaining the containers and how people can use them
>> to locally test with if they see an error in CI and want to reproduce it
>> locally.
>>
>> Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
>> ---
>>  extras/testing/README.md | 29 +++++++++++++++++++++++++++++
>>  1 file changed, 29 insertions(+)
>>  create mode 100644 extras/testing/README.md
>>
>> diff --git a/extras/testing/README.md b/extras/testing/README.md
>> new file mode 100644
>> index 0000000..0908a66
>> --- /dev/null
>> +++ b/extras/testing/README.md
>> @@ -0,0 +1,29 @@
>> +Docker Containers
>> +=================
>> +
>> +These Docker containers should make it possible to build Xen in
>> +any of the available environments on any system that supports
>> +running Docker. They are organized by distro and tagged with
>> +the version of that distro. They are available from the GitLab
>> +Container Registry under the Xen project at:
>> +
>> +registry.gitlab.com/cardoe/xen/DISTRO:VERSION
> 
> If we were to check something like this into the upstream tree, it would
> be better if this were some sort of official Xen account (of which I'd
> be happy for you to be the maintainer).  Is that possible?

Yep. I mentioned in the cover letter that if people gave me a +1 I'd
move it over to the "xen-project" tenant. I've already registered it to
reserve it for us to move forward with this if we so decide.

> 
> Other than that, this looks really good.  As you say, having FreeBSD
> would be a good thing to add, and with my CentOS hat on, I'd like to add
> a hook to build the CentOS Virt SIG packages against master as well (so
> that I could get early notification of issues).

Totally agree. I ultimately want to have every environment that's
supported / used run through this loop. I'm just going to need some help
enumerating them all out.


-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[George Dunlap]

On Tue, Mar 13, 2018 at 3:31 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
> Really early work on switching over to using GitLab CI over
> Travis CI. GitLab is a competitor to GitHub with some advantages
> such as an integrated CI system with a lot more flexibility
> and control. It additionally is fully open sourced unlike GitHub
> and Travis CI. We can even run an instance if that is preferred
> over using the hosted instance.
>
> This change uses GitLab CI's ability to use Docker based runners
> for running tests. With GitHub we also use a Docker based runner
> but we are limited to one Docker container that is then morphed
> a number of different ways. With this approach we can specify
> different Docker containers for every run (or use the same). By
> using different Docker containers we can build environments that
> match systems where Xen can and should build. Using this
> approach we should be able to cutdown on the number of surpise
> build failures encountered by users.
>
> An example run can be seen here:
> https://gitlab.com/cardoe/xen/pipelines/18789907
>
> If there is interest in this I will move it over to the "xen-project"
> name space in the next version.
>
> Doug Goldstein (6):
>   ci: add Dockerfile for CentOS 7.2
>   ci: add Dockerfile for Ubuntu 14.04
>   ci: add Dockerfile for Ubuntu 16.04
>   ci: add Dockerfile for Debian jessie
>   ci: add cfg to use GitLab CI to build
>   ci: add a README about the containers
>
>  .gitlab-ci.yml                          | 34 ++++++++++++++++++++++-
>  extras/testing/README.md                | 29 ++++++++++++++++++-
>  extras/testing/centos/CentOS-7.2.repo   | 35 ++++++++++++++++++++++-
>  extras/testing/centos/Dockerfile.7.2    | 41 ++++++++++++++++++++++++++-
>  extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++-
>  extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++-
>  extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++-

"extras" is a bit generic.  What about something like "automation/build"?

(You knew this bike shed wasn't going to get in without *some*
discussion of the color!)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[George Dunlap]

On Tue, Mar 13, 2018 at 3:31 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
> Really early work on switching over to using GitLab CI over
> Travis CI. GitLab is a competitor to GitHub with some advantages
> such as an integrated CI system with a lot more flexibility
> and control. It additionally is fully open sourced unlike GitHub
> and Travis CI. We can even run an instance if that is preferred
> over using the hosted instance.
>
> This change uses GitLab CI's ability to use Docker based runners
> for running tests. With GitHub we also use a Docker based runner
> but we are limited to one Docker container that is then morphed
> a number of different ways. With this approach we can specify
> different Docker containers for every run (or use the same). By
> using different Docker containers we can build environments that
> match systems where Xen can and should build. Using this
> approach we should be able to cutdown on the number of surpise
> build failures encountered by users.
>
> An example run can be seen here:
> https://gitlab.com/cardoe/xen/pipelines/18789907
>
> If there is interest in this I will move it over to the "xen-project"
> name space in the next version.
>
> Doug Goldstein (6):
>   ci: add Dockerfile for CentOS 7.2
>   ci: add Dockerfile for Ubuntu 14.04
>   ci: add Dockerfile for Ubuntu 16.04
>   ci: add Dockerfile for Debian jessie
>   ci: add cfg to use GitLab CI to build
>   ci: add a README about the containers
>
>  .gitlab-ci.yml                          | 34 ++++++++++++++++++++++-
>  extras/testing/README.md                | 29 ++++++++++++++++++-
>  extras/testing/centos/CentOS-7.2.repo   | 35 ++++++++++++++++++++++-
>  extras/testing/centos/Dockerfile.7.2    | 41 ++++++++++++++++++++++++++-
>  extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++-
>  extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++-
>  extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++-

"extras" is a bit generic.  What about something like "automation/build"?

(You knew this bike shed wasn't going to get in without *some*
discussion of the color!)

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 1/6] ci: add Dockerfile for CentOS 7.2

[Julien Grall]

Hi Doug,

On 03/13/2018 03:48 PM, Doug Goldstein wrote:
> On 3/13/18 9:52 AM, Julien Grall wrote:
>> Hi Doug,
>>
>> On 13/03/18 03:31, Doug Goldstein wrote:
>>> +# work around https://github.com/moby/moby/issues/10180
>>> +# and install Xen depends
>>> +RUN rpm --rebuilddb && \
>>> +    yum -y install \
>>> +        yum-plugin-ovl \
>>> +        gcc \
>>> +        gcc-c++ \
>>> +        ncurses-devel \
>>> +        zlib-devel \
>>> +        openssl-devel \
>>> +        python-devel \
>>> +        libuuid-devel \
>>> +        pkgconfig \
>>> +        gettext \
>>> +        flex \
>>> +        bison \
>>> +        libaio-devel \
>>> +        glib2-devel \
>>> +        yajl-devel \
>>> +        pixman-devel \
>>> +        glibc-devel \
>>> +        glibc-devel.i686 \
>>
>> This looks fairly x86. Do you have any plan to add arm support?
>>
>> In any case, the README should explain what architectures are supported.
>>
>> Cheers,
>>
> 
> Yes it definitely x86. I plan on adding ARM support I just haven't had
> enough time to gather up what the dependencies need to be. Since its
> going to be a cross compile environment I need to take the time to setup
> one. Do you have a recommendation? I'm sure there are docs I just
> haven't looked at them.

To summarize what was said on IRC, I would recommend to use QEMU 
user-emulation instead. Stefano mentioned he was working on it and will 
send an e-mail about it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 2502 bytes --]

On 3/14/18 4:45 AM, George Dunlap wrote:
> On Tue, Mar 13, 2018 at 3:31 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>> Really early work on switching over to using GitLab CI over
>> Travis CI. GitLab is a competitor to GitHub with some advantages
>> such as an integrated CI system with a lot more flexibility
>> and control. It additionally is fully open sourced unlike GitHub
>> and Travis CI. We can even run an instance if that is preferred
>> over using the hosted instance.
>>
>> This change uses GitLab CI's ability to use Docker based runners
>> for running tests. With GitHub we also use a Docker based runner
>> but we are limited to one Docker container that is then morphed
>> a number of different ways. With this approach we can specify
>> different Docker containers for every run (or use the same). By
>> using different Docker containers we can build environments that
>> match systems where Xen can and should build. Using this
>> approach we should be able to cutdown on the number of surpise
>> build failures encountered by users.
>>
>> An example run can be seen here:
>> https://gitlab.com/cardoe/xen/pipelines/18789907
>>
>> If there is interest in this I will move it over to the "xen-project"
>> name space in the next version.
>>
>> Doug Goldstein (6):
>>   ci: add Dockerfile for CentOS 7.2
>>   ci: add Dockerfile for Ubuntu 14.04
>>   ci: add Dockerfile for Ubuntu 16.04
>>   ci: add Dockerfile for Debian jessie
>>   ci: add cfg to use GitLab CI to build
>>   ci: add a README about the containers
>>
>>  .gitlab-ci.yml                          | 34 ++++++++++++++++++++++-
>>  extras/testing/README.md                | 29 ++++++++++++++++++-
>>  extras/testing/centos/CentOS-7.2.repo   | 35 ++++++++++++++++++++++-
>>  extras/testing/centos/Dockerfile.7.2    | 41 ++++++++++++++++++++++++++-
>>  extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++-
>>  extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++-
>>  extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++-
> 
> "extras" is a bit generic.  What about something like "automation/build"?
> 
> (You knew this bike shed wasn't going to get in without *some*
> discussion of the color!)
> 
>  -George
> 

Ha. Thank you. So in the same thread I'll move the helper script that
the CI will use into "automation/scripts"? Currently we have
"scripts/travis-build" but there are going to be some more (specifically
for ARM)

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 0/6] Using GitLab CI for build testing

[George Dunlap]

On 03/14/2018 01:28 PM, Doug Goldstein wrote:
> On 3/14/18 4:45 AM, George Dunlap wrote:
>> On Tue, Mar 13, 2018 at 3:31 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>> Really early work on switching over to using GitLab CI over
>>> Travis CI. GitLab is a competitor to GitHub with some advantages
>>> such as an integrated CI system with a lot more flexibility
>>> and control. It additionally is fully open sourced unlike GitHub
>>> and Travis CI. We can even run an instance if that is preferred
>>> over using the hosted instance.
>>>
>>> This change uses GitLab CI's ability to use Docker based runners
>>> for running tests. With GitHub we also use a Docker based runner
>>> but we are limited to one Docker container that is then morphed
>>> a number of different ways. With this approach we can specify
>>> different Docker containers for every run (or use the same). By
>>> using different Docker containers we can build environments that
>>> match systems where Xen can and should build. Using this
>>> approach we should be able to cutdown on the number of surpise
>>> build failures encountered by users.
>>>
>>> An example run can be seen here:
>>> https://gitlab.com/cardoe/xen/pipelines/18789907
>>>
>>> If there is interest in this I will move it over to the "xen-project"
>>> name space in the next version.
>>>
>>> Doug Goldstein (6):
>>>   ci: add Dockerfile for CentOS 7.2
>>>   ci: add Dockerfile for Ubuntu 14.04
>>>   ci: add Dockerfile for Ubuntu 16.04
>>>   ci: add Dockerfile for Debian jessie
>>>   ci: add cfg to use GitLab CI to build
>>>   ci: add a README about the containers
>>>
>>>  .gitlab-ci.yml                          | 34 ++++++++++++++++++++++-
>>>  extras/testing/README.md                | 29 ++++++++++++++++++-
>>>  extras/testing/centos/CentOS-7.2.repo   | 35 ++++++++++++++++++++++-
>>>  extras/testing/centos/Dockerfile.7.2    | 41 ++++++++++++++++++++++++++-
>>>  extras/testing/debian/Dockerfile.jessie | 21 +++++++++++++-
>>>  extras/testing/ubuntu/Dockerfile.trusty | 21 +++++++++++++-
>>>  extras/testing/ubuntu/Dockerfile.xenial | 21 +++++++++++++-
>>
>> "extras" is a bit generic.  What about something like "automation/build"?
>>
>> (You knew this bike shed wasn't going to get in without *some*
>> discussion of the color!)
>>
>>  -George
>>
> 
> Ha. Thank you. So in the same thread I'll move the helper script that
> the CI will use into "automation/scripts"? Currently we have
> "scripts/travis-build" but there are going to be some more (specifically
> for ARM)

Yes, we should try to group all the "hooks for external automated tools"
together somehow.

FWIW, although I still prefer "automation", upon reflection "extras" is
a nice color too. :-)

Thanks again for doing this.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel