[PATCH 0/5] QEMU Gating CI

[PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

The idea about a public facing Gating CI for QEMU was lastly
summarized in an RFC[1].  Since then, it was decided that a
simpler version should be attempted first.

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
  (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement
  (there's no static version of those libs in distro supplied
  packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
  doesn't fall into the general scope of gating job described by PMM
  (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs `make
  check`
- Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
  simple but effective way of speeding up the builds and tests by
  using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles,
  some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (5):
  tests/docker: add CentOS 8 Dockerfile
  tests/docker: make "buildah bud" output similar to "docker build"
  GitLab CI: avoid calling before_scripts on unintended jobs
  GitLab Gating CI: introduce pipeline-status contrib script
  GitLab Gating CI: initial set of jobs, documentation and scripts

 .gitlab-ci-gating.yml                         | 111 ++++++++++
 .gitlab-ci.yml                                |  32 ++-
 contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
 contrib/ci/orgs/qemu/inventory                |   2 +
 contrib/ci/orgs/qemu/vars.yml                 |  13 ++
 contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
 docs/devel/testing.rst                        | 142 ++++++++++++
 tests/docker/dockerfiles/centos8.docker       |  32 +++
 .../dockerfiles/debian-win32-cross.docker     |   2 +-
 10 files changed, 751 insertions(+), 4 deletions(-)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
 create mode 100644 tests/docker/dockerfiles/centos8.docker

-- 
2.24.1

[PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

[Cleber Rosa]

Which is currenly missing, and will be referenced later in the
contributed CI playbooks.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/docker/dockerfiles/centos8.docker

diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
new file mode 100644
index 0000000000..bfa0d33c9c
--- /dev/null
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -0,0 +1,32 @@
+FROM centos:8.1.1911
+
+RUN dnf -y update
+ENV PACKAGES \
+    SDL-devel \
+    bison \
+    bzip2 \
+    bzip2-devel \
+    dbus-daemon \
+    flex \
+    gcc \
+    gcc-c++ \
+    gettext \
+    git \
+    glib2-devel \
+    libaio-devel \
+    libepoxy-devel \
+    lzo-devel \
+    make \
+    mesa-libEGL-devel \
+    nettle-devel \
+    perl-Test-Harness \
+    pixman-devel \
+    python36 \
+    rdma-core-devel \
+    spice-glib-devel \
+    spice-server \
+    tar \
+    zlib-devel
+
+RUN dnf install -y $PACKAGES
+RUN rpm -q $PACKAGES | sort > /packages.txt
-- 
2.24.1

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

[Erik Skultety]

On Thu, Mar 12, 2020 at 03:36:12PM -0400, Cleber Rosa wrote:
> Which is currenly missing, and will be referenced later in the
> contributed CI playbooks.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/centos8.docker
> 
> diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
> new file mode 100644
> index 0000000000..bfa0d33c9c
> --- /dev/null
> +++ b/tests/docker/dockerfiles/centos8.docker
> @@ -0,0 +1,32 @@
> +FROM centos:8.1.1911
> +
> +RUN dnf -y update
> +ENV PACKAGES \
> +    SDL-devel \
> +    bison \
> +    bzip2 \
> +    bzip2-devel \
> +    dbus-daemon \
> +    flex \
> +    gcc \
> +    gcc-c++ \
> +    gettext \
> +    git \
> +    glib2-devel \
> +    libaio-devel \
> +    libepoxy-devel \
> +    lzo-devel \
> +    make \
> +    mesa-libEGL-devel \
> +    nettle-devel \
> +    perl-Test-Harness \
> +    pixman-devel \
> +    python36 \
> +    rdma-core-devel \
> +    spice-glib-devel \
> +    spice-server \
> +    tar \
> +    zlib-devel
> +
> +RUN dnf install -y $PACKAGES
> +RUN rpm -q $PACKAGES | sort > /packages.txt

How is the packages.txt consumed later?

Erik

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

[Alex Bennée]

Erik Skultety <eskultet@redhat.com> writes:

> On Thu, Mar 12, 2020 at 03:36:12PM -0400, Cleber Rosa wrote:
>> Which is currenly missing, and will be referenced later in the
>> contributed CI playbooks.
>> 
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
>>  1 file changed, 32 insertions(+)
>>  create mode 100644 tests/docker/dockerfiles/centos8.docker
>> 
>> diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
>> new file mode 100644
>> index 0000000000..bfa0d33c9c
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/centos8.docker
>> @@ -0,0 +1,32 @@
>> +FROM centos:8.1.1911
>> +
>> +RUN dnf -y update
>> +ENV PACKAGES \
>> +    SDL-devel \
>> +    bison \
>> +    bzip2 \
>> +    bzip2-devel \
>> +    dbus-daemon \
>> +    flex \
>> +    gcc \
>> +    gcc-c++ \
>> +    gettext \
>> +    git \
>> +    glib2-devel \
>> +    libaio-devel \
>> +    libepoxy-devel \
>> +    lzo-devel \
>> +    make \
>> +    mesa-libEGL-devel \
>> +    nettle-devel \
>> +    perl-Test-Harness \
>> +    pixman-devel \
>> +    python36 \
>> +    rdma-core-devel \
>> +    spice-glib-devel \
>> +    spice-server \
>> +    tar \
>> +    zlib-devel
>> +
>> +RUN dnf install -y $PACKAGES
>> +RUN rpm -q $PACKAGES | sort > /packages.txt
>
> How is the packages.txt consumed later?

I think it's just following the pattern of a number of the containers by dumping the
package list for later reference by the run support scripts (tests/docker/run):

  if test -n "$SHOW_ENV"; then
      if test -f /packages.txt; then
          echo "Packages installed:"
          cat /packages.txt
          echo
      fi
      echo "Environment variables:"
      env
      echo
  fi

>
> Erik


-- 
Alex Bennée

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

On Fri, Mar 13, 2020 at 09:46:55AM +0100, Erik Skultety wrote:
> On Thu, Mar 12, 2020 at 03:36:12PM -0400, Cleber Rosa wrote:
> > Which is currenly missing, and will be referenced later in the
> > contributed CI playbooks.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
> >  1 file changed, 32 insertions(+)
> >  create mode 100644 tests/docker/dockerfiles/centos8.docker
> > 
> > diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
> > new file mode 100644
> > index 0000000000..bfa0d33c9c
> > --- /dev/null
> > +++ b/tests/docker/dockerfiles/centos8.docker
> > @@ -0,0 +1,32 @@
> > +FROM centos:8.1.1911
> > +
> > +RUN dnf -y update
> > +ENV PACKAGES \
> > +    SDL-devel \
> > +    bison \
> > +    bzip2 \
> > +    bzip2-devel \
> > +    dbus-daemon \
> > +    flex \
> > +    gcc \
> > +    gcc-c++ \
> > +    gettext \
> > +    git \
> > +    glib2-devel \
> > +    libaio-devel \
> > +    libepoxy-devel \
> > +    lzo-devel \
> > +    make \
> > +    mesa-libEGL-devel \
> > +    nettle-devel \
> > +    perl-Test-Harness \
> > +    pixman-devel \
> > +    python36 \
> > +    rdma-core-devel \
> > +    spice-glib-devel \
> > +    spice-server \
> > +    tar \
> > +    zlib-devel
> > +
> > +RUN dnf install -y $PACKAGES
> > +RUN rpm -q $PACKAGES | sort > /packages.txt
> 
> How is the packages.txt consumed later?
> 
> Erik

The explanation given by Alex on this thread is better than I would be
able to provide. :)

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/5] tests/docker: add CentOS 8 Dockerfile

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> Which is currenly missing, and will be referenced later in the
> contributed CI playbooks.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/centos8.docker | 32 +++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/centos8.docker
>
> diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
> new file mode 100644
> index 0000000000..bfa0d33c9c
> --- /dev/null
> +++ b/tests/docker/dockerfiles/centos8.docker
> @@ -0,0 +1,32 @@
> +FROM centos:8.1.1911
> +
> +RUN dnf -y update
> +ENV PACKAGES \
> +    SDL-devel \
> +    bison \
> +    bzip2 \
> +    bzip2-devel \
> +    dbus-daemon \
> +    flex \
> +    gcc \
> +    gcc-c++ \
> +    gettext \
> +    git \
> +    glib2-devel \
> +    libaio-devel \
> +    libepoxy-devel \
> +    lzo-devel \
> +    make \
> +    mesa-libEGL-devel \
> +    nettle-devel \
> +    perl-Test-Harness \
> +    pixman-devel \
> +    python36 \
> +    rdma-core-devel \
> +    spice-glib-devel \
> +    spice-server \
> +    tar \
> +    zlib-devel
> +
> +RUN dnf install -y $PACKAGES
> +RUN rpm -q $PACKAGES | sort > /packages.txt


-- 
Alex Bennée

[PATCH 2/5] tests/docker: make "buildah bud" output similar to "docker build"

[Cleber Rosa]

Podman users will most often be using buildah to build containers.
Among the differences between "buildah bud|build-using-dockerfile" and
a traditional "docker build" is that buildah does not run a container
during build.

To the best of my knowledge and experiments, this means that runtime
variables, such as ENV from one base image will not propagate into
another.  The end result is that the location for the cross compiler
binaries, defined in the base "qemu/debian9-mxe" image, are not passed
through this image.  Consequently, the cross compilers are not on PATH
and the build fails.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/docker/dockerfiles/debian-win32-cross.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
index 9d7053e59d..d16d6431bc 100644
--- a/tests/docker/dockerfiles/debian-win32-cross.docker
+++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -9,7 +9,7 @@ MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
 ENV TARGET i686
 
-ENV PATH $PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
+ENV PATH $PATH:/usr/lib/mxe/usr/bin:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
 
 ENV PKG_CONFIG_PATH \
     $PKG_CONFIG_PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/lib/pkgconfig
-- 
2.24.1

Re: [PATCH 2/5] tests/docker: make "buildah bud" output similar to "docker build"

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> Podman users will most often be using buildah to build containers.
> Among the differences between "buildah bud|build-using-dockerfile" and
> a traditional "docker build" is that buildah does not run a container
> during build.
>
> To the best of my knowledge and experiments, this means that runtime
> variables, such as ENV from one base image will not propagate into
> another.  The end result is that the location for the cross compiler
> binaries, defined in the base "qemu/debian9-mxe" image, are not passed
> through this image.  Consequently, the cross compilers are not on PATH
> and the build fails.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Acked-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-win32-cross.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
> index 9d7053e59d..d16d6431bc 100644
> --- a/tests/docker/dockerfiles/debian-win32-cross.docker
> +++ b/tests/docker/dockerfiles/debian-win32-cross.docker
> @@ -9,7 +9,7 @@ MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>  
>  ENV TARGET i686
>  
> -ENV PATH $PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
> +ENV PATH $PATH:/usr/lib/mxe/usr/bin:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/bin
>  
>  ENV PKG_CONFIG_PATH \
>      $PKG_CONFIG_PATH:/usr/lib/mxe/usr/$TARGET-w64-mingw32.shared/lib/pkgconfig


-- 
Alex Bennée

[PATCH 3/5] GitLab CI: avoid calling before_scripts on unintended jobs

[Cleber Rosa]

At this point it seems that all jobs depend on those steps, with
maybe the EDK2 jobs as exceptions.

The jobs that will be added will not want those scripts to be
run, so let's move these steps to the appropriate jobs, while
still trying to avoid repetition.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 72f8b8aa51..3ef1551273 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,11 +1,10 @@
 include:
   - local: '/.gitlab-ci-edk2.yml'
 
-before_script:
+build-system1:
+ before_script: &before_scr_apt
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
-
-build-system1:
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
       libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
@@ -18,6 +17,8 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
       libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
@@ -31,6 +32,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ before_script:
+  *before_scr_apt
  script:
  - mkdir build
  - cd build
@@ -45,6 +48,8 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq clang libgtk-3-dev libusb-dev
  - mkdir build
@@ -63,6 +68,8 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ before_script:
+  *before_scr_apt
  script:
  - mkdir build
  - cd build
@@ -72,6 +79,8 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ before_script:
+  *before_scr_apt
  script:
  - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
       xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
@@ -84,6 +93,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ before_script:
+  *before_scr_apt
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
-- 
2.24.1

[PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Cleber Rosa]

This script is intended to be used right after a push to a branch.

By default, it will look for the pipeline associated with the commit
that is the HEAD of the *local* staging branch.  It can be used as a
one time check, or with the `--wait` option to wait until the pipeline
completes.

If the pipeline is successful, then a merge of the staging branch into
the master branch should be the next step.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100755 contrib/ci/scripts/gitlab-pipeline-status

diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status
new file mode 100755
index 0000000000..83d412daec
--- /dev/null
+++ b/contrib/ci/scripts/gitlab-pipeline-status
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+
+"""
+Checks the GitLab pipeline status for a given commit commit
+"""
+
+# pylint: disable=C0103
+
+import argparse
+import http.client
+import json
+import os
+import subprocess
+import time
+import sys
+
+
+def get_local_staging_branch_commit():
+    """
+    Returns the commit sha1 for the *local* branch named "staging"
+    """
+    result = subprocess.run(['git', 'rev-parse', 'staging'],
+                            stdin=subprocess.DEVNULL,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.DEVNULL,
+                            cwd=os.path.dirname(__file__),
+                            universal_newlines=True).stdout.strip()
+    if result == 'staging':
+        raise ValueError("There's no local staging branch")
+    if len(result) != 40:
+        raise ValueError("Branch staging HEAD doesn't look like a sha1")
+    return result
+
+
+def get_pipeline_status(project_id, commit_sha1):
+    """
+    Returns the JSON content of the pipeline status API response
+    """
+    url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id,
+                                                        commit_sha1)
+    connection = http.client.HTTPSConnection('gitlab.com')
+    connection.request('GET', url=url)
+    response = connection.getresponse()
+    if response.code != http.HTTPStatus.OK:
+        raise ValueError("Failed to receive a successful response")
+    json_response = json.loads(response.read())
+    # afaict, there should one one pipeline for the same project + commit
+    # if this assumption is false, we can add further filters to the
+    # url, such as username, and order_by.
+    if not json_response:
+        raise ValueError("No pipeline found")
+    return json_response[0]
+
+
+def wait_on_pipeline_success(timeout, interval,
+                             project_id, commit_sha):
+    """
+    Waits for the pipeline to end up to the timeout given
+    """
+    start = time.time()
+    while True:
+        if time.time() >= (start + timeout):
+            print("Waiting on the pipeline success timed out")
+            return False
+
+        status = get_pipeline_status(project_id, commit_sha)
+        if status['status'] == 'running':
+            time.sleep(interval)
+            print('running...')
+            continue
+
+        if status['status'] == 'success':
+            return True
+
+        msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url']
+        print(msg)
+        return False
+
+
+def main():
+    """
+    Script entry point
+    """
+    parser = argparse.ArgumentParser(
+        prog='pipeline-status',
+        description='check or wait on a pipeline status')
+
+    parser.add_argument('-t', '--timeout', type=int, default=7200,
+                        help=('Amount of time (in seconds) to wait for the '
+                              'pipeline to complete.  Defaults to '
+                              '%(default)s'))
+    parser.add_argument('-i', '--interval', type=int, default=60,
+                        help=('Amount of time (in seconds) to wait between '
+                              'checks of the pipeline status.  Defaults '
+                              'to %(default)s'))
+    parser.add_argument('-w', '--wait', action='store_true', default=False,
+                        help=('Wether to wait, instead of checking only once '
+                              'the status of a pipeline'))
+    parser.add_argument('-p', '--project-id', type=int, default=11167699,
+                        help=('The GitLab project ID. Defaults to the project '
+                              'for https://gitlab.com/qemu-project/qemu, that '
+                              'is, "%(default)s"'))
+    try:
+        default_commit = get_local_staging_branch_commit()
+        commit_required = False
+    except ValueError:
+        default_commit = ''
+        commit_required = True
+    parser.add_argument('-c', '--commit', required=commit_required,
+                        default=default_commit,
+                        help=('Look for a pipeline associated with the given '
+                              'commit.  If one is not explicitly given, the '
+                              'commit associated with the local branch named '
+                              '"staging" is used.  Default: %(default)s'))
+    parser.add_argument('--verbose', action='store_true', default=False,
+                        help=('A minimal verbosity level that prints the '
+                              'overall result of the check/wait'))
+
+    args = parser.parse_args()
+
+    try:
+        if args.wait:
+            success = wait_on_pipeline_success(
+                args.timeout,
+                args.interval,
+                args.project_id,
+                args.commit)
+        else:
+            status = get_pipeline_status(args.project_id,
+                                         args.commit)
+            success = status['status'] == 'success'
+    except Exception as error:      # pylint: disable=W0703
+        success = False
+        if args.verbose:
+            print("ERROR: %s" % error.args[0])
+
+    if success:
+        if args.verbose:
+            print('success')
+        sys.exit(0)
+    else:
+        if args.verbose:
+            print('failure')
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()
-- 
2.24.1

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Peter Maydell]

On Thu, 12 Mar 2020 at 19:37, Cleber Rosa <crosa@redhat.com> wrote:
>
> This script is intended to be used right after a push to a branch.
>
> By default, it will look for the pipeline associated with the commit
> that is the HEAD of the *local* staging branch.  It can be used as a
> one time check, or with the `--wait` option to wait until the pipeline
> completes.
>
> If the pipeline is successful, then a merge of the staging branch into
> the master branch should be the next step.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
>  1 file changed, 148 insertions(+)
>  create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
>
> diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status
> new file mode 100755
> index 0000000000..83d412daec
> --- /dev/null
> +++ b/contrib/ci/scripts/gitlab-pipeline-status
> @@ -0,0 +1,148 @@
> +#!/usr/bin/env python3
> +
> +"""
> +Checks the GitLab pipeline status for a given commit commit
> +"""

All new files, and particularly new scripts and source
files, should have the usual copyright-and-license
comment at the top, please.

thanks
-- PMM

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

On Fri, Mar 13, 2020 at 01:56:51PM +0000, Peter Maydell wrote:
> 
> All new files, and particularly new scripts and source
> files, should have the usual copyright-and-license
> comment at the top, please.
> 
> thanks
> -- PMM
> 

My bad.  I'll wait for further comments and included that in a v2.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Daniel P. Berrangé]

On Thu, Mar 12, 2020 at 03:36:15PM -0400, Cleber Rosa wrote:
> This script is intended to be used right after a push to a branch.
> 
> By default, it will look for the pipeline associated with the commit
> that is the HEAD of the *local* staging branch.  It can be used as a
> one time check, or with the `--wait` option to wait until the pipeline
> completes.
> 
> If the pipeline is successful, then a merge of the staging branch into
> the master branch should be the next step.

On IRC yesterday we were discussing the current .gitlab-ci.yml status,
and how frustrating it is that every time we get it green, more code is
soon merged that turns it red again.

It feels like it should be an easy win to declare that the current GitLab
CI jobs are to be used as a gating tests for merges to master.

As & when custom runners come online, their jobs can simply augment the
existing jobs. IOW, use of GitLab CI for gating master shouldn't be
dependant on setup of custom runners which we've been waiting on for a
long term.

Peter indicated that his main requirement is a way to automate the task
of kicking off the CI job & getting its status. It seems like the script
in this patch should fullfill that requirement.

Assuming Peter (or whomever is going todo the merge) has a fork of

   https://gitlab.com/qemu-project/qemu

then they need to find the "ID" number of their fork. This is
visible at the top for the page for their fork eg mine:

   https://gitlab.com/berrange/qemu

reports:

   "Project ID: 18588805 "

Assuming the fork is configured as a git remote called "gitlab", then
to use GitLab CI as gating test, all that appears to be needed is

   $ git push gitlab
   $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805

If this is an acceptable level of automation for Peter, then can we
get this specific patch merged right now and make current GitLab CI
be gating for master.


With GitLab as gating, then we have further incentive to move all
the jobs currently on Travis CI and Shippable, over to use GitLab
too, and also use cirrus-run  to make Cirrus CI jobs be proxied
from GitLab.  All this can be in parallel with adding custom GitLab
runners for expanding testing coverage still further.

> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++
>  1 file changed, 148 insertions(+)
>  create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
> 
> diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status
> new file mode 100755
> index 0000000000..83d412daec
> --- /dev/null
> +++ b/contrib/ci/scripts/gitlab-pipeline-status
> @@ -0,0 +1,148 @@
> +#!/usr/bin/env python3
> +
> +"""
> +Checks the GitLab pipeline status for a given commit commit
> +"""
> +
> +# pylint: disable=C0103
> +
> +import argparse
> +import http.client
> +import json
> +import os
> +import subprocess
> +import time
> +import sys
> +
> +
> +def get_local_staging_branch_commit():
> +    """
> +    Returns the commit sha1 for the *local* branch named "staging"
> +    """
> +    result = subprocess.run(['git', 'rev-parse', 'staging'],
> +                            stdin=subprocess.DEVNULL,
> +                            stdout=subprocess.PIPE,
> +                            stderr=subprocess.DEVNULL,
> +                            cwd=os.path.dirname(__file__),
> +                            universal_newlines=True).stdout.strip()
> +    if result == 'staging':
> +        raise ValueError("There's no local staging branch")
> +    if len(result) != 40:
> +        raise ValueError("Branch staging HEAD doesn't look like a sha1")
> +    return result
> +
> +
> +def get_pipeline_status(project_id, commit_sha1):
> +    """
> +    Returns the JSON content of the pipeline status API response
> +    """
> +    url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id,
> +                                                        commit_sha1)
> +    connection = http.client.HTTPSConnection('gitlab.com')
> +    connection.request('GET', url=url)
> +    response = connection.getresponse()
> +    if response.code != http.HTTPStatus.OK:
> +        raise ValueError("Failed to receive a successful response")
> +    json_response = json.loads(response.read())
> +    # afaict, there should one one pipeline for the same project + commit
> +    # if this assumption is false, we can add further filters to the
> +    # url, such as username, and order_by.
> +    if not json_response:
> +        raise ValueError("No pipeline found")
> +    return json_response[0]
> +
> +
> +def wait_on_pipeline_success(timeout, interval,
> +                             project_id, commit_sha):
> +    """
> +    Waits for the pipeline to end up to the timeout given
> +    """
> +    start = time.time()
> +    while True:
> +        if time.time() >= (start + timeout):
> +            print("Waiting on the pipeline success timed out")
> +            return False
> +
> +        status = get_pipeline_status(project_id, commit_sha)
> +        if status['status'] == 'running':
> +            time.sleep(interval)
> +            print('running...')
> +            continue
> +
> +        if status['status'] == 'success':
> +            return True
> +
> +        msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url']
> +        print(msg)
> +        return False
> +
> +
> +def main():
> +    """
> +    Script entry point
> +    """
> +    parser = argparse.ArgumentParser(
> +        prog='pipeline-status',
> +        description='check or wait on a pipeline status')
> +
> +    parser.add_argument('-t', '--timeout', type=int, default=7200,
> +                        help=('Amount of time (in seconds) to wait for the '
> +                              'pipeline to complete.  Defaults to '
> +                              '%(default)s'))
> +    parser.add_argument('-i', '--interval', type=int, default=60,
> +                        help=('Amount of time (in seconds) to wait between '
> +                              'checks of the pipeline status.  Defaults '
> +                              'to %(default)s'))
> +    parser.add_argument('-w', '--wait', action='store_true', default=False,
> +                        help=('Wether to wait, instead of checking only once '
> +                              'the status of a pipeline'))
> +    parser.add_argument('-p', '--project-id', type=int, default=11167699,
> +                        help=('The GitLab project ID. Defaults to the project '
> +                              'for https://gitlab.com/qemu-project/qemu, that '
> +                              'is, "%(default)s"'))
> +    try:
> +        default_commit = get_local_staging_branch_commit()
> +        commit_required = False
> +    except ValueError:
> +        default_commit = ''
> +        commit_required = True
> +    parser.add_argument('-c', '--commit', required=commit_required,
> +                        default=default_commit,
> +                        help=('Look for a pipeline associated with the given '
> +                              'commit.  If one is not explicitly given, the '
> +                              'commit associated with the local branch named '
> +                              '"staging" is used.  Default: %(default)s'))
> +    parser.add_argument('--verbose', action='store_true', default=False,
> +                        help=('A minimal verbosity level that prints the '
> +                              'overall result of the check/wait'))
> +
> +    args = parser.parse_args()
> +
> +    try:
> +        if args.wait:
> +            success = wait_on_pipeline_success(
> +                args.timeout,
> +                args.interval,
> +                args.project_id,
> +                args.commit)
> +        else:
> +            status = get_pipeline_status(args.project_id,
> +                                         args.commit)
> +            success = status['status'] == 'success'
> +    except Exception as error:      # pylint: disable=W0703
> +        success = False
> +        if args.verbose:
> +            print("ERROR: %s" % error.args[0])
> +
> +    if success:
> +        if args.verbose:
> +            print('success')
> +        sys.exit(0)
> +    else:
> +        if args.verbose:
> +            print('failure')
> +        sys.exit(1)
> +
> +
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.24.1
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]

On Thu, Jun 18, 2020 at 12:45:24PM +0100, Daniel P. Berrangé wrote:
> On Thu, Mar 12, 2020 at 03:36:15PM -0400, Cleber Rosa wrote:
> > This script is intended to be used right after a push to a branch.
> > 
> > By default, it will look for the pipeline associated with the commit
> > that is the HEAD of the *local* staging branch.  It can be used as a
> > one time check, or with the `--wait` option to wait until the pipeline
> > completes.
> > 
> > If the pipeline is successful, then a merge of the staging branch into
> > the master branch should be the next step.
> 
> On IRC yesterday we were discussing the current .gitlab-ci.yml status,
> and how frustrating it is that every time we get it green, more code is
> soon merged that turns it red again.
>

Hi Daniel,

I know this is nothing new to you, but "green" turning "red" is
something that can be minimized, but never completely abolished.
We've had discussions on how we could minimize those, and suggestions
on how to address those occurrences.  The points that I remember about
it are:

1. For jobs supposed to run on containers and vms, use images prepared
   to run the builds and tests, that is, don't do the prep steps
   (package installations and others) within the job itself - this is
   currently missing.

2. For bare metal, have the setup either documented or scripted and run
   before the jobs (in the patch series I sent, these were done in
   a number of Ansible based playbooks).

3. Have a mechanism for promoting non-gating jobs to gating jobs.  Keeping
   track of their reliability over a period of time is a way to forecast
   it future behavior, as much as it can be done.

4. Have a MAINTAINERS entry for those jobs, and have maintainers
   attemtp to address issues within a specified amount of time; if
   that fails, have a mechanism to downgrade those jobs to non-gating.
   A patch removing/moving entries from .gitlab-ci*.yml would do that
   and of course sping another set of CI jobs bringing things back to
   green.

> It feels like it should be an easy win to declare that the current GitLab
> CI jobs are to be used as a gating tests for merges to master.
>

I'm fine with that approach if most people agree.  This would mean
point #1 is ignored for now.  For everyone's sake, points #3 and #4
are a must IMO.  Point #2 will applied as we introduced our non-shared
gitlab runners.

> As & when custom runners come online, their jobs can simply augment the
> existing jobs. IOW, use of GitLab CI for gating master shouldn't be
> dependant on setup of custom runners which we've been waiting on for a
> long term.
> 
> Peter indicated that his main requirement is a way to automate the task
> of kicking off the CI job & getting its status. It seems like the script
> in this patch should fullfill that requirement.
> 
> Assuming Peter (or whomever is going todo the merge) has a fork of
> 
>    https://gitlab.com/qemu-project/qemu
> 
> then they need to find the "ID" number of their fork. This is
> visible at the top for the page for their fork eg mine:
> 
>    https://gitlab.com/berrange/qemu
> 
> reports:
> 
>    "Project ID: 18588805 "
> 
> Assuming the fork is configured as a git remote called "gitlab", then
> to use GitLab CI as gating test, all that appears to be needed is
> 
>    $ git push gitlab
>    $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805
> 
> If this is an acceptable level of automation for Peter, then can we
> get this specific patch merged right now and make current GitLab CI
> be gating for master.
> 
> 
> With GitLab as gating, then we have further incentive to move all
> the jobs currently on Travis CI and Shippable, over to use GitLab
> too, and also use cirrus-run  to make Cirrus CI jobs be proxied
> from GitLab.  All this can be in parallel with adding custom GitLab
> runners for expanding testing coverage still further.
>

The script should indeed work with the workflow you described.  I'll just
run a few tests, to make sure that are no caveats, and I'll let you and
Peter know about it.

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

On Mon, Jun 22, 2020 at 10:21:03AM -0400, Cleber Rosa wrote:
> 
> The script should indeed work with the workflow you described.  I'll just
> run a few tests, to make sure that are no caveats, and I'll let you and
> Peter know about it.
>

FIY, I've cherry-pick just the patch with the scripts (for no reason other
than having some content) and pushed it to a different branch than staging:

  https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners

Then I ran:

  ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8

Which worked as expected.  BTW, the commit wouldn't be necessary if I
was checking on a "staging" branch.  The pipeline triggered (and being
waited on by the script is at):

   https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485

So I believe it's now a matter of sorting out job the
inclusion/exclusion aspects, that is, if we start with all/some of the
current jobs and some or none of the jobs intended to run on the
s390/aarch64/x86_64 (non-shared) runners.

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 4/5] GitLab Gating CI: introduce pipeline-status contrib script

[Philippe Mathieu-Daudé]

On 6/23/20 7:59 PM, Cleber Rosa wrote:
> On Mon, Jun 22, 2020 at 10:21:03AM -0400, Cleber Rosa wrote:
>>
>> The script should indeed work with the workflow you described.  I'll just
>> run a few tests, to make sure that are no caveats, and I'll let you and
>> Peter know about it.
>>
> 
> FIY, I've cherry-pick just the patch with the scripts (for no reason other
> than having some content) and pushed it to a different branch than staging:
> 
>   https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners
> 
> Then I ran:
> 
>   ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8
> 
> Which worked as expected.

Great news!

Can you respin as a new series with this as single patch, and
explanation in the cover about how to setup it? This is the last
piece missing for Peter to use the effort done by Alex/Daniel/Thomas
on the GitLab side :)

> BTW, the commit wouldn't be necessary if I
> was checking on a "staging" branch.  The pipeline triggered (and being
> waited on by the script is at):
> 
>    https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485
> 
> So I believe it's now a matter of sorting out job the
> inclusion/exclusion aspects, that is, if we start with all/some of the
> current jobs and some or none of the jobs intended to run on the
> s390/aarch64/x86_64 (non-shared) runners.
> 
> Thanks,
> - Cleber.
> 

[PATCH 5/5] GitLab Gating CI: initial set of jobs, documentation and scripts

[Cleber Rosa]

This is a mapping of Peter's "remake-merge-builds" and
"pull-buildtest" scripts, gone through some updates, adding some build
option and removing others.

To add to some of the diversity of platforms that QEMU covers, this
initial set of jobs require three machines:

 - Ubuntu 18.04
 - Fedora 30
 - FreeBSD 12.1

All jobs have been marked as x86_64, but others can be introduced with
different architectures.  During the development of this set of jobs,
the GitLab CI was tested with many other architectures, including
ppc64, s390x and aarch64.

More information can be found in the documentation itself.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci-gating.yml                      | 111 +++++++++++
 .gitlab-ci.yml                             |  15 ++
 contrib/ci/orgs/qemu/build-environment.yml | 208 +++++++++++++++++++++
 contrib/ci/orgs/qemu/gitlab-runner.yml     |  65 +++++++
 contrib/ci/orgs/qemu/inventory             |   2 +
 contrib/ci/orgs/qemu/vars.yml              |  13 ++
 docs/devel/testing.rst                     | 142 ++++++++++++++
 7 files changed, 556 insertions(+)
 create mode 100644 .gitlab-ci-gating.yml
 create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
 create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
 create mode 100644 contrib/ci/orgs/qemu/inventory
 create mode 100644 contrib/ci/orgs/qemu/vars.yml

diff --git a/.gitlab-ci-gating.yml b/.gitlab-ci-gating.yml
new file mode 100644
index 0000000000..4c6010bcc4
--- /dev/null
+++ b/.gitlab-ci-gating.yml
@@ -0,0 +1,111 @@
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
+
+# All ubuntu-18.04.3 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04"
+ubuntu-18.04.3-x86_64-all-linux-static:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - ./configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04.3-x86_64-all:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-alldbg:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-clang:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04.3-x86_64-tci:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04.3-x86_64-notcg:
+ tags:
+ - ubuntu_18.04.3
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All fedora-30 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Fedora 30"
+fedora-30-x86_64-cross-w32:
+ tags:
+ - fedora_30
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --cross-prefix=i686-w64-mingw32-
+ - make --output-sync -j`nproc`
+
+fedora-30-x86_64-cross-w64:
+ tags:
+ - fedora_30
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure --cross-prefix=x86_64-w64-mingw32-
+ - make --output-sync -j`nproc`
+
+# All freebsd_12.1 jobs should run successfully in an environment
+# setup by the contrib/ci/orgs/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on FreeBSD 12.x"
+freebsd-12.1-x86_64-all:
+ tags:
+ - freebsd_12.1
+ - x86_64
+ rules:
+ - if: '$CI_COMMIT_REF_NAME == "staging"'
+ script:
+ - ./configure
+ - gmake --output-sync -j`sysctl -n hw.ncpu`
+ - gmake --output-sync -j`sysctl -n hw.ncpu` check V=1
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3ef1551273..8c51672dc1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,7 +1,10 @@
 include:
   - local: '/.gitlab-ci-edk2.yml'
+  - local: '/.gitlab-ci-gating.yml'
 
 build-system1:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script: &before_scr_apt
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
@@ -17,6 +20,8 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -32,6 +37,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -48,6 +55,8 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -68,6 +77,8 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -79,6 +90,8 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
@@ -93,6 +106,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ rules:
+ - if: '$CI_COMMIT_REF_NAME != "staging"'
  before_script:
   *before_scr_apt
  script:
diff --git a/contrib/ci/orgs/qemu/build-environment.yml b/contrib/ci/orgs/qemu/build-environment.yml
new file mode 100644
index 0000000000..ea839ded4c
--- /dev/null
+++ b/contrib/ci/orgs/qemu/build-environment.yml
@@ -0,0 +1,208 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - name: Install basic packages to build QEMU on Ubuntu 18.04
+      apt:
+        update_cache: yes
+        # This matches the packages on tests/docker/Dockerfiles/ubuntu1804.docker
+        pkg:
+          - flex
+          - bison
+          - ccache
+          - clang
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libspice-server-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libxen-dev
+          - libzstd-dev
+          - make
+          - python3-yaml
+          - python3-sphinx
+          - sparse
+          - texinfo
+          - xfslibs-dev
+        state: present
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Install basic packages to build QEMU on FreeBSD 12.x
+      pkgng:
+        # This matches the packages on .cirrus.yml under the freebsd_12_task
+        name: bash,bison,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,perl5,pixman,pkgconf,png,usbredir
+        state: present
+      when: "ansible_facts['os_family'] == 'FreeBSD'"
+
+    - name: Install basic packages to build QEMU on Fedora 30
+      dnf:
+        # This matches the packages on tests/docker/Dockerfiles/fedora.docker
+        name:
+          - bc
+          - bison
+          - brlapi-devel
+          - bzip2
+          - bzip2-devel
+          - ccache
+          - clang
+          - cyrus-sasl-devel
+          - dbus-daemon
+          - device-mapper-multipath-devel
+          - findutils
+          - flex
+          - gcc
+          - gcc-c++
+          - gettext
+          - git
+          - glib2-devel
+          - glusterfs-api-devel
+          - gnutls-devel
+          - gtk3-devel
+          - hostname
+          - libaio-devel
+          - libasan
+          - libattr-devel
+          - libblockdev-mpath-devel
+          - libcap-ng-devel
+          - libcurl-devel
+          - libfdt-devel
+          - libiscsi-devel
+          - libjpeg-devel
+          - libpmem-devel
+          - libpng-devel
+          - librbd-devel
+          - libseccomp-devel
+          - libssh-devel
+          - libubsan
+          - libusbx-devel
+          - libxml2-devel
+          - llvm
+          - lzo-devel
+          - make
+          - mingw32-bzip2
+          - mingw32-curl
+          - mingw32-glib2
+          - mingw32-gmp
+          - mingw32-gnutls
+          - mingw32-gtk3
+          - mingw32-libjpeg-turbo
+          - mingw32-libpng
+          - mingw32-libtasn1
+          - mingw32-nettle
+          - mingw32-nsis
+          - mingw32-pixman
+          - mingw32-pkg-config
+          - mingw32-SDL2
+          - mingw64-bzip2
+          - mingw64-curl
+          - mingw64-glib2
+          - mingw64-gmp
+          - mingw64-gnutls
+          - mingw64-gtk3
+          - mingw64-libjpeg-turbo
+          - mingw64-libpng
+          - mingw64-libtasn1
+          - mingw64-nettle
+          - mingw64-pixman
+          - mingw64-pkg-config
+          - mingw64-SDL2
+          - ncurses-devel
+          - nettle-devel
+          - nss-devel
+          - numactl-devel
+          - perl
+          - perl-Test-Harness
+          - pixman-devel
+          - python3
+          - python3-sphinx
+          - PyYAML
+          - rdma-core-devel
+          - SDL2-devel
+          - snappy-devel
+          - sparse
+          - spice-server-devel
+          - systemd-devel
+          - systemtap-sdt-devel
+          - tar
+          - texinfo
+          - usbredir-devel
+          - virglrenderer-devel
+          - vte291-devel
+          - which
+          - xen-devel
+          - zlib-devel
+          - libzstd-devel
+        state: present
+      when: "ansible_facts['distribution'] == 'Fedora'"
+
+    - name: Install basic packages to build QEMU on CentOS 8
+      dnf:
+        # This matches the packages on tests/docker/Dockerfiles/centos8.docker
+        name:
+          - SDL-devel
+          - bison
+          - bzip2
+          - bzip2-devel
+          - dbus-daemon
+          - flex
+          - gcc
+          - gcc-c++
+          - gettext
+          - git
+          - glib2-devel
+          - libaio-devel
+          - libepoxy-devel
+          - lzo-devel
+          - make
+          - mesa-libEGL-devel
+          - nettle-devel
+          - perl-Test-Harness
+          - pixman-devel
+          - python36
+          - rdma-core-devel
+          - spice-glib-devel
+          - spice-server
+          - tar
+          - zlib-devel
+        state: present
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
diff --git a/contrib/ci/orgs/qemu/gitlab-runner.yml b/contrib/ci/orgs/qemu/gitlab-runner.yml
new file mode 100644
index 0000000000..19dc31a299
--- /dev/null
+++ b/contrib/ci/orgs/qemu/gitlab-runner.yml
@@ -0,0 +1,65 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403
+      register: gitlab_runner_available_archive
+
+    - name: Update base url
+      set_fact:
+        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
+      when: gitlab_runner_available_archive.status == 200
+    - debug:
+        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
+
+    - name: Set OS name (FreeBSD)
+      set_fact:
+        gitlab_runner_os: freebsd
+      when: "ansible_facts['system'] == 'FreeBSD'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Downloads the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
diff --git a/contrib/ci/orgs/qemu/inventory b/contrib/ci/orgs/qemu/inventory
new file mode 100644
index 0000000000..8bb7ba6b33
--- /dev/null
+++ b/contrib/ci/orgs/qemu/inventory
@@ -0,0 +1,2 @@
+[local]
+localhost
diff --git a/contrib/ci/orgs/qemu/vars.yml b/contrib/ci/orgs/qemu/vars.yml
new file mode 100644
index 0000000000..0095851172
--- /dev/null
+++ b/contrib/ci/orgs/qemu/vars.yml
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 12.8.0
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 770a987ea4..6f8e04e079 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -896,3 +896,145 @@ exercise as many corner cases as possible. It is a useful test suite
 to run to exercise QEMU's linux-user code::
 
   https://linux-test-project.github.io/
+
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most update information about them and their status can be found
+at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Gating CI
+----------
+
+A Pull Requests will only to be merged if they successfully go through
+a different set of CI jobs.  GitLab's CI is the service/framework used
+for executing the gating jobs.
+
+The architecture of GitLab's CI service allows different machines to be
+setup with GitLab's "agent", called gitlab-runner, which will take care
+of running jobs created by events such as a push to a branch.
+
+Even though gitlab-runner can execute jobs on environments such as
+containers, this initial implementation assumes the shell executor is
+used, effectively running jobs on the same machine (be them physical
+or virtual) the gitlab-runner agent is running.  This means those
+machines must be setup in advance, with the requirements matching the
+jobs expected to be executed there.
+
+Machine configuration for gating jobs
+-------------------------------------
+
+The GitLab's CI architecture allows different parties to provide
+different machines that will run different jobs.  At this point, QEMU
+will deploy a limited set of machines and jobs.  Documentation and/or
+scripts to setup those machines is located under::
+
+  contrib/ci/orgs/qemu
+
+Ansible playbooks have been provided to perform two different tasks
+related to setting gitlab-runner and the build environment.
+
+Other organizations involved in QEMU development may, in the near
+future, contribute their own setup documentation/scripts under
+``contrib/ci/orgs/$ORG_NAME`` directory.
+
+The GitLab CI jobs definition for the gating jobs are located under::
+
+  .gitlab-ci-gating.yml
+
+It's expected that these will be moved to a ``.gitlab-ci.d/``
+directory shortly.  Once the gating CI has proved mature with this set
+of initial jobs, other members in the community may provide their own
+machine configuration documentation/scripts, and accompanying job
+definitions.  Those contributed jobs should run as non-gating, until
+their reliability is verified.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Start by adding your machines to
+the ``inventory`` file under ``contrib/ci/orgs/qemu``, such as this::
+
+  [local]
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  [local]
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``contrib/ci/orgs/qemu/build-environment.yml`` Ansible playbook
+will setup machines with the environment needed to perform builds of
+QEMU and runs of the tests defined in the current gating jobs.  It
+covers a number of different Linux distributions and FreeBSD.
+
+To run the playbook, execute::
+
+  cd contrib/ci/orgs/qemu
+  ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Edit the ``contrib/ci/orgs/qemu/vars.yml`` file, setting the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+To run the playbook, execute::
+
+  cd contrib/ci/orgs/qemu
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at
+          https://docs.gitlab.com/runner/install/freebsd.html
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+FreeBSD 12.1 x86_64 system, the tags should be set as::
+
+  freebsd12.1,x86_64
+
+Because the job definition at ``.gitlab-ci-gating.yml`` contains::
+
+  freebsd-12.1-x86_64-all:
+   tags:
+   - freebsd_12.1
+   - x86_64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * uncheck the "Run untagged jobs" check box
+ * give it a better Description
-- 
2.24.1

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Thu, 12 Mar 2020 at 19:36, Cleber Rosa <crosa@redhat.com> wrote:
>
> The idea about a public facing Gating CI for QEMU was lastly
> summarized in an RFC[1].  Since then, it was decided that a
> simpler version should be attempted first.

OK, so my question here is:
 * what are the instructions that I have to follow to be
able to say "ok, here's my branch, run it through these tests,
please" ?

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On Thu, Mar 12, 2020 at 10:00:42PM +0000, Peter Maydell wrote:
> On Thu, 12 Mar 2020 at 19:36, Cleber Rosa <crosa@redhat.com> wrote:
> >
> > The idea about a public facing Gating CI for QEMU was lastly
> > summarized in an RFC[1].  Since then, it was decided that a
> > simpler version should be attempted first.
> 
> OK, so my question here is:
>  * what are the instructions that I have to follow to be
> able to say "ok, here's my branch, run it through these tests,
> please" ?

The quick answer is:

 $ git push git@gitlab.com:qemu-project/qemu.git my-branch:staging

The longer explanation is that these jobs are limited to a "staging"
branch, so all you'd have to do is to push something to a branch
called "staging".  If that branch happens to be from the
"gitlab.com/qemu-project/qemu" repo, than the runners setup there
would be used.  The documentation an ansible playbooks are supposed
to help with this setup.

Once that push happens, you could use:

 $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait

Before doing something like:

 $ git push git@gitlab.com:qemu-project/qemu.git my-branch:master

> 
> thanks
> -- PMM
> 

Let me know if that makes sense.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Thu, 12 Mar 2020 at 22:16, Cleber Rosa <crosa@redhat.com> wrote:
>
> On Thu, Mar 12, 2020 at 10:00:42PM +0000, Peter Maydell wrote:
> > OK, so my question here is:
> >  * what are the instructions that I have to follow to be
> > able to say "ok, here's my branch, run it through these tests,
> > please" ?
>
> The quick answer is:
>
>  $ git push git@gitlab.com:qemu-project/qemu.git my-branch:staging
>
> The longer explanation is that these jobs are limited to a "staging"
> branch, so all you'd have to do is to push something to a branch
> called "staging".  If that branch happens to be from the
> "gitlab.com/qemu-project/qemu" repo, than the runners setup there
> would be used.  The documentation an ansible playbooks are supposed
> to help with this setup.

Great, thanks. Could I do that for testing purposes with a
staging branch that includes these patches, or would we have
to wait for them to be in master before it works?

-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

On Fri, Mar 13, 2020 at 01:55:49PM +0000, Peter Maydell wrote:
> On Thu, 12 Mar 2020 at 22:16, Cleber Rosa <crosa@redhat.com> wrote:
> >
> > On Thu, Mar 12, 2020 at 10:00:42PM +0000, Peter Maydell wrote:
> > > OK, so my question here is:
> > >  * what are the instructions that I have to follow to be
> > > able to say "ok, here's my branch, run it through these tests,
> > > please" ?
> >
> > The quick answer is:
> >
> >  $ git push git@gitlab.com:qemu-project/qemu.git my-branch:staging
> >
> > The longer explanation is that these jobs are limited to a "staging"
> > branch, so all you'd have to do is to push something to a branch
> > called "staging".  If that branch happens to be from the
> > "gitlab.com/qemu-project/qemu" repo, than the runners setup there
> > would be used.  The documentation an ansible playbooks are supposed
> > to help with this setup.
> 
> Great, thanks. Could I do that for testing purposes with a
> staging branch that includes these patches, or would we have
> to wait for them to be in master before it works?
>

You can definitely do that with a staging branch that includes these
patches.

And with regards to setting up machines with runners and other
requirements, I tried to make it easy to replicate, but YMMV.  So,
please let me know if you find any issues whatsoever.

> -- PMM
> 

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Thu, 12 Mar 2020 at 22:16, Cleber Rosa <crosa@redhat.com> wrote:
> The quick answer is:
>
>  $ git push git@gitlab.com:qemu-project/qemu.git my-branch:staging

So I did this bit...

> Once that push happens, you could use:
>
>  $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait

...but this script just says:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait
ERROR: No pipeline found
failure

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Peter Maydell" <peter.maydell@linaro.org>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Alex Bennée" <alex.bennee@linaro.org>, "QEMU Developers" <qemu-devel@nongnu.org>, "Fam Zheng" <fam@euphon.net>,
> "Eduardo Habkost" <ehabkost@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Philippe Mathieu-Daudé"
> <philmd@redhat.com>, "Thomas Huth" <thuth@redhat.com>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Erik
> Skultety" <eskultet@redhat.com>, "Willian Rampazzo" <wrampazz@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>
> Sent: Monday, March 16, 2020 7:57:33 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Thu, 12 Mar 2020 at 22:16, Cleber Rosa <crosa@redhat.com> wrote:
> > The quick answer is:
> >
> >  $ git push git@gitlab.com:qemu-project/qemu.git my-branch:staging
> 
> So I did this bit...
> 
> > Once that push happens, you could use:
> >
> >  $ contrib/ci/scripts/gitlab-pipeline-status --verbose --wait
> 
> ...but this script just says:
> 
> $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --wait
> ERROR: No pipeline found
> failure
> 

Hi Peter,

A few possible reasons come to my mind:

1) It usually takes a few seconds after the push for the pipeline to

2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu,
   you'd have to tweak the project ID (-p|--project-id).

3) The local branch is not called "staging", so the script can not find the
   commit ID, in that case you can use -c|--commit.

> thanks
> -- PMM
> 
> 

Please let me know if any of these points helps.

Cheers,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Mon, 16 Mar 2020 at 12:04, Cleber Rosa <crosa@redhat.com> wrote:
> A few possible reasons come to my mind:
>
> 1) It usually takes a few seconds after the push for the pipeline to
>
> 2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu,
>    you'd have to tweak the project ID (-p|--project-id).
>
> 3) The local branch is not called "staging", so the script can not find the
>    commit ID, in that case you can use -c|--commit.

Yes, the local branch is something else for the purposes of
testing this series. But using --commit doesn't work either:

$ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
81beaaab0851fe8c4db971 --wait
ERROR: No pipeline found
failure

On the web UI:
https://gitlab.com/qemu-project/qemu/pipelines
the pipelines are marked "stuck" (I don't know why there
are two of them for the same commit); drilling down,
the build part has completed but all the test parts are
pending with "This job is stuck because you don't have
any active runners online with any of these tags assigned
to them" type messages.

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Peter Maydell" <peter.maydell@linaro.org>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Alex Bennée" <alex.bennee@linaro.org>, "QEMU Developers" <qemu-devel@nongnu.org>, "Fam Zheng" <fam@euphon.net>,
> "Eduardo Habkost" <ehabkost@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Philippe Mathieu-Daudé"
> <philmd@redhat.com>, "Thomas Huth" <thuth@redhat.com>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Erik
> Skultety" <eskultet@redhat.com>, "Willian Rampazzo" <wrampazz@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>
> Sent: Monday, March 16, 2020 8:12:16 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Mon, 16 Mar 2020 at 12:04, Cleber Rosa <crosa@redhat.com> wrote:
> > A few possible reasons come to my mind:
> >
> > 1) It usually takes a few seconds after the push for the pipeline to
> >
> > 2) If you've pushed to a repo different than gitlab.com/qemu-project/qemu,
> >    you'd have to tweak the project ID (-p|--project-id).
> >
> > 3) The local branch is not called "staging", so the script can not find the
> >    commit ID, in that case you can use -c|--commit.
> 
> Yes, the local branch is something else for the purposes of
> testing this series. But using --commit doesn't work either:
> 
> $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
> 81beaaab0851fe8c4db971 --wait
> ERROR: No pipeline found
> failure
> 

This may be a bug in the script because I was not expecting two
pipelines for the same commit. Checking...

> On the web UI:
> https://gitlab.com/qemu-project/qemu/pipelines
> the pipelines are marked "stuck" (I don't know why there
> are two of them for the same commit); drilling down,
> the build part has completed but all the test parts are
> pending with "This job is stuck because you don't have
> any active runners online with any of these tags assigned
> to them" type messages.
> 

I had also not come across the duplicate pipelines, so I'm trying
to understand what's that about.

About the runners and the fact that the job is stuck without them,
the message seems straightforward enough, but I can't get to the
project configuration to look at the registered runners with my
current permissions (set as "developer").

> thanks
> -- PMM
> 
> 

Thanks,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Cleber Rosa" <crosa@redhat.com>
> To: "Peter Maydell" <peter.maydell@linaro.org>
> Cc: "Alex Bennée" <alex.bennee@linaro.org>, "QEMU Developers" <qemu-devel@nongnu.org>, "Fam Zheng" <fam@euphon.net>,
> "Eduardo Habkost" <ehabkost@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Philippe Mathieu-Daudé"
> <philmd@redhat.com>, "Thomas Huth" <thuth@redhat.com>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Erik
> Skultety" <eskultet@redhat.com>, "Willian Rampazzo" <wrampazz@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>
> Sent: Monday, March 16, 2020 8:26:46 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> 
> 
> ----- Original Message -----
> > From: "Peter Maydell" <peter.maydell@linaro.org>
> > To: "Cleber Rosa" <crosa@redhat.com>
> > Cc: "Alex Bennée" <alex.bennee@linaro.org>, "QEMU Developers"
> > <qemu-devel@nongnu.org>, "Fam Zheng" <fam@euphon.net>,
> > "Eduardo Habkost" <ehabkost@redhat.com>, "Beraldo Leal" <bleal@redhat.com>,
> > "Philippe Mathieu-Daudé"
> > <philmd@redhat.com>, "Thomas Huth" <thuth@redhat.com>, "Wainer dos Santos
> > Moschetta" <wainersm@redhat.com>, "Erik
> > Skultety" <eskultet@redhat.com>, "Willian Rampazzo" <wrampazz@redhat.com>,
> > "Wainer Moschetta" <wmoschet@redhat.com>
> > Sent: Monday, March 16, 2020 8:12:16 AM
> > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > 
> > On Mon, 16 Mar 2020 at 12:04, Cleber Rosa <crosa@redhat.com> wrote:
> > > A few possible reasons come to my mind:
> > >
> > > 1) It usually takes a few seconds after the push for the pipeline to
> > >
> > > 2) If you've pushed to a repo different than
> > > gitlab.com/qemu-project/qemu,
> > >    you'd have to tweak the project ID (-p|--project-id).
> > >
> > > 3) The local branch is not called "staging", so the script can not find
> > > the
> > >    commit ID, in that case you can use -c|--commit.
> > 
> > Yes, the local branch is something else for the purposes of
> > testing this series. But using --commit doesn't work either:
> > 
> > $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit
> > 81beaaab0851fe8c4db971 --wait
> > ERROR: No pipeline found
> > failure
> > 
> 
> This may be a bug in the script because I was not expecting two
> pipelines for the same commit. Checking...
> 

Looks like the GitLab API requires a 40 char commit ID, so this
should work:

 $ ./contrib/ci/scripts/gitlab-pipeline-status --verbose --commit 81beaaab0851fe8c4db971df555600152bb83a6c --wait

It'll stay silent at first, but then will print a message every 60 seconds.

Thanks,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Mon, 16 Mar 2020 at 12:26, Cleber Rosa <crosa@redhat.com> wrote:
> About the runners and the fact that the job is stuck without them,
> the message seems straightforward enough, but I can't get to the
> project configuration to look at the registered runners with my
> current permissions (set as "developer").

I've moved you up to 'maintainer' status, hopefully that is
sufficient to look at the relevant config ?

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Peter Maydell" <peter.maydell@linaro.org>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
> "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
> <wrampazz@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>
> Sent: Monday, March 16, 2020 10:57:30 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Mon, 16 Mar 2020 at 12:26, Cleber Rosa <crosa@redhat.com> wrote:
> > About the runners and the fact that the job is stuck without them,
> > the message seems straightforward enough, but I can't get to the
> > project configuration to look at the registered runners with my
> > current permissions (set as "developer").
> 
> I've moved you up to 'maintainer' status, hopefully that is
> sufficient to look at the relevant config ?
> 
> thanks
> -- PMM
> 
> 

Hi Peter,

Yes, that did the trick and I can now see the configuration.  What I can
*not* see is any "Specific Runner" configured.  So maybe:

1) The documentation I included is not clear enough about the fact that
setup steps need to be done on a machine so that it becomes a "Runner"

2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml)
is not working as intended

3) Some expectations misalignment on machines that would be available to run
those jobs

In any case, none of those should be big problems.  Please let me know what
you did/experienced/expected up to this point, and we can continue from there.

Regards,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Tue, 17 Mar 2020 at 04:59, Cleber Rosa <crosa@redhat.com> wrote:
> Yes, that did the trick and I can now see the configuration.  What I can
> *not* see is any "Specific Runner" configured.  So maybe:
>
> 1) The documentation I included is not clear enough about the fact that
> setup steps need to be done on a machine so that it becomes a "Runner"
>
> 2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml)
> is not working as intended
>
> 3) Some expectations misalignment on machines that would be available to run
> those jobs
>
> In any case, none of those should be big problems.  Please let me know what
> you did/experienced/expected up to this point, and we can continue from there.

Ah, I see. My assumption was that this was all stuff that you were
working on, so that I would then be able to test that it worked correctly,
not that I would need to do configuration of the gitlab.com setup.
I thought all the stuff about "how to set up runners" was only for
people who wanted to set up some 3rd-party CI for non-official
forks or for when we wanted to add extra runners in future (eg for
architectures not yet covered). So the only thing I did was follow
your "just push to staging" instructions.

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2729 bytes --]

On Tue, Mar 17, 2020 at 09:29:32AM +0000, Peter Maydell wrote:
> On Tue, 17 Mar 2020 at 04:59, Cleber Rosa <crosa@redhat.com> wrote:
> > Yes, that did the trick and I can now see the configuration.  What I can
> > *not* see is any "Specific Runner" configured.  So maybe:
> >
> > 1) The documentation I included is not clear enough about the fact that
> > setup steps need to be done on a machine so that it becomes a "Runner"
> >
> > 2) The (Ansible) playbooks (especially contrib/ci/orgs/qemu/gitlab-runner.yml)
> > is not working as intended
> >
> > 3) Some expectations misalignment on machines that would be available to run
> > those jobs
> >
> > In any case, none of those should be big problems.  Please let me know what
> > you did/experienced/expected up to this point, and we can continue from there.
> 
> Ah, I see. My assumption was that this was all stuff that you were
> working on, so that I would then be able to test that it worked correctly,
> not that I would need to do configuration of the gitlab.com setup.

Hi Peter,

So, I had to use temporary hardware resources to set this up (and set
it up countless times TBH).  I had the understanding based on the list
of machines you documented[1] that at least some of them would be used
for the permanent setup.

> I thought all the stuff about "how to set up runners" was only for
> people who wanted to set up some 3rd-party CI for non-official
> forks or for when we wanted to add extra runners in future (eg for
> architectures not yet covered). So the only thing I did was follow
> your "just push to staging" instructions.
>

OK, I see it, now it makes more sense.  So we're "only" missing the
setup for the machines we'll use for the more permanent setup.  Would
you like to do a staged setup/migration using one or some of the
machines you documented?  I'm 100% onboard to help with this, meaning
that I can assist you with instructions, or do "pair setup" of the
machines if needed.  I think a good part of the evaluation here comes
down to how manageable/reproducible the setup is, so it'd make sense
for one to be part of the setup itself.

FIY there's also the possibility of grabbing some free VMs on GCP,
Azure, etc and setting them up as GitLab runners in a temporary way
(because of the temporarily free and VM nature).  I have a few
problems with this approach, including the fact that it doesn't yield
the complete experience wrt using hardware one owns and will have to
manage, besides the hardware limitations themselves.

Please let me know how you want to move on from here.

Cheers,
- Cleber.

> thanks
> -- PMM
> 

[1] https://wiki.qemu.org/Requirements/GatingCI#Current_Tests

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Tue, 17 Mar 2020 at 14:13, Cleber Rosa <crosa@redhat.com> wrote:
>
> On Tue, Mar 17, 2020 at 09:29:32AM +0000, Peter Maydell wrote:
> > Ah, I see. My assumption was that this was all stuff that you were
> > working on, so that I would then be able to test that it worked correctly,
> > not that I would need to do configuration of the gitlab.com setup.

> So, I had to use temporary hardware resources to set this up (and set
> it up countless times TBH).  I had the understanding based on the list
> of machines you documented[1] that at least some of them would be used
> for the permanent setup.

Well, some of them will be (eg the s390 box), but some of them
are my personal ones that can't be reused easily. I'd assumed
in any case that gitlab would have at least support for x86 hosts:
we are definitely not going to continue to use my desktop machine
for running CI builds! Also IIRC RedHat said they'd be able to
provide some machines for runners.

> OK, I see it, now it makes more sense.  So we're "only" missing the
> setup for the machines we'll use for the more permanent setup.  Would
> you like to do a staged setup/migration using one or some of the
> machines you documented?  I'm 100% onboard to help with this, meaning
> that I can assist you with instructions, or do "pair setup" of the
> machines if needed.  I think a good part of the evaluation here comes
> down to how manageable/reproducible the setup is, so it'd make sense
> for one to be part of the setup itself.

I think we should start by getting the gitlab setup working
for the basic "x86 configs" first. Then we can try adding
a runner for s390 (that one's logistically easiest because
it is a project machine, not one owned by me personally or
by Linaro) once the basic framework is working, and expand
from there.

But to a large degree I really don't want to have to get
into the details of how gitlab works or setting up runners
myself if I can avoid it. We're going through this migration
because I want to be able to hand off the CI stuff to other
people, not to retain control of it.

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Tue, 17 Mar 2020 at 14:13, Cleber Rosa <crosa@redhat.com> wrote:
>>
>> On Tue, Mar 17, 2020 at 09:29:32AM +0000, Peter Maydell wrote:
>> > Ah, I see. My assumption was that this was all stuff that you were
>> > working on, so that I would then be able to test that it worked correctly,
>> > not that I would need to do configuration of the gitlab.com setup.
>
>> So, I had to use temporary hardware resources to set this up (and set
>> it up countless times TBH).  I had the understanding based on the list
>> of machines you documented[1] that at least some of them would be used
>> for the permanent setup.
>
> Well, some of them will be (eg the s390 box), but some of them
> are my personal ones that can't be reused easily. I'd assumed
> in any case that gitlab would have at least support for x86 hosts:
> we are definitely not going to continue to use my desktop machine
> for running CI builds! Also IIRC RedHat said they'd be able to
> provide some machines for runners.

Correct!  As discussed at the QEMU summit, we'll gladly chip in runners
to test the stuff we care about, but to match the coverage of your
private zoo of machines, others will have to chip in, too.

>> OK, I see it, now it makes more sense.  So we're "only" missing the
>> setup for the machines we'll use for the more permanent setup.  Would
>> you like to do a staged setup/migration using one or some of the
>> machines you documented?  I'm 100% onboard to help with this, meaning
>> that I can assist you with instructions, or do "pair setup" of the
>> machines if needed.  I think a good part of the evaluation here comes
>> down to how manageable/reproducible the setup is, so it'd make sense
>> for one to be part of the setup itself.
>
> I think we should start by getting the gitlab setup working
> for the basic "x86 configs" first. Then we can try adding
> a runner for s390 (that one's logistically easiest because
> it is a project machine, not one owned by me personally or
> by Linaro) once the basic framework is working, and expand
> from there.

Makes sense to me.

Next steps to get this off the ground:

* Red Hat provides runner(s) for x86 stuff we care about.

* If that doesn't cover 'basic "x86 configs" in your judgement, we
  fill the gaps as described below under "Expand from there".

* Add an s390 runner using the project machine you mentioned.

* Expand from there: identify the remaining gaps, map them to people /
  organizations interested in them, and solicit contributions from these
  guys.

A note on contributions: we need both hardware and people.  By people I
mean maintainers for the infrastructure, the tools and all the runners.
Cleber & team are willing to serve for the infrastructure, the tools and
the Red Hat runners.

Does this sound workable?

> But to a large degree I really don't want to have to get
> into the details of how gitlab works or setting up runners
> myself if I can avoid it. We're going through this migration
> because I want to be able to hand off the CI stuff to other
> people, not to retain control of it.

Understand.  We need contributions to gating CI, but the whole point of
this exercise is to make people other than *you* contribute to our
gating CI :)

Let me use this opportunity to say thank you for all your integration
work!

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 5280 bytes --]

On Thu, Mar 19, 2020 at 05:33:01PM +0100, Markus Armbruster wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On Tue, 17 Mar 2020 at 14:13, Cleber Rosa <crosa@redhat.com> wrote:
> >>
> >> On Tue, Mar 17, 2020 at 09:29:32AM +0000, Peter Maydell wrote:
> >> > Ah, I see. My assumption was that this was all stuff that you were
> >> > working on, so that I would then be able to test that it worked correctly,
> >> > not that I would need to do configuration of the gitlab.com setup.
> >
> >> So, I had to use temporary hardware resources to set this up (and set
> >> it up countless times TBH).  I had the understanding based on the list
> >> of machines you documented[1] that at least some of them would be used
> >> for the permanent setup.
> >
> > Well, some of them will be (eg the s390 box), but some of them
> > are my personal ones that can't be reused easily. I'd assumed
> > in any case that gitlab would have at least support for x86 hosts:
> > we are definitely not going to continue to use my desktop machine
> > for running CI builds! Also IIRC RedHat said they'd be able to
> > provide some machines for runners.

While GitLab let's you run x86 code for free with the "Linux Shared
Runners"[1], I don't think it would be suitable to what we're trying
to achieve.  It's limited to a single OS (CoreOS), single architecture
and really geared towards running containers.  BTW, if it isn't clear,
this is the approach being used today for the jobs defined on
".gitlab-ci.yml".

IMO we can leverage and still expand on the use of the "Linux Shared
Runners", but to really get a grasp oh how well this model can work
for QEMU, we'll need "Specific Runners", because we're validating
how/if we can depend on it for OS/architectures they don't support on
shared runners (and sometimes not even for the gitlab-runner agent).

> 
> Correct!  As discussed at the QEMU summit, we'll gladly chip in runners
> to test the stuff we care about, but to match the coverage of your
> private zoo of machines, others will have to chip in, too.
>

I'm sorry I missed the original discussions, and I'm even more sorry
if that led to any misunderstandings here.

> >> OK, I see it, now it makes more sense.  So we're "only" missing the
> >> setup for the machines we'll use for the more permanent setup.  Would
> >> you like to do a staged setup/migration using one or some of the
> >> machines you documented?  I'm 100% onboard to help with this, meaning
> >> that I can assist you with instructions, or do "pair setup" of the
> >> machines if needed.  I think a good part of the evaluation here comes
> >> down to how manageable/reproducible the setup is, so it'd make sense
> >> for one to be part of the setup itself.
> >
> > I think we should start by getting the gitlab setup working
> > for the basic "x86 configs" first. Then we can try adding
> > a runner for s390 (that one's logistically easiest because
> > it is a project machine, not one owned by me personally or
> > by Linaro) once the basic framework is working, and expand
> > from there.
> 
> Makes sense to me.
> 
> Next steps to get this off the ground:
> 
> * Red Hat provides runner(s) for x86 stuff we care about.
> 
> * If that doesn't cover 'basic "x86 configs" in your judgement, we
>   fill the gaps as described below under "Expand from there".
> 
> * Add an s390 runner using the project machine you mentioned.
> 
> * Expand from there: identify the remaining gaps, map them to people /
>   organizations interested in them, and solicit contributions from these
>   guys.
> 
> A note on contributions: we need both hardware and people.  By people I
> mean maintainers for the infrastructure, the tools and all the runners.
> Cleber & team are willing to serve for the infrastructure, the tools and
> the Red Hat runners.

Right, while we've tried to streamline the process of setting up the
machines, there will be plenty of changes to improve the automation.

More importantly, maintaining the machines is very important to the
super important goal of catching code regressions only, and not facing
other failures.  Mundane tasks such as making sure enough disk space
is always available can be completely change the perception of the
usefulness of a CI environment.  And for this maintenance, we need
help from people "owning" those machines.

> 
> Does this sound workable?
> 
> > But to a large degree I really don't want to have to get
> > into the details of how gitlab works or setting up runners
> > myself if I can avoid it. We're going through this migration
> > because I want to be able to hand off the CI stuff to other
> > people, not to retain control of it.
> 
> Understand.  We need contributions to gating CI, but the whole point of
> this exercise is to make people other than *you* contribute to our
> gating CI :)
> 
> Let me use this opportunity to say thank you for all your integration
> work!
> 
>

^ THIS.  I have to say that I'm still amazed as to how Peter has
managed to automate, integrate and run all those tests in such varied
environments for so long.  Major kudos!

- Cleber.

[1] https://docs.gitlab.com/ee/user/gitlab_com/#linux-shared-runners

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> > I think we should start by getting the gitlab setup working
> > for the basic "x86 configs" first. Then we can try adding
> > a runner for s390 (that one's logistically easiest because
> > it is a project machine, not one owned by me personally or
> > by Linaro) once the basic framework is working, and expand
> > from there.
>
> Makes sense to me.
>
> Next steps to get this off the ground:
>
> * Red Hat provides runner(s) for x86 stuff we care about.
>
> * If that doesn't cover 'basic "x86 configs" in your judgement, we
>   fill the gaps as described below under "Expand from there".
>
> * Add an s390 runner using the project machine you mentioned.
>
> * Expand from there: identify the remaining gaps, map them to people /
>   organizations interested in them, and solicit contributions from these
>   guys.
>
> A note on contributions: we need both hardware and people.  By people I
> mean maintainers for the infrastructure, the tools and all the runners.
> Cleber & team are willing to serve for the infrastructure, the tools and
> the Red Hat runners.

So, with 5.0 nearly out the door it seems like a good time to check
in on this thread again to ask where we are progress-wise with this.
My impression is that this patchset provides most of the scripting
and config side of the first step, so what we need is for RH to provide
an x86 runner machine and tell the gitlab CI it exists. I appreciate
that the whole coronavirus and working-from-home situation will have
upended everybody's plans, especially when actual hardware might
be involved, but how's it going ?

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Peter Maydell" <peter.maydell@linaro.org>
> To: "Markus Armbruster" <armbru@redhat.com>
> Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
> "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
> <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> Habkost" <ehabkost@redhat.com>
> Sent: Tuesday, April 21, 2020 8:53:49 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> > Peter Maydell <peter.maydell@linaro.org> writes:
> > > I think we should start by getting the gitlab setup working
> > > for the basic "x86 configs" first. Then we can try adding
> > > a runner for s390 (that one's logistically easiest because
> > > it is a project machine, not one owned by me personally or
> > > by Linaro) once the basic framework is working, and expand
> > > from there.
> >
> > Makes sense to me.
> >
> > Next steps to get this off the ground:
> >
> > * Red Hat provides runner(s) for x86 stuff we care about.
> >
> > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> >   fill the gaps as described below under "Expand from there".
> >
> > * Add an s390 runner using the project machine you mentioned.
> >
> > * Expand from there: identify the remaining gaps, map them to people /
> >   organizations interested in them, and solicit contributions from these
> >   guys.
> >
> > A note on contributions: we need both hardware and people.  By people I
> > mean maintainers for the infrastructure, the tools and all the runners.
> > Cleber & team are willing to serve for the infrastructure, the tools and
> > the Red Hat runners.
> 
> So, with 5.0 nearly out the door it seems like a good time to check
> in on this thread again to ask where we are progress-wise with this.
> My impression is that this patchset provides most of the scripting
> and config side of the first step, so what we need is for RH to provide
> an x86 runner machine and tell the gitlab CI it exists. I appreciate
> that the whole coronavirus and working-from-home situation will have
> upended everybody's plans, especially when actual hardware might
> be involved, but how's it going ?
> 

Hi Peter,

You hit the nail in the head here.  We were affected indeed with our ability
to move some machines from one lab to another (across the country), but we're
actively working on it.

From now on, I'll give you an update every time a significant event occurs on
our side.

> thanks
> -- PMM
> 
> 

Thanks for checking in!
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Daniel P. Berrangé]

On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Peter Maydell" <peter.maydell@linaro.org>
> > To: "Markus Armbruster" <armbru@redhat.com>
> > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> > Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
> > "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
> > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > Habkost" <ehabkost@redhat.com>
> > Sent: Tuesday, April 21, 2020 8:53:49 AM
> > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > 
> > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > I think we should start by getting the gitlab setup working
> > > > for the basic "x86 configs" first. Then we can try adding
> > > > a runner for s390 (that one's logistically easiest because
> > > > it is a project machine, not one owned by me personally or
> > > > by Linaro) once the basic framework is working, and expand
> > > > from there.
> > >
> > > Makes sense to me.
> > >
> > > Next steps to get this off the ground:
> > >
> > > * Red Hat provides runner(s) for x86 stuff we care about.
> > >
> > > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> > >   fill the gaps as described below under "Expand from there".
> > >
> > > * Add an s390 runner using the project machine you mentioned.
> > >
> > > * Expand from there: identify the remaining gaps, map them to people /
> > >   organizations interested in them, and solicit contributions from these
> > >   guys.
> > >
> > > A note on contributions: we need both hardware and people.  By people I
> > > mean maintainers for the infrastructure, the tools and all the runners.
> > > Cleber & team are willing to serve for the infrastructure, the tools and
> > > the Red Hat runners.
> > 
> > So, with 5.0 nearly out the door it seems like a good time to check
> > in on this thread again to ask where we are progress-wise with this.
> > My impression is that this patchset provides most of the scripting
> > and config side of the first step, so what we need is for RH to provide
> > an x86 runner machine and tell the gitlab CI it exists. I appreciate
> > that the whole coronavirus and working-from-home situation will have
> > upended everybody's plans, especially when actual hardware might
> > be involved, but how's it going ?
> > 
> 
> Hi Peter,
> 
> You hit the nail in the head here.  We were affected indeed with our ability
> to move some machines from one lab to another (across the country), but we're
> actively working on it.

For x86, do we really need to be using custom runners ?

With GitLab if someone forks the repo to their personal namespace, they
cannot use any custom runners setup by the origin project. So if we use
custom runners for x86, people forking won't be able to run the GitLab
CI jobs.

As a sub-system maintainer I wouldn't like this, because I ideally want
to be able to run the same jobs on my staging tree, that Peter will run
at merge time for the PULL request I send.

Thus my strong preference would be to use the GitLab runners in every
scenario where they are viable to use. Only use custom runners in the
cases where GitLab runners are clearly inadequate for our needs.

Based on what we've setup in GitLab for libvirt,  the shared runners
they have work fine for x86. Just need the environments you are testing
to be provided as Docker containers (you can actually build and cache
the container images during your CI job too).  IOW, any Linux distro
build and test jobs should be able to use shared runners on x86, and
likewise mingw builds. Custom runners should only be needed if the
jobs need todo *BSD / macOS builds, and/or have access to specific
hardware devices for some reason.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Peter Maydell" <peter.maydell@linaro.org>, "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>,
> "Beraldo Leal" <bleal@redhat.com>, "Erik Skultety" <eskultet@redhat.com>, "Philippe Mathieu-Daudé"
> <philmd@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>, "Markus Armbruster" <armbru@redhat.com>, "Wainer dos
> Santos Moschetta" <wainersm@redhat.com>, "QEMU Developers" <qemu-devel@nongnu.org>, "Willian Rampazzo"
> <wrampazz@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Eduardo Habkost" <ehabkost@redhat.com>
> Sent: Thursday, April 23, 2020 1:13:22 PM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Peter Maydell" <peter.maydell@linaro.org>
> > > To: "Markus Armbruster" <armbru@redhat.com>
> > > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>,
> > > "Beraldo Leal" <bleal@redhat.com>, "Erik
> > > Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>,
> > > "Wainer Moschetta" <wmoschet@redhat.com>,
> > > "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta"
> > > <wainersm@redhat.com>, "Willian Rampazzo"
> > > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe
> > > Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > > Habkost" <ehabkost@redhat.com>
> > > Sent: Tuesday, April 21, 2020 8:53:49 AM
> > > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > 
> > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com>
> > > wrote:
> > > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > > I think we should start by getting the gitlab setup working
> > > > > for the basic "x86 configs" first. Then we can try adding
> > > > > a runner for s390 (that one's logistically easiest because
> > > > > it is a project machine, not one owned by me personally or
> > > > > by Linaro) once the basic framework is working, and expand
> > > > > from there.
> > > >
> > > > Makes sense to me.
> > > >
> > > > Next steps to get this off the ground:
> > > >
> > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > >
> > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> > > >   fill the gaps as described below under "Expand from there".
> > > >
> > > > * Add an s390 runner using the project machine you mentioned.
> > > >
> > > > * Expand from there: identify the remaining gaps, map them to people /
> > > >   organizations interested in them, and solicit contributions from
> > > >   these
> > > >   guys.
> > > >
> > > > A note on contributions: we need both hardware and people.  By people I
> > > > mean maintainers for the infrastructure, the tools and all the runners.
> > > > Cleber & team are willing to serve for the infrastructure, the tools
> > > > and
> > > > the Red Hat runners.
> > > 
> > > So, with 5.0 nearly out the door it seems like a good time to check
> > > in on this thread again to ask where we are progress-wise with this.
> > > My impression is that this patchset provides most of the scripting
> > > and config side of the first step, so what we need is for RH to provide
> > > an x86 runner machine and tell the gitlab CI it exists. I appreciate
> > > that the whole coronavirus and working-from-home situation will have
> > > upended everybody's plans, especially when actual hardware might
> > > be involved, but how's it going ?
> > > 
> > 
> > Hi Peter,
> > 
> > You hit the nail in the head here.  We were affected indeed with our
> > ability
> > to move some machines from one lab to another (across the country), but
> > we're
> > actively working on it.
> 
> For x86, do we really need to be using custom runners ?
> 

Hi Daniel,

We're already using the shared x86 runners, but with a different goal.  The
goal of the "Gating CI" is indeed to expand on non-x86 environments.  We're
in a "chicken and egg" kind of situation, because we'd like to prove that
GitLab CI will allow QEMU to expand to very different runners and jobs, while
not really having all that hardware setup and publicly available at this time.

My experiments were really around that point, I mean, confirming that we can grow
the number of architectures/runners/jobs/configurations to provide a coverage
equal or greater to what Peter already does.

> With GitLab if someone forks the repo to their personal namespace, they
> cannot use any custom runners setup by the origin project. So if we use
> custom runners for x86, people forking won't be able to run the GitLab
> CI jobs.
> 

They will continue to be able use the jobs and runners already defined in
the .gitlab-ci.yml file.  This work will only affect people pushing to the/a
"staging" branch.

> As a sub-system maintainer I wouldn't like this, because I ideally want
> to be able to run the same jobs on my staging tree, that Peter will run
> at merge time for the PULL request I send.
> 

If you're looking for symmetry between any PR and "merge time" jobs, the
only solution is to allow any PR to access all the diverse set of non-shared
machines we're hoping to have in the near future.  This may be something
we'll get to, but I doubt we can tackle it in the near future now.

> Thus my strong preference would be to use the GitLab runners in every
> scenario where they are viable to use. Only use custom runners in the
> cases where GitLab runners are clearly inadequate for our needs.
> 
> Based on what we've setup in GitLab for libvirt,  the shared runners
> they have work fine for x86. Just need the environments you are testing
> to be provided as Docker containers (you can actually build and cache
> the container images during your CI job too).  IOW, any Linux distro
> build and test jobs should be able to use shared runners on x86, and
> likewise mingw builds. Custom runners should only be needed if the
> jobs need todo *BSD / macOS builds, and/or have access to specific
> hardware devices for some reason.
> 

We've discussed this before at the RFC time, wrt how the goal is for a wider
community to provide a wider range of jobs.  Even for x86, one may want
to require their jobs to run on a given accelerator, such as KVM, so we
need to consider that from the very beginning.

I don't see a problem with converging jobs with are being run on custom
runners back into shared runners as much as possible.  At the RFC discussion,
I actually pointed out how the build phase could be running essentially
on pre-built containers (on shared runners), but the test phase, say testing
KVM, should not be bound to that. 

So in essence, right now, moving everything to containers would invalidate the
exercise of being able to care for those custom architectures/builds/jobs we'll
need in the near future.  And that's really the whole point here.

Cheers,
- Cleber.

> 
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
> |:|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com
> |:|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
> |:|
> 

Re: [PATCH 0/5] QEMU Gating CI

[Peter Maydell]

On Thu, 23 Apr 2020 at 18:37, Cleber Rosa <crosa@redhat.com> wrote:
> We're already using the shared x86 runners, but with a different goal.  The
> goal of the "Gating CI" is indeed to expand on non-x86 environments.  We're
> in a "chicken and egg" kind of situation, because we'd like to prove that
> GitLab CI will allow QEMU to expand to very different runners and jobs, while
> not really having all that hardware setup and publicly available at this time.

We do have the S390 machine that IBM kindly made available to
the project -- that is not a personal or Linaro machine, so
there are no issues with giving you a login on that so you
can set it up as a CI runner. Drop me an email if you want
access to it.

thanks
-- PMM

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Thu, 23 Apr 2020 18:50:47 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> On Thu, 23 Apr 2020 at 18:37, Cleber Rosa <crosa@redhat.com> wrote:
> > We're already using the shared x86 runners, but with a different
> > goal.  The goal of the "Gating CI" is indeed to expand on non-x86
> > environments.  We're in a "chicken and egg" kind of situation,
> > because we'd like to prove that GitLab CI will allow QEMU to expand
> > to very different runners and jobs, while not really having all
> > that hardware setup and publicly available at this time.
> 
> We do have the S390 machine that IBM kindly made available to
> the project -- that is not a personal or Linaro machine, so
> there are no issues with giving you a login on that so you
> can set it up as a CI runner. Drop me an email if you want
> access to it.
> 

I sure do.  I'll shoot you an email off-list.

> thanks
> -- PMM
> 

Thanks,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Daniel P. Berrangé]

On Thu, Apr 23, 2020 at 01:36:48PM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Daniel P. Berrangé" <berrange@redhat.com>
> > To: "Cleber Rosa" <crosa@redhat.com>
> > Cc: "Peter Maydell" <peter.maydell@linaro.org>, "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>,
> > "Beraldo Leal" <bleal@redhat.com>, "Erik Skultety" <eskultet@redhat.com>, "Philippe Mathieu-Daudé"
> > <philmd@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>, "Markus Armbruster" <armbru@redhat.com>, "Wainer dos
> > Santos Moschetta" <wainersm@redhat.com>, "QEMU Developers" <qemu-devel@nongnu.org>, "Willian Rampazzo"
> > <wrampazz@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Eduardo Habkost" <ehabkost@redhat.com>
> > Sent: Thursday, April 23, 2020 1:13:22 PM
> > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > 
> > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Peter Maydell" <peter.maydell@linaro.org>
> > > > To: "Markus Armbruster" <armbru@redhat.com>
> > > > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>,
> > > > "Beraldo Leal" <bleal@redhat.com>, "Erik
> > > > Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>,
> > > > "Wainer Moschetta" <wmoschet@redhat.com>,
> > > > "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta"
> > > > <wainersm@redhat.com>, "Willian Rampazzo"
> > > > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe
> > > > Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > > > Habkost" <ehabkost@redhat.com>
> > > > Sent: Tuesday, April 21, 2020 8:53:49 AM
> > > > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > > 
> > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com>
> > > > wrote:
> > > > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > > > I think we should start by getting the gitlab setup working
> > > > > > for the basic "x86 configs" first. Then we can try adding
> > > > > > a runner for s390 (that one's logistically easiest because
> > > > > > it is a project machine, not one owned by me personally or
> > > > > > by Linaro) once the basic framework is working, and expand
> > > > > > from there.
> > > > >
> > > > > Makes sense to me.
> > > > >
> > > > > Next steps to get this off the ground:
> > > > >
> > > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > > >
> > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> > > > >   fill the gaps as described below under "Expand from there".
> > > > >
> > > > > * Add an s390 runner using the project machine you mentioned.
> > > > >
> > > > > * Expand from there: identify the remaining gaps, map them to people /
> > > > >   organizations interested in them, and solicit contributions from
> > > > >   these
> > > > >   guys.
> > > > >
> > > > > A note on contributions: we need both hardware and people.  By people I
> > > > > mean maintainers for the infrastructure, the tools and all the runners.
> > > > > Cleber & team are willing to serve for the infrastructure, the tools
> > > > > and
> > > > > the Red Hat runners.
> > > > 
> > > > So, with 5.0 nearly out the door it seems like a good time to check
> > > > in on this thread again to ask where we are progress-wise with this.
> > > > My impression is that this patchset provides most of the scripting
> > > > and config side of the first step, so what we need is for RH to provide
> > > > an x86 runner machine and tell the gitlab CI it exists. I appreciate
> > > > that the whole coronavirus and working-from-home situation will have
> > > > upended everybody's plans, especially when actual hardware might
> > > > be involved, but how's it going ?
> > > > 
> > > 
> > > Hi Peter,
> > > 
> > > You hit the nail in the head here.  We were affected indeed with our
> > > ability
> > > to move some machines from one lab to another (across the country), but
> > > we're
> > > actively working on it.
> > 
> > For x86, do we really need to be using custom runners ?
> > 
> 
> Hi Daniel,
> 
> We're already using the shared x86 runners, but with a different goal.  The
> goal of the "Gating CI" is indeed to expand on non-x86 environments.  We're
> in a "chicken and egg" kind of situation, because we'd like to prove that
> GitLab CI will allow QEMU to expand to very different runners and jobs, while
> not really having all that hardware setup and publicly available at this time.
> 
> My experiments were really around that point, I mean, confirming that we can grow
> the number of architectures/runners/jobs/configurations to provide a coverage
> equal or greater to what Peter already does.

So IIUC, you're saying that for x86 gating, the intention is to use shared
runners in general.

Your current work that you say is blocked on access to x86 hardware, is just
about demonstrating the concept of plugging in custom runners, while we wait
for access to non-x86 hardware ?

> > With GitLab if someone forks the repo to their personal namespace, they
> > cannot use any custom runners setup by the origin project. So if we use
> > custom runners for x86, people forking won't be able to run the GitLab
> > CI jobs.
> > 
> 
> They will continue to be able use the jobs and runners already defined in
> the .gitlab-ci.yml file.  This work will only affect people pushing to the/a
> "staging" branch.
> 
> > As a sub-system maintainer I wouldn't like this, because I ideally want
> > to be able to run the same jobs on my staging tree, that Peter will run
> > at merge time for the PULL request I send.
> > 
> 
> If you're looking for symmetry between any PR and "merge time" jobs, the
> only solution is to allow any PR to access all the diverse set of non-shared
> machines we're hoping to have in the near future.  This may be something
> we'll get to, but I doubt we can tackle it in the near future now.

It occurred to me that we could do this if we grant selective access to
the Gitlab repos, to people who are official subsystem maintainers.
GitLab has a concept of "protected branches", so you can control who is
allowed to push changes on a per-branch granularity.

So, for example, in the main qemu.git, we could create branches for each
subsystem tree eg

  staging-block
  staging-qapi
  staging-crypto
  staging-migration
  ....

and for each of these branches, we can grant access to relevant subsystem
maintainer(s).

When they're ready to send a pull request to Peter, they can push their
tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu
project namespace, this branch can run CI using the private QEMU runners.
The subsystem maintainer can thus see the full set of CI results across
all platforms required by Gating, before Peter even gets the pull request.

So when Peter then looks at merging the pull request to master, the only
he's likely to see are the non-deterministic bugs, or issues caused by
semantic conflicts with other recently merged code.

It would even be possible to do the final merge into master entirely from
GitLab, no need to go via email. When the source branch & target branch are
within the same git repo, GitLab has the ability to run CI jobs against the
resulting merge commit in a strict gating manner, before it hits master.
They call this "Merge trains" in their documentation.

IOW, from Peter's POV, merging pull requests could be as simple as hitting
the merge button in GitLab merge request UI. Everything wrt CI would be
completely automated, and the subsystem maintainers would have the
responsibility to dealing with merge conflicts & CI failures, which is
more scalable for the person co-ordinating the merges into master.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/5] QEMU Gating CI

[Philippe Mathieu-Daudé]

On Fri, Apr 24, 2020 at 11:30 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > ----- Original Message -----
> > > From: "Daniel P. Berrangé" <berrange@redhat.com>
> > > To: "Cleber Rosa" <crosa@redhat.com>
[...]
> > Hi Daniel,
> >
> > We're already using the shared x86 runners, but with a different goal.  The
> > goal of the "Gating CI" is indeed to expand on non-x86 environments.  We're
> > in a "chicken and egg" kind of situation, because we'd like to prove that
> > GitLab CI will allow QEMU to expand to very different runners and jobs, while
> > not really having all that hardware setup and publicly available at this time.
> >
> > My experiments were really around that point, I mean, confirming that we can grow
> > the number of architectures/runners/jobs/configurations to provide a coverage
> > equal or greater to what Peter already does.
>
> So IIUC, you're saying that for x86 gating, the intention is to use shared
> runners in general.
>
> Your current work that you say is blocked on access to x86 hardware, is just
> about demonstrating the concept of plugging in custom runners, while we wait
> for access to non-x86 hardware ?
>
> > > With GitLab if someone forks the repo to their personal namespace, they
> > > cannot use any custom runners setup by the origin project. So if we use
> > > custom runners for x86, people forking won't be able to run the GitLab
> > > CI jobs.
> > >
> >
> > They will continue to be able use the jobs and runners already defined in
> > the .gitlab-ci.yml file.  This work will only affect people pushing to the/a
> > "staging" branch.
> >
> > > As a sub-system maintainer I wouldn't like this, because I ideally want
> > > to be able to run the same jobs on my staging tree, that Peter will run
> > > at merge time for the PULL request I send.
> > >
> >
> > If you're looking for symmetry between any PR and "merge time" jobs, the
> > only solution is to allow any PR to access all the diverse set of non-shared
> > machines we're hoping to have in the near future.  This may be something
> > we'll get to, but I doubt we can tackle it in the near future now.
>
> It occurred to me that we could do this if we grant selective access to
> the Gitlab repos, to people who are official subsystem maintainers.
> GitLab has a concept of "protected branches", so you can control who is
> allowed to push changes on a per-branch granularity.
>
> So, for example, in the main qemu.git, we could create branches for each
> subsystem tree eg
>
>   staging-block
>   staging-qapi
>   staging-crypto
>   staging-migration
>   ....
>
> and for each of these branches, we can grant access to relevant subsystem
> maintainer(s).

The MAINTAINERS file could help us with that, we already have scripts
to parse its sections.
Maintainers should keep it up-to-date, then the merge script would check, i.e.:

<newline> // section separator
--------------- // ignored
Trivial patches // description ignored
M: Michael Tokarev <mjt@tls.msk.ru>
M: Laurent Vivier <laurent@vivier.eu> // must match commit author
T: git git://git.corpit.ru/qemu.git trivial-patches
T: git https://github.com/vivier/qemu.git trivial-patches // must
match MR source

>
> When they're ready to send a pull request to Peter, they can push their
> tree to this branch. Since the branch is in the main gitlab.com/qemu/qemu
> project namespace, this branch can run CI using the private QEMU runners.
> The subsystem maintainer can thus see the full set of CI results across
> all platforms required by Gating, before Peter even gets the pull request.
>
> So when Peter then looks at merging the pull request to master, the only
> he's likely to see are the non-deterministic bugs, or issues caused by
> semantic conflicts with other recently merged code.
>
> It would even be possible to do the final merge into master entirely from
> GitLab, no need to go via email. When the source branch & target branch are
> within the same git repo, GitLab has the ability to run CI jobs against the
> resulting merge commit in a strict gating manner, before it hits master.
> They call this "Merge trains" in their documentation.
>
> IOW, from Peter's POV, merging pull requests could be as simple as hitting
> the merge button in GitLab merge request UI. Everything wrt CI would be
> completely automated, and the subsystem maintainers would have the
> responsibility to dealing with merge conflicts & CI failures, which is
> more scalable for the person co-ordinating the merges into master.
>
>
> Regards,
> Daniel

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Fri, 24 Apr 2020 10:30:23 +0100
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Thu, Apr 23, 2020 at 01:36:48PM -0400, Cleber Rosa wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Daniel P. Berrangé" <berrange@redhat.com>
> > > To: "Cleber Rosa" <crosa@redhat.com>
> > > Cc: "Peter Maydell" <peter.maydell@linaro.org>, "Fam Zheng"
> > > <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo
> > > Leal" <bleal@redhat.com>, "Erik Skultety" <eskultet@redhat.com>,
> > > "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Wainer Moschetta"
> > > <wmoschet@redhat.com>, "Markus Armbruster" <armbru@redhat.com>,
> > > "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "QEMU
> > > Developers" <qemu-devel@nongnu.org>, "Willian Rampazzo"
> > > <wrampazz@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>,
> > > "Eduardo Habkost" <ehabkost@redhat.com> Sent: Thursday, April 23,
> > > 2020 1:13:22 PM Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > 
> > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Peter Maydell" <peter.maydell@linaro.org>
> > > > > To: "Markus Armbruster" <armbru@redhat.com>
> > > > > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth"
> > > > > <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> > > > > Skultety" <eskultet@redhat.com>, "Alex Bennée"
> > > > > <alex.bennee@linaro.org>, "Wainer Moschetta"
> > > > > <wmoschet@redhat.com>, "QEMU Developers"
> > > > > <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta"
> > > > > <wainersm@redhat.com>, "Willian Rampazzo"
> > > > > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > > > "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > > > > Habkost" <ehabkost@redhat.com> Sent: Tuesday, April 21, 2020
> > > > > 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > > > 
> > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster
> > > > > <armbru@redhat.com> wrote:
> > > > > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > > > > I think we should start by getting the gitlab setup
> > > > > > > working for the basic "x86 configs" first. Then we can
> > > > > > > try adding a runner for s390 (that one's logistically
> > > > > > > easiest because it is a project machine, not one owned by
> > > > > > > me personally or by Linaro) once the basic framework is
> > > > > > > working, and expand from there.
> > > > > >
> > > > > > Makes sense to me.
> > > > > >
> > > > > > Next steps to get this off the ground:
> > > > > >
> > > > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > > > >
> > > > > > * If that doesn't cover 'basic "x86 configs" in your
> > > > > > judgement, we fill the gaps as described below under
> > > > > > "Expand from there".
> > > > > >
> > > > > > * Add an s390 runner using the project machine you
> > > > > > mentioned.
> > > > > >
> > > > > > * Expand from there: identify the remaining gaps, map them
> > > > > > to people / organizations interested in them, and solicit
> > > > > > contributions from these
> > > > > >   guys.
> > > > > >
> > > > > > A note on contributions: we need both hardware and people.
> > > > > > By people I mean maintainers for the infrastructure, the
> > > > > > tools and all the runners. Cleber & team are willing to
> > > > > > serve for the infrastructure, the tools and
> > > > > > the Red Hat runners.
> > > > > 
> > > > > So, with 5.0 nearly out the door it seems like a good time to
> > > > > check in on this thread again to ask where we are
> > > > > progress-wise with this. My impression is that this patchset
> > > > > provides most of the scripting and config side of the first
> > > > > step, so what we need is for RH to provide an x86 runner
> > > > > machine and tell the gitlab CI it exists. I appreciate that
> > > > > the whole coronavirus and working-from-home situation will
> > > > > have upended everybody's plans, especially when actual
> > > > > hardware might be involved, but how's it going ?
> > > > > 
> > > > 
> > > > Hi Peter,
> > > > 
> > > > You hit the nail in the head here.  We were affected indeed
> > > > with our ability
> > > > to move some machines from one lab to another (across the
> > > > country), but we're
> > > > actively working on it.
> > > 
> > > For x86, do we really need to be using custom runners ?
> > > 
> > 
> > Hi Daniel,
> > 
> > We're already using the shared x86 runners, but with a different
> > goal.  The goal of the "Gating CI" is indeed to expand on non-x86
> > environments.  We're in a "chicken and egg" kind of situation,
> > because we'd like to prove that GitLab CI will allow QEMU to expand
> > to very different runners and jobs, while not really having all
> > that hardware setup and publicly available at this time.
> > 
> > My experiments were really around that point, I mean, confirming
> > that we can grow the number of
> > architectures/runners/jobs/configurations to provide a coverage
> > equal or greater to what Peter already does.
> 
> So IIUC, you're saying that for x86 gating, the intention is to use
> shared runners in general.
> 

No, I've said that whenever possible we could use containers and thus
shared runners.  For instance, testing QEMU running on the x86 CentOS 8
KVM is not something we could do with shared runners. 

> Your current work that you say is blocked on access to x86 hardware,
> is just about demonstrating the concept of plugging in custom
> runners, while we wait for access to non-x86 hardware ?
> 

Short answer is no.  The original scope and goal was to have the same or
very similar jobs that Peter runs himself in his own machines.  So it
was/is not about just x86 hardware, but x86 that can run a couple of
different OSs, and non-x86 hardware too.  We're basically scaling down
and changing the scope (for instance adding the s390 machine here) in
an attempt to get this moving forward.

> > > With GitLab if someone forks the repo to their personal
> > > namespace, they cannot use any custom runners setup by the origin
> > > project. So if we use custom runners for x86, people forking
> > > won't be able to run the GitLab CI jobs.
> > > 
> > 
> > They will continue to be able use the jobs and runners already
> > defined in the .gitlab-ci.yml file.  This work will only affect
> > people pushing to the/a "staging" branch.
> > 
> > > As a sub-system maintainer I wouldn't like this, because I
> > > ideally want to be able to run the same jobs on my staging tree,
> > > that Peter will run at merge time for the PULL request I send.
> > > 
> > 
> > If you're looking for symmetry between any PR and "merge time"
> > jobs, the only solution is to allow any PR to access all the
> > diverse set of non-shared machines we're hoping to have in the near
> > future.  This may be something we'll get to, but I doubt we can
> > tackle it in the near future now.
> 
> It occurred to me that we could do this if we grant selective access
> to the Gitlab repos, to people who are official subsystem maintainers.
> GitLab has a concept of "protected branches", so you can control who
> is allowed to push changes on a per-branch granularity.
> 
> So, for example, in the main qemu.git, we could create branches for
> each subsystem tree eg
> 
>   staging-block
>   staging-qapi
>   staging-crypto
>   staging-migration
>   ....
> 
> and for each of these branches, we can grant access to relevant
> subsystem maintainer(s).
> 
> When they're ready to send a pull request to Peter, they can push
> their tree to this branch. Since the branch is in the main
> gitlab.com/qemu/qemu project namespace, this branch can run CI using
> the private QEMU runners. The subsystem maintainer can thus see the
> full set of CI results across all platforms required by Gating,
> before Peter even gets the pull request.
> 

Sure, this is actually an extrapolation/extension of what we're
proposing to do here with the unique "staging" branch.  I see no issues
at all to have more than one (one per subsystem/maintainer) staging
branches.

> So when Peter then looks at merging the pull request to master, the
> only he's likely to see are the non-deterministic bugs, or issues
> caused by semantic conflicts with other recently merged code.
> 
> It would even be possible to do the final merge into master entirely
> from GitLab, no need to go via email. When the source branch & target
> branch are within the same git repo, GitLab has the ability to run CI
> jobs against the resulting merge commit in a strict gating manner,
> before it hits master. They call this "Merge trains" in their
> documentation.
> 
> IOW, from Peter's POV, merging pull requests could be as simple as
> hitting the merge button in GitLab merge request UI. Everything wrt
> CI would be completely automated, and the subsystem maintainers would
> have the responsibility to dealing with merge conflicts & CI
> failures, which is more scalable for the person co-ordinating the
> merges into master.
> 

This is very much aligned to some previous discussions, I believe, at
the RFC thread.  But for practical purposes, the general direction was
to scale down to the bare minimum to replicate Peter's setup and
workflow, and then move from there to possibly something very similar
to what you're describing here.

> 
> Regards,
> Daniel


Thanks a *whole lot* for the feedback Daniel!
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Philippe Mathieu-Daudé]

On 4/23/20 7:13 PM, Daniel P. Berrangé wrote:
> On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
>> ----- Original Message -----
>>> From: "Peter Maydell" <peter.maydell@linaro.org>
>>> To: "Markus Armbruster" <armbru@redhat.com>
>>> Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
>>> Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
>>> "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
>>> <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
>>> Habkost" <ehabkost@redhat.com>
>>> Sent: Tuesday, April 21, 2020 8:53:49 AM
>>> Subject: Re: [PATCH 0/5] QEMU Gating CI
>>>
>>> On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
>>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>>> I think we should start by getting the gitlab setup working
>>>>> for the basic "x86 configs" first. Then we can try adding
>>>>> a runner for s390 (that one's logistically easiest because
>>>>> it is a project machine, not one owned by me personally or
>>>>> by Linaro) once the basic framework is working, and expand
>>>>> from there.
>>>>
>>>> Makes sense to me.
>>>>
>>>> Next steps to get this off the ground:
>>>>
>>>> * Red Hat provides runner(s) for x86 stuff we care about.
>>>>
>>>> * If that doesn't cover 'basic "x86 configs" in your judgement, we
>>>>    fill the gaps as described below under "Expand from there".
>>>>
>>>> * Add an s390 runner using the project machine you mentioned.
>>>>
>>>> * Expand from there: identify the remaining gaps, map them to people /
>>>>    organizations interested in them, and solicit contributions from these
>>>>    guys.
>>>>
>>>> A note on contributions: we need both hardware and people.  By people I
>>>> mean maintainers for the infrastructure, the tools and all the runners.
>>>> Cleber & team are willing to serve for the infrastructure, the tools and
>>>> the Red Hat runners.
>>>
>>> So, with 5.0 nearly out the door it seems like a good time to check
>>> in on this thread again to ask where we are progress-wise with this.
>>> My impression is that this patchset provides most of the scripting
>>> and config side of the first step, so what we need is for RH to provide
>>> an x86 runner machine and tell the gitlab CI it exists. I appreciate
>>> that the whole coronavirus and working-from-home situation will have
>>> upended everybody's plans, especially when actual hardware might
>>> be involved, but how's it going ?
>>>
>>
>> Hi Peter,
>>
>> You hit the nail in the head here.  We were affected indeed with our ability
>> to move some machines from one lab to another (across the country), but we're
>> actively working on it.
> 
> For x86, do we really need to be using custom runners ?
> 
> With GitLab if someone forks the repo to their personal namespace, they
> cannot use any custom runners setup by the origin project. So if we use
> custom runners for x86, people forking won't be able to run the GitLab
> CI jobs.
> 
> As a sub-system maintainer I wouldn't like this, because I ideally want
> to be able to run the same jobs on my staging tree, that Peter will run
> at merge time for the PULL request I send.
> 
> Thus my strong preference would be to use the GitLab runners in every
> scenario where they are viable to use. Only use custom runners in the
> cases where GitLab runners are clearly inadequate for our needs.
> 
> Based on what we've setup in GitLab for libvirt,  the shared runners
> they have work fine for x86. Just need the environments you are testing
> to be provided as Docker containers (you can actually build and cache
> the container images during your CI job too).  IOW, any Linux distro
> build and test jobs should be able to use shared runners on x86, and
> likewise mingw builds. Custom runners should only be needed if the
> jobs need todo *BSD / macOS builds, and/or have access to specific
> hardware devices for some reason.

Thanks to insist with that point Daniel. I'd rather see every 
configuration reproducible, so if we loose a hardware sponsor, we can 
find another one and start another runner.
Also note, if it is not easy to reproduce a runner, it will be very hard 
to debug a reported build/test error.

A non-reproducible runner can not be used as gating, because if they 
fail it is not acceptable to lock the project development process.


In some cases custom runners are acceptable. These runners won't be 
"gating" but can post informative log and status.

[*] Specific hardware that is not easily available.

- Alistair at last KVM forum talked about a RISCV board
   (to test host TCG)
- Aleksandar said at last KVM forum Wavecomp could plug a CI20 MIPS
   (to test host TCG)
- Lemote seems interested to setup some Loongson MIPSr6 board
   (to test interaction with KVM)

[*] To run code requiring accepting License Agreements

[*] To run non Free / Open Source code


Owner of these runners take the responsibility to provide enough 
time/information about reported bugs, or to debug them themselves.


Now the problem is GitLab runner is not natively available on the 
architectures listed in this mail, so custom setup is required. A dumb 
script running ssh to a machine also works (tested) but lot of manual 
tuning/maintenance expected.

Re: [PATCH 0/5] QEMU Gating CI

[Erik Skultety]

On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé wrote:
> On 4/23/20 7:13 PM, Daniel P. Berrangé wrote:
> > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > > ----- Original Message -----
> > > > From: "Peter Maydell" <peter.maydell@linaro.org>
> > > > To: "Markus Armbruster" <armbru@redhat.com>
> > > > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> > > > Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
> > > > "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
> > > > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > > > Habkost" <ehabkost@redhat.com>
> > > > Sent: Tuesday, April 21, 2020 8:53:49 AM
> > > > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > > 
> > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> > > > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > > > I think we should start by getting the gitlab setup working
> > > > > > for the basic "x86 configs" first. Then we can try adding
> > > > > > a runner for s390 (that one's logistically easiest because
> > > > > > it is a project machine, not one owned by me personally or
> > > > > > by Linaro) once the basic framework is working, and expand
> > > > > > from there.
> > > > > 
> > > > > Makes sense to me.
> > > > > 
> > > > > Next steps to get this off the ground:
> > > > > 
> > > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > > > 
> > > > > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> > > > >    fill the gaps as described below under "Expand from there".
> > > > > 
> > > > > * Add an s390 runner using the project machine you mentioned.
> > > > > 
> > > > > * Expand from there: identify the remaining gaps, map them to people /
> > > > >    organizations interested in them, and solicit contributions from these
> > > > >    guys.
> > > > > 
> > > > > A note on contributions: we need both hardware and people.  By people I
> > > > > mean maintainers for the infrastructure, the tools and all the runners.
> > > > > Cleber & team are willing to serve for the infrastructure, the tools and
> > > > > the Red Hat runners.
> > > > 
> > > > So, with 5.0 nearly out the door it seems like a good time to check
> > > > in on this thread again to ask where we are progress-wise with this.
> > > > My impression is that this patchset provides most of the scripting
> > > > and config side of the first step, so what we need is for RH to provide
> > > > an x86 runner machine and tell the gitlab CI it exists. I appreciate
> > > > that the whole coronavirus and working-from-home situation will have
> > > > upended everybody's plans, especially when actual hardware might
> > > > be involved, but how's it going ?
> > > > 
> > > 
> > > Hi Peter,
> > > 
> > > You hit the nail in the head here.  We were affected indeed with our ability
> > > to move some machines from one lab to another (across the country), but we're
> > > actively working on it.
> > 
> > For x86, do we really need to be using custom runners ?
> > 
> > With GitLab if someone forks the repo to their personal namespace, they
> > cannot use any custom runners setup by the origin project. So if we use
> > custom runners for x86, people forking won't be able to run the GitLab
> > CI jobs.
> > 
> > As a sub-system maintainer I wouldn't like this, because I ideally want
> > to be able to run the same jobs on my staging tree, that Peter will run
> > at merge time for the PULL request I send.
> > 
> > Thus my strong preference would be to use the GitLab runners in every
> > scenario where they are viable to use. Only use custom runners in the
> > cases where GitLab runners are clearly inadequate for our needs.
> > 
> > Based on what we've setup in GitLab for libvirt,  the shared runners
> > they have work fine for x86. Just need the environments you are testing
> > to be provided as Docker containers (you can actually build and cache
> > the container images during your CI job too).  IOW, any Linux distro
> > build and test jobs should be able to use shared runners on x86, and
> > likewise mingw builds. Custom runners should only be needed if the
> > jobs need todo *BSD / macOS builds, and/or have access to specific
> > hardware devices for some reason.

Not just ^that, you also want custom VM runners to run integration tests, e.g.
in libvirt, we'd have to put systemd and a lof of other cruft into the
container to be able to run the tests at which point you must ask yourself,
whyt not go with a VM instead in which case we're limited in terms of
infrastructure...

> 
> Thanks to insist with that point Daniel. I'd rather see every configuration
> reproducible, so if we loose a hardware sponsor, we can find another one and
> start another runner.
> Also note, if it is not easy to reproduce a runner, it will be very hard to
> debug a reported build/test error.

(Thanks for bringing ^this point up Philippe)

...However, what we've been actively working on in libvirt is to extend the
lcitool we have (which can spawn local test VMs) to the point where we're able
to generate machines that would be the reproducible. Right now I'm playing with
cloud-init integration with lcitool (patches coming soon) that would allow us
to use the same machines locally as we'd want to have in, say, OpenStack and
share them as compressed images, so even when updated/managed by lcitool
locally, you'd get the same environment.

Regards,

-- 
Erik Skultety

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Fri, 24 Apr 2020 08:57:46 +0200
Erik Skultety <eskultet@redhat.com> wrote:

> On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé
> wrote:
> > On 4/23/20 7:13 PM, Daniel P. Berrangé wrote:
> > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > > > ----- Original Message -----
> > > > > From: "Peter Maydell" <peter.maydell@linaro.org>
> > > > > To: "Markus Armbruster" <armbru@redhat.com>
> > > > > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth"
> > > > > <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> > > > > Skultety" <eskultet@redhat.com>, "Alex Bennée"
> > > > > <alex.bennee@linaro.org>, "Wainer Moschetta"
> > > > > <wmoschet@redhat.com>, "QEMU Developers"
> > > > > <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta"
> > > > > <wainersm@redhat.com>, "Willian Rampazzo"
> > > > > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > > > "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > > > > Habkost" <ehabkost@redhat.com> Sent: Tuesday, April 21, 2020
> > > > > 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > > > 
> > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster
> > > > > <armbru@redhat.com> wrote:
> > > > > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > > > > I think we should start by getting the gitlab setup
> > > > > > > working for the basic "x86 configs" first. Then we can
> > > > > > > try adding a runner for s390 (that one's logistically
> > > > > > > easiest because it is a project machine, not one owned by
> > > > > > > me personally or by Linaro) once the basic framework is
> > > > > > > working, and expand from there.
> > > > > > 
> > > > > > Makes sense to me.
> > > > > > 
> > > > > > Next steps to get this off the ground:
> > > > > > 
> > > > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > > > > 
> > > > > > * If that doesn't cover 'basic "x86 configs" in your
> > > > > > judgement, we fill the gaps as described below under
> > > > > > "Expand from there".
> > > > > > 
> > > > > > * Add an s390 runner using the project machine you
> > > > > > mentioned.
> > > > > > 
> > > > > > * Expand from there: identify the remaining gaps, map them
> > > > > > to people / organizations interested in them, and solicit
> > > > > > contributions from these guys.
> > > > > > 
> > > > > > A note on contributions: we need both hardware and people.
> > > > > > By people I mean maintainers for the infrastructure, the
> > > > > > tools and all the runners. Cleber & team are willing to
> > > > > > serve for the infrastructure, the tools and the Red Hat
> > > > > > runners.
> > > > > 
> > > > > So, with 5.0 nearly out the door it seems like a good time to
> > > > > check in on this thread again to ask where we are
> > > > > progress-wise with this. My impression is that this patchset
> > > > > provides most of the scripting and config side of the first
> > > > > step, so what we need is for RH to provide an x86 runner
> > > > > machine and tell the gitlab CI it exists. I appreciate that
> > > > > the whole coronavirus and working-from-home situation will
> > > > > have upended everybody's plans, especially when actual
> > > > > hardware might be involved, but how's it going ?
> > > > > 
> > > > 
> > > > Hi Peter,
> > > > 
> > > > You hit the nail in the head here.  We were affected indeed
> > > > with our ability to move some machines from one lab to another
> > > > (across the country), but we're actively working on it.
> > > 
> > > For x86, do we really need to be using custom runners ?
> > > 
> > > With GitLab if someone forks the repo to their personal
> > > namespace, they cannot use any custom runners setup by the origin
> > > project. So if we use custom runners for x86, people forking
> > > won't be able to run the GitLab CI jobs.
> > > 
> > > As a sub-system maintainer I wouldn't like this, because I
> > > ideally want to be able to run the same jobs on my staging tree,
> > > that Peter will run at merge time for the PULL request I send.
> > > 
> > > Thus my strong preference would be to use the GitLab runners in
> > > every scenario where they are viable to use. Only use custom
> > > runners in the cases where GitLab runners are clearly inadequate
> > > for our needs.
> > > 
> > > Based on what we've setup in GitLab for libvirt,  the shared
> > > runners they have work fine for x86. Just need the environments
> > > you are testing to be provided as Docker containers (you can
> > > actually build and cache the container images during your CI job
> > > too).  IOW, any Linux distro build and test jobs should be able
> > > to use shared runners on x86, and likewise mingw builds. Custom
> > > runners should only be needed if the jobs need todo *BSD / macOS
> > > builds, and/or have access to specific hardware devices for some
> > > reason.
> 
> Not just ^that, you also want custom VM runners to run integration
> tests, e.g. in libvirt, we'd have to put systemd and a lof of other
> cruft into the container to be able to run the tests at which point
> you must ask yourself, whyt not go with a VM instead in which case
> we're limited in terms of infrastructure...
> 

I'm completely in agreement that a lot of the jobs will be suitable to
be run on containers, but like you exemplified here Erik, we must take
into account the ones that won't be suitable.

For instance, a very real use case is testing KVM on bare metal.  One
could argue that "QEMU running on a container making use of
KVM would suffice". It may be true, it may not.  But even that 
won't be possible Today on a CentOS/RHEL 8 machine, because
gitlab-runner knows nothing about podman, so full blown x86
physical boxes may be the "cheaper" and more practical solution, at
least initially. Trying to use other architectures would surely have
similar issues.

> > 
> > Thanks to insist with that point Daniel. I'd rather see every
> > configuration reproducible, so if we loose a hardware sponsor, we
> > can find another one and start another runner.
> > Also note, if it is not easy to reproduce a runner, it will be very
> > hard to debug a reported build/test error.
> 
> (Thanks for bringing ^this point up Philippe)
> 
> ...However, what we've been actively working on in libvirt is to
> extend the lcitool we have (which can spawn local test VMs) to the
> point where we're able to generate machines that would be the
> reproducible. Right now I'm playing with cloud-init integration with
> lcitool (patches coming soon) that would allow us to use the same
> machines locally as we'd want to have in, say, OpenStack and share
> them as compressed images, so even when updated/managed by lcitool
> locally, you'd get the same environment.
> 
> Regards,
> 

This is great, and it aligns with my previous point that reproducibility
it's not *just* about the hardware, but about diligently documenting
and automating the environments, be them mundane or super specialized.
And IMO that should score some points when it comes to being
promoted/demoted as a Gating machine/job.

Thanks for the feedback Erik!
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Andrea Bolognani]

On Mon, 2020-04-27 at 01:24 -0400, Cleber Rosa wrote:
> On Fri, 24 Apr 2020 08:57:46 +0200
> Erik Skultety <eskultet@redhat.com> wrote:
> > On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé
> > wrote:
> > > Thanks to insist with that point Daniel. I'd rather see every
> > > configuration reproducible, so if we loose a hardware sponsor, we
> > > can find another one and start another runner.
> > > Also note, if it is not easy to reproduce a runner, it will be very
> > > hard to debug a reported build/test error.
> > 
> > (Thanks for bringing ^this point up Philippe)
> > 
> > ...However, what we've been actively working on in libvirt is to
> > extend the lcitool we have (which can spawn local test VMs) to the
> > point where we're able to generate machines that would be the
> > reproducible. Right now I'm playing with cloud-init integration with
> > lcitool (patches coming soon) that would allow us to use the same
> > machines locally as we'd want to have in, say, OpenStack and share
> > them as compressed images, so even when updated/managed by lcitool
> > locally, you'd get the same environment.
> 
> This is great, and it aligns with my previous point that reproducibility
> it's not *just* about the hardware, but about diligently documenting
> and automating the environments, be them mundane or super specialized.
> And IMO that should score some points when it comes to being
> promoted/demoted as a Gating machine/job.

I think there's room to extend and rework lcitool so that it can be
used for QEMU CI as well, and we should definitely look into that.

Right now it only really covers VMs and containers, but there's
already one situation (FreeBSD) where the expectation is that you'd
import an existing VM image and then apply CI-related customizations
on top, so it's not too much of a stretch to imagine doing the same
for a bare metal machine. We use Ansible, so as long as we can
connect to the machine via ssh we're pretty much good to go.

Installation of VMs we already perform in an unattended fashion using
preseed/kickstart, and it should be relatively straighforward to
adapt those configurations to also work on real hardware. This way
we'd both be able to rely on having a sane OS as the base, and
relieve the administrator from the duty of manually installing the
machines.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Thu, 23 Apr 2020 23:28:21 +0200
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> On 4/23/20 7:13 PM, Daniel P. Berrangé wrote:
> > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> >> ----- Original Message -----
> >>> From: "Peter Maydell" <peter.maydell@linaro.org>
> >>> To: "Markus Armbruster" <armbru@redhat.com>
> >>> Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth"
> >>> <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> >>> Skultety" <eskultet@redhat.com>, "Alex Bennée"
> >>> <alex.bennee@linaro.org>, "Wainer Moschetta"
> >>> <wmoschet@redhat.com>, "QEMU Developers" <qemu-devel@nongnu.org>,
> >>> "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian
> >>> Rampazzo" <wrampazz@redhat.com>, "Cleber Rosa"
> >>> <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>,
> >>> "Eduardo Habkost" <ehabkost@redhat.com> Sent: Tuesday, April 21,
> >>> 2020 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI
> >>>
> >>> On Thu, 19 Mar 2020 at 16:33, Markus Armbruster
> >>> <armbru@redhat.com> wrote:
> >>>> Peter Maydell <peter.maydell@linaro.org> writes:
> >>>>> I think we should start by getting the gitlab setup working
> >>>>> for the basic "x86 configs" first. Then we can try adding
> >>>>> a runner for s390 (that one's logistically easiest because
> >>>>> it is a project machine, not one owned by me personally or
> >>>>> by Linaro) once the basic framework is working, and expand
> >>>>> from there.
> >>>>
> >>>> Makes sense to me.
> >>>>
> >>>> Next steps to get this off the ground:
> >>>>
> >>>> * Red Hat provides runner(s) for x86 stuff we care about.
> >>>>
> >>>> * If that doesn't cover 'basic "x86 configs" in your judgement,
> >>>> we fill the gaps as described below under "Expand from there".
> >>>>
> >>>> * Add an s390 runner using the project machine you mentioned.
> >>>>
> >>>> * Expand from there: identify the remaining gaps, map them to
> >>>> people / organizations interested in them, and solicit
> >>>> contributions from these guys.
> >>>>
> >>>> A note on contributions: we need both hardware and people.  By
> >>>> people I mean maintainers for the infrastructure, the tools and
> >>>> all the runners. Cleber & team are willing to serve for the
> >>>> infrastructure, the tools and the Red Hat runners.
> >>>
> >>> So, with 5.0 nearly out the door it seems like a good time to
> >>> check in on this thread again to ask where we are progress-wise
> >>> with this. My impression is that this patchset provides most of
> >>> the scripting and config side of the first step, so what we need
> >>> is for RH to provide an x86 runner machine and tell the gitlab CI
> >>> it exists. I appreciate that the whole coronavirus and
> >>> working-from-home situation will have upended everybody's plans,
> >>> especially when actual hardware might be involved, but how's it
> >>> going ?
> >>>
> >>
> >> Hi Peter,
> >>
> >> You hit the nail in the head here.  We were affected indeed with
> >> our ability to move some machines from one lab to another (across
> >> the country), but we're actively working on it.
> > 
> > For x86, do we really need to be using custom runners ?
> > 
> > With GitLab if someone forks the repo to their personal namespace,
> > they cannot use any custom runners setup by the origin project. So
> > if we use custom runners for x86, people forking won't be able to
> > run the GitLab CI jobs.
> > 
> > As a sub-system maintainer I wouldn't like this, because I ideally
> > want to be able to run the same jobs on my staging tree, that Peter
> > will run at merge time for the PULL request I send.
> > 
> > Thus my strong preference would be to use the GitLab runners in
> > every scenario where they are viable to use. Only use custom
> > runners in the cases where GitLab runners are clearly inadequate
> > for our needs.
> > 
> > Based on what we've setup in GitLab for libvirt,  the shared runners
> > they have work fine for x86. Just need the environments you are
> > testing to be provided as Docker containers (you can actually build
> > and cache the container images during your CI job too).  IOW, any
> > Linux distro build and test jobs should be able to use shared
> > runners on x86, and likewise mingw builds. Custom runners should
> > only be needed if the jobs need todo *BSD / macOS builds, and/or
> > have access to specific hardware devices for some reason.
> 
> Thanks to insist with that point Daniel. I'd rather see every 
> configuration reproducible, so if we loose a hardware sponsor, we can 
> find another one and start another runner.

I also believe that having jobs that can be reproducible is key, but I
differ when it comes to believing that the hardware *alone* should
define if a job is gating or not.

My point is that even with easily accessible systems and software,
different jobs can easily yield different results, because of how the
underlying system is configured.  Sure, containers, but again, we have
to consider non container usage too.

In the RFC I tried to gather feedback on a plan to promote and demote
jobs from a gating status.  IMO, most jobs would begin their lives
being non-gating, and would prove both their stability and their
mantainer's responsiveness.  Even when such jobs are already gating,
it should be trivial to simply demote a gating job because of (and
not limited to) any of the following reasons:

 * job fails in a non-reproducible way
 * hardware is unresponsive and takes too long to produce results
 * maintainer is MIA

Some or all of the gating runners could also pick up jobs sent to
a branch other than "staging", say, a branch called "reproducer". That
branch could be writable by maintainers that need to reproduce a given
failure.

> Also note, if it is not easy to reproduce a runner, it will be very
> hard to debug a reported build/test error.
> 

One of the goals of the patches you'll find on this series is to
propose (I would say *require*) that new jobs that require new hardware
(even easily accessible systems such as x86) should provide easy to run
scripts to recreate those environments.  This is inline with my previous
point that it's not enough to just have the same hardware.

> A non-reproducible runner can not be used as gating, because if they 
> fail it is not acceptable to lock the project development process.
> 

Other people may be more familiar with this, but I do remember projects
such as OpenStack deferring test of hardware/software combinations to
other entities.  One specific party won't be able to
reproduce all configurations unless it's decided to be kept really
small.  In my opinion, it's better to face it and acknowledge that
fact, and have plans to be put to action in the exceptional cases where
the environment to reproduce a test is now unavailable.

> 
> In some cases custom runners are acceptable. These runners won't be 
> "gating" but can post informative log and status.
> 

Well, I have the feeling that some people maintaining those runners
will *not* want to have them as "informational" only.  If they invest a
good amount of time on them, I believe they'll want to reap the
benefits such as other not breaking the code they rely on.  If their
system is not gating, they lose that and may find breakage that CI did
not catch.  Again, I don't think "easily accessible" hardware should be
the only criteria for gating/non-gating status.

For instance, would you consider, say, a "Raspberry Pi 4 Model
B", running KVM jobs to be a reproducible runner?  Would you blame a
developer that breaks a Gating CI job on such a platform and says that
he can not reproduce it?

> [*] Specific hardware that is not easily available.
> 
> - Alistair at last KVM forum talked about a RISCV board
>    (to test host TCG)
> - Aleksandar said at last KVM forum Wavecomp could plug a CI20 MIPS
>    (to test host TCG)
> - Lemote seems interested to setup some Loongson MIPSr6 board
>    (to test interaction with KVM)
> 
> [*] To run code requiring accepting License Agreements
> 
> [*] To run non Free / Open Source code
> 
> 
> Owner of these runners take the responsibility to provide enough 
> time/information about reported bugs, or to debug them themselves.
> 

I do think that the owner of such runners may *not* want to have them
with Gating jobs, but I don't think the opposite should be the case,
because I find it very hard to define, without some prejudice, what
"easily available runner" means.

> 
> Now the problem is GitLab runner is not natively available on the 
> architectures listed in this mail, so custom setup is required. A
> dumb script running ssh to a machine also works (tested) but lot of
> manual tuning/maintenance expected.
> 

That's where I'm trying to help.  I built and tested the gitlab-runner
for a number of non-supported environments, and I expect to build
further on that (say contributing code or feedback back to GitLab so
they become official builds?).

Cheers,
- Cleber. 

Re: [PATCH 0/5] QEMU Gating CI

[Philippe Mathieu-Daudé]

On 4/27/20 7:12 AM, Cleber Rosa wrote:
> On Thu, 23 Apr 2020 23:28:21 +0200
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
[...]
>> In some cases custom runners are acceptable. These runners won't be
>> "gating" but can post informative log and status.
>>
> 
> Well, I have the feeling that some people maintaining those runners
> will *not* want to have them as "informational" only.  If they invest a
> good amount of time on them, I believe they'll want to reap the
> benefits such as other not breaking the code they rely on.  If their
> system is not gating, they lose that and may find breakage that CI did
> not catch.  Again, I don't think "easily accessible" hardware should be
> the only criteria for gating/non-gating status.
> 
> For instance, would you consider, say, a "Raspberry Pi 4 Model
> B", running KVM jobs to be a reproducible runner?  Would you blame a
> developer that breaks a Gating CI job on such a platform and says that
> he can not reproduce it?

I'm not sure I understood the problem, as I'd answer "yes" but I guess 
you expect me to say "no"?

[...]
>> Now the problem is GitLab runner is not natively available on the
>> architectures listed in this mail, so custom setup is required. A
>> dumb script running ssh to a machine also works (tested) but lot of
>> manual tuning/maintenance expected.
>>
> 
> That's where I'm trying to help.  I built and tested the gitlab-runner
> for a number of non-supported environments, and I expect to build
> further on that (say contributing code or feedback back to GitLab so
> they become official builds?).

Good luck with that, it took more that 2 years to GitLab to officially 
support AMD64:
https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725

Hopefully the first non-x86 user was the hardest one who had to do all 
the bad work, and next architecture might get supported quicker...

> 
> Cheers,
> - Cleber.
> 

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Mon, 27 Apr 2020 12:51:36 +0200
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> On 4/27/20 7:12 AM, Cleber Rosa wrote:
> > On Thu, 23 Apr 2020 23:28:21 +0200
> > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> [...]
> >> In some cases custom runners are acceptable. These runners won't be
> >> "gating" but can post informative log and status.
> >>
> > 
> > Well, I have the feeling that some people maintaining those runners
> > will *not* want to have them as "informational" only.  If they
> > invest a good amount of time on them, I believe they'll want to
> > reap the benefits such as other not breaking the code they rely on.
> >  If their system is not gating, they lose that and may find
> > breakage that CI did not catch.  Again, I don't think "easily
> > accessible" hardware should be the only criteria for
> > gating/non-gating status.
> > 
> > For instance, would you consider, say, a "Raspberry Pi 4 Model
> > B", running KVM jobs to be a reproducible runner?  Would you blame a
> > developer that breaks a Gating CI job on such a platform and says
> > that he can not reproduce it?
> 
> I'm not sure I understood the problem, as I'd answer "yes" but I
> guess you expect me to say "no"?
> 

What I mean is: would you blame such a developer for *not* having a
machine himself/herself that he/she can try to reproduce the failure?
And would you consider a "Raspberry Pi 4 Model B" an easily available
hardware?

> [...]
> >> Now the problem is GitLab runner is not natively available on the
> >> architectures listed in this mail, so custom setup is required. A
> >> dumb script running ssh to a machine also works (tested) but lot of
> >> manual tuning/maintenance expected.
> >>
> > 
> > That's where I'm trying to help.  I built and tested the
> > gitlab-runner for a number of non-supported environments, and I
> > expect to build further on that (say contributing code or feedback
> > back to GitLab so they become official builds?).
> 
> Good luck with that, it took more that 2 years to GitLab to
> officially support AMD64:
> https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/725
> 

You mean aarch64, sure.  I'm not holding my breath, because we can
always have our own binaries/ports (or other executors such as ssh) but
I'm optimistic...

> Hopefully the first non-x86 user was the hardest one who had to do
> all the bad work, and next architecture might get supported quicker...
> 

... and this point is one of the reasons.  The other is competition
from Travis-CI (and others).

Cheers,
- Cleber.

Re: [PATCH 0/5] QEMU Gating CI

[Philippe Mathieu-Daudé]

On 4/27/20 4:28 PM, Cleber Rosa wrote:
> On Mon, 27 Apr 2020 12:51:36 +0200
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> On 4/27/20 7:12 AM, Cleber Rosa wrote:
>>> On Thu, 23 Apr 2020 23:28:21 +0200
>>> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>> [...]
>>>> In some cases custom runners are acceptable. These runners won't be
>>>> "gating" but can post informative log and status.
>>>>
>>>
>>> Well, I have the feeling that some people maintaining those runners
>>> will *not* want to have them as "informational" only.  If they
>>> invest a good amount of time on them, I believe they'll want to
>>> reap the benefits such as other not breaking the code they rely on.
>>>   If their system is not gating, they lose that and may find
>>> breakage that CI did not catch.  Again, I don't think "easily
>>> accessible" hardware should be the only criteria for
>>> gating/non-gating status.
>>>
>>> For instance, would you consider, say, a "Raspberry Pi 4 Model
>>> B", running KVM jobs to be a reproducible runner?  Would you blame a
>>> developer that breaks a Gating CI job on such a platform and says
>>> that he can not reproduce it?
>>
>> I'm not sure I understood the problem, as I'd answer "yes" but I
>> guess you expect me to say "no"?
>>
> 
> What I mean is: would you blame such a developer for *not* having a
> machine himself/herself that he/she can try to reproduce the failure?
> And would you consider a "Raspberry Pi 4 Model B" an easily available
> hardware?

My view on this is if someone merged code in mainstream QEMU and 
maintains it, and if it is not easy to reproduce the setup (for a bug 
reported by a CI script), then it is the responsibility of the 
maintainer to resolve it. Either by providing particular access to the 
hardware, or be ready to spend a long debugging session over email and 
multiple time zones.

If it is not possible, then this specific code/setup can not claim for 
gating CI, and eventually mainstream isn't the best place for it.

>> [...]

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Mon, 27 Apr 2020 16:41:38 +0200
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> On 4/27/20 4:28 PM, Cleber Rosa wrote:
> > 
> > What I mean is: would you blame such a developer for *not* having a
> > machine himself/herself that he/she can try to reproduce the
> > failure? And would you consider a "Raspberry Pi 4 Model B" an
> > easily available hardware?
> 
> My view on this is if someone merged code in mainstream QEMU and 
> maintains it, and if it is not easy to reproduce the setup (for a bug 
> reported by a CI script), then it is the responsibility of the 
> maintainer to resolve it. Either by providing particular access to
> the hardware, or be ready to spend a long debugging session over
> email and multiple time zones.
> 

Right, the "easy to reproduce" has a lot to with access to hardware,
and a lot to do with access to the same or reproducible setup.  And
yes, if I maintain platform/job "foobar" that was once upgraded to
gating status, has since then fallen behind and doesn't allow users to
easily reproduce it, it all falls unto the maintainer to resolve issues.

I'd even say that people having access to identical hardware could
proactively challenge a given job status as gating if they fail to
reproduce it with the provided documentation/scripts.

> If it is not possible, then this specific code/setup can not claim
> for gating CI, and eventually mainstream isn't the best place for it.
> 
> >> [...]
> 

IIUC, we're in agreement. :)

Thanks,
- CLeber.

Re: [PATCH 0/5] QEMU Gating CI

[Daniel P. Berrangé]

On Mon, Apr 27, 2020 at 04:41:38PM +0200, Philippe Mathieu-Daudé wrote:
> On 4/27/20 4:28 PM, Cleber Rosa wrote:
> > On Mon, 27 Apr 2020 12:51:36 +0200
> > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> > 
> > > On 4/27/20 7:12 AM, Cleber Rosa wrote:
> > > > On Thu, 23 Apr 2020 23:28:21 +0200
> > > > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> > > [...]
> > > > > In some cases custom runners are acceptable. These runners won't be
> > > > > "gating" but can post informative log and status.
> > > > > 
> > > > 
> > > > Well, I have the feeling that some people maintaining those runners
> > > > will *not* want to have them as "informational" only.  If they
> > > > invest a good amount of time on them, I believe they'll want to
> > > > reap the benefits such as other not breaking the code they rely on.
> > > >   If their system is not gating, they lose that and may find
> > > > breakage that CI did not catch.  Again, I don't think "easily
> > > > accessible" hardware should be the only criteria for
> > > > gating/non-gating status.
> > > > 
> > > > For instance, would you consider, say, a "Raspberry Pi 4 Model
> > > > B", running KVM jobs to be a reproducible runner?  Would you blame a
> > > > developer that breaks a Gating CI job on such a platform and says
> > > > that he can not reproduce it?
> > > 
> > > I'm not sure I understood the problem, as I'd answer "yes" but I
> > > guess you expect me to say "no"?
> > > 
> > 
> > What I mean is: would you blame such a developer for *not* having a
> > machine himself/herself that he/she can try to reproduce the failure?
> > And would you consider a "Raspberry Pi 4 Model B" an easily available
> > hardware?
> 
> My view on this is if someone merged code in mainstream QEMU and maintains
> it, and if it is not easy to reproduce the setup (for a bug reported by a CI
> script), then it is the responsibility of the maintainer to resolve it.
> Either by providing particular access to the hardware, or be ready to spend
> a long debugging session over email and multiple time zones.
> 
> If it is not possible, then this specific code/setup can not claim for
> gating CI, and eventually mainstream isn't the best place for it.

I'd caution to be wary about using gating CI as a big stick for hitting
contributors with. The more rules we put in place whicih contributors
have to follow before their work gets accepted for merge, the less likely
someone is to have a positive experiance contributing to the project, or
even be willing to try. This view of gating CI requirements was a negative
aspect of contributing to the OpenStack project, which drove people away.
There was pushback against contributing work because it lacked CI, but
there was often no viable way for to actually provide CI in a feasible
timeframe, especially for stuff only testable in physical hardware and
not VMs. Even if you work for a big company, it doesn't make it easy to
magic up money to spend on hardware & hosting to provide CI, as corporate
beaurcracy & priorities will get in your way.

I'd really encourage the more nuanced approach of thinking in terms of
tiered support levels:

  - Tier 1: features that we have gating CI tests for. Will always work.

  - Tier 2: features that we have non-gating CI test for. Should work at
            time of release, but may be broken for periods in git master.
	    
  - Tier 3: features that we don't have CI tests for. Compile tested only,
            relying on end user manual testing, so may or may not work
	    at any time or in release.

Obviously tier 1 is the gold standard that we would like everything to
achieve but we'll never achieve that reality unless we cull 90% of QEMU's
code. I don't think that's in the best interests of our users, because
clearly stuff in Tier 2 and Tier 3 is still useful for a large portion of
our end users - not least because Tier 3 is the level everything is at
right now in QEMU unless using a downstream vendor's packages.

The tier levels and CI are largely around setting reasonable quality
expectations. Right now we often have a problem that poeople want to
re-factor code but are afraid of breaking existing functionality that
guests rely on. This causes delays in merging code or causes people to
not even attempt the refactoring in the first place. This harms our
forward progress in QEMU.

With gating CI, we are declaring that contributors should feel free to
refactor anything as long as it passes gating CI. IOW, contributors only
have to care about Tier 1 features continuing to work. It would be nice
if refactoring does not breaks stuff in Tier 2 / 3, but if it does, then
that is acceptable collatoral damage. We would not block the merge on
stuff that is Tier 2 / 3.

Based on what I experianced in OpenStack the other big challenge is
deciding when something can be promoted from Tier 2 to Tier 1. They
had the official gating CI (for Tier 1) being maintained by the core
project infrastructure team. Any CI provided by third party companies
was non-gating (Tier 2) (at least in the time I was involved) because
they did not want code merge blocked on ability to communicate with
third party companies who were often hard to contact when CI broke.
So the only real path from Tier 2 to Tier 1 was to give the project
direct access to the CI hardware, instead of having the providing
company self-manage it.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 3896 bytes --]

On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> 
> 
> ----- Original Message -----
> > From: "Peter Maydell" <peter.maydell@linaro.org>
> > To: "Markus Armbruster" <armbru@redhat.com>
> > Cc: "Fam Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Beraldo Leal" <bleal@redhat.com>, "Erik
> > Skultety" <eskultet@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Wainer Moschetta" <wmoschet@redhat.com>,
> > "QEMU Developers" <qemu-devel@nongnu.org>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Willian Rampazzo"
> > <wrampazz@redhat.com>, "Cleber Rosa" <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, "Eduardo
> > Habkost" <ehabkost@redhat.com>
> > Sent: Tuesday, April 21, 2020 8:53:49 AM
> > Subject: Re: [PATCH 0/5] QEMU Gating CI
> > 
> > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster <armbru@redhat.com> wrote:
> > > Peter Maydell <peter.maydell@linaro.org> writes:
> > > > I think we should start by getting the gitlab setup working
> > > > for the basic "x86 configs" first. Then we can try adding
> > > > a runner for s390 (that one's logistically easiest because
> > > > it is a project machine, not one owned by me personally or
> > > > by Linaro) once the basic framework is working, and expand
> > > > from there.
> > >
> > > Makes sense to me.
> > >
> > > Next steps to get this off the ground:
> > >
> > > * Red Hat provides runner(s) for x86 stuff we care about.
> > >
> > > * If that doesn't cover 'basic "x86 configs" in your judgement, we
> > >   fill the gaps as described below under "Expand from there".
> > >
> > > * Add an s390 runner using the project machine you mentioned.
> > >
> > > * Expand from there: identify the remaining gaps, map them to people /
> > >   organizations interested in them, and solicit contributions from these
> > >   guys.
> > >
> > > A note on contributions: we need both hardware and people.  By people I
> > > mean maintainers for the infrastructure, the tools and all the runners.
> > > Cleber & team are willing to serve for the infrastructure, the tools and
> > > the Red Hat runners.
> > 
> > So, with 5.0 nearly out the door it seems like a good time to check
> > in on this thread again to ask where we are progress-wise with this.
> > My impression is that this patchset provides most of the scripting
> > and config side of the first step, so what we need is for RH to provide
> > an x86 runner machine and tell the gitlab CI it exists. I appreciate
> > that the whole coronavirus and working-from-home situation will have
> > upended everybody's plans, especially when actual hardware might
> > be involved, but how's it going ?
> > 
> 
> Hi Peter,
> 
> You hit the nail in the head here.  We were affected indeed with our ability
> to move some machines from one lab to another (across the country), but we're
> actively working on it.
> 
> From now on, I'll give you an update every time a significant event occurs on
> our side.
>

Hi all,

It took a while, but I finally have some updates, and they're pretty
good news.

Red Hat is sponsoring 3 x86_64 machines that will act as runners for
the QEMU CI, and together with QEMU's own s390 and aarch64 machines (1
each), we have enough hardware now for a reasonable build and test
coverage.

The s390 and aarch64 machines are already up and running, and the
x86_64 machines are being racked up, but should be up and running
in the next few days.  I'm working on an updated version of this
series that takes into account this new scenario, and some
fixes and improvements.

And as a reminder, if you (individual or organization) would like to
sponsor hardware or people to expand the QEMU build and test coverage,
please reach out to us.

Thanks,
- Cleber.

> > thanks
> > -- PMM
> > 
> > 
> 
> Thanks for checking in!
> - Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/5] QEMU Gating CI

[Daniel P. Berrangé]

On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote:
> The idea about a public facing Gating CI for QEMU was lastly
> summarized in an RFC[1].  Since then, it was decided that a
> simpler version should be attempted first.
> 
> Changes from the RFC patches[2] accompanying the RFC document:
> 
> - Moved gating job definitions to .gitlab-ci-gating.yml
> - Added info on "--disable-libssh" build option requirement
>   (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
> - Added info on "--disable-glusterfs" build option requirement
>   (there's no static version of those libs in distro supplied
>   packages) to one
> - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
>   doesn't fall into the general scope of gating job described by PMM
>   (and it did not run any test)
> - Added w32 and w64 cross builds based on Fedora 30
> - Added a FreeBSD based job that builds all targets and runs `make
>   check`
> - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
>   simple but effective way of speeding up the builds and tests by
>   using a number of make jobs matching the number of CPUs
> - Because the Ansible playbooks reference the content on Dockerfiles,
>   some fixes to some Dockerfiles caught in the process were included
> - New patch with script to check or wait on a pipeline execution
> 
> [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
> [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html
> 
> Cleber Rosa (5):
>   tests/docker: add CentOS 8 Dockerfile
>   tests/docker: make "buildah bud" output similar to "docker build"
>   GitLab CI: avoid calling before_scripts on unintended jobs
>   GitLab Gating CI: introduce pipeline-status contrib script
>   GitLab Gating CI: initial set of jobs, documentation and scripts
> 
>  .gitlab-ci-gating.yml                         | 111 ++++++++++
>  .gitlab-ci.yml                                |  32 ++-
>  contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
>  contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
>  contrib/ci/orgs/qemu/inventory                |   2 +
>  contrib/ci/orgs/qemu/vars.yml                 |  13 ++
>  contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++

FYI, the contrib/ directory is generally a place for arbitrary / adhoc
but interesting user contributed files/sources that are not officially
supported deliverables of the project.

IOW, this is not a good home for the official CI scripts.

We already have a .gitlab-ci.d/ directory that looks like it would
be good for this.  Or if that's not suitable, then scripts/ci/ is
a second choice.

>  docs/devel/testing.rst                        | 142 ++++++++++++
>  tests/docker/dockerfiles/centos8.docker       |  32 +++
>  .../dockerfiles/debian-win32-cross.docker     |   2 +-
>  10 files changed, 751 insertions(+), 4 deletions(-)
>  create mode 100644 .gitlab-ci-gating.yml
>  create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
>  create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
>  create mode 100644 contrib/ci/orgs/qemu/inventory
>  create mode 100644 contrib/ci/orgs/qemu/vars.yml
>  create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
>  create mode 100644 tests/docker/dockerfiles/centos8.docker

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

----- Original Message -----
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Alex Bennée" <alex.bennee@linaro.org>, "Peter Maydell" <peter.maydell@linaro.org>, qemu-devel@nongnu.org, "Fam
> Zheng" <fam@euphon.net>, "Thomas Huth" <thuth@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>, "Erik Skultety"
> <eskultet@redhat.com>, "Wainer Moschetta" <wmoschet@redhat.com>, "Wainer dos Santos Moschetta"
> <wainersm@redhat.com>, "Willian Rampazzo" <wrampazz@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>,
> "Beraldo Leal" <bleal@redhat.com>
> Sent: Monday, March 16, 2020 8:38:07 AM
> Subject: Re: [PATCH 0/5] QEMU Gating CI
> 
> On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote:
> > The idea about a public facing Gating CI for QEMU was lastly
> > summarized in an RFC[1].  Since then, it was decided that a
> > simpler version should be attempted first.
> > 
> > Changes from the RFC patches[2] accompanying the RFC document:
> > 
> > - Moved gating job definitions to .gitlab-ci-gating.yml
> > - Added info on "--disable-libssh" build option requirement
> >   (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
> > - Added info on "--disable-glusterfs" build option requirement
> >   (there's no static version of those libs in distro supplied
> >   packages) to one
> > - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
> >   doesn't fall into the general scope of gating job described by PMM
> >   (and it did not run any test)
> > - Added w32 and w64 cross builds based on Fedora 30
> > - Added a FreeBSD based job that builds all targets and runs `make
> >   check`
> > - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
> >   simple but effective way of speeding up the builds and tests by
> >   using a number of make jobs matching the number of CPUs
> > - Because the Ansible playbooks reference the content on Dockerfiles,
> >   some fixes to some Dockerfiles caught in the process were included
> > - New patch with script to check or wait on a pipeline execution
> > 
> > [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
> > [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html
> > 
> > Cleber Rosa (5):
> >   tests/docker: add CentOS 8 Dockerfile
> >   tests/docker: make "buildah bud" output similar to "docker build"
> >   GitLab CI: avoid calling before_scripts on unintended jobs
> >   GitLab Gating CI: introduce pipeline-status contrib script
> >   GitLab Gating CI: initial set of jobs, documentation and scripts
> > 
> >  .gitlab-ci-gating.yml                         | 111 ++++++++++
> >  .gitlab-ci.yml                                |  32 ++-
> >  contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
> >  contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
> >  contrib/ci/orgs/qemu/inventory                |   2 +
> >  contrib/ci/orgs/qemu/vars.yml                 |  13 ++
> >  contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
> 
> FYI, the contrib/ directory is generally a place for arbitrary / adhoc
> but interesting user contributed files/sources that are not officially
> supported deliverables of the project.
> 
> IOW, this is not a good home for the official CI scripts.
> 

Good point, reason is that I wasn't/ain't sure this script is going
to be "official" and really that useful.  I'm happy to move it
somewhere else though.

> We already have a .gitlab-ci.d/ directory that looks like it would
> be good for this.  Or if that's not suitable, then scripts/ci/ is
> a second choice.
> 

Ack.

Thanks!
- Cleber.

> >  docs/devel/testing.rst                        | 142 ++++++++++++
> >  tests/docker/dockerfiles/centos8.docker       |  32 +++
> >  .../dockerfiles/debian-win32-cross.docker     |   2 +-
> >  10 files changed, 751 insertions(+), 4 deletions(-)
> >  create mode 100644 .gitlab-ci-gating.yml
> >  create mode 100644 contrib/ci/orgs/qemu/build-environment.yml
> >  create mode 100644 contrib/ci/orgs/qemu/gitlab-runner.yml
> >  create mode 100644 contrib/ci/orgs/qemu/inventory
> >  create mode 100644 contrib/ci/orgs/qemu/vars.yml
> >  create mode 100755 contrib/ci/scripts/gitlab-pipeline-status
> >  create mode 100644 tests/docker/dockerfiles/centos8.docker
> 
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
> |:|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com
> |:|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
> |:|
> 

Re: [PATCH 0/5] QEMU Gating CI

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote:
>> The idea about a public facing Gating CI for QEMU was lastly
>> summarized in an RFC[1].  Since then, it was decided that a
>> simpler version should be attempted first.
>> 
>> Changes from the RFC patches[2] accompanying the RFC document:
>> 
>> - Moved gating job definitions to .gitlab-ci-gating.yml
>> - Added info on "--disable-libssh" build option requirement
>>   (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
>> - Added info on "--disable-glusterfs" build option requirement
>>   (there's no static version of those libs in distro supplied
>>   packages) to one
>> - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
>>   doesn't fall into the general scope of gating job described by PMM
>>   (and it did not run any test)
>> - Added w32 and w64 cross builds based on Fedora 30
>> - Added a FreeBSD based job that builds all targets and runs `make
>>   check`
>> - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
>>   simple but effective way of speeding up the builds and tests by
>>   using a number of make jobs matching the number of CPUs
>> - Because the Ansible playbooks reference the content on Dockerfiles,
>>   some fixes to some Dockerfiles caught in the process were included
>> - New patch with script to check or wait on a pipeline execution
>> 
>> [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
>> [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html
>> 
>> Cleber Rosa (5):
>>   tests/docker: add CentOS 8 Dockerfile
>>   tests/docker: make "buildah bud" output similar to "docker build"
>>   GitLab CI: avoid calling before_scripts on unintended jobs
>>   GitLab Gating CI: introduce pipeline-status contrib script
>>   GitLab Gating CI: initial set of jobs, documentation and scripts
>> 
>>  .gitlab-ci-gating.yml                         | 111 ++++++++++
>>  .gitlab-ci.yml                                |  32 ++-
>>  contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
>>  contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
>>  contrib/ci/orgs/qemu/inventory                |   2 +
>>  contrib/ci/orgs/qemu/vars.yml                 |  13 ++
>>  contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
>
> FYI, the contrib/ directory is generally a place for arbitrary / adhoc
> but interesting user contributed files/sources that are not officially
> supported deliverables of the project.
>
> IOW, this is not a good home for the official CI scripts.
>
> We already have a .gitlab-ci.d/ directory that looks like it would
> be good for this.  Or if that's not suitable, then scripts/ci/ is
> a second choice.

I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be
a little hidden.

-- 
Alex Bennée

Re: [PATCH 0/5] QEMU Gating CI

[Aleksandar Markovic]

On Mon, Mar 16, 2020 at 4:24 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Thu, Mar 12, 2020 at 03:36:11PM -0400, Cleber Rosa wrote:
> >> The idea about a public facing Gating CI for QEMU was lastly
> >> summarized in an RFC[1].  Since then, it was decided that a
> >> simpler version should be attempted first.
> >>
> >> Changes from the RFC patches[2] accompanying the RFC document:
> >>
> >> - Moved gating job definitions to .gitlab-ci-gating.yml
> >> - Added info on "--disable-libssh" build option requirement
> >>   (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
> >> - Added info on "--disable-glusterfs" build option requirement
> >>   (there's no static version of those libs in distro supplied
> >>   packages) to one
> >> - Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
> >>   doesn't fall into the general scope of gating job described by PMM
> >>   (and it did not run any test)
> >> - Added w32 and w64 cross builds based on Fedora 30
> >> - Added a FreeBSD based job that builds all targets and runs `make
> >>   check`
> >> - Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
> >>   simple but effective way of speeding up the builds and tests by
> >>   using a number of make jobs matching the number of CPUs
> >> - Because the Ansible playbooks reference the content on Dockerfiles,
> >>   some fixes to some Dockerfiles caught in the process were included
> >> - New patch with script to check or wait on a pipeline execution
> >>
> >> [1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
> >> [2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html
> >>
> >> Cleber Rosa (5):
> >>   tests/docker: add CentOS 8 Dockerfile
> >>   tests/docker: make "buildah bud" output similar to "docker build"
> >>   GitLab CI: avoid calling before_scripts on unintended jobs
> >>   GitLab Gating CI: introduce pipeline-status contrib script
> >>   GitLab Gating CI: initial set of jobs, documentation and scripts
> >>
> >>  .gitlab-ci-gating.yml                         | 111 ++++++++++
> >>  .gitlab-ci.yml                                |  32 ++-
> >>  contrib/ci/orgs/qemu/build-environment.yml    | 208 ++++++++++++++++++
> >>  contrib/ci/orgs/qemu/gitlab-runner.yml        |  65 ++++++
> >>  contrib/ci/orgs/qemu/inventory                |   2 +
> >>  contrib/ci/orgs/qemu/vars.yml                 |  13 ++
> >>  contrib/ci/scripts/gitlab-pipeline-status     | 148 +++++++++++++
> >
> > FYI, the contrib/ directory is generally a place for arbitrary / adhoc
> > but interesting user contributed files/sources that are not officially
> > supported deliverables of the project.
> >
> > IOW, this is not a good home for the official CI scripts.
> >
> > We already have a .gitlab-ci.d/ directory that looks like it would
> > be good for this.  Or if that's not suitable, then scripts/ci/ is
> > a second choice.
>
> I'd vote for scripts/ci/ or scripts/gitlab/ as the .gitlab-ci.d might be
> a little hidden.
>

I vote for scripts/ci/ or scripts/gitlab/ too. With a little
preference to scripts/ci/.

Aleksandar

> --
> Alex Bennée
>