[PATCH v4 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[PATCH v4 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Cleber Rosa]

TL;DR: this should allow the QEMU maintainer to push to the staging
branch, and have custom jobs running on the project's aarch64 and
s390x machines.  Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an
RFC[1].  Since then, it was decided that a simpler version should be
attempted first.

At this point, there are two specific runners (an aarch64 and an s390)
registered with GitLab, at https://gitlab.com/qemu-project, currently
setup to the "qemu" repository.

Changes from v3:

- Applied changes to match <20201014135416.1290679-1-pbonzini@redhat.com>,
  that is, added ninja-build to "build-environment.yml" list of packages
  and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners",
  given that the other jobs running on shared runners are also
  considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
 * update -> up to date (Erik)
 * a different set of CI jobs -> different CI jobs (Erik)
 * Pull requests will only be merged -> code will only be merged (Stefan)
 * Setup -> set up (Stefan)
 * them -> they (Stefan)
 * the -> where the (Stefan)
 * dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of
  the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that
  no external variable is used there

- Updated package list in "build-environment.yml" from current
  dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30),
  and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts"
  into (Phillipe):
  1) Basic documentation and configuration (gitlab-ci.yml) placeholder
  2) Playbooks for setting up a build environment
  3) Playbooks for setting up gitlab-runner
  4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine
  build and test.

- Set custom jobs to not depend on any other job, so they can start
  right away.

- Set rules for starting jobs so that all pushing to any branch that
  start with name "staging".  This allows the project maintainer to
  use the "push to staging" workflow, while also allowing others to
  generate similar jobs.  If this project has configured custom
  runners, the jobs will run, if not, the pipeline will be marked as
  "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern
  (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available
  (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available
  (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in
  ".gitlab-ci.yml", assumes that the additional jobs on the staging
  branch running on the freely available gitlab shared runner are
  positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to
  simply 18.04 (assumes no diverse minor levels will be used or
  specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
  (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement
  (there's no static version of those libs in distro supplied
  packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
  doesn't fall into the general scope of gating job described by PMM
  (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs `make
  check`
- Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
  simple but effective way of speeding up the builds and tests by
  using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles,
  some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
  Jobs based on custom runners: documentation and configuration
    placeholder
  Jobs based on custom runners: build environment docs and playbook
  Jobs based on custom runners: docs and gitlab-runner setup playbook
  Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 206 ++++++++++++++++++++++
 .gitlab-ci.yml                         |   1 +
 docs/devel/ci.rst                      | 149 ++++++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   1 +
 scripts/ci/setup/build-environment.yml | 233 +++++++++++++++++++++++++
 scripts/ci/setup/gitlab-runner.yml     |  72 ++++++++
 scripts/ci/setup/inventory             |   2 +
 scripts/ci/setup/vars.yml.template     |  13 ++
 9 files changed, 678 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory
 create mode 100644 scripts/ci/setup/vars.yml.template

-- 
2.25.4

[PATCH v4 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.

Those jobs are intended to run on hardware and/or Operating Systems
not provided by GitLab's shared runners.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 14 +++++++++
 .gitlab-ci.yml                  |  1 +
 docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 70 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
new file mode 100644
index 0000000000..3004da2bda
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the recursive submodule
+# strategy.
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8ffd415ca5..b33c433fd7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
new file mode 100644
index 0000000000..41a4bbddad
--- /dev/null
+++ b/docs/devel/ci.rst
@@ -0,0 +1,54 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner" here.
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Current Jobs
+------------
+
+The current CI jobs based on custom runners have the primary goal of
+catching and preventing regressions on a wider number of host systems
+than the ones provided by GitLab's shared runners.
+
+Also, the mechanics of reliability, capacity and overall maintanance
+of the machines provided by the QEMU project itself for those jobs
+will be evaluated.
+
+Future Plans and Jobs
+---------------------
+
+Once the CI Jobs based on custom runners have been proved mature with
+the initial set of jobs run on machines from the QEMU project, other
+members in the community should be able provide their own machine
+configuration documentation/scripts, and accompanying job definitions.
+
+As a general rule, those newly added contributed jobs should run as
+"non-gating", until their reliability is verified.
+
+The precise minimum requirements and exact rules for machine
+configuration documentation/scripts, and the success rate of jobs are
+still to be defined.
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 77baae5c77..2fdd36e751 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -21,6 +21,7 @@ Contents:
    atomics
    stable-process
    testing
+   ci
    qtest
    decodetree
    secure-coding-practices
-- 
2.25.4

[PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

To run basic jobs on custom runners, the environment needs to be
properly set up.  The most common requirement is having the right
packages installed.

The playbook introduced here covers a number of different Linux
distributions and FreeBSD, and are intended to provide a reproducible
environment.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/ci.rst                      |  32 ++++
 scripts/ci/setup/build-environment.yml | 233 +++++++++++++++++++++++++
 scripts/ci/setup/inventory             |   2 +
 3 files changed, 267 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 41a4bbddad..208b5e399b 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as
 The precise minimum requirements and exact rules for machine
 configuration documentation/scripts, and the success rate of jobs are
 still to be defined.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Start by adding your machines to
+the ``inventory`` file under ``scripts/ci/setup``, such as this::
+
+  [local]
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  [local]
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..074aaca927
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,233 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      apt:
+        update_cache: yes
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+        pkg:
+          - ccache
+          - clang
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libzstd-dev
+          - make
+          - ninja-build
+          - python3-yaml
+          - python3-sphinx
+          - sparse
+          - xfslibs-dev
+        state: present
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      apt:
+        update_cache: yes
+        pkg:
+         - libspice-server-dev
+         - libxen-dev
+        state: present
+      when:
+        - "ansible_facts['distribution'] == 'Ubuntu'"
+        - "ansible_facts['architecture'] != 's390x'"
+
+    - name: Install basic packages to build QEMU on FreeBSD 12.x
+      pkgng:
+        # Originally from packages on .cirrus.yml under the freebsd_12_task
+        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir
+        state: present
+      when: "ansible_facts['os_family'] == 'FreeBSD'"
+
+    - name: Install basic packages to build QEMU on Fedora 30, 31 and 32
+      dnf:
+        # Originally from tests/docker/dockerfiles/fedora.docker
+        name:
+          - SDL2-devel
+          - bc
+          - brlapi-devel
+          - bzip2
+          - bzip2-devel
+          - ccache
+          - clang
+          - cyrus-sasl-devel
+          - dbus-daemon
+          - device-mapper-multipath-devel
+          - diffutils
+          - findutils
+          - gcc
+          - gcc-c++
+          - genisoimage
+          - gettext
+          - git
+          - glib2-devel
+          - glusterfs-api-devel
+          - gnutls-devel
+          - gtk3-devel
+          - hostname
+          - libaio-devel
+          - libasan
+          - libattr-devel
+          - libblockdev-mpath-devel
+          - libcap-ng-devel
+          - libcurl-devel
+          - libepoxy-devel
+          - libfdt-devel
+          - libiscsi-devel
+          - libjpeg-devel
+          - libpmem-devel
+          - libpng-devel
+          - librbd-devel
+          - libseccomp-devel
+          - libssh-devel
+          - libubsan
+          - libudev-devel
+          - libusbx-devel
+          - libxml2-devel
+          - libzstd-devel
+          - llvm
+          - lzo-devel
+          - make
+          - mingw32-SDL2
+          - mingw32-bzip2
+          - mingw32-curl
+          - mingw32-glib2
+          - mingw32-gmp
+          - mingw32-gnutls
+          - mingw32-gtk3
+          - mingw32-libjpeg-turbo
+          - mingw32-libpng
+          - mingw32-libtasn1
+          - mingw32-nettle
+          - mingw32-nsis
+          - mingw32-pixman
+          - mingw32-pkg-config
+          - mingw64-SDL2
+          - mingw64-bzip2
+          - mingw64-curl
+          - mingw64-glib2
+          - mingw64-gmp
+          - mingw64-gnutls
+          - mingw64-gtk3
+          - mingw64-libjpeg-turbo
+          - mingw64-libpng
+          - mingw64-libtasn1
+          - mingw64-nettle
+          - mingw64-pixman
+          - mingw64-pkg-config
+          - ncurses-devel
+          - nettle-devel
+          - ninja-build
+          - nss-devel
+          - numactl-devel
+          - perl
+          - perl-Test-Harness
+          - pixman-devel
+          - python3
+          - python3-PyYAML
+          - python3-numpy
+          - python3-opencv
+          - python3-pillow
+          - python3-pip
+          - python3-sphinx
+          - python3-virtualenv
+          - rdma-core-devel
+          - snappy-devel
+          - sparse
+          - spice-server-devel
+          - systemd-devel
+          - systemtap-sdt-devel
+          - tar
+          - tesseract
+          - tesseract-langpack-eng
+          - usbredir-devel
+          - virglrenderer-devel
+          - vte291-devel
+          - which
+          - xen-devel
+          - zlib-devel
+        state: present
+      when: "ansible_facts['distribution'] == 'Fedora'"
+
+    - name: Enable PowerTools repo on CentOS 8
+      ini_file:
+        path: /etc/yum.repos.d/CentOS-PowerTools.repo
+        section: PowerTools
+        option: enabled
+        value: "1"
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
+
+    - name: Install basic packages to build QEMU on CentOS 8
+      dnf:
+        # Originally from tests/docker/dockerfiles/centos8.docker
+        name:
+          - SDL-devel
+          - bzip2
+          - bzip2-devel
+          - dbus-daemon
+          - gcc
+          - gcc-c++
+          - genisoimage
+          - gettext
+          - git
+          - glib2-devel
+          - libaio-devel
+          - libepoxy-devel
+          - libgcrypt-devel
+          - lzo-devel
+          - make
+          - mesa-libEGL-devel
+          - nettle-devel
+          - ninja-build
+          - perl-Test-Harness
+          - pixman-devel
+          - python36
+          - rdma-core-devel
+          - spice-glib-devel
+          - spice-server
+          - tar
+          - zlib-devel
+        state: present
+      when:
+        - "ansible_facts['distribution'] == 'CentOS'"
+        - "ansible_facts['distribution_major_version'] == '8'"
diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
new file mode 100644
index 0000000000..8bb7ba6b33
--- /dev/null
+++ b/scripts/ci/setup/inventory
@@ -0,0 +1,2 @@
+[local]
+localhost
-- 
2.25.4

[PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

To have the jobs dispatched to custom runners, gitlab-runner must
be installed, active as a service and properly configured.  The
variables file and playbook introduced here should help with those
steps.

The playbook introduced here covers a number of different Linux
distributions and FreeBSD, and are intended to provide a reproducible
environment.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 149 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 208b5e399b..a234a5e24c 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -84,3 +84,66 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+.. note:: there are currently limitations to gitlab-runner itself when
+          setting up a service under FreeBSD systems.  You will need to
+          perform steps 4 to 10 manually, as described at
+          https://docs.gitlab.com/runner/install/freebsd.html
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+FreeBSD 12.1 x86_64 system, the tags should be set as::
+
+  freebsd12.1,x86_64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  freebsd-12.1-x86_64-all:
+   tags:
+   - freebsd_12.1
+   - x86_64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * uncheck the "Run untagged jobs" check box
+ * give it a better Description
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
new file mode 100644
index 0000000000..f112d05dd0
--- /dev/null
+++ b/scripts/ci/setup/.gitignore
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
new file mode 100644
index 0000000000..c2f52dad10
--- /dev/null
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -0,0 +1,72 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403
+      register: gitlab_runner_available_archive
+
+    - name: Update base url
+      set_fact:
+        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
+      when: gitlab_runner_available_archive.status == 200
+    - debug:
+        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
+
+    - name: Set OS name (FreeBSD)
+      set_fact:
+        gitlab_runner_os: freebsd
+      when: "ansible_facts['system'] == 'FreeBSD'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Downloads the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
+      when: "ansible_facts['os_family'] != 'FreeBSD'"
diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
new file mode 100644
index 0000000000..621435d030
--- /dev/null
+++ b/scripts/ci/setup/vars.yml.template
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.1.1
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
-- 
2.25.4

[PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

The QEMU project has two machines (aarch64 and s390) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
 1 file changed, 192 insertions(+)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index 3004da2bda..5b51d1b336 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -12,3 +12,195 @@
 # strategy.
 variables:
   GIT_SUBMODULE_STRATEGY: recursive
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
-- 
2.25.4

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Erik Skultety]

On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---

In general, there's been put quite some effort into the playbooks - sorry I'm
late to the game - is there a plan to introduce QEMU as a project to lcitool?
We've taken care of most of the bits in the playbooks that are being introduced
and for the remaining ones I think it would be that big of an overhaul to do
the adjustments. One major re-factor though would IMO be to break the
dependency lcitool has on the machine naming, kind of restricting it to a
limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which
makes it inconvenient to prepare physical hosts.

More comments inline...

>  docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 149 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template
> 
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 208b5e399b..a234a5e24c 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -84,3 +84,66 @@ To run the playbook, execute::
>  
>    cd scripts/ci/setup
>    ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable
> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``

Yes, this can be suboptimal...Would it make sense to fall back to build the
binary of a given version from git as a fallback during this playbook if the
necessary arch version isn't provided the official way? Just an idea, I'd like
to avoid the need for you to become the maintainer of the binaries and keep up
with the releases.

> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +.. note:: there are currently limitations to gitlab-runner itself when
> +          setting up a service under FreeBSD systems.  You will need to
> +          perform steps 4 to 10 manually, as described at

Which one of them is considered an automation problem? In lcitool we made
gitlab-runner completely automated on all distros, including FreeBSD:

4) log file permissions - you're creating the user, you can as well create the
file with correct permissions

5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml
it will pull a bunch of dependencies which ensure the dir exists

6) gitlab_runner service script should IMO provided by this automation as a
template and install to the correct location

7) ensure the service script is executable - template module can do that

8) register the runner - you're doing that

9) enable the service - Ansible's service module is generic and supports
init/OpenRC

10) I don't see a step 10

IOW I think there should be as little manual intervention as possible and in
this case I don't think any manual steps are needed by the user.

Regards,
Erik

> +          https://docs.gitlab.com/runner/install/freebsd.html
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +FreeBSD 12.1 x86_64 system, the tags should be set as::
> +
> +  freebsd12.1,x86_64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  freebsd-12.1-x86_64-all:
> +   tags:
> +   - freebsd_12.1
> +   - x86_64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * uncheck the "Run untagged jobs" check box
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..c2f52dad10
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,72 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Checks the availability of official gitlab-runner builds in the archive
> +      uri:
> +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
> +        method: HEAD
> +        status_code:
> +          - 200
> +          - 403

Why is 403 an acceptable success status code?

> +      register: gitlab_runner_available_archive
> +
> +    - name: Update base url
> +      set_fact:
> +        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
> +      when: gitlab_runner_available_archive.status == 200
> +    - debug:
> +        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
> +
> +    - name: Set OS name (FreeBSD)
> +      set_fact:
> +        gitlab_runner_os: freebsd
> +      when: "ansible_facts['system'] == 'FreeBSD'"
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Downloads the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +      when: "ansible_facts['os_family'] != 'FreeBSD'"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> +      when: "ansible_facts['os_family'] != 'FreeBSD'"
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..621435d030
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,13 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.1.1
> +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
> +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# Defaults to linux, checks can be used to change this
> +gitlab_runner_os: linux
> +# Defaults to amd64 (x86_64), checks can be used to change this
> +gitlab_runner_arch: amd64
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
> -- 
> 2.25.4
> 

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Erik Skultety]

...

> diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> new file mode 100644
> index 0000000000..8bb7ba6b33
> --- /dev/null
> +++ b/scripts/ci/setup/inventory
> @@ -0,0 +1,2 @@
> +[local]

Nit pick, is a group for localhost actually needed?

Regards,
Erik

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Daniel P. Berrangé]

On Sun, Oct 18, 2020 at 09:50:03PM -0400, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
>  1 file changed, 192 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 10/19/20 3:50 AM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390) that can be used

"s390x" =)

> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 192 ++++++++++++++++++++++++++++++++
>   1 file changed, 192 insertions(+)

Re: [PATCH v4 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

On Mon, Oct 19, 2020 at 12:42:03PM +0200, Philippe Mathieu-Daudé wrote:
> On 10/19/20 3:50 AM, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390) that can be used
> 
> "s390x" =)
>

Sure!

Thanks!
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Mon, Oct 19, 2020 at 12:27:41PM +0200, Erik Skultety wrote:
> ...
> 
> > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> > new file mode 100644
> > index 0000000000..8bb7ba6b33
> > --- /dev/null
> > +++ b/scripts/ci/setup/inventory
> > @@ -0,0 +1,2 @@
> > +[local]
> 
> Nit pick, is a group for localhost actually needed?
>

You're right, it's not needed... I just thought it gave the
"localhost" entry some "shelter" and "context". :)

And, I think a mostly "ini-like" file without a section triggers an OCD
reaction in me.  I can remove it if it does something similar to you! :)

Thanks!
- Cleber.

> Regards,
> Erik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 6601 bytes --]

On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > To have the jobs dispatched to custom runners, gitlab-runner must
> > be installed, active as a service and properly configured.  The
> > variables file and playbook introduced here should help with those
> > steps.
> > 
> > The playbook introduced here covers a number of different Linux
> > distributions and FreeBSD, and are intended to provide a reproducible
> > environment.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> 
> In general, there's been put quite some effort into the playbooks - sorry I'm
> late to the game - is there a plan to introduce QEMU as a project to lcitool?

I think it's becoming quite clear that having so much duplication (in
the dockerfiles, tests/vm, this playbook, etc) is costly and error
prone.  I don't know if anyone has invested time in a PoC to
consolidate those (with lcitool), but I can certainly see the upside
to that.  BTW, are you volunteering (wink wink)? :)

> We've taken care of most of the bits in the playbooks that are being introduced
> and for the remaining ones I think it would be that big of an overhaul to do
> the adjustments. One major re-factor though would IMO be to break the
> dependency lcitool has on the machine naming, kind of restricting it to a
> limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which
> makes it inconvenient to prepare physical hosts.
>

Right... I wasn't aware of that depedency.  And, this may be a nice
project to make sure that lcitool doesn't have any other libvirt
specificities.

> More comments inline...
> 
> >  docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
> >  scripts/ci/setup/.gitignore        |  1 +
> >  scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
> >  scripts/ci/setup/vars.yml.template | 13 ++++++
> >  4 files changed, 149 insertions(+)
> >  create mode 100644 scripts/ci/setup/.gitignore
> >  create mode 100644 scripts/ci/setup/gitlab-runner.yml
> >  create mode 100644 scripts/ci/setup/vars.yml.template
> > 
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 208b5e399b..a234a5e24c 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -84,3 +84,66 @@ To run the playbook, execute::
> >  
> >    cd scripts/ci/setup
> >    ansible-playbook -i inventory build-environment.yml
> > +
> > +gitlab-runner setup and registration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +The gitlab-runner agent needs to be installed on each machine that
> > +will run jobs.  The association between a machine and a GitLab project
> > +happens with a registration token.  To find the registration token for
> > +your repository/project, navigate on GitLab's web UI to:
> > +
> > + * Settings (the gears like icon), then
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * Under "Set up a specific Runner manually", look for the value under
> > +   "Use the following registration token during setup"
> > +
> > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > +``scripts/ci/setup/vars.yml``.  Then, set the
> > +``gitlab_runner_registration_token`` variable to the value obtained
> > +earlier.
> > +
> > +.. note:: gitlab-runner is not available from the standard location
> > +          for all OS and architectures combinations.  For some systems,
> > +          a custom build may be necessary.  Some builds are avaiable
> > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > +          URI may be used as a value on ``vars.yml``
> 
> Yes, this can be suboptimal...Would it make sense to fall back to build the
> binary of a given version from git as a fallback during this playbook if the
> necessary arch version isn't provided the official way? Just an idea, I'd like
> to avoid the need for you to become the maintainer of the binaries and keep up
> with the releases.
>

Well, building them during the playbook would be a lot more
complex... You can have your own repo with your own builds, and just
tweak your vars.yml.

> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory gitlab-runner.yml
> > +
> > +.. note:: there are currently limitations to gitlab-runner itself when
> > +          setting up a service under FreeBSD systems.  You will need to
> > +          perform steps 4 to 10 manually, as described at
> 
> Which one of them is considered an automation problem? In lcitool we made
> gitlab-runner completely automated on all distros, including FreeBSD:
>

It looks like lcitool went the more practical route.  I was hoping to
not have to treat gitlab-runner in such a special way in any
"supported" OS.  What I mean is, I'd rather write the code within
gitlab-runner (or reespective libraries).  Of course, I did not get to
it, so that's why I just documented the steps here.

I'll take a look at lcitool's playbook to see if I can easily
canibalize some of that.  But, at this time, we don't runners for
FreeBSD anyway, so I guess this is not *that* urgent.

> 4) log file permissions - you're creating the user, you can as well create the
> file with correct permissions
> 
> 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml
> it will pull a bunch of dependencies which ensure the dir exists
> 
> 6) gitlab_runner service script should IMO provided by this automation as a
> template and install to the correct location
> 
> 7) ensure the service script is executable - template module can do that
> 
> 8) register the runner - you're doing that
> 
> 9) enable the service - Ansible's service module is generic and supports
> init/OpenRC
> 
> 10) I don't see a step 10
>

This was either a mistake or the installation steps got updated.  I'll
adjust that.

> IOW I think there should be as little manual intervention as possible and in
> this case I don't think any manual steps are needed by the user.
>

Agreed.  I was not super happy with the current state of this playbook
wrt FreeBSD, but I decided to pick other battles to fight.  Also,
newer gitlab-runner versions *may* be using the updated service
management lib, which may reduce/remove the need for custom handling.

Anyway, I'll look at what can be improved here, considering the cost.

Thanks!
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > +
> > +    - name: Checks the availability of official gitlab-runner builds in the archive
> > +      uri:
> > +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
> > +        method: HEAD
> > +        status_code:
> > +          - 200
> > +          - 403
> 
> Why is 403 an acceptable success status code?
>

I missed this question in my last reply, sorry.

s3 will throw a 403 if the URI doesn't exist... and we don't want to fail
the playbook because of that, given that we'll attempt the fallback
repo defined in vars.yml{,.template}.

> > +      register: gitlab_runner_available_archive
> > +
> > +    - name: Update base url
> > +      set_fact:
> > +        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
> > +      when: gitlab_runner_available_archive.status == 200
> > +    - debug:
> > +        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
> > +
> > +    - name: Set OS name (FreeBSD)
> > +      set_fact:
> > +        gitlab_runner_os: freebsd
> > +      when: "ansible_facts['system'] == 'FreeBSD'"
> > +
> > +    - name: Create a group for the gitlab-runner service
> > +      group:
> > +        name: gitlab-runner
> > +
> > +    - name: Create a user for the gitlab-runner service
> > +      user:
> > +        user: gitlab-runner
> > +        group: gitlab-runner
> > +        comment: GitLab Runner
> > +        home: /home/gitlab-runner
> > +        shell: /bin/bash
> > +
> > +    - name: Remove the .bash_logout file when on Ubuntu systems
> > +      file:
> > +        path: /home/gitlab-runner/.bash_logout
> > +        state: absent
> > +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> > +
> > +    - name: Downloads the matching gitlab-runner
> > +      get_url:
> > +        dest: /usr/local/bin/gitlab-runner
> > +        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> > +        owner: gitlab-runner
> > +        group: gitlab-runner
> > +        mode: u=rwx,g=rwx,o=rx
> > +

Here, the actual download is performed, and no 40x is considered OK.
I hope that explains it.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Erik Skultety]

On Mon, Oct 19, 2020 at 04:25:31PM -0400, Cleber Rosa wrote:
> On Mon, Oct 19, 2020 at 12:27:41PM +0200, Erik Skultety wrote:
> > ...
> > 
> > > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> > > new file mode 100644
> > > index 0000000000..8bb7ba6b33
> > > --- /dev/null
> > > +++ b/scripts/ci/setup/inventory
> > > @@ -0,0 +1,2 @@
> > > +[local]
> > 
> > Nit pick, is a group for localhost actually needed?
> >
> 
> You're right, it's not needed... I just thought it gave the
> "localhost" entry some "shelter" and "context". :)
> 
> And, I think a mostly "ini-like" file without a section triggers an OCD
> reaction in me.  I can remove it if it does something similar to you! :)
> 
> Thanks!
> - Cleber.

I understand, but even though it may be an ini-like config, "sections" have a
very distinct meaning in Ansible and that is grouping, so unless there are
hosts to group, I'd keep the inventory in a form of a simple list of individual
hosts.

Regards,
Erik

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Erik Skultety]

On Mon, Oct 19, 2020 at 04:41:38PM -0400, Cleber Rosa wrote:
> On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > > To have the jobs dispatched to custom runners, gitlab-runner must
> > > be installed, active as a service and properly configured.  The
> > > variables file and playbook introduced here should help with those
> > > steps.
> > > 
> > > The playbook introduced here covers a number of different Linux
> > > distributions and FreeBSD, and are intended to provide a reproducible
> > > environment.
> > > 
> > > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > ---
> > 
> > In general, there's been put quite some effort into the playbooks - sorry I'm
> > late to the game - is there a plan to introduce QEMU as a project to lcitool?
> 
> I think it's becoming quite clear that having so much duplication (in
> the dockerfiles, tests/vm, this playbook, etc) is costly and error
> prone.  I don't know if anyone has invested time in a PoC to
> consolidate those (with lcitool), but I can certainly see the upside
> to that.  BTW, are you volunteering (wink wink)? :)

I don't think I was trying to :), but sure, I can dedicate some time to it.
I'll need a bit of guidance in the QEMU world though for sure.

> 
> > We've taken care of most of the bits in the playbooks that are being introduced
> > and for the remaining ones I think it would be that big of an overhaul to do
> > the adjustments. One major re-factor though would IMO be to break the
> > dependency lcitool has on the machine naming, kind of restricting it to a
> > limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which
> > makes it inconvenient to prepare physical hosts.
> >
> 
> Right... I wasn't aware of that depedency.  And, this may be a nice
> project to make sure that lcitool doesn't have any other libvirt
> specificities.

Yeah, that one will be a major change. Thinking about this some more, while
having a QEMU project inside lcitool sounds definitely useful, it would
probably be a wasted effort wrt task priority right now, because without
expanding lcitool's range of applicable hosts I don't see how QEMU would
utilize the tool without using quirks.

> 
> > More comments inline...
> > 
> > >  docs/devel/ci.rst                  | 63 ++++++++++++++++++++++++++
> > >  scripts/ci/setup/.gitignore        |  1 +
> > >  scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++
> > >  scripts/ci/setup/vars.yml.template | 13 ++++++
> > >  4 files changed, 149 insertions(+)
> > >  create mode 100644 scripts/ci/setup/.gitignore
> > >  create mode 100644 scripts/ci/setup/gitlab-runner.yml
> > >  create mode 100644 scripts/ci/setup/vars.yml.template
> > > 
> > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > > index 208b5e399b..a234a5e24c 100644
> > > --- a/docs/devel/ci.rst
> > > +++ b/docs/devel/ci.rst
> > > @@ -84,3 +84,66 @@ To run the playbook, execute::
> > >  
> > >    cd scripts/ci/setup
> > >    ansible-playbook -i inventory build-environment.yml
> > > +
> > > +gitlab-runner setup and registration
> > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > +
> > > +The gitlab-runner agent needs to be installed on each machine that
> > > +will run jobs.  The association between a machine and a GitLab project
> > > +happens with a registration token.  To find the registration token for
> > > +your repository/project, navigate on GitLab's web UI to:
> > > +
> > > + * Settings (the gears like icon), then
> > > + * CI/CD, then
> > > + * Runners, and click on the "Expand" button, then
> > > + * Under "Set up a specific Runner manually", look for the value under
> > > +   "Use the following registration token during setup"
> > > +
> > > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > > +``scripts/ci/setup/vars.yml``.  Then, set the
> > > +``gitlab_runner_registration_token`` variable to the value obtained
> > > +earlier.
> > > +
> > > +.. note:: gitlab-runner is not available from the standard location
> > > +          for all OS and architectures combinations.  For some systems,
> > > +          a custom build may be necessary.  Some builds are avaiable
> > > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > > +          URI may be used as a value on ``vars.yml``
> > 
> > Yes, this can be suboptimal...Would it make sense to fall back to build the
> > binary of a given version from git as a fallback during this playbook if the
> > necessary arch version isn't provided the official way? Just an idea, I'd like
> > to avoid the need for you to become the maintainer of the binaries and keep up
> > with the releases.
> >
> 
> Well, building them during the playbook would be a lot more
> complex... You can have your own repo with your own builds, and just
> tweak your vars.yml.

I didn't want to suggest this as a replacement to vars.yml.
What if people wanting to run the playbook don't have their own repo and you
(I mean you personally) are not providing a fresh build of gitlab-runner for
the given architecture?
The automation machinery would surely be more complex, but at the same time,
I can image a dedicated playbook that would handle gitlab-runner building and
installation of the binary to the correct place, you'd then simply include that
playbook if vars.yml don't specify an alternative repo and download from AWS
failed.
Another thing to consider - you currently host the missing arch builds under
your personal namespace on fedorapeople, what if you decide one day you're not
interested in being involved in QEMU CI that much and suddenly you stop
building the binaries, someone has to take over. On the other hand, if you
automate this whole process, having exposed the builds would definitely not
slow down machine provisions, but would also not cause them to fail if nobody
exposes them.

> 
> > > +
> > > +To run the playbook, execute::
> > > +
> > > +  cd scripts/ci/setup
> > > +  ansible-playbook -i inventory gitlab-runner.yml
> > > +
> > > +.. note:: there are currently limitations to gitlab-runner itself when
> > > +          setting up a service under FreeBSD systems.  You will need to
> > > +          perform steps 4 to 10 manually, as described at
> > 
> > Which one of them is considered an automation problem? In lcitool we made
> > gitlab-runner completely automated on all distros, including FreeBSD:
> >
> 
> It looks like lcitool went the more practical route.  I was hoping to
> not have to treat gitlab-runner in such a special way in any
> "supported" OS.  What I mean is, I'd rather write the code within
> gitlab-runner (or reespective libraries).  Of course, I did not get to
> it, so that's why I just documented the steps here.
> 
> I'll take a look at lcitool's playbook to see if I can easily
> canibalize some of that.  But, at this time, we don't runners for
> FreeBSD anyway, so I guess this is not *that* urgent.

Okay, I was simply curious. If it's not something you don't need at the moment,
then let's not focus on it.

> 
> > 4) log file permissions - you're creating the user, you can as well create the
> > file with correct permissions
> > 
> > 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml
> > it will pull a bunch of dependencies which ensure the dir exists
> > 
> > 6) gitlab_runner service script should IMO provided by this automation as a
> > template and install to the correct location
> > 
> > 7) ensure the service script is executable - template module can do that
> > 
> > 8) register the runner - you're doing that
> > 
> > 9) enable the service - Ansible's service module is generic and supports
> > init/OpenRC
> > 
> > 10) I don't see a step 10
> >
> 
> This was either a mistake or the installation steps got updated.  I'll
> adjust that.
> 
> > IOW I think there should be as little manual intervention as possible and in
> > this case I don't think any manual steps are needed by the user.
> >
> 
> Agreed.  I was not super happy with the current state of this playbook
> wrt FreeBSD, but I decided to pick other battles to fight.  Also,
> newer gitlab-runner versions *may* be using the updated service
> management lib, which may reduce/remove the need for custom handling.

What do you mean by "service management lib"?

Regards,
Erik

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Erik Skultety]

On Mon, Oct 19, 2020 at 04:54:50PM -0400, Cleber Rosa wrote:
> On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > > +
> > > +    - name: Checks the availability of official gitlab-runner builds in the archive
> > > +      uri:
> > > +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
> > > +        method: HEAD
> > > +        status_code:
> > > +          - 200
> > > +          - 403
> > 
> > Why is 403 an acceptable success status code?
> >
> 
> I missed this question in my last reply, sorry.
> 
> s3 will throw a 403 if the URI doesn't exist... and we don't want to fail
> the playbook because of that, given that we'll attempt the fallback
> repo defined in vars.yml{,.template}.

Right, forgot about the repo in vars.yml fallback, thanks for explaining.

Regards,
Erik

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Daniel P. Berrangé]

On Tue, Oct 20, 2020 at 08:58:39AM +0200, Erik Skultety wrote:
> On Mon, Oct 19, 2020 at 04:41:38PM -0400, Cleber Rosa wrote:
> > On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote:
> > > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote:
> > > > To have the jobs dispatched to custom runners, gitlab-runner must
> > > > be installed, active as a service and properly configured.  The
> > > > variables file and playbook introduced here should help with those
> > > > steps.
> > > > 
> > > > The playbook introduced here covers a number of different Linux
> > > > distributions and FreeBSD, and are intended to provide a reproducible
> > > > environment.
> > > > 
> > > > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > 
> > > In general, there's been put quite some effort into the playbooks - sorry I'm
> > > late to the game - is there a plan to introduce QEMU as a project to lcitool?
> > 
> > I think it's becoming quite clear that having so much duplication (in
> > the dockerfiles, tests/vm, this playbook, etc) is costly and error
> > prone.  I don't know if anyone has invested time in a PoC to
> > consolidate those (with lcitool), but I can certainly see the upside
> > to that.  BTW, are you volunteering (wink wink)? :)
> 
> I don't think I was trying to :), but sure, I can dedicate some time to it.
> I'll need a bit of guidance in the QEMU world though for sure.

I think the obvious and easy place is start using lcitool is for the
tests/docker/dockerfiles/*.  All that's required is to add mappings
to lcitool for the various deps that QEMU has which libvirt does not
already have. Then we should be able to start auto-generating the
dockerfiles without too much difficulty. This will be a significant
step forward because it will help us keep te package lists in sync
across all the dockerfiles which is a major fail in QEMU right now.


Dealing with tests/vm replacement or these ansible recipes is likely
to be a significantly more challenging proposition. Perhaps we can
again start by just automating creation of the package lists that
the tests/vm and ansibile recipes need, as again those are all
inconsistent.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Andrea Bolognani]

On Sun, 2020-10-18 at 21:50 -0400, Cleber Rosa wrote:
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,233 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:

My Ansible-fu is a bit rusty at the moment, so please double-check my
claims and apologies in advance for the ones that I will get wrong O:-)

> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      apt:
> +        update_cache: yes
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +        pkg:
> +          - ccache
> +          - clang

Instead of using the 'apt' module here, and the equivalent module
further down, you could just do

  package:
    name:
      - pkg1
      - pkg2
        ...
    state: present

every single time and let Ansible take care of the differences for
you.

> +      when: "ansible_facts['distribution'] == 'Ubuntu'"

Quoting the entire condition is not necessary, you can just have

  when: ansible_facts['distribution'] == 'Ubuntu'

or, my preference,

  when:
    - ansible_facts['distribution'] == 'Ubuntu'

which results in a nicer diff when you add/remove conditions.

> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      apt:
> +        update_cache: yes
> +        pkg:
> +         - libspice-server-dev
> +         - libxen-dev

Indentation of list items is inconsistent here.

> +    - name: Install basic packages to build QEMU on FreeBSD 12.x
> +      pkgng:
> +        # Originally from packages on .cirrus.yml under the freebsd_12_task
> +        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir

See above for 'pkgng' vs 'package', but at the very least this should
be

  pkgng:
    name:
      - bash
      - curl
        ...

or each time the list is touched the resulting diff is going to be
unmanageable.

> +    - name: Enable PowerTools repo on CentOS 8
> +      ini_file:
> +        path: /etc/yum.repos.d/CentOS-PowerTools.repo
> +        section: PowerTools
> +        option: enabled
> +        value: "1"
> +      when:
> +        - "ansible_facts['distribution'] == 'CentOS'"
> +        - "ansible_facts['distribution_major_version'] == '8'"

Another option would be to use

  command: 'dnf config-manager --set-enabled Stream-PowerTools -y'
  args:
    warn: no

but I have to admit the way you're doing it is very clever ;)

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH v4 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Andrea Bolognani]

On Tue, 2020-10-20 at 09:29 +0100, Daniel P. Berrangé wrote:
> I think the obvious and easy place is start using lcitool is for the
> tests/docker/dockerfiles/*.  All that's required is to add mappings
> to lcitool for the various deps that QEMU has which libvirt does not
> already have. Then we should be able to start auto-generating the
> dockerfiles without too much difficulty. This will be a significant
> step forward because it will help us keep te package lists in sync
> across all the dockerfiles which is a major fail in QEMU right now.

I took a quick look at the contents of those dockerfiles and while I
agree that it's the obvious place to start, I'm not sure I would say
that generating them using lcitool is going to be easy :)

Basically there seem to be a number of assumptions both in lcitool
and in the QEMU dockerfiles, and getting the two to agree on those
will require quite some work from what I can tell.

But again, I agree it's something that should be worked on.

> Dealing with tests/vm replacement or these ansible recipes is likely
> to be a significantly more challenging proposition. Perhaps we can
> again start by just automating creation of the package lists that
> the tests/vm and ansibile recipes need, as again those are all
> inconsistent.

Perhaps 'lcitool variables' could be of use here.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH v4 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Thomas Huth]

On 19/10/2020 03.50, Cleber Rosa wrote:
> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.
> 
> Those jobs are intended to run on hardware and/or Operating Systems
> not provided by GitLab's shared runners.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 14 +++++++++
>  .gitlab-ci.yml                  |  1 +
>  docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
>  docs/devel/index.rst            |  1 +
>  4 files changed, 70 insertions(+)
>  create mode 100644 .gitlab-ci.d/custom-runners.yml
>  create mode 100644 docs/devel/ci.rst
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> new file mode 100644
> index 0000000000..3004da2bda
> --- /dev/null
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -0,0 +1,14 @@
> +# The CI jobs defined here require GitLab runners installed and
> +# registered on machines that match their operating system names,
> +# versions and architectures.  This is in contrast to the other CI
> +# jobs that are intended to run on GitLab's "shared" runners.
> +
> +# Different than the default approach on "shared" runners, based on
> +# containers, the custom runners have no such *requirement*, as those
> +# jobs should be capable of running on operating systems with no
> +# compatible container implementation, or no support from
> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> +# reusing the GIT repository, let's enable the recursive submodule
> +# strategy.
> +variables:
> +  GIT_SUBMODULE_STRATEGY: recursive
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 8ffd415ca5..b33c433fd7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -18,6 +18,7 @@ include:
>    - local: '/.gitlab-ci.d/opensbi.yml'
>    - local: '/.gitlab-ci.d/containers.yml'
>    - local: '/.gitlab-ci.d/crossbuilds.yml'
> +  - local: '/.gitlab-ci.d/custom-runners.yml'
>  
>  .native_build_job_template: &native_build_job_definition
>    stage: build
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> new file mode 100644
> index 0000000000..41a4bbddad
> --- /dev/null
> +++ b/docs/devel/ci.rst
> @@ -0,0 +1,54 @@
> +==
> +CI
> +==
> +
> +QEMU has configurations enabled for a number of different CI services.
> +The most up to date information about them and their status can be
> +found at::
> +
> +   https://wiki.qemu.org/Testing/CI
> +
> +Jobs on Custom Runners
> +======================
> +
> +Besides the jobs run under the various CI systems listed before, there
> +are a number additional jobs that will run before an actual merge.
> +These use the same GitLab CI's service/framework already used for all
> +other GitLab based CI jobs, but rely on additional systems, not the
> +ones provided by GitLab as "shared runners".
> +
> +The architecture of GitLab's CI service allows different machines to
> +be set up with GitLab's "agent", called gitlab-runner, which will take
> +care of running jobs created by events such as a push to a branch.
> +Here, the combination of a machine, properly configured with GitLab's
> +gitlab-runner, is called a "custom runner" here.

Nit: Remove one of the two "here" in the above sentence.

> +The GitLab CI jobs definition for the custom runners are located under::
> +
> +  .gitlab-ci.d/custom-runners.yml
> +
> +Current Jobs
> +------------
> +
> +The current CI jobs based on custom runners have the primary goal of
> +catching and preventing regressions on a wider number of host systems
> +than the ones provided by GitLab's shared runners.
> +
> +Also, the mechanics of reliability, capacity and overall maintanance

s/maintanance/maintenance/

> +of the machines provided by the QEMU project itself for those jobs
> +will be evaluated.

I'm not sure what this sentence is really good for... of course new stuff
has to prove its usefulness first, but that's always the case and does not
need to be mentioned in the documentation, I think? ... maybe that sentence
is better something for the patch description instead of (hopefully)
long-lasting documentation here?

> +Future Plans and Jobs
> +---------------------
> +
> +Once the CI Jobs based on custom runners have been proved mature with
> +the initial set of jobs run on machines from the QEMU project, other
> +members in the community should be able provide their own machine
> +configuration documentation/scripts, and accompanying job definitions.
> +
> +As a general rule, those newly added contributed jobs should run as
> +"non-gating", until their reliability is verified.
> +
> +The precise minimum requirements and exact rules for machine
> +configuration documentation/scripts, and the success rate of jobs are
> +still to be defined.
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index 77baae5c77..2fdd36e751 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -21,6 +21,7 @@ Contents:
>     atomics
>     stable-process
>     testing
> +   ci
>     qtest
>     decodetree
>     secure-coding-practices
> 

 Thomas

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Thomas Huth]

On 19/10/2020 03.50, Cleber Rosa wrote:
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
[...]
> +    - name: Enable PowerTools repo on CentOS 8
> +      ini_file:
> +        path: /etc/yum.repos.d/CentOS-PowerTools.repo
> +        section: PowerTools
> +        option: enabled
> +        value: "1"
> +      when:
> +        - "ansible_facts['distribution'] == 'CentOS'"
> +        - "ansible_facts['distribution_major_version'] == '8'"
> +
> +    - name: Install basic packages to build QEMU on CentOS 8
> +      dnf:
> +        # Originally from tests/docker/dockerfiles/centos8.docker
> +        name:
> +          - SDL-devel

We do not support SDL1 in QEMU anymore, so this should be SDL2-devel now.
Yes, we've also got it wrong in the docker files ... I'll send a patch to
fix it there.

 Thomas

Re: [PATCH v4 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 5100 bytes --]

On Wed, Oct 21, 2020 at 08:45:18AM +0200, Thomas Huth wrote:
> On 19/10/2020 03.50, Cleber Rosa wrote:
> > As described in the included documentation, the "custom runner" jobs
> > extend the GitLab CI jobs already in place.
> > 
> > Those jobs are intended to run on hardware and/or Operating Systems
> > not provided by GitLab's shared runners.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/custom-runners.yml | 14 +++++++++
> >  .gitlab-ci.yml                  |  1 +
> >  docs/devel/ci.rst               | 54 +++++++++++++++++++++++++++++++++
> >  docs/devel/index.rst            |  1 +
> >  4 files changed, 70 insertions(+)
> >  create mode 100644 .gitlab-ci.d/custom-runners.yml
> >  create mode 100644 docs/devel/ci.rst
> > 
> > diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> > new file mode 100644
> > index 0000000000..3004da2bda
> > --- /dev/null
> > +++ b/.gitlab-ci.d/custom-runners.yml
> > @@ -0,0 +1,14 @@
> > +# The CI jobs defined here require GitLab runners installed and
> > +# registered on machines that match their operating system names,
> > +# versions and architectures.  This is in contrast to the other CI
> > +# jobs that are intended to run on GitLab's "shared" runners.
> > +
> > +# Different than the default approach on "shared" runners, based on
> > +# containers, the custom runners have no such *requirement*, as those
> > +# jobs should be capable of running on operating systems with no
> > +# compatible container implementation, or no support from
> > +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> > +# reusing the GIT repository, let's enable the recursive submodule
> > +# strategy.
> > +variables:
> > +  GIT_SUBMODULE_STRATEGY: recursive
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 8ffd415ca5..b33c433fd7 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -18,6 +18,7 @@ include:
> >    - local: '/.gitlab-ci.d/opensbi.yml'
> >    - local: '/.gitlab-ci.d/containers.yml'
> >    - local: '/.gitlab-ci.d/crossbuilds.yml'
> > +  - local: '/.gitlab-ci.d/custom-runners.yml'
> >  
> >  .native_build_job_template: &native_build_job_definition
> >    stage: build
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > new file mode 100644
> > index 0000000000..41a4bbddad
> > --- /dev/null
> > +++ b/docs/devel/ci.rst
> > @@ -0,0 +1,54 @@
> > +==
> > +CI
> > +==
> > +
> > +QEMU has configurations enabled for a number of different CI services.
> > +The most up to date information about them and their status can be
> > +found at::
> > +
> > +   https://wiki.qemu.org/Testing/CI
> > +
> > +Jobs on Custom Runners
> > +======================
> > +
> > +Besides the jobs run under the various CI systems listed before, there
> > +are a number additional jobs that will run before an actual merge.
> > +These use the same GitLab CI's service/framework already used for all
> > +other GitLab based CI jobs, but rely on additional systems, not the
> > +ones provided by GitLab as "shared runners".
> > +
> > +The architecture of GitLab's CI service allows different machines to
> > +be set up with GitLab's "agent", called gitlab-runner, which will take
> > +care of running jobs created by events such as a push to a branch.
> > +Here, the combination of a machine, properly configured with GitLab's
> > +gitlab-runner, is called a "custom runner" here.
> 
> Nit: Remove one of the two "here" in the above sentence.
> 
> > +The GitLab CI jobs definition for the custom runners are located under::
> > +
> > +  .gitlab-ci.d/custom-runners.yml
> > +
> > +Current Jobs
> > +------------
> > +
> > +The current CI jobs based on custom runners have the primary goal of
> > +catching and preventing regressions on a wider number of host systems
> > +than the ones provided by GitLab's shared runners.
> > +
> > +Also, the mechanics of reliability, capacity and overall maintanance
> 
> s/maintanance/maintenance/
>

Oopsie... thanks.

> > +of the machines provided by the QEMU project itself for those jobs
> > +will be evaluated.
> 
> I'm not sure what this sentence is really good for... of course new stuff
> has to prove its usefulness first, but that's always the case and does not
> need to be mentioned in the documentation, I think? ... maybe that sentence
> is better something for the patch description instead of (hopefully)
> long-lasting documentation here?
>

I think the statement attempts to set the tone here, and answer a
question that I've seen more than once...  It certainly feels a bit
out of place here, but I also think the commit message would be too
transient and almost invisible.  I expect a lot will change in the CI
and in its docs, so I don't think this exact sentence will be too long
lasting.

Let me know if that convinces you it deserves to be in the docs, if not
I'll gladly moved it to the commit message.

Thanks!
- Cleber

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Tue, Oct 20, 2020 at 08:18:54AM +0200, Erik Skultety wrote:
> On Mon, Oct 19, 2020 at 04:25:31PM -0400, Cleber Rosa wrote:
> > On Mon, Oct 19, 2020 at 12:27:41PM +0200, Erik Skultety wrote:
> > > ...
> > > 
> > > > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> > > > new file mode 100644
> > > > index 0000000000..8bb7ba6b33
> > > > --- /dev/null
> > > > +++ b/scripts/ci/setup/inventory
> > > > @@ -0,0 +1,2 @@
> > > > +[local]
> > > 
> > > Nit pick, is a group for localhost actually needed?
> > >
> > 
> > You're right, it's not needed... I just thought it gave the
> > "localhost" entry some "shelter" and "context". :)
> > 
> > And, I think a mostly "ini-like" file without a section triggers an OCD
> > reaction in me.  I can remove it if it does something similar to you! :)
> > 
> > Thanks!
> > - Cleber.
> 
> I understand, but even though it may be an ini-like config, "sections" have a
> very distinct meaning in Ansible and that is grouping, so unless there are
> hosts to group, I'd keep the inventory in a form of a simple list of individual
> hosts.
>

Fair enough.  Changing it for v5.

Thanks!
- Cleber.

> Regards,
> Erik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 4317 bytes --]

On Tue, Oct 20, 2020 at 07:52:43PM +0200, Andrea Bolognani wrote:
> On Sun, 2020-10-18 at 21:50 -0400, Cleber Rosa wrote:
> > +++ b/scripts/ci/setup/build-environment.yml
> > @@ -0,0 +1,233 @@
> > +---
> > +- name: Installation of basic packages to build QEMU
> > +  hosts: all
> > +  tasks:
> 
> My Ansible-fu is a bit rusty at the moment, so please double-check my
> claims and apologies in advance for the ones that I will get wrong O:-)
> 
> > +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> > +      apt:
> > +        update_cache: yes
> > +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> > +        pkg:
> > +          - ccache
> > +          - clang
> 
> Instead of using the 'apt' module here, and the equivalent module
> further down, you could just do
> 
>   package:
>     name:
>       - pkg1
>       - pkg2
>         ...
>     state: present
> 
> every single time and let Ansible take care of the differences for
> you.
>

I'm almost sure that this was a conscious decision.  I remeber it had
to do with not being able to set `update_cache`, and failures on
recently installed systems and containers that did not update the APT
cache.  There may be something else, but I'll have to give it another
round of testing.

FIY, under the hood, package is not really a module, but an action
plugin that forwards these very limited options to the set or detected
package manager, so it brings uniformity in the playbook, but limits
the control too.  IMO, it's very low impact to leave it AS IS.

> > +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> 
> Quoting the entire condition is not necessary, you can just have
> 
>   when: ansible_facts['distribution'] == 'Ubuntu'
> 
> or, my preference,
> 
>   when:
>     - ansible_facts['distribution'] == 'Ubuntu'
>

Yep, I've used the explicit lists when there was more than one
condition, but having a standard style is better indeed.

> which results in a nicer diff when you add/remove conditions.
> 
> > +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> > +      apt:
> > +        update_cache: yes
> > +        pkg:
> > +         - libspice-server-dev
> > +         - libxen-dev
> 
> Indentation of list items is inconsistent here.
>

True.  Fixed, thanks!

> > +    - name: Install basic packages to build QEMU on FreeBSD 12.x
> > +      pkgng:
> > +        # Originally from packages on .cirrus.yml under the freebsd_12_task
> > +        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir
>
> See above for 'pkgng' vs 'package', but at the very least this should
> be
> 
>   pkgng:
>     name:
>       - bash
>       - curl
>         ...
> 
> or each time the list is touched the resulting diff is going to be
> unmanageable.
>

The documentation suggests a comma separated list of package names:

   https://docs.ansible.com/ansible/2.8/modules/pkgng_module.html#pkgng-module

And the reason is that this module is not as smart as others, and will
run one separate command for each individual package name value:

   https://github.com/ansible/ansible/blob/v2.10.3/test/support/integration/plugins/modules/pkgng.py#L214

It's a tradeoff indeed, but I think we're aligned with the docs.

> > +    - name: Enable PowerTools repo on CentOS 8
> > +      ini_file:
> > +        path: /etc/yum.repos.d/CentOS-PowerTools.repo
> > +        section: PowerTools
> > +        option: enabled
> > +        value: "1"
> > +      when:
> > +        - "ansible_facts['distribution'] == 'CentOS'"
> > +        - "ansible_facts['distribution_major_version'] == '8'"
> 
> Another option would be to use
> 
>   command: 'dnf config-manager --set-enabled Stream-PowerTools -y'
>   args:
>     warn: no
> 
> but I have to admit the way you're doing it is very clever ;)
>

Yeah, that would require another package to be installed, and then the
command executed... So I think this is cheaper and eaiser indeed.

> -- 
> Andrea Bolognani / Red Hat / Virtualization
> 

Thanks for the review, I'll report on the additional points as soon as
I test them.  If appropriate, I'll put notes on the v5.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Wed, Oct 21, 2020 at 09:00:27AM +0200, Thomas Huth wrote:
> On 19/10/2020 03.50, Cleber Rosa wrote:
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> > 
> > The playbook introduced here covers a number of different Linux
> > distributions and FreeBSD, and are intended to provide a reproducible
> > environment.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> [...]
> > +    - name: Enable PowerTools repo on CentOS 8
> > +      ini_file:
> > +        path: /etc/yum.repos.d/CentOS-PowerTools.repo
> > +        section: PowerTools
> > +        option: enabled
> > +        value: "1"
> > +      when:
> > +        - "ansible_facts['distribution'] == 'CentOS'"
> > +        - "ansible_facts['distribution_major_version'] == '8'"
> > +
> > +    - name: Install basic packages to build QEMU on CentOS 8
> > +      dnf:
> > +        # Originally from tests/docker/dockerfiles/centos8.docker
> > +        name:
> > +          - SDL-devel
> 
> We do not support SDL1 in QEMU anymore, so this should be SDL2-devel now.
> Yes, we've also got it wrong in the docker files ... I'll send a patch to
> fix it there.
> 
>  Thomas

Nice catch.  I see the dockerfile changes are already merged, so
updating them here.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/4] Jobs based on custom runners: build environment docs and playbook

[Andrea Bolognani]

On Mon, 2020-11-09 at 11:37 -0500, Cleber Rosa wrote:
> On Tue, Oct 20, 2020 at 07:52:43PM +0200, Andrea Bolognani wrote:
> > On Sun, 2020-10-18 at 21:50 -0400, Cleber Rosa wrote:
> > > +    - name: Install basic packages to build QEMU on FreeBSD 12.x
> > > +      pkgng:
> > > +        # Originally from packages on .cirrus.yml under the freebsd_12_task
> > > +        name: bash,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,ninja,perl5,pixman,pkgconf,png,usbredir
> > 
> > See above for 'pkgng' vs 'package', but at the very least this should
> > be
> > 
> >   pkgng:
> >     name:
> >       - bash
> >       - curl
> >         ...
> > 
> > or each time the list is touched the resulting diff is going to be
> > unmanageable.
> 
> The documentation suggests a comma separated list of package names:
> 
>    https://docs.ansible.com/ansible/2.8/modules/pkgng_module.html#pkgng-module
> 
> And the reason is that this module is not as smart as others, and will
> run one separate command for each individual package name value:
> 
>    https://github.com/ansible/ansible/blob/v2.10.3/test/support/integration/plugins/modules/pkgng.py#L214
> 
> It's a tradeoff indeed, but I think we're aligned with the docs.

I would probably take the performance hit over having to deal with an
unmaintainable blob. As a rule of thumb, reviewer time is more
valuable than machine time.

If the performance hit turns out to be unacceptably big, you could at
least do something along the lines of

  - set_fact:
      freebsd_packages:
        - bash
        - curl
          ...
  
  - pkgng:
      name: "{ freebsd_packages | join(',') }"

to keep things reviewable.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH v4 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Thomas Huth]

On 19/10/2020 03.49, Cleber Rosa wrote:
> TL;DR: this should allow the QEMU maintainer to push to the staging
> branch, and have custom jobs running on the project's aarch64 and
> s390x machines.  Simple usage looks like:
> 
>     git push remote staging
>     ./scripts/ci/gitlab-pipeline-status --verbose --wait

  Hi Cleber,

since the QEMU project is now using the gitlab repo as main repository, it 
should now be possible to finally enable these custom runners, too, I think.

Could you elaborate on what's left to do to get the s390x and aarch64 
runners enabled, and rebase your patches if necessary?

  Thanks,
   Thomas

Re: [PATCH v4 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Thu, Jan 28, 2021 at 12:51:20PM +0100, Thomas Huth wrote:
> On 19/10/2020 03.49, Cleber Rosa wrote:
> > TL;DR: this should allow the QEMU maintainer to push to the staging
> > branch, and have custom jobs running on the project's aarch64 and
> > s390x machines.  Simple usage looks like:
> > 
> >     git push remote staging
> >     ./scripts/ci/gitlab-pipeline-status --verbose --wait
> 
>  Hi Cleber,
> 
> since the QEMU project is now using the gitlab repo as main repository, it
> should now be possible to finally enable these custom runners, too, I think.
>

I couldn't agree more!

> Could you elaborate on what's left to do to get the s390x and aarch64
> runners enabled, and rebase your patches if necessary?
>

Basically what's needed is to enable the jobs, meaning committing the
relevant snippets to the "gitlab-ci.{yml,d/*}" files.

Both ansible playbooks have been run on both machines, so the gitlab-runner
and tooling to build QEMU are in place.  The gitlab-runners have also
been registered on the "qemu-project/qemu" GitLab repo.

I'll send a rebased/updated version of the playbooks, gitlab-ci.yml
configurations, etc.  In it, I'll propose that we enable the jobs, but
ignore errors for now (allow_failure: true), because it'll ease the
review.  Once the jobs prove mature, waiving of failures can be
removed.

How does that sound?

Thanks,
- Cleber.

>  Thanks,
>   Thomas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Thomas Huth]

On 03/02/2021 22.06, Cleber Rosa wrote:
> On Thu, Jan 28, 2021 at 12:51:20PM +0100, Thomas Huth wrote:
[...]
>> Could you elaborate on what's left to do to get the s390x and aarch64
>> runners enabled, and rebase your patches if necessary
> 
> Basically what's needed is to enable the jobs, meaning committing the
> relevant snippets to the "gitlab-ci.{yml,d/*}" files.
> 
> Both ansible playbooks have been run on both machines, so the gitlab-runner
> and tooling to build QEMU are in place.  The gitlab-runners have also
> been registered on the "qemu-project/qemu" GitLab repo.
> 
> I'll send a rebased/updated version of the playbooks, gitlab-ci.yml
> configurations, etc.  In it, I'll propose that we enable the jobs, but
> ignore errors for now (allow_failure: true), because it'll ease the
> review.  Once the jobs prove mature, waiving of failures can be
> removed.
> 
> How does that sound?

Sounds good, please go ahead!

  Thanks,
   Thomas