[PATCH v7 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[PATCH v7 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Cleber Rosa]

TL;DR: this should allow the QEMU maintainer to push to the staging
branch, and have custom jobs running on the project's aarch64 and
s390x machines.  Jobs in this version are allowed to fail, to allow
for the inclusion of the novel machines/jobs without CI disruption.
Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an
RFC[1].  Since then, it was decided that a simpler version should be
attempted first.

At this point, there are two specific runners (an aarch64 and an s390x)
registered with GitLab, at https://gitlab.com/qemu-project, currently
setup to the "qemu" repository.  To make it extra clear, the following
points deserve notice:

 - This work dealt with two different QEMU project machines:
    I) s390x running Ubuntu 18.04
   II) aarch64 running Ubuntu 20.04

 - All CI jobs introduced here are allowed to fail.  It should have
   no impact on the overall PASS/FAIL result of the pipeline.

 - Both machines have already been completely configured used the code
   on this series.  No further action (besides pushing this code to
   the staging branch) is needed to have CI jobs executed on them.

 - The actual CI jobs to be executed are defined in this series,
   and map as best as possible the tests run by Peter Maydell on
   the staging branch.  Peter, given the time delta since the
   beginning of this work, it may be needed to further tweak
   them.

 - The actual results of CI jobs run from these definitions are
   probably more fluid than the results from the CI jobs run on the
   shared runners.  Both changes to the code being tested, and the
   conditions/resources of the machine should be taken into account.

 - A pipeline, and some of the jobs, can be seen on the links bellow.
   Besides successful jobs, it also contains jobs not started (set to
   manual, with the specific reasons noted later on the changes
   section):

   * https://gitlab.com/cleber.gnu/qemu/-/pipelines/316471691
    - ubuntu-18.04-s390x-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698118
    - ubuntu-20.04-aarch64-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698124

Changes from v6:

 - Added minimum ansible version check to the build-environment.yml
   playbook, and a note to the documentation about the same
   requirement.

 - Removed FreeBSD from the list of covered systems in the
   documentation, and added a note saying that other systems can be
   supported by extending the current playbook.

 - Added note to documentation about required privileges to execute the
   playbook's tasks, and ansible-playbook options that can help with
   that.

 - Added a "apt upgrade" along with the "apt cache update" to reduce
   possible conflicts during package installation.

 - Removed note about custom builds of runners host at
   https://cleber.fedorapeople.org/gitlab-runner/, now that all
   architectures covered here have official builds.

 - Added more detailed instructions to the documentation on where to
   find the CI/CD settings.

 - Updated the header for the runner token page matching the current
   GitLab UI.

 - Added inventory to .gitignore.

 - Added missing newline to .gitignore.

 - Added comments with descriptions to both playbook files.

 - Added note about contacts for machine maintainers section on the
   QEMU wiki (https://wiki.qemu.org/AdminContacts).

Changes from v5:

 - Moved include of ".gitlab-ci.d/custom-runners.yml" from
   ".gitlab-ci.yml" to ".gitlab-ci.d/qemu-project.yml"

 - Changed git clone strategy from "submodule recursive" to "clone",
   to guarantee a fresh and clean repo on every job, and let
   QEMU handle eventual needed recursive submodule operations

 - Require user to create an Ansible inventory file, based on the
   now provided "inventory.template" one.  Previously, the
   "iventory" file itself was provided and users were asked to edit
   it.

 - Registered runners will now be given a default set of tags, with
   their OS and architectures.  This, besides automating another step,
   works around a "gitlab-runner register" command line issue with
   "--run-untagged=false" not being respected if no tags are given.

 - Added conditional for installing either "clang" or "clang-10" to
   match the package name on different versions of Ubuntu.

 - Changed Ubuntu 20.04 jobs to use clang-10 instead of clang.

 - Defaults to not running untagged jobs when registering a gitlab
   runner.

 - Added python3-sphinx-rtd-theme package installation to
   build-environment.yml, to match 73e6aec6522.

 - Added genisoimage package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match 7e86e5d5ccc.

 - Added liblttng-ust-dev package installation to
   build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match
   8e9419b7902.

 - Added libslirp-dev package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match 8e9419b7902.

 - Added netcat-openbsd package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match c4cb1c9f2e1.

 - Bumped gitlab-runner version to 13.12.0

 - Use only gitlab-runner binaries from the official distribution (they
   are now also available for s390x, previously missing).

 - Selection of the OS and architecture for the gitlab-runner binary
   is now done automatically.  If necessary, architecture chosen can be
   influenced by editing the "ansible_to_gitlab_arch" values on vars.yml
   (after you've created one from the vars.yml.template).

 - Marked job "ubuntu-18.04-s390x-clang" as manual, given that
   the latest executions are getting stuck with the last
   output being "Sanitizers are enabled ==> Not running the
   qemu-iotests."

 - Marked job "ubuntu-18.04-s390x-notcg" as manual, given that the
   latest executions are getting stuck with the last tests being
   "tests/qtest/pxe-test" and "tests/qtest/boot-serial-test" which
   contains commands lines with "-accel kvm -accel tcg" and "-accel
   qtest".  Obviously tcg is disabled and KVM is not available on the
   QEMU s390x machine, and the fallback to qtest gets the tests stuck.

 - Marked job "ubuntu-20.04-aarch64-notcg" as manual due to build
   failures (FAILED: libqemu-aarch64-softmmu.fa.p/target_arm_psci.c.o).

 - Marked job "ubuntu-20.04-aarch64-clang" as manual due to test
   failures:
      ../tests/qtest/libqtest.c:157: kill_qemu() tried to terminate
      QEMU process but encountered exit status 1 (expected 0)
      ERROR qtest-aarch64/qom-test - too few tests run (expected 80, got 6)

 - Cleared Daniel's R-B due to the number of changes in the
   corresponding patches.

Known issues:

 - tests/qtest/qos-test has been seen stuck a number of times on both
   the s390x and aarch64 machines.  Further investigation is needed.
   While a resolution is found, this should not affect the pipeline,
   given that all new jobs introduced here are currently allowed to
   fail.

Next development / discussion points:

 - Test docker executor
 - Test docker executor with podman
 - Configuration of more accurate job timeouts
 - Historical result analysis
 - Promotion of jobs (removing the "allow failure" setting)

Changes from v4:

 - Fixed typo in docs/devel/ci.rst, s/maintanance/maintenance/ (Thomas)
 - Removed "[local]" group from inventory file (Erik)
 - Removed sections from the playbooks which *would* be applied on
   hardware/OS that are currently not available to QEMU
 - Removed duplicated "here" on documentation (Thomas)
 - Moved description of current jobs, and possible direction of future
   jobs to the patch description (Thomas)
 - Remove comments around "when" conditions (Andrea)
 - Switch to always use explicit lists on "when" blocks (Andrea)
 - Switch from using module "apt" to using generic action module "package",
   which involved adding a new task to update the apt cache (Andrea)
 - Fix playbook indentation in the non-s390x package installation task (Andrea)
 - Changed gitlab-runner tags examples from FreeBSD to Ubuntu, which is
   covered by jobs added on this version
 - Fixed typo in commit message s/s390/s390x/ (Phil)
 - Allow all custom-runner jobs to fail at this time
 - Cleared "Reviewed-by" in one patch due to large changes

  Changes requested in v4 but *not* seen here due to sections of the
  playbook being removed:

 - Replace SDL-devel for SDL2-devel on CentOS, according to 5ed7ca3 (Thomas)
 - Correct missing step 10 on the FreeBSD gitlab-runner installation
   instructions (Erik)

Changes from v3:

- Applied changes to match <20201014135416.1290679-1-pbonzini@redhat.com>,
  that is, added ninja-build to "build-environment.yml" list of packages
  and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners",
  given that the other jobs running on shared runners are also
  considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
 * update -> up to date (Erik)
 * a different set of CI jobs -> different CI jobs (Erik)
 * Pull requests will only be merged -> code will only be merged (Stefan)
 * Setup -> set up (Stefan)
 * them -> they (Stefan)
 * the -> where the (Stefan)
 * dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of
  the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that
  no external variable is used there

- Updated package list in "build-environment.yml" from current
  dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30),
  and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts"
  into (Phillipe):
  1) Basic documentation and configuration (gitlab-ci.yml) placeholder
  2) Playbooks for setting up a build environment
  3) Playbooks for setting up gitlab-runner
  4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine
  build and test.

- Set custom jobs to not depend on any other job, so they can start
  right away.

- Set rules for starting jobs so that all pushing to any branch that
  start with name "staging".  This allows the project maintainer to
  use the "push to staging" workflow, while also allowing others to
  generate similar jobs.  If this project has configured custom
  runners, the jobs will run, if not, the pipeline will be marked as
  "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern
  (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available
  (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available
  (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in
  ".gitlab-ci.yml", assumes that the additional jobs on the staging
  branch running on the freely available gitlab shared runner are
  positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to
  simply 18.04 (assumes no diverse minor levels will be used or
  specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
  (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement
  (there's no static version of those libs in distro supplied
  packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
  doesn't fall into the general scope of gating job described by PMM
  (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs `make
  check`
- Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
  simple but effective way of speeding up the builds and tests by
  using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles,
  some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
  Jobs based on custom runners: documentation and configuration
    placeholder
  Jobs based on custom runners: build environment docs and playbook
  Jobs based on custom runners: docs and gitlab-runner setup playbook
  Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 222 +++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml          |   1 +
 docs/devel/ci.rst                      | 127 ++++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   2 +
 scripts/ci/setup/build-environment.yml | 116 +++++++++++++
 scripts/ci/setup/gitlab-runner.yml     |  71 ++++++++
 scripts/ci/setup/inventory.template    |   1 +
 scripts/ci/setup/vars.yml.template     |  12 ++
 9 files changed, 553 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory.template
 create mode 100644 scripts/ci/setup/vars.yml.template

-- 
2.31.1

[PATCH v7 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.  One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions.  As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
 .gitlab-ci.d/qemu-project.yml   |  1 +
 docs/devel/ci.rst               | 32 ++++++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 48 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
new file mode 100644
index 0000000000..a07b27384c
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+  GIT_STRATEGY: clone
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..dde8270301 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
new file mode 100644
index 0000000000..064ffa9988
--- /dev/null
+++ b/docs/devel/ci.rst
@@ -0,0 +1,32 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Custom runners entail custom machines.  To see a list of the machines
+currently deployed in the QEMU GitLab CI and their maintainers, please
+refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 977c3893bd..da15273786 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -27,6 +27,7 @@ Contents:
    migration
    atomics
    stable-process
+   ci
    qtest
    decodetree
    secure-coding-practices
-- 
2.31.1

[PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

To run basic jobs on custom runners, the environment needs to be
properly set up.  The most common requirement is having the right
packages installed.

The playbook introduced here covers the QEMU's project s390x and
aarch64 machines.  At the time this is being proposed, those machines
have already had this playbook applied to them.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/ci.rst                      |  40 +++++++++
 scripts/ci/setup/.gitignore            |   2 +
 scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |   1 +
 4 files changed, 159 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 064ffa9988..bfedbb1025 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
 Custom runners entail custom machines.  To see a list of the machines
 currently deployed in the QEMU GitLab CI and their maintainers, please
 refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  This playbook consists on the installation of various
+required packages (and a general package update while at it).  It
+currently covers a number of different Linux distributions, but it can
+be expanded to cover other systems.
+
+The minimum required version of Ansible successfully tested in this
+playbook is 2.8.0 (a version check is embedded within the playbook
+itself).  To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
+
+Please note that most of the tasks in the playbook require superuser
+privileges, such as those from the ``root`` account or those obtained
+by ``sudo``.  If necessary, please refer to ``ansible-playbook``
+options such as ``--become``, ``--become-method``, ``--become-user``
+and ``--ask-become-pass``.
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
new file mode 100644
index 0000000000..ee088604d1
--- /dev/null
+++ b/scripts/ci/setup/.gitignore
@@ -0,0 +1,2 @@
+inventory
+
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..581c1c75d1
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,116 @@
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+#
+# This is an ansible playbook file.  Run it to set up systems with the
+# environment needed to build QEMU.
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:
+    - name: Check for suitable ansible version
+      delegate_to: localhost
+      assert:
+        that:
+          - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)'
+        msg: "Unsuitable ansible version, please use version 2.8.0 or later"
+
+    - name: Update apt cache / upgrade packages via apt
+      apt:
+        update_cache: yes
+        upgrade: yes
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - ccache
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libzstd-dev
+          - make
+          - python3-yaml
+          - python3-sphinx
+          - python3-sphinx-rtd-theme
+          - ninja-build
+          - sparse
+          - xfslibs-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      package:
+        name:
+          - libspice-server-dev
+          - libxen-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] != 's390x'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - clang
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '18.04'
+
+    - name: Install basic packages to build QEMU on Ubuntu 20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
+          - clang-10
+          - genisoimage
+          - liblttng-ust-dev
+          - libslirp-dev
+          - netcat-openbsd
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '20.04'
diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
new file mode 100644
index 0000000000..2fbb50c4a8
--- /dev/null
+++ b/scripts/ci/setup/inventory.template
@@ -0,0 +1 @@
+localhost
-- 
2.31.1

[PATCH v7 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

To have the jobs dispatched to custom runners, gitlab-runner must
be installed, active as a service and properly configured.  The
variables file and playbook introduced here should help with those
steps.

The playbook introduced here covers the Linux distributions and
has been primarily tested on OS/machines that the QEMU project
has available to act as runners, namely:

 * Ubuntu 20.04 on aarch64
 * Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions.  Earlier
versions were tested on FreeBSD too, so chances of success are
high.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
---
 docs/devel/ci.rst                  | 55 +++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  2 +-
 scripts/ci/setup/gitlab-runner.yml | 71 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 12 +++++
 4 files changed, 139 insertions(+), 1 deletion(-)
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index bfedbb1025..b3bf3ef615 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -70,3 +70,58 @@ privileges, such as those from the ``root`` account or those obtained
 by ``sudo``.  If necessary, please refer to ``ansible-playbook``
 options such as ``--become``, ``--become-method``, ``--become-user``
 and ``--ask-become-pass``.
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears-like icon at the bottom of the left hand side
+   vertical toolbar), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "And this registration token:"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Tags are very important as they are used to route specific jobs to
+specific types of runners, so it's a good idea to double check that
+the automatically created tags are consistent with the OS and
+architecture.  For instance, an Ubuntu 20.04 aarch64 system should
+have tags set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
index ee088604d1..f4a6183f1f 100644
--- a/scripts/ci/setup/.gitignore
+++ b/scripts/ci/setup/.gitignore
@@ -1,2 +1,2 @@
 inventory
-
+vars.yml
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
new file mode 100644
index 0000000000..1127db516f
--- /dev/null
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -0,0 +1,71 @@
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+#
+# This is an ansible playbook file.  Run it to set up systems with the
+# gitlab-runner agent.
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Set the Operating System for gitlab-runner
+      set_fact:
+        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+    - debug:
+        msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+    - name: Set the architecture for gitlab-runner
+      set_fact:
+        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+    - debug:
+        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+    - name: Download the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
new file mode 100644
index 0000000000..e48089761f
--- /dev/null
+++ b/scripts/ci/setup/vars.yml.template
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
-- 
2.31.1

[PATCH v7 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index a07b27384c..061d3cdfed 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -12,3 +12,211 @@
 # guarantees a fresh repository on each job run.
 variables:
   GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
-- 
2.31.1

Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      |  40 +++++++++
>  scripts/ci/setup/.gitignore            |   2 +
>  scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
>  scripts/ci/setup/inventory.template    |   1 +
>  4 files changed, 159 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 064ffa9988..bfedbb1025 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
>  Custom runners entail custom machines.  To see a list of the machines
>  currently deployed in the QEMU GitLab CI and their maintainers, please
>  refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3

I was able to put root@foo for the machines I had in my .ssh/config

> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  This playbook consists on the installation of various
> +required packages (and a general package update while at it).  It
> +currently covers a number of different Linux distributions, but it can
> +be expanded to cover other systems.
> +
> +The minimum required version of Ansible successfully tested in this
> +playbook is 2.8.0 (a version check is embedded within the playbook
> +itself).  To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> +
> +Please note that most of the tasks in the playbook require superuser
> +privileges, such as those from the ``root`` account or those obtained
> +by ``sudo``.  If necessary, please refer to ``ansible-playbook``
> +options such as ``--become``, ``--become-method``, ``--become-user``
> +and ``--ask-become-pass``.

If the above works maybe worth mentioning here because just having root
ssh is probably the easiest way to manage a box. 

> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..ee088604d1
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1,2 @@
> +inventory
> +
> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..581c1c75d1
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,116 @@
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +#
> +# This is an ansible playbook file.  Run it to set up systems with the
> +# environment needed to build QEMU.
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Check for suitable ansible version
> +      delegate_to: localhost
> +      assert:
> +        that:
> +          - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)'
> +        msg: "Unsuitable ansible version, please use version 2.8.0 or later"
> +
> +    - name: Update apt cache / upgrade packages via apt
> +      apt:
> +        update_cache: yes
> +        upgrade: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - python3-yaml
> +          - python3-sphinx
> +          - python3-sphinx-rtd-theme
> +          - ninja-build
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - clang
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '18.04'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
> +          - clang-10
> +          - genisoimage
> +          - liblttng-ust-dev
> +          - libslirp-dev
> +          - netcat-openbsd
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '20.04'
> diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory.template
> @@ -0,0 +1 @@
> +localhost

Anyway:

Tested-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

Re: [PATCH v7 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 10:26 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
>
>  * Ubuntu 20.04 on aarch64
>  * Ubuntu 18.04 on s390x
>
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Willian Rampazzo <willianr@redhat.com>

Re-tested just in case and it looks really good, thanks!

Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 10:26 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      |  40 +++++++++
>  scripts/ci/setup/.gitignore            |   2 +
>  scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
>  scripts/ci/setup/inventory.template    |   1 +
>  4 files changed, 159 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory.template
>

Ran the playbook and built QEMU for testing without problems.

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Willian Rampazzo]

On Wed, Jun 30, 2021 at 8:28 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Cleber Rosa <crosa@redhat.com> writes:
>
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/ci.rst                      |  40 +++++++++
> >  scripts/ci/setup/.gitignore            |   2 +
> >  scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
> >  scripts/ci/setup/inventory.template    |   1 +
> >  4 files changed, 159 insertions(+)
> >  create mode 100644 scripts/ci/setup/.gitignore
> >  create mode 100644 scripts/ci/setup/build-environment.yml
> >  create mode 100644 scripts/ci/setup/inventory.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 064ffa9988..bfedbb1025 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
> >  Custom runners entail custom machines.  To see a list of the machines
> >  currently deployed in the QEMU GitLab CI and their maintainers, please
> >  refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Create an ``inventory`` file
> > +under ``scripts/ci/setup``, such as this::
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>
> I was able to put root@foo for the machines I had in my .ssh/config
>
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  This playbook consists on the installation of various
> > +required packages (and a general package update while at it).  It
> > +currently covers a number of different Linux distributions, but it can
> > +be expanded to cover other systems.
> > +
> > +The minimum required version of Ansible successfully tested in this
> > +playbook is 2.8.0 (a version check is embedded within the playbook
> > +itself).  To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
> > +
> > +Please note that most of the tasks in the playbook require superuser
> > +privileges, such as those from the ``root`` account or those obtained
> > +by ``sudo``.  If necessary, please refer to ``ansible-playbook``
> > +options such as ``--become``, ``--become-method``, ``--become-user``
> > +and ``--ask-become-pass``.
>
> If the above works maybe worth mentioning here because just having root
> ssh is probably the easiest way to manage a box.

If the host is internet-facing, there are lots of recommendations to
disable root access using ssh (eg.
https://www.redhat.com/sysadmin/administering-remote-systems). There
are also recommendations from NIST and SANS.

So, to avoid an unintended creation of an attack vector in the custom
runners, I would personally prefer to let just the ansible tricks in
the documentation than mentioning it is possible (and maybe easier) to
enable root access thru ssh.

Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Wainer dos Santos Moschetta]

On 6/29/21 10:26 PM, Cleber Rosa wrote:
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   docs/devel/ci.rst                      |  40 +++++++++
>   scripts/ci/setup/.gitignore            |   2 +
>   scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
>   scripts/ci/setup/inventory.template    |   1 +
>   4 files changed, 159 insertions(+)
>   create mode 100644 scripts/ci/setup/.gitignore
>   create mode 100644 scripts/ci/setup/build-environment.yml
>   create mode 100644 scripts/ci/setup/inventory.template

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 064ffa9988..bfedbb1025 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
>   Custom runners entail custom machines.  To see a list of the machines
>   currently deployed in the QEMU GitLab CI and their maintainers, please
>   refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  This playbook consists on the installation of various
> +required packages (and a general package update while at it).  It
> +currently covers a number of different Linux distributions, but it can
> +be expanded to cover other systems.
> +
> +The minimum required version of Ansible successfully tested in this
> +playbook is 2.8.0 (a version check is embedded within the playbook
> +itself).  To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> +
> +Please note that most of the tasks in the playbook require superuser
> +privileges, such as those from the ``root`` account or those obtained
> +by ``sudo``.  If necessary, please refer to ``ansible-playbook``
> +options such as ``--become``, ``--become-method``, ``--become-user``
> +and ``--ask-become-pass``.
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..ee088604d1
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1,2 @@
> +inventory
> +
> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..581c1c75d1
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,116 @@
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +#
> +# This is an ansible playbook file.  Run it to set up systems with the
> +# environment needed to build QEMU.
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Check for suitable ansible version
> +      delegate_to: localhost
> +      assert:
> +        that:
> +          - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)'
> +        msg: "Unsuitable ansible version, please use version 2.8.0 or later"
> +
> +    - name: Update apt cache / upgrade packages via apt
> +      apt:
> +        update_cache: yes
> +        upgrade: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - python3-yaml
> +          - python3-sphinx
> +          - python3-sphinx-rtd-theme
> +          - ninja-build
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - clang
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '18.04'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
> +          - clang-10
> +          - genisoimage
> +          - liblttng-ust-dev
> +          - libslirp-dev
> +          - netcat-openbsd
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '20.04'
> diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory.template
> @@ -0,0 +1 @@
> +localhost

Re: [PATCH v7 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Wainer dos Santos Moschetta]

On 6/29/21 10:26 PM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
>
>   * Ubuntu 20.04 on aarch64
>   * Ubuntu 18.04 on s390x
>
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Willian Rampazzo <willianr@redhat.com>
> ---
>   docs/devel/ci.rst                  | 55 +++++++++++++++++++++++
>   scripts/ci/setup/.gitignore        |  2 +-
>   scripts/ci/setup/gitlab-runner.yml | 71 ++++++++++++++++++++++++++++++
>   scripts/ci/setup/vars.yml.template | 12 +++++
>   4 files changed, 139 insertions(+), 1 deletion(-)
>   create mode 100644 scripts/ci/setup/gitlab-runner.yml
>   create mode 100644 scripts/ci/setup/vars.yml.template
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index bfedbb1025..b3bf3ef615 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -70,3 +70,58 @@ privileges, such as those from the ``root`` account or those obtained
>   by ``sudo``.  If necessary, please refer to ``ansible-playbook``
>   options such as ``--become``, ``--become-method``, ``--become-user``
>   and ``--ask-become-pass``.
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears-like icon at the bottom of the left hand side
> +   vertical toolbar), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "And this registration token:"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Tags are very important as they are used to route specific jobs to
> +specific types of runners, so it's a good idea to double check that
> +the automatically created tags are consistent with the OS and
> +architecture.  For instance, an Ubuntu 20.04 aarch64 system should
> +have tags set as::
> +
> +  ubuntu_20.04,aarch64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> index ee088604d1..f4a6183f1f 100644
> --- a/scripts/ci/setup/.gitignore
> +++ b/scripts/ci/setup/.gitignore
> @@ -1,2 +1,2 @@
>   inventory
> -
> +vars.yml
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..1127db516f
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,71 @@
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +#
> +# This is an ansible playbook file.  Run it to set up systems with the
> +# gitlab-runner agent.
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Set the Operating System for gitlab-runner
> +      set_fact:
> +        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
> +    - debug:
> +        msg: gitlab-runner OS is {{ gitlab_runner_os }}
> +
> +    - name: Set the architecture for gitlab-runner
> +      set_fact:
> +        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
> +    - debug:
> +        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
> +
> +    - name: Download the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..e48089761f
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,12 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.12.0
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# A mapping of the ansible to gitlab architecture nomenclature
> +ansible_to_gitlab_arch:
> +  x86_64: amd64
> +  aarch64: arm64
> +  s390x: s390x
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN

Re: [PATCH v7 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Wainer dos Santos Moschetta]

On 6/29/21 10:26 PM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>   1 file changed, 208 insertions(+)
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index a07b27384c..061d3cdfed 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -12,3 +12,211 @@
>   # guarantees a fresh repository on each job run.
>   variables:
>     GIT_STRATEGY: clone
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1
> +
> +ubuntu-18.04-s390x-all:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-18.04-s390x-alldbg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --disable-libssh
> + - make clean
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-18.04-s390x-clang:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-18.04-s390x-tci:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --enable-tcg-interpreter
> + - make --output-sync -j`nproc`
> +
> +ubuntu-18.04-s390x-notcg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --disable-tcg
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +# All ubuntu-20.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/qemu/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-20.04-aarch64-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1
> +
> +ubuntu-20.04-aarch64-all:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-alldbg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --disable-libssh
> + - make clean
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-clang:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-tci:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --enable-tcg-interpreter
> + - make --output-sync -j`nproc`
> +
> +ubuntu-20.04-aarch64-notcg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --disable-tcg
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1

Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Willian Rampazzo <wrampazz@redhat.com> writes:

> On Wed, Jun 30, 2021 at 8:28 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Cleber Rosa <crosa@redhat.com> writes:
>>
>> > To run basic jobs on custom runners, the environment needs to be
>> > properly set up.  The most common requirement is having the right
>> > packages installed.
>> >
>> > The playbook introduced here covers the QEMU's project s390x and
>> > aarch64 machines.  At the time this is being proposed, those machines
>> > have already had this playbook applied to them.
>> >
>> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> > ---
>> >  docs/devel/ci.rst                      |  40 +++++++++
>> >  scripts/ci/setup/.gitignore            |   2 +
>> >  scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
>> >  scripts/ci/setup/inventory.template    |   1 +
>> >  4 files changed, 159 insertions(+)
>> >  create mode 100644 scripts/ci/setup/.gitignore
>> >  create mode 100644 scripts/ci/setup/build-environment.yml
>> >  create mode 100644 scripts/ci/setup/inventory.template
>> >
>> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>> > index 064ffa9988..bfedbb1025 100644
>> > --- a/docs/devel/ci.rst
>> > +++ b/docs/devel/ci.rst
>> > @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
>> >  Custom runners entail custom machines.  To see a list of the machines
>> >  currently deployed in the QEMU GitLab CI and their maintainers, please
>> >  refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
>> > +
>> > +Machine Setup Howto
>> > +-------------------
>> > +
>> > +For all Linux based systems, the setup can be mostly automated by the
>> > +execution of two Ansible playbooks.  Create an ``inventory`` file
>> > +under ``scripts/ci/setup``, such as this::
>> > +
>> > +  fully.qualified.domain
>> > +  other.machine.hostname
>> > +
>> > +You may need to set some variables in the inventory file itself.  One
>> > +very common need is to tell Ansible to use a Python 3 interpreter on
>> > +those hosts.  This would look like::
>> > +
>> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>>
>> I was able to put root@foo for the machines I had in my .ssh/config
>>
>> > +
>> > +Build environment
>> > +~~~~~~~~~~~~~~~~~
>> > +
>> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>> > +set up machines with the environment needed to perform builds and run
>> > +QEMU tests.  This playbook consists on the installation of various
>> > +required packages (and a general package update while at it).  It
>> > +currently covers a number of different Linux distributions, but it can
>> > +be expanded to cover other systems.
>> > +
>> > +The minimum required version of Ansible successfully tested in this
>> > +playbook is 2.8.0 (a version check is embedded within the playbook
>> > +itself).  To run the playbook, execute::
>> > +
>> > +  cd scripts/ci/setup
>> > +  ansible-playbook -i inventory build-environment.yml
>> > +
>> > +Please note that most of the tasks in the playbook require superuser
>> > +privileges, such as those from the ``root`` account or those obtained
>> > +by ``sudo``.  If necessary, please refer to ``ansible-playbook``
>> > +options such as ``--become``, ``--become-method``, ``--become-user``
>> > +and ``--ask-become-pass``.
>>
>> If the above works maybe worth mentioning here because just having root
>> ssh is probably the easiest way to manage a box.
>
> If the host is internet-facing, there are lots of recommendations to
> disable root access using ssh (eg.
> https://www.redhat.com/sysadmin/administering-remote-systems). There
> are also recommendations from NIST and SANS.
>
> So, to avoid an unintended creation of an attack vector in the custom
> runners, I would personally prefer to let just the ansible tricks in
> the documentation than mentioning it is possible (and maybe easier) to
> enable root access thru ssh.

I agree you don't want remote password based authentication. I use
key-based authentication because it seems easier to log in directly as
root than to keep trusting my user password to the remote console to
gain sudo privileges. Anyway either way I'm happy.

-- 
Alex Bennée

Re: [PATCH v7 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
>
>  * Ubuntu 20.04 on aarch64
>  * Ubuntu 18.04 on s390x
>
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Willian Rampazzo <willianr@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

Re: [PATCH v7 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> TL;DR: this should allow the QEMU maintainer to push to the staging
> branch, and have custom jobs running on the project's aarch64 and
> s390x machines.  Jobs in this version are allowed to fail, to allow
> for the inclusion of the novel machines/jobs without CI disruption.
> Simple usage looks like:
>
>    git push remote staging
>    ./scripts/ci/gitlab-pipeline-status --verbose --wait

Queued to testing/next, thanks.

-- 
Alex Bennée