[PATCH v6 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[PATCH v6 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Cleber Rosa]

TL;DR: this should allow the QEMU maintainer to push to the staging
branch, and have custom jobs running on the project's aarch64 and
s390x machines.  Jobs in this version are allowed to fail, to allow
for the inclusion of the novel machines/jobs without CI disruption.
Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an
RFC[1].  Since then, it was decided that a simpler version should be
attempted first.

At this point, there are two specific runners (an aarch64 and an s390x)
registered with GitLab, at https://gitlab.com/qemu-project, currently
setup to the "qemu" repository.  To make it extra clear, the following
points deserve notice:

 - This work dealt with two different QEMU project machines:
    I) s390x running Ubuntu 18.04
   II) aarch64 running Ubuntu 20.04

 - All CI jobs introduced here are allowed to fail.  It should have
   no impact on the overall PASS/FAIL result of the pipeline.

 - Both machines have already been completely configured used the code
   on this series.  No further action (besides pushing this code to
   the staging branch) is needed to have CI jobs executed on them.

 - The actual CI jobs to be executed are defined in this series,
   and map as best as possible the tests run by Peter Maydell on
   the staging branch.  Peter, given the time delta since the
   beginning of this work, it may be needed to further tweak
   them.

 - The actual results of CI jobs run from these definitions are
   probably more fluid than the results from the CI jobs run on the
   shared runners.  Both changes to the code being tested, and the
   conditions/resources of the machine should be taken into account.

 - A pipeline, and some of the jobs, can be seen on the links bellow.
   Besides successful jobs, it also contains jobs not started (set to
   manual, with the specific reasons noted later on the changes
   section):

   * https://gitlab.com/cleber.gnu/qemu/-/pipelines/316471691
    - ubuntu-18.04-s390x-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698118
    - ubuntu-20.04-aarch64-all: https://gitlab.com/cleber.gnu/qemu/-/jobs/1325698124

Changes from v5:

 - Moved include of ".gitlab-ci.d/custom-runners.yml" from
   ".gitlab-ci.yml" to ".gitlab-ci.d/qemu-project.yml"

 - Changed git clone strategy from "submodule recursive" to "clone",
   to guarantee a fresh and clean repo on every job, and let
   QEMU handle eventual needed recursive submodule operations

 - Require user to create an Ansible inventory file, based on the
   now provided "inventory.template" one.  Previously, the
   "iventory" file itself was provided and users were asked to edit
   it.

 - Registered runners will now be given a default set of tags, with
   their OS and architectures.  This, besides automating another step,
   works around a "gitlab-runner register" command line issue with
   "--run-untagged=false" not being respected if no tags are given.

 - Added conditional for installing either "clang" or "clang-10" to
   match the package name on different versions of Ubuntu.

 - Changed Ubuntu 20.04 jobs to use clang-10 instead of clang.

 - Defaults to not running untagged jobs when registering a gitlab
   runner.

 - Added python3-sphinx-rtd-theme package installation to
   build-environment.yml, to match 73e6aec6522.

 - Added genisoimage package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match 7e86e5d5ccc.

 - Added liblttng-ust-dev package installation to
   build-environment.yml when on Ubuntu 20.04 (not on 18.04) to match
   8e9419b7902.

 - Added libslirp-dev package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match 8e9419b7902.

 - Added netcat-openbsd package installation to build-environment.yml
   when on Ubuntu 20.04 (not on 18.04) to match c4cb1c9f2e1.

 - Bumped gitlab-runner version to 13.12.0

 - Use only gitlab-runner binaries from the official distribution (they
   are now also available for s390x, previously missing).

 - Selection of the OS and architecture for the gitlab-runner binary
   is now done automatically.  If necessary, architecture chosen can be
   influenced by editing the "ansible_to_gitlab_arch" values on vars.yml
   (after you've created one from the vars.yml.template).

 - Marked job "ubuntu-18.04-s390x-clang" as manual, given that
   the latest executions are getting stuck with the last
   output being "Sanitizers are enabled ==> Not running the
   qemu-iotests."

 - Marked job "ubuntu-18.04-s390x-notcg" as manual, given that the
   latest executions are getting stuck with the last tests being
   "tests/qtest/pxe-test" and "tests/qtest/boot-serial-test" which
   contains commands lines with "-accel kvm -accel tcg" and "-accel
   qtest".  Obviously tcg is disabled and KVM is not available on the
   QEMU s390x machine, and the fallback to qtest gets the tests stuck.

 - Marked job "ubuntu-20.04-aarch64-notcg" as manual due to build
   failures (FAILED: libqemu-aarch64-softmmu.fa.p/target_arm_psci.c.o).

 - Marked job "ubuntu-20.04-aarch64-clang" as manual due to test
   failures:
      ../tests/qtest/libqtest.c:157: kill_qemu() tried to terminate
      QEMU process but encountered exit status 1 (expected 0)
      ERROR qtest-aarch64/qom-test - too few tests run (expected 80, got 6)

 - Cleared Daniel's R-B due to the number of changes in the
   corresponding patches.

Known issues:

 - tests/qtest/qos-test has been seen stuck a number of times on both
   the s390x and aarch64 machines.  Further investigation is needed.
   While a resolution is found, this should not affect the pipeline,
   given that all new jobs introduced here are currently allowed to
   fail.

Next development / discussion points:

 - Test docker executor
 - Test docker executor with podman
 - Configuration of more accurate job timeouts
 - Historical result analysis
 - Promotion of jobs (removing the "allow failure" setting)

Changes from v4:

 - Fixed typo in docs/devel/ci.rst, s/maintanance/maintenance/ (Thomas)
 - Removed "[local]" group from inventory file (Erik)
 - Removed sections from the playbooks which *would* be applied on
   hardware/OS that are currently not available to QEMU
 - Removed duplicated "here" on documentation (Thomas)
 - Moved description of current jobs, and possible direction of future
   jobs to the patch description (Thomas)
 - Remove comments around "when" conditions (Andrea)
 - Switch to always use explicit lists on "when" blocks (Andrea)
 - Switch from using module "apt" to using generic action module "package",
   which involved adding a new task to update the apt cache (Andrea)
 - Fix playbook indentation in the non-s390x package installation task (Andrea)
 - Changed gitlab-runner tags examples from FreeBSD to Ubuntu, which is
   covered by jobs added on this version
 - Fixed typo in commit message s/s390/s390x/ (Phil)
 - Allow all custom-runner jobs to fail at this time
 - Cleared "Reviewed-by" in one patch due to large changes

  Changes requested in v4 but *not* seen here due to sections of the
  playbook being removed:

 - Replace SDL-devel for SDL2-devel on CentOS, according to 5ed7ca3 (Thomas)
 - Correct missing step 10 on the FreeBSD gitlab-runner installation
   instructions (Erik)

Changes from v3:

- Applied changes to match <20201014135416.1290679-1-pbonzini@redhat.com>,
  that is, added ninja-build to "build-environment.yml" list of packages
  and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners",
  given that the other jobs running on shared runners are also
  considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
 * update -> up to date (Erik)
 * a different set of CI jobs -> different CI jobs (Erik)
 * Pull requests will only be merged -> code will only be merged (Stefan)
 * Setup -> set up (Stefan)
 * them -> they (Stefan)
 * the -> where the (Stefan)
 * dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of
  the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that
  no external variable is used there

- Updated package list in "build-environment.yml" from current
  dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30),
  and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts"
  into (Phillipe):
  1) Basic documentation and configuration (gitlab-ci.yml) placeholder
  2) Playbooks for setting up a build environment
  3) Playbooks for setting up gitlab-runner
  4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine
  build and test.

- Set custom jobs to not depend on any other job, so they can start
  right away.

- Set rules for starting jobs so that all pushing to any branch that
  start with name "staging".  This allows the project maintainer to
  use the "push to staging" workflow, while also allowing others to
  generate similar jobs.  If this project has configured custom
  runners, the jobs will run, if not, the pipeline will be marked as
  "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern
  (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available
  (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available
  (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in
  ".gitlab-ci.yml", assumes that the additional jobs on the staging
  branch running on the freely available gitlab shared runner are
  positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to
  simply 18.04 (assumes no diverse minor levels will be used or
  specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
  (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement
  (there's no static version of those libs in distro supplied
  packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
  doesn't fall into the general scope of gating job described by PMM
  (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs `make
  check`
- Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
  simple but effective way of speeding up the builds and tests by
  using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles,
  some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
  Jobs based on custom runners: documentation and configuration
    placeholder
  Jobs based on custom runners: build environment docs and playbook
  Jobs based on custom runners: docs and gitlab-runner setup playbook
  Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 222 +++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml          |   1 +
 docs/devel/ci.rst                      | 115 +++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   1 +
 scripts/ci/setup/build-environment.yml |  98 +++++++++++
 scripts/ci/setup/gitlab-runner.yml     |  61 +++++++
 scripts/ci/setup/inventory.template    |   1 +
 scripts/ci/setup/vars.yml.template     |  12 ++
 9 files changed, 512 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory.template
 create mode 100644 scripts/ci/setup/vars.yml.template

-- 
2.25.4

[PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.  One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions.  As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
 .gitlab-ci.d/qemu-project.yml   |  1 +
 docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 44 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
new file mode 100644
index 0000000000..a07b27384c
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+  GIT_STRATEGY: clone
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..dde8270301 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
new file mode 100644
index 0000000000..585b7bf4b8
--- /dev/null
+++ b/docs/devel/ci.rst
@@ -0,0 +1,28 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 791925dcda..c9a02e786e 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -27,6 +27,7 @@ Contents:
    migration
    atomics
    stable-process
+   ci
    qtest
    decodetree
    secure-coding-practices
-- 
2.25.4

[PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

To run basic jobs on custom runners, the environment needs to be
properly set up.  The most common requirement is having the right
packages installed.

The playbook introduced here covers the QEMU's project s390x and
aarch64 machines.  At the time this is being proposed, those machines
have already had this playbook applied to them.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/ci.rst                      | 30 ++++++++
 scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory.template    |  1 +
 3 files changed, 129 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 585b7bf4b8..35c6b5e269 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..664f2f0519
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,98 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:
+    - name: Update apt cache
+      apt:
+        update_cache: yes
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - ccache
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libzstd-dev
+          - make
+          - python3-yaml
+          - python3-sphinx
+          - python3-sphinx-rtd-theme
+          - ninja-build
+          - sparse
+          - xfslibs-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      package:
+        name:
+          - libspice-server-dev
+          - libxen-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] != 's390x'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - clang
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '18.04'
+
+    - name: Install basic packages to build QEMU on Ubuntu 20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
+          - clang-10
+          - genisoimage
+          - liblttng-ust-dev
+          - libslirp-dev
+          - netcat-openbsd
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '20.04'
diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
new file mode 100644
index 0000000000..2fbb50c4a8
--- /dev/null
+++ b/scripts/ci/setup/inventory.template
@@ -0,0 +1 @@
+localhost
-- 
2.25.4

[PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

To have the jobs dispatched to custom runners, gitlab-runner must
be installed, active as a service and properly configured.  The
variables file and playbook introduced here should help with those
steps.

The playbook introduced here covers the Linux distributions and
has been primarily tested on OS/machines that the QEMU project
has available to act as runners, namely:

 * Ubuntu 20.04 on aarch64
 * Ubuntu 18.04 on s390x

But, it should work on all other Linux distributions.  Earlier
versions were tested on FreeBSD too, so chances of success are
high.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 12 ++++++
 4 files changed, 131 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 35c6b5e269..bbd89e54d7 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -56,3 +56,60 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
new file mode 100644
index 0000000000..f112d05dd0
--- /dev/null
+++ b/scripts/ci/setup/.gitignore
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
new file mode 100644
index 0000000000..98dab92bb5
--- /dev/null
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -0,0 +1,61 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Set the Operating System for gitlab-runner
+      set_fact:
+        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+    - debug:
+        msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+    - name: Set the architecture for gitlab-runner
+      set_fact:
+        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+    - debug:
+        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+    - name: Download the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
new file mode 100644
index 0000000000..e48089761f
--- /dev/null
+++ b/scripts/ci/setup/vars.yml.template
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+  x86_64: amd64
+  aarch64: arm64
+  s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
-- 
2.25.4

[PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
 1 file changed, 208 insertions(+)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index a07b27384c..061d3cdfed 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -12,3 +12,211 @@
 # guarantees a fresh repository on each job run.
 variables:
   GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
-- 
2.25.4

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

Hi Alex, Stefan,

On 6/8/21 5:14 AM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.

AFAIK there is more hardware available to the project, so I'm wondering
what happened to the rest, is it a deliberate choice to start small?
What will happen with the rest, since we are wasting resources?
Who has access to what and should do what (setup)? How is this list of
hw managed btw? Should there be some public visibility (i.e. Wiki)?

Is there a document explaining what are the steps to follow for an
entity to donate / sponsor hardware? Where would it be stored, should
this hw be shipped somewhere? What documentation should be provided for
its system administration?

In case an entity manages hosting and maintenance, can the QEMU project
share the power bill? Up to which amount?
Similar question if a sysadmin has to be paid.

If the QEMU project can't spend money on CI, what is expected from
resource donators? Simply hardware + power (electricity) + network
traffic? Also sysadmining and monitoring? Do we expect some kind of
reliability on the data stored here or can we assume disposable /
transient runners?
Should donators also provide storage? Do we have a minimum storage
requirement?

Should we provide some guideline explaining any code might be run by
our contributors on these runners and some security measures have to
be taken / considered?

Sponsors usually expect some advertising to thanks them, and often
regular reports on how their resources are used, else they might not
renew their offer. Who should care to keep the relationship with
sponsors?

Where is defined what belong to the project, who is responsible, who can
request access to this hardware, what resource can be used?

More generically, what is the process for a private / corporate entity
to register a runner to the project? (how did it work for this aarch64
and s390x one?)

What else am I missing?

Thanks,

Phil.

> This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>  1 file changed, 208 insertions(+)

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa Junior]

[-- Attachment #1: Type: text/plain, Size: 4997 bytes --]

On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org>
wrote:

> Hi Alex, Stefan,
>
> On 6/8/21 5:14 AM, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390x) that can be used
> > for jobs that do build and run tests.
>
> AFAIK there is more hardware available to the project, so I'm wondering
> what happened to the rest, is it a deliberate choice to start small?
>

Hi Phil,

Yes, this series was deliberately focused on the first two machines owned
and available to QEMU.


> What will happen with the rest, since we are wasting resources?
>

The plan is to allow all machines (currently available and to-be available)
to be connected as custom runners.  This hopefully gets that started.

About "more hardware available to the project", there's one VM from
fosshost which was made available not long ago, and which I set up even
more recently, which could be used as a gitlab runner too.  But, even
though some new hardware resource is available (and wasted?), the human
resources to maintain them have not been properly determined, so I believe
it's a good decision to start with the machines that have been operational
for long, and that already have to the best of my knowledge, people
maintaining them.

I also see a "Debian unstable mips64el (Debian) @ cipunited.cn" registered
as a runner, but I don't have extra information about it.

Besides that, I'll send another series shortly, that builds upon this
series, and adds a Red Hat focused job, on a Red Hat managed machine.  This
should be what other entities should be capable of doing and allowed to do.


> Who has access to what and should do what (setup)? How is this list of
> hw managed btw? Should there be some public visibility (i.e. Wiki)?
>

These are good questions, and I believe Alex can answer them about those
two machines.  Even though I hooked them up to GitLab, AFAICT he is the
ultimate admin (maybe Peter too?).

About hardware management, it has been suggested to use either the Wiki or
a MAINTAINERS entry.  This is still unresolved and feedback would be
appreciated.  For me to propose a MAINTAINERS entry, say, on a v7, I'd need
the information on who is managing them.


> Is there a document explaining what are the steps to follow for an
> entity to donate / sponsor hardware? Where would it be stored, should
> this hw be shipped somewhere? What documentation should be provided for
> its system administration?
>
> In case an entity manages hosting and maintenance, can the QEMU project
> share the power bill? Up to which amount?
> Similar question if a sysadmin has to be paid.
>
> If the QEMU project can't spend money on CI, what is expected from
> resource donators? Simply hardware + power (electricity) + network
> traffic? Also sysadmining and monitoring? Do we expect some kind of
> reliability on the data stored here or can we assume disposable /
> transient runners?
> Should donators also provide storage? Do we have a minimum storage
> requirement?
>
> Should we provide some guideline explaining any code might be run by
> our contributors on these runners and some security measures have to
> be taken / considered?
>
> Sponsors usually expect some advertising to thanks them, and often
> regular reports on how their resources are used, else they might not
> renew their offer. Who should care to keep the relationship with
> sponsors?
>
> Where is defined what belong to the project, who is responsible, who can
> request access to this hardware, what resource can be used?
>
>
You obviously directed the question towards Alex and Stefan (rightfully
so), so I won't attempt to answer these ones at this point.


> More generically, what is the process for a private / corporate entity
> to register a runner to the project? (how did it work for this aarch64
> and s390x one?)
>

The steps are listed on the documentation.  Basically anyone with knowledge
of the "registration token" can add new machines to GitLab as runners.  For
the two aarch64 and s390x, it was a matter of following the documentation
steps which basically involve:

1) providing the hostname(s) in the inventory file
2) provide the "registration token" in the vars.yml file
3) running the playbooks


>
> What else am I missing?
>
>
I think you're missing the answers to all your good questions :).

And I understand that are a lot of them (from everyone, including myself).
The dilemma here is: should we activate the machines already available, and
learn in practice, what's missing?  I honestly believe we should.

Thanks,
- Cleber.


> Thanks,
>
> Phil.
>
> > This introduces those jobs,
> > which are a mapping of custom scripts used for the same purpose.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
> >  1 file changed, 208 insertions(+)
>
>

[-- Attachment #2: Type: text/html, Size: 6811 bytes --]

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Wainer dos Santos Moschetta]

Hi,

On 6/8/21 12:14 AM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>   1 file changed, 208 insertions(+)
>
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index a07b27384c..061d3cdfed 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -12,3 +12,211 @@
>   # guarantees a fresh repository on each job run.
>   variables:
>     GIT_STRATEGY: clone
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'

Should it restrict the job for pushes to qemu-project only? If yes, then 
it probably needs the statement:

'$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'

If you change that here, you will end it changing all over the jobs. In 
general, there are many boilerplates in this file. I'm ok to merge it as 
is as long as it is followed by another series to refactor the code.

> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1
> +
> +ubuntu-18.04-s390x-all:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-18.04-s390x-alldbg:
Maybe we don't need both ubuntu-18.04-s390x-all and 
ubuntu-18.04-s390x-alldbg jobs.
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --disable-libssh
> + - make clean
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +ubuntu-18.04-s390x-clang:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-18.04-s390x-tci:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --enable-tcg-interpreter
> + - make --output-sync -j`nproc`
I think it needs to `make check-tcg` at least. See "build-tci" in 
`.gitlab-ci.d/buildtest.yml` for other tests being executed on shared 
runners.
> +
> +ubuntu-18.04-s390x-notcg:
The "build-tcg-disabled" in `.gitlab-ci.d/buildtest.yml` could be 
mimic-ed here too.
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --disable-tcg
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +# All ubuntu-20.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/qemu/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-20.04-aarch64-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1
> +
> +ubuntu-20.04-aarch64-all:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-alldbg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --disable-libssh
> + - make clean
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-clang:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> +
> +ubuntu-20.04-aarch64-tci:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --enable-tcg-interpreter
> + - make --output-sync -j`nproc`
> +
> +ubuntu-20.04-aarch64-notcg:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> + script:
> + - mkdir build
> + - cd build
> + - ../configure --disable-libssh --disable-tcg
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Wainer dos Santos Moschetta]

On 6/8/21 12:14 AM, Cleber Rosa wrote:
> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
>
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>   .gitlab-ci.d/qemu-project.yml   |  1 +
>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>   docs/devel/index.rst            |  1 +
>   4 files changed, 44 insertions(+)
>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>   create mode 100644 docs/devel/ci.rst


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> new file mode 100644
> index 0000000000..a07b27384c
> --- /dev/null
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -0,0 +1,14 @@
> +# The CI jobs defined here require GitLab runners installed and
> +# registered on machines that match their operating system names,
> +# versions and architectures.  This is in contrast to the other CI
> +# jobs that are intended to run on GitLab's "shared" runners.
> +
> +# Different than the default approach on "shared" runners, based on
> +# containers, the custom runners have no such *requirement*, as those
> +# jobs should be capable of running on operating systems with no
> +# compatible container implementation, or no support from
> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> +# reusing the GIT repository, let's enable the clone strategy, which
> +# guarantees a fresh repository on each job run.
> +variables:
> +  GIT_STRATEGY: clone
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index 64cb2ba1da..dde8270301 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -9,3 +9,4 @@ include:
>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>     - local: '/.gitlab-ci.d/buildtest.yml'
>     - local: '/.gitlab-ci.d/static_checks.yml'
> +  - local: '/.gitlab-ci.d/custom-runners.yml'
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> new file mode 100644
> index 0000000000..585b7bf4b8
> --- /dev/null
> +++ b/docs/devel/ci.rst
> @@ -0,0 +1,28 @@
> +==
> +CI
> +==
> +
> +QEMU has configurations enabled for a number of different CI services.
> +The most up to date information about them and their status can be
> +found at::
> +
> +   https://wiki.qemu.org/Testing/CI
> +
> +Jobs on Custom Runners
> +======================
> +
> +Besides the jobs run under the various CI systems listed before, there
> +are a number additional jobs that will run before an actual merge.
> +These use the same GitLab CI's service/framework already used for all
> +other GitLab based CI jobs, but rely on additional systems, not the
> +ones provided by GitLab as "shared runners".
> +
> +The architecture of GitLab's CI service allows different machines to
> +be set up with GitLab's "agent", called gitlab-runner, which will take
> +care of running jobs created by events such as a push to a branch.
> +Here, the combination of a machine, properly configured with GitLab's
> +gitlab-runner, is called a "custom runner".
> +
> +The GitLab CI jobs definition for the custom runners are located under::
> +
> +  .gitlab-ci.d/custom-runners.yml
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index 791925dcda..c9a02e786e 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -27,6 +27,7 @@ Contents:
>      migration
>      atomics
>      stable-process
> +   ci
>      qtest
>      decodetree
>      secure-coding-practices

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Wainer dos Santos Moschetta]

Hi,

On 6/8/21 12:14 AM, Cleber Rosa wrote:
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   docs/devel/ci.rst                      | 30 ++++++++
>   scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>   scripts/ci/setup/inventory.template    |  1 +
>   3 files changed, 129 insertions(+)
>   create mode 100644 scripts/ci/setup/build-environment.yml
>   create mode 100644 scripts/ci/setup/inventory.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 585b7bf4b8..35c6b5e269 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>   The GitLab CI jobs definition for the custom runners are located under::
>   
>     .gitlab-ci.d/custom-runners.yml
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
Missing to mention the template file.
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  It covers a number of different Linux distributions and
> +FreeBSD.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..664f2f0519
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,98 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all

You will need to "become: yes" if the login user is not privileged, right?

Or mention on the documentation how the user should configure the 
connection for privileged login.

About privilege escalation with Ansible: 
https://docs.ansible.com/ansible/latest/user_guide/become.html

> +  tasks:

Just a tip: you can put all those task under a block 
(https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html) 
so check if "ansible_facts['distribution'] == 'Ubuntu'" only once.

I reviewed the remain of the playbook; it looks good to me.

> +    - name: Update apt cache
> +      apt:
> +        update_cache: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - python3-yaml
> +          - python3-sphinx
> +          - python3-sphinx-rtd-theme
> +          - ninja-build
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - clang
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '18.04'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
> +          - clang-10
> +          - genisoimage
> +          - liblttng-ust-dev
> +          - libslirp-dev
> +          - netcat-openbsd
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '20.04'
> diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory.template
> @@ -0,0 +1 @@
> +localhost

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Wainer dos Santos Moschetta]

Hi,

On 6/8/21 12:14 AM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
>
>   * Ubuntu 20.04 on aarch64
>   * Ubuntu 18.04 on s390x
>
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
>   scripts/ci/setup/.gitignore        |  1 +
>   scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
>   scripts/ci/setup/vars.yml.template | 12 ++++++
>   4 files changed, 131 insertions(+)
>   create mode 100644 scripts/ci/setup/.gitignore
>   create mode 100644 scripts/ci/setup/gitlab-runner.yml
>   create mode 100644 scripts/ci/setup/vars.yml.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 35c6b5e269..bbd89e54d7 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -56,3 +56,60 @@ To run the playbook, execute::
>   
>     cd scripts/ci/setup
>     ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable
> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``
I think you can remove the information about the gitlab-running being 
not available for some systems.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +Ubuntu 20.04 aarch64 system, the tags should be set as::
> +
> +  ubuntu_20.04,aarch64

Also users no longer need manually create the tags.

Remaining of the file looks good to me.

> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..98dab92bb5
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,61 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Set the Operating System for gitlab-runner
> +      set_fact:
> +        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
> +    - debug:
> +        msg: gitlab-runner OS is {{ gitlab_runner_os }}
> +
> +    - name: Set the architecture for gitlab-runner
> +      set_fact:
> +        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
> +    - debug:
> +        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
> +
> +    - name: Download the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..e48089761f
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,12 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.12.0
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# A mapping of the ansible to gitlab architecture nomenclature
> +ansible_to_gitlab_arch:
> +  x86_64: amd64
> +  aarch64: arm64
> +  s390x: s390x
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Wainer dos Santos Moschetta]

[-- Attachment #1: Type: text/plain, Size: 5539 bytes --]

Hi,

On 6/8/21 10:36 AM, Cleber Rosa Junior wrote:
>
>
> On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org 
> <mailto:f4bug@amsat.org>> wrote:
>
>     Hi Alex, Stefan,
>
>     On 6/8/21 5:14 AM, Cleber Rosa wrote:
>     > The QEMU project has two machines (aarch64 and s390x) that can
>     be used
>     > for jobs that do build and run tests.
>
>     AFAIK there is more hardware available to the project, so I'm
>     wondering
>     what happened to the rest, is it a deliberate choice to start small?
>
>
> Hi Phil,
>
> Yes, this series was deliberately focused on the first two machines 
> owned and available to QEMU.
>
>     What will happen with the rest, since we are wasting resources?
>
>
> The plan is to allow all machines (currently available and to-be 
> available) to be connected as custom runners. This hopefully gets that 
> started.
>
> About "more hardware available to the project", there's one VM from 
> fosshost which was made available not long ago, and which I set up 
> even more recently, which could be used as a gitlab runner too.  But, 
> even though some new hardware resource is available (and wasted?), the 
> human resources to maintain them have not been properly determined, so 
> I believe it's a good decision to start with the machines that have 
> been operational for long, and that already have to the best of my 
> knowledge, people maintaining them.
>
> I also see a "Debian unstable mips64el (Debian) @ cipunited.cn 
> <http://cipunited.cn>" registered as a runner, but I don't have extra 
> information about it.
>
> Besides that, I'll send another series shortly, that builds upon this 
> series, and adds a Red Hat focused job, on a Red Hat managed machine.  
> This should be what other entities should be capable of doing and 
> allowed to do.
>
>     Who has access to what and should do what (setup)? How is this list of
>     hw managed btw? Should there be some public visibility (i.e. Wiki)?
>
>
> These are good questions, and I believe Alex can answer them about 
> those two machines.  Even though I hooked them up to GitLab, AFAICT he 
> is the ultimate admin (maybe Peter too?).
>
> About hardware management, it has been suggested to use either the 
> Wiki or a MAINTAINERS entry.  This is still unresolved and feedback 
> would be appreciated.  For me to propose a MAINTAINERS entry, say, on 
> a v7, I'd need the information on who is managing them.
>
>
>     Is there a document explaining what are the steps to follow for an
>     entity to donate / sponsor hardware? Where would it be stored, should
>     this hw be shipped somewhere? What documentation should be
>     provided for
>     its system administration?
>
>     In case an entity manages hosting and maintenance, can the QEMU
>     project
>     share the power bill? Up to which amount?
>     Similar question if a sysadmin has to be paid.
>
>     If the QEMU project can't spend money on CI, what is expected from
>     resource donators? Simply hardware + power (electricity) + network
>     traffic? Also sysadmining and monitoring? Do we expect some kind of
>     reliability on the data stored here or can we assume disposable /
>     transient runners?
>     Should donators also provide storage? Do we have a minimum storage
>     requirement?
>
>     Should we provide some guideline explaining any code might be run by
>     our contributors on these runners and some security measures have to
>     be taken / considered?
>
>     Sponsors usually expect some advertising to thanks them, and often
>     regular reports on how their resources are used, else they might not
>     renew their offer. Who should care to keep the relationship with
>     sponsors?
>
>     Where is defined what belong to the project, who is responsible,
>     who can
>     request access to this hardware, what resource can be used?
>
>
> You obviously directed the question towards Alex and Stefan 
> (rightfully so), so I won't attempt to answer these ones at this point.
>
>     More generically, what is the process for a private / corporate entity
>     to register a runner to the project? (how did it work for this aarch64
>     and s390x one?)
>
>
> The steps are listed on the documentation.  Basically anyone with 
> knowledge of the "registration token" can add new machines to GitLab 
> as runners.  For the two aarch64 and s390x, it was a matter of 
> following the documentation steps which basically involve:
>
> 1) providing the hostname(s) in the inventory file
> 2) provide the "registration token" in the vars.yml file
> 3) running the playbooks
>
>
>     What else am I missing?
>
>
> I think you're missing the answers to all your good questions :).
>
> And I understand that are a lot of them (from everyone, including 
> myself).  The dilemma here is: should we activate the machines already 
> available, and learn in practice, what's missing?  I honestly believe 
> we should.


IMHO we should merge the minimum possible to start using the existing 
machines, then address the questions (good questions, btw) raised by 
Philippe as needed.

Thanks!

- Wainer

>
> Thanks,
> - Clr.
>
>     Thanks,
>
>     Phil.
>
>     > This introduces those jobs,
>     > which are a mapping of custom scripts used for the same purpose.
>     >
>     > Signed-off-by: Cleber Rosa <crosa@redhat.com
>     <mailto:crosa@redhat.com>>
>     > ---
>     >  .gitlab-ci.d/custom-runners.yml | 208
>     ++++++++++++++++++++++++++++++++
>     >  1 file changed, 208 insertions(+)
>

[-- Attachment #2: Type: text/html, Size: 10121 bytes --]

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
>
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      | 30 ++++++++
>  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>  scripts/ci/setup/inventory.template    |  1 +
>  3 files changed, 129 insertions(+)
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 585b7bf4b8..35c6b5e269 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>  The GitLab CI jobs definition for the custom runners are located under::
>  
>    .gitlab-ci.d/custom-runners.yml
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  It covers a number of different Linux distributions and
> +FreeBSD.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml

I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
have up and running as a VM and I got a failure when checking facts:

  14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory build-environment.yml

  PLAY [Installation of basic packages to build QEMU] *************************************************************************************************************************

  TASK [Gathering Facts] ******************************************************************************************************************************************************
  ok: [aarch64.ci.qemu.org]
  ok: [hackbox-ubuntu-2004]

  TASK [Update apt cache] *****************************************************************************************************************************************************
  fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}
  fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}
          to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry

  PLAY RECAP ******************************************************************************************************************************************************************
  aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
  hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1


-- 
Alex Bennée

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa Junior]

[-- Attachment #1: Type: text/plain, Size: 5716 bytes --]

On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Cleber Rosa <crosa@redhat.com> writes:
>
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/ci.rst                      | 30 ++++++++
> >  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
> >  scripts/ci/setup/inventory.template    |  1 +
> >  3 files changed, 129 insertions(+)
> >  create mode 100644 scripts/ci/setup/build-environment.yml
> >  create mode 100644 scripts/ci/setup/inventory.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 585b7bf4b8..35c6b5e269 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >  The GitLab CI jobs definition for the custom runners are located under::
> >
> >    .gitlab-ci.d/custom-runners.yml
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Create an ``inventory`` file
> > +under ``scripts/ci/setup``, such as this::
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  It covers a number of different Linux distributions and
> > +FreeBSD.
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
>
> I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
> have up and running as a VM and I got a failure when checking facts:
>
>   14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1)
> + ansible-playbook -i inventory build-environment.yml
>
>   PLAY [Installation of basic packages to build QEMU]
> *************************************************************************************************************************
>
>   TASK [Gathering Facts]
> ******************************************************************************************************************************************************
>   ok: [aarch64.ci.qemu.org]
>   ok: [hackbox-ubuntu-2004]
>
>   TASK [Update apt cache]
> *****************************************************************************************************************************************************
>   fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check
> 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error
> while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'):
> 'dict object' has no attribute 'distribution'\n\nThe error appears to have
> been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml':
> line 5, column 7, but may\nbe elsewhere in the file depending on the exact
> syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    -
> name: Update apt cache\n      ^ here\n"}
>   fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check
> 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error
> while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'):
> 'dict object' has no attribute 'distribution'\n\nThe error appears to have
> been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml':
> line 5, column 7, but may\nbe elsewhere in the file depending on the exact
> syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    -
> name: Update apt cache\n      ^ here\n"}
>           to retry, use: --limit
> @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry
>
>   PLAY RECAP
> ******************************************************************************************************************************************************************
>   aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0
> failed=1
>   hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0
> failed=1
>
>
>
Hi Alex,

Thanks for checking this version out.  It looks like this is similar to
what happened to you during v5 and hackmox-ubuntu-2004.  Because I had no
issues running the playbook against aarch64.ci.qemu.org, I am now betting
that this is an issue with the ansible installation on the "controller"
machine, that is, the one that drives the playbook execution (the "zen"
host, according to your output).

About the error, the syntax itself is correct[1], but I bet there may be
differences between ansible versions.  Could you please share the output of
"ansible --version" on that machine?

Thanks again,
- Cleber.

[1] -
https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution


> --
> Alex Bennée
>
>

[-- Attachment #2: Type: text/html, Size: 7440 bytes --]

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Tue, Jun 08, 2021 at 08:29:53AM +0200, Philippe Mathieu-Daudé wrote:
> On 6/8/21 5:14 AM, Cleber Rosa wrote:
> Sponsors usually expect some advertising to thanks them, and often
> regular reports on how their resources are used, else they might not
> renew their offer. Who should care to keep the relationship with
> sponsors?

Most sponsors I've encountered did not ask for advertising. Either they
have an interest in test coverage because they ship QEMU to customers
or they provide resources to open source projects and leave all the
detail up to us (they don't need test reports).

But in any case it's easy to have a page on the wiki or website listing
sponsors.

There needs to be a point of contact. Someone who answers questions and
coordinates with sponsors.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Thomas Huth]

On 08/06/2021 05.14, Cleber Rosa wrote:
> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
> 
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>   .gitlab-ci.d/qemu-project.yml   |  1 +
>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>   docs/devel/index.rst            |  1 +
>   4 files changed, 44 insertions(+)
>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>   create mode 100644 docs/devel/ci.rst
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> new file mode 100644
> index 0000000000..a07b27384c
> --- /dev/null
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -0,0 +1,14 @@
> +# The CI jobs defined here require GitLab runners installed and
> +# registered on machines that match their operating system names,
> +# versions and architectures.  This is in contrast to the other CI
> +# jobs that are intended to run on GitLab's "shared" runners.
> +
> +# Different than the default approach on "shared" runners, based on
> +# containers, the custom runners have no such *requirement*, as those
> +# jobs should be capable of running on operating systems with no
> +# compatible container implementation, or no support from
> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> +# reusing the GIT repository, let's enable the clone strategy, which
> +# guarantees a fresh repository on each job run.
> +variables:
> +  GIT_STRATEGY: clone

I assume this now does not influence the shared runners anymore, now that 
the file is handled as last entry in the list below? Or is this a global 
setting that will also be used for the shared runners? Anyway, let's keep it 
for now, we can still try to improve later...

> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index 64cb2ba1da..dde8270301 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -9,3 +9,4 @@ include:
>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>     - local: '/.gitlab-ci.d/buildtest.yml'
>     - local: '/.gitlab-ci.d/static_checks.yml'
> +  - local: '/.gitlab-ci.d/custom-runners.yml'

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v6 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Willian Rampazzo]

On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
>
> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
>
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>  .gitlab-ci.d/qemu-project.yml   |  1 +
>  docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>  docs/devel/index.rst            |  1 +
>  4 files changed, 44 insertions(+)
>  create mode 100644 .gitlab-ci.d/custom-runners.yml
>  create mode 100644 docs/devel/ci.rst
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4293 bytes --]

On Tue, Jun 08, 2021 at 09:36:37AM -0400, Cleber Rosa Junior wrote:
> On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org>
> wrote:

Here are my thoughts. It's just my opinion based on experience running
the old QEMU Buildbot infrastructure and interacting with our hosting
providers. I'm not planning to actively work on the CI infrastructure,
so if you have a strong opinion and want to do things differently
yourself, feel free.

> > Who has access to what and should do what (setup)? How is this list of
> > hw managed btw? Should there be some public visibility (i.e. Wiki)?
> >
> 
> These are good questions, and I believe Alex can answer them about those
> two machines.  Even though I hooked them up to GitLab, AFAICT he is the
> ultimate admin (maybe Peter too?).
> 
> About hardware management, it has been suggested to use either the Wiki or
> a MAINTAINERS entry.  This is still unresolved and feedback would be
> appreciated.  For me to propose a MAINTAINERS entry, say, on a v7, I'd need
> the information on who is managing them.

Here is the wiki page that lists project resources (machines, etc):
https://wiki.qemu.org/AdminContacts

We can continue to use this page.

> > Is there a document explaining what are the steps to follow for an
> > entity to donate / sponsor hardware? Where would it be stored, should
> > this hw be shipped somewhere? What documentation should be provided for
> > its system administration?

A document is needed that explains the process and the
roles/responsibilities of the people involved.

QEMU doesn't have a physical presence and we currently don't have a way
to host physical machines. We also probably shouldn't get involved in
that because it has a high overhead and puts the responsibility on the
project to maintain the hardware. There are hosting providers like
OSUOSL that offer non-x86 architectures, so I don't think we need to
deal with physical hardware even for other architectures. If someone
needs their special hardware covered, let them act as the sponsor and
sysadmin for that machine - they'll need to figure out where to host it.

> > In case an entity manages hosting and maintenance, can the QEMU project
> > share the power bill? Up to which amount?
> > Similar question if a sysadmin has to be paid.

No, it's too complicated. QEMU does not have regular income that could
be used for periodic expenses. We shouldn't spend time worrying about
this unless there is a real need and plenty of funding.

> > If the QEMU project can't spend money on CI, what is expected from
> > resource donators? Simply hardware + power (electricity) + network
> > traffic? Also sysadmining and monitoring? Do we expect some kind of
> > reliability on the data stored here or can we assume disposable /
> > transient runners?

Sponsors provide access to internet-connected machines. They can
designate a QEMU community volunteer to admin machines or they can admin
the machine themselves.

Sysadmins deal with keeping the machine online (security, network
traffic, monitoring, etc).

For simplicity it's best if sponsored machines are owned and paid for by
the sponsor. Billing electricity, bandwidth, etc to QEMU will make
things complicated and we don't have the admin infrastructure to support
that.

> > Should donators also provide storage? Do we have a minimum storage
> > requirement?

Sponsors provide storage. There is a minimum storage requirement that
depends on the nature of CI jobs (I don't know what the exact amount
is and it may change over time).

> > Where is defined what belong to the project, who is responsible, who can
> > request access to this hardware, what resource can be used?

Machines belong to their sponsors, not to the QEMU project. They could
go away in the future if the sponsor decides to withdraw them.

Only the sysadmin has ssh access to the machine. The CI system provides
access to logs so that ssh access to machines is not necessary for QEMU
developers. If ssh access is needed then the developer can ask the
sysadmin for help.

Regarding resource usage, that's up to the sysadmin. If they want to
apply resource limits to the CI environment they need to configure that.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

On Tue, Jun 08, 2021 at 04:07:57PM -0300, Wainer dos Santos Moschetta wrote:
> > And I understand that are a lot of them (from everyone, including
> > myself).  The dilemma here is: should we activate the machines already
> > available, and learn in practice, what's missing?  I honestly believe we
> > should.
> 
> 
> IMHO we should merge the minimum possible to start using the existing
> machines, then address the questions (good questions, btw) raised by
> Philippe as needed.

That sounds good.

Does anyone want to volunteer to be the QEMU CI runners point of contact
who is responsible for defining the process? Cleber's last email left a
lot for Alex and me to define, but I would prefer it if someone who is
not me takes this on since I'm already spread thin. Alex? Philippe? Cleber?
Wainer?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa Junior <crosa@redhat.com> writes:

> On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Cleber Rosa <crosa@redhat.com> writes:
>
>  > To run basic jobs on custom runners, the environment needs to be
>  > properly set up.  The most common requirement is having the right
>  > packages installed.
>  >
>  > The playbook introduced here covers the QEMU's project s390x and
>  > aarch64 machines.  At the time this is being proposed, those machines
>  > have already had this playbook applied to them.
>  >
>  > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>  > ---
>  >  docs/devel/ci.rst                      | 30 ++++++++
>  >  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>  >  scripts/ci/setup/inventory.template    |  1 +
>  >  3 files changed, 129 insertions(+)
>  >  create mode 100644 scripts/ci/setup/build-environment.yml
>  >  create mode 100644 scripts/ci/setup/inventory.template
>  >
>  > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>  > index 585b7bf4b8..35c6b5e269 100644
>  > --- a/docs/devel/ci.rst
>  > +++ b/docs/devel/ci.rst
>  > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>  >  The GitLab CI jobs definition for the custom runners are located under::
>  >  
>  >    .gitlab-ci.d/custom-runners.yml
>  > +
>  > +Machine Setup Howto
>  > +-------------------
>  > +
>  > +For all Linux based systems, the setup can be mostly automated by the
>  > +execution of two Ansible playbooks.  Create an ``inventory`` file
>  > +under ``scripts/ci/setup``, such as this::
>  > +
>  > +  fully.qualified.domain
>  > +  other.machine.hostname
>  > +
>  > +You may need to set some variables in the inventory file itself.  One
>  > +very common need is to tell Ansible to use a Python 3 interpreter on
>  > +those hosts.  This would look like::
>  > +
>  > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>  > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>  > +
>  > +Build environment
>  > +~~~~~~~~~~~~~~~~~
>  > +
>  > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>  > +set up machines with the environment needed to perform builds and run
>  > +QEMU tests.  It covers a number of different Linux distributions and
>  > +FreeBSD.
>  > +
>  > +To run the playbook, execute::
>  > +
>  > +  cd scripts/ci/setup
>  > +  ansible-playbook -i inventory build-environment.yml
>
>  I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
>  have up and running as a VM and I got a failure when checking facts:
>
>    14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory build-environment.yml
>
>    PLAY [Installation of basic packages to build QEMU]
>  *************************************************************************************************************************
>
>    TASK [Gathering Facts]
>  ******************************************************************************************************************************************************
>  
>    ok: [aarch64.ci.qemu.org]
>    ok: [hackbox-ubuntu-2004]
>
>    TASK [Update apt cache]
>  *****************************************************************************************************************************************************
>  
>    fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
>  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt
>  cache\n      ^ here\n"}
>    fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
>  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt
>  cache\n      ^ here\n"}
>            to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry
>
>    PLAY RECAP
>  ******************************************************************************************************************************************************************
>  
>    aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
>    hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1
>
> Hi Alex,
>
> Thanks for checking this version out.  It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004.  Because I
> had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation on the
> "controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).
>
> About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions.  Could you please share the
> output of "ansible --version" on that machine?

15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version
ansible 2.7.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

My machine is Debian stable (aka Buster)

>
> Thanks again,
> - Cleber.
>
> [1] - https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution
>  
>  -- 
>  Alex Bennée


-- 
Alex Bennée

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Alex Bennée]

Wainer dos Santos Moschetta <wainersm@redhat.com> writes:

> Hi,
>
> On 6/8/21 12:14 AM, Cleber Rosa wrote:
>> The QEMU project has two machines (aarch64 and s390x) that can be used
>> for jobs that do build and run tests.  This introduces those jobs,
>> which are a mapping of custom scripts used for the same purpose.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>>   1 file changed, 208 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
>> index a07b27384c..061d3cdfed 100644
>> --- a/.gitlab-ci.d/custom-runners.yml
>> +++ b/.gitlab-ci.d/custom-runners.yml
>> @@ -12,3 +12,211 @@
>>   # guarantees a fresh repository on each job run.
>>   variables:
>>     GIT_STRATEGY: clone
>> +
>> +# All ubuntu-18.04 jobs should run successfully in an environment
>> +# setup by the scripts/ci/setup/build-environment.yml task
>> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
>> +ubuntu-18.04-s390x-all-linux-static:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>
> Should it restrict the job for pushes to qemu-project only? If yes,
> then it probably needs the statement:
>
> '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
>
> If you change that here, you will end it changing all over the jobs.
> In general, there are many boilerplates in this file. I'm ok to merge
> it as is as long as it is followed by another series to refactor the
> code.
>
>> + script:
>> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
>> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
>> + - mkdir build
>> + - cd build
>> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> + - make --output-sync -j`nproc` check-tcg V=1
>> +
>> +ubuntu-18.04-s390x-all:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +ubuntu-18.04-s390x-alldbg:
> Maybe we don't need both ubuntu-18.04-s390x-all and
> ubuntu-18.04-s390x-alldbg jobs.
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --enable-debug --disable-libssh
>> + - make clean
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +ubuntu-18.04-s390x-clang:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> +   when: manual
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +ubuntu-18.04-s390x-tci:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --enable-tcg-interpreter
>> + - make --output-sync -j`nproc`
> I think it needs to `make check-tcg` at least. See "build-tci" in
> `.gitlab-ci.d/buildtest.yml` for other tests being executed on shared
> runners.

To get anything other than the s390x-linux-user tests we will need the
cross compilers installed. Currently we don't really use docker for
anything other than x86_64 hosts (and some aarch64 which I've tested).

>> +
>> +ubuntu-18.04-s390x-notcg:
> The "build-tcg-disabled" in `.gitlab-ci.d/buildtest.yml` could be
> mimic-ed here too.
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> +   when: manual
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --disable-tcg
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +# All ubuntu-20.04 jobs should run successfully in an environment
>> +# setup by the scripts/ci/setup/qemu/build-environment.yml task
>> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
>> +ubuntu-20.04-aarch64-all-linux-static:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
>> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
>> + - mkdir build
>> + - cd build
>> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> + - make --output-sync -j`nproc` check-tcg V=1
>> +
>> +ubuntu-20.04-aarch64-all:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +ubuntu-20.04-aarch64-alldbg:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --enable-debug --disable-libssh
>> + - make clean
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +ubuntu-20.04-aarch64-clang:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> +   when: manual
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> +
>> +ubuntu-20.04-aarch64-tci:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --enable-tcg-interpreter
>> + - make --output-sync -j`nproc`
>> +
>> +ubuntu-20.04-aarch64-notcg:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_20.04
>> + - aarch64
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>> +   when: manual
>> + script:
>> + - mkdir build
>> + - cd build
>> + - ../configure --disable-libssh --disable-tcg
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1


-- 
Alex Bennée

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Willian Rampazzo]

On Tue, Jun 8, 2021 at 3:48 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi,
>
> On 6/8/21 12:14 AM, Cleber Rosa wrote:
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   docs/devel/ci.rst                      | 30 ++++++++
> >   scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
> >   scripts/ci/setup/inventory.template    |  1 +
> >   3 files changed, 129 insertions(+)
> >   create mode 100644 scripts/ci/setup/build-environment.yml
> >   create mode 100644 scripts/ci/setup/inventory.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 585b7bf4b8..35c6b5e269 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >   The GitLab CI jobs definition for the custom runners are located under::
> >
> >     .gitlab-ci.d/custom-runners.yml
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Create an ``inventory`` file
> > +under ``scripts/ci/setup``, such as this::
> Missing to mention the template file.
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  It covers a number of different Linux distributions and
> > +FreeBSD.
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
> > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> > new file mode 100644
> > index 0000000000..664f2f0519
> > --- /dev/null
> > +++ b/scripts/ci/setup/build-environment.yml
> > @@ -0,0 +1,98 @@
> > +---
> > +- name: Installation of basic packages to build QEMU
> > +  hosts: all
>
> You will need to "become: yes" if the login user is not privileged, right?
>
> Or mention on the documentation how the user should configure the
> connection for privileged login.

As this will vary from system to system, I think it is worth
mentioning in the documentation it can be configured in the inventory
file, adding the variable ansible_become=yes and
ansible_become_password= if password is needed to sudo.

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa Junior]

[-- Attachment #1: Type: text/plain, Size: 7481 bytes --]

On Wed, Jun 9, 2021 at 11:26 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Cleber Rosa Junior <crosa@redhat.com> writes:
>
> > On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >
> >  Cleber Rosa <crosa@redhat.com> writes:
> >
> >  > To run basic jobs on custom runners, the environment needs to be
> >  > properly set up.  The most common requirement is having the right
> >  > packages installed.
> >  >
> >  > The playbook introduced here covers the QEMU's project s390x and
> >  > aarch64 machines.  At the time this is being proposed, those machines
> >  > have already had this playbook applied to them.
> >  >
> >  > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >  > ---
> >  >  docs/devel/ci.rst                      | 30 ++++++++
> >  >  scripts/ci/setup/build-environment.yml | 98
> ++++++++++++++++++++++++++
> >  >  scripts/ci/setup/inventory.template    |  1 +
> >  >  3 files changed, 129 insertions(+)
> >  >  create mode 100644 scripts/ci/setup/build-environment.yml
> >  >  create mode 100644 scripts/ci/setup/inventory.template
> >  >
> >  > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> >  > index 585b7bf4b8..35c6b5e269 100644
> >  > --- a/docs/devel/ci.rst
> >  > +++ b/docs/devel/ci.rst
> >  > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >  >  The GitLab CI jobs definition for the custom runners are located
> under::
> >  >
> >  >    .gitlab-ci.d/custom-runners.yml
> >  > +
> >  > +Machine Setup Howto
> >  > +-------------------
> >  > +
> >  > +For all Linux based systems, the setup can be mostly automated by the
> >  > +execution of two Ansible playbooks.  Create an ``inventory`` file
> >  > +under ``scripts/ci/setup``, such as this::
> >  > +
> >  > +  fully.qualified.domain
> >  > +  other.machine.hostname
> >  > +
> >  > +You may need to set some variables in the inventory file itself.  One
> >  > +very common need is to tell Ansible to use a Python 3 interpreter on
> >  > +those hosts.  This would look like::
> >  > +
> >  > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> >  > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> >  > +
> >  > +Build environment
> >  > +~~~~~~~~~~~~~~~~~
> >  > +
> >  > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> >  > +set up machines with the environment needed to perform builds and run
> >  > +QEMU tests.  It covers a number of different Linux distributions and
> >  > +FreeBSD.
> >  > +
> >  > +To run the playbook, execute::
> >  > +
> >  > +  cd scripts/ci/setup
> >  > +  ansible-playbook -i inventory build-environment.yml
> >
> >  I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
> >  have up and running as a VM and I got a failure when checking facts:
> >
> >    14:26:26 [alex@zen:~/l/q/s/c/setup]
> review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory
> build-environment.yml
> >
> >    PLAY [Installation of basic packages to build QEMU]
> >
> *************************************************************************************************************************
> >
> >    TASK [Gathering Facts]
> >
> ******************************************************************************************************************************************************
> >
> >    ok: [aarch64.ci.qemu.org]
> >    ok: [hackbox-ubuntu-2004]
> >
> >    TASK [Update apt cache]
> >
> *****************************************************************************************************************************************************
> >
> >    fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional
> check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
> >  was: error while evaluating conditional (ansible_facts['distribution']
> == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
> >  error appears to have been in
> '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5,
> column 7, but may\nbe
> >  elsewhere in the file depending on the exact syntax problem.\n\nThe
> offending line appears to be:\n\n  tasks:\n    - name: Update apt
> >  cache\n      ^ here\n"}
> >    fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional
> check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error
> >  was: error while evaluating conditional (ansible_facts['distribution']
> == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
> >  error appears to have been in
> '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5,
> column 7, but may\nbe
> >  elsewhere in the file depending on the exact syntax problem.\n\nThe
> offending line appears to be:\n\n  tasks:\n    - name: Update apt
> >  cache\n      ^ here\n"}
> >            to retry, use: --limit
> @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry
> >
> >    PLAY RECAP
> >
> ******************************************************************************************************************************************************************
> >
> >    aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0
> failed=1
> >    hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0
> failed=1
> >
> > Hi Alex,
> >
> > Thanks for checking this version out.  It looks like this is similar to
> what happened to you during v5 and hackmox-ubuntu-2004.  Because I
> > had no issues running the playbook against aarch64.ci.qemu.org, I am
> now betting that this is an issue with the ansible installation on the
> > "controller" machine, that is, the one that drives the playbook
> execution (the "zen" host, according to your output).
> >
> > About the error, the syntax itself is correct[1], but I bet there may be
> differences between ansible versions.  Could you please share the
> > output of "ansible --version" on that machine?
>
> 15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4
> + ansible --version
> ansible 2.7.7
>   config file = /etc/ansible/ansible.cfg
>   configured module search path = ['/home/alex/.ansible/plugins/modules',
> '/usr/share/ansible/plugins/modules']
>   ansible python module location = /usr/lib/python3/dist-packages/ansible
>   executable location = /usr/bin/ansible
>   python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
>
> My machine is Debian stable (aka Buster)
>
>
Hi Alex,

I found out, the hard way, that one will need ansible 2.8.x.  I'll make a
note about that in the documentation. Long version:

I was able to replicate this on Debian stable, using the exact same
versions as you did.  Then I tried the latest ansible 2.7.18 (the latest in
the 2.7.x series), installed with "pip install ansible==2.7.18 --user".
Still, the exact same issue.

Then, still on Debian stable, I tried 2.8.20 (installed with "pip install
ansible==2.8.20 --user"), and the playbook execution completed successfully.

I went through the 2.7x. and 2.8.x changelogs, and I could not spot one
clear entry about the change in behavior though.

Thanks,
- Cleber.


> >
> > Thanks again,
> > - Cleber.
> >
> > [1] -
> https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution
> >
> >  --
> >  Alex Bennée
>
>
> --
> Alex Bennée
>
>

[-- Attachment #2: Type: text/html, Size: 9974 bytes --]

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Willian Rampazzo]

On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
>
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      | 30 ++++++++
>  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>  scripts/ci/setup/inventory.template    |  1 +
>  3 files changed, 129 insertions(+)
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 585b7bf4b8..35c6b5e269 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>  The GitLab CI jobs definition for the custom runners are located under::
>
>    .gitlab-ci.d/custom-runners.yml
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +

As I mentioned to Wainer, my suggestion is to include a comment about
ansible_become=yes and
ansible_become_password= variables as some systems may need it.

> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  It covers a number of different Linux distributions and
> +FreeBSD.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..664f2f0519
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,98 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Update apt cache
> +      apt:
> +        update_cache: yes

On a freshly installed Ubuntu 20.04, the script failed for me with
dependency messages on the apt side. After I updated the packages on
the system, the playbook worked without problems.

So, my suggestion is to add the "update = yes" here, or add a note in
the documentation asking the user to update the system before running
the playbook.

Except for the above comment and Wainer's comments, it looks good to
me. With these changes:

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Willian Rampazzo]

On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
>
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
>
>  * Ubuntu 20.04 on aarch64
>  * Ubuntu 18.04 on s390x
>
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 12 ++++++
>  4 files changed, 131 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 35c6b5e269..bbd89e54d7 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -56,3 +56,60 @@ To run the playbook, execute::
>
>    cd scripts/ci/setup
>    ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears like icon), then

* Settings (the gears like icon in the end of the left menu), then

I took some time to find it as it was hidden at the end of the scrolling.

> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"

For me, it shows: "And this registration token:"

> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable

If you keep this block (see comment from Wainer), s/avaiable/available/

> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +Ubuntu 20.04 aarch64 system, the tags should be set as::
> +
> +  ubuntu_20.04,aarch64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..98dab92bb5
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,61 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Set the Operating System for gitlab-runner
> +      set_fact:
> +        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
> +    - debug:
> +        msg: gitlab-runner OS is {{ gitlab_runner_os }}
> +
> +    - name: Set the architecture for gitlab-runner
> +      set_fact:
> +        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
> +    - debug:
> +        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
> +
> +    - name: Download the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..e48089761f
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,12 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.12.0
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# A mapping of the ansible to gitlab architecture nomenclature
> +ansible_to_gitlab_arch:
> +  x86_64: amd64
> +  aarch64: arm64
> +  s390x: s390x
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
> --
> 2.25.4
>

With my suggestions and Wainer's:

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Willian Rampazzo]

On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
>
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>  1 file changed, 208 insertions(+)
>

Based on the comment from the cover letter that these jobs are defined
as trying to mimic what Peter runs on staging, the code looks good to
me, so:

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Thomas Huth]

On 08/06/2021 05.14, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
>   1 file changed, 208 insertions(+)
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index a07b27384c..061d3cdfed 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -12,3 +12,211 @@
>   # guarantees a fresh repository on each job run.
>   variables:
>     GIT_STRATEGY: clone
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'

I don't think this will work very well... sub-maintainers might want to push 
to a "staging" branch in their forked repositories, and without the s390x 
runner, the pipeline gets stuck now:

  https://gitlab.com/thuth/qemu/-/pipelines/317812558

We had the same issue in the kvm-unit-test CI, and we solved it there by 
rather making it depend on an environment variable that has to be set if the 
runner is available:

  only:
    variables:
     - $S390X_RUNNER_AVAILABLE

I think that's also nicer in case someone brings their own s390x runner and 
want to use the CI tests on other branches than staging.

Could you please change your patch accordingly?

  Thanks,
   Thomas

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Thomas Huth]

On 08/06/2021 05.14, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers the Linux distributions and
> has been primarily tested on OS/machines that the QEMU project
> has available to act as runners, namely:
> 
>   * Ubuntu 20.04 on aarch64
>   * Ubuntu 18.04 on s390x
> 
> But, it should work on all other Linux distributions.  Earlier
> versions were tested on FreeBSD too, so chances of success are
> high.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
>   scripts/ci/setup/.gitignore        |  1 +
>   scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
>   scripts/ci/setup/vars.yml.template | 12 ++++++
>   4 files changed, 131 insertions(+)
>   create mode 100644 scripts/ci/setup/.gitignore
>   create mode 100644 scripts/ci/setup/gitlab-runner.yml
>   create mode 100644 scripts/ci/setup/vars.yml.template
[...]
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file

Add a newline, please.

> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..98dab92bb5
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,61 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"

Could you please add a comment at the top of the file or name it differently 
so that it is clear from a quick glance that this is an ansible playbook? 
Poeple might later wonder otherwise...

  Thomas

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Erik Skultety]

...
> On a freshly installed Ubuntu 20.04, the script failed for me with
> dependency messages on the apt side. After I updated the packages on
> the system, the playbook worked without problems.
> 
> So, my suggestion is to add the "update = yes" here, or add a note in
> the documentation asking the user to update the system before running
> the playbook.

I'd advocate for always updating the system automatically rather than putting
it into the documentation and relying on users actually doing it.

Erik

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa Junior <crosa@redhat.com> writes:

> On Wed, Jun 9, 2021 at 11:26 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Cleber Rosa Junior <crosa@redhat.com> writes:
>
>  > On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>  >
>  >  Cleber Rosa <crosa@redhat.com> writes:
>  >
>  >  > To run basic jobs on custom runners, the environment needs to be
>  >  > properly set up.  The most common requirement is having the right
>  >  > packages installed.
>  >  >
>  >  > The playbook introduced here covers the QEMU's project s390x and
>  >  > aarch64 machines.  At the time this is being proposed, those machines
>  >  > have already had this playbook applied to them.
>  >  >
>  >  > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>  >  > ---
>  >  >  docs/devel/ci.rst                      | 30 ++++++++
>  >  >  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>  >  >  scripts/ci/setup/inventory.template    |  1 +
>  >  >  3 files changed, 129 insertions(+)
>  >  >  create mode 100644 scripts/ci/setup/build-environment.yml
>  >  >  create mode 100644 scripts/ci/setup/inventory.template
>  >  >
>  >  > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>  >  > index 585b7bf4b8..35c6b5e269 100644
>  >  > --- a/docs/devel/ci.rst
>  >  > +++ b/docs/devel/ci.rst
>  >  > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>  >  >  The GitLab CI jobs definition for the custom runners are located under::
>  >  >  
>  >  >    .gitlab-ci.d/custom-runners.yml
>  >  > +
>  >  > +Machine Setup Howto
>  >  > +-------------------
>  >  > +
>  >  > +For all Linux based systems, the setup can be mostly automated by the
>  >  > +execution of two Ansible playbooks.  Create an ``inventory`` file
>  >  > +under ``scripts/ci/setup``, such as this::
>  >  > +
>  >  > +  fully.qualified.domain
>  >  > +  other.machine.hostname
>  >  > +
>  >  > +You may need to set some variables in the inventory file itself.  One
>  >  > +very common need is to tell Ansible to use a Python 3 interpreter on
>  >  > +those hosts.  This would look like::
>  >  > +
>  >  > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>  >  > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>  >  > +
>  >  > +Build environment
>  >  > +~~~~~~~~~~~~~~~~~
>  >  > +
>  >  > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>  >  > +set up machines with the environment needed to perform builds and run
>  >  > +QEMU tests.  It covers a number of different Linux distributions and
>  >  > +FreeBSD.
>  >  > +
>  >  > +To run the playbook, execute::
>  >  > +
>  >  > +  cd scripts/ci/setup
>  >  > +  ansible-playbook -i inventory build-environment.yml
>  >
>  >  I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
>  >  have up and running as a VM and I got a failure when checking facts:
>  >
>  >    14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory
>  build-environment.yml
>  >
>  >    PLAY [Installation of basic packages to build QEMU]
>  >  *************************************************************************************************************************
>  >
>  >    TASK [Gathering Facts]
>  > 
>  ******************************************************************************************************************************************************
>  
>  >  
>  >    ok: [aarch64.ci.qemu.org]
>  >    ok: [hackbox-ubuntu-2004]
>  >
>  >    TASK [Update apt cache]
>  > 
>  *****************************************************************************************************************************************************
>  
>  >  
>  >    fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The
>  error
>  >  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>  >  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>  >  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update
>  apt
>  >  cache\n      ^ here\n"}
>  >    fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The
>  error
>  >  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>  >  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>  >  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update
>  apt
>  >  cache\n      ^ here\n"}
>  >            to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry
>  >
>  >    PLAY RECAP
>  > 
>  ******************************************************************************************************************************************************************
>  
>  >  
>  >    aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
>  >    hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1
>  >
>  > Hi Alex,
>  >
>  > Thanks for checking this version out.  It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004. 
>  Because I
>  > had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation
>  on the
>  > "controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).
>  >
>  > About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions.  Could you please share
>  the
>  > output of "ansible --version" on that machine?
>
>  15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version
>  ansible 2.7.7
>    config file = /etc/ansible/ansible.cfg
>    configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
>    ansible python module location = /usr/lib/python3/dist-packages/ansible
>    executable location = /usr/bin/ansible
>    python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
>
>  My machine is Debian stable (aka Buster)
>
> Hi Alex,
>
> I found out, the hard way, that one will need ansible 2.8.x.  I'll
> make a note about that in the documentation. Long version:

OK - yes please and I'll test with a local installed version. Is it
possible for the playbook to error out with a useful warning if the
version of ansible isn't new enough?

>
> I was able to replicate this on Debian stable, using the exact same versions as you did.  Then I tried the latest ansible 2.7.18 (the latest in
> the 2.7.x series), installed with "pip install ansible==2.7.18 --user".  Still, the exact same issue.
>
> Then, still on Debian stable, I tried 2.8.20 (installed with "pip install ansible==2.8.20 --user"), and the playbook execution completed
> successfully.
>
> I went through the 2.7x. and 2.8.x changelogs, and I could not spot one clear entry about the change in behavior though.
>
> Thanks,
> - Cleber.
>  
>  >
>  > Thanks again,
>  > - Cleber.
>  >
>  > [1] - https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#ansible-facts-distribution
>  >  
>  >  -- 
>  >  Alex Bennée
>
>  -- 
>  Alex Bennée


-- 
Alex Bennée

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Alex Bennée]

Cleber Rosa Junior <crosa@redhat.com> writes:

> On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
>  Hi Alex, Stefan,
>
>  On 6/8/21 5:14 AM, Cleber Rosa wrote:
>  > The QEMU project has two machines (aarch64 and s390x) that can be used
>  > for jobs that do build and run tests.
>
<snip>
>  
>  Who has access to what and should do what (setup)? How is this list of
>  hw managed btw? Should there be some public visibility (i.e. Wiki)?
>
> These are good questions, and I believe Alex can answer them about those two machines.  Even though I hooked them up to GitLab,
> AFAICT he is the ultimate admin (maybe Peter too?).
>
> About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry.  This is still unresolved and feedback
> would be appreciated.  For me to propose a MAINTAINERS entry, say, on
> a v7, I'd need the information on who is managing them.

I can only talk about aarch64.ci.qemu.org which is a donated Equinix
machine that comes from the WorksOnArm initiative. I applied for it on
behalf of the QEMU project and we can have it for as long as it's
useful.

I don't know if we need anything more that documenting the nominal
contacts in:

  https://wiki.qemu.org/AdminContacts

>  Is there a document explaining what are the steps to follow for an
>  entity to donate / sponsor hardware? Where would it be stored, should
>  this hw be shipped somewhere? What documentation should be provided for
>  its system administration?

I think the project can only really work with donations out of someones
data centre where they keep responsibility for the physical aspects of
any machines including the ongoing hosting and running costs.

-- 
Alex Bennée

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

On 6/11/21 6:40 AM, Alex Bennée wrote:

> Cleber Rosa Junior <crosa@redhat.com> writes:
>
>> On Wed, Jun 9, 2021 at 11:26 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>   Cleber Rosa Junior <crosa@redhat.com> writes:
>>
>>   > On Wed, Jun 9, 2021 at 9:36 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>   >
>>   >  Cleber Rosa <crosa@redhat.com> writes:
>>   >
>>   >  > To run basic jobs on custom runners, the environment needs to be
>>   >  > properly set up.  The most common requirement is having the right
>>   >  > packages installed.
>>   >  >
>>   >  > The playbook introduced here covers the QEMU's project s390x and
>>   >  > aarch64 machines.  At the time this is being proposed, those machines
>>   >  > have already had this playbook applied to them.
>>   >  >
>>   >  > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>   >  > ---
>>   >  >  docs/devel/ci.rst                      | 30 ++++++++
>>   >  >  scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>>   >  >  scripts/ci/setup/inventory.template    |  1 +
>>   >  >  3 files changed, 129 insertions(+)
>>   >  >  create mode 100644 scripts/ci/setup/build-environment.yml
>>   >  >  create mode 100644 scripts/ci/setup/inventory.template
>>   >  >
>>   >  > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>>   >  > index 585b7bf4b8..35c6b5e269 100644
>>   >  > --- a/docs/devel/ci.rst
>>   >  > +++ b/docs/devel/ci.rst
>>   >  > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>>   >  >  The GitLab CI jobs definition for the custom runners are located under::
>>   >  >
>>   >  >    .gitlab-ci.d/custom-runners.yml
>>   >  > +
>>   >  > +Machine Setup Howto
>>   >  > +-------------------
>>   >  > +
>>   >  > +For all Linux based systems, the setup can be mostly automated by the
>>   >  > +execution of two Ansible playbooks.  Create an ``inventory`` file
>>   >  > +under ``scripts/ci/setup``, such as this::
>>   >  > +
>>   >  > +  fully.qualified.domain
>>   >  > +  other.machine.hostname
>>   >  > +
>>   >  > +You may need to set some variables in the inventory file itself.  One
>>   >  > +very common need is to tell Ansible to use a Python 3 interpreter on
>>   >  > +those hosts.  This would look like::
>>   >  > +
>>   >  > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>>   >  > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>>   >  > +
>>   >  > +Build environment
>>   >  > +~~~~~~~~~~~~~~~~~
>>   >  > +
>>   >  > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>>   >  > +set up machines with the environment needed to perform builds and run
>>   >  > +QEMU tests.  It covers a number of different Linux distributions and
>>   >  > +FreeBSD.
>>   >  > +
>>   >  > +To run the playbook, execute::
>>   >  > +
>>   >  > +  cd scripts/ci/setup
>>   >  > +  ansible-playbook -i inventory build-environment.yml
>>   >
>>   >  I tried this to re-update aarch64.ci.qemu.org and another ubuntu box I
>>   >  have up and running as a VM and I got a failure when checking facts:
>>   >
>>   >    14:26:26 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) + ansible-playbook -i inventory
>>   build-environment.yml
>>   >
>>   >    PLAY [Installation of basic packages to build QEMU]
>>   >  *************************************************************************************************************************
>>   >
>>   >    TASK [Gathering Facts]
>>   >
>>   ******************************************************************************************************************************************************
>>   
>>   >
>>   >    ok: [aarch64.ci.qemu.org]
>>   >    ok: [hackbox-ubuntu-2004]
>>   >
>>   >    TASK [Update apt cache]
>>   >
>>   *****************************************************************************************************************************************************
>>   
>>   >
>>   >    fatal: [aarch64.ci.qemu.org]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The
>>   error
>>   >  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>>   >  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>>   >  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update
>>   apt
>>   >  cache\n      ^ here\n"}
>>   >    fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The
>>   error
>>   >  was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe
>>   >  error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe
>>   >  elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update
>>   apt
>>   >  cache\n      ^ here\n"}
>>   >            to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.retry
>>   >
>>   >    PLAY RECAP
>>   >
>>   ******************************************************************************************************************************************************************
>>   
>>   >
>>   >    aarch64.ci.qemu.org        : ok=1    changed=0    unreachable=0    failed=1
>>   >    hackbox-ubuntu-2004        : ok=1    changed=0    unreachable=0    failed=1
>>   >
>>   > Hi Alex,
>>   >
>>   > Thanks for checking this version out.  It looks like this is similar to what happened to you during v5 and hackmox-ubuntu-2004.
>>   Because I
>>   > had no issues running the playbook against aarch64.ci.qemu.org, I am now betting that this is an issue with the ansible installation
>>   on the
>>   > "controller" machine, that is, the one that drives the playbook execution (the "zen" host, according to your output).
>>   >
>>   > About the error, the syntax itself is correct[1], but I bet there may be differences between ansible versions.  Could you please share
>>   the
>>   > output of "ansible --version" on that machine?
>>
>>   15:41:21 [alex@zen:~/l/q/s/c/setup] review/custom-runners-v6|✚1…(+1/-1) 4 + ansible --version
>>   ansible 2.7.7
>>     config file = /etc/ansible/ansible.cfg
>>     configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
>>     ansible python module location = /usr/lib/python3/dist-packages/ansible
>>     executable location = /usr/bin/ansible
>>     python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
>>
>>   My machine is Debian stable (aka Buster)
>>
>> Hi Alex,
>>
>> I found out, the hard way, that one will need ansible 2.8.x.  I'll
>> make a note about that in the documentation. Long version:
> OK - yes please and I'll test with a local installed version. Is it
> possible for the playbook to error out with a useful warning if the
> version of ansible isn't new enough?


Hi Alex,


Yes, I'll add the version check on v7.  The planned error message will be:


"Unsuitable ansible version, please use version 2.8.0 or later"


- Cleber.

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

On Tue, Jun 8, 2021 at 2:48 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi,
>
> On 6/8/21 12:14 AM, Cleber Rosa wrote:
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   docs/devel/ci.rst                      | 30 ++++++++
> >   scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
> >   scripts/ci/setup/inventory.template    |  1 +
> >   3 files changed, 129 insertions(+)
> >   create mode 100644 scripts/ci/setup/build-environment.yml
> >   create mode 100644 scripts/ci/setup/inventory.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 585b7bf4b8..35c6b5e269 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >   The GitLab CI jobs definition for the custom runners are located under::
> >
> >     .gitlab-ci.d/custom-runners.yml
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Create an ``inventory`` file
> > +under ``scripts/ci/setup``, such as this::
> Missing to mention the template file.
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  It covers a number of different Linux distributions and
> > +FreeBSD.
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
> > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> > new file mode 100644
> > index 0000000000..664f2f0519
> > --- /dev/null
> > +++ b/scripts/ci/setup/build-environment.yml
> > @@ -0,0 +1,98 @@
> > +---
> > +- name: Installation of basic packages to build QEMU
> > +  hosts: all
>
> You will need to "become: yes" if the login user is not privileged, right?
>

Hi Wainer,

Yes, pretty much all of the actions here need a privileged user.  I'm
not sure how much of Ansible basics we should be replicating here.  In
my experience, users running "ansible-playbook" will resort to a
combination of --become, --become-method, --ask-become-pass, etc, if
needed.

> Or mention on the documentation how the user should configure the
> connection for privileged login.
>

Yes, I think it's better to just put a note/reference, instead of
inundating the user and the playbook with possibly unnecessary
configurations and details.  I've added the following to the
documentation:

Please note that most of the tasks in the playbook require superuser
privileges, such as those from the ``root`` account or those obtained
by ``sudo``.  If necessary, please refer to ``ansible-playbook`` options
such as ``--become``, ``--become-method``, ``--become-user`` and
``--ask-become-pass``.

Let me know if that sounds good to you.

> About privilege escalation with Ansible:
> https://docs.ansible.com/ansible/latest/user_guide/become.html
>
> > +  tasks:
>
> Just a tip: you can put all those task under a block
> (https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html)
> so check if "ansible_facts['distribution'] == 'Ubuntu'" only once.
>

There will be other non-Ubuntu tasks added later (such as in the
CentOS Stream 8 series), so I think it's better to keep those
separate.  Also, the Ansible version check which Alex suggested is not
limited to Ubuntu.

> I reviewed the remain of the playbook; it looks good to me.
>

Thanks!
- Cleber.

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

On 6/9/21 12:13 PM, Willian Rampazzo wrote:
> On Tue, Jun 8, 2021 at 3:48 PM Wainer dos Santos Moschetta
> <wainersm@redhat.com> wrote:
>> Hi,
>>
>> On 6/8/21 12:14 AM, Cleber Rosa wrote:
>>> To run basic jobs on custom runners, the environment needs to be
>>> properly set up.  The most common requirement is having the right
>>> packages installed.
>>>
>>> The playbook introduced here covers the QEMU's project s390x and
>>> aarch64 machines.  At the time this is being proposed, those machines
>>> have already had this playbook applied to them.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>    docs/devel/ci.rst                      | 30 ++++++++
>>>    scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>>>    scripts/ci/setup/inventory.template    |  1 +
>>>    3 files changed, 129 insertions(+)
>>>    create mode 100644 scripts/ci/setup/build-environment.yml
>>>    create mode 100644 scripts/ci/setup/inventory.template
>>>
>>> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>>> index 585b7bf4b8..35c6b5e269 100644
>>> --- a/docs/devel/ci.rst
>>> +++ b/docs/devel/ci.rst
>>> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>>>    The GitLab CI jobs definition for the custom runners are located under::
>>>
>>>      .gitlab-ci.d/custom-runners.yml
>>> +
>>> +Machine Setup Howto
>>> +-------------------
>>> +
>>> +For all Linux based systems, the setup can be mostly automated by the
>>> +execution of two Ansible playbooks.  Create an ``inventory`` file
>>> +under ``scripts/ci/setup``, such as this::
>> Missing to mention the template file.
>>> +
>>> +  fully.qualified.domain
>>> +  other.machine.hostname
>>> +
>>> +You may need to set some variables in the inventory file itself.  One
>>> +very common need is to tell Ansible to use a Python 3 interpreter on
>>> +those hosts.  This would look like::
>>> +
>>> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>>> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>>> +
>>> +Build environment
>>> +~~~~~~~~~~~~~~~~~
>>> +
>>> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>>> +set up machines with the environment needed to perform builds and run
>>> +QEMU tests.  It covers a number of different Linux distributions and
>>> +FreeBSD.
>>> +
>>> +To run the playbook, execute::
>>> +
>>> +  cd scripts/ci/setup
>>> +  ansible-playbook -i inventory build-environment.yml
>>> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
>>> new file mode 100644
>>> index 0000000000..664f2f0519
>>> --- /dev/null
>>> +++ b/scripts/ci/setup/build-environment.yml
>>> @@ -0,0 +1,98 @@
>>> +---
>>> +- name: Installation of basic packages to build QEMU
>>> +  hosts: all
>> You will need to "become: yes" if the login user is not privileged, right?
>>
>> Or mention on the documentation how the user should configure the
>> connection for privileged login.
> As this will vary from system to system, I think it is worth
> mentioning in the documentation it can be configured in the inventory
> file, adding the variable ansible_become=yes and
> ansible_become_password= if password is needed to sudo.
>
Agreed.  But I'll put a mention to the ansible-playbook command line 
options instead. I believe our savvy users will be able to decide 
whether to use the command line options, the inventory options, or even 
temporary changes to the playbook.


Thanks!

- Cleber.

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

On 6/9/21 1:16 PM, Willian Rampazzo wrote:
> On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
>> To run basic jobs on custom runners, the environment needs to be
>> properly set up.  The most common requirement is having the right
>> packages installed.
>>
>> The playbook introduced here covers the QEMU's project s390x and
>> aarch64 machines.  At the time this is being proposed, those machines
>> have already had this playbook applied to them.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   docs/devel/ci.rst                      | 30 ++++++++
>>   scripts/ci/setup/build-environment.yml | 98 ++++++++++++++++++++++++++
>>   scripts/ci/setup/inventory.template    |  1 +
>>   3 files changed, 129 insertions(+)
>>   create mode 100644 scripts/ci/setup/build-environment.yml
>>   create mode 100644 scripts/ci/setup/inventory.template
>>
>> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>> index 585b7bf4b8..35c6b5e269 100644
>> --- a/docs/devel/ci.rst
>> +++ b/docs/devel/ci.rst
>> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>>   The GitLab CI jobs definition for the custom runners are located under::
>>
>>     .gitlab-ci.d/custom-runners.yml
>> +
>> +Machine Setup Howto
>> +-------------------
>> +
>> +For all Linux based systems, the setup can be mostly automated by the
>> +execution of two Ansible playbooks.  Create an ``inventory`` file
>> +under ``scripts/ci/setup``, such as this::
>> +
>> +  fully.qualified.domain
>> +  other.machine.hostname
>> +
>> +You may need to set some variables in the inventory file itself.  One
>> +very common need is to tell Ansible to use a Python 3 interpreter on
>> +those hosts.  This would look like::
>> +
>> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>> +
> As I mentioned to Wainer, my suggestion is to include a comment about
> ansible_become=yes and
> ansible_become_password= variables as some systems may need it.


OK.  I mentioned in a previous reply that I'll plan to do this slightly 
different, that is, to mention the ansible-playbook command line 
options.  Let me know if that works for you.


>> +Build environment
>> +~~~~~~~~~~~~~~~~~
>> +
>> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>> +set up machines with the environment needed to perform builds and run
>> +QEMU tests.  It covers a number of different Linux distributions and
>> +FreeBSD.
>> +
>> +To run the playbook, execute::
>> +
>> +  cd scripts/ci/setup
>> +  ansible-playbook -i inventory build-environment.yml
>> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
>> new file mode 100644
>> index 0000000000..664f2f0519
>> --- /dev/null
>> +++ b/scripts/ci/setup/build-environment.yml
>> @@ -0,0 +1,98 @@
>> +---
>> +- name: Installation of basic packages to build QEMU
>> +  hosts: all
>> +  tasks:
>> +    - name: Update apt cache
>> +      apt:
>> +        update_cache: yes
> On a freshly installed Ubuntu 20.04, the script failed for me with
> dependency messages on the apt side. After I updated the packages on
> the system, the playbook worked without problems.


I know it's too late now, but it would have been helpful to document the 
exact error, at the very least here in this thread. I tried to replicate 
with an also fresh install, but resulted in:


PLAY RECAP 
**************************************************************************************************************************************
192.168.122.19             : ok=6    changed=4    unreachable=0 
failed=0    skipped=1    rescued=0    ignored=0


A second run resulted in:

PLAY [Installation of basic packages to build QEMU] 
*********************************************************************************************

TASK [Gathering Facts] 
**************************************************************************************************************************
ok: [192.168.122.19]

TASK [Check for suitable ansible version] 
*******************************************************************************************************
ok: [192.168.122.19] => {
     "changed": false,
     "msg": "All assertions passed"
}

TASK [Update apt cache] 
*************************************************************************************************************************
ok: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 18.04/20.04] 
*******************************************************************************
ok: [192.168.122.19]

TASK [Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x] 
************************************************************************
ok: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 18.04] 
*************************************************************************************
skipping: [192.168.122.19]

TASK [Install basic packages to build QEMU on Ubuntu 20.04] 
*************************************************************************************
ok: [192.168.122.19]

PLAY RECAP 
**************************************************************************************************************************************
192.168.122.19             : ok=6    changed=0    unreachable=0 
failed=0    skipped=1    rescued=0    ignored=0


> So, my suggestion is to add the "update = yes" here, or add a note in
> the documentation asking the user to update the system before running
> the playbook.


IIUC, you mean an "upgrade: yes" on the "Update apt cache" task, right?  
Something like:


-    - name: Update apt cache
+    - name: Update apt cache / upgrade packages via apt
        apt:
          update_cache: yes
+        upgrade: yes

I have mixed feelings about going beyond installing/updating the 
packages we care about.  On one hand, it's a good sysadmin practice, but 
on the other hand, development/test machines are sometimes peculiar.  
Since you and Erik are in favor of the updates, I'll change that and 
wait for feedback on v7.


So, besides the change to the playbook, I'm adding to the docs:


The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
set up machines with the environment needed to perform builds and run
QEMU tests.  This playbook consists on the installation of various
required packages (and a general package update while at it).  It
currently covers a number of different Linux distributions, but it can
be expanded to cover other systems.


> Except for the above comment and Wainer's comments, it looks good to
> me. With these changes:
>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Willian Rampazzo <willianr@redhat.com>
>

Thanks for the review and test!  Given the amount of changes, I'll hold 
a bit to take you Rb/Tb until you see/test v7.

Thanks again!

- Cleber

Re: [PATCH v6 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

On 6/10/21 4:13 AM, Erik Skultety wrote:
> ...
>> On a freshly installed Ubuntu 20.04, the script failed for me with
>> dependency messages on the apt side. After I updated the packages on
>> the system, the playbook worked without problems.
>>
>> So, my suggestion is to add the "update = yes" here, or add a note in
>> the documentation asking the user to update the system before running
>> the playbook.
> I'd advocate for always updating the system automatically rather than putting
> it into the documentation and relying on users actually doing it.
>
> Erik
>

ACK, I'm adding that to v7.


Thanks,

- Cleber.

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

On Tue, Jun 8, 2021 at 3:04 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi,
>
> On 6/8/21 12:14 AM, Cleber Rosa wrote:
> > To have the jobs dispatched to custom runners, gitlab-runner must
> > be installed, active as a service and properly configured.  The
> > variables file and playbook introduced here should help with those
> > steps.
> >
> > The playbook introduced here covers the Linux distributions and
> > has been primarily tested on OS/machines that the QEMU project
> > has available to act as runners, namely:
> >
> >   * Ubuntu 20.04 on aarch64
> >   * Ubuntu 18.04 on s390x
> >
> > But, it should work on all other Linux distributions.  Earlier
> > versions were tested on FreeBSD too, so chances of success are
> > high.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
> >   scripts/ci/setup/.gitignore        |  1 +
> >   scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
> >   scripts/ci/setup/vars.yml.template | 12 ++++++
> >   4 files changed, 131 insertions(+)
> >   create mode 100644 scripts/ci/setup/.gitignore
> >   create mode 100644 scripts/ci/setup/gitlab-runner.yml
> >   create mode 100644 scripts/ci/setup/vars.yml.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 35c6b5e269..bbd89e54d7 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -56,3 +56,60 @@ To run the playbook, execute::
> >
> >     cd scripts/ci/setup
> >     ansible-playbook -i inventory build-environment.yml
> > +
> > +gitlab-runner setup and registration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +The gitlab-runner agent needs to be installed on each machine that
> > +will run jobs.  The association between a machine and a GitLab project
> > +happens with a registration token.  To find the registration token for
> > +your repository/project, navigate on GitLab's web UI to:
> > +
> > + * Settings (the gears like icon), then
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * Under "Set up a specific Runner manually", look for the value under
> > +   "Use the following registration token during setup"
> > +
> > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > +``scripts/ci/setup/vars.yml``.  Then, set the
> > +``gitlab_runner_registration_token`` variable to the value obtained
> > +earlier.
> > +
> > +.. note:: gitlab-runner is not available from the standard location
> > +          for all OS and architectures combinations.  For some systems,
> > +          a custom build may be necessary.  Some builds are avaiable
> > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > +          URI may be used as a value on ``vars.yml``
> I think you can remove the information about the gitlab-running being
> not available for some systems.

Good catch!

> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory gitlab-runner.yml
> > +
> > +Following the registration, it's necessary to configure the runner tags,
> > +and optionally other configurations on the GitLab UI.  Navigate to:
> > +
> > + * Settings (the gears like icon), then
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * "Runners activated for this project", then
> > + * Click on the "Edit" icon (next to the "Lock" Icon)
> > +
> > +Under tags, add values matching the jobs a runner should run.  For a
> > +Ubuntu 20.04 aarch64 system, the tags should be set as::
> > +
> > +  ubuntu_20.04,aarch64
>
> Also users no longer need manually create the tags.
>

Of course.  But, given that this is a general rule, and there may be
OS/architectures for which Ansible may produce tag names one would not
expect, I'll ask the user to double check the tags. The text I'm
proposing on v7 is:

Tags are very important as they are used to route specific jobs to
specific types of runners, so it's a good idea to double check that
the automatically created tags are consistent with the OS and
architecture.  For instance, an Ubuntu 20.04 aarch64 system should
have tags set as::

  ubuntu_20.04,aarch64


> Remaining of the file looks good to me.
>

Thanks a lot!
- Cleber.

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

On Wed, Jun 9, 2021 at 1:46 PM Willian Rampazzo <wrampazz@redhat.com> wrote:
>
> On Tue, Jun 8, 2021 at 12:14 AM Cleber Rosa <crosa@redhat.com> wrote:
> >
> > To have the jobs dispatched to custom runners, gitlab-runner must
> > be installed, active as a service and properly configured.  The
> > variables file and playbook introduced here should help with those
> > steps.
> >
> > The playbook introduced here covers the Linux distributions and
> > has been primarily tested on OS/machines that the QEMU project
> > has available to act as runners, namely:
> >
> >  * Ubuntu 20.04 on aarch64
> >  * Ubuntu 18.04 on s390x
> >
> > But, it should work on all other Linux distributions.  Earlier
> > versions were tested on FreeBSD too, so chances of success are
> > high.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
> >  scripts/ci/setup/.gitignore        |  1 +
> >  scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
> >  scripts/ci/setup/vars.yml.template | 12 ++++++
> >  4 files changed, 131 insertions(+)
> >  create mode 100644 scripts/ci/setup/.gitignore
> >  create mode 100644 scripts/ci/setup/gitlab-runner.yml
> >  create mode 100644 scripts/ci/setup/vars.yml.template
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 35c6b5e269..bbd89e54d7 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -56,3 +56,60 @@ To run the playbook, execute::
> >
> >    cd scripts/ci/setup
> >    ansible-playbook -i inventory build-environment.yml
> > +
> > +gitlab-runner setup and registration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +The gitlab-runner agent needs to be installed on each machine that
> > +will run jobs.  The association between a machine and a GitLab project
> > +happens with a registration token.  To find the registration token for
> > +your repository/project, navigate on GitLab's web UI to:
> > +
> > + * Settings (the gears like icon), then
>
> * Settings (the gears like icon in the end of the left menu), then
>

ACK.  What about:

" * Settings (the gears-like icon at the bottom of the left hand side
   vertical toolbar), then"

> I took some time to find it as it was hidden at the end of the scrolling.
>
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * Under "Set up a specific Runner manually", look for the value under
> > +   "Use the following registration token during setup"
>
> For me, it shows: "And this registration token:"
>

Right, it must have changed.  Updating it on v7.

> > +
> > +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> > +``scripts/ci/setup/vars.yml``.  Then, set the
> > +``gitlab_runner_registration_token`` variable to the value obtained
> > +earlier.
> > +
> > +.. note:: gitlab-runner is not available from the standard location
> > +          for all OS and architectures combinations.  For some systems,
> > +          a custom build may be necessary.  Some builds are avaiable
>
> If you keep this block (see comment from Wainer), s/avaiable/available/
>

I'd rather remove it, but thanks for spotting the typo.

> > +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> > +          URI may be used as a value on ``vars.yml``
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory gitlab-runner.yml
> > +
> > +Following the registration, it's necessary to configure the runner tags,
> > +and optionally other configurations on the GitLab UI.  Navigate to:
> > +
> > + * Settings (the gears like icon), then
> > + * CI/CD, then
> > + * Runners, and click on the "Expand" button, then
> > + * "Runners activated for this project", then
> > + * Click on the "Edit" icon (next to the "Lock" Icon)
> > +
> > +Under tags, add values matching the jobs a runner should run.  For a
> > +Ubuntu 20.04 aarch64 system, the tags should be set as::
> > +
> > +  ubuntu_20.04,aarch64
> > +
> > +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> > +would contain::
> > +
> > +  ubuntu-20.04-aarch64-all:
> > +   tags:
> > +   - ubuntu_20.04
> > +   - aarch64
> > +
> > +It's also recommended to:
> > +
> > + * increase the "Maximum job timeout" to something like ``2h``
> > + * give it a better Description
> > diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> > new file mode 100644
> > index 0000000000..f112d05dd0
> > --- /dev/null
> > +++ b/scripts/ci/setup/.gitignore
> > @@ -0,0 +1 @@
> > +vars.yml
> > \ No newline at end of file
> > diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> > new file mode 100644
> > index 0000000000..98dab92bb5
> > --- /dev/null
> > +++ b/scripts/ci/setup/gitlab-runner.yml
> > @@ -0,0 +1,61 @@
> > +---
> > +- name: Installation of gitlab-runner
> > +  hosts: all
> > +  vars_files:
> > +    - vars.yml
> > +  tasks:
> > +    - debug:
> > +        msg: 'Checking for a valid GitLab registration token'
> > +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> > +
> > +    - name: Create a group for the gitlab-runner service
> > +      group:
> > +        name: gitlab-runner
> > +
> > +    - name: Create a user for the gitlab-runner service
> > +      user:
> > +        user: gitlab-runner
> > +        group: gitlab-runner
> > +        comment: GitLab Runner
> > +        home: /home/gitlab-runner
> > +        shell: /bin/bash
> > +
> > +    - name: Remove the .bash_logout file when on Ubuntu systems
> > +      file:
> > +        path: /home/gitlab-runner/.bash_logout
> > +        state: absent
> > +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> > +
> > +    - name: Set the Operating System for gitlab-runner
> > +      set_fact:
> > +        gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
> > +    - debug:
> > +        msg: gitlab-runner OS is {{ gitlab_runner_os }}
> > +
> > +    - name: Set the architecture for gitlab-runner
> > +      set_fact:
> > +        gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
> > +    - debug:
> > +        msg: gitlab-runner arch is {{ gitlab_runner_arch }}
> > +
> > +    - name: Download the matching gitlab-runner
> > +      get_url:
> > +        dest: /usr/local/bin/gitlab-runner
> > +        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> > +        owner: gitlab-runner
> > +        group: gitlab-runner
> > +        mode: u=rwx,g=rwx,o=rx
> > +
> > +    - name: Register the gitlab-runner
> > +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> > +
> > +    - name: Install the gitlab-runner service using its own functionality
> > +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> > +      register: gitlab_runner_install_service_result
> > +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> > +
> > +    - name: Enable the gitlab-runner service
> > +      service:
> > +        name: gitlab-runner
> > +        state: started
> > +        enabled: yes
> > diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> > new file mode 100644
> > index 0000000000..e48089761f
> > --- /dev/null
> > +++ b/scripts/ci/setup/vars.yml.template
> > @@ -0,0 +1,12 @@
> > +# The version of the gitlab-runner to use
> > +gitlab_runner_version: 13.12.0
> > +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> > +# using a private GitLab instance
> > +gitlab_runner_server_url: https://gitlab.com
> > +# A mapping of the ansible to gitlab architecture nomenclature
> > +ansible_to_gitlab_arch:
> > +  x86_64: amd64
> > +  aarch64: arm64
> > +  s390x: s390x
> > +# A unique token made available by GitLab to your project for registering runners
> > +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
> > --
> > 2.25.4
> >
>
> With my suggestions and Wainer's:
>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Willian Rampazzo <willianr@redhat.com>
>

Re: [PATCH v6 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

On Thu, Jun 10, 2021 at 2:24 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 08/06/2021 05.14, Cleber Rosa wrote:
> > To have the jobs dispatched to custom runners, gitlab-runner must
> > be installed, active as a service and properly configured.  The
> > variables file and playbook introduced here should help with those
> > steps.
> >
> > The playbook introduced here covers the Linux distributions and
> > has been primarily tested on OS/machines that the QEMU project
> > has available to act as runners, namely:
> >
> >   * Ubuntu 20.04 on aarch64
> >   * Ubuntu 18.04 on s390x
> >
> > But, it should work on all other Linux distributions.  Earlier
> > versions were tested on FreeBSD too, so chances of success are
> > high.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   docs/devel/ci.rst                  | 57 ++++++++++++++++++++++++++++
> >   scripts/ci/setup/.gitignore        |  1 +
> >   scripts/ci/setup/gitlab-runner.yml | 61 ++++++++++++++++++++++++++++++
> >   scripts/ci/setup/vars.yml.template | 12 ++++++
> >   4 files changed, 131 insertions(+)
> >   create mode 100644 scripts/ci/setup/.gitignore
> >   create mode 100644 scripts/ci/setup/gitlab-runner.yml
> >   create mode 100644 scripts/ci/setup/vars.yml.template
> [...]
> > diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> > new file mode 100644
> > index 0000000000..f112d05dd0
> > --- /dev/null
> > +++ b/scripts/ci/setup/.gitignore
> > @@ -0,0 +1 @@
> > +vars.yml
> > \ No newline at end of file
>
> Add a newline, please.
>

Sure! Thanks for spotting that.

> > diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> > new file mode 100644
> > index 0000000000..98dab92bb5
> > --- /dev/null
> > +++ b/scripts/ci/setup/gitlab-runner.yml
> > @@ -0,0 +1,61 @@
> > +---
> > +- name: Installation of gitlab-runner
> > +  hosts: all
> > +  vars_files:
> > +    - vars.yml
> > +  tasks:
> > +    - debug:
> > +        msg: 'Checking for a valid GitLab registration token'
> > +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
>
> Could you please add a comment at the top of the file or name it differently
> so that it is clear from a quick glance that this is an ansible playbook?
> Poeple might later wonder otherwise...
>

Good point.  I'm adding a proper header with copyright / comment.

>   Thomas
>

Thanks for the review.
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

On Tue, Jun 8, 2021 at 2:27 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi,
>
> On 6/8/21 12:14 AM, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390x) that can be used
> > for jobs that do build and run tests.  This introduces those jobs,
> > which are a mapping of custom scripts used for the same purpose.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
> >   1 file changed, 208 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> > index a07b27384c..061d3cdfed 100644
> > --- a/.gitlab-ci.d/custom-runners.yml
> > +++ b/.gitlab-ci.d/custom-runners.yml
> > @@ -12,3 +12,211 @@
> >   # guarantees a fresh repository on each job run.
> >   variables:
> >     GIT_STRATEGY: clone
> > +
> > +# All ubuntu-18.04 jobs should run successfully in an environment
> > +# setup by the scripts/ci/setup/build-environment.yml task
> > +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> > +ubuntu-18.04-s390x-all-linux-static:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>
> Should it restrict the job for pushes to qemu-project only? If yes, then
> it probably needs the statement:
>
> '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
>

I'm not sure we should.  In theory, if people have access to other
machines on their own accounts, they should be able to trigger the
same jobs just by using the "staging" prefix.

> If you change that here, you will end it changing all over the jobs. In
> general, there are many boilerplates in this file. I'm ok to merge it as
> is as long as it is followed by another series to refactor the code.
>

Absolutely, as mentioned before, this is a straightforward mapping of
Peter's jobs, so I don't want to go over too many inderaction and
abstraction levels initially.

> > + script:
> > + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> > + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> > + - mkdir build
> > + - cd build
> > + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > + - make --output-sync -j`nproc` check-tcg V=1
> > +
> > +ubuntu-18.04-s390x-all:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > + script:
> > + - mkdir build
> > + - cd build
> > + - ../configure --disable-libssh
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > +
> > +ubuntu-18.04-s390x-alldbg:
> Maybe we don't need both ubuntu-18.04-s390x-all and
> ubuntu-18.04-s390x-alldbg jobs.
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > + script:
> > + - mkdir build
> > + - cd build
> > + - ../configure --enable-debug --disable-libssh
> > + - make clean
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > +ubuntu-18.04-s390x-clang:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > +   when: manual
> > + script:
> > + - mkdir build
> > + - cd build
> > + - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > +
> > +ubuntu-18.04-s390x-tci:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > + script:
> > + - mkdir build
> > + - cd build
> > + - ../configure --disable-libssh --enable-tcg-interpreter
> > + - make --output-sync -j`nproc`
> I think it needs to `make check-tcg` at least. See "build-tci" in
> `.gitlab-ci.d/buildtest.yml` for other tests being executed on shared
> runners.
> > +
> > +ubuntu-18.04-s390x-notcg:
> The "build-tcg-disabled" in `.gitlab-ci.d/buildtest.yml` could be
> mimic-ed here too.

These are straightforward mappings of Peter's jobs... I honestly don't
think we should deviate any further at this time.  Let me know if you
think I'm missing something though.

Thanks for the review!
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

On Wed, Jun 9, 2021 at 10:55 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jun 08, 2021 at 09:36:37AM -0400, Cleber Rosa Junior wrote:
> > On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org>
> > wrote:
>
> Here are my thoughts. It's just my opinion based on experience running
> the old QEMU Buildbot infrastructure and interacting with our hosting
> providers. I'm not planning to actively work on the CI infrastructure,
> so if you have a strong opinion and want to do things differently
> yourself, feel free.
>
> > > Who has access to what and should do what (setup)? How is this list of
> > > hw managed btw? Should there be some public visibility (i.e. Wiki)?
> > >
> >
> > These are good questions, and I believe Alex can answer them about those
> > two machines.  Even though I hooked them up to GitLab, AFAICT he is the
> > ultimate admin (maybe Peter too?).
> >
> > About hardware management, it has been suggested to use either the Wiki or
> > a MAINTAINERS entry.  This is still unresolved and feedback would be
> > appreciated.  For me to propose a MAINTAINERS entry, say, on a v7, I'd need
> > the information on who is managing them.
>
> Here is the wiki page that lists project resources (machines, etc):
> https://wiki.qemu.org/AdminContacts
>
> We can continue to use this page.
>

ACK.  I'm adding a note to the documentation.

Thanks,
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

On Wed, Jun 9, 2021 at 11:09 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jun 08, 2021 at 04:07:57PM -0300, Wainer dos Santos Moschetta wrote:
> > > And I understand that are a lot of them (from everyone, including
> > > myself).  The dilemma here is: should we activate the machines already
> > > available, and learn in practice, what's missing?  I honestly believe we
> > > should.
> >
> >
> > IMHO we should merge the minimum possible to start using the existing
> > machines, then address the questions (good questions, btw) raised by
> > Philippe as needed.
>
> That sounds good.
>
> Does anyone want to volunteer to be the QEMU CI runners point of contact
> who is responsible for defining the process? Cleber's last email left a
> lot for Alex and me to define, but I would prefer it if someone who is
> not me takes this on since I'm already spread thin. Alex? Philippe? Cleber?
> Wainer?
>
> Stefan

Sure, I can do it.

I'll start by resurrecting the now ancient RFC that contains a lot of
proposals with regards to the process, in light of the current status
and developments:

  https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html

IMO we can start with documentation (either at ci.rst, or preferably
in the WIKI for now) for the process of onboarding new CI runners, and
track it with issues in GitLab itself.  I'll formalize those opinions.

Thanks,
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

On Thu, Jun 10, 2021 at 2:18 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 08/06/2021 05.14, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390x) that can be used
> > for jobs that do build and run tests.  This introduces those jobs,
> > which are a mapping of custom scripts used for the same purpose.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
> >   1 file changed, 208 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> > index a07b27384c..061d3cdfed 100644
> > --- a/.gitlab-ci.d/custom-runners.yml
> > +++ b/.gitlab-ci.d/custom-runners.yml
> > @@ -12,3 +12,211 @@
> >   # guarantees a fresh repository on each job run.
> >   variables:
> >     GIT_STRATEGY: clone
> > +
> > +# All ubuntu-18.04 jobs should run successfully in an environment
> > +# setup by the scripts/ci/setup/build-environment.yml task
> > +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> > +ubuntu-18.04-s390x-all-linux-static:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>
> I don't think this will work very well... sub-maintainers might want to push
> to a "staging" branch in their forked repositories, and without the s390x
> runner, the pipeline gets stuck now:
>
>   https://gitlab.com/thuth/qemu/-/pipelines/317812558
>

Hi Thomas,

As I put it in another response, I saw that actually as a feature, in
the sense that:

* people should indeed be allowed to push to their repos and leverage
their hardware, and
* "staging" is a pretty well scoped word, and has a reasonably well
defined meaning
* one would want to mimic as closely as possible what will be done
before a PR is merged

I agree that having the jobs stuck in any situation is not ideal, but
I honestly find that it would be reasonably hard to accidentally hit
that situation.  I also believe it will end up being inevitable for
entities to do a meta-analysis of the GitLab CI pipeline results,
possibly disregarding jobs that they can not run, or simply do not
care about, in their forks.

> We had the same issue in the kvm-unit-test CI, and we solved it there by
> rather making it depend on an environment variable that has to be set if the
> runner is available:
>
>   only:
>     variables:
>      - $S390X_RUNNER_AVAILABLE
>
> I think that's also nicer in case someone brings their own s390x runner and
> want to use the CI tests on other branches than staging.
>

The problem with this approach, is that it would not be enough to
protect the jobs based on variables for the architecture, as the OS
type and version also play a part in the possibility of running jobs.
For instance, suppose we get s390x machines from LinuxOne running
RHEL.  We'd need variables such as, say,
S390X_RHEL_8_4_RUNNER_AVAILABLE and S390X_RHEL_7_6_RUNNER_AVAILABLE.

> Could you please change your patch accordingly?
>

If you strongly believe now is the time to attempt to handle that
problem, I can go ahead and change it.  I stand behind my original
position that we should start with a simpler, "by convention" approach
and address the more complex scenarios as/if they come up.

>   Thanks,
>    Thomas
>

Thank you!
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

On Fri, Jun 11, 2021 at 7:04 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Cleber Rosa Junior <crosa@redhat.com> writes:
>
> > On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> >
> >  Hi Alex, Stefan,
> >
> >  On 6/8/21 5:14 AM, Cleber Rosa wrote:
> >  > The QEMU project has two machines (aarch64 and s390x) that can be used
> >  > for jobs that do build and run tests.
> >
> <snip>
> >
> >  Who has access to what and should do what (setup)? How is this list of
> >  hw managed btw? Should there be some public visibility (i.e. Wiki)?
> >
> > These are good questions, and I believe Alex can answer them about those two machines.  Even though I hooked them up to GitLab,
> > AFAICT he is the ultimate admin (maybe Peter too?).
> >
> > About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry.  This is still unresolved and feedback
> > would be appreciated.  For me to propose a MAINTAINERS entry, say, on
> > a v7, I'd need the information on who is managing them.
>
> I can only talk about aarch64.ci.qemu.org which is a donated Equinix
> machine that comes from the WorksOnArm initiative. I applied for it on
> behalf of the QEMU project and we can have it for as long as it's
> useful.
>
> I don't know if we need anything more that documenting the nominal
> contacts in:
>
>   https://wiki.qemu.org/AdminContacts
>

That's enough indeed, thanks.  I'll follow up with a proposal about
the expected duties of admins, which should be nothing but common
sense.

Is there anyone that can speak for the s390x machine?

> >  Is there a document explaining what are the steps to follow for an
> >  entity to donate / sponsor hardware? Where would it be stored, should
> >  this hw be shipped somewhere? What documentation should be provided for
> >  its system administration?
>
> I think the project can only really work with donations out of someones
> data centre where they keep responsibility for the physical aspects of
> any machines including the ongoing hosting and running costs.
>

Agreed.  Anything else is beyond what can be managed atm.

> --
> Alex Bennée
>

Thanks,
- Cleber.

Re: [PATCH v6 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Willian Rampazzo]

On Tue, Jun 29, 2021 at 10:08 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> On Fri, Jun 11, 2021 at 7:04 AM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >
> > Cleber Rosa Junior <crosa@redhat.com> writes:
> >
> > > On Tue, Jun 8, 2021 at 2:30 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> > >
> > >  Hi Alex, Stefan,
> > >
> > >  On 6/8/21 5:14 AM, Cleber Rosa wrote:
> > >  > The QEMU project has two machines (aarch64 and s390x) that can be used
> > >  > for jobs that do build and run tests.
> > >
> > <snip>
> > >
> > >  Who has access to what and should do what (setup)? How is this list of
> > >  hw managed btw? Should there be some public visibility (i.e. Wiki)?
> > >
> > > These are good questions, and I believe Alex can answer them about those two machines.  Even though I hooked them up to GitLab,
> > > AFAICT he is the ultimate admin (maybe Peter too?).
> > >
> > > About hardware management, it has been suggested to use either the Wiki or a MAINTAINERS entry.  This is still unresolved and feedback
> > > would be appreciated.  For me to propose a MAINTAINERS entry, say, on
> > > a v7, I'd need the information on who is managing them.
> >
> > I can only talk about aarch64.ci.qemu.org which is a donated Equinix
> > machine that comes from the WorksOnArm initiative. I applied for it on
> > behalf of the QEMU project and we can have it for as long as it's
> > useful.
> >
> > I don't know if we need anything more that documenting the nominal
> > contacts in:
> >
> >   https://wiki.qemu.org/AdminContacts
> >
>
> That's enough indeed, thanks.  I'll follow up with a proposal about
> the expected duties of admins, which should be nothing but common
> sense.
>
> Is there anyone that can speak for the s390x machine?
>

I can be the backup here, if needed.

Willian