[PATCH v5 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[PATCH v5 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Cleber Rosa]

TL;DR: this should allow the QEMU maintainer to push to the staging
branch, and have custom jobs running on the project's aarch64 and
s390x machines.  Jobs in this version are allowed to fail, to allow
for the inclusion of the novel machines/jobs without CI disruption.
Simple usage looks like:

   git push remote staging
   ./scripts/ci/gitlab-pipeline-status --verbose --wait

Long version:

The idea about a public facing Gating CI for QEMU was summarized in an
RFC[1].  Since then, it was decided that a simpler version should be
attempted first.

At this point, there are two specific runners (an aarch64 and an s390x)
registered with GitLab, at https://gitlab.com/qemu-project, currently
setup to the "qemu" repository.

Changes from v4:

 - Fixed typo in docs/devel/ci.rst, s/maintanance/maintenance/ (Thomas)
 - Removed "[local]" group from inventory file (Erik)
 - Removed sections from the playbooks which *would* be applied on
   hardware/OS that are currently not available to QEMU
 - Removed duplicated "here" on documentation (Thomas)
 - Moved description of current jobs, and possible direction of future
   jobs to the patch description (Thomas)
 - Remove comments around "when" conditions (Andrea)
 - Switch to always use explicit lists on "when" blocks (Andrea)
 - Switch from using module "apt" to using generic action module "package",
   which involved adding a new task to update the apt cache (Andrea)
 - Fix playbook indentation in the non-s390x package installation task (Andrea)
 - Changed gitlab-runner tags examples from FreeBSD to Ubuntu, which is
   covered by jobs added on this version
 - Fixed typo in commit message s/s390/s390x/ (Phil)
 - Allow all custom-runner jobs to fail at this time
 - Cleared "Reviewed-by" in one patch due to large changes

  Changes requested in v4 but *not* seen here due to sections of the
  playbook being removed:

 - Replace SDL-devel for SDL2-devel on CentOS, according to 5ed7ca3 (Thomas)
 - Correct missing step 10 on the FreeBSD gitlab-runner installation
   instructions (Erik)

Changes from v3:

- Applied changes to match <20201014135416.1290679-1-pbonzini@redhat.com>,
  that is, added ninja-build to "build-environment.yml" list of packages
  and enabled PowerTools repository on CentOS 8.

Changes from v2:

- The overall idea of "Gating CI" has been re-worded "custom runners",
  given that the other jobs running on shared runners are also
  considered gating (Daniel)

- Fixed wording and typos on the documentation, including:
 * update -> up to date (Erik)
 * a different set of CI jobs -> different CI jobs (Erik)
 * Pull requests will only be merged -> code will only be merged (Stefan)
 * Setup -> set up (Stefan)
 * them -> they (Stefan)
 * the -> where the (Stefan)
 * dropped "in the near future" (Stefan)

- Changed comment on "build-environment.yml" regarding the origin of
  the package list (Stefan)

- Removed inclusion of "vars.yml" from "build-environment.yml", given that
  no external variable is used there

- Updated package list in "build-environment.yml" from current
  dockerfiles

- Tested "build-environment" on Fedora 31 and 32 (in addition to Fedora 30),
  and noted that it's possible to use it on those distros

- Moved CI documentation from "testing.rst" to its own file (Phillipe)

- Split "GitLab Gating CI: initial set of jobs, documentation and scripts"
  into (Phillipe):
  1) Basic documentation and configuration (gitlab-ci.yml) placeholder
  2) Playbooks for setting up a build environment
  3) Playbooks for setting up gitlab-runner
  4) Actual GitLab CI jobs configuration

- Set custom jobs to be on the "build" stage, given that they combine
  build and test.

- Set custom jobs to not depend on any other job, so they can start
  right away.

- Set rules for starting jobs so that all pushing to any branch that
  start with name "staging".  This allows the project maintainer to
  use the "push to staging" workflow, while also allowing others to
  generate similar jobs.  If this project has configured custom
  runners, the jobs will run, if not, the pipeline will be marked as
  "stuck".

- Changed "scripts" on custom jobs to follow the now common pattern
  (on other jobs) of creating a "build" directory.

Changes from v1:

- Added jobs that require specific GitLab runners already available
  (Ubuntu 20.04 on aarch64, and Ubuntu 18.04 on s390x)
- Removed jobs that require specific GitLab runners not yet available
  (Fedora 30, FreeBSD 12.1)
- Updated documentation
- Added copyright and license to new scripts
- Moved script to from "contrib" to "scripts/ci/"
- Moved setup playbooks form "contrib" to "scripts/ci/setup"
- Moved "gating.yml" to ".gitlab-ci.d" directory
- Removed "staging" only branch restriction on jobs defined in
  ".gitlab-ci.yml", assumes that the additional jobs on the staging
  branch running on the freely available gitlab shared runner are
  positive
- Dropped patches 1-3 (already merged)
- Simplified amount of version specifity on Ubuntu, from 18.04.3 to
  simply 18.04 (assumes no diverse minor levels will be used or
  specific runners)

Changes from the RFC patches[2] accompanying the RFC document:

- Moved gating job definitions to .gitlab-ci-gating.yml
- Added info on "--disable-libssh" build option requirement
  (https://bugs.launchpad.net/qemu/+bug/1838763) to Ubuntu 18.04 jobs
- Added info on "--disable-glusterfs" build option requirement
  (there's no static version of those libs in distro supplied
  packages) to one
- Dropped ubuntu-18.04.3-x86_64-notools job definition, because it
  doesn't fall into the general scope of gating job described by PMM
  (and it did not run any test)
- Added w32 and w64 cross builds based on Fedora 30
- Added a FreeBSD based job that builds all targets and runs `make
  check`
- Added "-j`nproc`" and "-j`sysctl -n hw.ncpu`" options to make as a
  simple but effective way of speeding up the builds and tests by
  using a number of make jobs matching the number of CPUs
- Because the Ansible playbooks reference the content on Dockerfiles,
  some fixes to some Dockerfiles caught in the process were included
- New patch with script to check or wait on a pipeline execution

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg00231.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg00154.html

Cleber Rosa (4):
  Jobs based on custom runners: documentation and configuration
    placeholder
  Jobs based on custom runners: build environment docs and playbook
  Jobs based on custom runners: docs and gitlab-runner setup playbook
  Jobs based on custom runners: add job definitions for QEMU's machines

 .gitlab-ci.d/custom-runners.yml        | 218 +++++++++++++++++++++++++
 .gitlab-ci.yml                         |   1 +
 docs/devel/ci.rst                      | 116 +++++++++++++
 docs/devel/index.rst                   |   1 +
 scripts/ci/setup/.gitignore            |   1 +
 scripts/ci/setup/build-environment.yml |  76 +++++++++
 scripts/ci/setup/gitlab-runner.yml     |  65 ++++++++
 scripts/ci/setup/inventory             |   1 +
 scripts/ci/setup/vars.yml.template     |  13 ++
 9 files changed, 492 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/inventory
 create mode 100644 scripts/ci/setup/vars.yml.template

-- 
2.25.4

[PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.  One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions.  As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
 .gitlab-ci.yml                  |  1 +
 docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
 docs/devel/index.rst            |  1 +
 4 files changed, 44 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners.yml
 create mode 100644 docs/devel/ci.rst

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
new file mode 100644
index 0000000000..3004da2bda
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures.  This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner.  To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the recursive submodule
+# strategy.
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8b6d495288..ae19442e93 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/custom-runners.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
new file mode 100644
index 0000000000..585b7bf4b8
--- /dev/null
+++ b/docs/devel/ci.rst
@@ -0,0 +1,28 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+   https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 22854e334d..b178448a91 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -23,6 +23,7 @@ Contents:
    migration
    atomics
    stable-process
+   ci
    qtest
    decodetree
    secure-coding-practices
-- 
2.25.4

[PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

To run basic jobs on custom runners, the environment needs to be
properly set up.  The most common requirement is having the right
packages installed.

The playbook introduced here covers the QEMU's project s390x and
aarch64 machines.  At the time this is being proposed, those machines
have already had this playbook applied to them.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/ci.rst                      | 30 ++++++++++
 scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
 scripts/ci/setup/inventory             |  1 +
 3 files changed, 107 insertions(+)
 create mode 100644 scripts/ci/setup/build-environment.yml
 create mode 100644 scripts/ci/setup/inventory

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 585b7bf4b8..a556558435 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
 The GitLab CI jobs definition for the custom runners are located under::
 
   .gitlab-ci.d/custom-runners.yml
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Start by adding your machines to
+the ``inventory`` file under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..0197e0a48b
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,76 @@
+---
+- name: Installation of basic packages to build QEMU
+  hosts: all
+  tasks:
+    - name: Update apt cache
+      apt:
+        update_cache: yes
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+      package:
+        name:
+        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+          - ccache
+          - clang
+          - gcc
+          - gettext
+          - git
+          - glusterfs-common
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libbz2-dev
+          - libcacard-dev
+          - libcap-ng-dev
+          - libcurl4-gnutls-dev
+          - libdrm-dev
+          - libepoxy-dev
+          - libfdt-dev
+          - libgbm-dev
+          - libgtk-3-dev
+          - libibverbs-dev
+          - libiscsi-dev
+          - libjemalloc-dev
+          - libjpeg-turbo8-dev
+          - liblzo2-dev
+          - libncurses5-dev
+          - libncursesw5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libnuma-dev
+          - libpixman-1-dev
+          - librados-dev
+          - librbd-dev
+          - librdmacm-dev
+          - libsasl2-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - libsnappy-dev
+          - libspice-protocol-dev
+          - libssh-dev
+          - libusb-1.0-0-dev
+          - libusbredirhost-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          - libzstd-dev
+          - make
+          - ninja-build
+          - python3-yaml
+          - python3-sphinx
+          - sparse
+          - xfslibs-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+      package:
+        name:
+          - libspice-server-dev
+          - libxen-dev
+        state: present
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] != 's390x'
diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
new file mode 100644
index 0000000000..2fbb50c4a8
--- /dev/null
+++ b/scripts/ci/setup/inventory
@@ -0,0 +1 @@
+localhost
-- 
2.25.4

[PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Cleber Rosa]

To have the jobs dispatched to custom runners, gitlab-runner must
be installed, active as a service and properly configured.  The
variables file and playbook introduced here should help with those
steps.

The playbook introduced here covers a number of different Linux
distributions and FreeBSD, and are intended to provide a reproducible
environment.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
 scripts/ci/setup/.gitignore        |  1 +
 scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
 scripts/ci/setup/vars.yml.template | 13 ++++++
 4 files changed, 137 insertions(+)
 create mode 100644 scripts/ci/setup/.gitignore
 create mode 100644 scripts/ci/setup/gitlab-runner.yml
 create mode 100644 scripts/ci/setup/vars.yml.template

diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index a556558435..9f9c4bd3f9 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -56,3 +56,61 @@ To run the playbook, execute::
 
   cd scripts/ci/setup
   ansible-playbook -i inventory build-environment.yml
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "Use the following registration token during setup"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+.. note:: gitlab-runner is not available from the standard location
+          for all OS and architectures combinations.  For some systems,
+          a custom build may be necessary.  Some builds are avaiable
+          at https://cleber.fedorapeople.org/gitlab-runner/ and this
+          URI may be used as a value on ``vars.yml``
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Under tags, add values matching the jobs a runner should run.  For a
+Ubuntu 20.04 aarch64 system, the tags should be set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * uncheck the "Run untagged jobs" check box
+ * give it a better Description
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
new file mode 100644
index 0000000000..f112d05dd0
--- /dev/null
+++ b/scripts/ci/setup/.gitignore
@@ -0,0 +1 @@
+vars.yml
\ No newline at end of file
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
new file mode 100644
index 0000000000..ab1944965f
--- /dev/null
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -0,0 +1,65 @@
+---
+- name: Installation of gitlab-runner
+  hosts: all
+  vars_files:
+    - vars.yml
+  tasks:
+    - debug:
+        msg: 'Checking for a valid GitLab registration token'
+      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+    - name: Checks the availability of official gitlab-runner builds in the archive
+      uri:
+        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
+        method: HEAD
+        status_code:
+          - 200
+          - 403
+      register: gitlab_runner_available_archive
+
+    - name: Update base url
+      set_fact:
+        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
+      when: gitlab_runner_available_archive.status == 200
+    - debug:
+        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
+
+    - name: Create a group for the gitlab-runner service
+      group:
+        name: gitlab-runner
+
+    - name: Create a user for the gitlab-runner service
+      user:
+        user: gitlab-runner
+        group: gitlab-runner
+        comment: GitLab Runner
+        home: /home/gitlab-runner
+        shell: /bin/bash
+
+    - name: Remove the .bash_logout file when on Ubuntu systems
+      file:
+        path: /home/gitlab-runner/.bash_logout
+        state: absent
+      when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+    - name: Downloads the matching gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner
+        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+
+    - name: Register the gitlab-runner
+      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+    - name: Enable the gitlab-runner service
+      service:
+        name: gitlab-runner
+        state: started
+        enabled: yes
diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
new file mode 100644
index 0000000000..621435d030
--- /dev/null
+++ b/scripts/ci/setup/vars.yml.template
@@ -0,0 +1,13 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.1.1
+# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
+gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# Defaults to linux, checks can be used to change this
+gitlab_runner_os: linux
+# Defaults to amd64 (x86_64), checks can be used to change this
+gitlab_runner_arch: amd64
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
-- 
2.25.4

[PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index 3004da2bda..a9166c82a2 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -12,3 +12,207 @@
 # strategy.
 variables:
   GIT_SUBMODULE_STRATEGY: recursive
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
-- 
2.25.4

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Erik Skultety]

On Fri, Feb 19, 2021 at 04:58:37PM -0500, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 137 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template
> 
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index a556558435..9f9c4bd3f9 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -56,3 +56,61 @@ To run the playbook, execute::
>  
>    cd scripts/ci/setup
>    ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:

I think the word order should be "on GitLab's web UI navigate to:"

> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable

s/avaiable/available

> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +Ubuntu 20.04 aarch64 system, the tags should be set as::
> +
> +  ubuntu_20.04,aarch64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * uncheck the "Run untagged jobs" check box
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..ab1944965f
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,65 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Checks the availability of official gitlab-runner builds in the archive
> +      uri:
> +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
> +        method: HEAD
> +        status_code:
> +          - 200
> +          - 403
> +      register: gitlab_runner_available_archive
> +
> +    - name: Update base url
> +      set_fact:
> +        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
> +      when: gitlab_runner_available_archive.status == 200
> +    - debug:
> +        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash

Totally unimportant (you may as well ignore this comment), but depending on
how much in sync you want to be with libvirt's playbook, the user:group we
create is gitlab:gitlab.

> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Downloads the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner

I'm pretty sure I pointed this out in previous versions, but according to the
docs ^this won't install the runner on FreeBSD as a service. IIRC the answer
was that FreeBSD is not in the priority distro list at the moment and that it
can always be adjusted further down the road - that is fair, no objection, but
then the commit message says that this playbook is creating a reproducible
environment and covers both Linux and FreeBSD which is not true in its
entirety, so either drop it from the commit message or add a small comment here
that the command would actually only work as expected on Linux.

Reviewed-by: Erik Skultety <eskultet@redhat.com>

> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..621435d030
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,13 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.1.1
> +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
> +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# Defaults to linux, checks can be used to change this
> +gitlab_runner_os: linux
> +# Defaults to amd64 (x86_64), checks can be used to change this
> +gitlab_runner_arch: amd64
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
> -- 
> 2.25.4
> 

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Wainer dos Santos Moschetta]

Hi,

On 2/19/21 6:58 PM, Cleber Rosa wrote:
> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   docs/devel/ci.rst                      | 30 ++++++++++
>   scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
>   scripts/ci/setup/inventory             |  1 +
>   3 files changed, 107 insertions(+)
>   create mode 100644 scripts/ci/setup/build-environment.yml
>   create mode 100644 scripts/ci/setup/inventory


I tested the playbook in a Fedora 32 (aarch64) machine using containers, as:

dnf install -y podman podman-docker ansible
echo "" > inventory
for ver in 18.04 20.04; do
     name="ubuntu_$(echo $ver | sed 's/\./_/')_runner"
     podman run --rm -d --name "$name" docker.io/library/ubuntu:$ver 
tail -f /dev/null
     podman exec "$name" sh -c 'apt-get update && apt-get install -y 
python3'
     echo "$name ansible_connection=docker 
ansible_python_interpreter=/usr/bin/python3" >> inventory
done
ansible-playbook -i inventory build-environment.yml

So, Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 585b7bf4b8..a556558435 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>   The GitLab CI jobs definition for the custom runners are located under::
>   
>     .gitlab-ci.d/custom-runners.yml
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Start by adding your machines to
> +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  It covers a number of different Linux distributions and
> +FreeBSD.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..0197e0a48b
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,76 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Update apt cache
> +      apt:
> +        update_cache: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - clang
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - ninja-build
> +          - python3-yaml
> +          - python3-sphinx
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory
> @@ -0,0 +1 @@
> +localhost

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Wainer dos Santos Moschetta]

Hi,

On 2/19/21 6:58 PM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>   scripts/ci/setup/.gitignore        |  1 +
>   scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>   scripts/ci/setup/vars.yml.template | 13 ++++++
>   4 files changed, 137 insertions(+)
>   create mode 100644 scripts/ci/setup/.gitignore
>   create mode 100644 scripts/ci/setup/gitlab-runner.yml
>   create mode 100644 scripts/ci/setup/vars.yml.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index a556558435..9f9c4bd3f9 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -56,3 +56,61 @@ To run the playbook, execute::
>   
>     cd scripts/ci/setup
>     ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable
> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``
FYI the latest version (13.8.0) provides a s390x build.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +Ubuntu 20.04 aarch64 system, the tags should be set as::
> +
> +  ubuntu_20.04,aarch64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * uncheck the "Run untagged jobs" check box
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..ab1944965f
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,65 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Checks the availability of official gitlab-runner builds in the archive
> +      uri:
> +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386

Where it checks for 386 then later it uses gitlab_runner_arch (amd64 by 
default). It is not consistent.

Also, why not use ansible_machine + jinja2 to convert x86_64 -> amd64, 
aarch64 -> arm64...etc?

> +        method: HEAD
> +        status_code:
> +          - 200
> +          - 403
> +      register: gitlab_runner_available_archive
> +
> +    - name: Update base url
> +      set_fact:
> +        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
> +      when: gitlab_runner_available_archive.status == 200
> +    - debug:
> +        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner
> +
> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"
> +
> +    - name: Downloads the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"


And here instead of gitlab_runner_os, {{ ansible_system | lower }} 
should work out.

- Wainer

> +        owner: gitlab-runner
> +        group: gitlab-runner
> +        mode: u=rwx,g=rwx,o=rx
> +
> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
> +
> +    - name: Install the gitlab-runner service using its own functionality
> +      command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
> +      register: gitlab_runner_install_service_result
> +      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
> +
> +    - name: Enable the gitlab-runner service
> +      service:
> +        name: gitlab-runner
> +        state: started
> +        enabled: yes
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..621435d030
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,13 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.1.1
> +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
> +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-
> +# The URL of the gitlab server to use, usually https://gitlab.com unless you're
> +# using a private GitLab instance
> +gitlab_runner_server_url: https://gitlab.com
> +# Defaults to linux, checks can be used to change this
> +gitlab_runner_os: linux
> +# Defaults to amd64 (x86_64), checks can be used to change this
> +gitlab_runner_arch: amd64
> +# A unique token made available by GitLab to your project for registering runners
> +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
>
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Thomas Huth]

On 19/02/2021 22.58, Cleber Rosa wrote:
> As described in the included documentation, the "custom runner" jobs
> extend the GitLab CI jobs already in place.  One of their primary
> goals of catching and preventing regressions on a wider number of host
> systems than the ones provided by GitLab's shared runners.
> 
> This sets the stage in which other community members can add their own
> machine configuration documentation/scripts, and accompanying job
> definitions.  As a general rule, those newly added contributed jobs
> should run as "non-gating", until their reliability is verified (AKA
> "allow_failure: true").
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>   .gitlab-ci.yml                  |  1 +
>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>   docs/devel/index.rst            |  1 +
>   4 files changed, 44 insertions(+)
>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>   create mode 100644 docs/devel/ci.rst
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> new file mode 100644
> index 0000000000..3004da2bda
> --- /dev/null
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -0,0 +1,14 @@
> +# The CI jobs defined here require GitLab runners installed and
> +# registered on machines that match their operating system names,
> +# versions and architectures.  This is in contrast to the other CI
> +# jobs that are intended to run on GitLab's "shared" runners.
> +
> +# Different than the default approach on "shared" runners, based on
> +# containers, the custom runners have no such *requirement*, as those
> +# jobs should be capable of running on operating systems with no
> +# compatible container implementation, or no support from
> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> +# reusing the GIT repository, let's enable the recursive submodule
> +# strategy.
> +variables:
> +  GIT_SUBMODULE_STRATEGY: recursive

Is it really necessary? I thought our configure script would take care of 
the submodules?

Apart from that:
Acked-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Philippe Mathieu-Daudé]

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 137 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template
...

> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"

Is this only a problem with Ubuntu and not Debian?

> +    - name: Downloads the matching gitlab-runner
> +      get_url:
> +        dest: /usr/local/bin/gitlab-runner
> +        url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"

Can we move the dash at the end of gitlab_runner_base_url here before
gitlab_runner_os?

...
> diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
> new file mode 100644
> index 0000000000..621435d030
> --- /dev/null
> +++ b/scripts/ci/setup/vars.yml.template
> @@ -0,0 +1,13 @@
> +# The version of the gitlab-runner to use
> +gitlab_runner_version: 13.1.1
> +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH
> +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-

Are we using a specific feature from the official builds,
or can we use any runner?

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Philippe Mathieu-Daudé]

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 137 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template

> +    - name: Create a user for the gitlab-runner service
> +      user:
> +        user: gitlab-runner
> +        group: gitlab-runner
> +        comment: GitLab Runner
> +        home: /home/gitlab-runner
> +        shell: /bin/bash
> +
> +    - name: Remove the .bash_logout file when on Ubuntu systems
> +      file:
> +        path: /home/gitlab-runner/.bash_logout
> +        state: absent
> +      when: "ansible_facts['distribution'] == 'Ubuntu'"

Can we have a {{gitlab_runner_homedir}} in vars.yml?

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      | 30 ++++++++++
>  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
>  scripts/ci/setup/inventory             |  1 +
>  3 files changed, 107 insertions(+)
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 585b7bf4b8..a556558435 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>  The GitLab CI jobs definition for the custom runners are located under::
>  
>    .gitlab-ci.d/custom-runners.yml
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Start by adding your machines to
> +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname

Is this really needed? Can't the host list be passed in the command
line? I find it off to imagine users wanting to configure whole fleets
of runners.

> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  It covers a number of different Linux distributions and
> +FreeBSD.
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml

So I got somewhat there with a direct command line invocation:

  ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'

although for some reason a single host -i fails...

> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..0197e0a48b
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,76 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Update apt cache
> +      apt:
> +        update_cache: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'

So are we limiting to Ubuntu here rather than say a Debian base?

> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - clang
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - ninja-build
> +          - python3-yaml
> +          - python3-sphinx
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory
> @@ -0,0 +1 @@
> +localhost

I'm not sure we should have a default here because it will inevitably
cause someone to do something to their machine when trying to setup a
runner.

-- 
Alex Bennée

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Philippe Mathieu-Daudé]

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
> 
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 137 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template

> +    - name: Register the gitlab-runner
> +      command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell  --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"

Hmm maybe we want to register them with --run-untagged=false or
explicitly add tags like {{ ansible_facts[\"architecture\"] }}.

Also, maybe have --cache-shared by default?

And set a reasonable limits values...
 --maximum-timeout 10800 # 3h
 --output-limit 8192 # 8MiB

No CPU/memory limits yet.

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Erik Skultety]

On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
> 
> Cleber Rosa <crosa@redhat.com> writes:
> 
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/ci.rst                      | 30 ++++++++++
> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
> >  scripts/ci/setup/inventory             |  1 +
> >  3 files changed, 107 insertions(+)
> >  create mode 100644 scripts/ci/setup/build-environment.yml
> >  create mode 100644 scripts/ci/setup/inventory
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 585b7bf4b8..a556558435 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >  The GitLab CI jobs definition for the custom runners are located under::
> >  
> >    .gitlab-ci.d/custom-runners.yml
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Start by adding your machines to
> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> 
> Is this really needed? Can't the host list be passed in the command
> line? I find it off to imagine users wanting to configure whole fleets
> of runners.

Why not support both, since the playbook execution is not wrapped by anything,
giving the option of using either and inventory or direct cmdline invocation
seems like the proper way to do it.

> 
> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  It covers a number of different Linux distributions and
> > +FreeBSD.
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
> 
> So I got somewhat there with a direct command line invocation:
> 
>   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
> 
> although for some reason a single host -i fails...

The trick is to end it with a ',' like "-i host1,"

Erik

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Cleber Rosa <crosa@redhat.com> writes:
>
>> To run basic jobs on custom runners, the environment needs to be
>> properly set up.  The most common requirement is having the right
>> packages installed.
>>
<snip>
>
> So I got somewhat there with a direct command line invocation:
>
>   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
>
> although for some reason a single host -i fails...
>
>> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
>> new file mode 100644
>> index 0000000000..0197e0a48b
>> --- /dev/null
>> +++ b/scripts/ci/setup/build-environment.yml
>> @@ -0,0 +1,76 @@
>> +---
>> +- name: Installation of basic packages to build QEMU
>> +  hosts: all
>> +  tasks:
>> +    - name: Update apt cache
>> +      apt:
>> +        update_cache: yes
>> +      when:
>> +        - ansible_facts['distribution'] == 'Ubuntu'
>
> So are we limiting to Ubuntu here rather than say a Debian base?

Also I'm getting:

  TASK [Update apt cache] *****************************************************************************************************************************************************
  fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}

which is odd given that machine is definitely an Ubuntu one.

-- 
Alex Bennée

Re: [PATCH v5 3/4] Jobs based on custom runners: docs and gitlab-runner setup playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To have the jobs dispatched to custom runners, gitlab-runner must
> be installed, active as a service and properly configured.  The
> variables file and playbook introduced here should help with those
> steps.
>
> The playbook introduced here covers a number of different Linux
> distributions and FreeBSD, and are intended to provide a reproducible
> environment.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci.rst                  | 58 ++++++++++++++++++++++++++
>  scripts/ci/setup/.gitignore        |  1 +
>  scripts/ci/setup/gitlab-runner.yml | 65 ++++++++++++++++++++++++++++++
>  scripts/ci/setup/vars.yml.template | 13 ++++++
>  4 files changed, 137 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/vars.yml.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index a556558435..9f9c4bd3f9 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -56,3 +56,61 @@ To run the playbook, execute::
>  
>    cd scripts/ci/setup
>    ansible-playbook -i inventory build-environment.yml
> +
> +gitlab-runner setup and registration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The gitlab-runner agent needs to be installed on each machine that
> +will run jobs.  The association between a machine and a GitLab project
> +happens with a registration token.  To find the registration token for
> +your repository/project, navigate on GitLab's web UI to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * Under "Set up a specific Runner manually", look for the value under
> +   "Use the following registration token during setup"
> +
> +Copy the ``scripts/ci/setup/vars.yml.template`` file to
> +``scripts/ci/setup/vars.yml``.  Then, set the
> +``gitlab_runner_registration_token`` variable to the value obtained
> +earlier.
> +
> +.. note:: gitlab-runner is not available from the standard location
> +          for all OS and architectures combinations.  For some systems,
> +          a custom build may be necessary.  Some builds are avaiable
> +          at https://cleber.fedorapeople.org/gitlab-runner/ and this
> +          URI may be used as a value on ``vars.yml``
> +
> +To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory gitlab-runner.yml
> +
> +Following the registration, it's necessary to configure the runner tags,
> +and optionally other configurations on the GitLab UI.  Navigate to:
> +
> + * Settings (the gears like icon), then
> + * CI/CD, then
> + * Runners, and click on the "Expand" button, then
> + * "Runners activated for this project", then
> + * Click on the "Edit" icon (next to the "Lock" Icon)
> +
> +Under tags, add values matching the jobs a runner should run.  For a
> +Ubuntu 20.04 aarch64 system, the tags should be set as::
> +
> +  ubuntu_20.04,aarch64
> +
> +Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
> +would contain::
> +
> +  ubuntu-20.04-aarch64-all:
> +   tags:
> +   - ubuntu_20.04
> +   - aarch64
> +
> +It's also recommended to:
> +
> + * increase the "Maximum job timeout" to something like ``2h``
> + * uncheck the "Run untagged jobs" check box
> + * give it a better Description
> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..f112d05dd0
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1 @@
> +vars.yml
> \ No newline at end of file
> diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
> new file mode 100644
> index 0000000000..ab1944965f
> --- /dev/null
> +++ b/scripts/ci/setup/gitlab-runner.yml
> @@ -0,0 +1,65 @@
> +---
> +- name: Installation of gitlab-runner
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - debug:
> +        msg: 'Checking for a valid GitLab registration token'
> +      failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
> +
> +    - name: Checks the availability of official gitlab-runner builds in the archive
> +      uri:
> +        url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-linux-386
> +        method: HEAD
> +        status_code:
> +          - 200
> +          - 403
> +      register: gitlab_runner_available_archive
> +
> +    - name: Update base url
> +      set_fact:
> +        gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-
> +      when: gitlab_runner_available_archive.status == 200
> +    - debug:
> +        msg: Base gitlab-runner url is {{ gitlab_runner_base_url  }}
> +
> +    - name: Create a group for the gitlab-runner service
> +      group:
> +        name: gitlab-runner

I got this not particularly helpful error:

  TASK [Create a group for the gitlab-runner service] *************************************************************************************************************************
  fatal: [hackbox-ubuntu-2004]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.122.170 closed.\r\n", "module_stdout": "/root/.ansible/tmp/ansible
  -tmp-1614092629.906646-258936160555386/AnsiballZ_group.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alt
  ernative uses\r\n  import imp\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_group_payload_2xv1or12/ansible_group_payload.zip/ansible/module_utils/basic.py\"
  , line 279, in get_distribution\r\nAttributeError: module 'platform' has no attribute '_supported_dists'\r\n\r\nDuring handling of the above exception, another exception occ
  urred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/root/.ansible/tmp/ansible-tmp-1614092629.906646-258936160555386/AnsiballZ_group.py\", line 113, in <module>\r\n
      _ansiballz_main()\r\n  File \"/root/.ansible/tmp/ansible-tmp-1614092629.906646-258936160555386/AnsiballZ_group.py\", line 105, in _ansiballz_main\r\n    invoke_module(zi
  pped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/root/.ansible/tmp/ansible-tmp-1614092629.906646-258936160555386/AnsiballZ_group.py\", line 48, in invoke_module\r\n    im
  p.load_module('__main__', mod, module, MOD_DESC)\r\n  File \"/usr/lib/python3.8/imp.py\", line 234, in load_module\r\n    return load_source(name, filename, file)\r\n  File
  \"/usr/lib/python3.8/imp.py\", line 169, in load_source\r\n    module = _exec(spec, sys.modules[name])\r\n  File \"<frozen importlib._bootstrap>\", line 604, in _exec\r\n  F
  ile \"<frozen importlib._bootstrap_external>\", line 783, in exec_module\r\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\r\n  File \"/tmp
  /ansible_group_payload_2xv1or12/__main__.py\", line 501, in <module>\r\n  File \"/tmp/ansible_group_payload_2xv1or12/__main__.py\", line 449, in main\r\n  File \"/tmp/ansibl
  e_group_payload_2xv1or12/__main__.py\", line 89, in __new__\r\n  File \"/tmp/ansible_group_payload_2xv1or12/ansible_group_payload.zip/ansible/module_utils/basic.py\", line 3
  37, in load_platform_subclass\r\n  File \"/tmp/ansible_group_payload_2xv1or12/ansible_group_payload.zip/ansible/module_utils/basic.py\", line 289, in get_distribution\r\nAtt
  ributeError: module 'platform' has no attribute 'dist'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
          to retry, use: --limit @/home/alex/lsrc/qemu.git/scripts/ci/setup/gitlab-runner.retry


-- 
Alex Bennée

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index 3004da2bda..a9166c82a2 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -12,3 +12,207 @@
>  # strategy.
>  variables:
>    GIT_SUBMODULE_STRATEGY: recursive
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x

Where is this tag list filled upon registration?

> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1

Also this break the rest of the tests...

The first containers job (amd64-alpine-container) got
added to the custom runner and failed (because docker-dind
isn't there?):

$ export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
$ export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
$ apk add python3
bash: line 110: apk: command not found
Running after_script 00:01
Running after script...
$ docker logout
Removing login credentials for https://index.docker.io/v1/
ERROR: Job failed: exit status 1

Do we need to restrict the other jobs to the Gitlab public
(x86) runners? Maybe as:

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
@@ -1,6 +1,6 @@
 .container_job_template: &container_job_definition
+  tags:
+    - gitlab-org-docker
   image: docker:stable
   stage: containers
   services:

Daniel, you didn't hit this problem on the previous version
of this series?

Thanks,

Phil.

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Erik Skultety <eskultet@redhat.com> writes:

> On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
>> 
>> Cleber Rosa <crosa@redhat.com> writes:
>> 
>> > To run basic jobs on custom runners, the environment needs to be
>> > properly set up.  The most common requirement is having the right
>> > packages installed.
>> >
>> > The playbook introduced here covers the QEMU's project s390x and
>> > aarch64 machines.  At the time this is being proposed, those machines
>> > have already had this playbook applied to them.
>> >
>> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> > ---
>> >  docs/devel/ci.rst                      | 30 ++++++++++
>> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
>> >  scripts/ci/setup/inventory             |  1 +
>> >  3 files changed, 107 insertions(+)
>> >  create mode 100644 scripts/ci/setup/build-environment.yml
>> >  create mode 100644 scripts/ci/setup/inventory
>> >
>> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>> > index 585b7bf4b8..a556558435 100644
>> > --- a/docs/devel/ci.rst
>> > +++ b/docs/devel/ci.rst
>> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>> >  The GitLab CI jobs definition for the custom runners are located under::
>> >  
>> >    .gitlab-ci.d/custom-runners.yml
>> > +
>> > +Machine Setup Howto
>> > +-------------------
>> > +
>> > +For all Linux based systems, the setup can be mostly automated by the
>> > +execution of two Ansible playbooks.  Start by adding your machines to
>> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
>> > +
>> > +  fully.qualified.domain
>> > +  other.machine.hostname
>> 
>> Is this really needed? Can't the host list be passed in the command
>> line? I find it off to imagine users wanting to configure whole fleets
>> of runners.
>
> Why not support both, since the playbook execution is not wrapped by anything,
> giving the option of using either and inventory or direct cmdline invocation
> seems like the proper way to do it.

Sure - and I dare say people used to managing fleets of servers will
want to do it properly but in the first instance lets provide the simple
command line option so a user can get up and running without also
ensuring files are in the correct format.

>
>> 
>> > +
>> > +You may need to set some variables in the inventory file itself.  One
>> > +very common need is to tell Ansible to use a Python 3 interpreter on
>> > +those hosts.  This would look like::
>> > +
>> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>> > +
>> > +Build environment
>> > +~~~~~~~~~~~~~~~~~
>> > +
>> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>> > +set up machines with the environment needed to perform builds and run
>> > +QEMU tests.  It covers a number of different Linux distributions and
>> > +FreeBSD.
>> > +
>> > +To run the playbook, execute::
>> > +
>> > +  cd scripts/ci/setup
>> > +  ansible-playbook -i inventory build-environment.yml
>> 
>> So I got somewhat there with a direct command line invocation:
>> 
>>   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
>> 
>> although for some reason a single host -i fails...
>
> The trick is to end it with a ',' like "-i host1,"

Ahh found it thanks.

-- 
Alex Bennée

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> The QEMU project has two machines (aarch64 and s390x) that can be used
> for jobs that do build and run tests.  This introduces those jobs,
> which are a mapping of custom scripts used for the same purpose.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index 3004da2bda..a9166c82a2 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -12,3 +12,207 @@
>  # strategy.
>  variables:
>    GIT_SUBMODULE_STRATEGY: recursive
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'

Maybe this is too restrictive, we might want to test /master too.

> + script:
> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> + - mkdir build
> + - cd build
> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc` check-tcg V=1

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 2/23/21 4:27 PM, Philippe Mathieu-Daudé wrote:
> On 2/19/21 10:58 PM, Cleber Rosa wrote:
>> The QEMU project has two machines (aarch64 and s390x) that can be used
>> for jobs that do build and run tests.  This introduces those jobs,
>> which are a mapping of custom scripts used for the same purpose.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
>>  1 file changed, 204 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
>> index 3004da2bda..a9166c82a2 100644
>> --- a/.gitlab-ci.d/custom-runners.yml
>> +++ b/.gitlab-ci.d/custom-runners.yml
>> @@ -12,3 +12,207 @@
>>  # strategy.
>>  variables:
>>    GIT_SUBMODULE_STRATEGY: recursive
>> +
>> +# All ubuntu-18.04 jobs should run successfully in an environment
>> +# setup by the scripts/ci/setup/build-environment.yml task
>> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
>> +ubuntu-18.04-s390x-all-linux-static:
>> + allow_failure: true
>> + needs: []
>> + stage: build
>> + tags:
>> + - ubuntu_18.04
>> + - s390x
>> + rules:
>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> 
> Maybe this is too restrictive, we might want to test /master too.

Also now all fork pipelines are stuck...

  This job is stuck because you don't have any active runners online
  or available with any of these tags assigned to them: s390x
  ubuntu_18.04
  Go to project CI settings

https://gitlab.com/philmd/qemu/-/jobs/1050123478

What about using as starter:

  rules:
    if: '$CI_PROJECT_PATH == 'qemu-project/qemu'

> 
>> + script:
>> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
>> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
>> + - mkdir build
>> + - cd build
>> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
>> + - make --output-sync -j`nproc`
>> + - make --output-sync -j`nproc` check V=1
>> + - make --output-sync -j`nproc` check-tcg V=1

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Daniel P. Berrangé]

On Tue, Feb 23, 2021 at 04:35:41PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/23/21 4:27 PM, Philippe Mathieu-Daudé wrote:
> > On 2/19/21 10:58 PM, Cleber Rosa wrote:
> >> The QEMU project has two machines (aarch64 and s390x) that can be used
> >> for jobs that do build and run tests.  This introduces those jobs,
> >> which are a mapping of custom scripts used for the same purpose.
> >>
> >> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> >> ---
> >>  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
> >>  1 file changed, 204 insertions(+)
> >>
> >> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> >> index 3004da2bda..a9166c82a2 100644
> >> --- a/.gitlab-ci.d/custom-runners.yml
> >> +++ b/.gitlab-ci.d/custom-runners.yml
> >> @@ -12,3 +12,207 @@
> >>  # strategy.
> >>  variables:
> >>    GIT_SUBMODULE_STRATEGY: recursive
> >> +
> >> +# All ubuntu-18.04 jobs should run successfully in an environment
> >> +# setup by the scripts/ci/setup/build-environment.yml task
> >> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> >> +ubuntu-18.04-s390x-all-linux-static:
> >> + allow_failure: true
> >> + needs: []
> >> + stage: build
> >> + tags:
> >> + - ubuntu_18.04
> >> + - s390x
> >> + rules:
> >> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > 
> > Maybe this is too restrictive, we might want to test /master too.
> 
> Also now all fork pipelines are stuck...
> 
>   This job is stuck because you don't have any active runners online
>   or available with any of these tags assigned to them: s390x
>   ubuntu_18.04
>   Go to project CI settings
> 
> https://gitlab.com/philmd/qemu/-/jobs/1050123478
> 
> What about using as starter:
> 
>   rules:
>     if: '$CI_PROJECT_PATH == 'qemu-project/qemu'

I'm having dejavu about this exact problem previously this series was
posted. Restricting based on CI_PROJECT_PATH is not desirable, because
users should be free to bring up their own runners for this by following
the instructions earlier in the series. Having to hack the .gitlab-ci.yml
change this rule is going to be super unplesant.

If we can't make it auto-skip when no runners are available, then we
should set a rule based on a custom env variable. eg

   if "$QEMU_CI_RUNNER_S390" == "on"

then all a contributor needs todo is set the variable in their gitlab
repo preferences.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Daniel P. Berrangé]

On Tue, Feb 23, 2021 at 04:17:23PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/19/21 10:58 PM, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390x) that can be used
> > for jobs that do build and run tests.  This introduces those jobs,
> > which are a mapping of custom scripts used for the same purpose.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
> >  1 file changed, 204 insertions(+)
> > 
> > diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> > index 3004da2bda..a9166c82a2 100644
> > --- a/.gitlab-ci.d/custom-runners.yml
> > +++ b/.gitlab-ci.d/custom-runners.yml
> > @@ -12,3 +12,207 @@
> >  # strategy.
> >  variables:
> >    GIT_SUBMODULE_STRATEGY: recursive
> > +
> > +# All ubuntu-18.04 jobs should run successfully in an environment
> > +# setup by the scripts/ci/setup/build-environment.yml task
> > +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> > +ubuntu-18.04-s390x-all-linux-static:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> 
> Where is this tag list filled upon registration?
> 
> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > + script:
> > + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> > + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> > + - mkdir build
> > + - cd build
> > + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > + - make --output-sync -j`nproc` check-tcg V=1
> 
> Also this break the rest of the tests...
> 
> The first containers job (amd64-alpine-container) got
> added to the custom runner and failed (because docker-dind
> isn't there?):

Urgh, well that's a big problem. We certainly don't want *anything* being
placed on the custom runners without explicit opt-in, otherwise jobs run
in the main repo have a different environment from when users run on their
personal forks.

IOW, we need anti-affinity against our custom runners really.

> $ export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> $ export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> $ apk add python3
> bash: line 110: apk: command not found
> Running after_script 00:01
> Running after script...
> $ docker logout
> Removing login credentials for https://index.docker.io/v1/
> ERROR: Job failed: exit status 1
> 
> Do we need to restrict the other jobs to the Gitlab public
> (x86) runners? Maybe as:
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> @@ -1,6 +1,6 @@
>  .container_job_template: &container_job_definition
> +  tags:
> +    - gitlab-org-docker

Is that a real tag that exists on gitlab's shared runners, or something
you just invented ?

>    image: docker:stable
>    stage: containers
>    services:
> 
> Daniel, you didn't hit this problem on the previous version
> of this series?

I didn't try actually executing previous postings of this series.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Philippe Mathieu-Daudé]

On 2/23/21 12:25 PM, Thomas Huth wrote:
> On 19/02/2021 22.58, Cleber Rosa wrote:
>> As described in the included documentation, the "custom runner" jobs
>> extend the GitLab CI jobs already in place.  One of their primary
>> goals of catching and preventing regressions on a wider number of host
>> systems than the ones provided by GitLab's shared runners.
>>
>> This sets the stage in which other community members can add their own
>> machine configuration documentation/scripts, and accompanying job
>> definitions.  As a general rule, those newly added contributed jobs
>> should run as "non-gating", until their reliability is verified (AKA
>> "allow_failure: true").
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>>   .gitlab-ci.yml                  |  1 +
>>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>>   docs/devel/index.rst            |  1 +
>>   4 files changed, 44 insertions(+)
>>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>>   create mode 100644 docs/devel/ci.rst
>>
>> diff --git a/.gitlab-ci.d/custom-runners.yml
>> b/.gitlab-ci.d/custom-runners.yml
>> new file mode 100644
>> index 0000000000..3004da2bda
>> --- /dev/null
>> +++ b/.gitlab-ci.d/custom-runners.yml
>> @@ -0,0 +1,14 @@
>> +# The CI jobs defined here require GitLab runners installed and
>> +# registered on machines that match their operating system names,
>> +# versions and architectures.  This is in contrast to the other CI
>> +# jobs that are intended to run on GitLab's "shared" runners.
>> +
>> +# Different than the default approach on "shared" runners, based on
>> +# containers, the custom runners have no such *requirement*, as those
>> +# jobs should be capable of running on operating systems with no
>> +# compatible container implementation, or no support from
>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
>> +# reusing the GIT repository, let's enable the recursive submodule
>> +# strategy.
>> +variables:
>> +  GIT_SUBMODULE_STRATEGY: recursive
> 
> Is it really necessary? I thought our configure script would take care
> of the submodules?

Well, if there is a failure during the first clone (I got one network
timeout in the middle) then next time it doesn't work:

Updating/initializing submodules recursively...
Synchronizing submodule url for 'capstone'
Synchronizing submodule url for 'dtc'
Synchronizing submodule url for 'meson'
Synchronizing submodule url for 'roms/QemuMacDrivers'
Synchronizing submodule url for 'roms/SLOF'
Synchronizing submodule url for 'roms/edk2'
Synchronizing submodule url for
'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
Synchronizing submodule url for
'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
Synchronizing submodule url for
'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
Synchronizing submodule url for
'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
Synchronizing submodule url for
'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
Synchronizing submodule url for
'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
Synchronizing submodule url for
'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
Synchronizing submodule url for
'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
Synchronizing submodule url for 'roms/ipxe'
Synchronizing submodule url for 'roms/openbios'
Synchronizing submodule url for 'roms/opensbi'
Synchronizing submodule url for 'roms/qboot'
Synchronizing submodule url for 'roms/qemu-palcode'
Synchronizing submodule url for 'roms/seabios'
Synchronizing submodule url for 'roms/seabios-hppa'
Synchronizing submodule url for 'roms/sgabios'
Synchronizing submodule url for 'roms/skiboot'
Synchronizing submodule url for 'roms/u-boot'
Synchronizing submodule url for 'roms/u-boot-sam460ex'
Synchronizing submodule url for 'roms/vbootrom'
Synchronizing submodule url for 'slirp'
Synchronizing submodule url for 'tests/fp/berkeley-softfloat-3'
Synchronizing submodule url for 'tests/fp/berkeley-testfloat-3'
Synchronizing submodule url for 'ui/keycodemapdb'
Entering 'capstone'
Entering 'dtc'
Entering 'meson'
Entering 'roms/QemuMacDrivers'
Entering 'roms/SLOF'
Entering 'roms/edk2'
Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
Entering
'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
Entering 'roms/ipxe'
Entering 'roms/openbios'
Entering 'roms/opensbi'
Entering 'roms/qboot'
Entering 'roms/qemu-palcode'
Entering 'roms/seabios'
Entering 'roms/seabios-hppa'
Entering 'roms/sgabios'
Entering 'roms/skiboot'
Entering 'roms/u-boot'
Entering 'roms/u-boot-sam460ex'
Entering 'roms/vbootrom'
Entering 'slirp'
Entering 'tests/fp/berkeley-softfloat-3'
Entering 'tests/fp/berkeley-testfloat-3'
Entering 'ui/keycodemapdb'
Entering 'capstone'
HEAD is now at f8b1b833 fix CS_ mips_ OP structure comment error (#1674)
Entering 'dtc'
HEAD is now at 85e5d83 Makefile: when building libfdt only, do not add
unneeded deps
Entering 'meson'
HEAD is now at 776acd2a8 Bump versions to 0.55.3 for release
Entering 'roms/QemuMacDrivers'
HEAD is now at 90c488d Merge pull request #3 from
mcayland/fix/unbreak-256-color-mode
Entering 'roms/SLOF'
HEAD is now at e18ddad version: update to 20200717
Entering 'roms/edk2'
HEAD is now at 06dc822d04 Revert ".pytool/EccCheck: Disable Ecc error
code 10014 for open CI"
Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
HEAD is now at b64af41 Fix typo in function
'softfloat_propagateNaNF128M' for RISC-V.
Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
HEAD is now at 666c328 Make types of variable match (#796)
Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
HEAD is now at ca7cb33 move to git
Entering
'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
HEAD is now at 5f60d6f Merge pull request #7 from kloetzl/master
Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
HEAD is now at 63be8a9 unichr was removed in Python 3 because all str
are Unicode (#877)
Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
HEAD is now at b2c1da6 add ONIG_OPTION_CALLBACK_EACH_MATCH test
Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
HEAD is now at 160dffe Don't use non-literal format strings
Entering 'roms/ipxe'
HEAD is now at 4bd064de [build] Fix building on older versions of gcc
Entering 'roms/openbios'
HEAD is now at 7f28286 PPC: mark first 4 pages of physical and virtual
memory as unavailable
Entering 'roms/opensbi'
HEAD is now at a98258d include: Bump-up version to 0.8
Entering 'roms/qboot'
HEAD is now at a5300c4 qboot: Disable PIE for ELF binary build step
Entering 'roms/qemu-palcode'
HEAD is now at bf0e136 Report machine checks to the kernel
Entering 'roms/seabios'
HEAD is now at 155821a docs: Note v1.14.0 release
Entering 'roms/seabios-hppa'
HEAD is now at 73b740f7 parisc: Set text planes and used_bits in STI fields
Entering 'roms/sgabios'
HEAD is now at cbaee52 SGABIOS: fix wrong video attrs for int 10h, ah==13h
Entering 'roms/skiboot'
HEAD is now at 3a6fdede skiboot v6.4 release notes
Entering 'roms/u-boot'
HEAD is now at d3689267f9 Prepare v2019.01
Entering 'roms/u-boot-sam460ex'
HEAD is now at 60b3916 Add README to clarify relation to U-Boot and
ACube's version
Entering 'roms/vbootrom'
HEAD is now at 0c37a43 Merge pull request #1 from google/disable-build-id
Entering 'slirp'
HEAD is now at 8f43a99 Merge branch 'stable-4.2' into 'stable-4.2'
Entering 'tests/fp/berkeley-softfloat-3'
HEAD is now at b64af41 Fix typo in function
'softfloat_propagateNaNF128M' for RISC-V.
Entering 'tests/fp/berkeley-testfloat-3'
HEAD is now at 5a59dce fail: constify fail_programName
Entering 'ui/keycodemapdb'
HEAD is now at 6119e6e Fix scan codes for Korean keys
fatal: Needed a single revision
Unable to find current revision in submodule path
'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
Failed to recurse into submodule path 'roms/edk2'
ERROR: Job failed: exit status 1

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 2/23/21 4:56 PM, Daniel P. Berrangé wrote:
> On Tue, Feb 23, 2021 at 04:17:23PM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/19/21 10:58 PM, Cleber Rosa wrote:
>>> The QEMU project has two machines (aarch64 and s390x) that can be used
>>> for jobs that do build and run tests.  This introduces those jobs,
>>> which are a mapping of custom scripts used for the same purpose.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
>>>  1 file changed, 204 insertions(+)
>>>
>>> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
>>> index 3004da2bda..a9166c82a2 100644
>>> --- a/.gitlab-ci.d/custom-runners.yml
>>> +++ b/.gitlab-ci.d/custom-runners.yml
>>> @@ -12,3 +12,207 @@
>>>  # strategy.
>>>  variables:
>>>    GIT_SUBMODULE_STRATEGY: recursive
>>> +
>>> +# All ubuntu-18.04 jobs should run successfully in an environment
>>> +# setup by the scripts/ci/setup/build-environment.yml task
>>> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
>>> +ubuntu-18.04-s390x-all-linux-static:
>>> + allow_failure: true
>>> + needs: []
>>> + stage: build
>>> + tags:
>>> + - ubuntu_18.04
>>> + - s390x
>>
>> Where is this tag list filled upon registration?
>>
>>> + rules:
>>> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
>>> + script:
>>> + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
>>> + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
>>> + - mkdir build
>>> + - cd build
>>> + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
>>> + - make --output-sync -j`nproc`
>>> + - make --output-sync -j`nproc` check V=1
>>> + - make --output-sync -j`nproc` check-tcg V=1
>>
>> Also this break the rest of the tests...
>>
>> The first containers job (amd64-alpine-container) got
>> added to the custom runner and failed (because docker-dind
>> isn't there?):
> 
> Urgh, well that's a big problem. We certainly don't want *anything* being
> placed on the custom runners without explicit opt-in, otherwise jobs run
> in the main repo have a different environment from when users run on their
> personal forks.
> 
> IOW, we need anti-affinity against our custom runners really.
> 
>> $ export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>> $ export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>> $ apk add python3
>> bash: line 110: apk: command not found
>> Running after_script 00:01
>> Running after script...
>> $ docker logout
>> Removing login credentials for https://index.docker.io/v1/
>> ERROR: Job failed: exit status 1
>>
>> Do we need to restrict the other jobs to the Gitlab public
>> (x86) runners? Maybe as:
>>
>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> @@ -1,6 +1,6 @@
>>  .container_job_template: &container_job_definition
>> +  tags:
>> +    - gitlab-org-docker
> 
> Is that a real tag that exists on gitlab's shared runners, or something
> you just invented ?

This is not standardized yet:
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5420

I checked the available runners, some have 'docker' while other have
'gitlab-org-docker'. There are more 'gitlab-org-docker' than 'docker'
runners. The tag selection is not exclusive, this is a "all or nothing"
selection, so for my testing I choose 'gitlab-org-docker' which is
the most available.

> 
>>    image: docker:stable
>>    stage: containers
>>    services:
>>
>> Daniel, you didn't hit this problem on the previous version
>> of this series?
> 
> I didn't try actually executing previous postings of this series.
> 
> 
> Regards,
> Daniel
> 

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 9907 bytes --]

On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/23/21 12:25 PM, Thomas Huth wrote:
> > On 19/02/2021 22.58, Cleber Rosa wrote:
> >> As described in the included documentation, the "custom runner" jobs
> >> extend the GitLab CI jobs already in place.  One of their primary
> >> goals of catching and preventing regressions on a wider number of host
> >> systems than the ones provided by GitLab's shared runners.
> >>
> >> This sets the stage in which other community members can add their own
> >> machine configuration documentation/scripts, and accompanying job
> >> definitions.  As a general rule, those newly added contributed jobs
> >> should run as "non-gating", until their reliability is verified (AKA
> >> "allow_failure: true").
> >>
> >> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >> ---
> >>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
> >>   .gitlab-ci.yml                  |  1 +
> >>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
> >>   docs/devel/index.rst            |  1 +
> >>   4 files changed, 44 insertions(+)
> >>   create mode 100644 .gitlab-ci.d/custom-runners.yml
> >>   create mode 100644 docs/devel/ci.rst
> >>
> >> diff --git a/.gitlab-ci.d/custom-runners.yml
> >> b/.gitlab-ci.d/custom-runners.yml
> >> new file mode 100644
> >> index 0000000000..3004da2bda
> >> --- /dev/null
> >> +++ b/.gitlab-ci.d/custom-runners.yml
> >> @@ -0,0 +1,14 @@
> >> +# The CI jobs defined here require GitLab runners installed and
> >> +# registered on machines that match their operating system names,
> >> +# versions and architectures.  This is in contrast to the other CI
> >> +# jobs that are intended to run on GitLab's "shared" runners.
> >> +
> >> +# Different than the default approach on "shared" runners, based on
> >> +# containers, the custom runners have no such *requirement*, as those
> >> +# jobs should be capable of running on operating systems with no
> >> +# compatible container implementation, or no support from
> >> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> >> +# reusing the GIT repository, let's enable the recursive submodule
> >> +# strategy.
> >> +variables:
> >> +  GIT_SUBMODULE_STRATEGY: recursive
> > 
> > Is it really necessary? I thought our configure script would take care
> > of the submodules?
>

I've done a lot of testing on bare metal systems, and the problems
that come from reusing the same system and failed cleanups can be very
frustrating.  It's unfortunate that we need this, but it was the
simplest and most reliable solution I found.  :/

Having said that, I noticed after I posted this series that this is
affecting all other jobs.  We don't need it that in the jobs based
on containers (for obvious reasons), so I see two options:

1) have it enabled on all jobs for consistency

2) have it enabled only on jobs that will reuse the repo

> Well, if there is a failure during the first clone (I got one network
> timeout in the middle) then next time it doesn't work:
> 
> Updating/initializing submodules recursively...
> Synchronizing submodule url for 'capstone'
> Synchronizing submodule url for 'dtc'
> Synchronizing submodule url for 'meson'
> Synchronizing submodule url for 'roms/QemuMacDrivers'
> Synchronizing submodule url for 'roms/SLOF'
> Synchronizing submodule url for 'roms/edk2'
> Synchronizing submodule url for
> 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
> Synchronizing submodule url for
> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
> Synchronizing submodule url for
> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
> Synchronizing submodule url for
> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
> Synchronizing submodule url for
> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
> Synchronizing submodule url for
> 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
> Synchronizing submodule url for
> 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
> Synchronizing submodule url for
> 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
> Synchronizing submodule url for 'roms/ipxe'
> Synchronizing submodule url for 'roms/openbios'
> Synchronizing submodule url for 'roms/opensbi'
> Synchronizing submodule url for 'roms/qboot'
> Synchronizing submodule url for 'roms/qemu-palcode'
> Synchronizing submodule url for 'roms/seabios'
> Synchronizing submodule url for 'roms/seabios-hppa'
> Synchronizing submodule url for 'roms/sgabios'
> Synchronizing submodule url for 'roms/skiboot'
> Synchronizing submodule url for 'roms/u-boot'
> Synchronizing submodule url for 'roms/u-boot-sam460ex'
> Synchronizing submodule url for 'roms/vbootrom'
> Synchronizing submodule url for 'slirp'
> Synchronizing submodule url for 'tests/fp/berkeley-softfloat-3'
> Synchronizing submodule url for 'tests/fp/berkeley-testfloat-3'
> Synchronizing submodule url for 'ui/keycodemapdb'
> Entering 'capstone'
> Entering 'dtc'
> Entering 'meson'
> Entering 'roms/QemuMacDrivers'
> Entering 'roms/SLOF'
> Entering 'roms/edk2'
> Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
> Entering
> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
> Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
> Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
> Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
> Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
> Entering 'roms/ipxe'
> Entering 'roms/openbios'
> Entering 'roms/opensbi'
> Entering 'roms/qboot'
> Entering 'roms/qemu-palcode'
> Entering 'roms/seabios'
> Entering 'roms/seabios-hppa'
> Entering 'roms/sgabios'
> Entering 'roms/skiboot'
> Entering 'roms/u-boot'
> Entering 'roms/u-boot-sam460ex'
> Entering 'roms/vbootrom'
> Entering 'slirp'
> Entering 'tests/fp/berkeley-softfloat-3'
> Entering 'tests/fp/berkeley-testfloat-3'
> Entering 'ui/keycodemapdb'
> Entering 'capstone'
> HEAD is now at f8b1b833 fix CS_ mips_ OP structure comment error (#1674)
> Entering 'dtc'
> HEAD is now at 85e5d83 Makefile: when building libfdt only, do not add
> unneeded deps
> Entering 'meson'
> HEAD is now at 776acd2a8 Bump versions to 0.55.3 for release
> Entering 'roms/QemuMacDrivers'
> HEAD is now at 90c488d Merge pull request #3 from
> mcayland/fix/unbreak-256-color-mode
> Entering 'roms/SLOF'
> HEAD is now at e18ddad version: update to 20200717
> Entering 'roms/edk2'
> HEAD is now at 06dc822d04 Revert ".pytool/EccCheck: Disable Ecc error
> code 10014 for open CI"
> Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
> HEAD is now at b64af41 Fix typo in function
> 'softfloat_propagateNaNF128M' for RISC-V.
> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
> HEAD is now at 666c328 Make types of variable match (#796)
> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
> HEAD is now at ca7cb33 move to git
> Entering
> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
> HEAD is now at 5f60d6f Merge pull request #7 from kloetzl/master
> Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
> Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
> HEAD is now at 63be8a9 unichr was removed in Python 3 because all str
> are Unicode (#877)
> Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
> HEAD is now at b2c1da6 add ONIG_OPTION_CALLBACK_EACH_MATCH test
> Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
> HEAD is now at 160dffe Don't use non-literal format strings
> Entering 'roms/ipxe'
> HEAD is now at 4bd064de [build] Fix building on older versions of gcc
> Entering 'roms/openbios'
> HEAD is now at 7f28286 PPC: mark first 4 pages of physical and virtual
> memory as unavailable
> Entering 'roms/opensbi'
> HEAD is now at a98258d include: Bump-up version to 0.8
> Entering 'roms/qboot'
> HEAD is now at a5300c4 qboot: Disable PIE for ELF binary build step
> Entering 'roms/qemu-palcode'
> HEAD is now at bf0e136 Report machine checks to the kernel
> Entering 'roms/seabios'
> HEAD is now at 155821a docs: Note v1.14.0 release
> Entering 'roms/seabios-hppa'
> HEAD is now at 73b740f7 parisc: Set text planes and used_bits in STI fields
> Entering 'roms/sgabios'
> HEAD is now at cbaee52 SGABIOS: fix wrong video attrs for int 10h, ah==13h
> Entering 'roms/skiboot'
> HEAD is now at 3a6fdede skiboot v6.4 release notes
> Entering 'roms/u-boot'
> HEAD is now at d3689267f9 Prepare v2019.01
> Entering 'roms/u-boot-sam460ex'
> HEAD is now at 60b3916 Add README to clarify relation to U-Boot and
> ACube's version
> Entering 'roms/vbootrom'
> HEAD is now at 0c37a43 Merge pull request #1 from google/disable-build-id
> Entering 'slirp'
> HEAD is now at 8f43a99 Merge branch 'stable-4.2' into 'stable-4.2'
> Entering 'tests/fp/berkeley-softfloat-3'
> HEAD is now at b64af41 Fix typo in function
> 'softfloat_propagateNaNF128M' for RISC-V.
> Entering 'tests/fp/berkeley-testfloat-3'
> HEAD is now at 5a59dce fail: constify fail_programName
> Entering 'ui/keycodemapdb'
> HEAD is now at 6119e6e Fix scan codes for Korean keys
> fatal: Needed a single revision
> Unable to find current revision in submodule path
> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
> Failed to recurse into submodule path 'roms/edk2'
> ERROR: Job failed: exit status 1
> 

Yes, I've also found similar issues during my jobs.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 7394 bytes --]

On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
> 
> Cleber Rosa <crosa@redhat.com> writes:
> 
> > To run basic jobs on custom runners, the environment needs to be
> > properly set up.  The most common requirement is having the right
> > packages installed.
> >
> > The playbook introduced here covers the QEMU's project s390x and
> > aarch64 machines.  At the time this is being proposed, those machines
> > have already had this playbook applied to them.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  docs/devel/ci.rst                      | 30 ++++++++++
> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
> >  scripts/ci/setup/inventory             |  1 +
> >  3 files changed, 107 insertions(+)
> >  create mode 100644 scripts/ci/setup/build-environment.yml
> >  create mode 100644 scripts/ci/setup/inventory
> >
> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > index 585b7bf4b8..a556558435 100644
> > --- a/docs/devel/ci.rst
> > +++ b/docs/devel/ci.rst
> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >  The GitLab CI jobs definition for the custom runners are located under::
> >  
> >    .gitlab-ci.d/custom-runners.yml
> > +
> > +Machine Setup Howto
> > +-------------------
> > +
> > +For all Linux based systems, the setup can be mostly automated by the
> > +execution of two Ansible playbooks.  Start by adding your machines to
> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> > +
> > +  fully.qualified.domain
> > +  other.machine.hostname
> 
> Is this really needed? Can't the host list be passed in the command
> line? I find it off to imagine users wanting to configure whole fleets
> of runners.
>

No, it's not needed.

But, in my experience, it's the most common way people use
ansible-playbook.  As with all most tools QEMU relies on, that are
many different ways of using them.  IMO documenting more than one way
to perform the same task makes the documentation unclear.

> > +
> > +You may need to set some variables in the inventory file itself.  One
> > +very common need is to tell Ansible to use a Python 3 interpreter on
> > +those hosts.  This would look like::
> > +
> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > +
> > +Build environment
> > +~~~~~~~~~~~~~~~~~
> > +
> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > +set up machines with the environment needed to perform builds and run
> > +QEMU tests.  It covers a number of different Linux distributions and
> > +FreeBSD.
> > +
> > +To run the playbook, execute::
> > +
> > +  cd scripts/ci/setup
> > +  ansible-playbook -i inventory build-environment.yml
> 
> So I got somewhat there with a direct command line invocation:
> 
>   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
>

Yes, and the "-e" is another example of the multiple ways to achieve
the same task.

> although for some reason a single host -i fails...
> 
> > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml

It requires a comma separated list, even if it's a list with a single
item:

   https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#cmdoption-ansible-playbook-i

> > new file mode 100644
> > index 0000000000..0197e0a48b
> > --- /dev/null
> > +++ b/scripts/ci/setup/build-environment.yml
> > @@ -0,0 +1,76 @@
> > +---
> > +- name: Installation of basic packages to build QEMU
> > +  hosts: all
> > +  tasks:
> > +    - name: Update apt cache
> > +      apt:
> > +        update_cache: yes
> > +      when:
> > +        - ansible_facts['distribution'] == 'Ubuntu'
> 
> So are we limiting to Ubuntu here rather than say a Debian base?
>

You have a point, because this would certainly work and be applicable
to Debian systems too.  But, this is a new addition on v5, and I'm
limiting this patch to the machines that are available/connected right
now to the QEMU project on GitLab.

I can change that to "distribution_family == Debian" if you think
it's a good idea.  But IMO it'd make more sense for a patch
introducing the package list for Debian systems to change that.

> > +
> > +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> > +      package:
> > +        name:
> > +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> > +          - ccache
> > +          - clang
> > +          - gcc
> > +          - gettext
> > +          - git
> > +          - glusterfs-common
> > +          - libaio-dev
> > +          - libattr1-dev
> > +          - libbrlapi-dev
> > +          - libbz2-dev
> > +          - libcacard-dev
> > +          - libcap-ng-dev
> > +          - libcurl4-gnutls-dev
> > +          - libdrm-dev
> > +          - libepoxy-dev
> > +          - libfdt-dev
> > +          - libgbm-dev
> > +          - libgtk-3-dev
> > +          - libibverbs-dev
> > +          - libiscsi-dev
> > +          - libjemalloc-dev
> > +          - libjpeg-turbo8-dev
> > +          - liblzo2-dev
> > +          - libncurses5-dev
> > +          - libncursesw5-dev
> > +          - libnfs-dev
> > +          - libnss3-dev
> > +          - libnuma-dev
> > +          - libpixman-1-dev
> > +          - librados-dev
> > +          - librbd-dev
> > +          - librdmacm-dev
> > +          - libsasl2-dev
> > +          - libsdl2-dev
> > +          - libseccomp-dev
> > +          - libsnappy-dev
> > +          - libspice-protocol-dev
> > +          - libssh-dev
> > +          - libusb-1.0-0-dev
> > +          - libusbredirhost-dev
> > +          - libvdeplug-dev
> > +          - libvte-2.91-dev
> > +          - libzstd-dev
> > +          - make
> > +          - ninja-build
> > +          - python3-yaml
> > +          - python3-sphinx
> > +          - sparse
> > +          - xfslibs-dev
> > +        state: present
> > +      when:
> > +        - ansible_facts['distribution'] == 'Ubuntu'
> > +
> > +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> > +      package:
> > +        name:
> > +          - libspice-server-dev
> > +          - libxen-dev
> > +        state: present
> > +      when:
> > +        - ansible_facts['distribution'] == 'Ubuntu'
> > +        - ansible_facts['architecture'] != 's390x'
> > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
> > new file mode 100644
> > index 0000000000..2fbb50c4a8
> > --- /dev/null
> > +++ b/scripts/ci/setup/inventory
> > @@ -0,0 +1 @@
> > +localhost
> 
> I'm not sure we should have a default here because it will inevitably
> cause someone to do something to their machine when trying to setup a
> runner.
>

Fair enough.  Then I see two options:

1) follow the vars.yml.template example and only ship a
   inventory.template file

2) use a placeholder with an impossible hostname such as
   "my-qemu-runner.example.org" or "your-host-name-here"

> -- 
> Alex Bennée
> 

Let me know what you think is more reasonable, and thanks for the
review!

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 3658 bytes --]

On Tue, Feb 23, 2021 at 03:51:33PM +0100, Erik Skultety wrote:
> On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
> > 
> > Cleber Rosa <crosa@redhat.com> writes:
> > 
> > > To run basic jobs on custom runners, the environment needs to be
> > > properly set up.  The most common requirement is having the right
> > > packages installed.
> > >
> > > The playbook introduced here covers the QEMU's project s390x and
> > > aarch64 machines.  At the time this is being proposed, those machines
> > > have already had this playbook applied to them.
> > >
> > > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > > ---
> > >  docs/devel/ci.rst                      | 30 ++++++++++
> > >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
> > >  scripts/ci/setup/inventory             |  1 +
> > >  3 files changed, 107 insertions(+)
> > >  create mode 100644 scripts/ci/setup/build-environment.yml
> > >  create mode 100644 scripts/ci/setup/inventory
> > >
> > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> > > index 585b7bf4b8..a556558435 100644
> > > --- a/docs/devel/ci.rst
> > > +++ b/docs/devel/ci.rst
> > > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> > >  The GitLab CI jobs definition for the custom runners are located under::
> > >  
> > >    .gitlab-ci.d/custom-runners.yml
> > > +
> > > +Machine Setup Howto
> > > +-------------------
> > > +
> > > +For all Linux based systems, the setup can be mostly automated by the
> > > +execution of two Ansible playbooks.  Start by adding your machines to
> > > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> > > +
> > > +  fully.qualified.domain
> > > +  other.machine.hostname
> > 
> > Is this really needed? Can't the host list be passed in the command
> > line? I find it off to imagine users wanting to configure whole fleets
> > of runners.
> 
> Why not support both, since the playbook execution is not wrapped by anything,
> giving the option of using either and inventory or direct cmdline invocation
> seems like the proper way to do it.
>

Well, these two (and possibly many others) are supported by
ansible-playbook.  I don't think we should document more than one
though, as it leads to a more confusing documentation.

> > 
> > > +
> > > +You may need to set some variables in the inventory file itself.  One
> > > +very common need is to tell Ansible to use a Python 3 interpreter on
> > > +those hosts.  This would look like::
> > > +
> > > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> > > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
> > > +
> > > +Build environment
> > > +~~~~~~~~~~~~~~~~~
> > > +
> > > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> > > +set up machines with the environment needed to perform builds and run
> > > +QEMU tests.  It covers a number of different Linux distributions and
> > > +FreeBSD.
> > > +
> > > +To run the playbook, execute::
> > > +
> > > +  cd scripts/ci/setup
> > > +  ansible-playbook -i inventory build-environment.yml
> > 
> > So I got somewhat there with a direct command line invocation:
> > 
> >   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
> > 
> > although for some reason a single host -i fails...
> 
> The trick is to end it with a ',' like "-i host1,"
>

Yep, that is the trick!  A weird one nevertheless... :)

> Erik

Thanks for the review and comments so far Erik!

Best,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2886 bytes --]

On Tue, Feb 23, 2021 at 03:17:24PM +0000, Alex Bennée wrote:
> 
> Erik Skultety <eskultet@redhat.com> writes:
> 
> > On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
> >> 
> >> Cleber Rosa <crosa@redhat.com> writes:
> >> 
> >> > To run basic jobs on custom runners, the environment needs to be
> >> > properly set up.  The most common requirement is having the right
> >> > packages installed.
> >> >
> >> > The playbook introduced here covers the QEMU's project s390x and
> >> > aarch64 machines.  At the time this is being proposed, those machines
> >> > have already had this playbook applied to them.
> >> >
> >> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >> > ---
> >> >  docs/devel/ci.rst                      | 30 ++++++++++
> >> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
> >> >  scripts/ci/setup/inventory             |  1 +
> >> >  3 files changed, 107 insertions(+)
> >> >  create mode 100644 scripts/ci/setup/build-environment.yml
> >> >  create mode 100644 scripts/ci/setup/inventory
> >> >
> >> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> >> > index 585b7bf4b8..a556558435 100644
> >> > --- a/docs/devel/ci.rst
> >> > +++ b/docs/devel/ci.rst
> >> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
> >> >  The GitLab CI jobs definition for the custom runners are located under::
> >> >  
> >> >    .gitlab-ci.d/custom-runners.yml
> >> > +
> >> > +Machine Setup Howto
> >> > +-------------------
> >> > +
> >> > +For all Linux based systems, the setup can be mostly automated by the
> >> > +execution of two Ansible playbooks.  Start by adding your machines to
> >> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
> >> > +
> >> > +  fully.qualified.domain
> >> > +  other.machine.hostname
> >> 
> >> Is this really needed? Can't the host list be passed in the command
> >> line? I find it off to imagine users wanting to configure whole fleets
> >> of runners.
> >
> > Why not support both, since the playbook execution is not wrapped by anything,
> > giving the option of using either and inventory or direct cmdline invocation
> > seems like the proper way to do it.
> 
> Sure - and I dare say people used to managing fleets of servers will
> want to do it properly but in the first instance lets provide the simple
> command line option so a user can get up and running without also
> ensuring files are in the correct format.
>

Like I said before, I'm strongly in favor of a more straightforward
documentation, instead of documenting multiple ways to perform the
same task.  I clearly believe that writing the inventory file (which
will later be used for the second gitlab-runner playbook) is the best
choice here.

Do you think the command line approach is clearer?  Should we switch?

Regards,
Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Philippe Mathieu-Daudé]

On 2/23/21 5:47 PM, Cleber Rosa wrote:
> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/23/21 12:25 PM, Thomas Huth wrote:
>>> On 19/02/2021 22.58, Cleber Rosa wrote:
>>>> As described in the included documentation, the "custom runner" jobs
>>>> extend the GitLab CI jobs already in place.  One of their primary
>>>> goals of catching and preventing regressions on a wider number of host
>>>> systems than the ones provided by GitLab's shared runners.
>>>>
>>>> This sets the stage in which other community members can add their own
>>>> machine configuration documentation/scripts, and accompanying job
>>>> definitions.  As a general rule, those newly added contributed jobs
>>>> should run as "non-gating", until their reliability is verified (AKA
>>>> "allow_failure: true").
>>>>
>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>>>>   .gitlab-ci.yml                  |  1 +
>>>>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>>>>   docs/devel/index.rst            |  1 +
>>>>   4 files changed, 44 insertions(+)
>>>>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>>>>   create mode 100644 docs/devel/ci.rst
>>>>
>>>> diff --git a/.gitlab-ci.d/custom-runners.yml
>>>> b/.gitlab-ci.d/custom-runners.yml
>>>> new file mode 100644
>>>> index 0000000000..3004da2bda
>>>> --- /dev/null
>>>> +++ b/.gitlab-ci.d/custom-runners.yml
>>>> @@ -0,0 +1,14 @@
>>>> +# The CI jobs defined here require GitLab runners installed and
>>>> +# registered on machines that match their operating system names,
>>>> +# versions and architectures.  This is in contrast to the other CI
>>>> +# jobs that are intended to run on GitLab's "shared" runners.
>>>> +
>>>> +# Different than the default approach on "shared" runners, based on
>>>> +# containers, the custom runners have no such *requirement*, as those
>>>> +# jobs should be capable of running on operating systems with no
>>>> +# compatible container implementation, or no support from
>>>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
>>>> +# reusing the GIT repository, let's enable the recursive submodule
>>>> +# strategy.
>>>> +variables:
>>>> +  GIT_SUBMODULE_STRATEGY: recursive
>>>
>>> Is it really necessary? I thought our configure script would take care
>>> of the submodules?
>>
> 
> I've done a lot of testing on bare metal systems, and the problems
> that come from reusing the same system and failed cleanups can be very
> frustrating.  It's unfortunate that we need this, but it was the
> simplest and most reliable solution I found.  :/
> 
> Having said that, I noticed after I posted this series that this is
> affecting all other jobs.  We don't need it that in the jobs based
> on containers (for obvious reasons), so I see two options:
> 
> 1) have it enabled on all jobs for consistency
> 
> 2) have it enabled only on jobs that will reuse the repo
> 
>> Well, if there is a failure during the first clone (I got one network
>> timeout in the middle) 

[This network failure is pasted at the end]

>> then next time it doesn't work:
>>
>> Updating/initializing submodules recursively...
>> Synchronizing submodule url for 'capstone'
>> Synchronizing submodule url for 'dtc'
>> Synchronizing submodule url for 'meson'
>> Synchronizing submodule url for 'roms/QemuMacDrivers'
>> Synchronizing submodule url for 'roms/SLOF'
>> Synchronizing submodule url for 'roms/edk2'
>> Synchronizing submodule url for
>> 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
>> Synchronizing submodule url for
>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
>> Synchronizing submodule url for
>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
>> Synchronizing submodule url for
>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
>> Synchronizing submodule url for
>> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>> Synchronizing submodule url for
>> 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
>> Synchronizing submodule url for
>> 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
>> Synchronizing submodule url for
>> 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
>> Synchronizing submodule url for 'roms/ipxe'
>> Synchronizing submodule url for 'roms/openbios'
>> Synchronizing submodule url for 'roms/opensbi'
>> Synchronizing submodule url for 'roms/qboot'
>> Synchronizing submodule url for 'roms/qemu-palcode'
>> Synchronizing submodule url for 'roms/seabios'
>> Synchronizing submodule url for 'roms/seabios-hppa'
>> Synchronizing submodule url for 'roms/sgabios'
>> Synchronizing submodule url for 'roms/skiboot'
>> Synchronizing submodule url for 'roms/u-boot'
>> Synchronizing submodule url for 'roms/u-boot-sam460ex'
>> Synchronizing submodule url for 'roms/vbootrom'
>> Synchronizing submodule url for 'slirp'
>> Synchronizing submodule url for 'tests/fp/berkeley-softfloat-3'
>> Synchronizing submodule url for 'tests/fp/berkeley-testfloat-3'
>> Synchronizing submodule url for 'ui/keycodemapdb'
>> Entering 'capstone'
>> Entering 'dtc'
>> Entering 'meson'
>> Entering 'roms/QemuMacDrivers'
>> Entering 'roms/SLOF'
>> Entering 'roms/edk2'
>> Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
>> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
>> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
>> Entering
>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
>> Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>> Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
>> Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
>> Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
>> Entering 'roms/ipxe'
>> Entering 'roms/openbios'
>> Entering 'roms/opensbi'
>> Entering 'roms/qboot'
>> Entering 'roms/qemu-palcode'
>> Entering 'roms/seabios'
>> Entering 'roms/seabios-hppa'
>> Entering 'roms/sgabios'
>> Entering 'roms/skiboot'
>> Entering 'roms/u-boot'
>> Entering 'roms/u-boot-sam460ex'
>> Entering 'roms/vbootrom'
>> Entering 'slirp'
>> Entering 'tests/fp/berkeley-softfloat-3'
>> Entering 'tests/fp/berkeley-testfloat-3'
>> Entering 'ui/keycodemapdb'
>> Entering 'capstone'
>> HEAD is now at f8b1b833 fix CS_ mips_ OP structure comment error (#1674)
>> Entering 'dtc'
>> HEAD is now at 85e5d83 Makefile: when building libfdt only, do not add
>> unneeded deps
>> Entering 'meson'
>> HEAD is now at 776acd2a8 Bump versions to 0.55.3 for release
>> Entering 'roms/QemuMacDrivers'
>> HEAD is now at 90c488d Merge pull request #3 from
>> mcayland/fix/unbreak-256-color-mode
>> Entering 'roms/SLOF'
>> HEAD is now at e18ddad version: update to 20200717
>> Entering 'roms/edk2'
>> HEAD is now at 06dc822d04 Revert ".pytool/EccCheck: Disable Ecc error
>> code 10014 for open CI"
>> Entering 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
>> HEAD is now at b64af41 Fix typo in function
>> 'softfloat_propagateNaNF128M' for RISC-V.
>> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
>> HEAD is now at 666c328 Make types of variable match (#796)
>> Entering 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
>> HEAD is now at ca7cb33 move to git
>> Entering
>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
>> HEAD is now at 5f60d6f Merge pull request #7 from kloetzl/master
>> Entering 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>> Entering 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
>> HEAD is now at 63be8a9 unichr was removed in Python 3 because all str
>> are Unicode (#877)
>> Entering 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
>> HEAD is now at b2c1da6 add ONIG_OPTION_CALLBACK_EACH_MATCH test
>> Entering 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
>> HEAD is now at 160dffe Don't use non-literal format strings
>> Entering 'roms/ipxe'
>> HEAD is now at 4bd064de [build] Fix building on older versions of gcc
>> Entering 'roms/openbios'
>> HEAD is now at 7f28286 PPC: mark first 4 pages of physical and virtual
>> memory as unavailable
>> Entering 'roms/opensbi'
>> HEAD is now at a98258d include: Bump-up version to 0.8
>> Entering 'roms/qboot'
>> HEAD is now at a5300c4 qboot: Disable PIE for ELF binary build step
>> Entering 'roms/qemu-palcode'
>> HEAD is now at bf0e136 Report machine checks to the kernel
>> Entering 'roms/seabios'
>> HEAD is now at 155821a docs: Note v1.14.0 release
>> Entering 'roms/seabios-hppa'
>> HEAD is now at 73b740f7 parisc: Set text planes and used_bits in STI fields
>> Entering 'roms/sgabios'
>> HEAD is now at cbaee52 SGABIOS: fix wrong video attrs for int 10h, ah==13h
>> Entering 'roms/skiboot'
>> HEAD is now at 3a6fdede skiboot v6.4 release notes
>> Entering 'roms/u-boot'
>> HEAD is now at d3689267f9 Prepare v2019.01
>> Entering 'roms/u-boot-sam460ex'
>> HEAD is now at 60b3916 Add README to clarify relation to U-Boot and
>> ACube's version
>> Entering 'roms/vbootrom'
>> HEAD is now at 0c37a43 Merge pull request #1 from google/disable-build-id
>> Entering 'slirp'
>> HEAD is now at 8f43a99 Merge branch 'stable-4.2' into 'stable-4.2'
>> Entering 'tests/fp/berkeley-softfloat-3'
>> HEAD is now at b64af41 Fix typo in function
>> 'softfloat_propagateNaNF128M' for RISC-V.
>> Entering 'tests/fp/berkeley-testfloat-3'
>> HEAD is now at 5a59dce fail: constify fail_programName
>> Entering 'ui/keycodemapdb'
>> HEAD is now at 6119e6e Fix scan codes for Korean keys
>> fatal: Needed a single revision
>> Unable to find current revision in submodule path
>> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>> Failed to recurse into submodule path 'roms/edk2'
>> ERROR: Job failed: exit status 1
>>
> 
> Yes, I've also found similar issues during my jobs.

The problem with GIT_SUBMODULE_STRATEGY is it tries to
clone submodules from submodules (here EDK2) which QEMU
doesn't use:

Submodule 'boringssl' (https://boringssl.googlesource.com/boringssl)
registered for path
'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/boringssl'
Submodule 'krb5' (https://github.com/krb5/krb5) registered for path
'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/krb5'
Submodule 'pyca.cryptography' (https://github.com/pyca/cryptography.git)
registered for path
'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/pyca-cryptography'
Cloning into
'/var/lib/gitlab-runner/builds/shWMsY1a/0/philmd/qemu/roms/edk2/CryptoPkg/Library/OpensslLib/openssl/boringssl'...
fatal: unable to access 'https://boringssl.googlesource.com/boringssl/':
Failed to connect to boringssl.googlesource.com port 443: Connection
timed out
fatal: clone of 'https://boringssl.googlesource.com/boringssl' into
submodule path
'/var/lib/gitlab-runner/builds/shWMsY1a/0/philmd/qemu/roms/edk2/CryptoPkg/Library/OpensslLib/openssl/boringssl'
failed
Failed to clone 'boringssl'. Retry scheduled

I don't think we want to clone the unused boringssl on
QEMU namespace.

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Philippe Mathieu-Daudé]

On 2/23/21 6:24 PM, Philippe Mathieu-Daudé wrote:
> On 2/23/21 5:47 PM, Cleber Rosa wrote:
>> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
>>> On 2/23/21 12:25 PM, Thomas Huth wrote:
>>>> On 19/02/2021 22.58, Cleber Rosa wrote:
>>>>> As described in the included documentation, the "custom runner" jobs
>>>>> extend the GitLab CI jobs already in place.  One of their primary
>>>>> goals of catching and preventing regressions on a wider number of host
>>>>> systems than the ones provided by GitLab's shared runners.
>>>>>
>>>>> This sets the stage in which other community members can add their own
>>>>> machine configuration documentation/scripts, and accompanying job
>>>>> definitions.  As a general rule, those newly added contributed jobs
>>>>> should run as "non-gating", until their reliability is verified (AKA
>>>>> "allow_failure: true").
>>>>>
>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>>>>>   .gitlab-ci.yml                  |  1 +
>>>>>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>>>>>   docs/devel/index.rst            |  1 +
>>>>>   4 files changed, 44 insertions(+)
>>>>>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>>>>>   create mode 100644 docs/devel/ci.rst
>>>>>
>>>>> diff --git a/.gitlab-ci.d/custom-runners.yml
>>>>> b/.gitlab-ci.d/custom-runners.yml
>>>>> new file mode 100644
>>>>> index 0000000000..3004da2bda
>>>>> --- /dev/null
>>>>> +++ b/.gitlab-ci.d/custom-runners.yml
>>>>> @@ -0,0 +1,14 @@
>>>>> +# The CI jobs defined here require GitLab runners installed and
>>>>> +# registered on machines that match their operating system names,
>>>>> +# versions and architectures.  This is in contrast to the other CI
>>>>> +# jobs that are intended to run on GitLab's "shared" runners.
>>>>> +
>>>>> +# Different than the default approach on "shared" runners, based on
>>>>> +# containers, the custom runners have no such *requirement*, as those
>>>>> +# jobs should be capable of running on operating systems with no
>>>>> +# compatible container implementation, or no support from
>>>>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
>>>>> +# reusing the GIT repository, let's enable the recursive submodule
>>>>> +# strategy.
>>>>> +variables:
>>>>> +  GIT_SUBMODULE_STRATEGY: recursive
>>>>
>>>> Is it really necessary? I thought our configure script would take care
>>>> of the submodules?
>>>
>>
>> I've done a lot of testing on bare metal systems, and the problems
>> that come from reusing the same system and failed cleanups can be very
>> frustrating.  It's unfortunate that we need this, but it was the
>> simplest and most reliable solution I found.  :/
>>
>> Having said that, I noticed after I posted this series that this is
>> affecting all other jobs.  We don't need it that in the jobs based
>> on containers (for obvious reasons), so I see two options:
>>
>> 1) have it enabled on all jobs for consistency
>>
>> 2) have it enabled only on jobs that will reuse the repo
>>
>>> Well, if there is a failure during the first clone (I got one network
>>> timeout in the middle) 
> 
> [This network failure is pasted at the end]
> 
>>> then next time it doesn't work:
>>>
>>> Updating/initializing submodules recursively...
>>> Synchronizing submodule url for 'capstone'
>>> Synchronizing submodule url for 'dtc'
>>> Synchronizing submodule url for 'meson'
>>> Synchronizing submodule url for 'roms/QemuMacDrivers'
>>> Synchronizing submodule url for 'roms/SLOF'
>>> Synchronizing submodule url for 'roms/edk2'
>>> Synchronizing submodule url for
>>> 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
>>> Synchronizing submodule url for
>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
>>> Synchronizing submodule url for
>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
>>> Synchronizing submodule url for
>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
>>> Synchronizing submodule url for
>>> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>>> Synchronizing submodule url for
>>> 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
>>> Synchronizing submodule url for
>>> 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
>>> Synchronizing submodule url for
>>> 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'

So far, beside the repository useful for QEMU, I cloned:

- boringssl
- krb5
- pyca-cryptography
- esaxx
- libdivsufsort
- oniguruma
- openssl
- brotli
- cmocka

But reach the runner time limit of 2h.

The directory reports 3GB of source code.

I don't think the series has been tested enough before posting,
I'm stopping here my experiments.

Regards,

Phil.

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

On Tue, Feb 23, 2021 at 03:01:50PM +0000, Alex Bennée wrote:
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
> > Cleber Rosa <crosa@redhat.com> writes:
> >
> >> To run basic jobs on custom runners, the environment needs to be
> >> properly set up.  The most common requirement is having the right
> >> packages installed.
> >>
> <snip>
> >
> > So I got somewhat there with a direct command line invocation:
> >
> >   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
> >
> > although for some reason a single host -i fails...
> >
> >> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> >> new file mode 100644
> >> index 0000000000..0197e0a48b
> >> --- /dev/null
> >> +++ b/scripts/ci/setup/build-environment.yml
> >> @@ -0,0 +1,76 @@
> >> +---
> >> +- name: Installation of basic packages to build QEMU
> >> +  hosts: all
> >> +  tasks:
> >> +    - name: Update apt cache
> >> +      apt:
> >> +        update_cache: yes
> >> +      when:
> >> +        - ansible_facts['distribution'] == 'Ubuntu'
> >
> > So are we limiting to Ubuntu here rather than say a Debian base?
> 
> Also I'm getting:
> 
>   TASK [Update apt cache] *****************************************************************************************************************************************************
>   fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}
> 
> which is odd given that machine is definitely an Ubuntu one.
>

It's defintely odd.  This is what I get on a fresh machine:

   TASK [Update apt cache] *************************************************************************************************************************
   [WARNING]: Updating cache and auto-installing missing dependency: python3-apt
   ok: [localhost]

Could you please let me know the output of:

   $ ansible -m setup -u $YOUR_USERNAME -i $HOSTNAME, all | grep ansible_distribution

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Daniel P. Berrangé]

On Tue, Feb 23, 2021 at 11:47:18AM -0500, Cleber Rosa wrote:
> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
> > On 2/23/21 12:25 PM, Thomas Huth wrote:
> > > On 19/02/2021 22.58, Cleber Rosa wrote:
> > >> As described in the included documentation, the "custom runner" jobs
> > >> extend the GitLab CI jobs already in place.  One of their primary
> > >> goals of catching and preventing regressions on a wider number of host
> > >> systems than the ones provided by GitLab's shared runners.
> > >>
> > >> This sets the stage in which other community members can add their own
> > >> machine configuration documentation/scripts, and accompanying job
> > >> definitions.  As a general rule, those newly added contributed jobs
> > >> should run as "non-gating", until their reliability is verified (AKA
> > >> "allow_failure: true").
> > >>
> > >> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > >> ---
> > >>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
> > >>   .gitlab-ci.yml                  |  1 +
> > >>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
> > >>   docs/devel/index.rst            |  1 +
> > >>   4 files changed, 44 insertions(+)
> > >>   create mode 100644 .gitlab-ci.d/custom-runners.yml
> > >>   create mode 100644 docs/devel/ci.rst
> > >>
> > >> diff --git a/.gitlab-ci.d/custom-runners.yml
> > >> b/.gitlab-ci.d/custom-runners.yml
> > >> new file mode 100644
> > >> index 0000000000..3004da2bda
> > >> --- /dev/null
> > >> +++ b/.gitlab-ci.d/custom-runners.yml
> > >> @@ -0,0 +1,14 @@
> > >> +# The CI jobs defined here require GitLab runners installed and
> > >> +# registered on machines that match their operating system names,
> > >> +# versions and architectures.  This is in contrast to the other CI
> > >> +# jobs that are intended to run on GitLab's "shared" runners.
> > >> +
> > >> +# Different than the default approach on "shared" runners, based on
> > >> +# containers, the custom runners have no such *requirement*, as those
> > >> +# jobs should be capable of running on operating systems with no
> > >> +# compatible container implementation, or no support from
> > >> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> > >> +# reusing the GIT repository, let's enable the recursive submodule
> > >> +# strategy.
> > >> +variables:
> > >> +  GIT_SUBMODULE_STRATEGY: recursive
> > > 
> > > Is it really necessary? I thought our configure script would take care
> > > of the submodules?
> >
> 
> I've done a lot of testing on bare metal systems, and the problems
> that come from reusing the same system and failed cleanups can be very
> frustrating.  It's unfortunate that we need this, but it was the
> simplest and most reliable solution I found.  :/

Hmmm, this makes it sound like the job is not being run in a
fresh pristine checkout. IMHO we need to guarantee that in
general, at which point submodules should "just work", unless
the running is blocking network access ?



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 6468 bytes --]

On Tue, Feb 23, 2021 at 06:34:07PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/23/21 6:24 PM, Philippe Mathieu-Daudé wrote:
> > On 2/23/21 5:47 PM, Cleber Rosa wrote:
> >> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
> >>> On 2/23/21 12:25 PM, Thomas Huth wrote:
> >>>> On 19/02/2021 22.58, Cleber Rosa wrote:
> >>>>> As described in the included documentation, the "custom runner" jobs
> >>>>> extend the GitLab CI jobs already in place.  One of their primary
> >>>>> goals of catching and preventing regressions on a wider number of host
> >>>>> systems than the ones provided by GitLab's shared runners.
> >>>>>
> >>>>> This sets the stage in which other community members can add their own
> >>>>> machine configuration documentation/scripts, and accompanying job
> >>>>> definitions.  As a general rule, those newly added contributed jobs
> >>>>> should run as "non-gating", until their reliability is verified (AKA
> >>>>> "allow_failure: true").
> >>>>>
> >>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >>>>> ---
> >>>>>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
> >>>>>   .gitlab-ci.yml                  |  1 +
> >>>>>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
> >>>>>   docs/devel/index.rst            |  1 +
> >>>>>   4 files changed, 44 insertions(+)
> >>>>>   create mode 100644 .gitlab-ci.d/custom-runners.yml
> >>>>>   create mode 100644 docs/devel/ci.rst
> >>>>>
> >>>>> diff --git a/.gitlab-ci.d/custom-runners.yml
> >>>>> b/.gitlab-ci.d/custom-runners.yml
> >>>>> new file mode 100644
> >>>>> index 0000000000..3004da2bda
> >>>>> --- /dev/null
> >>>>> +++ b/.gitlab-ci.d/custom-runners.yml
> >>>>> @@ -0,0 +1,14 @@
> >>>>> +# The CI jobs defined here require GitLab runners installed and
> >>>>> +# registered on machines that match their operating system names,
> >>>>> +# versions and architectures.  This is in contrast to the other CI
> >>>>> +# jobs that are intended to run on GitLab's "shared" runners.
> >>>>> +
> >>>>> +# Different than the default approach on "shared" runners, based on
> >>>>> +# containers, the custom runners have no such *requirement*, as those
> >>>>> +# jobs should be capable of running on operating systems with no
> >>>>> +# compatible container implementation, or no support from
> >>>>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
> >>>>> +# reusing the GIT repository, let's enable the recursive submodule
> >>>>> +# strategy.
> >>>>> +variables:
> >>>>> +  GIT_SUBMODULE_STRATEGY: recursive
> >>>>
> >>>> Is it really necessary? I thought our configure script would take care
> >>>> of the submodules?
> >>>
> >>
> >> I've done a lot of testing on bare metal systems, and the problems
> >> that come from reusing the same system and failed cleanups can be very
> >> frustrating.  It's unfortunate that we need this, but it was the
> >> simplest and most reliable solution I found.  :/
> >>
> >> Having said that, I noticed after I posted this series that this is
> >> affecting all other jobs.  We don't need it that in the jobs based
> >> on containers (for obvious reasons), so I see two options:
> >>
> >> 1) have it enabled on all jobs for consistency
> >>
> >> 2) have it enabled only on jobs that will reuse the repo
> >>
> >>> Well, if there is a failure during the first clone (I got one network
> >>> timeout in the middle) 
> > 
> > [This network failure is pasted at the end]
> > 
> >>> then next time it doesn't work:
> >>>
> >>> Updating/initializing submodules recursively...
> >>> Synchronizing submodule url for 'capstone'
> >>> Synchronizing submodule url for 'dtc'
> >>> Synchronizing submodule url for 'meson'
> >>> Synchronizing submodule url for 'roms/QemuMacDrivers'
> >>> Synchronizing submodule url for 'roms/SLOF'
> >>> Synchronizing submodule url for 'roms/edk2'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
> >>> Synchronizing submodule url for
> >>> 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
> 
> So far, beside the repository useful for QEMU, I cloned:
> 
> - boringssl
> - krb5
> - pyca-cryptography
> - esaxx
> - libdivsufsort
> - oniguruma
> - openssl
> - brotli
> - cmocka
>

Hi Phil,

I'm not following what you meant by "I cloned"... Are you experimenting
with this on a machine of your own and manually cloning the submodules?

> But reach the runner time limit of 2h.
>
> The directory reports 3GB of source code.
> 
> I don't think the series has been tested enough before posting,

Please take into consideration that this series, although simple in
content, touches and interacts with a lot of moving pieces, and
possibly with personal systems that I did not have, or will have,
access to.  As far as public testing proof goes, you can see a
pipeline here with this version of this series here:

   https://gitlab.com/cleber.gnu/qemu/-/pipelines/258982039/builds

As I said elsewhere, I only noticed the recursive submodule being
applied to the existing jobs after I submitted the series.  Mea culpa.
But:

 * none of the jobs took noticeably longer than the previous baseline
 * there was one *container build failure* (safe to say it's not
   related)
 * all other jobs passed successfully

And, along with the previous versions, this series were tested on all
the previously included architectures and operating systems.  It's
unfortunate that because of your experience at this time (my
apologies), you don't realize the amount of testing done so far.

> I'm stopping here my experiments.
>
> Regards,
> 
> Phil.
> 

I honestly appreciate your help here up to this point.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
>> 
>> Cleber Rosa <crosa@redhat.com> writes:
>> 
>> > To run basic jobs on custom runners, the environment needs to be
>> > properly set up.  The most common requirement is having the right
>> > packages installed.
>> >
>> > The playbook introduced here covers the QEMU's project s390x and
>> > aarch64 machines.  At the time this is being proposed, those machines
>> > have already had this playbook applied to them.
>> >
>> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> > ---
>> >  docs/devel/ci.rst                      | 30 ++++++++++
>> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
>> >  scripts/ci/setup/inventory             |  1 +
>> >  3 files changed, 107 insertions(+)
>> >  create mode 100644 scripts/ci/setup/build-environment.yml
>> >  create mode 100644 scripts/ci/setup/inventory
>> >
>> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>> > index 585b7bf4b8..a556558435 100644
>> > --- a/docs/devel/ci.rst
>> > +++ b/docs/devel/ci.rst
>> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>> >  The GitLab CI jobs definition for the custom runners are located under::
>> >  
>> >    .gitlab-ci.d/custom-runners.yml
>> > +
>> > +Machine Setup Howto
>> > +-------------------
>> > +
>> > +For all Linux based systems, the setup can be mostly automated by the
>> > +execution of two Ansible playbooks.  Start by adding your machines to
>> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
>> > +
>> > +  fully.qualified.domain
>> > +  other.machine.hostname
>> 
>> Is this really needed? Can't the host list be passed in the command
>> line? I find it off to imagine users wanting to configure whole fleets
>> of runners.
>>
>
> No, it's not needed.
>
> But, in my experience, it's the most common way people use
> ansible-playbook.  As with all most tools QEMU relies on, that are
> many different ways of using them.  IMO documenting more than one way
> to perform the same task makes the documentation unclear.
>
>> > +
>> > +You may need to set some variables in the inventory file itself.  One
>> > +very common need is to tell Ansible to use a Python 3 interpreter on
>> > +those hosts.  This would look like::
>> > +
>> > +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
>> > +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
>> > +
>> > +Build environment
>> > +~~~~~~~~~~~~~~~~~
>> > +
>> > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
>> > +set up machines with the environment needed to perform builds and run
>> > +QEMU tests.  It covers a number of different Linux distributions and
>> > +FreeBSD.
>> > +
>> > +To run the playbook, execute::
>> > +
>> > +  cd scripts/ci/setup
>> > +  ansible-playbook -i inventory build-environment.yml
>> 
>> So I got somewhat there with a direct command line invocation:
>> 
>>   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
>>
>
> Yes, and the "-e" is another example of the multiple ways to achieve
> the same task.
>
>> although for some reason a single host -i fails...
>> 
>> > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
>
> It requires a comma separated list, even if it's a list with a single
> item:
>
>    https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#cmdoption-ansible-playbook-i
>
>> > new file mode 100644
>> > index 0000000000..0197e0a48b
>> > --- /dev/null
>> > +++ b/scripts/ci/setup/build-environment.yml
>> > @@ -0,0 +1,76 @@
>> > +---
>> > +- name: Installation of basic packages to build QEMU
>> > +  hosts: all
>> > +  tasks:
>> > +    - name: Update apt cache
>> > +      apt:
>> > +        update_cache: yes
>> > +      when:
>> > +        - ansible_facts['distribution'] == 'Ubuntu'
>> 
>> So are we limiting to Ubuntu here rather than say a Debian base?
>>
>
> You have a point, because this would certainly work and be applicable
> to Debian systems too.  But, this is a new addition on v5, and I'm
> limiting this patch to the machines that are available/connected right
> now to the QEMU project on GitLab.
>
> I can change that to "distribution_family == Debian" if you think
> it's a good idea.  But IMO it'd make more sense for a patch
> introducing the package list for Debian systems to change that.
>
>> > +
>> > +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
>> > +      package:
>> > +        name:
>> > +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
>> > +          - ccache
>> > +          - clang
>> > +          - gcc
>> > +          - gettext
>> > +          - git
>> > +          - glusterfs-common
>> > +          - libaio-dev
>> > +          - libattr1-dev
>> > +          - libbrlapi-dev
>> > +          - libbz2-dev
>> > +          - libcacard-dev
>> > +          - libcap-ng-dev
>> > +          - libcurl4-gnutls-dev
>> > +          - libdrm-dev
>> > +          - libepoxy-dev
>> > +          - libfdt-dev
>> > +          - libgbm-dev
>> > +          - libgtk-3-dev
>> > +          - libibverbs-dev
>> > +          - libiscsi-dev
>> > +          - libjemalloc-dev
>> > +          - libjpeg-turbo8-dev
>> > +          - liblzo2-dev
>> > +          - libncurses5-dev
>> > +          - libncursesw5-dev
>> > +          - libnfs-dev
>> > +          - libnss3-dev
>> > +          - libnuma-dev
>> > +          - libpixman-1-dev
>> > +          - librados-dev
>> > +          - librbd-dev
>> > +          - librdmacm-dev
>> > +          - libsasl2-dev
>> > +          - libsdl2-dev
>> > +          - libseccomp-dev
>> > +          - libsnappy-dev
>> > +          - libspice-protocol-dev
>> > +          - libssh-dev
>> > +          - libusb-1.0-0-dev
>> > +          - libusbredirhost-dev
>> > +          - libvdeplug-dev
>> > +          - libvte-2.91-dev
>> > +          - libzstd-dev
>> > +          - make
>> > +          - ninja-build
>> > +          - python3-yaml
>> > +          - python3-sphinx
>> > +          - sparse
>> > +          - xfslibs-dev
>> > +        state: present
>> > +      when:
>> > +        - ansible_facts['distribution'] == 'Ubuntu'
>> > +
>> > +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
>> > +      package:
>> > +        name:
>> > +          - libspice-server-dev
>> > +          - libxen-dev
>> > +        state: present
>> > +      when:
>> > +        - ansible_facts['distribution'] == 'Ubuntu'
>> > +        - ansible_facts['architecture'] != 's390x'
>> > diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
>> > new file mode 100644
>> > index 0000000000..2fbb50c4a8
>> > --- /dev/null
>> > +++ b/scripts/ci/setup/inventory
>> > @@ -0,0 +1 @@
>> > +localhost
>> 
>> I'm not sure we should have a default here because it will inevitably
>> cause someone to do something to their machine when trying to setup a
>> runner.
>>
>
> Fair enough.  Then I see two options:
>
> 1) follow the vars.yml.template example and only ship a
>    inventory.template file

I'd go with the template approach, that way someones local hacks can at
least live in their source tree without being overly bothered by
checkouts and updates.

>
> 2) use a placeholder with an impossible hostname such as
>    "my-qemu-runner.example.org" or "your-host-name-here"
>
>> -- 
>> Alex Bennée
>> 
>
> Let me know what you think is more reasonable, and thanks for the
> review!
>
> Regards,
> - Cleber.


-- 
Alex Bennée

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On Tue, Feb 23, 2021 at 03:17:24PM +0000, Alex Bennée wrote:
>> 
>> Erik Skultety <eskultet@redhat.com> writes:
>> 
>> > On Tue, Feb 23, 2021 at 02:01:53PM +0000, Alex Bennée wrote:
>> >> 
>> >> Cleber Rosa <crosa@redhat.com> writes:
>> >> 
>> >> > To run basic jobs on custom runners, the environment needs to be
>> >> > properly set up.  The most common requirement is having the right
>> >> > packages installed.
>> >> >
>> >> > The playbook introduced here covers the QEMU's project s390x and
>> >> > aarch64 machines.  At the time this is being proposed, those machines
>> >> > have already had this playbook applied to them.
>> >> >
>> >> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> >> > ---
>> >> >  docs/devel/ci.rst                      | 30 ++++++++++
>> >> >  scripts/ci/setup/build-environment.yml | 76 ++++++++++++++++++++++++++
>> >> >  scripts/ci/setup/inventory             |  1 +
>> >> >  3 files changed, 107 insertions(+)
>> >> >  create mode 100644 scripts/ci/setup/build-environment.yml
>> >> >  create mode 100644 scripts/ci/setup/inventory
>> >> >
>> >> > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
>> >> > index 585b7bf4b8..a556558435 100644
>> >> > --- a/docs/devel/ci.rst
>> >> > +++ b/docs/devel/ci.rst
>> >> > @@ -26,3 +26,33 @@ gitlab-runner, is called a "custom runner".
>> >> >  The GitLab CI jobs definition for the custom runners are located under::
>> >> >  
>> >> >    .gitlab-ci.d/custom-runners.yml
>> >> > +
>> >> > +Machine Setup Howto
>> >> > +-------------------
>> >> > +
>> >> > +For all Linux based systems, the setup can be mostly automated by the
>> >> > +execution of two Ansible playbooks.  Start by adding your machines to
>> >> > +the ``inventory`` file under ``scripts/ci/setup``, such as this::
>> >> > +
>> >> > +  fully.qualified.domain
>> >> > +  other.machine.hostname
>> >> 
>> >> Is this really needed? Can't the host list be passed in the command
>> >> line? I find it off to imagine users wanting to configure whole fleets
>> >> of runners.
>> >
>> > Why not support both, since the playbook execution is not wrapped by anything,
>> > giving the option of using either and inventory or direct cmdline invocation
>> > seems like the proper way to do it.
>> 
>> Sure - and I dare say people used to managing fleets of servers will
>> want to do it properly but in the first instance lets provide the simple
>> command line option so a user can get up and running without also
>> ensuring files are in the correct format.
>>
>
> Like I said before, I'm strongly in favor of a more straightforward
> documentation, instead of documenting multiple ways to perform the
> same task.  I clearly believe that writing the inventory file (which
> will later be used for the second gitlab-runner playbook) is the best
> choice here.
>
> Do you think the command line approach is clearer?  Should we switch?

I think the command line is $LESS_STEPS for a user to follow but I'm
happy to defer to the inventory approach if it's more idiomatic. I'd
rather avoid uses having their pristine source trees being polluted with
local customisations they have to keep stashing or loosing.

>
> Regards,
> Cleber.


-- 
Alex Bennée

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 3664 bytes --]

On Tue, Feb 23, 2021 at 04:17:23PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/19/21 10:58 PM, Cleber Rosa wrote:
> > The QEMU project has two machines (aarch64 and s390x) that can be used
> > for jobs that do build and run tests.  This introduces those jobs,
> > which are a mapping of custom scripts used for the same purpose.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/custom-runners.yml | 204 ++++++++++++++++++++++++++++++++
> >  1 file changed, 204 insertions(+)
> > 
> > diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> > index 3004da2bda..a9166c82a2 100644
> > --- a/.gitlab-ci.d/custom-runners.yml
> > +++ b/.gitlab-ci.d/custom-runners.yml
> > @@ -12,3 +12,207 @@
> >  # strategy.
> >  variables:
> >    GIT_SUBMODULE_STRATEGY: recursive
> > +
> > +# All ubuntu-18.04 jobs should run successfully in an environment
> > +# setup by the scripts/ci/setup/build-environment.yml task
> > +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> > +ubuntu-18.04-s390x-all-linux-static:
> > + allow_failure: true
> > + needs: []
> > + stage: build
> > + tags:
> > + - ubuntu_18.04
> > + - s390x
> 
> Where is this tag list filled upon registration?
>

The documentation on this series (previous patch) describes how one
should go about settings the tags.  Pasting it here for easier context:

---

Following the registration, it's necessary to configure the runner tags,
and optionally other configurations on the GitLab UI.  Navigate to:

 * Settings (the gears like icon), then
 * CI/CD, then
 * Runners, and click on the "Expand" button, then
 * "Runners activated for this project", then
 * Click on the "Edit" icon (next to the "Lock" Icon)

Under tags, add values matching the jobs a runner should run.  For a
Ubuntu 20.04 aarch64 system, the tags should be set as::

  ubuntu_20.04,aarch64

Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
would contain::

  ubuntu-20.04-aarch64-all:
   tags:
   - ubuntu_20.04
   - aarch64

---

Does that answer your question?

> > + rules:
> > + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> > + script:
> > + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
> > + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> > + - mkdir build
> > + - cd build
> > + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
> > + - make --output-sync -j`nproc`
> > + - make --output-sync -j`nproc` check V=1
> > + - make --output-sync -j`nproc` check-tcg V=1
> 
> Also this break the rest of the tests...
> 
> The first containers job (amd64-alpine-container) got
> added to the custom runner and failed (because docker-dind
> isn't there?):
>

The documentation explains that, saying that it's recommended to uncheck
the "Run untagged jobs" check box.

> $ export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> $ export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> $ apk add python3
> bash: line 110: apk: command not found
> Running after_script 00:01
> Running after script...
> $ docker logout
> Removing login credentials for https://index.docker.io/v1/
> ERROR: Job failed: exit status 1
> 
> Do we need to restrict the other jobs to the Gitlab public
> (x86) runners? Maybe as:
>

You just need to take care of the runners you add.  All the other jobs
are assumed to be running on the shared runners.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On Tue, Feb 23, 2021 at 03:01:50PM +0000, Alex Bennée wrote:
>> 
>> Alex Bennée <alex.bennee@linaro.org> writes:
>> 
>> > Cleber Rosa <crosa@redhat.com> writes:
>> >
>> >> To run basic jobs on custom runners, the environment needs to be
>> >> properly set up.  The most common requirement is having the right
>> >> packages installed.
>> >>
>> <snip>
>> >
>> > So I got somewhat there with a direct command line invocation:
>> >
>> >   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
>> >
>> > although for some reason a single host -i fails...
>> >
>> >> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
>> >> new file mode 100644
>> >> index 0000000000..0197e0a48b
>> >> --- /dev/null
>> >> +++ b/scripts/ci/setup/build-environment.yml
>> >> @@ -0,0 +1,76 @@
>> >> +---
>> >> +- name: Installation of basic packages to build QEMU
>> >> +  hosts: all
>> >> +  tasks:
>> >> +    - name: Update apt cache
>> >> +      apt:
>> >> +        update_cache: yes
>> >> +      when:
>> >> +        - ansible_facts['distribution'] == 'Ubuntu'
>> >
>> > So are we limiting to Ubuntu here rather than say a Debian base?
>> 
>> Also I'm getting:
>> 
>>   TASK [Update apt cache] *****************************************************************************************************************************************************
>>   fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}
>> 
>> which is odd given that machine is definitely an Ubuntu one.
>>
>
> It's defintely odd.  This is what I get on a fresh machine:
>
>    TASK [Update apt cache] *************************************************************************************************************************
>    [WARNING]: Updating cache and auto-installing missing dependency: python3-apt
>    ok: [localhost]
>
> Could you please let me know the output of:
>
>    $ ansible -m setup -u $YOUR_USERNAME -i $HOSTNAME, all | grep
>    ansible_distribution

The key doesn't exist:

hackbox-ubuntu-2004 | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "192.168.122.170"
        ],
        "ansible_all_ipv6_addresses": [
            "fe80::5054:ff:fe54:7cfe"
        ],
        "ansible_apparmor": {
            "status": "enabled"
        },
        "ansible_architecture": "x86_64",
        "ansible_bios_date": "04/01/2014",
        "ansible_bios_version": "1.10.2-1ubuntu1",
        "ansible_cmdline": {
            "BOOT_IMAGE": "/vmlinuz-5.4.0-65-generic",
            "maybe-ubiquity": true,
            "ro": true,
            "root": "/dev/mapper/ubuntu--vg-ubuntu--lv"
        },
        "ansible_date_time": {
            "date": "2021-02-23",
            "day": "23",
            "epoch": "1614104601",
            "hour": "18",
            "iso8601": "2021-02-23T18:23:21Z",
            "iso8601_basic": "20210223T182321857461",
            "iso8601_basic_short": "20210223T182321",
            "iso8601_micro": "2021-02-23T18:23:21.857529Z",
            "minute": "23",
            "month": "02",
            "second": "21",
            "time": "18:23:21",
            "tz": "UTC",
            "tz_offset": "+0000",
            "weekday": "Tuesday",
            "weekday_number": "2",
            "weeknumber": "08",
            "year": "2021"
        },
        "ansible_default_ipv4": {
            "address": "192.168.122.170",
            "alias": "enp1s0",
            "broadcast": "192.168.122.255",
            "gateway": "192.168.122.1",
            "interface": "enp1s0",
            "macaddress": "52:54:00:54:7c:fe",
            "mtu": 1500,
            "netmask": "255.255.255.0",
            "network": "192.168.122.0",
            "type": "ether"
        },
        "ansible_default_ipv6": {},
        "ansible_device_links": {
            "ids": {
                "dm-0": [
                    "dm-name-ubuntu--vg-ubuntu--lv",
                    "dm-uuid-LVM-filR1BfuX6Mpp9J7CP9cbVsTT2ICh7Apc9qZsFohnsqycocacS0Sm6HAhjTBEAkq"
                ],
                "sda": [
                    "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1"
                ],
                "sda1": [
                    "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part1"
                ],
                "sda2": [
                    "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part2"
                ],
                "sda3": [
                    "lvm-pv-uuid-agDdyQ-V5gQ-aaov-933l-SFAL-0rmD-SlOkYy",
                    "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part3"
                ],
                "sr0": [
                    "ata-QEMU_DVD-ROM_QM00001"
                ]
            },
            "labels": {},
            "masters": {
                "sda3": [
                    "dm-0"
                ]
            },
            "uuids": {
                "dm-0": [
                    "291656fe-bd87-484c-b4a9-4453471a17e8"
                ],
                "sda2": [
                    "45018994-9625-44ad-877a-3980bcf943a3"
                ]
            }
        },
        "ansible_devices": {
            "dm-0": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [
                        "dm-name-ubuntu--vg-ubuntu--lv",
                        "dm-uuid-LVM-filR1BfuX6Mpp9J7CP9cbVsTT2ICh7Apc9qZsFohnsqycocacS0Sm6HAhjTBEAkq"
                    ],
                    "labels": [],
                    "masters": [],
                    "uuids": [
                        "291656fe-bd87-484c-b4a9-4453471a17e8"
                    ]
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "",
                "sectors": "41943040",
                "sectorsize": "512",
                "size": "20.00 GB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop0": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "143120",
                "sectorsize": "512",
                "size": "69.88 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop1": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "0",
                "sectorsize": "512",
                "size": "0.00 Bytes",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop2": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "0",
                "sectorsize": "512",
                "size": "0.00 Bytes",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop3": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "113424",
                "sectorsize": "512",
                "size": "55.38 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop4": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "63672",
                "sectorsize": "512",
                "size": "31.09 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop5": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "142872",
                "sectorsize": "512",
                "size": "69.76 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop6": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "113592",
                "sectorsize": "512",
                "size": "55.46 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "loop7": {
                "holders": [],
                "host": "",
                "links": {
                    "ids": [],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": null,
                "partitions": {},
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "63664",
                "sectorsize": "512",
                "size": "31.09 MB",
                "support_discard": "4096",
                "vendor": null,
                "virtual": 1
            },
            "sda": {
                "holders": [],
                "host": "SCSI storage controller: Broadcom / LSI 53c895a",
                "links": {
                    "ids": [
                        "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1"
                    ],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": "QEMU HARDDISK",
                "partitions": {
                    "sda1": {
                        "holders": [],
                        "links": {
                            "ids": [
                                "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part1"
                            ],
                            "labels": [],
                            "masters": [],
                            "uuids": []
                        },
                        "sectors": "2048",
                        "sectorsize": 512,
                        "size": "1.00 MB",
                        "start": "2048",
                        "uuid": null
                    },
                    "sda2": {
                        "holders": [],
                        "links": {
                            "ids": [
                                "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part2"
                            ],
                            "labels": [],
                            "masters": [],
                            "uuids": [
                                "45018994-9625-44ad-877a-3980bcf943a3"
                            ]
                        },
                        "sectors": "2097152",
                        "sectorsize": 512,
                        "size": "1.00 GB",
                        "start": "4096",
                        "uuid": "45018994-9625-44ad-877a-3980bcf943a3"
                    },
                    "sda3": {
                        "holders": [
                            "ubuntu--vg-ubuntu--lv"
                        ],
                        "links": {
                            "ids": [
                                "lvm-pv-uuid-agDdyQ-V5gQ-aaov-933l-SFAL-0rmD-SlOkYy",
                                "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part3"
                            ],
                            "labels": [],
                            "masters": [
                                "dm-0"
                            ],
                            "uuids": []
                        },
                        "sectors": "81782784",
                        "sectorsize": 512,
                        "size": "39.00 GB",
                        "start": "2101248",
                        "uuid": null
                    }
                },
                "removable": "0",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "83886080",
                "sectorsize": "512",
                "size": "40.00 GB",
                "support_discard": "4096",
                "vendor": "QEMU",
                "virtual": 1
            },
            "sr0": {
                "holders": [],
                "host": "SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)",
                "links": {
                    "ids": [
                        "ata-QEMU_DVD-ROM_QM00001"
                    ],
                    "labels": [],
                    "masters": [],
                    "uuids": []
                },
                "model": "QEMU DVD-ROM",
                "partitions": {},
                "removable": "1",
                "rotational": "1",
                "sas_address": null,
                "sas_device_handle": null,
                "scheduler_mode": "mq-deadline",
                "sectors": "2097151",
                "sectorsize": "512",
                "size": "1024.00 MB",
                "support_discard": "0",
                "vendor": "QEMU",
                "virtual": 1
            }
        },
        "ansible_dns": {
            "nameservers": [
                "127.0.0.53"
            ],
            "options": {
                "edns0": true,
                "trust-ad": true
            }
        },
        "ansible_domain": "",
        "ansible_effective_group_id": 0,
        "ansible_effective_user_id": 0,
        "ansible_enp1s0": {
            "active": true,
            "device": "enp1s0",
            "features": {
                "esp_hw_offload": "off [fixed]",
                "esp_tx_csum_hw_offload": "off [fixed]",
                "fcoe_mtu": "off [fixed]",
                "generic_receive_offload": "on",
                "generic_segmentation_offload": "on",
                "highdma": "on [fixed]",
                "hw_tc_offload": "off [fixed]",
                "l2_fwd_offload": "off [fixed]",
                "large_receive_offload": "on",
                "loopback": "off [fixed]",
                "netns_local": "off [fixed]",
                "ntuple_filters": "off [fixed]",
                "receive_hashing": "off [fixed]",
                "rx_all": "off [fixed]",
                "rx_checksumming": "on [fixed]",
                "rx_fcs": "off [fixed]",
                "rx_gro_hw": "off [fixed]",
                "rx_udp_tunnel_port_offload": "off [fixed]",
                "rx_vlan_filter": "on [fixed]",
                "rx_vlan_offload": "off [fixed]",
                "rx_vlan_stag_filter": "off [fixed]",
                "rx_vlan_stag_hw_parse": "off [fixed]",
                "scatter_gather": "on",
                "tcp_segmentation_offload": "on",
                "tls_hw_record": "off [fixed]",
                "tls_hw_rx_offload": "off [fixed]",
                "tls_hw_tx_offload": "off [fixed]",
                "tx_checksum_fcoe_crc": "off [fixed]",
                "tx_checksum_ip_generic": "on",
                "tx_checksum_ipv4": "off [fixed]",
                "tx_checksum_ipv6": "off [fixed]",
                "tx_checksum_sctp": "off [fixed]",
                "tx_checksumming": "on",
                "tx_esp_segmentation": "off [fixed]",
                "tx_fcoe_segmentation": "off [fixed]",
                "tx_gre_csum_segmentation": "off [fixed]",
                "tx_gre_segmentation": "off [fixed]",
                "tx_gso_partial": "off [fixed]",
                "tx_gso_robust": "on [fixed]",
                "tx_ipxip4_segmentation": "off [fixed]",
                "tx_ipxip6_segmentation": "off [fixed]",
                "tx_lockless": "off [fixed]",
                "tx_nocache_copy": "off",
                "tx_scatter_gather": "on",
                "tx_scatter_gather_fraglist": "off [fixed]",
                "tx_sctp_segmentation": "off [fixed]",
                "tx_tcp6_segmentation": "on",
                "tx_tcp_ecn_segmentation": "on",
                "tx_tcp_mangleid_segmentation": "off",
                "tx_tcp_segmentation": "on",
                "tx_udp_segmentation": "off [fixed]",
                "tx_udp_tnl_csum_segmentation": "off [fixed]",
                "tx_udp_tnl_segmentation": "off [fixed]",
                "tx_vlan_offload": "off [fixed]",
                "tx_vlan_stag_hw_insert": "off [fixed]",
                "vlan_challenged": "off [fixed]"
            },
            "hw_timestamp_filters": [],
            "ipv4": {
                "address": "192.168.122.170",
                "broadcast": "192.168.122.255",
                "netmask": "255.255.255.0",
                "network": "192.168.122.0"
            },
            "ipv6": [
                {
                    "address": "fe80::5054:ff:fe54:7cfe",
                    "prefix": "64",
                    "scope": "link"
                }
            ],
            "macaddress": "52:54:00:54:7c:fe",
            "module": "virtio_net",
            "mtu": 1500,
            "pciid": "virtio0",
            "promisc": false,
            "speed": -1,
            "timestamping": [
                "tx_software",
                "rx_software",
                "software"
            ],
            "type": "ether"
        },
        "ansible_env": {
            "DBUS_SESSION_BUS_ADDRESS": "unix:path=/run/user/0/bus",
            "HOME": "/root",
            "LANG": "en_GB.UTF-8",
            "LOGNAME": "root",
            "MOTD_SHOWN": "pam",
            "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
            "PWD": "/root",
            "SHELL": "/bin/bash",
            "SHLVL": "0",
            "SSH_AUTH_SOCK": "/tmp/ssh-xGhYmKCci1/agent.4096",
            "SSH_CLIENT": "192.168.122.1 40374 22",
            "SSH_CONNECTION": "192.168.122.1 40374 192.168.122.170 22",
            "SSH_TTY": "/dev/pts/0",
            "TERM": "screen-256color",
            "USER": "root",
            "XDG_RUNTIME_DIR": "/run/user/0",
            "XDG_SESSION_CLASS": "user",
            "XDG_SESSION_ID": "17",
            "XDG_SESSION_TYPE": "tty",
            "_": "/bin/sh"
        },
        "ansible_fips": false,
        "ansible_form_factor": "Other",
        "ansible_fqdn": "ubuntu2004",
        "ansible_hostname": "ubuntu2004",
        "ansible_interfaces": [
            "enp1s0",
            "lo"
        ],
        "ansible_is_chroot": false,
        "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:af5bf2af245",
        "ansible_kernel": "5.4.0-65-generic",
        "ansible_lo": {
            "active": true,
            "device": "lo",
            "features": {
                "esp_hw_offload": "off [fixed]",
                "esp_tx_csum_hw_offload": "off [fixed]",
                "fcoe_mtu": "off [fixed]",
                "generic_receive_offload": "on",
                "generic_segmentation_offload": "on",
                "highdma": "on [fixed]",
                "hw_tc_offload": "off [fixed]",
                "l2_fwd_offload": "off [fixed]",
                "large_receive_offload": "off [fixed]",
                "loopback": "on [fixed]",
                "netns_local": "on [fixed]",
                "ntuple_filters": "off [fixed]",
                "receive_hashing": "off [fixed]",
                "rx_all": "off [fixed]",
                "rx_checksumming": "on [fixed]",
                "rx_fcs": "off [fixed]",
                "rx_gro_hw": "off [fixed]",
                "rx_udp_tunnel_port_offload": "off [fixed]",
                "rx_vlan_filter": "off [fixed]",
                "rx_vlan_offload": "off [fixed]",
                "rx_vlan_stag_filter": "off [fixed]",
                "rx_vlan_stag_hw_parse": "off [fixed]",
                "scatter_gather": "on",
                "tcp_segmentation_offload": "on",
                "tls_hw_record": "off [fixed]",
                "tls_hw_rx_offload": "off [fixed]",
                "tls_hw_tx_offload": "off [fixed]",
                "tx_checksum_fcoe_crc": "off [fixed]",
                "tx_checksum_ip_generic": "on [fixed]",
                "tx_checksum_ipv4": "off [fixed]",
                "tx_checksum_ipv6": "off [fixed]",
                "tx_checksum_sctp": "on [fixed]",
                "tx_checksumming": "on",
                "tx_esp_segmentation": "off [fixed]",
                "tx_fcoe_segmentation": "off [fixed]",
                "tx_gre_csum_segmentation": "off [fixed]",
                "tx_gre_segmentation": "off [fixed]",
                "tx_gso_partial": "off [fixed]",
                "tx_gso_robust": "off [fixed]",
                "tx_ipxip4_segmentation": "off [fixed]",
                "tx_ipxip6_segmentation": "off [fixed]",
                "tx_lockless": "on [fixed]",
                "tx_nocache_copy": "off [fixed]",
                "tx_scatter_gather": "on [fixed]",
                "tx_scatter_gather_fraglist": "on [fixed]",
                "tx_sctp_segmentation": "on",
                "tx_tcp6_segmentation": "on",
                "tx_tcp_ecn_segmentation": "on",
                "tx_tcp_mangleid_segmentation": "on",
                "tx_tcp_segmentation": "on",
                "tx_udp_segmentation": "off [fixed]",
                "tx_udp_tnl_csum_segmentation": "off [fixed]",
                "tx_udp_tnl_segmentation": "off [fixed]",
                "tx_vlan_offload": "off [fixed]",
                "tx_vlan_stag_hw_insert": "off [fixed]",
                "vlan_challenged": "on [fixed]"
            },
            "hw_timestamp_filters": [],
            "ipv4": {
                "address": "127.0.0.1",
                "broadcast": "host",
                "netmask": "255.0.0.0",
                "network": "127.0.0.0"
            },
            "ipv6": [
                {
                    "address": "::1",
                    "prefix": "128",
                    "scope": "host"
                }
            ],
            "mtu": 65536,
            "promisc": false,
            "timestamping": [
                "tx_software",
                "rx_software",
                "software"
            ],
            "type": "loopback"
        },
        "ansible_local": {},
        "ansible_lsb": {
            "codename": "focal",
            "description": "Ubuntu 20.04.2 LTS",
            "id": "Ubuntu",
            "major_release": "20",
            "release": "20.04"
        },
        "ansible_lvm": {
            "lvs": {
                "ubuntu-lv": {
                    "size_g": "20.00",
                    "vg": "ubuntu-vg"
                }
            },
            "pvs": {
                "/dev/sda3": {
                    "free_g": "19.00",
                    "size_g": "39.00",
                    "vg": "ubuntu-vg"
                }
            },
            "vgs": {
                "ubuntu-vg": {
                    "free_g": "19.00",
                    "num_lvs": "1",
                    "num_pvs": "1",
                    "size_g": "39.00"
                }
            }
        },
        "ansible_machine": "x86_64",
        "ansible_machine_id": "64d7747e869a45b09d0aae9a6d463611",
        "ansible_memfree_mb": 2765,
        "ansible_memory_mb": {
            "nocache": {
                "free": 3687,
                "used": 248
            },
            "real": {
                "free": 2765,
                "total": 3935,
                "used": 1170
            },
            "swap": {
                "cached": 0,
                "free": 3934,
                "total": 3934,
                "used": 0
            }
        },
        "ansible_memtotal_mb": 3935,
        "ansible_mounts": [
            {
                "block_available": 2357334,
                "block_size": 4096,
                "block_total": 5127828,
                "block_used": 2770494,
                "device": "/dev/mapper/ubuntu--vg-ubuntu--lv",
                "fstype": "ext4",
                "inode_available": 1130751,
                "inode_total": 1310720,
                "inode_used": 179969,
                "mount": "/",
                "options": "rw,relatime",
                "size_available": 9655640064,
                "size_total": 21003583488,
                "uuid": "291656fe-bd87-484c-b4a9-4453471a17e8"
            },
            {
                "block_available": 181527,
                "block_size": 4096,
                "block_total": 249830,
                "block_used": 68303,
                "device": "/dev/sda2",
                "fstype": "ext4",
                "inode_available": 65220,
                "inode_total": 65536,
                "inode_used": 316,
                "mount": "/boot",
                "options": "rw,relatime",
                "size_available": 743534592,
                "size_total": 1023303680,
                "uuid": "45018994-9625-44ad-877a-3980bcf943a3"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 444,
                "block_used": 444,
                "device": "/dev/loop3",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 10809,
                "inode_used": 10809,
                "mount": "/snap/core18/1944",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 58195968,
                "uuid": "N/A"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 249,
                "block_used": 249,
                "device": "/dev/loop4",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 472,
                "inode_used": 472,
                "mount": "/snap/snapd/10707",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 32636928,
                "uuid": "N/A"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 559,
                "block_used": 559,
                "device": "/dev/loop5",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 1578,
                "inode_used": 1578,
                "mount": "/snap/lxd/19032",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 73269248,
                "uuid": "N/A"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 444,
                "block_used": 444,
                "device": "/dev/loop6",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 10817,
                "inode_used": 10817,
                "mount": "/snap/core18/1988",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 58195968,
                "uuid": "N/A"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 249,
                "block_used": 249,
                "device": "/dev/loop7",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 470,
                "inode_used": 470,
                "mount": "/snap/snapd/11036",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 32636928,
                "uuid": "N/A"
            },
            {
                "block_available": 0,
                "block_size": 131072,
                "block_total": 560,
                "block_used": 560,
                "device": "/dev/loop0",
                "fstype": "squashfs",
                "inode_available": 0,
                "inode_total": 1578,
                "inode_used": 1578,
                "mount": "/snap/lxd/19188",
                "options": "ro,nodev,relatime",
                "size_available": 0,
                "size_total": 73400320,
                "uuid": "N/A"
            }
        ],
        "ansible_nodename": "ubuntu2004",
        "ansible_processor": [
            "0",
            "GenuineIntel",
            "Intel Xeon Processor (Skylake, IBRS)",
            "1",
            "GenuineIntel",
            "Intel Xeon Processor (Skylake, IBRS)",
            "2",
            "GenuineIntel",
            "Intel Xeon Processor (Skylake, IBRS)",
            "3",
            "GenuineIntel",
            "Intel Xeon Processor (Skylake, IBRS)"
        ],
        "ansible_processor_cores": 1,
        "ansible_processor_count": 4,
        "ansible_processor_threads_per_core": 1,
        "ansible_processor_vcpus": 4,
        "ansible_product_name": "Standard PC (Q35 + ICH9, 2009)",
        "ansible_product_serial": "NA",
        "ansible_product_uuid": "64d7747e-869a-45b0-9d0a-ae9a6d463611",
        "ansible_product_version": "pc-q35-2.11",
        "ansible_python": {
            "executable": "/usr/bin/python3",
            "has_sslcontext": true,
            "type": "cpython",
            "version": {
                "major": 3,
                "micro": 5,
                "minor": 8,
                "releaselevel": "final",
                "serial": 0
            },
            "version_info": [
                3,
                8,
                5,
                "final",
                0
            ]
        },
        "ansible_python_version": "3.8.5",
        "ansible_real_group_id": 0,
        "ansible_real_user_id": 0,
        "ansible_selinux": {
            "status": "Missing selinux Python library"
        },
        "ansible_selinux_python_present": false,
        "ansible_service_mgr": "systemd",
        "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANcblFlURYNVXrHiZ2ozUgS6NQWkL9q6dKRvhFV75WjqBQfZs4wAAd9qYdT/fAJfT+MHdaeKgAzIgCCEH0lwEOVJY5go1u3AOEuq6S2b9D1Tr6VufAjuVYuDbPqYCoYPMDepsgKfJIxLGcfs0SgaeJyCzKOh5prQrDfYPHIP3NRjAAAAFQDc3uAPrXGgg7q74VaX8yMC5evKjwAAAIEAy+8TP/tI05oSLikv6L5es6J/iIXouuCSSlpzYT+ZcA64PaXB7X9ziRUOF79fbWkVYGmCRutjayucFHfsnICwm17vLaA5Pdc18hKqgO64HLhX1fBf8BE3KKQFY2nqcop0ShRHsLHWoL5E8SJ0Jrjd+wqw/0SQ4EnxxdmW7mrf+KUAAACAeWRshM/sCGP/DDifYusYkhZ85d5vgeXK/h9d4V3WhnXa6TlNPTo7Y21rX842UJ8npSf+ZVZb9iRJMRxGJiGgQK3GPRvdopQPFM9Y+kTf4GfLS5Bmd4RdZXF0POEpe10xc0ewg5is9NsFnJI+mJFcEB9FH+TtS0T7PmP+l9ADkTs=",
        "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGumKEwBRlMPpWemu3oyScRXbm4/dH+q5iCKvhB4EsehElxsVTQXbNjQyv5Ei38yG34N2q5DvZSus+tD8LJEZW4=",
        "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIBJpeIq8MEf3YBN7NLxd/ss/iqbvH9q34eLjYP0tubup",
        "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABgQC8hyqeCETLm6kd/kG9lRy9HWIBFFRQTlsIUYdBDmb8dcA0Ye6JwcGhFbEJD5KaWKmyul0OP0dmV4BfLdf1dzDvilh0vTfgTTklbsPpEEjlfstqLHpKDZ4wL+Gj8eF54xW00oFwSR68CWNomyR0YrTczsN/CUb5HSejvYS48OzRP+it4iTyrlwVp8Lb7O7m/TnQFbys8uTaFFNpXFm4WrBtK0HlqVI/9LASXnuYqudCOgwkGlKamVnSwCO3Bt8MXFdhkgvXqoEp0sCdGZIM207jrN42hy6stXyjvn/43YbfTAiXwJDPUhllpbuUSTRF3zzlIvHbC0JRwq0wGd+eXS5kb9RS6v5QLptn0pA8kxQYg2uqO4I+Uc0R7akmgPu1S85jobS7MJIpZmNj57fGmUC7ZUvYTQ97lXcWrfNzk4pwl9TG85U/tQNwN6X5TmaFuSkqGSVRb+a86Z62//BH6lY8sOPEn+Ou883l3QBXjSkgQIjpWKy30GlUcd8Mn6nsgU0=",
        "ansible_swapfree_mb": 3934,
        "ansible_swaptotal_mb": 3934,
        "ansible_system": "Linux",
        "ansible_system_capabilities": [],
        "ansible_system_capabilities_enforced": "False",
        "ansible_system_vendor": "QEMU",
        "ansible_uptime_seconds": 12500,
        "ansible_user_dir": "/root",
        "ansible_user_gecos": "root",
        "ansible_user_gid": 0,
        "ansible_user_id": "root",
        "ansible_user_shell": "/bin/bash",
        "ansible_user_uid": 0,
        "ansible_userspace_architecture": "x86_64",
        "ansible_userspace_bits": "64",
        "ansible_virtualization_role": "guest",
        "ansible_virtualization_type": "kvm",
        "gather_subset": [
            "all"
        ],
        "module_setup": true
    },
    "changed": false
}


>
> Thanks,
> - Cleber.


-- 
Alex Bennée

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

On Tue, Feb 23, 2021 at 03:56:19PM +0000, Daniel P. Berrangé wrote:
> 
> Urgh, well that's a big problem. We certainly don't want *anything* being
> placed on the custom runners without explicit opt-in, otherwise jobs run
> in the main repo have a different environment from when users run on their
> personal forks.
> 
> IOW, we need anti-affinity against our custom runners really.
>

I'm assuming Phil missed that documentation, because that's a
non-issue, really.

Just unchecking the "Run untagged jobs" check box on the runner
configuration makes sure that the custom runners won't pickup any jobs
not *specifically* tagged for them.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 2/4] Jobs based on custom runners: build environment docs and playbook

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 39981 bytes --]

On Tue, Feb 23, 2021 at 06:23:25PM +0000, Alex Bennée wrote:
> 
> Cleber Rosa <crosa@redhat.com> writes:
> 
> > On Tue, Feb 23, 2021 at 03:01:50PM +0000, Alex Bennée wrote:
> >> 
> >> Alex Bennée <alex.bennee@linaro.org> writes:
> >> 
> >> > Cleber Rosa <crosa@redhat.com> writes:
> >> >
> >> >> To run basic jobs on custom runners, the environment needs to be
> >> >> properly set up.  The most common requirement is having the right
> >> >> packages installed.
> >> >>
> >> <snip>
> >> >
> >> > So I got somewhat there with a direct command line invocation:
> >> >
> >> >   ansible-playbook -u root -i 192.168.122.24,192.168.122.45 scripts/ci/setup/build-environment.yml -e 'ansible_python_interpreter=/usr/bin/python3'
> >> >
> >> > although for some reason a single host -i fails...
> >> >
> >> >> diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
> >> >> new file mode 100644
> >> >> index 0000000000..0197e0a48b
> >> >> --- /dev/null
> >> >> +++ b/scripts/ci/setup/build-environment.yml
> >> >> @@ -0,0 +1,76 @@
> >> >> +---
> >> >> +- name: Installation of basic packages to build QEMU
> >> >> +  hosts: all
> >> >> +  tasks:
> >> >> +    - name: Update apt cache
> >> >> +      apt:
> >> >> +        update_cache: yes
> >> >> +      when:
> >> >> +        - ansible_facts['distribution'] == 'Ubuntu'
> >> >
> >> > So are we limiting to Ubuntu here rather than say a Debian base?
> >> 
> >> Also I'm getting:
> >> 
> >>   TASK [Update apt cache] *****************************************************************************************************************************************************
> >>   fatal: [hackbox-ubuntu-2004]: FAILED! => {"msg": "The conditional check 'ansible_facts['distribution'] == 'Ubuntu'' failed. The error was: error while evaluating conditional (ansible_facts['distribution'] == 'Ubuntu'): 'dict object' has no attribute 'distribution'\n\nThe error appears to have been in '/home/alex/lsrc/qemu.git/scripts/ci/setup/build-environment.yml': line 5, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - name: Update apt cache\n      ^ here\n"}
> >> 
> >> which is odd given that machine is definitely an Ubuntu one.
> >>
> >
> > It's defintely odd.  This is what I get on a fresh machine:
> >
> >    TASK [Update apt cache] *************************************************************************************************************************
> >    [WARNING]: Updating cache and auto-installing missing dependency: python3-apt
> >    ok: [localhost]
> >
> > Could you please let me know the output of:
> >
> >    $ ansible -m setup -u $YOUR_USERNAME -i $HOSTNAME, all | grep
> >    ansible_distribution
> 
> The key doesn't exist:
> 
> hackbox-ubuntu-2004 | SUCCESS => {
>     "ansible_facts": {
>         "ansible_all_ipv4_addresses": [
>             "192.168.122.170"
>         ],
>         "ansible_all_ipv6_addresses": [
>             "fe80::5054:ff:fe54:7cfe"
>         ],
>         "ansible_apparmor": {
>             "status": "enabled"
>         },
>         "ansible_architecture": "x86_64",
>         "ansible_bios_date": "04/01/2014",
>         "ansible_bios_version": "1.10.2-1ubuntu1",
>         "ansible_cmdline": {
>             "BOOT_IMAGE": "/vmlinuz-5.4.0-65-generic",
>             "maybe-ubiquity": true,
>             "ro": true,
>             "root": "/dev/mapper/ubuntu--vg-ubuntu--lv"
>         },
>         "ansible_date_time": {
>             "date": "2021-02-23",
>             "day": "23",
>             "epoch": "1614104601",
>             "hour": "18",
>             "iso8601": "2021-02-23T18:23:21Z",
>             "iso8601_basic": "20210223T182321857461",
>             "iso8601_basic_short": "20210223T182321",
>             "iso8601_micro": "2021-02-23T18:23:21.857529Z",
>             "minute": "23",
>             "month": "02",
>             "second": "21",
>             "time": "18:23:21",
>             "tz": "UTC",
>             "tz_offset": "+0000",
>             "weekday": "Tuesday",
>             "weekday_number": "2",
>             "weeknumber": "08",
>             "year": "2021"
>         },
>         "ansible_default_ipv4": {
>             "address": "192.168.122.170",
>             "alias": "enp1s0",
>             "broadcast": "192.168.122.255",
>             "gateway": "192.168.122.1",
>             "interface": "enp1s0",
>             "macaddress": "52:54:00:54:7c:fe",
>             "mtu": 1500,
>             "netmask": "255.255.255.0",
>             "network": "192.168.122.0",
>             "type": "ether"
>         },
>         "ansible_default_ipv6": {},
>         "ansible_device_links": {
>             "ids": {
>                 "dm-0": [
>                     "dm-name-ubuntu--vg-ubuntu--lv",
>                     "dm-uuid-LVM-filR1BfuX6Mpp9J7CP9cbVsTT2ICh7Apc9qZsFohnsqycocacS0Sm6HAhjTBEAkq"
>                 ],
>                 "sda": [
>                     "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1"
>                 ],
>                 "sda1": [
>                     "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part1"
>                 ],
>                 "sda2": [
>                     "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part2"
>                 ],
>                 "sda3": [
>                     "lvm-pv-uuid-agDdyQ-V5gQ-aaov-933l-SFAL-0rmD-SlOkYy",
>                     "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part3"
>                 ],
>                 "sr0": [
>                     "ata-QEMU_DVD-ROM_QM00001"
>                 ]
>             },
>             "labels": {},
>             "masters": {
>                 "sda3": [
>                     "dm-0"
>                 ]
>             },
>             "uuids": {
>                 "dm-0": [
>                     "291656fe-bd87-484c-b4a9-4453471a17e8"
>                 ],
>                 "sda2": [
>                     "45018994-9625-44ad-877a-3980bcf943a3"
>                 ]
>             }
>         },
>         "ansible_devices": {
>             "dm-0": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [
>                         "dm-name-ubuntu--vg-ubuntu--lv",
>                         "dm-uuid-LVM-filR1BfuX6Mpp9J7CP9cbVsTT2ICh7Apc9qZsFohnsqycocacS0Sm6HAhjTBEAkq"
>                     ],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": [
>                         "291656fe-bd87-484c-b4a9-4453471a17e8"
>                     ]
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "",
>                 "sectors": "41943040",
>                 "sectorsize": "512",
>                 "size": "20.00 GB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop0": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "143120",
>                 "sectorsize": "512",
>                 "size": "69.88 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop1": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "0",
>                 "sectorsize": "512",
>                 "size": "0.00 Bytes",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop2": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "0",
>                 "sectorsize": "512",
>                 "size": "0.00 Bytes",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop3": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "113424",
>                 "sectorsize": "512",
>                 "size": "55.38 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop4": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "63672",
>                 "sectorsize": "512",
>                 "size": "31.09 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop5": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "142872",
>                 "sectorsize": "512",
>                 "size": "69.76 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop6": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "113592",
>                 "sectorsize": "512",
>                 "size": "55.46 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "loop7": {
>                 "holders": [],
>                 "host": "",
>                 "links": {
>                     "ids": [],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": null,
>                 "partitions": {},
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "63664",
>                 "sectorsize": "512",
>                 "size": "31.09 MB",
>                 "support_discard": "4096",
>                 "vendor": null,
>                 "virtual": 1
>             },
>             "sda": {
>                 "holders": [],
>                 "host": "SCSI storage controller: Broadcom / LSI 53c895a",
>                 "links": {
>                     "ids": [
>                         "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1"
>                     ],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": "QEMU HARDDISK",
>                 "partitions": {
>                     "sda1": {
>                         "holders": [],
>                         "links": {
>                             "ids": [
>                                 "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part1"
>                             ],
>                             "labels": [],
>                             "masters": [],
>                             "uuids": []
>                         },
>                         "sectors": "2048",
>                         "sectorsize": 512,
>                         "size": "1.00 MB",
>                         "start": "2048",
>                         "uuid": null
>                     },
>                     "sda2": {
>                         "holders": [],
>                         "links": {
>                             "ids": [
>                                 "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part2"
>                             ],
>                             "labels": [],
>                             "masters": [],
>                             "uuids": [
>                                 "45018994-9625-44ad-877a-3980bcf943a3"
>                             ]
>                         },
>                         "sectors": "2097152",
>                         "sectorsize": 512,
>                         "size": "1.00 GB",
>                         "start": "4096",
>                         "uuid": "45018994-9625-44ad-877a-3980bcf943a3"
>                     },
>                     "sda3": {
>                         "holders": [
>                             "ubuntu--vg-ubuntu--lv"
>                         ],
>                         "links": {
>                             "ids": [
>                                 "lvm-pv-uuid-agDdyQ-V5gQ-aaov-933l-SFAL-0rmD-SlOkYy",
>                                 "scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1-part3"
>                             ],
>                             "labels": [],
>                             "masters": [
>                                 "dm-0"
>                             ],
>                             "uuids": []
>                         },
>                         "sectors": "81782784",
>                         "sectorsize": 512,
>                         "size": "39.00 GB",
>                         "start": "2101248",
>                         "uuid": null
>                     }
>                 },
>                 "removable": "0",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "83886080",
>                 "sectorsize": "512",
>                 "size": "40.00 GB",
>                 "support_discard": "4096",
>                 "vendor": "QEMU",
>                 "virtual": 1
>             },
>             "sr0": {
>                 "holders": [],
>                 "host": "SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)",
>                 "links": {
>                     "ids": [
>                         "ata-QEMU_DVD-ROM_QM00001"
>                     ],
>                     "labels": [],
>                     "masters": [],
>                     "uuids": []
>                 },
>                 "model": "QEMU DVD-ROM",
>                 "partitions": {},
>                 "removable": "1",
>                 "rotational": "1",
>                 "sas_address": null,
>                 "sas_device_handle": null,
>                 "scheduler_mode": "mq-deadline",
>                 "sectors": "2097151",
>                 "sectorsize": "512",
>                 "size": "1024.00 MB",
>                 "support_discard": "0",
>                 "vendor": "QEMU",
>                 "virtual": 1
>             }
>         },
>         "ansible_dns": {
>             "nameservers": [
>                 "127.0.0.53"
>             ],
>             "options": {
>                 "edns0": true,
>                 "trust-ad": true
>             }
>         },
>         "ansible_domain": "",
>         "ansible_effective_group_id": 0,
>         "ansible_effective_user_id": 0,
>         "ansible_enp1s0": {
>             "active": true,
>             "device": "enp1s0",
>             "features": {
>                 "esp_hw_offload": "off [fixed]",
>                 "esp_tx_csum_hw_offload": "off [fixed]",
>                 "fcoe_mtu": "off [fixed]",
>                 "generic_receive_offload": "on",
>                 "generic_segmentation_offload": "on",
>                 "highdma": "on [fixed]",
>                 "hw_tc_offload": "off [fixed]",
>                 "l2_fwd_offload": "off [fixed]",
>                 "large_receive_offload": "on",
>                 "loopback": "off [fixed]",
>                 "netns_local": "off [fixed]",
>                 "ntuple_filters": "off [fixed]",
>                 "receive_hashing": "off [fixed]",
>                 "rx_all": "off [fixed]",
>                 "rx_checksumming": "on [fixed]",
>                 "rx_fcs": "off [fixed]",
>                 "rx_gro_hw": "off [fixed]",
>                 "rx_udp_tunnel_port_offload": "off [fixed]",
>                 "rx_vlan_filter": "on [fixed]",
>                 "rx_vlan_offload": "off [fixed]",
>                 "rx_vlan_stag_filter": "off [fixed]",
>                 "rx_vlan_stag_hw_parse": "off [fixed]",
>                 "scatter_gather": "on",
>                 "tcp_segmentation_offload": "on",
>                 "tls_hw_record": "off [fixed]",
>                 "tls_hw_rx_offload": "off [fixed]",
>                 "tls_hw_tx_offload": "off [fixed]",
>                 "tx_checksum_fcoe_crc": "off [fixed]",
>                 "tx_checksum_ip_generic": "on",
>                 "tx_checksum_ipv4": "off [fixed]",
>                 "tx_checksum_ipv6": "off [fixed]",
>                 "tx_checksum_sctp": "off [fixed]",
>                 "tx_checksumming": "on",
>                 "tx_esp_segmentation": "off [fixed]",
>                 "tx_fcoe_segmentation": "off [fixed]",
>                 "tx_gre_csum_segmentation": "off [fixed]",
>                 "tx_gre_segmentation": "off [fixed]",
>                 "tx_gso_partial": "off [fixed]",
>                 "tx_gso_robust": "on [fixed]",
>                 "tx_ipxip4_segmentation": "off [fixed]",
>                 "tx_ipxip6_segmentation": "off [fixed]",
>                 "tx_lockless": "off [fixed]",
>                 "tx_nocache_copy": "off",
>                 "tx_scatter_gather": "on",
>                 "tx_scatter_gather_fraglist": "off [fixed]",
>                 "tx_sctp_segmentation": "off [fixed]",
>                 "tx_tcp6_segmentation": "on",
>                 "tx_tcp_ecn_segmentation": "on",
>                 "tx_tcp_mangleid_segmentation": "off",
>                 "tx_tcp_segmentation": "on",
>                 "tx_udp_segmentation": "off [fixed]",
>                 "tx_udp_tnl_csum_segmentation": "off [fixed]",
>                 "tx_udp_tnl_segmentation": "off [fixed]",
>                 "tx_vlan_offload": "off [fixed]",
>                 "tx_vlan_stag_hw_insert": "off [fixed]",
>                 "vlan_challenged": "off [fixed]"
>             },
>             "hw_timestamp_filters": [],
>             "ipv4": {
>                 "address": "192.168.122.170",
>                 "broadcast": "192.168.122.255",
>                 "netmask": "255.255.255.0",
>                 "network": "192.168.122.0"
>             },
>             "ipv6": [
>                 {
>                     "address": "fe80::5054:ff:fe54:7cfe",
>                     "prefix": "64",
>                     "scope": "link"
>                 }
>             ],
>             "macaddress": "52:54:00:54:7c:fe",
>             "module": "virtio_net",
>             "mtu": 1500,
>             "pciid": "virtio0",
>             "promisc": false,
>             "speed": -1,
>             "timestamping": [
>                 "tx_software",
>                 "rx_software",
>                 "software"
>             ],
>             "type": "ether"
>         },
>         "ansible_env": {
>             "DBUS_SESSION_BUS_ADDRESS": "unix:path=/run/user/0/bus",
>             "HOME": "/root",
>             "LANG": "en_GB.UTF-8",
>             "LOGNAME": "root",
>             "MOTD_SHOWN": "pam",
>             "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
>             "PWD": "/root",
>             "SHELL": "/bin/bash",
>             "SHLVL": "0",
>             "SSH_AUTH_SOCK": "/tmp/ssh-xGhYmKCci1/agent.4096",
>             "SSH_CLIENT": "192.168.122.1 40374 22",
>             "SSH_CONNECTION": "192.168.122.1 40374 192.168.122.170 22",
>             "SSH_TTY": "/dev/pts/0",
>             "TERM": "screen-256color",
>             "USER": "root",
>             "XDG_RUNTIME_DIR": "/run/user/0",
>             "XDG_SESSION_CLASS": "user",
>             "XDG_SESSION_ID": "17",
>             "XDG_SESSION_TYPE": "tty",
>             "_": "/bin/sh"
>         },
>         "ansible_fips": false,
>         "ansible_form_factor": "Other",
>         "ansible_fqdn": "ubuntu2004",
>         "ansible_hostname": "ubuntu2004",
>         "ansible_interfaces": [
>             "enp1s0",
>             "lo"
>         ],
>         "ansible_is_chroot": false,
>         "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:af5bf2af245",
>         "ansible_kernel": "5.4.0-65-generic",
>         "ansible_lo": {
>             "active": true,
>             "device": "lo",
>             "features": {
>                 "esp_hw_offload": "off [fixed]",
>                 "esp_tx_csum_hw_offload": "off [fixed]",
>                 "fcoe_mtu": "off [fixed]",
>                 "generic_receive_offload": "on",
>                 "generic_segmentation_offload": "on",
>                 "highdma": "on [fixed]",
>                 "hw_tc_offload": "off [fixed]",
>                 "l2_fwd_offload": "off [fixed]",
>                 "large_receive_offload": "off [fixed]",
>                 "loopback": "on [fixed]",
>                 "netns_local": "on [fixed]",
>                 "ntuple_filters": "off [fixed]",
>                 "receive_hashing": "off [fixed]",
>                 "rx_all": "off [fixed]",
>                 "rx_checksumming": "on [fixed]",
>                 "rx_fcs": "off [fixed]",
>                 "rx_gro_hw": "off [fixed]",
>                 "rx_udp_tunnel_port_offload": "off [fixed]",
>                 "rx_vlan_filter": "off [fixed]",
>                 "rx_vlan_offload": "off [fixed]",
>                 "rx_vlan_stag_filter": "off [fixed]",
>                 "rx_vlan_stag_hw_parse": "off [fixed]",
>                 "scatter_gather": "on",
>                 "tcp_segmentation_offload": "on",
>                 "tls_hw_record": "off [fixed]",
>                 "tls_hw_rx_offload": "off [fixed]",
>                 "tls_hw_tx_offload": "off [fixed]",
>                 "tx_checksum_fcoe_crc": "off [fixed]",
>                 "tx_checksum_ip_generic": "on [fixed]",
>                 "tx_checksum_ipv4": "off [fixed]",
>                 "tx_checksum_ipv6": "off [fixed]",
>                 "tx_checksum_sctp": "on [fixed]",
>                 "tx_checksumming": "on",
>                 "tx_esp_segmentation": "off [fixed]",
>                 "tx_fcoe_segmentation": "off [fixed]",
>                 "tx_gre_csum_segmentation": "off [fixed]",
>                 "tx_gre_segmentation": "off [fixed]",
>                 "tx_gso_partial": "off [fixed]",
>                 "tx_gso_robust": "off [fixed]",
>                 "tx_ipxip4_segmentation": "off [fixed]",
>                 "tx_ipxip6_segmentation": "off [fixed]",
>                 "tx_lockless": "on [fixed]",
>                 "tx_nocache_copy": "off [fixed]",
>                 "tx_scatter_gather": "on [fixed]",
>                 "tx_scatter_gather_fraglist": "on [fixed]",
>                 "tx_sctp_segmentation": "on",
>                 "tx_tcp6_segmentation": "on",
>                 "tx_tcp_ecn_segmentation": "on",
>                 "tx_tcp_mangleid_segmentation": "on",
>                 "tx_tcp_segmentation": "on",
>                 "tx_udp_segmentation": "off [fixed]",
>                 "tx_udp_tnl_csum_segmentation": "off [fixed]",
>                 "tx_udp_tnl_segmentation": "off [fixed]",
>                 "tx_vlan_offload": "off [fixed]",
>                 "tx_vlan_stag_hw_insert": "off [fixed]",
>                 "vlan_challenged": "on [fixed]"
>             },
>             "hw_timestamp_filters": [],
>             "ipv4": {
>                 "address": "127.0.0.1",
>                 "broadcast": "host",
>                 "netmask": "255.0.0.0",
>                 "network": "127.0.0.0"
>             },
>             "ipv6": [
>                 {
>                     "address": "::1",
>                     "prefix": "128",
>                     "scope": "host"
>                 }
>             ],
>             "mtu": 65536,
>             "promisc": false,
>             "timestamping": [
>                 "tx_software",
>                 "rx_software",
>                 "software"
>             ],
>             "type": "loopback"
>         },
>         "ansible_local": {},
>         "ansible_lsb": {
>             "codename": "focal",
>             "description": "Ubuntu 20.04.2 LTS",
>             "id": "Ubuntu",
>             "major_release": "20",
>             "release": "20.04"
>         },
>         "ansible_lvm": {
>             "lvs": {
>                 "ubuntu-lv": {
>                     "size_g": "20.00",
>                     "vg": "ubuntu-vg"
>                 }
>             },
>             "pvs": {
>                 "/dev/sda3": {
>                     "free_g": "19.00",
>                     "size_g": "39.00",
>                     "vg": "ubuntu-vg"
>                 }
>             },
>             "vgs": {
>                 "ubuntu-vg": {
>                     "free_g": "19.00",
>                     "num_lvs": "1",
>                     "num_pvs": "1",
>                     "size_g": "39.00"
>                 }
>             }
>         },
>         "ansible_machine": "x86_64",
>         "ansible_machine_id": "64d7747e869a45b09d0aae9a6d463611",
>         "ansible_memfree_mb": 2765,
>         "ansible_memory_mb": {
>             "nocache": {
>                 "free": 3687,
>                 "used": 248
>             },
>             "real": {
>                 "free": 2765,
>                 "total": 3935,
>                 "used": 1170
>             },
>             "swap": {
>                 "cached": 0,
>                 "free": 3934,
>                 "total": 3934,
>                 "used": 0
>             }
>         },
>         "ansible_memtotal_mb": 3935,
>         "ansible_mounts": [
>             {
>                 "block_available": 2357334,
>                 "block_size": 4096,
>                 "block_total": 5127828,
>                 "block_used": 2770494,
>                 "device": "/dev/mapper/ubuntu--vg-ubuntu--lv",
>                 "fstype": "ext4",
>                 "inode_available": 1130751,
>                 "inode_total": 1310720,
>                 "inode_used": 179969,
>                 "mount": "/",
>                 "options": "rw,relatime",
>                 "size_available": 9655640064,
>                 "size_total": 21003583488,
>                 "uuid": "291656fe-bd87-484c-b4a9-4453471a17e8"
>             },
>             {
>                 "block_available": 181527,
>                 "block_size": 4096,
>                 "block_total": 249830,
>                 "block_used": 68303,
>                 "device": "/dev/sda2",
>                 "fstype": "ext4",
>                 "inode_available": 65220,
>                 "inode_total": 65536,
>                 "inode_used": 316,
>                 "mount": "/boot",
>                 "options": "rw,relatime",
>                 "size_available": 743534592,
>                 "size_total": 1023303680,
>                 "uuid": "45018994-9625-44ad-877a-3980bcf943a3"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 444,
>                 "block_used": 444,
>                 "device": "/dev/loop3",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 10809,
>                 "inode_used": 10809,
>                 "mount": "/snap/core18/1944",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 58195968,
>                 "uuid": "N/A"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 249,
>                 "block_used": 249,
>                 "device": "/dev/loop4",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 472,
>                 "inode_used": 472,
>                 "mount": "/snap/snapd/10707",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 32636928,
>                 "uuid": "N/A"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 559,
>                 "block_used": 559,
>                 "device": "/dev/loop5",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 1578,
>                 "inode_used": 1578,
>                 "mount": "/snap/lxd/19032",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 73269248,
>                 "uuid": "N/A"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 444,
>                 "block_used": 444,
>                 "device": "/dev/loop6",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 10817,
>                 "inode_used": 10817,
>                 "mount": "/snap/core18/1988",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 58195968,
>                 "uuid": "N/A"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 249,
>                 "block_used": 249,
>                 "device": "/dev/loop7",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 470,
>                 "inode_used": 470,
>                 "mount": "/snap/snapd/11036",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 32636928,
>                 "uuid": "N/A"
>             },
>             {
>                 "block_available": 0,
>                 "block_size": 131072,
>                 "block_total": 560,
>                 "block_used": 560,
>                 "device": "/dev/loop0",
>                 "fstype": "squashfs",
>                 "inode_available": 0,
>                 "inode_total": 1578,
>                 "inode_used": 1578,
>                 "mount": "/snap/lxd/19188",
>                 "options": "ro,nodev,relatime",
>                 "size_available": 0,
>                 "size_total": 73400320,
>                 "uuid": "N/A"
>             }
>         ],
>         "ansible_nodename": "ubuntu2004",
>         "ansible_processor": [
>             "0",
>             "GenuineIntel",
>             "Intel Xeon Processor (Skylake, IBRS)",
>             "1",
>             "GenuineIntel",
>             "Intel Xeon Processor (Skylake, IBRS)",
>             "2",
>             "GenuineIntel",
>             "Intel Xeon Processor (Skylake, IBRS)",
>             "3",
>             "GenuineIntel",
>             "Intel Xeon Processor (Skylake, IBRS)"
>         ],
>         "ansible_processor_cores": 1,
>         "ansible_processor_count": 4,
>         "ansible_processor_threads_per_core": 1,
>         "ansible_processor_vcpus": 4,
>         "ansible_product_name": "Standard PC (Q35 + ICH9, 2009)",
>         "ansible_product_serial": "NA",
>         "ansible_product_uuid": "64d7747e-869a-45b0-9d0a-ae9a6d463611",
>         "ansible_product_version": "pc-q35-2.11",
>         "ansible_python": {
>             "executable": "/usr/bin/python3",
>             "has_sslcontext": true,
>             "type": "cpython",
>             "version": {
>                 "major": 3,
>                 "micro": 5,
>                 "minor": 8,
>                 "releaselevel": "final",
>                 "serial": 0
>             },
>             "version_info": [
>                 3,
>                 8,
>                 5,
>                 "final",
>                 0
>             ]
>         },
>         "ansible_python_version": "3.8.5",
>         "ansible_real_group_id": 0,
>         "ansible_real_user_id": 0,
>         "ansible_selinux": {
>             "status": "Missing selinux Python library"
>         },
>         "ansible_selinux_python_present": false,
>         "ansible_service_mgr": "systemd",
>         "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANcblFlURYNVXrHiZ2ozUgS6NQWkL9q6dKRvhFV75WjqBQfZs4wAAd9qYdT/fAJfT+MHdaeKgAzIgCCEH0lwEOVJY5go1u3AOEuq6S2b9D1Tr6VufAjuVYuDbPqYCoYPMDepsgKfJIxLGcfs0SgaeJyCzKOh5prQrDfYPHIP3NRjAAAAFQDc3uAPrXGgg7q74VaX8yMC5evKjwAAAIEAy+8TP/tI05oSLikv6L5es6J/iIXouuCSSlpzYT+ZcA64PaXB7X9ziRUOF79fbWkVYGmCRutjayucFHfsnICwm17vLaA5Pdc18hKqgO64HLhX1fBf8BE3KKQFY2nqcop0ShRHsLHWoL5E8SJ0Jrjd+wqw/0SQ4EnxxdmW7mrf+KUAAACAeWRshM/sCGP/DDifYusYkhZ85d5vgeXK/h9d4V3WhnXa6TlNPTo7Y21rX842UJ8npSf+ZVZb9iRJMRxGJiGgQK3GPRvdopQPFM9Y+kTf4GfLS5Bmd4RdZXF0POEpe10xc0ewg5is9NsFnJI+mJFcEB9FH+TtS0T7PmP+l9ADkTs=",
>         "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGumKEwBRlMPpWemu3oyScRXbm4/dH+q5iCKvhB4EsehElxsVTQXbNjQyv5Ei38yG34N2q5DvZSus+tD8LJEZW4=",
>         "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIBJpeIq8MEf3YBN7NLxd/ss/iqbvH9q34eLjYP0tubup",
>         "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABgQC8hyqeCETLm6kd/kG9lRy9HWIBFFRQTlsIUYdBDmb8dcA0Ye6JwcGhFbEJD5KaWKmyul0OP0dmV4BfLdf1dzDvilh0vTfgTTklbsPpEEjlfstqLHpKDZ4wL+Gj8eF54xW00oFwSR68CWNomyR0YrTczsN/CUb5HSejvYS48OzRP+it4iTyrlwVp8Lb7O7m/TnQFbys8uTaFFNpXFm4WrBtK0HlqVI/9LASXnuYqudCOgwkGlKamVnSwCO3Bt8MXFdhkgvXqoEp0sCdGZIM207jrN42hy6stXyjvn/43YbfTAiXwJDPUhllpbuUSTRF3zzlIvHbC0JRwq0wGd+eXS5kb9RS6v5QLptn0pA8kxQYg2uqO4I+Uc0R7akmgPu1S85jobS7MJIpZmNj57fGmUC7ZUvYTQ97lXcWrfNzk4pwl9TG85U/tQNwN6X5TmaFuSkqGSVRb+a86Z62//BH6lY8sOPEn+Ou883l3QBXjSkgQIjpWKy30GlUcd8Mn6nsgU0=",
>         "ansible_swapfree_mb": 3934,
>         "ansible_swaptotal_mb": 3934,
>         "ansible_system": "Linux",
>         "ansible_system_capabilities": [],
>         "ansible_system_capabilities_enforced": "False",
>         "ansible_system_vendor": "QEMU",
>         "ansible_uptime_seconds": 12500,
>         "ansible_user_dir": "/root",
>         "ansible_user_gecos": "root",
>         "ansible_user_gid": 0,
>         "ansible_user_id": "root",
>         "ansible_user_shell": "/bin/bash",
>         "ansible_user_uid": 0,
>         "ansible_userspace_architecture": "x86_64",
>         "ansible_userspace_bits": "64",
>         "ansible_virtualization_role": "guest",
>         "ansible_virtualization_type": "kvm",
>         "gather_subset": [
>             "all"
>         ],
>         "module_setup": true
>     },
>     "changed": false
> }
> 
>
Hi Alex,

Thanks!  I've compared this to the output I get when running "ansible"
and connecting to an Ubuntu 20.04 VM, and it looks pretty much the
same... *but* the distribution related fields.

Thinking it could be an issue with the *ansible* code itself (maybe a
bug in a given version) I then went on and tried the following on
both x86_64 and aarch64:

  podman run --rm -ti ubuntu:20.04 /bin/sh -c 'apt update && apt -y install ansible && ansible -c local -i 127.0.0.1, -m setup all'

And the distribution keys are properly reported.  So both the ansible
code I'm running, and ansible shipped with Ubuntu 20.04 seem fine.

The next target for testing is the exact image you're using.  Ansible
probes the distribution largely based on the a "/etc/*release*" like
file[1], so I'd like to know how to replicate your machine.  Are you
using a cloud image?  Installing a given profile?  Is the actual image
something you could share?

Thanks,
- Cleber.

[1] - https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Wainer dos Santos Moschetta]

Hi,

On 2/23/21 2:45 PM, Daniel P. Berrangé wrote:
> On Tue, Feb 23, 2021 at 11:47:18AM -0500, Cleber Rosa wrote:
>> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
>>> On 2/23/21 12:25 PM, Thomas Huth wrote:
>>>> On 19/02/2021 22.58, Cleber Rosa wrote:
>>>>> As described in the included documentation, the "custom runner" jobs
>>>>> extend the GitLab CI jobs already in place.  One of their primary
>>>>> goals of catching and preventing regressions on a wider number of host
>>>>> systems than the ones provided by GitLab's shared runners.
>>>>>
>>>>> This sets the stage in which other community members can add their own
>>>>> machine configuration documentation/scripts, and accompanying job
>>>>> definitions.  As a general rule, those newly added contributed jobs
>>>>> should run as "non-gating", until their reliability is verified (AKA
>>>>> "allow_failure: true").
>>>>>
>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>    .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>>>>>    .gitlab-ci.yml                  |  1 +
>>>>>    docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>>>>>    docs/devel/index.rst            |  1 +
>>>>>    4 files changed, 44 insertions(+)
>>>>>    create mode 100644 .gitlab-ci.d/custom-runners.yml
>>>>>    create mode 100644 docs/devel/ci.rst
>>>>>
>>>>> diff --git a/.gitlab-ci.d/custom-runners.yml
>>>>> b/.gitlab-ci.d/custom-runners.yml
>>>>> new file mode 100644
>>>>> index 0000000000..3004da2bda
>>>>> --- /dev/null
>>>>> +++ b/.gitlab-ci.d/custom-runners.yml
>>>>> @@ -0,0 +1,14 @@
>>>>> +# The CI jobs defined here require GitLab runners installed and
>>>>> +# registered on machines that match their operating system names,
>>>>> +# versions and architectures.  This is in contrast to the other CI
>>>>> +# jobs that are intended to run on GitLab's "shared" runners.
>>>>> +
>>>>> +# Different than the default approach on "shared" runners, based on
>>>>> +# containers, the custom runners have no such *requirement*, as those
>>>>> +# jobs should be capable of running on operating systems with no
>>>>> +# compatible container implementation, or no support from
>>>>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
>>>>> +# reusing the GIT repository, let's enable the recursive submodule
>>>>> +# strategy.
>>>>> +variables:
>>>>> +  GIT_SUBMODULE_STRATEGY: recursive
>>>> Is it really necessary? I thought our configure script would take care
>>>> of the submodules?
>> I've done a lot of testing on bare metal systems, and the problems
>> that come from reusing the same system and failed cleanups can be very
>> frustrating.  It's unfortunate that we need this, but it was the
>> simplest and most reliable solution I found.  :/
> Hmmm, this makes it sound like the job is not being run in a
> fresh pristine checkout. IMHO we need to guarantee that in
> general, at which point submodules should "just work", unless
> the running is blocking network access ?

Setting the git strategy may work out:

https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy

- Wainer

>
>
>
> Regards,
> Daniel

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Philippe Mathieu-Daudé]

On 2/23/21 7:09 PM, Cleber Rosa wrote:
> On Tue, Feb 23, 2021 at 06:34:07PM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/23/21 6:24 PM, Philippe Mathieu-Daudé wrote:
>>> On 2/23/21 5:47 PM, Cleber Rosa wrote:
>>>> On Tue, Feb 23, 2021 at 05:37:04PM +0100, Philippe Mathieu-Daudé wrote:
>>>>> On 2/23/21 12:25 PM, Thomas Huth wrote:
>>>>>> On 19/02/2021 22.58, Cleber Rosa wrote:
>>>>>>> As described in the included documentation, the "custom runner" jobs
>>>>>>> extend the GitLab CI jobs already in place.  One of their primary
>>>>>>> goals of catching and preventing regressions on a wider number of host
>>>>>>> systems than the ones provided by GitLab's shared runners.
>>>>>>>
>>>>>>> This sets the stage in which other community members can add their own
>>>>>>> machine configuration documentation/scripts, and accompanying job
>>>>>>> definitions.  As a general rule, those newly added contributed jobs
>>>>>>> should run as "non-gating", until their reliability is verified (AKA
>>>>>>> "allow_failure: true").
>>>>>>>
>>>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>>>> ---
>>>>>>>   .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
>>>>>>>   .gitlab-ci.yml                  |  1 +
>>>>>>>   docs/devel/ci.rst               | 28 ++++++++++++++++++++++++++++
>>>>>>>   docs/devel/index.rst            |  1 +
>>>>>>>   4 files changed, 44 insertions(+)
>>>>>>>   create mode 100644 .gitlab-ci.d/custom-runners.yml
>>>>>>>   create mode 100644 docs/devel/ci.rst
>>>>>>>
>>>>>>> diff --git a/.gitlab-ci.d/custom-runners.yml
>>>>>>> b/.gitlab-ci.d/custom-runners.yml
>>>>>>> new file mode 100644
>>>>>>> index 0000000000..3004da2bda
>>>>>>> --- /dev/null
>>>>>>> +++ b/.gitlab-ci.d/custom-runners.yml
>>>>>>> @@ -0,0 +1,14 @@
>>>>>>> +# The CI jobs defined here require GitLab runners installed and
>>>>>>> +# registered on machines that match their operating system names,
>>>>>>> +# versions and architectures.  This is in contrast to the other CI
>>>>>>> +# jobs that are intended to run on GitLab's "shared" runners.
>>>>>>> +
>>>>>>> +# Different than the default approach on "shared" runners, based on
>>>>>>> +# containers, the custom runners have no such *requirement*, as those
>>>>>>> +# jobs should be capable of running on operating systems with no
>>>>>>> +# compatible container implementation, or no support from
>>>>>>> +# gitlab-runner.  To avoid problems that gitlab-runner can cause while
>>>>>>> +# reusing the GIT repository, let's enable the recursive submodule
>>>>>>> +# strategy.
>>>>>>> +variables:
>>>>>>> +  GIT_SUBMODULE_STRATEGY: recursive
>>>>>>
>>>>>> Is it really necessary? I thought our configure script would take care
>>>>>> of the submodules?
>>>>>
>>>>
>>>> I've done a lot of testing on bare metal systems, and the problems
>>>> that come from reusing the same system and failed cleanups can be very
>>>> frustrating.  It's unfortunate that we need this, but it was the
>>>> simplest and most reliable solution I found.  :/
>>>>
>>>> Having said that, I noticed after I posted this series that this is
>>>> affecting all other jobs.  We don't need it that in the jobs based
>>>> on containers (for obvious reasons), so I see two options:
>>>>
>>>> 1) have it enabled on all jobs for consistency
>>>>
>>>> 2) have it enabled only on jobs that will reuse the repo
>>>>
>>>>> Well, if there is a failure during the first clone (I got one network
>>>>> timeout in the middle) 
>>>
>>> [This network failure is pasted at the end]
>>>
>>>>> then next time it doesn't work:
>>>>>
>>>>> Updating/initializing submodules recursively...
>>>>> Synchronizing submodule url for 'capstone'
>>>>> Synchronizing submodule url for 'dtc'
>>>>> Synchronizing submodule url for 'meson'
>>>>> Synchronizing submodule url for 'roms/QemuMacDrivers'
>>>>> Synchronizing submodule url for 'roms/SLOF'
>>>>> Synchronizing submodule url for 'roms/edk2'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/esaxx'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/BaseTools/Source/C/BrotliCompress/brotli/research/libdivsufsort'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/MdeModulePkg/Universal/RegularExpressionDxe/oniguruma'
>>>>> Synchronizing submodule url for
>>>>> 'roms/edk2/UnitTestFrameworkPkg/Library/CmockaLib/cmocka'
>>
>> So far, beside the repository useful for QEMU, I cloned:
>>
>> - boringssl
>> - krb5
>> - pyca-cryptography
>> - esaxx
>> - libdivsufsort
>> - oniguruma
>> - openssl
>> - brotli
>> - cmocka
>>
> 
> Hi Phil,
> 
> I'm not following what you meant by "I cloned"... Are you experimenting
> with this on a machine of your own and manually cloning the submodules?

I meant "my test runner has been cloning ..."

>> But reach the runner time limit of 2h.

The first failure was 1h, I raised the job limit to the maximum
I could use for this runner, 2h.

>> The directory reports 3GB of source code.
>>
>> I don't think the series has been tested enough before posting,
> 
> Please take into consideration that this series, although simple in
> content, touches and interacts with a lot of moving pieces, and
> possibly with personal systems that I did not have, or will have,
> access to.  As far as public testing proof goes, you can see a
> pipeline here with this version of this series here:
> 
>    https://gitlab.com/cleber.gnu/qemu/-/pipelines/258982039/builds

Expand the timeout and retry the same job on the same runner
various times:

diff --git a/.gitlab-ci.d/custom-runners.yml
b/.gitlab-ci.d/custom-runners.yml
@@ -17,6 +17,7 @@ variables:
 # setup by the scripts/ci/setup/build-environment.yml task
 # "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
 ubuntu-18.04-s390x-all-linux-static:
+ timeout: 2h 30m
  allow_failure: true
  needs: []
  stage: build

Each time it will clone more submodules.

I stopped at the 3rd intent.

> As I said elsewhere, I only noticed the recursive submodule being
> applied to the existing jobs after I submitted the series.  Mea culpa.
> But:
> 
>  * none of the jobs took noticeably longer than the previous baseline
>  * there was one *container build failure* (safe to say it's not
>    related)
>  * all other jobs passed successfully

I had less luck then (see the docker-dind jobs started on the custom
runner commented elsewhere in this thread).

> And, along with the previous versions, this series were tested on all
> the previously included architectures and operating systems.  It's
> unfortunate that because of your experience at this time (my
> apologies), you don't realize the amount of testing done so far.

As I commented to Erik on IRC, the single difference I did
is use the distribution runner, not the official one:

$ sudo apt-get install gitlab-runner docker.io

Then registered changing the path (/usr/bin/gitlab-runner instead
of /usr/local/bin/gitlab-runner). Everything else left unchanged.

>> I'm stopping here my experiments.
>>
>> Regards,
>>
>> Phil.
>>
> 
> I honestly appreciate your help here up to this point.
> 
> Regards,
> - Cleber.
> 

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Philippe Mathieu-Daudé]

On 2/23/21 7:25 PM, Cleber Rosa wrote:
> On Tue, Feb 23, 2021 at 03:56:19PM +0000, Daniel P. Berrangé wrote:
>>
>> Urgh, well that's a big problem. We certainly don't want *anything* being
>> placed on the custom runners without explicit opt-in, otherwise jobs run
>> in the main repo have a different environment from when users run on their
>> personal forks.
>>
>> IOW, we need anti-affinity against our custom runners really.
>>
> 
> I'm assuming Phil missed that documentation, because that's a
> non-issue, really.
> 
> Just unchecking the "Run untagged jobs" check box on the runner
> configuration makes sure that the custom runners won't pickup any jobs
> not *specifically* tagged for them.

Can we explicit this when registering the runner instead of having to
access the WebUI?

$ gitlab-runner register --help

   --run-untagged
     Register to run untagged builds; defaults to 'true'
     when 'tag-list' is empty [$REGISTER_RUN_UNTAGGED]

Re: [PATCH v5 1/4] Jobs based on custom runners: documentation and configuration placeholder

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 3844 bytes --]

On Wed, Feb 24, 2021 at 12:57:52PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/23/21 7:09 PM, Cleber Rosa wrote:
> > Hi Phil,
> > 
> > I'm not following what you meant by "I cloned"... Are you experimenting
> > with this on a machine of your own and manually cloning the submodules?
> 
> I meant "my test runner has been cloning ..."
> 
> >> But reach the runner time limit of 2h.
> 
> The first failure was 1h, I raised the job limit to the maximum
> I could use for this runner, 2h.
> 
> >> The directory reports 3GB of source code.
> >>
> >> I don't think the series has been tested enough before posting,
> > 
> > Please take into consideration that this series, although simple in
> > content, touches and interacts with a lot of moving pieces, and
> > possibly with personal systems that I did not have, or will have,
> > access to.  As far as public testing proof goes, you can see a
> > pipeline here with this version of this series here:
> > 
> >    https://gitlab.com/cleber.gnu/qemu/-/pipelines/258982039/builds
> 
> Expand the timeout and retry the same job on the same runner
> various times:
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml
> b/.gitlab-ci.d/custom-runners.yml
> @@ -17,6 +17,7 @@ variables:
>  # setup by the scripts/ci/setup/build-environment.yml task
>  # "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
>  ubuntu-18.04-s390x-all-linux-static:
> + timeout: 2h 30m
>   allow_failure: true
>   needs: []
>   stage: build
> 
> Each time it will clone more submodules.
> 
> I stopped at the 3rd intent.
> 
> > As I said elsewhere, I only noticed the recursive submodule being
> > applied to the existing jobs after I submitted the series.  Mea culpa.
> > But:
> > 
> >  * none of the jobs took noticeably longer than the previous baseline
> >  * there was one *container build failure* (safe to say it's not
> >    related)
> >  * all other jobs passed successfully
> 
> I had less luck then (see the docker-dind jobs started on the custom
> runner commented elsewhere in this thread).
>

Hi Phil,

I replied to this issue elsewhere too... I assume you missed the
documentation and did not uncheck the "Run untagged jobs" as
instructed.

> > And, along with the previous versions, this series were tested on all
> > the previously included architectures and operating systems.  It's
> > unfortunate that because of your experience at this time (my
> > apologies), you don't realize the amount of testing done so far.
> 
> As I commented to Erik on IRC, the single difference I did
> is use the distribution runner, not the official one:
> 
> $ sudo apt-get install gitlab-runner docker.io
> 
> Then registered changing the path (/usr/bin/gitlab-runner instead
> of /usr/local/bin/gitlab-runner). Everything else left unchanged.
>

Assuming you did your experiments on Ubuntu 20.04:

   # dpkg -l gitlab-runner
  ||/ Name           Version              Architecture Description
  +++-==============-====================-============-=====================================================
  ii  gitlab-runner  11.2.0+dfsg-2ubuntu1 amd64        GitLab Runner - runs continuous integration (CI) jobs

This supposedly "single" difference, actually amounts to thousands of
changes (not counting the possible downstream patches, differences with
regards to packaging, etc):

  [gitlab-runner]$ git log --no-merges --oneline v11.2.0..v13.1.1 | wc -l
  1477

Version 13.1.1 referred above is what you'd get *if* using the
playbook.

Like I said before, I very much appreciate your help reviewing this,
but unfortunately what you used was *WAY OFF* what was proposed.

And you're right, this version was not tested enough (on an
environment similar to what you used) before it was posted.

Regards,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 4/4] Jobs based on custom runners: add job definitions for QEMU's machines

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

On Wed, Feb 24, 2021 at 01:00:54PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/23/21 7:25 PM, Cleber Rosa wrote:
> > On Tue, Feb 23, 2021 at 03:56:19PM +0000, Daniel P. Berrangé wrote:
> >>
> >> Urgh, well that's a big problem. We certainly don't want *anything* being
> >> placed on the custom runners without explicit opt-in, otherwise jobs run
> >> in the main repo have a different environment from when users run on their
> >> personal forks.
> >>
> >> IOW, we need anti-affinity against our custom runners really.
> >>
> > 
> > I'm assuming Phil missed that documentation, because that's a
> > non-issue, really.
> > 
> > Just unchecking the "Run untagged jobs" check box on the runner
> > configuration makes sure that the custom runners won't pickup any jobs
> > not *specifically* tagged for them.
> 
> Can we explicit this when registering the runner instead of having to
> access the WebUI?
> 
> $ gitlab-runner register --help
> 
>    --run-untagged
>      Register to run untagged builds; defaults to 'true'
>      when 'tag-list' is empty [$REGISTER_RUN_UNTAGGED]
> 

Sure thing, I will change the default behavior on the next version.

Thanks,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Philippe Mathieu-Daudé]

Hi Cleber,

On 2/19/21 10:58 PM, Cleber Rosa wrote:
> TL;DR: this should allow the QEMU maintainer to push to the staging
> branch, and have custom jobs running on the project's aarch64 and
> s390x machines.  Jobs in this version are allowed to fail, to allow
> for the inclusion of the novel machines/jobs without CI disruption.
> Simple usage looks like:
> 
>    git push remote staging
>    ./scripts/ci/gitlab-pipeline-status --verbose --wait
> 
> Long version:
> 
> The idea about a public facing Gating CI for QEMU was summarized in an
> RFC[1].  Since then, it was decided that a simpler version should be
> attempted first.
> 
> At this point, there are two specific runners (an aarch64 and an s390x)
> registered with GitLab, at https://gitlab.com/qemu-project, currently
> setup to the "qemu" repository.

Our CI is heavily based on containerized testing, your scripts/document
don't cover that.

Should we document how to install a container service (we mostly
use Docker and Podman)?

Or should we simply explicit these are only "native" runners and
container support will be considered later eventually?

Regards,

Phil.

Re: [PATCH v5 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Philippe Mathieu-Daudé]

On 3/5/21 11:14 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 2/19/21 10:58 PM, Cleber Rosa wrote:
>> TL;DR: this should allow the QEMU maintainer to push to the staging
>> branch, and have custom jobs running on the project's aarch64 and
>> s390x machines.  Jobs in this version are allowed to fail, to allow
>> for the inclusion of the novel machines/jobs without CI disruption.
>> Simple usage looks like:
>>
>>    git push remote staging
>>    ./scripts/ci/gitlab-pipeline-status --verbose --wait
>>
>> Long version:
>>
>> The idea about a public facing Gating CI for QEMU was summarized in an
>> RFC[1].  Since then, it was decided that a simpler version should be
>> attempted first.
>>
>> At this point, there are two specific runners (an aarch64 and an s390x)
>> registered with GitLab, at https://gitlab.com/qemu-project, currently
>> setup to the "qemu" repository.

Also we are interested in testing virtualization with these runners.

If KVM is available, we need to document the gitlab-runner user needs
to be in the KVM group, and it would be helpful to have a 'kvm' tag
in the runner taglist, so we could assign specific jobs to these
runners.

> Our CI is heavily based on containerized testing, your scripts/document
> don't cover that.
> 
> Should we document how to install a container service (we mostly
> use Docker and Podman)?
> 
> Or should we simply explicit these are only "native" runners and
> container support will be considered later eventually?
> 
> Regards,
> 
> Phil.
> 

Re: [PATCH v5 0/4] GitLab Custom Runners and Jobs (was: QEMU Gating CI)

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> TL;DR: this should allow the QEMU maintainer to push to the staging
> branch, and have custom jobs running on the project's aarch64 and
> s390x machines.  Jobs in this version are allowed to fail, to allow
> for the inclusion of the novel machines/jobs without CI disruption.
> Simple usage looks like:
>
>    git push remote staging
>    ./scripts/ci/gitlab-pipeline-status --verbose --wait
>
> Long version:
>
> The idea about a public facing Gating CI for QEMU was summarized in an
> RFC[1].  Since then, it was decided that a simpler version should be
> attempted first.
>
> At this point, there are two specific runners (an aarch64 and an s390x)
> registered with GitLab, at https://gitlab.com/qemu-project, currently
> setup to the "qemu" repository.
>
> Changes from v4:

Was there a v6 I missed?

-- 
Alex Bennée