* [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI
@ 2021-06-25 17:22 Daniel P. Berrangé
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
` (4 more replies)
0 siblings, 5 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-25 17:22 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Daniel P. Berrangé,
Alex Bennée, Li-Wen Hsu
Currently the Cirrus CI pipelines are completely separate from the
GitLab CI pipelines. This means contributors/maintainers have to
monitor two distinct places.
This series uses the 'cirrus-run' tool from within a GitLab CI job
to invoke a Cirrus CI job. Effectively Cirrus CI becomes a custom
runner for GitLab in this way.
The GitLab CI pipeline nows shows a complete picture for CI results
covering GitLab, Travis and Cirrus jobs.
The implementation used here is a direct copy of what is done in
libvirt with cirrus-run. In fact the jobs are using the container
image published by https://gitlab.com/libvirt/libvirt-ci for
getting the 'cirrus-run' tool.
Second, we use the 'lcitool' program to generate the
.gitlab-ci.d/cirrus/*.vars files which hold the package lists to
be installed on FreeBSD and macOS. As a result of the work done
with lcitool to generate the dockerfiles, this was easy to extend
to Cirrus CI, and as a result this new Cirrus config installs a
massively larger set of packages. Thus our testing coverage on
macOS and FreeBSD improves.
The MSys Windows job still remains in the .cirrus.yml file. This
can be addressed to, if we extend libvirt-ci to have package
mapping information for MSys.
Using this setup does require the contributor to do a one time
configuration task. They need to modify gitlab CI settings to
add two environment variables, providing the API token for the
Cirrus CI REST API.
I have a demo pipeline here showing the 4 cirrus CI jobs:
https://gitlab.com/berrange/qemu/-/pipelines/327362404
Note in this pipeline above, I temporarily disabled all the
normal GitLab CI jobs to reduce burden when I was debugging.
An earlier pipeline shows the full set of jobs:
https://gitlab.com/berrange/qemu/-/pipelines/327324780
This series adds cirrus as always on jobs, but we should probably
use 'allow_failure: true' for a few weeks to demonstrate that
they are stable enough to be part of the main gating set.
Daniel P. Berrangé (3):
build: validate that system capstone works before using it
gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
cirrus: delete FreeBSD and macOS jobs
.cirrus.yml | 55 ---------------
.gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
.gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
.gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
.gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
.gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
.gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
.gitlab-ci.d/qemu-project.yml | 1 +
meson.build | 13 ++++
9 files changed, 247 insertions(+), 55 deletions(-)
create mode 100644 .gitlab-ci.d/cirrus.yml
create mode 100644 .gitlab-ci.d/cirrus/README.rst
create mode 100644 .gitlab-ci.d/cirrus/build.yml
create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
--
2.31.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/3] build: validate that system capstone works before using it
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
@ 2021-06-25 17:22 ` Daniel P. Berrangé
2021-06-28 6:54 ` Thomas Huth
` (2 more replies)
2021-06-25 17:22 ` [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Daniel P. Berrangé
` (3 subsequent siblings)
4 siblings, 3 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-25 17:22 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Daniel P. Berrangé,
Alex Bennée, Li-Wen Hsu
Some versions of capstone have shipped a broken pkg-config file which
puts the -I path without the trailing '/capstone' suffix. This breaks
the ability to "#include <capstone.h>". Upstream and most distros have
fixed this, but a few stragglers remain, notably FreeBSD.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
meson.build | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/meson.build b/meson.build
index d8a92666fb..9979ddae71 100644
--- a/meson.build
+++ b/meson.build
@@ -1425,6 +1425,19 @@ if capstone_opt in ['enabled', 'auto', 'system']
kwargs: static_kwargs, method: 'pkg-config',
required: capstone_opt == 'system' or
capstone_opt == 'enabled' and not have_internal)
+
+ # Some versions of capstone have broken pkg-config file
+ # that reports a wrong -I path, causing the #include to
+ # fail later. If the system has such a broken version
+ # do not use it.
+ if capstone.found() and not cc.compiles('#include <capstone.h>',
+ dependencies: [capstone])
+ capstone = not_found
+ if capstone_opt == 'system'
+ error('system capstone requested, it it does not appear to work')
+ endif
+ endif
+
if capstone.found()
capstone_opt = 'system'
elif have_internal
--
2.31.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
@ 2021-06-25 17:22 ` Daniel P. Berrangé
2021-06-28 7:28 ` Thomas Huth
2021-06-30 18:58 ` Wainer dos Santos Moschetta
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
` (2 subsequent siblings)
4 siblings, 2 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-25 17:22 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Daniel P. Berrangé,
Alex Bennée, Li-Wen Hsu
This adds support for running 4 jobs via Cirrus CI runners:
* FreeBSD 12
* FreeBSD 13
* macOS 11 with default XCode
* macOS 11 with latest XCode
The gitlab job uses a container published by the libvirt-ci
project (https://gitlab.com/libvirt/libvirt-ci) that contains
the 'cirrus-run' command. This accepts a short yaml file that
describes a single Cirrus CI job, runs it using the Cirrus CI
REST API, and reports any output to the console.
In this way Cirrus CI is effectively working as an indirect
custom runner for GitLab CI pipelines. The key benefit is that
Cirrus CI job results affect the GitLab CI pipeline result and
so the user only has look at one CI dashboard.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
.gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
.gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
.gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
.gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
.gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
.gitlab-ci.d/qemu-project.yml | 1 +
7 files changed, 234 insertions(+)
create mode 100644 .gitlab-ci.d/cirrus.yml
create mode 100644 .gitlab-ci.d/cirrus/README.rst
create mode 100644 .gitlab-ci.d/cirrus/build.yml
create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
new file mode 100644
index 0000000000..d7b4cce79b
--- /dev/null
+++ b/.gitlab-ci.d/cirrus.yml
@@ -0,0 +1,103 @@
+# Jobs that we delegate to Cirrus CI because they require an operating
+# system other than Linux. These jobs will only run if the required
+# setup has been performed on the GitLab account.
+#
+# The Cirrus CI configuration is generated by replacing target-specific
+# variables in a generic template: some of these variables are provided
+# when the GitLab CI job is defined, others are taken from a shell
+# snippet generated using lcitool.
+#
+# Note that the $PATH environment variable has to be treated with
+# special care, because we can't just override it at the GitLab CI job
+# definition level or we risk breaking it completely.
+.cirrus_build_job:
+ stage: build
+ image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
+ needs: []
+ script:
+ - source .gitlab-ci.d/cirrus/$NAME.vars
+ - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
+ -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
+ -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
+ -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
+ -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
+ -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
+ -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
+ -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
+ -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
+ -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
+ -e "s|[@]PKGS@|$PKGS|g"
+ -e "s|[@]MAKE@|$MAKE|g"
+ -e "s|[@]PYTHON@|$PYTHON|g"
+ -e "s|[@]PIP3@|$PIP3|g"
+ -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
+ -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
+ -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
+ <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
+ - cat .gitlab-ci.d/cirrus/$NAME.yml
+ - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
+ rules:
+ - if: "$TEMPORARILY_DISABLED"
+ allow_failure: true
+ - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+
+x64-freebsd-12-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: freebsd-12
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-12-2
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
+ UPDATE_COMMAND: pkg update
+ INSTALL_COMMAND: pkg install -y
+ # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
+ # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
+ CONFIGURE_ARGS: --disable-gnutls
+ TEST_TARGETS: check
+
+x64-freebsd-13-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: freebsd-13
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-13-0
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
+ UPDATE_COMMAND: pkg update
+ INSTALL_COMMAND: pkg install -y
+ TEST_TARGETS: check
+
+x64-macos-11-base-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: macos-11
+ CIRRUS_VM_INSTANCE_TYPE: osx_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image
+ CIRRUS_VM_IMAGE_NAME: big-sur-base
+ CIRRUS_VM_CPUS: 12
+ CIRRUS_VM_RAM: 24G
+ UPDATE_COMMAND: brew update
+ INSTALL_COMMAND: brew install
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
+
+x64-macos-11-xcode-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: macos-11
+ CIRRUS_VM_INSTANCE_TYPE: osx_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image
+ CIRRUS_VM_IMAGE_NAME: big-sur-xcode
+ CIRRUS_VM_CPUS: 12
+ CIRRUS_VM_RAM: 24G
+ UPDATE_COMMAND: brew update
+ INSTALL_COMMAND: brew install
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
new file mode 100644
index 0000000000..657b0706d7
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/README.rst
@@ -0,0 +1,54 @@
+Cirrus CI integration
+=====================
+
+GitLab CI shared runners only provide a docker environment running on Linux.
+While it is possible to provide private runners for non-Linux platforms this
+is not something most contributors/maintainers will wish to do.
+
+To work around this limitation, we take advantage of `Cirrus CI`_'s free
+offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
+CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
+the main GitLab CI pipeline dashboard.
+
+There is, however, some one-time setup required. If you want FreeBSD and macOS
+builds to happen when you push to your GitLab repository, you need to
+
+* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
+ This repository needs to exist for cirrus-run to work, but it doesn't need to
+ be kept up to date, so you can create it and then forget about it;
+
+* enable the `Cirrus CI GitHub app`_ for your GitHub account;
+
+* sign up for Cirrus CI. It's enough to log into the website using your GitHub
+ account;
+
+* grab an API token from the `Cirrus CI settings`_ page;
+
+* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
+ for Cirrus CI to properly recognize the project. You can check whether
+ Cirrus CI knows about your project by navigating to:
+
+ ``https://cirrus-ci.com/yourusername/qemu``
+
+* in the *CI/CD / Variables* section of the settings page for your GitLab
+ repository, create two new variables:
+
+ * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
+ created earlier, eg. ``yourusername/qemu``;
+
+ * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
+ This variable **must** be marked as *Masked*, because anyone with knowledge
+ of it can impersonate you as far as Cirrus CI is concerned.
+
+ Neither of these variables should be marked as *Protected*, because in
+ general you'll want to be able to trigger Cirrus CI builds from non-protected
+ branches.
+
+Once this one-time setup is complete, you can just keep pushing to your GitLab
+repository as usual and you'll automatically get the additional CI coverage.
+
+
+.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
+.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
+.. _Cirrus CI: https://cirrus-ci.com/
+.. _cirrus-run: https://github.com/sio/cirrus-run/
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
new file mode 100644
index 0000000000..857bdc5536
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -0,0 +1,35 @@
+@CIRRUS_VM_INSTANCE_TYPE@:
+ @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
+ cpu: @CIRRUS_VM_CPUS@
+ memory: @CIRRUS_VM_RAM@
+
+env:
+ CIRRUS_CLONE_DEPTH: 1
+ CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
+ CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
+ CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
+ PATH: "@PATH@"
+ PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
+ PYTHON: "@PYTHON@"
+ MAKE: "@MAKE@"
+ CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
+
+build_task:
+ install_script:
+ - @UPDATE_COMMAND@
+ - @INSTALL_COMMAND@ @PKGS@
+ - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
+ clone_script:
+ - git clone --depth 100 "$CI_REPOSITORY_URL" .
+ - git fetch origin "$CI_COMMIT_REF_NAME"
+ - git reset --hard "$CI_COMMIT_SHA"
+ build_script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-werror $CONFIGURE_ARGS
+ || { cat config.log meson-logs/meson-log.txt; exit 1; }
+ - $MAKE -j$(sysctl -n hw.ncpu)
+ - for TARGET in $TEST_TARGETS ;
+ do
+ $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
+ done
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
new file mode 100644
index 0000000000..5e51e50ea9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables freebsd-12 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
new file mode 100644
index 0000000000..07716833d9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables freebsd-13 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars
new file mode 100644
index 0000000000..cbec8a44a3
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/macos-11.vars
@@ -0,0 +1,15 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables macos-11 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='brew'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip3'
+PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
+PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv'
+CPAN_PKGS='Test::Harness'
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..35d4e62c12 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
- local: '/.gitlab-ci.d/crossbuilds.yml'
- local: '/.gitlab-ci.d/buildtest.yml'
- local: '/.gitlab-ci.d/static_checks.yml'
+ - local: '/.gitlab-ci.d/cirrus.yml'
--
2.31.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
2021-06-25 17:22 ` [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Daniel P. Berrangé
@ 2021-06-25 17:22 ` Daniel P. Berrangé
2021-06-28 7:29 ` Thomas Huth
` (2 more replies)
2021-06-28 7:34 ` [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Thomas Huth
2021-07-05 10:32 ` Alex Bennée
4 siblings, 3 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-25 17:22 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Daniel P. Berrangé,
Alex Bennée, Li-Wen Hsu
The builds for these two platforms can now be performed from GitLab CI
using cirrus-run.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.cirrus.yml | 55 -----------------------------------------------------
1 file changed, 55 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f4bf49b704..02c43a074a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,61 +1,6 @@
env:
CIRRUS_CLONE_DEPTH: 1
-freebsd_12_task:
- freebsd_instance:
- image_family: freebsd-12-2
- cpu: 8
- memory: 8G
- install_script:
- - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ;
- - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed
- nettle perl5 pixman pkgconf png usbredir ninja
- script:
- - mkdir build
- - cd build
- # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
- # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
- - ../configure --enable-werror --disable-gnutls
- || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake -j$(sysctl -n hw.ncpu) check V=1
-
-macos_task:
- osx_instance:
- image: catalina-base
- install_script:
- - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja
- script:
- - mkdir build
- - cd build
- - ../configure --python=/usr/local/bin/python3 --enable-werror
- --extra-cflags='-Wno-error=deprecated-declarations'
- || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake check-unit V=1
- - gmake check-block V=1
- - gmake check-qapi-schema V=1
- - gmake check-softfloat V=1
- - gmake check-qtest-x86_64 V=1
-
-macos_xcode_task:
- osx_instance:
- # this is an alias for the latest Xcode
- image: catalina-xcode
- install_script:
- - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja
- script:
- - mkdir build
- - cd build
- - ../configure --extra-cflags='-Wno-error=deprecated-declarations' --enable-modules
- --enable-werror --cc=clang || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake check-unit V=1
- - gmake check-block V=1
- - gmake check-qapi-schema V=1
- - gmake check-softfloat V=1
- - gmake check-qtest-x86_64 V=1
-
windows_msys2_task:
timeout_in: 90m
windows_container:
--
2.31.1
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 1/3] build: validate that system capstone works before using it
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
@ 2021-06-28 6:54 ` Thomas Huth
2021-06-30 20:54 ` Willian Rampazzo
2021-07-05 12:06 ` Alex Bennée
2 siblings, 0 replies; 18+ messages in thread
From: Thomas Huth @ 2021-06-28 6:54 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Sean Bruno, Ed Maste, Warner Losh, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Nathan Whitehorn, Alex Bennée, Li-Wen Hsu
On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> Some versions of capstone have shipped a broken pkg-config file which
> puts the -I path without the trailing '/capstone' suffix. This breaks
> the ability to "#include <capstone.h>". Upstream and most distros have
> fixed this, but a few stragglers remain, notably FreeBSD.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> meson.build | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/meson.build b/meson.build
> index d8a92666fb..9979ddae71 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1425,6 +1425,19 @@ if capstone_opt in ['enabled', 'auto', 'system']
> kwargs: static_kwargs, method: 'pkg-config',
> required: capstone_opt == 'system' or
> capstone_opt == 'enabled' and not have_internal)
> +
> + # Some versions of capstone have broken pkg-config file
> + # that reports a wrong -I path, causing the #include to
> + # fail later. If the system has such a broken version
> + # do not use it.
> + if capstone.found() and not cc.compiles('#include <capstone.h>',
> + dependencies: [capstone])
> + capstone = not_found
> + if capstone_opt == 'system'
> + error('system capstone requested, it it does not appear to work')
> + endif
> + endif
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-25 17:22 ` [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Daniel P. Berrangé
@ 2021-06-28 7:28 ` Thomas Huth
2021-06-28 8:33 ` Daniel P. Berrangé
2021-06-30 18:58 ` Wainer dos Santos Moschetta
1 sibling, 1 reply; 18+ messages in thread
From: Thomas Huth @ 2021-06-28 7:28 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Alex Bennée, Li-Wen Hsu
On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> This adds support for running 4 jobs via Cirrus CI runners:
>
> * FreeBSD 12
> * FreeBSD 13
> * macOS 11 with default XCode
> * macOS 11 with latest XCode
>
> The gitlab job uses a container published by the libvirt-ci
> project (https://gitlab.com/libvirt/libvirt-ci) that contains
> the 'cirrus-run' command. This accepts a short yaml file that
> describes a single Cirrus CI job, runs it using the Cirrus CI
> REST API, and reports any output to the console.
>
> In this way Cirrus CI is effectively working as an indirect
> custom runner for GitLab CI pipelines. The key benefit is that
> Cirrus CI job results affect the GitLab CI pipeline result and
> so the user only has look at one CI dashboard.
Cool, thanks for tackling this!
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
> .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
> .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
> .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
> .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
> .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
> .gitlab-ci.d/qemu-project.yml | 1 +
> 7 files changed, 234 insertions(+)
> create mode 100644 .gitlab-ci.d/cirrus.yml
> create mode 100644 .gitlab-ci.d/cirrus/README.rst
> create mode 100644 .gitlab-ci.d/cirrus/build.yml
> create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
>
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> new file mode 100644
> index 0000000000..d7b4cce79b
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -0,0 +1,103 @@
> +# Jobs that we delegate to Cirrus CI because they require an operating
> +# system other than Linux. These jobs will only run if the required
> +# setup has been performed on the GitLab account.
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.
> +.cirrus_build_job:
> + stage: build
> + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> + needs: []
> + script:
> + - source .gitlab-ci.d/cirrus/$NAME.vars
> + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> + -e "s|[@]PKGS@|$PKGS|g"
> + -e "s|[@]MAKE@|$MAKE|g"
> + -e "s|[@]PYTHON@|$PYTHON|g"
> + -e "s|[@]PIP3@|$PIP3|g"
> + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> + - cat .gitlab-ci.d/cirrus/$NAME.yml
> + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> + rules:
> + - if: "$TEMPORARILY_DISABLED"
> + allow_failure: true
> + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> +
> +x64-freebsd-12-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: freebsd-12
> + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image_family
> + CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> + CIRRUS_VM_CPUS: 8
> + CIRRUS_VM_RAM: 8G
> + UPDATE_COMMAND: pkg update
> + INSTALL_COMMAND: pkg install -y
> + # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> + # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> + CONFIGURE_ARGS: --disable-gnutls
> + TEST_TARGETS: check
> +
> +x64-freebsd-13-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: freebsd-13
> + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image_family
> + CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> + CIRRUS_VM_CPUS: 8
> + CIRRUS_VM_RAM: 8G
> + UPDATE_COMMAND: pkg update
> + INSTALL_COMMAND: pkg install -y
> + TEST_TARGETS: check
> +
> +x64-macos-11-base-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: macos-11
> + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image
> + CIRRUS_VM_IMAGE_NAME: big-sur-base
> + CIRRUS_VM_CPUS: 12
> + CIRRUS_VM_RAM: 24G
> + UPDATE_COMMAND: brew update
> + INSTALL_COMMAND: brew install
> + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +x64-macos-11-xcode-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: macos-11
> + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image
> + CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> + CIRRUS_VM_CPUS: 12
> + CIRRUS_VM_RAM: 24G
> + UPDATE_COMMAND: brew update
> + INSTALL_COMMAND: brew install
> + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
Not directly related to your patch, but I wonder whether we really gain much
by having two macos build jobs in our CI ... they seem to be very similar,
e.g. compare the output of the "configure" step of the "macos" and the
"macos_xcode" job here:
https://cirrus-ci.com/build/4919300914937856
The diff basically looks like this:
diff -u /tmp/macos.txt /tmp/macos_xcode.txt
--- /tmp/macos.txt 2021-06-28 09:18:48.218698482 +0200
+++ /tmp/macos_xcode.txt 2021-06-28 09:21:01.811348076 +0200
@@ -1,6 +1,6 @@
mkdir build
cd build
-../configure --python=/usr/local/bin/python3 --enable-werror
--extra-cflags='-Wno-error=deprecated-declarations' || { cat config.log
meson-logs/meson-log.txt; exit 1; }
+../configure --extra-cflags='-Wno-error=deprecated-declarations'
--enable-modules --enable-werror --cc=clang || { cat config.log
meson-logs/meson-log.txt; exit 1; }
Disabling PIE due to missing toolchain support
Submodule 'capstone' (https://gitlab.com/qemu-project/capstone.git)
registered for path 'capstone'
Submodule 'dtc' (https://gitlab.com/qemu-project/dtc.git) registered for
path 'dtc'
@@ -23,18 +23,18 @@
Build type: native build
Project name: qemu
Project version: 6.0.50
-C compiler for the host machine: cc (clang 12.0.0 "Apple clang version
12.0.0 (clang-1200.0.32.28)")
-C linker for the host machine: cc ld64 609.8
+C compiler for the host machine: clang (clang 12.0.0 "Apple clang version
12.0.0 (clang-1200.0.32.27)")
+C linker for the host machine: clang ld64 609.7
Host machine cpu family: x86_64
Host machine cpu: x86_64
../meson.build:10: WARNING: Module unstable-keyval has no backwards or
forwards compatibility and might not exist in future releases.
Program sh found: YES
Program python3 found: YES (/usr/local/opt/python@3.9/bin/python3.9)
Program bzip2 found: YES
-C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version
12.0.0 (clang-1200.0.32.28)")
-C++ linker for the host machine: c++ ld64 609.8
+C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version
12.0.0 (clang-1200.0.32.27)")
+C++ linker for the host machine: c++ ld64 609.7
Objective-C compiler for the host machine: clang (clang 12.0.0)
-Objective-C linker for the host machine: clang ld64 609.8
+Objective-C linker for the host machine: clang ld64 609.7
Program cgcc found: NO
Library m found: YES
Library util found: YES
@@ -238,7 +238,7 @@
Program diff found: YES
Program dbus-daemon found: NO
Program initrd-stress.sh found: YES
-Build targets in project: 534
+Build targets in project: 556
qemu 6.0.50
@@ -273,7 +273,8 @@
user-mode emulation: NO
block layer: YES
Install blobs: YES
- module support: NO
+ module support: YES
+ alternative module path: NO
plugin support: NO
fuzzing support: NO
Audio drivers: coreaudio sdl
@@ -293,8 +294,8 @@
Compilation
host CPU: x86_64
host endianness: little
- C compiler: cc
- Host C compiler: cc
+ C compiler: clang
+ Host C compiler: clang
C++ compiler: c++
Objective-C compiler: clang
ARFLAGS: rv
@@ -376,7 +377,7 @@
pixman: YES
VTE support: NO
slirp support: internal
- libtasn1: NO
+ libtasn1: YES
PAM: YES
iconv support: YES
curses support: NO
@@ -403,7 +404,7 @@
xfsctl support: NO
smartcard support: NO
U2F support: NO
- libusb: NO
+ libusb: YES
usb net redir: NO
OpenGL support: NO
GBM: NO
... IMHO it's not worth wasting precious CI minutes here just because of
those small differences, and I'd like to suggest that we continue with only
on macos job in the future.
> diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
> new file mode 100644
> index 0000000000..657b0706d7
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/README.rst
> @@ -0,0 +1,54 @@
> +Cirrus CI integration
> +=====================
> +
> +GitLab CI shared runners only provide a docker environment running on Linux.
> +While it is possible to provide private runners for non-Linux platforms this
> +is not something most contributors/maintainers will wish to do.
> +
> +To work around this limitation, we take advantage of `Cirrus CI`_'s free
> +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
> +CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
> +the main GitLab CI pipeline dashboard.
> +
> +There is, however, some one-time setup required. If you want FreeBSD and macOS
> +builds to happen when you push to your GitLab repository, you need to
> +
> +* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
> + This repository needs to exist for cirrus-run to work, but it doesn't need to
> + be kept up to date, so you can create it and then forget about it;
> +
> +* enable the `Cirrus CI GitHub app`_ for your GitHub account;
> +
> +* sign up for Cirrus CI. It's enough to log into the website using your GitHub
> + account;
> +
> +* grab an API token from the `Cirrus CI settings`_ page;
> +
> +* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
> + for Cirrus CI to properly recognize the project. You can check whether
> + Cirrus CI knows about your project by navigating to:
> +
> + ``https://cirrus-ci.com/yourusername/qemu``
> +
> +* in the *CI/CD / Variables* section of the settings page for your GitLab
> + repository, create two new variables:
> +
> + * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
> + created earlier, eg. ``yourusername/qemu``;
> +
> + * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
> + This variable **must** be marked as *Masked*, because anyone with knowledge
> + of it can impersonate you as far as Cirrus CI is concerned.
> +
> + Neither of these variables should be marked as *Protected*, because in
> + general you'll want to be able to trigger Cirrus CI builds from non-protected
> + branches.
> +
> +Once this one-time setup is complete, you can just keep pushing to your GitLab
> +repository as usual and you'll automatically get the additional CI coverage.
> +
> +
> +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
> +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
> +.. _Cirrus CI: https://cirrus-ci.com/
> +.. _cirrus-run: https://github.com/sio/cirrus-run/
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> new file mode 100644
> index 0000000000..857bdc5536
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -0,0 +1,35 @@
> +@CIRRUS_VM_INSTANCE_TYPE@:
> + @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> + cpu: @CIRRUS_VM_CPUS@
> + memory: @CIRRUS_VM_RAM@
> +
> +env:
> + CIRRUS_CLONE_DEPTH: 1
> + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> + CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> + PATH: "@PATH@"
> + PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> + PYTHON: "@PYTHON@"
> + MAKE: "@MAKE@"
> + CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> +
> +build_task:
> + install_script:
> + - @UPDATE_COMMAND@
> + - @INSTALL_COMMAND@ @PKGS@
> + - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> + clone_script:
> + - git clone --depth 100 "$CI_REPOSITORY_URL" .
> + - git fetch origin "$CI_COMMIT_REF_NAME"
> + - git reset --hard "$CI_COMMIT_SHA"
> + build_script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-werror $CONFIGURE_ARGS
> + || { cat config.log meson-logs/meson-log.txt; exit 1; }
> + - $MAKE -j$(sysctl -n hw.ncpu)
> + - for TARGET in $TEST_TARGETS ;
> + do
> + $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> + done
You seem to try to enable ccache in the other files (e.g. by extending the
PATH) ... however, I don't see where you try to save the ccache directory
between the runs... so I guess that ccache won't be working with this setup
yet? In that case, I'd recommend to rather drop the other ccache changes
again, since the initial run with ccache (where the cache gets populated) is
rather slower than compiling without ccache.
Thomas
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
@ 2021-06-28 7:29 ` Thomas Huth
2021-06-30 19:00 ` Wainer dos Santos Moschetta
2021-06-30 20:52 ` Willian Rampazzo
2 siblings, 0 replies; 18+ messages in thread
From: Thomas Huth @ 2021-06-28 7:29 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Alex Bennée, Li-Wen Hsu
On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> The builds for these two platforms can now be performed from GitLab CI
> using cirrus-run.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .cirrus.yml | 55 -----------------------------------------------------
> 1 file changed, 55 deletions(-)
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index f4bf49b704..02c43a074a 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,61 +1,6 @@
> env:
> CIRRUS_CLONE_DEPTH: 1
>
> -freebsd_12_task:
> - freebsd_instance:
> - image_family: freebsd-12-2
> - cpu: 8
> - memory: 8G
> - install_script:
> - - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ;
> - - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed
> - nettle perl5 pixman pkgconf png usbredir ninja
> - script:
> - - mkdir build
> - - cd build
> - # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> - # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> - - ../configure --enable-werror --disable-gnutls
> - || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake -j$(sysctl -n hw.ncpu) check V=1
> -
> -macos_task:
> - osx_instance:
> - image: catalina-base
> - install_script:
> - - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --python=/usr/local/bin/python3 --enable-werror
> - --extra-cflags='-Wno-error=deprecated-declarations'
> - || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake check-unit V=1
> - - gmake check-block V=1
> - - gmake check-qapi-schema V=1
> - - gmake check-softfloat V=1
> - - gmake check-qtest-x86_64 V=1
> -
> -macos_xcode_task:
> - osx_instance:
> - # this is an alias for the latest Xcode
> - image: catalina-xcode
> - install_script:
> - - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --extra-cflags='-Wno-error=deprecated-declarations' --enable-modules
> - --enable-werror --cc=clang || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake check-unit V=1
> - - gmake check-block V=1
> - - gmake check-qapi-schema V=1
> - - gmake check-softfloat V=1
> - - gmake check-qtest-x86_64 V=1
> -
> windows_msys2_task:
> timeout_in: 90m
> windows_container:
>
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
` (2 preceding siblings ...)
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
@ 2021-06-28 7:34 ` Thomas Huth
2021-06-28 8:28 ` Daniel P. Berrangé
2021-07-05 10:32 ` Alex Bennée
4 siblings, 1 reply; 18+ messages in thread
From: Thomas Huth @ 2021-06-28 7:34 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Ed Maste, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Yonggang Luo,
Alex Bennée, Li-Wen Hsu
On 25/06/2021 19.22, Daniel P. Berrangé wrote:
[...]
> The MSys Windows job still remains in the .cirrus.yml file. This
> can be addressed to, if we extend libvirt-ci to have package
> mapping information for MSys.
I think gitlab-CI offers shared Windows runners, too, see e.g.:
https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta/
So I think we likely should rather convert that job to a shared gitlab-CI
Windows runner instead?
Thomas
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI
2021-06-28 7:34 ` [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Thomas Huth
@ 2021-06-28 8:28 ` Daniel P. Berrangé
0 siblings, 0 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-28 8:28 UTC (permalink / raw)
To: Thomas Huth
Cc: Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Alex Bennée, Li-Wen Hsu
On Mon, Jun 28, 2021 at 09:34:41AM +0200, Thomas Huth wrote:
> On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> [...]
> > The MSys Windows job still remains in the .cirrus.yml file. This
> > can be addressed to, if we extend libvirt-ci to have package
> > mapping information for MSys.
>
> I think gitlab-CI offers shared Windows runners, too, see e.g.:
>
> https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta/
>
> So I think we likely should rather convert that job to a shared gitlab-CI
> Windows runner instead?
There's quite a few caveats listed there, but agree it is worth looking
at it. We could start with having it 'allow_failure: true' and monitor
how reliable it is for a few months, before considering whether to let
it be gating. We can keep Cirrus CI job until we're comfortable with it.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-28 7:28 ` Thomas Huth
@ 2021-06-28 8:33 ` Daniel P. Berrangé
0 siblings, 0 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-28 8:33 UTC (permalink / raw)
To: Thomas Huth
Cc: Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Alex Bennée, Li-Wen Hsu
On Mon, Jun 28, 2021 at 09:28:18AM +0200, Thomas Huth wrote:
> On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> > This adds support for running 4 jobs via Cirrus CI runners:
> >
> > * FreeBSD 12
> > * FreeBSD 13
> > * macOS 11 with default XCode
> > * macOS 11 with latest XCode
> >
> > The gitlab job uses a container published by the libvirt-ci
> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
> > the 'cirrus-run' command. This accepts a short yaml file that
> > describes a single Cirrus CI job, runs it using the Cirrus CI
> > REST API, and reports any output to the console.
> >
> > In this way Cirrus CI is effectively working as an indirect
> > custom runner for GitLab CI pipelines. The key benefit is that
> > Cirrus CI job results affect the GitLab CI pipeline result and
> > so the user only has look at one CI dashboard.
>
> Cool, thanks for tackling this!
>
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > .gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
> > .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
> > .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
> > .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
> > .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
> > .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
> > .gitlab-ci.d/qemu-project.yml | 1 +
> > 7 files changed, 234 insertions(+)
> > create mode 100644 .gitlab-ci.d/cirrus.yml
> > create mode 100644 .gitlab-ci.d/cirrus/README.rst
> > create mode 100644 .gitlab-ci.d/cirrus/build.yml
> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> > create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
> >
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > new file mode 100644
> > index 0000000000..d7b4cce79b
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -0,0 +1,103 @@
> > +# Jobs that we delegate to Cirrus CI because they require an operating
> > +# system other than Linux. These jobs will only run if the required
> > +# setup has been performed on the GitLab account.
> > +#
> > +# The Cirrus CI configuration is generated by replacing target-specific
> > +# variables in a generic template: some of these variables are provided
> > +# when the GitLab CI job is defined, others are taken from a shell
> > +# snippet generated using lcitool.
> > +#
> > +# Note that the $PATH environment variable has to be treated with
> > +# special care, because we can't just override it at the GitLab CI job
> > +# definition level or we risk breaking it completely.
> > +.cirrus_build_job:
> > + stage: build
> > + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > + needs: []
> > + script:
> > + - source .gitlab-ci.d/cirrus/$NAME.vars
> > + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> > + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> > + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> > + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> > + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> > + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> > + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> > + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> > + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> > + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> > + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> > + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> > + -e "s|[@]PKGS@|$PKGS|g"
> > + -e "s|[@]MAKE@|$MAKE|g"
> > + -e "s|[@]PYTHON@|$PYTHON|g"
> > + -e "s|[@]PIP3@|$PIP3|g"
> > + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> > + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> > + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> > + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> > + - cat .gitlab-ci.d/cirrus/$NAME.yml
> > + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> > + rules:
> > + - if: "$TEMPORARILY_DISABLED"
> > + allow_failure: true
> > + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> > +
> > +x64-freebsd-12-build:
> > + extends: .cirrus_build_job
> > + variables:
> > + NAME: freebsd-12
> > + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> > + CIRRUS_VM_IMAGE_SELECTOR: image_family
> > + CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> > + CIRRUS_VM_CPUS: 8
> > + CIRRUS_VM_RAM: 8G
> > + UPDATE_COMMAND: pkg update
> > + INSTALL_COMMAND: pkg install -y
> > + # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> > + # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> > + CONFIGURE_ARGS: --disable-gnutls
> > + TEST_TARGETS: check
> > +
> > +x64-freebsd-13-build:
> > + extends: .cirrus_build_job
> > + variables:
> > + NAME: freebsd-13
> > + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> > + CIRRUS_VM_IMAGE_SELECTOR: image_family
> > + CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> > + CIRRUS_VM_CPUS: 8
> > + CIRRUS_VM_RAM: 8G
> > + UPDATE_COMMAND: pkg update
> > + INSTALL_COMMAND: pkg install -y
> > + TEST_TARGETS: check
> > +
> > +x64-macos-11-base-build:
> > + extends: .cirrus_build_job
> > + variables:
> > + NAME: macos-11
> > + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> > + CIRRUS_VM_IMAGE_SELECTOR: image
> > + CIRRUS_VM_IMAGE_NAME: big-sur-base
> > + CIRRUS_VM_CPUS: 12
> > + CIRRUS_VM_RAM: 24G
> > + UPDATE_COMMAND: brew update
> > + INSTALL_COMMAND: brew install
> > + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> > + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> > + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> > +
> > +x64-macos-11-xcode-build:
> > + extends: .cirrus_build_job
> > + variables:
> > + NAME: macos-11
> > + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> > + CIRRUS_VM_IMAGE_SELECTOR: image
> > + CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> > + CIRRUS_VM_CPUS: 12
> > + CIRRUS_VM_RAM: 24G
> > + UPDATE_COMMAND: brew update
> > + INSTALL_COMMAND: brew install
> > + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> > + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> > + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
>
> Not directly related to your patch, but I wonder whether we really gain much
> by having two macos build jobs in our CI ... they seem to be very similar,
> e.g. compare the output of the "configure" step of the "macos" and the
> "macos_xcode" job here:
>
> https://cirrus-ci.com/build/4919300914937856
>
> The diff basically looks like this:
snip
Looking back, the 2nd job was added by
commit fc84471ae2867823f56b1ad1705de324c2d8b725
Author: Alex Bennée <alex.bennee@linaro.org>
Date: Wed Oct 16 19:27:13 2019 +0100
cirrus.yml: add latest Xcode build target
CirrusCI provides a mojave-xcode alias for the latest Xcode available.
Let's use it to make sure we track the latest releases.
At the time this was added there was likely a significant difference
in XCode versions.
Since that time, both images were upgraded to catalina, instead of
mojave, and that appears to have eliminated the differences. My
changes upgrade again to big-sur.
So yes, we can drop the 2nd job at this time,
> > diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> > new file mode 100644
> > index 0000000000..857bdc5536
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus/build.yml
> > @@ -0,0 +1,35 @@
> > +@CIRRUS_VM_INSTANCE_TYPE@:
> > + @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> > + cpu: @CIRRUS_VM_CPUS@
> > + memory: @CIRRUS_VM_RAM@
> > +
> > +env:
> > + CIRRUS_CLONE_DEPTH: 1
> > + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> > + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> > + CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> > + PATH: "@PATH@"
> > + PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> > + PYTHON: "@PYTHON@"
> > + MAKE: "@MAKE@"
> > + CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> > +
> > +build_task:
> > + install_script:
> > + - @UPDATE_COMMAND@
> > + - @INSTALL_COMMAND@ @PKGS@
> > + - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> > + clone_script:
> > + - git clone --depth 100 "$CI_REPOSITORY_URL" .
> > + - git fetch origin "$CI_COMMIT_REF_NAME"
> > + - git reset --hard "$CI_COMMIT_SHA"
> > + build_script:
> > + - mkdir build
> > + - cd build
> > + - ../configure --enable-werror $CONFIGURE_ARGS
> > + || { cat config.log meson-logs/meson-log.txt; exit 1; }
> > + - $MAKE -j$(sysctl -n hw.ncpu)
> > + - for TARGET in $TEST_TARGETS ;
> > + do
> > + $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> > + done
>
> You seem to try to enable ccache in the other files (e.g. by extending the
> PATH) ... however, I don't see where you try to save the ccache directory
> between the runs... so I guess that ccache won't be working with this setup
> yet? In that case, I'd recommend to rather drop the other ccache changes
> again, since the initial run with ccache (where the cache gets populated) is
> rather slower than compiling without ccache.
I'll examine that. I think the path change is harmless though, because it
points to a location that I don't think exists.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-25 17:22 ` [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Daniel P. Berrangé
2021-06-28 7:28 ` Thomas Huth
@ 2021-06-30 18:58 ` Wainer dos Santos Moschetta
2021-06-30 20:35 ` Daniel P. Berrangé
1 sibling, 1 reply; 18+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-06-30 18:58 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Alex Bennée, Li-Wen Hsu
Hi,
On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
> This adds support for running 4 jobs via Cirrus CI runners:
>
> * FreeBSD 12
> * FreeBSD 13
> * macOS 11 with default XCode
> * macOS 11 with latest XCode
>
> The gitlab job uses a container published by the libvirt-ci
> project (https://gitlab.com/libvirt/libvirt-ci) that contains
> the 'cirrus-run' command. This accepts a short yaml file that
> describes a single Cirrus CI job, runs it using the Cirrus CI
> REST API, and reports any output to the console.
>
> In this way Cirrus CI is effectively working as an indirect
> custom runner for GitLab CI pipelines. The key benefit is that
> Cirrus CI job results affect the GitLab CI pipeline result and
> so the user only has look at one CI dashboard.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
> .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
> .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
> .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
> .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
> .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
> .gitlab-ci.d/qemu-project.yml | 1 +
> 7 files changed, 234 insertions(+)
> create mode 100644 .gitlab-ci.d/cirrus.yml
> create mode 100644 .gitlab-ci.d/cirrus/README.rst
> create mode 100644 .gitlab-ci.d/cirrus/build.yml
> create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
>
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> new file mode 100644
> index 0000000000..d7b4cce79b
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -0,0 +1,103 @@
> +# Jobs that we delegate to Cirrus CI because they require an operating
> +# system other than Linux. These jobs will only run if the required
> +# setup has been performed on the GitLab account.
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.
> +.cirrus_build_job:
> + stage: build
> + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> + needs: []
> + script:
> + - source .gitlab-ci.d/cirrus/$NAME.vars
> + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> + -e "s|[@]PKGS@|$PKGS|g"
> + -e "s|[@]MAKE@|$MAKE|g"
> + -e "s|[@]PYTHON@|$PYTHON|g"
> + -e "s|[@]PIP3@|$PIP3|g"
> + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> + - cat .gitlab-ci.d/cirrus/$NAME.yml
> + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> + rules:
> + - if: "$TEMPORARILY_DISABLED"
Reading 'TEMPORARILY_DISABLED' I immediately think the job is
malfunctioning or under maintenance.
But since the plan is to keep it running as 'non-gate' until it proves
reliable, so maybe you could rename the variable to 'NON_GATE' or
'STAGING_JOB' (i.e. some words to better express the intent).
Thanks!
- Wainer
> + allow_failure: true
> + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> +
> +x64-freebsd-12-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: freebsd-12
> + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image_family
> + CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> + CIRRUS_VM_CPUS: 8
> + CIRRUS_VM_RAM: 8G
> + UPDATE_COMMAND: pkg update
> + INSTALL_COMMAND: pkg install -y
> + # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> + # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> + CONFIGURE_ARGS: --disable-gnutls
> + TEST_TARGETS: check
> +
> +x64-freebsd-13-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: freebsd-13
> + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image_family
> + CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> + CIRRUS_VM_CPUS: 8
> + CIRRUS_VM_RAM: 8G
> + UPDATE_COMMAND: pkg update
> + INSTALL_COMMAND: pkg install -y
> + TEST_TARGETS: check
> +
> +x64-macos-11-base-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: macos-11
> + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image
> + CIRRUS_VM_IMAGE_NAME: big-sur-base
> + CIRRUS_VM_CPUS: 12
> + CIRRUS_VM_RAM: 24G
> + UPDATE_COMMAND: brew update
> + INSTALL_COMMAND: brew install
> + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +x64-macos-11-xcode-build:
> + extends: .cirrus_build_job
> + variables:
> + NAME: macos-11
> + CIRRUS_VM_INSTANCE_TYPE: osx_instance
> + CIRRUS_VM_IMAGE_SELECTOR: image
> + CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> + CIRRUS_VM_CPUS: 12
> + CIRRUS_VM_RAM: 24G
> + UPDATE_COMMAND: brew update
> + INSTALL_COMMAND: brew install
> + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
> new file mode 100644
> index 0000000000..657b0706d7
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/README.rst
> @@ -0,0 +1,54 @@
> +Cirrus CI integration
> +=====================
> +
> +GitLab CI shared runners only provide a docker environment running on Linux.
> +While it is possible to provide private runners for non-Linux platforms this
> +is not something most contributors/maintainers will wish to do.
> +
> +To work around this limitation, we take advantage of `Cirrus CI`_'s free
> +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
> +CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
> +the main GitLab CI pipeline dashboard.
> +
> +There is, however, some one-time setup required. If you want FreeBSD and macOS
> +builds to happen when you push to your GitLab repository, you need to
> +
> +* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
> + This repository needs to exist for cirrus-run to work, but it doesn't need to
> + be kept up to date, so you can create it and then forget about it;
> +
> +* enable the `Cirrus CI GitHub app`_ for your GitHub account;
> +
> +* sign up for Cirrus CI. It's enough to log into the website using your GitHub
> + account;
> +
> +* grab an API token from the `Cirrus CI settings`_ page;
> +
> +* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
> + for Cirrus CI to properly recognize the project. You can check whether
> + Cirrus CI knows about your project by navigating to:
> +
> + ``https://cirrus-ci.com/yourusername/qemu``
> +
> +* in the *CI/CD / Variables* section of the settings page for your GitLab
> + repository, create two new variables:
> +
> + * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
> + created earlier, eg. ``yourusername/qemu``;
> +
> + * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
> + This variable **must** be marked as *Masked*, because anyone with knowledge
> + of it can impersonate you as far as Cirrus CI is concerned.
> +
> + Neither of these variables should be marked as *Protected*, because in
> + general you'll want to be able to trigger Cirrus CI builds from non-protected
> + branches.
> +
> +Once this one-time setup is complete, you can just keep pushing to your GitLab
> +repository as usual and you'll automatically get the additional CI coverage.
> +
> +
> +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
> +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
> +.. _Cirrus CI: https://cirrus-ci.com/
> +.. _cirrus-run: https://github.com/sio/cirrus-run/
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> new file mode 100644
> index 0000000000..857bdc5536
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -0,0 +1,35 @@
> +@CIRRUS_VM_INSTANCE_TYPE@:
> + @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> + cpu: @CIRRUS_VM_CPUS@
> + memory: @CIRRUS_VM_RAM@
> +
> +env:
> + CIRRUS_CLONE_DEPTH: 1
> + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> + CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> + PATH: "@PATH@"
> + PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> + PYTHON: "@PYTHON@"
> + MAKE: "@MAKE@"
> + CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> +
> +build_task:
> + install_script:
> + - @UPDATE_COMMAND@
> + - @INSTALL_COMMAND@ @PKGS@
> + - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> + clone_script:
> + - git clone --depth 100 "$CI_REPOSITORY_URL" .
> + - git fetch origin "$CI_COMMIT_REF_NAME"
> + - git reset --hard "$CI_COMMIT_SHA"
> + build_script:
> + - mkdir build
> + - cd build
> + - ../configure --enable-werror $CONFIGURE_ARGS
> + || { cat config.log meson-logs/meson-log.txt; exit 1; }
> + - $MAKE -j$(sysctl -n hw.ncpu)
> + - for TARGET in $TEST_TARGETS ;
> + do
> + $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> + done
> diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
> new file mode 100644
> index 0000000000..5e51e50ea9
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
> @@ -0,0 +1,13 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +# $ lcitool variables freebsd-12 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='pkg'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip-3.7'
> +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
> diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
> new file mode 100644
> index 0000000000..07716833d9
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
> @@ -0,0 +1,13 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +# $ lcitool variables freebsd-13 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='pkg'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip-3.7'
> +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
> diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars
> new file mode 100644
> index 0000000000..cbec8a44a3
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/macos-11.vars
> @@ -0,0 +1,15 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +# $ lcitool variables macos-11 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='brew'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip3'
> +PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
> +PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv'
> +CPAN_PKGS='Test::Harness'
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index 64cb2ba1da..35d4e62c12 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -9,3 +9,4 @@ include:
> - local: '/.gitlab-ci.d/crossbuilds.yml'
> - local: '/.gitlab-ci.d/buildtest.yml'
> - local: '/.gitlab-ci.d/static_checks.yml'
> + - local: '/.gitlab-ci.d/cirrus.yml'
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
2021-06-28 7:29 ` Thomas Huth
@ 2021-06-30 19:00 ` Wainer dos Santos Moschetta
2021-06-30 20:52 ` Willian Rampazzo
2 siblings, 0 replies; 18+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-06-30 19:00 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Thomas Huth, Ed Maste, Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Alex Bennée, Li-Wen Hsu
On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
> The builds for these two platforms can now be performed from GitLab CI
> using cirrus-run.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .cirrus.yml | 55 -----------------------------------------------------
> 1 file changed, 55 deletions(-)
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index f4bf49b704..02c43a074a 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,61 +1,6 @@
> env:
> CIRRUS_CLONE_DEPTH: 1
>
> -freebsd_12_task:
> - freebsd_instance:
> - image_family: freebsd-12-2
> - cpu: 8
> - memory: 8G
> - install_script:
> - - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ;
> - - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed
> - nettle perl5 pixman pkgconf png usbredir ninja
> - script:
> - - mkdir build
> - - cd build
> - # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> - # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> - - ../configure --enable-werror --disable-gnutls
> - || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake -j$(sysctl -n hw.ncpu) check V=1
> -
> -macos_task:
> - osx_instance:
> - image: catalina-base
> - install_script:
> - - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --python=/usr/local/bin/python3 --enable-werror
> - --extra-cflags='-Wno-error=deprecated-declarations'
> - || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake check-unit V=1
> - - gmake check-block V=1
> - - gmake check-qapi-schema V=1
> - - gmake check-softfloat V=1
> - - gmake check-qtest-x86_64 V=1
> -
> -macos_xcode_task:
> - osx_instance:
> - # this is an alias for the latest Xcode
> - image: catalina-xcode
> - install_script:
> - - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --extra-cflags='-Wno-error=deprecated-declarations' --enable-modules
> - --enable-werror --cc=clang || { cat config.log meson-logs/meson-log.txt; exit 1; }
> - - gmake -j$(sysctl -n hw.ncpu)
> - - gmake check-unit V=1
> - - gmake check-block V=1
> - - gmake check-qapi-schema V=1
> - - gmake check-softfloat V=1
> - - gmake check-qtest-x86_64 V=1
> -
> windows_msys2_task:
> timeout_in: 90m
> windows_container:
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-30 18:58 ` Wainer dos Santos Moschetta
@ 2021-06-30 20:35 ` Daniel P. Berrangé
2021-07-05 10:35 ` Alex Bennée
0 siblings, 1 reply; 18+ messages in thread
From: Daniel P. Berrangé @ 2021-06-30 20:35 UTC (permalink / raw)
To: Wainer dos Santos Moschetta
Cc: Thomas Huth, Ed Maste, qemu-devel, Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Alex Bennée, Li-Wen Hsu
On Wed, Jun 30, 2021 at 03:58:57PM -0300, Wainer dos Santos Moschetta wrote:
> Hi,
>
> On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
> > This adds support for running 4 jobs via Cirrus CI runners:
> >
> > * FreeBSD 12
> > * FreeBSD 13
> > * macOS 11 with default XCode
> > * macOS 11 with latest XCode
> >
> > The gitlab job uses a container published by the libvirt-ci
> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
> > the 'cirrus-run' command. This accepts a short yaml file that
> > describes a single Cirrus CI job, runs it using the Cirrus CI
> > REST API, and reports any output to the console.
> >
> > In this way Cirrus CI is effectively working as an indirect
> > custom runner for GitLab CI pipelines. The key benefit is that
> > Cirrus CI job results affect the GitLab CI pipeline result and
> > so the user only has look at one CI dashboard.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > .gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
> > .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
> > .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
> > .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
> > .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
> > .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
> > .gitlab-ci.d/qemu-project.yml | 1 +
> > 7 files changed, 234 insertions(+)
> > create mode 100644 .gitlab-ci.d/cirrus.yml
> > create mode 100644 .gitlab-ci.d/cirrus/README.rst
> > create mode 100644 .gitlab-ci.d/cirrus/build.yml
> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> > create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
> >
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > new file mode 100644
> > index 0000000000..d7b4cce79b
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -0,0 +1,103 @@
> > +# Jobs that we delegate to Cirrus CI because they require an operating
> > +# system other than Linux. These jobs will only run if the required
> > +# setup has been performed on the GitLab account.
> > +#
> > +# The Cirrus CI configuration is generated by replacing target-specific
> > +# variables in a generic template: some of these variables are provided
> > +# when the GitLab CI job is defined, others are taken from a shell
> > +# snippet generated using lcitool.
> > +#
> > +# Note that the $PATH environment variable has to be treated with
> > +# special care, because we can't just override it at the GitLab CI job
> > +# definition level or we risk breaking it completely.
> > +.cirrus_build_job:
> > + stage: build
> > + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > + needs: []
> > + script:
> > + - source .gitlab-ci.d/cirrus/$NAME.vars
> > + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> > + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> > + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> > + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> > + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> > + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> > + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> > + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> > + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> > + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> > + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> > + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> > + -e "s|[@]PKGS@|$PKGS|g"
> > + -e "s|[@]MAKE@|$MAKE|g"
> > + -e "s|[@]PYTHON@|$PYTHON|g"
> > + -e "s|[@]PIP3@|$PIP3|g"
> > + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> > + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> > + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> > + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> > + - cat .gitlab-ci.d/cirrus/$NAME.yml
> > + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> > + rules:
> > + - if: "$TEMPORARILY_DISABLED"
>
> Reading 'TEMPORARILY_DISABLED' I immediately think the job is malfunctioning
> or under maintenance.
Actually this is cruft that I mistakenly copied from libvirt's rules.
> But since the plan is to keep it running as 'non-gate' until it proves
> reliable, so maybe you could rename the variable to 'NON_GATE' or
> 'STAGING_JOB' (i.e. some words to better express the intent).
We can just remove the 'if $TEMPORARILY_DISABLED' bit and
have only the 'allow_failure: true' bit
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
2021-06-28 7:29 ` Thomas Huth
2021-06-30 19:00 ` Wainer dos Santos Moschetta
@ 2021-06-30 20:52 ` Willian Rampazzo
2 siblings, 0 replies; 18+ messages in thread
From: Willian Rampazzo @ 2021-06-30 20:52 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Yonggang Luo, Alex Bennée, Li-Wen Hsu
On Fri, Jun 25, 2021 at 2:22 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> The builds for these two platforms can now be performed from GitLab CI
> using cirrus-run.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .cirrus.yml | 55 -----------------------------------------------------
> 1 file changed, 55 deletions(-)
>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/3] build: validate that system capstone works before using it
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
2021-06-28 6:54 ` Thomas Huth
@ 2021-06-30 20:54 ` Willian Rampazzo
2021-07-05 12:06 ` Alex Bennée
2 siblings, 0 replies; 18+ messages in thread
From: Willian Rampazzo @ 2021-06-30 20:54 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Yonggang Luo, Alex Bennée, Li-Wen Hsu
On Fri, Jun 25, 2021 at 2:22 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> Some versions of capstone have shipped a broken pkg-config file which
> puts the -I path without the trailing '/capstone' suffix. This breaks
> the ability to "#include <capstone.h>". Upstream and most distros have
> fixed this, but a few stragglers remain, notably FreeBSD.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> meson.build | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
` (3 preceding siblings ...)
2021-06-28 7:34 ` [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Thomas Huth
@ 2021-07-05 10:32 ` Alex Bennée
4 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2021-07-05 10:32 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Li-Wen Hsu
Daniel P. Berrangé <berrange@redhat.com> writes:
> Currently the Cirrus CI pipelines are completely separate from the
> GitLab CI pipelines. This means contributors/maintainers have to
> monitor two distinct places.
Queued to testing/next, thanks.
--
Alex Bennée
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-06-30 20:35 ` Daniel P. Berrangé
@ 2021-07-05 10:35 ` Alex Bennée
0 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2021-07-05 10:35 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Li-Wen Hsu
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Wed, Jun 30, 2021 at 03:58:57PM -0300, Wainer dos Santos Moschetta wrote:
>> Hi,
>>
>> On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
>> > This adds support for running 4 jobs via Cirrus CI runners:
>> >
>> > * FreeBSD 12
>> > * FreeBSD 13
>> > * macOS 11 with default XCode
>> > * macOS 11 with latest XCode
>> >
>> > The gitlab job uses a container published by the libvirt-ci
>> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
>> > the 'cirrus-run' command. This accepts a short yaml file that
>> > describes a single Cirrus CI job, runs it using the Cirrus CI
>> > REST API, and reports any output to the console.
>> >
>> > In this way Cirrus CI is effectively working as an indirect
>> > custom runner for GitLab CI pipelines. The key benefit is that
>> > Cirrus CI job results affect the GitLab CI pipeline result and
>> > so the user only has look at one CI dashboard.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> > .gitlab-ci.d/cirrus.yml | 103 ++++++++++++++++++++++++++++
>> > .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
>> > .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
>> > .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
>> > .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
>> > .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
>> > .gitlab-ci.d/qemu-project.yml | 1 +
>> > 7 files changed, 234 insertions(+)
>> > create mode 100644 .gitlab-ci.d/cirrus.yml
>> > create mode 100644 .gitlab-ci.d/cirrus/README.rst
>> > create mode 100644 .gitlab-ci.d/cirrus/build.yml
>> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
>> > create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
>> > create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
>> >
>> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
>> > new file mode 100644
>> > index 0000000000..d7b4cce79b
>> > --- /dev/null
>> > +++ b/.gitlab-ci.d/cirrus.yml
>> > @@ -0,0 +1,103 @@
>> > +# Jobs that we delegate to Cirrus CI because they require an operating
>> > +# system other than Linux. These jobs will only run if the required
>> > +# setup has been performed on the GitLab account.
>> > +#
>> > +# The Cirrus CI configuration is generated by replacing target-specific
>> > +# variables in a generic template: some of these variables are provided
>> > +# when the GitLab CI job is defined, others are taken from a shell
>> > +# snippet generated using lcitool.
>> > +#
>> > +# Note that the $PATH environment variable has to be treated with
>> > +# special care, because we can't just override it at the GitLab CI job
>> > +# definition level or we risk breaking it completely.
>> > +.cirrus_build_job:
>> > + stage: build
>> > + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>> > + needs: []
>> > + script:
>> > + - source .gitlab-ci.d/cirrus/$NAME.vars
>> > + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
>> > + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
>> > + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
>> > + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
>> > + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
>> > + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
>> > + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
>> > + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
>> > + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
>> > + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
>> > + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
>> > + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
>> > + -e "s|[@]PKGS@|$PKGS|g"
>> > + -e "s|[@]MAKE@|$MAKE|g"
>> > + -e "s|[@]PYTHON@|$PYTHON|g"
>> > + -e "s|[@]PIP3@|$PIP3|g"
>> > + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
>> > + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
>> > + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
>> > + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
>> > + - cat .gitlab-ci.d/cirrus/$NAME.yml
>> > + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
>> > + rules:
>> > + - if: "$TEMPORARILY_DISABLED"
>>
>> Reading 'TEMPORARILY_DISABLED' I immediately think the job is malfunctioning
>> or under maintenance.
>
> Actually this is cruft that I mistakenly copied from libvirt's rules.
>
>> But since the plan is to keep it running as 'non-gate' until it proves
>> reliable, so maybe you could rename the variable to 'NON_GATE' or
>> 'STAGING_JOB' (i.e. some words to better express the intent).
>
> We can just remove the 'if $TEMPORARILY_DISABLED' bit and
> have only the 'allow_failure: true' bit
I've cleaned that up on testing/next
--
Alex Bennée
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/3] build: validate that system capstone works before using it
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
2021-06-28 6:54 ` Thomas Huth
2021-06-30 20:54 ` Willian Rampazzo
@ 2021-07-05 12:06 ` Alex Bennée
2 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2021-07-05 12:06 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Ed Maste, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Yonggang Luo, Li-Wen Hsu
Daniel P. Berrangé <berrange@redhat.com> writes:
> Some versions of capstone have shipped a broken pkg-config file which
> puts the -I path without the trailing '/capstone' suffix. This breaks
> the ability to "#include <capstone.h>". Upstream and most distros have
> fixed this, but a few stragglers remain, notably FreeBSD.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2021-07-05 12:07 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-25 17:22 [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Daniel P. Berrangé
2021-06-25 17:22 ` [PATCH 1/3] build: validate that system capstone works before using it Daniel P. Berrangé
2021-06-28 6:54 ` Thomas Huth
2021-06-30 20:54 ` Willian Rampazzo
2021-07-05 12:06 ` Alex Bennée
2021-06-25 17:22 ` [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Daniel P. Berrangé
2021-06-28 7:28 ` Thomas Huth
2021-06-28 8:33 ` Daniel P. Berrangé
2021-06-30 18:58 ` Wainer dos Santos Moschetta
2021-06-30 20:35 ` Daniel P. Berrangé
2021-07-05 10:35 ` Alex Bennée
2021-06-25 17:22 ` [PATCH 3/3] cirrus: delete FreeBSD and macOS jobs Daniel P. Berrangé
2021-06-28 7:29 ` Thomas Huth
2021-06-30 19:00 ` Wainer dos Santos Moschetta
2021-06-30 20:52 ` Willian Rampazzo
2021-06-28 7:34 ` [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI Thomas Huth
2021-06-28 8:28 ` Daniel P. Berrangé
2021-07-05 10:32 ` Alex Bennée
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.