* [PATCH v3 01/40] Jobs based on custom runners: documentation and configuration placeholder
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 02/40] Jobs based on custom runners: build environment docs and playbook Alex Bennée
` (38 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Cleber Rosa <crosa@redhat.com>
As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place. One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.
This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions. As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-2-crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/devel/ci.rst | 32 ++++++++++++++++++++++++++++++++
docs/devel/index.rst | 1 +
.gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++
.gitlab-ci.d/qemu-project.yml | 1 +
4 files changed, 48 insertions(+)
create mode 100644 docs/devel/ci.rst
create mode 100644 .gitlab-ci.d/custom-runners.yml
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
new file mode 100644
index 0000000000..064ffa9988
--- /dev/null
+++ b/docs/devel/ci.rst
@@ -0,0 +1,32 @@
+==
+CI
+==
+
+QEMU has configurations enabled for a number of different CI services.
+The most up to date information about them and their status can be
+found at::
+
+ https://wiki.qemu.org/Testing/CI
+
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+ .gitlab-ci.d/custom-runners.yml
+
+Custom runners entail custom machines. To see a list of the machines
+currently deployed in the QEMU GitLab CI and their maintainers, please
+refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 977c3893bd..da15273786 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -27,6 +27,7 @@ Contents:
migration
atomics
stable-process
+ ci
qtest
decodetree
secure-coding-practices
diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
new file mode 100644
index 0000000000..a07b27384c
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -0,0 +1,14 @@
+# The CI jobs defined here require GitLab runners installed and
+# registered on machines that match their operating system names,
+# versions and architectures. This is in contrast to the other CI
+# jobs that are intended to run on GitLab's "shared" runners.
+
+# Different than the default approach on "shared" runners, based on
+# containers, the custom runners have no such *requirement*, as those
+# jobs should be capable of running on operating systems with no
+# compatible container implementation, or no support from
+# gitlab-runner. To avoid problems that gitlab-runner can cause while
+# reusing the GIT repository, let's enable the clone strategy, which
+# guarantees a fresh repository on each job run.
+variables:
+ GIT_STRATEGY: clone
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..dde8270301 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
- local: '/.gitlab-ci.d/crossbuilds.yml'
- local: '/.gitlab-ci.d/buildtest.yml'
- local: '/.gitlab-ci.d/static_checks.yml'
+ - local: '/.gitlab-ci.d/custom-runners.yml'
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 02/40] Jobs based on custom runners: build environment docs and playbook
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
2021-07-09 14:29 ` [PATCH v3 01/40] Jobs based on custom runners: documentation and configuration placeholder Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 03/40] Jobs based on custom runners: docs and gitlab-runner setup playbook Alex Bennée
` (37 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Cleber Rosa <crosa@redhat.com>
To run basic jobs on custom runners, the environment needs to be
properly set up. The most common requirement is having the right
packages installed.
The playbook introduced here covers the QEMU's project s390x and
aarch64 machines. At the time this is being proposed, those machines
have already had this playbook applied to them.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210630012619.115262-3-crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/devel/ci.rst | 40 +++++++++
scripts/ci/setup/.gitignore | 2 +
scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
scripts/ci/setup/inventory.template | 1 +
4 files changed, 159 insertions(+)
create mode 100644 scripts/ci/setup/.gitignore
create mode 100644 scripts/ci/setup/build-environment.yml
create mode 100644 scripts/ci/setup/inventory.template
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 064ffa9988..bfedbb1025 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under::
Custom runners entail custom machines. To see a list of the machines
currently deployed in the QEMU GitLab CI and their maintainers, please
refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks. Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+ fully.qualified.domain
+ other.machine.hostname
+
+You may need to set some variables in the inventory file itself. One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts. This would look like::
+
+ fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+ other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests. This playbook consists on the installation of various
+required packages (and a general package update while at it). It
+currently covers a number of different Linux distributions, but it can
+be expanded to cover other systems.
+
+The minimum required version of Ansible successfully tested in this
+playbook is 2.8.0 (a version check is embedded within the playbook
+itself). To run the playbook, execute::
+
+ cd scripts/ci/setup
+ ansible-playbook -i inventory build-environment.yml
+
+Please note that most of the tasks in the playbook require superuser
+privileges, such as those from the ``root`` account or those obtained
+by ``sudo``. If necessary, please refer to ``ansible-playbook``
+options such as ``--become``, ``--become-method``, ``--become-user``
+and ``--ask-become-pass``.
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
new file mode 100644
index 0000000000..ee088604d1
--- /dev/null
+++ b/scripts/ci/setup/.gitignore
@@ -0,0 +1,2 @@
+inventory
+
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..581c1c75d1
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,116 @@
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+# Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+#
+# This is an ansible playbook file. Run it to set up systems with the
+# environment needed to build QEMU.
+---
+- name: Installation of basic packages to build QEMU
+ hosts: all
+ tasks:
+ - name: Check for suitable ansible version
+ delegate_to: localhost
+ assert:
+ that:
+ - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)'
+ msg: "Unsuitable ansible version, please use version 2.8.0 or later"
+
+ - name: Update apt cache / upgrade packages via apt
+ apt:
+ update_cache: yes
+ upgrade: yes
+ when:
+ - ansible_facts['distribution'] == 'Ubuntu'
+
+ - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+ package:
+ name:
+ # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+ - ccache
+ - gcc
+ - gettext
+ - git
+ - glusterfs-common
+ - libaio-dev
+ - libattr1-dev
+ - libbrlapi-dev
+ - libbz2-dev
+ - libcacard-dev
+ - libcap-ng-dev
+ - libcurl4-gnutls-dev
+ - libdrm-dev
+ - libepoxy-dev
+ - libfdt-dev
+ - libgbm-dev
+ - libgtk-3-dev
+ - libibverbs-dev
+ - libiscsi-dev
+ - libjemalloc-dev
+ - libjpeg-turbo8-dev
+ - liblzo2-dev
+ - libncurses5-dev
+ - libncursesw5-dev
+ - libnfs-dev
+ - libnss3-dev
+ - libnuma-dev
+ - libpixman-1-dev
+ - librados-dev
+ - librbd-dev
+ - librdmacm-dev
+ - libsasl2-dev
+ - libsdl2-dev
+ - libseccomp-dev
+ - libsnappy-dev
+ - libspice-protocol-dev
+ - libssh-dev
+ - libusb-1.0-0-dev
+ - libusbredirhost-dev
+ - libvdeplug-dev
+ - libvte-2.91-dev
+ - libzstd-dev
+ - make
+ - python3-yaml
+ - python3-sphinx
+ - python3-sphinx-rtd-theme
+ - ninja-build
+ - sparse
+ - xfslibs-dev
+ state: present
+ when:
+ - ansible_facts['distribution'] == 'Ubuntu'
+
+ - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+ package:
+ name:
+ - libspice-server-dev
+ - libxen-dev
+ state: present
+ when:
+ - ansible_facts['distribution'] == 'Ubuntu'
+ - ansible_facts['architecture'] != 's390x'
+
+ - name: Install basic packages to build QEMU on Ubuntu 18.04
+ package:
+ name:
+ # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+ - clang
+ when:
+ - ansible_facts['distribution'] == 'Ubuntu'
+ - ansible_facts['distribution_version'] == '18.04'
+
+ - name: Install basic packages to build QEMU on Ubuntu 20.04
+ package:
+ name:
+ # Originally from tests/docker/dockerfiles/ubuntu2004.docker
+ - clang-10
+ - genisoimage
+ - liblttng-ust-dev
+ - libslirp-dev
+ - netcat-openbsd
+ when:
+ - ansible_facts['distribution'] == 'Ubuntu'
+ - ansible_facts['distribution_version'] == '20.04'
diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template
new file mode 100644
index 0000000000..2fbb50c4a8
--- /dev/null
+++ b/scripts/ci/setup/inventory.template
@@ -0,0 +1 @@
+localhost
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 03/40] Jobs based on custom runners: docs and gitlab-runner setup playbook
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
2021-07-09 14:29 ` [PATCH v3 01/40] Jobs based on custom runners: documentation and configuration placeholder Alex Bennée
2021-07-09 14:29 ` [PATCH v3 02/40] Jobs based on custom runners: build environment docs and playbook Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 04/40] Jobs based on custom runners: add job definitions for QEMU's machines Alex Bennée
` (36 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Cleber Rosa <crosa@redhat.com>
To have the jobs dispatched to custom runners, gitlab-runner must
be installed, active as a service and properly configured. The
variables file and playbook introduced here should help with those
steps.
The playbook introduced here covers the Linux distributions and
has been primarily tested on OS/machines that the QEMU project
has available to act as runners, namely:
* Ubuntu 20.04 on aarch64
* Ubuntu 18.04 on s390x
But, it should work on all other Linux distributions. Earlier
versions were tested on FreeBSD too, so chances of success are
high.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210630012619.115262-4-crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/devel/ci.rst | 55 +++++++++++++++++++++++
scripts/ci/setup/.gitignore | 2 +-
scripts/ci/setup/gitlab-runner.yml | 71 ++++++++++++++++++++++++++++++
scripts/ci/setup/vars.yml.template | 12 +++++
4 files changed, 139 insertions(+), 1 deletion(-)
create mode 100644 scripts/ci/setup/gitlab-runner.yml
create mode 100644 scripts/ci/setup/vars.yml.template
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index bfedbb1025..b3bf3ef615 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -70,3 +70,58 @@ privileges, such as those from the ``root`` account or those obtained
by ``sudo``. If necessary, please refer to ``ansible-playbook``
options such as ``--become``, ``--become-method``, ``--become-user``
and ``--ask-become-pass``.
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs. The association between a machine and a GitLab project
+happens with a registration token. To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears-like icon at the bottom of the left hand side
+ vertical toolbar), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+ "And this registration token:"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``. Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+To run the playbook, execute::
+
+ cd scripts/ci/setup
+ ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI. Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Tags are very important as they are used to route specific jobs to
+specific types of runners, so it's a good idea to double check that
+the automatically created tags are consistent with the OS and
+architecture. For instance, an Ubuntu 20.04 aarch64 system should
+have tags set as::
+
+ ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+ ubuntu-20.04-aarch64-all:
+ tags:
+ - ubuntu_20.04
+ - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
index ee088604d1..f4a6183f1f 100644
--- a/scripts/ci/setup/.gitignore
+++ b/scripts/ci/setup/.gitignore
@@ -1,2 +1,2 @@
inventory
-
+vars.yml
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
new file mode 100644
index 0000000000..1127db516f
--- /dev/null
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -0,0 +1,71 @@
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Author:
+# Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+#
+# This is an ansible playbook file. Run it to set up systems with the
+# gitlab-runner agent.
+---
+- name: Installation of gitlab-runner
+ hosts: all
+ vars_files:
+ - vars.yml
+ tasks:
+ - debug:
+ msg: 'Checking for a valid GitLab registration token'
+ failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'"
+
+ - name: Create a group for the gitlab-runner service
+ group:
+ name: gitlab-runner
+
+ - name: Create a user for the gitlab-runner service
+ user:
+ user: gitlab-runner
+ group: gitlab-runner
+ comment: GitLab Runner
+ home: /home/gitlab-runner
+ shell: /bin/bash
+
+ - name: Remove the .bash_logout file when on Ubuntu systems
+ file:
+ path: /home/gitlab-runner/.bash_logout
+ state: absent
+ when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+ - name: Set the Operating System for gitlab-runner
+ set_fact:
+ gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}"
+ - debug:
+ msg: gitlab-runner OS is {{ gitlab_runner_os }}
+
+ - name: Set the architecture for gitlab-runner
+ set_fact:
+ gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}"
+ - debug:
+ msg: gitlab-runner arch is {{ gitlab_runner_arch }}
+
+ - name: Download the matching gitlab-runner
+ get_url:
+ dest: /usr/local/bin/gitlab-runner
+ url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}"
+ owner: gitlab-runner
+ group: gitlab-runner
+ mode: u=rwx,g=rwx,o=rx
+
+ - name: Register the gitlab-runner
+ command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+ - name: Install the gitlab-runner service using its own functionality
+ command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner
+ register: gitlab_runner_install_service_result
+ failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+
+ - name: Enable the gitlab-runner service
+ service:
+ name: gitlab-runner
+ state: started
+ enabled: yes
diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template
new file mode 100644
index 0000000000..e48089761f
--- /dev/null
+++ b/scripts/ci/setup/vars.yml.template
@@ -0,0 +1,12 @@
+# The version of the gitlab-runner to use
+gitlab_runner_version: 13.12.0
+# The URL of the gitlab server to use, usually https://gitlab.com unless you're
+# using a private GitLab instance
+gitlab_runner_server_url: https://gitlab.com
+# A mapping of the ansible to gitlab architecture nomenclature
+ansible_to_gitlab_arch:
+ x86_64: amd64
+ aarch64: arm64
+ s390x: s390x
+# A unique token made available by GitLab to your project for registering runners
+gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 04/40] Jobs based on custom runners: add job definitions for QEMU's machines
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (2 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 03/40] Jobs based on custom runners: docs and gitlab-runner setup playbook Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 05/40] tests/tcg: also disable the signals test for plugins Alex Bennée
` (35 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Cleber Rosa <crosa@redhat.com>
The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests. This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-5-crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++
1 file changed, 208 insertions(+)
diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index a07b27384c..061d3cdfed 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -12,3 +12,211 @@
# guarantees a fresh repository on each job run.
variables:
GIT_STRATEGY: clone
+
+# All ubuntu-18.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-18.04-s390x-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-18.04-s390x-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-18.04-s390x-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-18.04-s390x-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_18.04
+ - s390x
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+ubuntu-20.04-aarch64-all-linux-static:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
+ # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc` check-tcg V=1
+
+ubuntu-20.04-aarch64-all:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-alldbg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-debug --disable-libssh
+ - make clean
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-clang:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
+
+ubuntu-20.04-aarch64-tci:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --enable-tcg-interpreter
+ - make --output-sync -j`nproc`
+
+ubuntu-20.04-aarch64-notcg:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch64
+ rules:
+ - if: '$CI_COMMIT_BRANCH =~ /^staging/'
+ when: manual
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --disable-libssh --disable-tcg
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 05/40] tests/tcg: also disable the signals test for plugins
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (3 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 04/40] Jobs based on custom runners: add job definitions for QEMU's machines Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 06/40] build: validate that system capstone works before using it Alex Bennée
` (34 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, Richard Henderson, f4bug,
robhenry, mahmoudabdalghany, aaron, cota, stefanha, crosa,
pbonzini, ma.mandourr, Alex Bennée, aurelien
This will be more important when plugins is enabled by default.
Fixes: eba61056e4 ("tests/tcg: generalise the disabling of the signals test")
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/tcg/multiarch/Makefile.target | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target
index d57a115873..85a6fb7a2e 100644
--- a/tests/tcg/multiarch/Makefile.target
+++ b/tests/tcg/multiarch/Makefile.target
@@ -37,6 +37,8 @@ signals: LDFLAGS+=-lrt -lpthread
run-signals: signals
$(call skip-test, $<, "BROKEN awaiting sigframe clean-ups and vdso support")
+run-plugin-signals-with-%:
+ $(call skip-test, $<, "BROKEN awaiting sigframe clean-ups and vdso support")
# We define the runner for test-mmap after the individual
# architectures have defined their supported pages sizes. If no
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 06/40] build: validate that system capstone works before using it
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (4 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 05/40] tests/tcg: also disable the signals test for plugins Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 07/40] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Alex Bennée
` (33 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Richard Henderson, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
Some versions of capstone have shipped a broken pkg-config file which
puts the -I path without the trailing '/capstone' suffix. This breaks
the ability to "#include <capstone.h>". Upstream and most distros have
fixed this, but a few stragglers remain, notably FreeBSD.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210625172211.451010-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
---
v3
- fix double it.
---
meson.build | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/meson.build b/meson.build
index 7e12de01be..ff7a9f8b4f 100644
--- a/meson.build
+++ b/meson.build
@@ -1641,6 +1641,19 @@ if capstone_opt in ['enabled', 'auto', 'system']
kwargs: static_kwargs, method: 'pkg-config',
required: capstone_opt == 'system' or
capstone_opt == 'enabled' and not have_internal)
+
+ # Some versions of capstone have broken pkg-config file
+ # that reports a wrong -I path, causing the #include to
+ # fail later. If the system has such a broken version
+ # do not use it.
+ if capstone.found() and not cc.compiles('#include <capstone.h>',
+ dependencies: [capstone])
+ capstone = not_found
+ if capstone_opt == 'system'
+ error('system capstone requested, it does not appear to work')
+ endif
+ endif
+
if capstone.found()
capstone_opt = 'system'
elif have_internal
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 07/40] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (5 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 06/40] build: validate that system capstone works before using it Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 08/40] cirrus: delete FreeBSD and macOS jobs Alex Bennée
` (32 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This adds support for running 4 jobs via Cirrus CI runners:
* FreeBSD 12
* FreeBSD 13
* macOS 11 with default XCode
* macOS 11 with latest XCode
The gitlab job uses a container published by the libvirt-ci
project (https://gitlab.com/libvirt/libvirt-ci) that contains
the 'cirrus-run' command. This accepts a short yaml file that
describes a single Cirrus CI job, runs it using the Cirrus CI
REST API, and reports any output to the console.
In this way Cirrus CI is effectively working as an indirect
custom runner for GitLab CI pipelines. The key benefit is that
Cirrus CI job results affect the GitLab CI pipeline result and
so the user only has look at one CI dashboard.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210625172211.451010-3-berrange@redhat.com>
[AJB: remove $TEMPORARILY_DISABLED condition]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++
.gitlab-ci.d/cirrus.yml | 102 ++++++++++++++++++++++++++++
.gitlab-ci.d/cirrus/build.yml | 35 ++++++++++
.gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++
.gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++
.gitlab-ci.d/cirrus/macos-11.vars | 15 ++++
.gitlab-ci.d/qemu-project.yml | 1 +
7 files changed, 233 insertions(+)
create mode 100644 .gitlab-ci.d/cirrus/README.rst
create mode 100644 .gitlab-ci.d/cirrus.yml
create mode 100644 .gitlab-ci.d/cirrus/build.yml
create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
new file mode 100644
index 0000000000..657b0706d7
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/README.rst
@@ -0,0 +1,54 @@
+Cirrus CI integration
+=====================
+
+GitLab CI shared runners only provide a docker environment running on Linux.
+While it is possible to provide private runners for non-Linux platforms this
+is not something most contributors/maintainers will wish to do.
+
+To work around this limitation, we take advantage of `Cirrus CI`_'s free
+offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
+CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
+the main GitLab CI pipeline dashboard.
+
+There is, however, some one-time setup required. If you want FreeBSD and macOS
+builds to happen when you push to your GitLab repository, you need to
+
+* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
+ This repository needs to exist for cirrus-run to work, but it doesn't need to
+ be kept up to date, so you can create it and then forget about it;
+
+* enable the `Cirrus CI GitHub app`_ for your GitHub account;
+
+* sign up for Cirrus CI. It's enough to log into the website using your GitHub
+ account;
+
+* grab an API token from the `Cirrus CI settings`_ page;
+
+* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
+ for Cirrus CI to properly recognize the project. You can check whether
+ Cirrus CI knows about your project by navigating to:
+
+ ``https://cirrus-ci.com/yourusername/qemu``
+
+* in the *CI/CD / Variables* section of the settings page for your GitLab
+ repository, create two new variables:
+
+ * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
+ created earlier, eg. ``yourusername/qemu``;
+
+ * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
+ This variable **must** be marked as *Masked*, because anyone with knowledge
+ of it can impersonate you as far as Cirrus CI is concerned.
+
+ Neither of these variables should be marked as *Protected*, because in
+ general you'll want to be able to trigger Cirrus CI builds from non-protected
+ branches.
+
+Once this one-time setup is complete, you can just keep pushing to your GitLab
+repository as usual and you'll automatically get the additional CI coverage.
+
+
+.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
+.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
+.. _Cirrus CI: https://cirrus-ci.com/
+.. _cirrus-run: https://github.com/sio/cirrus-run/
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
new file mode 100644
index 0000000000..60b13ed83f
--- /dev/null
+++ b/.gitlab-ci.d/cirrus.yml
@@ -0,0 +1,102 @@
+# Jobs that we delegate to Cirrus CI because they require an operating
+# system other than Linux. These jobs will only run if the required
+# setup has been performed on the GitLab account.
+#
+# The Cirrus CI configuration is generated by replacing target-specific
+# variables in a generic template: some of these variables are provided
+# when the GitLab CI job is defined, others are taken from a shell
+# snippet generated using lcitool.
+#
+# Note that the $PATH environment variable has to be treated with
+# special care, because we can't just override it at the GitLab CI job
+# definition level or we risk breaking it completely.
+.cirrus_build_job:
+ stage: build
+ image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
+ needs: []
+ allow_failure: true
+ script:
+ - source .gitlab-ci.d/cirrus/$NAME.vars
+ - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
+ -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
+ -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
+ -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
+ -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
+ -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
+ -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
+ -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
+ -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
+ -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
+ -e "s|[@]PKGS@|$PKGS|g"
+ -e "s|[@]MAKE@|$MAKE|g"
+ -e "s|[@]PYTHON@|$PYTHON|g"
+ -e "s|[@]PIP3@|$PIP3|g"
+ -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
+ -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
+ -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
+ <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
+ - cat .gitlab-ci.d/cirrus/$NAME.yml
+ - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
+ rules:
+ - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+
+x64-freebsd-12-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: freebsd-12
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-12-2
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
+ UPDATE_COMMAND: pkg update
+ INSTALL_COMMAND: pkg install -y
+ # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
+ # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
+ CONFIGURE_ARGS: --disable-gnutls
+ TEST_TARGETS: check
+
+x64-freebsd-13-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: freebsd-13
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-13-0
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
+ UPDATE_COMMAND: pkg update
+ INSTALL_COMMAND: pkg install -y
+ TEST_TARGETS: check
+
+x64-macos-11-base-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: macos-11
+ CIRRUS_VM_INSTANCE_TYPE: osx_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image
+ CIRRUS_VM_IMAGE_NAME: big-sur-base
+ CIRRUS_VM_CPUS: 12
+ CIRRUS_VM_RAM: 24G
+ UPDATE_COMMAND: brew update
+ INSTALL_COMMAND: brew install
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
+
+x64-macos-11-xcode-build:
+ extends: .cirrus_build_job
+ variables:
+ NAME: macos-11
+ CIRRUS_VM_INSTANCE_TYPE: osx_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image
+ CIRRUS_VM_IMAGE_NAME: big-sur-xcode
+ CIRRUS_VM_CPUS: 12
+ CIRRUS_VM_RAM: 24G
+ UPDATE_COMMAND: brew update
+ INSTALL_COMMAND: brew install
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
new file mode 100644
index 0000000000..857bdc5536
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -0,0 +1,35 @@
+@CIRRUS_VM_INSTANCE_TYPE@:
+ @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
+ cpu: @CIRRUS_VM_CPUS@
+ memory: @CIRRUS_VM_RAM@
+
+env:
+ CIRRUS_CLONE_DEPTH: 1
+ CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
+ CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
+ CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
+ PATH: "@PATH@"
+ PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
+ PYTHON: "@PYTHON@"
+ MAKE: "@MAKE@"
+ CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
+
+build_task:
+ install_script:
+ - @UPDATE_COMMAND@
+ - @INSTALL_COMMAND@ @PKGS@
+ - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
+ clone_script:
+ - git clone --depth 100 "$CI_REPOSITORY_URL" .
+ - git fetch origin "$CI_COMMIT_REF_NAME"
+ - git reset --hard "$CI_COMMIT_SHA"
+ build_script:
+ - mkdir build
+ - cd build
+ - ../configure --enable-werror $CONFIGURE_ARGS
+ || { cat config.log meson-logs/meson-log.txt; exit 1; }
+ - $MAKE -j$(sysctl -n hw.ncpu)
+ - for TARGET in $TEST_TARGETS ;
+ do
+ $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
+ done
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
new file mode 100644
index 0000000000..5e51e50ea9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables freebsd-12 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
new file mode 100644
index 0000000000..07716833d9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables freebsd-13 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars
new file mode 100644
index 0000000000..cbec8a44a3
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/macos-11.vars
@@ -0,0 +1,15 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool variables macos-11 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='brew'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip3'
+PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
+PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv'
+CPAN_PKGS='Test::Harness'
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index dde8270301..b3d79bc429 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -10,3 +10,4 @@ include:
- local: '/.gitlab-ci.d/buildtest.yml'
- local: '/.gitlab-ci.d/static_checks.yml'
- local: '/.gitlab-ci.d/custom-runners.yml'
+ - local: '/.gitlab-ci.d/cirrus.yml'
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 08/40] cirrus: delete FreeBSD and macOS jobs
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (6 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 07/40] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 09/40] hw/usb/ccid: remove references to NSS Alex Bennée
` (31 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Ed Maste, Willian Rampazzo, mahmoudabdalghany, aaron,
cota, Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, Li-Wen Hsu, aurelien,
Yonggang Luo
From: Daniel P. Berrangé <berrange@redhat.com>
The builds for these two platforms can now be performed from GitLab CI
using cirrus-run.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210625172211.451010-4-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.cirrus.yml | 55 -----------------------------------------------------
1 file changed, 55 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f4bf49b704..02c43a074a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,61 +1,6 @@
env:
CIRRUS_CLONE_DEPTH: 1
-freebsd_12_task:
- freebsd_instance:
- image_family: freebsd-12-2
- cpu: 8
- memory: 8G
- install_script:
- - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ;
- - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed
- nettle perl5 pixman pkgconf png usbredir ninja
- script:
- - mkdir build
- - cd build
- # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
- # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
- - ../configure --enable-werror --disable-gnutls
- || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake -j$(sysctl -n hw.ncpu) check V=1
-
-macos_task:
- osx_instance:
- image: catalina-base
- install_script:
- - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja
- script:
- - mkdir build
- - cd build
- - ../configure --python=/usr/local/bin/python3 --enable-werror
- --extra-cflags='-Wno-error=deprecated-declarations'
- || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake check-unit V=1
- - gmake check-block V=1
- - gmake check-qapi-schema V=1
- - gmake check-softfloat V=1
- - gmake check-qtest-x86_64 V=1
-
-macos_xcode_task:
- osx_instance:
- # this is an alias for the latest Xcode
- image: catalina-xcode
- install_script:
- - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja
- script:
- - mkdir build
- - cd build
- - ../configure --extra-cflags='-Wno-error=deprecated-declarations' --enable-modules
- --enable-werror --cc=clang || { cat config.log meson-logs/meson-log.txt; exit 1; }
- - gmake -j$(sysctl -n hw.ncpu)
- - gmake check-unit V=1
- - gmake check-block V=1
- - gmake check-qapi-schema V=1
- - gmake check-softfloat V=1
- - gmake check-qtest-x86_64 V=1
-
windows_msys2_task:
timeout_in: 90m
windows_container:
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 09/40] hw/usb/ccid: remove references to NSS
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (7 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 08/40] cirrus: delete FreeBSD and macOS jobs Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 10/40] tests/docker: don't use BUILDKIT in GitLab either Alex Bennée
` (30 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Peter Maydell, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
The NSS package was previously pre-requisite for building CCID related
features, however, this became obsolete when the libcacard library was
spun off to a separate project:
commit 7b02f5447c64d1854468f758398c9f6fe9e5721f
Author: Marc-André Lureau <marcandre.lureau@redhat.com>
Date: Sun Aug 30 11:48:40 2015 +0200
libcacard: use the standalone project
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/ccid.txt | 15 +++++++--------
.travis.yml | 12 ++++++------
scripts/coverity-scan/coverity-scan.docker | 1 -
tests/docker/dockerfiles/fedora.docker | 2 +-
tests/docker/dockerfiles/ubuntu.docker | 1 -
tests/docker/dockerfiles/ubuntu1804.docker | 1 -
tests/docker/dockerfiles/ubuntu2004.docker | 1 -
7 files changed, 14 insertions(+), 19 deletions(-)
diff --git a/docs/ccid.txt b/docs/ccid.txt
index c97fbd2de0..2b85b1bd42 100644
--- a/docs/ccid.txt
+++ b/docs/ccid.txt
@@ -34,15 +34,14 @@ reader and smart card (i.e. not backed by a physical device) using this device.
2. Building
-The cryptographic functions and access to the physical card is done via NSS.
-
-Installing NSS:
+The cryptographic functions and access to the physical card is done via the
+libcacard library, whose development package must be installed prior to
+building QEMU:
In redhat/fedora:
- yum install nss-devel
-In ubuntu/debian:
- apt-get install libnss3-dev
- (not tested on ubuntu)
+ yum install libcacard-devel
+In ubuntu:
+ apt-get install libcacard-dev
Configuring and building:
./configure --enable-smartcard && make
@@ -51,7 +50,7 @@ Configuring and building:
3. Using ccid-card-emulated with hardware
Assuming you have a working smartcard on the host with the current
-user, using NSS, qemu acts as another NSS client using ccid-card-emulated:
+user, using libcacard, QEMU acts as another client using ccid-card-emulated:
qemu -usb -device usb-ccid -device ccid-card-emulated
diff --git a/.travis.yml b/.travis.yml
index 4609240b5a..0faddf7b4e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -27,6 +27,7 @@ addons:
- libattr1-dev
- libbrlapi-dev
- libcap-ng-dev
+ - libcacard-dev
- libgcc-7-dev
- libgnutls28-dev
- libgtk-3-dev
@@ -34,7 +35,6 @@ addons:
- liblttng-ust-dev
- libncurses5-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libpng-dev
- librados-dev
@@ -129,6 +129,7 @@ jobs:
- libaio-dev
- libattr1-dev
- libbrlapi-dev
+ - libcacard-dev
- libcap-ng-dev
- libgcrypt20-dev
- libgnutls28-dev
@@ -137,7 +138,6 @@ jobs:
- liblttng-ust-dev
- libncurses5-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libpng-dev
- librados-dev
@@ -163,6 +163,7 @@ jobs:
- libaio-dev
- libattr1-dev
- libbrlapi-dev
+ - libcacard-dev
- libcap-ng-dev
- libgcrypt20-dev
- libgnutls28-dev
@@ -171,7 +172,6 @@ jobs:
- liblttng-ust-dev
- libncurses5-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libpng-dev
- librados-dev
@@ -196,6 +196,7 @@ jobs:
- libaio-dev
- libattr1-dev
- libbrlapi-dev
+ - libcacard-dev
- libcap-ng-dev
- libgcrypt20-dev
- libgnutls28-dev
@@ -204,7 +205,6 @@ jobs:
- liblttng-ust-dev
- libncurses5-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libpng-dev
- librados-dev
@@ -238,6 +238,7 @@ jobs:
apt_packages:
- libaio-dev
- libattr1-dev
+ - libcacard-dev
- libcap-ng-dev
- libgnutls28-dev
- libiscsi-dev
@@ -245,7 +246,6 @@ jobs:
- liblzo2-dev
- libncurses-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libsdl2-dev
- libsdl2-image-dev
@@ -281,6 +281,7 @@ jobs:
- libaio-dev
- libattr1-dev
- libbrlapi-dev
+ - libcacard-dev
- libcap-ng-dev
- libgcrypt20-dev
- libgnutls28-dev
@@ -289,7 +290,6 @@ jobs:
- liblttng-ust-dev
- libncurses5-dev
- libnfs-dev
- - libnss3-dev
- libpixman-1-dev
- libpng-dev
- librados-dev
diff --git a/scripts/coverity-scan/coverity-scan.docker b/scripts/coverity-scan/coverity-scan.docker
index 501ac67233..ecff6ac5b4 100644
--- a/scripts/coverity-scan/coverity-scan.docker
+++ b/scripts/coverity-scan/coverity-scan.docker
@@ -93,7 +93,6 @@ ENV PACKAGES \
mingw64-SDL2 \
ncurses-devel \
nettle-devel \
- nss-devel \
numactl-devel \
perl \
perl-Test-Harness \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 00cac5d61c..9dde3f6a78 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -28,6 +28,7 @@ ENV PACKAGES \
libasan \
libattr-devel \
libblockdev-mpath-devel \
+ libcacard-devel \
libcap-ng-devel \
libcurl-devel \
libepoxy-devel \
@@ -82,7 +83,6 @@ ENV PACKAGES \
ncurses-devel \
nettle-devel \
ninja-build \
- nss-devel \
numactl-devel \
perl \
perl-Test-Harness \
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index 24d1647a65..100cfa76e3 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -40,7 +40,6 @@ ENV PACKAGES \
libncurses5-dev \
libncursesw5-dev \
libnfs-dev \
- libnss3-dev \
libnuma-dev \
libpixman-1-dev \
libpng-dev \
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index 2f1ec7c42b..86114be23a 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -27,7 +27,6 @@ ENV PACKAGES \
libncurses5-dev \
libncursesw5-dev \
libnfs-dev \
- libnss3-dev \
libnuma-dev \
libpixman-1-dev \
librados-dev \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index fe993fe2a3..b33ed139d9 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -31,7 +31,6 @@ ENV PACKAGES flex bison \
libncurses5-dev \
libncursesw5-dev \
libnfs-dev \
- libnss3-dev \
libnuma-dev \
libpixman-1-dev \
librados-dev \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 10/40] tests/docker: don't use BUILDKIT in GitLab either
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (8 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 09/40] hw/usb/ccid: remove references to NSS Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 11/40] tests/docker: use project specific container registries Alex Bennée
` (29 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
Using BUILDKIT breaks with certain container registries such as CentOS,
with docker build reporting an error such as
failed to solve with frontend dockerfile.v0:
failed to build LLB: failed to load cache key:
unexpected status code
https://registry.centos.org/v2/centos/manifests/7:
403 Forbidden
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210623142245.307776-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/docker.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 4d9bb7c7ed..78dd13171e 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -228,7 +228,9 @@ class Docker(object):
def __init__(self):
self._command = _guess_engine_command()
- if "docker" in self._command and "TRAVIS" not in os.environ:
+ if ("docker" in self._command and
+ "TRAVIS" not in os.environ and
+ "GITLAB_CI" not in os.environ):
os.environ["DOCKER_BUILDKIT"] = "1"
self._buildkit = True
else:
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 11/40] tests/docker: use project specific container registries
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (9 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 10/40] tests/docker: don't use BUILDKIT in GitLab either Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 12/40] tests/docker: use explicit docker.io registry Alex Bennée
` (28 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
Since Docker Hub has started to enforce pull rate limits on clients, it
is preferrable to use project specific container registries where they
are available. Fedora and OpenSUSE projects provide registries.
The images in these registries are also refreshed on a more regular
basis than the ones in docker hub, so the package update should
generally be faster.
While CentOS also has a registry it is considerably outdated compared
to docker.io, and also only provides x86 images, while docker.io images
are multi-arch.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-4-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/fedora-cris-cross.docker | 2 +-
tests/docker/dockerfiles/fedora-i386-cross.docker | 2 +-
tests/docker/dockerfiles/fedora-win32-cross.docker | 2 +-
tests/docker/dockerfiles/fedora-win64-cross.docker | 2 +-
tests/docker/dockerfiles/fedora.docker | 2 +-
tests/docker/dockerfiles/opensuse-leap.docker | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/tests/docker/dockerfiles/fedora-cris-cross.docker b/tests/docker/dockerfiles/fedora-cris-cross.docker
index 1dfff6e0b9..91c373fdd3 100644
--- a/tests/docker/dockerfiles/fedora-cris-cross.docker
+++ b/tests/docker/dockerfiles/fedora-cris-cross.docker
@@ -2,7 +2,7 @@
# Cross compiler for cris system tests
#
-FROM fedora:33
+FROM registry.fedoraproject.org/fedora:33
ENV PACKAGES gcc-cris-linux-gnu
RUN dnf install -y $PACKAGES
RUN rpm -q $PACKAGES | sort > /packages.txt
diff --git a/tests/docker/dockerfiles/fedora-i386-cross.docker b/tests/docker/dockerfiles/fedora-i386-cross.docker
index 8004fd8ee5..dbb8195eb1 100644
--- a/tests/docker/dockerfiles/fedora-i386-cross.docker
+++ b/tests/docker/dockerfiles/fedora-i386-cross.docker
@@ -1,4 +1,4 @@
-FROM fedora:33
+FROM registry.fedoraproject.org/fedora:33
ENV PACKAGES \
bzip2 \
ccache \
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index a638afb525..9fed35f4e1 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -1,4 +1,4 @@
-FROM fedora:33
+FROM registry.fedoraproject.org/fedora:33
# Please keep this list sorted alphabetically
ENV PACKAGES \
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index f53007ac86..fb641f6104 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -1,4 +1,4 @@
-FROM fedora:33
+FROM registry.fedoraproject.org/fedora:33
# Please keep this list sorted alphabetically
ENV PACKAGES \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 9dde3f6a78..a506e68d33 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -1,4 +1,4 @@
-FROM fedora:33
+FROM registry.fedoraproject.org/fedora:33
# Please keep this list sorted alphabetically
ENV PACKAGES \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index f7e1cbfbe6..7ebff1b3a8 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -1,4 +1,4 @@
-FROM opensuse/leap:15.2
+FROM registry.opensuse.org/opensuse/leap:15.2
# Please keep this list sorted alphabetically
ENV PACKAGES \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 12/40] tests/docker: use explicit docker.io registry
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (10 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 11/40] tests/docker: use project specific container registries Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 13/40] tests/docker: remove FEATURES env var from templates Alex Bennée
` (27 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
It is good practice to use an explicit registry for referencing the base
image. This is because some distros will inject their own registries
into the search path. For example registry.fedoraproject.org comes ahead
of docker.io. Using an explicit registry avoids wasting time querying
multiple registries for images that they won't have.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-5-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/centos8.docker | 2 +-
tests/docker/dockerfiles/debian-xtensa-cross.docker | 2 +-
tests/docker/dockerfiles/debian10.docker | 2 +-
tests/docker/dockerfiles/debian11.docker | 2 +-
tests/docker/dockerfiles/ubuntu.docker | 2 +-
tests/docker/dockerfiles/ubuntu1804.docker | 2 +-
tests/docker/dockerfiles/ubuntu2004.docker | 2 +-
7 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 03e0440e03..8f810810f3 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -1,4 +1,4 @@
-FROM centos:8.3.2011
+FROM docker.io/centos:8
RUN dnf -y update
ENV PACKAGES \
diff --git a/tests/docker/dockerfiles/debian-xtensa-cross.docker b/tests/docker/dockerfiles/debian-xtensa-cross.docker
index ba4148299c..2f11b3b7bc 100644
--- a/tests/docker/dockerfiles/debian-xtensa-cross.docker
+++ b/tests/docker/dockerfiles/debian-xtensa-cross.docker
@@ -5,7 +5,7 @@
# using a prebuilt toolchains for Xtensa cores from:
# https://github.com/foss-xtensa/toolchain/releases
#
-FROM debian:stretch-slim
+FROM docker.io/library/debian:stretch-slim
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
diff --git a/tests/docker/dockerfiles/debian10.docker b/tests/docker/dockerfiles/debian10.docker
index 4ffe47671e..a27b88df55 100644
--- a/tests/docker/dockerfiles/debian10.docker
+++ b/tests/docker/dockerfiles/debian10.docker
@@ -7,7 +7,7 @@
# On its own you can't build much but the docker-foo-cross targets
# build on top of the base debian image.
#
-FROM debian:buster-slim
+FROM docker.io/library/debian:buster-slim
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
diff --git a/tests/docker/dockerfiles/debian11.docker b/tests/docker/dockerfiles/debian11.docker
index 5adfd62d55..febf884f8f 100644
--- a/tests/docker/dockerfiles/debian11.docker
+++ b/tests/docker/dockerfiles/debian11.docker
@@ -8,7 +8,7 @@
# On its own you can't build much but the docker-foo-cross targets
# build on top of the base debian image.
#
-FROM debian:bullseye-slim
+FROM docker.io/library/debian:bullseye-slim
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index 100cfa76e3..e0ff425c01 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -9,7 +9,7 @@
# system won't pick up that it has changed.
#
-FROM ubuntu:20.04
+FROM docker.io/library/ubuntu:20.04
ENV PACKAGES \
ccache \
clang \
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index 86114be23a..adcdef8244 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -1,4 +1,4 @@
-FROM ubuntu:18.04
+FROM docker.io/library/ubuntu:18.04
ENV PACKAGES \
ccache \
clang \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index b33ed139d9..78755bc2e3 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,4 +1,4 @@
-FROM ubuntu:20.04
+FROM docker.io/library/ubuntu:20.04
ENV PACKAGES flex bison \
bsdmainutils \
ccache \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 13/40] tests/docker: remove FEATURES env var from templates
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (11 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 12/40] tests/docker: use explicit docker.io registry Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 14/40] tests/docker: fix sorting in package lists Alex Bennée
` (26 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
In preparation for switching to auto-generated dockerfiles, remove the
FEATURES env variable. The equivalent functionality can be achieved in
most cases by just looking for existance of a binary.
The cases which don't correspond to binaries are simply dropped because
configure/meson will probe for any requested feature anyway.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-6-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/common.rc | 19 ++++++++++++++-----
tests/docker/dockerfiles/debian10.docker | 2 --
.../dockerfiles/fedora-win32-cross.docker | 1 -
.../dockerfiles/fedora-win64-cross.docker | 1 -
tests/docker/dockerfiles/fedora.docker | 1 -
tests/docker/dockerfiles/ubuntu.docker | 1 -
tests/docker/dockerfiles/ubuntu1804.docker | 1 -
tests/docker/dockerfiles/ubuntu2004.docker | 1 -
tests/docker/run | 3 ---
tests/docker/test-clang | 2 +-
tests/docker/test-debug | 2 +-
tests/docker/test-mingw | 3 ++-
tests/docker/test-misc | 2 +-
tests/docker/test-tsan | 2 +-
14 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index ebc5b97ecf..c5cc33d366 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -15,14 +15,23 @@
# overriden by TARGET_LIST if the user sets it.
DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"}
-requires()
+requires_binary()
{
+ found=0
for c in $@; do
- if ! echo "$FEATURES" | grep -wq -e "$c"; then
- echo "Prerequisite '$c' not present, skip"
- exit 0
- fi
+ for d in /bin /usr/bin /usr/local/bin
+ do
+ if test -f "$d/$c"
+ then
+ found=1
+ fi
+ done
done
+ if test "$found" != "1"
+ then
+ echo "Prerequisite '$c' not present, skip"
+ exit 0
+ fi
}
configure_qemu()
diff --git a/tests/docker/dockerfiles/debian10.docker b/tests/docker/dockerfiles/debian10.docker
index a27b88df55..b414af1b9f 100644
--- a/tests/docker/dockerfiles/debian10.docker
+++ b/tests/docker/dockerfiles/debian10.docker
@@ -35,5 +35,3 @@ RUN apt update && \
python3-sphinx \
python3-sphinx-rtd-theme \
$(apt-get -s build-dep --arch-only qemu | egrep ^Inst | fgrep '[all]' | cut -d\ -f2)
-
-ENV FEATURES docs
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index 9fed35f4e1..5a03e1af43 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -37,7 +37,6 @@ ENV PACKAGES \
RUN dnf install -y $PACKAGES
RUN rpm -q $PACKAGES | sort > /packages.txt
-ENV FEATURES mingw
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=i686-w64-mingw32-
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index fb641f6104..ff706040c4 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -33,7 +33,6 @@ ENV PACKAGES \
RUN dnf install -y $PACKAGES
RUN rpm -q $PACKAGES | sort > /packages.txt
-ENV FEATURES mingw
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=x86_64-w64-mingw32- --disable-capstone
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index a506e68d33..64a413f5e0 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -117,4 +117,3 @@ ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3
RUN dnf install -y $PACKAGES
RUN rpm -q $PACKAGES | sort > /packages.txt
ENV PATH $PATH:/usr/libexec/python3-sphinx/
-ENV FEATURES mingw clang pyyaml asan docs
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index e0ff425c01..f0e0180d21 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -69,4 +69,3 @@ ENV PACKAGES \
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
-ENV FEATURES clang pyyaml sdl2 docs
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index adcdef8244..0acdb0d9ad 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -55,7 +55,6 @@ ENV PACKAGES \
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
-ENV FEATURES clang pyyaml sdl2 docs
# https://bugs.launchpad.net/qemu/+bug/1838763
ENV QEMU_CONFIGURE_OPTS --disable-libssh
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 78755bc2e3..88b3cfa136 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -70,7 +70,6 @@ ENV PACKAGES flex bison \
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
-ENV FEATURES clang tsan pyyaml sdl2
# Apply patch https://reviews.llvm.org/D75820
# This is required for TSan in clang-10 to compile with QEMU.
diff --git a/tests/docker/run b/tests/docker/run
index 8edc7026ee..421393046b 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -30,9 +30,6 @@ mkdir -p $TEST_DIR/{src,build,install}
# Extract the source tarballs
tar -C $TEST_DIR/src -xf $BASE/qemu.tar || { echo "Failed to untar source"; exit 2; }
-if test -f $TEST_DIR/src/Makefile; then
- export FEATURES="$FEATURES dtc"
-fi
if test -n "$SHOW_ENV"; then
if test -f /packages.txt; then
diff --git a/tests/docker/test-clang b/tests/docker/test-clang
index 8c51ead518..b57e0119d9 100755
--- a/tests/docker/test-clang
+++ b/tests/docker/test-clang
@@ -13,7 +13,7 @@
. common.rc
-requires clang
+requires_binary clang
cd "$BUILD_DIR"
diff --git a/tests/docker/test-debug b/tests/docker/test-debug
index c050fa0d93..f52f16328c 100755
--- a/tests/docker/test-debug
+++ b/tests/docker/test-debug
@@ -14,7 +14,7 @@
. common.rc
-requires clang asan
+requires_binary clang
cd "$BUILD_DIR"
diff --git a/tests/docker/test-mingw b/tests/docker/test-mingw
index c30eb654eb..0bc6d78872 100755
--- a/tests/docker/test-mingw
+++ b/tests/docker/test-mingw
@@ -13,7 +13,8 @@
. common.rc
-requires mingw dtc
+requires_binary x86_64-w64-mingw32-gcc
+requires_binary i686-w64-mingw32-gcc
cd "$BUILD_DIR"
diff --git a/tests/docker/test-misc b/tests/docker/test-misc
index cc94a738dd..2a3c2c2e1c 100755
--- a/tests/docker/test-misc
+++ b/tests/docker/test-misc
@@ -14,7 +14,7 @@
. common.rc
-requires docs
+requires_binary sphinx-build-3 sphinx-build
cd "$BUILD_DIR"
diff --git a/tests/docker/test-tsan b/tests/docker/test-tsan
index eb40ac45b7..53d90d2f79 100755
--- a/tests/docker/test-tsan
+++ b/tests/docker/test-tsan
@@ -17,7 +17,7 @@
setup_tsan()
{
- requires clang tsan
+ requires_binary clang
tsan_log_dir="/tmp/qemu-test/build/tsan"
mkdir -p $tsan_log_dir > /dev/null || true
EXTRA_CONFIGURE_OPTS="${EXTRA_CONFIGURE_OPTS} --enable-tsan \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 14/40] tests/docker: fix sorting in package lists
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (12 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 13/40] tests/docker: remove FEATURES env var from templates Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 15/40] tests/docker: fix mistakes in centos " Alex Bennée
` (25 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This will make diffs in later patches clearer.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-7-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/centos8.docker | 2 +-
tests/docker/dockerfiles/fedora.docker | 4 ++--
tests/docker/dockerfiles/opensuse-leap.docker | 16 ++++++++--------
tests/docker/dockerfiles/ubuntu1804.docker | 4 ++--
tests/docker/dockerfiles/ubuntu2004.docker | 8 +++++---
5 files changed, 18 insertions(+), 16 deletions(-)
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 8f810810f3..ee52305646 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -22,9 +22,9 @@ ENV PACKAGES \
lzo-devel \
make \
mesa-libEGL-devel \
- nmap-ncat \
nettle-devel \
ninja-build \
+ nmap-ncat \
perl-Test-Harness \
pixman-devel \
python36 \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 64a413f5e0..4a0a84eb43 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -2,6 +2,7 @@ FROM registry.fedoraproject.org/fedora:33
# Please keep this list sorted alphabetically
ENV PACKAGES \
+ SDL2-devel \
bc \
brlapi-devel \
bzip2 \
@@ -79,10 +80,10 @@ ENV PACKAGES \
mingw64-pixman \
mingw64-pkg-config \
mingw64-SDL2 \
- nmap-ncat \
ncurses-devel \
nettle-devel \
ninja-build \
+ nmap-ncat \
numactl-devel \
perl \
perl-Test-Harness \
@@ -97,7 +98,6 @@ ENV PACKAGES \
python3-sphinx_rtd_theme \
python3-virtualenv \
rdma-core-devel \
- SDL2-devel \
snappy-devel \
sparse \
spice-server-devel \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 7ebff1b3a8..6011447181 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -9,46 +9,46 @@ ENV PACKAGES \
cyrus-sasl-devel \
gcc \
gcc-c++ \
- mkisofs \
gettext-runtime \
git \
glib2-devel \
glusterfs-devel \
- libgnutls-devel \
gtk3-devel \
+ libSDL2-devel \
+ libSDL2_image-devel \
libaio-devel \
libattr-devel \
libcap-ng-devel \
libepoxy-devel \
libfdt-devel \
+ libgnutls-devel \
libiscsi-devel \
libjpeg8-devel \
+ libnuma-devel \
+ libpixman-1-0-devel \
libpmem-devel \
libpng16-devel \
librbd-devel \
libseccomp-devel \
+ libspice-server-devel \
libssh-devel \
lzo-devel \
make \
- libSDL2_image-devel \
+ mkisofs \
ncurses-devel \
ninja \
- libnuma-devel \
perl \
- libpixman-1-0-devel \
python3-base \
python3-virtualenv \
rdma-core-devel \
- libSDL2-devel \
snappy-devel \
- libspice-server-devel \
systemd-devel \
systemtap-sdt-devel \
tar \
usbredir-devel \
virglrenderer-devel \
- xen-devel \
vte-devel \
+ xen-devel \
zlib-devel
ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index 0acdb0d9ad..a50a35e6fe 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -46,10 +46,10 @@ ENV PACKAGES \
libxen-dev \
libzstd-dev \
make \
- python3-yaml \
+ ninja-build \
python3-sphinx \
python3-sphinx-rtd-theme \
- ninja-build \
+ python3-yaml \
sparse \
xfslibs-dev
RUN apt-get update && \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 88b3cfa136..eee2ef3cac 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,8 +1,10 @@
FROM docker.io/library/ubuntu:20.04
-ENV PACKAGES flex bison \
+ENV PACKAGES \
+ bison \
bsdmainutils \
ccache \
clang-10\
+ flex \
gcc \
gcovr \
genisoimage \
@@ -65,8 +67,8 @@ ENV PACKAGES flex bison \
sparse \
tesseract-ocr \
tesseract-ocr-eng \
- xfslibs-dev\
- vim
+ vim \
+ xfslibs-dev
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 15/40] tests/docker: fix mistakes in centos package lists
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (13 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 14/40] tests/docker: fix sorting in package lists Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 16/40] tests/docker: fix mistakes in fedora package list Alex Bennée
` (24 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
mesa-libEGL-devel is not used in QEMU at all, but mesa-libgbm-devel is.
spice-glib-devel is not use in QEMU at all, but spice-protocol is.
We also need the -devel package for spice-server, not the runtime.
There is no need to specifically refer to python36, we can just
use python3 as in other distros.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-8-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/centos8.docker | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index ee52305646..5f1c57b4ad 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -21,16 +21,16 @@ ENV PACKAGES \
libgcrypt-devel \
lzo-devel \
make \
- mesa-libEGL-devel \
+ mesa-libgbm-devel \
nettle-devel \
ninja-build \
nmap-ncat \
perl-Test-Harness \
pixman-devel \
- python36 \
+ python3 \
rdma-core-devel \
- spice-glib-devel \
- spice-server \
+ spice-protocol \
+ spice-server-devel \
systemtap-sdt-devel \
tar \
zlib-devel
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 16/40] tests/docker: fix mistakes in fedora package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (14 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 15/40] tests/docker: fix mistakes in centos " Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 17/40] tests/docker: fix mistakes in ubuntu package lists Alex Bennée
` (23 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
libblockdev-mpath-devel is not used by QEMU, rather it wants
device-mapper-multipath-devel.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-9-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/fedora.docker | 1 -
1 file changed, 1 deletion(-)
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 4a0a84eb43..f667f03cc5 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -28,7 +28,6 @@ ENV PACKAGES \
libaio-devel \
libasan \
libattr-devel \
- libblockdev-mpath-devel \
libcacard-devel \
libcap-ng-devel \
libcurl-devel \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 17/40] tests/docker: fix mistakes in ubuntu package lists
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (15 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 16/40] tests/docker: fix mistakes in fedora package list Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 18/40] tests/docker: remove mingw packages from Fedora Alex Bennée
` (22 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
librados-dev is not required by QEMU directly, only librbd-dev.
glusterfs-common is not directly needed by QEMU.
QEMU uses ncursesw only on non-Windows hosts.
The clang package is clang 10.
flex and bison are not required by QEMU.
Standardize on nmap ncat implementation to match Fedora/CentOS.
Remove vim since it is not a build pre-requisite and no other containers
include it.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-10-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/ubuntu1804.docker | 3 ---
tests/docker/dockerfiles/ubuntu2004.docker | 10 ++--------
2 files changed, 2 insertions(+), 11 deletions(-)
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index a50a35e6fe..ee8545e4b1 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -5,7 +5,6 @@ ENV PACKAGES \
gcc \
gettext \
git \
- glusterfs-common \
libaio-dev \
libattr1-dev \
libbrlapi-dev \
@@ -24,12 +23,10 @@ ENV PACKAGES \
libjemalloc-dev \
libjpeg-turbo8-dev \
liblzo2-dev \
- libncurses5-dev \
libncursesw5-dev \
libnfs-dev \
libnuma-dev \
libpixman-1-dev \
- librados-dev \
librbd-dev \
librdmacm-dev \
libsasl2-dev \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index eee2ef3cac..25f56adfb2 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,16 +1,13 @@
FROM docker.io/library/ubuntu:20.04
ENV PACKAGES \
- bison \
bsdmainutils \
ccache \
- clang-10\
- flex \
+ clang \
gcc \
gcovr \
genisoimage \
gettext \
git \
- glusterfs-common \
libaio-dev \
libattr1-dev \
libbrlapi-dev \
@@ -30,12 +27,10 @@ ENV PACKAGES \
libjpeg-turbo8-dev \
liblttng-ust-dev \
liblzo2-dev \
- libncurses5-dev \
libncursesw5-dev \
libnfs-dev \
libnuma-dev \
libpixman-1-dev \
- librados-dev \
librbd-dev \
librdmacm-dev \
libsasl2-dev \
@@ -53,7 +48,7 @@ ENV PACKAGES \
libxen-dev \
libzstd-dev \
make \
- netcat-openbsd \
+ ncat \
ninja-build \
python3-numpy \
python3-opencv \
@@ -67,7 +62,6 @@ ENV PACKAGES \
sparse \
tesseract-ocr \
tesseract-ocr-eng \
- vim \
xfslibs-dev
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 18/40] tests/docker: remove mingw packages from Fedora
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (16 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 17/40] tests/docker: fix mistakes in ubuntu package lists Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 19/40] tests/docker: expand centos8 package list Alex Bennée
` (21 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
There are dedicated containers providing mingw packages for Fedora.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-11-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/fedora.docker | 27 --------------------------
1 file changed, 27 deletions(-)
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index f667f03cc5..5849ea7617 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -52,33 +52,6 @@ ENV PACKAGES \
lzo-devel \
make \
meson \
- mingw32-bzip2 \
- mingw32-curl \
- mingw32-glib2 \
- mingw32-gmp \
- mingw32-gnutls \
- mingw32-gtk3 \
- mingw32-libjpeg-turbo \
- mingw32-libpng \
- mingw32-libtasn1 \
- mingw32-nettle \
- mingw32-nsis \
- mingw32-pixman \
- mingw32-pkg-config \
- mingw32-SDL2 \
- mingw64-bzip2 \
- mingw64-curl \
- mingw64-glib2 \
- mingw64-gmp \
- mingw64-gnutls \
- mingw64-gtk3 \
- mingw64-libjpeg-turbo \
- mingw64-libpng \
- mingw64-libtasn1 \
- mingw64-nettle \
- mingw64-pixman \
- mingw64-pkg-config \
- mingw64-SDL2 \
ncurses-devel \
nettle-devel \
ninja-build \
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 19/40] tests/docker: expand centos8 package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (17 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 18/40] tests/docker: remove mingw packages from Fedora Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 20/40] tests/docker: expand fedora " Alex Bennée
` (20 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This is the fully expanded list of build pre-requisites QEMU can
conceivably use in any scenario.
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210623142245.307776-12-berrange@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: added centos-release-advanced-virtualization/epel-release]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/centos8.docker | 70 +++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 5f1c57b4ad..46398c61ee 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -3,39 +3,109 @@ FROM docker.io/centos:8
RUN dnf -y update
ENV PACKAGES \
SDL2-devel \
+ alsa-lib-devel \
+ bc \
+ brlapi-devel \
bzip2 \
bzip2-devel \
+ ca-certificates \
+ capstone-devel \
+ ccache \
+ clang \
+ ctags \
+ cyrus-sasl-devel \
+ daxctl-devel \
dbus-daemon \
+ device-mapper-multipath-devel \
diffutils \
+ findutils \
gcc \
gcc-c++ \
genisoimage \
gettext \
git \
glib2-devel \
+ glibc-langpack-en \
+ glibc-static \
+ glusterfs-api-devel \
+ gnutls-devel \
+ gtk3-devel \
+ hostname \
+ jemalloc-devel \
libaio-devel \
+ libasan \
+ libattr-devel \
libbpf-devel \
+ libcacard-devel \
+ libcap-ng-devel \
+ libcurl-devel \
+ libdrm-devel \
libepoxy-devel \
libfdt-devel \
libffi-devel \
libgcrypt-devel \
+ libiscsi-devel \
+ libjpeg-devel \
+ libnfs-devel \
+ libpmem-devel \
+ libpng-devel \
+ librbd-devel \
+ libseccomp-devel \
+ libslirp-devel \
+ libssh-devel \
+ libtasn1-devel \
+ libubsan \
+ libudev-devel \
+ libusbx-devel \
+ libxml2-devel \
+ libzstd-devel \
+ llvm \
lzo-devel \
make \
mesa-libgbm-devel \
+ ncurses-devel \
nettle-devel \
ninja-build \
nmap-ncat \
+ numactl-devel \
+ openssh-clients \
+ pam-devel \
+ perl \
perl-Test-Harness \
pixman-devel \
+ pkgconfig \
+ pulseaudio-libs-devel \
python3 \
+ python3-PyYAML \
+ python3-numpy \
+ python3-pillow \
+ python3-pip \
+ python3-setuptools \
+ python3-sphinx \
+ python3-sphinx_rtd_theme \
+ python3-virtualenv \
+ python3-wheel \
rdma-core-devel \
+ rpm \
+ sed \
+ snappy-devel \
spice-protocol \
spice-server-devel \
+ systemd-devel \
systemtap-sdt-devel \
tar \
+ texinfo \
+ usbredir-devel \
+ util-linux \
+ virglrenderer-devel \
+ vte291-devel \
+ which \
+ xfsprogs-devel \
zlib-devel
RUN dnf install -y dnf-plugins-core && \
dnf config-manager --set-enabled powertools && \
+ dnf install -y centos-release-advanced-virtualization && \
+ dnf install -y epel-release && \
dnf install -y $PACKAGES
RUN rpm -q $PACKAGES | sort > /packages.txt
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 20/40] tests/docker: expand fedora package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (18 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 19/40] tests/docker: expand centos8 package list Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 21/40] tests/docker: expand ubuntu1804 " Alex Bennée
` (19 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This is the fully expanded list of build pre-requisites QEMU can
conceivably use in any scenario.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-13-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/fedora.docker | 30 ++++++++++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 5849ea7617..eec1add7f6 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -3,63 +3,83 @@ FROM registry.fedoraproject.org/fedora:33
# Please keep this list sorted alphabetically
ENV PACKAGES \
SDL2-devel \
+ SDL2_image-devel \
+ alsa-lib-devel \
bc \
brlapi-devel \
bzip2 \
bzip2-devel \
+ ca-certificates \
capstone-devel \
ccache \
clang \
+ ctags \
cyrus-sasl-devel \
+ daxctl-devel \
dbus-daemon \
device-mapper-multipath-devel \
diffutils \
findutils \
gcc \
gcc-c++ \
+ gcovr \
genisoimage \
gettext \
git \
glib2-devel \
+ glibc-langpack-en \
+ glibc-static \
glusterfs-api-devel \
gnutls-devel \
gtk3-devel \
hostname \
+ jemalloc-devel \
libaio-devel \
libasan \
libattr-devel \
+ libbpf-devel \
libcacard-devel \
libcap-ng-devel \
libcurl-devel \
+ libdrm-devel \
libepoxy-devel \
libfdt-devel \
- libbpf-devel \
libffi-devel \
+ libgcrypt-devel \
libiscsi-devel \
libjpeg-devel \
+ libnfs-devel \
libpmem-devel \
libpng-devel \
librbd-devel \
libseccomp-devel \
libslirp-devel \
libssh-devel \
+ libtasn1-devel \
libubsan \
libudev-devel \
+ liburing-devel \
libusbx-devel \
libxml2-devel \
libzstd-devel \
llvm \
+ lttng-ust-devel \
lzo-devel \
make \
+ mesa-libgbm-devel \
meson \
ncurses-devel \
nettle-devel \
ninja-build \
nmap-ncat \
numactl-devel \
- perl \
+ openssh-clients \
+ pam-devel \
perl-Test-Harness \
+ perl-base \
pixman-devel \
+ pkgconfig \
+ pulseaudio-libs-devel \
python3 \
python3-PyYAML \
python3-numpy \
@@ -70,19 +90,25 @@ ENV PACKAGES \
python3-sphinx_rtd_theme \
python3-virtualenv \
rdma-core-devel \
+ rpm \
+ sed \
snappy-devel \
sparse \
+ spice-protocol \
spice-server-devel \
systemd-devel \
systemtap-sdt-devel \
tar \
tesseract \
tesseract-langpack-eng \
+ texinfo \
usbredir-devel \
+ util-linux \
virglrenderer-devel \
vte291-devel \
which \
xen-devel \
+ xfsprogs-devel \
zlib-devel
ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 21/40] tests/docker: expand ubuntu1804 package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (19 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 20/40] tests/docker: expand fedora " Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 22/40] tests/docker: expand ubuntu2004 " Alex Bennée
` (18 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This is the fully expanded list of build pre-requisites QEMU can
conceivably use in any scenario.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-14-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/ubuntu1804.docker | 61 +++++++++++++++++++++-
1 file changed, 60 insertions(+), 1 deletion(-)
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index ee8545e4b1..0880bf3e29 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -1,54 +1,113 @@
FROM docker.io/library/ubuntu:18.04
ENV PACKAGES \
+ bc \
+ bsdmainutils \
+ bzip2 \
+ ca-certificates \
ccache \
clang \
+ dbus \
+ debianutils \
+ diffutils \
+ exuberant-ctags \
+ findutils \
+ g++ \
gcc \
+ gcovr \
+ genisoimage \
gettext \
git \
+ glusterfs-common \
+ hostname \
libaio-dev \
+ libasan5 \
+ libasound2-dev \
libattr1-dev \
libbrlapi-dev \
libbz2-dev \
+ libc6-dev \
libcacard-dev \
libcap-ng-dev \
+ libcapstone-dev \
libcurl4-gnutls-dev \
+ libdaxctl-dev \
libdrm-dev \
libepoxy-dev \
libfdt-dev \
libffi-dev \
libgbm-dev \
+ libgcrypt20-dev \
+ libglib2.0-dev \
+ libgnutls28-dev \
libgtk-3-dev \
libibverbs-dev \
libiscsi-dev \
libjemalloc-dev \
libjpeg-turbo8-dev \
+ liblttng-ust-dev \
liblzo2-dev \
libncursesw5-dev \
libnfs-dev \
libnuma-dev \
+ libpam0g-dev \
libpixman-1-dev \
+ libpmem-dev \
+ libpng-dev \
+ libpulse-dev \
librbd-dev \
librdmacm-dev \
libsasl2-dev \
libsdl2-dev \
+ libsdl2-image-dev \
libseccomp-dev \
libsnappy-dev \
libspice-protocol-dev \
libspice-server-dev \
libssh-dev \
+ libsystemd-dev \
+ libtasn1-6-dev \
+ libtest-harness-perl \
+ libubsan1 \
+ libudev-dev \
libusb-1.0-0-dev \
libusbredirhost-dev \
libvdeplug-dev \
+ libvirglrenderer-dev \
libvte-2.91-dev \
libxen-dev \
+ libxml2-dev \
libzstd-dev \
+ llvm \
+ locales \
make \
+ multipath-tools \
+ netcat-openbsd \
+ nettle-dev \
ninja-build \
+ openssh-client \
+ perl-base \
+ pkgconf \
+ python3 \
+ python3-numpy \
+ python3-opencv \
+ python3-pillow \
+ python3-pip \
+ python3-setuptools \
python3-sphinx \
python3-sphinx-rtd-theme \
+ python3-venv \
+ python3-wheel \
python3-yaml \
+ rpm2cpio \
+ sed \
sparse \
- xfslibs-dev
+ systemtap-sdt-dev \
+ tar \
+ tesseract-ocr \
+ tesseract-ocr-eng \
+ texinfo \
+ xfslibs-dev \
+ zlib1g-dev
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 22/40] tests/docker: expand ubuntu2004 package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (20 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 21/40] tests/docker: expand ubuntu1804 " Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 23/40] tests/docker: expand opensuse-leap " Alex Bennée
` (17 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This is the fully expanded list of build pre-requisites QEMU can
conceivably use in any scenario.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-15-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/ubuntu2004.docker | 50 +++++++++++++++++++++-
1 file changed, 48 insertions(+), 2 deletions(-)
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 25f56adfb2..39de63d012 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,25 +1,44 @@
FROM docker.io/library/ubuntu:20.04
ENV PACKAGES \
+ bc \
bsdmainutils \
+ bzip2 \
+ ca-certificates \
ccache \
clang \
+ dbus \
+ debianutils \
+ diffutils \
+ exuberant-ctags \
+ findutils \
+ g++ \
gcc \
gcovr \
genisoimage \
gettext \
git \
+ hostname \
libaio-dev \
+ libasan5 \
+ libasound2-dev \
libattr1-dev \
libbrlapi-dev \
libbz2-dev \
+ libc6-dev \
libcacard-dev \
libcap-ng-dev \
+ libcapstone-dev \
libcurl4-gnutls-dev \
+ libdaxctl-dev \
libdrm-dev \
libepoxy-dev \
libfdt-dev \
libffi-dev \
libgbm-dev \
+ libgcrypt20-dev \
+ libglib2.0-dev \
+ libglusterfs-dev \
+ libgnutls28-dev \
libgtk-3-dev \
libibverbs-dev \
libiscsi-dev \
@@ -30,39 +49,66 @@ ENV PACKAGES \
libncursesw5-dev \
libnfs-dev \
libnuma-dev \
+ libpam0g-dev \
libpixman-1-dev \
+ libpmem-dev \
+ libpng-dev \
+ libpulse-dev \
librbd-dev \
librdmacm-dev \
libsasl2-dev \
libsdl2-dev \
+ libsdl2-image-dev \
libseccomp-dev \
libslirp-dev \
libsnappy-dev \
libspice-protocol-dev \
libspice-server-dev \
libssh-dev \
+ libsystemd-dev \
+ libtasn1-6-dev \
+ libtest-harness-perl \
+ libubsan1 \
+ libudev-dev \
libusb-1.0-0-dev \
libusbredirhost-dev \
libvdeplug-dev \
+ libvirglrenderer-dev \
libvte-2.91-dev \
libxen-dev \
+ libxml2-dev \
libzstd-dev \
+ llvm \
+ locales \
make \
+ multipath-tools \
ncat \
+ nettle-dev \
ninja-build \
+ openssh-client \
+ perl-base \
+ pkgconf \
+ python3 \
python3-numpy \
python3-opencv \
- python3-pil \
+ python3-pillow \
python3-pip \
+ python3-setuptools \
python3-sphinx \
python3-sphinx-rtd-theme \
python3-venv \
+ python3-wheel \
python3-yaml \
rpm2cpio \
+ sed \
sparse \
+ systemtap-sdt-dev \
+ tar \
tesseract-ocr \
tesseract-ocr-eng \
- xfslibs-dev
+ texinfo \
+ xfslibs-dev \
+ zlib1g-dev
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
RUN dpkg -l $PACKAGES | sort > /packages.txt
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 23/40] tests/docker: expand opensuse-leap package list
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (21 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 22/40] tests/docker: expand ubuntu2004 " Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 24/40] tests/vm: update NetBSD to 9.2 Alex Bennée
` (16 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Alex Bennée, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Wainer dos Santos Moschetta,
stefanha, crosa, pbonzini, ma.mandourr,
Philippe Mathieu-Daudé,
aurelien
From: Daniel P. Berrangé <berrange@redhat.com>
This is the fully expanded list of build pre-requisites QEMU can
conceivably use in any scenario.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-16-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/dockerfiles/opensuse-leap.docker | 59 ++++++++++++++++++-
1 file changed, 58 insertions(+), 1 deletion(-)
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 6011447181..5a8bee0289 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -2,53 +2,110 @@ FROM registry.opensuse.org/opensuse/leap:15.2
# Please keep this list sorted alphabetically
ENV PACKAGES \
+ Mesa-devel \
+ alsa-lib-devel \
bc \
brlapi-devel \
bzip2 \
+ ca-certificates \
ccache \
+ clang \
+ ctags \
cyrus-sasl-devel \
+ dbus-1 \
+ diffutils \
+ findutils \
gcc \
gcc-c++ \
+ gcovr \
gettext-runtime \
git \
glib2-devel \
+ glibc-locale \
+ glibc-static \
glusterfs-devel \
gtk3-devel \
+ hostname \
+ jemalloc-devel \
libSDL2-devel \
libSDL2_image-devel \
libaio-devel \
+ libasan6 \
libattr-devel \
+ libbpf-devel \
+ libbz2-devel \
+ libcacard-devel \
libcap-ng-devel \
+ libcurl-devel \
+ libdrm-devel \
libepoxy-devel \
libfdt-devel \
+ libffi-devel \
+ libgcrypt-devel \
libgnutls-devel \
libiscsi-devel \
libjpeg8-devel \
+ libndctl-devel \
+ libnettle-devel \
+ libnfs-devel \
libnuma-devel \
libpixman-1-0-devel \
libpmem-devel \
libpng16-devel \
+ libpulse-devel \
librbd-devel \
libseccomp-devel \
libspice-server-devel \
libssh-devel \
+ libtasn1-devel \
+ libubsan1 \
+ libudev-devel \
+ libusb-1_0-devel \
+ libxml2-devel \
+ libzstd-devel \
+ llvm \
+ lttng-ust-devel \
lzo-devel \
make \
mkisofs \
+ ncat \
ncurses-devel \
ninja \
- perl \
+ openssh \
+ pam-devel \
+ perl-Test-Harness \
+ perl-base \
+ pkgconfig \
+ python3-Pillow \
+ python3-PyYAML \
+ python3-Sphinx \
python3-base \
+ python3-numpy \
+ python3-opencv \
+ python3-pip \
+ python3-setuptools \
+ python3-sphinx_rtd_theme \
python3-virtualenv \
+ python3-wheel \
rdma-core-devel \
+ rpm \
+ sed \
snappy-devel \
+ sparse \
+ spice-protocol-devel \
systemd-devel \
systemtap-sdt-devel \
tar \
+ tesseract-ocr \
+ tesseract-ocr-traineddata-english \
+ texinfo \
usbredir-devel \
+ util-linux \
virglrenderer-devel \
vte-devel \
+ which \
xen-devel \
+ xfsprogs-devel \
zlib-devel
ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 24/40] tests/vm: update NetBSD to 9.2
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (22 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 23/40] tests/docker: expand opensuse-leap " Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 25/40] tests/vm: update openbsd to release 6.9 Alex Bennée
` (15 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, aaron, mahmoudabdalghany, Gerd Hoffmann, Kamil Rytarowski,
Wainer dos Santos Moschetta, Willian Rampazzo, minyihh, cota,
Reinoud Zandijk, Brad Smith, Thomas Huth, stefanha, crosa,
kuhn.chenqun, Alex Bennée, berrange, f4bug, robhenry,
pbonzini, ma.mandourr, Ryo ONODERA, aurelien
From: Brad Smith <brad@comstyle.com>
tests/vm: update NetBSD to 9.2
Signed-off-by: Brad Smith <brad@comstyle.com>
Message-Id: <YLe+LZ/j0pxVGeBi@humpty.home.comstyle.com>
Tested-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/vm/netbsd | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index b9efc269d2..4cc58df130 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -22,8 +22,8 @@ class NetBSDVM(basevm.BaseVM):
name = "netbsd"
arch = "x86_64"
- link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/images/NetBSD-9.1-amd64.iso"
- csum = "65bddc95945991c3b2021f9c8ded7f34c25f0a7611b7aa15a15fe23399e902307e926ae97fcd01dc1662ac67b5f6e4be643c6a2b581692ddcb616d30125066f9"
+ link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.2/images/NetBSD-9.2-amd64.iso"
+ csum = "5ee0ea101f73386b9b424f5d1041e371db3c42fdd6f4e4518dc79c4a08f31d43091ebe93425c9f0dcaaed2b51131836fe6774f33f89030b58d64709b35fda72f"
size = "20G"
pkgs = [
# tools
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 25/40] tests/vm: update openbsd to release 6.9
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (23 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 24/40] tests/vm: update NetBSD to 9.2 Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive Alex Bennée
` (14 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Wainer dos Santos Moschetta, f4bug, robhenry, Willian Rampazzo,
mahmoudabdalghany, aaron, cota, Gerd Hoffmann, stefanha, crosa,
pbonzini, ma.mandourr, Alex Bennée, aurelien, Brad Smith
From: Brad Smith <brad@comstyle.com>
tests/vm: update openbsd to release 6.9
Signed-off-by: Brad Smith <brad@comstyle.com>
Tested-by: Gerd Hoffmann <kraxel@redhat.com>
Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <YLRDeJV8qBrt9++c@humpty.home.comstyle.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/vm/openbsd | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index 4d1399378e..c4c78a80f1 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -22,8 +22,8 @@ class OpenBSDVM(basevm.BaseVM):
name = "openbsd"
arch = "x86_64"
- link = "https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/install68.iso"
- csum = "47e291fcc2d0c1a8ae0b66329f040b33af755b6adbd21739e20bb5ad56f62b6c"
+ link = "https://cdn.openbsd.org/pub/OpenBSD/6.9/amd64/install69.iso"
+ csum = "140d26548aec680e34bb5f82295414228e7f61e4f5e7951af066014fda2d6e43"
size = "20G"
pkgs = [
# tools
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (24 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 25/40] tests/vm: update openbsd to release 6.9 Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 15:04 ` Richard Henderson
2021-07-11 23:59 ` Taylor Simpson
2021-07-09 14:29 ` [PATCH v3 27/40] plugins: fix-up handling of internal hostaddr for 32 bit Alex Bennée
` (13 subsequent siblings)
39 siblings, 2 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, Taylor Simpson, aurelien
The check_aligned_anonymous_unfixed_mmaps and
check_aligned_anonymous_unfixed_colliding_mmaps do a lot of mmap's and
copying of data. This is especially unfriendly to targets like hexagon
which have quite large pages and need to do sanity checks on each
memory access.
While we are at it clean-up the white space and style issues from the
legacy code. As we no longer do quite so much needless memory access
we can also remove the hexagon timeout hack.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
v3
- style and whitespace cleanups, reduce loop even further.
- remove hexagon timeout hack
---
tests/tcg/multiarch/test-mmap.c | 208 +++++++++++++++---------------
tests/tcg/hexagon/Makefile.target | 9 --
2 files changed, 104 insertions(+), 113 deletions(-)
diff --git a/tests/tcg/multiarch/test-mmap.c b/tests/tcg/multiarch/test-mmap.c
index 11d0e777b1..96257f8ebe 100644
--- a/tests/tcg/multiarch/test-mmap.c
+++ b/tests/tcg/multiarch/test-mmap.c
@@ -49,64 +49,62 @@ size_t test_fsize;
void check_aligned_anonymous_unfixed_mmaps(void)
{
- void *p1;
- void *p2;
- void *p3;
- void *p4;
- void *p5;
- uintptr_t p;
- int i;
-
- fprintf(stdout, "%s", __func__);
- for (i = 0; i < 0x1fff; i++)
- {
- size_t len;
-
- len = pagesize + (pagesize * i & 7);
- p1 = mmap(NULL, len, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- p2 = mmap(NULL, len, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- p3 = mmap(NULL, len, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- p4 = mmap(NULL, len, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- p5 = mmap(NULL, len, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
-
- /* Make sure we get pages aligned with the pagesize. The
- target expects this. */
- fail_unless (p1 != MAP_FAILED);
- fail_unless (p2 != MAP_FAILED);
- fail_unless (p3 != MAP_FAILED);
- fail_unless (p4 != MAP_FAILED);
- fail_unless (p5 != MAP_FAILED);
- p = (uintptr_t) p1;
- D(printf ("p=%x\n", p));
- fail_unless ((p & pagemask) == 0);
- p = (uintptr_t) p2;
- fail_unless ((p & pagemask) == 0);
- p = (uintptr_t) p3;
- fail_unless ((p & pagemask) == 0);
- p = (uintptr_t) p4;
- fail_unless ((p & pagemask) == 0);
- p = (uintptr_t) p5;
- fail_unless ((p & pagemask) == 0);
-
- /* Make sure we can read from the entire area. */
- memcpy (dummybuf, p1, pagesize);
- memcpy (dummybuf, p2, pagesize);
- memcpy (dummybuf, p3, pagesize);
- memcpy (dummybuf, p4, pagesize);
- memcpy (dummybuf, p5, pagesize);
-
- munmap (p1, len);
- munmap (p2, len);
- munmap (p3, len);
- munmap (p4, len);
- munmap (p5, len);
- }
- fprintf(stdout, " passed\n");
+ void *p1;
+ void *p2;
+ void *p3;
+ void *p4;
+ void *p5;
+ uintptr_t p;
+ int i;
+ fprintf(stdout, "%s", __func__);
+ for (i = 0; i < 8; i++) {
+ size_t len;
+ len = pagesize + (pagesize * i);
+ p1 = mmap(NULL, len, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ p2 = mmap(NULL, len, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ p3 = mmap(NULL, len, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ p4 = mmap(NULL, len, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ p5 = mmap(NULL, len, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+
+ /*
+ * Make sure we get pages aligned with the pagesize. The
+ * target expects this.
+ */
+ fail_unless(p1 != MAP_FAILED);
+ fail_unless(p2 != MAP_FAILED);
+ fail_unless(p3 != MAP_FAILED);
+ fail_unless(p4 != MAP_FAILED);
+ fail_unless(p5 != MAP_FAILED);
+ p = (uintptr_t) p1;
+ D(printf("p=%x\n", p));
+ fail_unless((p & pagemask) == 0);
+ p = (uintptr_t) p2;
+ fail_unless((p & pagemask) == 0);
+ p = (uintptr_t) p3;
+ fail_unless((p & pagemask) == 0);
+ p = (uintptr_t) p4;
+ fail_unless((p & pagemask) == 0);
+ p = (uintptr_t) p5;
+ fail_unless((p & pagemask) == 0);
+
+ /* Make sure we can read from the entire area. */
+ memcpy(dummybuf, p1, pagesize);
+ memcpy(dummybuf, p2, pagesize);
+ memcpy(dummybuf, p3, pagesize);
+ memcpy(dummybuf, p4, pagesize);
+ memcpy(dummybuf, p5, pagesize);
+ munmap(p1, len);
+ munmap(p2, len);
+ munmap(p3, len);
+ munmap(p4, len);
+ munmap(p5, len);
+ }
+ fprintf(stdout, " passed\n");
}
void check_large_anonymous_unfixed_mmap(void)
@@ -135,52 +133,54 @@ void check_large_anonymous_unfixed_mmap(void)
void check_aligned_anonymous_unfixed_colliding_mmaps(void)
{
- char *p1;
- char *p2;
- char *p3;
- uintptr_t p;
- int i;
-
- fprintf(stdout, "%s", __func__);
- for (i = 0; i < 0x2fff; i++)
- {
- int nlen;
- p1 = mmap(NULL, pagesize, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- fail_unless (p1 != MAP_FAILED);
- p = (uintptr_t) p1;
- fail_unless ((p & pagemask) == 0);
- memcpy (dummybuf, p1, pagesize);
-
- p2 = mmap(NULL, pagesize, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- fail_unless (p2 != MAP_FAILED);
- p = (uintptr_t) p2;
- fail_unless ((p & pagemask) == 0);
- memcpy (dummybuf, p2, pagesize);
-
-
- munmap (p1, pagesize);
- nlen = pagesize * 8;
- p3 = mmap(NULL, nlen, PROT_READ,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- fail_unless (p3 != MAP_FAILED);
-
- /* Check if the mmaped areas collide. */
- if (p3 < p2
- && (p3 + nlen) > p2)
- fail_unless (0);
-
- memcpy (dummybuf, p3, pagesize);
-
- /* Make sure we get pages aligned with the pagesize. The
- target expects this. */
- p = (uintptr_t) p3;
- fail_unless ((p & pagemask) == 0);
- munmap (p2, pagesize);
- munmap (p3, nlen);
- }
- fprintf(stdout, " passed\n");
+ char *p1;
+ char *p2;
+ char *p3;
+ uintptr_t p;
+ int i;
+
+ fprintf(stdout, "%s", __func__);
+ for (i = 0; i < 2; i++) {
+ int nlen;
+ p1 = mmap(NULL, pagesize, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ fail_unless(p1 != MAP_FAILED);
+ p = (uintptr_t) p1;
+ fail_unless((p & pagemask) == 0);
+ memcpy(dummybuf, p1, pagesize);
+
+ p2 = mmap(NULL, pagesize, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ fail_unless(p2 != MAP_FAILED);
+ p = (uintptr_t) p2;
+ fail_unless((p & pagemask) == 0);
+ memcpy(dummybuf, p2, pagesize);
+
+
+ munmap(p1, pagesize);
+ nlen = pagesize * 8;
+ p3 = mmap(NULL, nlen, PROT_READ,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ fail_unless(p3 != MAP_FAILED);
+
+ /* Check if the mmaped areas collide. */
+ if (p3 < p2
+ && (p3 + nlen) > p2) {
+ fail_unless(0);
+ }
+
+ memcpy(dummybuf, p3, pagesize);
+
+ /*
+ * Make sure we get pages aligned with the pagesize. The
+ * target expects this.
+ */
+ p = (uintptr_t) p3;
+ fail_unless((p & pagemask) == 0);
+ munmap(p2, pagesize);
+ munmap(p3, nlen);
+ }
+ fprintf(stdout, " passed\n");
}
void check_aligned_anonymous_fixed_mmaps(void)
diff --git a/tests/tcg/hexagon/Makefile.target b/tests/tcg/hexagon/Makefile.target
index 0992787d50..050cd61c1a 100644
--- a/tests/tcg/hexagon/Makefile.target
+++ b/tests/tcg/hexagon/Makefile.target
@@ -18,15 +18,6 @@
# Hexagon doesn't support gdb, so skip the EXTRA_RUNS
EXTRA_RUNS =
-# Hexagon has 64K pages, so increase the timeout to keep
-# test-mmap from timing out
-ifeq ($(CONFIG_DEBUG_TCG),y)
-TIMEOUT=800
-else
-TIMEOUT=500
-endif
-
-
CFLAGS += -Wno-incompatible-pointer-types -Wno-undefined-internal
CFLAGS += -fno-unroll-loops
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive
2021-07-09 14:29 ` [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive Alex Bennée
@ 2021-07-09 15:04 ` Richard Henderson
2021-07-11 23:59 ` Taylor Simpson
1 sibling, 0 replies; 45+ messages in thread
From: Richard Henderson @ 2021-07-09 15:04 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, pbonzini, f4bug, robhenry, aaron,
mahmoudabdalghany, minyihh, cota, stefanha, crosa, kuhn.chenqun,
ma.mandourr, Taylor Simpson, aurelien
On 7/9/21 7:29 AM, Alex Bennée wrote:
> The check_aligned_anonymous_unfixed_mmaps and
> check_aligned_anonymous_unfixed_colliding_mmaps do a lot of mmap's and
> copying of data. This is especially unfriendly to targets like hexagon
> which have quite large pages and need to do sanity checks on each
> memory access.
>
> While we are at it clean-up the white space and style issues from the
> legacy code. As we no longer do quite so much needless memory access
> we can also remove the hexagon timeout hack.
>
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
>
> ---
> v3
> - style and whitespace cleanups, reduce loop even further.
> - remove hexagon timeout hack
> ---
> tests/tcg/multiarch/test-mmap.c | 208 +++++++++++++++---------------
> tests/tcg/hexagon/Makefile.target | 9 --
> 2 files changed, 104 insertions(+), 113 deletions(-)
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
r~
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive
2021-07-09 14:29 ` [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive Alex Bennée
2021-07-09 15:04 ` Richard Henderson
@ 2021-07-11 23:59 ` Taylor Simpson
1 sibling, 0 replies; 45+ messages in thread
From: Taylor Simpson @ 2021-07-11 23:59 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, aurelien
> -----Original Message-----
> From: Alex Bennée <alex.bennee@linaro.org>
> Sent: Friday, July 9, 2021 8:30 AM
> To: qemu-devel@nongnu.org
> Cc: fam@euphon.net; berrange@redhat.com; f4bug@amsat.org;
> aurelien@aurel32.net; pbonzini@redhat.com; stefanha@redhat.com;
> crosa@redhat.com; cota@braap.org; aaron@os.amperecomputing.com;
> kuhn.chenqun@huawei.com; robhenry@microsoft.com;
> mahmoudabdalghany@outlook.com; minyihh@uci.edu;
> ma.mandourr@gmail.com; Alex Bennée <alex.bennee@linaro.org>; Taylor
> Simpson <tsimpson@quicinc.com>
> Subject: [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive
>
> The check_aligned_anonymous_unfixed_mmaps and
> check_aligned_anonymous_unfixed_colliding_mmaps do a lot of mmap's
> and copying of data. This is especially unfriendly to targets like hexagon
> which have quite large pages and need to do sanity checks on each memory
> access.
>
> While we are at it clean-up the white space and style issues from the legacy
> code. As we no longer do quite so much needless memory access we can
> also remove the hexagon timeout hack.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> ---
> v3
> - style and whitespace cleanups, reduce loop even further.
> - remove hexagon timeout hack
> ---
> tests/tcg/multiarch/test-mmap.c | 208 +++++++++++++++---------------
> tests/tcg/hexagon/Makefile.target | 9 --
> 2 files changed, 104 insertions(+), 113 deletions(-)
Reviewed-by: Taylor Simpson <tsimpson@quicinc.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH v3 27/40] plugins: fix-up handling of internal hostaddr for 32 bit
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (25 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 26/40] tests/tcg: make test-mmap a little less aggressive Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 28/40] meson.build: move TCG plugin summary output Alex Bennée
` (12 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, Richard Henderson, f4bug,
robhenry, mahmoudabdalghany, aaron, cota, stefanha, crosa,
pbonzini, ma.mandourr, Alexandre Iooss, Alex Bennée,
aurelien
The compiler rightly complains when we build on 32 bit that casting
uint64_t into a void is a bad idea. We are really dealing with a host
pointer at this point so treat it as such. This does involve
a uintptr_t cast of the result of the TLB addend as we know that has
to point to the host memory.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
include/qemu/plugin-memory.h | 2 +-
accel/tcg/cputlb.c | 2 +-
plugins/api.c | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/qemu/plugin-memory.h b/include/qemu/plugin-memory.h
index b36def27d7..0f59226727 100644
--- a/include/qemu/plugin-memory.h
+++ b/include/qemu/plugin-memory.h
@@ -18,7 +18,7 @@ struct qemu_plugin_hwaddr {
hwaddr offset;
} io;
struct {
- uint64_t hostaddr;
+ void *hostaddr;
} ram;
} v;
};
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index b6d5fc6326..b4e15b6aad 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -1728,7 +1728,7 @@ bool tlb_plugin_lookup(CPUState *cpu, target_ulong addr, int mmu_idx,
data->v.io.offset = (iotlbentry->addr & TARGET_PAGE_MASK) + addr;
} else {
data->is_io = false;
- data->v.ram.hostaddr = addr + tlbe->addend;
+ data->v.ram.hostaddr = (void *)((uintptr_t)addr + tlbe->addend);
}
return true;
} else {
diff --git a/plugins/api.c b/plugins/api.c
index 332e2c60e2..78b563c5c5 100644
--- a/plugins/api.c
+++ b/plugins/api.c
@@ -308,11 +308,11 @@ uint64_t qemu_plugin_hwaddr_phys_addr(const struct qemu_plugin_hwaddr *haddr)
if (!haddr->is_io) {
RAMBlock *block;
ram_addr_t offset;
- void *hostaddr = (void *) haddr->v.ram.hostaddr;
+ void *hostaddr = haddr->v.ram.hostaddr;
block = qemu_ram_block_from_host(hostaddr, false, &offset);
if (!block) {
- error_report("Bad ram pointer %"PRIx64"", haddr->v.ram.hostaddr);
+ error_report("Bad host ram pointer %p", haddr->v.ram.hostaddr);
abort();
}
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 28/40] meson.build: move TCG plugin summary output
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (26 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 27/40] plugins: fix-up handling of internal hostaddr for 32 bit Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 29/40] configure: don't allow plugins to be enabled for a non-TCG build Alex Bennée
` (11 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, Richard Henderson, f4bug,
robhenry, mahmoudabdalghany, aaron, cota, stefanha, crosa,
pbonzini, ma.mandourr, Alex Bennée, aurelien
Let's put it with the rest of the TCG related output with the
accelerator.
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meson.build b/meson.build
index ff7a9f8b4f..ba680addda 100644
--- a/meson.build
+++ b/meson.build
@@ -2737,7 +2737,6 @@ summary_info += {'module support': config_host.has_key('CONFIG_MODULES')}
if config_host.has_key('CONFIG_MODULES')
summary_info += {'alternative module path': config_host.has_key('CONFIG_MODULE_UPGRADES')}
endif
-summary_info += {'plugin support': config_host.has_key('CONFIG_PLUGIN')}
summary_info += {'fuzzing support': config_host.has_key('CONFIG_FUZZ')}
if have_system
summary_info += {'Audio drivers': config_host['CONFIG_AUDIO_DRIVERS']}
@@ -2853,6 +2852,7 @@ if config_all.has_key('CONFIG_TCG')
else
summary_info += {'TCG backend': 'native (@0@)'.format(cpu)}
endif
+ summary_info += {'TCG plugins': config_host.has_key('CONFIG_PLUGIN')}
summary_info += {'TCG debug enabled': config_host.has_key('CONFIG_DEBUG_TCG')}
endif
summary_info += {'target list': ' '.join(target_dirs)}
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 29/40] configure: don't allow plugins to be enabled for a non-TCG build
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (27 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 28/40] meson.build: move TCG plugin summary output Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 30/40] configure: add an explicit static and plugins check Alex Bennée
` (10 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, Richard Henderson, f4bug,
robhenry, mahmoudabdalghany, aaron, cota, stefanha, crosa,
pbonzini, ma.mandourr, Alex Bennée, aurelien
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
v2
- error out after all options have been processed
---
configure | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/configure b/configure
index 650d9c0735..7f906be68d 100755
--- a/configure
+++ b/configure
@@ -1098,6 +1098,7 @@ for opt do
--enable-cap-ng) cap_ng="enabled"
;;
--disable-tcg) tcg="disabled"
+ plugins="no"
;;
--enable-tcg) tcg="enabled"
;;
@@ -1563,6 +1564,11 @@ for opt do
esac
done
+# test for any invalid configuration combinations
+if test "$plugins" = "yes" -a "$tcg" = "disabled"; then
+ error_exit "Can't enable plugins on non-TCG builds"
+fi
+
case $git_submodules_action in
update|validate)
if test ! -e "$source_path/.git"; then
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 30/40] configure: add an explicit static and plugins check
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (28 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 29/40] configure: don't allow plugins to be enabled for a non-TCG build Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 15:06 ` Richard Henderson
2021-07-09 14:29 ` [PATCH v3 31/40] configure: stop user enabling plugins on Windows for now Alex Bennée
` (9 subsequent siblings)
39 siblings, 1 reply; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
Moving this check earlier will make the later re-factor for enabling
by default a bit neater.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
configure | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/configure b/configure
index 7f906be68d..25daabd0ff 100755
--- a/configure
+++ b/configure
@@ -923,6 +923,7 @@ for opt do
;;
--static)
static="yes"
+ plugins="no"
QEMU_PKG_CONFIG_FLAGS="--static $QEMU_PKG_CONFIG_FLAGS"
;;
--mandir=*) mandir="$optarg"
@@ -2184,11 +2185,14 @@ if test "$modules" = "no" && test "$module_upgrades" = "yes" ; then
error_exit "Can't enable module-upgrades as Modules are not enabled"
fi
-# Static linking is not possible with modules or PIE
+# Static linking is not possible with plugins, modules or PIE
if test "$static" = "yes" ; then
if test "$modules" = "yes" ; then
error_exit "static and modules are mutually incompatible"
fi
+ if test "$plugins" = "yes"; then
+ error_exit "static and plugins are mutually incompatible"
+ fi
fi
# Unconditional check for compiler __thread support
@@ -3979,7 +3983,7 @@ if test "$plugins" = "yes" &&
"Plugin support requires dynamic linking and specifying a set of symbols " \
"that are exported to plugins. Unfortunately your linker doesn't " \
"support the flag (--dynamic-list or -exported_symbols_list) used " \
- "for this purpose. You can't build with --static."
+ "for this purpose."
fi
########################################
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH v3 30/40] configure: add an explicit static and plugins check
2021-07-09 14:29 ` [PATCH v3 30/40] configure: add an explicit static and plugins check Alex Bennée
@ 2021-07-09 15:06 ` Richard Henderson
0 siblings, 0 replies; 45+ messages in thread
From: Richard Henderson @ 2021-07-09 15:06 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, pbonzini, f4bug, robhenry, aaron,
mahmoudabdalghany, minyihh, cota, stefanha, crosa, kuhn.chenqun,
ma.mandourr, aurelien
On 7/9/21 7:29 AM, Alex Bennée wrote:
> Moving this check earlier will make the later re-factor for enabling
> by default a bit neater.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> configure | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/configure b/configure
> index 7f906be68d..25daabd0ff 100755
> --- a/configure
> +++ b/configure
> @@ -923,6 +923,7 @@ for opt do
> ;;
> --static)
> static="yes"
> + plugins="no"
This is wrong. This means that
--enable-plugins --static
will not error out.
With that line removed,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
r~
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH v3 31/40] configure: stop user enabling plugins on Windows for now
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (29 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 30/40] configure: add an explicit static and plugins check Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 32/40] tcg/plugins: enable by default for most TCG builds Alex Bennée
` (8 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, Richard Henderson, f4bug,
robhenry, mahmoudabdalghany, aaron, cota, stefanha, crosa,
pbonzini, ma.mandourr, Alex Bennée, aurelien, Yonggang Luo
There are some patches on the list that enable plugins on Windows but
they still need some changes to be ready:
https://patchew.org/QEMU/20201013002806.1447-1-luoyonggang@gmail.com/
In the meantime lets stop the user from being able to configure the
support so they don't get confused by the weird linker error messages
later.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Yonggang Luo <luoyonggang@gmail.com>
---
v2
- move targetos check to if we try to --enable-plugins, default no otherwise
v3
- fix missing test, s/targetos/mingw32/
---
configure | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/configure b/configure
index 25daabd0ff..1eb8ef5341 100755
--- a/configure
+++ b/configure
@@ -708,6 +708,7 @@ MINGW32*)
audio_drv_list=""
fi
supported_os="yes"
+ plugins="no"
pie="no"
;;
GNU/kFreeBSD)
@@ -1511,7 +1512,11 @@ for opt do
;;
--disable-xkbcommon) xkbcommon="disabled"
;;
- --enable-plugins) plugins="yes"
+ --enable-plugins) if test "$mingw32" = "yes"; then
+ error_exit "TCG plugins not currently supported on Windows platforms"
+ else
+ plugins="yes"
+ fi
;;
--disable-plugins) plugins="no"
;;
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 32/40] tcg/plugins: enable by default for most TCG builds
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (30 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 31/40] configure: stop user enabling plugins on Windows for now Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 15:12 ` Richard Henderson
2021-07-09 14:29 ` [PATCH v3 33/40] contrib/plugins: enable -Wall for building plugins Alex Bennée
` (7 subsequent siblings)
39 siblings, 1 reply; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun, f4bug,
robhenry, Willian Rampazzo, mahmoudabdalghany, aaron, cota,
Wainer dos Santos Moschetta, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
Aside from a minor bloat to file size the ability to have TCG plugins
has no real impact on performance unless a plugin is actively loaded.
Even then the libempty.so plugin shows only a minor degradation in
performance caused by the extra book keeping the TCG has to do to keep
track of instructions. As it's a useful feature lets just enable it by
default and reduce our testing matrix a little.
We need to move our linker testing earlier so we can be sure we can
enable the loader module required. As we have ruled out static &
plugins in an earlier patch we can also reduce the indent a little.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
v2
- properly handle plugins being ""
- make the test for linker support handle default case
- move linker tests to before the glib-modules test
v3
- remove stray debug ($plugins)
- remove no longer needed if static = no check
- update commit message
---
docs/devel/tcg-plugins.rst | 3 +-
configure | 122 ++++++++++++++++++++-----------------
.gitlab-ci.d/buildtest.yml | 23 -------
3 files changed, 67 insertions(+), 81 deletions(-)
diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst
index 18c6581d85..0cd77c77d2 100644
--- a/docs/devel/tcg-plugins.rst
+++ b/docs/devel/tcg-plugins.rst
@@ -71,7 +71,8 @@ API
Usage
=====
-The QEMU binary needs to be compiled for plugin support::
+Any QEMU binary with TCG support has plugins enabled by default.
+Earlier releases needed to be explicitly enabled with::
configure --enable-plugins
diff --git a/configure b/configure
index 1eb8ef5341..b960dd5f77 100755
--- a/configure
+++ b/configure
@@ -429,7 +429,7 @@ libxml2="auto"
debug_mutex="no"
libpmem="auto"
default_devices="true"
-plugins="no"
+plugins="$default_feature"
fuzzing="no"
rng_none="no"
secret_keyring="$default_feature"
@@ -3090,6 +3090,69 @@ for drv in $audio_drv_list; do
esac
done
+##########################################
+# plugin linker support probe
+
+if test "$plugins" != "no"; then
+
+ #########################################
+ # See if --dynamic-list is supported by the linker
+
+ ld_dynamic_list="no"
+ cat > $TMPTXT <<EOF
+{
+ foo;
+};
+EOF
+
+ cat > $TMPC <<EOF
+#include <stdio.h>
+void foo(void);
+
+void foo(void)
+{
+ printf("foo\n");
+}
+
+int main(void)
+{
+ foo();
+ return 0;
+}
+EOF
+
+ if compile_prog "" "-Wl,--dynamic-list=$TMPTXT" ; then
+ ld_dynamic_list="yes"
+ fi
+
+ #########################################
+ # See if -exported_symbols_list is supported by the linker
+
+ ld_exported_symbols_list="no"
+ cat > $TMPTXT <<EOF
+ _foo
+EOF
+
+ if compile_prog "" "-Wl,-exported_symbols_list,$TMPTXT" ; then
+ ld_exported_symbols_list="yes"
+ fi
+
+ if test "$ld_dynamic_list" = "no" &&
+ test "$ld_exported_symbols_list" = "no" ; then
+ if test "$plugins" = "yes"; then
+ error_exit \
+ "Plugin support requires dynamic linking and specifying a set of symbols " \
+ "that are exported to plugins. Unfortunately your linker doesn't " \
+ "support the flag (--dynamic-list or -exported_symbols_list) used " \
+ "for this purpose."
+ else
+ plugins="no"
+ fi
+ else
+ plugins="yes"
+ fi
+fi
+
##########################################
# glib support probe
@@ -3098,7 +3161,7 @@ glib_modules=gthread-2.0
if test "$modules" = yes; then
glib_modules="$glib_modules gmodule-export-2.0"
fi
-if test "$plugins" = yes; then
+if test "$plugins" = "yes"; then
glib_modules="$glib_modules gmodule-2.0"
fi
@@ -3936,61 +3999,6 @@ if compile_prog "" "" ; then
atomic64=yes
fi
-#########################################
-# See if --dynamic-list is supported by the linker
-ld_dynamic_list="no"
-if test "$static" = "no" ; then
- cat > $TMPTXT <<EOF
-{
- foo;
-};
-EOF
-
- cat > $TMPC <<EOF
-#include <stdio.h>
-void foo(void);
-
-void foo(void)
-{
- printf("foo\n");
-}
-
-int main(void)
-{
- foo();
- return 0;
-}
-EOF
-
- if compile_prog "" "-Wl,--dynamic-list=$TMPTXT" ; then
- ld_dynamic_list="yes"
- fi
-fi
-
-#########################################
-# See if -exported_symbols_list is supported by the linker
-
-ld_exported_symbols_list="no"
-if test "$static" = "no" ; then
- cat > $TMPTXT <<EOF
- _foo
-EOF
-
- if compile_prog "" "-Wl,-exported_symbols_list,$TMPTXT" ; then
- ld_exported_symbols_list="yes"
- fi
-fi
-
-if test "$plugins" = "yes" &&
- test "$ld_dynamic_list" = "no" &&
- test "$ld_exported_symbols_list" = "no" ; then
- error_exit \
- "Plugin support requires dynamic linking and specifying a set of symbols " \
- "that are exported to plugins. Unfortunately your linker doesn't " \
- "support the flag (--dynamic-list or -exported_symbols_list) used " \
- "for this purpose."
-fi
-
########################################
# check if getauxval is available.
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index d9b834c848..89df51517c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -354,29 +354,6 @@ build-some-softmmu:
TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
MAKE_CHECK_ARGS: check-tcg
-# Run check-tcg against linux-user (with plugins)
-# we skip sparc64-linux-user until it has been fixed somewhat
-# we skip cris-linux-user as it doesn't use the common run loop
-build-user-plugins:
- extends: .native_build_job_template
- needs:
- job: amd64-debian-user-cross-container
- variables:
- IMAGE: debian-all-test-cross
- CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
- MAKE_CHECK_ARGS: check-tcg
- timeout: 1h 30m
-
-build-some-softmmu-plugins:
- extends: .native_build_job_template
- needs:
- job: amd64-debian-user-cross-container
- variables:
- IMAGE: debian-all-test-cross
- CONFIGURE_ARGS: --disable-tools --disable-user --enable-plugins --enable-debug-tcg
- TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
- MAKE_CHECK_ARGS: check-tcg
-
clang-system:
extends: .native_build_job_template
needs:
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH v3 32/40] tcg/plugins: enable by default for most TCG builds
2021-07-09 14:29 ` [PATCH v3 32/40] tcg/plugins: enable by default for most TCG builds Alex Bennée
@ 2021-07-09 15:12 ` Richard Henderson
0 siblings, 0 replies; 45+ messages in thread
From: Richard Henderson @ 2021-07-09 15:12 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, Thomas Huth, berrange, pbonzini, f4bug, robhenry, aaron,
Willian Rampazzo, mahmoudabdalghany, minyihh, cota,
Wainer dos Santos Moschetta, stefanha, crosa, kuhn.chenqun,
ma.mandourr, Alexandre Iooss, aurelien
On 7/9/21 7:29 AM, Alex Bennée wrote:
> Aside from a minor bloat to file size the ability to have TCG plugins
> has no real impact on performance unless a plugin is actively loaded.
> Even then the libempty.so plugin shows only a minor degradation in
> performance caused by the extra book keeping the TCG has to do to keep
> track of instructions. As it's a useful feature lets just enable it by
> default and reduce our testing matrix a little.
>
> We need to move our linker testing earlier so we can be sure we can
> enable the loader module required. As we have ruled out static &
> plugins in an earlier patch we can also reduce the indent a little.
>
> Cc: Paolo Bonzini<pbonzini@redhat.com>
> Reviewed-by: Thomas Huth<thuth@redhat.com>
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
>
> ---
> v2
> - properly handle plugins being ""
> - make the test for linker support handle default case
> - move linker tests to before the glib-modules test
> v3
> - remove stray debug ($plugins)
> - remove no longer needed if static = no check
> - update commit message
> ---
> docs/devel/tcg-plugins.rst | 3 +-
> configure | 122 ++++++++++++++++++++-----------------
> .gitlab-ci.d/buildtest.yml | 23 -------
> 3 files changed, 67 insertions(+), 81 deletions(-)
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
r~
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH v3 33/40] contrib/plugins: enable -Wall for building plugins
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (31 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 32/40] tcg/plugins: enable by default for most TCG builds Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:29 ` [PATCH v3 34/40] contrib/plugins: add execlog to log instruction execution and memory access Alex Bennée
` (6 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, minyihh, berrange, kuhn.chenqun,
Richard Henderson, f4bug, robhenry, mahmoudabdalghany, aaron,
cota, stefanha, crosa, pbonzini, ma.mandourr, Alexandre Iooss,
Alex Bennée, aurelien
Lets spot the obvious errors.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
v2
- also $(filter -W%, $(QEMU_CFLAGS)) from the main build
---
contrib/plugins/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile
index b9d7935e5e..3431bc1ce9 100644
--- a/contrib/plugins/Makefile
+++ b/contrib/plugins/Makefile
@@ -24,7 +24,7 @@ SONAMES := $(addsuffix .so,$(addprefix lib,$(NAMES)))
# The main QEMU uses Glib extensively so it's perfectly fine to use it
# in plugins (which many example do).
CFLAGS = $(GLIB_CFLAGS)
-CFLAGS += -fPIC
+CFLAGS += -fPIC -Wall $(filter -W%, $(QEMU_CFLAGS))
CFLAGS += $(if $(findstring no-psabi,$(QEMU_CFLAGS)),-Wpsabi)
CFLAGS += -I$(SRC_PATH)/include/qemu
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 34/40] contrib/plugins: add execlog to log instruction execution and memory access
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (32 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 33/40] contrib/plugins: enable -Wall for building plugins Alex Bennée
@ 2021-07-09 14:29 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 35/40] docs/devel: tcg-plugins: add execlog plugin description Alex Bennée
` (5 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:29 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Alexandre Iooss <erdnaxe@crans.org>
Log instruction execution and memory access to a file.
This plugin can be used for reverse engineering or for side-channel analysis
using QEMU.
Signed-off-by: Alexandre Iooss <erdnaxe@crans.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210702081307.1653644-2-erdnaxe@crans.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
contrib/plugins/execlog.c | 153 ++++++++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
contrib/plugins/Makefile | 1 +
3 files changed, 155 insertions(+)
create mode 100644 contrib/plugins/execlog.c
diff --git a/contrib/plugins/execlog.c b/contrib/plugins/execlog.c
new file mode 100644
index 0000000000..2de9f0d7d4
--- /dev/null
+++ b/contrib/plugins/execlog.c
@@ -0,0 +1,153 @@
+/*
+ * Copyright (C) 2021, Alexandre Iooss <erdnaxe@crans.org>
+ *
+ * Log instruction execution with memory access.
+ *
+ * License: GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include <glib.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <qemu-plugin.h>
+
+QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
+
+/* Store last executed instruction on each vCPU as a GString */
+GArray *last_exec;
+
+/**
+ * Add memory read or write information to current instruction log
+ */
+static void vcpu_mem(unsigned int cpu_index, qemu_plugin_meminfo_t info,
+ uint64_t vaddr, void *udata)
+{
+ GString *s;
+
+ /* Find vCPU in array */
+ g_assert(cpu_index < last_exec->len);
+ s = g_array_index(last_exec, GString *, cpu_index);
+
+ /* Indicate type of memory access */
+ if (qemu_plugin_mem_is_store(info)) {
+ g_string_append(s, ", store");
+ } else {
+ g_string_append(s, ", load");
+ }
+
+ /* If full system emulation log physical address and device name */
+ struct qemu_plugin_hwaddr *hwaddr = qemu_plugin_get_hwaddr(info, vaddr);
+ if (hwaddr) {
+ uint64_t addr = qemu_plugin_hwaddr_phys_addr(hwaddr);
+ const char *name = qemu_plugin_hwaddr_device_name(hwaddr);
+ g_string_append_printf(s, ", 0x%08"PRIx64", %s", addr, name);
+ } else {
+ g_string_append_printf(s, ", 0x%08"PRIx64, vaddr);
+ }
+}
+
+/**
+ * Log instruction execution
+ */
+static void vcpu_insn_exec(unsigned int cpu_index, void *udata)
+{
+ GString *s;
+
+ /* Find or create vCPU in array */
+ while (cpu_index >= last_exec->len) {
+ s = g_string_new(NULL);
+ g_array_append_val(last_exec, s);
+ }
+ s = g_array_index(last_exec, GString *, cpu_index);
+
+ /* Print previous instruction in cache */
+ if (s->len) {
+ qemu_plugin_outs(s->str);
+ qemu_plugin_outs("s\n");
+ }
+
+ /* Store new instruction in cache */
+ /* vcpu_mem will add memory access information to last_exec */
+ g_string_printf(s, "%u, ", cpu_index);
+ g_string_append(s, (char *)udata);
+}
+
+/**
+ * On translation block new translation
+ *
+ * QEMU convert code by translation block (TB). By hooking here we can then hook
+ * a callback on each instruction and memory access.
+ */
+static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
+{
+ struct qemu_plugin_insn *insn;
+ uint64_t insn_vaddr;
+ uint32_t insn_opcode;
+ char *insn_disas;
+
+ size_t n = qemu_plugin_tb_n_insns(tb);
+ for (size_t i = 0; i < n; i++) {
+ /*
+ * `insn` is shared between translations in QEMU, copy needed data here.
+ * `output` is never freed as it might be used multiple times during
+ * the emulation lifetime.
+ * We only consider the first 32 bits of the instruction, this may be
+ * a limitation for CISC architectures.
+ */
+ insn = qemu_plugin_tb_get_insn(tb, i);
+ insn_vaddr = qemu_plugin_insn_vaddr(insn);
+ insn_opcode = *((uint32_t *)qemu_plugin_insn_data(insn));
+ insn_disas = qemu_plugin_insn_disas(insn);
+ char *output = g_strdup_printf("0x%"PRIx64", 0x%"PRIx32", \"%s\"",
+ insn_vaddr, insn_opcode, insn_disas);
+
+ /* Register callback on memory read or write */
+ qemu_plugin_register_vcpu_mem_cb(insn, vcpu_mem,
+ QEMU_PLUGIN_CB_NO_REGS,
+ QEMU_PLUGIN_MEM_RW, NULL);
+
+ /* Register callback on instruction */
+ qemu_plugin_register_vcpu_insn_exec_cb(insn, vcpu_insn_exec,
+ QEMU_PLUGIN_CB_NO_REGS, output);
+ }
+}
+
+/**
+ * On plugin exit, print last instruction in cache
+ */
+static void plugin_exit(qemu_plugin_id_t id, void *p)
+{
+ guint i;
+ GString *s;
+ for (i = 0; i < last_exec->len; i++) {
+ s = g_array_index(last_exec, GString *, i);
+ if (s->str) {
+ qemu_plugin_outs(s->str);
+ qemu_plugin_outs("\n");
+ }
+ }
+}
+
+/**
+ * Install the plugin
+ */
+QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id,
+ const qemu_info_t *info, int argc,
+ char **argv)
+{
+ /*
+ * Initialize dynamic array to cache vCPU instruction. In user mode
+ * we don't know the size before emulation.
+ */
+ last_exec = g_array_new(FALSE, FALSE, sizeof(GString *));
+
+ /* Register translation block and exit callbacks */
+ qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans);
+ qemu_plugin_register_atexit_cb(id, plugin_exit, NULL);
+
+ return 0;
+}
diff --git a/MAINTAINERS b/MAINTAINERS
index 684142e12e..ee456e5fb4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2981,6 +2981,7 @@ F: include/tcg/
TCG Plugins
M: Alex Bennée <alex.bennee@linaro.org>
+R: Alexandre Iooss <erdnaxe@crans.org>
S: Maintained
F: docs/devel/tcg-plugins.rst
F: plugins/
diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile
index 3431bc1ce9..3c9209b6b0 100644
--- a/contrib/plugins/Makefile
+++ b/contrib/plugins/Makefile
@@ -13,6 +13,7 @@ include $(BUILD_DIR)/config-host.mak
VPATH += $(SRC_PATH)/contrib/plugins
NAMES :=
+NAMES += execlog
NAMES += hotblocks
NAMES += hotpages
NAMES += howvec
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 35/40] docs/devel: tcg-plugins: add execlog plugin description
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (33 preceding siblings ...)
2021-07-09 14:29 ` [PATCH v3 34/40] contrib/plugins: add execlog to log instruction execution and memory access Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 36/40] plugins: Added a new cache modelling plugin Alex Bennée
` (4 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Alexandre Iooss <erdnaxe@crans.org>
This adds description of the execlog TCG plugin with an example.
Signed-off-by: Alexandre Iooss <erdnaxe@crans.org>
Message-Id: <20210702081307.1653644-3-erdnaxe@crans.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/devel/tcg-plugins.rst | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst
index 0cd77c77d2..179867e9c1 100644
--- a/docs/devel/tcg-plugins.rst
+++ b/docs/devel/tcg-plugins.rst
@@ -320,3 +320,27 @@ the user to see what hardware is accessed how often. It has a number of options:
off:0000001c, 1, 2
off:00000020, 1, 2
...
+
+- contrib/plugins/execlog.c
+
+The execlog tool traces executed instructions with memory access. It can be used
+for debugging and security analysis purposes.
+Please be aware that this will generate a lot of output.
+
+The plugin takes no argument::
+
+ qemu-system-arm $(QEMU_ARGS) \
+ -plugin ./contrib/plugins/libexeclog.so -d plugin
+
+which will output an execution trace following this structure::
+
+ # vCPU, vAddr, opcode, disassembly[, load/store, memory addr, device]...
+ 0, 0xa12, 0xf8012400, "movs r4, #0"
+ 0, 0xa14, 0xf87f42b4, "cmp r4, r6"
+ 0, 0xa16, 0xd206, "bhs #0xa26"
+ 0, 0xa18, 0xfff94803, "ldr r0, [pc, #0xc]", load, 0x00010a28, RAM
+ 0, 0xa1a, 0xf989f000, "bl #0xd30"
+ 0, 0xd30, 0xfff9b510, "push {r4, lr}", store, 0x20003ee0, RAM, store, 0x20003ee4, RAM
+ 0, 0xd32, 0xf9893014, "adds r0, #0x14"
+ 0, 0xd34, 0xf9c8f000, "bl #0x10c8"
+ 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 36/40] plugins: Added a new cache modelling plugin
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (34 preceding siblings ...)
2021-07-09 14:30 ` [PATCH v3 35/40] docs/devel: tcg-plugins: add execlog plugin description Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 37/40] plugins/cache: Enable cache parameterization Alex Bennée
` (3 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Mahmoud Mandour <ma.mandourr@gmail.com>
Added a cache modelling plugin that uses a static configuration used in
many of the commercial microprocessors and uses random eviction policy.
The purpose of the plugin is to identify the most cache-thrashing
instructions for both instruction cache and data cache.
Signed-off-by: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623125458.450462-2-ma.mandourr@gmail.com>
---
contrib/plugins/cache.c | 419 +++++++++++++++++++++++++++++++++++++++
contrib/plugins/Makefile | 1 +
2 files changed, 420 insertions(+)
create mode 100644 contrib/plugins/cache.c
diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c
new file mode 100644
index 0000000000..e9955cdc3a
--- /dev/null
+++ b/contrib/plugins/cache.c
@@ -0,0 +1,419 @@
+/*
+ * Copyright (C) 2021, Mahmoud Mandour <ma.mandourr@gmail.com>
+ *
+ * License: GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include <inttypes.h>
+#include <stdio.h>
+#include <glib.h>
+
+#include <qemu-plugin.h>
+
+QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
+
+static enum qemu_plugin_mem_rw rw = QEMU_PLUGIN_MEM_RW;
+
+static GHashTable *miss_ht;
+
+static GMutex mtx;
+static GRand *rng;
+
+static int limit;
+static bool sys;
+
+static uint64_t dmem_accesses;
+static uint64_t dmisses;
+
+static uint64_t imem_accesses;
+static uint64_t imisses;
+
+/*
+ * A CacheSet is a set of cache blocks. A memory block that maps to a set can be
+ * put in any of the blocks inside the set. The number of block per set is
+ * called the associativity (assoc).
+ *
+ * Each block contains the the stored tag and a valid bit. Since this is not
+ * a functional simulator, the data itself is not stored. We only identify
+ * whether a block is in the cache or not by searching for its tag.
+ *
+ * In order to search for memory data in the cache, the set identifier and tag
+ * are extracted from the address and the set is probed to see whether a tag
+ * match occur.
+ *
+ * An address is logically divided into three portions: The block offset,
+ * the set number, and the tag.
+ *
+ * The set number is used to identify the set in which the block may exist.
+ * The tag is compared against all the tags of a set to search for a match. If a
+ * match is found, then the access is a hit.
+ */
+
+typedef struct {
+ uint64_t tag;
+ bool valid;
+} CacheBlock;
+
+typedef struct {
+ CacheBlock *blocks;
+} CacheSet;
+
+typedef struct {
+ CacheSet *sets;
+ int num_sets;
+ int cachesize;
+ int assoc;
+ int blksize_shift;
+ uint64_t set_mask;
+ uint64_t tag_mask;
+} Cache;
+
+typedef struct {
+ char *disas_str;
+ const char *symbol;
+ uint64_t addr;
+ uint64_t dmisses;
+ uint64_t imisses;
+} InsnData;
+
+Cache *dcache, *icache;
+
+static int pow_of_two(int num)
+{
+ g_assert((num & (num - 1)) == 0);
+ int ret = 0;
+ while (num /= 2) {
+ ret++;
+ }
+ return ret;
+}
+
+static inline uint64_t extract_tag(Cache *cache, uint64_t addr)
+{
+ return addr & cache->tag_mask;
+}
+
+static inline uint64_t extract_set(Cache *cache, uint64_t addr)
+{
+ return (addr & cache->set_mask) >> cache->blksize_shift;
+}
+
+static Cache *cache_init(int blksize, int assoc, int cachesize)
+{
+ Cache *cache;
+ int i;
+ uint64_t blk_mask;
+
+ cache = g_new(Cache, 1);
+ cache->assoc = assoc;
+ cache->cachesize = cachesize;
+ cache->num_sets = cachesize / (blksize * assoc);
+ cache->sets = g_new(CacheSet, cache->num_sets);
+ cache->blksize_shift = pow_of_two(blksize);
+
+ for (i = 0; i < cache->num_sets; i++) {
+ cache->sets[i].blocks = g_new0(CacheBlock, assoc);
+ }
+
+ blk_mask = blksize - 1;
+ cache->set_mask = ((cache->num_sets - 1) << cache->blksize_shift);
+ cache->tag_mask = ~(cache->set_mask | blk_mask);
+ return cache;
+}
+
+static int get_invalid_block(Cache *cache, uint64_t set)
+{
+ int i;
+
+ for (i = 0; i < cache->assoc; i++) {
+ if (!cache->sets[set].blocks[i].valid) {
+ return i;
+ }
+ }
+
+ return -1;
+}
+
+static int get_replaced_block(Cache *cache)
+{
+ return g_rand_int_range(rng, 0, cache->assoc);
+}
+
+static bool in_cache(Cache *cache, uint64_t addr)
+{
+ int i;
+ uint64_t tag, set;
+
+ tag = extract_tag(cache, addr);
+ set = extract_set(cache, addr);
+
+ for (i = 0; i < cache->assoc; i++) {
+ if (cache->sets[set].blocks[i].tag == tag &&
+ cache->sets[set].blocks[i].valid) {
+ return true;
+ }
+ }
+
+ return false;
+}
+
+/**
+ * access_cache(): Simulate a cache access
+ * @cache: The cache under simulation
+ * @addr: The address of the requested memory location
+ *
+ * Returns true if the requsted data is hit in the cache and false when missed.
+ * The cache is updated on miss for the next access.
+ */
+static bool access_cache(Cache *cache, uint64_t addr)
+{
+ uint64_t tag, set;
+ int replaced_blk;
+
+ if (in_cache(cache, addr)) {
+ return true;
+ }
+
+ tag = extract_tag(cache, addr);
+ set = extract_set(cache, addr);
+
+ replaced_blk = get_invalid_block(cache, set);
+
+ if (replaced_blk == -1) {
+ replaced_blk = get_replaced_block(cache);
+ }
+
+ cache->sets[set].blocks[replaced_blk].tag = tag;
+ cache->sets[set].blocks[replaced_blk].valid = true;
+
+ return false;
+}
+
+static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
+ uint64_t vaddr, void *userdata)
+{
+ uint64_t effective_addr;
+ struct qemu_plugin_hwaddr *hwaddr;
+ InsnData *insn;
+
+ g_mutex_lock(&mtx);
+ hwaddr = qemu_plugin_get_hwaddr(info, vaddr);
+ if (hwaddr && qemu_plugin_hwaddr_is_io(hwaddr)) {
+ g_mutex_unlock(&mtx);
+ return;
+ }
+
+ effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
+
+ if (!access_cache(dcache, effective_addr)) {
+ insn = (InsnData *) userdata;
+ insn->dmisses++;
+ dmisses++;
+ }
+ dmem_accesses++;
+ g_mutex_unlock(&mtx);
+}
+
+static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
+{
+ uint64_t insn_addr;
+ InsnData *insn;
+
+ g_mutex_lock(&mtx);
+ insn_addr = ((InsnData *) userdata)->addr;
+
+ if (!access_cache(icache, insn_addr)) {
+ insn = (InsnData *) userdata;
+ insn->imisses++;
+ imisses++;
+ }
+ imem_accesses++;
+ g_mutex_unlock(&mtx);
+}
+
+static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
+{
+ size_t n_insns;
+ size_t i;
+ InsnData *data;
+
+ n_insns = qemu_plugin_tb_n_insns(tb);
+ for (i = 0; i < n_insns; i++) {
+ struct qemu_plugin_insn *insn = qemu_plugin_tb_get_insn(tb, i);
+ uint64_t effective_addr;
+
+ if (sys) {
+ effective_addr = (uint64_t) qemu_plugin_insn_haddr(insn);
+ } else {
+ effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
+ }
+
+ /*
+ * Instructions might get translated multiple times, we do not create
+ * new entries for those instructions. Instead, we fetch the same
+ * entry from the hash table and register it for the callback again.
+ */
+ g_mutex_lock(&mtx);
+ data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr));
+ if (data == NULL) {
+ data = g_new0(InsnData, 1);
+ data->disas_str = qemu_plugin_insn_disas(insn);
+ data->symbol = qemu_plugin_insn_symbol(insn);
+ data->addr = effective_addr;
+ g_hash_table_insert(miss_ht, GUINT_TO_POINTER(effective_addr),
+ (gpointer) data);
+ }
+ g_mutex_unlock(&mtx);
+
+ qemu_plugin_register_vcpu_mem_cb(insn, vcpu_mem_access,
+ QEMU_PLUGIN_CB_NO_REGS,
+ rw, data);
+
+ qemu_plugin_register_vcpu_insn_exec_cb(insn, vcpu_insn_exec,
+ QEMU_PLUGIN_CB_NO_REGS, data);
+ }
+}
+
+static void insn_free(gpointer data)
+{
+ InsnData *insn = (InsnData *) data;
+ g_free(insn->disas_str);
+ g_free(insn);
+}
+
+static void cache_free(Cache *cache)
+{
+ for (int i = 0; i < cache->num_sets; i++) {
+ g_free(cache->sets[i].blocks);
+ }
+
+ g_free(cache->sets);
+ g_free(cache);
+}
+
+static int dcmp(gconstpointer a, gconstpointer b)
+{
+ InsnData *insn_a = (InsnData *) a;
+ InsnData *insn_b = (InsnData *) b;
+
+ return insn_a->dmisses < insn_b->dmisses ? 1 : -1;
+}
+
+static int icmp(gconstpointer a, gconstpointer b)
+{
+ InsnData *insn_a = (InsnData *) a;
+ InsnData *insn_b = (InsnData *) b;
+
+ return insn_a->imisses < insn_b->imisses ? 1 : -1;
+}
+
+static void log_stats()
+{
+ g_autoptr(GString) rep = g_string_new("");
+ g_string_append_printf(rep,
+ "Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
+ dmem_accesses,
+ dmisses,
+ ((double) dmisses / (double) dmem_accesses) * 100.0);
+
+ g_string_append_printf(rep,
+ "Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
+ imem_accesses,
+ imisses,
+ ((double) imisses / (double) imem_accesses) * 100.0);
+
+ qemu_plugin_outs(rep->str);
+}
+
+static void log_top_insns()
+{
+ int i;
+ GList *curr, *miss_insns;
+ InsnData *insn;
+
+ miss_insns = g_hash_table_get_values(miss_ht);
+ miss_insns = g_list_sort(miss_insns, dcmp);
+ g_autoptr(GString) rep = g_string_new("");
+ g_string_append_printf(rep, "%s", "address, data misses, instruction\n");
+
+ for (curr = miss_insns, i = 0; curr && i < limit; i++, curr = curr->next) {
+ insn = (InsnData *) curr->data;
+ g_string_append_printf(rep, "0x%" PRIx64, insn->addr);
+ if (insn->symbol) {
+ g_string_append_printf(rep, " (%s)", insn->symbol);
+ }
+ g_string_append_printf(rep, ", %ld, %s\n", insn->dmisses,
+ insn->disas_str);
+ }
+
+ miss_insns = g_list_sort(miss_insns, icmp);
+ g_string_append_printf(rep, "%s", "\naddress, fetch misses, instruction\n");
+
+ for (curr = miss_insns, i = 0; curr && i < limit; i++, curr = curr->next) {
+ insn = (InsnData *) curr->data;
+ g_string_append_printf(rep, "0x%" PRIx64, insn->addr);
+ if (insn->symbol) {
+ g_string_append_printf(rep, " (%s)", insn->symbol);
+ }
+ g_string_append_printf(rep, ", %ld, %s\n", insn->imisses,
+ insn->disas_str);
+ }
+
+ qemu_plugin_outs(rep->str);
+ g_list_free(miss_insns);
+}
+
+static void plugin_exit(qemu_plugin_id_t id, void *p)
+{
+ log_stats();
+ log_top_insns();
+
+ cache_free(dcache);
+ cache_free(icache);
+
+ g_hash_table_destroy(miss_ht);
+}
+
+QEMU_PLUGIN_EXPORT
+int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
+ int argc, char **argv)
+{
+ int i;
+ int iassoc, iblksize, icachesize;
+ int dassoc, dblksize, dcachesize;
+
+ limit = 32;
+ sys = info->system_emulation;
+
+ dassoc = 8;
+ dblksize = 64;
+ dcachesize = dblksize * dassoc * 32;
+
+ iassoc = 8;
+ iblksize = 64;
+ icachesize = iblksize * iassoc * 32;
+
+
+ for (i = 0; i < argc; i++) {
+ char *opt = argv[i];
+ if (g_str_has_prefix(opt, "limit=")) {
+ limit = g_ascii_strtoll(opt + 6, NULL, 10);
+ } else {
+ fprintf(stderr, "option parsing failed: %s\n", opt);
+ return -1;
+ }
+ }
+
+ dcache = cache_init(dblksize, dassoc, dcachesize);
+ icache = cache_init(iblksize, iassoc, icachesize);
+
+ rng = g_rand_new();
+
+ qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans);
+ qemu_plugin_register_atexit_cb(id, plugin_exit, NULL);
+
+ miss_ht = g_hash_table_new_full(NULL, g_direct_equal, NULL, insn_free);
+
+ return 0;
+}
diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile
index 3c9209b6b0..54ac5ccd9f 100644
--- a/contrib/plugins/Makefile
+++ b/contrib/plugins/Makefile
@@ -19,6 +19,7 @@ NAMES += hotpages
NAMES += howvec
NAMES += lockstep
NAMES += hwprofile
+NAMES += cache
SONAMES := $(addsuffix .so,$(addprefix lib,$(NAMES)))
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 37/40] plugins/cache: Enable cache parameterization
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (35 preceding siblings ...)
2021-07-09 14:30 ` [PATCH v3 36/40] plugins: Added a new cache modelling plugin Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 38/40] plugins/cache: Added FIFO and LRU eviction policies Alex Bennée
` (2 subsequent siblings)
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Mahmoud Mandour <ma.mandourr@gmail.com>
Enabled configuring both icache and dcache parameters using plugin
parameters.
Signed-off-by: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623125458.450462-3-ma.mandourr@gmail.com>
---
contrib/plugins/cache.c | 47 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 46 insertions(+), 1 deletion(-)
diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c
index e9955cdc3a..b550ef31b0 100644
--- a/contrib/plugins/cache.c
+++ b/contrib/plugins/cache.c
@@ -99,8 +99,28 @@ static inline uint64_t extract_set(Cache *cache, uint64_t addr)
return (addr & cache->set_mask) >> cache->blksize_shift;
}
+static const char *cache_config_error(int blksize, int assoc, int cachesize)
+{
+ if (cachesize % blksize != 0) {
+ return "cache size must be divisible by block size";
+ } else if (cachesize % (blksize * assoc) != 0) {
+ return "cache size must be divisible by set size (assoc * block size)";
+ } else {
+ return NULL;
+ }
+}
+
+static bool bad_cache_params(int blksize, int assoc, int cachesize)
+{
+ return (cachesize % blksize) != 0 || (cachesize % (blksize * assoc) != 0);
+}
+
static Cache *cache_init(int blksize, int assoc, int cachesize)
{
+ if (bad_cache_params(blksize, assoc, cachesize)) {
+ return NULL;
+ }
+
Cache *cache;
int i;
uint64_t blk_mask;
@@ -397,7 +417,19 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
for (i = 0; i < argc; i++) {
char *opt = argv[i];
- if (g_str_has_prefix(opt, "limit=")) {
+ if (g_str_has_prefix(opt, "iblksize=")) {
+ iblksize = g_ascii_strtoll(opt + 9, NULL, 10);
+ } else if (g_str_has_prefix(opt, "iassoc=")) {
+ iassoc = g_ascii_strtoll(opt + 7, NULL, 10);
+ } else if (g_str_has_prefix(opt, "icachesize=")) {
+ icachesize = g_ascii_strtoll(opt + 11, NULL, 10);
+ } else if (g_str_has_prefix(opt, "dblksize=")) {
+ dblksize = g_ascii_strtoll(opt + 9, NULL, 10);
+ } else if (g_str_has_prefix(opt, "dassoc=")) {
+ dassoc = g_ascii_strtoll(opt + 7, NULL, 10);
+ } else if (g_str_has_prefix(opt, "dcachesize=")) {
+ dcachesize = g_ascii_strtoll(opt + 11, NULL, 10);
+ } else if (g_str_has_prefix(opt, "limit=")) {
limit = g_ascii_strtoll(opt + 6, NULL, 10);
} else {
fprintf(stderr, "option parsing failed: %s\n", opt);
@@ -406,7 +438,20 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
}
dcache = cache_init(dblksize, dassoc, dcachesize);
+ if (!dcache) {
+ const char *err = cache_config_error(dblksize, dassoc, dcachesize);
+ fprintf(stderr, "dcache cannot be constructed from given parameters\n");
+ fprintf(stderr, "%s\n", err);
+ return -1;
+ }
+
icache = cache_init(iblksize, iassoc, icachesize);
+ if (!icache) {
+ const char *err = cache_config_error(iblksize, iassoc, icachesize);
+ fprintf(stderr, "icache cannot be constructed from given parameters\n");
+ fprintf(stderr, "%s\n", err);
+ return -1;
+ }
rng = g_rand_new();
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 38/40] plugins/cache: Added FIFO and LRU eviction policies
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (36 preceding siblings ...)
2021-07-09 14:30 ` [PATCH v3 37/40] plugins/cache: Enable cache parameterization Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 39/40] docs/devel: Added cache plugin to the plugins docs Alex Bennée
2021-07-09 14:30 ` [PATCH v3 40/40] MAINTAINERS: Added myself as a reviewer for TCG Plugins Alex Bennée
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Mahmoud Mandour <ma.mandourr@gmail.com>
Implemented FIFO and LRU eviction policies. Now one of the three
eviction policies can be chosen as an argument. On not specifying an
argument, LRU is used by default.
Signed-off-by: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623125458.450462-4-ma.mandourr@gmail.com>
---
contrib/plugins/cache.c | 203 +++++++++++++++++++++++++++++++++++++---
1 file changed, 190 insertions(+), 13 deletions(-)
diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c
index b550ef31b0..bf0d2f6097 100644
--- a/contrib/plugins/cache.c
+++ b/contrib/plugins/cache.c
@@ -29,6 +29,14 @@ static uint64_t dmisses;
static uint64_t imem_accesses;
static uint64_t imisses;
+enum EvictionPolicy {
+ LRU,
+ FIFO,
+ RAND,
+};
+
+enum EvictionPolicy policy;
+
/*
* A CacheSet is a set of cache blocks. A memory block that maps to a set can be
* put in any of the blocks inside the set. The number of block per set is
@@ -48,6 +56,8 @@ static uint64_t imisses;
* The set number is used to identify the set in which the block may exist.
* The tag is compared against all the tags of a set to search for a match. If a
* match is found, then the access is a hit.
+ *
+ * The CacheSet also contains bookkeaping information about eviction details.
*/
typedef struct {
@@ -57,6 +67,9 @@ typedef struct {
typedef struct {
CacheBlock *blocks;
+ uint64_t *lru_priorities;
+ uint64_t lru_gen_counter;
+ GQueue *fifo_queue;
} CacheSet;
typedef struct {
@@ -77,6 +90,12 @@ typedef struct {
uint64_t imisses;
} InsnData;
+void (*update_hit)(Cache *cache, int set, int blk);
+void (*update_miss)(Cache *cache, int set, int blk);
+
+void (*metadata_init)(Cache *cache);
+void (*metadata_destroy)(Cache *cache);
+
Cache *dcache, *icache;
static int pow_of_two(int num)
@@ -89,6 +108,103 @@ static int pow_of_two(int num)
return ret;
}
+/*
+ * LRU evection policy: For each set, a generation counter is maintained
+ * alongside a priority array.
+ *
+ * On each set access, the generation counter is incremented.
+ *
+ * On a cache hit: The hit-block is assigned the current generation counter,
+ * indicating that it is the most recently used block.
+ *
+ * On a cache miss: The block with the least priority is searched and replaced
+ * with the newly-cached block, of which the priority is set to the current
+ * generation number.
+ */
+
+static void lru_priorities_init(Cache *cache)
+{
+ int i;
+
+ for (i = 0; i < cache->num_sets; i++) {
+ cache->sets[i].lru_priorities = g_new0(uint64_t, cache->assoc);
+ cache->sets[i].lru_gen_counter = 0;
+ }
+}
+
+static void lru_update_blk(Cache *cache, int set_idx, int blk_idx)
+{
+ CacheSet *set = &cache->sets[set_idx];
+ set->lru_priorities[blk_idx] = cache->sets[set_idx].lru_gen_counter;
+ set->lru_gen_counter++;
+}
+
+static int lru_get_lru_block(Cache *cache, int set_idx)
+{
+ int i, min_idx, min_priority;
+
+ min_priority = cache->sets[set_idx].lru_priorities[0];
+ min_idx = 0;
+
+ for (i = 1; i < cache->assoc; i++) {
+ if (cache->sets[set_idx].lru_priorities[i] < min_priority) {
+ min_priority = cache->sets[set_idx].lru_priorities[i];
+ min_idx = i;
+ }
+ }
+ return min_idx;
+}
+
+static void lru_priorities_destroy(Cache *cache)
+{
+ int i;
+
+ for (i = 0; i < cache->num_sets; i++) {
+ g_free(cache->sets[i].lru_priorities);
+ }
+}
+
+/*
+ * FIFO eviction policy: a FIFO queue is maintained for each CacheSet that
+ * stores accesses to the cache.
+ *
+ * On a compulsory miss: The block index is enqueued to the fifo_queue to
+ * indicate that it's the latest cached block.
+ *
+ * On a conflict miss: The first-in block is removed from the cache and the new
+ * block is put in its place and enqueued to the FIFO queue.
+ */
+
+static void fifo_init(Cache *cache)
+{
+ int i;
+
+ for (i = 0; i < cache->num_sets; i++) {
+ cache->sets[i].fifo_queue = g_queue_new();
+ }
+}
+
+static int fifo_get_first_block(Cache *cache, int set)
+{
+ GQueue *q = cache->sets[set].fifo_queue;
+ return GPOINTER_TO_INT(g_queue_pop_tail(q));
+}
+
+static void fifo_update_on_miss(Cache *cache, int set, int blk_idx)
+{
+ GQueue *q = cache->sets[set].fifo_queue;
+ g_queue_push_head(q, GINT_TO_POINTER(blk_idx));
+}
+
+static void fifo_destroy(Cache *cache)
+{
+ int i;
+
+ for (i = 0; i < cache->assoc; i++) {
+ g_queue_free(cache->sets[i].fifo_queue);
+ }
+}
+
static inline uint64_t extract_tag(Cache *cache, uint64_t addr)
{
return addr & cache->tag_mask;
@@ -139,6 +255,11 @@ static Cache *cache_init(int blksize, int assoc, int cachesize)
blk_mask = blksize - 1;
cache->set_mask = ((cache->num_sets - 1) << cache->blksize_shift);
cache->tag_mask = ~(cache->set_mask | blk_mask);
+
+ if (metadata_init) {
+ metadata_init(cache);
+ }
+
return cache;
}
@@ -155,12 +276,21 @@ static int get_invalid_block(Cache *cache, uint64_t set)
return -1;
}
-static int get_replaced_block(Cache *cache)
+static int get_replaced_block(Cache *cache, int set)
{
- return g_rand_int_range(rng, 0, cache->assoc);
+ switch (policy) {
+ case RAND:
+ return g_rand_int_range(rng, 0, cache->assoc);
+ case LRU:
+ return lru_get_lru_block(cache, set);
+ case FIFO:
+ return fifo_get_first_block(cache, set);
+ default:
+ g_assert_not_reached();
+ }
}
-static bool in_cache(Cache *cache, uint64_t addr)
+static int in_cache(Cache *cache, uint64_t addr)
{
int i;
uint64_t tag, set;
@@ -171,11 +301,11 @@ static bool in_cache(Cache *cache, uint64_t addr)
for (i = 0; i < cache->assoc; i++) {
if (cache->sets[set].blocks[i].tag == tag &&
cache->sets[set].blocks[i].valid) {
- return true;
+ return i;
}
}
- return false;
+ return -1;
}
/**
@@ -188,20 +318,28 @@ static bool in_cache(Cache *cache, uint64_t addr)
*/
static bool access_cache(Cache *cache, uint64_t addr)
{
+ int hit_blk, replaced_blk;
uint64_t tag, set;
- int replaced_blk;
-
- if (in_cache(cache, addr)) {
- return true;
- }
tag = extract_tag(cache, addr);
set = extract_set(cache, addr);
+ hit_blk = in_cache(cache, addr);
+ if (hit_blk != -1) {
+ if (update_hit) {
+ update_hit(cache, set, hit_blk);
+ }
+ return true;
+ }
+
replaced_blk = get_invalid_block(cache, set);
if (replaced_blk == -1) {
- replaced_blk = get_replaced_block(cache);
+ replaced_blk = get_replaced_block(cache, set);
+ }
+
+ if (update_miss) {
+ update_miss(cache, set, replaced_blk);
}
cache->sets[set].blocks[replaced_blk].tag = tag;
@@ -308,6 +446,10 @@ static void cache_free(Cache *cache)
g_free(cache->sets[i].blocks);
}
+ if (metadata_destroy) {
+ metadata_destroy(cache);
+ }
+
g_free(cache->sets);
g_free(cache);
}
@@ -395,6 +537,28 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)
g_hash_table_destroy(miss_ht);
}
+static void policy_init()
+{
+ switch (policy) {
+ case LRU:
+ update_hit = lru_update_blk;
+ update_miss = lru_update_blk;
+ metadata_init = lru_priorities_init;
+ metadata_destroy = lru_priorities_destroy;
+ break;
+ case FIFO:
+ update_miss = fifo_update_on_miss;
+ metadata_init = fifo_init;
+ metadata_destroy = fifo_destroy;
+ break;
+ case RAND:
+ rng = g_rand_new();
+ break;
+ default:
+ g_assert_not_reached();
+ }
+}
+
QEMU_PLUGIN_EXPORT
int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
int argc, char **argv)
@@ -414,6 +578,7 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
iblksize = 64;
icachesize = iblksize * iassoc * 32;
+ policy = LRU;
for (i = 0; i < argc; i++) {
char *opt = argv[i];
@@ -431,12 +596,26 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
dcachesize = g_ascii_strtoll(opt + 11, NULL, 10);
} else if (g_str_has_prefix(opt, "limit=")) {
limit = g_ascii_strtoll(opt + 6, NULL, 10);
+ } else if (g_str_has_prefix(opt, "evict=")) {
+ gchar *p = opt + 6;
+ if (g_strcmp0(p, "rand") == 0) {
+ policy = RAND;
+ } else if (g_strcmp0(p, "lru") == 0) {
+ policy = LRU;
+ } else if (g_strcmp0(p, "fifo") == 0) {
+ policy = FIFO;
+ } else {
+ fprintf(stderr, "invalid eviction policy: %s\n", opt);
+ return -1;
+ }
} else {
fprintf(stderr, "option parsing failed: %s\n", opt);
return -1;
}
}
+ policy_init();
+
dcache = cache_init(dblksize, dassoc, dcachesize);
if (!dcache) {
const char *err = cache_config_error(dblksize, dassoc, dcachesize);
@@ -453,8 +632,6 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
return -1;
}
- rng = g_rand_new();
-
qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans);
qemu_plugin_register_atexit_cb(id, plugin_exit, NULL);
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 39/40] docs/devel: Added cache plugin to the plugins docs
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (37 preceding siblings ...)
2021-07-09 14:30 ` [PATCH v3 38/40] plugins/cache: Added FIFO and LRU eviction policies Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
2021-07-09 14:30 ` [PATCH v3 40/40] MAINTAINERS: Added myself as a reviewer for TCG Plugins Alex Bennée
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alexandre Iooss, Alex Bennée, aurelien
From: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Mahmoud Mandour <ma.mandourr@gmail.com>
Message-Id: <20210628053808.17422-1-ma.mandourr@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
docs/devel/tcg-plugins.rst | 59 ++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst
index 179867e9c1..7e54f12837 100644
--- a/docs/devel/tcg-plugins.rst
+++ b/docs/devel/tcg-plugins.rst
@@ -344,3 +344,62 @@ which will output an execution trace following this structure::
0, 0xd32, 0xf9893014, "adds r0, #0x14"
0, 0xd34, 0xf9c8f000, "bl #0x10c8"
0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM
+
+- contrib/plugins/cache
+
+Cache modelling plugin that measures the performance of a given cache
+configuration when a given working set is run::
+
+ qemu-x86_64 -plugin ./contrib/plugins/libcache.so \
+ -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs
+
+will report the following::
+
+ Data accesses: 996479, Misses: 507
+ Miss rate: 0.050879%
+
+ Instruction accesses: 2641737, Misses: 18617
+ Miss rate: 0.704726%
+
+ address, data misses, instruction
+ 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx)
+ 0x41f395 (_IO_default_xsputn), 49, movb %dl, (%rdi, %rax)
+ 0x42584d (ptmalloc_init.part.0), 33, movaps %xmm0, (%rax)
+ 0x454d48 (__tunables_init), 20, cmpb $0, (%r8)
+ ...
+
+ address, fetch misses, instruction
+ 0x4160a0 (__vfprintf_internal), 744, movl $1, %ebx
+ 0x41f0a0 (_IO_setb), 744, endbr64
+ 0x415882 (__vfprintf_internal), 744, movq %r12, %rdi
+ 0x4268a0 (__malloc), 696, andq $0xfffffffffffffff0, %rax
+ ...
+
+The plugin has a number of arguments, all of them are optional:
+
+ * arg="limit=N"
+
+ Print top N icache and dcache thrashing instructions along with their
+ address, number of misses, and its disassembly. (default: 32)
+
+ * arg="icachesize=N"
+ * arg="iblksize=B"
+ * arg="iassoc=A"
+
+ Instruction cache configuration arguments. They specify the cache size, block
+ size, and associativity of the instruction cache, respectively.
+ (default: N = 16384, B = 64, A = 8)
+
+ * arg="dcachesize=N"
+ * arg="dblksize=B"
+ * arg="dassoc=A"
+
+ Data cache configuration arguments. They specify the cache size, block size,
+ and associativity of the data cache, respectively.
+ (default: N = 16384, B = 64, A = 8)
+
+ * arg="evict=POLICY"
+
+ Sets the eviction policy to POLICY. Available policies are: :code:`lru`,
+ :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for
+ both instruction and data caches. (default: POLICY = :code:`lru`)
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH v3 40/40] MAINTAINERS: Added myself as a reviewer for TCG Plugins
2021-07-09 14:29 [PATCH for 6.1 v3 00/40] testing and plugins pre-PR Alex Bennée
` (38 preceding siblings ...)
2021-07-09 14:30 ` [PATCH v3 39/40] docs/devel: Added cache plugin to the plugins docs Alex Bennée
@ 2021-07-09 14:30 ` Alex Bennée
39 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2021-07-09 14:30 UTC (permalink / raw)
To: qemu-devel
Cc: fam, minyihh, berrange, kuhn.chenqun, f4bug, robhenry,
mahmoudabdalghany, aaron, cota, stefanha, crosa, pbonzini,
ma.mandourr, Alex Bennée, aurelien
From: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Mahmoud Mandour <ma.mandourr@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623125458.450462-6-ma.mandourr@gmail.com>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index ee456e5fb4..77b304afac 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2982,6 +2982,7 @@ F: include/tcg/
TCG Plugins
M: Alex Bennée <alex.bennee@linaro.org>
R: Alexandre Iooss <erdnaxe@crans.org>
+R: Mahmoud Mandour <ma.mandourr@gmail.com>
S: Maintained
F: docs/devel/tcg-plugins.rst
F: plugins/
--
2.20.1
^ permalink raw reply related [flat|nested] 45+ messages in thread