All of lore.kernel.org
 help / color / mirror / Atom feed
* [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile)
@ 2022-06-01 18:05 Alex Bennée
  2022-06-01 18:05 ` [PULL 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages Alex Bennée
                   ` (33 more replies)
  0 siblings, 34 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson; +Cc: qemu-devel, Alex Bennée

The following changes since commit 7077fcb9b68f058809c9dd9fd1dacae1881e886c:

  Merge tag 'vmbus-maint-20220530' of https://github.com/maciejsszmigiero/qemu into staging (2022-05-30 12:40:36 -0700)

are available in the Git repository at:

  https://github.com/stsquad/qemu.git tags/pull-testing-next-010622-3

for you to fetch changes up to 7266ecce502c31387a3cbf83d7297bc9cf27b139:

  docs/devel: clean-up the CI links in the docs (2022-06-01 18:55:04 +0100)

----------------------------------------------------------------
Various testing updates

  - fix some gitlab container dependencies
  - report meson test results via JUnit
  - fix meson display of enabled cross compilers
  - convert more cross build containers to lcitool and Debian 11
  - re-factor cross compiler detection
  - use test cross-compilers for building ROMs
  - disable CI runs by default (see docs)
  - fix some broken links in development documentation

----------------------------------------------------------------
Alex Bennée (9):
      meson.build: fix summary display of test compilers
      tests/lcitool: fix up indentation to correct style
      tests/docker: update debian-armhf-cross with lcitool
      tests/docker: update debian-armel-cross with lcitool
      tests/docker: update debian-mipsel-cross with lcitool
      tests/docker: update debian-mips64el-cross with lcitool
      tests/docker: update debian-ppc64el-cross with lcitool
      tests/docker: update debian-amd64 with lcitool
      docs/devel: clean-up the CI links in the docs

Daniel P. Berrangé (5):
      gitlab: introduce a common base job template
      gitlab: convert Cirrus jobs to .base_job_template
      gitlab: convert static checks to .base_job_template
      gitlab: convert build/container jobs to .base_job_template
      gitlab: don't run CI jobs in forks by default

Marc-André Lureau (1):
      gitlab-ci: add meson JUnit test result into report

Paolo Bonzini (16):
      configure: do not define or use the CPP variable
      build: clean up ninja invocation
      build: add a more generic way to specify make->ninja dependencies
      build: do a full build before running TCG tests
      configure, meson: move symlinking of ROMs to meson
      tests/tcg: correct target CPU for sparc32
      tests/tcg: merge configure.sh back into main configure script
      configure: add missing cross compiler fallbacks
      configure: handle host compiler in probe_target_compiler
      configure: introduce --cross-prefix-*=
      configure: include more binutils in tests/tcg makefile
      configure: move symlink configuration earlier
      configure: enable cross-compilation of s390-ccw
      configure: enable cross-compilation of optionrom
      configure: enable cross compilation of vof
      configure: remove unused variables from config-host.mak

Thomas Huth (2):
      .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages
      .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job

 docs/devel/ci-jobs.rst.inc                         | 116 +++-
 docs/devel/ci.rst                                  |  11 +-
 docs/devel/submitting-a-patch.rst                  |  36 +-
 docs/devel/testing.rst                             |   2 +
 configure                                          | 606 ++++++++++++++++++---
 Makefile                                           |   9 +-
 pc-bios/s390-ccw/netboot.mak                       |   2 +-
 meson.build                                        |   8 +-
 .gitlab-ci.d/base.yml                              |  72 +++
 .gitlab-ci.d/buildtest-template.yml                |  18 +-
 .gitlab-ci.d/buildtest.yml                         |  28 +-
 .gitlab-ci.d/cirrus.yml                            |  16 +-
 .gitlab-ci.d/container-cross.yml                   |  24 +-
 .gitlab-ci.d/container-template.yml                |   1 +
 .gitlab-ci.d/containers.yml                        |   3 +-
 .gitlab-ci.d/crossbuild-template.yml               |   3 +
 .gitlab-ci.d/crossbuilds.yml                       |   2 +
 .gitlab-ci.d/qemu-project.yml                      |   1 +
 .gitlab-ci.d/static_checks.yml                     |  19 +-
 .gitlab-ci.d/windows.yml                           |   1 +
 pc-bios/meson.build                                |  17 +-
 pc-bios/optionrom/Makefile                         |   4 +-
 pc-bios/s390-ccw/Makefile                          |   9 +-
 pc-bios/vof/Makefile                               |  17 +-
 scripts/mtest2make.py                              |   8 +-
 tests/Makefile.include                             |   4 +-
 tests/docker/Makefile.include                      |   5 -
 tests/docker/dockerfiles/debian-amd64.docker       | 194 +++++--
 tests/docker/dockerfiles/debian-armel-cross.docker | 178 +++++-
 tests/docker/dockerfiles/debian-armhf-cross.docker | 184 ++++++-
 .../dockerfiles/debian-mips64el-cross.docker       | 177 +++++-
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 179 +++++-
 .../docker/dockerfiles/debian-ppc64el-cross.docker | 178 +++++-
 tests/lcitool/refresh                              | 178 ++++--
 tests/tcg/configure.sh                             | 376 -------------
 35 files changed, 1885 insertions(+), 801 deletions(-)
 create mode 100644 .gitlab-ci.d/base.yml
 delete mode 100755 tests/tcg/configure.sh

-- 
2.30.2



^ permalink raw reply	[flat|nested] 45+ messages in thread

* [PULL 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 02/33] .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job Alex Bennée
                   ` (32 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Thomas Huth, Alex Bennée, Alistair Francis,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Thomas Huth <thuth@redhat.com>

The "riscv64-debian-cross-container" job does not depend on any other
container job from the first stage, so we can move it to the first
stage, too.

The "riscv64-debian-test-cross-container" job needs the debian11
container, so we should add a proper "needs:" statement here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220524093141.91012-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <20220527153603.887929-2-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index e622ac2d21..ac15fce9b6 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -125,7 +125,7 @@ ppc64el-debian-cross-container:
 
 riscv64-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
+  stage: containers
   # as we are currently based on 'sid/unstable' we may break so...
   allow_failure: true
   variables:
@@ -135,6 +135,7 @@ riscv64-debian-cross-container:
 riscv64-debian-test-cross-container:
   extends: .container_job_template
   stage: containers-layer2
+  needs: ['amd64-debian11-container']
   variables:
     NAME: debian-riscv64-test-cross
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 02/33] .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
  2022-06-01 18:05 ` [PULL 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 03/33] gitlab-ci: add meson JUnit test result into report Alex Bennée
                   ` (31 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Thomas Huth, Alex Bennée,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Thomas Huth <thuth@redhat.com>

The cross-i386-tci job uses the fedora-i386-cross image, so we should make sure
that the corresponding job that builds it (the i386-fedora-cross-container job)
has finished before we start the TCI job.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220524092600.89997-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-3-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 17d6cb3e45..4a5fb6ea2a 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -62,6 +62,8 @@ cross-i386-user:
 cross-i386-tci:
   extends: .cross_accel_build_job
   timeout: 60m
+  needs:
+    job: i386-fedora-cross-container
   variables:
     IMAGE: fedora-i386-cross
     ACCEL: tcg-interpreter
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 03/33] gitlab-ci: add meson JUnit test result into report
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
  2022-06-01 18:05 ` [PULL 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages Alex Bennée
  2022-06-01 18:05 ` [PULL 02/33] .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 04/33] meson.build: fix summary display of test compilers Alex Bennée
                   ` (30 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Marc-André Lureau, Alex Bennée,
	Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This allows the gitlab UI to show the test results in different ways,
see doc:

  https://docs.gitlab.com/ee/ci/unit_test_reports.html#how-it-works

Previous we only reports avocado test results (.avocado_test_job_template),
with this change, the qemu/meson tests are also covered.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20220525173411.612224-1-marcandre.lureau@redhat.com>
[AJB: expand the commit description]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-4-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index dc6d67aacf..b381345dbc 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -44,6 +44,8 @@
     expire_in: 7 days
     paths:
       - build/meson-logs/testlog.txt
+    reports:
+      junit: build/meson-logs/testlog.junit.xml
 
 .avocado_test_job_template:
   extends: .common_test_job_template
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 04/33] meson.build: fix summary display of test compilers
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (2 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 03/33] gitlab-ci: add meson JUnit test result into report Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 05/33] tests/lcitool: fix up indentation to correct style Alex Bennée
                   ` (29 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson; +Cc: qemu-devel, Alex Bennée

The recent refactoring of configure.sh dropped a number of variables
we relied on for printing out information. Make it simpler.

Fixes: eebf199c09 (tests/tcg: invoke Makefile.target directly from QEMU's makefile)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220527153603.887929-5-alex.bennee@linaro.org>

diff --git a/meson.build b/meson.build
index bf318d9cbb..bc6234c85e 100644
--- a/meson.build
+++ b/meson.build
@@ -3735,12 +3735,8 @@ foreach target: target_dirs
     config_cross_tcg = keyval.load(tcg_mak)
     target = config_cross_tcg['TARGET_NAME']
     compiler = ''
-    if 'DOCKER_CROSS_CC_GUEST' in config_cross_tcg
-      summary_info += {target + ' tests': config_cross_tcg['DOCKER_CROSS_CC_GUEST'] +
-                                          ' via ' + config_cross_tcg['DOCKER_IMAGE']}
-    elif 'CROSS_CC_GUEST' in config_cross_tcg
-      summary_info += {target + ' tests'
-                                : config_cross_tcg['CROSS_CC_GUEST'] }
+    if 'CC' in config_cross_tcg
+      summary_info += {target + ' tests': config_cross_tcg['CC']}
     endif
    endif
 endforeach
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 05/33] tests/lcitool: fix up indentation to correct style
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (3 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 04/33] meson.build: fix summary display of test compilers Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 06/33] tests/docker: update debian-armhf-cross with lcitool Alex Bennée
                   ` (28 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

3 space indentation snuck into the initial commit. Clean it up before
we let it get established. I've also:

  - removed unused os import
  - added double lines between functions
  - added some comments and grouped and sorted the generation stanzas

My lint tool is also recommending using f-strings but that requires
python 3.6.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-6-alex.bennee@linaro.org>

diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index fb49bbc441..dc1fc21ef9 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -13,14 +13,13 @@
 # the top-level directory.
 
 import sys
-import os
 import subprocess
 
 from pathlib import Path
 
 if len(sys.argv) != 1:
-   print("syntax: %s" % sys.argv[0], file=sys.stderr)
-   sys.exit(1)
+    print("syntax: %s" % sys.argv[0], file=sys.stderr)
+    sys.exit(1)
 
 self_dir = Path(__file__).parent
 src_dir = self_dir.parent.parent
@@ -30,76 +29,95 @@ lcitool_path = Path(self_dir, "libvirt-ci", "lcitool")
 
 lcitool_cmd = [lcitool_path, "--data-dir", self_dir]
 
+
 def atomic_write(filename, content):
-   tmp = filename.with_suffix(filename.suffix + ".tmp")
-   try:
-      with tmp.open("w") as fp:
-         print(content, file=fp, end="")
-         tmp.rename(filename)
-   except Exception as ex:
-      tmp.unlink()
-      raise
+    tmp = filename.with_suffix(filename.suffix + ".tmp")
+    try:
+        with tmp.open("w") as fp:
+            print(content, file=fp, end="")
+            tmp.rename(filename)
+    except Exception as ex:
+        tmp.unlink()
+        raise
+
 
 def generate(filename, cmd, trailer):
-   print("Generate %s" % filename)
-   lcitool=subprocess.run(cmd, capture_output=True)
+    print("Generate %s" % filename)
+    lcitool = subprocess.run(cmd, capture_output=True)
 
-   if lcitool.returncode != 0:
-      raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr))
+    if lcitool.returncode != 0:
+        raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr))
+
+    content = lcitool.stdout.decode("utf8")
+    if trailer is not None:
+        content += trailer
+    atomic_write(filename, content)
 
-   content = lcitool.stdout.decode("utf8")
-   if trailer is not None:
-      content += trailer
-   atomic_write(filename, content)
 
 def generate_dockerfile(host, target, cross=None, trailer=None):
-   filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker")
-   cmd = lcitool_cmd + ["dockerfile"]
-   if cross is not None:
-      cmd.extend(["--cross", cross])
-   cmd.extend([target, "qemu"])
-   generate(filename, cmd, trailer)
+    filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker")
+    cmd = lcitool_cmd + ["dockerfile"]
+    if cross is not None:
+        cmd.extend(["--cross", cross])
+    cmd.extend([target, "qemu"])
+    generate(filename, cmd, trailer)
+
 
 def generate_cirrus(target, trailer=None):
-   filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars")
-   cmd = lcitool_cmd + ["variables", target, "qemu"]
-   generate(filename, cmd, trailer)
+    filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars")
+    cmd = lcitool_cmd + ["variables", target, "qemu"]
+    generate(filename, cmd, trailer)
+
 
 ubuntu2004_tsanhack = [
-   "# Apply patch https://reviews.llvm.org/D75820\n",
-   "# This is required for TSan in clang-10 to compile with QEMU.\n",
-   "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"
+    "# Apply patch https://reviews.llvm.org/D75820\n",
+    "# This is required for TSan in clang-10 to compile with QEMU.\n",
+    "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"
 ]
 
+
 def debian_cross_build(prefix, targets):
-   conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix)
-   targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
-   return "".join([conf, targets])
+    conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix)
+    targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
+    return "".join([conf, targets])
 
+#
+# Update all the various build configurations.
+# Please keep each group sorted alphabetically for easy reading.
+#
 
 try:
-   generate_dockerfile("centos8", "centos-stream-8")
-   generate_dockerfile("fedora", "fedora-35")
-   generate_dockerfile("ubuntu2004", "ubuntu-2004",
-                       trailer="".join(ubuntu2004_tsanhack))
-   generate_dockerfile("opensuse-leap", "opensuse-leap-152")
-   generate_dockerfile("alpine", "alpine-edge")
-
-   generate_dockerfile("debian-arm64-cross", "debian-11",
-                       cross="aarch64",
-                       trailer=debian_cross_build("aarch64-linux-gnu-",
-                                                  "aarch64-softmmu,aarch64-linux-user"))
-
-   generate_dockerfile("debian-s390x-cross", "debian-11",
-                       cross="s390x",
-                       trailer=debian_cross_build("s390x-linux-gnu-",
-                                                  "s390x-softmmu,s390x-linux-user"))
-
-   generate_cirrus("freebsd-12")
-   generate_cirrus("freebsd-13")
-   generate_cirrus("macos-11")
-
-   sys.exit(0)
+    #
+    # Standard native builds
+    #
+    generate_dockerfile("alpine", "alpine-edge")
+    generate_dockerfile("centos8", "centos-stream-8")
+    generate_dockerfile("fedora", "fedora-35")
+    generate_dockerfile("opensuse-leap", "opensuse-leap-152")
+    generate_dockerfile("ubuntu2004", "ubuntu-2004",
+                        trailer="".join(ubuntu2004_tsanhack))
+
+    #
+    # Cross compiling builds
+    #
+    generate_dockerfile("debian-arm64-cross", "debian-11",
+                        cross="aarch64",
+                        trailer=debian_cross_build("aarch64-linux-gnu-",
+                                                   "aarch64-softmmu,aarch64-linux-user"))
+
+    generate_dockerfile("debian-s390x-cross", "debian-11",
+                        cross="s390x",
+                        trailer=debian_cross_build("s390x-linux-gnu-",
+                                                   "s390x-softmmu,s390x-linux-user"))
+
+    #
+    # Cirrus packages lists for GitLab
+    #
+    generate_cirrus("freebsd-12")
+    generate_cirrus("freebsd-13")
+    generate_cirrus("macos-11")
+
+    sys.exit(0)
 except Exception as ex:
-   print(str(ex), file=sys.stderr)
-   sys.exit(1)
+    print(str(ex), file=sys.stderr)
+    sys.exit(1)
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 06/33] tests/docker: update debian-armhf-cross with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (4 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 05/33] tests/lcitool: fix up indentation to correct style Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 07/33] tests/docker: update debian-armel-cross " Alex Bennée
                   ` (27 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

Use lcitool to update debian-armhf-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-7-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index ac15fce9b6..4d1830f3fc 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -34,8 +34,7 @@ armel-debian-cross-container:
 
 armhf-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-armhf-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ca2157db46..d6e0710554 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -90,7 +90,6 @@ endif
 
 docker-image-debian-alpha-cross: docker-image-debian10
 docker-image-debian-armel-cross: docker-image-debian10
-docker-image-debian-armhf-cross: docker-image-debian10
 docker-image-debian-hppa-cross: docker-image-debian10
 docker-image-debian-m68k-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 25d7618833..a2ebce96f8 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -1,29 +1,165 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker armhf cross-compiler target
+#  $ lcitool dockerfile --layers all --cross armv7l debian-11 qemu
 #
-# This docker target builds on the debian Stretch base image.
-#
-FROM qemu/debian10
+# https://gitlab.com/libvirt/libvirt-ci
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture armhf
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        crossbuild-essential-armhf
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a armhf --arch-only qemu
+FROM docker.io/library/debian:11-slim
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
-ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user,armeb-linux-user
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
+
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:armhf \
-        liblzo2-dev:armhf \
-        librdmacm-dev:armhf \
-        libsnappy-dev:armhf \
-        libxen-dev:armhf
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture armhf && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-arm-linux-gnueabihf \
+            gcc-arm-linux-gnueabihf \
+            libaio-dev:armhf \
+            libasan5:armhf \
+            libasound2-dev:armhf \
+            libattr1-dev:armhf \
+            libbpf-dev:armhf \
+            libbrlapi-dev:armhf \
+            libbz2-dev:armhf \
+            libc6-dev:armhf \
+            libcacard-dev:armhf \
+            libcap-ng-dev:armhf \
+            libcapstone-dev:armhf \
+            libcurl4-gnutls-dev:armhf \
+            libdaxctl-dev:armhf \
+            libdrm-dev:armhf \
+            libepoxy-dev:armhf \
+            libfdt-dev:armhf \
+            libffi-dev:armhf \
+            libfuse3-dev:armhf \
+            libgbm-dev:armhf \
+            libgcrypt20-dev:armhf \
+            libglib2.0-dev:armhf \
+            libglusterfs-dev:armhf \
+            libgnutls28-dev:armhf \
+            libgtk-3-dev:armhf \
+            libibumad-dev:armhf \
+            libibverbs-dev:armhf \
+            libiscsi-dev:armhf \
+            libjemalloc-dev:armhf \
+            libjpeg62-turbo-dev:armhf \
+            liblttng-ust-dev:armhf \
+            liblzo2-dev:armhf \
+            libncursesw5-dev:armhf \
+            libnfs-dev:armhf \
+            libnuma-dev:armhf \
+            libpam0g-dev:armhf \
+            libpixman-1-dev:armhf \
+            libpng-dev:armhf \
+            libpulse-dev:armhf \
+            librbd-dev:armhf \
+            librdmacm-dev:armhf \
+            libsasl2-dev:armhf \
+            libsdl2-dev:armhf \
+            libsdl2-image-dev:armhf \
+            libseccomp-dev:armhf \
+            libselinux1-dev:armhf \
+            libslirp-dev:armhf \
+            libsnappy-dev:armhf \
+            libspice-server-dev:armhf \
+            libssh-gcrypt-dev:armhf \
+            libsystemd-dev:armhf \
+            libtasn1-6-dev:armhf \
+            libubsan1:armhf \
+            libudev-dev:armhf \
+            liburing-dev:armhf \
+            libusb-1.0-0-dev:armhf \
+            libusbredirhost-dev:armhf \
+            libvdeplug-dev:armhf \
+            libvirglrenderer-dev:armhf \
+            libvte-2.91-dev:armhf \
+            libxen-dev:armhf \
+            libzstd-dev:armhf \
+            nettle-dev:armhf \
+            systemtap-sdt-dev:armhf \
+            xfslibs-dev:armhf \
+            zlib1g-dev:armhf && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/arm-linux-gnueabihf-gcc'\n\
+ar = '/usr/bin/arm-linux-gnueabihf-gcc-ar'\n\
+strip = '/usr/bin/arm-linux-gnueabihf-strip'\n\
+pkgconfig = '/usr/bin/arm-linux-gnueabihf-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'arm'\n\
+cpu = 'armhf'\n\
+endian = 'little'" > /usr/local/share/meson/cross/arm-linux-gnueabihf && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabihf-gcc
+
+ENV ABI "arm-linux-gnueabihf"
+ENV MESON_OPTS "--cross-file=arm-linux-gnueabihf"
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index dc1fc21ef9..7e4a92f630 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -105,6 +105,11 @@ try:
                         trailer=debian_cross_build("aarch64-linux-gnu-",
                                                    "aarch64-softmmu,aarch64-linux-user"))
 
+    generate_dockerfile("debian-armhf-cross", "debian-11",
+                        cross="armv7l",
+                        trailer=debian_cross_build("arm-linux-gnueabihf-",
+                                                   "arm-softmmu,arm-linux-user"))
+
     generate_dockerfile("debian-s390x-cross", "debian-11",
                         cross="s390x",
                         trailer=debian_cross_build("s390x-linux-gnu-",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 07/33] tests/docker: update debian-armel-cross with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (5 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 06/33] tests/docker: update debian-armhf-cross with lcitool Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 08/33] tests/docker: update debian-mipsel-cross " Alex Bennée
                   ` (26 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

Use lcitool to update debian-armel-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-8-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 4d1830f3fc..caef7decf4 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -27,8 +27,7 @@ arm64-debian-cross-container:
 
 armel-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-armel-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d6e0710554..d9109bcc77 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -89,7 +89,6 @@ DOCKER_PARTIAL_IMAGES += fedora
 endif
 
 docker-image-debian-alpha-cross: docker-image-debian10
-docker-image-debian-armel-cross: docker-image-debian10
 docker-image-debian-hppa-cross: docker-image-debian10
 docker-image-debian-m68k-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index b7b1a3585f..a6153e5a83 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -1,26 +1,164 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker armel cross-compiler target
+#  $ lcitool dockerfile --layers all --cross armv6l debian-11 qemu
 #
-# This docker target builds on the debian Stretch base image.
-#
-FROM qemu/debian10
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+# https://gitlab.com/libvirt/libvirt-ci
+
+FROM docker.io/library/debian:11-slim
+
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture armel && \
-    apt update && \
-    apt install -yy crossbuild-essential-armel && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a armel --arch-only qemu
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture armel && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-arm-linux-gnueabi \
+            gcc-arm-linux-gnueabi \
+            libaio-dev:armel \
+            libasan5:armel \
+            libasound2-dev:armel \
+            libattr1-dev:armel \
+            libbpf-dev:armel \
+            libbrlapi-dev:armel \
+            libbz2-dev:armel \
+            libc6-dev:armel \
+            libcacard-dev:armel \
+            libcap-ng-dev:armel \
+            libcapstone-dev:armel \
+            libcurl4-gnutls-dev:armel \
+            libdaxctl-dev:armel \
+            libdrm-dev:armel \
+            libepoxy-dev:armel \
+            libfdt-dev:armel \
+            libffi-dev:armel \
+            libfuse3-dev:armel \
+            libgbm-dev:armel \
+            libgcrypt20-dev:armel \
+            libglib2.0-dev:armel \
+            libglusterfs-dev:armel \
+            libgnutls28-dev:armel \
+            libgtk-3-dev:armel \
+            libibumad-dev:armel \
+            libibverbs-dev:armel \
+            libiscsi-dev:armel \
+            libjemalloc-dev:armel \
+            libjpeg62-turbo-dev:armel \
+            liblttng-ust-dev:armel \
+            liblzo2-dev:armel \
+            libncursesw5-dev:armel \
+            libnfs-dev:armel \
+            libnuma-dev:armel \
+            libpam0g-dev:armel \
+            libpixman-1-dev:armel \
+            libpng-dev:armel \
+            libpulse-dev:armel \
+            librbd-dev:armel \
+            librdmacm-dev:armel \
+            libsasl2-dev:armel \
+            libsdl2-dev:armel \
+            libsdl2-image-dev:armel \
+            libseccomp-dev:armel \
+            libselinux1-dev:armel \
+            libslirp-dev:armel \
+            libsnappy-dev:armel \
+            libspice-server-dev:armel \
+            libssh-gcrypt-dev:armel \
+            libsystemd-dev:armel \
+            libtasn1-6-dev:armel \
+            libubsan1:armel \
+            libudev-dev:armel \
+            liburing-dev:armel \
+            libusb-1.0-0-dev:armel \
+            libusbredirhost-dev:armel \
+            libvdeplug-dev:armel \
+            libvirglrenderer-dev:armel \
+            libvte-2.91-dev:armel \
+            libzstd-dev:armel \
+            nettle-dev:armel \
+            systemtap-sdt-dev:armel \
+            xfslibs-dev:armel \
+            zlib1g-dev:armel && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/arm-linux-gnueabi-gcc'\n\
+ar = '/usr/bin/arm-linux-gnueabi-gcc-ar'\n\
+strip = '/usr/bin/arm-linux-gnueabi-strip'\n\
+pkgconfig = '/usr/bin/arm-linux-gnueabi-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'arm'\n\
+cpu = 'arm'\n\
+endian = 'little'" > /usr/local/share/meson/cross/arm-linux-gnueabi && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/arm-linux-gnueabi-gcc
+
+ENV ABI "arm-linux-gnueabi"
+ENV MESON_OPTS "--cross-file=arm-linux-gnueabi"
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabi-
 ENV DEF_TARGET_LIST arm-softmmu,arm-linux-user,armeb-linux-user
-
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:armel \
-        liblzo2-dev:armel \
-        librdmacm-dev:armel \
-        libsnappy-dev:armel
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 7e4a92f630..4dc5527234 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -105,6 +105,11 @@ try:
                         trailer=debian_cross_build("aarch64-linux-gnu-",
                                                    "aarch64-softmmu,aarch64-linux-user"))
 
+    generate_dockerfile("debian-armel-cross", "debian-11",
+                        cross="armv6l",
+                        trailer=debian_cross_build("arm-linux-gnueabi-",
+                                                   "arm-softmmu,arm-linux-user,armeb-linux-user"))
+
     generate_dockerfile("debian-armhf-cross", "debian-11",
                         cross="armv7l",
                         trailer=debian_cross_build("arm-linux-gnueabihf-",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 08/33] tests/docker: update debian-mipsel-cross with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (6 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 07/33] tests/docker: update debian-armel-cross " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 09/33] tests/docker: update debian-mips64el-cross " Alex Bennée
                   ` (25 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal,
	Jiaxun Yang

Use lcitool to update debian-mipsel-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-9-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index caef7decf4..1a533e6fc0 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -102,8 +102,7 @@ mips-debian-cross-container:
 
 mipsel-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-mipsel-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d9109bcc77..0ac5975419 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -94,7 +94,6 @@ docker-image-debian-m68k-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian10
 docker-image-debian-mips64-cross: docker-image-debian10
 docker-image-debian-mips64el-cross: docker-image-debian10
-docker-image-debian-mipsel-cross: docker-image-debian10
 docker-image-debian-ppc64el-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index 0e5dd42d3c..b6d99ae324 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -1,31 +1,162 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker mipsel cross-compiler target
+#  $ lcitool dockerfile --layers all --cross mipsel debian-11 qemu
 #
-# This docker target builds on the debian Stretch base image.
-#
-FROM qemu/debian10
+# https://gitlab.com/libvirt/libvirt-ci
 
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+FROM docker.io/library/debian:11-slim
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture mipsel
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        gcc-mipsel-linux-gnu
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a mipsel --arch-only qemu
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture mipsel && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-mipsel-linux-gnu \
+            gcc-mipsel-linux-gnu \
+            libaio-dev:mipsel \
+            libasound2-dev:mipsel \
+            libattr1-dev:mipsel \
+            libbpf-dev:mipsel \
+            libbrlapi-dev:mipsel \
+            libbz2-dev:mipsel \
+            libc6-dev:mipsel \
+            libcacard-dev:mipsel \
+            libcap-ng-dev:mipsel \
+            libcapstone-dev:mipsel \
+            libcurl4-gnutls-dev:mipsel \
+            libdaxctl-dev:mipsel \
+            libdrm-dev:mipsel \
+            libepoxy-dev:mipsel \
+            libfdt-dev:mipsel \
+            libffi-dev:mipsel \
+            libfuse3-dev:mipsel \
+            libgbm-dev:mipsel \
+            libgcrypt20-dev:mipsel \
+            libglib2.0-dev:mipsel \
+            libglusterfs-dev:mipsel \
+            libgnutls28-dev:mipsel \
+            libgtk-3-dev:mipsel \
+            libibumad-dev:mipsel \
+            libibverbs-dev:mipsel \
+            libiscsi-dev:mipsel \
+            libjemalloc-dev:mipsel \
+            libjpeg62-turbo-dev:mipsel \
+            liblttng-ust-dev:mipsel \
+            liblzo2-dev:mipsel \
+            libncursesw5-dev:mipsel \
+            libnfs-dev:mipsel \
+            libnuma-dev:mipsel \
+            libpam0g-dev:mipsel \
+            libpixman-1-dev:mipsel \
+            libpng-dev:mipsel \
+            libpulse-dev:mipsel \
+            librbd-dev:mipsel \
+            librdmacm-dev:mipsel \
+            libsasl2-dev:mipsel \
+            libsdl2-dev:mipsel \
+            libsdl2-image-dev:mipsel \
+            libseccomp-dev:mipsel \
+            libselinux1-dev:mipsel \
+            libslirp-dev:mipsel \
+            libsnappy-dev:mipsel \
+            libspice-server-dev:mipsel \
+            libssh-gcrypt-dev:mipsel \
+            libsystemd-dev:mipsel \
+            libtasn1-6-dev:mipsel \
+            libudev-dev:mipsel \
+            liburing-dev:mipsel \
+            libusb-1.0-0-dev:mipsel \
+            libusbredirhost-dev:mipsel \
+            libvdeplug-dev:mipsel \
+            libvirglrenderer-dev:mipsel \
+            libvte-2.91-dev:mipsel \
+            libzstd-dev:mipsel \
+            nettle-dev:mipsel \
+            systemtap-sdt-dev:mipsel \
+            xfslibs-dev:mipsel \
+            zlib1g-dev:mipsel && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/mipsel-linux-gnu-gcc'\n\
+ar = '/usr/bin/mipsel-linux-gnu-gcc-ar'\n\
+strip = '/usr/bin/mipsel-linux-gnu-strip'\n\
+pkgconfig = '/usr/bin/mipsel-linux-gnu-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'mips'\n\
+cpu = 'mipsel'\n\
+endian = 'little'" > /usr/local/share/meson/cross/mipsel-linux-gnu && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mipsel-linux-gnu-gcc
 
-# Install extra libraries to increase code coverage
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:mipsel \
-        liblzo2-dev:mipsel \
-        librdmacm-dev:mipsel \
-        libsnappy-dev:mipsel
+ENV ABI "mipsel-linux-gnu"
+ENV MESON_OPTS "--cross-file=mipsel-linux-gnu"
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+ENV DEF_TARGET_LIST mipsel-softmmu,mipsel-linux-user
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 4dc5527234..a4b7452189 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -115,6 +115,11 @@ try:
                         trailer=debian_cross_build("arm-linux-gnueabihf-",
                                                    "arm-softmmu,arm-linux-user"))
 
+    generate_dockerfile("debian-mipsel-cross", "debian-11",
+                        cross="mipsel",
+                        trailer=debian_cross_build("mipsel-linux-gnu-",
+                                                   "mipsel-softmmu,mipsel-linux-user"))
+
     generate_dockerfile("debian-s390x-cross", "debian-11",
                         cross="s390x",
                         trailer=debian_cross_build("s390x-linux-gnu-",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 09/33] tests/docker: update debian-mips64el-cross with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (7 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 08/33] tests/docker: update debian-mipsel-cross " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 10/33] tests/docker: update debian-ppc64el-cross " Alex Bennée
                   ` (24 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal,
	Jiaxun Yang

Use lcitool to update debian-mips64el-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-10-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 1a533e6fc0..411dc06bf8 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -88,8 +88,7 @@ mips64-debian-cross-container:
 
 mips64el-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-mips64el-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 0ac5975419..d9f37ae8fa 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -93,7 +93,6 @@ docker-image-debian-hppa-cross: docker-image-debian10
 docker-image-debian-m68k-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian10
 docker-image-debian-mips64-cross: docker-image-debian10
-docker-image-debian-mips64el-cross: docker-image-debian10
 docker-image-debian-ppc64el-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index c990b683b7..b02dcb7fd9 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -1,33 +1,162 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker mips64el cross-compiler target
-#
-# This docker target builds on the debian Stretch base image.
+#  $ lcitool dockerfile --layers all --cross mips64el debian-11 qemu
 #
+# https://gitlab.com/libvirt/libvirt-ci
 
-FROM qemu/debian10
+FROM docker.io/library/debian:11-slim
 
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture mips64el && \
-    apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        gcc-mips64el-linux-gnuabi64
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a mips64el --arch-only qemu
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture mips64el && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-mips64el-linux-gnuabi64 \
+            gcc-mips64el-linux-gnuabi64 \
+            libaio-dev:mips64el \
+            libasound2-dev:mips64el \
+            libattr1-dev:mips64el \
+            libbpf-dev:mips64el \
+            libbrlapi-dev:mips64el \
+            libbz2-dev:mips64el \
+            libc6-dev:mips64el \
+            libcacard-dev:mips64el \
+            libcap-ng-dev:mips64el \
+            libcapstone-dev:mips64el \
+            libcurl4-gnutls-dev:mips64el \
+            libdaxctl-dev:mips64el \
+            libdrm-dev:mips64el \
+            libepoxy-dev:mips64el \
+            libfdt-dev:mips64el \
+            libffi-dev:mips64el \
+            libfuse3-dev:mips64el \
+            libgbm-dev:mips64el \
+            libgcrypt20-dev:mips64el \
+            libglib2.0-dev:mips64el \
+            libglusterfs-dev:mips64el \
+            libgnutls28-dev:mips64el \
+            libgtk-3-dev:mips64el \
+            libibumad-dev:mips64el \
+            libibverbs-dev:mips64el \
+            libiscsi-dev:mips64el \
+            libjemalloc-dev:mips64el \
+            libjpeg62-turbo-dev:mips64el \
+            liblttng-ust-dev:mips64el \
+            liblzo2-dev:mips64el \
+            libncursesw5-dev:mips64el \
+            libnfs-dev:mips64el \
+            libnuma-dev:mips64el \
+            libpam0g-dev:mips64el \
+            libpixman-1-dev:mips64el \
+            libpng-dev:mips64el \
+            libpulse-dev:mips64el \
+            librbd-dev:mips64el \
+            librdmacm-dev:mips64el \
+            libsasl2-dev:mips64el \
+            libsdl2-dev:mips64el \
+            libsdl2-image-dev:mips64el \
+            libseccomp-dev:mips64el \
+            libselinux1-dev:mips64el \
+            libslirp-dev:mips64el \
+            libsnappy-dev:mips64el \
+            libspice-server-dev:mips64el \
+            libssh-gcrypt-dev:mips64el \
+            libsystemd-dev:mips64el \
+            libtasn1-6-dev:mips64el \
+            libudev-dev:mips64el \
+            liburing-dev:mips64el \
+            libusb-1.0-0-dev:mips64el \
+            libusbredirhost-dev:mips64el \
+            libvdeplug-dev:mips64el \
+            libvirglrenderer-dev:mips64el \
+            libvte-2.91-dev:mips64el \
+            libzstd-dev:mips64el \
+            nettle-dev:mips64el \
+            systemtap-sdt-dev:mips64el \
+            xfslibs-dev:mips64el \
+            zlib1g-dev:mips64el && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/mips64el-linux-gnuabi64-gcc'\n\
+ar = '/usr/bin/mips64el-linux-gnuabi64-gcc-ar'\n\
+strip = '/usr/bin/mips64el-linux-gnuabi64-strip'\n\
+pkgconfig = '/usr/bin/mips64el-linux-gnuabi64-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'mips64'\n\
+cpu = 'mips64el'\n\
+endian = 'little'" > /usr/local/share/meson/cross/mips64el-linux-gnuabi64 && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/mips64el-linux-gnuabi64-gcc
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV ABI "mips64el-linux-gnuabi64"
+ENV MESON_OPTS "--cross-file=mips64el-linux-gnuabi64"
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=mips64el-linux-gnuabi64-
 ENV DEF_TARGET_LIST mips64el-softmmu,mips64el-linux-user
-
-# Install extra libraries to increase code coverage
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:mips64el \
-        liblzo2-dev:mips64el \
-        librdmacm-dev:mips64el \
-        libsnappy-dev:mips64el
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index a4b7452189..341a07e677 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -115,6 +115,11 @@ try:
                         trailer=debian_cross_build("arm-linux-gnueabihf-",
                                                    "arm-softmmu,arm-linux-user"))
 
+    generate_dockerfile("debian-mips64el-cross", "debian-11",
+                        cross="mips64el",
+                        trailer=debian_cross_build("mips64el-linux-gnuabi64-",
+                                                  "mips64el-softmmu,mips64el-linux-user"))
+
     generate_dockerfile("debian-mipsel-cross", "debian-11",
                         cross="mipsel",
                         trailer=debian_cross_build("mipsel-linux-gnu-",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 10/33] tests/docker: update debian-ppc64el-cross with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (8 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 09/33] tests/docker: update debian-mips64el-cross " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 11/33] tests/docker: update debian-amd64 " Alex Bennée
                   ` (23 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé,
	Philippe Mathieu-Daudé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

Use lcitool to update debian-ppc64el-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-11-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 411dc06bf8..147e667744 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -114,8 +114,7 @@ powerpc-test-cross-container:
 
 ppc64el-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-ppc64el-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d9f37ae8fa..e68f91b853 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -93,7 +93,6 @@ docker-image-debian-hppa-cross: docker-image-debian10
 docker-image-debian-m68k-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian10
 docker-image-debian-mips64-cross: docker-image-debian10
-docker-image-debian-ppc64el-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
 
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index 5de12b01cd..bcf04bc90b 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -1,28 +1,164 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker ppc64el cross-compiler target
+#  $ lcitool dockerfile --layers all --cross ppc64le debian-11 qemu
 #
-# This docker target builds on the debian Stretch base image.
-#
-FROM qemu/debian10
+# https://gitlab.com/libvirt/libvirt-ci
+
+FROM docker.io/library/debian:11-slim
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture ppc64el && \
-    apt update && \
-    apt install -yy crossbuild-essential-ppc64el
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a ppc64el --arch-only qemu
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture ppc64el && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-powerpc64le-linux-gnu \
+            gcc-powerpc64le-linux-gnu \
+            libaio-dev:ppc64el \
+            libasan5:ppc64el \
+            libasound2-dev:ppc64el \
+            libattr1-dev:ppc64el \
+            libbpf-dev:ppc64el \
+            libbrlapi-dev:ppc64el \
+            libbz2-dev:ppc64el \
+            libc6-dev:ppc64el \
+            libcacard-dev:ppc64el \
+            libcap-ng-dev:ppc64el \
+            libcapstone-dev:ppc64el \
+            libcurl4-gnutls-dev:ppc64el \
+            libdaxctl-dev:ppc64el \
+            libdrm-dev:ppc64el \
+            libepoxy-dev:ppc64el \
+            libfdt-dev:ppc64el \
+            libffi-dev:ppc64el \
+            libfuse3-dev:ppc64el \
+            libgbm-dev:ppc64el \
+            libgcrypt20-dev:ppc64el \
+            libglib2.0-dev:ppc64el \
+            libglusterfs-dev:ppc64el \
+            libgnutls28-dev:ppc64el \
+            libgtk-3-dev:ppc64el \
+            libibumad-dev:ppc64el \
+            libibverbs-dev:ppc64el \
+            libiscsi-dev:ppc64el \
+            libjemalloc-dev:ppc64el \
+            libjpeg62-turbo-dev:ppc64el \
+            liblttng-ust-dev:ppc64el \
+            liblzo2-dev:ppc64el \
+            libncursesw5-dev:ppc64el \
+            libnfs-dev:ppc64el \
+            libnuma-dev:ppc64el \
+            libpam0g-dev:ppc64el \
+            libpixman-1-dev:ppc64el \
+            libpng-dev:ppc64el \
+            libpulse-dev:ppc64el \
+            librbd-dev:ppc64el \
+            librdmacm-dev:ppc64el \
+            libsasl2-dev:ppc64el \
+            libsdl2-dev:ppc64el \
+            libsdl2-image-dev:ppc64el \
+            libseccomp-dev:ppc64el \
+            libselinux1-dev:ppc64el \
+            libslirp-dev:ppc64el \
+            libsnappy-dev:ppc64el \
+            libspice-server-dev:ppc64el \
+            libssh-gcrypt-dev:ppc64el \
+            libsystemd-dev:ppc64el \
+            libtasn1-6-dev:ppc64el \
+            libubsan1:ppc64el \
+            libudev-dev:ppc64el \
+            liburing-dev:ppc64el \
+            libusb-1.0-0-dev:ppc64el \
+            libusbredirhost-dev:ppc64el \
+            libvdeplug-dev:ppc64el \
+            libvirglrenderer-dev:ppc64el \
+            libvte-2.91-dev:ppc64el \
+            libzstd-dev:ppc64el \
+            nettle-dev:ppc64el \
+            systemtap-sdt-dev:ppc64el \
+            xfslibs-dev:ppc64el \
+            zlib1g-dev:ppc64el && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/powerpc64le-linux-gnu-gcc'\n\
+ar = '/usr/bin/powerpc64le-linux-gnu-gcc-ar'\n\
+strip = '/usr/bin/powerpc64le-linux-gnu-strip'\n\
+pkgconfig = '/usr/bin/powerpc64le-linux-gnu-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'ppc64'\n\
+cpu = 'powerpc64le'\n\
+endian = 'little'" > /usr/local/share/meson/cross/powerpc64le-linux-gnu && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/powerpc64le-linux-gnu-gcc
+
+ENV ABI "powerpc64le-linux-gnu"
+ENV MESON_OPTS "--cross-file=powerpc64le-linux-gnu"
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc64le-linux-gnu-
 ENV DEF_TARGET_LIST ppc64-softmmu,ppc64-linux-user
-
-# Install extra libraries to increase code coverage
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:ppc64el \
-        liblzo2-dev:ppc64el \
-        librdmacm-dev:ppc64el \
-        libsnappy-dev:ppc64el
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 341a07e677..b947bb170d 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -125,6 +125,11 @@ try:
                         trailer=debian_cross_build("mipsel-linux-gnu-",
                                                    "mipsel-softmmu,mipsel-linux-user"))
 
+    generate_dockerfile("debian-ppc64el-cross", "debian-11",
+                        cross="ppc64le",
+                        trailer=debian_cross_build("powerpc64le-linux-gnu-",
+                                                   "ppc64-softmmu,ppc64-linux-user"))
+
     generate_dockerfile("debian-s390x-cross", "debian-11",
                         cross="s390x",
                         trailer=debian_cross_build("s390x-linux-gnu-",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 11/33] tests/docker: update debian-amd64 with lcitool
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (9 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 10/33] tests/docker: update debian-ppc64el-cross " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 12/33] configure: do not define or use the CPP variable Alex Bennée
                   ` (22 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Philippe Mathieu-Daudé,
	Luigi Rizzo, Giuseppe Lettieri, Vincenzo Maffione,
	Daniel P . Berrangé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

The one minor wrinkle we need to account for is the netmap support
still requires building from source. We also include cscope and GNU
global as they are used in one of the builds.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Luigi Rizzo <rizzo@iet.unipi.it>
Cc: Giuseppe Lettieri <g.lettieri@iet.unipi.it>
Cc: Vincenzo Maffione <v.maffione@gmail.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-12-alex.bennee@linaro.org>

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index e9df90bbdd..be34cbc7ba 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -14,8 +14,7 @@ amd64-debian11-container:
 
 amd64-debian-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-amd64
 
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index ed546edcd6..503e282802 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -1,59 +1,153 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker x86_64 target
+#  $ lcitool dockerfile --layers all debian-11 qemu
 #
-# This docker target builds on the Debian Buster base image. Further
-# libraries which are not widely available are installed by hand.
-#
-FROM qemu/debian10
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
-
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy qemu
+# https://gitlab.com/libvirt/libvirt-ci
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        cscope \
-        genisoimage \
-        exuberant-ctags \
-        global \
-        libbz2-dev \
-        liblzo2-dev \
-        libgcrypt20-dev \
-        libfdt-dev \
-        librdmacm-dev \
-        libsasl2-dev \
-        libsnappy-dev \
-        libvte-dev \
-        netcat-openbsd \
-        openssh-client \
-        python3-numpy \
-        python3-opencv \
-        python3-venv
+FROM docker.io/library/debian:11-slim
 
-# virgl
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libegl1-mesa-dev \
-        libepoxy-dev \
-        libgbm-dev
-RUN git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git /usr/src/virglrenderer && \
-    cd /usr/src/virglrenderer && git checkout virglrenderer-0.8.0
-RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --disable-tests && make install
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            clang \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            g++ \
+            gcc \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libaio-dev \
+            libasan5 \
+            libasound2-dev \
+            libattr1-dev \
+            libbpf-dev \
+            libbrlapi-dev \
+            libbz2-dev \
+            libc6-dev \
+            libcacard-dev \
+            libcap-ng-dev \
+            libcapstone-dev \
+            libcurl4-gnutls-dev \
+            libdaxctl-dev \
+            libdrm-dev \
+            libepoxy-dev \
+            libfdt-dev \
+            libffi-dev \
+            libfuse3-dev \
+            libgbm-dev \
+            libgcrypt20-dev \
+            libglib2.0-dev \
+            libglusterfs-dev \
+            libgnutls28-dev \
+            libgtk-3-dev \
+            libibumad-dev \
+            libibverbs-dev \
+            libiscsi-dev \
+            libjemalloc-dev \
+            libjpeg62-turbo-dev \
+            liblttng-ust-dev \
+            liblzo2-dev \
+            libncursesw5-dev \
+            libnfs-dev \
+            libnuma-dev \
+            libpam0g-dev \
+            libpcre2-dev \
+            libpixman-1-dev \
+            libpmem-dev \
+            libpng-dev \
+            libpulse-dev \
+            librbd-dev \
+            librdmacm-dev \
+            libsasl2-dev \
+            libsdl2-dev \
+            libsdl2-image-dev \
+            libseccomp-dev \
+            libselinux1-dev \
+            libslirp-dev \
+            libsnappy-dev \
+            libspice-protocol-dev \
+            libspice-server-dev \
+            libssh-gcrypt-dev \
+            libsystemd-dev \
+            libtasn1-6-dev \
+            libubsan1 \
+            libudev-dev \
+            liburing-dev \
+            libusb-1.0-0-dev \
+            libusbredirhost-dev \
+            libvdeplug-dev \
+            libvirglrenderer-dev \
+            libvte-2.91-dev \
+            libxen-dev \
+            libzstd-dev \
+            llvm \
+            locales \
+            make \
+            meson \
+            multipath-tools \
+            ncat \
+            nettle-dev \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            systemtap-sdt-dev \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo \
+            xfslibs-dev \
+            zlib1g-dev && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-# netmap
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        linux-headers-amd64
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
+# netmap/cscope/global
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+  apt install -y --no-install-recommends \
+  cscope\
+  global\
+  linux-headers-amd64
 RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
 RUN cd /usr/src/netmap && git checkout v11.3
 RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
 ENV QEMU_CONFIGURE_OPTS --enable-netmap
-
-RUN ldconfig
-
-# gcrypt
-ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-gcrypt
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index b947bb170d..5e260f8cd6 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -76,6 +76,23 @@ ubuntu2004_tsanhack = [
 ]
 
 
+# Netmap still needs to be manually built as it is yet to be packaged
+# into a distro. We also add cscope and gtags which are used in the CI
+# test
+debian11_extras = [
+    "# netmap/cscope/global\n",
+    "RUN DEBIAN_FRONTEND=noninteractive eatmydata \\\n",
+    "  apt install -y --no-install-recommends \\\n",
+    "  cscope\\\n",
+    "  global\\\n",
+    "  linux-headers-amd64\n",
+    "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n",
+    "RUN cd /usr/src/netmap && git checkout v11.3\n",
+    "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n",
+    "ENV QEMU_CONFIGURE_OPTS --enable-netmap\n"
+]
+
+
 def debian_cross_build(prefix, targets):
     conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix)
     targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
@@ -92,6 +109,8 @@ try:
     #
     generate_dockerfile("alpine", "alpine-edge")
     generate_dockerfile("centos8", "centos-stream-8")
+    generate_dockerfile("debian-amd64", "debian-11",
+                        trailer="".join(debian11_extras))
     generate_dockerfile("fedora", "fedora-35")
     generate_dockerfile("opensuse-leap", "opensuse-leap-152")
     generate_dockerfile("ubuntu2004", "ubuntu-2004",
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 12/33] configure: do not define or use the CPP variable
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (10 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 11/33] tests/docker: update debian-amd64 " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 13/33] build: clean up ninja invocation Alex Bennée
                   ` (21 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

Just hardcode $(CC) -E, it should be enough.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-2-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-13-alex.bennee@linaro.org>

diff --git a/configure b/configure
index f2baf2f526..c88ef94fec 100755
--- a/configure
+++ b/configure
@@ -376,7 +376,6 @@ fi
 ar="${AR-${cross_prefix}ar}"
 as="${AS-${cross_prefix}as}"
 ccas="${CCAS-$cc}"
-cpp="${CPP-$cc -E}"
 objcopy="${OBJCOPY-${cross_prefix}objcopy}"
 ld="${LD-${cross_prefix}ld}"
 ranlib="${RANLIB-${cross_prefix}ranlib}"
@@ -2014,7 +2013,6 @@ echo "CC=$cc" >> $config_host_mak
 echo "AR=$ar" >> $config_host_mak
 echo "AS=$as" >> $config_host_mak
 echo "CCAS=$ccas" >> $config_host_mak
-echo "CPP=$cpp" >> $config_host_mak
 echo "OBJCOPY=$objcopy" >> $config_host_mak
 echo "LD=$ld" >> $config_host_mak
 echo "QEMU_CFLAGS=$QEMU_CFLAGS" >> $config_host_mak
@@ -2257,7 +2255,6 @@ preserve_env() {
 preserve_env AR
 preserve_env AS
 preserve_env CC
-preserve_env CPP
 preserve_env CFLAGS
 preserve_env CXX
 preserve_env CXXFLAGS
diff --git a/pc-bios/optionrom/Makefile b/pc-bios/optionrom/Makefile
index 2494ad9c25..17ccc76241 100644
--- a/pc-bios/optionrom/Makefile
+++ b/pc-bios/optionrom/Makefile
@@ -50,7 +50,7 @@ override LDFLAGS = -m $(LD_I386_EMULATION) -T $(SRC_DIR)/flat.lds
 pvh.img: pvh.o pvh_main.o
 
 %.o: %.S
-	$(call quiet-command,$(CPP) $(CPPFLAGS) -c -o - $< | $(AS) $(ASFLAGS) -o $@,"AS","$@")
+	$(call quiet-command,$(CC) $(CPPFLAGS) -E -o - $< | $(AS) $(ASFLAGS) -o $@,"AS","$@")
 
 %.o: %.c
 	$(call quiet-command,$(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@,"CC","$@")
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 13/33] build: clean up ninja invocation
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (11 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 12/33] configure: do not define or use the CPP variable Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 14/33] build: add a more generic way to specify make->ninja dependencies Alex Bennée
                   ` (20 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

Fix an incorrect "@@:" and move "-d keepdepfile" to the NINJAFLAGS variable.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-3-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-14-alex.bennee@linaro.org>

diff --git a/Makefile b/Makefile
index b842dbccdb..fad312040f 100644
--- a/Makefile
+++ b/Makefile
@@ -143,7 +143,7 @@ MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS))))
 MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq)
 NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
         $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \
-
+        -d keepdepfile
 ninja-cmd-goals = $(or $(MAKECMDGOALS), all)
 ninja-cmd-goals += $(foreach t, $(.check.build-suites), $(.check-$t.deps))
 ninja-cmd-goals += $(foreach t, $(.bench.build-suites), $(.bench-$t.deps))
@@ -160,8 +160,8 @@ $(ninja-targets): run-ninja
 # --output-sync line.
 run-ninja: config-host.mak
 ifneq ($(filter $(ninja-targets), $(ninja-cmd-goals)),)
-	+$(quiet-@)$(if $(MAKE.nq),@:, $(NINJA) -d keepdepfile \
-	   $(NINJAFLAGS) $(sort $(filter $(ninja-targets), $(ninja-cmd-goals))) | cat)
+	+$(if $(MAKE.nq),@:,$(quiet-@)$(NINJA) $(NINJAFLAGS) \
+	   $(sort $(filter $(ninja-targets), $(ninja-cmd-goals))) | cat)
 endif
 endif
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 14/33] build: add a more generic way to specify make->ninja dependencies
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (12 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 13/33] build: clean up ninja invocation Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 15/33] build: do a full build before running TCG tests Alex Bennée
                   ` (19 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée, John Snow, Cleber Rosa

From: Paolo Bonzini <pbonzini@redhat.com>

Let any make target specify ninja goals that needs to be built for it
(though selecting the goals is _not_ recursive on depending targets)
instead of having a custom mechanism only for "make check" and "make
bench".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-4-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-15-alex.bennee@linaro.org>

diff --git a/Makefile b/Makefile
index fad312040f..3c0d89057e 100644
--- a/Makefile
+++ b/Makefile
@@ -145,8 +145,7 @@ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
         $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \
         -d keepdepfile
 ninja-cmd-goals = $(or $(MAKECMDGOALS), all)
-ninja-cmd-goals += $(foreach t, $(.check.build-suites), $(.check-$t.deps))
-ninja-cmd-goals += $(foreach t, $(.bench.build-suites), $(.bench-$t.deps))
+ninja-cmd-goals += $(foreach g, $(MAKECMDGOALS), $(.ninja-goals.$g))))
 
 makefile-targets := build.ninja ctags TAGS cscope dist clean uninstall
 # "ninja -t targets" also lists all prerequisites.  If build system
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 304634b71e..0fe81efbbc 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -81,12 +81,12 @@ def emit_prolog(suites, prefix):
 
 def emit_suite_deps(name, suite, prefix):
     deps = ' '.join(suite.deps)
-    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml {prefix} {prefix}-report.junit.xml'
+    targets = [f'{prefix}-{name}', f'{prefix}-report-{name}.junit.xml', f'{prefix}', f'{prefix}-report.junit.xml',
+               f'{prefix}-build']
     print()
     print(f'.{prefix}-{name}.deps = {deps}')
-    print(f'ifneq ($(filter {prefix}-build {targets}, $(MAKECMDGOALS)),)')
-    print(f'.{prefix}.build-suites += {name}')
-    print(f'endif')
+    for t in targets:
+        print(f'.ninja-goals.{t} += $(.{prefix}-{name}.deps)')
 
 def emit_suite(name, suite, prefix):
     emit_suite_deps(name, suite, prefix)
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 15/33] build: do a full build before running TCG tests
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (13 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 14/33] build: add a more generic way to specify make->ninja dependencies Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 16/33] configure, meson: move symlinking of ROMs to meson Alex Bennée
                   ` (18 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

TCG tests need both QEMU and firmware to be built, so do "ninja all" before
trying to run them.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org>

diff --git a/tests/Makefile.include b/tests/Makefile.include
index ec84b2ebc0..72ce0561f4 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -57,7 +57,7 @@ $(TCG_TESTS_TARGETS:%=build-tcg-tests-%): build-tcg-tests-%: $(BUILD_DIR)/tests/
         "BUILD","$* guest-tests")
 
 .PHONY: $(TCG_TESTS_TARGETS:%=run-tcg-tests-%)
-$(TCG_TESTS_TARGETS:%=run-tcg-tests-%): run-tcg-tests-%: build-tcg-tests-% $(if $(CONFIG_PLUGIN),test-plugins)
+$(TCG_TESTS_TARGETS:%=run-tcg-tests-%): run-tcg-tests-%: build-tcg-tests-%
 	$(call quiet-command, \
            $(MAKE) -C tests/tcg/$* -f ../Makefile.target $(SUBDIR_MAKEFLAGS) \
                         TARGET="$*" SRC_PATH="$(SRC_PATH)" SPEED=$(SPEED) run, \
@@ -74,6 +74,7 @@ $(TCG_TESTS_TARGETS:%=clean-tcg-tests-%): clean-tcg-tests-%:
 build-tcg: $(BUILD_TCG_TARGET_RULES)
 
 .PHONY: check-tcg
+.ninja-goals.check-tcg = all $(if $(CONFIG_PLUGIN),test-plugins)
 check-tcg: $(RUN_TCG_TARGET_RULES)
 
 .PHONY: clean-tcg
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 16/33] configure, meson: move symlinking of ROMs to meson
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (14 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 15/33] build: do a full build before running TCG tests Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 17/33] tests/tcg: correct target CPU for sparc32 Alex Bennée
                   ` (17 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

This is useful because pc-bios/meson.build already has a list of all ROM
files, and thus does not need to use wildcards.  The problems with
wildcards are mentioned above the definition of the LINKS variable,
but then the recommendation is disattended.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-6-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-17-alex.bennee@linaro.org>

diff --git a/configure b/configure
index c88ef94fec..7b6adc29fe 100755
--- a/configure
+++ b/configure
@@ -2100,21 +2100,6 @@ LINKS="$LINKS tests/avocado tests/data"
 LINKS="$LINKS tests/qemu-iotests/check"
 LINKS="$LINKS python"
 LINKS="$LINKS contrib/plugins/Makefile "
-for bios_file in \
-    $source_path/pc-bios/*.bin \
-    $source_path/pc-bios/*.elf \
-    $source_path/pc-bios/*.lid \
-    $source_path/pc-bios/*.rom \
-    $source_path/pc-bios/*.dtb \
-    $source_path/pc-bios/*.img \
-    $source_path/pc-bios/openbios-* \
-    $source_path/pc-bios/u-boot.* \
-    $source_path/pc-bios/palcode-* \
-    $source_path/pc-bios/qemu_vga.ndrv
-
-do
-    LINKS="$LINKS pc-bios/$(basename $bios_file)"
-done
 for f in $LINKS ; do
     if [ -e "$source_path/$f" ]; then
         mkdir -p `dirname ./$f`
diff --git a/pc-bios/meson.build b/pc-bios/meson.build
index c86dedf7df..41ba1c0ec7 100644
--- a/pc-bios/meson.build
+++ b/pc-bios/meson.build
@@ -23,7 +23,7 @@ if unpack_edk2_blobs
   endforeach
 endif
 
-blobs = files(
+blobs = [
   'bios.bin',
   'bios-256k.bin',
   'bios-microvm.bin',
@@ -83,11 +83,18 @@ blobs = files(
   'npcm7xx_bootrom.bin',
   'vof.bin',
   'vof-nvram.bin',
-)
+]
 
-if get_option('install_blobs')
-  install_data(blobs, install_dir: qemu_datadir)
-endif
+ln_s = [find_program('ln', required: true), '-sf']
+foreach f : blobs
+  roms += custom_target(f,
+                build_by_default: have_system,
+                output: f,
+                input: files('meson.build'),            # dummy input
+                install: get_option('install_blobs'),
+                install_dir: qemu_datadir,
+                command: [ ln_s, meson.project_source_root() / 'pc-bios' / f, '@OUTPUT@' ])
+endforeach
 
 subdir('descriptors')
 subdir('keymaps')
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 17/33] tests/tcg: correct target CPU for sparc32
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (15 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 16/33] configure, meson: move symlinking of ROMs to meson Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 18/33] tests/tcg: merge configure.sh back into main configure script Alex Bennée
                   ` (16 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

We do not want v8plus for pure sparc32, as the difference with the V8 ABI
are only meaningful on 64-bit CPUs suh as ultrasparc; supersparc is the
best CPU to use for 32-bit.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-7-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-18-alex.bennee@linaro.org>

diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
index 691d90abac..59f2403d1a 100755
--- a/tests/tcg/configure.sh
+++ b/tests/tcg/configure.sh
@@ -70,7 +70,7 @@ fi
 : ${cross_cc_riscv64="riscv64-linux-gnu-gcc"}
 : ${cross_cc_s390x="s390x-linux-gnu-gcc"}
 : ${cross_cc_sh4="sh4-linux-gnu-gcc"}
-: ${cross_cc_cflags_sparc="-m32 -mv8plus -mcpu=ultrasparc"}
+: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
 : ${cross_cc_sparc64="sparc64-linux-gnu-gcc"}
 : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
 : ${cross_cc_x86_64="x86_64-linux-gnu-gcc"}
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 18/33] tests/tcg: merge configure.sh back into main configure script
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (16 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 17/33] tests/tcg: correct target CPU for sparc32 Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 19/33] configure: add missing cross compiler fallbacks Alex Bennée
                   ` (15 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

tests/tcg/configure.sh has a complicated story.

In the beginning its code ran as part of the creation of config-target.mak
files, and that is where it placed the information on the target compiler.
However, probing for the buildability of TCG tests required multiple
inclusions of config-target.mak in the _main_ Makefile (not in
Makefile.target, which took care of building the QEMU executables in
the pre-Meson era), which polluted the namespace.

Thus, it was moved to a separate directory.  It created small config-*.mak
files in $(BUILD_DIR)/tests/tcg.  Those were also included multiple
times, but at least they were small and manageable; this was also an
important step in disentangling the TCG tests from Makefile.target.

Since then, Meson has allowed the configure script to go on a diet.
A few compilation tests survive (mostly for sanitizers) but these days
it mostly takes care of command line parsing, looking for tools, and
setting up the environment for Meson to do its stuff.

It's time to extend configure with the capability to build for more
than just one target: not just tests, but also firmware.  As a first
step, integrate all the logic to find cross compilers in the configure
script, and move tests/tcg/configure.sh back there (though as a
separate loop, not integrated in the one that generates target
configurations for Meson).

tests/tcg is actually very close to being buildable as a standalone
project, so I actually expect the compiler tests to move back to
tests/tcg, as a "configure" script of sorts which would run at Make
time after the docker images are built.  The GCC tree has a similar idea
of doing only bare-bones tree-wide configuration and leaving the rest
for Make time.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org>

diff --git a/configure b/configure
index 7b6adc29fe..f91ac632e7 100755
--- a/configure
+++ b/configure
@@ -109,6 +109,20 @@ error_exit() {
 }
 
 do_compiler() {
+  # Run the compiler, capturing its output to the log. First argument
+  # is compiler binary to execute.
+  local compiler="$1"
+  shift
+  if test -n "$BASH_VERSION"; then eval '
+      echo >>config.log "
+funcs: ${FUNCNAME[*]}
+lines: ${BASH_LINENO[*]}"
+  '; fi
+  echo $compiler "$@" >> config.log
+  $compiler "$@" >> config.log 2>&1 || return $?
+}
+
+do_compiler_werror() {
     # Run the compiler, capturing its output to the log. First argument
     # is compiler binary to execute.
     compiler="$1"
@@ -142,15 +156,15 @@ lines: ${BASH_LINENO[*]}"
 }
 
 do_cc() {
-    do_compiler "$cc" $CPU_CFLAGS "$@"
+    do_compiler_werror "$cc" $CPU_CFLAGS "$@"
 }
 
 do_cxx() {
-    do_compiler "$cxx" $CPU_CFLAGS "$@"
+    do_compiler_werror "$cxx" $CPU_CFLAGS "$@"
 }
 
 do_objc() {
-    do_compiler "$objcc" $CPU_CFLAGS "$@"
+    do_compiler_werror "$objcc" $CPU_CFLAGS "$@"
 }
 
 # Append $2 to the variable named $1, with space separation
@@ -345,11 +359,9 @@ for opt do
   ;;
   --cross-cc-cflags-*) cc_arch=${opt#--cross-cc-cflags-}; cc_arch=${cc_arch%%=*}
                       eval "cross_cc_cflags_${cc_arch}=\$optarg"
-                      cross_cc_vars="$cross_cc_vars cross_cc_cflags_${cc_arch}"
   ;;
   --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*}
                 eval "cross_cc_${cc_arch}=\$optarg"
-                cross_cc_vars="$cross_cc_vars cross_cc_${cc_arch}"
   ;;
   esac
 done
@@ -944,7 +956,6 @@ esac
 
 if eval test -z "\${cross_cc_$cpu}"; then
     eval "cross_cc_${cpu}=\$cc"
-    cross_cc_vars="$cross_cc_vars cross_cc_${cpu}"
 fi
 
 default_target_list=""
@@ -1800,6 +1811,248 @@ case "$slirp" in
     ;;
 esac
 
+##########################################
+# functions to probe cross compilers
+
+container="no"
+if test $use_containers = "yes"; then
+    if has "docker" || has "podman"; then
+        container=$($python $source_path/tests/docker/docker.py probe)
+    fi
+fi
+
+# cross compilers defaults, can be overridden with --cross-cc-ARCH
+: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"}
+: ${cross_cc_aarch64_be="$cross_cc_aarch64"}
+: ${cross_cc_cflags_aarch64_be="-mbig-endian"}
+: ${cross_cc_alpha="alpha-linux-gnu-gcc"}
+: ${cross_cc_arm="arm-linux-gnueabihf-gcc"}
+: ${cross_cc_cflags_armeb="-mbig-endian"}
+: ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"}
+: ${cross_cc_cflags_hexagon="-mv67 -O2 -static"}
+: ${cross_cc_hppa="hppa-linux-gnu-gcc"}
+: ${cross_cc_i386="i686-linux-gnu-gcc"}
+: ${cross_cc_cflags_i386="-m32"}
+: ${cross_cc_m68k="m68k-linux-gnu-gcc"}
+: ${cross_cc_microblaze="microblaze-linux-musl-gcc"}
+: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"}
+: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"}
+: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"}
+: ${cross_cc_mips="mips-linux-gnu-gcc"}
+: ${cross_cc_nios2="nios2-linux-gnu-gcc"}
+: ${cross_cc_ppc="powerpc-linux-gnu-gcc"}
+: ${cross_cc_cflags_ppc="-m32"}
+: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"}
+: ${cross_cc_cflags_ppc64="-m64 -mbig-endian"}
+: ${cross_cc_ppc64le="$cross_cc_ppc64"}
+: ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"}
+: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"}
+: ${cross_cc_s390x="s390x-linux-gnu-gcc"}
+: ${cross_cc_sh4="sh4-linux-gnu-gcc"}
+: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
+: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"}
+: ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
+: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"}
+: ${cross_cc_cflags_x86_64="-m64"}
+
+# tricore is special as it doesn't have a compiler
+: ${cross_as_tricore="tricore-as"}
+: ${cross_ld_tricore="tricore-ld"}
+
+probe_target_compiler() {
+  # reset all output variables
+  container_image=
+  container_hosts=
+  container_cross_cc=
+  container_cross_as=
+  container_cross_ld=
+  target_cc=
+  target_as=
+  target_ld=
+
+  case $1 in
+    aarch64) container_hosts="x86_64 aarch64" ;;
+    alpha) container_hosts=x86_64 ;;
+    arm) container_hosts="x86_64 aarch64" ;;
+    cris) container_hosts=x86_64 ;;
+    hexagon) container_hosts=x86_64 ;;
+    hppa) container_hosts=x86_64 ;;
+    i386) container_hosts=x86_64 ;;
+    m68k) container_hosts=x86_64 ;;
+    microblaze) container_hosts=x86_64 ;;
+    mips64el) container_hosts=x86_64 ;;
+    mips64) container_hosts=x86_64 ;;
+    mipsel) container_hosts=x86_64 ;;
+    mips) container_hosts=x86_64 ;;
+    nios2) container_hosts=x86_64 ;;
+    ppc) container_hosts=x86_64 ;;
+    ppc64|ppc64le) container_hosts=x86_64 ;;
+    riscv64) container_hosts=x86_64 ;;
+    s390x) container_hosts=x86_64 ;;
+    sh4) container_hosts=x86_64 ;;
+    sparc64) container_hosts=x86_64 ;;
+    tricore) container_hosts=x86_64 ;;
+    x86_64) container_hosts="aarch64 ppc64el x86_64" ;;
+    xtensa*) container_hosts=x86_64 ;;
+  esac
+
+  for host in $container_hosts; do
+    test "$container" != no || continue
+    test "$host" = "$cpu" || continue
+    case $1 in
+      aarch64)
+        # We don't have any bigendian build tools so we only use this for AArch64
+        container_image=debian-arm64-cross
+        container_cross_cc=aarch64-linux-gnu-gcc-10
+        ;;
+      alpha)
+        container_image=debian-alpha-cross
+        container_cross_cc=alpha-linux-gnu-gcc
+        ;;
+      arm)
+        # We don't have any bigendian build tools so we only use this for ARM
+        container_image=debian-armhf-cross
+        container_cross_cc=arm-linux-gnueabihf-gcc
+        ;;
+      cris)
+        container_image=fedora-cris-cross
+        container_cross_cc=cris-linux-gnu-gcc
+        ;;
+      hexagon)
+        container_image=debian-hexagon-cross
+        container_cross_cc=hexagon-unknown-linux-musl-clang
+        ;;
+      hppa)
+        container_image=debian-hppa-cross
+        container_cross_cc=hppa-linux-gnu-gcc
+        ;;
+      i386)
+        container_image=fedora-i386-cross
+        container_cross_cc=gcc
+        ;;
+      m68k)
+        container_image=debian-m68k-cross
+        container_cross_cc=m68k-linux-gnu-gcc
+        ;;
+      microblaze)
+        container_image=debian-microblaze-cross
+        container_cross_cc=microblaze-linux-musl-gcc
+        ;;
+      mips64el)
+        container_image=debian-mips64el-cross
+        container_cross_cc=mips64el-linux-gnuabi64-gcc
+        ;;
+      mips64)
+        container_image=debian-mips64-cross
+        container_cross_cc=mips64-linux-gnuabi64-gcc
+        ;;
+      mipsel)
+        container_image=debian-mipsel-cross
+        container_cross_cc=mipsel-linux-gnu-gcc
+        ;;
+      mips)
+        container_image=debian-mips-cross
+        container_cross_cc=mips-linux-gnu-gcc
+        ;;
+      nios2)
+        container_image=debian-nios2-cross
+        container_cross_cc=nios2-linux-gnu-gcc
+        ;;
+      ppc)
+        container_image=debian-powerpc-test-cross
+        container_cross_cc=powerpc-linux-gnu-gcc-10
+        ;;
+      ppc64|ppc64le)
+        container_image=debian-powerpc-test-cross
+        container_cross_cc=powerpc${1#ppc}-linux-gnu-gcc-10
+        ;;
+      riscv64)
+        container_image=debian-riscv64-test-cross
+        container_cross_cc=riscv64-linux-gnu-gcc
+        ;;
+      s390x)
+        container_image=debian-s390x-cross
+        container_cross_cc=s390x-linux-gnu-gcc
+        ;;
+      sh4)
+        container_image=debian-sh4-cross
+        container_cross_cc=sh4-linux-gnu-gcc
+        ;;
+      sparc64)
+        container_image=debian-sparc64-cross
+        container_cross_cc=sparc64-linux-gnu-gcc
+        ;;
+      tricore)
+        container_image=debian-tricore-cross
+        container_cross_as=tricore-as
+        container_cross_ld=tricore-ld
+        ;;
+      x86_64)
+        container_image=debian-amd64-cross
+        container_cross_cc=x86_64-linux-gnu-gcc
+        ;;
+      xtensa*)
+        # FIXME: xtensa-linux-user?
+        container_hosts=x86_64
+        container_image=debian-xtensa-cross
+
+        # default to the dc232b cpu
+        container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc
+        ;;
+    esac
+  done
+
+  eval "target_cflags=\${cross_cc_cflags_$1}"
+  if eval test -n "\"\${cross_cc_$1}\""; then
+    if eval has "\"\${cross_cc_$1}\""; then
+      eval "target_cc=\"\${cross_cc_$1}\""
+      case $1 in
+        i386|x86_64)
+          if $target_cc --version | grep -qi "clang"; then
+            unset target_cc
+          fi
+          ;;
+      esac
+    fi
+  fi
+  if eval test -n "\"\${cross_as_$1}\""; then
+    if eval has "\"\${cross_as_$1}\""; then
+      eval "target_as=\"\${cross_as_$1}\""
+    fi
+  fi
+  if eval test -n "\"\${cross_ld_$1}\""; then
+    if eval has "\"\${cross_ld_$1}\""; then
+      eval "target_ld=\"\${cross_ld_$1}\""
+    fi
+  fi
+}
+
+write_target_makefile() {
+  if test -n "$target_cc"; then
+    echo "CC=$target_cc"
+  fi
+  if test -n "$target_as"; then
+    echo "AS=$target_as"
+  fi
+  if test -n "$target_ld"; then
+    echo "LD=$target_ld"
+  fi
+}
+
+write_container_target_makefile() {
+  if test -n "$container_cross_cc"; then
+    echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
+  fi
+  if test -n "$container_cross_as"; then
+    echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --"
+  fi
+  if test -n "$container_cross_ld"; then
+    echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --"
+  fi
+}
+
+
+
 ##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
@@ -2107,11 +2360,136 @@ for f in $LINKS ; do
     fi
 done
 
-(for i in $cross_cc_vars; do
-  export $i
+# tests/tcg configuration
+(makefile=tests/tcg/Makefile.prereqs
+echo "# Automatically generated by configure - do not modify" > $makefile
+
+config_host_mak=tests/tcg/config-host.mak
+echo "# Automatically generated by configure - do not modify" > $config_host_mak
+echo "SRC_PATH=$source_path" >> $config_host_mak
+echo "HOST_CC=$host_cc" >> $config_host_mak
+
+tcg_tests_targets=
+for target in $target_list; do
+  arch=${target%%-*}
+
+  probe_target_compiler ${arch}
+  config_target_mak=tests/tcg/config-$target.mak
+
+  echo "# Automatically generated by configure - do not modify" > $config_target_mak
+  echo "TARGET_NAME=$arch" >> $config_target_mak
+  case $target in
+    *-softmmu)
+      test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue
+      qemu="qemu-system-$arch"
+      ;;
+    *-linux-user|*-bsd-user)
+      qemu="qemu-$arch"
+      ;;
+  esac
+
+  got_cross_cc=no
+  unset build_static
+
+  if test -n "$target_cc"; then
+      write_c_skeleton
+      if ! do_compiler "$target_cc" $target_cflags \
+           -o $TMPE $TMPC -static ; then
+          # For host systems we might get away with building without -static
+          if do_compiler "$target_cc" $target_cflags \
+                         -o $TMPE $TMPC ; then
+              got_cross_cc=yes
+          fi
+      else
+          got_cross_cc=yes
+          build_static=y
+      fi
+  elif test -n "$target_as" && test -n "$target_ld"; then
+      # Special handling for assembler only tests
+      case $target in
+          tricore-softmmu) got_cross_cc=yes ;;
+      esac
+  fi
+
+  if test $got_cross_cc = yes; then
+      # Test for compiler features for optional tests. We only do this
+      # for cross compilers because ensuring the docker containers based
+      # compilers is a requirememt for adding a new test that needs a
+      # compiler feature.
+
+      echo "BUILD_STATIC=$build_static" >> $config_target_mak
+      write_target_makefile >> $config_target_mak
+      case $target in
+          aarch64-*)
+              if do_compiler "$target_cc" $target_cflags \
+                             -march=armv8.1-a+sve -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak
+              fi
+              if do_compiler "$target_cc" $target_cflags \
+                             -march=armv8.1-a+sve2 -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak
+              fi
+              if do_compiler "$target_cc" $target_cflags \
+                             -march=armv8.3-a -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak
+              fi
+              if do_compiler "$target_cc" $target_cflags \
+                             -mbranch-protection=standard -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak
+              fi
+              if do_compiler "$target_cc" $target_cflags \
+                             -march=armv8.5-a+memtag -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak
+              fi
+              ;;
+          ppc*)
+              if do_compiler "$target_cc" $target_cflags \
+                             -mpower8-vector -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak
+              fi
+              if do_compiler "$target_cc" $target_cflags \
+                             -mpower10 -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak
+              fi
+              ;;
+          i386-linux-user)
+              if do_compiler "$target_cc" $target_cflags \
+                             -Werror -fno-pie -o $TMPE $TMPC; then
+                  echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak
+              fi
+              ;;
+      esac
+  elif test -n "$container_image"; then
+      echo "build-tcg-tests-$target: docker-image-$container_image" >> $makefile
+      echo "BUILD_STATIC=y" >> $config_target_mak
+      write_container_target_makefile >> $config_target_mak
+      case $target in
+          aarch64-*)
+              echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak
+              echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak
+              echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak
+              echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak
+              echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak
+              ;;
+          ppc*)
+              echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak
+              echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak
+              ;;
+          i386-linux-user)
+              echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak
+              ;;
+      esac
+      got_cross_cc=yes
+  fi
+  if test $got_cross_cc = yes; then
+      mkdir -p tests/tcg/$target
+      echo "QEMU=$PWD/$qemu" >> $config_target_mak
+      echo "EXTRA_CFLAGS=$target_cflags" >> $config_target_mak
+      echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> $makefile
+      tcg_tests_targets="$tcg_tests_targets $target"
+  fi
 done
-export target_list source_path use_containers cpu host_cc
-$source_path/tests/tcg/configure.sh)
+echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile)
 
 config_mak=pc-bios/optionrom/config.mak
 echo "# Automatically generated by configure - do not modify" > $config_mak
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 72ce0561f4..6a1688e33e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -37,7 +37,6 @@ export SRC_PATH
 SPEED = quick
 
 -include tests/tcg/Makefile.prereqs
-config-host.mak: $(SRC_PATH)/tests/tcg/configure.sh
 tests/tcg/Makefile.prereqs: config-host.mak
 
 # Per guest TCG tests
diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
deleted file mode 100755
index 59f2403d1a..0000000000
--- a/tests/tcg/configure.sh
+++ /dev/null
@@ -1,376 +0,0 @@
-#! /bin/sh
-
-if test -z "$source_path"; then
-  echo Do not invoke this script directly.  It is called
-  echo automatically by configure.
-  exit 1
-fi
-
-write_c_skeleton() {
-    cat > $TMPC <<EOF
-int main(void) { return 0; }
-EOF
-}
-
-has() {
-  command -v "$1" >/dev/null 2>&1
-}
-
-do_compiler() {
-  # Run the compiler, capturing its output to the log. First argument
-  # is compiler binary to execute.
-  local compiler="$1"
-  shift
-  if test -n "$BASH_VERSION"; then eval '
-      echo >>config.log "
-funcs: ${FUNCNAME[*]}
-lines: ${BASH_LINENO[*]}"
-  '; fi
-  echo $compiler "$@" >> config.log
-  $compiler "$@" >> config.log 2>&1 || return $?
-}
-
-
-TMPDIR1="config-temp"
-TMPC="${TMPDIR1}/qemu-conf.c"
-TMPE="${TMPDIR1}/qemu-conf.exe"
-
-container="no"
-if test $use_containers = "yes"; then
-    if has "docker" || has "podman"; then
-        container=$($python $source_path/tests/docker/docker.py probe)
-    fi
-fi
-
-# cross compilers defaults, can be overridden with --cross-cc-ARCH
-: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"}
-: ${cross_cc_aarch64_be="$cross_cc_aarch64"}
-: ${cross_cc_cflags_aarch64_be="-mbig-endian"}
-: ${cross_cc_alpha="alpha-linux-gnu-gcc"}
-: ${cross_cc_arm="arm-linux-gnueabihf-gcc"}
-: ${cross_cc_cflags_armeb="-mbig-endian"}
-: ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"}
-: ${cross_cc_cflags_hexagon="-mv67 -O2 -static"}
-: ${cross_cc_hppa="hppa-linux-gnu-gcc"}
-: ${cross_cc_i386="i686-linux-gnu-gcc"}
-: ${cross_cc_cflags_i386="-m32"}
-: ${cross_cc_m68k="m68k-linux-gnu-gcc"}
-: ${cross_cc_microblaze="microblaze-linux-musl-gcc"}
-: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"}
-: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"}
-: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"}
-: ${cross_cc_mips="mips-linux-gnu-gcc"}
-: ${cross_cc_nios2="nios2-linux-gnu-gcc"}
-: ${cross_cc_ppc="powerpc-linux-gnu-gcc"}
-: ${cross_cc_cflags_ppc="-m32"}
-: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"}
-: ${cross_cc_cflags_ppc64="-m64 -mbig-endian"}
-: ${cross_cc_ppc64le="$cross_cc_ppc64"}
-: ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"}
-: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"}
-: ${cross_cc_s390x="s390x-linux-gnu-gcc"}
-: ${cross_cc_sh4="sh4-linux-gnu-gcc"}
-: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
-: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"}
-: ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
-: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"}
-: ${cross_cc_cflags_x86_64="-m64"}
-
-# tricore is special as it doesn't have a compiler
-: ${cross_as_tricore="tricore-as"}
-: ${cross_ld_tricore="tricore-ld"}
-
-makefile=tests/tcg/Makefile.prereqs
-echo "# Automatically generated by configure - do not modify" > $makefile
-
-config_host_mak=tests/tcg/config-host.mak
-echo "# Automatically generated by configure - do not modify" > $config_host_mak
-echo "SRC_PATH=$source_path" >> $config_host_mak
-echo "HOST_CC=$host_cc" >> $config_host_mak
-
-tcg_tests_targets=
-for target in $target_list; do
-  arch=${target%%-*}
-
-  # reset all container fields
-  container_image=
-  container_hosts=
-  container_cross_cc=
-  container_cross_as=
-  container_cross_ld=
-
-  # suppress clang
-  supress_clang=
-
-  case $target in
-    aarch64-*)
-      # We don't have any bigendian build tools so we only use this for AArch64
-      container_hosts="x86_64 aarch64"
-      container_image=debian-arm64-cross
-      container_cross_cc=aarch64-linux-gnu-gcc-10
-      ;;
-    alpha-*)
-      container_hosts=x86_64
-      container_image=debian-alpha-cross
-      container_cross_cc=alpha-linux-gnu-gcc
-      ;;
-    arm-*)
-      # We don't have any bigendian build tools so we only use this for ARM
-      container_hosts="x86_64 aarch64"
-      container_image=debian-armhf-cross
-      container_cross_cc=arm-linux-gnueabihf-gcc
-      ;;
-    cris-*)
-      container_hosts=x86_64
-      container_image=fedora-cris-cross
-      container_cross_cc=cris-linux-gnu-gcc
-      ;;
-    hexagon-*)
-      container_hosts=x86_64
-      container_image=debian-hexagon-cross
-      container_cross_cc=hexagon-unknown-linux-musl-clang
-      ;;
-    hppa-*)
-      container_hosts=x86_64
-      container_image=debian-hppa-cross
-      container_cross_cc=hppa-linux-gnu-gcc
-      ;;
-    i386-*)
-      container_hosts=x86_64
-      container_image=fedora-i386-cross
-      container_cross_cc=gcc
-      supress_clang=yes
-      ;;
-    m68k-*)
-      container_hosts=x86_64
-      container_image=debian-m68k-cross
-      container_cross_cc=m68k-linux-gnu-gcc
-      ;;
-    microblaze-*)
-      container_hosts=x86_64
-      container_image=debian-microblaze-cross
-      container_cross_cc=microblaze-linux-musl-gcc
-      ;;
-    mips64el-*)
-      container_hosts=x86_64
-      container_image=debian-mips64el-cross
-      container_cross_cc=mips64el-linux-gnuabi64-gcc
-      ;;
-    mips64-*)
-      container_hosts=x86_64
-      container_image=debian-mips64-cross
-      container_cross_cc=mips64-linux-gnuabi64-gcc
-      ;;
-    mipsel-*)
-      container_hosts=x86_64
-      container_image=debian-mipsel-cross
-      container_cross_cc=mipsel-linux-gnu-gcc
-      ;;
-    mips-*)
-      container_hosts=x86_64
-      container_image=debian-mips-cross
-      container_cross_cc=mips-linux-gnu-gcc
-      ;;
-    nios2-*)
-      container_hosts=x86_64
-      container_image=debian-nios2-cross
-      container_cross_cc=nios2-linux-gnu-gcc
-      ;;
-    ppc-*)
-      container_hosts=x86_64
-      container_image=debian-powerpc-test-cross
-      container_cross_cc=powerpc-linux-gnu-gcc-10
-      ;;
-    ppc64-*|ppc64le-*)
-      container_hosts=x86_64
-      container_image=debian-powerpc-test-cross
-      container_cross_cc=${target%%-*}-linux-gnu-gcc-10
-      container_cross_cc=powerpc${container_cross_cc#ppc}
-      ;;
-    riscv64-*)
-      container_hosts=x86_64
-      container_image=debian-riscv64-test-cross
-      container_cross_cc=riscv64-linux-gnu-gcc
-      ;;
-    s390x-*)
-      container_hosts=x86_64
-      container_image=debian-s390x-cross
-      container_cross_cc=s390x-linux-gnu-gcc
-      ;;
-    sh4-*)
-      container_hosts=x86_64
-      container_image=debian-sh4-cross
-      container_cross_cc=sh4-linux-gnu-gcc
-      ;;
-    sparc64-*)
-      container_hosts=x86_64
-      container_image=debian-sparc64-cross
-      container_cross_cc=sparc64-linux-gnu-gcc
-      ;;
-    tricore-softmmu)
-      container_hosts=x86_64
-      container_image=debian-tricore-cross
-      container_cross_as=tricore-as
-      container_cross_ld=tricore-ld
-      ;;
-    x86_64-*)
-      container_hosts="aarch64 ppc64el x86_64"
-      container_image=debian-amd64-cross
-      container_cross_cc=x86_64-linux-gnu-gcc
-      supress_clang=yes
-      ;;
-    xtensa*-softmmu)
-      container_hosts=x86_64
-      container_image=debian-xtensa-cross
-
-      # default to the dc232b cpu
-      container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc
-      ;;
-  esac
-
-  config_target_mak=tests/tcg/config-$target.mak
-
-  echo "# Automatically generated by configure - do not modify" > $config_target_mak
-  echo "TARGET_NAME=$arch" >> $config_target_mak
-  case $target in
-    *-softmmu)
-      test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue
-      qemu="qemu-system-$arch"
-      ;;
-    *-linux-user|*-bsd-user)
-      qemu="qemu-$arch"
-      ;;
-  esac
-
-  eval "target_compiler_cflags=\${cross_cc_cflags_$arch}"
-
-  got_cross_cc=no
-
-  if eval test "x\"\${cross_cc_$arch}\"" != xyes; then
-      eval "target_compiler=\"\${cross_cc_$arch}\""
-
-      if has $target_compiler; then
-          if test "$supress_clang" = yes &&
-                  $target_compiler --version | grep -qi "clang"; then
-              got_cross_cc=no
-          else
-              write_c_skeleton
-              if ! do_compiler "$target_compiler" $target_compiler_cflags \
-                   -o $TMPE $TMPC -static ; then
-                  # For host systems we might get away with building without -static
-                  if do_compiler "$target_compiler" $target_compiler_cflags \
-                                 -o $TMPE $TMPC ; then
-                      got_cross_cc=yes
-                      echo "CC=$target_compiler" >> $config_target_mak
-                  fi
-              else
-                  got_cross_cc=yes
-                  echo "BUILD_STATIC=y" >> $config_target_mak
-                  echo "CC=$target_compiler" >> $config_target_mak
-              fi
-          fi
-      fi
-
-      # Special handling for assembler only tests
-      eval "target_as=\"\${cross_as_$arch}\""
-      eval "target_ld=\"\${cross_ld_$arch}\""
-      if has $target_as && has $target_ld; then
-          case $target in
-              tricore-softmmu)
-                  echo "AS=$target_as" >> $config_target_mak
-                  echo "LD=$target_ld" >> $config_target_mak
-                  got_cross_cc=yes
-                  ;;
-          esac
-      fi
-  fi
-
-  if test $got_cross_cc = yes; then
-      # Test for compiler features for optional tests. We only do this
-      # for cross compilers because ensuring the docker containers based
-      # compilers is a requirememt for adding a new test that needs a
-      # compiler feature.
-
-      case $target in
-          aarch64-*)
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -march=armv8.1-a+sve -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak
-              fi
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -march=armv8.1-a+sve2 -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak
-              fi
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -march=armv8.3-a -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak
-              fi
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -mbranch-protection=standard -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak
-              fi
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -march=armv8.5-a+memtag -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak
-              fi
-              ;;
-          ppc*)
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -mpower8-vector -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak
-              fi
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -mpower10 -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak
-              fi
-              ;;
-          i386-linux-user)
-              if do_compiler "$target_compiler" $target_compiler_cflags \
-                             -Werror -fno-pie -o $TMPE $TMPC; then
-                  echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak
-              fi
-              ;;
-      esac
-  elif test $got_cross_cc = no && test "$container" != no && \
-          test -n "$container_image"; then
-      for host in $container_hosts; do
-          if test "$host" = "$cpu"; then
-              echo "build-tcg-tests-$target: docker-image-$container_image" >> $makefile
-              echo "BUILD_STATIC=y" >> $config_target_mak
-              echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --" >> $config_target_mak
-              if test -n "$container_cross_as"; then
-                  echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --" >> $config_target_mak
-              fi
-              if test -n "$container_cross_ld"; then
-                  echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --" >> $config_target_mak
-              fi
-              case $target in
-                  aarch64-*)
-                      echo "CROSS_CC_HAS_SVE=y" >> $config_target_mak
-                      echo "CROSS_CC_HAS_SVE2=y" >> $config_target_mak
-                      echo "CROSS_CC_HAS_ARMV8_3=y" >> $config_target_mak
-                      echo "CROSS_CC_HAS_ARMV8_BTI=y" >> $config_target_mak
-                      echo "CROSS_CC_HAS_ARMV8_MTE=y" >> $config_target_mak
-                      ;;
-                  ppc*)
-                      echo "CROSS_CC_HAS_POWER8_VECTOR=y" >> $config_target_mak
-                      echo "CROSS_CC_HAS_POWER10=y" >> $config_target_mak
-                      ;;
-                  i386-linux-user)
-                      echo "CROSS_CC_HAS_I386_NOPIE=y" >> $config_target_mak
-                      ;;
-              esac
-              got_cross_cc=yes
-              break
-          fi
-      done
-  fi
-  if test $got_cross_cc = yes; then
-      mkdir -p tests/tcg/$target
-      echo "QEMU=$PWD/$qemu" >> $config_target_mak
-      echo "EXTRA_CFLAGS=$target_compiler_cflags" >> $config_target_mak
-      echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> $makefile
-      tcg_tests_targets="$tcg_tests_targets $target"
-  fi
-done
-echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 19/33] configure: add missing cross compiler fallbacks
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (17 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 18/33] tests/tcg: merge configure.sh back into main configure script Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 20/33] configure: handle host compiler in probe_target_compiler Alex Bennée
                   ` (14 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

The arm compiler can be used for armeb, and the sparc64 compiler
can be used for sparc.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-9-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-20-alex.bennee@linaro.org>

diff --git a/configure b/configure
index f91ac632e7..fbf6d39f96 100755
--- a/configure
+++ b/configure
@@ -1827,6 +1827,7 @@ fi
 : ${cross_cc_cflags_aarch64_be="-mbig-endian"}
 : ${cross_cc_alpha="alpha-linux-gnu-gcc"}
 : ${cross_cc_arm="arm-linux-gnueabihf-gcc"}
+: ${cross_cc_armeb="$cross_cc_arm"}
 : ${cross_cc_cflags_armeb="-mbig-endian"}
 : ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"}
 : ${cross_cc_cflags_hexagon="-mv67 -O2 -static"}
@@ -1849,9 +1850,10 @@ fi
 : ${cross_cc_riscv64="riscv64-linux-gnu-gcc"}
 : ${cross_cc_s390x="s390x-linux-gnu-gcc"}
 : ${cross_cc_sh4="sh4-linux-gnu-gcc"}
-: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
 : ${cross_cc_sparc64="sparc64-linux-gnu-gcc"}
 : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
+: ${cross_cc_sparc="$cross_cc_sparc64"}
+: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
 : ${cross_cc_x86_64="x86_64-linux-gnu-gcc"}
 : ${cross_cc_cflags_x86_64="-m64"}
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (18 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 19/33] configure: add missing cross compiler fallbacks Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-15 11:57   ` Matheus Kowalczuk Ferst
  2022-06-01 18:05 ` [PULL 21/33] configure: introduce --cross-prefix-*= Alex Bennée
                   ` (13 subsequent siblings)
  33 siblings, 1 reply; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

In preparation for handling more binaries than just cc, handle
the case of "probe_target_compiler $cpu" directly in the function,
setting the target_* variables based on the ones that are used to
build QEMU.  The clang check also needs to be moved after this
fallback.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>

diff --git a/configure b/configure
index fbf6d39f96..217c8b3cac 100755
--- a/configure
+++ b/configure
@@ -954,10 +954,6 @@ case $git_submodules_action in
     ;;
 esac
 
-if eval test -z "\${cross_cc_$cpu}"; then
-    eval "cross_cc_${cpu}=\$cc"
-fi
-
 default_target_list=""
 mak_wilds=""
 
@@ -2008,13 +2004,6 @@ probe_target_compiler() {
   if eval test -n "\"\${cross_cc_$1}\""; then
     if eval has "\"\${cross_cc_$1}\""; then
       eval "target_cc=\"\${cross_cc_$1}\""
-      case $1 in
-        i386|x86_64)
-          if $target_cc --version | grep -qi "clang"; then
-            unset target_cc
-          fi
-          ;;
-      esac
     fi
   fi
   if eval test -n "\"\${cross_as_$1}\""; then
@@ -2027,6 +2016,20 @@ probe_target_compiler() {
       eval "target_ld=\"\${cross_ld_$1}\""
     fi
   fi
+  if test "$1" = $cpu; then
+    : ${target_cc:=$cc}
+    : ${target_as:=$as}
+    : ${target_ld:=$ld}
+  fi
+  if test -n "$target_cc"; then
+    case $1 in
+      i386|x86_64)
+        if $target_cc --version | grep -qi "clang"; then
+          unset target_cc
+        fi
+        ;;
+    esac
+  fi
 }
 
 write_target_makefile() {
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 21/33] configure: introduce --cross-prefix-*=
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (19 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 20/33] configure: handle host compiler in probe_target_compiler Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 22/33] configure: include more binutils in tests/tcg makefile Alex Bennée
                   ` (12 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

Also in preparation for handling more binaries from the cross binutils,
support an option --cross-prefix-ARCH.  All cross_cc_* defaults are
replaced with cross_prefix_*; the cross_cc_* fallbacks are extended
to the cross-compilation prefix, but the compiler fallbacks remain
as well.  This way, for example, --cross-cc-arm=arm-linux-gnueabihf-clang
also applies to armeb binaries.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-11-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-22-alex.bennee@linaro.org>

diff --git a/configure b/configure
index 217c8b3cac..f55ae82a5d 100755
--- a/configure
+++ b/configure
@@ -363,6 +363,11 @@ for opt do
   --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*}
                 eval "cross_cc_${cc_arch}=\$optarg"
   ;;
+  --cross-prefix-*[!a-zA-Z0-9_-]*=*) error_exit "Passed bad --cross-prefix-FOO option"
+  ;;
+  --cross-prefix-*) cc_arch=${opt#--cross-prefix-}; cc_arch=${cc_arch%%=*}
+                    eval "cross_prefix_${cc_arch}=\$optarg"
+  ;;
   esac
 done
 # OS specific
@@ -728,6 +733,8 @@ for opt do
   ;;
   --cross-cc-*)
   ;;
+  --cross-prefix-*)
+  ;;
   --enable-debug-info) meson_option_add -Ddebug=true
   ;;
   --disable-debug-info) meson_option_add -Ddebug=false
@@ -1016,6 +1023,7 @@ Advanced options (experts only):
   --extra-ldflags=LDFLAGS  append extra linker flags LDFLAGS
   --cross-cc-ARCH=CC       use compiler when building ARCH guest test cases
   --cross-cc-cflags-ARCH=  use compiler flags when building ARCH guest tests
+  --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases
   --make=MAKE              use specified make [$make]
   --python=PYTHON          use specified python [$python]
   --meson=MESON            use specified meson [$meson]
@@ -1818,44 +1826,54 @@ if test $use_containers = "yes"; then
 fi
 
 # cross compilers defaults, can be overridden with --cross-cc-ARCH
-: ${cross_cc_aarch64="aarch64-linux-gnu-gcc"}
+: ${cross_prefix_aarch64="aarch64-linux-gnu-"}
+: ${cross_prefix_aarch64_be="$cross_prefix_aarch64"}
+: ${cross_prefix_alpha="alpha-linux-gnu-"}
+: ${cross_prefix_arm="arm-linux-gnueabihf-"}
+: ${cross_prefix_armeb="$cross_prefix_arm"}
+: ${cross_prefix_hexagon="hexagon-unknown-linux-musl-"}
+: ${cross_prefix_hppa="hppa-linux-gnu-"}
+: ${cross_prefix_i386="i686-linux-gnu-"}
+: ${cross_prefix_m68k="m68k-linux-gnu-"}
+: ${cross_prefix_microblaze="microblaze-linux-musl-"}
+: ${cross_prefix_mips64el="mips64el-linux-gnuabi64-"}
+: ${cross_prefix_mips64="mips64-linux-gnuabi64-"}
+: ${cross_prefix_mipsel="mipsel-linux-gnu-"}
+: ${cross_prefix_mips="mips-linux-gnu-"}
+: ${cross_prefix_nios2="nios2-linux-gnu-"}
+: ${cross_prefix_ppc="powerpc-linux-gnu-"}
+: ${cross_prefix_ppc64="powerpc64-linux-gnu-"}
+: ${cross_prefix_ppc64le="$cross_prefix_ppc64"}
+: ${cross_prefix_riscv64="riscv64-linux-gnu-"}
+: ${cross_prefix_s390x="s390x-linux-gnu-"}
+: ${cross_prefix_sh4="sh4-linux-gnu-"}
+: ${cross_prefix_sparc64="sparc64-linux-gnu-"}
+: ${cross_prefix_sparc="$cross_prefix_sparc64"}
+: ${cross_prefix_x86_64="x86_64-linux-gnu-"}
+
 : ${cross_cc_aarch64_be="$cross_cc_aarch64"}
 : ${cross_cc_cflags_aarch64_be="-mbig-endian"}
-: ${cross_cc_alpha="alpha-linux-gnu-gcc"}
-: ${cross_cc_arm="arm-linux-gnueabihf-gcc"}
 : ${cross_cc_armeb="$cross_cc_arm"}
 : ${cross_cc_cflags_armeb="-mbig-endian"}
 : ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"}
 : ${cross_cc_cflags_hexagon="-mv67 -O2 -static"}
-: ${cross_cc_hppa="hppa-linux-gnu-gcc"}
-: ${cross_cc_i386="i686-linux-gnu-gcc"}
 : ${cross_cc_cflags_i386="-m32"}
-: ${cross_cc_m68k="m68k-linux-gnu-gcc"}
-: ${cross_cc_microblaze="microblaze-linux-musl-gcc"}
-: ${cross_cc_mips64el="mips64el-linux-gnuabi64-gcc"}
-: ${cross_cc_mips64="mips64-linux-gnuabi64-gcc"}
-: ${cross_cc_mipsel="mipsel-linux-gnu-gcc"}
-: ${cross_cc_mips="mips-linux-gnu-gcc"}
-: ${cross_cc_nios2="nios2-linux-gnu-gcc"}
-: ${cross_cc_ppc="powerpc-linux-gnu-gcc"}
 : ${cross_cc_cflags_ppc="-m32"}
-: ${cross_cc_ppc64="powerpc64-linux-gnu-gcc"}
 : ${cross_cc_cflags_ppc64="-m64 -mbig-endian"}
 : ${cross_cc_ppc64le="$cross_cc_ppc64"}
 : ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"}
-: ${cross_cc_riscv64="riscv64-linux-gnu-gcc"}
-: ${cross_cc_s390x="s390x-linux-gnu-gcc"}
-: ${cross_cc_sh4="sh4-linux-gnu-gcc"}
-: ${cross_cc_sparc64="sparc64-linux-gnu-gcc"}
 : ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
 : ${cross_cc_sparc="$cross_cc_sparc64"}
 : ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
-: ${cross_cc_x86_64="x86_64-linux-gnu-gcc"}
 : ${cross_cc_cflags_x86_64="-m64"}
 
-# tricore is special as it doesn't have a compiler
-: ${cross_as_tricore="tricore-as"}
-: ${cross_ld_tricore="tricore-ld"}
+compute_target_variable() {
+  if eval test -n "\"\${cross_prefix_$1}\""; then
+    if eval has "\"\${cross_prefix_$1}\$3\""; then
+      eval "$2=\"\${cross_prefix_$1}\$3\""
+    fi
+  fi
+}
 
 probe_target_compiler() {
   # reset all output variables
@@ -1901,93 +1919,99 @@ probe_target_compiler() {
       aarch64)
         # We don't have any bigendian build tools so we only use this for AArch64
         container_image=debian-arm64-cross
-        container_cross_cc=aarch64-linux-gnu-gcc-10
+        container_cross_prefix=aarch64-linux-gnu-
+        container_cross_cc=${container_cross_prefix}gcc-10
         ;;
       alpha)
         container_image=debian-alpha-cross
-        container_cross_cc=alpha-linux-gnu-gcc
+        container_cross_prefix=alpha-linux-gnu-
         ;;
       arm)
         # We don't have any bigendian build tools so we only use this for ARM
         container_image=debian-armhf-cross
-        container_cross_cc=arm-linux-gnueabihf-gcc
+        container_cross_prefix=arm-linux-gnueabihf-
         ;;
       cris)
         container_image=fedora-cris-cross
-        container_cross_cc=cris-linux-gnu-gcc
+        container_cross_prefix=cris-linux-gnu-
         ;;
       hexagon)
         container_image=debian-hexagon-cross
-        container_cross_cc=hexagon-unknown-linux-musl-clang
+        container_cross_prefix=hexagon-unknown-linux-musl-
+        container_cross_cc=${container_cross_prefix}clang
         ;;
       hppa)
         container_image=debian-hppa-cross
-        container_cross_cc=hppa-linux-gnu-gcc
+        container_cross_prefix=hppa-linux-gnu-
         ;;
       i386)
         container_image=fedora-i386-cross
-        container_cross_cc=gcc
+        container_cross_prefix=
         ;;
       m68k)
         container_image=debian-m68k-cross
-        container_cross_cc=m68k-linux-gnu-gcc
+        container_cross_prefix=m68k-linux-gnu-
         ;;
       microblaze)
         container_image=debian-microblaze-cross
-        container_cross_cc=microblaze-linux-musl-gcc
+        container_cross_prefix=microblaze-linux-musl-
         ;;
       mips64el)
         container_image=debian-mips64el-cross
-        container_cross_cc=mips64el-linux-gnuabi64-gcc
+        container_cross_prefix=mips64el-linux-gnuabi64-
         ;;
       mips64)
         container_image=debian-mips64-cross
-        container_cross_cc=mips64-linux-gnuabi64-gcc
+        container_cross_prefix=mips64-linux-gnuabi64-
         ;;
       mipsel)
         container_image=debian-mipsel-cross
-        container_cross_cc=mipsel-linux-gnu-gcc
+        container_cross_prefix=mipsel-linux-gnu-
         ;;
       mips)
         container_image=debian-mips-cross
-        container_cross_cc=mips-linux-gnu-gcc
+        container_cross_prefix=mips-linux-gnu-
         ;;
       nios2)
         container_image=debian-nios2-cross
-        container_cross_cc=nios2-linux-gnu-gcc
+        container_cross_prefix=nios2-linux-gnu-
         ;;
       ppc)
         container_image=debian-powerpc-test-cross
-        container_cross_cc=powerpc-linux-gnu-gcc-10
+        container_cross_prefix=powerpc-linux-gnu-
+        container_cross_cc=${container_cross_prefix}gcc-10
         ;;
       ppc64|ppc64le)
         container_image=debian-powerpc-test-cross
-        container_cross_cc=powerpc${1#ppc}-linux-gnu-gcc-10
+        container_cross_prefix=powerpc${1#ppc}-linux-gnu-
+        container_cross_cc=${container_cross_prefix}gcc-10
         ;;
       riscv64)
         container_image=debian-riscv64-test-cross
-        container_cross_cc=riscv64-linux-gnu-gcc
+        container_cross_prefix=riscv64-linux-gnu-
         ;;
       s390x)
         container_image=debian-s390x-cross
-        container_cross_cc=s390x-linux-gnu-gcc
+        container_cross_prefix=s390x-linux-gnu-
         ;;
       sh4)
         container_image=debian-sh4-cross
-        container_cross_cc=sh4-linux-gnu-gcc
+        container_cross_prefix=sh4-linux-gnu-
         ;;
       sparc64)
         container_image=debian-sparc64-cross
-        container_cross_cc=sparc64-linux-gnu-gcc
+        container_cross_prefix=sparc64-linux-gnu-
         ;;
       tricore)
         container_image=debian-tricore-cross
+        container_cross_prefix=tricore-
         container_cross_as=tricore-as
         container_cross_ld=tricore-ld
+        break
         ;;
       x86_64)
         container_image=debian-amd64-cross
-        container_cross_cc=x86_64-linux-gnu-gcc
+        container_cross_prefix=x86_64-linux-gnu-
         ;;
       xtensa*)
         # FIXME: xtensa-linux-user?
@@ -1995,9 +2019,12 @@ probe_target_compiler() {
         container_image=debian-xtensa-cross
 
         # default to the dc232b cpu
-        container_cross_cc=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-gcc
+        container_cross_prefix=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-
         ;;
     esac
+    : ${container_cross_cc:=${container_cross_prefix}gcc}
+    : ${container_cross_as:=${container_cross_prefix}as}
+    : ${container_cross_ld:=${container_cross_prefix}ld}
   done
 
   eval "target_cflags=\${cross_cc_cflags_$1}"
@@ -2005,17 +2032,11 @@ probe_target_compiler() {
     if eval has "\"\${cross_cc_$1}\""; then
       eval "target_cc=\"\${cross_cc_$1}\""
     fi
+  else
+    compute_target_variable $1 target_cc gcc
   fi
-  if eval test -n "\"\${cross_as_$1}\""; then
-    if eval has "\"\${cross_as_$1}\""; then
-      eval "target_as=\"\${cross_as_$1}\""
-    fi
-  fi
-  if eval test -n "\"\${cross_ld_$1}\""; then
-    if eval has "\"\${cross_ld_$1}\""; then
-      eval "target_ld=\"\${cross_ld_$1}\""
-    fi
-  fi
+  compute_target_variable $1 target_as as
+  compute_target_variable $1 target_ld ld
   if test "$1" = $cpu; then
     : ${target_cc:=$cc}
     : ${target_as:=$as}
@@ -2048,12 +2069,8 @@ write_container_target_makefile() {
   if test -n "$container_cross_cc"; then
     echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
   fi
-  if test -n "$container_cross_as"; then
-    echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --"
-  fi
-  if test -n "$container_cross_ld"; then
-    echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --"
-  fi
+  echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --"
+  echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --"
 }
 
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 22/33] configure: include more binutils in tests/tcg makefile
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (20 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 21/33] configure: introduce --cross-prefix-*= Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 23/33] configure: move symlink configuration earlier Alex Bennée
                   ` (11 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

Firmware builds require paths to all the binutils; it is not enough to
use only cc, or even as/ld as in the case of tests/tcg/tricore.
Adjust the cross-compiler configurator to detect also ar, nm, objcopy,
ranlib and strip.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-12-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-23-alex.bennee@linaro.org>

diff --git a/configure b/configure
index f55ae82a5d..ced8283b98 100755
--- a/configure
+++ b/configure
@@ -1880,11 +1880,21 @@ probe_target_compiler() {
   container_image=
   container_hosts=
   container_cross_cc=
+  container_cross_ar=
   container_cross_as=
   container_cross_ld=
+  container_cross_nm=
+  container_cross_objcopy=
+  container_cross_ranlib=
+  container_cross_strip=
   target_cc=
+  target_ar=
   target_as=
   target_ld=
+  target_nm=
+  target_objcopy=
+  target_ranlib=
+  target_strip=
 
   case $1 in
     aarch64) container_hosts="x86_64 aarch64" ;;
@@ -2023,8 +2033,13 @@ probe_target_compiler() {
         ;;
     esac
     : ${container_cross_cc:=${container_cross_prefix}gcc}
+    : ${container_cross_ar:=${container_cross_prefix}ar}
     : ${container_cross_as:=${container_cross_prefix}as}
     : ${container_cross_ld:=${container_cross_prefix}ld}
+    : ${container_cross_nm:=${container_cross_prefix}nm}
+    : ${container_cross_objcopy:=${container_cross_prefix}objcopy}
+    : ${container_cross_ranlib:=${container_cross_prefix}ranlib}
+    : ${container_cross_strip:=${container_cross_prefix}strip}
   done
 
   eval "target_cflags=\${cross_cc_cflags_$1}"
@@ -2035,12 +2050,26 @@ probe_target_compiler() {
   else
     compute_target_variable $1 target_cc gcc
   fi
+  target_ccas=$target_cc
+  compute_target_variable $1 target_ar ar
   compute_target_variable $1 target_as as
   compute_target_variable $1 target_ld ld
+  compute_target_variable $1 target_nm nm
+  compute_target_variable $1 target_objcopy objcopy
+  compute_target_variable $1 target_ranlib ranlib
+  compute_target_variable $1 target_strip strip
   if test "$1" = $cpu; then
     : ${target_cc:=$cc}
+    : ${target_ccas:=$ccas}
     : ${target_as:=$as}
     : ${target_ld:=$ld}
+    : ${target_ar:=$ar}
+    : ${target_as:=$as}
+    : ${target_ld:=$ld}
+    : ${target_nm:=$nm}
+    : ${target_objcopy:=$objcopy}
+    : ${target_ranlib:=$ranlib}
+    : ${target_strip:=$strip}
   fi
   if test -n "$target_cc"; then
     case $1 in
@@ -2056,6 +2085,10 @@ probe_target_compiler() {
 write_target_makefile() {
   if test -n "$target_cc"; then
     echo "CC=$target_cc"
+    echo "CCAS=$target_ccas"
+  fi
+  if test -n "$target_ar"; then
+    echo "AR=$target_ar"
   fi
   if test -n "$target_as"; then
     echo "AS=$target_as"
@@ -2063,14 +2096,32 @@ write_target_makefile() {
   if test -n "$target_ld"; then
     echo "LD=$target_ld"
   fi
+  if test -n "$target_nm"; then
+    echo "NM=$target_nm"
+  fi
+  if test -n "$target_objcopy"; then
+    echo "OBJCOPY=$target_objcopy"
+  fi
+  if test -n "$target_ranlib"; then
+    echo "RANLIB=$target_ranlib"
+  fi
+  if test -n "$target_strip"; then
+    echo "STRIP=$target_strip"
+  fi
 }
 
 write_container_target_makefile() {
   if test -n "$container_cross_cc"; then
     echo "CC=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
+    echo "CCAS=\$(DOCKER_SCRIPT) cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
   fi
+  echo "AR=\$(DOCKER_SCRIPT) cc --cc $container_cross_ar -i qemu/$container_image -s $source_path --"
   echo "AS=\$(DOCKER_SCRIPT) cc --cc $container_cross_as -i qemu/$container_image -s $source_path --"
   echo "LD=\$(DOCKER_SCRIPT) cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --"
+  echo "NM=\$(DOCKER_SCRIPT) cc --cc $container_cross_nm -i qemu/$container_image -s $source_path --"
+  echo "OBJCOPY=\$(DOCKER_SCRIPT) cc --cc $container_cross_objcopy -i qemu/$container_image -s $source_path --"
+  echo "RANLIB=\$(DOCKER_SCRIPT) cc --cc $container_cross_ranlib -i qemu/$container_image -s $source_path --"
+  echo "STRIP=\$(DOCKER_SCRIPT) cc --cc $container_cross_strip -i qemu/$container_image -s $source_path --"
 }
 
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 23/33] configure: move symlink configuration earlier
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (21 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 22/33] configure: include more binutils in tests/tcg makefile Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 24/33] configure: enable cross-compilation of s390-ccw Alex Bennée
                   ` (10 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

Ensure that the pc-bios/optionrom and pc-bios/s390-ccw directory
exist at the time when we'll write out the compiler configuration
for them.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-13-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-24-alex.bennee@linaro.org>

diff --git a/configure b/configure
index ced8283b98..aa17b0fa55 100755
--- a/configure
+++ b/configure
@@ -2190,6 +2190,30 @@ if test "$QEMU_GA_VERSION" = ""; then
     QEMU_GA_VERSION=$(cat $source_path/VERSION)
 fi
 
+# Set up build tree symlinks that point back into the source tree
+# (these can be both files and directories).
+# Caution: avoid adding files or directories here using wildcards. This
+# will result in problems later if a new file matching the wildcard is
+# added to the source tree -- nothing will cause configure to be rerun
+# so the build tree will be missing the link back to the new file, and
+# tests might fail. Prefer to keep the relevant files in their own
+# directory and symlink the directory instead.
+LINKS="Makefile"
+LINKS="$LINKS tests/tcg/Makefile.target"
+LINKS="$LINKS pc-bios/optionrom/Makefile"
+LINKS="$LINKS pc-bios/s390-ccw/Makefile"
+LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
+LINKS="$LINKS tests/avocado tests/data"
+LINKS="$LINKS tests/qemu-iotests/check"
+LINKS="$LINKS python"
+LINKS="$LINKS contrib/plugins/Makefile "
+for f in $LINKS ; do
+    if [ -e "$source_path/$f" ]; then
+        mkdir -p `dirname ./$f`
+        symlink "$source_path/$f" "$f"
+    fi
+done
+
 # Mac OS X ships with a broken assembler
 roms=
 if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \
@@ -2408,31 +2432,6 @@ if test "$safe_stack" = "yes"; then
   echo "CONFIG_SAFESTACK=y" >> $config_host_mak
 fi
 
-# If we're using a separate build tree, set it up now.
-# LINKS are things to symlink back into the source tree
-# (these can be both files and directories).
-# Caution: do not add files or directories here using wildcards. This
-# will result in problems later if a new file matching the wildcard is
-# added to the source tree -- nothing will cause configure to be rerun
-# so the build tree will be missing the link back to the new file, and
-# tests might fail. Prefer to keep the relevant files in their own
-# directory and symlink the directory instead.
-LINKS="Makefile"
-LINKS="$LINKS tests/tcg/Makefile.target"
-LINKS="$LINKS pc-bios/optionrom/Makefile"
-LINKS="$LINKS pc-bios/s390-ccw/Makefile"
-LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
-LINKS="$LINKS tests/avocado tests/data"
-LINKS="$LINKS tests/qemu-iotests/check"
-LINKS="$LINKS python"
-LINKS="$LINKS contrib/plugins/Makefile "
-for f in $LINKS ; do
-    if [ -e "$source_path/$f" ]; then
-        mkdir -p `dirname ./$f`
-        symlink "$source_path/$f" "$f"
-    fi
-done
-
 # tests/tcg configuration
 (makefile=tests/tcg/Makefile.prereqs
 echo "# Automatically generated by configure - do not modify" > $makefile
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 24/33] configure: enable cross-compilation of s390-ccw
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (22 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 23/33] configure: move symlink configuration earlier Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 25/33] configure: enable cross-compilation of optionrom Alex Bennée
                   ` (9 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée, Thomas Huth,
	Christian Borntraeger, Cornelia Huck, open list:S390-ccw boot

From: Paolo Bonzini <pbonzini@redhat.com>

While container-based cross compilers are not supported, this already makes
it possible to build s390-ccw on any machine that has s390x GCC and binutils
installed.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-14-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-25-alex.bennee@linaro.org>

diff --git a/configure b/configure
index aa17b0fa55..f509bce304 100755
--- a/configure
+++ b/configure
@@ -2231,24 +2231,32 @@ if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \
     done
 fi
 
-# Only build s390-ccw bios if we're on s390x and the compiler has -march=z900
-# or -march=z10 (which is the lowest architecture level that Clang supports)
-if test "$cpu" = "s390x" ; then
+# Only build s390-ccw bios if the compiler has -march=z900 or -march=z10
+# (which is the lowest architecture level that Clang supports)
+probe_target_compiler s390x
+if test -n "$target_cc" && test "$softmmu" = yes; then
   write_c_skeleton
-  compile_prog "-march=z900" ""
+  do_compiler "$target_cc" $target_cc_cflags -march=z900 -o $TMPO -c $TMPC
   has_z900=$?
-  if [ $has_z900 = 0 ] || compile_object "-march=z10 -msoft-float -Werror"; then
+  if [ $has_z900 = 0 ] || do_compiler "$target_cc" $target_cc_cflags -march=z10 -msoft-float -Werror -o $TMPO -c $TMPC; then
     if [ $has_z900 != 0 ]; then
       echo "WARNING: Your compiler does not support the z900!"
       echo "         The s390-ccw bios will only work with guest CPUs >= z10."
     fi
     roms="$roms s390-ccw"
+    config_mak=pc-bios/s390-ccw/config-host.mak
+    echo "# Automatically generated by configure - do not modify" > $config_mak
+    echo "SRC_PATH=$source_path/pc-bios/s390-ccw" >> $config_mak
+    write_target_makefile >> $config_mak
     # SLOF is required for building the s390-ccw firmware on s390x,
     # since it is using the libnet code from SLOF for network booting.
     git_submodules="${git_submodules} roms/SLOF"
   fi
 fi
 
+#######################################
+# generate config-host.mak
+
 # Check that the C++ compiler exists and works with the C compiler.
 # All the QEMU_CXXFLAGS are based on QEMU_CFLAGS. Keep this at the end to don't miss any other that could be added.
 if has $cxx; then
diff --git a/pc-bios/s390-ccw/netboot.mak b/pc-bios/s390-ccw/netboot.mak
index 68b4d7edcb..1a06befa4b 100644
--- a/pc-bios/s390-ccw/netboot.mak
+++ b/pc-bios/s390-ccw/netboot.mak
@@ -1,5 +1,5 @@
 
-SLOF_DIR := $(SRC_PATH)/roms/SLOF
+SLOF_DIR := $(SRC_PATH)/../../roms/SLOF
 
 NETOBJS := start.o sclp.o cio.o virtio.o virtio-net.o jump2ipl.o netmain.o
 
diff --git a/pc-bios/s390-ccw/Makefile b/pc-bios/s390-ccw/Makefile
index 0eb68efc7b..6eb713bf37 100644
--- a/pc-bios/s390-ccw/Makefile
+++ b/pc-bios/s390-ccw/Makefile
@@ -2,8 +2,9 @@ all: build-all
 # Dummy command so that make thinks it has done something
 	@true
 
-include ../../config-host.mak
+include config-host.mak
 CFLAGS = -O2 -g
+MAKEFLAGS += -rR
 
 quiet-command = $(if $(V),$1,$(if $(2),@printf "  %-7s %s\n" $2 $3 && $1, @$1))
 cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc /dev/null \
@@ -11,7 +12,7 @@ cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc /dev/null \
 
 VPATH_SUFFIXES = %.c %.h %.S %.m %.mak %.sh %.rc Kconfig% %.json.in
 set-vpath = $(if $1,$(foreach PATTERN,$(VPATH_SUFFIXES),$(eval vpath $(PATTERN) $1)))
-$(call set-vpath, $(SRC_PATH)/pc-bios/s390-ccw)
+$(call set-vpath, $(SRC_PATH))
 
 # Flags for dependency generation
 QEMU_DGFLAGS = -MMD -MP -MT $@ -MF $(@D)/$(*F).d
@@ -49,8 +50,8 @@ s390-ccw.img: s390-ccw.elf
 
 $(OBJECTS): Makefile
 
-ifneq ($(wildcard $(SRC_PATH)/roms/SLOF/lib/libnet),)
-include $(SRC_PATH)/pc-bios/s390-ccw/netboot.mak
+ifneq ($(wildcard $(SRC_PATH)/../../roms/SLOF/lib/libnet),)
+include $(SRC_PATH)/netboot.mak
 else
 s390-netboot.img:
 	@echo "s390-netboot.img not built since roms/SLOF/ is not available."
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 25/33] configure: enable cross-compilation of optionrom
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (23 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 24/33] configure: enable cross-compilation of s390-ccw Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 26/33] configure: enable cross compilation of vof Alex Bennée
                   ` (8 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

While container-based cross compilers are not supported, this already makes
it possible to build x86 optionroms on any machine that has an installation
of GCC and binutils for 32- or 64-bit x86.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-15-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-26-alex.bennee@linaro.org>

diff --git a/configure b/configure
index f509bce304..1ae5c950f0 100755
--- a/configure
+++ b/configure
@@ -2082,6 +2082,13 @@ probe_target_compiler() {
   fi
 }
 
+probe_target_compilers() {
+  for i; do
+    probe_target_compiler $i
+    test -n "$target_cc" && return 0
+  done
+}
+
 write_target_makefile() {
   if test -n "$target_cc"; then
     echo "CC=$target_cc"
@@ -2190,6 +2197,10 @@ if test "$QEMU_GA_VERSION" = ""; then
     QEMU_GA_VERSION=$(cat $source_path/VERSION)
 fi
 
+
+#######################################
+# cross-compiled firmware targets
+
 # Set up build tree symlinks that point back into the source tree
 # (these can be both files and directories).
 # Caution: avoid adding files or directories here using wildcards. This
@@ -2216,19 +2227,27 @@ done
 
 # Mac OS X ships with a broken assembler
 roms=
-if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \
+probe_target_compilers i386 x86_64
+if test -n "$target_cc" &&
         test "$targetos" != "darwin" && test "$targetos" != "sunos" && \
         test "$targetos" != "haiku" && test "$softmmu" = yes ; then
     # Different host OS linkers have different ideas about the name of the ELF
     # emulation. Linux and OpenBSD/amd64 use 'elf_i386'; FreeBSD uses the _fbsd
     # variant; OpenBSD/i386 uses the _obsd variant; and Windows uses i386pe.
     for emu in elf_i386 elf_i386_fbsd elf_i386_obsd i386pe; do
-        if "$ld" -verbose 2>&1 | grep -q "^[[:space:]]*$emu[[:space:]]*$"; then
+        if "$target_ld" -verbose 2>&1 | grep -q "^[[:space:]]*$emu[[:space:]]*$"; then
             ld_i386_emulation="$emu"
-            roms="optionrom"
             break
         fi
     done
+    if test -n "$ld_i386_emulation"; then
+        roms="optionrom"
+        config_mak=pc-bios/optionrom/config.mak
+        echo "# Automatically generated by configure - do not modify" > $config_mak
+        echo "TOPSRC_DIR=$source_path" >> $config_mak
+        echo "LD_I386_EMULATION=$ld_i386_emulation" >> $config_mak
+        write_target_makefile >> $config_mak
+    fi
 fi
 
 # Only build s390-ccw bios if the compiler has -march=z900 or -march=z10
@@ -2381,7 +2400,6 @@ echo "GLIB_LIBS=$glib_libs" >> $config_host_mak
 echo "GLIB_BINDIR=$glib_bindir" >> $config_host_mak
 echo "GLIB_VERSION=$(pkg-config --modversion glib-2.0)" >> $config_host_mak
 echo "QEMU_LDFLAGS=$QEMU_LDFLAGS" >> $config_host_mak
-echo "LD_I386_EMULATION=$ld_i386_emulation" >> $config_host_mak
 echo "STRIP=$strip" >> $config_host_mak
 echo "EXESUF=$EXESUF" >> $config_host_mak
 
@@ -2571,10 +2589,6 @@ for target in $target_list; do
 done
 echo "TCG_TESTS_TARGETS=$tcg_tests_targets" >> $makefile)
 
-config_mak=pc-bios/optionrom/config.mak
-echo "# Automatically generated by configure - do not modify" > $config_mak
-echo "TOPSRC_DIR=$source_path" >> $config_mak
-
 if test "$skip_meson" = no; then
   cross="config-meson.cross.new"
   meson_quote() {
diff --git a/pc-bios/optionrom/Makefile b/pc-bios/optionrom/Makefile
index 17ccc76241..f639915b4f 100644
--- a/pc-bios/optionrom/Makefile
+++ b/pc-bios/optionrom/Makefile
@@ -6,7 +6,6 @@ all: multiboot.bin multiboot_dma.bin linuxboot.bin linuxboot_dma.bin kvmvapic.bi
 # Dummy command so that make thinks it has done something
 	@true
 
-include ../../config-host.mak
 CFLAGS = -O2 -g
 
 quiet-command = $(if $(V),$1,$(if $(2),@printf "  %-7s %s\n" $2 $3 && $1, @$1))
@@ -44,7 +43,6 @@ Wa = -Wa,
 override ASFLAGS += -32
 override CFLAGS += $(call cc-option, $(Wa)-32)
 
-LD_I386_EMULATION ?= elf_i386
 override LDFLAGS = -m $(LD_I386_EMULATION) -T $(SRC_DIR)/flat.lds
 
 pvh.img: pvh.o pvh_main.o
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 26/33] configure: enable cross compilation of vof
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (24 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 25/33] configure: enable cross-compilation of optionrom Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 27/33] configure: remove unused variables from config-host.mak Alex Bennée
                   ` (7 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée,
	Alexey Kardashevskiy, Cédric Le Goater,
	Daniel Henrique Barboza, David Gibson, Greg Kurz,
	open list:Virtual Open Firm...

From: Paolo Bonzini <pbonzini@redhat.com>

While container-based cross compilers are not supported, this already
makes it possible to build vof on any machine that has an installation
of GCC and binutils for 32- or 64-bit PowerPC.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-16-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-27-alex.bennee@linaro.org>

diff --git a/configure b/configure
index 1ae5c950f0..11e7f30553 100755
--- a/configure
+++ b/configure
@@ -2213,6 +2213,7 @@ LINKS="Makefile"
 LINKS="$LINKS tests/tcg/Makefile.target"
 LINKS="$LINKS pc-bios/optionrom/Makefile"
 LINKS="$LINKS pc-bios/s390-ccw/Makefile"
+LINKS="$LINKS pc-bios/vof/Makefile"
 LINKS="$LINKS .gdbinit scripts" # scripts needed by relative path in .gdbinit
 LINKS="$LINKS tests/avocado tests/data"
 LINKS="$LINKS tests/qemu-iotests/check"
@@ -2250,6 +2251,15 @@ if test -n "$target_cc" &&
     fi
 fi
 
+probe_target_compilers ppc ppc64
+if test -n "$target_cc" && test "$softmmu" = yes; then
+    roms="$roms vof"
+    config_mak=pc-bios/vof/config.mak
+    echo "# Automatically generated by configure - do not modify" > $config_mak
+    echo "SRC_DIR=$source_path/pc-bios/vof" >> $config_mak
+    write_target_makefile >> $config_mak
+fi
+
 # Only build s390-ccw bios if the compiler has -march=z900 or -march=z10
 # (which is the lowest architecture level that Clang supports)
 probe_target_compiler s390x
diff --git a/pc-bios/vof/Makefile b/pc-bios/vof/Makefile
index aa1678c4d8..391ac0d600 100644
--- a/pc-bios/vof/Makefile
+++ b/pc-bios/vof/Makefile
@@ -1,11 +1,10 @@
-all: build-all
+include config.mak
+VPATH=$(SRC_DIR)
+all: vof.bin
 
-build-all: vof.bin
-
-CROSS ?=
-CC = $(CROSS)gcc
-LD = $(CROSS)ld
-OBJCOPY = $(CROSS)objcopy
+CC ?= $(CROSS)gcc
+LD ?= $(CROSS)ld
+OBJCOPY ?= $(CROSS)objcopy
 
 %.o: %.S
 	$(CC) -m32 -mbig-endian -mcpu=power4 -c -o $@ $<
@@ -14,10 +13,12 @@ OBJCOPY = $(CROSS)objcopy
 	$(CC) -m32 -mbig-endian -mcpu=power4 -c -fno-stack-protector -o $@ $<
 
 vof.elf: entry.o main.o ci.o bootmem.o libc.o
-	$(LD) -nostdlib -e_start -Tvof.lds -EB -o $@ $^
+	$(LD) -nostdlib -e_start -T$(SRC_DIR)/vof.lds -EB -o $@ $^
 
 %.bin: %.elf
 	$(OBJCOPY) -O binary -j .text -j .data -j .toc -j .got2 $^ $@
 
 clean:
 	rm -f *.o vof.bin vof.elf *~
+
+.PHONY: all clean
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 27/33] configure: remove unused variables from config-host.mak
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (25 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 26/33] configure: enable cross compilation of vof Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 28/33] gitlab: introduce a common base job template Alex Bennée
                   ` (6 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Alex Bennée

From: Paolo Bonzini <pbonzini@redhat.com>

The only compiler variable that is still needed is $(CC), for
contrib/plugins/Makefile.  All firmware builds have their own
config-host.mak file.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-17-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-28-alex.bennee@linaro.org>

diff --git a/configure b/configure
index 11e7f30553..b9ccff9067 100755
--- a/configure
+++ b/configure
@@ -2397,11 +2397,6 @@ echo "GENISOIMAGE=$genisoimage" >> $config_host_mak
 echo "MESON=$meson" >> $config_host_mak
 echo "NINJA=$ninja" >> $config_host_mak
 echo "CC=$cc" >> $config_host_mak
-echo "AR=$ar" >> $config_host_mak
-echo "AS=$as" >> $config_host_mak
-echo "CCAS=$ccas" >> $config_host_mak
-echo "OBJCOPY=$objcopy" >> $config_host_mak
-echo "LD=$ld" >> $config_host_mak
 echo "QEMU_CFLAGS=$QEMU_CFLAGS" >> $config_host_mak
 echo "QEMU_CXXFLAGS=$QEMU_CXXFLAGS" >> $config_host_mak
 echo "QEMU_OBJCFLAGS=$QEMU_OBJCFLAGS" >> $config_host_mak
@@ -2410,7 +2405,6 @@ echo "GLIB_LIBS=$glib_libs" >> $config_host_mak
 echo "GLIB_BINDIR=$glib_bindir" >> $config_host_mak
 echo "GLIB_VERSION=$(pkg-config --modversion glib-2.0)" >> $config_host_mak
 echo "QEMU_LDFLAGS=$QEMU_LDFLAGS" >> $config_host_mak
-echo "STRIP=$strip" >> $config_host_mak
 echo "EXESUF=$EXESUF" >> $config_host_mak
 
 # use included Linux headers
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 28/33] gitlab: introduce a common base job template
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (26 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 27/33] configure: remove unused variables from config-host.mak Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 29/33] gitlab: convert Cirrus jobs to .base_job_template Alex Bennée
                   ` (5 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Daniel P. Berrangé,
	Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Daniel P. Berrangé <berrange@redhat.com>

Currently job rules are spread across the various templates
and jobs, making it hard to understand exactly what runs in
what scenario. This leads to inconsistency in the rules and
increased maint burden.

The intent is that we introduce a common '.base_job_template'
which will have a general purpose 'rules:' block. No other
template or job should define 'rules:', but instead they must
rely on the inherited rules. To allow behaviour to be tweaked,
rules will be influenced by a number of variables with the
naming scheme 'QEMU_JOB_nnnn'.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-29-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 92e25872aa..eb6a9e6122 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -28,7 +28,35 @@ For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
-Here is a list of the most used variables:
+Variable naming and grouping
+----------------------------
+
+The variables used by QEMU's CI configuration are grouped together
+in a handful of namespaces
+
+ * QEMU_JOB_nnnn - variables to be defined in individual jobs
+   or templates, to influence the shared rules defined in the
+   .base_job_template.
+
+ * QEMU_CI_nnn - variables to be set by contributors in their
+   repository CI settings, or as git push variables, to influence
+   which jobs get run in a pipeline
+
+ * nnn - other misc variables not falling into the above
+   categories, or using different names for historical reasons
+   and not yet converted.
+
+Maintainer controlled job variables
+-----------------------------------
+
+The following variables may be set when defining a job in the
+CI configuration file.
+
+Contributor controlled runtime variables
+----------------------------------------
+
+The following variables may be set by contributors to control
+job execution
 
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -38,6 +66,12 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+Other misc variables
+--------------------
+
+These variables are primarily to control execution of jobs on
+private runners
+
 AARCH64_RUNNER_AVAILABLE
 ~~~~~~~~~~~~~~~~~~~~~~~~
 If you've got access to an aarch64 host that can be used as a gitlab-CI
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
new file mode 100644
index 0000000000..10eb6ab8bc
--- /dev/null
+++ b/.gitlab-ci.d/base.yml
@@ -0,0 +1,28 @@
+
+# The order of rules defined here is critically important.
+# They are evaluated in order and first match wins.
+#
+# Thus we group them into a number of stages, ordered from
+# most restrictive to least restrictive
+#
+.base_job_template:
+  rules:
+    #############################################################
+    # Stage 1: exclude scenarios where we definitely don't
+    # want jobs to run
+    #############################################################
+
+
+    #############################################################
+    # Stage 2: fine tune execution of jobs in specific scenarios
+    # where the catch all logic is inapprorpaite
+    #############################################################
+
+
+    #############################################################
+    # Stage 3: catch all logic applying to any job not matching
+    # an earlier criteria
+    #############################################################
+
+    # Jobs can run if any jobs they depend on were successfull
+    - when: on_success
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 871262fe0e..691d9bf5dc 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -2,6 +2,7 @@
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
 include:
+  - local: '/.gitlab-ci.d/base.yml'
   - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 29/33] gitlab: convert Cirrus jobs to .base_job_template
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (27 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 28/33] gitlab: introduce a common base job template Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 30/33] gitlab: convert static checks " Alex Bennée
                   ` (4 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Daniel P. Berrangé,
	Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Daniel P. Berrangé <berrange@redhat.com>

This folds the Cirrus job rules into the base job
template, introducing two new variables

  - QEMU_JOB_CIRRUS - identifies the job as making
    use of Cirrus CI via cirrus-run

  - QEMU_JOB_OPTIONAL - identifies the job as one
    that is not run by default, primarily due to
    resource constraints. It can be manually invoked
    by users if they wish to validate that scenario.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-30-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index eb6a9e6122..a539f502da 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -52,6 +52,20 @@ Maintainer controlled job variables
 The following variables may be set when defining a job in the
 CI configuration file.
 
+QEMU_JOB_CIRRUS
+~~~~~~~~~~~~~~~
+
+The job makes use of Cirrus CI infrastructure, requiring the
+configuration setup for cirrus-run to be present in the repository
+
+QEMU_JOB_OPTIONAL
+~~~~~~~~~~~~~~~~~
+
+The job is expected to be successful in general, but is not run
+by default due to need to conserve limited CI resources. It is
+available to be started manually by the contributor in the CI
+pipelines UI.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 10eb6ab8bc..5734caf9fe 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -12,12 +12,21 @@
     # want jobs to run
     #############################################################
 
+    # Cirrus jobs can't run unless the creds / target repo are set
+    - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
     # where the catch all logic is inapprorpaite
     #############################################################
 
+    # Optional jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_OPTIONAL'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index b96b22e269..609c364308 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -11,6 +11,7 @@
 # special care, because we can't just override it at the GitLab CI job
 # definition level or we risk breaking it completely.
 .cirrus_build_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
@@ -40,11 +41,8 @@
       <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    # Allow on 'staging' branch and 'stable-X.Y-staging' branches only
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
-      when: never
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+  variables:
+    QEMU_JOB_CIRRUS: 1
 
 x64-freebsd-12-build:
   extends: .cirrus_build_job
@@ -90,11 +88,11 @@ x64-macos-11-base-build:
 
 # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
 .cirrus_kvm_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
   timeout: 80m
-  allow_failure: true
   script:
     - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
           -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
@@ -105,8 +103,10 @@ x64-macos-11-base-build:
       <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - when: manual
+  variables:
+    QEMU_JOB_CIRRUS: 1
+    QEMU_JOB_OPTIONAL: 1
+
 
 x86-netbsd:
   extends: .cirrus_kvm_job
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 30/33] gitlab: convert static checks to .base_job_template
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (28 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 29/33] gitlab: convert Cirrus jobs to .base_job_template Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 31/33] gitlab: convert build/container jobs " Alex Bennée
                   ` (3 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Daniel P. Berrangé,
	Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Daniel P. Berrangé <berrange@redhat.com>

This folds the static checks into using the base job
template rules, introducing one new variable

 - QEMU_JOB_ONLY_FORKS - a job that should never run
   on an upstream pipeline. The information it reports
   is only applicable to contributors in a pre-submission
   scenario, not time of merge.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-4-berrange@redhat.com>
[AJB: fix typo]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-31-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index a539f502da..4c7e30ab08 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -66,6 +66,13 @@ by default due to need to conserve limited CI resources. It is
 available to be started manually by the contributor in the CI
 pipelines UI.
 
+QEMU_JOB_ONLY_FORKS
+~~~~~~~~~~~~~~~~~~~
+
+The job results are only of interest to contributors prior to
+submitting code. They are not required as part of the gating
+CI pipeline.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 5734caf9fe..e6953c77ae 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -16,6 +16,10 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Jobs only intended for forks should always be skipped on upstream
+    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 94858e3272..289ad1359e 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -1,4 +1,5 @@
 check-patch:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -6,15 +7,13 @@ check-patch:
     - .gitlab-ci.d/check-patch.py
   variables:
     GIT_DEPTH: 1000
+    QEMU_JOB_ONLY_FORKS: 1
   before_script:
     - apk -U add git perl
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
 check-dco:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -23,12 +22,9 @@ check-dco:
     GIT_DEPTH: 1000
   before_script:
     - apk -U add git
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
 check-python-pipenv:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -39,6 +35,7 @@ check-python-pipenv:
     job: python-container
 
 check-python-tox:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -46,8 +43,6 @@ check-python-tox:
   variables:
     GIT_DEPTH: 1
     QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
+    QEMU_JOB_OPTIONAL: 1
   needs:
     job: python-container
-  rules:
-    - when: manual
-  allow_failure: true
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 31/33] gitlab: convert build/container jobs to .base_job_template
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (29 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 30/33] gitlab: convert static checks " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 18:05 ` [PULL 32/33] gitlab: don't run CI jobs in forks by default Alex Bennée
                   ` (2 subsequent siblings)
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Daniel P. Berrangé,
	Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Daniel P. Berrangé <berrange@redhat.com>

This converts the main build and container jobs to use the
base job rules, defining the following new variables

 - QEMU_JOB_SKIPPED - jobs that are known to be currently
   broken and should not be run. Can still be manually
   launched if desired.

 - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
   test harness.

 - QEMU_JOB_PUBLISH - jobs that publish content after the
   branch is merged upstream

As build-tools-and-docs runs on master we declare the requirement of
building amd64-debian-container optional as it should already exits
once we merge.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-5-berrange@redhat.com>
[AJB: fix upstream typo, mention optional container req]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 4c7e30ab08..0b4926e537 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -73,6 +73,25 @@ The job results are only of interest to contributors prior to
 submitting code. They are not required as part of the gating
 CI pipeline.
 
+QEMU_JOB_SKIPPED
+~~~~~~~~~~~~~~~~
+
+The job is not reliably successsful in general, so is not
+currently suitable to be run by default. Ideally this should
+be a temporary marker until the problems can be addressed, or
+the job permanently removed.
+
+QEMU_JOB_PUBLISH
+~~~~~~~~~~~~~~~~
+
+The job is for publishing content after a branch has been
+merged into the upstream default branch.
+
+QEMU_JOB_AVOCADO
+~~~~~~~~~~~~~~~~
+
+The job runs the Avocado integration test suite
+
 Contributor controlled runtime variables
 ----------------------------------------
 
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index e6953c77ae..4f091d5aad 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -16,10 +16,22 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Publishing jobs should only run on the default branch in upstream
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+      when: never
+
+    # Non-publishing jobs should only run on staging branches in upstream
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+      when: never
+
     # Jobs only intended for forks should always be skipped on upstream
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
@@ -31,6 +43,16 @@
       when: manual
       allow_failure: true
 
+    # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED'
+      when: manual
+      allow_failure: true
+
+    # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index b381345dbc..73ecfabb8d 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,4 +1,5 @@
 .native_build_job_template:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +28,7 @@
       fi
 
 .common_test_job_template:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -77,15 +79,5 @@
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_AVOCADO: 1
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index e9620c3074..ecac3ec50c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -360,12 +360,11 @@ build-cfi-aarch64:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
@@ -579,6 +577,7 @@ build-without-default-features:
     MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
@@ -595,10 +594,13 @@ build-tools-and-docs-debian:
   extends: .native_build_job_template
   needs:
     job: amd64-debian-container
+    # when running on 'master' we use pre-existing container
+    optional: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+    QEMU_JOB_PUBLISH: 1
   artifacts:
     expire_in: 2 days
     paths:
@@ -618,6 +620,7 @@ build-tools-and-docs-debian:
 # that users can see the results of their commits, regardless
 # of what topic branch they're currently using
 pages:
+  extends: .base_job_template
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
@@ -635,10 +638,5 @@ pages:
   artifacts:
     paths:
       - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
+  variables:
+    QEMU_JOB_PUBLISH: 1
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 147e667744..b7963498a3 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -40,15 +40,13 @@ armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
+    QEMU_JOB_ONLY_FORKS: 1
   services:
     - docker:dind
   before_script:
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 1baecd9460..c434b9c8f3 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,4 +1,5 @@
 .container_job_template:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
   services:
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 29c3c2b826..28b2142ec2 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -24,6 +25,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -36,6 +38,7 @@
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index cf7724b8e5..1b2ede49e1 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -1,4 +1,5 @@
 .shared_msys2_builder:
+  extends: .base_job_template
   tags:
   - shared-windows
   - windows
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 32/33] gitlab: don't run CI jobs in forks by default
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (30 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 31/33] gitlab: convert build/container jobs " Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-02 10:50   ` Philippe Mathieu-Daudé via
  2022-06-01 18:05 ` [PULL 33/33] docs/devel: clean-up the CI links in the docs Alex Bennée
  2022-06-01 21:21 ` [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Richard Henderson
  33 siblings, 1 reply; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Daniel P. Berrangé,
	Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Beraldo Leal

From: Daniel P. Berrangé <berrange@redhat.com>

To preserve CI shared runner credits we don't want to run
pipelines on every push.

This sets up the config so that pipelines are never created
for contributors by default. To override this the QEMU_CI
variable can be set to a non-zero value. If set to 1, the
pipeline will be created but all jobs will remain manually
started. The contributor can selectively run jobs that they
care about. If set to 2, the pipeline will be created and
all jobs will immediately start.

This behavior can be controlled using push variables

  git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

  git push-ci

to create the pipeline, or

  git push-ci-now

to create and run the pipeline

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-6-berrange@redhat.com>
[AJB: fix typo, replicate alias tips in ci.rst]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 0b4926e537..9118a61a17 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -28,6 +28,32 @@ For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
+Setting aliases in your git config
+----------------------------------
+
+You can use aliases to make it easier to push branches with different
+CI configurations. For example define an alias for triggering CI:
+
+.. code::
+
+   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
+   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
+
+Which lets you run:
+
+.. code::
+
+   git push-ci
+
+to create the pipeline, or:
+
+.. code::
+
+   git push-ci-now
+
+to create and run the pipeline
+
+
 Variable naming and grouping
 ----------------------------
 
@@ -98,6 +124,18 @@ Contributor controlled runtime variables
 The following variables may be set by contributors to control
 job execution
 
+QEMU_CI
+~~~~~~~
+
+By default, no pipelines will be created on contributor forks
+in order to preserve CI credits
+
+Set this variable to 1 to create the pipelines, but leave all
+the jobs to be manually started from the UI
+
+Set this variable to 2 to create the pipelines and run all
+the jobs immediately, as was historicaly behaviour
+
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
 By default, tests using the Avocado framework are not run automatically in
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 4f091d5aad..f334f3ded7 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -28,6 +28,10 @@
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
     - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
       when: never
@@ -59,5 +63,10 @@
     # an earlier criteria
     #############################################################
 
+    # Forks pipeline jobs don't start automatically unless
+    # QEMU_CI=2 is set
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successfull
     - when: on_success
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* [PULL 33/33] docs/devel: clean-up the CI links in the docs
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (31 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 32/33] gitlab: don't run CI jobs in forks by default Alex Bennée
@ 2022-06-01 18:05 ` Alex Bennée
  2022-06-01 21:21 ` [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Richard Henderson
  33 siblings, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-01 18:05 UTC (permalink / raw)
  To: peter.maydell, richard.henderson
  Cc: qemu-devel, Alex Bennée, Daniel P . Berrangé

There where some broken links so fix those up with proper references
to the devel docs. I also did a little light copy-editing to reflect
the current state and broke up a paragraph to reduce the "wall of
text" effect.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-34-alex.bennee@linaro.org>

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 9118a61a17..1f28fec0d0 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -1,3 +1,5 @@
+.. _ci_var:
+
 Custom CI/CD variables
 ======================
 
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index d106610096..ed88a2010b 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -1,12 +1,13 @@
+.. _ci:
+
 ==
 CI
 ==
 
-QEMU has configurations enabled for a number of different CI services.
-The most up to date information about them and their status can be
-found at::
-
-   https://wiki.qemu.org/Testing/CI
+Most of QEMU's CI is run on GitLab's infrastructure although a number
+of other CI services are used for specialised purposes. The most up to
+date information about them and their status can be found on the
+`project wiki testing page <https://wiki.qemu.org/Testing/CI>`_.
 
 .. include:: ci-definitions.rst.inc
 .. include:: ci-jobs.rst.inc
diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
index e51259eb9c..d3876ec1b7 100644
--- a/docs/devel/submitting-a-patch.rst
+++ b/docs/devel/submitting-a-patch.rst
@@ -204,23 +204,25 @@ log`` for these keywords for example usage.
 Test your patches
 ~~~~~~~~~~~~~~~~~
 
-Although QEMU has `continuous integration
-services <Testing#Continuous_Integration>`__ that attempt to test
-patches submitted to the list, it still saves everyone time if you have
-already tested that your patch compiles and works. Because QEMU is such
-a large project, it's okay to use configure arguments to limit what is
-built for faster turnaround during your development time; but it is
-still wise to also check that your patches work with a full build before
-submitting a series, especially if your changes might have an unintended
-effect on other areas of the code you don't normally experiment with.
-See `Testing <Testing>`__ for more details on what tests are available.
-Also, it is a wise idea to include a testsuite addition as part of your
-patches - either to ensure that future changes won't regress your new
-feature, or to add a test which exposes the bug that the rest of your
-series fixes. Keeping separate commits for the test and the fix allows
-reviewers to rebase the test to occur first to prove it catches the
-problem, then again to place it last in the series so that bisection
-doesn't land on a known-broken state.
+Although QEMU uses various :ref:`ci` services that attempt to test
+patches submitted to the list, it still saves everyone time if you
+have already tested that your patch compiles and works. Because QEMU
+is such a large project the default configuration won't create a
+testing pipeline on GitLab when a branch is pushed. See the :ref:`CI
+variable documentation<ci_var>` for details on how to control the
+running of tests; but it is still wise to also check that your patches
+work with a full build before submitting a series, especially if your
+changes might have an unintended effect on other areas of the code you
+don't normally experiment with. See :ref:`testing` for more details on
+what tests are available.
+
+Also, it is a wise idea to include a testsuite addition as part of
+your patches - either to ensure that future changes won't regress your
+new feature, or to add a test which exposes the bug that the rest of
+your series fixes. Keeping separate commits for the test and the fix
+allows reviewers to rebase the test to occur first to prove it catches
+the problem, then again to place it last in the series so that
+bisection doesn't land on a known-broken state.
 
 .. _submitting_your_patches:
 
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 5b60a31807..3f6ebd5073 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1,3 +1,5 @@
+.. _testing:
+
 Testing in QEMU
 ===============
 
-- 
2.30.2



^ permalink raw reply related	[flat|nested] 45+ messages in thread

* Re: [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile)
  2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
                   ` (32 preceding siblings ...)
  2022-06-01 18:05 ` [PULL 33/33] docs/devel: clean-up the CI links in the docs Alex Bennée
@ 2022-06-01 21:21 ` Richard Henderson
  33 siblings, 0 replies; 45+ messages in thread
From: Richard Henderson @ 2022-06-01 21:21 UTC (permalink / raw)
  To: Alex Bennée, peter.maydell; +Cc: qemu-devel

On 6/1/22 11:05, Alex Bennée wrote:
> The following changes since commit 7077fcb9b68f058809c9dd9fd1dacae1881e886c:
> 
>    Merge tag 'vmbus-maint-20220530' of https://github.com/maciejsszmigiero/qemu into staging (2022-05-30 12:40:36 -0700)
> 
> are available in the Git repository at:
> 
>    https://github.com/stsquad/qemu.git tags/pull-testing-next-010622-3
> 
> for you to fetch changes up to 7266ecce502c31387a3cbf83d7297bc9cf27b139:
> 
>    docs/devel: clean-up the CI links in the docs (2022-06-01 18:55:04 +0100)
> 
> ----------------------------------------------------------------
> Various testing updates
> 
>    - fix some gitlab container dependencies
>    - report meson test results via JUnit
>    - fix meson display of enabled cross compilers
>    - convert more cross build containers to lcitool and Debian 11
>    - re-factor cross compiler detection
>    - use test cross-compilers for building ROMs
>    - disable CI runs by default (see docs)
>    - fix some broken links in development documentation

Applied, thanks.  Please update https://wiki.qemu.org/ChangeLog/7.1 as appropriate.


r~



> 
> ----------------------------------------------------------------
> Alex Bennée (9):
>        meson.build: fix summary display of test compilers
>        tests/lcitool: fix up indentation to correct style
>        tests/docker: update debian-armhf-cross with lcitool
>        tests/docker: update debian-armel-cross with lcitool
>        tests/docker: update debian-mipsel-cross with lcitool
>        tests/docker: update debian-mips64el-cross with lcitool
>        tests/docker: update debian-ppc64el-cross with lcitool
>        tests/docker: update debian-amd64 with lcitool
>        docs/devel: clean-up the CI links in the docs
> 
> Daniel P. Berrangé (5):
>        gitlab: introduce a common base job template
>        gitlab: convert Cirrus jobs to .base_job_template
>        gitlab: convert static checks to .base_job_template
>        gitlab: convert build/container jobs to .base_job_template
>        gitlab: don't run CI jobs in forks by default
> 
> Marc-André Lureau (1):
>        gitlab-ci: add meson JUnit test result into report
> 
> Paolo Bonzini (16):
>        configure: do not define or use the CPP variable
>        build: clean up ninja invocation
>        build: add a more generic way to specify make->ninja dependencies
>        build: do a full build before running TCG tests
>        configure, meson: move symlinking of ROMs to meson
>        tests/tcg: correct target CPU for sparc32
>        tests/tcg: merge configure.sh back into main configure script
>        configure: add missing cross compiler fallbacks
>        configure: handle host compiler in probe_target_compiler
>        configure: introduce --cross-prefix-*=
>        configure: include more binutils in tests/tcg makefile
>        configure: move symlink configuration earlier
>        configure: enable cross-compilation of s390-ccw
>        configure: enable cross-compilation of optionrom
>        configure: enable cross compilation of vof
>        configure: remove unused variables from config-host.mak
> 
> Thomas Huth (2):
>        .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages
>        .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job
> 
>   docs/devel/ci-jobs.rst.inc                         | 116 +++-
>   docs/devel/ci.rst                                  |  11 +-
>   docs/devel/submitting-a-patch.rst                  |  36 +-
>   docs/devel/testing.rst                             |   2 +
>   configure                                          | 606 ++++++++++++++++++---
>   Makefile                                           |   9 +-
>   pc-bios/s390-ccw/netboot.mak                       |   2 +-
>   meson.build                                        |   8 +-
>   .gitlab-ci.d/base.yml                              |  72 +++
>   .gitlab-ci.d/buildtest-template.yml                |  18 +-
>   .gitlab-ci.d/buildtest.yml                         |  28 +-
>   .gitlab-ci.d/cirrus.yml                            |  16 +-
>   .gitlab-ci.d/container-cross.yml                   |  24 +-
>   .gitlab-ci.d/container-template.yml                |   1 +
>   .gitlab-ci.d/containers.yml                        |   3 +-
>   .gitlab-ci.d/crossbuild-template.yml               |   3 +
>   .gitlab-ci.d/crossbuilds.yml                       |   2 +
>   .gitlab-ci.d/qemu-project.yml                      |   1 +
>   .gitlab-ci.d/static_checks.yml                     |  19 +-
>   .gitlab-ci.d/windows.yml                           |   1 +
>   pc-bios/meson.build                                |  17 +-
>   pc-bios/optionrom/Makefile                         |   4 +-
>   pc-bios/s390-ccw/Makefile                          |   9 +-
>   pc-bios/vof/Makefile                               |  17 +-
>   scripts/mtest2make.py                              |   8 +-
>   tests/Makefile.include                             |   4 +-
>   tests/docker/Makefile.include                      |   5 -
>   tests/docker/dockerfiles/debian-amd64.docker       | 194 +++++--
>   tests/docker/dockerfiles/debian-armel-cross.docker | 178 +++++-
>   tests/docker/dockerfiles/debian-armhf-cross.docker | 184 ++++++-
>   .../dockerfiles/debian-mips64el-cross.docker       | 177 +++++-
>   .../docker/dockerfiles/debian-mipsel-cross.docker  | 179 +++++-
>   .../docker/dockerfiles/debian-ppc64el-cross.docker | 178 +++++-
>   tests/lcitool/refresh                              | 178 ++++--
>   tests/tcg/configure.sh                             | 376 -------------
>   35 files changed, 1885 insertions(+), 801 deletions(-)
>   create mode 100644 .gitlab-ci.d/base.yml
>   delete mode 100755 tests/tcg/configure.sh
> 



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 32/33] gitlab: don't run CI jobs in forks by default
  2022-06-01 18:05 ` [PULL 32/33] gitlab: don't run CI jobs in forks by default Alex Bennée
@ 2022-06-02 10:50   ` Philippe Mathieu-Daudé via
  2022-06-06 11:01     ` Daniel P. Berrangé
  0 siblings, 1 reply; 45+ messages in thread
From: Philippe Mathieu-Daudé via @ 2022-06-02 10:50 UTC (permalink / raw)
  To: Alex Bennée, peter.maydell, richard.henderson, qemu-stable
  Cc: qemu-devel, Daniel P. Berrangé,
	Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal

Cc'ing qemu-stable@

Thank you Daniel / Thomas / Alex!

On 1/6/22 20:05, Alex Bennée wrote:
> From: Daniel P. Berrangé <berrange@redhat.com>
> 
> To preserve CI shared runner credits we don't want to run
> pipelines on every push.
> 
> This sets up the config so that pipelines are never created
> for contributors by default. To override this the QEMU_CI
> variable can be set to a non-zero value. If set to 1, the
> pipeline will be created but all jobs will remain manually
> started. The contributor can selectively run jobs that they
> care about. If set to 2, the pipeline will be created and
> all jobs will immediately start.
> 
> This behavior can be controlled using push variables
> 
>    git push -o ci.variable=QEMU_CI=1
> 
> To make this more convenient define an alias
> 
>     git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
>     git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
> 
> Which lets you run
> 
>    git push-ci
> 
> to create the pipeline, or
> 
>    git push-ci-now
> 
> to create and run the pipeline
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> Message-Id: <20220526110705.59952-6-berrange@redhat.com>
> [AJB: fix typo, replicate alias tips in ci.rst]
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org>
> 
> diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
> index 0b4926e537..9118a61a17 100644
> --- a/docs/devel/ci-jobs.rst.inc
> +++ b/docs/devel/ci-jobs.rst.inc
> @@ -28,6 +28,32 @@ For further information about how to set these variables, please refer to::
>   
>     https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
>   
> +Setting aliases in your git config
> +----------------------------------
> +
> +You can use aliases to make it easier to push branches with different
> +CI configurations. For example define an alias for triggering CI:
> +
> +.. code::
> +
> +   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
> +   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
> +
> +Which lets you run:
> +
> +.. code::
> +
> +   git push-ci
> +
> +to create the pipeline, or:
> +
> +.. code::
> +
> +   git push-ci-now
> +
> +to create and run the pipeline
> +
> +
>   Variable naming and grouping
>   ----------------------------
>   
> @@ -98,6 +124,18 @@ Contributor controlled runtime variables
>   The following variables may be set by contributors to control
>   job execution
>   
> +QEMU_CI
> +~~~~~~~
> +
> +By default, no pipelines will be created on contributor forks
> +in order to preserve CI credits
> +
> +Set this variable to 1 to create the pipelines, but leave all
> +the jobs to be manually started from the UI
> +
> +Set this variable to 2 to create the pipelines and run all
> +the jobs immediately, as was historicaly behaviour
> +
>   QEMU_CI_AVOCADO_TESTING
>   ~~~~~~~~~~~~~~~~~~~~~~~
>   By default, tests using the Avocado framework are not run automatically in
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 4f091d5aad..f334f3ded7 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -28,6 +28,10 @@
>       - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
>         when: never
>   
> +    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> +      when: never
> +
>       # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
>       - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
>         when: never
> @@ -59,5 +63,10 @@
>       # an earlier criteria
>       #############################################################
>   
> +    # Forks pipeline jobs don't start automatically unless
> +    # QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> +      when: manual
> +
>       # Jobs can run if any jobs they depend on were successfull
>       - when: on_success



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 32/33] gitlab: don't run CI jobs in forks by default
  2022-06-02 10:50   ` Philippe Mathieu-Daudé via
@ 2022-06-06 11:01     ` Daniel P. Berrangé
  0 siblings, 0 replies; 45+ messages in thread
From: Daniel P. Berrangé @ 2022-06-06 11:01 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Alex Bennée, peter.maydell, richard.henderson, qemu-stable,
	qemu-devel, Thomas Huth, Wainer dos Santos Moschetta,
	Beraldo Leal

On Thu, Jun 02, 2022 at 12:50:21PM +0200, Philippe Mathieu-Daudé wrote:
> Cc'ing qemu-stable@

If you want this in stable, you'll need the 5 preceeding patches
too. I wouldn't be surprised if there are conflicts too. So it
might well need a custom set of patches spinning just for stable.

> 
> Thank you Daniel / Thomas / Alex!
> 
> On 1/6/22 20:05, Alex Bennée wrote:
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > 
> > To preserve CI shared runner credits we don't want to run
> > pipelines on every push.
> > 
> > This sets up the config so that pipelines are never created
> > for contributors by default. To override this the QEMU_CI
> > variable can be set to a non-zero value. If set to 1, the
> > pipeline will be created but all jobs will remain manually
> > started. The contributor can selectively run jobs that they
> > care about. If set to 2, the pipeline will be created and
> > all jobs will immediately start.
> > 
> > This behavior can be controlled using push variables
> > 
> >    git push -o ci.variable=QEMU_CI=1
> > 
> > To make this more convenient define an alias
> > 
> >     git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
> >     git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
> > 
> > Which lets you run
> > 
> >    git push-ci
> > 
> > to create the pipeline, or
> > 
> >    git push-ci-now
> > 
> > to create and run the pipeline
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > Message-Id: <20220526110705.59952-6-berrange@redhat.com>
> > [AJB: fix typo, replicate alias tips in ci.rst]
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > Reviewed-by: Thomas Huth <thuth@redhat.com>
> > Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org>
> > 
> > diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
> > index 0b4926e537..9118a61a17 100644
> > --- a/docs/devel/ci-jobs.rst.inc
> > +++ b/docs/devel/ci-jobs.rst.inc
> > @@ -28,6 +28,32 @@ For further information about how to set these variables, please refer to::
> >     https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
> > +Setting aliases in your git config
> > +----------------------------------
> > +
> > +You can use aliases to make it easier to push branches with different
> > +CI configurations. For example define an alias for triggering CI:
> > +
> > +.. code::
> > +
> > +   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
> > +   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
> > +
> > +Which lets you run:
> > +
> > +.. code::
> > +
> > +   git push-ci
> > +
> > +to create the pipeline, or:
> > +
> > +.. code::
> > +
> > +   git push-ci-now
> > +
> > +to create and run the pipeline
> > +
> > +
> >   Variable naming and grouping
> >   ----------------------------
> > @@ -98,6 +124,18 @@ Contributor controlled runtime variables
> >   The following variables may be set by contributors to control
> >   job execution
> > +QEMU_CI
> > +~~~~~~~
> > +
> > +By default, no pipelines will be created on contributor forks
> > +in order to preserve CI credits
> > +
> > +Set this variable to 1 to create the pipelines, but leave all
> > +the jobs to be manually started from the UI
> > +
> > +Set this variable to 2 to create the pipelines and run all
> > +the jobs immediately, as was historicaly behaviour
> > +
> >   QEMU_CI_AVOCADO_TESTING
> >   ~~~~~~~~~~~~~~~~~~~~~~~
> >   By default, tests using the Avocado framework are not run automatically in
> > diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> > index 4f091d5aad..f334f3ded7 100644
> > --- a/.gitlab-ci.d/base.yml
> > +++ b/.gitlab-ci.d/base.yml
> > @@ -28,6 +28,10 @@
> >       - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
> >         when: never
> > +    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
> > +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> > +      when: never
> > +
> >       # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
> >       - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> >         when: never
> > @@ -59,5 +63,10 @@
> >       # an earlier criteria
> >       #############################################################
> > +    # Forks pipeline jobs don't start automatically unless
> > +    # QEMU_CI=2 is set
> > +    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> > +      when: manual
> > +
> >       # Jobs can run if any jobs they depend on were successfull
> >       - when: on_success
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-01 18:05 ` [PULL 20/33] configure: handle host compiler in probe_target_compiler Alex Bennée
@ 2022-06-15 11:57   ` Matheus Kowalczuk Ferst
  2022-06-16  1:20     ` Alex Bennée
  2022-06-17 10:12     ` Paolo Bonzini
  0 siblings, 2 replies; 45+ messages in thread
From: Matheus Kowalczuk Ferst @ 2022-06-15 11:57 UTC (permalink / raw)
  To: Alex Bennée, peter.maydell, richard.henderson
  Cc: qemu-devel, Paolo Bonzini, Leandro Lupori

On 01/06/2022 15:05, Alex Bennée wrote:
> From: Paolo Bonzini <pbonzini@redhat.com>
> 
> In preparation for handling more binaries than just cc, handle
> the case of "probe_target_compiler $cpu" directly in the function,
> setting the target_* variables based on the ones that are used to
> build QEMU.  The clang check also needs to be moved after this
> fallback.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>

Hi,

After this patch, a clean build in ppc64le hosts will not build 
ppc64{,le}-linux-user tests with "make check-tcg"

> 
> diff --git a/configure b/configure
> index fbf6d39f96..217c8b3cac 100755
> --- a/configure
> +++ b/configure
> @@ -954,10 +954,6 @@ case $git_submodules_action in
>       ;;
>   esac
> 
> -if eval test -z "\${cross_cc_$cpu}"; then
> -    eval "cross_cc_${cpu}=\$cc"
> -fi
> -
>   default_target_list=""
>   mak_wilds=""
> 
> @@ -2008,13 +2004,6 @@ probe_target_compiler() {
>     if eval test -n "\"\${cross_cc_$1}\""; then
>       if eval has "\"\${cross_cc_$1}\""; then
>         eval "target_cc=\"\${cross_cc_$1}\""
> -      case $1 in
> -        i386|x86_64)
> -          if $target_cc --version | grep -qi "clang"; then
> -            unset target_cc
> -          fi
> -          ;;
> -      esac
>       fi
>     fi
>     if eval test -n "\"\${cross_as_$1}\""; then
> @@ -2027,6 +2016,20 @@ probe_target_compiler() {
>         eval "target_ld=\"\${cross_ld_$1}\""
>       fi
>     fi
> +  if test "$1" = $cpu; then > +    : ${target_cc:=$cc}
> +    : ${target_as:=$as}
> +    : ${target_ld:=$ld}
> +  fi

$cpu is normalized[1] to ppc64 on little-endian hosts, so 
ppc64le-linux-user will not have $target_{cc,as,ld} set, and 
ppc64-linux-user will have them set to a toolchain that may not support 
-mbig-endian. I suppose we have a similar problem with MIPS targets on 
MIPS hosts.

[1] 
https://gitlab.com/qemu-project/qemu/-/blob/2ad60f6f8c12ca0acd8834fdd70e088361b8791f/configure#L611

-- 
Matheus K. Ferst
Instituto de Pesquisas ELDORADO <http://www.eldorado.org.br/>
Analista de Software
Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-15 11:57   ` Matheus Kowalczuk Ferst
@ 2022-06-16  1:20     ` Alex Bennée
  2022-06-17 10:12     ` Paolo Bonzini
  1 sibling, 0 replies; 45+ messages in thread
From: Alex Bennée @ 2022-06-16  1:20 UTC (permalink / raw)
  To: Matheus Kowalczuk Ferst
  Cc: peter.maydell, richard.henderson, qemu-devel, Paolo Bonzini,
	Leandro Lupori


Matheus Kowalczuk Ferst <matheus.ferst@eldorado.org.br> writes:

> On 01/06/2022 15:05, Alex Bennée wrote:
>> From: Paolo Bonzini <pbonzini@redhat.com>
>> 
>> In preparation for handling more binaries than just cc, handle
>> the case of "probe_target_compiler $cpu" directly in the function,
>> setting the target_* variables based on the ones that are used to
>> build QEMU.  The clang check also needs to be moved after this
>> fallback.
>> 
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>
>
> Hi,
>
> After this patch, a clean build in ppc64le hosts will not build 
> ppc64{,le}-linux-user tests with "make check-tcg"
>
>> 
>> diff --git a/configure b/configure
>> index fbf6d39f96..217c8b3cac 100755
>> --- a/configure
>> +++ b/configure
>> @@ -954,10 +954,6 @@ case $git_submodules_action in
>>       ;;
>>   esac
>> 
>> -if eval test -z "\${cross_cc_$cpu}"; then
>> -    eval "cross_cc_${cpu}=\$cc"
>> -fi
>> -
>>   default_target_list=""
>>   mak_wilds=""
>> 
>> @@ -2008,13 +2004,6 @@ probe_target_compiler() {
>>     if eval test -n "\"\${cross_cc_$1}\""; then
>>       if eval has "\"\${cross_cc_$1}\""; then
>>         eval "target_cc=\"\${cross_cc_$1}\""
>> -      case $1 in
>> -        i386|x86_64)
>> -          if $target_cc --version | grep -qi "clang"; then
>> -            unset target_cc
>> -          fi
>> -          ;;
>> -      esac
>>       fi
>>     fi
>>     if eval test -n "\"\${cross_as_$1}\""; then
>> @@ -2027,6 +2016,20 @@ probe_target_compiler() {
>>         eval "target_ld=\"\${cross_ld_$1}\""
>>       fi
>>     fi
>> +  if test "$1" = $cpu; then > +    : ${target_cc:=$cc}
>> +    : ${target_as:=$as}
>> +    : ${target_ld:=$ld}
>> +  fi
>
> $cpu is normalized[1] to ppc64 on little-endian hosts, so 
> ppc64le-linux-user will not have $target_{cc,as,ld} set, and 
> ppc64-linux-user will have them set to a toolchain that may not support 
> -mbig-endian. I suppose we have a similar problem with MIPS targets on 
> MIPS hosts.

For now you can always explicitly tell configure about the host compiler
with:

 --cross-cc-ppc64le=gcc

but we should fix the broken detection. It seems the var cpu has an
overloaded meaning so I wonder if we just need an explicit host_cpu
setting when we normalize cpu?

>
> [1] 
> https://gitlab.com/qemu-project/qemu/-/blob/2ad60f6f8c12ca0acd8834fdd70e088361b8791f/configure#L611


-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-15 11:57   ` Matheus Kowalczuk Ferst
  2022-06-16  1:20     ` Alex Bennée
@ 2022-06-17 10:12     ` Paolo Bonzini
  2022-06-20 16:41       ` Matheus Kowalczuk Ferst
  1 sibling, 1 reply; 45+ messages in thread
From: Paolo Bonzini @ 2022-06-17 10:12 UTC (permalink / raw)
  To: Matheus Kowalczuk Ferst
  Cc: Alex Bennée, peter.maydell, richard.henderson, qemu-devel,
	Leandro Lupori

Hi Matheus,

could you please test the tests-tcg-next branch at
https://gitlab.com/bonzini/qemu?

Thanks,

Paolo

On Thu, Jun 16, 2022 at 3:23 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Matheus Kowalczuk Ferst <matheus.ferst@eldorado.org.br> writes:
>
> > On 01/06/2022 15:05, Alex Bennée wrote:
> >> From: Paolo Bonzini <pbonzini@redhat.com>
> >>
> >> In preparation for handling more binaries than just cc, handle
> >> the case of "probe_target_compiler $cpu" directly in the function,
> >> setting the target_* variables based on the ones that are used to
> >> build QEMU.  The clang check also needs to be moved after this
> >> fallback.
> >>
> >> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> >> Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>
> >
> > Hi,
> >
> > After this patch, a clean build in ppc64le hosts will not build
> > ppc64{,le}-linux-user tests with "make check-tcg"
> >
> >>
> >> diff --git a/configure b/configure
> >> index fbf6d39f96..217c8b3cac 100755
> >> --- a/configure
> >> +++ b/configure
> >> @@ -954,10 +954,6 @@ case $git_submodules_action in
> >>       ;;
> >>   esac
> >>
> >> -if eval test -z "\${cross_cc_$cpu}"; then
> >> -    eval "cross_cc_${cpu}=\$cc"
> >> -fi
> >> -
> >>   default_target_list=""
> >>   mak_wilds=""
> >>
> >> @@ -2008,13 +2004,6 @@ probe_target_compiler() {
> >>     if eval test -n "\"\${cross_cc_$1}\""; then
> >>       if eval has "\"\${cross_cc_$1}\""; then
> >>         eval "target_cc=\"\${cross_cc_$1}\""
> >> -      case $1 in
> >> -        i386|x86_64)
> >> -          if $target_cc --version | grep -qi "clang"; then
> >> -            unset target_cc
> >> -          fi
> >> -          ;;
> >> -      esac
> >>       fi
> >>     fi
> >>     if eval test -n "\"\${cross_as_$1}\""; then
> >> @@ -2027,6 +2016,20 @@ probe_target_compiler() {
> >>         eval "target_ld=\"\${cross_ld_$1}\""
> >>       fi
> >>     fi
> >> +  if test "$1" = $cpu; then > +    : ${target_cc:=$cc}
> >> +    : ${target_as:=$as}
> >> +    : ${target_ld:=$ld}
> >> +  fi
> >
> > $cpu is normalized[1] to ppc64 on little-endian hosts, so
> > ppc64le-linux-user will not have $target_{cc,as,ld} set, and
> > ppc64-linux-user will have them set to a toolchain that may not support
> > -mbig-endian. I suppose we have a similar problem with MIPS targets on
> > MIPS hosts.
>
> For now you can always explicitly tell configure about the host compiler
> with:
>
>  --cross-cc-ppc64le=gcc
>
> but we should fix the broken detection. It seems the var cpu has an
> overloaded meaning so I wonder if we just need an explicit host_cpu
> setting when we normalize cpu?
>
> >
> > [1]
> > https://gitlab.com/qemu-project/qemu/-/blob/2ad60f6f8c12ca0acd8834fdd70e088361b8791f/configure#L611
>
>
> --
> Alex Bennée
>

On Wed, Jun 15, 2022 at 1:57 PM Matheus Kowalczuk Ferst
<matheus.ferst@eldorado.org.br> wrote:
>
> On 01/06/2022 15:05, Alex Bennée wrote:
> > From: Paolo Bonzini <pbonzini@redhat.com>
> >
> > In preparation for handling more binaries than just cc, handle
> > the case of "probe_target_compiler $cpu" directly in the function,
> > setting the target_* variables based on the ones that are used to
> > build QEMU.  The clang check also needs to be moved after this
> > fallback.
> >
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> > Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> > Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>
>
> Hi,
>
> After this patch, a clean build in ppc64le hosts will not build
> ppc64{,le}-linux-user tests with "make check-tcg"
>
> >
> > diff --git a/configure b/configure
> > index fbf6d39f96..217c8b3cac 100755
> > --- a/configure
> > +++ b/configure
> > @@ -954,10 +954,6 @@ case $git_submodules_action in
> >       ;;
> >   esac
> >
> > -if eval test -z "\${cross_cc_$cpu}"; then
> > -    eval "cross_cc_${cpu}=\$cc"
> > -fi
> > -
> >   default_target_list=""
> >   mak_wilds=""
> >
> > @@ -2008,13 +2004,6 @@ probe_target_compiler() {
> >     if eval test -n "\"\${cross_cc_$1}\""; then
> >       if eval has "\"\${cross_cc_$1}\""; then
> >         eval "target_cc=\"\${cross_cc_$1}\""
> > -      case $1 in
> > -        i386|x86_64)
> > -          if $target_cc --version | grep -qi "clang"; then
> > -            unset target_cc
> > -          fi
> > -          ;;
> > -      esac
> >       fi
> >     fi
> >     if eval test -n "\"\${cross_as_$1}\""; then
> > @@ -2027,6 +2016,20 @@ probe_target_compiler() {
> >         eval "target_ld=\"\${cross_ld_$1}\""
> >       fi
> >     fi
> > +  if test "$1" = $cpu; then > +    : ${target_cc:=$cc}
> > +    : ${target_as:=$as}
> > +    : ${target_ld:=$ld}
> > +  fi
>
> $cpu is normalized[1] to ppc64 on little-endian hosts, so
> ppc64le-linux-user will not have $target_{cc,as,ld} set, and
> ppc64-linux-user will have them set to a toolchain that may not support
> -mbig-endian. I suppose we have a similar problem with MIPS targets on
> MIPS hosts.
>
> [1]
> https://gitlab.com/qemu-project/qemu/-/blob/2ad60f6f8c12ca0acd8834fdd70e088361b8791f/configure#L611
>
> --
> Matheus K. Ferst
> Instituto de Pesquisas ELDORADO <http://www.eldorado.org.br/>
> Analista de Software
> Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>



^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-17 10:12     ` Paolo Bonzini
@ 2022-06-20 16:41       ` Matheus Kowalczuk Ferst
  2022-06-21  1:38         ` Richard Henderson
  2022-06-21 16:58         ` Paolo Bonzini
  0 siblings, 2 replies; 45+ messages in thread
From: Matheus Kowalczuk Ferst @ 2022-06-20 16:41 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: Alex Bennée, peter.maydell, richard.henderson, qemu-devel,
	Leandro Lupori

On 17/06/2022 07:12, Paolo Bonzini wrote:
> Hi Matheus,
> 
> could you please test the tests-tcg-next branch at
> https://gitlab.com/bonzini/qemu?

At be6090bcac10, it works if no BE toolchain is present. Otherwise, the 
script probes powerpc64-linux-gnu-gcc instead of the native tools for 
ppc64le-linux-user, and then do_compiler fails because the 
$target_cflags contains -mlittle-endian.

> 
> On Thu, Jun 16, 2022 at 3:23 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Matheus Kowalczuk Ferst <matheus.ferst@eldorado.org.br> writes:
>>
>>> On 01/06/2022 15:05, Alex Bennée wrote:
>>>> From: Paolo Bonzini <pbonzini@redhat.com>
>>>>
>>>> In preparation for handling more binaries than just cc, handle
>>>> the case of "probe_target_compiler $cpu" directly in the function,
>>>> setting the target_* variables based on the ones that are used to
>>>> build QEMU.  The clang check also needs to be moved after this
>>>> fallback.
>>>>
>>>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>> Message-Id: <20220517092616.1272238-10-pbonzini@redhat.com>
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> Message-Id: <20220527153603.887929-21-alex.bennee@linaro.org>
>>>
>>> Hi,
>>>
>>> After this patch, a clean build in ppc64le hosts will not build
>>> ppc64{,le}-linux-user tests with "make check-tcg"
>>>
>>>>
>>>> diff --git a/configure b/configure
>>>> index fbf6d39f96..217c8b3cac 100755
>>>> --- a/configure
>>>> +++ b/configure
>>>> @@ -954,10 +954,6 @@ case $git_submodules_action in
>>>>        ;;
>>>>    esac
>>>>
>>>> -if eval test -z "\${cross_cc_$cpu}"; then
>>>> -    eval "cross_cc_${cpu}=\$cc"
>>>> -fi
>>>> -
>>>>    default_target_list=""
>>>>    mak_wilds=""
>>>>
>>>> @@ -2008,13 +2004,6 @@ probe_target_compiler() {
>>>>      if eval test -n "\"\${cross_cc_$1}\""; then
>>>>        if eval has "\"\${cross_cc_$1}\""; then
>>>>          eval "target_cc=\"\${cross_cc_$1}\""
>>>> -      case $1 in
>>>> -        i386|x86_64)
>>>> -          if $target_cc --version | grep -qi "clang"; then
>>>> -            unset target_cc
>>>> -          fi
>>>> -          ;;
>>>> -      esac
>>>>        fi
>>>>      fi
>>>>      if eval test -n "\"\${cross_as_$1}\""; then
>>>> @@ -2027,6 +2016,20 @@ probe_target_compiler() {
>>>>          eval "target_ld=\"\${cross_ld_$1}\""
>>>>        fi
>>>>      fi
>>>> +  if test "$1" = $cpu; then > +    : ${target_cc:=$cc}
>>>> +    : ${target_as:=$as}
>>>> +    : ${target_ld:=$ld}
>>>> +  fi
>>>
>>> $cpu is normalized[1] to ppc64 on little-endian hosts, so
>>> ppc64le-linux-user will not have $target_{cc,as,ld} set, and
>>> ppc64-linux-user will have them set to a toolchain that may not support
>>> -mbig-endian. I suppose we have a similar problem with MIPS targets on
>>> MIPS hosts.
>>
>> For now you can always explicitly tell configure about the host compiler
>> with:
>>
>>   --cross-cc-ppc64le=gcc
>>
>> but we should fix the broken detection.

With this workaround we can run the tests on master. However, it works 
because linux-user tests only use CC. I wonder if the --cross-cc-* 
options will remain useful in the future; it seems that only 
--cross-prefix-* should be used now that more tools are exported in the 
config-<target>.mak

>> It seems the var cpu has an
>> overloaded meaning so I wonder if we just need an explicit host_cpu
>> setting when we normalize cpu?
>>

AFAICT, $cpu is always host-related, but after the normalization it 
seems more like the architecture than the CPU name... Maybe we should 
call them $host_cpu and $host_arch?

>>>
>>> [1]
>>> https://gitlab.com/qemu-project/qemu/-/blob/2ad60f6f8c12ca0acd8834fdd70e088361b8791f/configure#L611
>>
>>
>> --
>> Alex Bennée
>>

Thanks,
Matheus K. Ferst
Instituto de Pesquisas ELDORADO <http://www.eldorado.org.br/>
Analista de Software
Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>

^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-20 16:41       ` Matheus Kowalczuk Ferst
@ 2022-06-21  1:38         ` Richard Henderson
  2022-06-21 16:58         ` Paolo Bonzini
  1 sibling, 0 replies; 45+ messages in thread
From: Richard Henderson @ 2022-06-21  1:38 UTC (permalink / raw)
  To: Matheus Kowalczuk Ferst, Paolo Bonzini
  Cc: Alex Bennée, peter.maydell, qemu-devel, Leandro Lupori

On 6/20/22 09:41, Matheus Kowalczuk Ferst wrote:
> On 17/06/2022 07:12, Paolo Bonzini wrote:
>> Hi Matheus,
>>
>> could you please test the tests-tcg-next branch at
>> https://gitlab.com/bonzini/qemu?
> 
> At be6090bcac10, it works if no BE toolchain is present. Otherwise, the
> script probes powerpc64-linux-gnu-gcc instead of the native tools for
> ppc64le-linux-user, and then do_compiler fails because the
> $target_cflags contains -mlittle-endian.

Also testing this branch, -m32 may not be supported:

make[1]: Entering directory `/home/rth/qemu/bld/pc-bios/vof'

cc -m32 -mbig-endian -mcpu=power4 -c -o entry.o /home/rth/qemu/src/pc-bios/vof/entry.S

cc1: error: ‘-m32’ not supported in this configuration

make[1]: *** [entry.o] Error 1



This is with a self-built compiler, rather than distro packaged, but with default 
configure options to gcc.


r~


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-20 16:41       ` Matheus Kowalczuk Ferst
  2022-06-21  1:38         ` Richard Henderson
@ 2022-06-21 16:58         ` Paolo Bonzini
  2022-06-21 18:30           ` Richard Henderson
  1 sibling, 1 reply; 45+ messages in thread
From: Paolo Bonzini @ 2022-06-21 16:58 UTC (permalink / raw)
  To: Matheus Kowalczuk Ferst
  Cc: Alex Bennée, peter.maydell, richard.henderson, qemu-devel,
	Leandro Lupori

On 6/20/22 18:41, Matheus Kowalczuk Ferst wrote:
> On 17/06/2022 07:12, Paolo Bonzini wrote:
>> Hi Matheus,
>>
>> could you please test the tests-tcg-next branch at
>> https://gitlab.com/bonzini/qemu?
> At be6090bcac10, it works if no BE toolchain is present. Otherwise, the
> script probes powerpc64-linux-gnu-gcc instead of the native tools for
> ppc64le-linux-user, and then do_compiler fails because the
> $target_cflags contains -mlittle-endian.
> 

So the problem in that case is that powerpc64-linux-gnu-gcc is _not_
biarch and thus does not support -mlittle-endian?  (I thought PPC
compilers were all 32/64-bit and bi-endian).

If so, I guess this would fix your case

diff --git a/configure b/configure
index 0fd2838e82..6ad888ed97 100755
--- a/configure
+++ b/configure
@@ -2041,22 +2041,6 @@ probe_target_compiler() {
      : ${container_cross_strip:=${container_cross_prefix}strip}
    done

-  eval "target_cflags=\${cross_cc_cflags_$1}"
-  if eval test -n "\"\${cross_cc_$1}\""; then
-    if eval has "\"\${cross_cc_$1}\""; then
-      eval "target_cc=\"\${cross_cc_$1}\""
-    fi
-  else
-    compute_target_variable $1 target_cc gcc
-  fi
-  target_ccas=$target_cc
-  compute_target_variable $1 target_ar ar
-  compute_target_variable $1 target_as as
-  compute_target_variable $1 target_ld ld
-  compute_target_variable $1 target_nm nm
-  compute_target_variable $1 target_objcopy objcopy
-  compute_target_variable $1 target_ranlib ranlib
-  compute_target_variable $1 target_strip strip
    case "$1:$cpu" in
      aarch64_be:aarch64 | \
      armeb:arm | \
@@ -2065,17 +2049,35 @@ probe_target_compiler() {
      ppc*:ppc64 | \
      sparc:sparc64 | \
      "$cpu:$cpu")
-      : ${target_cc:=$cc}
-      : ${target_ccas:=$ccas}
-      : ${target_as:=$as}
-      : ${target_ld:=$ld}
-      : ${target_ar:=$ar}
-      : ${target_as:=$as}
-      : ${target_ld:=$ld}
-      : ${target_nm:=$nm}
-      : ${target_objcopy:=$objcopy}
-      : ${target_ranlib:=$ranlib}
-      : ${target_strip:=$strip}
+      target_cc=$cc
+      target_ccas=$ccas
+      target_as=$as
+      target_ld=$ld
+      target_ar=$ar
+      target_as=$as
+      target_ld=$ld
+      target_nm=$nm
+      target_objcopy=$objcopy
+      target_ranlib=$ranlib
+      target_strip=$strip
+      ;;
+    *)
+      eval "target_cflags=\${cross_cc_cflags_$1}"
+      if eval test -n "\"\${cross_cc_$1}\""; then
+        if eval has "\"\${cross_cc_$1}\""; then
+          eval "target_cc=\"\${cross_cc_$1}\""
+        fi
+      else
+        compute_target_variable $1 target_cc gcc
+      fi
+      target_ccas=$target_cc
+      compute_target_variable $1 target_ar ar
+      compute_target_variable $1 target_as as
+      compute_target_variable $1 target_ld ld
+      compute_target_variable $1 target_nm nm
+      compute_target_variable $1 target_objcopy objcopy
+      compute_target_variable $1 target_ranlib ranlib
+      compute_target_variable $1 target_strip strip
        ;;
    esac
    if test -n "$target_cc"; then

but it would break the opposite of compiling tests/tcg QEMU
with a big-endian non-biarch compiler.

In any case, the series is already a step forward so I've sent it out.

Paolo


^ permalink raw reply related	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-21 16:58         ` Paolo Bonzini
@ 2022-06-21 18:30           ` Richard Henderson
  2022-06-22  8:39             ` Paolo Bonzini
  0 siblings, 1 reply; 45+ messages in thread
From: Richard Henderson @ 2022-06-21 18:30 UTC (permalink / raw)
  To: Paolo Bonzini, Matheus Kowalczuk Ferst
  Cc: Alex Bennée, peter.maydell, qemu-devel, Leandro Lupori

On 6/21/22 09:58, Paolo Bonzini wrote:
> On 6/20/22 18:41, Matheus Kowalczuk Ferst wrote:
>> On 17/06/2022 07:12, Paolo Bonzini wrote:
>>> Hi Matheus,
>>>
>>> could you please test the tests-tcg-next branch at
>>> https://gitlab.com/bonzini/qemu?
>> At be6090bcac10, it works if no BE toolchain is present. Otherwise, the
>> script probes powerpc64-linux-gnu-gcc instead of the native tools for
>> ppc64le-linux-user, and then do_compiler fails because the
>> $target_cflags contains -mlittle-endian.
>>
> 
> So the problem in that case is that powerpc64-linux-gnu-gcc is _not_
> biarch and thus does not support -mlittle-endian?  (I thought PPC
> compilers were all 32/64-bit and bi-endian).

Nit: this is ppc64le-linux-gnu-gcc, built for gcc135, a power9 ppc64le host.  It *does* 
support -mbig-endian, but not -m32.

At least gcc11 as release was not biarch, with no special configure arguments.  I can try 
upgrading to the current gcc12 release to see if that changed...


r~


^ permalink raw reply	[flat|nested] 45+ messages in thread

* Re: [PULL 20/33] configure: handle host compiler in probe_target_compiler
  2022-06-21 18:30           ` Richard Henderson
@ 2022-06-22  8:39             ` Paolo Bonzini
  0 siblings, 0 replies; 45+ messages in thread
From: Paolo Bonzini @ 2022-06-22  8:39 UTC (permalink / raw)
  To: Richard Henderson, Matheus Kowalczuk Ferst
  Cc: Alex Bennée, peter.maydell, qemu-devel, Leandro Lupori

On 6/21/22 20:30, Richard Henderson wrote:
> On 6/21/22 09:58, Paolo Bonzini wrote:
>> On 6/20/22 18:41, Matheus Kowalczuk Ferst wrote:
>>> On 17/06/2022 07:12, Paolo Bonzini wrote:
>>>> Hi Matheus,
>>>>
>>>> could you please test the tests-tcg-next branch at
>>>> https://gitlab.com/bonzini/qemu?
>>> At be6090bcac10, it works if no BE toolchain is present. Otherwise, the
>>> script probes powerpc64-linux-gnu-gcc instead of the native tools for
>>> ppc64le-linux-user, and then do_compiler fails because the
>>> $target_cflags contains -mlittle-endian.
>>>
>>
>> So the problem in that case is that powerpc64-linux-gnu-gcc is _not_
>> biarch and thus does not support -mlittle-endian?  (I thought PPC
>> compilers were all 32/64-bit and bi-endian).
> 
> Nit: this is ppc64le-linux-gnu-gcc, built for gcc135, a power9 ppc64le 
> host.  It *does* support -mbig-endian, but not -m32.
> 
> At least gcc11 as release was not biarch, with no special configure 
> arguments.  I can try upgrading to the current gcc12 release to see if 
> that changed...

Ok, yesterday I had tested with RHEL and there the ppc64le compiler is 
bi-arch and bi-endian, but multilibs are disabled.  So it cannot build 
32-bit hosted binaries like those for tests/tcg, only freestanding 
binaries (vof).

On the other hand the powerpc64-linux-gnu-gcc binary from the cross-gcc 
package is theoretically multilib-friendly, but it cannot find the CRT 
files on the host because they are not in the .../le multilib subdirectory.

The simplest way out is to just test both native and cross.  There is 
already code to check for a working compiler in the tests/tcg stanza, 
and it can be moved to probe_target_compiler.

Paolo


^ permalink raw reply	[flat|nested] 45+ messages in thread

end of thread, other threads:[~2022-06-22  8:40 UTC | newest]

Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-01 18:05 [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Alex Bennée
2022-06-01 18:05 ` [PULL 01/33] .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages Alex Bennée
2022-06-01 18:05 ` [PULL 02/33] .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job Alex Bennée
2022-06-01 18:05 ` [PULL 03/33] gitlab-ci: add meson JUnit test result into report Alex Bennée
2022-06-01 18:05 ` [PULL 04/33] meson.build: fix summary display of test compilers Alex Bennée
2022-06-01 18:05 ` [PULL 05/33] tests/lcitool: fix up indentation to correct style Alex Bennée
2022-06-01 18:05 ` [PULL 06/33] tests/docker: update debian-armhf-cross with lcitool Alex Bennée
2022-06-01 18:05 ` [PULL 07/33] tests/docker: update debian-armel-cross " Alex Bennée
2022-06-01 18:05 ` [PULL 08/33] tests/docker: update debian-mipsel-cross " Alex Bennée
2022-06-01 18:05 ` [PULL 09/33] tests/docker: update debian-mips64el-cross " Alex Bennée
2022-06-01 18:05 ` [PULL 10/33] tests/docker: update debian-ppc64el-cross " Alex Bennée
2022-06-01 18:05 ` [PULL 11/33] tests/docker: update debian-amd64 " Alex Bennée
2022-06-01 18:05 ` [PULL 12/33] configure: do not define or use the CPP variable Alex Bennée
2022-06-01 18:05 ` [PULL 13/33] build: clean up ninja invocation Alex Bennée
2022-06-01 18:05 ` [PULL 14/33] build: add a more generic way to specify make->ninja dependencies Alex Bennée
2022-06-01 18:05 ` [PULL 15/33] build: do a full build before running TCG tests Alex Bennée
2022-06-01 18:05 ` [PULL 16/33] configure, meson: move symlinking of ROMs to meson Alex Bennée
2022-06-01 18:05 ` [PULL 17/33] tests/tcg: correct target CPU for sparc32 Alex Bennée
2022-06-01 18:05 ` [PULL 18/33] tests/tcg: merge configure.sh back into main configure script Alex Bennée
2022-06-01 18:05 ` [PULL 19/33] configure: add missing cross compiler fallbacks Alex Bennée
2022-06-01 18:05 ` [PULL 20/33] configure: handle host compiler in probe_target_compiler Alex Bennée
2022-06-15 11:57   ` Matheus Kowalczuk Ferst
2022-06-16  1:20     ` Alex Bennée
2022-06-17 10:12     ` Paolo Bonzini
2022-06-20 16:41       ` Matheus Kowalczuk Ferst
2022-06-21  1:38         ` Richard Henderson
2022-06-21 16:58         ` Paolo Bonzini
2022-06-21 18:30           ` Richard Henderson
2022-06-22  8:39             ` Paolo Bonzini
2022-06-01 18:05 ` [PULL 21/33] configure: introduce --cross-prefix-*= Alex Bennée
2022-06-01 18:05 ` [PULL 22/33] configure: include more binutils in tests/tcg makefile Alex Bennée
2022-06-01 18:05 ` [PULL 23/33] configure: move symlink configuration earlier Alex Bennée
2022-06-01 18:05 ` [PULL 24/33] configure: enable cross-compilation of s390-ccw Alex Bennée
2022-06-01 18:05 ` [PULL 25/33] configure: enable cross-compilation of optionrom Alex Bennée
2022-06-01 18:05 ` [PULL 26/33] configure: enable cross compilation of vof Alex Bennée
2022-06-01 18:05 ` [PULL 27/33] configure: remove unused variables from config-host.mak Alex Bennée
2022-06-01 18:05 ` [PULL 28/33] gitlab: introduce a common base job template Alex Bennée
2022-06-01 18:05 ` [PULL 29/33] gitlab: convert Cirrus jobs to .base_job_template Alex Bennée
2022-06-01 18:05 ` [PULL 30/33] gitlab: convert static checks " Alex Bennée
2022-06-01 18:05 ` [PULL 31/33] gitlab: convert build/container jobs " Alex Bennée
2022-06-01 18:05 ` [PULL 32/33] gitlab: don't run CI jobs in forks by default Alex Bennée
2022-06-02 10:50   ` Philippe Mathieu-Daudé via
2022-06-06 11:01     ` Daniel P. Berrangé
2022-06-01 18:05 ` [PULL 33/33] docs/devel: clean-up the CI links in the docs Alex Bennée
2022-06-01 21:21 ` [PULL 00/33] testing updates (gitlab, junit, lcitool, x-compile) Richard Henderson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.