[PATCH v2 00/18] testing and semihosting pre-PR

[PATCH v2 00/18] testing and semihosting pre-PR

[Alex Bennée]

Hi,

This is the current state of my testing and semihosting trees which I
intend to generate a PR for next week. I'd like to have a clean plate
by the time we hit softfreeze. I'll probably have to drop the sha512 vector
test patches as they break TCI unless someone can figure out what is
going wrong.

The following patches still need review:

 - tests/tcg: completely disable threadcount for sh4
 - tests/tcg: add sha512 test
 - tests/tcg/ppc64: clean-up handling of byte-reverse
 - gitlab: add a new aarch32 custom runner definition
 - scripts/ci: allow for a secondary runner
 - scripts/ci: add build env rules for aarch32 on aarch64
 - tests/docker: update debian-s390x-cross with lcitool
 - tests/lcitool: update to latest version
 - tests/docker: restore TESTS/IMAGES filtering

Alex Bennée (17):
  tests/docker: restore TESTS/IMAGES filtering
  tests/docker: add NOUSER for alpine image
  tests/lcitool: update to latest version
  tests/docker: update debian-arm64-cross with lci-tool
  tests/docker: update debian-s390x-cross with lcitool
  tests/docker: introduce debian-riscv64-test-cross
  scripts/ci: add build env rules for aarch32 on aarch64
  scripts/ci: allow for a secondary runner
  gitlab: add a new aarch32 custom runner definition
  tests/tcg/ppc64: clean-up handling of byte-reverse
  tests/tcg: build sha1-vector with O3 and compare
  tests/tcg: add sha512 test
  tests/tcg: add vectorised sha512 versions
  gitlab: upgrade the job definition for s390x to 20.04
  tests/tcg: completely disable threadcount for sh4
  semihosting/arm-compat: replace heuristic for softmmu SYS_HEAPINFO
  tests/tcg: port SYS_HEAPINFO to a system test

Thomas Huth (1):
  travis.yml: Update the s390x jobs to Ubuntu Focal

 docs/devel/ci-jobs.rst.inc                    |   7 +
 include/hw/loader.h                           |  14 +
 hw/core/loader.c                              |  86 ++
 semihosting/arm-compat-semi.c                 | 124 +--
 tests/tcg/aarch64/system/semiheap.c           |  93 ++
 tests/tcg/multiarch/sha512.c                  | 990 ++++++++++++++++++
 .gitlab-ci.d/container-cross.yml              |  20 +-
 .gitlab-ci.d/custom-runners.yml               |   2 +-
 ...18.04-s390x.yml => ubuntu-20.04-s390x.yml} |  28 +-
 .../custom-runners/ubuntu-20.40-aarch32.yml   |  23 +
 .travis.yml                                   |  12 +-
 MAINTAINERS                                   |   1 +
 scripts/ci/setup/build-environment.yml        |  25 +
 scripts/ci/setup/gitlab-runner.yml            |  38 +
 tests/docker/Makefile.include                 |  29 +-
 .../dockerfiles/debian-arm64-cross.docker     | 186 +++-
 .../debian-arm64-test-cross.docker            |  13 -
 .../debian-riscv64-test-cross.docker          |  12 +
 .../dockerfiles/debian-s390x-cross.docker     | 181 +++-
 tests/docker/dockerfiles/opensuse-leap.docker |   3 +-
 tests/docker/dockerfiles/ubuntu1804.docker    |   3 +-
 tests/docker/dockerfiles/ubuntu2004.docker    |   3 +-
 tests/lcitool/libvirt-ci                      |   2 +-
 tests/lcitool/refresh                         |  16 +
 tests/tcg/aarch64/Makefile.target             |  17 +
 tests/tcg/arm/Makefile.target                 |  17 +
 tests/tcg/configure.sh                        |   4 +-
 tests/tcg/i386/Makefile.target                |   6 +
 tests/tcg/ppc64/Makefile.target               |  20 +-
 tests/tcg/ppc64le/Makefile.target             |   9 +-
 tests/tcg/s390x/Makefile.target               |   9 +
 tests/tcg/sh4/Makefile.target                 |   2 +
 tests/tcg/x86_64/Makefile.target              |   7 +
 33 files changed, 1813 insertions(+), 189 deletions(-)
 create mode 100644 tests/tcg/aarch64/system/semiheap.c
 create mode 100644 tests/tcg/multiarch/sha512.c
 rename .gitlab-ci.d/custom-runners/{ubuntu-18.04-s390x.yml => ubuntu-20.04-s390x.yml} (87%)
 create mode 100644 .gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml
 delete mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-riscv64-test-cross.docker

-- 
2.30.2

[PATCH v2 01/18] tests/docker: restore TESTS/IMAGES filtering

[Alex Bennée]

This was broken in the re-factor:

  e86c9a64f4 ("tests/docker/Makefile.include: add a generic docker-run target")

Rather than unwind the changes just apply the filters to the total set
of available images and tests. That way we don't inadvertently build
images only not to use them later.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Alex Williamson <alex.williamson@redhat.com>
Message-Id: <20220211160309.335014-2-alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f1a0c5db7a..0ec59b2193 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -8,13 +8,19 @@ COMMA := ,
 
 HOST_ARCH = $(if $(ARCH),$(ARCH),$(shell uname -m))
 
+# These variables can be set by the user to limit the set of docker
+# images and tests to a more restricted subset
+TESTS ?= %
+IMAGES ?= %
+
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
 DOCKER_PARTIAL_IMAGES := debian10 debian11
 # we don't directly build virtual images (they are used to build other images)
 DOCKER_VIRTUAL_IMAGES := debian-bootstrap debian-toolchain empty
-DOCKER_IMAGES := $(sort $(filter-out $(DOCKER_VIRTUAL_IMAGES), $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
+__IMAGES := $(sort $(filter-out $(DOCKER_VIRTUAL_IMAGES), $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
+DOCKER_IMAGES := $(if $(IMAGES), $(filter $(IMAGES), $(__IMAGES)), $(__IMAGES))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
 DOCKER_CCACHE_DIR := $$HOME/.cache/qemu-docker-ccache
@@ -23,16 +29,14 @@ DOCKER_DEFAULT_REGISTRY := registry.gitlab.com/qemu-project/qemu
 endif
 DOCKER_REGISTRY := $(if $(REGISTRY),$(REGISTRY),$(DOCKER_DEFAULT_REGISTRY))
 
-DOCKER_TESTS := $(notdir $(shell \
-	find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f))
+__TESTS := $(notdir $(shell \
+		find $(SRC_PATH)/tests/docker/ -name 'test-*' -type f))
+DOCKER_TESTS := $(if $(TESTS), $(filter $(TESTS), $(__TESTS)), $(__TESTS))
 
 ENGINE := auto
 
 DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(ENGINE)
 
-TESTS ?= %
-IMAGES ?= %
-
 CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
 DOCKER_SRC_COPY := $(BUILD_DIR)/docker-src.$(CUR_TIME)
 
@@ -274,8 +278,8 @@ endif
 	@echo '    TARGET_LIST=a,b,c    Override target list in builds.'
 	@echo '    EXTRA_CONFIGURE_OPTS="..."'
 	@echo '                         Extra configure options.'
-	@echo '    IMAGES="a b c ..":   Filters which images to build or run.'
-	@echo '    TESTS="x y z .."     Filters which tests to run (for docker-test).'
+	@echo '    IMAGES="a b c ..":   Restrict available images to subset.'
+	@echo '    TESTS="x y z .."     Restrict available tests to subset.'
 	@echo '    J=[0..9]*            Overrides the -jN parameter for make commands'
 	@echo '                         (default is 1)'
 	@echo '    DEBUG=1              Stop and drop to shell in the created container'
-- 
2.30.2

Re: [PATCH v2 01/18] tests/docker: restore TESTS/IMAGES filtering

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> This was broken in the re-factor:
> 
>    e86c9a64f4 ("tests/docker/Makefile.include: add a generic docker-run target")
> 
> Rather than unwind the changes just apply the filters to the total set
> of available images and tests. That way we don't inadvertently build
> images only not to use them later.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Reported-by: Alex Williamson<alex.williamson@redhat.com>
> Message-Id:<20220211160309.335014-2-alex.bennee@linaro.org>
> ---
>   tests/docker/Makefile.include | 20 ++++++++++++--------
>   1 file changed, 12 insertions(+), 8 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 02/18] tests/docker: add NOUSER for alpine image

[Alex Bennée]

The alpine image doesn't have a standard useradd binary so disable
this convenience feature for it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220211160309.335014-3-alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 0ec59b2193..286f0ac5b5 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -158,6 +158,9 @@ docker-image-debian-native: DOCKER_REGISTRY=
 docker-image-debian10: NOUSER=1
 docker-image-debian11: NOUSER=1
 
+# alpine has no adduser
+docker-image-alpine: NOUSER=1
+
 #
 # The build rule for hexagon-cross is special in so far for most of
 # the time we don't want to build it. While dockers caching does avoid
-- 
2.30.2

Re: [PATCH v2 02/18] tests/docker: add NOUSER for alpine image

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> The alpine image doesn't have a standard useradd binary so disable
> this convenience feature for it.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Reviewed-by: Daniel P. Berrangé<berrange@redhat.com>
> Message-Id:<20220211160309.335014-3-alex.bennee@linaro.org>
> ---
>   tests/docker/Makefile.include | 3 +++
>   1 file changed, 3 insertions(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 03/18] tests/lcitool: update to latest version

[Alex Bennée]

We will need an update shortly for some new images.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220211160309.335014-4-alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/opensuse-leap.docker | 3 +--
 tests/docker/dockerfiles/ubuntu1804.docker    | 3 +--
 tests/docker/dockerfiles/ubuntu2004.docker    | 3 +--
 tests/lcitool/libvirt-ci                      | 2 +-
 4 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 1b78d8369a..e1ad9434a3 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -127,8 +127,7 @@ RUN zypper update -y && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-RUN pip3 install \
-         meson==0.56.0
+RUN pip3 install meson==0.56.0
 
 ENV LANG "en_US.UTF-8"
 ENV MAKE "/usr/bin/make"
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index 699f2dfc6a..0a622b467c 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -134,8 +134,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-RUN pip3 install \
-         meson==0.56.0
+RUN pip3 install meson==0.56.0
 
 ENV LANG "en_US.UTF-8"
 ENV MAKE "/usr/bin/make"
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 87513125b8..b9d06cb040 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -136,8 +136,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-RUN pip3 install \
-         meson==0.56.0
+RUN pip3 install meson==0.56.0
 
 ENV LANG "en_US.UTF-8"
 ENV MAKE "/usr/bin/make"
diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
index 6dd9b6fab1..f83b916d5e 160000
--- a/tests/lcitool/libvirt-ci
+++ b/tests/lcitool/libvirt-ci
@@ -1 +1 @@
-Subproject commit 6dd9b6fab1fe081b16bc975485d7a02c81ba5fbe
+Subproject commit f83b916d5efa4bd33fbf4b7ea41bf6d535cc63fb
-- 
2.30.2

Re: [PATCH v2 03/18] tests/lcitool: update to latest version

[Peter Maydell]

On Fri, 25 Feb 2022 at 17:36, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> We will need an update shortly for some new images.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20220211160309.335014-4-alex.bennee@linaro.org>
> ---
>  tests/docker/dockerfiles/opensuse-leap.docker | 3 +--
>  tests/docker/dockerfiles/ubuntu1804.docker    | 3 +--
>  tests/docker/dockerfiles/ubuntu2004.docker    | 3 +--
>  tests/lcitool/libvirt-ci                      | 2 +-
>  4 files changed, 4 insertions(+), 7 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
> index 1b78d8369a..e1ad9434a3 100644
> --- a/tests/docker/dockerfiles/opensuse-leap.docker
> +++ b/tests/docker/dockerfiles/opensuse-leap.docker
> @@ -127,8 +127,7 @@ RUN zypper update -y && \
>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
>
> -RUN pip3 install \
> -         meson==0.56.0
> +RUN pip3 install meson==0.56.0

Why are these formatting changes to the dockerfiles in this commit ?

thanks
-- PMM

Re: [PATCH v2 03/18] tests/lcitool: update to latest version

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Fri, 25 Feb 2022 at 17:36, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> We will need an update shortly for some new images.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Message-Id: <20220211160309.335014-4-alex.bennee@linaro.org>
>> ---
>>  tests/docker/dockerfiles/opensuse-leap.docker | 3 +--
>>  tests/docker/dockerfiles/ubuntu1804.docker    | 3 +--
>>  tests/docker/dockerfiles/ubuntu2004.docker    | 3 +--
>>  tests/lcitool/libvirt-ci                      | 2 +-
>>  4 files changed, 4 insertions(+), 7 deletions(-)
>>
>> diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
>> index 1b78d8369a..e1ad9434a3 100644
>> --- a/tests/docker/dockerfiles/opensuse-leap.docker
>> +++ b/tests/docker/dockerfiles/opensuse-leap.docker
>> @@ -127,8 +127,7 @@ RUN zypper update -y && \
>>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
>>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
>>
>> -RUN pip3 install \
>> -         meson==0.56.0
>> +RUN pip3 install meson==0.56.0
>
> Why are these formatting changes to the dockerfiles in this commit ?

The latest lcitool I assume didn't see the need for the continuation
line. I ran:

  make lcitool-refresh

when I updated.

-- 
Alex Bennée

Re: [PATCH v2 03/18] tests/lcitool: update to latest version

[Peter Maydell]

On Mon, 28 Feb 2022 at 16:46, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Fri, 25 Feb 2022 at 17:36, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >> We will need an update shortly for some new images.
> >>
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> Message-Id: <20220211160309.335014-4-alex.bennee@linaro.org>
> >> ---
> >>  tests/docker/dockerfiles/opensuse-leap.docker | 3 +--
> >>  tests/docker/dockerfiles/ubuntu1804.docker    | 3 +--
> >>  tests/docker/dockerfiles/ubuntu2004.docker    | 3 +--
> >>  tests/lcitool/libvirt-ci                      | 2 +-
> >>  4 files changed, 4 insertions(+), 7 deletions(-)
> >>
> >> diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
> >> index 1b78d8369a..e1ad9434a3 100644
> >> --- a/tests/docker/dockerfiles/opensuse-leap.docker
> >> +++ b/tests/docker/dockerfiles/opensuse-leap.docker
> >> @@ -127,8 +127,7 @@ RUN zypper update -y && \
> >>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
> >>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
> >>
> >> -RUN pip3 install \
> >> -         meson==0.56.0
> >> +RUN pip3 install meson==0.56.0
> >
> > Why are these formatting changes to the dockerfiles in this commit ?
>
> The latest lcitool I assume didn't see the need for the continuation
> line. I ran:
>
>   make lcitool-refresh
>
> when I updated.

Oh, I see. If you put something in the commit message along
the lines of

"This commit was automatically generated by running 'make lcitool-refresh'"

that will help reviewers, since the default assumption is
generally that changes are hand-written.

thanks
-- PMM

[PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Alex Bennée]

Using lci-tool update debian-arm64-cross to a Debian 11 based system.
As a result we can drop debian-arm64-test-cross just for building
tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220211160309.335014-5-alex.bennee@linaro.org>
---
 .gitlab-ci.d/container-cross.yml              |  10 +-
 tests/docker/Makefile.include                 |   3 -
 .../dockerfiles/debian-arm64-cross.docker     | 186 +++++++++++++++---
 .../debian-arm64-test-cross.docker            |  13 --
 tests/lcitool/refresh                         |  11 ++
 tests/tcg/configure.sh                        |   2 +-
 6 files changed, 173 insertions(+), 52 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index a3b5b90552..ed620620f8 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -21,18 +21,10 @@ amd64-debian-user-cross-container:
 
 arm64-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-arm64-cross
 
-arm64-test-debian-cross-container:
-  extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian11-container']
-  variables:
-    NAME: debian-arm64-test-cross
-
 armel-debian-cross-container:
   extends: .container_job_template
   stage: containers-layer2
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 286f0ac5b5..eeee1e6bdf 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -137,7 +137,6 @@ DOCKER_PARTIAL_IMAGES += fedora
 endif
 
 docker-image-debian-alpha-cross: docker-image-debian10
-docker-image-debian-arm64-cross: docker-image-debian10
 docker-image-debian-armel-cross: docker-image-debian10
 docker-image-debian-armhf-cross: docker-image-debian10
 docker-image-debian-hppa-cross: docker-image-debian10
@@ -213,14 +212,12 @@ docker-image-debian-nios2-cross: $(DOCKER_FILES_DIR)/debian-toolchain.docker \
 # Specialist build images, sometimes very limited tools
 docker-image-debian-tricore-cross: docker-image-debian10
 docker-image-debian-all-test-cross: docker-image-debian10
-docker-image-debian-arm64-test-cross: docker-image-debian11
 docker-image-debian-microblaze-cross: docker-image-debian10
 docker-image-debian-nios2-cross: docker-image-debian10
 docker-image-debian-powerpc-test-cross: docker-image-debian11
 
 # These images may be good enough for building tests but not for test builds
 DOCKER_PARTIAL_IMAGES += debian-alpha-cross
-DOCKER_PARTIAL_IMAGES += debian-arm64-test-cross
 DOCKER_PARTIAL_IMAGES += debian-powerpc-test-cross
 DOCKER_PARTIAL_IMAGES += debian-hppa-cross
 DOCKER_PARTIAL_IMAGES += debian-m68k-cross debian-mips64-cross
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 166e24df13..589510a7be 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -1,32 +1,166 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker arm64 cross-compiler target
+#  $ lcitool dockerfile --layers all --cross aarch64 debian-11 qemu
 #
-# This docker target builds on the debian Buster base image.
-#
-FROM qemu/debian10
+# https://gitlab.com/libvirt/libvirt-ci
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture arm64
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        crossbuild-essential-arm64
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a arm64 --arch-only qemu
+FROM docker.io/library/debian:11-slim
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
-ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            libtest-harness-perl \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:arm64 \
-        liblzo2-dev:arm64 \
-        librdmacm-dev:arm64 \
-        libsnappy-dev:arm64 \
-        libxen-dev:arm64
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-# nettle
-ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-nettle
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture arm64 && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-aarch64-linux-gnu \
+            gcc-aarch64-linux-gnu \
+            libaio-dev:arm64 \
+            libasan5:arm64 \
+            libasound2-dev:arm64 \
+            libattr1-dev:arm64 \
+            libbpf-dev:arm64 \
+            libbrlapi-dev:arm64 \
+            libbz2-dev:arm64 \
+            libc6-dev:arm64 \
+            libcacard-dev:arm64 \
+            libcap-ng-dev:arm64 \
+            libcapstone-dev:arm64 \
+            libcurl4-gnutls-dev:arm64 \
+            libdaxctl-dev:arm64 \
+            libdrm-dev:arm64 \
+            libepoxy-dev:arm64 \
+            libfdt-dev:arm64 \
+            libffi-dev:arm64 \
+            libfuse3-dev:arm64 \
+            libgbm-dev:arm64 \
+            libgcrypt20-dev:arm64 \
+            libglib2.0-dev:arm64 \
+            libglusterfs-dev:arm64 \
+            libgnutls28-dev:arm64 \
+            libgtk-3-dev:arm64 \
+            libibumad-dev:arm64 \
+            libibverbs-dev:arm64 \
+            libiscsi-dev:arm64 \
+            libjemalloc-dev:arm64 \
+            libjpeg62-turbo-dev:arm64 \
+            liblttng-ust-dev:arm64 \
+            liblzo2-dev:arm64 \
+            libncursesw5-dev:arm64 \
+            libnfs-dev:arm64 \
+            libnuma-dev:arm64 \
+            libpam0g-dev:arm64 \
+            libpixman-1-dev:arm64 \
+            libpng-dev:arm64 \
+            libpulse-dev:arm64 \
+            librbd-dev:arm64 \
+            librdmacm-dev:arm64 \
+            libsasl2-dev:arm64 \
+            libsdl2-dev:arm64 \
+            libsdl2-image-dev:arm64 \
+            libseccomp-dev:arm64 \
+            libselinux1-dev:arm64 \
+            libslirp-dev:arm64 \
+            libsnappy-dev:arm64 \
+            libspice-server-dev:arm64 \
+            libssh-gcrypt-dev:arm64 \
+            libsystemd-dev:arm64 \
+            libtasn1-6-dev:arm64 \
+            libubsan1:arm64 \
+            libudev-dev:arm64 \
+            liburing-dev:arm64 \
+            libusb-1.0-0-dev:arm64 \
+            libusbredirhost-dev:arm64 \
+            libvdeplug-dev:arm64 \
+            libvirglrenderer-dev:arm64 \
+            libvte-2.91-dev:arm64 \
+            libxen-dev:arm64 \
+            libzstd-dev:arm64 \
+            nettle-dev:arm64 \
+            systemtap-sdt-dev:arm64 \
+            xfslibs-dev:arm64 \
+            zlib1g-dev:arm64 && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/aarch64-linux-gnu-gcc'\n\
+ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'\n\
+strip = '/usr/bin/aarch64-linux-gnu-strip'\n\
+pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 'aarch64'\n\
+cpu = 'aarch64'\n\
+endian = 'little'" > /usr/local/share/meson/cross/aarch64-linux-gnu && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-gcc
+
+ENV ABI "aarch64-linux-gnu"
+ENV MESON_OPTS "--cross-file=aarch64-linux-gnu"
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user
diff --git a/tests/docker/dockerfiles/debian-arm64-test-cross.docker b/tests/docker/dockerfiles/debian-arm64-test-cross.docker
deleted file mode 100644
index 53a9012beb..0000000000
--- a/tests/docker/dockerfiles/debian-arm64-test-cross.docker
+++ /dev/null
@@ -1,13 +0,0 @@
-#
-# Docker arm64 cross-compiler target (tests only)
-#
-# This docker target builds on the debian Bullseye base image.
-#
-FROM qemu/debian11
-
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture arm64
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-        apt install -y --no-install-recommends \
-        crossbuild-essential-arm64 gcc-10-aarch64-linux-gnu
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 4ab90a310a..ada73f7045 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -76,6 +76,12 @@ ubuntu2004_tsanhack = [
    "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"
 ]
 
+def debian_cross_build(prefix, targets):
+   conf = "ENV QEMU_CONFIGURE_OPTS --cross-prefix=%s\n" % (prefix)
+   targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
+   return "".join([conf, targets])
+
+
 try:
    generate_dockerfile("centos8", "centos-stream-8")
    generate_dockerfile("fedora", "fedora-35")
@@ -86,6 +92,11 @@ try:
    generate_dockerfile("opensuse-leap", "opensuse-leap-152")
    generate_dockerfile("alpine", "alpine-edge")
 
+   generate_dockerfile("debian-arm64-cross", "debian-11",
+                       cross="aarch64",
+                       trailer=debian_cross_build("aarch64-linux-gnu-",
+                                                  "aarch64-softmmu,aarch64-linux-user"))
+
    generate_cirrus("freebsd-12")
    generate_cirrus("freebsd-13")
    generate_cirrus("macos-11")
diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
index 763e9b6ad8..adc95d6a44 100755
--- a/tests/tcg/configure.sh
+++ b/tests/tcg/configure.sh
@@ -97,7 +97,7 @@ for target in $target_list; do
     aarch64-*)
       # We don't have any bigendian build tools so we only use this for AArch64
       container_hosts="x86_64 aarch64"
-      container_image=debian-arm64-test-cross
+      container_image=debian-arm64-cross
       container_cross_cc=aarch64-linux-gnu-gcc-10
       ;;
     alpha-*)
-- 
2.30.2

Re: [PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Thomas Huth]

On 25/02/2022 18.20, Alex Bennée wrote:
> Using lci-tool update debian-arm64-cross to a Debian 11 based system.
> As a result we can drop debian-arm64-test-cross just for building
> tests.

Good idea!

One comment below...

[...]
> diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
> index 166e24df13..589510a7be 100644
> --- a/tests/docker/dockerfiles/debian-arm64-cross.docker
> +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
> @@ -1,32 +1,166 @@
> +# THIS FILE WAS AUTO-GENERATED
>   #
> -# Docker arm64 cross-compiler target
> +#  $ lcitool dockerfile --layers all --cross aarch64 debian-11 qemu
>   #
> -# This docker target builds on the debian Buster base image.
> -#
> -FROM qemu/debian10
> +# https://gitlab.com/libvirt/libvirt-ci
>   
> -# Add the foreign architecture we want and install dependencies
> -RUN dpkg --add-architecture arm64
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt install -y --no-install-recommends \
> -        crossbuild-essential-arm64
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt build-dep -yy -a arm64 --arch-only qemu
> +FROM docker.io/library/debian:11-slim
>   
> -# Specify the cross prefix for this image (see tests/docker/common.rc)
> -ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
> -ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user
> +RUN export DEBIAN_FRONTEND=noninteractive && \
> +    apt-get update && \
> +    apt-get install -y eatmydata && \
> +    eatmydata apt-get dist-upgrade -y && \
> +    eatmydata apt-get install --no-install-recommends -y \
> +            bash \
> +            bc \
> +            bsdextrautils \
> +            bzip2 \
> +            ca-certificates \
> +            ccache \
> +            dbus \
> +            debianutils \
> +            diffutils \
> +            exuberant-ctags \
> +            findutils \
> +            gcovr \
> +            genisoimage \
> +            gettext \
> +            git \
> +            hostname \
> +            libpcre2-dev \
> +            libspice-protocol-dev \
> +            libtest-harness-perl \
> +            llvm \
> +            locales \
> +            make \
> +            meson \
> +            ncat \
> +            ninja-build \
> +            openssh-client \
> +            perl-base \
> +            pkgconf \
> +            python3 \
> +            python3-numpy \
> +            python3-opencv \
> +            python3-pillow \
> +            python3-pip \
> +            python3-sphinx \
> +            python3-sphinx-rtd-theme \
> +            python3-venv \
> +            python3-yaml \
> +            rpm2cpio \
> +            sed \
> +            sparse \
> +            tar \
> +            tesseract-ocr \
> +            tesseract-ocr-eng \
> +            texinfo && \
> +    eatmydata apt-get autoremove -y && \
> +    eatmydata apt-get autoclean -y && \
> +    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
> +    dpkg-reconfigure locales
>   
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt install -y --no-install-recommends \
> -        libbz2-dev:arm64 \
> -        liblzo2-dev:arm64 \
> -        librdmacm-dev:arm64 \
> -        libsnappy-dev:arm64 \
> -        libxen-dev:arm64
> +ENV LANG "en_US.UTF-8"
> +ENV MAKE "/usr/bin/make"
> +ENV NINJA "/usr/bin/ninja"
> +ENV PYTHON "/usr/bin/python3"
> +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
>   
> -# nettle
> -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-nettle
> +RUN export DEBIAN_FRONTEND=noninteractive && \
> +    dpkg --add-architecture arm64 && \
> +    eatmydata apt-get update && \
> +    eatmydata apt-get dist-upgrade -y && \
> +    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
> +    eatmydata apt-get install --no-install-recommends -y \
> +            g++-aarch64-linux-gnu \
> +            gcc-aarch64-linux-gnu \
> +            libaio-dev:arm64 \
> +            libasan5:arm64 \
> +            libasound2-dev:arm64 \
> +            libattr1-dev:arm64 \
> +            libbpf-dev:arm64 \
> +            libbrlapi-dev:arm64 \
> +            libbz2-dev:arm64 \
> +            libc6-dev:arm64 \
> +            libcacard-dev:arm64 \
> +            libcap-ng-dev:arm64 \
> +            libcapstone-dev:arm64 \
> +            libcurl4-gnutls-dev:arm64 \
> +            libdaxctl-dev:arm64 \
> +            libdrm-dev:arm64 \
> +            libepoxy-dev:arm64 \
> +            libfdt-dev:arm64 \
> +            libffi-dev:arm64 \
> +            libfuse3-dev:arm64 \
> +            libgbm-dev:arm64 \
> +            libgcrypt20-dev:arm64 \
> +            libglib2.0-dev:arm64 \
> +            libglusterfs-dev:arm64 \
> +            libgnutls28-dev:arm64 \
> +            libgtk-3-dev:arm64 \
> +            libibumad-dev:arm64 \
> +            libibverbs-dev:arm64 \
> +            libiscsi-dev:arm64 \
> +            libjemalloc-dev:arm64 \
> +            libjpeg62-turbo-dev:arm64 \
> +            liblttng-ust-dev:arm64 \
> +            liblzo2-dev:arm64 \
> +            libncursesw5-dev:arm64 \
> +            libnfs-dev:arm64 \
> +            libnuma-dev:arm64 \
> +            libpam0g-dev:arm64 \
> +            libpixman-1-dev:arm64 \
> +            libpng-dev:arm64 \
> +            libpulse-dev:arm64 \
> +            librbd-dev:arm64 \
> +            librdmacm-dev:arm64 \
> +            libsasl2-dev:arm64 \
> +            libsdl2-dev:arm64 \
> +            libsdl2-image-dev:arm64 \
> +            libseccomp-dev:arm64 \
> +            libselinux1-dev:arm64 \
> +            libslirp-dev:arm64 \
> +            libsnappy-dev:arm64 \
> +            libspice-server-dev:arm64 \
> +            libssh-gcrypt-dev:arm64 \
> +            libsystemd-dev:arm64 \
> +            libtasn1-6-dev:arm64 \
> +            libubsan1:arm64 \
> +            libudev-dev:arm64 \
> +            liburing-dev:arm64 \
> +            libusb-1.0-0-dev:arm64 \
> +            libusbredirhost-dev:arm64 \
> +            libvdeplug-dev:arm64 \
> +            libvirglrenderer-dev:arm64 \
> +            libvte-2.91-dev:arm64 \
> +            libxen-dev:arm64 \
> +            libzstd-dev:arm64 \
> +            nettle-dev:arm64 \
> +            systemtap-sdt-dev:arm64 \
> +            xfslibs-dev:arm64 \
> +            zlib1g-dev:arm64 && \
> +    eatmydata apt-get autoremove -y && \
> +    eatmydata apt-get autoclean -y && \
> +    mkdir -p /usr/local/share/meson/cross && \
> +    echo "[binaries]\n\
> +c = '/usr/bin/aarch64-linux-gnu-gcc'\n\

Is it really just "c =" or should this be "cc =" ?

> +ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'\n\
> +strip = '/usr/bin/aarch64-linux-gnu-strip'\n\
> +pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'\n\
> +\n\
> +[host_machine]\n\
> +system = 'linux'\n\
> +cpu_family = 'aarch64'\n\
> +cpu = 'aarch64'\n\
> +endian = 'little'" > /usr/local/share/meson/cross/aarch64-linux-gnu && \
> +    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
> +    mkdir -p /usr/libexec/ccache-wrappers && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-c++ && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-cc && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-g++ && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-gcc
> +
> +ENV ABI "aarch64-linux-gnu"
> +ENV MESON_OPTS "--cross-file=aarch64-linux-gnu"
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
> +ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user

  Thomas

Re: [PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Daniel P. Berrangé]

On Mon, Feb 28, 2022 at 09:39:29AM +0100, Thomas Huth wrote:
> On 25/02/2022 18.20, Alex Bennée wrote:
> > Using lci-tool update debian-arm64-cross to a Debian 11 based system.
> > As a result we can drop debian-arm64-test-cross just for building
> > tests.
> 
> Good idea!
> 
> One comment below...
> 
> [...]
> > diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
> > index 166e24df13..589510a7be 100644
> > --- a/tests/docker/dockerfiles/debian-arm64-cross.docker
> > +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
> > @@ -1,32 +1,166 @@
> > +# THIS FILE WAS AUTO-GENERATED
> >   #
> > -# Docker arm64 cross-compiler target
> > +#  $ lcitool dockerfile --layers all --cross aarch64 debian-11 qemu
> >   #
> > -# This docker target builds on the debian Buster base image.
> > -#
> > -FROM qemu/debian10
> > +# https://gitlab.com/libvirt/libvirt-ci
> > -# Add the foreign architecture we want and install dependencies
> > -RUN dpkg --add-architecture arm64
> > -RUN apt update && \
> > -    DEBIAN_FRONTEND=noninteractive eatmydata \
> > -    apt install -y --no-install-recommends \
> > -        crossbuild-essential-arm64
> > -RUN apt update && \
> > -    DEBIAN_FRONTEND=noninteractive eatmydata \
> > -    apt build-dep -yy -a arm64 --arch-only qemu
> > +FROM docker.io/library/debian:11-slim
> > -# Specify the cross prefix for this image (see tests/docker/common.rc)
> > -ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
> > -ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user
> > +RUN export DEBIAN_FRONTEND=noninteractive && \
> > +    apt-get update && \
> > +    apt-get install -y eatmydata && \
> > +    eatmydata apt-get dist-upgrade -y && \
> > +    eatmydata apt-get install --no-install-recommends -y \
> > +            bash \
> > +            bc \
> > +            bsdextrautils \
> > +            bzip2 \
> > +            ca-certificates \
> > +            ccache \
> > +            dbus \
> > +            debianutils \
> > +            diffutils \
> > +            exuberant-ctags \
> > +            findutils \
> > +            gcovr \
> > +            genisoimage \
> > +            gettext \
> > +            git \
> > +            hostname \
> > +            libpcre2-dev \
> > +            libspice-protocol-dev \
> > +            libtest-harness-perl \
> > +            llvm \
> > +            locales \
> > +            make \
> > +            meson \
> > +            ncat \
> > +            ninja-build \
> > +            openssh-client \
> > +            perl-base \
> > +            pkgconf \
> > +            python3 \
> > +            python3-numpy \
> > +            python3-opencv \
> > +            python3-pillow \
> > +            python3-pip \
> > +            python3-sphinx \
> > +            python3-sphinx-rtd-theme \
> > +            python3-venv \
> > +            python3-yaml \
> > +            rpm2cpio \
> > +            sed \
> > +            sparse \
> > +            tar \
> > +            tesseract-ocr \
> > +            tesseract-ocr-eng \
> > +            texinfo && \
> > +    eatmydata apt-get autoremove -y && \
> > +    eatmydata apt-get autoclean -y && \
> > +    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
> > +    dpkg-reconfigure locales
> > -RUN apt update && \
> > -    DEBIAN_FRONTEND=noninteractive eatmydata \
> > -    apt install -y --no-install-recommends \
> > -        libbz2-dev:arm64 \
> > -        liblzo2-dev:arm64 \
> > -        librdmacm-dev:arm64 \
> > -        libsnappy-dev:arm64 \
> > -        libxen-dev:arm64
> > +ENV LANG "en_US.UTF-8"
> > +ENV MAKE "/usr/bin/make"
> > +ENV NINJA "/usr/bin/ninja"
> > +ENV PYTHON "/usr/bin/python3"
> > +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
> > -# nettle
> > -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-nettle
> > +RUN export DEBIAN_FRONTEND=noninteractive && \
> > +    dpkg --add-architecture arm64 && \
> > +    eatmydata apt-get update && \
> > +    eatmydata apt-get dist-upgrade -y && \
> > +    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
> > +    eatmydata apt-get install --no-install-recommends -y \
> > +            g++-aarch64-linux-gnu \
> > +            gcc-aarch64-linux-gnu \
> > +            libaio-dev:arm64 \
> > +            libasan5:arm64 \
> > +            libasound2-dev:arm64 \
> > +            libattr1-dev:arm64 \
> > +            libbpf-dev:arm64 \
> > +            libbrlapi-dev:arm64 \
> > +            libbz2-dev:arm64 \
> > +            libc6-dev:arm64 \
> > +            libcacard-dev:arm64 \
> > +            libcap-ng-dev:arm64 \
> > +            libcapstone-dev:arm64 \
> > +            libcurl4-gnutls-dev:arm64 \
> > +            libdaxctl-dev:arm64 \
> > +            libdrm-dev:arm64 \
> > +            libepoxy-dev:arm64 \
> > +            libfdt-dev:arm64 \
> > +            libffi-dev:arm64 \
> > +            libfuse3-dev:arm64 \
> > +            libgbm-dev:arm64 \
> > +            libgcrypt20-dev:arm64 \
> > +            libglib2.0-dev:arm64 \
> > +            libglusterfs-dev:arm64 \
> > +            libgnutls28-dev:arm64 \
> > +            libgtk-3-dev:arm64 \
> > +            libibumad-dev:arm64 \
> > +            libibverbs-dev:arm64 \
> > +            libiscsi-dev:arm64 \
> > +            libjemalloc-dev:arm64 \
> > +            libjpeg62-turbo-dev:arm64 \
> > +            liblttng-ust-dev:arm64 \
> > +            liblzo2-dev:arm64 \
> > +            libncursesw5-dev:arm64 \
> > +            libnfs-dev:arm64 \
> > +            libnuma-dev:arm64 \
> > +            libpam0g-dev:arm64 \
> > +            libpixman-1-dev:arm64 \
> > +            libpng-dev:arm64 \
> > +            libpulse-dev:arm64 \
> > +            librbd-dev:arm64 \
> > +            librdmacm-dev:arm64 \
> > +            libsasl2-dev:arm64 \
> > +            libsdl2-dev:arm64 \
> > +            libsdl2-image-dev:arm64 \
> > +            libseccomp-dev:arm64 \
> > +            libselinux1-dev:arm64 \
> > +            libslirp-dev:arm64 \
> > +            libsnappy-dev:arm64 \
> > +            libspice-server-dev:arm64 \
> > +            libssh-gcrypt-dev:arm64 \
> > +            libsystemd-dev:arm64 \
> > +            libtasn1-6-dev:arm64 \
> > +            libubsan1:arm64 \
> > +            libudev-dev:arm64 \
> > +            liburing-dev:arm64 \
> > +            libusb-1.0-0-dev:arm64 \
> > +            libusbredirhost-dev:arm64 \
> > +            libvdeplug-dev:arm64 \
> > +            libvirglrenderer-dev:arm64 \
> > +            libvte-2.91-dev:arm64 \
> > +            libxen-dev:arm64 \
> > +            libzstd-dev:arm64 \
> > +            nettle-dev:arm64 \
> > +            systemtap-sdt-dev:arm64 \
> > +            xfslibs-dev:arm64 \
> > +            zlib1g-dev:arm64 && \
> > +    eatmydata apt-get autoremove -y && \
> > +    eatmydata apt-get autoclean -y && \
> > +    mkdir -p /usr/local/share/meson/cross && \
> > +    echo "[binaries]\n\
> > +c = '/usr/bin/aarch64-linux-gnu-gcc'\n\
> 
> Is it really just "c =" or should this be "cc =" ?

I believe this really does want to be 'c', as in this is the path
for the 'c' programming language  in Meson.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Daniel P. Berrangé]

$SUBJECT  =~ s/lci-tool/lcitool/

On Fri, Feb 25, 2022 at 05:20:07PM +0000, Alex Bennée wrote:
> Using lci-tool update debian-arm64-cross to a Debian 11 based system.

Likewise

> As a result we can drop debian-arm64-test-cross just for building
> tests.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Message-Id: <20220211160309.335014-5-alex.bennee@linaro.org>
> ---
>  .gitlab-ci.d/container-cross.yml              |  10 +-
>  tests/docker/Makefile.include                 |   3 -
>  .../dockerfiles/debian-arm64-cross.docker     | 186 +++++++++++++++---
>  .../debian-arm64-test-cross.docker            |  13 --
>  tests/lcitool/refresh                         |  11 ++
>  tests/tcg/configure.sh                        |   2 +-
>  6 files changed, 173 insertions(+), 52 deletions(-)
>  delete mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker
> 



> +RUN export DEBIAN_FRONTEND=noninteractive && \
> +    apt-get update && \
> +    apt-get install -y eatmydata && \
> +    eatmydata apt-get dist-upgrade -y && \
> +    eatmydata apt-get install --no-install-recommends -y \
> +            bash \

    ...snip native packages...

> +            texinfo && \
> +    eatmydata apt-get autoremove -y && \
> +    eatmydata apt-get autoclean -y && \
> +    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
> +    dpkg-reconfigure locales
>  
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt install -y --no-install-recommends \
> -        libbz2-dev:arm64 \
> -        liblzo2-dev:arm64 \
> -        librdmacm-dev:arm64 \
> -        libsnappy-dev:arm64 \
> -        libxen-dev:arm64
> +ENV LANG "en_US.UTF-8"
> +ENV MAKE "/usr/bin/make"
> +ENV NINJA "/usr/bin/ninja"
> +ENV PYTHON "/usr/bin/python3"
> +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
>  
> -# nettle
> -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-nettle
> +RUN export DEBIAN_FRONTEND=noninteractive && \
> +    dpkg --add-architecture arm64 && \
> +    eatmydata apt-get update && \
> +    eatmydata apt-get dist-upgrade -y && \
> +    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
> +    eatmydata apt-get install --no-install-recommends -y \
> +            g++-aarch64-linux-gnu \

  ...snip cross packages...

> +            zlib1g-dev:arm64 && \
> +    eatmydata apt-get autoremove -y && \
> +    eatmydata apt-get autoclean -y && \
> +    mkdir -p /usr/local/share/meson/cross && \
> +    echo "[binaries]\n\
> +c = '/usr/bin/aarch64-linux-gnu-gcc'\n\
> +ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'\n\
> +strip = '/usr/bin/aarch64-linux-gnu-strip'\n\
> +pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'\n\
> +\n\
> +[host_machine]\n\
> +system = 'linux'\n\
> +cpu_family = 'aarch64'\n\
> +cpu = 'aarch64'\n\
> +endian = 'little'" > /usr/local/share/meson/cross/aarch64-linux-gnu && \
> +    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
> +    mkdir -p /usr/libexec/ccache-wrappers && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-c++ && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-cc && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-g++ && \
> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-gcc
> +
> +ENV ABI "aarch64-linux-gnu"
> +ENV MESON_OPTS "--cross-file=aarch64-linux-gnu"
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
> +ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user

This cross dockerfile is a fully self-contained image.

Traditionally QEMU has had a split image for Debian cross targets,
where there is a base with common native packages, and then a
layer for the cross packages.

lcitool is capable of generating the image in this split format
using the arg

   --layers {all,native,foreign}

Personally I think it is simpler to just use the fully self
contained image, as it would simplify our gitlab pipeline
to only need 1 build stage for containers.  The cost is that
we'll not be sharing layers for native packages and more wall
clock time building since we're installing the same native
packages over & over.

I'm not saying to change your patch, I just wanted to point
out the possibility in case someone cares strongly about
keeping a split layer model for cross containers.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> $SUBJECT  =~ s/lci-tool/lcitool/
>
> On Fri, Feb 25, 2022 at 05:20:07PM +0000, Alex Bennée wrote:
>> Using lci-tool update debian-arm64-cross to a Debian 11 based system.
>
> Likewise
>
>> As a result we can drop debian-arm64-test-cross just for building
>> tests.
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> Message-Id: <20220211160309.335014-5-alex.bennee@linaro.org>
>> ---
>>  .gitlab-ci.d/container-cross.yml              |  10 +-
>>  tests/docker/Makefile.include                 |   3 -
>>  .../dockerfiles/debian-arm64-cross.docker     | 186 +++++++++++++++---
>>  .../debian-arm64-test-cross.docker            |  13 --
>>  tests/lcitool/refresh                         |  11 ++
>>  tests/tcg/configure.sh                        |   2 +-
>>  6 files changed, 173 insertions(+), 52 deletions(-)
>>  delete mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker
>> 
>
>
>
>> +RUN export DEBIAN_FRONTEND=noninteractive && \
>> +    apt-get update && \
>> +    apt-get install -y eatmydata && \
>> +    eatmydata apt-get dist-upgrade -y && \
>> +    eatmydata apt-get install --no-install-recommends -y \
>> +            bash \
>
>     ...snip native packages...
>
>> +            texinfo && \
>> +    eatmydata apt-get autoremove -y && \
>> +    eatmydata apt-get autoclean -y && \
>> +    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
>> +    dpkg-reconfigure locales
>>  
>> -RUN apt update && \
>> -    DEBIAN_FRONTEND=noninteractive eatmydata \
>> -    apt install -y --no-install-recommends \
>> -        libbz2-dev:arm64 \
>> -        liblzo2-dev:arm64 \
>> -        librdmacm-dev:arm64 \
>> -        libsnappy-dev:arm64 \
>> -        libxen-dev:arm64
>> +ENV LANG "en_US.UTF-8"
>> +ENV MAKE "/usr/bin/make"
>> +ENV NINJA "/usr/bin/ninja"
>> +ENV PYTHON "/usr/bin/python3"
>> +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
>>  
>> -# nettle
>> -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS --enable-nettle
>> +RUN export DEBIAN_FRONTEND=noninteractive && \
>> +    dpkg --add-architecture arm64 && \
>> +    eatmydata apt-get update && \
>> +    eatmydata apt-get dist-upgrade -y && \
>> +    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
>> +    eatmydata apt-get install --no-install-recommends -y \
>> +            g++-aarch64-linux-gnu \
>
>   ...snip cross packages...
>
>> +            zlib1g-dev:arm64 && \
>> +    eatmydata apt-get autoremove -y && \
>> +    eatmydata apt-get autoclean -y && \
>> +    mkdir -p /usr/local/share/meson/cross && \
>> +    echo "[binaries]\n\
>> +c = '/usr/bin/aarch64-linux-gnu-gcc'\n\
>> +ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'\n\
>> +strip = '/usr/bin/aarch64-linux-gnu-strip'\n\
>> +pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'\n\
>> +\n\
>> +[host_machine]\n\
>> +system = 'linux'\n\
>> +cpu_family = 'aarch64'\n\
>> +cpu = 'aarch64'\n\
>> +endian = 'little'" > /usr/local/share/meson/cross/aarch64-linux-gnu && \
>> +    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
>> +    mkdir -p /usr/libexec/ccache-wrappers && \
>> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-c++ && \
>> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-cc && \
>> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-g++ && \
>> +    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/aarch64-linux-gnu-gcc
>> +
>> +ENV ABI "aarch64-linux-gnu"
>> +ENV MESON_OPTS "--cross-file=aarch64-linux-gnu"
>> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
>> +ENV DEF_TARGET_LIST aarch64-softmmu,aarch64-linux-user
>
> This cross dockerfile is a fully self-contained image.
>
> Traditionally QEMU has had a split image for Debian cross targets,
> where there is a base with common native packages, and then a
> layer for the cross packages.
>
> lcitool is capable of generating the image in this split format
> using the arg
>
>    --layers {all,native,foreign}
>
> Personally I think it is simpler to just use the fully self
> contained image, as it would simplify our gitlab pipeline
> to only need 1 build stage for containers.  The cost is that
> we'll not be sharing layers for native packages and more wall
> clock time building since we're installing the same native
> packages over & over.
>
> I'm not saying to change your patch, I just wanted to point
> out the possibility in case someone cares strongly about
> keeping a split layer model for cross containers.

My thinking on our layered approach has evolved over the years. One of
the problems is when the two layers get out of sync and you run into
build issues due to different states of cached layers.

I think now for full cross-builds I'm fine with just going for the full
single recipe. For cross-test building there may be more argument for
having a base layer (although in the end we also ended up with
all-test-cross because we can't have containers running containers in
GitLab).

>
> Regards,
> Daniel


-- 
Alex Bennée

Re: [PATCH v2 04/18] tests/docker: update debian-arm64-cross with lci-tool

[Daniel P. Berrangé]

On Mon, Feb 28, 2022 at 02:39:17PM +0000, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > $SUBJECT  =~ s/lci-tool/lcitool/
> >
> > On Fri, Feb 25, 2022 at 05:20:07PM +0000, Alex Bennée wrote:
> >> Using lci-tool update debian-arm64-cross to a Debian 11 based system.
> >
> > Likewise
> >
> >> As a result we can drop debian-arm64-test-cross just for building
> >> tests.
> >> 
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> >> Message-Id: <20220211160309.335014-5-alex.bennee@linaro.org>
> >> ---
> >>  .gitlab-ci.d/container-cross.yml              |  10 +-
> >>  tests/docker/Makefile.include                 |   3 -
> >>  .../dockerfiles/debian-arm64-cross.docker     | 186 +++++++++++++++---
> >>  .../debian-arm64-test-cross.docker            |  13 --
> >>  tests/lcitool/refresh                         |  11 ++
> >>  tests/tcg/configure.sh                        |   2 +-
> >>  6 files changed, 173 insertions(+), 52 deletions(-)
> >>  delete mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker
> >> 


> > This cross dockerfile is a fully self-contained image.
> >
> > Traditionally QEMU has had a split image for Debian cross targets,
> > where there is a base with common native packages, and then a
> > layer for the cross packages.
> >
> > lcitool is capable of generating the image in this split format
> > using the arg
> >
> >    --layers {all,native,foreign}
> >
> > Personally I think it is simpler to just use the fully self
> > contained image, as it would simplify our gitlab pipeline
> > to only need 1 build stage for containers.  The cost is that
> > we'll not be sharing layers for native packages and more wall
> > clock time building since we're installing the same native
> > packages over & over.
> >
> > I'm not saying to change your patch, I just wanted to point
> > out the possibility in case someone cares strongly about
> > keeping a split layer model for cross containers.
> 
> My thinking on our layered approach has evolved over the years. One of
> the problems is when the two layers get out of sync and you run into
> build issues due to different states of cached layers.

Oh, I'd not even thought about that possibility but yes, it makes
sense. We could have cached the base layer and when we do an
'apt-get update' in the cross layer we'll end up pulling in new
copies of packages otherwise present in the base layer, partly
defeating the point of having two layers.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 05/18] tests/docker: update debian-s390x-cross with lcitool

[Alex Bennée]

A later compiler is needed for some upcomming tests so we might as
well migrate to an lcitool generated docker file.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: David Hildenbrand <david@redhat.com>
---
 .gitlab-ci.d/container-cross.yml              |   3 +-
 tests/docker/Makefile.include                 |   1 -
 .../dockerfiles/debian-s390x-cross.docker     | 181 +++++++++++++++---
 tests/lcitool/refresh                         |   5 +
 4 files changed, 162 insertions(+), 28 deletions(-)

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index ed620620f8..d38f657131 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -133,8 +133,7 @@ riscv64-debian-cross-container:
 
 s390x-debian-cross-container:
   extends: .container_job_template
-  stage: containers-layer2
-  needs: ['amd64-debian10-container']
+  stage: containers
   variables:
     NAME: debian-s390x-cross
 
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index eeee1e6bdf..cce9faab36 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -146,7 +146,6 @@ docker-image-debian-mips64-cross: docker-image-debian10
 docker-image-debian-mips64el-cross: docker-image-debian10
 docker-image-debian-mipsel-cross: docker-image-debian10
 docker-image-debian-ppc64el-cross: docker-image-debian10
-docker-image-debian-s390x-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
 
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 9f2ab51eb0..aa1bd6eb4c 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -1,33 +1,164 @@
+# THIS FILE WAS AUTO-GENERATED
 #
-# Docker s390 cross-compiler target
+#  $ lcitool dockerfile --layers all --cross s390x debian-11 qemu
 #
-# This docker target builds on the debian Stretch base image.
-#
-FROM qemu/debian10
+# https://gitlab.com/libvirt/libvirt-ci
+
+FROM docker.io/library/debian:11-slim
 
-# Add the s390x architecture
-RUN dpkg --add-architecture s390x
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \
+    apt-get install -y eatmydata && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y \
+            bash \
+            bc \
+            bsdextrautils \
+            bzip2 \
+            ca-certificates \
+            ccache \
+            dbus \
+            debianutils \
+            diffutils \
+            exuberant-ctags \
+            findutils \
+            gcovr \
+            genisoimage \
+            gettext \
+            git \
+            hostname \
+            libpcre2-dev \
+            libspice-protocol-dev \
+            libtest-harness-perl \
+            llvm \
+            locales \
+            make \
+            meson \
+            ncat \
+            ninja-build \
+            openssh-client \
+            perl-base \
+            pkgconf \
+            python3 \
+            python3-numpy \
+            python3-opencv \
+            python3-pillow \
+            python3-pip \
+            python3-sphinx \
+            python3-sphinx-rtd-theme \
+            python3-venv \
+            python3-yaml \
+            rpm2cpio \
+            sed \
+            sparse \
+            tar \
+            tesseract-ocr \
+            tesseract-ocr-eng \
+            texinfo && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
+    dpkg-reconfigure locales
 
-# Grab the updated list of packages
-RUN apt update && apt dist-upgrade -yy
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        gcc-multilib-s390x-linux-gnu
+ENV LANG "en_US.UTF-8"
+ENV MAKE "/usr/bin/make"
+ENV NINJA "/usr/bin/ninja"
+ENV PYTHON "/usr/bin/python3"
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy -a s390x --arch-only qemu
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    dpkg --add-architecture s390x && \
+    eatmydata apt-get update && \
+    eatmydata apt-get dist-upgrade -y && \
+    eatmydata apt-get install --no-install-recommends -y dpkg-dev && \
+    eatmydata apt-get install --no-install-recommends -y \
+            g++-s390x-linux-gnu \
+            gcc-s390x-linux-gnu \
+            libaio-dev:s390x \
+            libasan5:s390x \
+            libasound2-dev:s390x \
+            libattr1-dev:s390x \
+            libbpf-dev:s390x \
+            libbrlapi-dev:s390x \
+            libbz2-dev:s390x \
+            libc6-dev:s390x \
+            libcacard-dev:s390x \
+            libcap-ng-dev:s390x \
+            libcapstone-dev:s390x \
+            libcurl4-gnutls-dev:s390x \
+            libdaxctl-dev:s390x \
+            libdrm-dev:s390x \
+            libepoxy-dev:s390x \
+            libfdt-dev:s390x \
+            libffi-dev:s390x \
+            libfuse3-dev:s390x \
+            libgbm-dev:s390x \
+            libgcrypt20-dev:s390x \
+            libglib2.0-dev:s390x \
+            libglusterfs-dev:s390x \
+            libgnutls28-dev:s390x \
+            libgtk-3-dev:s390x \
+            libibumad-dev:s390x \
+            libibverbs-dev:s390x \
+            libiscsi-dev:s390x \
+            libjemalloc-dev:s390x \
+            libjpeg62-turbo-dev:s390x \
+            liblttng-ust-dev:s390x \
+            liblzo2-dev:s390x \
+            libncursesw5-dev:s390x \
+            libnfs-dev:s390x \
+            libnuma-dev:s390x \
+            libpam0g-dev:s390x \
+            libpixman-1-dev:s390x \
+            libpng-dev:s390x \
+            libpulse-dev:s390x \
+            librbd-dev:s390x \
+            librdmacm-dev:s390x \
+            libsasl2-dev:s390x \
+            libsdl2-dev:s390x \
+            libsdl2-image-dev:s390x \
+            libseccomp-dev:s390x \
+            libselinux1-dev:s390x \
+            libslirp-dev:s390x \
+            libsnappy-dev:s390x \
+            libssh-gcrypt-dev:s390x \
+            libsystemd-dev:s390x \
+            libtasn1-6-dev:s390x \
+            libubsan1:s390x \
+            libudev-dev:s390x \
+            liburing-dev:s390x \
+            libusb-1.0-0-dev:s390x \
+            libusbredirhost-dev:s390x \
+            libvdeplug-dev:s390x \
+            libvirglrenderer-dev:s390x \
+            libvte-2.91-dev:s390x \
+            libzstd-dev:s390x \
+            nettle-dev:s390x \
+            systemtap-sdt-dev:s390x \
+            xfslibs-dev:s390x \
+            zlib1g-dev:s390x && \
+    eatmydata apt-get autoremove -y && \
+    eatmydata apt-get autoclean -y && \
+    mkdir -p /usr/local/share/meson/cross && \
+    echo "[binaries]\n\
+c = '/usr/bin/s390x-linux-gnu-gcc'\n\
+ar = '/usr/bin/s390x-linux-gnu-gcc-ar'\n\
+strip = '/usr/bin/s390x-linux-gnu-strip'\n\
+pkgconfig = '/usr/bin/s390x-linux-gnu-pkg-config'\n\
+\n\
+[host_machine]\n\
+system = 'linux'\n\
+cpu_family = 's390x'\n\
+cpu = 's390x'\n\
+endian = 'big'" > /usr/local/share/meson/cross/s390x-linux-gnu && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt && \
+    mkdir -p /usr/libexec/ccache-wrappers && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/s390x-linux-gnu-c++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/s390x-linux-gnu-cc && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/s390x-linux-gnu-g++ && \
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/s390x-linux-gnu-gcc
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV ABI "s390x-linux-gnu"
+ENV MESON_OPTS "--cross-file=s390x-linux-gnu"
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
 ENV DEF_TARGET_LIST s390x-softmmu,s390x-linux-user
-
-# Install extra libraries to increase code coverage
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        libbz2-dev:s390x \
-        liblzo2-dev:s390x \
-        librdmacm-dev:s390x \
-        libsnappy-dev:s390x
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index ada73f7045..1f00281b44 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -97,6 +97,11 @@ try:
                        trailer=debian_cross_build("aarch64-linux-gnu-",
                                                   "aarch64-softmmu,aarch64-linux-user"))
 
+   generate_dockerfile("debian-s390x-cross", "debian-11",
+                       cross="s390x",
+                       trailer=debian_cross_build("s390x-linux-gnu-",
+                                                  "s390x-softmmu,s390x-linux-user"))
+
    generate_cirrus("freebsd-12")
    generate_cirrus("freebsd-13")
    generate_cirrus("macos-11")
-- 
2.30.2

Re: [PATCH v2 05/18] tests/docker: update debian-s390x-cross with lcitool

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> A later compiler is needed for some upcomming tests so we might as
> well migrate to an lcitool generated docker file.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Cc: David Hildenbrand<david@redhat.com>
> ---
>   .gitlab-ci.d/container-cross.yml              |   3 +-
>   tests/docker/Makefile.include                 |   1 -
>   .../dockerfiles/debian-s390x-cross.docker     | 181 +++++++++++++++---
>   tests/lcitool/refresh                         |   5 +
>   4 files changed, 162 insertions(+), 28 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 06/18] tests/docker: introduce debian-riscv64-test-cross

[Alex Bennée]

Cross building QEMU for riscv64 still involves messing about with sid
and ports. However for building tests we can have a slimmer compiler
only container which should be more stable.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220211160309.335014-6-alex.bennee@linaro.org>
---
 .gitlab-ci.d/container-cross.yml                     |  7 +++++++
 tests/docker/Makefile.include                        |  2 ++
 .../dockerfiles/debian-riscv64-test-cross.docker     | 12 ++++++++++++
 tests/tcg/configure.sh                               |  2 +-
 4 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 tests/docker/dockerfiles/debian-riscv64-test-cross.docker

diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index d38f657131..e622ac2d21 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -131,6 +131,13 @@ riscv64-debian-cross-container:
   variables:
     NAME: debian-riscv64-cross
 
+# we can however build TCG tests using a non-sid base
+riscv64-debian-test-cross-container:
+  extends: .container_job_template
+  stage: containers-layer2
+  variables:
+    NAME: debian-riscv64-test-cross
+
 s390x-debian-cross-container:
   extends: .container_job_template
   stage: containers
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index cce9faab36..e495b163a0 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -214,6 +214,7 @@ docker-image-debian-all-test-cross: docker-image-debian10
 docker-image-debian-microblaze-cross: docker-image-debian10
 docker-image-debian-nios2-cross: docker-image-debian10
 docker-image-debian-powerpc-test-cross: docker-image-debian11
+docker-image-debian-riscv64-test-cross: docker-image-debian11
 
 # These images may be good enough for building tests but not for test builds
 DOCKER_PARTIAL_IMAGES += debian-alpha-cross
@@ -222,6 +223,7 @@ DOCKER_PARTIAL_IMAGES += debian-hppa-cross
 DOCKER_PARTIAL_IMAGES += debian-m68k-cross debian-mips64-cross
 DOCKER_PARTIAL_IMAGES += debian-microblaze-cross
 DOCKER_PARTIAL_IMAGES += debian-nios2-cross
+DOCKER_PARTIAL_IMAGES += debian-riscv64-test-cross
 DOCKER_PARTIAL_IMAGES += debian-sh4-cross debian-sparc64-cross
 DOCKER_PARTIAL_IMAGES += debian-tricore-cross
 DOCKER_PARTIAL_IMAGES += debian-xtensa-cross
diff --git a/tests/docker/dockerfiles/debian-riscv64-test-cross.docker b/tests/docker/dockerfiles/debian-riscv64-test-cross.docker
new file mode 100644
index 0000000000..1d90901298
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-riscv64-test-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the Debian Bullseye base image.
+#
+FROM qemu/debian11
+
+RUN apt update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt install -y --no-install-recommends \
+        gcc-riscv64-linux-gnu \
+        libc6-dev-riscv64-cross
diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
index adc95d6a44..0663bd19f4 100755
--- a/tests/tcg/configure.sh
+++ b/tests/tcg/configure.sh
@@ -180,7 +180,7 @@ for target in $target_list; do
       ;;
     riscv64-*)
       container_hosts=x86_64
-      container_image=debian-riscv64-cross
+      container_image=debian-riscv64-test-cross
       container_cross_cc=riscv64-linux-gnu-gcc
       ;;
     s390x-*)
-- 
2.30.2

[PATCH v2 07/18] scripts/ci: add build env rules for aarch32 on aarch64

[Alex Bennée]

At least the current crop of Aarch64 HW can support running 32 bit EL0
code. Before we can build and test we need a minimal set of packages
installed. We can't use "apt build-dep" because it currently gets
confused trying to keep two sets of build-deps installed at once.
Instead we install a minimal set of libraries that will allow us to
continue.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220211160309.335014-7-alex.bennee@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 599896cc5b..9182e0c253 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -19,6 +19,13 @@
           - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)'
         msg: "Unsuitable ansible version, please use version 2.8.0 or later"
 
+    - name: Add armhf foreign architecture to aarch64 hosts
+      command: dpkg --add-architecture armhf
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] == 'aarch64'
+        - ansible_facts['distribution_version'] == '20.04'
+
     - name: Update apt cache / upgrade packages via apt
       apt:
         update_cache: yes
@@ -115,6 +122,24 @@
         - ansible_facts['distribution'] == 'Ubuntu'
         - ansible_facts['distribution_version'] == '20.04'
 
+    - name: Install armhf cross-compile packages to build QEMU on AArch64 Ubuntu 20.04
+      package:
+        name:
+          - binutils-arm-linux-gnueabihf
+          - gcc-arm-linux-gnueabihf
+          - libblkid-dev:armhf
+          - libc6-dev:armhf
+          - libffi-dev:armhf
+          - libglib2.0-dev:armhf
+          - libmount-dev:armhf
+          - libpcre2-dev:armhf
+          - libpixman-1-dev:armhf
+          - zlib1g-dev:armhf
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '20.04'
+        - ansible_facts['architecture'] == 'aarch64'
+
     - name: Install basic packages to build QEMU on EL8
       dnf:
         # This list of packages start with tests/docker/dockerfiles/centos8.docker
-- 
2.30.2

Re: [PATCH v2 07/18] scripts/ci: add build env rules for aarch32 on aarch64

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> At least the current crop of Aarch64 HW can support running 32 bit EL0
> code. Before we can build and test we need a minimal set of packages
> installed. We can't use "apt build-dep" because it currently gets
> confused trying to keep two sets of build-deps installed at once.
> Instead we install a minimal set of libraries that will allow us to
> continue.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Message-Id:<20220211160309.335014-7-alex.bennee@linaro.org>
> ---
>   scripts/ci/setup/build-environment.yml | 25 +++++++++++++++++++++++++
>   1 file changed, 25 insertions(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 08/18] scripts/ci: allow for a secondary runner

[Alex Bennée]

Some HW can run multiple architecture profiles so we can install a
secondary runner to build and run tests for those profiles. This
allows setting up secondary service.

[AJB: this is a fairly ugly hack, I'm sure this could be expressed in
a neater way]

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220211160309.335014-8-alex.bennee@linaro.org>
---
 scripts/ci/setup/gitlab-runner.yml | 34 ++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
index 1127db516f..19bfd68f7e 100644
--- a/scripts/ci/setup/gitlab-runner.yml
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -69,3 +69,37 @@
         name: gitlab-runner
         state: started
         enabled: yes
+
+    - name: Download secondary gitlab-runner
+      get_url:
+        dest: /usr/local/bin/gitlab-runner-arm
+        url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version  }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-arm"
+        owner: gitlab-runner
+        group: gitlab-runner
+        mode: u=rwx,g=rwx,o=rx
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] == 'aarch64'
+        - ansible_facts['distribution_version'] == '20.04'
+
+    - name: Register secondary gitlab-runner
+      command: "/usr/local/bin/gitlab-runner-arm register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+
+    - name: Install the secondary gitlab-runner service using its own functionality
+      command: /usr/local/bin/gitlab-runner-arm install --user gitlab-runner --working-directory /home/gitlab-runner/arm -n gitlab-runner-arm
+      register: gitlab_runner_install_service_result
+      failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] == 'aarch64'
+        - ansible_facts['distribution_version'] == '20.04'
+
+    - name: Enable the secondary gitlab-runner service
+      service:
+        name: gitlab-runner-arm
+        state: started
+        enabled: yes
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] == 'aarch64'
+        - ansible_facts['distribution_version'] == '20.04'
-- 
2.30.2

Re: [PATCH v2 08/18] scripts/ci: allow for a secondary runner

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> Some HW can run multiple architecture profiles so we can install a
> secondary runner to build and run tests for those profiles. This
> allows setting up secondary service.
> 
> [AJB: this is a fairly ugly hack, I'm sure this could be expressed in
> a neater way]
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Message-Id:<20220211160309.335014-8-alex.bennee@linaro.org>
> ---
>   scripts/ci/setup/gitlab-runner.yml | 34 ++++++++++++++++++++++++++++++
>   1 file changed, 34 insertions(+)

Ugly, but working I suppose.  :-)

Acked-by: Richard Henderson <richard.henderson@linaro.org>


r~

[PATCH v2 09/18] gitlab: add a new aarch32 custom runner definition

[Alex Bennée]

Although running on aarch64 hardware we can still target 32bit builds
with a cross compiler and run the resulting binaries.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220211160309.335014-9-alex.bennee@linaro.org>
---
 docs/devel/ci-jobs.rst.inc                    |  7 ++++++
 .../custom-runners/ubuntu-20.40-aarch32.yml   | 23 +++++++++++++++++++
 scripts/ci/setup/gitlab-runner.yml            |  4 ++++
 3 files changed, 34 insertions(+)
 create mode 100644 .gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml

diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index db3f571d5f..92e25872aa 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -44,6 +44,13 @@ If you've got access to an aarch64 host that can be used as a gitlab-CI
 runner, you can set this variable to enable the tests that require this
 kind of host. The runner should be tagged with "aarch64".
 
+AARCH32_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an armhf host or an arch64 host that can run
+aarch32 EL0 code to be used as a gitlab-CI runner, you can set this
+variable to enable the tests that require this kind of host. The
+runner should be tagged with "aarch32".
+
 S390X_RUNNER_AVAILABLE
 ~~~~~~~~~~~~~~~~~~~~~~
 If you've got access to an IBM Z host that can be used as a gitlab-CI
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml
new file mode 100644
index 0000000000..9c589bc4cf
--- /dev/null
+++ b/.gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml
@@ -0,0 +1,23 @@
+# All ubuntu-20.04 jobs should run successfully in an environment
+# setup by the scripts/ci/setup/qemu/build-environment.yml task
+# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+
+ubuntu-20.04-aarch32-all:
+ needs: []
+ stage: build
+ tags:
+ - ubuntu_20.04
+ - aarch32
+ rules:
+ - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
+   when: manual
+   allow_failure: true
+ - if: "$AARCH32_RUNNER_AVAILABLE"
+   when: manual
+   allow_failure: true
+ script:
+ - mkdir build
+ - cd build
+ - ../configure --cross-prefix=arm-linux-gnueabihf-
+ - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc` check V=1
diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
index 19bfd68f7e..33128be85d 100644
--- a/scripts/ci/setup/gitlab-runner.yml
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -84,6 +84,10 @@
 
     - name: Register secondary gitlab-runner
       command: "/usr/local/bin/gitlab-runner-arm register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['architecture'] == 'aarch64'
+        - ansible_facts['distribution_version'] == '20.04'
 
     - name: Install the secondary gitlab-runner service using its own functionality
       command: /usr/local/bin/gitlab-runner-arm install --user gitlab-runner --working-directory /home/gitlab-runner/arm -n gitlab-runner-arm
-- 
2.30.2

Re: [PATCH v2 09/18] gitlab: add a new aarch32 custom runner definition

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> +++ b/.gitlab-ci.d/custom-runners/ubuntu-20.40-aarch32.yml
> @@ -0,0 +1,23 @@
> +# All ubuntu-20.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/qemu/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +
> +ubuntu-20.04-aarch32-all:
> + needs: []
> + stage: build
> + tags:
> + - ubuntu_20.04
> + - aarch32
> + rules:
> + - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
> +   when: manual
> +   allow_failure: true

Why only manually run?

We do this for some, but not all, of the aarch64 jobs, to cut down on the runner load. 
However this is the only job defined for aarch32.


r~

[PATCH v2 10/18] tests/tcg/ppc64: clean-up handling of byte-reverse

[Alex Bennée]

Rather than having an else leg for the missing compiler case we can
simply just not add the test - the same way as is done for ppc64le.
Also while we are at it fix up the compiler invocation.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/ppc64/Makefile.target | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/tests/tcg/ppc64/Makefile.target b/tests/tcg/ppc64/Makefile.target
index 0368007028..9d6dfc1e26 100644
--- a/tests/tcg/ppc64/Makefile.target
+++ b/tests/tcg/ppc64/Makefile.target
@@ -10,19 +10,14 @@ PPC64_TESTS=bcdsub non_signalling_xscv
 endif
 $(PPC64_TESTS): CFLAGS += -mpower8-vector
 
-PPC64_TESTS += byte_reverse
 PPC64_TESTS += mtfsf
+
 ifneq ($(DOCKER_IMAGE)$(CROSS_CC_HAS_POWER10),)
+PPC64_TESTS += byte_reverse
+endif
+byte_reverse: CFLAGS += -mcpu=power10
 run-byte_reverse: QEMU_OPTS+=-cpu POWER10
 run-plugin-byte_reverse-with-%: QEMU_OPTS+=-cpu POWER10
-else
-byte_reverse:
-	$(call skip-test, "BUILD of $@", "missing compiler support")
-run-byte_reverse:
-	$(call skip-test, "RUN of byte_reverse", "not built")
-run-plugin-byte_reverse-with-%:
-	$(call skip-test, "RUN of byte_reverse ($*)", "not built")
-endif
 
 PPC64_TESTS += signal_save_restore_xer
 
-- 
2.30.2

Re: [PATCH v2 10/18] tests/tcg/ppc64: clean-up handling of byte-reverse

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> Rather than having an else leg for the missing compiler case we can
> simply just not add the test - the same way as is done for ppc64le.
> Also while we are at it fix up the compiler invocation.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> ---
>   tests/tcg/ppc64/Makefile.target | 13 ++++---------
>   1 file changed, 4 insertions(+), 9 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 11/18] tests/tcg: build sha1-vector with O3 and compare

[Alex Bennée]

The aim of this is to test code generation for vectorised operations.
Unfortunately gcc struggles to do much with the messy sha1 code (try
-fopt-info-vec-missed to see why). However it's better than nothing.

We assume the non-vectorised output is gold and baring compiler bugs
the outputs should match.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220202191242.652607-3-alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220211160309.335014-10-alex.bennee@linaro.org>
---
 tests/tcg/aarch64/Makefile.target | 10 ++++++++++
 tests/tcg/arm/Makefile.target     |  9 +++++++++
 2 files changed, 19 insertions(+)

diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target
index 1d967901bd..df3f8e9438 100644
--- a/tests/tcg/aarch64/Makefile.target
+++ b/tests/tcg/aarch64/Makefile.target
@@ -50,6 +50,16 @@ sysregs: CFLAGS+=-march=armv8.1-a+sve
 AARCH64_TESTS += sve-ioctls
 sve-ioctls: CFLAGS+=-march=armv8.1-a+sve
 
+# Vector SHA1
+sha1-vector: CFLAGS=-O3
+sha1-vector: sha1.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+run-sha1-vector: sha1-vector run-sha1
+	$(call run-test, $<, $(QEMU) $(QEMU_OPTS) $<, "$< on $(TARGET_NAME)")
+	$(call diff-out, sha1-vector, sha1.out)
+
+TESTS += sha1-vector
+
 ifneq ($(HAVE_GDB_BIN),)
 GDB_SCRIPT=$(SRC_PATH)/tests/guest-debug/run-test.py
 
diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
index f509d823d4..2dc94931c3 100644
--- a/tests/tcg/arm/Makefile.target
+++ b/tests/tcg/arm/Makefile.target
@@ -70,6 +70,15 @@ endif
 
 ARM_TESTS += commpage
 
+# Vector SHA1
+sha1-vector: CFLAGS=-O3
+sha1-vector: sha1.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+run-sha1-vector: sha1-vector run-sha1
+	$(call run-test, $<, $(QEMU) $(QEMU_OPTS) $<, "$< on $(TARGET_NAME)")
+	$(call diff-out, sha1-vector, sha1.out)
+
+ARM_TESTS += sha1-vector
 TESTS += $(ARM_TESTS)
 
 # On ARM Linux only supports 4k pages
-- 
2.30.2

[PATCH v2 12/18] tests/tcg: add sha512 test

[Alex Bennée]

This imports the sha512 algorithm and related tests from ccan which
offers a cleaner hash implementation with its own validation tests
with which we can exercise TCG code generations.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220211160309.335014-11-alex.bennee@linaro.org>

---

v2
  - s/#if BYTE_ORDER == BIG_ENDIAN/#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
---
 tests/tcg/multiarch/sha512.c | 990 +++++++++++++++++++++++++++++++++++
 1 file changed, 990 insertions(+)
 create mode 100644 tests/tcg/multiarch/sha512.c

diff --git a/tests/tcg/multiarch/sha512.c b/tests/tcg/multiarch/sha512.c
new file mode 100644
index 0000000000..e1729828b9
--- /dev/null
+++ b/tests/tcg/multiarch/sha512.c
@@ -0,0 +1,990 @@
+/*
+ * sha512 test based on CCAN: https://ccodearchive.net/info/crypto/sha512.html
+ *
+ * src/crypto/sha512.cpp commit f914f1a746d7f91951c1da262a4a749dd3ebfa71
+ * Copyright (c) 2014 The Bitcoin Core developers
+ * Distributed under the MIT software license, see:
+ *  http://www.opensource.org/licenses/mit-license.php.
+ *
+ * SPDX-License-Identifier: MIT CC0-1.0
+ */
+#define _GNU_SOURCE         /* See feature_test_macros(7) */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <ctype.h>
+#include <stdarg.h>
+
+/* Required portions from endian.h */
+
+/**
+ * BSWAP_64 - reverse bytes in a constant uint64_t value.
+ * @val: constantvalue whose bytes to swap.
+ *
+ * Designed to be usable in constant-requiring initializers.
+ *
+ * Example:
+ *  struct mystruct {
+ *      char buf[BSWAP_64(0xff00000000000000ULL)];
+ *  };
+ */
+#define BSWAP_64(val)                       \
+    ((((uint64_t)(val) & 0x00000000000000ffULL) << 56)  \
+     | (((uint64_t)(val) & 0x000000000000ff00ULL) << 40)    \
+     | (((uint64_t)(val) & 0x0000000000ff0000ULL) << 24)    \
+     | (((uint64_t)(val) & 0x00000000ff000000ULL) << 8)     \
+     | (((uint64_t)(val) & 0x000000ff00000000ULL) >> 8)     \
+     | (((uint64_t)(val) & 0x0000ff0000000000ULL) >> 24)    \
+     | (((uint64_t)(val) & 0x00ff000000000000ULL) >> 40)    \
+     | (((uint64_t)(val) & 0xff00000000000000ULL) >> 56))
+
+
+typedef uint64_t beint64_t;
+
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+
+/**
+ * CPU_TO_BE64 - convert a constant uint64_t value to big-endian
+ * @native: constant to convert
+ */
+#define CPU_TO_BE64(native) ((beint64_t)(native))
+/**
+ * BE64_TO_CPU - convert a big-endian uint64_t constant
+ * @le_val: big-endian constant to convert
+ */
+#define BE64_TO_CPU(le_val) ((uint64_t)(le_val))
+
+#else /* ... HAVE_LITTLE_ENDIAN */
+#define CPU_TO_BE64(native) ((beint64_t)BSWAP_64(native))
+#define BE64_TO_CPU(le_val) BSWAP_64((uint64_t)le_val)
+#endif /* HAVE_LITTE_ENDIAN */
+
+/**
+ * cpu_to_be64 - convert a uint64_t value to big endian.
+ * @native: value to convert
+ */
+static inline beint64_t cpu_to_be64(uint64_t native)
+{
+    return CPU_TO_BE64(native);
+}
+
+/**
+ * be64_to_cpu - convert a big-endian uint64_t value
+ * @be_val: big-endian value to convert
+ */
+static inline uint64_t be64_to_cpu(beint64_t be_val)
+{
+    return BE64_TO_CPU(be_val);
+}
+
+/* From compiler.h */
+
+#ifndef UNUSED
+/**
+ * UNUSED - a parameter is unused
+ *
+ * Some compilers (eg. gcc with -W or -Wunused) warn about unused
+ * function parameters.  This suppresses such warnings and indicates
+ * to the reader that it's deliberate.
+ *
+ * Example:
+ *  // This is used as a callback, so needs to have this prototype.
+ *  static int some_callback(void *unused UNUSED)
+ *  {
+ *      return 0;
+ *  }
+ */
+#define UNUSED __attribute__((__unused__))
+#endif
+
+/* From sha512.h */
+
+/**
+ * struct sha512 - structure representing a completed SHA512.
+ * @u.u8: an unsigned char array.
+ * @u.u64: a 64-bit integer array.
+ *
+ * Other fields may be added to the union in future.
+ */
+struct sha512 {
+    union {
+        uint64_t u64[8];
+        unsigned char u8[64];
+    } u;
+};
+
+/**
+ * sha512 - return sha512 of an object.
+ * @sha512: the sha512 to fill in
+ * @p: pointer to memory,
+ * @size: the number of bytes pointed to by @p
+ *
+ * The bytes pointed to by @p is SHA512 hashed into @sha512.  This is
+ * equivalent to sha512_init(), sha512_update() then sha512_done().
+ */
+void sha512(struct sha512 *sha, const void *p, size_t size);
+
+/**
+ * struct sha512_ctx - structure to store running context for sha512
+ */
+struct sha512_ctx {
+    uint64_t s[8];
+    union {
+        uint64_t u64[16];
+        unsigned char u8[128];
+    } buf;
+    size_t bytes;
+};
+
+/**
+ * sha512_init - initialize an SHA512 context.
+ * @ctx: the sha512_ctx to initialize
+ *
+ * This must be called before sha512_update or sha512_done, or
+ * alternately you can assign SHA512_INIT.
+ *
+ * If it was already initialized, this forgets anything which was
+ * hashed before.
+ *
+ * Example:
+ * static void hash_all(const char **arr, struct sha512 *hash)
+ * {
+ *  size_t i;
+ *  struct sha512_ctx ctx;
+ *
+ *  sha512_init(&ctx);
+ *  for (i = 0; arr[i]; i++)
+ *      sha512_update(&ctx, arr[i], strlen(arr[i]));
+ *  sha512_done(&ctx, hash);
+ * }
+ */
+void sha512_init(struct sha512_ctx *ctx);
+
+/**
+ * SHA512_INIT - initializer for an SHA512 context.
+ *
+ * This can be used to statically initialize an SHA512 context (instead
+ * of sha512_init()).
+ *
+ * Example:
+ * static void hash_all(const char **arr, struct sha512 *hash)
+ * {
+ *  size_t i;
+ *  struct sha512_ctx ctx = SHA512_INIT;
+ *
+ *  for (i = 0; arr[i]; i++)
+ *      sha512_update(&ctx, arr[i], strlen(arr[i]));
+ *  sha512_done(&ctx, hash);
+ * }
+ */
+#define SHA512_INIT                                         \
+    { { 0x6a09e667f3bcc908ull, 0xbb67ae8584caa73bull,   \
+        0x3c6ef372fe94f82bull, 0xa54ff53a5f1d36f1ull,   \
+        0x510e527fade682d1ull, 0x9b05688c2b3e6c1full,   \
+        0x1f83d9abfb41bd6bull, 0x5be0cd19137e2179ull }, \
+      { { 0 } }, 0 }
+
+/**
+ * sha512_update - include some memory in the hash.
+ * @ctx: the sha512_ctx to use
+ * @p: pointer to memory,
+ * @size: the number of bytes pointed to by @p
+ *
+ * You can call this multiple times to hash more data, before calling
+ * sha512_done().
+ */
+void sha512_update(struct sha512_ctx *ctx, const void *p, size_t size);
+
+/**
+ * sha512_done - finish SHA512 and return the hash
+ * @ctx: the sha512_ctx to complete
+ * @res: the hash to return.
+ *
+ * Note that @ctx is *destroyed* by this, and must be reinitialized.
+ * To avoid that, pass a copy instead.
+ */
+void sha512_done(struct sha512_ctx *sha512, struct sha512 *res);
+
+/* From sha512.c */
+
+/*
+ * SHA512 core code translated from the Bitcoin project's C++:
+ *
+ * src/crypto/sha512.cpp commit f914f1a746d7f91951c1da262a4a749dd3ebfa71
+ * Copyright (c) 2014 The Bitcoin Core developers
+ * Distributed under the MIT software license, see the accompanying
+ * file COPYING or http://www.opensource.org/licenses/mit-license.php.
+ */
+/* #include <ccan/endian/endian.h> */
+/* #include <ccan/compiler/compiler.h> */
+#include <stdbool.h>
+#include <assert.h>
+#include <string.h>
+
+static void invalidate_sha512(struct sha512_ctx *ctx)
+{
+    ctx->bytes = (size_t)-1;
+}
+
+static void check_sha512(struct sha512_ctx *ctx UNUSED)
+{
+    assert(ctx->bytes != (size_t)-1);
+}
+
+static uint64_t Ch(uint64_t x, uint64_t y, uint64_t z)
+{
+    return z ^ (x & (y ^ z));
+}
+static uint64_t Maj(uint64_t x, uint64_t y, uint64_t z)
+{
+    return (x & y) | (z & (x | y));
+}
+static uint64_t Sigma0(uint64_t x)
+{
+    return (x >> 28 | x << 36) ^ (x >> 34 | x << 30) ^ (x >> 39 | x << 25);
+}
+static uint64_t Sigma1(uint64_t x)
+{
+    return (x >> 14 | x << 50) ^ (x >> 18 | x << 46) ^ (x >> 41 | x << 23);
+}
+static uint64_t sigma0(uint64_t x)
+{
+    return (x >> 1 | x << 63) ^ (x >> 8 | x << 56) ^ (x >> 7);
+}
+static uint64_t sigma1(uint64_t x)
+{
+    return (x >> 19 | x << 45) ^ (x >> 61 | x << 3) ^ (x >> 6);
+}
+
+/** One round of SHA-512. */
+static void Round(uint64_t a, uint64_t b, uint64_t c, uint64_t *d, uint64_t e, uint64_t f, uint64_t g, uint64_t *h, uint64_t k, uint64_t w)
+{
+    uint64_t t1 = *h + Sigma1(e) + Ch(e, f, g) + k + w;
+    uint64_t t2 = Sigma0(a) + Maj(a, b, c);
+    *d += t1;
+    *h = t1 + t2;
+}
+
+/** Perform one SHA-512 transformation, processing a 128-byte chunk. */
+static void Transform(uint64_t *s, const uint64_t *chunk)
+{
+    uint64_t a = s[0], b = s[1], c = s[2], d = s[3], e = s[4], f = s[5], g = s[6], h = s[7];
+    uint64_t w0, w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12, w13, w14, w15;
+
+    Round(a, b, c, &d, e, f, g, &h, 0x428a2f98d728ae22ull, w0 = be64_to_cpu(chunk[0]));
+    Round(h, a, b, &c, d, e, f, &g, 0x7137449123ef65cdull, w1 = be64_to_cpu(chunk[1]));
+    Round(g, h, a, &b, c, d, e, &f, 0xb5c0fbcfec4d3b2full, w2 = be64_to_cpu(chunk[2]));
+    Round(f, g, h, &a, b, c, d, &e, 0xe9b5dba58189dbbcull, w3 = be64_to_cpu(chunk[3]));
+    Round(e, f, g, &h, a, b, c, &d, 0x3956c25bf348b538ull, w4 = be64_to_cpu(chunk[4]));
+    Round(d, e, f, &g, h, a, b, &c, 0x59f111f1b605d019ull, w5 = be64_to_cpu(chunk[5]));
+    Round(c, d, e, &f, g, h, a, &b, 0x923f82a4af194f9bull, w6 = be64_to_cpu(chunk[6]));
+    Round(b, c, d, &e, f, g, h, &a, 0xab1c5ed5da6d8118ull, w7 = be64_to_cpu(chunk[7]));
+    Round(a, b, c, &d, e, f, g, &h, 0xd807aa98a3030242ull, w8 = be64_to_cpu(chunk[8]));
+    Round(h, a, b, &c, d, e, f, &g, 0x12835b0145706fbeull, w9 = be64_to_cpu(chunk[9]));
+    Round(g, h, a, &b, c, d, e, &f, 0x243185be4ee4b28cull, w10 = be64_to_cpu(chunk[10]));
+    Round(f, g, h, &a, b, c, d, &e, 0x550c7dc3d5ffb4e2ull, w11 = be64_to_cpu(chunk[11]));
+    Round(e, f, g, &h, a, b, c, &d, 0x72be5d74f27b896full, w12 = be64_to_cpu(chunk[12]));
+    Round(d, e, f, &g, h, a, b, &c, 0x80deb1fe3b1696b1ull, w13 = be64_to_cpu(chunk[13]));
+    Round(c, d, e, &f, g, h, a, &b, 0x9bdc06a725c71235ull, w14 = be64_to_cpu(chunk[14]));
+    Round(b, c, d, &e, f, g, h, &a, 0xc19bf174cf692694ull, w15 = be64_to_cpu(chunk[15]));
+
+    Round(a, b, c, &d, e, f, g, &h, 0xe49b69c19ef14ad2ull, w0 += sigma1(w14) + w9 + sigma0(w1));
+    Round(h, a, b, &c, d, e, f, &g, 0xefbe4786384f25e3ull, w1 += sigma1(w15) + w10 + sigma0(w2));
+    Round(g, h, a, &b, c, d, e, &f, 0x0fc19dc68b8cd5b5ull, w2 += sigma1(w0) + w11 + sigma0(w3));
+    Round(f, g, h, &a, b, c, d, &e, 0x240ca1cc77ac9c65ull, w3 += sigma1(w1) + w12 + sigma0(w4));
+    Round(e, f, g, &h, a, b, c, &d, 0x2de92c6f592b0275ull, w4 += sigma1(w2) + w13 + sigma0(w5));
+    Round(d, e, f, &g, h, a, b, &c, 0x4a7484aa6ea6e483ull, w5 += sigma1(w3) + w14 + sigma0(w6));
+    Round(c, d, e, &f, g, h, a, &b, 0x5cb0a9dcbd41fbd4ull, w6 += sigma1(w4) + w15 + sigma0(w7));
+    Round(b, c, d, &e, f, g, h, &a, 0x76f988da831153b5ull, w7 += sigma1(w5) + w0 + sigma0(w8));
+    Round(a, b, c, &d, e, f, g, &h, 0x983e5152ee66dfabull, w8 += sigma1(w6) + w1 + sigma0(w9));
+    Round(h, a, b, &c, d, e, f, &g, 0xa831c66d2db43210ull, w9 += sigma1(w7) + w2 + sigma0(w10));
+    Round(g, h, a, &b, c, d, e, &f, 0xb00327c898fb213full, w10 += sigma1(w8) + w3 + sigma0(w11));
+    Round(f, g, h, &a, b, c, d, &e, 0xbf597fc7beef0ee4ull, w11 += sigma1(w9) + w4 + sigma0(w12));
+    Round(e, f, g, &h, a, b, c, &d, 0xc6e00bf33da88fc2ull, w12 += sigma1(w10) + w5 + sigma0(w13));
+    Round(d, e, f, &g, h, a, b, &c, 0xd5a79147930aa725ull, w13 += sigma1(w11) + w6 + sigma0(w14));
+    Round(c, d, e, &f, g, h, a, &b, 0x06ca6351e003826full, w14 += sigma1(w12) + w7 + sigma0(w15));
+    Round(b, c, d, &e, f, g, h, &a, 0x142929670a0e6e70ull, w15 += sigma1(w13) + w8 + sigma0(w0));
+
+    Round(a, b, c, &d, e, f, g, &h, 0x27b70a8546d22ffcull, w0 += sigma1(w14) + w9 + sigma0(w1));
+    Round(h, a, b, &c, d, e, f, &g, 0x2e1b21385c26c926ull, w1 += sigma1(w15) + w10 + sigma0(w2));
+    Round(g, h, a, &b, c, d, e, &f, 0x4d2c6dfc5ac42aedull, w2 += sigma1(w0) + w11 + sigma0(w3));
+    Round(f, g, h, &a, b, c, d, &e, 0x53380d139d95b3dfull, w3 += sigma1(w1) + w12 + sigma0(w4));
+    Round(e, f, g, &h, a, b, c, &d, 0x650a73548baf63deull, w4 += sigma1(w2) + w13 + sigma0(w5));
+    Round(d, e, f, &g, h, a, b, &c, 0x766a0abb3c77b2a8ull, w5 += sigma1(w3) + w14 + sigma0(w6));
+    Round(c, d, e, &f, g, h, a, &b, 0x81c2c92e47edaee6ull, w6 += sigma1(w4) + w15 + sigma0(w7));
+    Round(b, c, d, &e, f, g, h, &a, 0x92722c851482353bull, w7 += sigma1(w5) + w0 + sigma0(w8));
+    Round(a, b, c, &d, e, f, g, &h, 0xa2bfe8a14cf10364ull, w8 += sigma1(w6) + w1 + sigma0(w9));
+    Round(h, a, b, &c, d, e, f, &g, 0xa81a664bbc423001ull, w9 += sigma1(w7) + w2 + sigma0(w10));
+    Round(g, h, a, &b, c, d, e, &f, 0xc24b8b70d0f89791ull, w10 += sigma1(w8) + w3 + sigma0(w11));
+    Round(f, g, h, &a, b, c, d, &e, 0xc76c51a30654be30ull, w11 += sigma1(w9) + w4 + sigma0(w12));
+    Round(e, f, g, &h, a, b, c, &d, 0xd192e819d6ef5218ull, w12 += sigma1(w10) + w5 + sigma0(w13));
+    Round(d, e, f, &g, h, a, b, &c, 0xd69906245565a910ull, w13 += sigma1(w11) + w6 + sigma0(w14));
+    Round(c, d, e, &f, g, h, a, &b, 0xf40e35855771202aull, w14 += sigma1(w12) + w7 + sigma0(w15));
+    Round(b, c, d, &e, f, g, h, &a, 0x106aa07032bbd1b8ull, w15 += sigma1(w13) + w8 + sigma0(w0));
+
+    Round(a, b, c, &d, e, f, g, &h, 0x19a4c116b8d2d0c8ull, w0 += sigma1(w14) + w9 + sigma0(w1));
+    Round(h, a, b, &c, d, e, f, &g, 0x1e376c085141ab53ull, w1 += sigma1(w15) + w10 + sigma0(w2));
+    Round(g, h, a, &b, c, d, e, &f, 0x2748774cdf8eeb99ull, w2 += sigma1(w0) + w11 + sigma0(w3));
+    Round(f, g, h, &a, b, c, d, &e, 0x34b0bcb5e19b48a8ull, w3 += sigma1(w1) + w12 + sigma0(w4));
+    Round(e, f, g, &h, a, b, c, &d, 0x391c0cb3c5c95a63ull, w4 += sigma1(w2) + w13 + sigma0(w5));
+    Round(d, e, f, &g, h, a, b, &c, 0x4ed8aa4ae3418acbull, w5 += sigma1(w3) + w14 + sigma0(w6));
+    Round(c, d, e, &f, g, h, a, &b, 0x5b9cca4f7763e373ull, w6 += sigma1(w4) + w15 + sigma0(w7));
+    Round(b, c, d, &e, f, g, h, &a, 0x682e6ff3d6b2b8a3ull, w7 += sigma1(w5) + w0 + sigma0(w8));
+    Round(a, b, c, &d, e, f, g, &h, 0x748f82ee5defb2fcull, w8 += sigma1(w6) + w1 + sigma0(w9));
+    Round(h, a, b, &c, d, e, f, &g, 0x78a5636f43172f60ull, w9 += sigma1(w7) + w2 + sigma0(w10));
+    Round(g, h, a, &b, c, d, e, &f, 0x84c87814a1f0ab72ull, w10 += sigma1(w8) + w3 + sigma0(w11));
+    Round(f, g, h, &a, b, c, d, &e, 0x8cc702081a6439ecull, w11 += sigma1(w9) + w4 + sigma0(w12));
+    Round(e, f, g, &h, a, b, c, &d, 0x90befffa23631e28ull, w12 += sigma1(w10) + w5 + sigma0(w13));
+    Round(d, e, f, &g, h, a, b, &c, 0xa4506cebde82bde9ull, w13 += sigma1(w11) + w6 + sigma0(w14));
+    Round(c, d, e, &f, g, h, a, &b, 0xbef9a3f7b2c67915ull, w14 += sigma1(w12) + w7 + sigma0(w15));
+    Round(b, c, d, &e, f, g, h, &a, 0xc67178f2e372532bull, w15 += sigma1(w13) + w8 + sigma0(w0));
+
+    Round(a, b, c, &d, e, f, g, &h, 0xca273eceea26619cull, w0 += sigma1(w14) + w9 + sigma0(w1));
+    Round(h, a, b, &c, d, e, f, &g, 0xd186b8c721c0c207ull, w1 += sigma1(w15) + w10 + sigma0(w2));
+    Round(g, h, a, &b, c, d, e, &f, 0xeada7dd6cde0eb1eull, w2 += sigma1(w0) + w11 + sigma0(w3));
+    Round(f, g, h, &a, b, c, d, &e, 0xf57d4f7fee6ed178ull, w3 += sigma1(w1) + w12 + sigma0(w4));
+    Round(e, f, g, &h, a, b, c, &d, 0x06f067aa72176fbaull, w4 += sigma1(w2) + w13 + sigma0(w5));
+    Round(d, e, f, &g, h, a, b, &c, 0x0a637dc5a2c898a6ull, w5 += sigma1(w3) + w14 + sigma0(w6));
+    Round(c, d, e, &f, g, h, a, &b, 0x113f9804bef90daeull, w6 += sigma1(w4) + w15 + sigma0(w7));
+    Round(b, c, d, &e, f, g, h, &a, 0x1b710b35131c471bull, w7 += sigma1(w5) + w0 + sigma0(w8));
+    Round(a, b, c, &d, e, f, g, &h, 0x28db77f523047d84ull, w8 += sigma1(w6) + w1 + sigma0(w9));
+    Round(h, a, b, &c, d, e, f, &g, 0x32caab7b40c72493ull, w9 += sigma1(w7) + w2 + sigma0(w10));
+    Round(g, h, a, &b, c, d, e, &f, 0x3c9ebe0a15c9bebcull, w10 += sigma1(w8) + w3 + sigma0(w11));
+    Round(f, g, h, &a, b, c, d, &e, 0x431d67c49c100d4cull, w11 += sigma1(w9) + w4 + sigma0(w12));
+    Round(e, f, g, &h, a, b, c, &d, 0x4cc5d4becb3e42b6ull, w12 += sigma1(w10) + w5 + sigma0(w13));
+    Round(d, e, f, &g, h, a, b, &c, 0x597f299cfc657e2aull, w13 += sigma1(w11) + w6 + sigma0(w14));
+    Round(c, d, e, &f, g, h, a, &b, 0x5fcb6fab3ad6faecull, w14 + sigma1(w12) + w7 + sigma0(w15));
+    Round(b, c, d, &e, f, g, h, &a, 0x6c44198c4a475817ull, w15 + sigma1(w13) + w8 + sigma0(w0));
+
+    s[0] += a;
+    s[1] += b;
+    s[2] += c;
+    s[3] += d;
+    s[4] += e;
+    s[5] += f;
+    s[6] += g;
+    s[7] += h;
+}
+
+static bool alignment_ok(const void *p UNUSED, size_t n UNUSED)
+{
+#if HAVE_UNALIGNED_ACCESS
+    return true;
+#else
+    return ((size_t)p % n == 0);
+#endif
+}
+
+static void add(struct sha512_ctx *ctx, const void *p, size_t len)
+{
+    const unsigned char *data = p;
+    size_t bufsize = ctx->bytes % 128;
+
+    if (bufsize + len >= 128) {
+        /* Fill the buffer, and process it. */
+        memcpy(ctx->buf.u8 + bufsize, data, 128 - bufsize);
+        ctx->bytes += 128 - bufsize;
+        data += 128 - bufsize;
+        len -= 128 - bufsize;
+        Transform(ctx->s, ctx->buf.u64);
+        bufsize = 0;
+    }
+
+    while (len >= 128) {
+        /* Process full chunks directly from the source. */
+        if (alignment_ok(data, sizeof(uint64_t)))
+            Transform(ctx->s, (const uint64_t *)data);
+        else {
+            memcpy(ctx->buf.u8, data, sizeof(ctx->buf));
+            Transform(ctx->s, ctx->buf.u64);
+        }
+        ctx->bytes += 128;
+        data += 128;
+        len -= 128;
+    }
+
+    if (len) {
+        /* Fill the buffer with what remains. */
+        memcpy(ctx->buf.u8 + bufsize, data, len);
+        ctx->bytes += len;
+    }
+}
+
+void sha512_init(struct sha512_ctx *ctx)
+{
+    struct sha512_ctx init = SHA512_INIT;
+    *ctx = init;
+}
+
+void sha512_update(struct sha512_ctx *ctx, const void *p, size_t size)
+{
+    check_sha512(ctx);
+    add(ctx, p, size);
+}
+
+void sha512_done(struct sha512_ctx *ctx, struct sha512 *res)
+{
+    static const unsigned char pad[128] = { 0x80 };
+    uint64_t sizedesc[2] = { 0, 0 };
+    size_t i;
+
+    sizedesc[1] = cpu_to_be64((uint64_t)ctx->bytes << 3);
+
+    /* Add '1' bit to terminate, then all 0 bits, up to next block - 16. */
+    add(ctx, pad, 1 + ((256 - 16 - (ctx->bytes % 128) - 1) % 128));
+    /* Add number of bits of data (big endian) */
+    add(ctx, sizedesc, sizeof(sizedesc));
+    for (i = 0; i < sizeof(ctx->s) / sizeof(ctx->s[0]); i++)
+        res->u.u64[i] = cpu_to_be64(ctx->s[i]);
+    invalidate_sha512(ctx);
+}
+
+void sha512(struct sha512 *sha, const void *p, size_t size)
+{
+    struct sha512_ctx ctx;
+
+    sha512_init(&ctx);
+    sha512_update(&ctx, p, size);
+    sha512_done(&ctx, sha);
+}
+
+/* From hex.h */
+/**
+ * hex_decode - Unpack a hex string.
+ * @str: the hexidecimal string
+ * @slen: the length of @str
+ * @buf: the buffer to write the data into
+ * @bufsize: the length of @buf
+ *
+ * Returns false if there are any characters which aren't 0-9, a-f or A-F,
+ * of the string wasn't the right length for @bufsize.
+ *
+ * Example:
+ *  unsigned char data[20];
+ *
+ *  if (!hex_decode(argv[1], strlen(argv[1]), data, 20))
+ *      printf("String is malformed!\n");
+ */
+bool hex_decode(const char *str, size_t slen, void *buf, size_t bufsize);
+
+/**
+ * hex_encode - Create a nul-terminated hex string
+ * @buf: the buffer to read the data from
+ * @bufsize: the length of @buf
+ * @dest: the string to fill
+ * @destsize: the max size of the string
+ *
+ * Returns true if the string, including terminator, fit in @destsize;
+ *
+ * Example:
+ *  unsigned char buf[] = { 0x1F, 0x2F };
+ *  char str[5];
+ *
+ *  if (!hex_encode(buf, sizeof(buf), str, sizeof(str)))
+ *      abort();
+ */
+bool hex_encode(const void *buf, size_t bufsize, char *dest, size_t destsize);
+
+/**
+ * hex_str_size - Calculate how big a nul-terminated hex string is
+ * @bytes: bytes of data to represent
+ *
+ * Example:
+ *  unsigned char buf[] = { 0x1F, 0x2F };
+ *  char str[hex_str_size(sizeof(buf))];
+ *
+ *  hex_encode(buf, sizeof(buf), str, sizeof(str));
+ */
+static inline size_t hex_str_size(size_t bytes)
+{
+    return 2 * bytes + 1;
+}
+
+/* From hex.c */
+static bool char_to_hex(unsigned char *val, char c)
+{
+    if (c >= '0' && c <= '9') {
+        *val = c - '0';
+        return true;
+    }
+    if (c >= 'a' && c <= 'f') {
+        *val = c - 'a' + 10;
+        return true;
+    }
+    if (c >= 'A' && c <= 'F') {
+        *val = c - 'A' + 10;
+        return true;
+    }
+    return false;
+}
+
+bool hex_decode(const char *str, size_t slen, void *buf, size_t bufsize)
+{
+    unsigned char v1, v2;
+    unsigned char *p = buf;
+
+    while (slen > 1) {
+        if (!char_to_hex(&v1, str[0]) || !char_to_hex(&v2, str[1]))
+            return false;
+        if (!bufsize)
+            return false;
+        *(p++) = (v1 << 4) | v2;
+        str += 2;
+        slen -= 2;
+        bufsize--;
+    }
+    return slen == 0 && bufsize == 0;
+}
+
+static char hexchar(unsigned int val)
+{
+    if (val < 10)
+        return '0' + val;
+    if (val < 16)
+        return 'a' + val - 10;
+    abort();
+}
+
+bool hex_encode(const void *buf, size_t bufsize, char *dest, size_t destsize)
+{
+    size_t i;
+
+    if (destsize < hex_str_size(bufsize))
+        return false;
+
+    for (i = 0; i < bufsize; i++) {
+        unsigned int c = ((const unsigned char *)buf)[i];
+        *(dest++) = hexchar(c >> 4);
+        *(dest++) = hexchar(c & 0xF);
+    }
+    *dest = '\0';
+
+    return true;
+}
+
+/* From tap.h */
+/**
+ * plan_tests - announce the number of tests you plan to run
+ * @tests: the number of tests
+ *
+ * This should be the first call in your test program: it allows tracing
+ * of failures which mean that not all tests are run.
+ *
+ * If you don't know how many tests will actually be run, assume all of them
+ * and use skip() if you don't actually run some tests.
+ *
+ * Example:
+ *  plan_tests(13);
+ */
+void plan_tests(unsigned int tests);
+
+/**
+ * ok1 - Simple conditional test
+ * @e: the expression which we expect to be true.
+ *
+ * This is the simplest kind of test: if the expression is true, the
+ * test passes.  The name of the test which is printed will simply be
+ * file name, line number, and the expression itself.
+ *
+ * Example:
+ *  ok1(somefunc() == 1);
+ */
+# define ok1(e) ((e) ?                          \
+         _gen_result(1, __func__, __FILE__, __LINE__, "%s", #e) : \
+         _gen_result(0, __func__, __FILE__, __LINE__, "%s", #e))
+
+/**
+ * exit_status - the value that main should return.
+ *
+ * For maximum compatibility your test program should return a particular exit
+ * code (ie. 0 if all tests were run, and every test which was expected to
+ * succeed succeeded).
+ *
+ * Example:
+ *  exit(exit_status());
+ */
+int exit_status(void);
+
+/**
+ * tap_fail_callback - function to call when we fail
+ *
+ * This can be used to ease debugging, or exit on the first failure.
+ */
+void (*tap_fail_callback)(void);
+
+/* From tap.c */
+
+static int no_plan = 0;
+static int skip_all = 0;
+static int have_plan = 0;
+static unsigned int test_count = 0; /* Number of tests that have been run */
+static unsigned int e_tests = 0; /* Expected number of tests to run */
+static unsigned int failures = 0; /* Number of tests that failed */
+static char *todo_msg = NULL;
+static const char *todo_msg_fixed = "libtap malloc issue";
+static int todo = 0;
+static int test_died = 0;
+static int test_pid;
+
+static void
+_expected_tests(unsigned int tests)
+{
+    printf("1..%d\n", tests);
+    e_tests = tests;
+}
+
+static void
+diagv(const char *fmt, va_list ap)
+{
+    fputs("# ", stdout);
+    vfprintf(stdout, fmt, ap);
+    fputs("\n", stdout);
+}
+
+static void
+_diag(const char *fmt, ...)
+{
+    va_list ap;
+    va_start(ap, fmt);
+    diagv(fmt, ap);
+    va_end(ap);
+}
+
+/*
+ * Generate a test result.
+ *
+ * ok -- boolean, indicates whether or not the test passed.
+ * test_name -- the name of the test, may be NULL
+ * test_comment -- a comment to print afterwards, may be NULL
+ */
+unsigned int
+_gen_result(int ok, const char *func, const char *file, unsigned int line,
+        const char *test_name, ...)
+{
+    va_list ap;
+    char *local_test_name = NULL;
+    char *c;
+    int name_is_digits;
+
+    test_count++;
+
+    /* Start by taking the test name and performing any printf()
+       expansions on it */
+    if(test_name != NULL) {
+        va_start(ap, test_name);
+        if (vasprintf(&local_test_name, test_name, ap) < 0)
+            local_test_name = NULL;
+        va_end(ap);
+
+        /* Make sure the test name contains more than digits
+           and spaces.  Emit an error message and exit if it
+           does */
+        if(local_test_name) {
+            name_is_digits = 1;
+            for(c = local_test_name; *c != '\0'; c++) {
+                if(!isdigit((unsigned char)*c)
+                   && !isspace((unsigned char)*c)) {
+                    name_is_digits = 0;
+                    break;
+                }
+            }
+
+            if(name_is_digits) {
+                _diag("    You named your test '%s'.  You shouldn't use numbers for your test names.", local_test_name);
+                _diag("    Very confusing.");
+            }
+        }
+    }
+
+    if(!ok) {
+        printf("not ");
+        failures++;
+    }
+
+    printf("ok %d", test_count);
+
+    if(test_name != NULL) {
+        printf(" - ");
+
+        /* Print the test name, escaping any '#' characters it
+           might contain */
+        if(local_test_name != NULL) {
+            flockfile(stdout);
+            for(c = local_test_name; *c != '\0'; c++) {
+                if(*c == '#')
+                    fputc('\\', stdout);
+                fputc((int)*c, stdout);
+            }
+            funlockfile(stdout);
+        } else {    /* vasprintf() failed, use a fixed message */
+            printf("%s", todo_msg_fixed);
+        }
+    }
+
+    /* If we're in a todo_start() block then flag the test as being
+       TODO.  todo_msg should contain the message to print at this
+       point.  If it's NULL then asprintf() failed, and we should
+       use the fixed message.
+
+       This is not counted as a failure, so decrement the counter if
+       the test failed. */
+    if(todo) {
+        printf(" # TODO %s", todo_msg ? todo_msg : todo_msg_fixed);
+        if(!ok)
+            failures--;
+    }
+
+    printf("\n");
+
+    if(!ok)
+        _diag("    Failed %stest (%s:%s() at line %d)",
+              todo ? "(TODO) " : "", file, func, line);
+
+    free(local_test_name);
+
+    if (!ok && tap_fail_callback)
+        tap_fail_callback();
+
+    /* We only care (when testing) that ok is positive, but here we
+       specifically only want to return 1 or 0 */
+    return ok ? 1 : 0;
+}
+
+/*
+ * Cleanup at the end of the run, produce any final output that might be
+ * required.
+ */
+static void
+_cleanup(void)
+{
+    /* If we forked, don't do cleanup in child! */
+    if (getpid() != test_pid)
+        return;
+
+    /* If plan_no_plan() wasn't called, and we don't have a plan,
+       and we're not skipping everything, then something happened
+       before we could produce any output */
+    if(!no_plan && !have_plan && !skip_all) {
+        _diag("Looks like your test died before it could output anything.");
+        return;
+    }
+
+    if(test_died) {
+        _diag("Looks like your test died just after %d.", test_count);
+        return;
+    }
+
+
+    /* No plan provided, but now we know how many tests were run, and can
+       print the header at the end */
+    if(!skip_all && (no_plan || !have_plan)) {
+        printf("1..%d\n", test_count);
+    }
+
+    if((have_plan && !no_plan) && e_tests < test_count) {
+        _diag("Looks like you planned %d tests but ran %d extra.",
+              e_tests, test_count - e_tests);
+        return;
+    }
+
+    if((have_plan || !no_plan) && e_tests > test_count) {
+        _diag("Looks like you planned %d tests but only ran %d.",
+              e_tests, test_count);
+        if(failures) {
+            _diag("Looks like you failed %d tests of %d run.",
+                  failures, test_count);
+        }
+        return;
+    }
+
+    if(failures)
+        _diag("Looks like you failed %d tests of %d.",
+              failures, test_count);
+
+}
+
+/*
+ * Initialise the TAP library.  Will only do so once, however many times it's
+ * called.
+ */
+static void
+_tap_init(void)
+{
+    static int run_once = 0;
+
+    if(!run_once) {
+        test_pid = getpid();
+        atexit(_cleanup);
+
+        /* stdout needs to be unbuffered so that the output appears
+           in the same place relative to stderr output as it does
+           with Test::Harness */
+//      setbuf(stdout, 0);
+        run_once = 1;
+    }
+}
+
+/*
+ * Note the number of tests that will be run.
+ */
+void
+plan_tests(unsigned int tests)
+{
+
+    _tap_init();
+
+    if(have_plan != 0) {
+        fprintf(stderr, "You tried to plan twice!\n");
+        test_died = 1;
+        exit(255);
+    }
+
+    if(tests == 0) {
+        fprintf(stderr, "You said to run 0 tests!  You've got to run something.\n");
+        test_died = 1;
+        exit(255);
+    }
+
+    have_plan = 1;
+
+    _expected_tests(tests);
+}
+
+static int
+exit_status_(void)
+{
+    int r;
+
+    /* If there's no plan, just return the number of failures */
+    if(no_plan || !have_plan) {
+        return failures;
+    }
+
+    /* Ran too many tests?  Return the number of tests that were run
+       that shouldn't have been */
+    if(e_tests < test_count) {
+        r = test_count - e_tests;
+        return r;
+    }
+
+    /* Return the number of tests that failed + the number of tests
+       that weren't run */
+    r = failures + e_tests - test_count;
+
+    return r;
+}
+
+int
+exit_status(void)
+{
+    int r = exit_status_();
+    if (r > 255)
+        r = 255;
+    return r;
+}
+
+/* From run-test-vectors.c */
+
+/* Test vectors. */
+struct test {
+    const char *vector;
+    size_t repetitions;
+    const char *expected;
+};
+
+static const char ZEROES[] =
+    "0000000000000000000000000000000000000000000000000000000000000000"
+    "0000000000000000000000000000000000000000000000000000000000000000";
+
+static struct test tests[] = {
+    /* http://csrc.nist.gov/groups/STM/cavp/secure-hashing.html ShortMsg */
+    { "21", 1,
+      "3831a6a6155e509dee59a7f451eb35324d8f8f2df6e3708894740f98fdee2388"
+      "9f4de5adb0c5010dfb555cda77c8ab5dc902094c52de3278f35a75ebc25f093a" },
+    { "9083", 1,
+      "55586ebba48768aeb323655ab6f4298fc9f670964fc2e5f2731e34dfa4b0c09e"
+      "6e1e12e3d7286b3145c61c2047fb1a2a1297f36da64160b31fa4c8c2cddd2fb4" },
+    { "0a55db", 1,
+      "7952585e5330cb247d72bae696fc8a6b0f7d0804577e347d99bc1b11e52f3849"
+      "85a428449382306a89261ae143c2f3fb613804ab20b42dc097e5bf4a96ef919b" },
+    { "23be86d5", 1,
+      "76d42c8eadea35a69990c63a762f330614a4699977f058adb988f406fb0be8f2"
+      "ea3dce3a2bbd1d827b70b9b299ae6f9e5058ee97b50bd4922d6d37ddc761f8eb" },
+    { "eb0ca946c1", 1,
+      "d39ecedfe6e705a821aee4f58bfc489c3d9433eb4ac1b03a97e321a2586b40dd"
+      "0522f40fa5aef36afff591a78c916bfc6d1ca515c4983dd8695b1ec7951d723e" },
+    { "38667f39277b", 1,
+      "85708b8ff05d974d6af0801c152b95f5fa5c06af9a35230c5bea2752f031f9bd"
+      "84bd844717b3add308a70dc777f90813c20b47b16385664eefc88449f04f2131" },
+    { "b39f71aaa8a108", 1,
+      "258b8efa05b4a06b1e63c7a3f925c5ef11fa03e3d47d631bf4d474983783d8c0"
+      "b09449009e842fc9fa15de586c67cf8955a17d790b20f41dadf67ee8cdcdfce6" },
+    { "dc28484ebfd293d62ac759d5754bdf502423e4d419fa79020805134b2ce3dff7"
+      "38c7556c91d810adbad8dd210f041296b73c2185d4646c97fc0a5b69ed49ac8c"
+      "7ced0bd1cfd7e3c3cca47374d189247da6811a40b0ab097067ed4ad40ade2e47"
+      "91e39204e398b3204971445822a1be0dd93af8", 1,
+      "615115d2e8b62e345adaa4bdb95395a3b4fe27d71c4a111b86c1841463c5f03d"
+      "6b20d164a39948ab08ae060720d05c10f6022e5c8caf2fa3bca2e04d9c539ded" },
+    { "fd2203e467574e834ab07c9097ae164532f24be1eb5d88f1af7748ceff0d2c67"
+      "a21f4e4097f9d3bb4e9fbf97186e0db6db0100230a52b453d421f8ab9c9a6043"
+      "aa3295ea20d2f06a2f37470d8a99075f1b8a8336f6228cf08b5942fc1fb4299c"
+      "7d2480e8e82bce175540bdfad7752bc95b577f229515394f3ae5cec870a4b2f8",
+      1,
+      "a21b1077d52b27ac545af63b32746c6e3c51cb0cb9f281eb9f3580a6d4996d5c"
+      "9917d2a6e484627a9d5a06fa1b25327a9d710e027387fc3e07d7c4d14c6086cc" },
+    /* http://www.di-mgt.com.au/sha_testvectors.html */
+    { ZEROES, 1,
+      "7be9fda48f4179e611c698a73cff09faf72869431efee6eaad14de0cb44bbf66"
+      "503f752b7a8eb17083355f3ce6eb7d2806f236b25af96a24e22b887405c20081" }
+};
+
+static void *xmalloc(size_t size)
+{
+    char * ret;
+    ret = malloc(size);
+    if (ret == NULL) {
+        perror("malloc");
+        abort();
+    }
+    return ret;
+}
+
+static bool do_test(const struct test *t)
+{
+    struct sha512 h;
+    char got[128 + 1];
+    bool passed;
+    size_t i, vector_len = strlen(t->vector) / 2;
+    void *vector = xmalloc(vector_len);
+
+    hex_decode(t->vector, vector_len * 2, vector, vector_len);
+
+    for (i = 0; i < t->repetitions; i++) {
+        sha512(&h, vector, vector_len);
+        if (t->repetitions > 1)
+            memcpy(vector, &h, sizeof(h));
+    }
+
+    hex_encode(&h, sizeof(h), got, sizeof(got));
+
+    passed = strcmp(t->expected, got) == 0;
+    free(vector);
+    return passed;
+}
+
+int main(void)
+{
+    const size_t num_tests = sizeof(tests) / sizeof(tests[0]);
+    size_t i;
+
+    /* This is how many tests you plan to run */
+    plan_tests(num_tests);
+
+    for (i = 0; i < num_tests; i++)
+        ok1(do_test(&tests[i]));
+
+    /* This exits depending on whether all tests passed */
+    return exit_status();
+}
-- 
2.30.2

[PATCH v2 13/18] tests/tcg: add vectorised sha512 versions

[Alex Bennée]

This builds vectorised versions of sha512 to exercise the vector code:

  - aarch64 (AdvSimd)
  - i386 (SSE)
  - s390x (MVX)
  - ppc64/ppc64le (power10 vectors)

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220202191242.652607-5-alex.bennee@linaro.org>
Message-Id: <20220211160309.335014-12-alex.bennee@linaro.org>

---
v2
  - expanded to include both flavours of ppc64
---
 tests/tcg/aarch64/Makefile.target | 7 +++++++
 tests/tcg/arm/Makefile.target     | 8 ++++++++
 tests/tcg/i386/Makefile.target    | 6 ++++++
 tests/tcg/ppc64/Makefile.target   | 9 ++++++++-
 tests/tcg/ppc64le/Makefile.target | 9 ++++++++-
 tests/tcg/s390x/Makefile.target   | 9 +++++++++
 tests/tcg/x86_64/Makefile.target  | 7 +++++++
 7 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target
index df3f8e9438..ac07acde66 100644
--- a/tests/tcg/aarch64/Makefile.target
+++ b/tests/tcg/aarch64/Makefile.target
@@ -60,6 +60,13 @@ run-sha1-vector: sha1-vector run-sha1
 
 TESTS += sha1-vector
 
+# Vector versions of sha512 (-O3 triggers vectorisation)
+sha512-vector: CFLAGS=-O3
+sha512-vector: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+TESTS += sha512-vector
+
 ifneq ($(HAVE_GDB_BIN),)
 GDB_SCRIPT=$(SRC_PATH)/tests/guest-debug/run-test.py
 
diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
index 2dc94931c3..2f815120a5 100644
--- a/tests/tcg/arm/Makefile.target
+++ b/tests/tcg/arm/Makefile.target
@@ -79,6 +79,14 @@ run-sha1-vector: sha1-vector run-sha1
 	$(call diff-out, sha1-vector, sha1.out)
 
 ARM_TESTS += sha1-vector
+
+# Vector versions of sha512 (-O3 triggers vectorisation)
+sha512-vector: CFLAGS=-O3
+sha512-vector: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+ARM_TESTS += sha512-vector
+
 TESTS += $(ARM_TESTS)
 
 # On ARM Linux only supports 4k pages
diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index 38c10379af..099556efdc 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -71,3 +71,9 @@ TESTS=$(MULTIARCH_TESTS) $(I386_TESTS)
 
 # On i386 and x86_64 Linux only supports 4k pages (large pages are a different hack)
 EXTRA_RUNS+=run-test-mmap-4096
+
+sha512-sse: CFLAGS=-msse4.1 -O3
+sha512-sse: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+TESTS+=sha512-sse
diff --git a/tests/tcg/ppc64/Makefile.target b/tests/tcg/ppc64/Makefile.target
index 9d6dfc1e26..c9498053df 100644
--- a/tests/tcg/ppc64/Makefile.target
+++ b/tests/tcg/ppc64/Makefile.target
@@ -13,12 +13,19 @@ $(PPC64_TESTS): CFLAGS += -mpower8-vector
 PPC64_TESTS += mtfsf
 
 ifneq ($(DOCKER_IMAGE)$(CROSS_CC_HAS_POWER10),)
-PPC64_TESTS += byte_reverse
+PPC64_TESTS += byte_reverse sha512-vector
 endif
 byte_reverse: CFLAGS += -mcpu=power10
 run-byte_reverse: QEMU_OPTS+=-cpu POWER10
 run-plugin-byte_reverse-with-%: QEMU_OPTS+=-cpu POWER10
 
+sha512-vector: CFLAGS +=-mcpu=power10 -O3
+sha512-vector: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+run-sha512-vector: QEMU_OPTS+=-cpu POWER10
+run-plugin-sha512-vector-with-%: QEMU_OPTS+=-cpu POWER10
+
 PPC64_TESTS += signal_save_restore_xer
 
 TESTS += $(PPC64_TESTS)
diff --git a/tests/tcg/ppc64le/Makefile.target b/tests/tcg/ppc64le/Makefile.target
index 480ff0898d..12d85e946b 100644
--- a/tests/tcg/ppc64le/Makefile.target
+++ b/tests/tcg/ppc64le/Makefile.target
@@ -10,12 +10,19 @@ endif
 $(PPC64LE_TESTS): CFLAGS += -mpower8-vector
 
 ifneq ($(DOCKER_IMAGE)$(CROSS_CC_HAS_POWER10),)
-PPC64LE_TESTS += byte_reverse
+PPC64LE_TESTS += byte_reverse sha512-vector
 endif
 byte_reverse: CFLAGS += -mcpu=power10
 run-byte_reverse: QEMU_OPTS+=-cpu POWER10
 run-plugin-byte_reverse-with-%: QEMU_OPTS+=-cpu POWER10
 
+sha512-vector: CFLAGS +=-mcpu=power10 -O3
+sha512-vector: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+run-sha512-vector: QEMU_OPTS+=-cpu POWER10
+run-plugin-sha512-vector-with-%: QEMU_OPTS+=-cpu POWER10
+
 PPC64LE_TESTS += mtfsf
 PPC64LE_TESTS += signal_save_restore_xer
 
diff --git a/tests/tcg/s390x/Makefile.target b/tests/tcg/s390x/Makefile.target
index 1a7238b4eb..e53b599b22 100644
--- a/tests/tcg/s390x/Makefile.target
+++ b/tests/tcg/s390x/Makefile.target
@@ -25,3 +25,12 @@ run-gdbstub-signals-s390x: signals-s390x
 
 EXTRA_RUNS += run-gdbstub-signals-s390x
 endif
+
+# MVX versions of sha512
+sha512-mvx: CFLAGS=-march=z13 -mvx -O3
+sha512-mvx: sha512.c
+	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+run-sha512-mvx: QEMU_OPTS+=-cpu max
+
+TESTS+=sha512-mvx
diff --git a/tests/tcg/x86_64/Makefile.target b/tests/tcg/x86_64/Makefile.target
index 4a8a464c57..17cf168f0a 100644
--- a/tests/tcg/x86_64/Makefile.target
+++ b/tests/tcg/x86_64/Makefile.target
@@ -22,3 +22,10 @@ test-x86_64: test-i386.c test-i386.h test-i386-shift.h test-i386-muldiv.h
 
 vsyscall: $(SRC_PATH)/tests/tcg/x86_64/vsyscall.c
 	$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+
+# TCG does not yet support all SSE (SIGILL on pshufb)
+# sha512-sse: CFLAGS=-march=core2 -O3
+# sha512-sse: sha512.c
+# 	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+TESTS+=sha512-sse
-- 
2.30.2

Re: [PATCH v2 13/18] tests/tcg: add vectorised sha512 versions

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> +++ b/tests/tcg/i386/Makefile.target
> @@ -71,3 +71,9 @@ TESTS=$(MULTIARCH_TESTS) $(I386_TESTS)
>   
>   # On i386 and x86_64 Linux only supports 4k pages (large pages are a different hack)
>   EXTRA_RUNS+=run-test-mmap-4096
> +
> +sha512-sse: CFLAGS=-msse4.1 -O3
> +sha512-sse: sha512.c
> +	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
> +
> +TESTS+=sha512-sse

The default cpu, qemu32, only implements sse3, not sse4.1, so we get a guest SIGILL.  We 
can execute this with -cpu max, or we could limit the vectorization.


r~

Re: [PATCH v2 13/18] tests/tcg: add vectorised sha512 versions

[Alex Bennée]

Richard Henderson <richard.henderson@linaro.org> writes:

> On 2/25/22 07:20, Alex Bennée wrote:
>> +++ b/tests/tcg/i386/Makefile.target
>> @@ -71,3 +71,9 @@ TESTS=$(MULTIARCH_TESTS) $(I386_TESTS)
>>     # On i386 and x86_64 Linux only supports 4k pages (large pages
>> are a different hack)
>>   EXTRA_RUNS+=run-test-mmap-4096
>> +
>> +sha512-sse: CFLAGS=-msse4.1 -O3
>> +sha512-sse: sha512.c
>> +	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
>> +
>> +TESTS+=sha512-sse
>
> The default cpu, qemu32, only implements sse3, not sse4.1, so we get a
> guest SIGILL.  We can execute this with -cpu max, or we could limit
> the vectorization.

OK I'll add:

run-sha512-vector: QEMU_OPTS+=-cpu max

TESTS+=sha512-sse                                                          

>
>
> r~


-- 
Alex Bennée

Re: [PATCH v2 13/18] tests/tcg: add vectorised sha512 versions

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Richard Henderson <richard.henderson@linaro.org> writes:
>
>> On 2/25/22 07:20, Alex Bennée wrote:
>>> +++ b/tests/tcg/i386/Makefile.target
>>> @@ -71,3 +71,9 @@ TESTS=$(MULTIARCH_TESTS) $(I386_TESTS)
>>>     # On i386 and x86_64 Linux only supports 4k pages (large pages
>>> are a different hack)
>>>   EXTRA_RUNS+=run-test-mmap-4096
>>> +
>>> +sha512-sse: CFLAGS=-msse4.1 -O3
>>> +sha512-sse: sha512.c
>>> +	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
>>> +
>>> +TESTS+=sha512-sse
>>
>> The default cpu, qemu32, only implements sse3, not sse4.1, so we get a
>> guest SIGILL.  We can execute this with -cpu max, or we could limit
>> the vectorization.
>
> OK I'll add:
>
> run-sha512-vector: QEMU_OPTS+=-cpu max
>
> TESTS+=sha512-sse

Or you know something that will actually work :-/

  sha512-sse: CFLAGS=-msse4.1 -O3
  sha512-sse: sha512.c
          $(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)

  run-sha512-sse: QEMU_OPTS+=-cpu max
  run-plugin-sha512-sse-with-%: QEMU_OPTS+=-cpu max

  TESTS+=sha512-sse

>
>>
>>
>> r~


-- 
Alex Bennée

Re: [PATCH v2 13/18] tests/tcg: add vectorised sha512 versions

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> +# TCG does not yet support all SSE (SIGILL on pshufb)
> +# sha512-sse: CFLAGS=-march=core2 -O3
> +# sha512-sse: sha512.c
> +# 	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
> +

Given

ops_sse.h:void glue(helper_pshufb, SUFFIX)(CPUX86State *env, Reg *d, Reg *s)
ops_sse_header.h:DEF_HELPER_3(glue(pshufb, SUFFIX), void, env, Reg, Reg)
tcg/translate.c:    [0x00] = SSSE3_OP(pshufb),

I'm inclined to think -cpu max might work here as well.  Yep:

static const X86CPUDefinition builtin_x86_defs[] = {
     {
         .name = "qemu64",
...
         .features[FEAT_1_ECX] =
             CPUID_EXT_SSE3 | CPUID_EXT_CX16,

does not include SSE4.


r~

[PATCH v2 14/18] travis.yml: Update the s390x jobs to Ubuntu Focal

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

QEMU will soon drop the support for Ubuntu 18.04, so let's update
the Travis jobs that were still using this version to 20.04 instead.

While we're at it, also remove an obsolete comment about Ubuntu
Xenial being the default for our Travis jobs.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220221153423.1028465-1-thuth@redhat.com>
---
 .travis.yml | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 41010ebe6b..c3c8048842 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,6 +1,3 @@
-# The current Travis default is a VM based 16.04 Xenial on GCE
-# Additional builds with specific requirements for a full VM need to
-# be added as additional matrix: entries later on
 os: linux
 dist: focal
 language: c
@@ -190,7 +187,7 @@ jobs:
 
     - name: "[s390x] GCC check-tcg"
       arch: s390x
-      dist: bionic
+      dist: focal
       addons:
         apt_packages:
           - libaio-dev
@@ -233,7 +230,7 @@ jobs:
 
     - name: "[s390x] GCC (other-softmmu)"
       arch: s390x
-      dist: bionic
+      dist: focal
       addons:
         apt_packages:
           - libaio-dev
@@ -263,10 +260,11 @@ jobs:
 
     - name: "[s390x] GCC (user)"
       arch: s390x
-      dist: bionic
+      dist: focal
       addons:
         apt_packages:
           - libgcrypt20-dev
+          - libglib2.0-dev
           - libgnutls28-dev
           - ninja-build
       env:
@@ -274,7 +272,7 @@ jobs:
 
     - name: "[s390x] Clang (disable-tcg)"
       arch: s390x
-      dist: bionic
+      dist: focal
       compiler: clang
       addons:
         apt_packages:
-- 
2.30.2

Re: [PATCH v2 14/18] travis.yml: Update the s390x jobs to Ubuntu Focal

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> From: Thomas Huth<thuth@redhat.com>
> 
> QEMU will soon drop the support for Ubuntu 18.04, so let's update
> the Travis jobs that were still using this version to 20.04 instead.
> 
> While we're at it, also remove an obsolete comment about Ubuntu
> Xenial being the default for our Travis jobs.
> 
> Signed-off-by: Thomas Huth<thuth@redhat.com>
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Message-Id:<20220221153423.1028465-1-thuth@redhat.com>
> ---
>   .travis.yml | 12 +++++-------
>   1 file changed, 5 insertions(+), 7 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 15/18] gitlab: upgrade the job definition for s390x to 20.04

[Alex Bennée]

The new s390x machine has more of everything including the OS. As
18.04 will soon be going we might as well get onto something moderately
modern.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Peter Maydell <peter.maydell@linaro.org>
---
 .gitlab-ci.d/custom-runners.yml               |  2 +-
 ...18.04-s390x.yml => ubuntu-20.04-s390x.yml} | 28 +++++++++----------
 2 files changed, 15 insertions(+), 15 deletions(-)
 rename .gitlab-ci.d/custom-runners/{ubuntu-18.04-s390x.yml => ubuntu-20.04-s390x.yml} (87%)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index 056c374619..3e76a2034a 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -14,6 +14,6 @@ variables:
   GIT_STRATEGY: clone
 
 include:
-  - local: '/.gitlab-ci.d/custom-runners/ubuntu-18.04-s390x.yml'
+  - local: '/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml'
   - local: '/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml'
   - local: '/.gitlab-ci.d/custom-runners/centos-stream-8-x86_64.yml'
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-18.04-s390x.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
similarity index 87%
rename from .gitlab-ci.d/custom-runners/ubuntu-18.04-s390x.yml
rename to .gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
index f39d874a1e..0333872113 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-18.04-s390x.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
@@ -1,12 +1,12 @@
-# All ubuntu-18.04 jobs should run successfully in an environment
+# All ubuntu-20.04 jobs should run successfully in an environment
 # setup by the scripts/ci/setup/build-environment.yml task
-# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
+# "Install basic packages to build QEMU on Ubuntu 20.04/20.04"
 
-ubuntu-18.04-s390x-all-linux-static:
+ubuntu-20.04-s390x-all-linux-static:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
@@ -21,11 +21,11 @@ ubuntu-18.04-s390x-all-linux-static:
  - make --output-sync -j`nproc` check V=1
  - make --output-sync -j`nproc` check-tcg V=1
 
-ubuntu-18.04-s390x-all:
+ubuntu-20.04-s390x-all:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
@@ -37,11 +37,11 @@ ubuntu-18.04-s390x-all:
  - make --output-sync -j`nproc`
  - make --output-sync -j`nproc` check V=1
 
-ubuntu-18.04-s390x-alldbg:
+ubuntu-20.04-s390x-alldbg:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
@@ -58,11 +58,11 @@ ubuntu-18.04-s390x-alldbg:
  - make --output-sync -j`nproc`
  - make --output-sync -j`nproc` check V=1
 
-ubuntu-18.04-s390x-clang:
+ubuntu-20.04-s390x-clang:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
@@ -78,11 +78,11 @@ ubuntu-18.04-s390x-clang:
  - make --output-sync -j`nproc`
  - make --output-sync -j`nproc` check V=1
 
-ubuntu-18.04-s390x-tci:
+ubuntu-20.04-s390x-tci:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
@@ -97,11 +97,11 @@ ubuntu-18.04-s390x-tci:
  - ../configure --disable-libssh --enable-tcg-interpreter
  - make --output-sync -j`nproc`
 
-ubuntu-18.04-s390x-notcg:
+ubuntu-20.04-s390x-notcg:
  needs: []
  stage: build
  tags:
- - ubuntu_18.04
+ - ubuntu_20.04
  - s390x
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
-- 
2.30.2

Re: [PATCH v2 15/18] gitlab: upgrade the job definition for s390x to 20.04

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> The new s390x machine has more of everything including the OS. As
> 18.04 will soon be going we might as well get onto something moderately
> modern.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Acked-by: Christian Borntraeger<borntraeger@de.ibm.com>
> Reviewed-by: Thomas Huth<thuth@redhat.com>
> Acked-by: Cornelia Huck<cohuck@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé<f4bug@amsat.org>
> Cc: Peter Maydell<peter.maydell@linaro.org>
> ---
>   .gitlab-ci.d/custom-runners.yml               |  2 +-
>   ...18.04-s390x.yml => ubuntu-20.04-s390x.yml} | 28 +++++++++----------
>   2 files changed, 15 insertions(+), 15 deletions(-)
>   rename .gitlab-ci.d/custom-runners/{ubuntu-18.04-s390x.yml => ubuntu-20.04-s390x.yml} (87%)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 16/18] tests/tcg: completely disable threadcount for sh4

[Alex Bennée]

The previous disabling of threadcount 3bdc19af00 ("tests/tcg/sh4:
disable another unreliable test") just for plugins was being too
conservative. It's all broken so skip it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/sh4/Makefile.target | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/tcg/sh4/Makefile.target b/tests/tcg/sh4/Makefile.target
index 620ccc23c1..35ebe6b4e3 100644
--- a/tests/tcg/sh4/Makefile.target
+++ b/tests/tcg/sh4/Makefile.target
@@ -20,5 +20,7 @@ run-plugin-linux-test-with-%:
 	$(call skip-test, $<, "BROKEN")
 
 # This test is currently unreliable: https://gitlab.com/qemu-project/qemu/-/issues/856
+run-threadcount:
+	$(call skip-test, $<, "BROKEN")
 run-plugin-threadcount-with-%:
 	$(call skip-test, $<, "BROKEN")
-- 
2.30.2

Re: [PATCH v2 16/18] tests/tcg: completely disable threadcount for sh4

[Richard Henderson]

On 2/25/22 07:20, Alex Bennée wrote:
> The previous disabling of threadcount 3bdc19af00 ("tests/tcg/sh4:
> disable another unreliable test") just for plugins was being too
> conservative. It's all broken so skip it.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> ---
>   tests/tcg/sh4/Makefile.target | 2 ++
>   1 file changed, 2 insertions(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 17/18] semihosting/arm-compat: replace heuristic for softmmu SYS_HEAPINFO

[Alex Bennée]

The previous numbers were a guess at best and rather arbitrary without
taking into account anything that might be loaded. Instead of using
guesses based on the state of registers implement a new function that:

 a) scans the MemoryRegions for the largest RAM block
 b) iterates through all "ROM" blobs looking for the biggest gap

The "ROM" blobs include all code loaded via -kernel and the various
-device loader techniques.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Andrew Strauss <astrauss11@gmail.com>
Cc: Keith Packard <keithp@keithp.com>
Message-Id: <20210601090715.22330-1-alex.bennee@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220210113021.3799514-2-alex.bennee@linaro.org>

---
v7
  - remove stray comment
---
 include/hw/loader.h           |  14 ++++
 hw/core/loader.c              |  86 +++++++++++++++++++++++
 semihosting/arm-compat-semi.c | 124 +++++++++++++++++-----------------
 3 files changed, 163 insertions(+), 61 deletions(-)

diff --git a/include/hw/loader.h b/include/hw/loader.h
index 4fa485bd61..5572108ba5 100644
--- a/include/hw/loader.h
+++ b/include/hw/loader.h
@@ -343,4 +343,18 @@ int rom_add_option(const char *file, int32_t bootindex);
  * overflow on real hardware too. */
 #define UBOOT_MAX_GUNZIP_BYTES (64 << 20)
 
+typedef struct RomGap {
+    hwaddr base;
+    size_t size;
+} RomGap;
+
+/**
+ * rom_find_largest_gap_between: return largest gap between ROMs in given range
+ *
+ * Given a range of addresses, this function finds the largest
+ * contiguous subrange which has no ROMs loaded to it. That is,
+ * it finds the biggest gap which is free for use for other things.
+ */
+RomGap rom_find_largest_gap_between(hwaddr base, size_t size);
+
 #endif
diff --git a/hw/core/loader.c b/hw/core/loader.c
index 19edb928e9..ca2f2431fb 100644
--- a/hw/core/loader.c
+++ b/hw/core/loader.c
@@ -1333,6 +1333,92 @@ static Rom *find_rom(hwaddr addr, size_t size)
     return NULL;
 }
 
+typedef struct RomSec {
+    hwaddr base;
+    int se; /* start/end flag */
+} RomSec;
+
+
+/*
+ * Sort into address order. We break ties between rom-startpoints
+ * and rom-endpoints in favour of the startpoint, by sorting the 0->1
+ * transition before the 1->0 transition. Either way round would
+ * work, but this way saves a little work later by avoiding
+ * dealing with "gaps" of 0 length.
+ */
+static gint sort_secs(gconstpointer a, gconstpointer b)
+{
+    RomSec *ra = (RomSec *) a;
+    RomSec *rb = (RomSec *) b;
+
+    if (ra->base == rb->base) {
+        return ra->se - rb->se;
+    }
+    return ra->base > rb->base ? 1 : -1;
+}
+
+static GList *add_romsec_to_list(GList *secs, hwaddr base, int se)
+{
+   RomSec *cand = g_new(RomSec, 1);
+   cand->base = base;
+   cand->se = se;
+   return g_list_prepend(secs, cand);
+}
+
+RomGap rom_find_largest_gap_between(hwaddr base, size_t size)
+{
+    Rom *rom;
+    RomSec *cand;
+    RomGap res = {0, 0};
+    hwaddr gapstart = base;
+    GList *it, *secs = NULL;
+    int count = 0;
+
+    QTAILQ_FOREACH(rom, &roms, next) {
+        /* Ignore blobs being loaded to special places */
+        if (rom->mr || rom->fw_file) {
+            continue;
+        }
+        /* ignore anything finishing bellow base */
+        if (rom->addr + rom->romsize <= base) {
+            continue;
+        }
+        /* ignore anything starting above the region */
+        if (rom->addr >= base + size) {
+            continue;
+        }
+
+        /* Save the start and end of each relevant ROM */
+        secs = add_romsec_to_list(secs, rom->addr, 1);
+
+        if (rom->addr + rom->romsize < base + size) {
+            secs = add_romsec_to_list(secs, rom->addr + rom->romsize, -1);
+        }
+    }
+
+    /* sentinel */
+    secs = add_romsec_to_list(secs, base + size, 1);
+
+    secs = g_list_sort(secs, sort_secs);
+
+    for (it = g_list_first(secs); it; it = g_list_next(it)) {
+        cand = (RomSec *) it->data;
+        if (count == 0 && count + cand->se == 1) {
+            size_t gap = cand->base - gapstart;
+            if (gap > res.size) {
+                res.base = gapstart;
+                res.size = gap;
+            }
+        } else if (count == 1 && count + cand->se == 0) {
+            gapstart = cand->base;
+        }
+        count += cand->se;
+    }
+
+    g_list_free_full(secs, g_free);
+    return res;
+}
+
 /*
  * Copies memory from registered ROMs to dest. Any memory that is contained in
  * a ROM between addr and addr + size is copied. Note that this can involve
diff --git a/semihosting/arm-compat-semi.c b/semihosting/arm-compat-semi.c
index 37963becae..7a51fd0737 100644
--- a/semihosting/arm-compat-semi.c
+++ b/semihosting/arm-compat-semi.c
@@ -44,6 +44,7 @@
 #define COMMON_SEMI_HEAP_SIZE (128 * 1024 * 1024)
 #else
 #include "qemu/cutils.h"
+#include "hw/loader.h"
 #ifdef TARGET_ARM
 #include "hw/arm/boot.h"
 #endif
@@ -144,33 +145,69 @@ typedef struct GuestFD {
 static GArray *guestfd_array;
 
 #ifndef CONFIG_USER_ONLY
-#include "exec/address-spaces.h"
-/*
- * Find the base of a RAM region containing the specified address
+
+/**
+ * common_semi_find_bases: find information about ram and heap base
+ *
+ * This function attempts to provide meaningful numbers for RAM and
+ * HEAP base addresses. The rambase is simply the lowest addressable
+ * RAM position. For the heapbase we ask the loader to scan the
+ * address space and the largest available gap by querying the "ROM"
+ * regions.
+ *
+ * Returns: a structure with the numbers we need.
  */
-static inline hwaddr
-common_semi_find_region_base(hwaddr addr)
+
+typedef struct LayoutInfo {
+    target_ulong rambase;
+    size_t ramsize;
+    hwaddr heapbase;
+    hwaddr heaplimit;
+} LayoutInfo;
+
+static bool find_ram_cb(Int128 start, Int128 len, const MemoryRegion *mr,
+                        hwaddr offset_in_region, void *opaque)
 {
-    MemoryRegion *subregion;
+    LayoutInfo *info = (LayoutInfo *) opaque;
+    uint64_t size = int128_get64(len);
+
+    if (!mr->ram || mr->readonly) {
+        return false;
+    }
+
+    if (size > info->ramsize) {
+        info->rambase = int128_get64(start);
+        info->ramsize = size;
+    }
+
+    /* search exhaustively for largest RAM */
+    return false;
+}
+
+static LayoutInfo common_semi_find_bases(CPUState *cs)
+{
+    FlatView *fv;
+    LayoutInfo info = { 0, 0, 0, 0 };
+
+    RCU_READ_LOCK_GUARD();
+
+    fv = address_space_to_flatview(cs->as);
+    flatview_for_each_range(fv, find_ram_cb, &info);
 
     /*
-     * Find the chunk of R/W memory containing the address.  This is
-     * used for the SYS_HEAPINFO semihosting call, which should
-     * probably be using information from the loaded application.
+     * If we have found the RAM lets iterate through the ROM blobs to
+     * work out the best place for the remainder of RAM and split it
+     * equally between stack and heap.
      */
-    QTAILQ_FOREACH(subregion, &get_system_memory()->subregions,
-                   subregions_link) {
-        if (subregion->ram && !subregion->readonly) {
-            Int128 top128 = int128_add(int128_make64(subregion->addr),
-                                       subregion->size);
-            Int128 addr128 = int128_make64(addr);
-            if (subregion->addr <= addr && int128_lt(addr128, top128)) {
-                return subregion->addr;
-            }
-        }
+    if (info.rambase || info.ramsize > 0) {
+        RomGap gap = rom_find_largest_gap_between(info.rambase, info.ramsize);
+        info.heapbase = gap.base;
+        info.heaplimit = gap.base + gap.size;
     }
-    return 0;
+
+    return info;
 }
+
 #endif
 
 #ifdef TARGET_ARM
@@ -204,28 +241,6 @@ common_semi_sys_exit_extended(CPUState *cs, int nr)
     return (nr == TARGET_SYS_EXIT_EXTENDED || is_a64(cs->env_ptr));
 }
 
-#ifndef CONFIG_USER_ONLY
-#include "hw/arm/boot.h"
-static inline target_ulong
-common_semi_rambase(CPUState *cs)
-{
-    CPUArchState *env = cs->env_ptr;
-    const struct arm_boot_info *info = env->boot_info;
-    target_ulong sp;
-
-    if (info) {
-        return info->loader_start;
-    }
-
-    if (is_a64(env)) {
-        sp = env->xregs[31];
-    } else {
-        sp = env->regs[13];
-    }
-    return common_semi_find_region_base(sp);
-}
-#endif
-
 #endif /* TARGET_ARM */
 
 #ifdef TARGET_RISCV
@@ -251,17 +266,6 @@ common_semi_sys_exit_extended(CPUState *cs, int nr)
     return (nr == TARGET_SYS_EXIT_EXTENDED || sizeof(target_ulong) == 8);
 }
 
-#ifndef CONFIG_USER_ONLY
-
-static inline target_ulong
-common_semi_rambase(CPUState *cs)
-{
-    RISCVCPU *cpu = RISCV_CPU(cs);
-    CPURISCVState *env = &cpu->env;
-    return common_semi_find_region_base(env->gpr[xSP]);
-}
-#endif
-
 #endif
 
 /*
@@ -1165,12 +1169,12 @@ target_ulong do_common_semihosting(CPUState *cs)
     case TARGET_SYS_HEAPINFO:
         {
             target_ulong retvals[4];
-            target_ulong limit;
             int i;
 #ifdef CONFIG_USER_ONLY
             TaskState *ts = cs->opaque;
+            target_ulong limit;
 #else
-            target_ulong rambase = common_semi_rambase(cs);
+            LayoutInfo info = common_semi_find_bases(cs);
 #endif
 
             GET_ARG(0);
@@ -1201,12 +1205,10 @@ target_ulong do_common_semihosting(CPUState *cs)
             retvals[2] = ts->stack_base;
             retvals[3] = 0; /* Stack limit.  */
 #else
-            limit = current_machine->ram_size;
-            /* TODO: Make this use the limit of the loaded application.  */
-            retvals[0] = rambase + limit / 2;
-            retvals[1] = rambase + limit;
-            retvals[2] = rambase + limit; /* Stack base */
-            retvals[3] = rambase; /* Stack limit.  */
+            retvals[0] = info.heapbase;  /* Heap Base */
+            retvals[1] = info.heaplimit; /* Heap Limit */
+            retvals[2] = info.heaplimit; /* Stack base */
+            retvals[3] = info.heapbase;  /* Stack limit.  */
 #endif
 
             for (i = 0; i < ARRAY_SIZE(retvals); i++) {
-- 
2.30.2

[PATCH v2 18/18] tests/tcg: port SYS_HEAPINFO to a system test

[Alex Bennée]

This allows us to check our new SYS_HEAPINFO implementation generates
sane values.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20220210113021.3799514-3-alex.bennee@linaro.org>
---
 tests/tcg/aarch64/system/semiheap.c | 93 +++++++++++++++++++++++++++++
 MAINTAINERS                         |  1 +
 2 files changed, 94 insertions(+)
 create mode 100644 tests/tcg/aarch64/system/semiheap.c

diff --git a/tests/tcg/aarch64/system/semiheap.c b/tests/tcg/aarch64/system/semiheap.c
new file mode 100644
index 0000000000..4ed258476d
--- /dev/null
+++ b/tests/tcg/aarch64/system/semiheap.c
@@ -0,0 +1,93 @@
+/*
+ * Semihosting System HEAPINFO Test
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <minilib.h>
+
+#define SYS_HEAPINFO    0x16
+
+uintptr_t __semi_call(uintptr_t type, uintptr_t arg0)
+{
+    register uintptr_t t asm("x0") = type;
+    register uintptr_t a0 asm("x1") = arg0;
+    asm("hlt 0xf000"
+        : "=r" (t)
+        : "r" (t), "r" (a0)
+        : "memory" );
+
+    return t;
+}
+
+int main(int argc, char *argv[argc])
+{
+    struct {
+        void *heap_base;
+        void *heap_limit;
+        void *stack_base;
+        void *stack_limit;
+    } info = { };
+    void *ptr_to_info = (void *) &info;
+    uint32_t *ptr_to_heap;
+    int i;
+
+    ml_printf("Semihosting Heap Info Test\n");
+
+    __semi_call(SYS_HEAPINFO, (uintptr_t) &ptr_to_info);
+
+    if (info.heap_base == NULL || info.heap_limit == NULL) {
+        ml_printf("null heap: %p -> %p\n", info.heap_base, info.heap_limit);
+        return -1;
+    }
+
+    /* Error if heap base is above limit */
+    if ((uintptr_t) info.heap_base >= (uintptr_t) info.heap_limit) {
+        ml_printf("heap base %p >= heap_limit %p\n",
+               info.heap_base, info.heap_limit);
+        return -2;
+    }
+
+    if (info.stack_base == NULL) {
+        ml_printf("null stack: %p -> %p\n", info.stack_base, info.stack_limit);
+        return -3;
+    }
+
+    /*
+     * boot.S put our stack somewhere inside the data segment of the
+     * ELF file, and we know that SYS_HEAPINFO won't pick a range
+     * that overlaps with part of a loaded ELF file. So the info
+     * struct (on the stack) should not be inside the reported heap.
+     */
+    if (ptr_to_info > info.heap_base && ptr_to_info < info.heap_limit) {
+        ml_printf("info appears to be inside the heap: %p in %p:%p\n",
+               ptr_to_info, info.heap_base, info.heap_limit);
+        return -4;
+    }
+
+    ml_printf("heap: %p -> %p\n", info.heap_base, info.heap_limit);
+    ml_printf("stack: %p <- %p\n", info.stack_limit, info.stack_base);
+
+    /* finally can we read/write the heap */
+    ptr_to_heap = (uint32_t *) info.heap_base;
+    for (i = 0; i < 512; i++) {
+        *ptr_to_heap++ = i;
+    }
+    ptr_to_heap = (uint32_t *) info.heap_base;
+    for (i = 0; i < 512; i++) {
+        uint32_t tmp = *ptr_to_heap;
+        if (tmp != i) {
+            ml_printf("unexpected value in heap: %d @ %p", tmp, ptr_to_heap);
+            return -5;
+        }
+        ptr_to_heap++;
+    }
+    ml_printf("r/w to heap upto %p\n", ptr_to_heap);
+
+    ml_printf("Passed HeapInfo checks\n");
+    return 0;
+}
diff --git a/MAINTAINERS b/MAINTAINERS
index fa8adc2618..68adaac373 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3549,6 +3549,7 @@ S: Maintained
 F: semihosting/
 F: include/semihosting/
 F: tests/tcg/multiarch/arm-compat-semi/
+F: tests/tcg/aarch64/system/semiheap.c
 
 Multi-process QEMU
 M: Elena Ufimtseva <elena.ufimtseva@oracle.com>
-- 
2.30.2