[PULL 00/12] testing and plugin fixes

[PULL 00/12] testing and plugin fixes

[Alex Bennée]

The following changes since commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:

  Merge remote-tracking branch 'remotes/gkurz/tags/9p-next-2020-05-26' into staging (2020-05-26 14:05:53 +0100)

are available in the Git repository at:

  https://github.com/stsquad/qemu.git tags/pull-testing-tcg-plugins-270520-1

for you to fetch changes up to 919bfbf5d6569b63a374332292cf3d2355a6d6c3:

  tests/tcg: add new threadcount test (2020-05-27 14:26:49 +0100)

----------------------------------------------------------------
Testing and one plugin fix:

  - support alternates for genisoimage to test/vm
  - add clang++ to clang tests
  - fix record/replay smoke test
  - enable more softfloat tests
  - better detection of hung gdb
  - upgrade aarch64 tcg test x-compile to gcc-10
  - fix plugin cpu_index clash vs threads

----------------------------------------------------------------
Alex Bennée (11):
      configure: add alternate binary for genisoimage
      tests/vm: pass --genisoimage to basevm script
      tests/tcg: fix invocation of the memory record/replay tests
      tests/fp: enable extf80_le_quite tests
      tests/fp: split and audit the conversion tests
      tests/tcg: better detect confused gdb which can't connect
      tests/docker: add debian11 base image
      tests/docker: use a gcc-10 based image for arm64 tests
      cpus-common: ensure auto-assigned cpu_indexes don't clash
      linux-user: properly "unrealize" vCPU object
      tests/tcg: add new threadcount test

Thomas Huth (1):
      travis.yml: Use clang++ in the Clang tests

 configure                                          |  2 +-
 cpus-common.c                                      | 10 ++--
 linux-user/syscall.c                               | 19 ++++---
 tests/tcg/multiarch/threadcount.c                  | 64 ++++++++++++++++++++++
 .travis.yml                                        |  8 ++-
 tests/Makefile.include                             | 44 +++++++++++----
 tests/docker/Makefile.include                      |  4 +-
 .../dockerfiles/debian-arm64-test-cross.docker     | 13 +++++
 tests/docker/dockerfiles/debian11.docker           | 18 ++++++
 tests/tcg/aarch64/Makefile.softmmu-target          |  2 +-
 tests/tcg/configure.sh                             |  4 +-
 tests/tcg/multiarch/Makefile.target                |  2 +
 tests/tcg/multiarch/gdbstub/sha1.py                |  4 ++
 tests/vm/Makefile.include                          |  1 +
 tests/vm/basevm.py                                 | 16 ++++--
 15 files changed, 172 insertions(+), 39 deletions(-)
 create mode 100644 tests/tcg/multiarch/threadcount.c
 create mode 100644 tests/docker/dockerfiles/debian-arm64-test-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian11.docker

-- 
2.20.1

[PULL 01/12] configure: add alternate binary for genisoimage

[Alex Bennée]

Not all distros ship genisoimage which is a Debian fork from the
original cdrtools. As the options are pretty much the same support it
as a fallback binary.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Message-Id: <20200519132259.405-2-robert.foley@linaro.org>

diff --git a/configure b/configure
index b969dee675b..af2ba83f0e2 100755
--- a/configure
+++ b/configure
@@ -941,7 +941,7 @@ done
 
 # Check for ancillary tools used in testing
 genisoimage=
-for binary in genisoimage
+for binary in genisoimage mkisofs
 do
     if has $binary
     then
-- 
2.20.1

[PULL 02/12] tests/vm: pass --genisoimage to basevm script

[Alex Bennée]

If we have an alternative to genisoimage we really need to tell the
script about it as well so it can use it. It will still default to
genisoimage in case it is run outside our build machinery.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Robert Foley <robert.foley@linaro.org>
Message-Id: <20200519132259.405-3-robert.foley@linaro.org>

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 1bf9693d195..74ab522c55d 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -56,6 +56,7 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
 	$(call quiet-command, \
 		$(PYTHON) $< \
 		$(if $(V)$(DEBUG), --debug) \
+		$(if $(GENISOIMAGE),--genisoimage $(GENISOIMAGE)) \
 		--image "$@" \
 		--force \
 		--build-image $@, \
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 756ccf7acae..a2d4054d72b 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -61,8 +61,9 @@ class BaseVM(object):
     # 4 is arbitrary, but greater than 2,
     # since we found we need to wait more than twice as long.
     tcg_ssh_timeout_multiplier = 4
-    def __init__(self, debug=False, vcpus=None):
+    def __init__(self, debug=False, vcpus=None, genisoimage=None):
         self._guest = None
+        self._genisoimage = genisoimage
         self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",
                                                          suffix=".tmp",
                                                          dir="."))
@@ -381,12 +382,12 @@ class BaseVM(object):
             udata.writelines(["apt:\n",
                               "  proxy: %s" % proxy])
         udata.close()
-        subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
+        subprocess.check_call([self._genisoimage, "-output", "cloud-init.iso",
                                "-volid", "cidata", "-joliet", "-rock",
                                "user-data", "meta-data"],
-                               cwd=cidir,
-                               stdin=self._devnull, stdout=self._stdout,
-                               stderr=self._stdout)
+                              cwd=cidir,
+                              stdin=self._devnull, stdout=self._stdout,
+                              stderr=self._stdout)
 
         return os.path.join(cidir, "cloud-init.iso")
 
@@ -424,6 +425,8 @@ def parse_args(vmcls):
                       help="Interactively run command")
     parser.add_option("--snapshot", "-s", action="store_true",
                       help="run tests with a snapshot")
+    parser.add_option("--genisoimage", default="genisoimage",
+                      help="iso imaging tool")
     parser.disable_interspersed_args()
     return parser.parse_args()
 
@@ -435,7 +438,8 @@ def main(vmcls):
             return 1
         logging.basicConfig(level=(logging.DEBUG if args.debug
                                    else logging.WARN))
-        vm = vmcls(debug=args.debug, vcpus=args.jobs)
+        vm = vmcls(debug=args.debug, vcpus=args.jobs,
+                   genisoimage=args.genisoimage)
         if args.build_image:
             if os.path.exists(args.image) and not args.force:
                 sys.stderr.writelines(["Image file exists: %s\n" % args.image,
-- 
2.20.1

[PULL 03/12] travis.yml: Use clang++ in the Clang tests

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

Our configure script does not look for clang++ automatically, so we
should use --cxx=clang++ to make sure that we test our C++ code with
Clang, too. And while we're at it, also use --host-cc=clang here
to avoid that we use the normal "cc" as host C compiler.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200518083316.25065-1-thuth@redhat.com>
Message-Id: <20200520140541.30256-4-alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 1ec8a7b4657..564be50a3c1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -205,14 +205,15 @@ jobs:
     # Test with Clang for compile portability (Travis uses clang-5.0)
     - name: "Clang (user)"
       env:
-        - CONFIG="--disable-system"
+        - CONFIG="--disable-system --host-cc=clang --cxx=clang++"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
     - name: "Clang (main-softmmu)"
       env:
-        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS} "
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS}
+                  --host-cc=clang --cxx=clang++"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-sanitize"
       compiler: clang
       before_script:
@@ -222,7 +223,8 @@ jobs:
 
     - name: "Clang (other-softmmu)"
       env:
-        - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+        - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}
+                  --host-cc=clang --cxx=clang++"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
-- 
2.20.1

[PULL 04/12] tests/tcg: fix invocation of the memory record/replay tests

[Alex Bennée]

I'm not sure when this broke but we should use EXTRA_RUNS for
"virtual" tests which are not generated from the binary names.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

Message-Id: <20200520140541.30256-5-alex.bennee@linaro.org>

diff --git a/tests/tcg/aarch64/Makefile.softmmu-target b/tests/tcg/aarch64/Makefile.softmmu-target
index 71f72cfbe34..1057a8ac498 100644
--- a/tests/tcg/aarch64/Makefile.softmmu-target
+++ b/tests/tcg/aarch64/Makefile.softmmu-target
@@ -61,7 +61,7 @@ run-memory-replay: memory-replay run-memory-record
 	   	  $(QEMU_OPTS) memory, \
 	  "$< on $(TARGET_NAME)")
 
-EXTRA_TESTS+=memory-record memory-replay
+EXTRA_RUNS+=run-memory-replay
 
 ifneq ($(DOCKER_IMAGE)$(CROSS_CC_HAS_ARMV8_3),)
 pauth-3: CFLAGS += -march=armv8.3-a
-- 
2.20.1

[PULL 05/12] tests/fp: enable extf80_le_quite tests

[Alex Bennée]

These have been fixed now so we no longer need a special version of
the le_quiet rule to skip the test.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200520140541.30256-6-alex.bennee@linaro.org>

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 03a74b60f6b..e6d87fcbf0e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -741,13 +741,6 @@ check-softfloat-%: $(FP_TEST_BIN)
 SF_COMPARE_OPS=eq eq_signaling le le_quiet lt_quiet
 SF_COMPARE_RULES=$(patsubst %,check-softfloat-%, $(SF_COMPARE_OPS))
 
-# FIXME: extF80_le_quiet (broken)
-check-softfloat-le_quiet: $(FP_TEST_BIN)
-	$(call test-softfloat, 				\
-		f16_le_quiet f32_le_quiet f64_le_quiet  \
-		f128_le_quiet, 				\
-		le_quiet)
-
 # FIXME: extF80_lt_quiet (broken)
 check-softfloat-lt_quiet: $(FP_TEST_BIN)
 	$(call test-softfloat, 				\
-- 
2.20.1

[PULL 06/12] tests/fp: split and audit the conversion tests

[Alex Bennée]

Split the float conversion tests into separate groups and audit the
tests to check what is still broken. I was able to enable a bunch of
tests that had been missed before:

  all the float to float conversions
  ui32_to_extF80
  ui64_to_extF80
  extF80_to_ui32
  extF80_to_ui32_r_minMag
  extF80_to_ui64
  extF80_to_ui64_r_minMag

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200520140541.30256-7-alex.bennee@linaro.org>

diff --git a/tests/Makefile.include b/tests/Makefile.include
index e6d87fcbf0e..a00ccc94b8b 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -687,11 +687,26 @@ test-softfloat = $(call quiet-command, \
 			(cat $2.out && exit 1;), \
 			"FLOAT TEST", $2)
 
-# Conversion Routines:
+# Conversion Routines: Float to Float
+# FIXME: f32_to_f128 (broken), f64_to_f128 (broken)
+# FIXME: f128_to_f32(broken), f128_to_f64 (broken)
+# FIXME: f128_to_extF80 (broken)
+check-softfloat-conv-f2f: $(FP_TEST_BIN)
+	$(call test-softfloat, \
+		f16_to_f32 f16_to_f64 \
+		f16_to_extF80 f16_to_f128 \
+		f32_to_f16 f32_to_f64 \
+		f32_to_extF80 \
+		f64_to_f16 f64_to_f32 \
+		extF80_to_f16 extF80_to_f32 \
+		extF80_to_f64 extF80_to_f128 \
+		f128_to_f16, \
+		float-to-float)
+
+# Conversion Routines: Int and Uint to Float
 # FIXME: i32_to_extF80 (broken), i64_to_extF80 (broken)
-#        ui32_to_f128 (not implemented), extF80_roundToInt (broken)
-#
-check-softfloat-conv: $(FP_TEST_BIN)
+#        ui32_to_f128 (not implemented)
+check-softfloat-conv-to-float: $(FP_TEST_BIN)
 	$(call test-softfloat, \
 		i32_to_f16 i64_to_f16 \
 		i32_to_f32 i64_to_f32 \
@@ -701,7 +716,12 @@ check-softfloat-conv: $(FP_TEST_BIN)
 		ui32_to_f16 ui64_to_f16 \
 		ui32_to_f32 ui64_to_f32 \
 		ui32_to_f64 ui64_to_f64 \
+		ui32_to_extF80 ui64_to_extF80 \
 		ui64_to_f128, uint-to-float)
+
+# Conversion Routines: Float to integers
+# FIXME: extF80_roundToInt (broken)
+check-softfloat-conv-to-int: $(FP_TEST_BIN)
 	$(call test-softfloat, \
 		f16_to_i32 f16_to_i32_r_minMag \
 		f32_to_i32 f32_to_i32_r_minMag \
@@ -718,10 +738,12 @@ check-softfloat-conv: $(FP_TEST_BIN)
 		f16_to_ui32 f16_to_ui32_r_minMag \
 		f32_to_ui32 f32_to_ui32_r_minMag \
 		f64_to_ui32 f64_to_ui32_r_minMag \
+		extF80_to_ui32 extF80_to_ui32_r_minMag \
 		f128_to_ui32 f128_to_ui32_r_minMag \
 		f16_to_ui64 f16_to_ui64_r_minMag \
 		f32_to_ui64 f32_to_ui64_r_minMag \
 		f64_to_ui64 f64_to_ui64_r_minMag \
+		extF80_to_ui64 extF80_to_ui64_r_minMag \
 		f128_to_ui64 f128_to_ui64_r_minMag, \
 		float-to-uint)
 	$(call test-softfloat, \
@@ -729,9 +751,14 @@ check-softfloat-conv: $(FP_TEST_BIN)
 		f64_roundToInt f128_roundToInt, \
 		round-to-integer)
 
+.PHONY: check-softfloat-conv
+check-softfloat-conv: check-softfloat-conv-f2f
+check-softfloat-conv: check-softfloat-conv-to-float
+check-softfloat-conv: check-softfloat-conv-to-int
+
 # Generic rule for all float operations
 #
-# Some patterns are overidden due to broken or missing tests.
+# Some patterns are overridden due to broken or missing tests.
 # Hopefully these can be removed over time.
 
 check-softfloat-%: $(FP_TEST_BIN)
-- 
2.20.1

[PULL 07/12] tests/tcg: better detect confused gdb which can't connect

[Alex Bennée]

While we may gamely give the right information it can still confuse
the wide range of GDBs out there. For example ppc64abi32-linux-user
reports:

  warning: Selected architecture powerpc:common is not compatible with reported target architecture powerpc:common64
  warning: Architecture rejected target-supplied description

but still connects. Add a test for a 0 pc and exit early if that is
the case. This may actually be a bug we need to fix?

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200520140541.30256-8-alex.bennee@linaro.org>

diff --git a/tests/tcg/multiarch/gdbstub/sha1.py b/tests/tcg/multiarch/gdbstub/sha1.py
index 734553b98bb..2bfde49633c 100644
--- a/tests/tcg/multiarch/gdbstub/sha1.py
+++ b/tests/tcg/multiarch/gdbstub/sha1.py
@@ -65,6 +65,10 @@ except (gdb.error, AttributeError):
     print("SKIPPING (not connected)", file=sys.stderr)
     exit(0)
 
+if gdb.parse_and_eval('$pc') == 0:
+    print("SKIP: PC not set")
+    exit(0)
+
 try:
     # These are not very useful in scripts
     gdb.execute("set pagination off")
-- 
2.20.1

[PULL 08/12] tests/docker: add debian11 base image

[Alex Bennée]

We won't use this for building QEMU but we do need newer GCC's and
binutils for building some of our test cases.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200520140541.30256-10-alex.bennee@linaro.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 43a86786882..3596b589307 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -7,7 +7,7 @@ HOST_ARCH = $(if $(ARCH),$(ARCH),$(shell uname -m))
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_PARTIAL_IMAGES := debian9 debian10
+DOCKER_PARTIAL_IMAGES := debian9 debian10 debian11
 DOCKER_PARTIAL_IMAGES += debian9-mxe debian-bootstrap
 DOCKER_IMAGES := $(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
diff --git a/tests/docker/dockerfiles/debian11.docker b/tests/docker/dockerfiles/debian11.docker
new file mode 100644
index 00000000000..5adfd62d553
--- /dev/null
+++ b/tests/docker/dockerfiles/debian11.docker
@@ -0,0 +1,18 @@
+#
+# Docker multiarch cross-compiler target
+#
+# This docker target uses the current development version of Debian as
+# a base for cross compilers for building test binaries. We won't
+# attempt to build QEMU on it yet given it is still in development.
+#
+# On its own you can't build much but the docker-foo-cross targets
+# build on top of the base debian image.
+#
+FROM debian:bullseye-slim
+
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
+
+# Install common build utilities
+RUN apt update && \
+    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
-- 
2.20.1

[PULL 09/12] tests/docker: use a gcc-10 based image for arm64 tests

[Alex Bennée]

As we enable newer features that we want to test on arm64 targets we
need newer compilers. Split off a new debian-arm64-test-cross image
which we can use to build these new tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200520140541.30256-11-alex.bennee@linaro.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 3596b589307..ed46bd98eb5 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -131,9 +131,11 @@ docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
+docker-image-debian-arm64-test-cross: docker-image-debian11
 
 # These images may be good enough for building tests but not for test builds
 DOCKER_PARTIAL_IMAGES += debian-alpha-cross
+DOCKER_PARTIAL_IMAGES += debian-arm64-test-cross
 DOCKER_PARTIAL_IMAGES += debian-hppa-cross
 DOCKER_PARTIAL_IMAGES += debian-m68k-cross debian-mips64-cross
 DOCKER_PARTIAL_IMAGES += debian-powerpc-cross debian-ppc64-cross
diff --git a/tests/docker/dockerfiles/debian-arm64-test-cross.docker b/tests/docker/dockerfiles/debian-arm64-test-cross.docker
new file mode 100644
index 00000000000..a44e76d9421
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-arm64-test-cross.docker
@@ -0,0 +1,13 @@
+#
+# Docker arm64 cross-compiler target (tests only)
+#
+# This docker target builds on the debian Bullseye base image.
+#
+FROM qemu:debian11
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture arm64
+RUN apt update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata \
+        apt install -y --no-install-recommends \
+        crossbuild-essential-arm64 gcc-10-aarch64-linux-gnu
diff --git a/tests/tcg/configure.sh b/tests/tcg/configure.sh
index eaaaff6233a..2326f978562 100755
--- a/tests/tcg/configure.sh
+++ b/tests/tcg/configure.sh
@@ -97,8 +97,8 @@ for target in $target_list; do
   case $target in
     aarch64-*)
       # We don't have any bigendian build tools so we only use this for AArch64
-      container_image=debian-arm64-cross
-      container_cross_cc=aarch64-linux-gnu-gcc
+      container_image=debian-arm64-test-cross
+      container_cross_cc=aarch64-linux-gnu-gcc-10
       ;;
     alpha-*)
       container_image=debian-alpha-cross
-- 
2.20.1

[PULL 10/12] cpus-common: ensure auto-assigned cpu_indexes don't clash

[Alex Bennée]

Basing the cpu_index on the number of currently allocated vCPUs fails
when vCPUs aren't removed in a LIFO manner. This is especially true
when we are allocating a cpu_index for each guest thread in
linux-user where there is no ordering constraint on their allocation
and de-allocation.

[I've dropped the assert which is there to guard against out-of-order
removal as this should probably be caught higher up the stack. Maybe
we could just ifdef CONFIG_SOFTTMU it?]

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Igor Mammedow <imammedo@redhat.com>
Cc: Nikolay Igotti <igotti@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20200520140541.30256-13-alex.bennee@linaro.org>

diff --git a/cpus-common.c b/cpus-common.c
index 55d5df89237..70a9d12981a 100644
--- a/cpus-common.c
+++ b/cpus-common.c
@@ -61,13 +61,15 @@ static bool cpu_index_auto_assigned;
 static int cpu_get_free_index(void)
 {
     CPUState *some_cpu;
-    int cpu_index = 0;
+    int max_cpu_index = 0;
 
     cpu_index_auto_assigned = true;
     CPU_FOREACH(some_cpu) {
-        cpu_index++;
+        if (some_cpu->cpu_index >= max_cpu_index) {
+            max_cpu_index = some_cpu->cpu_index + 1;
+        }
     }
-    return cpu_index;
+    return max_cpu_index;
 }
 
 void cpu_list_add(CPUState *cpu)
@@ -90,8 +92,6 @@ void cpu_list_remove(CPUState *cpu)
         return;
     }
 
-    assert(!(cpu_index_auto_assigned && cpu != QTAILQ_LAST(&cpus)));
-
     QTAILQ_REMOVE_RCU(&cpus, cpu, node);
     cpu->cpu_index = UNASSIGNED_CPU_INDEX;
 }
-- 
2.20.1

[PULL 11/12] linux-user: properly "unrealize" vCPU object

[Alex Bennée]

We shouldn't be messing around with the CPU list in linux-user save
for the very special case of do_fork(). When threads end we need to
properly follow QOM object lifetime handling and allow the eventual
cpu_common_unrealizefn to both remove the CPU and ensure any clean-up
actions are taken place, for example calling plugin exit hooks.

There is still a race condition to avoid so use the linux-user
specific clone_lock instead of the cpu_list_lock to avoid it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Nikolay Igotti <igotti@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Daniel P. Berrange <berrange@redhat.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200520140541.30256-14-alex.bennee@linaro.org>

diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index 05f03919ff0..7f6700c54e3 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -7635,30 +7635,33 @@ static abi_long do_syscall1(void *cpu_env, int num, abi_long arg1,
             return -TARGET_ERESTARTSYS;
         }
 
-        cpu_list_lock();
+        pthread_mutex_lock(&clone_lock);
 
         if (CPU_NEXT(first_cpu)) {
-            TaskState *ts;
+            TaskState *ts = cpu->opaque;
 
-            /* Remove the CPU from the list.  */
-            QTAILQ_REMOVE_RCU(&cpus, cpu, node);
+            object_property_set_bool(OBJECT(cpu), false, "realized", NULL);
+            object_unref(OBJECT(cpu));
+            /*
+             * At this point the CPU should be unrealized and removed
+             * from cpu lists. We can clean-up the rest of the thread
+             * data without the lock held.
+             */
 
-            cpu_list_unlock();
+            pthread_mutex_unlock(&clone_lock);
 
-            ts = cpu->opaque;
             if (ts->child_tidptr) {
                 put_user_u32(0, ts->child_tidptr);
                 do_sys_futex(g2h(ts->child_tidptr), FUTEX_WAKE, INT_MAX,
                           NULL, NULL, 0);
             }
             thread_cpu = NULL;
-            object_unref(OBJECT(cpu));
             g_free(ts);
             rcu_unregister_thread();
             pthread_exit(NULL);
         }
 
-        cpu_list_unlock();
+        pthread_mutex_unlock(&clone_lock);
         preexit_cleanup(cpu_env, arg1);
         _exit(arg1);
         return 0; /* avoid warning */
-- 
2.20.1

[PULL 12/12] tests/tcg: add new threadcount test

[Alex Bennée]

Based on the original testcase by Nikolay Igotti.

Message-ID: <CAEme+7GLKg_dNsHizzTKDymX9HyD+Ph2iZ=WKhOw2XG+zhViXg@mail.gmail.com>
Signed-off-by: Nikolay Igotti <igotti@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Message-Id: <20200520140541.30256-15-alex.bennee@linaro.org>

diff --git a/tests/tcg/multiarch/threadcount.c b/tests/tcg/multiarch/threadcount.c
new file mode 100644
index 00000000000..545a1c81466
--- /dev/null
+++ b/tests/tcg/multiarch/threadcount.c
@@ -0,0 +1,64 @@
+/*
+ * Thread Exerciser
+ *
+ * Unlike testthread which is mainly concerned about testing thread
+ * semantics this test is used to exercise the thread creation and
+ * accounting. A version of this test found a problem with clashing
+ * cpu_indexes which caused a break in plugin handling.
+ *
+ * Based on the original test case by Nikolay Igotti.
+ *
+ * Copyright (c) 2020 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <pthread.h>
+
+int max_threads = 10;
+
+typedef struct {
+    int delay;
+} ThreadArg;
+
+static void *thread_fn(void* varg)
+{
+    ThreadArg *arg = varg;
+    usleep(arg->delay);
+    free(arg);
+    return NULL;
+}
+
+int main(int argc, char **argv)
+{
+    int i;
+    pthread_t *threads;
+
+    if (argc > 1) {
+        max_threads = atoi(argv[1]);
+    }
+    threads = calloc(sizeof(pthread_t), max_threads);
+
+    for (i = 0; i < max_threads; i++) {
+        ThreadArg *arg = calloc(sizeof(ThreadArg), 1);
+        arg->delay = i * 100;
+        pthread_create(threads + i, NULL, thread_fn, arg);
+    }
+
+    printf("Created %d threads\n", max_threads);
+
+    /* sleep until roughly half the threads have "finished" */
+    usleep(max_threads * 50);
+
+    for (i = 0; i < max_threads; i++) {
+        pthread_join(threads[i], NULL);
+    }
+
+    printf("Done\n");
+
+    return 0;
+}
diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target
index 51fb75ecfdd..cb49cc9ccb2 100644
--- a/tests/tcg/multiarch/Makefile.target
+++ b/tests/tcg/multiarch/Makefile.target
@@ -28,6 +28,8 @@ run-float_%: float_%
 
 testthread: LDFLAGS+=-lpthread
 
+threadcount: LDFLAGS+=-lpthread
+
 # We define the runner for test-mmap after the individual
 # architectures have defined their supported pages sizes. If no
 # additional page sizes are defined we only run the default test.
-- 
2.20.1

Re: [PULL 00/12] testing and plugin fixes

[Peter Maydell]

On Wed, 27 May 2020 at 15:54, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>
>   Merge remote-tracking branch 'remotes/gkurz/tags/9p-next-2020-05-26' into staging (2020-05-26 14:05:53 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-testing-tcg-plugins-270520-1
>
> for you to fetch changes up to 919bfbf5d6569b63a374332292cf3d2355a6d6c3:
>
>   tests/tcg: add new threadcount test (2020-05-27 14:26:49 +0100)
>
> ----------------------------------------------------------------
> Testing and one plugin fix:
>
>   - support alternates for genisoimage to test/vm
>   - add clang++ to clang tests
>   - fix record/replay smoke test
>   - enable more softfloat tests
>   - better detection of hung gdb
>   - upgrade aarch64 tcg test x-compile to gcc-10
>   - fix plugin cpu_index clash vs threads

This makes the NetBSD tests/vm setup spit out a huge pile of extra warnings:
every link has something like:

ld: warning: libintl.so.1, needed by /usr/pkg/lib/libglib-2.0.so, may
conflict with libintl.so.8

or some variant like:
ld: warning: libintl.so.1, needed by /usr/pkg/lib/libcurl.so, may
conflict with libintl.so.8
ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may
conflict with libintl.so.8

thanks
-- PMM

Re: [PULL 00/12] testing and plugin fixes

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 27 May 2020 at 15:54, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> The following changes since commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>>
>>   Merge remote-tracking branch 'remotes/gkurz/tags/9p-next-2020-05-26' into staging (2020-05-26 14:05:53 +0100)
>>
>> are available in the Git repository at:
>>
>>   https://github.com/stsquad/qemu.git tags/pull-testing-tcg-plugins-270520-1
>>
>> for you to fetch changes up to 919bfbf5d6569b63a374332292cf3d2355a6d6c3:
>>
>>   tests/tcg: add new threadcount test (2020-05-27 14:26:49 +0100)
>>
>> ----------------------------------------------------------------
>> Testing and one plugin fix:
>>
>>   - support alternates for genisoimage to test/vm
>>   - add clang++ to clang tests
>>   - fix record/replay smoke test
>>   - enable more softfloat tests
>>   - better detection of hung gdb
>>   - upgrade aarch64 tcg test x-compile to gcc-10
>>   - fix plugin cpu_index clash vs threads
>
> This makes the NetBSD tests/vm setup spit out a huge pile of extra warnings:
> every link has something like:
>
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libglib-2.0.so, may
> conflict with libintl.so.8
>
> or some variant like:
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libcurl.so, may
> conflict with libintl.so.8
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may
> conflict with libintl.so.8

Weird... I'll have a look at whats going on.

>
> thanks
> -- PMM


-- 
Alex Bennée

Re: [PULL 00/12] testing and plugin fixes

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 27 May 2020 at 15:54, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> The following changes since commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>>
>>   Merge remote-tracking branch 'remotes/gkurz/tags/9p-next-2020-05-26' into staging (2020-05-26 14:05:53 +0100)
>>
>> are available in the Git repository at:
>>
>>   https://github.com/stsquad/qemu.git tags/pull-testing-tcg-plugins-270520-1
>>
>> for you to fetch changes up to 919bfbf5d6569b63a374332292cf3d2355a6d6c3:
>>
>>   tests/tcg: add new threadcount test (2020-05-27 14:26:49 +0100)
>>
>> ----------------------------------------------------------------
>> Testing and one plugin fix:
>>
>>   - support alternates for genisoimage to test/vm
>>   - add clang++ to clang tests
>>   - fix record/replay smoke test
>>   - enable more softfloat tests
>>   - better detection of hung gdb
>>   - upgrade aarch64 tcg test x-compile to gcc-10
>>   - fix plugin cpu_index clash vs threads
>
> This makes the NetBSD tests/vm setup spit out a huge pile of extra warnings:
> every link has something like:
>
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libglib-2.0.so, may
> conflict with libintl.so.8
>
> or some variant like:
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libcurl.so, may
> conflict with libintl.so.8
> ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may
> conflict with libintl.so.8

I get these on commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:

    LINK    arm-softmmu/qemu-system-arm
  ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple common of `environ'
  ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
  ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
  ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
  ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
  ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8

which makes me think the build environment has changed and it was simply
that my PR triggered a re-build of the NetBSD VM.

-- 
Alex Bennée

Re: [PULL 00/12] testing and plugin fixes

[Peter Maydell]

On Thu, 28 May 2020 at 15:57, Alex Bennée <alex.bennee@linaro.org> wrote:
> I get these on commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>
>     LINK    arm-softmmu/qemu-system-arm
>   ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple common of `environ'
>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>
> which makes me think the build environment has changed and it was simply
> that my PR triggered a re-build of the NetBSD VM.

Yes. I see them also now on the next pullreq I'm trying to
process. How do I get back to an environment that doesn't
produce all these warnings?

thanks
-- PMM

Re: [PULL 00/12] testing and plugin fixes

[Philippe Mathieu-Daudé]

+Gerd

On 5/28/20 5:09 PM, Peter Maydell wrote:
> On Thu, 28 May 2020 at 15:57, Alex Bennée <alex.bennee@linaro.org> wrote:
>> I get these on commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>>
>>     LINK    arm-softmmu/qemu-system-arm
>>   ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple common of `environ'
>>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
>>
>> which makes me think the build environment has changed and it was simply
>> that my PR triggered a re-build of the NetBSD VM.
> 
> Yes. I see them also now on the next pullreq I'm trying to
> process. How do I get back to an environment that doesn't
> produce all these warnings?

Maybe building back NetBSD 8.1?

$ git checkout 2cc3e591b3321a~ tests/vm/netbsd
$ make vm-build-netbsd
$ git checkout HEAD tests/vm/netbsd

> 
> thanks
> -- PMM
> 

Re: [PULL 00/12] testing and plugin fixes

[Peter Maydell]

On Thu, 28 May 2020 at 16:18, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> +Gerd
>
> On 5/28/20 5:09 PM, Peter Maydell wrote:
> > Yes. I see them also now on the next pullreq I'm trying to
> > process. How do I get back to an environment that doesn't
> > produce all these warnings?
>
> Maybe building back NetBSD 8.1?

Did that change not itself trigger the VM to be rebuilt ?
If not, can we make it so that updates to the images always
do trigger rebuilds, so that we can catch new warnings
that they introduce at the time rather than later?

thanks
-- PMM

Re: [PULL 00/12] testing and plugin fixes

[Philippe Mathieu-Daudé]

On 5/28/20 5:20 PM, Peter Maydell wrote:
> On Thu, 28 May 2020 at 16:18, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>>
>> +Gerd
>>
>> On 5/28/20 5:09 PM, Peter Maydell wrote:
>>> Yes. I see them also now on the next pullreq I'm trying to
>>> process. How do I get back to an environment that doesn't
>>> produce all these warnings?
>>
>> Maybe building back NetBSD 8.1?
> 
> Did that change not itself trigger the VM to be rebuilt ?
> If not, can we make it so that updates to the images always
> do trigger rebuilds, so that we can catch new warnings
> that they introduce at the time rather than later?

This is how it should work... How do you run your VM builds?

Re: [PULL 00/12] testing and plugin fixes

[Gerd Hoffmann]

On Thu, May 28, 2020 at 05:18:42PM +0200, Philippe Mathieu-Daudé wrote:
> +Gerd
> 
> On 5/28/20 5:09 PM, Peter Maydell wrote:
> > On Thu, 28 May 2020 at 15:57, Alex Bennée <alex.bennee@linaro.org> wrote:
> >> I get these on commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
> >>
> >>     LINK    arm-softmmu/qemu-system-arm
> >>   ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple common of `environ'
> >>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
> >>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
> >>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
> >>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8
> >>   ld: warning: libintl.so.1, needed by /usr/pkg/lib/libgio-2.0.so, may conflict with libintl.so.8

Hmm, on a quick glance this looks like a netbsd packaging bug ...

> Maybe building back NetBSD 8.1?
> 
> $ git checkout 2cc3e591b3321a~ tests/vm/netbsd
> $ make vm-build-netbsd
> $ git checkout HEAD tests/vm/netbsd

The *nsd installs fetch and install a bunch of addon packages, so when
these are updated a netbsd image rebuild will use the newer versions.

IIRC both tests/vm/netbsd and tests/vm/basevm.py changes will trigger
and netbsd image rebuild.  If you want avoid that you have to touch the
disk image after git checkout so make thinks it is up-to-date and
doesn't try to rebuild it.

take care,
  Gerd

Re: [PULL 00/12] testing and plugin fixes

[Peter Maydell]

On Thu, 28 May 2020 at 16:26, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> > Did that change not itself trigger the VM to be rebuilt ?
> > If not, can we make it so that updates to the images always
> > do trigger rebuilds, so that we can catch new warnings
> > that they introduce at the time rather than later?
>
> This is how it should work... How do you run your VM builds?

git fetch pmaydell
git checkout --quiet remotes/pmaydell/staging
make -C build vm-build-netbsd -j8 V=1

thanks
-- PMM

Re: [PULL 00/12] testing and plugin fixes

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 28 May 2020 at 16:18, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>>
>> +Gerd
>>
>> On 5/28/20 5:09 PM, Peter Maydell wrote:
>> > Yes. I see them also now on the next pullreq I'm trying to
>> > process. How do I get back to an environment that doesn't
>> > produce all these warnings?
>>
>> Maybe building back NetBSD 8.1?
>
> Did that change not itself trigger the VM to be rebuilt ?
> If not, can we make it so that updates to the images always
> do trigger rebuilds, so that we can catch new warnings
> that they introduce at the time rather than later?

$(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \

So netbsdb.img should have been rebuilt when tests/vm/netbsd was
touched. However maybe the packages have been updated upstream since? As
we also have:

			$(SRC_PATH)/tests/vm/basevm.py \
			$(SRC_PATH)/tests/vm/Makefile.include

I suspect the genisoimage change to basevm triggered a re-build grabbing
the latest packages.

-- 
Alex Bennée

Re: [PULL 00/12] testing and plugin fixes

[Peter Maydell]

On Wed, 27 May 2020 at 15:54, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit ddc760832fa8cf5e93b9d9e6e854a5114ac63510:
>
>   Merge remote-tracking branch 'remotes/gkurz/tags/9p-next-2020-05-26' into staging (2020-05-26 14:05:53 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-testing-tcg-plugins-270520-1
>
> for you to fetch changes up to 919bfbf5d6569b63a374332292cf3d2355a6d6c3:
>
>   tests/tcg: add new threadcount test (2020-05-27 14:26:49 +0100)
>
> ----------------------------------------------------------------
> Testing and one plugin fix:
>
>   - support alternates for genisoimage to test/vm
>   - add clang++ to clang tests
>   - fix record/replay smoke test
>   - enable more softfloat tests
>   - better detection of hung gdb
>   - upgrade aarch64 tcg test x-compile to gcc-10
>   - fix plugin cpu_index clash vs threads


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.1
for any user-visible changes.

-- PMM