[PATCH v2 00/29] tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86

[PATCH v2 00/29] tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86

[Philippe Mathieu-Daudé]

Currently our CI is mostly restricted to Linux/X86.
We want to have more, but until there we have to run these tests
manually, which is why we didn't noticed earlier this test was
failing on OSX.

This series contains the patches I used to debug why the
virtio_seg_max_adjust had broken the OSX CI, but then I
kept testing on other archs, thus added more patches...

Currently our CI is mostly restricted to Linux/X86.
We want to have more, but until there we have to run these tests
manually, which is why we didn't noticed earlier some tests are
failing (on OSX, using --without-default-devices, ...).

Tested on Linux/FreeBSD/OSX, X86/Aarch64.

Regards,

Phil.

Supersedes: <20200122223247.30419-1-philmd@redhat.com>
"tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86"

Denis Plotnikov (1):
  tests: rename virtio_seg_max_adjust to virtio_check_params

Philippe Mathieu-Daudé (28):
  hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host
  accel/accel: Make TYPE_ACCEL abstract
  python/qemu: Add binutils::binary_get_version()
  tests/acceptance: Use 'version-min' tag to verify QEMU binary version
  tests/acceptance: Restrict X86CPUModelAliases test to QEMU >= 4.1
  python/qemu: Add binutils::binary_get_arch()
  tests/acceptance: Use the 'arch' tag to verify QEMU binary target
  python/qemu: Add binutils::binary_get_machines()
  tests/acceptance: Use 'machine' tag to check if available in QEMU
    binary
  python/qemu: Add binutils::binary_get_qom_implementations()
  python/qemu: Add binutils::binary_get_accels()
  python/qemu/accel: Use binutils::binary_get_accels()
  python/qemu: Add binutils::binary_get_devices()
  tests/acceptance: Use 'device' tags to check availability in QEMU
    binary
  tests/acceptance/virtio_check_params: Only remove listed machines
  tests/acceptance/virtio_check_params: Improve exception logging
  tests/acceptance/virtio_check_params: List machine being tested
  tests/acceptance/virtio_check_params: Only test Xen as superuser
  tests/acceptance/virtio_check_params: Skip test if arch is not
    supported
  tests/acceptance/virtio_check_params: Kludge to skip tests on MIPS
  tests/acceptance/virtio_check_params: Support the s390x architecture
  tests/acceptance/virtio_check_params: Support the ppc architecture
  tests/acceptance/virtio_check_params: Default to -nodefaults
  tests/acceptance/virtio_check_params: Require a virtio-scsi-pci device
  tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine
  tests/acceptance/migration: Default to -nodefaults
  tests/acceptance/version: Default to -nodefaults
  MAINTAINERS: Add Acceptance tests reviewers

 accel/accel.c                                 |   1 +
 hw/core/machine.c                             |   3 +-
 MAINTAINERS                                   |   6 +
 python/qemu/accel.py                          |  26 +----
 python/qemu/binutils.py                       | 107 ++++++++++++++++++
 tests/acceptance/avocado_qemu/__init__.py     |  51 ++++++++-
 tests/acceptance/boot_linux_console.py        |   2 +-
 tests/acceptance/core_scripts.py              |  79 +++++++++++++
 tests/acceptance/migration.py                 |   4 +-
 tests/acceptance/version.py                   |   1 +
 ...g_max_adjust.py => virtio_check_params.py} |  70 +++++++++---
 tests/acceptance/x86_cpu_model_versions.py    |   1 +
 12 files changed, 306 insertions(+), 45 deletions(-)
 create mode 100644 python/qemu/binutils.py
 create mode 100644 tests/acceptance/core_scripts.py
 rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py} (64%)

-- 
2.21.1

[PATCH v2 01/29] hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host

[Philippe Mathieu-Daudé]

Commit ed65fd1a27 restricted the 'scsi' property to Linux hosts:

  1261 static Property virtio_blk_properties[] = {
  1262     DEFINE_BLOCK_PROPERTIES(VirtIOBlock, conf.conf),
  ...
  1268 #ifdef __linux__
  1269     DEFINE_PROP_BIT64("scsi", VirtIOBlock, host_features,
  1270                       VIRTIO_BLK_F_SCSI, false),
  1271 #endif

When we try to set this property on a pre-2.4 machine on OSX,
we get:

   Unexpected error in object_property_find() at qom/object.c:1201:
   qemu-system-x86_64: -device virtio-blk-pci,id=scsi0,drive=drive0: can't apply global virtio-blk-device.scsi=true: Property '.scsi' not found

Fix this error by marking the property optional.

Fixes: ed65fd1a27
Suggested-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 hw/core/machine.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index 3e288bfceb..d8e30e4895 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -148,7 +148,8 @@ GlobalProperty hw_compat_2_5[] = {
 const size_t hw_compat_2_5_len = G_N_ELEMENTS(hw_compat_2_5);
 
 GlobalProperty hw_compat_2_4[] = {
-    { "virtio-blk-device", "scsi", "true" },
+    /* Optional because the 'scsi' property is Linux-only */
+    { "virtio-blk-device", "scsi", "true", .optional = true },
     { "e1000", "extra_mac_registers", "off" },
     { "virtio-pci", "x-disable-pcie", "on" },
     { "virtio-pci", "migrate-extra", "off" },
-- 
2.21.1

[PATCH v2 02/29] accel/accel: Make TYPE_ACCEL abstract

[Philippe Mathieu-Daudé]

There is no generic acceleration, we have to use specific
implementations. Make the base class abstract.

Fixes: b14a0b7469f
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 accel/accel.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/accel/accel.c b/accel/accel.c
index cb555e3b06..a0169b4e69 100644
--- a/accel/accel.c
+++ b/accel/accel.c
@@ -33,6 +33,7 @@
 static const TypeInfo accel_type = {
     .name = TYPE_ACCEL,
     .parent = TYPE_OBJECT,
+    .abstract = true,
     .class_size = sizeof(AccelClass),
     .instance_size = sizeof(AccelState),
 };
-- 
2.21.1

[PATCH v2 03/29] python/qemu: Add binutils::binary_get_version()

[Philippe Mathieu-Daudé]

Add a helper to query the version of a QEMU binary.
We simply send the 'query-version' command over a QMP
socket.
Introduce the PythonQemuCoreScripts class to test our
new helper.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 38 ++++++++++++++++++++++++++++++++
 tests/acceptance/core_scripts.py | 31 ++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
 create mode 100644 python/qemu/binutils.py
 create mode 100644 tests/acceptance/core_scripts.py

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
new file mode 100644
index 0000000000..96b200eef4
--- /dev/null
+++ b/python/qemu/binutils.py
@@ -0,0 +1,38 @@
+"""
+QEMU binary utility module:
+
+The binary utility module provides helpers to query QEMU binary for
+build-dependent configuration options at runtime.
+"""
+#
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Philippe Mathieu-Daudé <philmd@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import logging
+
+from .machine import QEMUMachine
+
+LOG = logging.getLogger(__name__)
+
+
+def binary_get_version(qemu_bin):
+    '''
+    Get QEMU binary version
+
+    @param qemu_bin (str): path to the QEMU binary
+    @return binary version (dictionary with major/minor/micro keys)
+    '''
+    with QEMUMachine(qemu_bin) as vm:
+        vm.set_machine('none')
+        vm.launch()
+        res = vm.command('query-version')
+        LOG.info(res)
+        vm.shutdown()
+        return res['qemu']
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
new file mode 100644
index 0000000000..3f253337cd
--- /dev/null
+++ b/tests/acceptance/core_scripts.py
@@ -0,0 +1,31 @@
+# Tests covering various python/qemu/ scripts
+#
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Philippe Mathieu-Daudé <philmd@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import sys
+import os
+import logging
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from avocado_qemu import Test
+from qemu.binutils import binary_get_version
+
+
+class PythonQemuCoreScripts(Test):
+
+    def test_get_version(self):
+        logger = logging.getLogger('core')
+        version = binary_get_version(self.qemu_bin)
+        logger.debug('version: {}'.format(version))
+        # QMP 'query-version' introduced with QEMU v0.14
+        self.assertGreaterEqual(version['major'], 0)
+        if version['major'] == 0:
+            self.assertGreaterEqual(version['minor'], 14)
-- 
2.21.1

[PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Philippe Mathieu-Daudé]

Introduce the 'version-min' tag.
Tests can set it to the minimum version of QEMU they require.
If QEMU is older, the tests will be cancelled (skipped):

  $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
   (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
   (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
   (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
   ...

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 6618ea67c1..67c75822d5 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -19,6 +19,7 @@ import avocado
 SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
 
+from qemu.binutils import binary_get_version
 from qemu.machine import QEMUMachine
 
 def is_readable_executable_file(path):
@@ -111,6 +112,7 @@ class Test(avocado.Test):
 
     def setUp(self):
         self._vms = {}
+        logger = logging.getLogger('core')
 
         self.arch = self.params.get('arch',
                                     default=self._get_unique_tag_val('arch'))
@@ -118,12 +120,28 @@ class Test(avocado.Test):
         self.machine = self.params.get('machine',
                                        default=self._get_unique_tag_val('machine'))
 
+        # Verify qemu_bin
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
 
+        # Verify version
+        min_version = self._get_unique_tag_val('version-min')
+        logger.debug('version-min: {}'.format(min_version))
+        if min_version:
+            min_v = [int(n) for n in min_version.split('.')]
+            if len(min_v) < 3:
+                min_v.append(0)
+            logger.debug(min_v)
+            bin_v = binary_get_version(self.qemu_bin)
+            bin_v = (bin_v['major'], bin_v['minor'], bin_v['micro'])
+            logger.debug(bin_v)
+            fmt = "Test expects version '{}' but QEMU binary is '{}.{}.{}'"
+            if bin_v[0] < min_v[0] or bin_v[1] < min_v[1] or bin_v[2] < min_v[2]:
+                self.cancel(fmt.format(min_version, bin_v[0], bin_v[1], bin_v[2]))
+
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
         if args:
-- 
2.21.1

[PATCH v2 05/29] tests/acceptance: Restrict X86CPUModelAliases test to QEMU >= 4.1

[Philippe Mathieu-Daudé]

The 'alias-of' option has been introduced with QEMU 4.1 (see
commit 7d753f61544). By using the 'version-min' Avocado tag
we can restrict this test to versions >= 4.1.

This fixes when running the test with a older QEMU:

  tests/acceptance/x86_cpu_model_versions.py:X86CPUModelAliases.test_none_alias:  ERROR: 'alias-of' (0.45 s)

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/x86_cpu_model_versions.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/x86_cpu_model_versions.py b/tests/acceptance/x86_cpu_model_versions.py
index 90558d9a71..e7604bab7c 100644
--- a/tests/acceptance/x86_cpu_model_versions.py
+++ b/tests/acceptance/x86_cpu_model_versions.py
@@ -30,6 +30,7 @@ class X86CPUModelAliases(avocado_qemu.Test):
     Validation of PC CPU model versions and CPU model aliases
 
     :avocado: tags=arch:x86_64
+    :avocado: tags=version-min:4.1
     """
     def validate_aliases(self, cpus):
         for c in cpus.values():
-- 
2.21.1

[PATCH v2 06/29] python/qemu: Add binutils::binary_get_arch()

[Philippe Mathieu-Daudé]

Add a helper to query the architecture of a QEMU binary.
We simply send the 'query-target' command over a QMP socket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 15 +++++++++++++++
 tests/acceptance/core_scripts.py | 11 +++++++++++
 2 files changed, 26 insertions(+)

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
index 96b200eef4..905d393ba5 100644
--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -36,3 +36,18 @@ def binary_get_version(qemu_bin):
         LOG.info(res)
         vm.shutdown()
         return res['qemu']
+
+def binary_get_arch(qemu_bin):
+    '''
+    Get target architecture for a QEMU binary
+
+    @param qemu_bin (str): path to the QEMU binary
+    @return binary target architecture
+    '''
+    with QEMUMachine(qemu_bin) as vm:
+        vm.set_machine('none')
+        vm.launch()
+        res = vm.command('query-target')
+        LOG.info(res)
+        vm.shutdown()
+        return res['arch']
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
index 3f253337cd..93dd822368 100644
--- a/tests/acceptance/core_scripts.py
+++ b/tests/acceptance/core_scripts.py
@@ -16,6 +16,7 @@ import logging
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from avocado_qemu import Test
+from qemu.binutils import binary_get_arch
 from qemu.binutils import binary_get_version
 
 
@@ -29,3 +30,13 @@ class PythonQemuCoreScripts(Test):
         self.assertGreaterEqual(version['major'], 0)
         if version['major'] == 0:
             self.assertGreaterEqual(version['minor'], 14)
+
+    def test_get_arch_x86(self):
+        """
+        :avocado: tags=arch:i386
+        :avocado: tags=arch:x86_64
+        """
+        logger = logging.getLogger('core')
+        a = binary_get_arch(self.qemu_bin)
+        logger.debug('arch: {}'.format(a))
+        self.assertIn(a, ['i386', 'x86_64'])
-- 
2.21.1

[PATCH v2 07/29] tests/acceptance: Use the 'arch' tag to verify QEMU binary target

[Philippe Mathieu-Daudé]

We already use the 'arch' tag in Avocado tests. Tests can set
it to restrict their use on a particular target architecture.

If the QEMU binary does not target the required architecture,
the tests will be cancelled (skipped):

  $ avocado --show=app run tests/acceptance/boot_linux_console.py
   ...
   (14/19) BootLinuxConsole.test_arm_cubieboard_initrd: CANCEL: Test expects arch 'arm' while QEMU binary is targetting 'mips' (0.16 s)
   (15/19) BootLinuxConsole.test_arm_cubieboard_sata: CANCEL: Test expects arch 'arm' while QEMU binary is targetting 'mips' (0.18 s)
   (16/19) BootLinuxConsole.test_s390x_s390_ccw_virtio: CANCEL: Test expects arch 's390x' while QEMU binary is targetting 'mips' (0.14 s)
   (17/19) BootLinuxConsole.test_alpha_clipper: CANCEL: Test expects arch 'alpha' while QEMU binary is targetting 'mips' (0.16 s)
   (18/19) BootLinuxConsole.test_ppc64_pseries: CANCEL: Test expects arch 'ppc64' while QEMU binary is targetting 'mips' (0.18 s)
   (19/19) BootLinuxConsole.test_m68k_q800: CANCEL: Test expects arch 'm68k' while QEMU binary is targetting 'mips' (0.17 s)
  RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 16
  JOB TIME   : 19.18 s

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 67c75822d5..e7d5affe24 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -19,6 +19,7 @@ import avocado
 SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
 
+from qemu.binutils import binary_get_arch
 from qemu.binutils import binary_get_version
 from qemu.machine import QEMUMachine
 
@@ -142,6 +143,14 @@ class Test(avocado.Test):
             if bin_v[0] < min_v[0] or bin_v[1] < min_v[1] or bin_v[2] < min_v[2]:
                 self.cancel(fmt.format(min_version, bin_v[0], bin_v[1], bin_v[2]))
 
+        # Verify architecture
+        if self.arch:
+            fmt = "Test expects arch '{}' while QEMU binary is targetting '{}'"
+            bin_arch = binary_get_arch(self.qemu_bin)
+            logger.debug('arch: {}'.format(bin_arch))
+            if bin_arch != self.arch:
+                self.cancel(fmt.format(self.arch, bin_arch))
+
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
         if args:
-- 
2.21.1

[PATCH v2 08/29] python/qemu: Add binutils::binary_get_machines()

[Philippe Mathieu-Daudé]

Add a helper to query the list of machines built into a QEMU binary.
We simply send the 'query-machines' command over a QMP socket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 15 +++++++++++++++
 tests/acceptance/core_scripts.py |  9 +++++++++
 2 files changed, 24 insertions(+)

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
index 905d393ba5..7bb57c521b 100644
--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -51,3 +51,18 @@ def binary_get_arch(qemu_bin):
         LOG.info(res)
         vm.shutdown()
         return res['arch']
+
+def binary_get_machines(qemu_bin):
+    '''
+    Get list of machines supported by a QEMU binary
+
+    @param qemu_bin (str): path to the QEMU binary
+    @return list of machines supported by the binary
+    '''
+    with QEMUMachine(qemu_bin) as vm:
+        vm.set_machine('none')
+        vm.launch()
+        res = vm.command('query-machines')
+        LOG.info(res)
+        vm.shutdown()
+        return [m['name'] for m in res]
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
index 93dd822368..a5b112f928 100644
--- a/tests/acceptance/core_scripts.py
+++ b/tests/acceptance/core_scripts.py
@@ -17,6 +17,7 @@ import logging
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from avocado_qemu import Test
 from qemu.binutils import binary_get_arch
+from qemu.binutils import binary_get_machines
 from qemu.binutils import binary_get_version
 
 
@@ -40,3 +41,11 @@ class PythonQemuCoreScripts(Test):
         a = binary_get_arch(self.qemu_bin)
         logger.debug('arch: {}'.format(a))
         self.assertIn(a, ['i386', 'x86_64'])
+
+    def test_get_machines(self):
+        logger = logging.getLogger('core')
+        machines = binary_get_machines(self.qemu_bin)
+        for m in sorted(machines):
+            logger.debug('machine: {}'.format(m))
+        # The 'none' machine is always available
+        self.assertIn('none', machines)
-- 
2.21.1

[PATCH v2 09/29] tests/acceptance: Use 'machine' tag to check if available in QEMU binary

[Philippe Mathieu-Daudé]

We already use the 'machine' tag in Avocado tests.
If the requested machine is not available in the QEMU binary,
the tests will be cancelled (skipped):

  $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
   ...
   (04/11) CascadelakeArchCapabilities.test_4_1: CANCEL: Test expects machine 'pc-i440fx-4.1' which is missing from QEMU binary (0.10 s)
   (05/11) CascadelakeArchCapabilities.test_4_0: CANCEL: Test expects machine 'pc-i440fx-4.0' which is missing from QEMU binary (0.11 s)
   ...

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index e7d5affe24..53ec8512d1 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -20,6 +20,7 @@ SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
 
 from qemu.binutils import binary_get_arch
+from qemu.binutils import binary_get_machines
 from qemu.binutils import binary_get_version
 from qemu.machine import QEMUMachine
 
@@ -118,9 +119,6 @@ class Test(avocado.Test):
         self.arch = self.params.get('arch',
                                     default=self._get_unique_tag_val('arch'))
 
-        self.machine = self.params.get('machine',
-                                       default=self._get_unique_tag_val('machine'))
-
         # Verify qemu_bin
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
@@ -151,6 +149,15 @@ class Test(avocado.Test):
             if bin_arch != self.arch:
                 self.cancel(fmt.format(self.arch, bin_arch))
 
+        # Verify machine
+        self.machine = self.params.get('machine',
+                                       default=self._get_unique_tag_val('machine'))
+        logger.debug('machine: {}'.format(self.machine))
+        if self.machine:
+            fmt = "Test expects machine '{}' which is missing from QEMU binary"
+            if self.machine not in binary_get_machines(self.qemu_bin):
+                self.cancel(fmt.format(self.machine))
+
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
         if args:
-- 
2.21.1

[PATCH v2 10/29] python/qemu: Add binutils::binary_get_qom_implementations()

[Philippe Mathieu-Daudé]

Since QEMU binaries can be built with various configurations,
the list of QOM objects linked can vary.
Add a helper to query the list of all QOM types implementing a
particular interface.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 20 ++++++++++++++++++++
 tests/acceptance/core_scripts.py | 10 ++++++++++
 2 files changed, 30 insertions(+)

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
index 7bb57c521b..76f256f5d2 100644
--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -66,3 +66,23 @@ def binary_get_machines(qemu_bin):
         LOG.info(res)
         vm.shutdown()
         return [m['name'] for m in res]
+
+def binary_get_qom_implementations(qemu_bin, type_name, include_abstract=False):
+    '''
+    Get list of QOM types implementing a particular interface
+
+    @param qemu_bin (str): path to the QEMU binary
+    @param type_name (str): QOM interface name
+    @param include_abstract (bool): if True, abstract interfaces are also
+                                    returned in the list
+    @return list of QOM types implementing the interface @type_name
+    '''
+    with QEMUMachine(qemu_bin) as vm:
+        vm.set_machine('none')
+        vm.launch()
+        res = vm.command('qom-list-types',
+                         implements=type_name,
+                         abstract=include_abstract)
+        LOG.info(res)
+        vm.shutdown()
+        return [m['name'] for m in res]
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
index a5b112f928..c2fe4acf1d 100644
--- a/tests/acceptance/core_scripts.py
+++ b/tests/acceptance/core_scripts.py
@@ -18,6 +18,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from avocado_qemu import Test
 from qemu.binutils import binary_get_arch
 from qemu.binutils import binary_get_machines
+from qemu.binutils import binary_get_qom_implementations
 from qemu.binutils import binary_get_version
 
 
@@ -49,3 +50,12 @@ class PythonQemuCoreScripts(Test):
             logger.debug('machine: {}'.format(m))
         # The 'none' machine is always available
         self.assertIn('none', machines)
+
+    def test_get_qom_implementation(self):
+        logger = logging.getLogger('core')
+        type_name = "accel"
+        type_impl = binary_get_qom_implementations(self.qemu_bin,
+                                                   type_name, True)
+        for t in type_impl:
+            logger.debug('type: {}'.format(t))
+        self.assertIn(type_name, type_impl)
-- 
2.21.1

[PATCH v2 11/29] python/qemu: Add binutils::binary_get_accels()

[Philippe Mathieu-Daudé]

Add a helper to query the list of accelerators built into a
QEMU binary.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 10 ++++++++++
 tests/acceptance/core_scripts.py | 10 ++++++++++
 2 files changed, 20 insertions(+)

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
index 76f256f5d2..bba203bc8d 100644
--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -86,3 +86,13 @@ def binary_get_qom_implementations(qemu_bin, type_name, include_abstract=False):
         LOG.info(res)
         vm.shutdown()
         return [m['name'] for m in res]
+
+def binary_get_accels(qemu_bin):
+    '''
+    Get list of accelerators supported by a QEMU binary
+
+    @param qemu_bin (str): path to the QEMU binary
+    @return list of accelerators supported by the binary
+    '''
+    accel_types = binary_get_qom_implementations(qemu_bin, "accel", False)
+    return [a.strip("-accel") for a in accel_types]
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
index c2fe4acf1d..7380f2f49b 100644
--- a/tests/acceptance/core_scripts.py
+++ b/tests/acceptance/core_scripts.py
@@ -16,6 +16,7 @@ import logging
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from avocado_qemu import Test
+from qemu.binutils import binary_get_accels
 from qemu.binutils import binary_get_arch
 from qemu.binutils import binary_get_machines
 from qemu.binutils import binary_get_qom_implementations
@@ -59,3 +60,12 @@ class PythonQemuCoreScripts(Test):
         for t in type_impl:
             logger.debug('type: {}'.format(t))
         self.assertIn(type_name, type_impl)
+
+    def test_get_accels(self):
+        logger = logging.getLogger('core')
+        accels = binary_get_accels(self.qemu_bin)
+        for a in accels:
+            logger.debug('accel: {}'.format(a))
+        self.assertNotIn("accel", accels)
+        # qtest is always available
+        self.assertIn("qtest", accels)
-- 
2.21.1

[PATCH v2 12/29] python/qemu/accel: Use binutils::binary_get_accels()

[Philippe Mathieu-Daudé]

Instead of parsing the process help output, use the
binary_get_accels() helper which queries the list of
accelerators over a QMP socket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/accel.py | 26 ++++----------------------
 1 file changed, 4 insertions(+), 22 deletions(-)

diff --git a/python/qemu/accel.py b/python/qemu/accel.py
index 0b38ddf0ab..cde51ae159 100644
--- a/python/qemu/accel.py
+++ b/python/qemu/accel.py
@@ -16,7 +16,8 @@ accelerators.
 
 import logging
 import os
-import subprocess
+
+from .binutils import binary_get_accels
 
 LOG = logging.getLogger(__name__)
 
@@ -27,25 +28,6 @@ ADDITIONAL_ARCHES = {
     "aarch64" : "armhf"
 }
 
-def list_accel(qemu_bin):
-    """
-    List accelerators enabled in the QEMU binary.
-
-    @param qemu_bin (str): path to the QEMU binary.
-    @raise Exception: if failed to run `qemu -accel help`
-    @return a list of accelerator names.
-    """
-    if not qemu_bin:
-        return []
-    try:
-        out = subprocess.check_output([qemu_bin, '-accel', 'help'],
-                                      universal_newlines=True)
-    except:
-        LOG.debug("Failed to get the list of accelerators in %s", qemu_bin)
-        raise
-    # Skip the first line which is the header.
-    return [acc.strip() for acc in out.splitlines()[1:]]
-
 def kvm_available(target_arch=None, qemu_bin=None):
     """
     Check if KVM is available using the following heuristic:
@@ -64,7 +46,7 @@ def kvm_available(target_arch=None, qemu_bin=None):
         if target_arch != host_arch:
             if target_arch != ADDITIONAL_ARCHES.get(host_arch):
                 return False
-    if qemu_bin and "kvm" not in list_accel(qemu_bin):
+    if qemu_bin and "kvm" not in binary_get_accels(qemu_bin):
         return False
     return True
 
@@ -74,4 +56,4 @@ def tcg_available(qemu_bin):
 
     @param qemu_bin (str): path to the QEMU binary
     """
-    return 'tcg' in list_accel(qemu_bin)
+    return 'tcg' in binary_get_accels(qemu_bin)
-- 
2.21.1

[PATCH v2 13/29] python/qemu: Add binutils::binary_get_devices()

[Philippe Mathieu-Daudé]

Since QEMU binaries can be built with various configurations,
the list of devices linked in can vary.
Add a helper to query the list of devices built into a
QEMU binary.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 python/qemu/binutils.py          | 9 +++++++++
 tests/acceptance/core_scripts.py | 8 ++++++++
 2 files changed, 17 insertions(+)

diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
index bba203bc8d..9633ba8efd 100644
--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -96,3 +96,12 @@ def binary_get_accels(qemu_bin):
     '''
     accel_types = binary_get_qom_implementations(qemu_bin, "accel", False)
     return [a.strip("-accel") for a in accel_types]
+
+def binary_get_devices(qemu_bin):
+    '''
+    Get list of devices supported by a QEMU binary
+
+    @param qemu_bin (str): path to the QEMU binary
+    @return list of devices supported by the binary
+    '''
+    return binary_get_qom_implementations(qemu_bin, "device", False)
diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
index 7380f2f49b..a62eb2e46b 100644
--- a/tests/acceptance/core_scripts.py
+++ b/tests/acceptance/core_scripts.py
@@ -18,6 +18,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from avocado_qemu import Test
 from qemu.binutils import binary_get_accels
 from qemu.binutils import binary_get_arch
+from qemu.binutils import binary_get_devices
 from qemu.binutils import binary_get_machines
 from qemu.binutils import binary_get_qom_implementations
 from qemu.binutils import binary_get_version
@@ -69,3 +70,10 @@ class PythonQemuCoreScripts(Test):
         self.assertNotIn("accel", accels)
         # qtest is always available
         self.assertIn("qtest", accels)
+
+    def test_get_devices(self):
+        logger = logging.getLogger('core')
+        devices = binary_get_devices(self.qemu_bin)
+        for d in devices:
+            logger.debug('device: {}'.format(d))
+        self.assertIn("migration", devices)
-- 
2.21.1

[RFC PATCH v2 14/29] tests/acceptance: Use 'device' tags to check availability in QEMU binary

[Philippe Mathieu-Daudé]

We already use the 'device' tag in Avocado tests.
If any of the requested device is not available in the QEMU binary,
the tests will be cancelled (skipped).

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
RFC because I'm not certain, we might want to find bugs where
    a device is missing in the built binary...
---
 tests/acceptance/avocado_qemu/__init__.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 53ec8512d1..44062aea56 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -20,6 +20,7 @@ SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
 sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
 
 from qemu.binutils import binary_get_arch
+from qemu.binutils import binary_get_devices
 from qemu.binutils import binary_get_machines
 from qemu.binutils import binary_get_version
 from qemu.machine import QEMUMachine
@@ -158,6 +159,16 @@ class Test(avocado.Test):
             if self.machine not in binary_get_machines(self.qemu_bin):
                 self.cancel(fmt.format(self.machine))
 
+        # Verify devices
+        self.devices = self.tags.get('device', [])
+        bin_devices = binary_get_devices(self.qemu_bin)
+        logger.debug('devices: {}'.format(self.devices))
+        fmt = "Test expects device '{}' which is missing from QEMU binary"
+        for dev in self.devices:
+            logger.debug(dev)
+            if dev not in bin_devices:
+                self.cancel(fmt.format(dev))
+
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
         if args:
-- 
2.21.1

[PATCH v2 15/29] tests: rename virtio_seg_max_adjust to virtio_check_params

[Philippe Mathieu-Daudé]

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

Since, virtio_seg_max_adjust checks not only seg_max, but also
virtqueue_size parameter, let's make the test more general and
add new parameters to be checked there in the future.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Message-Id: <20200129140702.5411-5-dplotnikov@virtuozzo.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .../{virtio_seg_max_adjust.py => virtio_check_params.py}          | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py} (100%)

diff --git a/tests/acceptance/virtio_seg_max_adjust.py b/tests/acceptance/virtio_check_params.py
similarity index 100%
rename from tests/acceptance/virtio_seg_max_adjust.py
rename to tests/acceptance/virtio_check_params.py
-- 
2.21.1

[PATCH v2 16/29] tests/acceptance/virtio_check_params: Only remove listed machines

[Philippe Mathieu-Daudé]

Do not remove unavailable machines, this fixes:

  VirtioMaxSegSettingsCheck.test_machine_types: ERROR: list.remove(x): x not in list (0.12 s)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
TODO check how to use attribute (Wainer)
---
 tests/acceptance/virtio_check_params.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 5458573138..4a417b8ef5 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -109,14 +109,15 @@ class VirtioMaxSegSettingsCheck(Test):
         return False
 
     def test_machine_types(self):
-        # collect all machine types except 'none', 'isapc', 'microvm'
+        EXCLUDED_MACHINES = ['none', 'isapc', 'microvm']
+        # collect all machine types except the ones in EXCLUDED_MACHINES
         with QEMUMachine(self.qemu_bin) as vm:
             vm.launch()
             machines = [m['name'] for m in vm.command('query-machines')]
             vm.shutdown()
-        machines.remove('none')
-        machines.remove('isapc')
-        machines.remove('microvm')
+        for m in EXCLUDED_MACHINES:
+            if m in machines:
+                machines.remove(m)
 
         for dev_type in DEV_TYPES:
             # create the list of machine types and their parameters.
-- 
2.21.1

[PATCH v2 17/29] tests/acceptance/virtio_check_params: Improve exception logging

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 4a417b8ef5..51a2dd76e8 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -77,8 +77,12 @@ class VirtioMaxSegSettingsCheck(Test):
             vm.set_machine(mt["name"])
             for s in VM_DEV_PARAMS[dev_type_name]:
                 vm.add_args(s)
-            vm.launch()
-            query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
+            try:
+                vm.launch()
+                query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
+            except:
+                query_ok = False
+                error = sys.exc_info()[0]
 
         if not query_ok:
             self.fail('machine type {0}: {1}'.format(mt['name'], error))
-- 
2.21.1

[PATCH v2 18/29] tests/acceptance/virtio_check_params: List machine being tested

[Philippe Mathieu-Daudé]

Add logging for easier debugging of failures:

  $ avocado --show=machine run tests/acceptance/virtio_check_params.py
   (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
  machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'pc-q35-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'pc-i440fx-2.5', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'pc-i440fx-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  ...

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 51a2dd76e8..f679b0eec7 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -21,6 +21,7 @@
 import sys
 import os
 import re
+import logging
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu.machine import QEMUMachine
@@ -73,6 +74,9 @@ class VirtioMaxSegSettingsCheck(Test):
         return query_ok, props, error
 
     def check_mt(self, mt, dev_type_name):
+        mt['device'] = dev_type_name # Only for the debug() call.
+        logger = logging.getLogger('machine')
+        logger.debug(mt)
         with QEMUMachine(self.qemu_bin) as vm:
             vm.set_machine(mt["name"])
             for s in VM_DEV_PARAMS[dev_type_name]:
-- 
2.21.1

[PATCH v2 19/29] tests/acceptance/virtio_check_params: Only test Xen as superuser

[Philippe Mathieu-Daudé]

When running the test unprivileged, we get:

  $ avocado --show=app,machine run tests/acceptance/virtio_check_params.py
  JOB ID     : b631d5d692e49b791b211d33b80730315d561d45
  JOB LOG    : job-results/job-2020-01-22T17.56-b631d5d/job.log
   (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
  machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  machine: {'name': 'xenpv', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
  FAIL: machine type xenpv: <class 'qemu.qmp.QMPConnectError'> (0.40 s)

Looking at the job.log file we find:

  xencall: error: Could not obtain handle on privileged command interface: No such file or directory
  xen be core: xen be core: can't open xen interface

Do not run this test on Xen machines if not superuser.

Acked-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index f679b0eec7..ad736bcda3 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -118,6 +118,8 @@ class VirtioMaxSegSettingsCheck(Test):
 
     def test_machine_types(self):
         EXCLUDED_MACHINES = ['none', 'isapc', 'microvm']
+        if os.geteuid() != 0:
+            EXCLUDED_MACHINES += ['xenfv', 'xenpv']
         # collect all machine types except the ones in EXCLUDED_MACHINES
         with QEMUMachine(self.qemu_bin) as vm:
             vm.launch()
-- 
2.21.1

[PATCH v2 20/29] tests/acceptance/virtio_check_params: Skip test if arch is not supported

[Philippe Mathieu-Daudé]

Refactor the X86 specific code. If we run this test on an
architecture which is not explicitly supported, the test
will be cancelled (skipped).

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 37 +++++++++++++++----------
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index ad736bcda3..8752e25f08 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -26,6 +26,8 @@ import logging
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 from qemu.machine import QEMUMachine
 from avocado_qemu import Test
+from qemu.binutils import binary_get_arch
+from avocado.core.exceptions import TestCancel
 
 #list of machine types and virtqueue properties to test
 VIRTIO_SCSI_PROPS = {'seg_max_adjust': 'seg_max_adjust'}
@@ -96,18 +98,21 @@ class VirtioMaxSegSettingsCheck(Test):
             self.assertEqual(expected_val, prop_val)
 
     @staticmethod
-    def seg_max_adjust_enabled(mt):
-        # machine types >= 5.0 should have seg_max_adjust = true
-        # others seg_max_adjust = false
-        mt = mt.split("-")
+    def seg_max_adjust_enabled(arch, mt):
+        if arch in ['i386', 'x86_64']:
+            # machine types >= 5.0 should have seg_max_adjust = true
+            # others seg_max_adjust = false
+            mt = mt.split("-")
 
-        # machine types with one line name and name like pc-x.x
-        if len(mt) <= 2:
-            return False
+            # machine types with one line name and name like pc-x.x
+            if len(mt) <= 2:
+                return False
 
-        # machine types like pc-<chip_name>-x.x[.x]
-        ver = mt[2]
-        ver = ver.split(".");
+            # machine types like pc-<chip_name>-x.x[.x]
+            ver = mt[2]
+            ver = ver.split(".");
+        else:
+            raise TestCancel('Unsupported architecture: %s' % arch)
 
         # versions >= 5.0 goes with seg_max_adjust enabled
         major = int(ver[0])
@@ -117,9 +122,13 @@ class VirtioMaxSegSettingsCheck(Test):
         return False
 
     def test_machine_types(self):
-        EXCLUDED_MACHINES = ['none', 'isapc', 'microvm']
-        if os.geteuid() != 0:
-            EXCLUDED_MACHINES += ['xenfv', 'xenpv']
+        arch = binary_get_arch(self.qemu_bin)
+
+        EXCLUDED_MACHINES = ['none']
+        if arch in ['i386', 'x86_64']:
+            EXCLUDED_MACHINES += ['isapc', 'microvm']
+            if os.geteuid() != 0:
+                EXCLUDED_MACHINES += ['xenfv', 'xenpv']
         # collect all machine types except the ones in EXCLUDED_MACHINES
         with QEMUMachine(self.qemu_bin) as vm:
             vm.launch()
@@ -133,7 +142,7 @@ class VirtioMaxSegSettingsCheck(Test):
             # create the list of machine types and their parameters.
             mtypes = list()
             for m in machines:
-                if self.seg_max_adjust_enabled(m):
+                if self.seg_max_adjust_enabled(arch, m):
                     enabled = 'true'
                 else:
                     enabled = 'false'
-- 
2.21.1

[RFC PATCH v2 21/29] tests/acceptance/virtio_check_params: Kludge to skip tests on MIPS

[Philippe Mathieu-Daudé]

The MIPS default machine expects a BIOS.
To be able to run QMP queries on the machine, we have to use
the '-bios' command line option.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
RFC: This is not the correct place, this should be generic,
     or the MIPS machine need some rework.
---
 tests/acceptance/virtio_check_params.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 8752e25f08..73200c9c5b 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -131,6 +131,9 @@ class VirtioMaxSegSettingsCheck(Test):
                 EXCLUDED_MACHINES += ['xenfv', 'xenpv']
         # collect all machine types except the ones in EXCLUDED_MACHINES
         with QEMUMachine(self.qemu_bin) as vm:
+            if arch.startswith('mips'):
+                # FIXME kludge for MIPS default machine
+                vm.add_args('-bios', '/dev/null')
             vm.launch()
             machines = [m['name'] for m in vm.command('query-machines')]
             vm.shutdown()
-- 
2.21.1

[PATCH v2 22/29] tests/acceptance/virtio_check_params: Support the s390x architecture

[Philippe Mathieu-Daudé]

We can run these tests on the s390x architecture too,
we were just missing to properly parse the machine names.

Suggested-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 73200c9c5b..b14cfb5958 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -111,6 +111,8 @@ class VirtioMaxSegSettingsCheck(Test):
             # machine types like pc-<chip_name>-x.x[.x]
             ver = mt[2]
             ver = ver.split(".");
+        elif arch == 's390x':
+            ver = mt[len('s390-ccw-virtio-'):].split('.')
         else:
             raise TestCancel('Unsupported architecture: %s' % arch)
 
-- 
2.21.1

[PATCH v2 23/29] tests/acceptance/virtio_check_params: Support the ppc architecture

[Philippe Mathieu-Daudé]

We can run these tests on the pSeries machines too,
we were just missing to properly parse the machine names.

Note: we have to exclude the versioned machines from QEMU 2.8
to 2.11 which fail, such:

  TestFail: machine type pseries-2.8: No Transactional Memory support in TCG, try appending -machine cap-htm=off

Suggested-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index b14cfb5958..2c14af825d 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -113,6 +113,8 @@ class VirtioMaxSegSettingsCheck(Test):
             ver = ver.split(".");
         elif arch == 's390x':
             ver = mt[len('s390-ccw-virtio-'):].split('.')
+        elif arch.startswith('ppc'):
+            ver = mt[len('pseries-'):].split('.')
         else:
             raise TestCancel('Unsupported architecture: %s' % arch)
 
@@ -131,6 +133,9 @@ class VirtioMaxSegSettingsCheck(Test):
             EXCLUDED_MACHINES += ['isapc', 'microvm']
             if os.geteuid() != 0:
                 EXCLUDED_MACHINES += ['xenfv', 'xenpv']
+        elif arch.startswith('ppc'):
+            EXCLUDED_MACHINES += ['pseries-2.8', 'pseries-2.9',
+                                  'pseries-2.10', 'pseries-2.11']
         # collect all machine types except the ones in EXCLUDED_MACHINES
         with QEMUMachine(self.qemu_bin) as vm:
             if arch.startswith('mips'):
@@ -142,6 +147,8 @@ class VirtioMaxSegSettingsCheck(Test):
         for m in EXCLUDED_MACHINES:
             if m in machines:
                 machines.remove(m)
+        if arch.startswith('ppc'):
+            machines = [m for m in machines if m.startswith('pseries')]
 
         for dev_type in DEV_TYPES:
             # create the list of machine types and their parameters.
-- 
2.21.1

[PATCH v2 24/29] tests/acceptance/virtio_check_params: Default to -nodefaults

[Philippe Mathieu-Daudé]

We don't need the default options to run this test.

This fixes errors when running a binary built with
--without-default-devices such:

  ERROR: qemu-system-s390x: Unknown device 'virtio-net-ccw' for bus 'virtual-css-bus'

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 2c14af825d..3466aa5b32 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -81,6 +81,7 @@ class VirtioMaxSegSettingsCheck(Test):
         logger.debug(mt)
         with QEMUMachine(self.qemu_bin) as vm:
             vm.set_machine(mt["name"])
+            vm.add_args('-nodefaults')
             for s in VM_DEV_PARAMS[dev_type_name]:
                 vm.add_args(s)
             try:
-- 
2.21.1

[PATCH v2 25/29] tests/acceptance/virtio_check_params: Require a virtio-scsi-pci device

[Philippe Mathieu-Daudé]

When running this test with QEMU configured using
--without-default-devices, we get:

  ERROR: qemu-system-s390x: -device virtio-scsi-pci,id=scsi0: 'virtio-scsi-pci' is not a valid device model name

To be able to run the acceptance tests with a binary built using
--without-default-devices, add a 'device' tag. The Test::setUp()
method will verify the device is available, else cancel (skip)
the test.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/virtio_check_params.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 3466aa5b32..95a9048786 100755
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -44,6 +44,9 @@ VM_DEV_PARAMS = {'virtio-scsi-pci': ['-device', 'virtio-scsi-pci,id=scsi0'],
 
 
 class VirtioMaxSegSettingsCheck(Test):
+    """
+    :avocado: tags=device:virtio-scsi-pci
+    """
     @staticmethod
     def make_pattern(props):
         pattern_items = ['{0} = \w+'.format(prop) for prop in props]
-- 
2.21.1

[PATCH v2 26/29] tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine

[Philippe Mathieu-Daudé]

As we only read the serial console, we don't need to force a
VGA display. This fixes when running a binary built with
--without-default-devices:

  ERROR: qemu-system-alpha: standard VGA not available

We also need the '-nodefaults' argument to avoid:

  ERROR: qemu-system-alpha: Unsupported NIC model: e1000

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index e40b84651b..7d6065828c 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -519,7 +519,7 @@ class BootLinuxConsole(Test):
 
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
-        self.vm.add_args('-vga', 'std',
+        self.vm.add_args('-nodefaults',
                          '-kernel', uncompressed_kernel,
                          '-append', kernel_command_line)
         self.vm.launch()
-- 
2.21.1

[PATCH v2 27/29] tests/acceptance/migration: Default to -nodefaults

[Philippe Mathieu-Daudé]

We don't need the default options to run this test.

This fixes errors when running a binary built with
--without-default-devices such:

  ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/migration.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
index a44c1ae58f..d09e5d3fe7 100644
--- a/tests/acceptance/migration.py
+++ b/tests/acceptance/migration.py
@@ -32,10 +32,12 @@ class Migration(Test):
 
 
     def test_migration_with_tcp_localhost(self):
-        source_vm = self.get_vm()
         dest_uri = 'tcp:localhost:%u' % self._get_free_port()
         dest_vm = self.get_vm('-incoming', dest_uri)
+        dest_vm.add_args('-nodefaults')
         dest_vm.launch()
+        source_vm = self.get_vm()
+        source_vm.add_args('-nodefaults')
         source_vm.launch()
         source_vm.qmp('migrate', uri=dest_uri)
         wait.wait_for(
-- 
2.21.1

[PATCH v2 28/29] tests/acceptance/version: Default to -nodefaults

[Philippe Mathieu-Daudé]

We don't need the default options to run this test.

This fixes errors when running a binary built with
--without-default-devices such:

  ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/version.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/version.py b/tests/acceptance/version.py
index 67c2192c93..79b923d4fc 100644
--- a/tests/acceptance/version.py
+++ b/tests/acceptance/version.py
@@ -17,6 +17,7 @@ class Version(Test):
     :avocado: tags=quick
     """
     def test_qmp_human_info_version(self):
+        self.vm.add_args('-nodefaults')
         self.vm.launch()
         res = self.vm.command('human-monitor-command',
                               command_line='info version')
-- 
2.21.1

[PATCH v2 29/29] MAINTAINERS: Add Acceptance tests reviewers

[Philippe Mathieu-Daudé]

Acceptance tests can test any piece of the QEMU codebase.
As such, the directory holding them does not belong to a specific
subsystem with designated maintainers.

Each subsystem covered by a test is welcomed to add the test path
to its section.
See for example commits 71b290e70, b11785ca2 or 5d480ddde.

Since Eduardo, Cleber and Philippe have been reviewing the patches
on the list, add them as reviewers.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
Cc: Wainer dos Santos Moschetta <wainersm@redhat.com>
Wainer do you want to be listed?
---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index efd3f3875f..4a3cc3f137 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2733,6 +2733,12 @@ S: Maintained
 F: tests/tcg/Makefile
 F: tests/tcg/Makefile.include
 
+Acceptance Tests
+R: Eduardo Habkost <ehabkost@redhat.com>
+R: Cleber Rosa <crosa@redhat.com>
+R: Philippe Mathieu-Daudé <philmd@redhat.com>
+F: tests/acceptance/
+
 Documentation
 -------------
 Build system architecture
-- 
2.21.1

Re: [PATCH v2 00/29] tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86

[Philippe Mathieu-Daudé]

Wrong subject... should be:

   "Improve multi-arch/OS support of acceptance tests"

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
> Currently our CI is mostly restricted to Linux/X86.
> We want to have more, but until there we have to run these tests
> manually, which is why we didn't noticed earlier this test was
> failing on OSX.
> 
> This series contains the patches I used to debug why the
> virtio_seg_max_adjust had broken the OSX CI, but then I
> kept testing on other archs, thus added more patches...
> 
> Currently our CI is mostly restricted to Linux/X86.
> We want to have more, but until there we have to run these tests
> manually, which is why we didn't noticed earlier some tests are
> failing (on OSX, using --without-default-devices, ...).
> 
> Tested on Linux/FreeBSD/OSX, X86/Aarch64.
> 
> Regards,
> 
> Phil.
> 
> Supersedes: <20200122223247.30419-1-philmd@redhat.com>
> "tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86"
> 
> Denis Plotnikov (1):
>    tests: rename virtio_seg_max_adjust to virtio_check_params
> 
> Philippe Mathieu-Daudé (28):
>    hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host
>    accel/accel: Make TYPE_ACCEL abstract
>    python/qemu: Add binutils::binary_get_version()
>    tests/acceptance: Use 'version-min' tag to verify QEMU binary version
>    tests/acceptance: Restrict X86CPUModelAliases test to QEMU >= 4.1
>    python/qemu: Add binutils::binary_get_arch()
>    tests/acceptance: Use the 'arch' tag to verify QEMU binary target
>    python/qemu: Add binutils::binary_get_machines()
>    tests/acceptance: Use 'machine' tag to check if available in QEMU
>      binary
>    python/qemu: Add binutils::binary_get_qom_implementations()
>    python/qemu: Add binutils::binary_get_accels()
>    python/qemu/accel: Use binutils::binary_get_accels()
>    python/qemu: Add binutils::binary_get_devices()
>    tests/acceptance: Use 'device' tags to check availability in QEMU
>      binary
>    tests/acceptance/virtio_check_params: Only remove listed machines
>    tests/acceptance/virtio_check_params: Improve exception logging
>    tests/acceptance/virtio_check_params: List machine being tested
>    tests/acceptance/virtio_check_params: Only test Xen as superuser
>    tests/acceptance/virtio_check_params: Skip test if arch is not
>      supported
>    tests/acceptance/virtio_check_params: Kludge to skip tests on MIPS
>    tests/acceptance/virtio_check_params: Support the s390x architecture
>    tests/acceptance/virtio_check_params: Support the ppc architecture
>    tests/acceptance/virtio_check_params: Default to -nodefaults
>    tests/acceptance/virtio_check_params: Require a virtio-scsi-pci device
>    tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine
>    tests/acceptance/migration: Default to -nodefaults
>    tests/acceptance/version: Default to -nodefaults
>    MAINTAINERS: Add Acceptance tests reviewers
> 
>   accel/accel.c                                 |   1 +
>   hw/core/machine.c                             |   3 +-
>   MAINTAINERS                                   |   6 +
>   python/qemu/accel.py                          |  26 +----
>   python/qemu/binutils.py                       | 107 ++++++++++++++++++
>   tests/acceptance/avocado_qemu/__init__.py     |  51 ++++++++-
>   tests/acceptance/boot_linux_console.py        |   2 +-
>   tests/acceptance/core_scripts.py              |  79 +++++++++++++
>   tests/acceptance/migration.py                 |   4 +-
>   tests/acceptance/version.py                   |   1 +
>   ...g_max_adjust.py => virtio_check_params.py} |  70 +++++++++---
>   tests/acceptance/x86_cpu_model_versions.py    |   1 +
>   12 files changed, 306 insertions(+), 45 deletions(-)
>   create mode 100644 python/qemu/binutils.py
>   create mode 100644 tests/acceptance/core_scripts.py
>   rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py} (64%)
> 

Re: [PATCH v2 02/29] accel/accel: Make TYPE_ACCEL abstract

[Philippe Mathieu-Daudé]

Since this file isn't covered in MAINTAINERS, manually Cc'ing Paolo.

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
> There is no generic acceleration, we have to use specific
> implementations. Make the base class abstract.
> 
> Fixes: b14a0b7469f
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   accel/accel.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/accel/accel.c b/accel/accel.c
> index cb555e3b06..a0169b4e69 100644
> --- a/accel/accel.c
> +++ b/accel/accel.c
> @@ -33,6 +33,7 @@
>   static const TypeInfo accel_type = {
>       .name = TYPE_ACCEL,
>       .parent = TYPE_OBJECT,
> +    .abstract = true,
>       .class_size = sizeof(AccelClass),
>       .instance_size = sizeof(AccelState),
>   };
> 

Re: [PATCH v2 29/29] MAINTAINERS: Add Acceptance tests reviewers

[Thomas Huth]

On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
> Acceptance tests can test any piece of the QEMU codebase.
> As such, the directory holding them does not belong to a specific
> subsystem with designated maintainers.
> 
> Each subsystem covered by a test is welcomed to add the test path
> to its section.
> See for example commits 71b290e70, b11785ca2 or 5d480ddde.
> 
> Since Eduardo, Cleber and Philippe have been reviewing the patches
> on the list, add them as reviewers.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> Cc: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Wainer do you want to be listed?
> ---
>  MAINTAINERS | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index efd3f3875f..4a3cc3f137 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2733,6 +2733,12 @@ S: Maintained
>  F: tests/tcg/Makefile
>  F: tests/tcg/Makefile.include
>  
> +Acceptance Tests
> +R: Eduardo Habkost <ehabkost@redhat.com>
> +R: Cleber Rosa <crosa@redhat.com>
> +R: Philippe Mathieu-Daudé <philmd@redhat.com>
> +F: tests/acceptance/

Many files under tests/acceptance/ are not covered by a maintainer at
all, e.g.:

 $ scripts/get_maintainer.pl -f tests/acceptance/boot_linux_console.py
 get_maintainer.pl: No maintainers found, printing recent contributors.

... so I think it would be good to have someone who feels at least a
little bit responsible to do the maintainers work here and picks up
patches to put them into a pull request. Any volunteers?

 Thomas

Re: [PATCH v2 01/29] hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:17 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Commit ed65fd1a27 restricted the 'scsi' property to Linux hosts:

It was restricted to Linux hosts before that commit; that only became
problematic because we set it unconditionally in the compat code.

> 
>   1261 static Property virtio_blk_properties[] = {
>   1262     DEFINE_BLOCK_PROPERTIES(VirtIOBlock, conf.conf),
>   ...
>   1268 #ifdef __linux__
>   1269     DEFINE_PROP_BIT64("scsi", VirtIOBlock, host_features,
>   1270                       VIRTIO_BLK_F_SCSI, false),
>   1271 #endif

What about:

Commit ed65fd1a2750 ("virtio-blk: switch off scsi-passthrough by
default") changed the default value of the 'scsi' property of
virtio-blk, which is only available on Linux hosts. It also added an
unconditional compat entry for 2.4 or earlier machines.

> 
> When we try to set this property on a pre-2.4 machine on OSX,

Pre-2.5, I think?

> we get:
> 
>    Unexpected error in object_property_find() at qom/object.c:1201:
>    qemu-system-x86_64: -device virtio-blk-pci,id=scsi0,drive=drive0: can't apply global virtio-blk-device.scsi=true: Property '.scsi' not found
> 
> Fix this error by marking the property optional.
> 
> Fixes: ed65fd1a27

Fixes: ed65fd1a2750 ("virtio-blk: switch off scsi-passthrough by default")

> Suggested-by: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  hw/core/machine.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index 3e288bfceb..d8e30e4895 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -148,7 +148,8 @@ GlobalProperty hw_compat_2_5[] = {
>  const size_t hw_compat_2_5_len = G_N_ELEMENTS(hw_compat_2_5);
>  
>  GlobalProperty hw_compat_2_4[] = {
> -    { "virtio-blk-device", "scsi", "true" },
> +    /* Optional because the 'scsi' property is Linux-only */
> +    { "virtio-blk-device", "scsi", "true", .optional = true },
>      { "e1000", "extra_mac_registers", "off" },
>      { "virtio-pci", "x-disable-pcie", "on" },
>      { "virtio-pci", "migrate-extra", "off" },

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 02/29] accel/accel: Make TYPE_ACCEL abstract

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:18 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> There is no generic acceleration, we have to use specific
> implementations. Make the base class abstract.
> 
> Fixes: b14a0b7469f

Fixes: b14a0b7469fa ("accel: Use QOM classes for accel types")

> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  accel/accel.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/accel/accel.c b/accel/accel.c
> index cb555e3b06..a0169b4e69 100644
> --- a/accel/accel.c
> +++ b/accel/accel.c
> @@ -33,6 +33,7 @@
>  static const TypeInfo accel_type = {
>      .name = TYPE_ACCEL,
>      .parent = TYPE_OBJECT,
> +    .abstract = true,
>      .class_size = sizeof(AccelClass),
>      .instance_size = sizeof(AccelState),
>  };

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:20 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Introduce the 'version-min' tag.
> Tests can set it to the minimum version of QEMU they require.
> If QEMU is older, the tests will be cancelled (skipped):
> 
>   $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>    (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>    (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>    (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>    ...
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)

Hm... this might cause headaches if someone tries to backport tests on
distro versions; but we probably won't be able to test for the feature
in many cases anyway?

Re: [PATCH v2 15/29] tests: rename virtio_seg_max_adjust to virtio_check_params

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:31 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> From: Denis Plotnikov <dplotnikov@virtuozzo.com>
> 
> Since, virtio_seg_max_adjust checks not only seg_max, but also
> virtqueue_size parameter, let's make the test more general and
> add new parameters to be checked there in the future.
> 
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Message-Id: <20200129140702.5411-5-dplotnikov@virtuozzo.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .../{virtio_seg_max_adjust.py => virtio_check_params.py}          | 0
>  1 file changed, 0 insertions(+), 0 deletions(-)
>  rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py} (100%)
> 
> diff --git a/tests/acceptance/virtio_seg_max_adjust.py b/tests/acceptance/virtio_check_params.py
> similarity index 100%
> rename from tests/acceptance/virtio_seg_max_adjust.py
> rename to tests/acceptance/virtio_check_params.py

Makes sense, especially if we want to add non-blk/scsi checks in there.

Acked-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 18/29] tests/acceptance/virtio_check_params: List machine being tested

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:34 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Add logging for easier debugging of failures:
> 
>   $ avocado --show=machine run tests/acceptance/virtio_check_params.py
>    (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
>   machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'pc-q35-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'pc-i440fx-2.5', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'pc-i440fx-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   ...
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/virtio_check_params.py | 4 ++++
>  1 file changed, 4 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 19/29] tests/acceptance/virtio_check_params: Only test Xen as superuser

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:35 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> When running the test unprivileged, we get:
> 
>   $ avocado --show=app,machine run tests/acceptance/virtio_check_params.py
>   JOB ID     : b631d5d692e49b791b211d33b80730315d561d45
>   JOB LOG    : job-results/job-2020-01-22T17.56-b631d5d/job.log
>    (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
>   machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   machine: {'name': 'xenpv', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>   FAIL: machine type xenpv: <class 'qemu.qmp.QMPConnectError'> (0.40 s)
> 
> Looking at the job.log file we find:
> 
>   xencall: error: Could not obtain handle on privileged command interface: No such file or directory
>   xen be core: xen be core: can't open xen interface
> 
> Do not run this test on Xen machines if not superuser.
> 
> Acked-by: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/virtio_check_params.py | 2 ++
>  1 file changed, 2 insertions(+)

I still think changing the title to "Test Xen only as superuser" would
be an improvement.

Re: [PATCH v2 01/29] hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host

[Philippe Mathieu-Daudé]

On 1/30/20 1:20 PM, Cornelia Huck wrote:
> On Wed, 29 Jan 2020 22:23:17 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> Commit ed65fd1a27 restricted the 'scsi' property to Linux hosts:
> 
> It was restricted to Linux hosts before that commit; that only became
> problematic because we set it unconditionally in the compat code.
> 
>>
>>    1261 static Property virtio_blk_properties[] = {
>>    1262     DEFINE_BLOCK_PROPERTIES(VirtIOBlock, conf.conf),
>>    ...
>>    1268 #ifdef __linux__
>>    1269     DEFINE_PROP_BIT64("scsi", VirtIOBlock, host_features,
>>    1270                       VIRTIO_BLK_F_SCSI, false),
>>    1271 #endif
> 
> What about:
> 
> Commit ed65fd1a2750 ("virtio-blk: switch off scsi-passthrough by
> default") changed the default value of the 'scsi' property of
> virtio-blk, which is only available on Linux hosts. It also added an
> unconditional compat entry for 2.4 or earlier machines.

OK!

>>
>> When we try to set this property on a pre-2.4 machine on OSX,
> 
> Pre-2.5, I think?

Yes :S

>> we get:
>>
>>     Unexpected error in object_property_find() at qom/object.c:1201:
>>     qemu-system-x86_64: -device virtio-blk-pci,id=scsi0,drive=drive0: can't apply global virtio-blk-device.scsi=true: Property '.scsi' not found
>>
>> Fix this error by marking the property optional.
>>
>> Fixes: ed65fd1a27
> 
> Fixes: ed65fd1a2750 ("virtio-blk: switch off scsi-passthrough by default")

OK.

> 
>> Suggested-by: Cornelia Huck <cohuck@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   hw/core/machine.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/core/machine.c b/hw/core/machine.c
>> index 3e288bfceb..d8e30e4895 100644
>> --- a/hw/core/machine.c
>> +++ b/hw/core/machine.c
>> @@ -148,7 +148,8 @@ GlobalProperty hw_compat_2_5[] = {
>>   const size_t hw_compat_2_5_len = G_N_ELEMENTS(hw_compat_2_5);
>>   
>>   GlobalProperty hw_compat_2_4[] = {
>> -    { "virtio-blk-device", "scsi", "true" },
>> +    /* Optional because the 'scsi' property is Linux-only */
>> +    { "virtio-blk-device", "scsi", "true", .optional = true },
>>       { "e1000", "extra_mac_registers", "off" },
>>       { "virtio-pci", "x-disable-pcie", "on" },
>>       { "virtio-pci", "migrate-extra", "off" },
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Thanks!

Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Philippe Mathieu-Daudé]

On 1/30/20 1:26 PM, Cornelia Huck wrote:
> On Wed, 29 Jan 2020 22:23:20 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> Introduce the 'version-min' tag.
>> Tests can set it to the minimum version of QEMU they require.
>> If QEMU is older, the tests will be cancelled (skipped):
>>
>>    $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>>     (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>     (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>     (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>     ...
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
>>   1 file changed, 18 insertions(+)
> 
> Hm... this might cause headaches if someone tries to backport tests on
> distro versions; but we probably won't be able to test for the feature
> in many cases anyway?
> 

Are you saying a distro version might not match upstream version when a 
feature is introduced? Isn't it the point of versioning? If you are 
thinking of some case you point me at it so I can improve this patch?

Thanks,

Phil.

Re: [PATCH v2 19/29] tests/acceptance/virtio_check_params: Only test Xen as superuser

[Philippe Mathieu-Daudé]

On 1/30/20 1:37 PM, Cornelia Huck wrote:
> On Wed, 29 Jan 2020 22:23:35 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> When running the test unprivileged, we get:
>>
>>    $ avocado --show=app,machine run tests/acceptance/virtio_check_params.py
>>    JOB ID     : b631d5d692e49b791b211d33b80730315d561d45
>>    JOB LOG    : job-results/job-2020-01-22T17.56-b631d5d/job.log
>>     (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
>>    machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>>    machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>>    machine: {'name': 'xenpv', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>>    FAIL: machine type xenpv: <class 'qemu.qmp.QMPConnectError'> (0.40 s)
>>
>> Looking at the job.log file we find:
>>
>>    xencall: error: Could not obtain handle on privileged command interface: No such file or directory
>>    xen be core: xen be core: can't open xen interface
>>
>> Do not run this test on Xen machines if not superuser.
>>
>> Acked-by: Cornelia Huck <cohuck@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/acceptance/virtio_check_params.py | 2 ++
>>   1 file changed, 2 insertions(+)
> 
> I still think changing the title to "Test Xen only as superuser" would
> be an improvement.

If it fits the 72chars in patch subject, I'm happy to change :)

Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Cornelia Huck]

On Thu, 30 Jan 2020 14:28:19 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> On 1/30/20 1:26 PM, Cornelia Huck wrote:
> > On Wed, 29 Jan 2020 22:23:20 +0100
> > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> >   
> >> Introduce the 'version-min' tag.
> >> Tests can set it to the minimum version of QEMU they require.
> >> If QEMU is older, the tests will be cancelled (skipped):
> >>
> >>    $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
> >>     (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> >>     (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> >>     (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> >>     ...
> >>
> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >> ---
> >>   tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
> >>   1 file changed, 18 insertions(+)  
> > 
> > Hm... this might cause headaches if someone tries to backport tests on
> > distro versions; but we probably won't be able to test for the feature
> > in many cases anyway?
> >   
> 
> Are you saying a distro version might not match upstream version when a 
> feature is introduced? Isn't it the point of versioning? If you are 
> thinking of some case you point me at it so I can improve this patch?

Something like the following:

- a feature together with a test is introduced in QEMU version n
- a distro is using m (where m < n), but wants the new feature and
  backports it

If that distro now wants to include the acceptance test as well (to run
in their internal testing), they need to adapt the version check as
well, or the test simply won't run. Depending on how this is reported,
it might not be easily noticed. If the test depends on another feature
(but the feature does not), it might get a bit more complicated.

It's probably not a big problem, but I wanted to point it out.

Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Philippe Mathieu-Daudé]

Cc'ing Markus

On 1/30/20 6:08 PM, Cornelia Huck wrote:
> On Thu, 30 Jan 2020 14:28:19 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> On 1/30/20 1:26 PM, Cornelia Huck wrote:
>>> On Wed, 29 Jan 2020 22:23:20 +0100
>>> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>>>    
>>>> Introduce the 'version-min' tag.
>>>> Tests can set it to the minimum version of QEMU they require.
>>>> If QEMU is older, the tests will be cancelled (skipped):
>>>>
>>>>     $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>>>>      (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      ...
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>> ---
>>>>    tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
>>>>    1 file changed, 18 insertions(+)
>>>
>>> Hm... this might cause headaches if someone tries to backport tests on
>>> distro versions; but we probably won't be able to test for the feature
>>> in many cases anyway?
>>>    
>>
>> Are you saying a distro version might not match upstream version when a
>> feature is introduced? Isn't it the point of versioning? If you are
>> thinking of some case you point me at it so I can improve this patch?
> 
> Something like the following:
> 
> - a feature together with a test is introduced in QEMU version n
> - a distro is using m (where m < n), but wants the new feature and
>    backports it
> 
> If that distro now wants to include the acceptance test as well (to run
> in their internal testing), they need to adapt the version check as
> well, or the test simply won't run. Depending on how this is reported,
> it might not be easily noticed. If the test depends on another feature
> (but the feature does not), it might get a bit more complicated.
> 
> It's probably not a big problem, but I wanted to point it out.

OK now I understand better.

I wrote that patch having QAPI versioned features in mind, which are 
versioned/documented.
For example:

   ##
   # @qom-list-types:
   #
   ...
   #
   # Since: 1.1

Now it is true that we add a new feature to a device, we do not document 
that in QAPI, and I don't see how we can test it via QMP.

Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Eduardo Habkost]

On Thu, Jan 30, 2020 at 06:18:22PM +0100, Philippe Mathieu-Daudé wrote:
> Cc'ing Markus
> 
> On 1/30/20 6:08 PM, Cornelia Huck wrote:
> > On Thu, 30 Jan 2020 14:28:19 +0100
> > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> > 
> > > On 1/30/20 1:26 PM, Cornelia Huck wrote:
> > > > On Wed, 29 Jan 2020 22:23:20 +0100
> > > > Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> > > > > Introduce the 'version-min' tag.
> > > > > Tests can set it to the minimum version of QEMU they require.
> > > > > If QEMU is older, the tests will be cancelled (skipped):
> > > > > 
> > > > >     $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
> > > > >      (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > > >      (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > > >      (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > > >      ...
> > > > > 
> > > > > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > > > > ---
> > > > >    tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
> > > > >    1 file changed, 18 insertions(+)
> > > > 
> > > > Hm... this might cause headaches if someone tries to backport tests on
> > > > distro versions; but we probably won't be able to test for the feature
> > > > in many cases anyway?
> > > 
> > > Are you saying a distro version might not match upstream version when a
> > > feature is introduced? Isn't it the point of versioning? If you are
> > > thinking of some case you point me at it so I can improve this patch?
> > 
> > Something like the following:
> > 
> > - a feature together with a test is introduced in QEMU version n
> > - a distro is using m (where m < n), but wants the new feature and
> >    backports it
> > 
> > If that distro now wants to include the acceptance test as well (to run
> > in their internal testing), they need to adapt the version check as
> > well, or the test simply won't run. Depending on how this is reported,
> > it might not be easily noticed. If the test depends on another feature
> > (but the feature does not), it might get a bit more complicated.
> > 
> > It's probably not a big problem, but I wanted to point it out.
> 
> OK now I understand better.
> 
> I wrote that patch having QAPI versioned features in mind, which are
> versioned/documented.
> For example:
> 
>   ##
>   # @qom-list-types:
>   #
>   ...
>   #
>   # Since: 1.1
> 
> Now it is true that we add a new feature to a device, we do not document
> that in QAPI, and I don't see how we can test it via QMP.

The features are documented as introduced in specific QEMU
versions to help humans, but the best way to test for their
existence is to check the QAPI schema.

-- 
Eduardo

Re: [PATCH v2 15/29] tests: rename virtio_seg_max_adjust to virtio_check_params

[Philippe Mathieu-Daudé]

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
> From: Denis Plotnikov <dplotnikov@virtuozzo.com>
> 
> Since, virtio_seg_max_adjust checks not only seg_max, but also
> virtqueue_size parameter, let's make the test more general and
> add new parameters to be checked there in the future.
> 
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Message-Id: <20200129140702.5411-5-dplotnikov@virtuozzo.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   .../{virtio_seg_max_adjust.py => virtio_check_params.py}          | 0
>   1 file changed, 0 insertions(+), 0 deletions(-)
>   rename tests/acceptance/{virtio_seg_max_adjust.py => virtio_check_params.py} (100%)
> 
> diff --git a/tests/acceptance/virtio_seg_max_adjust.py b/tests/acceptance/virtio_check_params.py
> similarity index 100%
> rename from tests/acceptance/virtio_seg_max_adjust.py
> rename to tests/acceptance/virtio_check_params.py
> 

Thanks, applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next

Re: [PATCH v2 18/29] tests/acceptance/virtio_check_params: List machine being tested

[Philippe Mathieu-Daudé]

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
> Add logging for easier debugging of failures:
> 
>    $ avocado --show=machine run tests/acceptance/virtio_check_params.py
>     (1/1) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
>    machine: {'name': 'pc-i440fx-2.12', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>    machine: {'name': 'pc-i440fx-2.0', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>    machine: {'name': 'pc-q35-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>    machine: {'name': 'pc-i440fx-2.5', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>    machine: {'name': 'pc-i440fx-4.2', 'seg_max_adjust': 'false', 'device': 'virtio-scsi-pci'}
>    ...
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   tests/acceptance/virtio_check_params.py | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
> index 51a2dd76e8..f679b0eec7 100755
> --- a/tests/acceptance/virtio_check_params.py
> +++ b/tests/acceptance/virtio_check_params.py
> @@ -21,6 +21,7 @@
>   import sys
>   import os
>   import re
> +import logging
>   
>   sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
>   from qemu.machine import QEMUMachine
> @@ -73,6 +74,9 @@ class VirtioMaxSegSettingsCheck(Test):
>           return query_ok, props, error
>   
>       def check_mt(self, mt, dev_type_name):
> +        mt['device'] = dev_type_name # Only for the debug() call.
> +        logger = logging.getLogger('machine')
> +        logger.debug(mt)
>           with QEMUMachine(self.qemu_bin) as vm:
>               vm.set_machine(mt["name"])
>               for s in VM_DEV_PARAMS[dev_type_name]:
> 

Thanks, applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next

Re: [PATCH v2 06/29] python/qemu: Add binutils::binary_get_arch()

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:22 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Add a helper to query the architecture of a QEMU binary.
> We simply send the 'query-target' command over a QMP socket.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  python/qemu/binutils.py          | 15 +++++++++++++++
>  tests/acceptance/core_scripts.py | 11 +++++++++++
>  2 files changed, 26 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 07/29] tests/acceptance: Use the 'arch' tag to verify QEMU binary target

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:23 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> We already use the 'arch' tag in Avocado tests. Tests can set
> it to restrict their use on a particular target architecture.
> 
> If the QEMU binary does not target the required architecture,
> the tests will be cancelled (skipped):
> 
>   $ avocado --show=app run tests/acceptance/boot_linux_console.py
>    ...
>    (14/19) BootLinuxConsole.test_arm_cubieboard_initrd: CANCEL: Test expects arch 'arm' while QEMU binary is targetting 'mips' (0.16 s)
>    (15/19) BootLinuxConsole.test_arm_cubieboard_sata: CANCEL: Test expects arch 'arm' while QEMU binary is targetting 'mips' (0.18 s)
>    (16/19) BootLinuxConsole.test_s390x_s390_ccw_virtio: CANCEL: Test expects arch 's390x' while QEMU binary is targetting 'mips' (0.14 s)
>    (17/19) BootLinuxConsole.test_alpha_clipper: CANCEL: Test expects arch 'alpha' while QEMU binary is targetting 'mips' (0.16 s)
>    (18/19) BootLinuxConsole.test_ppc64_pseries: CANCEL: Test expects arch 'ppc64' while QEMU binary is targetting 'mips' (0.18 s)
>    (19/19) BootLinuxConsole.test_m68k_q800: CANCEL: Test expects arch 'm68k' while QEMU binary is targetting 'mips' (0.17 s)
>   RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 16
>   JOB TIME   : 19.18 s
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 9 +++++++++
>  1 file changed, 9 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 08/29] python/qemu: Add binutils::binary_get_machines()

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:24 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Add a helper to query the list of machines built into a QEMU binary.
> We simply send the 'query-machines' command over a QMP socket.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  python/qemu/binutils.py          | 15 +++++++++++++++
>  tests/acceptance/core_scripts.py |  9 +++++++++
>  2 files changed, 24 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 09/29] tests/acceptance: Use 'machine' tag to check if available in QEMU binary

[Cornelia Huck]

On Wed, 29 Jan 2020 22:23:25 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> We already use the 'machine' tag in Avocado tests.
> If the requested machine is not available in the QEMU binary,
> the tests will be cancelled (skipped):
> 
>   $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>    ...
>    (04/11) CascadelakeArchCapabilities.test_4_1: CANCEL: Test expects machine 'pc-i440fx-4.1' which is missing from QEMU binary (0.10 s)
>    (05/11) CascadelakeArchCapabilities.test_4_0: CANCEL: Test expects machine 'pc-i440fx-4.0' which is missing from QEMU binary (0.11 s)
>    ...
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 29/29] MAINTAINERS: Add Acceptance tests reviewers

[Wainer dos Santos Moschetta]

On 1/30/20 6:41 AM, Thomas Huth wrote:
> On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
>> Acceptance tests can test any piece of the QEMU codebase.
>> As such, the directory holding them does not belong to a specific
>> subsystem with designated maintainers.
>>
>> Each subsystem covered by a test is welcomed to add the test path
>> to its section.
>> See for example commits 71b290e70, b11785ca2 or 5d480ddde.
>>
>> Since Eduardo, Cleber and Philippe have been reviewing the patches
>> on the list, add them as reviewers.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>> Cc: Wainer dos Santos Moschetta <wainersm@redhat.com>
>> Wainer do you want to be listed?

Yes sir, please consider my name for reviewer of all tests/acceptance.

>> ---
>>   MAINTAINERS | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index efd3f3875f..4a3cc3f137 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -2733,6 +2733,12 @@ S: Maintained
>>   F: tests/tcg/Makefile
>>   F: tests/tcg/Makefile.include
>>   
>> +Acceptance Tests
>> +R: Eduardo Habkost <ehabkost@redhat.com>
>> +R: Cleber Rosa <crosa@redhat.com>
>> +R: Philippe Mathieu-Daudé <philmd@redhat.com>
>> +F: tests/acceptance/
> Many files under tests/acceptance/ are not covered by a maintainer at
> all, e.g.:
>
>   $ scripts/get_maintainer.pl -f tests/acceptance/boot_linux_console.py
>   get_maintainer.pl: No maintainers found, printing recent contributors.

I'll maintain tests/acceptance/linux_initrd.py, that I created.

Also I feel comfortable on co-maintaining 
tests/acceptance/avocado_qemu/. Cleber?

The migration.py, virtio_version.py, version.py, 
virtio_seg_max_adjust.py tests can be put in my account if no one else 
step up.


> ... so I think it would be good to have someone who feels at least a
> little bit responsible to do the maintainers work here and picks up
> patches to put them into a pull request. Any volunteers?

Cleber has pulled the Python + Acceptance patches. Maybe we just need to 
turn it official in the MAINTAINERS?

Thanks,

Wainer

>
>   Thomas

Re: [PATCH v2 03/29] python/qemu: Add binutils::binary_get_version()

[Wainer dos Santos Moschetta]

On 1/29/20 7:23 PM, Philippe Mathieu-Daudé wrote:
> Add a helper to query the version of a QEMU binary.
> We simply send the 'query-version' command over a QMP
> socket.
> Introduce the PythonQemuCoreScripts class to test our
> new helper.
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   python/qemu/binutils.py          | 38 ++++++++++++++++++++++++++++++++

I'm not sure about creating the file with that name, it reminds me to 
GNU Binutils rather than the QEMU binary. :)

Perhaps it could be named qemu_bin.py?

Another suggestion is to encapsulate the methods you propose in this 
series in an object. For example:

class QEMUBin:
     def __init__(self, bin_path):
         # Check bin exists.
         self.bin_path = bin_path

     def get_version(self):
         # binutils.binary_get_version() goes here.
         pass

     def get_arch(self):
         # binutils.binary_get_arch() goes here.
         pass

     def list_accel(self):
         # move accel.list_accel() method to here.
         pass

     def get_vm(self, args):
         # Return an QEMUMachine object...
         return QEMUMachine(self.bin_path, *args)

     def get_build_config_host(self):
         # Detect if self.bin_path is in a build directory,
         # attempt to read the host-config.mak and return
         # as hash. Or fail...
         pass


>   tests/acceptance/core_scripts.py | 31 ++++++++++++++++++++++++++
>   2 files changed, 69 insertions(+)
>   create mode 100644 python/qemu/binutils.py
>   create mode 100644 tests/acceptance/core_scripts.py
>
> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
> new file mode 100644
> index 0000000000..96b200eef4
> --- /dev/null
> +++ b/python/qemu/binutils.py
> @@ -0,0 +1,38 @@
> +"""
> +QEMU binary utility module:
> +
> +The binary utility module provides helpers to query QEMU binary for
> +build-dependent configuration options at runtime.
> +"""
> +#
> +# Copyright (c) 2020 Red Hat, Inc.
> +#
> +# Author:
> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or later.
> +# See the COPYING file in the top-level directory.
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import logging
> +
> +from .machine import QEMUMachine
> +
> +LOG = logging.getLogger(__name__)
> +
> +
> +def binary_get_version(qemu_bin):
> +    '''
> +    Get QEMU binary version
> +
> +    @param qemu_bin (str): path to the QEMU binary

It could check that qemu_bin file exists, otherwise raise an exception 
or return None.

> +    @return binary version (dictionary with major/minor/micro keys)
> +    '''
> +    with QEMUMachine(qemu_bin) as vm:
> +        vm.set_machine('none')
> +        vm.launch()
> +        res = vm.command('query-version')
> +        LOG.info(res)
> +        vm.shutdown()

Don't need this, the vm will be shutdown anyway (see QEMUMachine.__exit__())

Thanks!

- Wainer


> +        return res['qemu']
> diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
> new file mode 100644
> index 0000000000..3f253337cd
> --- /dev/null
> +++ b/tests/acceptance/core_scripts.py
> @@ -0,0 +1,31 @@
> +# Tests covering various python/qemu/ scripts
> +#
> +# Copyright (c) 2020 Red Hat, Inc.
> +#
> +# Author:
> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or later.
> +# See the COPYING file in the top-level directory.
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import sys
> +import os
> +import logging
> +
> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
> +from avocado_qemu import Test
> +from qemu.binutils import binary_get_version
> +
> +
> +class PythonQemuCoreScripts(Test):
> +
> +    def test_get_version(self):
> +        logger = logging.getLogger('core')
> +        version = binary_get_version(self.qemu_bin)
> +        logger.debug('version: {}'.format(version))
> +        # QMP 'query-version' introduced with QEMU v0.14
> +        self.assertGreaterEqual(version['major'], 0)
> +        if version['major'] == 0:
> +            self.assertGreaterEqual(version['minor'], 14)

Re: [PATCH v2 28/29] tests/acceptance/version: Default to -nodefaults

[Thomas Huth]

On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
> We don't need the default options to run this test.
> 
> This fixes errors when running a binary built with
> --without-default-devices such:
> 
>   ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/version.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tests/acceptance/version.py b/tests/acceptance/version.py
> index 67c2192c93..79b923d4fc 100644
> --- a/tests/acceptance/version.py
> +++ b/tests/acceptance/version.py
> @@ -17,6 +17,7 @@ class Version(Test):
>      :avocado: tags=quick
>      """
>      def test_qmp_human_info_version(self):
> +        self.vm.add_args('-nodefaults')
>          self.vm.launch()
>          res = self.vm.command('human-monitor-command',
>                                command_line='info version')

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 26/29] tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine

[Thomas Huth]

On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
> As we only read the serial console, we don't need to force a
> VGA display. This fixes when running a binary built with
> --without-default-devices:
> 
>   ERROR: qemu-system-alpha: standard VGA not available
> 
> We also need the '-nodefaults' argument to avoid:
> 
>   ERROR: qemu-system-alpha: Unsupported NIC model: e1000
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/boot_linux_console.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e40b84651b..7d6065828c 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -519,7 +519,7 @@ class BootLinuxConsole(Test):
>  
>          self.vm.set_console()
>          kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
> -        self.vm.add_args('-vga', 'std',
> +        self.vm.add_args('-nodefaults',
>                           '-kernel', uncompressed_kernel,
>                           '-append', kernel_command_line)
>          self.vm.launch()

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 24/29] tests/acceptance/virtio_check_params: Default to -nodefaults

[Thomas Huth]

On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
> We don't need the default options to run this test.
> 
> This fixes errors when running a binary built with
> --without-default-devices such:
> 
>   ERROR: qemu-system-s390x: Unknown device 'virtio-net-ccw' for bus 'virtual-css-bus'
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/virtio_check_params.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
> index 2c14af825d..3466aa5b32 100755
> --- a/tests/acceptance/virtio_check_params.py
> +++ b/tests/acceptance/virtio_check_params.py
> @@ -81,6 +81,7 @@ class VirtioMaxSegSettingsCheck(Test):
>          logger.debug(mt)
>          with QEMUMachine(self.qemu_bin) as vm:
>              vm.set_machine(mt["name"])
> +            vm.add_args('-nodefaults')
>              for s in VM_DEV_PARAMS[dev_type_name]:
>                  vm.add_args(s)
>              try:
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 27/29] tests/acceptance/migration: Default to -nodefaults

[Thomas Huth]

On 29/01/2020 22.23, Philippe Mathieu-Daudé wrote:
> We don't need the default options to run this test.
> 
> This fixes errors when running a binary built with
> --without-default-devices such:
> 
>   ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/migration.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
> index a44c1ae58f..d09e5d3fe7 100644
> --- a/tests/acceptance/migration.py
> +++ b/tests/acceptance/migration.py
> @@ -32,10 +32,12 @@ class Migration(Test):
>  
>  
>      def test_migration_with_tcp_localhost(self):
> -        source_vm = self.get_vm()
>          dest_uri = 'tcp:localhost:%u' % self._get_free_port()
>          dest_vm = self.get_vm('-incoming', dest_uri)
> +        dest_vm.add_args('-nodefaults')
>          dest_vm.launch()
> +        source_vm = self.get_vm()
> +        source_vm.add_args('-nodefaults')
>          source_vm.launch()
>          source_vm.qmp('migrate', uri=dest_uri)
>          wait.wait_for(
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 03/29] python/qemu: Add binutils::binary_get_version()

[Philippe Mathieu-Daudé]

On 2/3/20 8:17 PM, Wainer dos Santos Moschetta wrote:
> 
> On 1/29/20 7:23 PM, Philippe Mathieu-Daudé wrote:
>> Add a helper to query the version of a QEMU binary.
>> We simply send the 'query-version' command over a QMP
>> socket.
>> Introduce the PythonQemuCoreScripts class to test our
>> new helper.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   python/qemu/binutils.py          | 38 ++++++++++++++++++++++++++++++++
> 
> I'm not sure about creating the file with that name, it reminds me to 
> GNU Binutils rather than the QEMU binary. :)
> 
> Perhaps it could be named qemu_bin.py?

OK, I don't mind.

> 
> Another suggestion is to encapsulate the methods you propose in this 
> series in an object. For example:
> 
> class QEMUBin:
>      def __init__(self, bin_path):
>          # Check bin exists.
>          self.bin_path = bin_path
> 
>      def get_version(self):
>          # binutils.binary_get_version() goes here.
>          pass
> 
>      def get_arch(self):
>          # binutils.binary_get_arch() goes here.
>          pass
> 
>      def list_accel(self):
>          # move accel.list_accel() method to here.
>          pass
> 
>      def get_vm(self, args):
>          # Return an QEMUMachine object...
>          return QEMUMachine(self.bin_path, *args)
> 
>      def get_build_config_host(self):
>          # Detect if self.bin_path is in a build directory,
>          # attempt to read the host-config.mak and return
>          # as hash. Or fail...
>          pass

OK, I'll try.

> 
>>   tests/acceptance/core_scripts.py | 31 ++++++++++++++++++++++++++
>>   2 files changed, 69 insertions(+)
>>   create mode 100644 python/qemu/binutils.py
>>   create mode 100644 tests/acceptance/core_scripts.py
>>
>> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
>> new file mode 100644
>> index 0000000000..96b200eef4
>> --- /dev/null
>> +++ b/python/qemu/binutils.py
>> @@ -0,0 +1,38 @@
>> +"""
>> +QEMU binary utility module:
>> +
>> +The binary utility module provides helpers to query QEMU binary for
>> +build-dependent configuration options at runtime.
>> +"""
>> +#
>> +# Copyright (c) 2020 Red Hat, Inc.
>> +#
>> +# Author:
>> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or 
>> later.
>> +# See the COPYING file in the top-level directory.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import logging
>> +
>> +from .machine import QEMUMachine
>> +
>> +LOG = logging.getLogger(__name__)
>> +
>> +
>> +def binary_get_version(qemu_bin):
>> +    '''
>> +    Get QEMU binary version
>> +
>> +    @param qemu_bin (str): path to the QEMU binary
> 
> It could check that qemu_bin file exists, otherwise raise an exception 
> or return None.

OK.

> 
>> +    @return binary version (dictionary with major/minor/micro keys)
>> +    '''
>> +    with QEMUMachine(qemu_bin) as vm:
>> +        vm.set_machine('none')
>> +        vm.launch()
>> +        res = vm.command('query-version')
>> +        LOG.info(res)
>> +        vm.shutdown()
> 
> Don't need this, the vm will be shutdown anyway (see 
> QEMUMachine.__exit__())

OK.

> 
> Thanks!
> 
> - Wainer
> 
> 
>> +        return res['qemu']
>> diff --git a/tests/acceptance/core_scripts.py 
>> b/tests/acceptance/core_scripts.py
>> new file mode 100644
>> index 0000000000..3f253337cd
>> --- /dev/null
>> +++ b/tests/acceptance/core_scripts.py
>> @@ -0,0 +1,31 @@
>> +# Tests covering various python/qemu/ scripts
>> +#
>> +# Copyright (c) 2020 Red Hat, Inc.
>> +#
>> +# Author:
>> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or 
>> later.
>> +# See the COPYING file in the top-level directory.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import sys
>> +import os
>> +import logging
>> +
>> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 
>> 'python'))
>> +from avocado_qemu import Test
>> +from qemu.binutils import binary_get_version
>> +
>> +
>> +class PythonQemuCoreScripts(Test):
>> +
>> +    def test_get_version(self):
>> +        logger = logging.getLogger('core')
>> +        version = binary_get_version(self.qemu_bin)
>> +        logger.debug('version: {}'.format(version))
>> +        # QMP 'query-version' introduced with QEMU v0.14
>> +        self.assertGreaterEqual(version['major'], 0)
>> +        if version['major'] == 0:
>> +            self.assertGreaterEqual(version['minor'], 14)
> 

Re: [PATCH v2 00/29] tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86

[Philippe Mathieu-Daudé]

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
> Currently our CI is mostly restricted to Linux/X86.
> We want to have more, but until there we have to run these tests
> manually, which is why we didn't noticed earlier this test was
> failing on OSX.
> 
> This series contains the patches I used to debug why the
> virtio_seg_max_adjust had broken the OSX CI, but then I
> kept testing on other archs, thus added more patches...
> 
> Currently our CI is mostly restricted to Linux/X86.
> We want to have more, but until there we have to run these tests
> manually, which is why we didn't noticed earlier some tests are
> failing (on OSX, using --without-default-devices, ...).
> 
> Tested on Linux/FreeBSD/OSX, X86/Aarch64.
> 
> Regards,
> 
> Phil.
> 
> Supersedes: <20200122223247.30419-1-philmd@redhat.com>
> "tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86"
> 
> Denis Plotnikov (1):
>    tests: rename virtio_seg_max_adjust to virtio_check_params
> 
> Philippe Mathieu-Daudé (28):
>    hw/core: Allow setting 'virtio-blk-device.scsi' property on OSX host
>    accel/accel: Make TYPE_ACCEL abstract
>    python/qemu: Add binutils::binary_get_version()
>    tests/acceptance: Use 'version-min' tag to verify QEMU binary version
>    tests/acceptance: Restrict X86CPUModelAliases test to QEMU >= 4.1
>    python/qemu: Add binutils::binary_get_arch()
>    tests/acceptance: Use the 'arch' tag to verify QEMU binary target
>    python/qemu: Add binutils::binary_get_machines()
>    tests/acceptance: Use 'machine' tag to check if available in QEMU
>      binary
>    python/qemu: Add binutils::binary_get_qom_implementations()
>    python/qemu: Add binutils::binary_get_accels()
>    python/qemu/accel: Use binutils::binary_get_accels()
>    python/qemu: Add binutils::binary_get_devices()
>    tests/acceptance: Use 'device' tags to check availability in QEMU
>      binary
>    tests/acceptance/virtio_check_params: Only remove listed machines
>    tests/acceptance/virtio_check_params: Improve exception logging
>    tests/acceptance/virtio_check_params: List machine being tested
>    tests/acceptance/virtio_check_params: Only test Xen as superuser
>    tests/acceptance/virtio_check_params: Skip test if arch is not
>      supported
>    tests/acceptance/virtio_check_params: Kludge to skip tests on MIPS
>    tests/acceptance/virtio_check_params: Support the s390x architecture
>    tests/acceptance/virtio_check_params: Support the ppc architecture
>    tests/acceptance/virtio_check_params: Default to -nodefaults
>    tests/acceptance/virtio_check_params: Require a virtio-scsi-pci device
>    tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine
>    tests/acceptance/migration: Default to -nodefaults
>    tests/acceptance/version: Default to -nodefaults
>    MAINTAINERS: Add Acceptance tests reviewers

I have applied the following patches on my python-next tree [*]:
15/29 tests: rename virtio_seg_max_adjust to virtio_check_params
17/29 tests/acceptance/virtio_check_params: Improve exception logging
18/29 tests/acceptance/virtio_check_params: List machine being tested
24/29 tests/acceptance/virtio_check_params: Default to -nodefaults
26/29 tests/acceptance/boot_linux_console: Do not use VGA on Clipper machine
27/29 tests/acceptance/migration: Default to -nodefaults
28/29 tests/acceptance/version: Default to -nodefaults

I plan to disable the virtio_check_params.py test for now, because it 
broke the CI and the Virtuozzo folks didn't reply to this series 
intending to fix it. We will restore it once it later.

[*] https://gitlab.com/philmd/qemu/commits/python-next

Re: [PATCH v2 09/29] tests/acceptance: Use 'machine' tag to check if available in QEMU binary

[Liam Merwick]

On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
> We already use the 'machine' tag in Avocado tests.
> If the requested machine is not available in the QEMU binary,
> the tests will be cancelled (skipped):
> 
>    $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>     ...
>     (04/11) CascadelakeArchCapabilities.test_4_1: CANCEL: Test expects machine 'pc-i440fx-4.1' which is missing from QEMU binary (0.10 s)
>     (05/11) CascadelakeArchCapabilities.test_4_0: CANCEL: Test expects machine 'pc-i440fx-4.0' which is missing from QEMU binary (0.11 s)
>     ...
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   tests/acceptance/avocado_qemu/__init__.py | 13 ++++++++++---
>   1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index e7d5affe24..53ec8512d1 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -20,6 +20,7 @@ SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
>   sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
>   
>   from qemu.binutils import binary_get_arch
> +from qemu.binutils import binary_get_machines
>   from qemu.binutils import binary_get_version
>   from qemu.machine import QEMUMachine
>   
> @@ -118,9 +119,6 @@ class Test(avocado.Test):
>           self.arch = self.params.get('arch',
>                                       default=self._get_unique_tag_val('arch'))
>   
> -        self.machine = self.params.get('machine',
> -                                       default=self._get_unique_tag_val('machine'))
> -
>           # Verify qemu_bin
>           default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
>           self.qemu_bin = self.params.get('qemu_bin',
> @@ -151,6 +149,15 @@ class Test(avocado.Test):
>               if bin_arch != self.arch:
>                   self.cancel(fmt.format(self.arch, bin_arch))
>   
> +        # Verify machine
> +        self.machine = self.params.get('machine',
> +                                       default=self._get_unique_tag_val('machine'))
> +        logger.debug('machine: {}'.format(self.machine))
> +        if self.machine:
> +            fmt = "Test expects machine '{}' which is missing from QEMU binary"
> +            if self.machine not in binary_get_machines(self.qemu_bin):
> +                self.cancel(fmt.format(self.machine))
> +

Starting with this patch:

$ avocado run --filter-by-tags arch:x86_64 tests/acceptance
...
/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: CANCEL: Test 
expects machine 'pc' which is missing from QEMU binary (0.05 s)
...

$ x86_64-softmmu/qemu-system-x86_64 -machine help
Supported machines are:
microvm              microvm (i386)
pc                   Standard PC (i440FX + PIIX, 1996) (alias of 
pc-i440fx-5.0)
...

but checking via QMP, 'pc' (and 'q35') is listed as 'alias'
...
         {
             "hotpluggable-cpus": true,
             "name": "pc-i440fx-5.0",
             "numa-mem-supported": true,
             "default-cpu-type": "qemu64-x86_64-cpu",
             "is-default": true,
             "cpu-max": 255,
             "deprecated": false,
             "alias": "pc"
         },
...

Does 'alias' need to be checked by binary_get_machines() in Patch8 as 
well as 'name'?

Regards,
Liam

Re: [PATCH v2 17/29] tests/acceptance/virtio_check_params: Improve exception logging

[Eduardo Habkost]

On Wed, Jan 29, 2020 at 10:23:33PM +0100, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/acceptance/virtio_check_params.py | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
> index 4a417b8ef5..51a2dd76e8 100755
> --- a/tests/acceptance/virtio_check_params.py
> +++ b/tests/acceptance/virtio_check_params.py
> @@ -77,8 +77,12 @@ class VirtioMaxSegSettingsCheck(Test):
>              vm.set_machine(mt["name"])
>              for s in VM_DEV_PARAMS[dev_type_name]:
>                  vm.add_args(s)
> -            vm.launch()
> -            query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
> +            try:
> +                vm.launch()
> +                query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
> +            except:
> +                query_ok = False
> +                error = sys.exc_info()[0]

I would prefer to do this inside query_virtqueue(), but:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v2 17/29] tests/acceptance/virtio_check_params: Improve exception logging

[Philippe Mathieu-Daudé]

On 2/6/20 8:54 PM, Eduardo Habkost wrote:
> On Wed, Jan 29, 2020 at 10:23:33PM +0100, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/acceptance/virtio_check_params.py | 8 ++++++--
>>   1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
>> index 4a417b8ef5..51a2dd76e8 100755
>> --- a/tests/acceptance/virtio_check_params.py
>> +++ b/tests/acceptance/virtio_check_params.py
>> @@ -77,8 +77,12 @@ class VirtioMaxSegSettingsCheck(Test):
>>               vm.set_machine(mt["name"])
>>               for s in VM_DEV_PARAMS[dev_type_name]:
>>                   vm.add_args(s)
>> -            vm.launch()
>> -            query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
>> +            try:
>> +                vm.launch()
>> +                query_ok, props, error = self.query_virtqueue(vm, dev_type_name)
>> +            except:
>> +                query_ok = False
>> +                error = sys.exc_info()[0]
> 
> I would prefer to do this inside query_virtqueue(), but:

The problem is in vm.launch():

DEBUG| Output: "xencall: error: Could not obtain handle on privileged 
command interface: No such file or directory\nxen be core: xen be core: 
can't open xen interface\ncan't open xen interface\nqemu-system-x86_64: 
failed to initialize xen: Operation not permitted\n"

> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 

Thanks, I'll keep this patch queued.

Phil.

Re: [PATCH v2 27/29] tests/acceptance/migration: Default to -nodefaults

[Wainer dos Santos Moschetta]

On 1/29/20 7:23 PM, Philippe Mathieu-Daudé wrote:
> We don't need the default options to run this test.
>
> This fixes errors when running a binary built with
> --without-default-devices such:
>
>    ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci


If you look at all the other acceptance tests the majority is going to 
fail with --without-default-devices. So could we do better on this problem?

Although I would prefer a solution that cover all the tests, nothing is 
wrong with this change:

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   tests/acceptance/migration.py | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
> index a44c1ae58f..d09e5d3fe7 100644
> --- a/tests/acceptance/migration.py
> +++ b/tests/acceptance/migration.py
> @@ -32,10 +32,12 @@ class Migration(Test):
>   
>   
>       def test_migration_with_tcp_localhost(self):
> -        source_vm = self.get_vm()
>           dest_uri = 'tcp:localhost:%u' % self._get_free_port()
>           dest_vm = self.get_vm('-incoming', dest_uri)
> +        dest_vm.add_args('-nodefaults')
>           dest_vm.launch()
> +        source_vm = self.get_vm()
> +        source_vm.add_args('-nodefaults')
>           source_vm.launch()
>           source_vm.qmp('migrate', uri=dest_uri)
>           wait.wait_for(

Re: [PATCH v2 28/29] tests/acceptance/version: Default to -nodefaults

[Wainer dos Santos Moschetta]

On 1/29/20 7:23 PM, Philippe Mathieu-Daudé wrote:
> We don't need the default options to run this test.
>
> This fixes errors when running a binary built with
> --without-default-devices such:
>
>    ERROR: qemu-system-arm: Unsupported NIC model: virtio-net-pci
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   tests/acceptance/version.py | 1 +
>   1 file changed, 1 insertion(+)

My comments in [1] also applies for this change. So:

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

[1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg675104.html


>
> diff --git a/tests/acceptance/version.py b/tests/acceptance/version.py
> index 67c2192c93..79b923d4fc 100644
> --- a/tests/acceptance/version.py
> +++ b/tests/acceptance/version.py
> @@ -17,6 +17,7 @@ class Version(Test):
>       :avocado: tags=quick
>       """
>       def test_qmp_human_info_version(self):
> +        self.vm.add_args('-nodefaults')
>           self.vm.launch()
>           res = self.vm.command('human-monitor-command',
>                                 command_line='info version')

Re: [PATCH v2 10/29] python/qemu: Add binutils::binary_get_qom_implementations()

[Liam Merwick]

On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
> Since QEMU binaries can be built with various configurations,
> the list of QOM objects linked can vary.
> Add a helper to query the list of all QOM types implementing a
> particular interface.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   python/qemu/binutils.py          | 20 ++++++++++++++++++++
>   tests/acceptance/core_scripts.py | 10 ++++++++++
>   2 files changed, 30 insertions(+)
> 
> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
> index 7bb57c521b..76f256f5d2 100644
> --- a/python/qemu/binutils.py
> +++ b/python/qemu/binutils.py
> @@ -66,3 +66,23 @@ def binary_get_machines(qemu_bin):
>           LOG.info(res)
>           vm.shutdown()
>           return [m['name'] for m in res]
> +
> +def binary_get_qom_implementations(qemu_bin, type_name, include_abstract=False):
> +    '''
> +    Get list of QOM types implementing a particular interface
> +
> +    @param qemu_bin (str): path to the QEMU binary
> +    @param type_name (str): QOM interface name
> +    @param include_abstract (bool): if True, abstract interfaces are also
> +                                    returned in the list
> +    @return list of QOM types implementing the interface @type_name
> +    '''
> +    with QEMUMachine(qemu_bin) as vm:
> +        vm.set_machine('none')
> +        vm.launch()
> +        res = vm.command('qom-list-types',
> +                         implements=type_name,
> +                         abstract=include_abstract)
> +        LOG.info(res)
> +        vm.shutdown()


Based on Wainer's comment on patch3 - is this vm.shutdown() needed?

otherwise

Reviewed-by: Liam Merwick <liam.merwick@oracle.com>


> +        return [m['name'] for m in res]
> diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
> index a5b112f928..c2fe4acf1d 100644
> --- a/tests/acceptance/core_scripts.py
> +++ b/tests/acceptance/core_scripts.py
> @@ -18,6 +18,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
>   from avocado_qemu import Test
>   from qemu.binutils import binary_get_arch
>   from qemu.binutils import binary_get_machines
> +from qemu.binutils import binary_get_qom_implementations
>   from qemu.binutils import binary_get_version
>   
>   
> @@ -49,3 +50,12 @@ class PythonQemuCoreScripts(Test):
>               logger.debug('machine: {}'.format(m))
>           # The 'none' machine is always available
>           self.assertIn('none', machines)
> +
> +    def test_get_qom_implementation(self):
> +        logger = logging.getLogger('core')
> +        type_name = "accel"
> +        type_impl = binary_get_qom_implementations(self.qemu_bin,
> +                                                   type_name, True)
> +        for t in type_impl:
> +            logger.debug('type: {}'.format(t))
> +        self.assertIn(type_name, type_impl)
> 

Re: [PATCH v2 10/29] python/qemu: Add binutils::binary_get_qom_implementations()

[Philippe Mathieu-Daudé]

On 2/7/20 3:28 PM, Liam Merwick wrote:
> On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
>> Since QEMU binaries can be built with various configurations,
>> the list of QOM objects linked can vary.
>> Add a helper to query the list of all QOM types implementing a
>> particular interface.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   python/qemu/binutils.py          | 20 ++++++++++++++++++++
>>   tests/acceptance/core_scripts.py | 10 ++++++++++
>>   2 files changed, 30 insertions(+)
>>
>> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
>> index 7bb57c521b..76f256f5d2 100644
>> --- a/python/qemu/binutils.py
>> +++ b/python/qemu/binutils.py
>> @@ -66,3 +66,23 @@ def binary_get_machines(qemu_bin):
>>           LOG.info(res)
>>           vm.shutdown()
>>           return [m['name'] for m in res]
>> +
>> +def binary_get_qom_implementations(qemu_bin, type_name, 
>> include_abstract=False):
>> +    '''
>> +    Get list of QOM types implementing a particular interface
>> +
>> +    @param qemu_bin (str): path to the QEMU binary
>> +    @param type_name (str): QOM interface name
>> +    @param include_abstract (bool): if True, abstract interfaces are 
>> also
>> +                                    returned in the list
>> +    @return list of QOM types implementing the interface @type_name
>> +    '''
>> +    with QEMUMachine(qemu_bin) as vm:
>> +        vm.set_machine('none')
>> +        vm.launch()
>> +        res = vm.command('qom-list-types',
>> +                         implements=type_name,
>> +                         abstract=include_abstract)
>> +        LOG.info(res)
>> +        vm.shutdown()
> 
> 
> Based on Wainer's comment on patch3 - is this vm.shutdown() needed?

Nop.

> 
> otherwise
> 
> Reviewed-by: Liam Merwick <liam.merwick@oracle.com>

Thanks :)

> 
> 
>> +        return [m['name'] for m in res]
>> diff --git a/tests/acceptance/core_scripts.py 
>> b/tests/acceptance/core_scripts.py
>> index a5b112f928..c2fe4acf1d 100644
>> --- a/tests/acceptance/core_scripts.py
>> +++ b/tests/acceptance/core_scripts.py
>> @@ -18,6 +18,7 @@ 
>> sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 
>> 'python'))
>>   from avocado_qemu import Test
>>   from qemu.binutils import binary_get_arch
>>   from qemu.binutils import binary_get_machines
>> +from qemu.binutils import binary_get_qom_implementations
>>   from qemu.binutils import binary_get_version
>> @@ -49,3 +50,12 @@ class PythonQemuCoreScripts(Test):
>>               logger.debug('machine: {}'.format(m))
>>           # The 'none' machine is always available
>>           self.assertIn('none', machines)
>> +
>> +    def test_get_qom_implementation(self):
>> +        logger = logging.getLogger('core')
>> +        type_name = "accel"
>> +        type_impl = binary_get_qom_implementations(self.qemu_bin,
>> +                                                   type_name, True)
>> +        for t in type_impl:
>> +            logger.debug('type: {}'.format(t))
>> +        self.assertIn(type_name, type_impl)
>>
> 

Re: [PATCH v2 11/29] python/qemu: Add binutils::binary_get_accels()

[Liam Merwick]

On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
> Add a helper to query the list of accelerators built into a
> QEMU binary.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>


Reviewed-by: Liam Merwick <liam.merwick@oracle.com>


> ---
>   python/qemu/binutils.py          | 10 ++++++++++
>   tests/acceptance/core_scripts.py | 10 ++++++++++
>   2 files changed, 20 insertions(+)
> 
> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
> index 76f256f5d2..bba203bc8d 100644
> --- a/python/qemu/binutils.py
> +++ b/python/qemu/binutils.py
> @@ -86,3 +86,13 @@ def binary_get_qom_implementations(qemu_bin, type_name, include_abstract=False):
>           LOG.info(res)
>           vm.shutdown()
>           return [m['name'] for m in res]
> +
> +def binary_get_accels(qemu_bin):
> +    '''
> +    Get list of accelerators supported by a QEMU binary
> +
> +    @param qemu_bin (str): path to the QEMU binary
> +    @return list of accelerators supported by the binary
> +    '''
> +    accel_types = binary_get_qom_implementations(qemu_bin, "accel", False)
> +    return [a.strip("-accel") for a in accel_types]
> diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
> index c2fe4acf1d..7380f2f49b 100644
> --- a/tests/acceptance/core_scripts.py
> +++ b/tests/acceptance/core_scripts.py
> @@ -16,6 +16,7 @@ import logging
>   
>   sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
>   from avocado_qemu import Test
> +from qemu.binutils import binary_get_accels
>   from qemu.binutils import binary_get_arch
>   from qemu.binutils import binary_get_machines
>   from qemu.binutils import binary_get_qom_implementations
> @@ -59,3 +60,12 @@ class PythonQemuCoreScripts(Test):
>           for t in type_impl:
>               logger.debug('type: {}'.format(t))
>           self.assertIn(type_name, type_impl)
> +
> +    def test_get_accels(self):
> +        logger = logging.getLogger('core')
> +        accels = binary_get_accels(self.qemu_bin)
> +        for a in accels:
> +            logger.debug('accel: {}'.format(a))
> +        self.assertNotIn("accel", accels)
> +        # qtest is always available
> +        self.assertIn("qtest", accels)
> 

Re: [PATCH v2 12/29] python/qemu/accel: Use binutils::binary_get_accels()

[Liam Merwick]

On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
> Instead of parsing the process help output, use the
> binary_get_accels() helper which queries the list of
> accelerators over a QMP socket.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>


Reviewed-by: Liam Merwick <liam.merwick@oracle.com>


> ---
>   python/qemu/accel.py | 26 ++++----------------------
>   1 file changed, 4 insertions(+), 22 deletions(-)
> 
> diff --git a/python/qemu/accel.py b/python/qemu/accel.py
> index 0b38ddf0ab..cde51ae159 100644
> --- a/python/qemu/accel.py
> +++ b/python/qemu/accel.py
> @@ -16,7 +16,8 @@ accelerators.
>   
>   import logging
>   import os
> -import subprocess
> +
> +from .binutils import binary_get_accels
>   
>   LOG = logging.getLogger(__name__)
>   
> @@ -27,25 +28,6 @@ ADDITIONAL_ARCHES = {
>       "aarch64" : "armhf"
>   }
>   
> -def list_accel(qemu_bin):
> -    """
> -    List accelerators enabled in the QEMU binary.
> -
> -    @param qemu_bin (str): path to the QEMU binary.
> -    @raise Exception: if failed to run `qemu -accel help`
> -    @return a list of accelerator names.
> -    """
> -    if not qemu_bin:
> -        return []
> -    try:
> -        out = subprocess.check_output([qemu_bin, '-accel', 'help'],
> -                                      universal_newlines=True)
> -    except:
> -        LOG.debug("Failed to get the list of accelerators in %s", qemu_bin)
> -        raise
> -    # Skip the first line which is the header.
> -    return [acc.strip() for acc in out.splitlines()[1:]]
> -
>   def kvm_available(target_arch=None, qemu_bin=None):
>       """
>       Check if KVM is available using the following heuristic:
> @@ -64,7 +46,7 @@ def kvm_available(target_arch=None, qemu_bin=None):
>           if target_arch != host_arch:
>               if target_arch != ADDITIONAL_ARCHES.get(host_arch):
>                   return False
> -    if qemu_bin and "kvm" not in list_accel(qemu_bin):
> +    if qemu_bin and "kvm" not in binary_get_accels(qemu_bin):
>           return False
>       return True
>   
> @@ -74,4 +56,4 @@ def tcg_available(qemu_bin):
>   
>       @param qemu_bin (str): path to the QEMU binary
>       """
> -    return 'tcg' in list_accel(qemu_bin)
> +    return 'tcg' in binary_get_accels(qemu_bin)
> 

Re: [PATCH v2 13/29] python/qemu: Add binutils::binary_get_devices()

[Liam Merwick]

On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
> Since QEMU binaries can be built with various configurations,
> the list of devices linked in can vary.
> Add a helper to query the list of devices built into a
> QEMU binary.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>


Reviewed-by: Liam Merwick <liam.merwick@oracle.com>


> ---
>   python/qemu/binutils.py          | 9 +++++++++
>   tests/acceptance/core_scripts.py | 8 ++++++++
>   2 files changed, 17 insertions(+)
> 
> diff --git a/python/qemu/binutils.py b/python/qemu/binutils.py
> index bba203bc8d..9633ba8efd 100644
> --- a/python/qemu/binutils.py
> +++ b/python/qemu/binutils.py
> @@ -96,3 +96,12 @@ def binary_get_accels(qemu_bin):
>       '''
>       accel_types = binary_get_qom_implementations(qemu_bin, "accel", False)
>       return [a.strip("-accel") for a in accel_types]
> +
> +def binary_get_devices(qemu_bin):
> +    '''
> +    Get list of devices supported by a QEMU binary
> +
> +    @param qemu_bin (str): path to the QEMU binary
> +    @return list of devices supported by the binary
> +    '''
> +    return binary_get_qom_implementations(qemu_bin, "device", False)
> diff --git a/tests/acceptance/core_scripts.py b/tests/acceptance/core_scripts.py
> index 7380f2f49b..a62eb2e46b 100644
> --- a/tests/acceptance/core_scripts.py
> +++ b/tests/acceptance/core_scripts.py
> @@ -18,6 +18,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
>   from avocado_qemu import Test
>   from qemu.binutils import binary_get_accels
>   from qemu.binutils import binary_get_arch
> +from qemu.binutils import binary_get_devices
>   from qemu.binutils import binary_get_machines
>   from qemu.binutils import binary_get_qom_implementations
>   from qemu.binutils import binary_get_version
> @@ -69,3 +70,10 @@ class PythonQemuCoreScripts(Test):
>           self.assertNotIn("accel", accels)
>           # qtest is always available
>           self.assertIn("qtest", accels)
> +
> +    def test_get_devices(self):
> +        logger = logging.getLogger('core')
> +        devices = binary_get_devices(self.qemu_bin)
> +        for d in devices:
> +            logger.debug('device: {}'.format(d))
> +        self.assertIn("migration", devices)
> 

Re: [PATCH v2 09/29] tests/acceptance: Use 'machine' tag to check if available in QEMU binary

[Liam Merwick]

On 06/02/2020 18:17, Liam Merwick wrote:
> On 29/01/2020 21:23, Philippe Mathieu-Daudé wrote:
>> We already use the 'machine' tag in Avocado tests.
>> If the requested machine is not available in the QEMU binary,
>> the tests will be cancelled (skipped):
>>
>>    $ python -m avocado --show=app run 
>> tests/acceptance/x86_cpu_model_versions.py
>>     ...
>>     (04/11) CascadelakeArchCapabilities.test_4_1: CANCEL: Test expects 
>> machine 'pc-i440fx-4.1' which is missing from QEMU binary (0.10 s)
>>     (05/11) CascadelakeArchCapabilities.test_4_0: CANCEL: Test expects 
>> machine 'pc-i440fx-4.0' which is missing from QEMU binary (0.11 s)
>>     ...
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/acceptance/avocado_qemu/__init__.py | 13 ++++++++++---
>>   1 file changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/tests/acceptance/avocado_qemu/__init__.py 
>> b/tests/acceptance/avocado_qemu/__init__.py
>> index e7d5affe24..53ec8512d1 100644
>> --- a/tests/acceptance/avocado_qemu/__init__.py
>> +++ b/tests/acceptance/avocado_qemu/__init__.py
>> @@ -20,6 +20,7 @@ SRC_ROOT_DIR = 
>> os.path.join(os.path.dirname(__file__), '..', '..', '..')
>>   sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
>>   from qemu.binutils import binary_get_arch
>> +from qemu.binutils import binary_get_machines
>>   from qemu.binutils import binary_get_version
>>   from qemu.machine import QEMUMachine
>> @@ -118,9 +119,6 @@ class Test(avocado.Test):
>>           self.arch = self.params.get('arch',
>>                                       
>> default=self._get_unique_tag_val('arch'))
>> -        self.machine = self.params.get('machine',
>> -                                       
>> default=self._get_unique_tag_val('machine'))
>> -
>>           # Verify qemu_bin
>>           default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
>>           self.qemu_bin = self.params.get('qemu_bin',
>> @@ -151,6 +149,15 @@ class Test(avocado.Test):
>>               if bin_arch != self.arch:
>>                   self.cancel(fmt.format(self.arch, bin_arch))
>> +        # Verify machine
>> +        self.machine = self.params.get('machine',
>> +                                       
>> default=self._get_unique_tag_val('machine'))
>> +        logger.debug('machine: {}'.format(self.machine))
>> +        if self.machine:
>> +            fmt = "Test expects machine '{}' which is missing from 
>> QEMU binary"
>> +            if self.machine not in binary_get_machines(self.qemu_bin):
>> +                self.cancel(fmt.format(self.machine))
>> +
> 
> Starting with this patch:
> 
> $ avocado run --filter-by-tags arch:x86_64 tests/acceptance
> ...
> /boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: CANCEL: Test 
> expects machine 'pc' which is missing from QEMU binary (0.05 s)
> ...
> 
> $ x86_64-softmmu/qemu-system-x86_64 -machine help
> Supported machines are:
> microvm              microvm (i386)
> pc                   Standard PC (i440FX + PIIX, 1996) (alias of 
> pc-i440fx-5.0)
> ...
> 
> but checking via QMP, 'pc' (and 'q35') is listed as 'alias'
> ...
>          {
>              "hotpluggable-cpus": true,
>              "name": "pc-i440fx-5.0",
>              "numa-mem-supported": true,
>              "default-cpu-type": "qemu64-x86_64-cpu",
>              "is-default": true,
>              "cpu-max": 255,
>              "deprecated": false,
>              "alias": "pc"
>          },
> ...
> 
> Does 'alias' need to be checked by binary_get_machines() in Patch8 as 
> well as 'name'?
> 

When I made this change (I'm sure there is a more Pythonesque way)

--- a/python/qemu/binutils.py
+++ b/python/qemu/binutils.py
@@ -65,7 +65,12 @@ def binary_get_machines(qemu_bin):
          res = vm.command('query-machines')
          LOG.info(res)
          vm.shutdown()
-        return [m['name'] for m in res]
+        d = []
+        for m in res:
+            d += [m['name']]
+            if 'alias' in m:
+                d += [m['alias']]
+        return d



I can then use binary_get_machines() in my PVH acceptance test patches.

@@ -72,11 +74,16 @@ class BootLinuxConsole(Test):
          os.chdir(cwd)
          return os.path.normpath(os.path.join(self.workdir, path))

-    def test_x86_64_pc(self):
+    def do_test_x86_64_machine(self):
          """
-        :avocado: tags=arch:x86_64
-        :avocado: tags=machine:pc
+        Common routine to boot an x86_64 guest.
+        Caller must specify tags=arch and tags=machine
          """
+
+        if self.machine not in binary_get_machines(self.qemu_bin):
+            raise TestCancel('QEMU binary %s does not support machine 
class %s'
+                             % (self.qemu_bin, self.machine))
+

otherwise I was getting

  (1/7) 
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: 
CANCEL: Test expects machine 'pc' which is missing from QEMU binary (0.03 s)

Regards,
Liam

python/qemu: Refactor QemuBinaryInfo

[Philippe Mathieu-Daudé]

On 1/29/20 10:23 PM, Philippe Mathieu-Daudé wrote:
[...]
>   accel/accel: Make TYPE_ACCEL abstract
>   python/qemu: Add binutils::binary_get_version()
>   tests/acceptance: Use 'version-min' tag to verify QEMU binary version
>   tests/acceptance: Restrict X86CPUModelAliases test to QEMU >= 4.1
>   python/qemu: Add binutils::binary_get_arch()
>   tests/acceptance: Use the 'arch' tag to verify QEMU binary target
>   python/qemu: Add binutils::binary_get_machines()
>   tests/acceptance: Use 'machine' tag to check if available in QEMU
>     binary
>   python/qemu: Add binutils::binary_get_qom_implementations()
>   python/qemu: Add binutils::binary_get_accels()
>   python/qemu/accel: Use binutils::binary_get_accels()
>   python/qemu: Add binutils::binary_get_devices()

I just noticed various of these methods are already implemented in
QemuBinaryInfo (see scripts/device-crash-test), so I guess it'd be
better to start refactoring from there (to the recent QEMU package).

> 
>  accel/accel.c                                 |   1 +
>  hw/core/machine.c                             |   3 +-
>  MAINTAINERS                                   |   6 +
>  python/qemu/accel.py                          |  26 +----
>  python/qemu/binutils.py                       | 107 ++++++++++++++++++
[...]