[PULL 00/15] Python queue 2019-12-17

[PULL 00/15] Python queue 2019-12-17

[Cleber Rosa]

The following changes since commit 856ffa6465ad38a31603223eb057a253114ceaea:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20191216-1' into staging (2019-12-16 13:04:34 +0000)

are available in the Git repository at:

  git://github.com/clebergnu/qemu.git tags/python-next-pull-request

for you to fetch changes up to e42cb9678cee7ff7c28afe7917c1002d6d8fdc0d:

  python/qemu: Remove unneeded imports in __init__ (2019-12-16 18:39:51 -0500)

----------------------------------------------------------------

Cleber Rosa (4):
  Acceptance test x86_cpu_model_versions: use default vm
  Acceptance tests: introduce utility method for tags unique vals
  Acceptance tests: use avocado tags for machine type
  Acceptance tests: use relative location for tests

Marc-André Lureau (2):
  analyze-migration.py: fix find() type error
  analyze-migration.py: replace numpy with python 3.2

Philippe Mathieu-Daudé (2):
  tests/boot_linux_console: Fetch assets from Debian snapshot archives
  Revert "Acceptance test: cancel test if m68k kernel packages goes
    missing"

Wainer dos Santos Moschetta (7):
  python/qemu: Add set_qmp_monitor() to QEMUMachine
  tests/acceptance: Makes linux_initrd and empty_cpu_model use
    QEMUMachine
  python/qemu: Move kvm_available() to its own module
  python/qemu: accel: Add list_accel() method
  python/qemu: accel: Strengthen kvm_available() checks
  python/qemu: accel: Add tcg_available() method
  python/qemu: Remove unneeded imports in __init__

 docs/devel/testing.rst                     |  18 +++
 python/qemu/__init__.py                    |  24 ----
 python/qemu/accel.py                       |  77 ++++++++++++
 python/qemu/machine.py                     |  70 +++++++----
 scripts/analyze-migration.py               |  39 +++---
 tests/Makefile.include                     |   2 +-
 tests/acceptance/avocado_qemu/__init__.py  |  24 +++-
 tests/acceptance/boot_linux_console.py     |  27 +---
 tests/acceptance/cpu_queries.py            |   2 +-
 tests/acceptance/empty_cpu_model.py        |  12 +-
 tests/acceptance/linux_initrd.py           |  15 +--
 tests/acceptance/linux_ssh_mips_malta.py   |   5 -
 tests/acceptance/machine_m68k_nextcube.py  |  21 +---
 tests/acceptance/machine_sparc_leon3.py    |   3 +-
 tests/acceptance/ppc_prep_40p.py           |   3 -
 tests/acceptance/x86_cpu_model_versions.py | 137 ++++++++++++---------
 tests/vm/basevm.py                         |   2 +-
 17 files changed, 284 insertions(+), 197 deletions(-)
 create mode 100644 python/qemu/accel.py

-- 
2.21.0

[PULL 01/15] tests/boot_linux_console: Fetch assets from Debian snapshot archives

[Cleber Rosa]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

The kernel packaged was fetched from an unstable repository.
Use the stable snapshot archive instead.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191126223810.20180-2-philmd@redhat.com>
Tested-by: Willian Rampazzo <wrampazz@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 7e41cebd47..752f776f68 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -479,7 +479,8 @@ class BootLinuxConsole(Test):
         :avocado: tags=arch:m68k
         :avocado: tags=machine:q800
         """
-        deb_url = ('http://ftp.ports.debian.org/debian-ports/pool-m68k/main'
+        deb_url = ('https://snapshot.debian.org/archive/debian-ports'
+                   '/20191021T083923Z/pool-m68k/main'
                    '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
         deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
         try:
-- 
2.21.0

[PULL 02/15] Revert "Acceptance test: cancel test if m68k kernel packages goes missing"

[Cleber Rosa]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

Now than we use the stable snapshot archive, we can remove this check.

This reverts commit d2499aca4bac613d837e2720e7bbe3378bc91245.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191126223810.20180-3-philmd@redhat.com>
Tested-by: Willian Rampazzo <wrampazz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 752f776f68..ab4c8a5e5a 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -483,10 +483,7 @@ class BootLinuxConsole(Test):
                    '/20191021T083923Z/pool-m68k/main'
                    '/l/linux/kernel-image-5.3.0-1-m68k-di_5.3.7-1_m68k.udeb')
         deb_hash = '044954bb9be4160a3ce81f8bc1b5e856b75cccd1'
-        try:
-            deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        except OSError as exp:
-            self.cancel(exp)
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
         kernel_path = self.extract_from_deb(deb_path,
                                             '/boot/vmlinux-5.3.0-1-m68k')
 
-- 
2.21.0

[PULL 03/15] analyze-migration.py: fix find() type error

[Cleber Rosa]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Traceback (most recent call last):
  File "../scripts/analyze-migration.py", line 611, in <module>
    dump.read(desc_only = True)
  File "../scripts/analyze-migration.py", line 513, in read
    self.load_vmsd_json(file)
  File "../scripts/analyze-migration.py", line 556, in load_vmsd_json
    vmsd_json = file.read_migration_debug_json()
  File "../scripts/analyze-migration.py", line 89, in read_migration_debug_json
    nulpos = data.rfind("\0")
TypeError: argument should be integer or bytes-like object, not 'str'

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20191127101038.327080-2-marcandre.lureau@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/analyze-migration.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py
index e527eb168e..2b835d9b70 100755
--- a/scripts/analyze-migration.py
+++ b/scripts/analyze-migration.py
@@ -86,8 +86,8 @@ class MigrationFile(object):
 
         # Find the last NULL byte, then the first brace after that. This should
         # be the beginning of our JSON data.
-        nulpos = data.rfind("\0")
-        jsonpos = data.find("{", nulpos)
+        nulpos = data.rfind(b'\0')
+        jsonpos = data.find(b'{', nulpos)
 
         # Check backwards from there and see whether we guessed right
         self.file.seek(datapos + jsonpos - 5, 0)
-- 
2.21.0

[PULL 04/15] analyze-migration.py: replace numpy with python 3.2

[Cleber Rosa]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Use int.from_bytes() from python 3.2 instead.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20191127101038.327080-3-marcandre.lureau@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/analyze-migration.py | 35 +++++++++++++++++++----------------
 1 file changed, 19 insertions(+), 16 deletions(-)

diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py
index 2b835d9b70..96a31d3974 100755
--- a/scripts/analyze-migration.py
+++ b/scripts/analyze-migration.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 #  Migration Stream Analyzer
 #
@@ -17,12 +17,18 @@
 # You should have received a copy of the GNU Lesser General Public
 # License along with this library; if not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import print_function
-import numpy as np
 import json
 import os
 import argparse
 import collections
+import struct
+import sys
+
+
+MIN_PYTHON = (3, 2)
+if sys.version_info < MIN_PYTHON:
+    sys.exit("Python %s.%s or later is required.\n" % MIN_PYTHON)
+
 
 def mkdir_p(path):
     try:
@@ -30,29 +36,26 @@ def mkdir_p(path):
     except OSError:
         pass
 
+
 class MigrationFile(object):
     def __init__(self, filename):
         self.filename = filename
         self.file = open(self.filename, "rb")
 
     def read64(self):
-        return np.asscalar(np.fromfile(self.file, count=1, dtype='>i8')[0])
+        return int.from_bytes(self.file.read(8), byteorder='big', signed=True)
 
     def read32(self):
-        return np.asscalar(np.fromfile(self.file, count=1, dtype='>i4')[0])
+        return int.from_bytes(self.file.read(4), byteorder='big', signed=True)
 
     def read16(self):
-        return np.asscalar(np.fromfile(self.file, count=1, dtype='>i2')[0])
+        return int.from_bytes(self.file.read(2), byteorder='big', signed=True)
 
     def read8(self):
-        return np.asscalar(np.fromfile(self.file, count=1, dtype='>i1')[0])
+        return int.from_bytes(self.file.read(1), byteorder='big', signed=True)
 
     def readstr(self, len = None):
-        if len is None:
-            len = self.read8()
-        if len == 0:
-            return ""
-        return np.fromfile(self.file, count=1, dtype=('S%d' % len))[0]
+        return self.readvar(len).decode('utf-8')
 
     def readvar(self, size = None):
         if size is None:
@@ -275,7 +278,7 @@ class VMSDFieldGeneric(object):
         return str(self.__str__())
 
     def __str__(self):
-        return " ".join("{0:02x}".format(ord(c)) for c in self.data)
+        return " ".join("{0:02x}".format(c) for c in self.data)
 
     def getDict(self):
         return self.__str__()
@@ -307,8 +310,8 @@ class VMSDFieldInt(VMSDFieldGeneric):
 
     def read(self):
         super(VMSDFieldInt, self).read()
-        self.sdata = np.fromstring(self.data, count=1, dtype=(self.sdtype))[0]
-        self.udata = np.fromstring(self.data, count=1, dtype=(self.udtype))[0]
+        self.sdata = int.from_bytes(self.data, byteorder='big', signed=True)
+        self.udata = int.from_bytes(self.data, byteorder='big', signed=False)
         self.data = self.sdata
         return self.data
 
@@ -363,7 +366,7 @@ class VMSDFieldStruct(VMSDFieldGeneric):
             array_len = field.pop('array_len')
             field['index'] = 0
             new_fields.append(field)
-            for i in xrange(1, array_len):
+            for i in range(1, array_len):
                 c = field.copy()
                 c['index'] = i
                 new_fields.append(c)
-- 
2.21.0

[PULL 05/15] python/qemu: Add set_qmp_monitor() to QEMUMachine

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

The QEMUMachine VM has a monitor setup on which an QMP
connection is always attempted on _post_launch() (executed
by launch()). In case the QEMU process immediatly exits
then the qmp.accept() (used to establish the connection) stalls
until it reaches timeout and consequently an exception raises.

That behavior is undesirable when, for instance, it needs to
gather information from the QEMU binary ($ qemu -cpu list) or a
test which launches the VM expecting its failure.

This patch adds the set_qmp_monitor() method to QEMUMachine that
allows turn off the creation of the monitor machinery on VM launch.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191211185536.16962-2-wainersm@redhat.com>
[Cleber: trivial indentation fix]
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/machine.py | 70 +++++++++++++++++++++++++++---------------
 1 file changed, 45 insertions(+), 25 deletions(-)

diff --git a/python/qemu/machine.py b/python/qemu/machine.py
index a4631d6934..734efd8536 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine.py
@@ -104,6 +104,7 @@ class QEMUMachine(object):
         self._events = []
         self._iolog = None
         self._socket_scm_helper = socket_scm_helper
+        self._qmp_set = True   # Enable QMP monitor by default.
         self._qmp = None
         self._qemu_full_args = None
         self._test_dir = test_dir
@@ -228,15 +229,16 @@ class QEMUMachine(object):
                 self._iolog = iolog.read()
 
     def _base_args(self):
-        if isinstance(self._monitor_address, tuple):
-            moncdev = "socket,id=mon,host=%s,port=%s" % (
-                self._monitor_address[0],
-                self._monitor_address[1])
-        else:
-            moncdev = 'socket,id=mon,path=%s' % self._vm_monitor
-        args = ['-chardev', moncdev,
-                '-mon', 'chardev=mon,mode=control',
-                '-display', 'none', '-vga', 'none']
+        args = ['-display', 'none', '-vga', 'none']
+        if self._qmp_set:
+            if isinstance(self._monitor_address, tuple):
+                moncdev = "socket,id=mon,host=%s,port=%s" % (
+                    self._monitor_address[0],
+                    self._monitor_address[1])
+            else:
+                moncdev = 'socket,id=mon,path=%s' % self._vm_monitor
+            args.extend(['-chardev', moncdev, '-mon',
+                         'chardev=mon,mode=control'])
         if self._machine is not None:
             args.extend(['-machine', self._machine])
         if self._console_set:
@@ -255,20 +257,21 @@ class QEMUMachine(object):
 
     def _pre_launch(self):
         self._temp_dir = tempfile.mkdtemp(dir=self._test_dir)
-        if self._monitor_address is not None:
-            self._vm_monitor = self._monitor_address
-        else:
-            self._vm_monitor = os.path.join(self._sock_dir,
-                                            self._name + "-monitor.sock")
-            self._remove_files.append(self._vm_monitor)
         self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log")
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
-        self._qmp = qmp.QEMUMonitorProtocol(self._vm_monitor,
-                                            server=True)
+        if self._qmp_set:
+            if self._monitor_address is not None:
+                self._vm_monitor = self._monitor_address
+            else:
+                self._vm_monitor = os.path.join(self._sock_dir,
+                                                self._name + "-monitor.sock")
+                self._remove_files.append(self._vm_monitor)
+            self._qmp = qmp.QEMUMonitorProtocol(self._vm_monitor, server=True)
 
     def _post_launch(self):
-        self._qmp.accept()
+        if self._qmp:
+            self._qmp.accept()
 
     def _post_shutdown(self):
         if self._qemu_log_file is not None:
@@ -330,7 +333,8 @@ class QEMUMachine(object):
         Wait for the VM to power off
         """
         self._popen.wait()
-        self._qmp.close()
+        if self._qmp:
+            self._qmp.close()
         self._load_io_log()
         self._post_shutdown()
 
@@ -346,12 +350,13 @@ class QEMUMachine(object):
             self._console_socket = None
 
         if self.is_running():
-            try:
-                if not has_quit:
-                    self._qmp.cmd('quit')
-                self._qmp.close()
-            except:
-                self._popen.kill()
+            if self._qmp:
+                try:
+                    if not has_quit:
+                        self._qmp.cmd('quit')
+                    self._qmp.close()
+                except:
+                    self._popen.kill()
             self._popen.wait()
 
         self._load_io_log()
@@ -368,6 +373,21 @@ class QEMUMachine(object):
 
         self._launched = False
 
+    def set_qmp_monitor(self, enabled=True):
+        """
+        Set the QMP monitor.
+
+        @param enabled: if False, qmp monitor options will be removed from
+                        the base arguments of the resulting QEMU command
+                        line. Default is True.
+        @note: call this function before launch().
+        """
+        if enabled:
+            self._qmp_set = True
+        else:
+            self._qmp_set = False
+            self._qmp = None
+
     def qmp(self, cmd, conv_keys=True, **args):
         """
         Invoke a QMP command and return the response dict
-- 
2.21.0

[PULL 06/15] tests/acceptance: Makes linux_initrd and empty_cpu_model use QEMUMachine

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

On linux_initrd and empty_cpu_model tests the same effect of
calling QEMU through run() to inspect the terminated process is
achieved with a sequence of set_qmp_monitor() / launch() / wait()
commands on an QEMUMachine object. This patch changes those
tests to use QEMUMachine instead, so they follow the same pattern
to launch QEMU found on other acceptance tests.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191211185536.16962-3-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/empty_cpu_model.py | 12 ++++++------
 tests/acceptance/linux_initrd.py    | 13 +++++++------
 2 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/tests/acceptance/empty_cpu_model.py b/tests/acceptance/empty_cpu_model.py
index 3f4f663582..a1e59e45e4 100644
--- a/tests/acceptance/empty_cpu_model.py
+++ b/tests/acceptance/empty_cpu_model.py
@@ -7,13 +7,13 @@
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-import subprocess
 from avocado_qemu import Test
 
 class EmptyCPUModel(Test):
     def test(self):
-        cmd = [self.qemu_bin, '-S', '-display', 'none', '-machine', 'none', '-cpu', '']
-        r = subprocess.run(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
-        self.assertEquals(r.returncode, 1, "QEMU exit code should be 1")
-        self.assertEquals(r.stdout, b'', "QEMU stdout should be empty")
-        self.assertNotEquals(r.stderr, b'', "QEMU stderr shouldn't be empty")
+        self.vm.add_args('-S', '-display', 'none', '-machine', 'none', '-cpu', '')
+        self.vm.set_qmp_monitor(enabled=False)
+        self.vm.launch()
+        self.vm.wait()
+        self.assertEquals(self.vm.exitcode(), 1, "QEMU exit code should be 1")
+        self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index c61d9826a4..aaa4eb9698 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -10,7 +10,6 @@
 
 import logging
 import tempfile
-from avocado.utils.process import run
 
 from avocado_qemu import Test
 
@@ -41,13 +40,15 @@ class LinuxInitrd(Test):
             initrd.seek(max_size)
             initrd.write(b'\0')
             initrd.flush()
-            cmd = "%s -kernel %s -initrd %s -m 4096" % (
-                  self.qemu_bin, kernel_path, initrd.name)
-            res = run(cmd, ignore_status=True)
-            self.assertEqual(res.exit_status, 1)
+            self.vm.add_args('-kernel', kernel_path, '-initrd', initrd.name,
+                             '-m', '4096')
+            self.vm.set_qmp_monitor(enabled=False)
+            self.vm.launch()
+            self.vm.wait()
+            self.assertEqual(self.vm.exitcode(), 1)
             expected_msg = r'.*initrd is too large.*max: \d+, need %s.*' % (
                 max_size + 1)
-            self.assertRegex(res.stderr_text, expected_msg)
+            self.assertRegex(self.vm.get_log(), expected_msg)
 
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
-- 
2.21.0

[PULL 07/15] Acceptance test x86_cpu_model_versions: use default vm

[Cleber Rosa]

The default vm provided by the test, available as self.vm, serves the
same purpose of the one obtained by self.get_vm(), but saves a line
and matches the style of other tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20190924194501.9303-2-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/x86_cpu_model_versions.py | 100 ++++++++++-----------
 1 file changed, 46 insertions(+), 54 deletions(-)

diff --git a/tests/acceptance/x86_cpu_model_versions.py b/tests/acceptance/x86_cpu_model_versions.py
index 5fc9ca4bc6..6eb977954d 100644
--- a/tests/acceptance/x86_cpu_model_versions.py
+++ b/tests/acceptance/x86_cpu_model_versions.py
@@ -25,10 +25,6 @@
 import avocado_qemu
 import re
 
-def get_cpu_prop(vm, prop):
-    cpu_path = vm.command('query-cpus')[0].get('qom_path')
-    return vm.command('qom-get', path=cpu_path, property=prop)
-
 class X86CPUModelAliases(avocado_qemu.Test):
     """
     Validation of PC CPU model versions and CPU model aliases
@@ -241,78 +237,74 @@ class CascadelakeArchCapabilities(avocado_qemu.Test):
 
     :avocado: tags=arch:x86_64
     """
+    def get_cpu_prop(self, prop):
+        cpu_path = self.vm.command('query-cpus')[0].get('qom_path')
+        return self.vm.command('qom-get', path=cpu_path, property=prop)
+
     def test_4_1(self):
         # machine-type only:
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.1')
-        vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
-        vm.launch()
-        self.assertFalse(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.1')
+        self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
+        self.vm.launch()
+        self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.1 + Cascadelake-Server should not have arch-capabilities')
 
     def test_4_0(self):
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.0')
-        vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
-        vm.launch()
-        self.assertFalse(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.0')
+        self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
+        self.vm.launch()
+        self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.0 + Cascadelake-Server should not have arch-capabilities')
 
     def test_set_4_0(self):
         # command line must override machine-type if CPU model is not versioned:
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.0')
-        vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,+arch-capabilities')
-        vm.launch()
-        self.assertTrue(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.0')
+        self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,+arch-capabilities')
+        self.vm.launch()
+        self.assertTrue(self.get_cpu_prop('arch-capabilities'),
                         'pc-i440fx-4.0 + Cascadelake-Server,+arch-capabilities should have arch-capabilities')
 
     def test_unset_4_1(self):
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.1')
-        vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,-arch-capabilities')
-        vm.launch()
-        self.assertFalse(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.1')
+        self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,-arch-capabilities')
+        self.vm.launch()
+        self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.1 + Cascadelake-Server,-arch-capabilities should not have arch-capabilities')
 
     def test_v1_4_0(self):
         # versioned CPU model overrides machine-type:
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.0')
-        vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off')
-        vm.launch()
-        self.assertFalse(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.0')
+        self.vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off')
+        self.vm.launch()
+        self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.0 + Cascadelake-Server-v1 should not have arch-capabilities')
 
     def test_v2_4_0(self):
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.0')
-        vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off')
-        vm.launch()
-        self.assertTrue(get_cpu_prop(vm, 'arch-capabilities'),
-                         'pc-i440fx-4.0 + Cascadelake-Server-v2 should have arch-capabilities')
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.0')
+        self.vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off')
+        self.vm.launch()
+        self.assertTrue(self.get_cpu_prop('arch-capabilities'),
+                        'pc-i440fx-4.0 + Cascadelake-Server-v2 should have arch-capabilities')
 
     def test_v1_set_4_0(self):
         # command line must override machine-type and versioned CPU model:
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.0')
-        vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off,+arch-capabilities')
-        vm.launch()
-        self.assertTrue(get_cpu_prop(vm, 'arch-capabilities'),
-                         'pc-i440fx-4.0 + Cascadelake-Server-v1,+arch-capabilities should have arch-capabilities')
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.0')
+        self.vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off,+arch-capabilities')
+        self.vm.launch()
+        self.assertTrue(self.get_cpu_prop('arch-capabilities'),
+                        'pc-i440fx-4.0 + Cascadelake-Server-v1,+arch-capabilities should have arch-capabilities')
 
     def test_v2_unset_4_1(self):
-        vm = self.get_vm()
-        vm.add_args('-S')
-        vm.set_machine('pc-i440fx-4.1')
-        vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off,-arch-capabilities')
-        vm.launch()
-        self.assertFalse(get_cpu_prop(vm, 'arch-capabilities'),
+        self.vm.add_args('-S')
+        self.vm.set_machine('pc-i440fx-4.1')
+        self.vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off,-arch-capabilities')
+        self.vm.launch()
+        self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.1 + Cascadelake-Server-v2,-arch-capabilities should not have arch-capabilities')
-- 
2.21.0

[PULL 08/15] Acceptance tests: introduce utility method for tags unique vals

[Cleber Rosa]

Currently a test can describe the target architecture binary that it
should primarily be run with, be setting a single tag value.

The same approach is expected to be done with other QEMU aspects to be
tested, for instance, the machine type and accelerator, so let's
generalize the logic into a utility method.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20190924194501.9303-3-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 9a57c020d8..e676d9c4e7 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -100,14 +100,21 @@ def exec_command_and_wait_for_pattern(test, command,
 
 
 class Test(avocado.Test):
+    def _get_unique_tag_val(self, tag_name):
+        """
+        Gets a tag value, if unique for a key
+        """
+        vals = self.tags.get(tag_name, [])
+        if len(vals) == 1:
+            return vals.pop()
+        return None
+
     def setUp(self):
         self._vms = {}
-        arches = self.tags.get('arch', [])
-        if len(arches) == 1:
-            arch = arches.pop()
-        else:
-            arch = None
-        self.arch = self.params.get('arch', default=arch)
+
+        self.arch = self.params.get('arch',
+                                    default=self._get_unique_tag_val('arch'))
+
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
-- 
2.21.0

[PULL 09/15] Acceptance tests: use avocado tags for machine type

[Cleber Rosa]

The same way the arch tag is being used as a fallback for the arch
parameter, let's do the same for QEMU's machine and avoid some boiler
plate code.

This is now possible because, since Avocado 72.0, it's possible to use
tags with names that match the machine types on QEMU.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191104151323.9883-4-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst                     | 18 ++++++++
 tests/acceptance/avocado_qemu/__init__.py  |  5 ++
 tests/acceptance/boot_linux_console.py     | 19 +-------
 tests/acceptance/cpu_queries.py            |  2 +-
 tests/acceptance/linux_initrd.py           |  2 +-
 tests/acceptance/linux_ssh_mips_malta.py   |  5 --
 tests/acceptance/machine_m68k_nextcube.py  | 21 ++-------
 tests/acceptance/machine_sparc_leon3.py    |  3 +-
 tests/acceptance/ppc_prep_40p.py           |  3 --
 tests/acceptance/x86_cpu_model_versions.py | 53 ++++++++++++++++------
 10 files changed, 72 insertions(+), 59 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 8e981e062d..27f286858a 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -746,6 +746,17 @@ name.  If one is not given explicitly, it will either be set to
 ``None``, or, if the test is tagged with one (and only one)
 ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
 
+machine
+~~~~~~~
+
+The machine type that will be set to all QEMUMachine instances created
+by the test.
+
+The ``machine`` attribute will be set to the test parameter of the same
+name.  If one is not given explicitly, it will either be set to
+``None``, or, if the test is tagged with one (and only one)
+``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
+
 qemu_bin
 ~~~~~~~~
 
@@ -781,6 +792,13 @@ architecture of a kernel or disk image to boot a VM with.
 This parameter has a direct relation with the ``arch`` attribute.  If
 not given, it will default to None.
 
+machine
+~~~~~~~
+
+The machine type that will be set to all QEMUMachine instances created
+by the test.
+
+
 qemu_bin
 ~~~~~~~~
 
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index e676d9c4e7..6618ea67c1 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -115,6 +115,9 @@ class Test(avocado.Test):
         self.arch = self.params.get('arch',
                                     default=self._get_unique_tag_val('arch'))
 
+        self.machine = self.params.get('machine',
+                                       default=self._get_unique_tag_val('machine'))
+
         default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
@@ -136,6 +139,8 @@ class Test(avocado.Test):
             name = str(uuid.uuid4())
         if self._vms.get(name) is None:
             self._vms[name] = self._new_vm(*args)
+            if self.machine is not None:
+                self._vms[name].set_machine(self.machine)
         return self._vms[name]
 
     def tearDown(self):
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index ab4c8a5e5a..9c6aa2040a 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -62,7 +62,6 @@ class BootLinuxConsole(Test):
         kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('pc')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-kernel', kernel_path,
@@ -85,7 +84,6 @@ class BootLinuxConsole(Test):
         kernel_path = self.extract_from_deb(deb_path,
                                             '/boot/vmlinux-2.6.32-5-4kc-malta')
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-kernel', kernel_path,
@@ -118,7 +116,6 @@ class BootLinuxConsole(Test):
         kernel_path = self.extract_from_deb(deb_path,
                                             '/boot/vmlinux-2.6.32-5-5kc-malta')
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-kernel', kernel_path,
@@ -148,7 +145,6 @@ class BootLinuxConsole(Test):
         initrd_path = self.workdir + "rootfs.cpio"
         archive.gzip_uncompress(initrd_path_gz, initrd_path)
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
                                + 'console=ttyS0 console=tty '
@@ -188,7 +184,6 @@ class BootLinuxConsole(Test):
         initrd_path = self.workdir + "rootfs.cpio"
         archive.gzip_uncompress(initrd_path_gz, initrd_path)
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
                                + 'console=ttyS0 console=tty '
@@ -215,7 +210,6 @@ class BootLinuxConsole(Test):
             with open(kernel_path, 'wb') as f_out:
                 shutil.copyfileobj(f_in, f_out)
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
                                + 'mem=256m@@0x0 '
@@ -275,7 +269,6 @@ class BootLinuxConsole(Test):
         kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('virt')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'console=ttyAMA0')
@@ -297,7 +290,6 @@ class BootLinuxConsole(Test):
         kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('virt')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'console=ttyAMA0')
@@ -310,7 +302,7 @@ class BootLinuxConsole(Test):
     def test_arm_emcraft_sf2(self):
         """
         :avocado: tags=arch:arm
-        :avocado: tags=machine:emcraft_sf2
+        :avocado: tags=machine:emcraft-sf2
         :avocado: tags=endian:little
         """
         uboot_url = ('https://raw.githubusercontent.com/'
@@ -324,7 +316,6 @@ class BootLinuxConsole(Test):
         spi_hash = '85f698329d38de63aea6e884a86fbde70890a78a'
         spi_path = self.fetch_asset(spi_url, asset_hash=spi_hash)
 
-        self.vm.set_machine('emcraft-sf2')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
         self.vm.add_args('-kernel', uboot_path,
@@ -351,7 +342,6 @@ class BootLinuxConsole(Test):
         kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
         dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
 
-        self.vm.set_machine('raspi2')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                serial_kernel_cmdline[uart_id])
@@ -393,7 +383,6 @@ class BootLinuxConsole(Test):
         initrd_path = os.path.join(self.workdir, 'rootfs.cpio')
         archive.gzip_uncompress(initrd_path_gz, initrd_path)
 
-        self.vm.set_machine('smdkc210')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'earlycon=exynos4210,0x13800000 earlyprintk ' +
@@ -414,7 +403,7 @@ class BootLinuxConsole(Test):
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x
-        :avocado: tags=machine:s390_ccw_virtio
+        :avocado: tags=machine:s390-ccw-virtio
         """
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
                       '/fedora-secondary/releases/29/Everything/s390x/os/images'
@@ -422,7 +411,6 @@ class BootLinuxConsole(Test):
         kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('s390-ccw-virtio')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
         self.vm.add_args('-nodefaults',
@@ -444,7 +432,6 @@ class BootLinuxConsole(Test):
 
         uncompressed_kernel = archive.uncompress(kernel_path, self.workdir)
 
-        self.vm.set_machine('clipper')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
         self.vm.add_args('-vga', 'std',
@@ -465,7 +452,6 @@ class BootLinuxConsole(Test):
         kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('pseries')
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
         self.vm.add_args('-kernel', kernel_path,
@@ -487,7 +473,6 @@ class BootLinuxConsole(Test):
         kernel_path = self.extract_from_deb(deb_path,
                                             '/boot/vmlinux-5.3.0-1-m68k')
 
-        self.vm.set_machine('q800')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
                                'console=ttyS0 vga=off')
diff --git a/tests/acceptance/cpu_queries.py b/tests/acceptance/cpu_queries.py
index af47d2795a..293dccb89a 100644
--- a/tests/acceptance/cpu_queries.py
+++ b/tests/acceptance/cpu_queries.py
@@ -20,8 +20,8 @@ class QueryCPUModelExpansion(Test):
     def test(self):
         """
         :avocado: tags=arch:x86_64
+        :avocado: tags=machine:none
         """
-        self.vm.set_machine('none')
         self.vm.add_args('-S')
         self.vm.launch()
 
diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py
index aaa4eb9698..a3e54d3fc9 100644
--- a/tests/acceptance/linux_initrd.py
+++ b/tests/acceptance/linux_initrd.py
@@ -19,6 +19,7 @@ class LinuxInitrd(Test):
     Checks QEMU evaluates correctly the initrd file passed as -initrd option.
 
     :avocado: tags=arch:x86_64
+    :avocado: tags=machine:pc
     """
 
     timeout = 300
@@ -67,7 +68,6 @@ class LinuxInitrd(Test):
             initrd.write(b'\0')
             initrd.flush()
 
-            self.vm.set_machine('pc')
             self.vm.set_console()
             kernel_command_line = 'console=ttyS0'
             self.vm.add_args('-kernel', kernel_path,
diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py
index fc13f9e4d4..1d570deb00 100644
--- a/tests/acceptance/linux_ssh_mips_malta.py
+++ b/tests/acceptance/linux_ssh_mips_malta.py
@@ -111,7 +111,6 @@ class LinuxSSH(Test):
         image_url, image_hash = self.get_image_info(endianess)
         image_path = self.fetch_asset(image_url, asset_hash=image_hash)
 
-        self.vm.set_machine('malta')
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
                                + 'console=ttyS0 root=/dev/sda1')
@@ -215,7 +214,6 @@ class LinuxSSH(Test):
     def test_mips_malta32eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips
-        :avocado: tags=machine:malta
         :avocado: tags=endian:big
         :avocado: tags=device:pcnet32
         """
@@ -224,7 +222,6 @@ class LinuxSSH(Test):
     def test_mips_malta32el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mipsel
-        :avocado: tags=machine:malta
         :avocado: tags=endian:little
         :avocado: tags=device:pcnet32
         """
@@ -233,7 +230,6 @@ class LinuxSSH(Test):
     def test_mips_malta64eb_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64
-        :avocado: tags=machine:malta
         :avocado: tags=endian:big
         :avocado: tags=device:pcnet32
         """
@@ -242,7 +238,6 @@ class LinuxSSH(Test):
     def test_mips_malta64el_kernel3_2_0(self):
         """
         :avocado: tags=arch:mips64el
-        :avocado: tags=machine:malta
         :avocado: tags=endian:little
         :avocado: tags=device:pcnet32
         """
diff --git a/tests/acceptance/machine_m68k_nextcube.py b/tests/acceptance/machine_m68k_nextcube.py
index fcd2c58ee7..32cf571f94 100644
--- a/tests/acceptance/machine_m68k_nextcube.py
+++ b/tests/acceptance/machine_m68k_nextcube.py
@@ -43,6 +43,11 @@ def tesseract_available(expected_version):
 
 
 class NextCubeMachine(Test):
+    """
+    :avocado: tags=arch:m68k
+    :avocado: tags=machine:next-cube
+    :avocado: tags=device:framebuffer
+    """
 
     timeout = 15
 
@@ -52,7 +57,6 @@ class NextCubeMachine(Test):
         rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
         rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
 
-        self.vm.set_machine('next-cube')
         self.vm.add_args('-bios', rom_path)
         self.vm.launch()
 
@@ -66,11 +70,6 @@ class NextCubeMachine(Test):
 
     @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
     def test_bootrom_framebuffer_size(self):
-        """
-        :avocado: tags=arch:m68k
-        :avocado: tags=machine:next_cube
-        :avocado: tags=device:framebuffer
-        """
         screenshot_path = os.path.join(self.workdir, "dump.png")
         self.check_bootrom_framebuffer(screenshot_path)
 
@@ -80,11 +79,6 @@ class NextCubeMachine(Test):
 
     @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
-        """
-        :avocado: tags=arch:m68k
-        :avocado: tags=machine:next_cube
-        :avocado: tags=device:framebuffer
-        """
         screenshot_path = os.path.join(self.workdir, "dump.png")
         self.check_bootrom_framebuffer(screenshot_path)
 
@@ -101,11 +95,6 @@ class NextCubeMachine(Test):
     # that it is still alpha-level software.
     @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available')
     def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
-        """
-        :avocado: tags=arch:m68k
-        :avocado: tags=machine:next_cube
-        :avocado: tags=device:framebuffer
-        """
         screenshot_path = os.path.join(self.workdir, "dump.png")
         self.check_bootrom_framebuffer(screenshot_path)
 
diff --git a/tests/acceptance/machine_sparc_leon3.py b/tests/acceptance/machine_sparc_leon3.py
index 298f1e25e6..f77e210ccb 100644
--- a/tests/acceptance/machine_sparc_leon3.py
+++ b/tests/acceptance/machine_sparc_leon3.py
@@ -16,7 +16,7 @@ class Leon3Machine(Test):
     def test_leon3_helenos_uimage(self):
         """
         :avocado: tags=arch:sparc
-        :avocado: tags=machine:leon3
+        :avocado: tags=machine:leon3_generic
         :avocado: tags=binfmt:uimage
         """
         kernel_url = ('http://www.helenos.org/releases/'
@@ -24,7 +24,6 @@ class Leon3Machine(Test):
         kernel_hash = 'a88c9cfdb8430c66650e5290a08765f9bf049a30'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
-        self.vm.set_machine('leon3_generic')
         self.vm.set_console()
         self.vm.add_args('-kernel', kernel_path)
 
diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index 6f507fb0a6..b27572f212 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -39,7 +39,6 @@ class IbmPrep40pMachine(Test):
         drive_hash = 'dbcfc09912e71bd5f0d82c7c1ee43082fb596ceb'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
 
-        self.vm.set_machine('40p')
         self.vm.set_console()
         self.vm.add_args('-bios', bios_path,
                          '-fda', drive_path)
@@ -53,7 +52,6 @@ class IbmPrep40pMachine(Test):
         :avocado: tags=arch:ppc
         :avocado: tags=machine:40p
         """
-        self.vm.set_machine('40p')
         self.vm.set_console()
         self.vm.add_args('-m', '192') # test fw_cfg
 
@@ -73,7 +71,6 @@ class IbmPrep40pMachine(Test):
         drive_hash = 'ac6fa2707d888b36d6fa64de6e7fe48e'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash,
                                       algorithm='md5')
-        self.vm.set_machine('40p')
         self.vm.set_console()
         self.vm.add_args('-cdrom', drive_path,
                          '-boot', 'd')
diff --git a/tests/acceptance/x86_cpu_model_versions.py b/tests/acceptance/x86_cpu_model_versions.py
index 6eb977954d..90558d9a71 100644
--- a/tests/acceptance/x86_cpu_model_versions.py
+++ b/tests/acceptance/x86_cpu_model_versions.py
@@ -75,12 +75,15 @@ class X86CPUModelAliases(avocado_qemu.Test):
                          "EPYC-IBPB shouldn't be versioned")
 
     def test_4_0_alias_compatibility(self):
-        """Check if pc-*-4.0 unversioned CPU model won't be reported as aliases"""
+        """
+        Check if pc-*-4.0 unversioned CPU model won't be reported as aliases
+
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         # pc-*-4.0 won't expose non-versioned CPU models as aliases
         # We do this to help management software to keep compatibility
         # with older QEMU versions that didn't have the versioned CPU model
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.launch()
         cpus = dict((m['name'], m) for m in self.vm.command('query-cpu-definitions'))
 
@@ -105,9 +108,12 @@ class X86CPUModelAliases(avocado_qemu.Test):
             self.assertNotIn('alias-of', c, "%s shouldn't be an alias" % (name))
 
     def test_4_1_alias(self):
-        """Check if unversioned CPU model is an alias pointing to right version"""
+        """
+        Check if unversioned CPU model is an alias pointing to right version
+
+        :avocado: tags=machine:pc-i440fx-4.1
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.1')
         self.vm.launch()
 
         cpus = dict((m['name'], m) for m in self.vm.command('query-cpu-definitions'))
@@ -207,9 +213,12 @@ class X86CPUModelAliases(avocado_qemu.Test):
         self.validate_aliases(cpus)
 
     def test_none_alias(self):
-        """Check if unversioned CPU model is an alias pointing to some version"""
+        """
+        Check if unversioned CPU model is an alias pointing to some version
+
+        :avocado: tags=machine:none
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('none')
         self.vm.launch()
 
         cpus = dict((m['name'], m) for m in self.vm.command('query-cpu-definitions'))
@@ -242,68 +251,84 @@ class CascadelakeArchCapabilities(avocado_qemu.Test):
         return self.vm.command('qom-get', path=cpu_path, property=prop)
 
     def test_4_1(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.1
+        """
         # machine-type only:
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.1')
         self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
         self.vm.launch()
         self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.1 + Cascadelake-Server should not have arch-capabilities')
 
     def test_4_0(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off')
         self.vm.launch()
         self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.0 + Cascadelake-Server should not have arch-capabilities')
 
     def test_set_4_0(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         # command line must override machine-type if CPU model is not versioned:
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,+arch-capabilities')
         self.vm.launch()
         self.assertTrue(self.get_cpu_prop('arch-capabilities'),
                         'pc-i440fx-4.0 + Cascadelake-Server,+arch-capabilities should have arch-capabilities')
 
     def test_unset_4_1(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.1
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.1')
         self.vm.add_args('-cpu', 'Cascadelake-Server,x-force-features=on,check=off,enforce=off,-arch-capabilities')
         self.vm.launch()
         self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.1 + Cascadelake-Server,-arch-capabilities should not have arch-capabilities')
 
     def test_v1_4_0(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         # versioned CPU model overrides machine-type:
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off')
         self.vm.launch()
         self.assertFalse(self.get_cpu_prop('arch-capabilities'),
                          'pc-i440fx-4.0 + Cascadelake-Server-v1 should not have arch-capabilities')
 
     def test_v2_4_0(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off')
         self.vm.launch()
         self.assertTrue(self.get_cpu_prop('arch-capabilities'),
                         'pc-i440fx-4.0 + Cascadelake-Server-v2 should have arch-capabilities')
 
     def test_v1_set_4_0(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.0
+        """
         # command line must override machine-type and versioned CPU model:
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.0')
         self.vm.add_args('-cpu', 'Cascadelake-Server-v1,x-force-features=on,check=off,enforce=off,+arch-capabilities')
         self.vm.launch()
         self.assertTrue(self.get_cpu_prop('arch-capabilities'),
                         'pc-i440fx-4.0 + Cascadelake-Server-v1,+arch-capabilities should have arch-capabilities')
 
     def test_v2_unset_4_1(self):
+        """
+        :avocado: tags=machine:pc-i440fx-4.1
+        """
         self.vm.add_args('-S')
-        self.vm.set_machine('pc-i440fx-4.1')
         self.vm.add_args('-cpu', 'Cascadelake-Server-v2,x-force-features=on,check=off,enforce=off,-arch-capabilities')
         self.vm.launch()
         self.assertFalse(self.get_cpu_prop('arch-capabilities'),
-- 
2.21.0

[PULL 10/15] Acceptance tests: use relative location for tests

[Cleber Rosa]

An Avocado Test ID[1] is composed by a number of components, but it
starts with the Test Name, usually a file system location that was
given to the loader.

Because the source directory is being given as a prefix to the
"tests/acceptance" directory containing the acceptance tests, the test
names will needlessly include the directory the user is using to host
the QEMU sources (and/or build tree).

Let's remove the source dir (or a build dir) from the path given to
the test loader.  This should give more constant names, and when using
result servers and databases, it should give the same test names
across executions from different people or from different directories.

[1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191104151323.9883-5-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8566f5f119..82dab2a36a 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1185,7 +1185,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
             $(AVOCADO_TAGS) \
-            --failfast=on $(SRC_PATH)/tests/acceptance, \
+            --failfast=on tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
 # Consolidated targets
-- 
2.21.0

[PULL 11/15] python/qemu: Move kvm_available() to its own module

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

This creates the 'accel' Python module to be the home for
utilities that deal with accelerators. Also moved kvm_available()
from __init__.py to this new module.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191216191438.93418-2-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/__init__.py               | 20 +-------------------
 python/qemu/{__init__.py => accel.py} | 16 ++++++----------
 tests/vm/basevm.py                    |  2 +-
 3 files changed, 8 insertions(+), 30 deletions(-)
 copy python/qemu/{__init__.py => accel.py} (84%)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index 6c919a3d56..eff17a306e 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -12,24 +12,6 @@
 # Based on qmp.py.
 #
 
-import logging
-import os
-
 from . import qmp
 from . import machine
-
-LOG = logging.getLogger(__name__)
-
-# Mapping host architecture to any additional architectures it can
-# support which often includes its 32 bit cousin.
-ADDITIONAL_ARCHES = {
-    "x86_64" : "i386",
-    "aarch64" : "armhf"
-}
-
-def kvm_available(target_arch=None):
-    host_arch = os.uname()[4]
-    if target_arch and target_arch != host_arch:
-        if target_arch != ADDITIONAL_ARCHES.get(host_arch):
-            return False
-    return os.access("/dev/kvm", os.R_OK | os.W_OK)
+from . import accel
diff --git a/python/qemu/__init__.py b/python/qemu/accel.py
similarity index 84%
copy from python/qemu/__init__.py
copy to python/qemu/accel.py
index 6c919a3d56..cbeac10dd1 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/accel.py
@@ -1,5 +1,9 @@
-# QEMU library
-#
+"""
+QEMU accel module:
+
+This module provides utilities for discover and check the availability of
+accelerators.
+"""
 # Copyright (C) 2015-2016 Red Hat Inc.
 # Copyright (C) 2012 IBM Corp.
 #
@@ -9,17 +13,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.  See
 # the COPYING file in the top-level directory.
 #
-# Based on qmp.py.
-#
 
-import logging
 import os
 
-from . import qmp
-from . import machine
-
-LOG = logging.getLogger(__name__)
-
 # Mapping host architecture to any additional architectures it can
 # support which often includes its 32 bit cousin.
 ADDITIONAL_ARCHES = {
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 0b8c1b2657..53b9515ee2 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -21,7 +21,7 @@ import logging
 import time
 import datetime
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu import kvm_available
+from qemu.accel import kvm_available
 from qemu.machine import QEMUMachine
 import subprocess
 import hashlib
-- 
2.21.0

[PULL 12/15] python/qemu: accel: Add list_accel() method

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

Since commit cbe6d6365a48 the command `qemu -accel help` returns
the list of accelerators enabled in the QEMU binary. This adds
the list_accel() method which return that same list.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20191216191438.93418-3-wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/accel.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/python/qemu/accel.py b/python/qemu/accel.py
index cbeac10dd1..7395cee793 100644
--- a/python/qemu/accel.py
+++ b/python/qemu/accel.py
@@ -14,7 +14,11 @@ accelerators.
 # the COPYING file in the top-level directory.
 #
 
+import logging
 import os
+import subprocess
+
+LOG = logging.getLogger(__name__)
 
 # Mapping host architecture to any additional architectures it can
 # support which often includes its 32 bit cousin.
@@ -23,6 +27,25 @@ ADDITIONAL_ARCHES = {
     "aarch64" : "armhf"
 }
 
+def list_accel(qemu_bin):
+    """
+    List accelerators enabled in the QEMU binary.
+
+    @param qemu_bin (str): path to the QEMU binary.
+    @raise Exception: if failed to run `qemu -accel help`
+    @return a list of accelerator names.
+    """
+    if not qemu_bin:
+        return []
+    try:
+        out = subprocess.check_output([qemu_bin, '-accel', 'help'],
+                                      universal_newlines=True)
+    except:
+        LOG.debug("Failed to get the list of accelerators in %s", qemu_bin)
+        raise
+    # Skip the first line which is the header.
+    return [acc.strip() for acc in out.splitlines()[1:]]
+
 def kvm_available(target_arch=None):
     host_arch = os.uname()[4]
     if target_arch and target_arch != host_arch:
-- 
2.21.0

[PULL 13/15] python/qemu: accel: Strengthen kvm_available() checks

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

Currently kvm_available() checks for the presence of kvm module
and, if target and host arches don't mismatch. This patch adds
an 3rd checking: if QEMU binary was compiled with kvm
support.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-4-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/accel.py | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/python/qemu/accel.py b/python/qemu/accel.py
index 7395cee793..5fce3aa3dd 100644
--- a/python/qemu/accel.py
+++ b/python/qemu/accel.py
@@ -46,9 +46,24 @@ def list_accel(qemu_bin):
     # Skip the first line which is the header.
     return [acc.strip() for acc in out.splitlines()[1:]]
 
-def kvm_available(target_arch=None):
-    host_arch = os.uname()[4]
-    if target_arch and target_arch != host_arch:
-        if target_arch != ADDITIONAL_ARCHES.get(host_arch):
-            return False
-    return os.access("/dev/kvm", os.R_OK | os.W_OK)
+def kvm_available(target_arch=None, qemu_bin=None):
+    """
+    Check if KVM is available using the following heuristic:
+      - Kernel module is present in the host;
+      - Target and host arches don't mismatch;
+      - KVM is enabled in the QEMU binary.
+
+    @param target_arch (str): target architecture
+    @param qemu_bin (str): path to the QEMU binary
+    @return True if kvm is available, otherwise False.
+    """
+    if not os.access("/dev/kvm", os.R_OK | os.W_OK):
+        return False
+    if target_arch:
+        host_arch = os.uname()[4]
+        if target_arch != host_arch:
+            if target_arch != ADDITIONAL_ARCHES.get(host_arch):
+                return False
+    if qemu_bin and "kvm" not in list_accel(qemu_bin):
+        return False
+    return True
-- 
2.21.0

[PULL 14/15] python/qemu: accel: Add tcg_available() method

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

This adds a method to check if the tcg accelerator is enabled
in the QEMU binary.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-5-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/accel.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/python/qemu/accel.py b/python/qemu/accel.py
index 5fce3aa3dd..0b38ddf0ab 100644
--- a/python/qemu/accel.py
+++ b/python/qemu/accel.py
@@ -67,3 +67,11 @@ def kvm_available(target_arch=None, qemu_bin=None):
     if qemu_bin and "kvm" not in list_accel(qemu_bin):
         return False
     return True
+
+def tcg_available(qemu_bin):
+    """
+    Check if TCG is available.
+
+    @param qemu_bin (str): path to the QEMU binary
+    """
+    return 'tcg' in list_accel(qemu_bin)
-- 
2.21.0

[PULL 15/15] python/qemu: Remove unneeded imports in __init__

[Cleber Rosa]

From: Wainer dos Santos Moschetta <wainersm@redhat.com>

__init_.py import some sub-modules unnecessarily. So let's
clean it up.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Suggested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20191216191438.93418-6-wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/__init__.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index eff17a306e..4ca06c34a4 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -9,9 +9,3 @@
 # This work is licensed under the terms of the GNU GPL, version 2.  See
 # the COPYING file in the top-level directory.
 #
-# Based on qmp.py.
-#
-
-from . import qmp
-from . import machine
-from . import accel
-- 
2.21.0

Re: [PULL 00/15] Python queue 2019-12-17

[Peter Maydell]

On Tue, 17 Dec 2019 at 05:15, Cleber Rosa <crosa@redhat.com> wrote:
>
> The following changes since commit 856ffa6465ad38a31603223eb057a253114ceaea:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20191216-1' into staging (2019-12-16 13:04:34 +0000)
>
> are available in the Git repository at:
>
>   git://github.com/clebergnu/qemu.git tags/python-next-pull-request
>
> for you to fetch changes up to e42cb9678cee7ff7c28afe7917c1002d6d8fdc0d:
>
>   python/qemu: Remove unneeded imports in __init__ (2019-12-16 18:39:51 -0500)
>
> ----------------------------------------------------------------
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.0
for any user-visible changes.

-- PMM