[Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[Eduardo Habkost]

Rationale
---------

Today we have a few test cases in the QEMU tree that just run
QEMU with a few command-line options, run some QMP or HMP
commands, and check if QEMU didn't crash.

Some examples:
* scripts/device-crash-test
* The test case suggested by Thomas at:
  Subject: [RFC PATCH] tests: Add a device_add/del HMP test
* Some test cases in tests/device-introspect-test.c
* tests/cpu-plug-test.c
* tests/display-vga-test.c
* tests/drive_del-test.c
* tests/machine-none-test.c
* tests/test-hmp.c
* Some device test skeletons: ac97-test.c, e1000-test.c,
  eepro100-test.c, es1370-test.c, i82801b11-test.c,
  intel-hda-test.c, ioh3420-test.c, ipoctal232-test.c,
  ne2000-test.c, nvme-test.c, pcnet-test.c, spapr-phb-test.c,
  test-netfilter.c, tpci200-test.c, usb-hcd-ohci-test.c,
  usb-hcd-xhci-test.c, virtio-balloon-test.c,
  virtio-console-test.c, virtio-serial-test.c, vmxnet3-test.c.

All of those test cases require lots of boilerplate code just to
run some combinations of QEMU command-line options and monitor
commands.

In addition to that, writing and reviewing test code like the
examples above is more complex than it could be.  Some past
examples from qemu-devel:
* Subject: [RFC 5/6] device-crash-test: Basic device_add support
* Subject: [RFC 6/6] device-crash-test: Multi-device device_add test
* Subject: [RFC PATCH] tests/device-introspect: Test devices with all machines,
           not only with "none"

The Proposal
------------

This series introduces the scripts/validator.py test runner, and
a set of example test specifications in tests/validator.

Each test specification is a YAML file containing the QEMU
command-line and monitor commands to run, and can contain
variables like $MACHINE, $DEVICE, $CPU, that can be expanded
automatically by the test runner.

This way, scripts/device-crash-test can be replaced by a one-line
test specification:

    command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL -device $DEVICE'

The device_add/device_del test case written by Thomas can be specified as:

    command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL'
    monitor-commands:
    - qmp:
      - execute: 'device_add'
        arguments:
          driver: '$DEVICE'
          id: 'test-device-for-$DEVICE'
      - execute: 'device_del'
        arguments:
          id: 'test-device-for-$DEVICE'

and the test runner will take care of running test cases with
with all valid machine-types, device-types, and accelerators.

For other examples, see the last patch in this series
("Collection of validator.py test cases").  Most of the examples
replace existing C or Python test cases with simple declarative
test specifications.

I expect this to be very useful when somebody is fixing a QEMU
crash and just want to add a very simple reproducer to ensure the
crash will never be introduced again.

(I actually suggest we _require_ people to include a
tests/validator test specifications every time they fix a crash
that can be reproduced using just QMP commands).

Features
--------

Current features:
* Automatic variable expansion, as described above
* A simple method to specify defaults (e.g. to test only
  "-machine none" or only "accel=kvm:tcg" unless running in --full mode)
* A simple method to represent expected failures (it can't
  replace the device-crash-test whitelist mechanism, yet)

Caveats
-------

The test runner code still needs some clean up.  I'm not happy with the current
variable enumeration/expansion code, and plan to make it cleaner.

How to test
-----------

You can run all example test cases on /usr/bin/qemu-system-x86_64 with:

  $ ./scripts/validator.py tests/validator/* -V QEMU=/usr/bin/qemu-system-x86_64

However, the device-crash-test.yaml test specification still
takes too long to run and reports too many false positives.  So I
suggest doing this instead:

  $ ./scripts/validator.py $(ls tests/validator/* | grep -v device-crash-test) \
        -V QEMU=/usr/bin/qemu-system-x86_64

On my system, this runs ~700 test cases in ~15 seconds.

What's out of scope?
--------------------

The following are _not_ goals of this system:
* Replacing the work done by the Avocado team to support
  Avocado-based tests.
* Making the YAML test specification a fully-featured imperative
  language.  Anything that requires extra logic still needs to be
  written in C or Python.

Future Plans
------------

Short term:
* Choose a better name for the test runner
* Support a list of known-failures similar to the whitelist
  in device-crash-test.
* Replace device-crash-test completely
* "make check" rules

Long term:
* stdout and QMP command output validation.
  I plan to use this to write machine-type/guest-ABI
  compatibility test cases that will detect guest ABI breakage as
  soon as possible.
* See if the Avocado multiplexer API can be used to implement
  variable expansion.
* See if we can make some of the features (e.g. automatic
  variable expansion) available to Python test cases
* Fuzzing QMP commands and QEMU command-line arguments.
* See if code can be shared with the iotests test
  runner (especially the variable expansion logic)

Eduardo Habkost (18):
  qmp.py: Make it safe to call close() any time
  qmp.py: Fix error handling for Python 3
  qmp.py: Cleanly handle unexpectedly closed socket
  qemu.py: Make _vm_monitor a method
  qemu.py: Split _base_args()
  qemu.py: Move _load_io_log() call to _post_shutdown()
  qemu.py: Use wait() logic inside shutdown()
  qemu.py: Close _qmp inside _post_shutdown()
  qemu.py: Make monitor optional
  qemu.py: Set _launched = False on _post_shutdown
  qemu.py: Log crashes inside _post_shutdown()
  qemu.py: Only wait for process if it's still running
  qemu.py: 'force' parameter on shutdown()
  qemu.py: Don't try to quit cleanly on exceptions
  qemu.py: qmp_obj() method
  qemu.py: is_launched() method
  validator.py script
  Collection of validator.py test cases

 scripts/qemu.py                        | 112 +++--
 scripts/qmp/qmp.py                     |  16 +-
 scripts/validator.py                   | 724 +++++++++++++++++++++++++++++++++
 tests/validator/cpu.yaml               |   5 +
 tests/validator/device-add.yaml        |  17 +
 tests/validator/device-crash-test.yaml |   7 +
 tests/validator/device-help.yaml       |   3 +
 tests/validator/device-introspect.yaml |  23 ++
 tests/validator/device-list.yaml       |  10 +
 tests/validator/just-help.yaml         |   3 +
 10 files changed, 876 insertions(+), 44 deletions(-)
 create mode 100755 scripts/validator.py
 create mode 100644 tests/validator/cpu.yaml
 create mode 100644 tests/validator/device-add.yaml
 create mode 100644 tests/validator/device-crash-test.yaml
 create mode 100644 tests/validator/device-help.yaml
 create mode 100644 tests/validator/device-introspect.yaml
 create mode 100644 tests/validator/device-list.yaml
 create mode 100644 tests/validator/just-help.yaml

-- 
2.14.3

[Qemu-devel] [RFC 01/18] qmp.py: Make it safe to call close() any time

[Eduardo Habkost]

This will allow us to simplify the error handling and shutdown
logic in qemu.py.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qmp/qmp.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
index 5c8cf6a056..078ce65f3b 100644
--- a/scripts/qmp/qmp.py
+++ b/scripts/qmp/qmp.py
@@ -244,7 +244,9 @@ class QEMUMonitorProtocol(object):
 
     def close(self):
         self.__sock.close()
-        self.__sockfile.close()
+        if self.__sockfile is not None:
+            self.__sockfile.close()
+            self.__sockfile = None
 
     def settimeout(self, timeout):
         self.__sock.settimeout(timeout)
-- 
2.14.3

[Qemu-devel] [RFC 02/18] qmp.py: Fix error handling for Python 3

[Eduardo Habkost]

socket.error doesn't behave like a tuple in Python 3, but we can
use error.args on both Python 2.7 and 3.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qmp/qmp.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
index 078ce65f3b..e9cb6b2683 100644
--- a/scripts/qmp/qmp.py
+++ b/scripts/qmp/qmp.py
@@ -108,7 +108,7 @@ class QEMUMonitorProtocol(object):
         try:
             self.__json_read()
         except socket.error as err:
-            if err[0] == errno.EAGAIN:
+            if err.args[0] == errno.EAGAIN:
                 # No data available
                 pass
         self.__sock.setblocking(1)
@@ -168,9 +168,9 @@ class QEMUMonitorProtocol(object):
         try:
             self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
         except socket.error as err:
-            if err[0] == errno.EPIPE:
+            if err.args[0] == errno.EPIPE:
                 return
-            raise socket.error(err)
+            raise
         resp = self.__json_read()
         self.logger.debug("<<< %s", resp)
         return resp
-- 
2.14.3

[Qemu-devel] [RFC 03/18] qmp.py: Cleanly handle unexpectedly closed socket

[Eduardo Habkost]

QEMUMonitorProtocol.cmd() returns None if the socket was closed,
so callers must handle this case explicltly.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qmp/qmp.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
index e9cb6b2683..91b537ea04 100644
--- a/scripts/qmp/qmp.py
+++ b/scripts/qmp/qmp.py
@@ -73,6 +73,8 @@ class QEMUMonitorProtocol(object):
             raise QMPConnectError
         # Greeting seems ok, negotiate capabilities
         resp = self.cmd('qmp_capabilities')
+        if resp is None:
+            raise QMPConnectError("QMP connection unexpectedly closed")
         if "return" in resp:
             return greeting
         raise QMPCapabilitiesError
@@ -182,6 +184,8 @@ class QEMUMonitorProtocol(object):
         @param name: command name (string)
         @param args: command arguments (dict)
         @param cmd_id: command id (dict, list, string or int)
+        @return QMP response as a Python dict or None if the connection has
+                been closed
         """
         qmp_cmd = {'execute': name}
         if args:
@@ -195,6 +199,8 @@ class QEMUMonitorProtocol(object):
         Build and send a QMP command to the monitor, report errors if any
         """
         ret = self.cmd(cmd, kwds)
+        if ret is None:
+            raise QMPConnectError("QMP connection unexpectedly closed")
         if "error" in ret:
             raise Exception(ret['error']['desc'])
         return ret['return']
-- 
2.14.3

[Qemu-devel] [RFC 04/18] qemu.py: Make _vm_monitor a method

[Eduardo Habkost]

Use a function instead of a data field.  Less state to keep track
of, less chance of bugs.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 08a3e9af5a..c051c4525a 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -76,7 +76,6 @@ class QEMUMachine(object):
             name = "qemu-%d" % os.getpid()
         self._name = name
         self._monitor_address = monitor_address
-        self._vm_monitor = None
         self._qemu_log_path = None
         self._qemu_log_file = None
         self._popen = None
@@ -168,29 +167,28 @@ class QEMUMachine(object):
             with open(self._qemu_log_path, "r") as iolog:
                 self._iolog = iolog.read()
 
+    def _vm_monitor(self):
+        if self._monitor_address is not None:
+            return self._monitor_address
+        else:
+            return os.path.join(self._temp_dir, self._name + "-monitor.sock")
+
     def _base_args(self):
-        if isinstance(self._monitor_address, tuple):
-            moncdev = "socket,id=mon,host=%s,port=%s" % (
-                self._monitor_address[0],
-                self._monitor_address[1])
+        addr = self._vm_monitor()
+        if isinstance(addr, tuple):
+            moncdev = "socket,id=mon,host=%s,port=%s" % (addr[0], addr[1])
         else:
-            moncdev = 'socket,id=mon,path=%s' % self._vm_monitor
+            moncdev = 'socket,id=mon,path=%s' % (addr)
         return ['-chardev', moncdev,
                 '-mon', 'chardev=mon,mode=control',
                 '-display', 'none', '-vga', 'none']
 
     def _pre_launch(self):
         self._temp_dir = tempfile.mkdtemp(dir=self._test_dir)
-        if self._monitor_address is not None:
-            self._vm_monitor = self._monitor_address
-        else:
-            self._vm_monitor = os.path.join(self._temp_dir,
-                                            self._name + "-monitor.sock")
         self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log")
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
-        self._qmp = qmp.qmp.QEMUMonitorProtocol(self._vm_monitor,
-                                                server=True)
+        self._qmp = qmp.qmp.QEMUMonitorProtocol(self._vm_monitor(), server=True)
 
     def _post_launch(self):
         self._qmp.accept()
-- 
2.14.3

[Qemu-devel] [RFC 05/18] qemu.py: Split _base_args()

[Eduardo Habkost]

Split it into _monitor_args() and _display_args(), so we can make
monitor args optional later.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index c051c4525a..d9f85bb153 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -173,15 +173,20 @@ class QEMUMachine(object):
         else:
             return os.path.join(self._temp_dir, self._name + "-monitor.sock")
 
-    def _base_args(self):
+    def _monitor_args(self):
         addr = self._vm_monitor()
         if isinstance(addr, tuple):
             moncdev = "socket,id=mon,host=%s,port=%s" % (addr[0], addr[1])
         else:
             moncdev = 'socket,id=mon,path=%s' % (addr)
         return ['-chardev', moncdev,
-                '-mon', 'chardev=mon,mode=control',
-                '-display', 'none', '-vga', 'none']
+                '-mon', 'chardev=mon,mode=control']
+
+    def _display_args(self):
+        return ['-display', 'none', '-vga', 'none']
+
+    def _base_args(self):
+        return self._monitor_args() + self._display_args()
 
     def _pre_launch(self):
         self._temp_dir = tempfile.mkdtemp(dir=self._test_dir)
-- 
2.14.3

[Qemu-devel] [RFC 06/18] qemu.py: Move _load_io_log() call to _post_shutdown()

[Eduardo Habkost]

All callers of _post_shutdown() call _load_io_log(), so it's
easier to simply call it inside _post_shutdown().

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index d9f85bb153..84bb3da613 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -199,6 +199,7 @@ class QEMUMachine(object):
         self._qmp.accept()
 
     def _post_shutdown(self):
+        self._load_io_log()
         if self._qemu_log_file is not None:
             self._qemu_log_file.close()
             self._qemu_log_file = None
@@ -250,7 +251,6 @@ class QEMUMachine(object):
         '''Wait for the VM to power off'''
         self._popen.wait()
         self._qmp.close()
-        self._load_io_log()
         self._post_shutdown()
 
     def shutdown(self):
@@ -263,7 +263,6 @@ class QEMUMachine(object):
                 self._popen.kill()
             self._popen.wait()
 
-        self._load_io_log()
         self._post_shutdown()
 
         exitcode = self.exitcode()
-- 
2.14.3

[Qemu-devel] [RFC 07/18] qemu.py: Use wait() logic inside shutdown()

[Eduardo Habkost]

This way we will have a single method where we wait for the QEMU
process to finish.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 84bb3da613..4a71542c6b 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -258,12 +258,10 @@ class QEMUMachine(object):
         if self.is_running():
             try:
                 self._qmp.cmd('quit')
-                self._qmp.close()
             except:
                 self._popen.kill()
-            self._popen.wait()
 
-        self._post_shutdown()
+        self.wait()
 
         exitcode = self.exitcode()
         if exitcode is not None and exitcode < 0:
-- 
2.14.3

[Qemu-devel] [RFC 08/18] qemu.py: Close _qmp inside _post_shutdown()

[Eduardo Habkost]

This way all shutdown-related cleanup is kept in a single place.

While at it, set _qmp to None after closing the socket, to avoid
trying to reuse it by mistake later.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 4a71542c6b..1241e7f10b 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -199,6 +199,10 @@ class QEMUMachine(object):
         self._qmp.accept()
 
     def _post_shutdown(self):
+        if self._qmp is not None:
+            self._qmp.close()
+            self._qmp = None
+
         self._load_io_log()
         if self._qemu_log_file is not None:
             self._qemu_log_file.close()
@@ -250,7 +254,6 @@ class QEMUMachine(object):
     def wait(self):
         '''Wait for the VM to power off'''
         self._popen.wait()
-        self._qmp.close()
         self._post_shutdown()
 
     def shutdown(self):
-- 
2.14.3

[Qemu-devel] [RFC 09/18] qemu.py: Make monitor optional

[Eduardo Habkost]

This will allow us to use the QEMUMachine class on test cases
that don't use QMP at all.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 1241e7f10b..e82540a235 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -54,7 +54,7 @@ class QEMUMachine(object):
     '''
 
     def __init__(self, binary, args=None, wrapper=None, name=None,
-                 test_dir="/var/tmp", monitor_address=None,
+                 test_dir="/var/tmp", monitor_address=None, qmp=True,
                  socket_scm_helper=None):
         '''
         Initialize a QEMUMachine
@@ -65,6 +65,7 @@ class QEMUMachine(object):
         @param name: prefix for socket and log file names (default: qemu-PID)
         @param test_dir: where to create socket and log file
         @param monitor_address: address for QMP monitor
+        @param qmp: if False, disable QMP monitor
         @param socket_scm_helper: helper program, required for send_fd_scm()"
         @note: Qemu process is not started until launch() is used.
         '''
@@ -90,6 +91,7 @@ class QEMUMachine(object):
         self._test_dir = test_dir
         self._temp_dir = None
         self._launched = False
+        self._qmp_enabled = qmp
 
         # just in case logging wasn't configured by the main script:
         logging.basicConfig()
@@ -168,13 +170,18 @@ class QEMUMachine(object):
                 self._iolog = iolog.read()
 
     def _vm_monitor(self):
-        if self._monitor_address is not None:
+        if not self._qmp_enabled:
+            return None
+        elif self._monitor_address is not None:
             return self._monitor_address
         else:
             return os.path.join(self._temp_dir, self._name + "-monitor.sock")
 
     def _monitor_args(self):
         addr = self._vm_monitor()
+        if addr is None:
+            return []
+
         if isinstance(addr, tuple):
             moncdev = "socket,id=mon,host=%s,port=%s" % (addr[0], addr[1])
         else:
@@ -193,10 +200,14 @@ class QEMUMachine(object):
         self._qemu_log_path = os.path.join(self._temp_dir, self._name + ".log")
         self._qemu_log_file = open(self._qemu_log_path, 'wb')
 
-        self._qmp = qmp.qmp.QEMUMonitorProtocol(self._vm_monitor(), server=True)
+        self._qmp = \
+            qmp.qmp.QEMUMonitorProtocol(self._vm_monitor(), server=True) \
+            if self._qmp_enabled \
+            else None
 
     def _post_launch(self):
-        self._qmp.accept()
+        if self._qmp:
+            self._qmp.accept()
 
     def _post_shutdown(self):
         if self._qmp is not None:
-- 
2.14.3

[Qemu-devel] [RFC 10/18] qemu.py: Set _launched = False on _post_shutdown

[Eduardo Habkost]

This will allow a sequence like:

  vm.launch()
  vm.cmd('quit')
  vm.wait()  # triggers post-shutdown code
  vm.launch()

to work.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index e82540a235..226d2c4d48 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -225,6 +225,8 @@ class QEMUMachine(object):
             shutil.rmtree(self._temp_dir)
             self._temp_dir = None
 
+        self._launched = False
+
     def launch(self):
         """
         Launch the VM and make sure we cleanup and expose the
@@ -286,8 +288,6 @@ class QEMUMachine(object):
                 command = ''
             LOG.warn(msg, exitcode, command)
 
-        self._launched = False
-
     def qmp(self, cmd, conv_keys=True, **args):
         '''Invoke a QMP command and return the response dict'''
         qmp_args = dict()
-- 
2.14.3

[Qemu-devel] [RFC 11/18] qemu.py: Log crashes inside _post_shutdown()

[Eduardo Habkost]

This will allow us to log QEMU crashes even if the test code uses
.wait() instead of .shutdown().

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 226d2c4d48..e19e4b34d0 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -225,6 +225,15 @@ class QEMUMachine(object):
             shutil.rmtree(self._temp_dir)
             self._temp_dir = None
 
+        exitcode = self.exitcode()
+        if exitcode is not None and exitcode < 0:
+            msg = 'qemu received signal %i: %s'
+            if self._qemu_full_args:
+                command = ' '.join(self._qemu_full_args)
+            else:
+                command = ''
+            LOG.warn(msg, exitcode, command)
+
         self._launched = False
 
     def launch(self):
@@ -279,15 +288,6 @@ class QEMUMachine(object):
 
         self.wait()
 
-        exitcode = self.exitcode()
-        if exitcode is not None and exitcode < 0:
-            msg = 'qemu received signal %i: %s'
-            if self._qemu_full_args:
-                command = ' '.join(self._qemu_full_args)
-            else:
-                command = ''
-            LOG.warn(msg, exitcode, command)
-
     def qmp(self, cmd, conv_keys=True, **args):
         '''Invoke a QMP command and return the response dict'''
         qmp_args = dict()
-- 
2.14.3

[Qemu-devel] [RFC 12/18] qemu.py: Only wait for process if it's still running

[Eduardo Habkost]

TODO: explain why

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index e19e4b34d0..aaba04b3c1 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -275,7 +275,8 @@ class QEMUMachine(object):
 
     def wait(self):
         '''Wait for the VM to power off'''
-        self._popen.wait()
+        if self.is_running():
+            self._popen.wait()
         self._post_shutdown()
 
     def shutdown(self):
-- 
2.14.3

[Qemu-devel] [RFC 13/18] qemu.py: 'force' parameter on shutdown()

[Eduardo Habkost]

This will allow test code to shut down the VM without trying to
run a 'quit' QMP command.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index aaba04b3c1..045dca5d02 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -279,13 +279,24 @@ class QEMUMachine(object):
             self._popen.wait()
         self._post_shutdown()
 
-    def shutdown(self):
-        '''Terminate the VM and clean up'''
-        if self.is_running():
+    def shutdown(self, force=None):
+        '''Terminate the VM and clean up
+
+        @param force: If True, terminate QEMU process, if False always try
+                      clean shutdown, if None terminate QEMU process if
+                      clean shutdown fails.
+        '''
+        if not force and self.is_running():
             try:
                 self._qmp.cmd('quit')
             except:
-                self._popen.kill()
+                if force is None:
+                    force = True
+                else:
+                    raise
+
+        if force and self.is_running():
+            self._popen.kill()
 
         self.wait()
 
-- 
2.14.3

[Qemu-devel] [RFC 14/18] qemu.py: Don't try to quit cleanly on exceptions

[Eduardo Habkost]

Exceptions when starting a VM probably mean QEMU exited and the
monitor won't work.  It's better to simply terminate the process
instead of trying to communicate using QMP.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 045dca5d02..00b44ea0f4 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -251,7 +251,7 @@ class QEMUMachine(object):
             self._launch()
             self._launched = True
         except:
-            self.shutdown()
+            self.shutdown(force=True)
 
             LOG.debug('Error launching VM')
             if self._qemu_full_args:
-- 
2.14.3

[Qemu-devel] [RFC 15/18] qemu.py: qmp_obj() method

[Eduardo Habkost]

A wrapper for the QEMUMonitorProtocol.cmd_obj() method.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 00b44ea0f4..d25fe030bb 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -311,6 +311,9 @@ class QEMUMachine(object):
 
         return self._qmp.cmd(cmd, args=qmp_args)
 
+    def qmp_obj(self, *args, **kwargs):
+        return self._qmp.cmd_obj(*args, **kwargs)
+
     def command(self, cmd, conv_keys=True, **args):
         '''
         Invoke a QMP command.
-- 
2.14.3

[Qemu-devel] [RFC 16/18] qemu.py: is_launched() method

[Eduardo Habkost]

Useful if some code gets a QEMUMachine object in unknown state,
and wants to launch it only if necessary.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index d25fe030bb..d32b923a15 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -236,6 +236,9 @@ class QEMUMachine(object):
 
         self._launched = False
 
+    def is_launched(self):
+        return self._launched
+
     def launch(self):
         """
         Launch the VM and make sure we cleanup and expose the
-- 
2.14.3

[Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

See cover letter for a description of the new test system.

TODO: code clean up
TODO: write description.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/validator.py | 724 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 724 insertions(+)
 create mode 100755 scripts/validator.py

diff --git a/scripts/validator.py b/scripts/validator.py
new file mode 100755
index 0000000000..4312571feb
--- /dev/null
+++ b/scripts/validator.py
@@ -0,0 +1,724 @@
+#!/usr/bin/env python
+#
+#  Copyright (c) 2018 Red Hat Inc
+#
+# Author:
+#  Eduardo Habkost <ehabkost@redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""
+QEMU validator script
+=====================
+
+This script will get test YAML test case specifications or Python
+modules as input, and generate/run test cases based on them.
+
+USAGE
+-----
+
+validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3
+
+specification-file is a YAML file containing the test specification.
+
+Example::
+
+    # this test specification is equivalent to the
+    # "device/introspect/list" test case in device-introspect-test.c
+    command-line: '$QEMU -nodefaults -machine none'
+    monitor-commands:
+    - qmp:
+      - execute: qom-list-types
+        arguments:
+          implements: 'device'
+          abstract: true
+    - hmp: 'device_add help'
+
+
+VARIABLE EXPANSION
+------------------
+
+The test runner will try to run the test cases with all possible values
+for variables appearing in the test specification.
+
+Some built-in variables are automatically expanded:
+
+* `$MACHINE` - Expands to a machine-type name supported by $QEMU
+* `$ACCEL` - Expands to an accelerator name supported by $QEMU
+* `$DEVICE` - Expands to a (user-creatable) device type name supported by $QEMU
+* `$CPU` - Expands to a CPU model name supported by $QEMU
+
+Note that the $QEMU variable must be specified in th
+
+TEST SPECIFICATION FIELDS
+-------------------------
+
+command-line
+~~~~~~~~~~~~
+
+List or string, containing the QEMU command-line to be run.
+
+Default: '$QEMU'
+
+
+monitor-commands
+~~~~~~~~~~~~~~~~
+
+Mapping or list-of-mappings containing monitor commands to run.  The key on each
+item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
+mapping, or list.
+
+Default: None.
+
+TODO: not implemented yet.
+
+
+qmp
+~~~
+
+Boolean.  If true (the default), a QMP monitor is configured on the command-line
+automatically.
+
+If true, the test runner will issue a ``quit`` command automatically when
+testing is finished.  If false, the test runner will wait until QEMU exits by
+itself.
+
+Example::
+
+    # just run $QEMU -help and ensure it won't crash
+    command-line: ['$QEMU', '-help']
+    qmp: false
+
+
+TODO: whitelist
+TODO: validate output against reference output
+TODO: configure defaults for variables
+TODO: compatibility with Avocado multiplexer?
+"""
+
+import sys
+import os
+import string
+import argparse
+import pprint
+import yaml
+import logging
+import shlex
+import pipes
+import re
+import itertools
+import traceback
+import socket
+from collections import OrderedDict
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
+from qemu import QEMUMachine
+from qmp.qmp import QMPError
+
+logger = logging.getLogger('qemu.tests.validator')
+dbg = logger.debug
+
+# Python 2.7 compatibility:
+shquote = getattr(shlex, 'quote', pipes.quote)
+
+class InvalidSpecification(Exception):
+    pass
+
+def qom_type_names(vm, **kwargs):
+    """Run qom-list-types QMP command, return type names"""
+    types = vm.command('qom-list-types', **kwargs)
+    return [t['name'] for t in types]
+
+
+def info_qdm(vm):
+    """Parse 'info qdm' output"""
+    args = {'command-line': 'info qdm'}
+    devhelp = vm.command('human-monitor-command', **args)
+    for l in devhelp.split('\n'):
+        l = l.strip()
+        if l == '' or l.endswith(':'):
+            continue
+        d = {'name': re.search(r'name "([^"]+)"', l).group(1),
+             'no-user': (re.search(', no-user', l) is not None)}
+        yield d
+
+
+class QemuBinaryInfo(object):
+    """Information for a specific QEMU binary"""
+    def __init__(self, binary):
+        """Don't instantiate this directly, use get_binary_info()"""
+        self.binary = binary
+
+        args = ['-S', '-machine', 'none,accel=kvm:tcg']
+        dbg("querying info for QEMU binary: %s", binary)
+        vm = QEMUMachine(binary=binary, args=args)
+        vm.launch()
+        try:
+            self.alldevs = qom_type_names(vm, implements='device', abstract=False)
+            # there's no way to query DeviceClass::user_creatable using QMP,
+            # so use 'info qdm':
+            self.no_user_devs = [d['name'] for d in info_qdm(vm, ) if d['no-user']]
+            self.machines = [m['name'] for m in vm.command('query-machines')]
+            self.user_devs = [dev for dev in self.alldevs if dev not in self.no_user_devs]
+            self.kvm_available = vm.command('query-kvm')['enabled']
+            self.cpu_models = [c['name'] for c in vm.command('query-cpu-definitions')]
+        finally:
+            vm.shutdown()
+
+    def available_accels(self):
+        if self.kvm_available:
+            yield 'kvm'
+        yield 'tcg'
+
+BINARY_INFO = {}
+
+def get_binary_info(binary):
+    """Lookup info for QEMU binary, caching data"""
+    if binary not in BINARY_INFO:
+        BINARY_INFO[binary] = QemuBinaryInfo(binary)
+    return BINARY_INFO[binary]
+
+
+# TEST CASE GENERATION LOGIC
+#TODO: the functions below are not used yet, refactor code to use
+#      the simpler variable expansion logic
+
+def updatedict(d1, d2):
+    """Return updated dictionary, after d1.update(d2)"""
+    d = d1.copy()
+    d.update(d2)
+    return d
+
+def newdict(d, k, v):
+    """Return new dictionary with d[k] = v
+
+    >>> a = {'a':1, 'b':2}
+    >>> b = newdict(a, 'a', 100)
+    >>> sorted(a.items())
+    [('a', 1), ('b', 2)]
+    >>> sorted(b.items())
+    [('a', 100), ('b', 2)]
+    """
+    d = d.copy()
+    d[k] = v
+    return d
+
+def mapchain(fn, l):
+    """map(fn, l) and then chain the results together
+
+    >>> multiples = lambda i: [i, i*2, i*3]
+    >>> list(mapchain(multiples, [2, 3, 5]))
+    [2, 4, 6, 3, 6, 9, 5, 10, 15]
+    """
+    return itertools.chain(*map(fn, l))
+
+def call_func(v):
+    """Call function, if v is callable"""
+    if callable(v):
+        return v()
+    return v
+
+def fnitem(tc, var, fn):
+    """Replace tc[var] with fn(tc[var])"""
+    return newdict(tc, var, fn(tc[var]))
+
+def call_var_func(tc, var):
+    """Update `var` with result of its enumeration func, if necessary
+
+    >>> retlist = lambda tc: [{'a':[1,2,3], 'c':['hi']}]
+    >>> tc = {'a':retlist, 'b':[100]}
+    >>> tc = call_var_func(tc, 'a')
+    >>> list(tc) == [{'a':[1,2,3], 'b':[100], 'c':['hi']}]
+    True
+    """
+    if callable(tc[var]):
+        r = tc[var](tc)
+        assert all(var in t for t in r)
+        return (updatedict(tc, t) for t in r)
+    else:
+        return [tc]
+
+#def simple_enum(fn):
+#    """Wrapper for simple enumeration functions that return a list of values
+#
+#    >>> fn = simple_enum(lambda: [1,2,3])
+#    >>> tc = {'a':fn}
+#    >>> tc = call_var_func(tc, 'a')
+#    >>> tc['a']
+#    [1, 2, 3]
+#    """
+#    def enum_func(tc):
+#        return {}
+
+def split_var(tc, var):
+    """Split test case into multiple items
+
+    >>> tc = {'a':[1,2,3], 'b':100}
+    >>> tc = split_var(tc, 'a')
+    >>> [r['a'] for r in tc]
+    [[1], [2], [3]]
+    """
+    return (newdict(tc, var, [v]) for v in tc[var])
+
+def split_vars(tc, vars):
+    """Split test case for multiple variables
+
+    >>> list1 = [1,2]
+    >>> list2 = [10, 20]
+    >>> tc = {'a':list1, 'b':list2, 'c':[100, 200, 300]}
+    >>> tc = split_vars(tc, ['a', 'b'])
+    >>> list(tc) == [{'a':[i], 'b':[j], 'c':[100, 200, 300]} for i in [1,2] for j in [10, 20]]
+    True
+    """
+    tcs = [tc]
+    for var in vars:
+        #TODO: call_var_func()
+        tcs = mapchain(lambda tc: split_var(tc, var), tcs)
+    return tcs
+
+class VariableDefinition(object):
+    # variable names that must be set before this variable
+    deps = []
+
+    def __init__(self):
+        self.values = None
+
+    @staticmethod
+    def enumerate(values):
+        """Enumerate possible values for the variable"""
+        raise NotImplementedError
+
+class BuiltinVars(object):
+    """Namespace for builtin variables"""
+    class QEMU(VariableDefinition):
+        """QEMU binary path, must be set explicitly by the user"""
+        def enumerate(self, values):
+            raise Exception("QEMU variable not set")
+
+    class MACHINE(VariableDefinition):
+        """Machine-type name.  Don't use with $MACHINE_OPT"""
+        deps = ['QEMU']
+
+        @staticmethod
+        def enumerate(values):
+            return get_binary_info(values['QEMU']).machines
+
+    class ACCEL(VariableDefinition):
+        deps = ['QEMU']
+
+        @staticmethod
+        def enumerate(values):
+            return get_binary_info(values['QEMU']).available_accels()
+
+    class DEVICE(VariableDefinition):
+        deps = ['QEMU']
+
+        @staticmethod
+        def enumerate(values):
+            return get_binary_info(values['QEMU']).user_devs
+
+    class CPU(VariableDefinition):
+        deps = ['QEMU']
+
+        @staticmethod
+        def enumerate(values):
+            return get_binary_info(values['QEMU']).cpu_models
+
+class VariableEnumerator(object):
+    """Helper class that will enumerate possible values for variables"""
+    def __init__(self):
+        # start with built-in variables:
+        self._vars = dict((v, getattr(BuiltinVars, v)()) for v in dir(BuiltinVars) if not v.startswith('_'))
+        self._defaults = {}
+
+    def _add_var(self, var):
+        return self._vars.setdefault(var, VariableDefinition())
+
+    def set_values(self, var, values):
+        """Set default values for variable `var`
+
+        `values` can be a list of values, or a single item
+        Default values override the values returned by VariableDefinition.enumerate()
+        """
+        if not isinstance(values, list):
+            values = [values]
+        self._add_var(var).values = values
+
+    def update_values(self, valuedict):
+        dbg("update_values: %r", valuedict)
+        for k,v in valuedict.items():
+            self.set_values(k, v)
+
+    def lookup_var(self, var):
+        """Lookup variable"""
+        return self._vars.get(var)
+
+    def var_deps(self, var):
+        """Return variables need to be set before `var`"""
+        return self.lookup_var(var).deps
+
+    def order_deps(self, vars):
+        """Return full list of variables, including dependencies in the right order
+        """
+        dbg("START order_deps %r", vars)
+        result = OrderedDict()
+        vars = OrderedDict(((v,1) for v in reversed(vars)))
+        while vars:
+            dbg("queue: %r", list(vars.keys()))
+            v,_ = vars.popitem(last=True)
+            dbg("next var: %s", v)
+            deps = self.var_deps(v)
+            dbg("deps: %r", deps)
+            if all(dep in result for dep in deps):
+                dbg("var %s is ready", v)
+                result[v] = 1
+                continue
+            # dependencies not met:
+            vars[v] = 1
+            for dep in deps:
+                if dep in vars:
+                    raise Exception("Variable dependency cycle: %s" % (' -> '.join(vars.keys())))
+                vars[dep] = 1
+        return list(result.keys())
+
+    def enumerate(self, env, var, values):
+        """Enumerate possible values for variable `var`
+
+        May be called only if all requirements for `var` are set in `values`.
+        """
+        v = self.lookup_var(var)
+        assert all(dep in values for dep in v.deps)
+        dbg("var: %r", v)
+        if v.values:
+            r = v.values
+        else:
+            r = v.enumerate(values)
+        logger.debug('Values for %s: %r', var, r)
+        return r
+
+# HELPER FUNCTIONS FOR TEMPLATE STRINGS:
+
+def apply_template(templ, values):
+    """Apply variables to a template, supporting strings and lists
+
+    >>> apply_template('$QEMU -machine X', {'QEMU':'qemu-system-x86_64'})
+    'qemu-system-x86_64 -machine X'
+    >>> apply_template({"$TEST": ["$FOO", "is $BAR"]}, \
+                       {'FOO':'XX', 'BAR':'YY', 'TEST':'TT'})
+    {'$TEST': ['XX', 'is YY']}
+    """
+    if isinstance(templ, str):
+        return string.Template(templ).substitute(values)
+    elif isinstance(templ, list):
+        return [apply_template(s, values) for s in templ]
+    elif isinstance(templ, dict):
+        return dict( (k, apply_template(v, values)) for (k, v) in templ.items())
+    else:
+        return templ
+
+def vars_for_template(templ):
+    """Return list of variables used by s when used as template string
+
+    >>> vars_for_template('abcde fgh')
+    []
+    >>> vars_for_template('$A is ${A}, not ${B} or $C')
+    ['A', 'B', 'C']
+    >>> vars_for_template(['$QEMU', '-machine' , '$MACHINE$MACHINE_OPT'])
+    ['QEMU', 'MACHINE', 'MACHINE_OPT']
+    """
+    usedKeys = OrderedDict()
+    class LoggingDict(object):
+        def __getitem__(self, k):
+            usedKeys[k] = 1
+            return 'X'
+    apply_template(templ, LoggingDict())
+    return list(usedKeys.keys())
+
+class TestSpecification(object):
+    def __init__(self, data):
+        self._data = data
+        self.normalize()
+
+    def normalize(self):
+        """Normalize test specification data
+
+        * ensure 'command-line' is a list of arguments
+        * 'monitor-commands' will be an array
+        """
+        # if command-line is omitted, just run QEMU with no arguments:
+        self._data.setdefault('command-line', ['$QEMU'])
+        self._data.setdefault('monitor-commands', [])
+
+        # 'monitor-commands' must be a list
+        if not isinstance(self.get('monitor-commands'), list):
+            self._data['monitor-commands'] = [self.get('monitor-commands')]
+
+    @classmethod
+    def load_file(cls, file):
+        data = yaml.load(open(file))
+        return cls(data)
+
+    def get(self, key, default=None):
+        return self._data.get(key, default)
+
+    def _gen_subtests(self, testcases, env, vars):
+        """Call _gen_test_cases() for each test case in `testcases`"""
+        #import pdb; pdb.set_trace()
+        for tc in testcases:
+            for st in self._gen_test_cases(env, vars, tc):
+                yield st
+
+    def _gen_var_testcases(self, env, var, values):
+        """Generate one test case for each possible value for `var`"""
+        for value in self.var_enum.enumerate(env, var, values):
+            newvalues = values.copy()
+            newvalues[var] = value
+            yield newvalues
+
+    def _gen_test_cases(self, env, vars, values):
+        """Generate list of test cases
+
+        :param vars: List of variable names that are not set yet
+        :param values: values of variables that are already set
+        """
+        if not vars:
+            # No unset variables -> only 1 test case
+            return [values.copy()]
+
+        # pick next unset variable, enumerate values, set it:
+        var = vars[0]
+        cases = self._gen_var_testcases(env, var, values)
+        cases = self._gen_subtests(cases, env, vars[1:])
+        return cases
+
+    def gen_test_cases(self, env):
+        """Generate all test cases for this test specification"""
+        # we generate combinations for the command-line and monitor commands
+        # in separate steps, so test cases using the same QEMU command-line
+        # are grouped together
+        vars = vars_for_template(self.get('command-line')) + vars_for_template(self.get('monitor-commands'))
+
+        self.var_enum = VariableEnumerator()
+        if not env.args.full:
+            self.var_enum.update_values(self.get('defaults', {}))
+        self.var_enum.update_values(env.var_values)
+
+        # put dependencies in right order:
+        vars = self.var_enum.order_deps(vars)
+        cases = self._gen_test_cases(env, vars, {})
+        return (TestCase(self, c) for c in cases)
+
+class TestCase(object):
+    def __init__(self, spec, values):
+        self.spec = spec
+        self.values = values
+
+    def __str__(self):
+        return ' '.join('%s=%s' % (k, shquote(v)) for k,v in self.values.items())
+
+    def is_expected_entry(self, expected_entry):
+        """Check if `expected_entry` matches the testcase/results"""
+        expected_vars = expected_entry.copy()
+        for var,value in expected_vars.items():
+            if self.values.get(var) != value:
+                return False
+        return True
+
+    def is_expected_failure(self):
+        for e in self.getField('expected-failures', []):
+            if self.is_expected_entry(e):
+                return True
+
+    def getField(self, var, default=None):
+        """Get value of test spec field, expanding variables"""
+        return apply_template(self.spec.get(var, default), self.values)
+
+    def qmp_cmd(self, vm, cmd):
+        if isinstance(cmd, list):
+            for c in cmd:
+                self.qmp_cmd(vm, c)
+        elif isinstance(cmd, dict):
+            return vm.qmp_obj(cmd)
+        else:
+            raise InvalidSpecification("QMP command must be dict: %r" % (cmd))
+
+    def hmp_cmd(self, vm, cmd):
+        return vm.command('human-monitor-command', command_line=cmd)
+
+    def monitor_cmd(self, vm, cmd):
+        dbg("monitor cmd: %r", cmd)
+        if isinstance(cmd, dict):
+            for k,v in cmd.items():
+                if k == 'qmp':
+                    self.qmp_cmd(vm, v)
+                elif k == 'hmp':
+                    self.hmp_cmd(vm, v)
+                else:
+                    raise InvalidSpecification("Invalid monitor command: %r: %r" % (k, v))
+
+    def run(self, env):
+        """Check one specific test case
+
+        Returns a dictionary containing failure information on error,
+        or None on success
+        """
+        result = {'success': True }
+        result['is-expected-failure'] = self.is_expected_failure()
+
+        cmdline = self.getField('command-line')
+        if not isinstance(cmdline, list):
+            cmdline = shlex.split(cmdline)
+
+        qmp = self.getField('qmp', True)
+        #TODO: use context manager to enter/exit borrowed VM from env
+        vm = env.get_vm(cmdline, qmp)
+        try:
+            if not vm.is_launched():
+                vm.launch()
+            #TODO: generate/enumerate variables inside monitor commands too
+            for cmd in self.getField('monitor-commands', []):
+                self.monitor_cmd(vm, cmd)
+            if not qmp:
+                vm.wait()
+                env.drop_vm()
+        except KeyboardInterrupt:
+            raise
+        except QMPError as err:
+            result['exception'] = repr(err)
+            result['success'] = False
+        except socket.error as err:
+            result['exception'] = repr(err)
+            result['success'] = False
+
+        dbg('vm is %r', vm)
+        ec = vm.exitcode()
+        dbg("exit code: %r", ec)
+        if ec is not None and ec != 0:
+            result['success'] = False
+        result['exitcode'] = ec
+        result['log'] = vm.get_log()
+
+        #TODO: use context manager to enter/exit borrowed VM from env
+        if not result['success']:
+            env.drop_vm()
+
+        return result
+
+
+class TestEnv(object):
+    def __init__(self, args):
+        self.args = args
+        self._last_vm_args = None
+        self._last_vm = None
+
+    def qemu_binaries(self):
+        return self.args.qemu_binaries
+
+    def drop_vm(self):
+        """Drop existing VM object"""
+        if self._last_vm:
+            #TODO: record failures here
+            self._last_vm.shutdown()
+            self._last_vm = None
+            self._last_vm_args = None
+
+    def get_vm(self, cmdline, qmp):
+        """Get VM object for test case"""
+        if self._last_vm_args == (cmdline, qmp) and self._last_vm.is_running():
+            dbg("Reusing VM object for cmdline %r", cmdline)
+            return self._last_vm
+
+        dbg("Starting new VM for cmdline %r", cmdline)
+        #FIXME: need to catch exitcode/segfaults here somehow  :(
+        self.drop_vm()
+        vm = QEMUMachine(binary=cmdline[0], args=cmdline[1:], qmp=qmp)
+        self._last_vm = vm
+        self._last_vm_args = (cmdline, qmp)
+        return vm
+
+def main():
+    parser = argparse.ArgumentParser(description="Generic QEMU validator")
+    parser.set_defaults(loglevel=logging.INFO)
+    parser.add_argument('-V', metavar='VAR=VALUE', nargs='*',
+                        help="Force variabie VAR to VALUE",
+                        action='append', dest='vars', default=[])
+    parser.add_argument('-d', '--debug',action='store_const',
+                        dest='loglevel', const=logging.DEBUG,
+                        help='debug output')
+    parser.add_argument('-v', '--verbose',action='store_const',
+                        dest='loglevel', const=logging.INFO,
+                        help='verbose output')
+    parser.add_argument('-q', '--quiet',action='store_const',
+                        dest='loglevel', const=logging.WARN,
+                        help='non-verbose output')
+    parser.add_argument("--dry-run", action="store_true",
+                        help="Don't run test cases")
+    parser.add_argument("--full", action="store_true",
+                        help="Run all test case combinations, not just the default for the test specification")
+    parser.add_argument("testfiles", nargs="+", metavar="FILE",
+                        help="Load test case specification from FILE")
+    args = parser.parse_args()
+
+    env = TestEnv(args)
+
+    vars = {}
+    if args.vars:
+        for varval in itertools.chain(*args.vars):
+            var,val = varval.split('=', 1)
+            vars.setdefault(var, []).append(val)
+    env.var_values = vars
+
+    logging.basicConfig(stream=sys.stdout, level=args.loglevel, format='%(levelname)s: %(message)s')
+    resultdict = {}
+    try:
+        for testfile in args.testfiles:
+            specname = os.path.basename(testfile)
+            #TODO: support test specifications pointing to Python modules
+            spec = TestSpecification.load_file(testfile)
+            logger.debug("Test specification:")
+            logger.debug(pprint.pformat(spec._data))
+            logger.debug('---')
+            for tc in spec.gen_test_cases(env):
+                if tc.is_expected_failure():
+                    logger.info("%s: Skipped: %s", specname, str(tc))
+                    continue
+                logger.info("%s: Running: %s", specname, str(tc))
+                if not args.dry_run:
+                    r = tc.run(env)
+                    logger.debug("Result:")
+                    logger.debug(pprint.pformat(r))
+                    if not r['success']:
+                        logger.error("%s: failed: %s", specname, tc)
+                    resultdict.setdefault(r['success'], []).append( (tc, r) )
+    except KeyboardInterrupt:
+        # Print partial test result summary on interrupt
+        logger.info("Interrupted. Partial test summary follows")
+        pass
+
+    env.drop_vm()
+
+    if not args.dry_run:
+        logger.info('%d successes', len(resultdict.get(True, [])))
+        failures = resultdict.get(False, [])
+        if failures:
+            logger.error('%d failures', len(failures))
+            for tc,r in failures:
+                logger.error("Failed: %s", tc)
+                logger.error("Result:")
+                pprint.pprint(r)
+                dbg("Result: %r", r)
+
+if __name__ == '__main__':
+    sys.exit(main())
-- 
2.14.3

[Qemu-devel] [RFC 18/18] Collection of validator.py test cases

[Eduardo Habkost]

These test case specifications demonstrate what kind of tests can
be specified using validator.py.

Most of them can replace existing unit tests written in C, and
device-crash-test.yaml can replace most of the functionality of
scripts/device-crash-test.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/validator/cpu.yaml               |  5 +++++
 tests/validator/device-add.yaml        | 17 +++++++++++++++++
 tests/validator/device-crash-test.yaml |  7 +++++++
 tests/validator/device-help.yaml       |  3 +++
 tests/validator/device-introspect.yaml | 23 +++++++++++++++++++++++
 tests/validator/device-list.yaml       | 10 ++++++++++
 tests/validator/just-help.yaml         |  3 +++
 7 files changed, 68 insertions(+)
 create mode 100644 tests/validator/cpu.yaml
 create mode 100644 tests/validator/device-add.yaml
 create mode 100644 tests/validator/device-crash-test.yaml
 create mode 100644 tests/validator/device-help.yaml
 create mode 100644 tests/validator/device-introspect.yaml
 create mode 100644 tests/validator/device-list.yaml
 create mode 100644 tests/validator/just-help.yaml

diff --git a/tests/validator/cpu.yaml b/tests/validator/cpu.yaml
new file mode 100644
index 0000000000..48032995ff
--- /dev/null
+++ b/tests/validator/cpu.yaml
@@ -0,0 +1,5 @@
+# Simply check if QEMU will run without errors using a CPU model:
+command-line: '$QEMU -S -cpu $CPU'
+expected-failures:
+    # "host" doesn't always work
+    - CPU: 'host'
diff --git a/tests/validator/device-add.yaml b/tests/validator/device-add.yaml
new file mode 100644
index 0000000000..b4c0daf70c
--- /dev/null
+++ b/tests/validator/device-add.yaml
@@ -0,0 +1,17 @@
+# This test specification will try device_add with all device types
+command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL'
+monitor-commands:
+- qmp:
+  - execute: 'device_add'
+    arguments:
+      driver: '$DEVICE'
+      id: 'test-device-for-$DEVICE'
+  - execute: 'device_del'
+    arguments:
+      id: 'test-device-for-$DEVICE'
+defaults:
+  MACHINE: 'none'
+  ACCEL: 'kvm:tcg'
+expected-failures:
+  - MACHINE: 'xenpv'
+  - MACHINE: 'xenfv'
diff --git a/tests/validator/device-crash-test.yaml b/tests/validator/device-crash-test.yaml
new file mode 100644
index 0000000000..edaab0ee8f
--- /dev/null
+++ b/tests/validator/device-crash-test.yaml
@@ -0,0 +1,7 @@
+# this is a very simple test case generator that will run QEMU
+# using all combinations of -machine and -device.
+# This replaces scripts/device-crash-test.
+# TODO: use a $MACHINE_OPT variable to make -machine optional
+command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL -device $DEVICE'
+
+#TODO: represent the whitelist from device-crash-test script somehow
diff --git a/tests/validator/device-help.yaml b/tests/validator/device-help.yaml
new file mode 100644
index 0000000000..95aa5c9f5b
--- /dev/null
+++ b/tests/validator/device-help.yaml
@@ -0,0 +1,3 @@
+# Just check if -device ...,help works:
+command-line: '$QEMU -device $DEVICE,help'
+qmp: false
diff --git a/tests/validator/device-introspect.yaml b/tests/validator/device-introspect.yaml
new file mode 100644
index 0000000000..f5b10aff9a
--- /dev/null
+++ b/tests/validator/device-introspect.yaml
@@ -0,0 +1,23 @@
+# This test specification is equivalent to "device/introspect/concrete" in
+# tests/device-introspect-test.c
+command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL'
+monitor-commands:
+- qmp:
+  - execute: 'device-list-properties'
+    arguments:
+      typename: '$DEVICE'
+- hmp: 'device_add $DEVICE,help'
+- hmp: 'info qom-tree'
+defaults:
+  MACHINE: 'none'
+  ACCEL: 'kvm:tcg'
+expected-failures:
+  - MACHINE: 'xenpv'
+  - MACHINE: 'xenfv'
+
+#TODO: the test runner could support something like:
+# extra-cases:
+#   # like "device/introspect/none":
+#   - DEVICE: 'nonexistent'
+#   # like "device/introspect/abstract":
+#   - DEVICE: 'device'
diff --git a/tests/validator/device-list.yaml b/tests/validator/device-list.yaml
new file mode 100644
index 0000000000..ac0004c05d
--- /dev/null
+++ b/tests/validator/device-list.yaml
@@ -0,0 +1,10 @@
+# this test specification is equivalent to the
+# "device/introspect/list" test case in device-introspect-test.c
+command-line: '$QEMU -nodefaults -machine none'
+monitor-commands:
+- qmp:
+  - execute: qom-list-types
+    arguments:
+      implements: 'device'
+      abstract: true
+- hmp: 'device_add help'
diff --git a/tests/validator/just-help.yaml b/tests/validator/just-help.yaml
new file mode 100644
index 0000000000..84ec8d7090
--- /dev/null
+++ b/tests/validator/just-help.yaml
@@ -0,0 +1,3 @@
+# just run $QEMU -help and ensure it won't crash
+command-line: '$QEMU -help'
+qmp: false
-- 
2.14.3

Re: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180329213857.15499-1-ehabkost@redhat.com
Subject: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 t [tag update]            patchew/20180330170209.20627-1-rkagan@virtuozzo.com -> patchew/20180330170209.20627-1-rkagan@virtuozzo.com
 * [new tag]               patchew/20180330195928.15607-1-jcmvbkbc@gmail.com -> patchew/20180330195928.15607-1-jcmvbkbc@gmail.com
Switched to a new branch 'test'
0fdb1a2f05 Collection of validator.py test cases
962c7b9a45 validator.py script
11125125d8 qemu.py: is_launched() method
e7b9360fc7 qemu.py: qmp_obj() method
04f1d03e56 qemu.py: Don't try to quit cleanly on exceptions
b504207869 qemu.py: 'force' parameter on shutdown()
9cfedb7337 qemu.py: Only wait for process if it's still running
8c7e843d6e qemu.py: Log crashes inside _post_shutdown()
992c1b6bb2 qemu.py: Set _launched = False on _post_shutdown
9e4bcb4215 qemu.py: Make monitor optional
4d78dabfe2 qemu.py: Close _qmp inside _post_shutdown()
3937ce28e2 qemu.py: Use wait() logic inside shutdown()
c2b5140e3d qemu.py: Move _load_io_log() call to _post_shutdown()
a20254c9e0 qemu.py: Split _base_args()
bb5e609a6b qemu.py: Make _vm_monitor a method
a571feea0c qmp.py: Cleanly handle unexpectedly closed socket
9a92759c0c qmp.py: Fix error handling for Python 3
1f8f56095d qmp.py: Make it safe to call close() any time

=== OUTPUT BEGIN ===
Checking PATCH 1/18: qmp.py: Make it safe to call close() any time...
Checking PATCH 2/18: qmp.py: Fix error handling for Python 3...
Checking PATCH 3/18: qmp.py: Cleanly handle unexpectedly closed socket...
Checking PATCH 4/18: qemu.py: Make _vm_monitor a method...
Checking PATCH 5/18: qemu.py: Split _base_args()...
Checking PATCH 6/18: qemu.py: Move _load_io_log() call to _post_shutdown()...
Checking PATCH 7/18: qemu.py: Use wait() logic inside shutdown()...
Checking PATCH 8/18: qemu.py: Close _qmp inside _post_shutdown()...
Checking PATCH 9/18: qemu.py: Make monitor optional...
Checking PATCH 10/18: qemu.py: Set _launched = False on _post_shutdown...
Checking PATCH 11/18: qemu.py: Log crashes inside _post_shutdown()...
Checking PATCH 12/18: qemu.py: Only wait for process if it's still running...
Checking PATCH 13/18: qemu.py: 'force' parameter on shutdown()...
Checking PATCH 14/18: qemu.py: Don't try to quit cleanly on exceptions...
Checking PATCH 15/18: qemu.py: qmp_obj() method...
Checking PATCH 16/18: qemu.py: is_launched() method...
Checking PATCH 17/18: validator.py script...
WARNING: line over 80 characters
#188: FILE: scripts/validator.py:169:
+            self.alldevs = qom_type_names(vm, implements='device', abstract=False)

WARNING: line over 80 characters
#191: FILE: scripts/validator.py:172:
+            self.no_user_devs = [d['name'] for d in info_qdm(vm, ) if d['no-user']]

WARNING: line over 80 characters
#193: FILE: scripts/validator.py:174:
+            self.user_devs = [dev for dev in self.alldevs if dev not in self.no_user_devs]

WARNING: line over 80 characters
#195: FILE: scripts/validator.py:176:
+            self.cpu_models = [c['name'] for c in vm.command('query-cpu-definitions')]

ERROR: line over 90 characters
#301: FILE: scripts/validator.py:282:
+    >>> list(tc) == [{'a':[i], 'b':[j], 'c':[100, 200, 300]} for i in [1,2] for j in [10, 20]]

ERROR: line over 90 characters
#362: FILE: scripts/validator.py:343:
+        self._vars = dict((v, getattr(BuiltinVars, v)()) for v in dir(BuiltinVars) if not v.startswith('_'))

WARNING: line over 80 characters
#372: FILE: scripts/validator.py:353:
+        Default values override the values returned by VariableDefinition.enumerate()

WARNING: line over 80 characters
#392: FILE: scripts/validator.py:373:
+        """Return full list of variables, including dependencies in the right order

ERROR: line over 90 characters
#411: FILE: scripts/validator.py:392:
+                    raise Exception("Variable dependency cycle: %s" % (' -> '.join(vars.keys())))

ERROR: line over 90 characters
#530: FILE: scripts/validator.py:511:
+        vars = vars_for_template(self.get('command-line')) + vars_for_template(self.get('monitor-commands'))

WARNING: line over 80 characters
#548: FILE: scripts/validator.py:529:
+        return ' '.join('%s=%s' % (k, shquote(v)) for k,v in self.values.items())

WARNING: line over 80 characters
#588: FILE: scripts/validator.py:569:
+                    raise InvalidSpecification("Invalid monitor command: %r: %r" % (k, v))

ERROR: line over 90 characters
#688: FILE: scripts/validator.py:669:
+                        help="Run all test case combinations, not just the default for the test specification")

ERROR: line over 90 characters
#702: FILE: scripts/validator.py:683:
+    logging.basicConfig(stream=sys.stdout, level=args.loglevel, format='%(levelname)s: %(message)s')

total: 6 errors, 8 warnings, 724 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 18/18: Collection of validator.py test cases...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[no-reply]

Hi,

This series failed docker-quick@centos6 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180329213857.15499-1-ehabkost@redhat.com
Subject: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-quick@centos6
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
0fdb1a2f05 Collection of validator.py test cases
962c7b9a45 validator.py script
11125125d8 qemu.py: is_launched() method
e7b9360fc7 qemu.py: qmp_obj() method
04f1d03e56 qemu.py: Don't try to quit cleanly on exceptions
b504207869 qemu.py: 'force' parameter on shutdown()
9cfedb7337 qemu.py: Only wait for process if it's still running
8c7e843d6e qemu.py: Log crashes inside _post_shutdown()
992c1b6bb2 qemu.py: Set _launched = False on _post_shutdown
9e4bcb4215 qemu.py: Make monitor optional
4d78dabfe2 qemu.py: Close _qmp inside _post_shutdown()
3937ce28e2 qemu.py: Use wait() logic inside shutdown()
c2b5140e3d qemu.py: Move _load_io_log() call to _post_shutdown()
a20254c9e0 qemu.py: Split _base_args()
bb5e609a6b qemu.py: Make _vm_monitor a method
a571feea0c qmp.py: Cleanly handle unexpectedly closed socket
9a92759c0c qmp.py: Fix error handling for Python 3
1f8f56095d qmp.py: Make it safe to call close() any time

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-_gt5exqw/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   centos6
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-_gt5exqw/src'
  GEN     /var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200/qemu.tar.vroot'...
done.
Checking out files:  44% (2693/6074)   
Checking out files:  45% (2734/6074)   
Checking out files:  46% (2795/6074)   
Checking out files:  47% (2855/6074)   
Checking out files:  47% (2882/6074)   
Checking out files:  48% (2916/6074)   
Checking out files:  49% (2977/6074)   
Checking out files:  50% (3037/6074)   
Checking out files:  51% (3098/6074)   
Checking out files:  52% (3159/6074)   
Checking out files:  53% (3220/6074)   
Checking out files:  54% (3280/6074)   
Checking out files:  55% (3341/6074)   
Checking out files:  56% (3402/6074)   
Checking out files:  57% (3463/6074)   
Checking out files:  58% (3523/6074)   
Checking out files:  59% (3584/6074)   
Checking out files:  60% (3645/6074)   
Checking out files:  61% (3706/6074)   
Checking out files:  62% (3766/6074)   
Checking out files:  63% (3827/6074)   
Checking out files:  64% (3888/6074)   
Checking out files:  65% (3949/6074)   
Checking out files:  66% (4009/6074)   
Checking out files:  67% (4070/6074)   
Checking out files:  68% (4131/6074)   
Checking out files:  69% (4192/6074)   
Checking out files:  70% (4252/6074)   
Checking out files:  71% (4313/6074)   
Checking out files:  72% (4374/6074)   
Checking out files:  73% (4435/6074)   
Checking out files:  74% (4495/6074)   
Checking out files:  75% (4556/6074)   
Checking out files:  76% (4617/6074)   
Checking out files:  77% (4677/6074)   
Checking out files:  78% (4738/6074)   
Checking out files:  79% (4799/6074)   
Checking out files:  80% (4860/6074)   
Checking out files:  81% (4920/6074)   
Checking out files:  82% (4981/6074)   
Checking out files:  83% (5042/6074)   
Checking out files:  84% (5103/6074)   
Checking out files:  85% (5163/6074)   
Checking out files:  86% (5224/6074)   
Checking out files:  86% (5278/6074)   
Checking out files:  87% (5285/6074)   
Checking out files:  88% (5346/6074)   
Checking out files:  89% (5406/6074)   
Checking out files:  90% (5467/6074)   
Checking out files:  91% (5528/6074)   
Checking out files:  92% (5589/6074)   
Checking out files:  93% (5649/6074)   
Checking out files:  94% (5710/6074)   
Checking out files:  95% (5771/6074)   
Checking out files:  96% (5832/6074)   
Checking out files:  97% (5892/6074)   
Checking out files:  98% (5953/6074)   
Checking out files:  99% (6014/6074)   
Checking out files: 100% (6074/6074)   
Checking out files: 100% (6074/6074), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
tar: /var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200/qemu.tar: Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now
failed to create tar file
  COPY    RUNNER
    RUN test-quick in qemu:centos6 
tar: Unexpected EOF in archive
tar: rmtlseek not stopped at a record boundary
tar: Error is not recoverable: exiting now
/var/tmp/qemu/run: line 32: prep_fail: command not found
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
bison-2.4.1-5.el6.x86_64
bzip2-devel-1.0.5-7.el6_0.x86_64
ccache-3.1.6-2.el6.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el6.x86_64
flex-2.5.35-9.el6.x86_64
gcc-4.4.7-18.el6.x86_64
gettext-0.17-18.el6.x86_64
git-1.7.1-9.el6_9.x86_64
glib2-devel-2.28.8-9.el6.x86_64
libepoxy-devel-1.2-3.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
librdmacm-devel-1.0.21-0.el6.x86_64
lzo-devel-2.03-3.1.el6_5.1.x86_64
make-3.81-23.el6.x86_64
mesa-libEGL-devel-11.0.7-4.el6.x86_64
mesa-libgbm-devel-11.0.7-4.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
spice-glib-devel-0.26-8.el6.x86_64
spice-server-devel-0.12.4-16.el6.x86_64
tar-1.23-15.el6_8.x86_64
vte-devel-0.25.1-9.el6.x86_64
xen-devel-4.6.6-2.el6.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=bison     bzip2-devel     ccache     csnappy-devel     flex     g++     gcc     gettext     git     glib2-devel     libepoxy-devel     libfdt-devel     librdmacm-devel     lzo-devel     make     mesa-libEGL-devel     mesa-libgbm-devel     pixman-devel     SDL-devel     spice-glib-devel     spice-server-devel     tar     vte-devel     xen-devel     zlib-devel
HOSTNAME=bbfcbf497505
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
FEATURES= dtc
DEBUG=
_=/usr/bin/env

/var/tmp/qemu/run: line 52: cd: /tmp/qemu-test/src/tests/docker: No such file or directory
/var/tmp/qemu/run: line 57: /test-quick: No such file or directory
/var/tmp/qemu/run: line 57: exec: /test-quick: cannot execute: No such file or directory
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=bca478da34be11e8875d52540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-_gt5exqw/src/docker-src.2018-03-31-04.37.05.10200:/var/tmp/qemu:z,ro', 'qemu:centos6', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 126
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-_gt5exqw/src'
make: *** [tests/docker/Makefile.include:163: docker-run-test-quick@centos6] Error 2

real	0m33.155s
user	0m8.989s
sys	0m6.511s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[no-reply]

Hi,

This series failed docker-build@min-glib build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

N/A. Internal error while reading log file



---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[Philippe Mathieu-Daudé]

[/var]/tmp full due to some jobs not cleaning well their workspaces?

On 03/31/2018 06:04 AM, no-reply@patchew.org wrote:
> Hi,
> 
> This series failed docker-build@min-glib build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> N/A. Internal error while reading log file
> 
> 
> 
> ---
> Email generated automatically by Patchew [http://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
> 

Re: [Qemu-devel] [RFC 00/18] QEMU validator: A method to specify QEMU crash-test cases

[Fam Zheng]

On Mon, Apr 2, 2018 at 7:10 AM, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> [/var]/tmp full due to some jobs not cleaning well their workspaces?

Somehow our "make check" now keeps hanging in the container,
preventing it from being cleaned up correctly...

Will try to find time for it this week.

Fam

>
> On 03/31/2018 06:04 AM, no-reply@patchew.org wrote:
>> Hi,
>>
>> This series failed docker-build@min-glib build test. Please find the testing commands and
>> their output below. If you have Docker installed, you can probably reproduce it
>> locally.
>>
>> N/A. Internal error while reading log file
>>
>>
>>
>> ---
>> Email generated automatically by Patchew [http://patchew.org/].
>> Please send your feedback to patchew-devel@redhat.com
>>

Re: [Qemu-devel] [RFC 02/18] qmp.py: Fix error handling for Python 3

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> socket.error doesn't behave like a tuple in Python 3, but we can
> use error.args on both Python 2.7 and 3.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qmp/qmp.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
> index 078ce65f3b..e9cb6b2683 100644
> --- a/scripts/qmp/qmp.py
> +++ b/scripts/qmp/qmp.py
> @@ -108,7 +108,7 @@ class QEMUMonitorProtocol(object):
>          try:
>              self.__json_read()
>          except socket.error as err:
> -            if err[0] == errno.EAGAIN:
> +            if err.args[0] == errno.EAGAIN:
>                  # No data available
>                  pass
>          self.__sock.setblocking(1)
> @@ -168,9 +168,9 @@ class QEMUMonitorProtocol(object):
>          try:
>              self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
>          except socket.error as err:
> -            if err[0] == errno.EPIPE:
> +            if err.args[0] == errno.EPIPE:
>                  return
> -            raise socket.error(err)
> +            raise
>          resp = self.__json_read()
>          self.logger.debug("<<< %s", resp)
>          return resp
> 

Re: [Qemu-devel] [RFC 03/18] qmp.py: Cleanly handle unexpectedly closed socket

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> QEMUMonitorProtocol.cmd() returns None if the socket was closed,
> so callers must handle this case explicltly.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qmp/qmp.py | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
> index e9cb6b2683..91b537ea04 100644
> --- a/scripts/qmp/qmp.py
> +++ b/scripts/qmp/qmp.py
> @@ -73,6 +73,8 @@ class QEMUMonitorProtocol(object):
>              raise QMPConnectError
>          # Greeting seems ok, negotiate capabilities
>          resp = self.cmd('qmp_capabilities')
> +        if resp is None:
> +            raise QMPConnectError("QMP connection unexpectedly closed")
>          if "return" in resp:
>              return greeting
>          raise QMPCapabilitiesError
> @@ -182,6 +184,8 @@ class QEMUMonitorProtocol(object):
>          @param name: command name (string)
>          @param args: command arguments (dict)
>          @param cmd_id: command id (dict, list, string or int)
> +        @return QMP response as a Python dict or None if the connection has
> +                been closed
>          """
>          qmp_cmd = {'execute': name}
>          if args:
> @@ -195,6 +199,8 @@ class QEMUMonitorProtocol(object):
>          Build and send a QMP command to the monitor, report errors if any
>          """
>          ret = self.cmd(cmd, kwds)
> +        if ret is None:
> +            raise QMPConnectError("QMP connection unexpectedly closed")
>          if "error" in ret:
>              raise Exception(ret['error']['desc'])
>          return ret['return']
> 

Re: [Qemu-devel] [RFC 05/18] qemu.py: Split _base_args()

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> Split it into _monitor_args() and _display_args(), so we can make
> monitor args optional later.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qemu.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index c051c4525a..d9f85bb153 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -173,15 +173,20 @@ class QEMUMachine(object):
>          else:
>              return os.path.join(self._temp_dir, self._name + "-monitor.sock")
>  
> -    def _base_args(self):
> +    def _monitor_args(self):
>          addr = self._vm_monitor()
>          if isinstance(addr, tuple):
>              moncdev = "socket,id=mon,host=%s,port=%s" % (addr[0], addr[1])
>          else:
>              moncdev = 'socket,id=mon,path=%s' % (addr)
>          return ['-chardev', moncdev,
> -                '-mon', 'chardev=mon,mode=control',
> -                '-display', 'none', '-vga', 'none']
> +                '-mon', 'chardev=mon,mode=control']
> +
> +    def _display_args(self):
> +        return ['-display', 'none', '-vga', 'none']
> +
> +    def _base_args(self):
> +        return self._monitor_args() + self._display_args()
>  
>      def _pre_launch(self):
>          self._temp_dir = tempfile.mkdtemp(dir=self._test_dir)
> 

Re: [Qemu-devel] [RFC 06/18] qemu.py: Move _load_io_log() call to _post_shutdown()

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> All callers of _post_shutdown() call _load_io_log(), so it's
> easier to simply call it inside _post_shutdown().
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qemu.py | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index d9f85bb153..84bb3da613 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -199,6 +199,7 @@ class QEMUMachine(object):
>          self._qmp.accept()
>  
>      def _post_shutdown(self):
> +        self._load_io_log()
>          if self._qemu_log_file is not None:
>              self._qemu_log_file.close()
>              self._qemu_log_file = None
> @@ -250,7 +251,6 @@ class QEMUMachine(object):
>          '''Wait for the VM to power off'''
>          self._popen.wait()
>          self._qmp.close()
> -        self._load_io_log()
>          self._post_shutdown()
>  
>      def shutdown(self):
> @@ -263,7 +263,6 @@ class QEMUMachine(object):
>                  self._popen.kill()
>              self._popen.wait()
>  
> -        self._load_io_log()
>          self._post_shutdown()
>  
>          exitcode = self.exitcode()
> 

Re: [Qemu-devel] [RFC 08/18] qemu.py: Close _qmp inside _post_shutdown()

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> This way all shutdown-related cleanup is kept in a single place.
> 
> While at it, set _qmp to None after closing the socket, to avoid
> trying to reuse it by mistake later.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qemu.py | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index 4a71542c6b..1241e7f10b 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -199,6 +199,10 @@ class QEMUMachine(object):
>          self._qmp.accept()
>  
>      def _post_shutdown(self):
> +        if self._qmp is not None:
> +            self._qmp.close()
> +            self._qmp = None
> +
>          self._load_io_log()
>          if self._qemu_log_file is not None:
>              self._qemu_log_file.close()
> @@ -250,7 +254,6 @@ class QEMUMachine(object):
>      def wait(self):
>          '''Wait for the VM to power off'''
>          self._popen.wait()
> -        self._qmp.close()
>          self._post_shutdown()
>  
>      def shutdown(self):
> 

Re: [Qemu-devel] [RFC 11/18] qemu.py: Log crashes inside _post_shutdown()

[Philippe Mathieu-Daudé]

On 03/29/2018 06:38 PM, Eduardo Habkost wrote:
> This will allow us to log QEMU crashes even if the test code uses
> .wait() instead of .shutdown().
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  scripts/qemu.py | 18 +++++++++---------
>  1 file changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index 226d2c4d48..e19e4b34d0 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -225,6 +225,15 @@ class QEMUMachine(object):
>              shutil.rmtree(self._temp_dir)
>              self._temp_dir = None
>  
> +        exitcode = self.exitcode()
> +        if exitcode is not None and exitcode < 0:
> +            msg = 'qemu received signal %i: %s'
> +            if self._qemu_full_args:
> +                command = ' '.join(self._qemu_full_args)
> +            else:
> +                command = ''
> +            LOG.warn(msg, exitcode, command)
> +
>          self._launched = False
>  
>      def launch(self):
> @@ -279,15 +288,6 @@ class QEMUMachine(object):
>  
>          self.wait()
>  
> -        exitcode = self.exitcode()
> -        if exitcode is not None and exitcode < 0:
> -            msg = 'qemu received signal %i: %s'
> -            if self._qemu_full_args:
> -                command = ' '.join(self._qemu_full_args)
> -            else:
> -                command = ''
> -            LOG.warn(msg, exitcode, command)
> -
>      def qmp(self, cmd, conv_keys=True, **args):
>          '''Invoke a QMP command and return the response dict'''
>          qmp_args = dict()
> 

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> See cover letter for a description of the new test system.
>
> TODO: code clean up
> TODO: write description.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  scripts/validator.py | 724 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 724 insertions(+)
>  create mode 100755 scripts/validator.py
>
> diff --git a/scripts/validator.py b/scripts/validator.py
> new file mode 100755
> index 0000000000..4312571feb
> --- /dev/null
> +++ b/scripts/validator.py
> @@ -0,0 +1,724 @@
> +#!/usr/bin/env python
> +#
> +#  Copyright (c) 2018 Red Hat Inc
> +#
> +# Author:
> +#  Eduardo Habkost <ehabkost@redhat.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +"""
> +QEMU validator script
> +=====================
> +
> +This script will get test YAML test case specifications or Python
> +modules as input, and generate/run test cases based on them.
> +
> +USAGE
> +-----
> +
> +validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3
> +
> +specification-file is a YAML file containing the test specification.

I can see the "test YAML test case specifications", but not the "Python
modules".

Should we introduce yet another markup language into QEMU?  (pardon the
pun).

> +
> +Example::
> +
> +    # this test specification is equivalent to the
> +    # "device/introspect/list" test case in device-introspect-test.c

If we merge this, not for long :)

> +    command-line: '$QEMU -nodefaults -machine none'
> +    monitor-commands:
> +    - qmp:
> +      - execute: qom-list-types
> +        arguments:
> +          implements: 'device'
> +          abstract: true
> +    - hmp: 'device_add help'
> +
> +
> +VARIABLE EXPANSION
> +------------------
> +
> +The test runner will try to run the test cases with all possible values
> +for variables appearing in the test specification.
> +
> +Some built-in variables are automatically expanded:
> +
> +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> +* `$DEVICE` - Expands to a (user-creatable) device type name supported by $QEMU
> +* `$CPU` - Expands to a CPU model name supported by $QEMU
> +
> +Note that the $QEMU variable must be specified in th

Yes?

> +
> +TEST SPECIFICATION FIELDS
> +-------------------------
> +
> +command-line
> +~~~~~~~~~~~~
> +
> +List or string, containing the QEMU command-line to be run.
> +
> +Default: '$QEMU'
> +
> +
> +monitor-commands
> +~~~~~~~~~~~~~~~~
> +
> +Mapping or list-of-mappings containing monitor commands to run.  The key on each
> +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> +mapping, or list.
> +
> +Default: None.
> +
> +TODO: not implemented yet.

The whole monitor-commands feature?

Can I do

    monitor-commands:
    - qmp:
      - execute: eins
    - hmp: zwei
    - qmp:
      - execute: drei

to execute eins, zwei, drei in this order?

> +
> +
> +qmp
> +~~~
> +
> +Boolean.  If true (the default), a QMP monitor is configured on the command-line
> +automatically.
> +
> +If true, the test runner will issue a ``quit`` command automatically when
> +testing is finished.  If false, the test runner will wait until QEMU exits by
> +itself.
> +
> +Example::
> +
> +    # just run $QEMU -help and ensure it won't crash
> +    command-line: ['$QEMU', '-help']
> +    qmp: false
> +
> +
> +TODO: whitelist
> +TODO: validate output against reference output
> +TODO: configure defaults for variables
> +TODO: compatibility with Avocado multiplexer?
> +"""

This is a DSL to write tests.  I applaud the idea to write more tests at
a higher level than C.  We just need to be careful not to create too
many different ways to write tests, or else readability will suffer.
Ideally, the way to use for a test should be fairly obvious.

Looking forward to your next iteration.

[...]

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > See cover letter for a description of the new test system.
> >
> > TODO: code clean up
> > TODO: write description.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  scripts/validator.py | 724 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 724 insertions(+)
> >  create mode 100755 scripts/validator.py
> >
> > diff --git a/scripts/validator.py b/scripts/validator.py
> > new file mode 100755
> > index 0000000000..4312571feb
> > --- /dev/null
> > +++ b/scripts/validator.py
> > @@ -0,0 +1,724 @@
> > +#!/usr/bin/env python
> > +#
> > +#  Copyright (c) 2018 Red Hat Inc
> > +#
> > +# Author:
> > +#  Eduardo Habkost <ehabkost@redhat.com>
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License along
> > +# with this program; if not, write to the Free Software Foundation, Inc.,
> > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> > +
> > +"""
> > +QEMU validator script
> > +=====================
> > +
> > +This script will get test YAML test case specifications or Python
> > +modules as input, and generate/run test cases based on them.
> > +
> > +USAGE
> > +-----
> > +
> > +validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3
> > +
> > +specification-file is a YAML file containing the test specification.
> 
> I can see the "test YAML test case specifications", but not the "Python
> modules".
> 
> Should we introduce yet another markup language into QEMU?  (pardon the
> pun).

Fair question.  What are the existing markup languages in QEMU we
could use?

JSON is an option, but I believe YAML is more readable.

> 
> > +
> > +Example::
> > +
> > +    # this test specification is equivalent to the
> > +    # "device/introspect/list" test case in device-introspect-test.c
> 
> If we merge this, not for long :)
> 
> > +    command-line: '$QEMU -nodefaults -machine none'
> > +    monitor-commands:
> > +    - qmp:
> > +      - execute: qom-list-types
> > +        arguments:
> > +          implements: 'device'
> > +          abstract: true
> > +    - hmp: 'device_add help'
> > +
> > +
> > +VARIABLE EXPANSION
> > +------------------
> > +
> > +The test runner will try to run the test cases with all possible values
> > +for variables appearing in the test specification.
> > +
> > +Some built-in variables are automatically expanded:
> > +
> > +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> > +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> > +* `$DEVICE` - Expands to a (user-creatable) device type name supported by $QEMU
> > +* `$CPU` - Expands to a CPU model name supported by $QEMU
> > +
> > +Note that the $QEMU variable must be specified in th
> 
> Yes?

Heh.  It must be specified in the command-line.  I will fix that.

> 
> > +
> > +TEST SPECIFICATION FIELDS
> > +-------------------------
> > +
> > +command-line
> > +~~~~~~~~~~~~
> > +
> > +List or string, containing the QEMU command-line to be run.
> > +
> > +Default: '$QEMU'
> > +
> > +
> > +monitor-commands
> > +~~~~~~~~~~~~~~~~
> > +
> > +Mapping or list-of-mappings containing monitor commands to run.  The key on each
> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> > +mapping, or list.
> > +
> > +Default: None.
> > +
> > +TODO: not implemented yet.
> 
> The whole monitor-commands feature?

Outdated comment, sorry.

> 
> Can I do
> 
>     monitor-commands:
>     - qmp:
>       - execute: eins
>     - hmp: zwei
>     - qmp:
>       - execute: drei
> 
> to execute eins, zwei, drei in this order?

Yes, it can.  See the test specification examples.

> 
> > +
> > +
> > +qmp
> > +~~~
> > +
> > +Boolean.  If true (the default), a QMP monitor is configured on the command-line
> > +automatically.
> > +
> > +If true, the test runner will issue a ``quit`` command automatically when
> > +testing is finished.  If false, the test runner will wait until QEMU exits by
> > +itself.
> > +
> > +Example::
> > +
> > +    # just run $QEMU -help and ensure it won't crash
> > +    command-line: ['$QEMU', '-help']
> > +    qmp: false
> > +
> > +
> > +TODO: whitelist
> > +TODO: validate output against reference output
> > +TODO: configure defaults for variables
> > +TODO: compatibility with Avocado multiplexer?
> > +"""
> 
> This is a DSL to write tests.  I applaud the idea to write more tests at
> a higher level than C.  We just need to be careful not to create too
> many different ways to write tests, or else readability will suffer.
> Ideally, the way to use for a test should be fairly obvious.

Agreed we need to keep this in mind.

Replacing the DSL with very short and simple Python code is a
possibility I was considering, but I'm not sure yet if this is
the way to go.  I don't want this to create a whole new test
framework.


> 
> Looking forward to your next iteration.

Thanks!

-- 
Eduardo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Paolo Bonzini]

On 17/04/2018 16:42, Eduardo Habkost wrote:
>> Should we introduce yet another markup language into QEMU?  (pardon the
>> pun).
> Fair question.  What are the existing markup languages in QEMU we
> could use?
> 
> JSON is an option, but I believe YAML is more readable.

Isn't YAML actually a superset of JSON that adds "magic indent", bullet
lists and a bunch of other stuff that you're not using?  You can still
use JSON to express a complicated dictionary for example.

Paolo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 05:17:13PM +0200, Paolo Bonzini wrote:
> On 17/04/2018 16:42, Eduardo Habkost wrote:
> >> Should we introduce yet another markup language into QEMU?  (pardon the
> >> pun).
> > Fair question.  What are the existing markup languages in QEMU we
> > could use?
> > 
> > JSON is an option, but I believe YAML is more readable.
> 
> Isn't YAML actually a superset of JSON that adds "magic indent", bullet
> lists and a bunch of other stuff that you're not using?  You can still
> use JSON to express a complicated dictionary for example.

Things I like about YAML in comparison to JSON:
- Comments
- No need to count number of closing brackets or worry about
  trailing commas
- Convenient ways to write long strings without
  "\\"another layer of escaping syntax\\""

JSON is not a language for humans, YAML is.

-- 
Eduardo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
[...]
>> > +    command-line: '$QEMU -nodefaults -machine none'
>> > +    monitor-commands:
>> > +    - qmp:
>> > +      - execute: qom-list-types
>> > +        arguments:
>> > +          implements: 'device'
>> > +          abstract: true
>> > +    - hmp: 'device_add help'
[...]
>> > +monitor-commands
>> > +~~~~~~~~~~~~~~~~
>> > +
>> > +Mapping or list-of-mappings containing monitor commands to run.

http://yaml.org/spec/1.2/spec.html talks about mappings and sequences.
Recommend to say "sequence" rather than "list".

I can't see how sequences come into play.  Your example's
monitor-command is a mapping, not a sequence of mappings.  Am I
confused?

>> >                                                                   The key on each
>> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
>> > +mapping, or list.
>> > +
>> > +Default: None.
[...]
>> Can I do
>> 
>>     monitor-commands:
>>     - qmp:
>>       - execute: eins
>>     - hmp: zwei
>>     - qmp:
>>       - execute: drei
>> 
>> to execute eins, zwei, drei in this order?
>
> Yes, it can.  See the test specification examples.

I asked because the YAML spec I quoted above says mappings are unordered
and must be unique.  My example violates "unique", and your "yes, it
can" violates "unordered".

[...]

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Wed, Apr 18, 2018 at 08:58:00AM +0200, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> [...]
> >> > +    command-line: '$QEMU -nodefaults -machine none'
> >> > +    monitor-commands:
> >> > +    - qmp:
> >> > +      - execute: qom-list-types
> >> > +        arguments:
> >> > +          implements: 'device'
> >> > +          abstract: true
> >> > +    - hmp: 'device_add help'
> [...]
> >> > +monitor-commands
> >> > +~~~~~~~~~~~~~~~~
> >> > +
> >> > +Mapping or list-of-mappings containing monitor commands to run.
> 
> http://yaml.org/spec/1.2/spec.html talks about mappings and sequences.
> Recommend to say "sequence" rather than "list".

Noted.

> 
> I can't see how sequences come into play.  Your example's
> monitor-command is a mapping, not a sequence of mappings.  Am I
> confused?

The example is supposed to be a sequence of mappings.

> 
> >> >                                                                   The key on each
> >> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> >> > +mapping, or list.
> >> > +
> >> > +Default: None.
> [...]
> >> Can I do
> >> 
> >>     monitor-commands:
> >>     - qmp:
> >>       - execute: eins
> >>     - hmp: zwei
> >>     - qmp:
> >>       - execute: drei
> >> 
> >> to execute eins, zwei, drei in this order?
> >
> > Yes, it can.  See the test specification examples.
> 
> I asked because the YAML spec I quoted above says mappings are unordered
> and must be unique.  My example violates "unique", and your "yes, it
> can" violates "unordered".
> 
> [...]

The above is a sequence of one-key mappings.


This is a (not very useful) mapping:

>>> yaml.load("""
... qmp:
... - execute: eins
... hmp: zwei
... """)
{'qmp': [{'execute': 'eins'}], 'hmp': 'zwei'}


This is a sequence of one-key mappings:

>>> yaml.load("""
... - qmp:
...   - execute: eins
... - hmp: zwei
... - qmp:
...   - execute: drei
... """)
[{'qmp': [{'execute': 'eins'}]}, {'hmp': 'zwei'}, {'qmp': [{'execute': 'drei'}]}]
>>>

-- 
Eduardo