* [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device
@ 2017-05-13 3:33 Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown Eduardo Habkost
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-13 3:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Thomas Huth, Marcel Apfelbaum, Markus Armbruster
Changes v1 -> v2:
* Use a simpler method to query QEMU exit code in qemu.py
* Use only qemu.py module, instead of qtest.py
* New whitelist entries:
* "could not find stage1 bootloader"
* Segfaults when using devices: a15mpcore_priv, sb16, cs4231a, arm-gicv3
* Format "success" line using formatTestCase(), and using DEBUG
loglevel
* Reword "test case:" line with "running test case:", for clarity
* Fix "pc-.*" whitelist to include "q35" too
* Add --devtype option to test only a specific device type
* Send all log messages to stdout instead of stderr
* Avoid printing "obsolete whitelist entry?" messages if we know
we are not testing every single accel/machine/device
combination
* --quick mode, to skip cases where failures are always expected,
and to print a warning in case we don't get an expected failure
* Use qemu.QEMUMachine instead of qtest.QEMUQtestMachine, as we don't
use any of the QEMUQtestMachine features
* Fix handling of multiple '-t' options
* Simplify code that generate random sample of test cases
This series adds scripts/device-crashtest.py, that can be used to
crash-test -device with multiple machine/accel/device
combinations.
The script found a few crashes on some machines/devices. A dump
of existing cases can be seen here:
https://gist.github.com/ehabkost/503b0af0375f0d98d3e84017e8ca54eb
The script contains a whitelist that can also be useful as
documentation of existing ways -device can fail or crash.
Note that the script takes a few hours to run on the default mode
(testing all accel/machine/device combinations), but the "-r N"
option can be used to make it only test N random samples.
Example script output:
$ ../scripts/device-crash-test.py -v --shuffle
INFO: test case: machine=verdex binary=./aarch64-softmmu/qemu-system-aarch64 device=exynos4210-ehci-usb accel=tcg
INFO: test case: machine=none binary=./aarch64-softmmu/qemu-system-aarch64 device=onenand accel=tcg
INFO: test case: machine=pc-i440fx-2.2 binary=./x86_64-softmmu/qemu-system-x86_64 device=ide-cd accel=kvm
INFO: success: ./x86_64-softmmu/qemu-system-x86_64 -S -machine pc-i440fx-2.2,accel=kvm -device ide-cd
INFO: test case: machine=SPARCClassic binary=./sparc-softmmu/qemu-system-sparc device=memory accel=tcg
qemu received signal 6: -S -machine SPARCClassic,accel=tcg -device memory
ERROR: failed: machine=SPARCClassic binary=./sparc-softmmu/qemu-system-sparc device=memory accel=tcg
ERROR: cmdline: ./sparc-softmmu/qemu-system-sparc -S -machine SPARCClassic,accel=tcg -device memory
ERROR: log: qemu-system-sparc: /root/qemu-build/exec.c:1500: find_ram_offset: Assertion `size != 0' failed.
ERROR: exit code: -6
INFO: test case: machine=romulus-bmc binary=./arm-softmmu/qemu-system-arm device=ich9-usb-uhci6 accel=tcg
INFO: test case: machine=ref405ep binary=./ppc-softmmu/qemu-system-ppc device=ivshmem-doorbell accel=tcg
INFO: test case: machine=romulus-bmc binary=./aarch64-softmmu/qemu-system-aarch64 device=l2x0 accel=tcg
INFO: test case: machine=pc-i440fx-1.7 binary=./x86_64-softmmu/qemu-system-x86_64 device=virtio-input-host-pci accel=tcg
INFO: test case: machine=none binary=./ppc-softmmu/qemu-system-ppc device=virtio-tablet-pci accel=tcg
INFO: test case: machine=terrier binary=./aarch64-softmmu/qemu-system-aarch64 device=sst25vf016b accel=tcg
INFO: success: ./aarch64-softmmu/qemu-system-aarch64 -S -machine terrier,accel=tcg -device sst25vf016b
INFO: test case: machine=none binary=./i386-softmmu/qemu-system-i386 device=intel-iommu accel=kvm
qemu received signal 6: -S -machine none,accel=kvm -device intel-iommu
ERROR: failed: machine=none binary=./i386-softmmu/qemu-system-i386 device=intel-iommu accel=kvm
ERROR: cmdline: ./i386-softmmu/qemu-system-i386 -S -machine none,accel=kvm -device intel-iommu
ERROR: log: /root/qemu-build/hw/i386/intel_iommu.c:2565:vtd_realize: Object 0x7fe117fabfb0 is not an instance of type generic-pc-machine
ERROR: exit code: -6
INFO: test case: machine=tosa binary=./aarch64-softmmu/qemu-system-aarch64 device=integrator_core accel=tcg
INFO: test case: machine=isapc binary=./i386-softmmu/qemu-system-i386 device=i82550 accel=kvm
INFO: test case: machine=xlnx-ep108 binary=./aarch64-softmmu/qemu-system-aarch64 device=digic accel=tcg
qemu received signal 6: -S -machine xlnx-ep108,accel=tcg -device digic
ERROR: failed: machine=xlnx-ep108 binary=./aarch64-softmmu/qemu-system-aarch64 device=digic accel=tcg
ERROR: cmdline: ./aarch64-softmmu/qemu-system-aarch64 -S -machine xlnx-ep108,accel=tcg -device digic
ERROR: log: audio: Could not init `oss' audio driver
ERROR: log: Unexpected error in qemu_chr_fe_init() at /root/qemu-build/chardev/char.c:512:
ERROR: log: qemu-system-aarch64: -device digic: Device 'serial0' is in use
ERROR: exit code: -6
INFO: test case: machine=raspi2 binary=./arm-softmmu/qemu-system-arm device=sd-card accel=tcg
INFO: success: ./arm-softmmu/qemu-system-arm -S -machine raspi2,accel=tcg -device sd-card
[...]
Eduardo Habkost (3):
qemu.py: Don't set _popen=None on error/shutdown
qemu.py: Add QEMUMachine.exitcode() method
scripts: Test script to look for -device crashes
scripts/device-crash-test.py | 520 +++++++++++++++++++++++++++++++++++++++++++
scripts/qemu.py | 16 +-
2 files changed, 531 insertions(+), 5 deletions(-)
create mode 100755 scripts/device-crash-test.py
--
2.11.0.259.g40922b1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown
2017-05-13 3:33 [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device Eduardo Habkost
@ 2017-05-13 3:33 ` Eduardo Habkost
2017-05-23 14:23 ` Markus Armbruster
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 2/3] qemu.py: Add QEMUMachine.exitcode() method Eduardo Habkost
` (2 subsequent siblings)
3 siblings, 1 reply; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-13 3:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Thomas Huth, Marcel Apfelbaum, Markus Armbruster
Keep the Popen object around to we can query its exit code later.
To keep the existing 'self._popen is None' checks working, add a
is_running() method, that will check if the process is still running.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
scripts/qemu.py | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/scripts/qemu.py b/scripts/qemu.py
index 6d1b6230b7..16934f1e02 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -85,8 +85,11 @@ class QEMUMachine(object):
return
raise
+ def is_running(self):
+ return self._popen and (self._popen.returncode is None)
+
def get_pid(self):
- if not self._popen:
+ if not self.is_running():
return None
return self._popen.pid
@@ -128,16 +131,16 @@ class QEMUMachine(object):
stderr=subprocess.STDOUT, shell=False)
self._post_launch()
except:
- if self._popen:
+ if self.is_running():
self._popen.kill()
+ self._popen.wait()
self._load_io_log()
self._post_shutdown()
- self._popen = None
raise
def shutdown(self):
'''Terminate the VM and clean up'''
- if not self._popen is None:
+ if self.is_running():
try:
self._qmp.cmd('quit')
self._qmp.close()
@@ -149,7 +152,6 @@ class QEMUMachine(object):
sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode, ' '.join(self._args)))
self._load_io_log()
self._post_shutdown()
- self._popen = None
underscore_to_dash = string.maketrans('_', '-')
def qmp(self, cmd, conv_keys=True, **args):
--
2.11.0.259.g40922b1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v2 2/3] qemu.py: Add QEMUMachine.exitcode() method
2017-05-13 3:33 [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown Eduardo Habkost
@ 2017-05-13 3:33 ` Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes Eduardo Habkost
2017-05-13 4:05 ` [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device no-reply
3 siblings, 0 replies; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-13 3:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Thomas Huth, Marcel Apfelbaum, Markus Armbruster
Allow the exit code of QEMU to be queried by scripts.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
scripts/qemu.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/qemu.py b/scripts/qemu.py
index 16934f1e02..ebe1c4b919 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -88,6 +88,10 @@ class QEMUMachine(object):
def is_running(self):
return self._popen and (self._popen.returncode is None)
+ def exitcode(self):
+ if self._popen:
+ return self._popen.returncode
+
def get_pid(self):
if not self.is_running():
return None
--
2.11.0.259.g40922b1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes
2017-05-13 3:33 [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 2/3] qemu.py: Add QEMUMachine.exitcode() method Eduardo Habkost
@ 2017-05-13 3:33 ` Eduardo Habkost
2017-05-23 14:52 ` Markus Armbruster
2017-05-13 4:05 ` [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device no-reply
3 siblings, 1 reply; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-13 3:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Thomas Huth, Marcel Apfelbaum, Markus Armbruster
Test code to check if we can crash QEMU using -device. It will
test all accel/machine/device combinations by default, which may
take a few hours (it's more than 90k test cases). There's a "-r"
option that makes it test a random sample of combinations.
The scripts contains a whitelist for: 1) known error messages
that make QEMU exit cleanly; 2) known QEMU crashes.
This is the behavior when the script finds a failure:
* Known clean (exitcode=1) error messages generate INFO messages
(visible only in verbose mode), to make script output shorter
* Unknown clean error messages generate warnings
(visible by default)
* Known crashes generate error messages, but are not fatal
* Unknown crashes generate fatal error messages
I'm unsure about the need to maintain a list of known clean error
messages, but I wanted to at least document all existing failure
cases to use as base to build more comprehensive test code.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
Changes v1 -> v2:
* New whitelist entries:
* "could not find stage1 bootloader"
* Segfaults when using devices: a15mpcore_priv, sb16, cs4231a, arm-gicv3
* Format "success" line using formatTestCase(), and using DEBUg
loglevel
* Reword "test case:" line with "running test case:", for clarity
* Fix "pc-.*" whitelist to include "q35" too
* Add --devtype option to test only a specific device type
* Send all log messages to stdout instead of stderr
* Avoid printing "obsolete whitelist entry?" messages if we know
we are not testing every single accel/machine/device
combination
* --quick mode, to skip cases where failures are always expected,
and to print a warning in case we don't get an expected failure
* Use qemu.QEMUMachine instead of qtest.QEMUQtestMachine, as we don't
use any of the QEMUQtestMachine features
* Fix handling of multiple '-t' options
* Simplify code that generate random sample of test cases
---
scripts/device-crash-test.py | 520 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 520 insertions(+)
create mode 100755 scripts/device-crash-test.py
diff --git a/scripts/device-crash-test.py b/scripts/device-crash-test.py
new file mode 100755
index 0000000000..550da70ec7
--- /dev/null
+++ b/scripts/device-crash-test.py
@@ -0,0 +1,520 @@
+#!/usr/bin/env python2.7
+#
+# Run QEMU with all combinations of -machine and -device types,
+# check for crashes and unexpected errors.
+#
+# Copyright (c) 2017 Red Hat Inc
+#
+# Author:
+# Eduardo Habkost <ehabkost@redhat.com>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys, os, glob
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
+
+from itertools import chain
+from qemu import QEMUMachine
+import logging, traceback, re, random, argparse
+
+logger = logging.getLogger('device-crash-test')
+dbg = logger.debug
+
+# Valid whitelist entry keys:
+# - accel: regexp, full match only
+# - machine: regexp, full match only
+# - device: regexp, full match only
+# - log: regexp, partial match allowed
+# - exitcode: if not present, defaults to 1. If None, matches any exitcode
+# - warn: if True, matching failures will be logged as warnings
+# - expected: if True, QEMU is expected to always fail every time
+# when testing the corresponding test case
+ERROR_WHITELIST = [
+ # Machines that won't work out of the box:
+ # MACHINE | ERROR MESSAGE
+ dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
+ dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
+ dict(machine='leon3_generic', expected=True), # Can't read bios image (null)
+
+ # devices that don't work out of the box because they require extra options to "-device DEV":
+ # DEVICE | ERROR MESSAGE
+ dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
+ dict(device='ARM,bitband-memory', expected=True), # source-memory property not set
+ dict(device='arm.cortex-a9-global-timer', expected=True), # a9_gtimer_realize: num-cpu must be between 1 and 4
+ dict(device='arm_mptimer', expected=True), # num-cpu must be between 1 and 4
+ dict(device='armv7m', expected=True), # memory property was not set
+ dict(device='aspeed.scu', expected=True), # Unknown silicon revision: 0x0
+ dict(device='aspeed.sdmc', expected=True), # Unknown silicon revision: 0x0
+ dict(device='bcm2835-dma', expected=True), # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
+ dict(device='bcm2835-fb', expected=True), # bcm2835_fb_realize: required vcram-base property not set
+ dict(device='bcm2835-mbox', expected=True), # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
+ dict(device='bcm2835-peripherals', expected=True), # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
+ dict(device='bcm2835-property', expected=True), # bcm2835_property_realize: required fb link not found: Property '.fb' not found
+ dict(device='bcm2835_gpio', expected=True), # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
+ dict(device='bcm2836', expected=True), # bcm2836_realize: required ram link not found: Property '.ram' not found
+ dict(device='cfi.pflash01', expected=True), # attribute "sector-length" not specified or zero.
+ dict(device='cfi.pflash02', expected=True), # attribute "sector-length" not specified or zero.
+ dict(device='icp', expected=True), # icp_realize: required link 'xics' not found: Property '.xics' not found
+ dict(device='ics', expected=True), # ics_base_realize: required link 'xics' not found: Property '.xics' not found
+ dict(device='ide-drive', expected=True), # No drive specified
+ dict(device='ide-hd', expected=True), # No drive specified
+ dict(device='ipmi-bmc-extern', expected=True), # IPMI external bmc requires chardev attribute
+ dict(device='isa-debugcon', expected=True), # Can't create serial device, empty char device
+ dict(device='isa-ipmi-bt', expected=True), # IPMI device requires a bmc attribute to be set
+ dict(device='isa-ipmi-kcs', expected=True), # IPMI device requires a bmc attribute to be set
+ dict(device='isa-parallel', expected=True), # Can't create serial device, empty char device
+ dict(device='isa-serial', expected=True), # Can't create serial device, empty char device
+ dict(device='ivshmem', expected=True), # You must specify either 'shm' or 'chardev'
+ dict(device='ivshmem-doorbell', expected=True), # You must specify a 'chardev'
+ dict(device='ivshmem-plain', expected=True), # You must specify a 'memdev'
+ dict(device='kvm-pci-assign', expected=True), # no host device specified
+ dict(device='loader', expected=True), # please include valid arguments
+ dict(device='nand', expected=True), #' Unsupported NAND block size 0x1
+ dict(device='nvdimm', expected=True), # 'memdev' property is not set
+ dict(device='nvme', expected=True), # Device initialization failed
+ dict(device='pc-dimm', expected=True), # 'memdev' property is not set
+ dict(device='pci-bridge', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ dict(device='pci-bridge-seat', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ dict(device='pci-serial', expected=True), # Can't create serial device, empty char device
+ dict(device='pci-serial-2x', expected=True), # Can't create serial device, empty char device
+ dict(device='pci-serial-4x', expected=True), # Can't create serial device, empty char device
+ dict(device='pxa2xx-dma', expected=True), # channels value invalid
+ dict(device='pxb', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ dict(device='scsi-block', expected=True), # drive property not set
+ dict(device='scsi-disk', expected=True), # drive property not set
+ dict(device='scsi-generic', expected=True), # drive property not set
+ dict(device='scsi-hd', expected=True), # drive property not set
+ dict(device='spapr-pci-host-bridge', expected=True), # BUID not specified for PHB
+ dict(device='spapr-pci-vfio-host-bridge', expected=True), # BUID not specified for PHB
+ dict(device='spapr-rng', expected=True), #' spapr-rng needs an RNG backend!
+ dict(device='spapr-vty', expected=True), # chardev property not set
+ dict(device='tpm-tis', expected=True), # tpm_tis: backend driver with id (null) could not be found
+ dict(device='unimplemented-device', expected=True), # property 'size' not specified or zero
+ dict(device='usb-braille', expected=True), # Property chardev is required
+ dict(device='usb-mtp', expected=True), # x-root property must be configured
+ dict(device='usb-redir', expected=True), # Parameter 'chardev' is missing
+ dict(device='usb-serial', expected=True), # Property chardev is required
+ dict(device='usb-storage', expected=True), # drive property not set
+ dict(device='vfio-amd-xgbe', expected=True), # -device vfio-amd-xgbe: vfio error: wrong host device name
+ dict(device='vfio-calxeda-xgmac', expected=True), # -device vfio-calxeda-xgmac: vfio error: wrong host device name
+ dict(device='vfio-pci', expected=True), # No provided host device
+ dict(device='vfio-pci-igd-lpc-bridge', expected=True), # VFIO dummy ISA/LPC bridge must have address 1f.0
+ dict(device='vhost-scsi.*', expected=True), # vhost-scsi: missing wwpn
+ dict(device='vhost-vsock-device', expected=True), # guest-cid property must be greater than 2
+ dict(device='vhost-vsock-pci', expected=True), # guest-cid property must be greater than 2
+ dict(device='virtio-9p-ccw', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
+ dict(device='virtio-9p-device', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
+ dict(device='virtio-9p-pci', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
+ dict(device='virtio-blk-ccw', expected=True), # drive property not set
+ dict(device='virtio-blk-device', expected=True), # drive property not set
+ dict(device='virtio-blk-device', expected=True), # drive property not set
+ dict(device='virtio-blk-pci', expected=True), # drive property not set
+ dict(device='virtio-crypto-ccw', expected=True), # 'cryptodev' parameter expects a valid object
+ dict(device='virtio-crypto-device', expected=True), # 'cryptodev' parameter expects a valid object
+ dict(device='virtio-crypto-pci', expected=True), # 'cryptodev' parameter expects a valid object
+ dict(device='virtio-input-host-device', expected=True), # evdev property is required
+ dict(device='virtio-input-host-pci', expected=True), # evdev property is required
+ dict(device='xen-pvdevice', expected=True), # Device ID invalid, it must always be supplied
+ dict(device='vhost-vsock-ccw', expected=True), # guest-cid property must be greater than 2
+ dict(device='ALTR.timer', expected=True), # "clock-frequency" property must be provided
+ dict(device='zpci', expected=True), # target must be defined
+
+ # ioapic devices are already created by pc and will fail:
+ dict(machine='q35|pc.*', device='kvm-ioapic', expected=True), # Only 1 ioapics allowed
+ dict(machine='q35|pc.*', device='ioapic', expected=True), # Only 1 ioapics allowed
+
+ # KVM-specific devices shouldn't be tried without accel=kvm:
+ dict(accel='(?!kvm).*', device='kvmclock', expected=True),
+ dict(accel='(?!kvm).*', device='kvm-pci-assign', expected=True),
+
+ # xen-specific machines and devices:
+ dict(accel='(?!xen).*', machine='xen.*', expected=True),
+ dict(accel='(?!xen).*', device='xen-.*', expected=True),
+
+ # KNOWN CRASHES:
+ # known crashes will generate error messages, but won't be fatal:
+ dict(exitcode=-6, log=r"Device 'serial0' is in use", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"qemu_net_client_setup: Assertion `!peer->peer' failed", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r'RAMBlock "[\w.-]+" already registered', loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"find_ram_offset: Assertion `size != 0' failed.", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"add_cpreg_to_hashtable: code should not be reached", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"qemu_alloc_display: Assertion `surface->image != NULL' failed", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"Unexpected error in error_set_from_qdev_prop_error", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"Object .* is not an instance of type spapr-machine", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"Object .* is not an instance of type generic-pc-machine", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"Object .* is not an instance of type e500-ccsr", loglevel=logging.ERROR),
+ dict(exitcode=-6, log=r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", loglevel=logging.ERROR),
+ dict(exitcode=-11, device='stm32f205-soc', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='xlnx,zynqmp', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='mips-cps', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='gus', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='a9mpcore_priv', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='a15mpcore_priv', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='isa-serial', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='sb16', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='cs4231a', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, device='arm-gicv3', loglevel=logging.ERROR, expected=True),
+ dict(exitcode=-11, machine='isapc', device='.*-iommu', loglevel=logging.ERROR, expected=True),
+
+ # Some error messages that are common on multiple devices/machines:
+ dict(log=r"No '[\w-]+' bus found for device '[\w-]+'"),
+ dict(log=r"images* must be given with the 'pflash' parameter"),
+ dict(log=r"(Guest|ROM|Flash|Kernel) image must be specified"),
+ dict(log=r"[cC]ould not load [\w ]+ (BIOS|bios) '[\w-]+\.bin'"),
+ dict(log=r"Couldn't find rom image '[\w-]+\.bin'"),
+ dict(log=r"speed mismatch trying to attach usb device"),
+ dict(log=r"Can't create a second ISA bus"),
+ dict(log=r"duplicate fw_cfg file name"),
+ # sysbus-related error messages: most machines reject most dynamic sysbus devices:
+ #TODO: expected=True entries for unsupported sysbus devices
+ dict(log=r"Option '-device [\w.,-]+' cannot be handled by this machine"),
+ dict(log=r"Device [\w.,-]+ is not supported by this machine yet"),
+ dict(log=r"Device [\w.,-]+ can not be dynamically instantiated"),
+ dict(log=r"Platform Bus: Can not fit MMIO region of size "),
+ # other more specific errors we will ignore:
+ dict(device='allwinner-a10', log="Unsupported NIC model:"),
+ dict(device='.*-spapr-cpu-core', log=r"CPU core type should be"),
+ dict(log=r"MSI(-X)? is not supported by interrupt controller"),
+ dict(log=r"pxb-pcie? devices cannot reside on a PCIe? bus"),
+ dict(log=r"Ignoring smp_cpus value"),
+ dict(log=r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"),
+ dict(log=r"This CPU requires a smaller page size than the system is using"),
+ dict(log=r"MSI-X support is mandatory in the S390 architecture"),
+ dict(log=r"rom check and register reset failed"),
+ dict(log=r"Unable to initialize GIC, CPUState for CPU#0 not valid"),
+ dict(log=r"Multiple VT220 operator consoles are not supported"),
+ dict(log=r"core 0 already populated"),
+ dict(log=r"could not find stage1 bootloader"),
+
+ # other exitcode=1 failures not listed above will generate warnings:
+ dict(exitcode=1, loglevel=logging.WARN),
+
+ # everything else (including SIGABRT and SIGSEGV) will be a fatal error:
+ dict(exitcode=None, fatal=True, loglevel=logging.FATAL),
+]
+
+def whitelistMatch(wl, f):
+ t = f.get('testcase', {})
+ return (not wl.has_key('machine') \
+ or not t.has_key('machine') \
+ or re.match(wl['machine'] +'$', t['machine'])) \
+ and (not wl.has_key('accel') \
+ or not t.has_key('accel') \
+ or re.match(wl['accel'] +'$', t['accel'])) \
+ and (not wl.has_key('device') \
+ or not t.has_key('device') \
+ or re.match(wl['device'] +'$', t['device'])) \
+ and (wl.get('exitcode', 1) is None \
+ or not f.has_key('exitcode')
+ or f['exitcode'] == wl.get('exitcode', 1)) \
+ and (not wl.has_key('log') \
+ or not f.has_key('log') \
+ or re.search(wl['log'], f['log'], re.MULTILINE))
+
+def checkWhitelist(f):
+ """Look up whitelist entry for failure dictionary
+
+ Returns index in ERROR_WHITELIST
+ """
+ for i,wl in enumerate(ERROR_WHITELIST):
+ #dbg("whitelist entry: %r", wl)
+ if whitelistMatch(wl, f):
+ return i, wl
+
+ raise Exception("this should never happen")
+
+def qemuOptsEscape(s):
+ return s.replace(",", ",,")
+
+def formatTestCase(t):
+ return ' '.join('%s=%s' % (k, v) for k,v in t.items())
+
+def qomListTypeNames(vm, **kwargs):
+ """Run qom-list-types QMP command, return type names"""
+ types = vm.command('qom-list-types', **kwargs)
+ return [t['name'] for t in types]
+
+def infoQDM(vm):
+ """Parse 'info qdm' output"""
+ args = {'command-line': 'info qdm'}
+ devhelp = vm.command('human-monitor-command', **args)
+ for l in devhelp.split('\n'):
+ l = l.strip()
+ if l == '' or l.endswith(':'):
+ continue
+ d = {'name': re.search(r'name "([^"]+)"', l).group(1),
+ 'no-user': (re.search(', no-user', l) is not None)}
+ #dbg('info qdm item: %r', d)
+ yield d
+
+
+class QemuBinaryInfo:
+ def __init__(self, binary, devtype):
+ if devtype is None:
+ devtype = 'device'
+
+ dbg("devtype: %r", devtype)
+ args = ['-S', '-machine', 'none,accel=kvm:tcg']
+ dbg("querying info for QEMU binary: %s", binary)
+ vm = QEMUMachine(binary=binary, args=args)
+ vm.launch()
+ try:
+ self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False))
+ # there's no way to query cannot_instantiate_with_device_add_yet using QMP,
+ # so use 'info qdm':
+ self.no_user_devs = set([d['name'] for d in infoQDM(vm, ) if d['no-user']])
+ self.machines = list(m['name'] for m in vm.command('query-machines'))
+ self.user_devs = self.alldevs.difference(self.no_user_devs)
+ self.kvm_available = vm.command('query-kvm')['enabled']
+ finally:
+ vm.shutdown()
+
+BINARY_INFO = {}
+def getBinaryInfo(args, binary):
+ if not BINARY_INFO.has_key(binary):
+ BINARY_INFO[binary] = QemuBinaryInfo(binary, args.devtype)
+ return BINARY_INFO[binary]
+
+def checkOneCase(args, testcase):
+ """Check one specific case
+
+ Returns a dictionary containing failure information on error,
+ or None on success
+ """
+ binary = testcase['binary']
+ accel = testcase['accel']
+ machine = testcase['machine']
+ device = testcase['device']
+ info = getBinaryInfo(args, binary)
+
+ dbg("will test: %r", testcase)
+
+ args = ['-S', '-machine', '%s,accel=%s' % (machine, accel),
+ '-device', qemuOptsEscape(device)]
+ cmdline = ' '.join([binary] + args)
+ dbg("will launch QEMU: %s", cmdline)
+ vm = QEMUMachine(binary=binary, args=args)
+
+ exception = None
+ try:
+ vm.launch()
+ except KeyboardInterrupt:
+ raise
+ except Exception,e:
+ exception = e
+ finally:
+ vm.shutdown()
+ ec = vm.exitcode()
+ log = vm.get_log()
+
+ if exception is not None or ec != 0:
+ f = dict(exception = exception,
+ exitcode = ec,
+ log = log,
+ testcase = testcase,
+ cmdline=cmdline)
+ return f
+
+def binariesToTest(args, testcase):
+ if args.qemu:
+ r = args.qemu
+ else:
+ r = glob.glob('./*-softmmu/qemu-system-*')
+ return r
+
+def accelsToTest(args, testcase):
+ if getBinaryInfo(args, testcase['binary']).kvm_available:
+ yield 'kvm'
+ yield 'tcg'
+
+def machinesToTest(args, testcase):
+ return getBinaryInfo(args, testcase['binary']).machines
+
+def devicesToTest(args, testcase):
+ return getBinaryInfo(args, testcase['binary']).user_devs
+
+TESTCASE_VARIABLES = [
+ ('binary', binariesToTest),
+ ('accel', accelsToTest),
+ ('machine', machinesToTest),
+ ('device', devicesToTest),
+]
+
+def genCases1(args, testcases, var, fn):
+ """Generate new testcases for one variable
+
+ If an existing item already has a variable set, don't
+ generate new items and just return it directly. This
+ allows the "-t" command-line option to be used to choose
+ a specific test case.
+ """
+ for testcase in testcases:
+ t = testcase.copy
+ if testcase.has_key(var):
+ yield testcase.copy()
+ else:
+ for i in fn(args, testcase):
+ t = testcase.copy()
+ t[var] = i
+ yield t
+
+def genCases(args, testcase):
+ """Generate test cases for all variables
+ """
+ cases = [testcase.copy()]
+ for var,fn in TESTCASE_VARIABLES:
+ dbg("var: %r, fn: %r", var, fn)
+ cases = genCases1(args, cases, var, fn)
+ return cases
+
+def genAllCases(args, testcase):
+ return genCases(args, testcase)
+
+def pickRandomCase(args, testcase):
+ cases = list(genCases(args, testcase))
+ if cases:
+ assert len(cases) == 1
+ return cases[0]
+
+def casesToTest(args, testcase):
+ cases = genAllCases(args, testcase)
+ if args.random:
+ cases = random.sample(list(cases), args.random)
+ if args.debug:
+ cases = list(cases)
+ dbg("%d test cases to test", len(cases))
+ if args.shuffle:
+ cases = list(cases)
+ random.shuffle(cases)
+ return cases
+
+def logFailure(f, level):
+ t = f['testcase']
+ logger.log(level, "failed: %s", formatTestCase(t))
+ logger.log(level, "cmdline: %s", f['cmdline'])
+ for l in f['log'].strip().split('\n'):
+ logger.log(level, "log: %s", l)
+ logger.log(level, "exit code: %r", f['exitcode'])
+
+def main():
+ parser = argparse.ArgumentParser(description="QEMU -device crash test")
+ parser.add_argument('-t', metavar='KEY=VALUE', nargs='*',
+ help="Limit test cases to KEY=VALUE",
+ action='append', dest='testcases', default=[])
+ parser.add_argument('-d', '--debug', action='store_true',
+ help='debug output')
+ parser.add_argument('-v', '--verbose', action='store_true',
+ help='verbose output')
+ parser.add_argument('-r', '--random', type=int, metavar='COUNT',
+ help='run a random sample of COUNT test cases',
+ default=0)
+ parser.add_argument('--shuffle', action='store_true',
+ help='Run test cases in random order')
+ parser.add_argument('--dry-run', action='store_true',
+ help="Don't run any tests, just generate list")
+ parser.add_argument('-D', '--devtype', metavar='TYPE',
+ help="Test only device types that implement TYPE")
+ parser.add_argument('-Q', '--quick', action='store_true',
+ help="Quick mode, skip test cases that are expected to fail")
+ parser.add_argument('qemu', nargs='*', metavar='QEMU',
+ help='QEMU binary to run')
+ args = parser.parse_args()
+
+ if args.debug:
+ lvl = logging.DEBUG
+ elif args.verbose:
+ lvl = level=logging.INFO
+ else:
+ lvl = level=logging.WARN
+ logging.basicConfig(stream=sys.stdout, level=lvl, format='%(levelname)s: %(message)s')
+
+
+ interrupted = False
+ fatal_failures = []
+ wl_stats = {}
+ skipped = 0
+ total = 0
+
+ tc = {}
+ dbg("testcases: %r", args.testcases)
+ if args.testcases:
+ for t in chain(*args.testcases):
+ for kv in t.split():
+ k,v = kv.split('=', 1)
+ tc[k] = v
+
+ if len(binariesToTest(args, tc)) == 0:
+ print >>sys.stderr, "No QEMU binary found"
+ parser.print_usage(sys.stderr)
+ return 1
+
+ for t in casesToTest(args, tc):
+ logger.info("running test case: %s", formatTestCase(t))
+ total += 1
+
+ _,expected_match = checkWhitelist({'testcase': t})
+ if args.quick and expected_match.get('expected'):
+ dbg("Skipped: %s", formatTestCase(t))
+ skipped += 1
+ continue
+
+ if args.dry_run:
+ continue
+
+ try:
+ f = checkOneCase(args, t)
+ except KeyboardInterrupt:
+ interrupted = True
+ break
+
+ if f:
+ i,wl = checkWhitelist(f)
+ dbg("testcase: %r, whitelist match: %r", t, wl)
+ wl_stats.setdefault(i, []).append(f)
+ logFailure(f, wl.get('loglevel', logging.DEBUG))
+ if wl.get('fatal'):
+ fatal_failures.append(f)
+ else:
+ dbg("success: %s", formatTestCase(t))
+ if expected_match.get('expected'):
+ logger.warn("Didn't fail as expected: %s", formatTestCase(t))
+
+ logger.info("Total: %d test cases", total)
+ if skipped:
+ logger.info("Skipped %d test cases", skipped)
+
+ # tell us about obsolete whitelist entries so we can clean it up after
+ # bugs are fixed:
+ if not (interrupted or args.random or args.dry_run \
+ or args.testcases or args.qemu or args.devtype):
+ for i,wl in enumerate(ERROR_WHITELIST):
+ if not wl_stats.get(i) and not wl.get('fatal'):
+ logger.info("Obsolete whitelist entry? %r", wl)
+
+ stats = sorted([(len(wl_stats[i]), i) for i in wl_stats])
+ for count,i in stats:
+ dbg("whitelist entry stats: %d: %r", count, ERROR_WHITELIST[i])
+
+ if fatal_failures:
+ for f in fatal_failures:
+ t = f['testcase']
+ logger.error("Fatal failure: %s", formatTestCase(t))
+ logger.error("Fatal failures on some machine/device combinations")
+ return 1
+
+if __name__ == '__main__':
+ sys.exit(main())
--
2.11.0.259.g40922b1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device
2017-05-13 3:33 [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device Eduardo Habkost
` (2 preceding siblings ...)
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes Eduardo Habkost
@ 2017-05-13 4:05 ` no-reply
3 siblings, 0 replies; 12+ messages in thread
From: no-reply @ 2017-05-13 4:05 UTC (permalink / raw)
To: ehabkost; +Cc: famz, qemu-devel, marcel, thuth, armbru
Hi,
This series seems to have some coding style problems. See output below for
more information:
Type: series
Message-id: 20170513033316.22395-1-ehabkost@redhat.com
Subject: [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device
=== TEST SCRIPT BEGIN ===
#!/bin/bash
BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0
git config --local diff.renamelimit 0
git config --local diff.renames True
commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done
exit $failed
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
67ffc03 scripts: Test script to look for -device crashes
0be0012 qemu.py: Add QEMUMachine.exitcode() method
2dea278 qemu.py: Don't set _popen=None on error/shutdown
=== OUTPUT BEGIN ===
Checking PATCH 1/3: qemu.py: Don't set _popen=None on error/shutdown...
Checking PATCH 2/3: qemu.py: Add QEMUMachine.exitcode() method...
Checking PATCH 3/3: scripts: Test script to look for -device crashes...
WARNING: line over 80 characters
#82: FILE: scripts/device-crash-test.py:47:
+ dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
ERROR: line over 90 characters
#83: FILE: scripts/device-crash-test.py:48:
+ dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
ERROR: line over 90 characters
#86: FILE: scripts/device-crash-test.py:51:
+ # devices that don't work out of the box because they require extra options to "-device DEV":
WARNING: line over 80 characters
#88: FILE: scripts/device-crash-test.py:53:
+ dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
WARNING: line over 80 characters
#89: FILE: scripts/device-crash-test.py:54:
+ dict(device='ARM,bitband-memory', expected=True), # source-memory property not set
ERROR: line over 90 characters
#90: FILE: scripts/device-crash-test.py:55:
+ dict(device='arm.cortex-a9-global-timer', expected=True), # a9_gtimer_realize: num-cpu must be between 1 and 4
WARNING: line over 80 characters
#91: FILE: scripts/device-crash-test.py:56:
+ dict(device='arm_mptimer', expected=True), # num-cpu must be between 1 and 4
WARNING: line over 80 characters
#92: FILE: scripts/device-crash-test.py:57:
+ dict(device='armv7m', expected=True), # memory property was not set
WARNING: line over 80 characters
#93: FILE: scripts/device-crash-test.py:58:
+ dict(device='aspeed.scu', expected=True), # Unknown silicon revision: 0x0
WARNING: line over 80 characters
#94: FILE: scripts/device-crash-test.py:59:
+ dict(device='aspeed.sdmc', expected=True), # Unknown silicon revision: 0x0
ERROR: line over 90 characters
#95: FILE: scripts/device-crash-test.py:60:
+ dict(device='bcm2835-dma', expected=True), # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
ERROR: line over 90 characters
#96: FILE: scripts/device-crash-test.py:61:
+ dict(device='bcm2835-fb', expected=True), # bcm2835_fb_realize: required vcram-base property not set
ERROR: line over 90 characters
#97: FILE: scripts/device-crash-test.py:62:
+ dict(device='bcm2835-mbox', expected=True), # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
ERROR: line over 90 characters
#98: FILE: scripts/device-crash-test.py:63:
+ dict(device='bcm2835-peripherals', expected=True), # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
ERROR: line over 90 characters
#99: FILE: scripts/device-crash-test.py:64:
+ dict(device='bcm2835-property', expected=True), # bcm2835_property_realize: required fb link not found: Property '.fb' not found
ERROR: line over 90 characters
#100: FILE: scripts/device-crash-test.py:65:
+ dict(device='bcm2835_gpio', expected=True), # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
ERROR: line over 90 characters
#101: FILE: scripts/device-crash-test.py:66:
+ dict(device='bcm2836', expected=True), # bcm2836_realize: required ram link not found: Property '.ram' not found
ERROR: line over 90 characters
#102: FILE: scripts/device-crash-test.py:67:
+ dict(device='cfi.pflash01', expected=True), # attribute "sector-length" not specified or zero.
ERROR: line over 90 characters
#103: FILE: scripts/device-crash-test.py:68:
+ dict(device='cfi.pflash02', expected=True), # attribute "sector-length" not specified or zero.
ERROR: line over 90 characters
#104: FILE: scripts/device-crash-test.py:69:
+ dict(device='icp', expected=True), # icp_realize: required link 'xics' not found: Property '.xics' not found
ERROR: line over 90 characters
#105: FILE: scripts/device-crash-test.py:70:
+ dict(device='ics', expected=True), # ics_base_realize: required link 'xics' not found: Property '.xics' not found
ERROR: line over 90 characters
#108: FILE: scripts/device-crash-test.py:73:
+ dict(device='ipmi-bmc-extern', expected=True), # IPMI external bmc requires chardev attribute
ERROR: line over 90 characters
#109: FILE: scripts/device-crash-test.py:74:
+ dict(device='isa-debugcon', expected=True), # Can't create serial device, empty char device
ERROR: line over 90 characters
#110: FILE: scripts/device-crash-test.py:75:
+ dict(device='isa-ipmi-bt', expected=True), # IPMI device requires a bmc attribute to be set
ERROR: line over 90 characters
#111: FILE: scripts/device-crash-test.py:76:
+ dict(device='isa-ipmi-kcs', expected=True), # IPMI device requires a bmc attribute to be set
ERROR: line over 90 characters
#112: FILE: scripts/device-crash-test.py:77:
+ dict(device='isa-parallel', expected=True), # Can't create serial device, empty char device
ERROR: line over 90 characters
#113: FILE: scripts/device-crash-test.py:78:
+ dict(device='isa-serial', expected=True), # Can't create serial device, empty char device
ERROR: line over 90 characters
#114: FILE: scripts/device-crash-test.py:79:
+ dict(device='ivshmem', expected=True), # You must specify either 'shm' or 'chardev'
WARNING: line over 80 characters
#115: FILE: scripts/device-crash-test.py:80:
+ dict(device='ivshmem-doorbell', expected=True), # You must specify a 'chardev'
WARNING: line over 80 characters
#116: FILE: scripts/device-crash-test.py:81:
+ dict(device='ivshmem-plain', expected=True), # You must specify a 'memdev'
WARNING: line over 80 characters
#117: FILE: scripts/device-crash-test.py:82:
+ dict(device='kvm-pci-assign', expected=True), # no host device specified
WARNING: line over 80 characters
#118: FILE: scripts/device-crash-test.py:83:
+ dict(device='loader', expected=True), # please include valid arguments
ERROR: line over 90 characters
#119: FILE: scripts/device-crash-test.py:84:
+ dict(device='nand', expected=True), #' Unsupported NAND block size 0x1
WARNING: line over 80 characters
#120: FILE: scripts/device-crash-test.py:85:
+ dict(device='nvdimm', expected=True), # 'memdev' property is not set
WARNING: line over 80 characters
#121: FILE: scripts/device-crash-test.py:86:
+ dict(device='nvme', expected=True), # Device initialization failed
WARNING: line over 80 characters
#122: FILE: scripts/device-crash-test.py:87:
+ dict(device='pc-dimm', expected=True), # 'memdev' property is not set
ERROR: line over 90 characters
#123: FILE: scripts/device-crash-test.py:88:
+ dict(device='pci-bridge', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
ERROR: line over 90 characters
#124: FILE: scripts/device-crash-test.py:89:
+ dict(device='pci-bridge-seat', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
ERROR: line over 90 characters
#125: FILE: scripts/device-crash-test.py:90:
+ dict(device='pci-serial', expected=True), # Can't create serial device, empty char device
ERROR: line over 90 characters
#126: FILE: scripts/device-crash-test.py:91:
+ dict(device='pci-serial-2x', expected=True), # Can't create serial device, empty char device
ERROR: line over 90 characters
#127: FILE: scripts/device-crash-test.py:92:
+ dict(device='pci-serial-4x', expected=True), # Can't create serial device, empty char device
WARNING: line over 80 characters
#128: FILE: scripts/device-crash-test.py:93:
+ dict(device='pxa2xx-dma', expected=True), # channels value invalid
ERROR: line over 90 characters
#129: FILE: scripts/device-crash-test.py:94:
+ dict(device='pxb', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
WARNING: line over 80 characters
#130: FILE: scripts/device-crash-test.py:95:
+ dict(device='scsi-block', expected=True), # drive property not set
WARNING: line over 80 characters
#131: FILE: scripts/device-crash-test.py:96:
+ dict(device='scsi-disk', expected=True), # drive property not set
WARNING: line over 80 characters
#132: FILE: scripts/device-crash-test.py:97:
+ dict(device='scsi-generic', expected=True), # drive property not set
WARNING: line over 80 characters
#133: FILE: scripts/device-crash-test.py:98:
+ dict(device='scsi-hd', expected=True), # drive property not set
WARNING: line over 80 characters
#134: FILE: scripts/device-crash-test.py:99:
+ dict(device='spapr-pci-host-bridge', expected=True), # BUID not specified for PHB
WARNING: line over 80 characters
#135: FILE: scripts/device-crash-test.py:100:
+ dict(device='spapr-pci-vfio-host-bridge', expected=True), # BUID not specified for PHB
ERROR: line over 90 characters
#136: FILE: scripts/device-crash-test.py:101:
+ dict(device='spapr-rng', expected=True), #' spapr-rng needs an RNG backend!
WARNING: line over 80 characters
#137: FILE: scripts/device-crash-test.py:102:
+ dict(device='spapr-vty', expected=True), # chardev property not set
ERROR: line over 90 characters
#138: FILE: scripts/device-crash-test.py:103:
+ dict(device='tpm-tis', expected=True), # tpm_tis: backend driver with id (null) could not be found
ERROR: line over 90 characters
#139: FILE: scripts/device-crash-test.py:104:
+ dict(device='unimplemented-device', expected=True), # property 'size' not specified or zero
WARNING: line over 80 characters
#140: FILE: scripts/device-crash-test.py:105:
+ dict(device='usb-braille', expected=True), # Property chardev is required
ERROR: line over 90 characters
#141: FILE: scripts/device-crash-test.py:106:
+ dict(device='usb-mtp', expected=True), # x-root property must be configured
WARNING: line over 80 characters
#142: FILE: scripts/device-crash-test.py:107:
+ dict(device='usb-redir', expected=True), # Parameter 'chardev' is missing
WARNING: line over 80 characters
#143: FILE: scripts/device-crash-test.py:108:
+ dict(device='usb-serial', expected=True), # Property chardev is required
WARNING: line over 80 characters
#144: FILE: scripts/device-crash-test.py:109:
+ dict(device='usb-storage', expected=True), # drive property not set
ERROR: line over 90 characters
#145: FILE: scripts/device-crash-test.py:110:
+ dict(device='vfio-amd-xgbe', expected=True), # -device vfio-amd-xgbe: vfio error: wrong host device name
ERROR: line over 90 characters
#146: FILE: scripts/device-crash-test.py:111:
+ dict(device='vfio-calxeda-xgmac', expected=True), # -device vfio-calxeda-xgmac: vfio error: wrong host device name
WARNING: line over 80 characters
#147: FILE: scripts/device-crash-test.py:112:
+ dict(device='vfio-pci', expected=True), # No provided host device
ERROR: line over 90 characters
#148: FILE: scripts/device-crash-test.py:113:
+ dict(device='vfio-pci-igd-lpc-bridge', expected=True), # VFIO dummy ISA/LPC bridge must have address 1f.0
WARNING: line over 80 characters
#149: FILE: scripts/device-crash-test.py:114:
+ dict(device='vhost-scsi.*', expected=True), # vhost-scsi: missing wwpn
ERROR: line over 90 characters
#150: FILE: scripts/device-crash-test.py:115:
+ dict(device='vhost-vsock-device', expected=True), # guest-cid property must be greater than 2
ERROR: line over 90 characters
#151: FILE: scripts/device-crash-test.py:116:
+ dict(device='vhost-vsock-pci', expected=True), # guest-cid property must be greater than 2
ERROR: line over 90 characters
#152: FILE: scripts/device-crash-test.py:117:
+ dict(device='virtio-9p-ccw', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
ERROR: line over 90 characters
#153: FILE: scripts/device-crash-test.py:118:
+ dict(device='virtio-9p-device', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
ERROR: line over 90 characters
#154: FILE: scripts/device-crash-test.py:119:
+ dict(device='virtio-9p-pci', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
WARNING: line over 80 characters
#155: FILE: scripts/device-crash-test.py:120:
+ dict(device='virtio-blk-ccw', expected=True), # drive property not set
WARNING: line over 80 characters
#156: FILE: scripts/device-crash-test.py:121:
+ dict(device='virtio-blk-device', expected=True), # drive property not set
WARNING: line over 80 characters
#157: FILE: scripts/device-crash-test.py:122:
+ dict(device='virtio-blk-device', expected=True), # drive property not set
WARNING: line over 80 characters
#158: FILE: scripts/device-crash-test.py:123:
+ dict(device='virtio-blk-pci', expected=True), # drive property not set
ERROR: line over 90 characters
#159: FILE: scripts/device-crash-test.py:124:
+ dict(device='virtio-crypto-ccw', expected=True), # 'cryptodev' parameter expects a valid object
ERROR: line over 90 characters
#160: FILE: scripts/device-crash-test.py:125:
+ dict(device='virtio-crypto-device', expected=True), # 'cryptodev' parameter expects a valid object
ERROR: line over 90 characters
#161: FILE: scripts/device-crash-test.py:126:
+ dict(device='virtio-crypto-pci', expected=True), # 'cryptodev' parameter expects a valid object
WARNING: line over 80 characters
#162: FILE: scripts/device-crash-test.py:127:
+ dict(device='virtio-input-host-device', expected=True), # evdev property is required
WARNING: line over 80 characters
#163: FILE: scripts/device-crash-test.py:128:
+ dict(device='virtio-input-host-pci', expected=True), # evdev property is required
ERROR: line over 90 characters
#164: FILE: scripts/device-crash-test.py:129:
+ dict(device='xen-pvdevice', expected=True), # Device ID invalid, it must always be supplied
ERROR: line over 90 characters
#165: FILE: scripts/device-crash-test.py:130:
+ dict(device='vhost-vsock-ccw', expected=True), # guest-cid property must be greater than 2
ERROR: line over 90 characters
#166: FILE: scripts/device-crash-test.py:131:
+ dict(device='ALTR.timer', expected=True), # "clock-frequency" property must be provided
WARNING: line over 80 characters
#167: FILE: scripts/device-crash-test.py:132:
+ dict(device='zpci', expected=True), # target must be defined
WARNING: line over 80 characters
#170: FILE: scripts/device-crash-test.py:135:
+ dict(machine='q35|pc.*', device='kvm-ioapic', expected=True), # Only 1 ioapics allowed
WARNING: line over 80 characters
#171: FILE: scripts/device-crash-test.py:136:
+ dict(machine='q35|pc.*', device='ioapic', expected=True), # Only 1 ioapics allowed
ERROR: line over 90 characters
#184: FILE: scripts/device-crash-test.py:149:
+ dict(exitcode=-6, log=r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", loglevel=logging.ERROR),
ERROR: line over 90 characters
#185: FILE: scripts/device-crash-test.py:150:
+ dict(exitcode=-6, log=r"qemu_net_client_setup: Assertion `!peer->peer' failed", loglevel=logging.ERROR),
WARNING: line over 80 characters
#186: FILE: scripts/device-crash-test.py:151:
+ dict(exitcode=-6, log=r'RAMBlock "[\w.-]+" already registered', loglevel=logging.ERROR),
ERROR: line over 90 characters
#187: FILE: scripts/device-crash-test.py:152:
+ dict(exitcode=-6, log=r"find_ram_offset: Assertion `size != 0' failed.", loglevel=logging.ERROR),
ERROR: line over 90 characters
#188: FILE: scripts/device-crash-test.py:153:
+ dict(exitcode=-6, log=r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", loglevel=logging.ERROR),
ERROR: line over 90 characters
#189: FILE: scripts/device-crash-test.py:154:
+ dict(exitcode=-6, log=r"add_cpreg_to_hashtable: code should not be reached", loglevel=logging.ERROR),
ERROR: line over 90 characters
#190: FILE: scripts/device-crash-test.py:155:
+ dict(exitcode=-6, log=r"qemu_alloc_display: Assertion `surface->image != NULL' failed", loglevel=logging.ERROR),
ERROR: line over 90 characters
#191: FILE: scripts/device-crash-test.py:156:
+ dict(exitcode=-6, log=r"Unexpected error in error_set_from_qdev_prop_error", loglevel=logging.ERROR),
ERROR: line over 90 characters
#192: FILE: scripts/device-crash-test.py:157:
+ dict(exitcode=-6, log=r"Object .* is not an instance of type spapr-machine", loglevel=logging.ERROR),
ERROR: line over 90 characters
#193: FILE: scripts/device-crash-test.py:158:
+ dict(exitcode=-6, log=r"Object .* is not an instance of type generic-pc-machine", loglevel=logging.ERROR),
ERROR: line over 90 characters
#194: FILE: scripts/device-crash-test.py:159:
+ dict(exitcode=-6, log=r"Object .* is not an instance of type e500-ccsr", loglevel=logging.ERROR),
ERROR: line over 90 characters
#195: FILE: scripts/device-crash-test.py:160:
+ dict(exitcode=-6, log=r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", loglevel=logging.ERROR),
WARNING: line over 80 characters
#196: FILE: scripts/device-crash-test.py:161:
+ dict(exitcode=-11, device='stm32f205-soc', loglevel=logging.ERROR, expected=True),
WARNING: line over 80 characters
#197: FILE: scripts/device-crash-test.py:162:
+ dict(exitcode=-11, device='xlnx,zynqmp', loglevel=logging.ERROR, expected=True),
WARNING: line over 80 characters
#200: FILE: scripts/device-crash-test.py:165:
+ dict(exitcode=-11, device='a9mpcore_priv', loglevel=logging.ERROR, expected=True),
WARNING: line over 80 characters
#201: FILE: scripts/device-crash-test.py:166:
+ dict(exitcode=-11, device='a15mpcore_priv', loglevel=logging.ERROR, expected=True),
WARNING: line over 80 characters
#202: FILE: scripts/device-crash-test.py:167:
+ dict(exitcode=-11, device='isa-serial', loglevel=logging.ERROR, expected=True),
ERROR: line over 90 characters
#206: FILE: scripts/device-crash-test.py:171:
+ dict(exitcode=-11, machine='isapc', device='.*-iommu', loglevel=logging.ERROR, expected=True),
WARNING: line over 80 characters
#217: FILE: scripts/device-crash-test.py:182:
+ # sysbus-related error messages: most machines reject most dynamic sysbus devices:
ERROR: line over 90 characters
#229: FILE: scripts/device-crash-test.py:194:
+ dict(log=r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"),
WARNING: line over 80 characters
#311: FILE: scripts/device-crash-test.py:276:
+ self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False))
WARNING: line over 80 characters
#312: FILE: scripts/device-crash-test.py:277:
+ # there's no way to query cannot_instantiate_with_device_add_yet using QMP,
WARNING: line over 80 characters
#314: FILE: scripts/device-crash-test.py:279:
+ self.no_user_devs = set([d['name'] for d in infoQDM(vm, ) if d['no-user']])
WARNING: line over 80 characters
#315: FILE: scripts/device-crash-test.py:280:
+ self.machines = list(m['name'] for m in vm.command('query-machines'))
WARNING: line over 80 characters
#467: FILE: scripts/device-crash-test.py:432:
+ help="Quick mode, skip test cases that are expected to fail")
WARNING: line over 80 characters
#478: FILE: scripts/device-crash-test.py:443:
+ logging.basicConfig(stream=sys.stdout, level=lvl, format='%(levelname)s: %(message)s')
total: 59 errors, 50 warnings, 520 lines checked
Your patch has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===
Test command exited with code: 1
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown Eduardo Habkost
@ 2017-05-23 14:23 ` Markus Armbruster
2017-05-25 18:04 ` Eduardo Habkost
0 siblings, 1 reply; 12+ messages in thread
From: Markus Armbruster @ 2017-05-23 14:23 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: qemu-devel, Marcel Apfelbaum, Thomas Huth
Eduardo Habkost <ehabkost@redhat.com> writes:
> Keep the Popen object around to we can query its exit code later.
>
> To keep the existing 'self._popen is None' checks working, add a
> is_running() method, that will check if the process is still running.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Looks harmless enough. Have you tested the scripts using this module
still work?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes Eduardo Habkost
@ 2017-05-23 14:52 ` Markus Armbruster
2017-05-24 16:39 ` Eduardo Habkost
0 siblings, 1 reply; 12+ messages in thread
From: Markus Armbruster @ 2017-05-23 14:52 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: qemu-devel, Marcel Apfelbaum, Thomas Huth
Beware, my review is rather superficial.
Eduardo Habkost <ehabkost@redhat.com> writes:
> Test code to check if we can crash QEMU using -device. It will
> test all accel/machine/device combinations by default, which may
> take a few hours (it's more than 90k test cases). There's a "-r"
> option that makes it test a random sample of combinations.
>
> The scripts contains a whitelist for: 1) known error messages
> that make QEMU exit cleanly; 2) known QEMU crashes.
>
> This is the behavior when the script finds a failure:
>
> * Known clean (exitcode=1) error messages generate INFO messages
> (visible only in verbose mode), to make script output shorter
> * Unknown clean error messages generate warnings
> (visible by default)
> * Known crashes generate error messages, but are not fatal
> * Unknown crashes generate fatal error messages
>
> I'm unsure about the need to maintain a list of known clean error
> messages, but I wanted to at least document all existing failure
> cases to use as base to build more comprehensive test code.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> Changes v1 -> v2:
> * New whitelist entries:
> * "could not find stage1 bootloader"
> * Segfaults when using devices: a15mpcore_priv, sb16, cs4231a, arm-gicv3
> * Format "success" line using formatTestCase(), and using DEBUg
> loglevel
> * Reword "test case:" line with "running test case:", for clarity
> * Fix "pc-.*" whitelist to include "q35" too
> * Add --devtype option to test only a specific device type
> * Send all log messages to stdout instead of stderr
> * Avoid printing "obsolete whitelist entry?" messages if we know
> we are not testing every single accel/machine/device
> combination
> * --quick mode, to skip cases where failures are always expected,
> and to print a warning in case we don't get an expected failure
> * Use qemu.QEMUMachine instead of qtest.QEMUQtestMachine, as we don't
> use any of the QEMUQtestMachine features
> * Fix handling of multiple '-t' options
> * Simplify code that generate random sample of test cases
> ---
> scripts/device-crash-test.py | 520 +++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 520 insertions(+)
> create mode 100755 scripts/device-crash-test.py
>
> diff --git a/scripts/device-crash-test.py b/scripts/device-crash-test.py
> new file mode 100755
> index 0000000000..550da70ec7
> --- /dev/null
> +++ b/scripts/device-crash-test.py
> @@ -0,0 +1,520 @@
> +#!/usr/bin/env python2.7
> +#
> +# Run QEMU with all combinations of -machine and -device types,
> +# check for crashes and unexpected errors.
> +#
> +# Copyright (c) 2017 Red Hat Inc
> +#
> +# Author:
> +# Eduardo Habkost <ehabkost@redhat.com>
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2 of the License, or (at your option) any later version.
Any particular reason for "Lesser"?
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import sys, os, glob
> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
> +
> +from itertools import chain
> +from qemu import QEMUMachine
> +import logging, traceback, re, random, argparse
> +
> +logger = logging.getLogger('device-crash-test')
> +dbg = logger.debug
> +
> +# Valid whitelist entry keys:
> +# - accel: regexp, full match only
> +# - machine: regexp, full match only
> +# - device: regexp, full match only
> +# - log: regexp, partial match allowed
> +# - exitcode: if not present, defaults to 1. If None, matches any exitcode
> +# - warn: if True, matching failures will be logged as warnings
> +# - expected: if True, QEMU is expected to always fail every time
> +# when testing the corresponding test case
> +ERROR_WHITELIST = [
> + # Machines that won't work out of the box:
> + # MACHINE | ERROR MESSAGE
> + dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
What's wrong with dictionary displays?
{'expected': True, 'machine': 'niagara'}
> + dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
> + dict(machine='leon3_generic', expected=True), # Can't read bios image (null)
> +
> + # devices that don't work out of the box because they require extra options to "-device DEV":
> + # DEVICE | ERROR MESSAGE
> + dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
> + dict(device='ARM,bitband-memory', expected=True), # source-memory property not set
> + dict(device='arm.cortex-a9-global-timer', expected=True), # a9_gtimer_realize: num-cpu must be between 1 and 4
> + dict(device='arm_mptimer', expected=True), # num-cpu must be between 1 and 4
> + dict(device='armv7m', expected=True), # memory property was not set
> + dict(device='aspeed.scu', expected=True), # Unknown silicon revision: 0x0
> + dict(device='aspeed.sdmc', expected=True), # Unknown silicon revision: 0x0
> + dict(device='bcm2835-dma', expected=True), # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
> + dict(device='bcm2835-fb', expected=True), # bcm2835_fb_realize: required vcram-base property not set
> + dict(device='bcm2835-mbox', expected=True), # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
> + dict(device='bcm2835-peripherals', expected=True), # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
> + dict(device='bcm2835-property', expected=True), # bcm2835_property_realize: required fb link not found: Property '.fb' not found
> + dict(device='bcm2835_gpio', expected=True), # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
> + dict(device='bcm2836', expected=True), # bcm2836_realize: required ram link not found: Property '.ram' not found
> + dict(device='cfi.pflash01', expected=True), # attribute "sector-length" not specified or zero.
> + dict(device='cfi.pflash02', expected=True), # attribute "sector-length" not specified or zero.
> + dict(device='icp', expected=True), # icp_realize: required link 'xics' not found: Property '.xics' not found
> + dict(device='ics', expected=True), # ics_base_realize: required link 'xics' not found: Property '.xics' not found
> + dict(device='ide-drive', expected=True), # No drive specified
> + dict(device='ide-hd', expected=True), # No drive specified
> + dict(device='ipmi-bmc-extern', expected=True), # IPMI external bmc requires chardev attribute
> + dict(device='isa-debugcon', expected=True), # Can't create serial device, empty char device
> + dict(device='isa-ipmi-bt', expected=True), # IPMI device requires a bmc attribute to be set
> + dict(device='isa-ipmi-kcs', expected=True), # IPMI device requires a bmc attribute to be set
> + dict(device='isa-parallel', expected=True), # Can't create serial device, empty char device
> + dict(device='isa-serial', expected=True), # Can't create serial device, empty char device
> + dict(device='ivshmem', expected=True), # You must specify either 'shm' or 'chardev'
> + dict(device='ivshmem-doorbell', expected=True), # You must specify a 'chardev'
> + dict(device='ivshmem-plain', expected=True), # You must specify a 'memdev'
> + dict(device='kvm-pci-assign', expected=True), # no host device specified
> + dict(device='loader', expected=True), # please include valid arguments
> + dict(device='nand', expected=True), #' Unsupported NAND block size 0x1
> + dict(device='nvdimm', expected=True), # 'memdev' property is not set
> + dict(device='nvme', expected=True), # Device initialization failed
> + dict(device='pc-dimm', expected=True), # 'memdev' property is not set
> + dict(device='pci-bridge', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> + dict(device='pci-bridge-seat', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> + dict(device='pci-serial', expected=True), # Can't create serial device, empty char device
> + dict(device='pci-serial-2x', expected=True), # Can't create serial device, empty char device
> + dict(device='pci-serial-4x', expected=True), # Can't create serial device, empty char device
> + dict(device='pxa2xx-dma', expected=True), # channels value invalid
> + dict(device='pxb', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> + dict(device='scsi-block', expected=True), # drive property not set
> + dict(device='scsi-disk', expected=True), # drive property not set
> + dict(device='scsi-generic', expected=True), # drive property not set
> + dict(device='scsi-hd', expected=True), # drive property not set
> + dict(device='spapr-pci-host-bridge', expected=True), # BUID not specified for PHB
> + dict(device='spapr-pci-vfio-host-bridge', expected=True), # BUID not specified for PHB
> + dict(device='spapr-rng', expected=True), #' spapr-rng needs an RNG backend!
> + dict(device='spapr-vty', expected=True), # chardev property not set
> + dict(device='tpm-tis', expected=True), # tpm_tis: backend driver with id (null) could not be found
> + dict(device='unimplemented-device', expected=True), # property 'size' not specified or zero
> + dict(device='usb-braille', expected=True), # Property chardev is required
> + dict(device='usb-mtp', expected=True), # x-root property must be configured
> + dict(device='usb-redir', expected=True), # Parameter 'chardev' is missing
> + dict(device='usb-serial', expected=True), # Property chardev is required
> + dict(device='usb-storage', expected=True), # drive property not set
> + dict(device='vfio-amd-xgbe', expected=True), # -device vfio-amd-xgbe: vfio error: wrong host device name
> + dict(device='vfio-calxeda-xgmac', expected=True), # -device vfio-calxeda-xgmac: vfio error: wrong host device name
> + dict(device='vfio-pci', expected=True), # No provided host device
> + dict(device='vfio-pci-igd-lpc-bridge', expected=True), # VFIO dummy ISA/LPC bridge must have address 1f.0
> + dict(device='vhost-scsi.*', expected=True), # vhost-scsi: missing wwpn
> + dict(device='vhost-vsock-device', expected=True), # guest-cid property must be greater than 2
> + dict(device='vhost-vsock-pci', expected=True), # guest-cid property must be greater than 2
> + dict(device='virtio-9p-ccw', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> + dict(device='virtio-9p-device', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> + dict(device='virtio-9p-pci', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> + dict(device='virtio-blk-ccw', expected=True), # drive property not set
> + dict(device='virtio-blk-device', expected=True), # drive property not set
> + dict(device='virtio-blk-device', expected=True), # drive property not set
> + dict(device='virtio-blk-pci', expected=True), # drive property not set
> + dict(device='virtio-crypto-ccw', expected=True), # 'cryptodev' parameter expects a valid object
> + dict(device='virtio-crypto-device', expected=True), # 'cryptodev' parameter expects a valid object
> + dict(device='virtio-crypto-pci', expected=True), # 'cryptodev' parameter expects a valid object
> + dict(device='virtio-input-host-device', expected=True), # evdev property is required
> + dict(device='virtio-input-host-pci', expected=True), # evdev property is required
> + dict(device='xen-pvdevice', expected=True), # Device ID invalid, it must always be supplied
> + dict(device='vhost-vsock-ccw', expected=True), # guest-cid property must be greater than 2
> + dict(device='ALTR.timer', expected=True), # "clock-frequency" property must be provided
> + dict(device='zpci', expected=True), # target must be defined
> +
> + # ioapic devices are already created by pc and will fail:
> + dict(machine='q35|pc.*', device='kvm-ioapic', expected=True), # Only 1 ioapics allowed
> + dict(machine='q35|pc.*', device='ioapic', expected=True), # Only 1 ioapics allowed
> +
> + # KVM-specific devices shouldn't be tried without accel=kvm:
> + dict(accel='(?!kvm).*', device='kvmclock', expected=True),
> + dict(accel='(?!kvm).*', device='kvm-pci-assign', expected=True),
> +
> + # xen-specific machines and devices:
> + dict(accel='(?!xen).*', machine='xen.*', expected=True),
> + dict(accel='(?!xen).*', device='xen-.*', expected=True),
> +
> + # KNOWN CRASHES:
> + # known crashes will generate error messages, but won't be fatal:
> + dict(exitcode=-6, log=r"Device 'serial0' is in use", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"qemu_net_client_setup: Assertion `!peer->peer' failed", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r'RAMBlock "[\w.-]+" already registered', loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"find_ram_offset: Assertion `size != 0' failed.", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"add_cpreg_to_hashtable: code should not be reached", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"qemu_alloc_display: Assertion `surface->image != NULL' failed", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"Unexpected error in error_set_from_qdev_prop_error", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"Object .* is not an instance of type spapr-machine", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"Object .* is not an instance of type generic-pc-machine", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"Object .* is not an instance of type e500-ccsr", loglevel=logging.ERROR),
> + dict(exitcode=-6, log=r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", loglevel=logging.ERROR),
> + dict(exitcode=-11, device='stm32f205-soc', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='xlnx,zynqmp', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='mips-cps', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='gus', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='a9mpcore_priv', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='a15mpcore_priv', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='isa-serial', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='sb16', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='cs4231a', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, device='arm-gicv3', loglevel=logging.ERROR, expected=True),
> + dict(exitcode=-11, machine='isapc', device='.*-iommu', loglevel=logging.ERROR, expected=True),
> +
> + # Some error messages that are common on multiple devices/machines:
> + dict(log=r"No '[\w-]+' bus found for device '[\w-]+'"),
> + dict(log=r"images* must be given with the 'pflash' parameter"),
> + dict(log=r"(Guest|ROM|Flash|Kernel) image must be specified"),
> + dict(log=r"[cC]ould not load [\w ]+ (BIOS|bios) '[\w-]+\.bin'"),
> + dict(log=r"Couldn't find rom image '[\w-]+\.bin'"),
> + dict(log=r"speed mismatch trying to attach usb device"),
> + dict(log=r"Can't create a second ISA bus"),
> + dict(log=r"duplicate fw_cfg file name"),
> + # sysbus-related error messages: most machines reject most dynamic sysbus devices:
> + #TODO: expected=True entries for unsupported sysbus devices
> + dict(log=r"Option '-device [\w.,-]+' cannot be handled by this machine"),
> + dict(log=r"Device [\w.,-]+ is not supported by this machine yet"),
> + dict(log=r"Device [\w.,-]+ can not be dynamically instantiated"),
> + dict(log=r"Platform Bus: Can not fit MMIO region of size "),
> + # other more specific errors we will ignore:
> + dict(device='allwinner-a10', log="Unsupported NIC model:"),
> + dict(device='.*-spapr-cpu-core', log=r"CPU core type should be"),
> + dict(log=r"MSI(-X)? is not supported by interrupt controller"),
> + dict(log=r"pxb-pcie? devices cannot reside on a PCIe? bus"),
> + dict(log=r"Ignoring smp_cpus value"),
> + dict(log=r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"),
> + dict(log=r"This CPU requires a smaller page size than the system is using"),
> + dict(log=r"MSI-X support is mandatory in the S390 architecture"),
> + dict(log=r"rom check and register reset failed"),
> + dict(log=r"Unable to initialize GIC, CPUState for CPU#0 not valid"),
> + dict(log=r"Multiple VT220 operator consoles are not supported"),
> + dict(log=r"core 0 already populated"),
> + dict(log=r"could not find stage1 bootloader"),
> +
> + # other exitcode=1 failures not listed above will generate warnings:
> + dict(exitcode=1, loglevel=logging.WARN),
> +
> + # everything else (including SIGABRT and SIGSEGV) will be a fatal error:
> + dict(exitcode=None, fatal=True, loglevel=logging.FATAL),
> +]
How will we keep this whitelist up-to-date?
Hmm, there seems to be code for detecting obsolete entries. Can you
explain how it works, for dummies?
> +
> +def whitelistMatch(wl, f):
> + t = f.get('testcase', {})
> + return (not wl.has_key('machine') \
scripts/device-crash-test.py:212:19: W601 .has_key() is deprecated, use 'in'
Please feed this module to pylint and pep8, then use your judgement.
> + or not t.has_key('machine') \
> + or re.match(wl['machine'] +'$', t['machine'])) \
> + and (not wl.has_key('accel') \
> + or not t.has_key('accel') \
> + or re.match(wl['accel'] +'$', t['accel'])) \
> + and (not wl.has_key('device') \
> + or not t.has_key('device') \
> + or re.match(wl['device'] +'$', t['device'])) \
> + and (wl.get('exitcode', 1) is None \
> + or not f.has_key('exitcode')
> + or f['exitcode'] == wl.get('exitcode', 1)) \
Could this be simplified to
and (wl.get('exitcode', 1) is None
or f.get('exitcode') == wl.get('exitcode', 1))
?
> + and (not wl.has_key('log') \
> + or not f.has_key('log') \
> + or re.search(wl['log'], f['log'], re.MULTILINE))
> +
> +def checkWhitelist(f):
> + """Look up whitelist entry for failure dictionary
> +
> + Returns index in ERROR_WHITELIST
> + """
> + for i,wl in enumerate(ERROR_WHITELIST):
Umm, sure you need to wrap enumerate() around ERROR_WHITELIST?
> + #dbg("whitelist entry: %r", wl)
Debugging turd? More of the same below.
> + if whitelistMatch(wl, f):
> + return i, wl
> +
> + raise Exception("this should never happen")
> +
> +def qemuOptsEscape(s):
> + return s.replace(",", ",,")
> +
> +def formatTestCase(t):
> + return ' '.join('%s=%s' % (k, v) for k,v in t.items())
> +
> +def qomListTypeNames(vm, **kwargs):
> + """Run qom-list-types QMP command, return type names"""
> + types = vm.command('qom-list-types', **kwargs)
> + return [t['name'] for t in types]
> +
> +def infoQDM(vm):
> + """Parse 'info qdm' output"""
> + args = {'command-line': 'info qdm'}
> + devhelp = vm.command('human-monitor-command', **args)
> + for l in devhelp.split('\n'):
> + l = l.strip()
> + if l == '' or l.endswith(':'):
> + continue
> + d = {'name': re.search(r'name "([^"]+)"', l).group(1),
> + 'no-user': (re.search(', no-user', l) is not None)}
> + #dbg('info qdm item: %r', d)
> + yield d
The need for HMP is sad, but you already proposed patches to remove it.
> +
> +
> +class QemuBinaryInfo:
> + def __init__(self, binary, devtype):
> + if devtype is None:
> + devtype = 'device'
> +
> + dbg("devtype: %r", devtype)
> + args = ['-S', '-machine', 'none,accel=kvm:tcg']
> + dbg("querying info for QEMU binary: %s", binary)
> + vm = QEMUMachine(binary=binary, args=args)
> + vm.launch()
> + try:
> + self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False))
> + # there's no way to query cannot_instantiate_with_device_add_yet using QMP,
Update needed for the rename to user_creatable.
> + # so use 'info qdm':
> + self.no_user_devs = set([d['name'] for d in infoQDM(vm, ) if d['no-user']])
> + self.machines = list(m['name'] for m in vm.command('query-machines'))
> + self.user_devs = self.alldevs.difference(self.no_user_devs)
> + self.kvm_available = vm.command('query-kvm')['enabled']
> + finally:
> + vm.shutdown()
> +
> +BINARY_INFO = {}
> +def getBinaryInfo(args, binary):
> + if not BINARY_INFO.has_key(binary):
> + BINARY_INFO[binary] = QemuBinaryInfo(binary, args.devtype)
> + return BINARY_INFO[binary]
> +
> +def checkOneCase(args, testcase):
> + """Check one specific case
> +
> + Returns a dictionary containing failure information on error,
> + or None on success
> + """
> + binary = testcase['binary']
> + accel = testcase['accel']
> + machine = testcase['machine']
> + device = testcase['device']
> + info = getBinaryInfo(args, binary)
> +
> + dbg("will test: %r", testcase)
> +
> + args = ['-S', '-machine', '%s,accel=%s' % (machine, accel),
> + '-device', qemuOptsEscape(device)]
> + cmdline = ' '.join([binary] + args)
> + dbg("will launch QEMU: %s", cmdline)
> + vm = QEMUMachine(binary=binary, args=args)
> +
> + exception = None
> + try:
> + vm.launch()
> + except KeyboardInterrupt:
> + raise
> + except Exception,e:
> + exception = e
> + finally:
> + vm.shutdown()
> + ec = vm.exitcode()
> + log = vm.get_log()
> +
> + if exception is not None or ec != 0:
> + f = dict(exception = exception,
> + exitcode = ec,
> + log = log,
> + testcase = testcase,
> + cmdline=cmdline)
> + return f
Why not simply
return {'exception': exception,
'exitcode': ec,
'log': log,
'testcase': testcase,
'cmdline': cmdline}
?
> +
> +def binariesToTest(args, testcase):
> + if args.qemu:
> + r = args.qemu
> + else:
> + r = glob.glob('./*-softmmu/qemu-system-*')
> + return r
> +
> +def accelsToTest(args, testcase):
> + if getBinaryInfo(args, testcase['binary']).kvm_available:
> + yield 'kvm'
> + yield 'tcg'
If only "--accel help" showed just the ones that can actually work...
> +
> +def machinesToTest(args, testcase):
> + return getBinaryInfo(args, testcase['binary']).machines
> +
> +def devicesToTest(args, testcase):
> + return getBinaryInfo(args, testcase['binary']).user_devs
> +
> +TESTCASE_VARIABLES = [
> + ('binary', binariesToTest),
> + ('accel', accelsToTest),
> + ('machine', machinesToTest),
> + ('device', devicesToTest),
> +]
> +
> +def genCases1(args, testcases, var, fn):
> + """Generate new testcases for one variable
> +
> + If an existing item already has a variable set, don't
> + generate new items and just return it directly. This
> + allows the "-t" command-line option to be used to choose
> + a specific test case.
> + """
> + for testcase in testcases:
> + t = testcase.copy
> + if testcase.has_key(var):
> + yield testcase.copy()
> + else:
> + for i in fn(args, testcase):
> + t = testcase.copy()
> + t[var] = i
> + yield t
> +
> +def genCases(args, testcase):
> + """Generate test cases for all variables
> + """
> + cases = [testcase.copy()]
> + for var,fn in TESTCASE_VARIABLES:
> + dbg("var: %r, fn: %r", var, fn)
> + cases = genCases1(args, cases, var, fn)
> + return cases
> +
> +def genAllCases(args, testcase):
> + return genCases(args, testcase)
> +
> +def pickRandomCase(args, testcase):
> + cases = list(genCases(args, testcase))
> + if cases:
> + assert len(cases) == 1
> + return cases[0]
> +
> +def casesToTest(args, testcase):
> + cases = genAllCases(args, testcase)
> + if args.random:
> + cases = random.sample(list(cases), args.random)
> + if args.debug:
> + cases = list(cases)
> + dbg("%d test cases to test", len(cases))
> + if args.shuffle:
> + cases = list(cases)
> + random.shuffle(cases)
> + return cases
> +
> +def logFailure(f, level):
> + t = f['testcase']
> + logger.log(level, "failed: %s", formatTestCase(t))
> + logger.log(level, "cmdline: %s", f['cmdline'])
> + for l in f['log'].strip().split('\n'):
> + logger.log(level, "log: %s", l)
> + logger.log(level, "exit code: %r", f['exitcode'])
> +
> +def main():
> + parser = argparse.ArgumentParser(description="QEMU -device crash test")
> + parser.add_argument('-t', metavar='KEY=VALUE', nargs='*',
> + help="Limit test cases to KEY=VALUE",
> + action='append', dest='testcases', default=[])
> + parser.add_argument('-d', '--debug', action='store_true',
> + help='debug output')
> + parser.add_argument('-v', '--verbose', action='store_true',
> + help='verbose output')
> + parser.add_argument('-r', '--random', type=int, metavar='COUNT',
> + help='run a random sample of COUNT test cases',
> + default=0)
> + parser.add_argument('--shuffle', action='store_true',
> + help='Run test cases in random order')
> + parser.add_argument('--dry-run', action='store_true',
> + help="Don't run any tests, just generate list")
> + parser.add_argument('-D', '--devtype', metavar='TYPE',
> + help="Test only device types that implement TYPE")
> + parser.add_argument('-Q', '--quick', action='store_true',
> + help="Quick mode, skip test cases that are expected to fail")
> + parser.add_argument('qemu', nargs='*', metavar='QEMU',
> + help='QEMU binary to run')
> + args = parser.parse_args()
> +
> + if args.debug:
> + lvl = logging.DEBUG
> + elif args.verbose:
> + lvl = level=logging.INFO
> + else:
> + lvl = level=logging.WARN
> + logging.basicConfig(stream=sys.stdout, level=lvl, format='%(levelname)s: %(message)s')
> +
> +
> + interrupted = False
> + fatal_failures = []
> + wl_stats = {}
> + skipped = 0
> + total = 0
> +
> + tc = {}
> + dbg("testcases: %r", args.testcases)
> + if args.testcases:
> + for t in chain(*args.testcases):
> + for kv in t.split():
> + k,v = kv.split('=', 1)
> + tc[k] = v
> +
> + if len(binariesToTest(args, tc)) == 0:
> + print >>sys.stderr, "No QEMU binary found"
> + parser.print_usage(sys.stderr)
> + return 1
> +
> + for t in casesToTest(args, tc):
> + logger.info("running test case: %s", formatTestCase(t))
> + total += 1
> +
> + _,expected_match = checkWhitelist({'testcase': t})
> + if args.quick and expected_match.get('expected'):
> + dbg("Skipped: %s", formatTestCase(t))
> + skipped += 1
> + continue
> +
> + if args.dry_run:
> + continue
> +
> + try:
> + f = checkOneCase(args, t)
> + except KeyboardInterrupt:
> + interrupted = True
> + break
> +
> + if f:
> + i,wl = checkWhitelist(f)
> + dbg("testcase: %r, whitelist match: %r", t, wl)
> + wl_stats.setdefault(i, []).append(f)
> + logFailure(f, wl.get('loglevel', logging.DEBUG))
> + if wl.get('fatal'):
> + fatal_failures.append(f)
> + else:
> + dbg("success: %s", formatTestCase(t))
> + if expected_match.get('expected'):
> + logger.warn("Didn't fail as expected: %s", formatTestCase(t))
> +
> + logger.info("Total: %d test cases", total)
> + if skipped:
> + logger.info("Skipped %d test cases", skipped)
> +
> + # tell us about obsolete whitelist entries so we can clean it up after
> + # bugs are fixed:
> + if not (interrupted or args.random or args.dry_run \
> + or args.testcases or args.qemu or args.devtype):
> + for i,wl in enumerate(ERROR_WHITELIST):
> + if not wl_stats.get(i) and not wl.get('fatal'):
> + logger.info("Obsolete whitelist entry? %r", wl)
> +
> + stats = sorted([(len(wl_stats[i]), i) for i in wl_stats])
> + for count,i in stats:
> + dbg("whitelist entry stats: %d: %r", count, ERROR_WHITELIST[i])
> +
> + if fatal_failures:
> + for f in fatal_failures:
> + t = f['testcase']
> + logger.error("Fatal failure: %s", formatTestCase(t))
> + logger.error("Fatal failures on some machine/device combinations")
> + return 1
> +
> +if __name__ == '__main__':
> + sys.exit(main())
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes
2017-05-23 14:52 ` Markus Armbruster
@ 2017-05-24 16:39 ` Eduardo Habkost
2017-05-29 9:25 ` Markus Armbruster
0 siblings, 1 reply; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-24 16:39 UTC (permalink / raw)
To: Markus Armbruster; +Cc: qemu-devel, Marcel Apfelbaum, Thomas Huth
On Tue, May 23, 2017 at 04:52:47PM +0200, Markus Armbruster wrote:
> Beware, my review is rather superficial.
>
> Eduardo Habkost <ehabkost@redhat.com> writes:
>
> > Test code to check if we can crash QEMU using -device. It will
> > test all accel/machine/device combinations by default, which may
> > take a few hours (it's more than 90k test cases). There's a "-r"
> > option that makes it test a random sample of combinations.
> >
> > The scripts contains a whitelist for: 1) known error messages
> > that make QEMU exit cleanly; 2) known QEMU crashes.
> >
> > This is the behavior when the script finds a failure:
> >
> > * Known clean (exitcode=1) error messages generate INFO messages
> > (visible only in verbose mode), to make script output shorter
> > * Unknown clean error messages generate warnings
> > (visible by default)
> > * Known crashes generate error messages, but are not fatal
> > * Unknown crashes generate fatal error messages
> >
> > I'm unsure about the need to maintain a list of known clean error
> > messages, but I wanted to at least document all existing failure
> > cases to use as base to build more comprehensive test code.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> > Changes v1 -> v2:
> > * New whitelist entries:
> > * "could not find stage1 bootloader"
> > * Segfaults when using devices: a15mpcore_priv, sb16, cs4231a, arm-gicv3
> > * Format "success" line using formatTestCase(), and using DEBUg
> > loglevel
> > * Reword "test case:" line with "running test case:", for clarity
> > * Fix "pc-.*" whitelist to include "q35" too
> > * Add --devtype option to test only a specific device type
> > * Send all log messages to stdout instead of stderr
> > * Avoid printing "obsolete whitelist entry?" messages if we know
> > we are not testing every single accel/machine/device
> > combination
> > * --quick mode, to skip cases where failures are always expected,
> > and to print a warning in case we don't get an expected failure
> > * Use qemu.QEMUMachine instead of qtest.QEMUQtestMachine, as we don't
> > use any of the QEMUQtestMachine features
> > * Fix handling of multiple '-t' options
> > * Simplify code that generate random sample of test cases
> > ---
> > scripts/device-crash-test.py | 520 +++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 520 insertions(+)
> > create mode 100755 scripts/device-crash-test.py
> >
> > diff --git a/scripts/device-crash-test.py b/scripts/device-crash-test.py
> > new file mode 100755
> > index 0000000000..550da70ec7
> > --- /dev/null
> > +++ b/scripts/device-crash-test.py
> > @@ -0,0 +1,520 @@
> > +#!/usr/bin/env python2.7
> > +#
> > +# Run QEMU with all combinations of -machine and -device types,
> > +# check for crashes and unexpected errors.
> > +#
> > +# Copyright (c) 2017 Red Hat Inc
> > +#
> > +# Author:
> > +# Eduardo Habkost <ehabkost@redhat.com>
> > +#
> > +# This library is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU Lesser General Public
> > +# License as published by the Free Software Foundation; either
> > +# version 2 of the License, or (at your option) any later version.
>
> Any particular reason for "Lesser"?
I don't remember, probably I used another script I had written
before as reference. Probably it's not even a valid license as
I'm using a GPL module (qemu.py). I will change it to GPL.
>
> > +#
> > +# This library is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > +# Lesser General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU Lesser General Public
> > +# License along with this library; if not, see <http://www.gnu.org/licenses/>.
> > +#
> > +
> > +import sys, os, glob
> > +sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
> > +
> > +from itertools import chain
> > +from qemu import QEMUMachine
> > +import logging, traceback, re, random, argparse
> > +
> > +logger = logging.getLogger('device-crash-test')
> > +dbg = logger.debug
> > +
> > +# Valid whitelist entry keys:
> > +# - accel: regexp, full match only
> > +# - machine: regexp, full match only
> > +# - device: regexp, full match only
> > +# - log: regexp, partial match allowed
> > +# - exitcode: if not present, defaults to 1. If None, matches any exitcode
> > +# - warn: if True, matching failures will be logged as warnings
> > +# - expected: if True, QEMU is expected to always fail every time
> > +# when testing the corresponding test case
> > +ERROR_WHITELIST = [
> > + # Machines that won't work out of the box:
> > + # MACHINE | ERROR MESSAGE
> > + dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
>
> What's wrong with dictionary displays?
>
> {'expected': True, 'machine': 'niagara'}
Nothing wrong. I just prefer the dict(key=value) syntax to the
colons and extra quotes in {'key': value}.
>
> > + dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
> > + dict(machine='leon3_generic', expected=True), # Can't read bios image (null)
> > +
> > + # devices that don't work out of the box because they require extra options to "-device DEV":
> > + # DEVICE | ERROR MESSAGE
> > + dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
> > + dict(device='ARM,bitband-memory', expected=True), # source-memory property not set
> > + dict(device='arm.cortex-a9-global-timer', expected=True), # a9_gtimer_realize: num-cpu must be between 1 and 4
> > + dict(device='arm_mptimer', expected=True), # num-cpu must be between 1 and 4
> > + dict(device='armv7m', expected=True), # memory property was not set
> > + dict(device='aspeed.scu', expected=True), # Unknown silicon revision: 0x0
> > + dict(device='aspeed.sdmc', expected=True), # Unknown silicon revision: 0x0
> > + dict(device='bcm2835-dma', expected=True), # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
> > + dict(device='bcm2835-fb', expected=True), # bcm2835_fb_realize: required vcram-base property not set
> > + dict(device='bcm2835-mbox', expected=True), # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
> > + dict(device='bcm2835-peripherals', expected=True), # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
> > + dict(device='bcm2835-property', expected=True), # bcm2835_property_realize: required fb link not found: Property '.fb' not found
> > + dict(device='bcm2835_gpio', expected=True), # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
> > + dict(device='bcm2836', expected=True), # bcm2836_realize: required ram link not found: Property '.ram' not found
> > + dict(device='cfi.pflash01', expected=True), # attribute "sector-length" not specified or zero.
> > + dict(device='cfi.pflash02', expected=True), # attribute "sector-length" not specified or zero.
> > + dict(device='icp', expected=True), # icp_realize: required link 'xics' not found: Property '.xics' not found
> > + dict(device='ics', expected=True), # ics_base_realize: required link 'xics' not found: Property '.xics' not found
> > + dict(device='ide-drive', expected=True), # No drive specified
> > + dict(device='ide-hd', expected=True), # No drive specified
> > + dict(device='ipmi-bmc-extern', expected=True), # IPMI external bmc requires chardev attribute
> > + dict(device='isa-debugcon', expected=True), # Can't create serial device, empty char device
> > + dict(device='isa-ipmi-bt', expected=True), # IPMI device requires a bmc attribute to be set
> > + dict(device='isa-ipmi-kcs', expected=True), # IPMI device requires a bmc attribute to be set
> > + dict(device='isa-parallel', expected=True), # Can't create serial device, empty char device
> > + dict(device='isa-serial', expected=True), # Can't create serial device, empty char device
> > + dict(device='ivshmem', expected=True), # You must specify either 'shm' or 'chardev'
> > + dict(device='ivshmem-doorbell', expected=True), # You must specify a 'chardev'
> > + dict(device='ivshmem-plain', expected=True), # You must specify a 'memdev'
> > + dict(device='kvm-pci-assign', expected=True), # no host device specified
> > + dict(device='loader', expected=True), # please include valid arguments
> > + dict(device='nand', expected=True), #' Unsupported NAND block size 0x1
> > + dict(device='nvdimm', expected=True), # 'memdev' property is not set
> > + dict(device='nvme', expected=True), # Device initialization failed
> > + dict(device='pc-dimm', expected=True), # 'memdev' property is not set
> > + dict(device='pci-bridge', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> > + dict(device='pci-bridge-seat', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> > + dict(device='pci-serial', expected=True), # Can't create serial device, empty char device
> > + dict(device='pci-serial-2x', expected=True), # Can't create serial device, empty char device
> > + dict(device='pci-serial-4x', expected=True), # Can't create serial device, empty char device
> > + dict(device='pxa2xx-dma', expected=True), # channels value invalid
> > + dict(device='pxb', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
> > + dict(device='scsi-block', expected=True), # drive property not set
> > + dict(device='scsi-disk', expected=True), # drive property not set
> > + dict(device='scsi-generic', expected=True), # drive property not set
> > + dict(device='scsi-hd', expected=True), # drive property not set
> > + dict(device='spapr-pci-host-bridge', expected=True), # BUID not specified for PHB
> > + dict(device='spapr-pci-vfio-host-bridge', expected=True), # BUID not specified for PHB
> > + dict(device='spapr-rng', expected=True), #' spapr-rng needs an RNG backend!
> > + dict(device='spapr-vty', expected=True), # chardev property not set
> > + dict(device='tpm-tis', expected=True), # tpm_tis: backend driver with id (null) could not be found
> > + dict(device='unimplemented-device', expected=True), # property 'size' not specified or zero
> > + dict(device='usb-braille', expected=True), # Property chardev is required
> > + dict(device='usb-mtp', expected=True), # x-root property must be configured
> > + dict(device='usb-redir', expected=True), # Parameter 'chardev' is missing
> > + dict(device='usb-serial', expected=True), # Property chardev is required
> > + dict(device='usb-storage', expected=True), # drive property not set
> > + dict(device='vfio-amd-xgbe', expected=True), # -device vfio-amd-xgbe: vfio error: wrong host device name
> > + dict(device='vfio-calxeda-xgmac', expected=True), # -device vfio-calxeda-xgmac: vfio error: wrong host device name
> > + dict(device='vfio-pci', expected=True), # No provided host device
> > + dict(device='vfio-pci-igd-lpc-bridge', expected=True), # VFIO dummy ISA/LPC bridge must have address 1f.0
> > + dict(device='vhost-scsi.*', expected=True), # vhost-scsi: missing wwpn
> > + dict(device='vhost-vsock-device', expected=True), # guest-cid property must be greater than 2
> > + dict(device='vhost-vsock-pci', expected=True), # guest-cid property must be greater than 2
> > + dict(device='virtio-9p-ccw', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> > + dict(device='virtio-9p-device', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> > + dict(device='virtio-9p-pci', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
> > + dict(device='virtio-blk-ccw', expected=True), # drive property not set
> > + dict(device='virtio-blk-device', expected=True), # drive property not set
> > + dict(device='virtio-blk-device', expected=True), # drive property not set
> > + dict(device='virtio-blk-pci', expected=True), # drive property not set
> > + dict(device='virtio-crypto-ccw', expected=True), # 'cryptodev' parameter expects a valid object
> > + dict(device='virtio-crypto-device', expected=True), # 'cryptodev' parameter expects a valid object
> > + dict(device='virtio-crypto-pci', expected=True), # 'cryptodev' parameter expects a valid object
> > + dict(device='virtio-input-host-device', expected=True), # evdev property is required
> > + dict(device='virtio-input-host-pci', expected=True), # evdev property is required
> > + dict(device='xen-pvdevice', expected=True), # Device ID invalid, it must always be supplied
> > + dict(device='vhost-vsock-ccw', expected=True), # guest-cid property must be greater than 2
> > + dict(device='ALTR.timer', expected=True), # "clock-frequency" property must be provided
> > + dict(device='zpci', expected=True), # target must be defined
> > +
> > + # ioapic devices are already created by pc and will fail:
> > + dict(machine='q35|pc.*', device='kvm-ioapic', expected=True), # Only 1 ioapics allowed
> > + dict(machine='q35|pc.*', device='ioapic', expected=True), # Only 1 ioapics allowed
> > +
> > + # KVM-specific devices shouldn't be tried without accel=kvm:
> > + dict(accel='(?!kvm).*', device='kvmclock', expected=True),
> > + dict(accel='(?!kvm).*', device='kvm-pci-assign', expected=True),
> > +
> > + # xen-specific machines and devices:
> > + dict(accel='(?!xen).*', machine='xen.*', expected=True),
> > + dict(accel='(?!xen).*', device='xen-.*', expected=True),
> > +
> > + # KNOWN CRASHES:
> > + # known crashes will generate error messages, but won't be fatal:
> > + dict(exitcode=-6, log=r"Device 'serial0' is in use", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"qemu_net_client_setup: Assertion `!peer->peer' failed", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r'RAMBlock "[\w.-]+" already registered', loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"find_ram_offset: Assertion `size != 0' failed.", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"add_cpreg_to_hashtable: code should not be reached", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"qemu_alloc_display: Assertion `surface->image != NULL' failed", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"Unexpected error in error_set_from_qdev_prop_error", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"Object .* is not an instance of type spapr-machine", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"Object .* is not an instance of type generic-pc-machine", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"Object .* is not an instance of type e500-ccsr", loglevel=logging.ERROR),
> > + dict(exitcode=-6, log=r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", loglevel=logging.ERROR),
> > + dict(exitcode=-11, device='stm32f205-soc', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='xlnx,zynqmp', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='mips-cps', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='gus', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='a9mpcore_priv', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='a15mpcore_priv', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='isa-serial', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='sb16', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='cs4231a', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, device='arm-gicv3', loglevel=logging.ERROR, expected=True),
> > + dict(exitcode=-11, machine='isapc', device='.*-iommu', loglevel=logging.ERROR, expected=True),
> > +
> > + # Some error messages that are common on multiple devices/machines:
> > + dict(log=r"No '[\w-]+' bus found for device '[\w-]+'"),
> > + dict(log=r"images* must be given with the 'pflash' parameter"),
> > + dict(log=r"(Guest|ROM|Flash|Kernel) image must be specified"),
> > + dict(log=r"[cC]ould not load [\w ]+ (BIOS|bios) '[\w-]+\.bin'"),
> > + dict(log=r"Couldn't find rom image '[\w-]+\.bin'"),
> > + dict(log=r"speed mismatch trying to attach usb device"),
> > + dict(log=r"Can't create a second ISA bus"),
> > + dict(log=r"duplicate fw_cfg file name"),
> > + # sysbus-related error messages: most machines reject most dynamic sysbus devices:
> > + #TODO: expected=True entries for unsupported sysbus devices
> > + dict(log=r"Option '-device [\w.,-]+' cannot be handled by this machine"),
> > + dict(log=r"Device [\w.,-]+ is not supported by this machine yet"),
> > + dict(log=r"Device [\w.,-]+ can not be dynamically instantiated"),
> > + dict(log=r"Platform Bus: Can not fit MMIO region of size "),
> > + # other more specific errors we will ignore:
> > + dict(device='allwinner-a10', log="Unsupported NIC model:"),
> > + dict(device='.*-spapr-cpu-core', log=r"CPU core type should be"),
> > + dict(log=r"MSI(-X)? is not supported by interrupt controller"),
> > + dict(log=r"pxb-pcie? devices cannot reside on a PCIe? bus"),
> > + dict(log=r"Ignoring smp_cpus value"),
> > + dict(log=r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"),
> > + dict(log=r"This CPU requires a smaller page size than the system is using"),
> > + dict(log=r"MSI-X support is mandatory in the S390 architecture"),
> > + dict(log=r"rom check and register reset failed"),
> > + dict(log=r"Unable to initialize GIC, CPUState for CPU#0 not valid"),
> > + dict(log=r"Multiple VT220 operator consoles are not supported"),
> > + dict(log=r"core 0 already populated"),
> > + dict(log=r"could not find stage1 bootloader"),
> > +
> > + # other exitcode=1 failures not listed above will generate warnings:
> > + dict(exitcode=1, loglevel=logging.WARN),
^^^ [*]
> > +
> > + # everything else (including SIGABRT and SIGSEGV) will be a fatal error:
> > + dict(exitcode=None, fatal=True, loglevel=logging.FATAL),
> > +]
>
> How will we keep this whitelist up-to-date?
We don't really have to. Making a complete whitelist was useful
to me when validating the script, but I don't think it will be
worth the effort to keep it up to date. The only consequence of
an incomplete whitelist will be the warnings triggered by [*].
We may consider changing the entry at [*] to logging.INFO, if it
starts triggering too many warnings.
I will add comments to the script noting
>
> Hmm, there seems to be code for detecting obsolete entries. Can you
> explain how it works, for dummies?
The script simply lists the whitelist entries that didn't have
any matches, because it is useful if somebody wants to remove
obsolete entries from the whitelist. The script can't be sure
the entry is really obsolete, because the user might be testing
just a subset of the QEMU binaries, or building it with some
devices/features disabled.
I will change these messages to be printed only in debug mode,
and add comments explaining why they are there.
>
> > +
> > +def whitelistMatch(wl, f):
> > + t = f.get('testcase', {})
> > + return (not wl.has_key('machine') \
>
> scripts/device-crash-test.py:212:19: W601 .has_key() is deprecated, use 'in'
>
> Please feed this module to pylint and pep8, then use your judgement.
Will do.
>
> > + or not t.has_key('machine') \
> > + or re.match(wl['machine'] +'$', t['machine'])) \
> > + and (not wl.has_key('accel') \
> > + or not t.has_key('accel') \
> > + or re.match(wl['accel'] +'$', t['accel'])) \
> > + and (not wl.has_key('device') \
> > + or not t.has_key('device') \
> > + or re.match(wl['device'] +'$', t['device'])) \
> > + and (wl.get('exitcode', 1) is None \
> > + or not f.has_key('exitcode')
> > + or f['exitcode'] == wl.get('exitcode', 1)) \
>
> Could this be simplified to
>
> and (wl.get('exitcode', 1) is None
> or f.get('exitcode') == wl.get('exitcode', 1))
>
> ?
Nope, because this:
checkWhitelist({'testcase':{'machine':'niagara'}})
should return:
(0, {'expected': True, 'machine': 'niagara'})
to make the check at [***] properly skip expected=True entries.
Maybe I should write two separate functions, for clarity:
whitelistTestCaseMatch() and whitelistTestCaseResultsMatch().
The check at [***] then would use whitelistTestCaseMatch(). The
check at [****] would use whitelistTestCaseResultsMatch().
>
> > + and (not wl.has_key('log') \
> > + or not f.has_key('log') \
> > + or re.search(wl['log'], f['log'], re.MULTILINE))
> > +
> > +def checkWhitelist(f):
> > + """Look up whitelist entry for failure dictionary
> > +
> > + Returns index in ERROR_WHITELIST
> > + """
> > + for i,wl in enumerate(ERROR_WHITELIST):
>
> Umm, sure you need to wrap enumerate() around ERROR_WHITELIST?
I do, because I want to return the index of the entry also[**].
>
> > + #dbg("whitelist entry: %r", wl)
>
> Debugging turd? More of the same below.
I will delete that.
>
> > + if whitelistMatch(wl, f):
> > + return i, wl
^^^ [**]
> > +
> > + raise Exception("this should never happen")
> > +
> > +def qemuOptsEscape(s):
> > + return s.replace(",", ",,")
> > +
> > +def formatTestCase(t):
> > + return ' '.join('%s=%s' % (k, v) for k,v in t.items())
> > +
> > +def qomListTypeNames(vm, **kwargs):
> > + """Run qom-list-types QMP command, return type names"""
> > + types = vm.command('qom-list-types', **kwargs)
> > + return [t['name'] for t in types]
> > +
> > +def infoQDM(vm):
> > + """Parse 'info qdm' output"""
> > + args = {'command-line': 'info qdm'}
> > + devhelp = vm.command('human-monitor-command', **args)
> > + for l in devhelp.split('\n'):
> > + l = l.strip()
> > + if l == '' or l.endswith(':'):
> > + continue
> > + d = {'name': re.search(r'name "([^"]+)"', l).group(1),
> > + 'no-user': (re.search(', no-user', l) is not None)}
> > + #dbg('info qdm item: %r', d)
> > + yield d
>
> The need for HMP is sad, but you already proposed patches to remove it.
Yep.
>
> > +
> > +
> > +class QemuBinaryInfo:
> > + def __init__(self, binary, devtype):
> > + if devtype is None:
> > + devtype = 'device'
> > +
> > + dbg("devtype: %r", devtype)
> > + args = ['-S', '-machine', 'none,accel=kvm:tcg']
> > + dbg("querying info for QEMU binary: %s", binary)
> > + vm = QEMUMachine(binary=binary, args=args)
> > + vm.launch()
> > + try:
> > + self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False))
> > + # there's no way to query cannot_instantiate_with_device_add_yet using QMP,
>
> Update needed for the rename to user_creatable.
Right. I will update it.
>
> > + # so use 'info qdm':
> > + self.no_user_devs = set([d['name'] for d in infoQDM(vm, ) if d['no-user']])
> > + self.machines = list(m['name'] for m in vm.command('query-machines'))
> > + self.user_devs = self.alldevs.difference(self.no_user_devs)
> > + self.kvm_available = vm.command('query-kvm')['enabled']
> > + finally:
> > + vm.shutdown()
> > +
> > +BINARY_INFO = {}
> > +def getBinaryInfo(args, binary):
> > + if not BINARY_INFO.has_key(binary):
> > + BINARY_INFO[binary] = QemuBinaryInfo(binary, args.devtype)
> > + return BINARY_INFO[binary]
> > +
> > +def checkOneCase(args, testcase):
> > + """Check one specific case
> > +
> > + Returns a dictionary containing failure information on error,
> > + or None on success
> > + """
> > + binary = testcase['binary']
> > + accel = testcase['accel']
> > + machine = testcase['machine']
> > + device = testcase['device']
> > + info = getBinaryInfo(args, binary)
> > +
> > + dbg("will test: %r", testcase)
> > +
> > + args = ['-S', '-machine', '%s,accel=%s' % (machine, accel),
> > + '-device', qemuOptsEscape(device)]
> > + cmdline = ' '.join([binary] + args)
> > + dbg("will launch QEMU: %s", cmdline)
> > + vm = QEMUMachine(binary=binary, args=args)
> > +
> > + exception = None
> > + try:
> > + vm.launch()
> > + except KeyboardInterrupt:
> > + raise
> > + except Exception,e:
> > + exception = e
> > + finally:
> > + vm.shutdown()
> > + ec = vm.exitcode()
> > + log = vm.get_log()
> > +
> > + if exception is not None or ec != 0:
> > + f = dict(exception = exception,
> > + exitcode = ec,
> > + log = log,
> > + testcase = testcase,
> > + cmdline=cmdline)
> > + return f
>
> Why not simply
>
> return {'exception': exception,
> 'exitcode': ec,
> 'log': log,
> 'testcase': testcase,
> 'cmdline': cmdline}
About the 'f' variable: probably I had a debugging statement here
in the past. I will remove it.
About the dict(key=value) syntax instead of {'key':value}: like
above, it was just personal taste.
>
> ?
>
> > +
> > +def binariesToTest(args, testcase):
> > + if args.qemu:
> > + r = args.qemu
> > + else:
> > + r = glob.glob('./*-softmmu/qemu-system-*')
> > + return r
> > +
> > +def accelsToTest(args, testcase):
> > + if getBinaryInfo(args, testcase['binary']).kvm_available:
> > + yield 'kvm'
> > + yield 'tcg'
>
> If only "--accel help" showed just the ones that can actually work...
I refuse to parse help output unless there's no equivalent QMP
alternative. But we could add a query-accelerators QMP command
in the future.
>
> > +
> > +def machinesToTest(args, testcase):
> > + return getBinaryInfo(args, testcase['binary']).machines
> > +
> > +def devicesToTest(args, testcase):
> > + return getBinaryInfo(args, testcase['binary']).user_devs
> > +
> > +TESTCASE_VARIABLES = [
> > + ('binary', binariesToTest),
> > + ('accel', accelsToTest),
> > + ('machine', machinesToTest),
> > + ('device', devicesToTest),
> > +]
> > +
> > +def genCases1(args, testcases, var, fn):
> > + """Generate new testcases for one variable
> > +
> > + If an existing item already has a variable set, don't
> > + generate new items and just return it directly. This
> > + allows the "-t" command-line option to be used to choose
> > + a specific test case.
> > + """
> > + for testcase in testcases:
> > + t = testcase.copy
> > + if testcase.has_key(var):
> > + yield testcase.copy()
> > + else:
> > + for i in fn(args, testcase):
> > + t = testcase.copy()
> > + t[var] = i
> > + yield t
> > +
> > +def genCases(args, testcase):
> > + """Generate test cases for all variables
> > + """
> > + cases = [testcase.copy()]
> > + for var,fn in TESTCASE_VARIABLES:
> > + dbg("var: %r, fn: %r", var, fn)
> > + cases = genCases1(args, cases, var, fn)
> > + return cases
> > +
> > +def genAllCases(args, testcase):
> > + return genCases(args, testcase)
> > +
> > +def pickRandomCase(args, testcase):
> > + cases = list(genCases(args, testcase))
> > + if cases:
> > + assert len(cases) == 1
> > + return cases[0]
> > +
> > +def casesToTest(args, testcase):
> > + cases = genAllCases(args, testcase)
> > + if args.random:
> > + cases = random.sample(list(cases), args.random)
> > + if args.debug:
> > + cases = list(cases)
> > + dbg("%d test cases to test", len(cases))
> > + if args.shuffle:
> > + cases = list(cases)
> > + random.shuffle(cases)
> > + return cases
> > +
> > +def logFailure(f, level):
> > + t = f['testcase']
> > + logger.log(level, "failed: %s", formatTestCase(t))
> > + logger.log(level, "cmdline: %s", f['cmdline'])
> > + for l in f['log'].strip().split('\n'):
> > + logger.log(level, "log: %s", l)
> > + logger.log(level, "exit code: %r", f['exitcode'])
> > +
> > +def main():
> > + parser = argparse.ArgumentParser(description="QEMU -device crash test")
> > + parser.add_argument('-t', metavar='KEY=VALUE', nargs='*',
> > + help="Limit test cases to KEY=VALUE",
> > + action='append', dest='testcases', default=[])
> > + parser.add_argument('-d', '--debug', action='store_true',
> > + help='debug output')
> > + parser.add_argument('-v', '--verbose', action='store_true',
> > + help='verbose output')
> > + parser.add_argument('-r', '--random', type=int, metavar='COUNT',
> > + help='run a random sample of COUNT test cases',
> > + default=0)
> > + parser.add_argument('--shuffle', action='store_true',
> > + help='Run test cases in random order')
> > + parser.add_argument('--dry-run', action='store_true',
> > + help="Don't run any tests, just generate list")
> > + parser.add_argument('-D', '--devtype', metavar='TYPE',
> > + help="Test only device types that implement TYPE")
> > + parser.add_argument('-Q', '--quick', action='store_true',
> > + help="Quick mode, skip test cases that are expected to fail")
> > + parser.add_argument('qemu', nargs='*', metavar='QEMU',
> > + help='QEMU binary to run')
> > + args = parser.parse_args()
> > +
> > + if args.debug:
> > + lvl = logging.DEBUG
> > + elif args.verbose:
> > + lvl = level=logging.INFO
> > + else:
> > + lvl = level=logging.WARN
> > + logging.basicConfig(stream=sys.stdout, level=lvl, format='%(levelname)s: %(message)s')
> > +
> > +
> > + interrupted = False
> > + fatal_failures = []
> > + wl_stats = {}
> > + skipped = 0
> > + total = 0
> > +
> > + tc = {}
> > + dbg("testcases: %r", args.testcases)
> > + if args.testcases:
> > + for t in chain(*args.testcases):
> > + for kv in t.split():
> > + k,v = kv.split('=', 1)
> > + tc[k] = v
> > +
> > + if len(binariesToTest(args, tc)) == 0:
> > + print >>sys.stderr, "No QEMU binary found"
> > + parser.print_usage(sys.stderr)
> > + return 1
> > +
> > + for t in casesToTest(args, tc):
> > + logger.info("running test case: %s", formatTestCase(t))
> > + total += 1
> > +
> > + _,expected_match = checkWhitelist({'testcase': t})
^^^ [***]
> > + if args.quick and expected_match.get('expected'):
> > + dbg("Skipped: %s", formatTestCase(t))
> > + skipped += 1
> > + continue
> > +
> > + if args.dry_run:
> > + continue
> > +
> > + try:
> > + f = checkOneCase(args, t)
> > + except KeyboardInterrupt:
> > + interrupted = True
> > + break
> > +
> > + if f:
> > + i,wl = checkWhitelist(f)
^^^ [****]
> > + dbg("testcase: %r, whitelist match: %r", t, wl)
> > + wl_stats.setdefault(i, []).append(f)
> > + logFailure(f, wl.get('loglevel', logging.DEBUG))
> > + if wl.get('fatal'):
> > + fatal_failures.append(f)
> > + else:
> > + dbg("success: %s", formatTestCase(t))
> > + if expected_match.get('expected'):
> > + logger.warn("Didn't fail as expected: %s", formatTestCase(t))
> > +
> > + logger.info("Total: %d test cases", total)
> > + if skipped:
> > + logger.info("Skipped %d test cases", skipped)
> > +
> > + # tell us about obsolete whitelist entries so we can clean it up after
> > + # bugs are fixed:
> > + if not (interrupted or args.random or args.dry_run \
> > + or args.testcases or args.qemu or args.devtype):
> > + for i,wl in enumerate(ERROR_WHITELIST):
> > + if not wl_stats.get(i) and not wl.get('fatal'):
> > + logger.info("Obsolete whitelist entry? %r", wl)
> > +
> > + stats = sorted([(len(wl_stats[i]), i) for i in wl_stats])
> > + for count,i in stats:
> > + dbg("whitelist entry stats: %d: %r", count, ERROR_WHITELIST[i])
> > +
> > + if fatal_failures:
> > + for f in fatal_failures:
> > + t = f['testcase']
> > + logger.error("Fatal failure: %s", formatTestCase(t))
> > + logger.error("Fatal failures on some machine/device combinations")
> > + return 1
> > +
> > +if __name__ == '__main__':
> > + sys.exit(main())
--
Eduardo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown
2017-05-23 14:23 ` Markus Armbruster
@ 2017-05-25 18:04 ` Eduardo Habkost
2017-05-29 11:40 ` Markus Armbruster
0 siblings, 1 reply; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-25 18:04 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, Marcel Apfelbaum, Thomas Huth, Daniel P. Berrange
On Tue, May 23, 2017 at 04:23:08PM +0200, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
>
> > Keep the Popen object around to we can query its exit code later.
> >
> > To keep the existing 'self._popen is None' checks working, add a
> > is_running() method, that will check if the process is still running.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>
> Looks harmless enough. Have you tested the scripts using this module
> still work?
Now I did. :)
I see only two users of qemu.py and qtest.py:
tests/migration/guestperf/engine.py:import qemu
tests/qemu-iotests/iotests.py:import qtest
I just tested both using:
$ make check-tests/qemu-iotests-quick.sh
$ make tests/migration/initrd-stress.img PTHREAD_LIB=-pthread
$ ./tests/migration/guestperf.py > /tmp/guestperf.json
and they seem to be working.
(It looks like there's no Makefile rule to run guestperf.py)
BTW, it seems to be impossible to build initrd-stress.img on a out-of-tree
build. I had to run ./configure inside the source tree to be able to run
guestperf.py.
--
Eduardo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes
2017-05-24 16:39 ` Eduardo Habkost
@ 2017-05-29 9:25 ` Markus Armbruster
2017-05-31 17:41 ` Eduardo Habkost
0 siblings, 1 reply; 12+ messages in thread
From: Markus Armbruster @ 2017-05-29 9:25 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: Marcel Apfelbaum, Thomas Huth, qemu-devel
Eduardo Habkost <ehabkost@redhat.com> writes:
> On Tue, May 23, 2017 at 04:52:47PM +0200, Markus Armbruster wrote:
>> Beware, my review is rather superficial.
>>
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>>
>> > Test code to check if we can crash QEMU using -device. It will
>> > test all accel/machine/device combinations by default, which may
>> > take a few hours (it's more than 90k test cases). There's a "-r"
>> > option that makes it test a random sample of combinations.
>> >
>> > The scripts contains a whitelist for: 1) known error messages
>> > that make QEMU exit cleanly; 2) known QEMU crashes.
>> >
>> > This is the behavior when the script finds a failure:
>> >
>> > * Known clean (exitcode=1) error messages generate INFO messages
>> > (visible only in verbose mode), to make script output shorter
>> > * Unknown clean error messages generate warnings
>> > (visible by default)
>> > * Known crashes generate error messages, but are not fatal
>> > * Unknown crashes generate fatal error messages
>> >
>> > I'm unsure about the need to maintain a list of known clean error
>> > messages, but I wanted to at least document all existing failure
>> > cases to use as base to build more comprehensive test code.
>> >
>> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> > ---
>> > Changes v1 -> v2:
>> > * New whitelist entries:
>> > * "could not find stage1 bootloader"
>> > * Segfaults when using devices: a15mpcore_priv, sb16, cs4231a, arm-gicv3
>> > * Format "success" line using formatTestCase(), and using DEBUg
>> > loglevel
>> > * Reword "test case:" line with "running test case:", for clarity
>> > * Fix "pc-.*" whitelist to include "q35" too
>> > * Add --devtype option to test only a specific device type
>> > * Send all log messages to stdout instead of stderr
>> > * Avoid printing "obsolete whitelist entry?" messages if we know
>> > we are not testing every single accel/machine/device
>> > combination
>> > * --quick mode, to skip cases where failures are always expected,
>> > and to print a warning in case we don't get an expected failure
>> > * Use qemu.QEMUMachine instead of qtest.QEMUQtestMachine, as we don't
>> > use any of the QEMUQtestMachine features
>> > * Fix handling of multiple '-t' options
>> > * Simplify code that generate random sample of test cases
>> > ---
>> > scripts/device-crash-test.py | 520 +++++++++++++++++++++++++++++++++++++++++++
>> > 1 file changed, 520 insertions(+)
>> > create mode 100755 scripts/device-crash-test.py
>> >
>> > diff --git a/scripts/device-crash-test.py b/scripts/device-crash-test.py
>> > new file mode 100755
>> > index 0000000000..550da70ec7
>> > --- /dev/null
>> > +++ b/scripts/device-crash-test.py
>> > @@ -0,0 +1,520 @@
>> > +#!/usr/bin/env python2.7
>> > +#
>> > +# Run QEMU with all combinations of -machine and -device types,
>> > +# check for crashes and unexpected errors.
>> > +#
>> > +# Copyright (c) 2017 Red Hat Inc
>> > +#
>> > +# Author:
>> > +# Eduardo Habkost <ehabkost@redhat.com>
>> > +#
>> > +# This library is free software; you can redistribute it and/or
>> > +# modify it under the terms of the GNU Lesser General Public
>> > +# License as published by the Free Software Foundation; either
>> > +# version 2 of the License, or (at your option) any later version.
>>
>> Any particular reason for "Lesser"?
>
> I don't remember, probably I used another script I had written
> before as reference. Probably it's not even a valid license as
> I'm using a GPL module (qemu.py). I will change it to GPL.
Thanks.
>> > +#
>> > +# This library is distributed in the hope that it will be useful,
>> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
>> > +# Lesser General Public License for more details.
>> > +#
>> > +# You should have received a copy of the GNU Lesser General Public
>> > +# License along with this library; if not, see <http://www.gnu.org/licenses/>.
>> > +#
>> > +
>> > +import sys, os, glob
>> > +sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
>> > +
>> > +from itertools import chain
>> > +from qemu import QEMUMachine
>> > +import logging, traceback, re, random, argparse
>> > +
>> > +logger = logging.getLogger('device-crash-test')
>> > +dbg = logger.debug
>> > +
>> > +# Valid whitelist entry keys:
>> > +# - accel: regexp, full match only
>> > +# - machine: regexp, full match only
>> > +# - device: regexp, full match only
>> > +# - log: regexp, partial match allowed
>> > +# - exitcode: if not present, defaults to 1. If None, matches any exitcode
>> > +# - warn: if True, matching failures will be logged as warnings
>> > +# - expected: if True, QEMU is expected to always fail every time
>> > +# when testing the corresponding test case
>> > +ERROR_WHITELIST = [
>> > + # Machines that won't work out of the box:
>> > + # MACHINE | ERROR MESSAGE
>> > + dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
>>
>> What's wrong with dictionary displays?
>>
>> {'expected': True, 'machine': 'niagara'}
>
> Nothing wrong. I just prefer the dict(key=value) syntax to the
> colons and extra quotes in {'key': value}.
Since Python provides syntactic sugar ("dictionary displays")
specifically for creating dictionaries, shouldn't we use it?
As far as I can tell, existing Python code always uses dictionary
displays, never the equivalent constructor call.
q>> > + dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
>> > + dict(machine='leon3_generic', expected=True), # Can't read bios image (null)
>> > +
>> > + # devices that don't work out of the box because they require extra options to "-device DEV":
>> > + # DEVICE | ERROR MESSAGE
>> > + dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
[...]
>> > + dict(log=r"could not find stage1 bootloader"),
>> > +
>> > + # other exitcode=1 failures not listed above will generate warnings:
>> > + dict(exitcode=1, loglevel=logging.WARN),
> ^^^ [*]
>
>> > +
>> > + # everything else (including SIGABRT and SIGSEGV) will be a fatal error:
>> > + dict(exitcode=None, fatal=True, loglevel=logging.FATAL),
>> > +]
>>
>> How will we keep this whitelist up-to-date?
>
> We don't really have to. Making a complete whitelist was useful
> to me when validating the script, but I don't think it will be
> worth the effort to keep it up to date. The only consequence of
> an incomplete whitelist will be the warnings triggered by [*].
>
> We may consider changing the entry at [*] to logging.INFO, if it
> starts triggering too many warnings.
>
> I will add comments to the script noting
>
>>
>> Hmm, there seems to be code for detecting obsolete entries. Can you
>> explain how it works, for dummies?
>
> The script simply lists the whitelist entries that didn't have
> any matches, because it is useful if somebody wants to remove
> obsolete entries from the whitelist. The script can't be sure
> the entry is really obsolete, because the user might be testing
> just a subset of the QEMU binaries, or building it with some
> devices/features disabled.
Okay, so the script can tell us about failures not covered by the
whitelist, and about whitelist entries that appear unused. The latter
is of course useful only when your testing with a sufficiently complete
QEMU. Good enough for me.
> I will change these messages to be printed only in debug mode,
> and add comments explaining why they are there.
>
>>
>> > +
>> > +def whitelistMatch(wl, f):
>> > + t = f.get('testcase', {})
>> > + return (not wl.has_key('machine') \
>>
>> scripts/device-crash-test.py:212:19: W601 .has_key() is deprecated, use 'in'
>>
>> Please feed this module to pylint and pep8, then use your judgement.
>
> Will do.
>
>>
>> > + or not t.has_key('machine') \
>> > + or re.match(wl['machine'] +'$', t['machine'])) \
>> > + and (not wl.has_key('accel') \
>> > + or not t.has_key('accel') \
>> > + or re.match(wl['accel'] +'$', t['accel'])) \
>> > + and (not wl.has_key('device') \
>> > + or not t.has_key('device') \
>> > + or re.match(wl['device'] +'$', t['device'])) \
>> > + and (wl.get('exitcode', 1) is None \
>> > + or not f.has_key('exitcode')
>> > + or f['exitcode'] == wl.get('exitcode', 1)) \
>>
>> Could this be simplified to
>>
>> and (wl.get('exitcode', 1) is None
>> or f.get('exitcode') == wl.get('exitcode', 1))
>>
>> ?
>
> Nope, because this:
> checkWhitelist({'testcase':{'machine':'niagara'}})
> should return:
> (0, {'expected': True, 'machine': 'niagara'})
> to make the check at [***] properly skip expected=True entries.
>
> Maybe I should write two separate functions, for clarity:
> whitelistTestCaseMatch() and whitelistTestCaseResultsMatch().
>
> The check at [***] then would use whitelistTestCaseMatch(). The
> check at [****] would use whitelistTestCaseResultsMatch().
>
>>
>> > + and (not wl.has_key('log') \
>> > + or not f.has_key('log') \
>> > + or re.search(wl['log'], f['log'], re.MULTILINE))
>> > +
>> > +def checkWhitelist(f):
>> > + """Look up whitelist entry for failure dictionary
>> > +
>> > + Returns index in ERROR_WHITELIST
>> > + """
>> > + for i,wl in enumerate(ERROR_WHITELIST):
>>
>> Umm, sure you need to wrap enumerate() around ERROR_WHITELIST?
>
> I do, because I want to return the index of the entry also[**].
>
>>
>> > + #dbg("whitelist entry: %r", wl)
>>
>> Debugging turd? More of the same below.
>
> I will delete that.
>
>>
>> > + if whitelistMatch(wl, f):
>> > + return i, wl
> ^^^ [**]
>
>> > +
>> > + raise Exception("this should never happen")
>> > +
>> > +def qemuOptsEscape(s):
>> > + return s.replace(",", ",,")
>> > +
>> > +def formatTestCase(t):
>> > + return ' '.join('%s=%s' % (k, v) for k,v in t.items())
>> > +
>> > +def qomListTypeNames(vm, **kwargs):
>> > + """Run qom-list-types QMP command, return type names"""
>> > + types = vm.command('qom-list-types', **kwargs)
>> > + return [t['name'] for t in types]
>> > +
>> > +def infoQDM(vm):
>> > + """Parse 'info qdm' output"""
>> > + args = {'command-line': 'info qdm'}
>> > + devhelp = vm.command('human-monitor-command', **args)
>> > + for l in devhelp.split('\n'):
>> > + l = l.strip()
>> > + if l == '' or l.endswith(':'):
>> > + continue
>> > + d = {'name': re.search(r'name "([^"]+)"', l).group(1),
>> > + 'no-user': (re.search(', no-user', l) is not None)}
>> > + #dbg('info qdm item: %r', d)
>> > + yield d
>>
>> The need for HMP is sad, but you already proposed patches to remove it.
>
> Yep.
>
>>
>> > +
>> > +
>> > +class QemuBinaryInfo:
>> > + def __init__(self, binary, devtype):
>> > + if devtype is None:
>> > + devtype = 'device'
>> > +
>> > + dbg("devtype: %r", devtype)
>> > + args = ['-S', '-machine', 'none,accel=kvm:tcg']
>> > + dbg("querying info for QEMU binary: %s", binary)
>> > + vm = QEMUMachine(binary=binary, args=args)
>> > + vm.launch()
>> > + try:
>> > + self.alldevs = set(qomListTypeNames(vm, implements=devtype, abstract=False))
>> > + # there's no way to query cannot_instantiate_with_device_add_yet using QMP,
>>
>> Update needed for the rename to user_creatable.
>
> Right. I will update it.
>
>>
>> > + # so use 'info qdm':
>> > + self.no_user_devs = set([d['name'] for d in infoQDM(vm, ) if d['no-user']])
>> > + self.machines = list(m['name'] for m in vm.command('query-machines'))
>> > + self.user_devs = self.alldevs.difference(self.no_user_devs)
>> > + self.kvm_available = vm.command('query-kvm')['enabled']
>> > + finally:
>> > + vm.shutdown()
>> > +
>> > +BINARY_INFO = {}
>> > +def getBinaryInfo(args, binary):
>> > + if not BINARY_INFO.has_key(binary):
>> > + BINARY_INFO[binary] = QemuBinaryInfo(binary, args.devtype)
>> > + return BINARY_INFO[binary]
>> > +
>> > +def checkOneCase(args, testcase):
>> > + """Check one specific case
>> > +
>> > + Returns a dictionary containing failure information on error,
>> > + or None on success
>> > + """
>> > + binary = testcase['binary']
>> > + accel = testcase['accel']
>> > + machine = testcase['machine']
>> > + device = testcase['device']
>> > + info = getBinaryInfo(args, binary)
>> > +
>> > + dbg("will test: %r", testcase)
>> > +
>> > + args = ['-S', '-machine', '%s,accel=%s' % (machine, accel),
>> > + '-device', qemuOptsEscape(device)]
>> > + cmdline = ' '.join([binary] + args)
>> > + dbg("will launch QEMU: %s", cmdline)
>> > + vm = QEMUMachine(binary=binary, args=args)
>> > +
>> > + exception = None
>> > + try:
>> > + vm.launch()
>> > + except KeyboardInterrupt:
>> > + raise
>> > + except Exception,e:
>> > + exception = e
>> > + finally:
>> > + vm.shutdown()
>> > + ec = vm.exitcode()
>> > + log = vm.get_log()
>> > +
>> > + if exception is not None or ec != 0:
>> > + f = dict(exception = exception,
>> > + exitcode = ec,
>> > + log = log,
>> > + testcase = testcase,
>> > + cmdline=cmdline)
>> > + return f
>>
>> Why not simply
>>
>> return {'exception': exception,
>> 'exitcode': ec,
>> 'log': log,
>> 'testcase': testcase,
>> 'cmdline': cmdline}
>
> About the 'f' variable: probably I had a debugging statement here
> in the past. I will remove it.
>
> About the dict(key=value) syntax instead of {'key':value}: like
> above, it was just personal taste.
>
>>
>> ?
>>
>> > +
>> > +def binariesToTest(args, testcase):
>> > + if args.qemu:
>> > + r = args.qemu
>> > + else:
>> > + r = glob.glob('./*-softmmu/qemu-system-*')
>> > + return r
>> > +
>> > +def accelsToTest(args, testcase):
>> > + if getBinaryInfo(args, testcase['binary']).kvm_available:
>> > + yield 'kvm'
>> > + yield 'tcg'
>>
>> If only "--accel help" showed just the ones that can actually work...
>
> I refuse to parse help output unless there's no equivalent QMP
> alternative. But we could add a query-accelerators QMP command
> in the future.
Understand.
What you have is good enough for now.
[...]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown
2017-05-25 18:04 ` Eduardo Habkost
@ 2017-05-29 11:40 ` Markus Armbruster
0 siblings, 0 replies; 12+ messages in thread
From: Markus Armbruster @ 2017-05-29 11:40 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: Marcel Apfelbaum, Thomas Huth, qemu-devel
Eduardo Habkost <ehabkost@redhat.com> writes:
> On Tue, May 23, 2017 at 04:23:08PM +0200, Markus Armbruster wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>>
>> > Keep the Popen object around to we can query its exit code later.
>> >
>> > To keep the existing 'self._popen is None' checks working, add a
>> > is_running() method, that will check if the process is still running.
>> >
>> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>
>> Looks harmless enough. Have you tested the scripts using this module
>> still work?
>
> Now I did. :)
>
> I see only two users of qemu.py and qtest.py:
>
> tests/migration/guestperf/engine.py:import qemu
> tests/qemu-iotests/iotests.py:import qtest
>
> I just tested both using:
>
> $ make check-tests/qemu-iotests-quick.sh
> $ make tests/migration/initrd-stress.img PTHREAD_LIB=-pthread
> $ ./tests/migration/guestperf.py > /tmp/guestperf.json
>
> and they seem to be working.
Thanks!
> (It looks like there's no Makefile rule to run guestperf.py)
>
> BTW, it seems to be impossible to build initrd-stress.img on a out-of-tree
> build. I had to run ./configure inside the source tree to be able to run
> guestperf.py.
I consider that a bug. But it's clearly not one you need to fix to get
your series accepted.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes
2017-05-29 9:25 ` Markus Armbruster
@ 2017-05-31 17:41 ` Eduardo Habkost
0 siblings, 0 replies; 12+ messages in thread
From: Eduardo Habkost @ 2017-05-31 17:41 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Marcel Apfelbaum, Thomas Huth, qemu-devel
On Mon, May 29, 2017 at 11:25:03AM +0200, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> > On Tue, May 23, 2017 at 04:52:47PM +0200, Markus Armbruster wrote:
[...]
> >> > +ERROR_WHITELIST = [
> >> > + # Machines that won't work out of the box:
> >> > + # MACHINE | ERROR MESSAGE
> >> > + dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
> >>
> >> What's wrong with dictionary displays?
> >>
> >> {'expected': True, 'machine': 'niagara'}
> >
> > Nothing wrong. I just prefer the dict(key=value) syntax to the
> > colons and extra quotes in {'key': value}.
>
> Since Python provides syntactic sugar ("dictionary displays")
> specifically for creating dictionaries, shouldn't we use it?
>
> As far as I can tell, existing Python code always uses dictionary
> displays, never the equivalent constructor call.
Well, if dict literals is less surprising, I can change it.
I will squash this in:
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 3b089099ce..5f90e9bb54 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -65,175 +65,175 @@ dbg = logger.debug
ERROR_WHITELIST = [
# Machines that won't work out of the box:
# MACHINE | ERROR MESSAGE
- dict(machine='niagara', expected=True), # Unable to load a firmware for -M niagara
- dict(machine='boston', expected=True), # Please provide either a -kernel or -bios argument
- dict(machine='leon3_generic', expected=True), # Can't read bios image (null)
+ {'machine':'niagara', 'expected':True}, # Unable to load a firmware for -M niagara
+ {'machine':'boston', 'expected':True}, # Please provide either a -kernel or -bios argument
+ {'machine':'leon3_generic', 'expected':True}, # Can't read bios image (null)
# devices that don't work out of the box because they require extra options to "-device DEV":
# DEVICE | ERROR MESSAGE
- dict(device='.*-(i386|x86_64)-cpu', expected=True), # CPU socket-id is not set
- dict(device='ARM,bitband-memory', expected=True), # source-memory property not set
- dict(device='arm.cortex-a9-global-timer', expected=True), # a9_gtimer_realize: num-cpu must be between 1 and 4
- dict(device='arm_mptimer', expected=True), # num-cpu must be between 1 and 4
- dict(device='armv7m', expected=True), # memory property was not set
- dict(device='aspeed.scu', expected=True), # Unknown silicon revision: 0x0
- dict(device='aspeed.sdmc', expected=True), # Unknown silicon revision: 0x0
- dict(device='bcm2835-dma', expected=True), # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
- dict(device='bcm2835-fb', expected=True), # bcm2835_fb_realize: required vcram-base property not set
- dict(device='bcm2835-mbox', expected=True), # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
- dict(device='bcm2835-peripherals', expected=True), # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
- dict(device='bcm2835-property', expected=True), # bcm2835_property_realize: required fb link not found: Property '.fb' not found
- dict(device='bcm2835_gpio', expected=True), # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
- dict(device='bcm2836', expected=True), # bcm2836_realize: required ram link not found: Property '.ram' not found
- dict(device='cfi.pflash01', expected=True), # attribute "sector-length" not specified or zero.
- dict(device='cfi.pflash02', expected=True), # attribute "sector-length" not specified or zero.
- dict(device='icp', expected=True), # icp_realize: required link 'xics' not found: Property '.xics' not found
- dict(device='ics', expected=True), # ics_base_realize: required link 'xics' not found: Property '.xics' not found
+ {'device':'.*-(i386|x86_64)-cpu', 'expected':True}, # CPU socket-id is not set
+ {'device':'ARM,bitband-memory', 'expected':True}, # source-memory property not set
+ {'device':'arm.cortex-a9-global-timer', 'expected':True}, # a9_gtimer_realize: num-cpu must be between 1 and 4
+ {'device':'arm_mptimer', 'expected':True}, # num-cpu must be between 1 and 4
+ {'device':'armv7m', 'expected':True}, # memory property was not set
+ {'device':'aspeed.scu', 'expected':True}, # Unknown silicon revision: 0x0
+ {'device':'aspeed.sdmc', 'expected':True}, # Unknown silicon revision: 0x0
+ {'device':'bcm2835-dma', 'expected':True}, # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
+ {'device':'bcm2835-fb', 'expected':True}, # bcm2835_fb_realize: required vcram-base property not set
+ {'device':'bcm2835-mbox', 'expected':True}, # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
+ {'device':'bcm2835-peripherals', 'expected':True}, # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
+ {'device':'bcm2835-property', 'expected':True}, # bcm2835_property_realize: required fb link not found: Property '.fb' not found
+ {'device':'bcm2835_gpio', 'expected':True}, # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
+ {'device':'bcm2836', 'expected':True}, # bcm2836_realize: required ram link not found: Property '.ram' not found
+ {'device':'cfi.pflash01', 'expected':True}, # attribute "sector-length" not specified or zero.
+ {'device':'cfi.pflash02', 'expected':True}, # attribute "sector-length" not specified or zero.
+ {'device':'icp', 'expected':True}, # icp_realize: required link 'xics' not found: Property '.xics' not found
+ {'device':'ics', 'expected':True}, # ics_base_realize: required link 'xics' not found: Property '.xics' not found
# "-device ide-cd" does work on more recent QEMU versions, so it doesn't have expected=True
- dict(device='ide-cd'), # No drive specified
- dict(device='ide-drive', expected=True), # No drive specified
- dict(device='ide-hd', expected=True), # No drive specified
- dict(device='ipmi-bmc-extern', expected=True), # IPMI external bmc requires chardev attribute
- dict(device='isa-debugcon', expected=True), # Can't create serial device, empty char device
- dict(device='isa-ipmi-bt', expected=True), # IPMI device requires a bmc attribute to be set
- dict(device='isa-ipmi-kcs', expected=True), # IPMI device requires a bmc attribute to be set
- dict(device='isa-parallel', expected=True), # Can't create serial device, empty char device
- dict(device='isa-serial', expected=True), # Can't create serial device, empty char device
- dict(device='ivshmem', expected=True), # You must specify either 'shm' or 'chardev'
- dict(device='ivshmem-doorbell', expected=True), # You must specify a 'chardev'
- dict(device='ivshmem-plain', expected=True), # You must specify a 'memdev'
- dict(device='kvm-pci-assign', expected=True), # no host device specified
- dict(device='loader', expected=True), # please include valid arguments
- dict(device='nand', expected=True), # Unsupported NAND block size 0x1
- dict(device='nvdimm', expected=True), # 'memdev' property is not set
- dict(device='nvme', expected=True), # Device initialization failed
- dict(device='pc-dimm', expected=True), # 'memdev' property is not set
- dict(device='pci-bridge', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
- dict(device='pci-bridge-seat', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
- dict(device='pci-serial', expected=True), # Can't create serial device, empty char device
- dict(device='pci-serial-2x', expected=True), # Can't create serial device, empty char device
- dict(device='pci-serial-4x', expected=True), # Can't create serial device, empty char device
- dict(device='pxa2xx-dma', expected=True), # channels value invalid
- dict(device='pxb', expected=True), # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
- dict(device='scsi-block', expected=True), # drive property not set
- dict(device='scsi-disk', expected=True), # drive property not set
- dict(device='scsi-generic', expected=True), # drive property not set
- dict(device='scsi-hd', expected=True), # drive property not set
- dict(device='spapr-pci-host-bridge', expected=True), # BUID not specified for PHB
- dict(device='spapr-pci-vfio-host-bridge', expected=True), # BUID not specified for PHB
- dict(device='spapr-rng', expected=True), # spapr-rng needs an RNG backend!
- dict(device='spapr-vty', expected=True), # chardev property not set
- dict(device='tpm-tis', expected=True), # tpm_tis: backend driver with id (null) could not be found
- dict(device='unimplemented-device', expected=True), # property 'size' not specified or zero
- dict(device='usb-braille', expected=True), # Property chardev is required
- dict(device='usb-mtp', expected=True), # x-root property must be configured
- dict(device='usb-redir', expected=True), # Parameter 'chardev' is missing
- dict(device='usb-serial', expected=True), # Property chardev is required
- dict(device='usb-storage', expected=True), # drive property not set
- dict(device='vfio-amd-xgbe', expected=True), # -device vfio-amd-xgbe: vfio error: wrong host device name
- dict(device='vfio-calxeda-xgmac', expected=True), # -device vfio-calxeda-xgmac: vfio error: wrong host device name
- dict(device='vfio-pci', expected=True), # No provided host device
- dict(device='vfio-pci-igd-lpc-bridge', expected=True), # VFIO dummy ISA/LPC bridge must have address 1f.0
- dict(device='vhost-scsi.*', expected=True), # vhost-scsi: missing wwpn
- dict(device='vhost-vsock-device', expected=True), # guest-cid property must be greater than 2
- dict(device='vhost-vsock-pci', expected=True), # guest-cid property must be greater than 2
- dict(device='virtio-9p-ccw', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
- dict(device='virtio-9p-device', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
- dict(device='virtio-9p-pci', expected=True), # 9pfs device couldn't find fsdev with the id = NULL
- dict(device='virtio-blk-ccw', expected=True), # drive property not set
- dict(device='virtio-blk-device', expected=True), # drive property not set
- dict(device='virtio-blk-device', expected=True), # drive property not set
- dict(device='virtio-blk-pci', expected=True), # drive property not set
- dict(device='virtio-crypto-ccw', expected=True), # 'cryptodev' parameter expects a valid object
- dict(device='virtio-crypto-device', expected=True), # 'cryptodev' parameter expects a valid object
- dict(device='virtio-crypto-pci', expected=True), # 'cryptodev' parameter expects a valid object
- dict(device='virtio-input-host-device', expected=True), # evdev property is required
- dict(device='virtio-input-host-pci', expected=True), # evdev property is required
- dict(device='xen-pvdevice', expected=True), # Device ID invalid, it must always be supplied
- dict(device='vhost-vsock-ccw', expected=True), # guest-cid property must be greater than 2
- dict(device='ALTR.timer', expected=True), # "clock-frequency" property must be provided
- dict(device='zpci', expected=True), # target must be defined
- dict(device='pnv-(occ|icp|lpc)', expected=True), # required link 'xics' not found: Property '.xics' not found
- dict(device='powernv-cpu-.*', expected=True), # pnv_core_realize: required link 'xics' not found: Property '.xics' not found
+ {'device':'ide-cd'}, # No drive specified
+ {'device':'ide-drive', 'expected':True}, # No drive specified
+ {'device':'ide-hd', 'expected':True}, # No drive specified
+ {'device':'ipmi-bmc-extern', 'expected':True}, # IPMI external bmc requires chardev attribute
+ {'device':'isa-debugcon', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'isa-ipmi-bt', 'expected':True}, # IPMI device requires a bmc attribute to be set
+ {'device':'isa-ipmi-kcs', 'expected':True}, # IPMI device requires a bmc attribute to be set
+ {'device':'isa-parallel', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'isa-serial', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'ivshmem', 'expected':True}, # You must specify either 'shm' or 'chardev'
+ {'device':'ivshmem-doorbell', 'expected':True}, # You must specify a 'chardev'
+ {'device':'ivshmem-plain', 'expected':True}, # You must specify a 'memdev'
+ {'device':'kvm-pci-assign', 'expected':True}, # no host device specified
+ {'device':'loader', 'expected':True}, # please include valid arguments
+ {'device':'nand', 'expected':True}, # Unsupported NAND block size 0x1
+ {'device':'nvdimm', 'expected':True}, # 'memdev' property is not set
+ {'device':'nvme', 'expected':True}, # Device initialization failed
+ {'device':'pc-dimm', 'expected':True}, # 'memdev' property is not set
+ {'device':'pci-bridge', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ {'device':'pci-bridge-seat', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ {'device':'pci-serial', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'pci-serial-2x', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'pci-serial-4x', 'expected':True}, # Can't create serial device, empty char device
+ {'device':'pxa2xx-dma', 'expected':True}, # channels value invalid
+ {'device':'pxb', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ {'device':'scsi-block', 'expected':True}, # drive property not set
+ {'device':'scsi-disk', 'expected':True}, # drive property not set
+ {'device':'scsi-generic', 'expected':True}, # drive property not set
+ {'device':'scsi-hd', 'expected':True}, # drive property not set
+ {'device':'spapr-pci-host-bridge', 'expected':True}, # BUID not specified for PHB
+ {'device':'spapr-pci-vfio-host-bridge', 'expected':True}, # BUID not specified for PHB
+ {'device':'spapr-rng', 'expected':True}, # spapr-rng needs an RNG backend!
+ {'device':'spapr-vty', 'expected':True}, # chardev property not set
+ {'device':'tpm-tis', 'expected':True}, # tpm_tis: backend driver with id (null) could not be found
+ {'device':'unimplemented-device', 'expected':True}, # property 'size' not specified or zero
+ {'device':'usb-braille', 'expected':True}, # Property chardev is required
+ {'device':'usb-mtp', 'expected':True}, # x-root property must be configured
+ {'device':'usb-redir', 'expected':True}, # Parameter 'chardev' is missing
+ {'device':'usb-serial', 'expected':True}, # Property chardev is required
+ {'device':'usb-storage', 'expected':True}, # drive property not set
+ {'device':'vfio-amd-xgbe', 'expected':True}, # -device vfio-amd-xgbe: vfio error: wrong host device name
+ {'device':'vfio-calxeda-xgmac', 'expected':True}, # -device vfio-calxeda-xgmac: vfio error: wrong host device name
+ {'device':'vfio-pci', 'expected':True}, # No provided host device
+ {'device':'vfio-pci-igd-lpc-bridge', 'expected':True}, # VFIO dummy ISA/LPC bridge must have address 1f.0
+ {'device':'vhost-scsi.*', 'expected':True}, # vhost-scsi: missing wwpn
+ {'device':'vhost-vsock-device', 'expected':True}, # guest-cid property must be greater than 2
+ {'device':'vhost-vsock-pci', 'expected':True}, # guest-cid property must be greater than 2
+ {'device':'virtio-9p-ccw', 'expected':True}, # 9pfs device couldn't find fsdev with the id = NULL
+ {'device':'virtio-9p-device', 'expected':True}, # 9pfs device couldn't find fsdev with the id = NULL
+ {'device':'virtio-9p-pci', 'expected':True}, # 9pfs device couldn't find fsdev with the id = NULL
+ {'device':'virtio-blk-ccw', 'expected':True}, # drive property not set
+ {'device':'virtio-blk-device', 'expected':True}, # drive property not set
+ {'device':'virtio-blk-device', 'expected':True}, # drive property not set
+ {'device':'virtio-blk-pci', 'expected':True}, # drive property not set
+ {'device':'virtio-crypto-ccw', 'expected':True}, # 'cryptodev' parameter expects a valid object
+ {'device':'virtio-crypto-device', 'expected':True}, # 'cryptodev' parameter expects a valid object
+ {'device':'virtio-crypto-pci', 'expected':True}, # 'cryptodev' parameter expects a valid object
+ {'device':'virtio-input-host-device', 'expected':True}, # evdev property is required
+ {'device':'virtio-input-host-pci', 'expected':True}, # evdev property is required
+ {'device':'xen-pvdevice', 'expected':True}, # Device ID invalid, it must always be supplied
+ {'device':'vhost-vsock-ccw', 'expected':True}, # guest-cid property must be greater than 2
+ {'device':'ALTR.timer', 'expected':True}, # "clock-frequency" property must be provided
+ {'device':'zpci', 'expected':True}, # target must be defined
+ {'device':'pnv-(occ|icp|lpc)', 'expected':True}, # required link 'xics' not found: Property '.xics' not found
+ {'device':'powernv-cpu-.*', 'expected':True}, # pnv_core_realize: required link 'xics' not found: Property '.xics' not found
# ioapic devices are already created by pc and will fail:
- dict(machine='q35|pc.*', device='kvm-ioapic', expected=True), # Only 1 ioapics allowed
- dict(machine='q35|pc.*', device='ioapic', expected=True), # Only 1 ioapics allowed
+ {'machine':'q35|pc.*', 'device':'kvm-ioapic', 'expected':True}, # Only 1 ioapics allowed
+ {'machine':'q35|pc.*', 'device':'ioapic', 'expected':True}, # Only 1 ioapics allowed
# KVM-specific devices shouldn't be tried without accel=kvm:
- dict(accel='(?!kvm).*', device='kvmclock', expected=True),
- dict(accel='(?!kvm).*', device='kvm-pci-assign', expected=True),
+ {'accel':'(?!kvm).*', 'device':'kvmclock', 'expected':True},
+ {'accel':'(?!kvm).*', 'device':'kvm-pci-assign', 'expected':True},
# xen-specific machines and devices:
- dict(accel='(?!xen).*', machine='xen.*', expected=True),
- dict(accel='(?!xen).*', device='xen-.*', expected=True),
+ {'accel':'(?!xen).*', 'machine':'xen.*', 'expected':True},
+ {'accel':'(?!xen).*', 'device':'xen-.*', 'expected':True},
# this fails on some machine-types, but not all, so they don't have expected=True:
- dict(device='vmgenid'), # vmgenid requires DMA write support in fw_cfg, which this machine type does not provide
+ {'device':'vmgenid'}, # vmgenid requires DMA write support in fw_cfg, which this machine type does not provide
# Silence INFO messages for errors that are common on multiple
# devices/machines:
- dict(log=r"No '[\w-]+' bus found for device '[\w-]+'"),
- dict(log=r"images* must be given with the 'pflash' parameter"),
- dict(log=r"(Guest|ROM|Flash|Kernel) image must be specified"),
- dict(log=r"[cC]ould not load [\w ]+ (BIOS|bios) '[\w-]+\.bin'"),
- dict(log=r"Couldn't find rom image '[\w-]+\.bin'"),
- dict(log=r"speed mismatch trying to attach usb device"),
- dict(log=r"Can't create a second ISA bus"),
- dict(log=r"duplicate fw_cfg file name"),
+ {'log':r"No '[\w-]+' bus found for device '[\w-]+'"},
+ {'log':r"images* must be given with the 'pflash' parameter"},
+ {'log':r"(Guest|ROM|Flash|Kernel) image must be specified"},
+ {'log':r"[cC]ould not load [\w ]+ (BIOS|bios) '[\w-]+\.bin'"},
+ {'log':r"Couldn't find rom image '[\w-]+\.bin'"},
+ {'log':r"speed mismatch trying to attach usb device"},
+ {'log':r"Can't create a second ISA bus"},
+ {'log':r"duplicate fw_cfg file name"},
# sysbus-related error messages: most machines reject most dynamic sysbus devices:
- dict(log=r"Option '-device [\w.,-]+' cannot be handled by this machine"),
- dict(log=r"Device [\w.,-]+ is not supported by this machine yet"),
- dict(log=r"Device [\w.,-]+ can not be dynamically instantiated"),
- dict(log=r"Platform Bus: Can not fit MMIO region of size "),
+ {'log':r"Option '-device [\w.,-]+' cannot be handled by this machine"},
+ {'log':r"Device [\w.,-]+ is not supported by this machine yet"},
+ {'log':r"Device [\w.,-]+ can not be dynamically instantiated"},
+ {'log':r"Platform Bus: Can not fit MMIO region of size "},
# other more specific errors we will ignore:
- dict(device='allwinner-a10', log="Unsupported NIC model:"),
- dict(device='.*-spapr-cpu-core', log=r"CPU core type should be"),
- dict(log=r"MSI(-X)? is not supported by interrupt controller"),
- dict(log=r"pxb-pcie? devices cannot reside on a PCIe? bus"),
- dict(log=r"Ignoring smp_cpus value"),
- dict(log=r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"),
- dict(log=r"This CPU requires a smaller page size than the system is using"),
- dict(log=r"MSI-X support is mandatory in the S390 architecture"),
- dict(log=r"rom check and register reset failed"),
- dict(log=r"Unable to initialize GIC, CPUState for CPU#0 not valid"),
- dict(log=r"Multiple VT220 operator consoles are not supported"),
- dict(log=r"core 0 already populated"),
- dict(log=r"could not find stage1 bootloader"),
+ {'device':'allwinner-a10', 'log':"Unsupported NIC model:"},
+ {'device':'.*-spapr-cpu-core', 'log':r"CPU core type should be"},
+ {'log':r"MSI(-X)? is not supported by interrupt controller"},
+ {'log':r"pxb-pcie? devices cannot reside on a PCIe? bus"},
+ {'log':r"Ignoring smp_cpus value"},
+ {'log':r"sd_init failed: Drive 'sd0' is already in use because it has been automatically connected to another device"},
+ {'log':r"This CPU requires a smaller page size than the system is using"},
+ {'log':r"MSI-X support is mandatory in the S390 architecture"},
+ {'log':r"rom check and register reset failed"},
+ {'log':r"Unable to initialize GIC, CPUState for CPU#0 not valid"},
+ {'log':r"Multiple VT220 operator consoles are not supported"},
+ {'log':r"core 0 already populated"},
+ {'log':r"could not find stage1 bootloader"},
# other exitcode=1 failures not listed above will just generate INFO messages:
- dict(exitcode=1, loglevel=logging.INFO),
+ {'exitcode':1, 'loglevel':logging.INFO},
# KNOWN CRASHES:
# Known crashes will generate error messages, but won't be fatal.
# Those entries must be removed once we fix the crashes.
- dict(exitcode=-6, log=r"Device 'serial0' is in use", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"qemu_net_client_setup: Assertion `!peer->peer' failed", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r'RAMBlock "[\w.-]+" already registered', loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"find_ram_offset: Assertion `size != 0' failed.", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"add_cpreg_to_hashtable: code should not be reached", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"qemu_alloc_display: Assertion `surface->image != NULL' failed", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"Unexpected error in error_set_from_qdev_prop_error", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"Object .* is not an instance of type spapr-machine", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"Object .* is not an instance of type generic-pc-machine", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"Object .* is not an instance of type e500-ccsr", loglevel=logging.ERROR),
- dict(exitcode=-6, log=r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", loglevel=logging.ERROR),
- dict(exitcode=-11, device='stm32f205-soc', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='xlnx,zynqmp', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='mips-cps', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='gus', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='a9mpcore_priv', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='a15mpcore_priv', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='isa-serial', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='sb16', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='cs4231a', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, device='arm-gicv3', loglevel=logging.ERROR, expected=True),
- dict(exitcode=-11, machine='isapc', device='.*-iommu', loglevel=logging.ERROR, expected=True),
+ {'exitcode':-6, 'log':r"Device 'serial0' is in use", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"spapr_rtas_register: Assertion .*rtas_table\[token\]\.name.* failed", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"qemu_net_client_setup: Assertion `!peer->peer' failed", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r'RAMBlock "[\w.-]+" already registered', 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"find_ram_offset: Assertion `size != 0' failed.", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"puv3_load_kernel: Assertion `kernel_filename != NULL' failed", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"add_cpreg_to_hashtable: code should not be reached", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"qemu_alloc_display: Assertion `surface->image != NULL' failed", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"Unexpected error in error_set_from_qdev_prop_error", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"Object .* is not an instance of type spapr-machine", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"Object .* is not an instance of type generic-pc-machine", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"Object .* is not an instance of type e500-ccsr", 'loglevel':logging.ERROR},
+ {'exitcode':-6, 'log':r"vmstate_register_with_alias_id: Assertion `!se->compat || se->instance_id == 0' failed", 'loglevel':logging.ERROR},
+ {'exitcode':-11, 'device':'stm32f205-soc', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'xlnx,zynqmp', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'mips-cps', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'gus', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'a9mpcore_priv', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'a15mpcore_priv', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'isa-serial', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'sb16', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'cs4231a', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'device':'arm-gicv3', 'loglevel':logging.ERROR, 'expected':True},
+ {'exitcode':-11, 'machine':'isapc', 'device':'.*-iommu', 'loglevel':logging.ERROR, 'expected':True},
# everything else (including SIGABRT and SIGSEGV) will be a fatal error:
- dict(exitcode=None, fatal=True, loglevel=logging.FATAL),
+ {'exitcode':None, 'fatal':True, 'loglevel':logging.FATAL},
]
@@ -423,11 +423,11 @@ def checkOneCase(args, testcase):
log = vm.get_log()
if exc_traceback is not None or ec != 0:
- return dict(exc_traceback=exc_traceback,
- exitcode=ec,
- log=log,
- testcase=testcase,
- cmdline=cmdline)
+ return {'exc_traceback':exc_traceback,
+ 'exitcode':ec,
+ 'log':log,
+ 'testcase':testcase,
+ 'cmdline':cmdline}
def binariesToTest(args, testcase):
--
Eduardo
^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-05-31 17:42 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-13 3:33 [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 1/3] qemu.py: Don't set _popen=None on error/shutdown Eduardo Habkost
2017-05-23 14:23 ` Markus Armbruster
2017-05-25 18:04 ` Eduardo Habkost
2017-05-29 11:40 ` Markus Armbruster
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 2/3] qemu.py: Add QEMUMachine.exitcode() method Eduardo Habkost
2017-05-13 3:33 ` [Qemu-devel] [PATCH v2 3/3] scripts: Test script to look for -device crashes Eduardo Habkost
2017-05-23 14:52 ` Markus Armbruster
2017-05-24 16:39 ` Eduardo Habkost
2017-05-29 9:25 ` Markus Armbruster
2017-05-31 17:41 ` Eduardo Habkost
2017-05-13 4:05 ` [Qemu-devel] [PATCH v2 0/3] script for crash-testing -device no-reply
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.